Edmundo-Pavel Soriano-Morales
email: edmundo.soriano-morales@univ-lyon2.fr

Julien Ah-Pine
email: julien.ah-pine@univ-lyon2.fr

Sabine Loudcher
email: sabine.loudcher@univ-lyon2.fr

Fusion Techniques for Named Entity Recognition and Word Sense Induction and Disambiguation

In this paper we explore the use of well-known multimodal fusion techniques to solve two prominent Natural Language Processing tasks. Specifically, we focus on solving Named Entity Recognition and Word Sense Induction and Disambiguation by applying feature-combination methods that have already shown their efficiency in the multimedia analysis domain. We present a series of experiments employing fusion techniques in order to combine textual linguistic features. Our intuition is that by combining different types of features we may find semantic relatedness among words at different levels and thus, the combination (and recombination) of these levels may yield gains in terms of metrics' performance. To our knowledge, employing these techniques has not been studied for the tasks we address in this paper. We test the proposed fusion techniques on three datasets for named entity recognition and one for word sense disambiguation and induction. Our results show that the combination of textual features indeed improves the performance compared to single feature representation and the trivial feature concatenation.

Introduction

Named Entity Recognition (NER) and Word Sense Induction and Disambiguation (WSI/ WSD) requires textual features to represent the similarities between words in order to discern between different words' meanings. NER goal is to automatically discover, within a text, mentions that belong to a well-defined semantic category. The classic task of NER involves detecting entities of type Location, Organization, Person and Miscellaneous. The task is of great importance for more complex NLP systems, e.g, relation extraction, opinion mining. Common solutions to NER consist on one of the following: via matching patterns created manually or extracted semi-automatically; or by training a supervised machine learning algorithm with large quantities of annotated text. The latter being the currently more popular solution to this task.

Word Sense Induction and Disambiguation involves two closely related tasks 1 . WSI aims to automatically discover the set of possible senses for a target word given a text corpus containing several occurrences of said target word. Meanwhile, WSD takes a set of possible senses and determines the most appropriate sense for each instance of the target word according to the instance's context. WSI is usually approached as an unsupervised learning task, i.e., a cluster method is applied to the words occurring in the instances of a target word. The groups found are interpreted as the senses of the target word. The WSD task is usually solved with knowledge-based approaches, based on WordNet; or more recently with supervised models which require annotated data.

As stated before, both tasks rely on features extracted from text. Usually, these representations are obtained from the surrounding context of the words in the input corpus. Mainly two types of representations are used. According to their nature we call these features lexical and syntactical. The first type requires no extra information than that contained already in the analyzed text itself. It consists merely on the tokens surrounding a word, i.e., those tokens that come before and after within a fixed window. The second type, syntactical features, is similar to the lexical representation in that we also consider as features the tokens that appear next to the corpus' words. Nonetheless, it requires a deeper degree of language understanding. In particular, these features are based on part of speech tags, phrase constituents information, and syntactical functionality between words, portrayed by syntactical dependencies. Likewise, specific features, particular to a task are also employed. These features later on become standard features in the literature.

Most of the approaches in the literature dealing with these tasks use each of these features independently or stacked together, i.e., different feature columns in an input representation space matrix. In the latter case, features are usually combined without regards to their nature.

The main intuition of the present work is that word similarities may be found at different levels according to the type of features employed. In order to exploit these similarities, we look into multimedia fusion methods. In order to better perform an analysis task, these techniques combine multimodal representations, their corresponding similarities, or the decisions coming from models fitted with these features. In this paper, we try to mutually complement independent representations by utilizing said fusion techniques to combine (or fuse) features in the hope of improving the performance of the tasks at hand, specially compared to the use of features independently.

Fusion techniques have previously shown their efficiency, mainly on text and image related tasks, where there is a need to model the relation between images and text extracts. Here, in order to apply multimedia fusion techniques, we consider textual features as different modalities, i.e., instead of having textual and image features we have lexical and syntactical features. The main contribution of this work is to assess the effectiveness of simple yet untested techniques to combine classical and easy to obtain textual features. As a second contribution, we propose a series of feature combination and recombination to attain better results. We test our intuitions on both NER and WSI/WSD tasks and over four different corpora: CoNLL-2003 [START_REF] Sang | Introduction to the conll-2003 shared task: Languageindependent named entity recognition[END_REF], WikiNER and Wikigold [START_REF] Balasuriya | Named entity recognition in wikipedia[END_REF] for NER; Semeval-2007 [START_REF] Agirre | Semeval-2007 task 02: Evaluating word sense induction and discrimination systems[END_REF] for WSI/WSD.

The rest of the paper is organized as follows: in Section 2, we go into further details about fusion techniques. We introduce the fusion operators that we use in our experiments in Section 3. Then, in Section 4 we show the effectiveness of the presented methods by testing them on NER and WSI/WSD and their respective datasets. Finally, in Section 5 we present our conclusions and future directions to explore.

In this section, we describe the fusion techniques we use in our methodology as well as relevant use-cases where they have been employed.

Multimodal Fusion Techniques

Multimodal fusion is a set of popular techniques used in multimedia analysis tasks. These methods integrate multiple media features, the affinities among these attributes or the decisions obtained from systems trained with said features, to obtain rich insights about the data being used and thus to solve a given analysis task [START_REF] Ah-Pine | Unsupervised visual and textual information fusion in CBMIR using graph-based methods[END_REF][START_REF] Atrey | Multimodal fusion for multimedia analysis: a survey[END_REF]. We note that these techniques come at the price of augmenting the training time of a system by increasing both the dimension space and/or the density of a given feature matrix.

In the multimodal fusion literature we can discern two main common types of techniques: early fusion and late fusion.

Early Fusion This technique is the most widely used fusion method. The principle is simple: we take both modal features and concatenate them into a single representation matrix. More formally, we consider two matrices that represent different modality features each over the same set of individuals. To perform early fusion we concatenate them column-wise, such that we form a new matrix having the same number of lines but increasing the number of columns to the sum of the number of columns of both matrices. The matrices may also be weighted as to control the influence of each modality.

The main advantage of early fusion is that a single unique model is fitted while leveraging the correlations among the concatenated features. The method is also easy to integrate into an analysis system. The main drawback is that we increase the representation space and may make it harder to fit models over it.

Late Fusion

In contrast to early fusion, in late fusion the combination of multimodal features are generally performed at the decision level, i.e., using the output of independent models trained each with an unique set of features [START_REF] Clinchant | Semantic combination of textual and visual information in multimedia retrieval[END_REF]. In this setting, decisions produced by each model are combined into a single final result set. The methods used to combine preliminary decisions usually involve one of two types: rule-based (where modalities are combined according to domain-specific knowledge) or linear fusion (e.g., weighting and then adding or multiplying both matrices together). This type of fusion is very close to the so-called ensemble methods in the machine learning literature. Late fusion combines both modalities in the same semantic space. In that sense, we may also combine modalities via an affinity representation instead of final decision sets. In other words, we can combine two modality matrices by means of their respective similarities. A final representation is then usually obtained by adding the weighted similarity matrices.

The advantages of late fusion include the combination of features at the same level of representation (either the fusion of decisions or similarity matrices). Also, given that independent models are trained separately, we can chose which algorithm is more adequate for each type of features.

Cross-media Similarity Fusion A third type of fusion technique, cross-media similarity fusion (or simply cross fusion), introduced in [START_REF] Ah-Pine | Unsupervised visual and textual information fusion in CBMIR using graph-based methods[END_REF][START_REF] Clinchant | Semantic combination of textual and visual information in multimedia retrieval[END_REF], is defined and employed to propagate a single similarity matrix into a second similarity matrix. In their paper, the authors propagated information from textual media towards visual media. In our case, we transfer information among textual features. For example, to perform a cross fusion between lexical and syntactical features, we perform the following steps:

1. Compute the corresponding similarity matrices for each type of feature. 2. Select only the k-nearest neighbor for each word within the lexical similarity matrix. These neighbors are to be used as lexical representatives to enrich the syntactical similarities. 3. Linearly combine both similarity matrices (lexical k-nearest lexical neighbors with the syntactical features) via a matrix product.

Cross fusion aims to bridge the semantic gap between two modalities by using the most similar neighbors as proxies to transfer valuable information from one modality onto another one. Usually, the result of a cross fusion is combined with the previous techniques, early and late fusion. In this paper we perform experiment in that sense.

Hybrid Fusion

We may leverage the advantages of the previous two types of fusion techniques by combining them once more in a hybrid setting. As described in [START_REF] Atrey | Multimodal fusion for multimedia analysis: a survey[END_REF][START_REF] Yu | Informedia@ trecvid 2014 med and mer[END_REF], the main idea is to simultaneously combine features at the feature level, i.e., early fusion, and at the same semantic space or decision level. Nonetheless, they define a specific type of hybrid fusion. In this paper, we adopt a looser definition of hybrid fusion. That is, we perform hybrid fusion by leveraging the combination of the fusion strategies described before.

We consider the first three types of fusion techniques (early fusion, late fusion and cross fusion) as the building blocks to the experiments we conduct. While we work with a single modality, i.e., textual data, we consider the different kinds of features extracted from it as distinct modalities. Our intuition being that the semantic similarities among words in these different spaces can be combined in order to exploit the latent complementarity between the lexical and syntactical representations. The fusion should therefore improve the performance of the NLP tasks at hand, NER and WSI/WSD.

Our first goal is to assess the effectiveness of the classic fusion methods and then, as a second goal, to propose new combinations that yield better outcomes in terms of performance than the simpler approaches. The new combinations are found empirically. Nonetheless, as we will show, their effectiveness replicates across different datasets and NLP tasks.

NER and WSI/WSD

To the best of our knowledge, there is no work that addresses both NER and WSI/WSD explicitly while using fusion techniques from the multimedia analysis domain. Still, we base our experiments on those carried on in [START_REF] Collins | Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms[END_REF][START_REF] Daume | Practical Structured Learning Techniques for Natural Language Processing[END_REF][START_REF] Goyal | Unsupervised word sense induction using distributional statistics[END_REF] using well-known supervised (structured perceptron) and unsupervised (spectral clustering) learning algorithms. A thorough review on NER and WSI/WSD can be found in [START_REF] Nadeau | A survey of named entity recognition and classification[END_REF] and [START_REF] Navigli | Word sense disambiguation: A survey[END_REF], respectively.

Input Corpus Preprocessing Fusion Matrix Generation IF 1F 2F NF M LEX M SYN M STD Fusion Matrix Learning Model Results Evaluation
Fig. 1. Steps followed on our experiments. First the corpus is preprocessed, then features are extracted from the text. A fusion matrix is generated, which in turn is used as input to a learning algorithm. Finally, the system yields its results and to be analyzed.

Methodology

In the present section we address the core of the work performed in this paper. We formally describe the fusion techniques we employ in the next section. Also, we delineate the procedure followed in our experiments.

The experiments we carry on consist in generating fusion matrices that will serve as input to a learning algorithm in order to solve NER and WSI/WSD. These input feature matrices are based upon lexical, syntactical, or other types of representation. The procedure can be seen in Figure 1.

Fusion Strategies

We begin by presenting a formal definition of the fusion techniques employed and described in the previous sections. We define (weighted) early fusion, late fusion and cross fusion as follows:

Early Fusion E(A, B) = hstack(A, B) (1)
Weighted Early Fusion

wE α (A, B) = hstack(α • A, (1 -α) • B) (2) Late Fusion L β (A, B) = β • A + (1 -β) • B (3)
Cross fusion

X γ (A, B) = K(A, γ) × B (4)
Parameters A and B are arbitrary input matrices. They may initially represent, for example, the lexical (M LEX ) or syntactical based (M SY N ) features matrix, or their corresponding similarity matrices, S LEX and S SY N , respectively. In a broader sense, matrices A and B may represent any pair of valid 2 fusion matrices.

In early fusion, E(A, B), the matrices A and B are combined together via a function called hstack which concatenates, column-wise, both matrices A and B. Weighted early fusion represents the same operation as before with an extra parameter: α, which controls the relative importance of each matrix. In the following, we refer to both operations as early fusion. When α is determined, we refer to weighted early fusion.

Regarding late fusion L β (A, B), the β parameter determines again the importance of the matrix A, and consequently also the relevance of matrix B.

In cross fusion X γ (A, B), the K(•) function keeps the top-γ closest words (columns) to each word (lines) while the rest of the values are set to zero.

Using the previously defined operators, we distinguish four levels of experiments:

1. Single Features: in this phase we consider the modalities independently as input to the learning methods. For instance, we may train a model for NER using only the lexical features matrix M LEX . 2. First Degree Fusion: we consider the three elementary fusion techniques by themselves (early fusion, late fusion, cross fusion) without any recombination. These experiments, as well as those from the previous level, serve as the baselines we set to surpass in order to show the efficacy of the rest of the fusion approaches. As an example, we may obtain a representation matrix by performing an early fusion between the lexical matrix and the syntactical features matrix: E(M LEX , M SY N ).

In this level we distinguish two types of cross fusion: Cross Early Fusion (XEF) and Cross Late Fusion (XLF). The first one combines a similarity matrix with a feature matrix: X(S LEX , M SY N ). The second one joins a similarity matrix with a similarity matrix: X(S SY N , S LEX ). 3. Second Degree Fusion: we recombine the outputs of the previous two levels with the elementary techniques. This procedure then yields a recombination of "seconddegree" among fusion methods. We introduce the four types of second degree fusions in the following list. Each one is illustrated with an example: (a) Cross Late Early Fusion (XLEF): X(X(S ST D , S SY N ),

M ST D ) (b) Cross Early Early Fusion (XEEF: X(S ST D , X(S ST D , S SY N )) (c) Early Cross Early Fusion (EXEF): E(M ST D , X(S LEX , M ST D )) (d) Late Cross Early Fusion (LXEF): L(M ST D , X(S ST D , M ST D )) 4.
N-Degree Fusion: in this last level we follow a similar approach to the previous level by combining the output of the second-degree fusion level multiple times (more than two times) with other second-degree fusion outputs. Again, in this level we test the following two fusion operations: (a) Early Late Cross Early Fusion (ELXEF):

E(M ST D , L(M ST D , X(S ST D , M ST D ))) (b) Early ELXEF (EELXEF): E(M LEX , E(E(M ST D , L(M ST D , X(S ST D , M ST D ))), L(M LEX , X(S SY N , M LEX ))))
2 Valid in terms of having compatible shapes while computing a matrix sum or multiplication.

Feature Matrices

In the previous subsection we presented the fusion operators used in our experiments. Below we detail the three types of features used to describe the words of each of the tested corpus.

Lexical Matrix (LEX) For each token in the corpus, we use a lexical window of two words to the left and two words to the right, plus the token itself. Specifically, for a target word w, its lexical context is (w -2 , w -2 , w, w +1 , w +2 ). This type of context features is typical for most systems studying the surroundings of a word, i.e., using a distributional approach [START_REF] Levy | Dependency-based word embeddings[END_REF].

Syntactical Matrix (SYN) Based on the syntactic features used in [START_REF] Levy | Dependency-based word embeddings[END_REF][START_REF] Panchenko | Unsupervised does not mean uninterpretable: The case for word sense induction and disambiguation[END_REF], we derive contexts based on the syntactic relations a word participates in, as well as including the part of speech (PoS) of the arguments of these relations. Formally, for a word w with modifiers m 1 , . . . , m k and their corresponding PoS tags p m 1 , . . . , p m k ; a head h and its corresponding PoS tag p h , we consider the context features (m 1 , p m1 , lbl 1 ), . . . , (m k , p m k , lbl k ), (h, p h , lbl inv h ). In this case, lbl and lbl inv indicate the label of the dependency relation and its inverse, correspondingly. Using syntactic dependencies as features should yield more specific similarities, closer to synonymy, instead of the broader topical similarity found through lexical contexts.

NER Standard Features Matrix (STD)

The features used for NER are based on those used in [START_REF] Daume | Practical Structured Learning Techniques for Natural Language Processing[END_REF][START_REF] Balasuriya | Named entity recognition in wikipedia[END_REF]. The feature set consists of: the word itself, whether the word begins with capital letter, prefix and suffix up to three characters (within a window of two words to the left and two words to the right), and the PoS tag of the current word. These features are considered to be standard in the literature. We note that the matrix generated with these features is exclusively used in the experiments regarding NER.

Learning Methods

We use supervised and unsupervised learning methods for NER and WSI/WSD respectively. On the one hand, for NER, as supervised algorithm, we use an averaged structured perceptron [START_REF] Collins | Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms[END_REF][START_REF] Daume | Practical Structured Learning Techniques for Natural Language Processing[END_REF] to determine the tags of the named entities. We considered Logistic Regression and linear SVM. We chose the perceptron because of its performance and its lower training time.

On the other hand, for WSD/WSI, specifically for the induction part, we applied spectral clustering, as in [START_REF] Goyal | Unsupervised word sense induction using distributional statistics[END_REF], on the input matrices in order to automatically discover senses (a cluster is considered a sense). Regarding disambiguation, we trivially assign senses to the target word instances according to the number of common words in each cluster and the context words of the target word. In other words, for each test instance of a target word, we select the cluster (sense) with the maximum number of shared words with the current instance context.

We experiment with four levels of fusion: Single Features (SF), First-degree Fusion (1F), Second-degree Fusion (2F) and N-degree Fusion (NF). The representation matrices for NER come from lexical context features M LEX , syntactical context features M SY N or standard features M ST D . On the other hand, experiments on WSI/WSD exclusively employ matrices M LEX and M SY N .

Our first goal is to compare the efficiency of the basic multimedia fusion techniques applied to single-modality multi-feature NLP tasks, namely NER and WSI/WSD. A second goal is to empirically determine a fusion combination setting able to leverage the complementarity of our features.

To this end, we evaluate the aforementioned 4 fusion levels. We note that the fusion combinations in the third and fourth level (2F and NF) are proposed based on the results obtained in the previous levels. In other words, in order to reduce the number of experiments, we restrict our tests to the best performing configurations. This is due to the large number of possible combinations (an argument to a fusion operation may be any valid output of a second fusion operation).

Named Entity Recognition

Pre-processing As is usual when preprocessing text before performing named entity recognition, [START_REF] Ratinov | Design challenges and misconceptions in named entity recognition[END_REF], we normalize tokens that include numbers. This allows a degree of abstraction to tokens that contain years, phone numbers, etc.

Features The linguistic information we use are extracted with the Stanford's CoreNLP parser [START_REF] Manning | The Stanford CoreNLP natural language processing toolkit[END_REF]. Again, the features used for these experiments on NER are those described before: lexical, syntactic and standard features, i.e., M LEX , M SY N , and M ST D , respectively.

Test Datasets We work with three corpora coming from two different domains:

(1) CoNLL-2003 (CONLL): This dataset was used in the language-independent named entity recognition CoNLL-2003 shared task [START_REF] Sang | Introduction to the conll-2003 shared task: Languageindependent named entity recognition[END_REF]. It contains selected news-wire articles from the Reuters Corpus. Each article is annotated manually. It is divided in three parts: training (train) and two testing sets (testa and testb). The training part contains 219,554 lines, while the test sets contain 55,044 and 50,350 lines, respectively. The task was evaluated on the testb file, as in the original task. (2) WikiNER (WNER): A more recent dataset [START_REF] Balasuriya | Named entity recognition in wikipedia[END_REF] of selected English Wikipedia articles, all of them annotated automatically with the author's semi-supervised method. In total, it contains 3,656,439 words. (3) Wikigold (WGLD): Also a corpus of Wikipedia articles, from the same authors of the previous corpus. Nonetheless, this was annotated manually. This dataset is the smaller, with 41,011 words. We used this corpus to validate human-tagged Wikipedia text. These three datasets are tagged with the same four types of entities: Location, Organization, Person and Miscellaneous.

Results

We present in this subsection the results obtained in the named entity recognition task, while employing the 4 levels of fusion proposed in the previous section.

In contrast to other related fusion works [2, 5, 9], we do not focus our analysis on the impact of the parameters of the fusion operators. Instead, we focus our analysis on the effect of the type of linguistic data being used and how, by transferring information from one feature type to another, they can be experimentally recombined to generate more complete representations.

Regarding the fusion operators' parameters, we empirically found the best configuration for β, from late fusion L β (A, B) = β • A + (1 -β) • B, is β = 0.5. This implies that an equal combination is the best linear fusion for two different types of features.

In respect of the γ parameter, used in cross fusion X γ (A, B) = K(A, γ) × B, we set γ = 5. This indicates that just few high quality similarities attain better results than utilizing a larger quantity of lower quality similarities.

Single Features Looking at Table 1, we see that the best independent features, in terms of F-measure come from the standard representation matrix M ST D . This is not surprising as these features, simple as they may be, have been used and proved extensively in the NER community. On the other hand, M LEX performs relatively well, considering it only includes information contained in the dataset itself. Nevertheless, this representation that this kind of lexical context features are the foundation of most word embedding techniques used nowadays. While we expected better results from the syntactical features M SY N , as they are able to provide not only general word similarity, but also functional, getting close to synonymy-level [START_REF] Levy | Dependency-based word embeddings[END_REF], we believe that the relatively small size of the datasets do not provide enough information to generalize First Degree Fusion In Table 2 we present the First Degree fusion level. The best performance is obtained by trivially concatenating the representation matrices. This baseline proved to be the toughest result to beat. Late fusion does not perform well in this setting, still, we see further on that by linearly combining weighted representation 3. NER F-measure results using second degree fusion (2F). In XLEF, a * corresponds to the best performing matrix in the set {X(S ST D , S LEX ), X(S LEX , S ST D ), X(S ST D , S SY N )}.

For XEEF, bXEEF = E(M LEX , M ST D ). In EXEF, b * EXEF takes the best performing matrix from {X(S SY N , M LEX ), matrices, we can add information to an already strong representation. Finally, regarding the cross fusion techniques, cross early and late fusion, we see that they depend directly on the information contained in the similarity matrices. We note that, as is the case on single features, the combinations with matrix S ST D yield almost always the best results. While these fusion techniques by themselves may not offer the best results, we see below that by recombining them with other types of fusion we can improve the general performance of a representation.

X(S LEX , M LEX ), X(S LEX , M ST D ), X(S SY N , M LEX ), X(S SY N , M ST D )}. Finally, in LXEF, bLXEF takes the best possible matrix from {X(S LEX , M ST D ), X(S SY N , M ST D ), X(S SY N , M LEX )}. A B

Second Degree Fusion

The second degree fusion techniques presented in Table 3 show that the recombination of cross fusion techniques gets us closer to the early fusion baseline. With the exception of cross late early fusion, the rest of the recombination schemes yield interesting results. First, in cross early fusion, the best results, for the most part, are obtained while using the S LEX matrix combined with the output of E(M LEX , M ST D ), which is still far from the baseline values. Concerning, EXEF, we get already close to surpass the baselines with the M ST D matrix, with the exception of the CONLL dataset.

In LXEF, even though the cross fusion X(S SY N , M LEX ) is not the best performing, we found experimentally that by combining it with M LEX through a late fusion, it gets a strong complementary representation. Our intuition in this case was to complement M LEX with itself but enriched with the S SY N information. In the N-degree fusion results we discover that indeed this propagation of information helps us beat the baselines we set before.

N-degree Fusion Finally, the last set of experiments are shown in Table 4. Using a recombination of fusion techniques, a so-called hybrid approach, we finally beat the baselines (single features and early fusion) for each dataset. We note that the best configuration made use of a weighted early fusion with α = 0.95. This indicates that the single feature matrix, M LEX is enriched a small amount by the fusion recombination, which is enough to improve the results of said baselines. In CONLL, the early fusion (see Table 2) baseline being 78.13, we reached 78.69, the lowest improvement of the three datasets. Regarding the Wikipedia corpus, in WNER, we passed from 79.78 to 81.75; and in WGLD, from 61.96 to 67.29, the largest improvement of all. It is important that we tried the weighted Early Fusion operator with different α and the best result does not beat these fusion results.

In the next section we transfer the knowledge gained in this task to a new one, word sense induction and disambiguation.

Word Sense Induction and Disambiguation

Having learned the best fusion configuration from the previous task, in this experiments we set to test if the improvements achieved can be transfered into another NLP task, namely Word Sensed Induction and Disambiguation (WSI/WSD).

Pre-processing We simply remove stopwords and tokens with less than three letters.

Features We use the same set of features from the previous task, with the exception of the standard NER features, that is, those represented by M ST D , as they are specifically designed to tackle NER.

Test Dataset

The WSI/WSD model is tested on the dataset of the Semeval-2007 WSID task [START_REF] Agirre | Semeval-2007 task 02: Evaluating word sense induction and discrimination systems[END_REF]. The task was based on a set of 100 target words (65 nouns and 35 verbs), each word having a set of instances, which are specific contexts where the word appear. Senses are induced from these contexts and applied to each one of the instances. Table 4.

F-measure results using N-degree fusion (NF). In ELXEF, bELXEF = L(M LEX , X(S SY N , M LEX )). For EELXEF, bEELXEF = E(E(M ST D , L(M LEX , X(S SY N , M LEX ))), L(M LEX , X(S ST D , M LEX ))) for CONLL and bEELXEF = E(E(M ST D , L(M ST D , X(S SY N , M ST D ))), L(M LEX , X(S SY N , M LEX )))

for WNER and WGLD. The best result is obtained in EELXEF when α = 0.95. Evaluation Measures Being an unsupervised task, the evaluation metrics of WSI/WSD are debated in terms of quality [START_REF] De Cruys | Latent semantic word sense induction and disambiguation[END_REF]. We consider supervised recall and unsupervised Fmeasure, as in the competition original paper [START_REF] Agirre | Semeval-2007 task 02: Evaluating word sense induction and discrimination systems[END_REF]. The first one maps the output of a system to the true senses of the target words' instances and the second one measures the quality of the correspondence between the automatically found clusters and the senses. We consider that the number of senses found by the system is also a rather good indicator of performance: the best competition baseline assigns the most frequent sense to each target word (this baseline is called MFS), thus this baseline system would have an average of 1 sense (cluster) per word. A system that goes near this average may be indeed not resolving the task efficiently but finding the MFS trivial solution. Consequently, to show that we do not fall in the MFS solution, we display in our results the average number of clusters.

A

Results Word sense induction and disambiguation results are found in Table 5. Again, we aim to surpass the baseline of the single features and early fusion. We experimentally set β = 0.90 and γ = 50. In this task, in late fusion, when the first matrix is deemed more relevant than the second one, the performance is higher. This may be due to the fact that, in this task, the feature matrices rows contain types (that is, each line represent an unique word), and thus they are more dense, which may entail more noisy data. By reducing the relevance of the second matrix in late fusion, we are effectively attenuating the less important information. Regarding γ = 50, again due to the denser characteristic of the matrices, there is a larger quantity of true similar words that are useful to project information into another matrix, through cross fusion.

The WSI/WSD results are shown in Table 5. In the following paragraph, we will discuss these result all at once. Due to the page limit constraint, we omit certain configurations that do not yield interesting results either by converging to the MFS solution (1 sense found per target word) or because the performance shown by those configurations is not interesting. Regarding Single Features, M LEX comes on top of M SY N again. Nonetheless, M SY N is much closer in terms of performance, and as expected, it is actually higher with regards to verbs.

On the 1F level, we see that the early fusion technique in this task does not surpass the independent features representation. Our intuition is that the similarities of both matrices seem to be correlated. In cross early fusion, the best result is obtained by X(S LEX , M LEX ), regarding the unsupervised F-measure. This configuration already beats our baselines, improving both noun and verb results on the unsupervised evaluation, improving the supervised recall of nouns, and staying on the same level considering all words. Also, it produces more senses than the MSF average number of senses (1 sense per target word), which is good but not indicative of results correctness. Regarding cross late fusion, given the average number of clusters produced, it seems that both results converge towards the MFS, therefore we do not consider these results.

Beginning with the fusion recombinations, in level 2F, both cross late early fusions yield average results. cross early cross early fusion, the early fusion of M LEX with X(S LEX , M LEX ) yields very similar results than X(S LEX , M LEX ). The next natural step is to test this fusion via a linear combination, with a late fusion. The result obtained confirmed the intuition of enriching a single feature matrix with another weighted-down matrix to improve the performance. Indeed, we consider that L(M LEX , X(S LEX , M LEX )) gets the best results in terms of all-words supervised recall and the second best all-words unsupervised F-measure (we do not consider solutions that are too close to the MFS baseline).

We test the same configurations as in NER, within the NF level, to try and improve our results. Nonetheless, in general, they do not overcome the best result found previously.

In general, we found that the recombination fusion techniques work in terms of improving the performance of the tasks addressed. In the following, we make our final remarks and the future work to be done regarding fusion techniques on NLP tasks.

Conclusion and Future Work

In this paper, we presented a comparative study of multimedia fusion techniques applied to two NLP tasks: Named Entity Recognition and Word Sense Induction and Disambiguation. We also proposed new fusion recombinations in order to complement the information contained in the single representation matrices. In order to accomplish this goal, we built upon basic fusion techniques such as early and late fusion, as well as cross media fusion to transfer quality information from one set of features to another.

We found that by taking a strong feature, in our case lexical context, M LEX , and enriching it with the output of rather complex fusion combinations, we can improve the performance of the tasks addressed. The enrichment has to give more relevance to the strong feature matrix, by selecting the right parameters.

While there is an improvement, we do note that fusion techniques augment the computing time and memory consumption of the tasks at hand by enlarging the feature space or by making it more dense. In that sense, more intelligent ways of finding the most appropriate fusion must be researched. This is indeed one of our future work paths: determining an optimal fusion path from single features to a N-degree fusion recombination. Coupled with this, the automatic determination of the parameters is still ongoing research in the multimedia fusion community. Consequently, we believe that efficiently determining both parameters and fusion combinations is the general domain of our future work. Another route we would like to explore is testing these techniques on other tasks and with datasets from different domains, in order to assert its effectiveness.

Table 1 .

 1 NER F-measure results using the Single Features over the three datasets. These values serve as a first set of baselines.

	A	Single Features
		CONLL WNER WGLD
	M ST D 77.41 77.50 59.66
	M LEX 69.40 69.17 52.34
	M SY N 32.95 28.47 25.49
	Evaluation Measures We evaluate our NER models following the standard CoNLL-
	2003 evaluation script. Given the amount of experiments we carried on, and the size
	constraints, we report exclusively the total F-measure for the four types of entities (Lo-
	cation, Organization, Person, Miscellaneous). WNER and WGLD datasets are evalu-
	ated on a 5-fold cross validation.

Table 2 .

 2 NER F-measure results using first degree fusion (1F). B is either indicated on the table or specified as follows. Looking at EF, bEF = E(M SY N , M ST D ). In XEF, b * XEF takes the matrix from the set {M LEX , M ST D } which yields the best performing result. In XLF, b * XLF corresponds to the best performing matrix in {S LEX , S SY N }. These configurations serve as the main set of baseline results.

				Table
	A	B		Early Fusion
			CONLL WNER WGLD
	M LEX M SY N 72.01	70.59 59.38
	M LEX M ST D 78.13	79.78 61.96
	M SY N M ST D 77.70	78.10 60.93
	M LEX bEF	78.90	80.04 63.20
				Late Fusion
			CONLL WNER WGLD
	S LEX S SY N	61.65	58.79 44.29
	S LEX S ST D	55.64	67.70 48.00
	S SY N S ST D	50.21	58.41 49.81
			Cross Early Fusion
			CONLL WNER WGLD
	S LEX M ST D 49.90	70.27 62.69
	S SY N M ST D 47.27	51.38 48.53
	S ST D b * XEF	52.89	62.21 50.15
			Cross Late Fusion
			CONLL WNER WGLD
	S LEX S ST D	27.75	59.12 38.35
	S SY N b * XLF	36.87	40.92 39.62
	S ST D b * XLF	41.89	52.03 39.92

Table 5 .

 5 Supervised Recall and Unsupervised F-measure for the Semeval-2007 corpus. We also display the average number of clusters found by each fusion configuration. LEX , S SY N ), M LEX ) 78.40 80.40 76.10 70.00 68.70 71.40 3.11 X(X(S LEX , S SY N ), M SY N ) 78.90 81.80 75.60 75.20 77.40 72.80 3.16 LEX , L(M SY N , X(S LEX , M SY N ))) 78.50 81.40 75.40 74.20 78.20 69.80 4.26 E(M LEX , L(M LEX , X(S LEX , M LEX ))) 79.50 82.70 75.90 75.80 78.50 72.70 3.99

	Method	Recall (%)	FM (%)	# cl
		all noun verb	all noun verb
			Single Features
	M LEX	79.20 82.10 75.80 72.70 76.90 67.90 4.13
	M SY N	79.10 81.60 76.20 69.30 69.40 69.20 4.47
			Early Fusion
	E(M LEX , M SY N )	78.70 81.11 76.10 74.00 76.66 71.11 4.46
			Cross Early Fusion
	X(S LEX , M LEX )	79.20 82.30 75.70 76.20 79.60 72.50 3.63
	X(S LEX , M SY N )	78.30 80.90 75.30 74.60 75.10 73.90 3.08
	X(S SY N , M LEX )	78.60 80.90 76.10 78.90 80.70 76.90 1.08
	X(S SY N , M SY N )	78.90 81.40 76.10 73.70 77.70 70.00 2.72
			Cross Late Fusion
	X(S SY N , S LEX )	78.70 80.90 76.20 78.90 80.80 76.80 1.01
	X(S LEX , S SY N )	78.80 80.90 76.06 78.70 80.50 76.80 1.33
			Cross Late Early Fusion
	X(X(S Early Cross Early Fusion
	E(M LEX , X(S LEX , M LEX ))	79.20 82.40 75.70 76.00 79.50 72.10 3.57
	E(M SY N , X(S LEX , M LEX ))	78.30 80.50 75.80 75.20 75.40 75.00 1.95
			Late Cross Early Fusion
	L(M SY N , X(S LEX , M SY N ))	78.60 81.10 75.80 67.80 71.40 63.80 4.22
	L(M LEX , X(S LEX , M LEX ))	79.50 82.80 75.70 76.09 79.10 72.70 3.96
		Early Late Cross Early Fusion
	E(M

Even though these tasks are related, they are independent from one another. Still, in this paper we consider them to be a single one.


Arthur Gaudron
email: arthur.gaudron@mines-paristech.fr

Simon Tamayo

Arnaud De

L A Fortelle

Interactive Simulation To Explore Urban Distribution Schemes

Keywords: city logistics, interactive simulation, decision support, multimodality

The goal of building a shared vision about city logistics is often obstructed by the diversity of its actors (authorities, carriers, shippers, etc.). As an example, urban deliveries are sometimes not taken into account in the urban planning of cities or are simply misunderstood. This reality highlights the need for accessible and straightforward decision support tools in this field. In this work an interactive simulation is proposed as a playful tool to disseminate theoretical results from academia and lessons learned towards non-experts, who can easily interact with a model. The proposed interactive simulator aims at helping actors to broadly share and enlarge their perspective about a specific problem. That is, identifying the different objectives and constraints of other actors and converging towards a collective solution to the problem at hand. A trial was performed with 28 participants that explored and compared two different distribution schemes using interactive simulation. Positive feedback from the participants shows promising perspectives for this type of tool.

Introduction

The challenges for the mobility of goods and passengers are getting bigger as cities face greater constraints. In 2050, urban population is expected to represent 80% of the total population in Europe (United Nations, 2014). At the same time, in regards to this growing need from the population, the European Commission targets to reduce emissions related to transport to more than 60% of 1990 levels by 2050 (European Commission, 2011). Aside from switching to low-carbon energy, city logistics experts point out the "poor and inappropriate" urban logistics service in large European cities [START_REF] Dablanc | Goods transport in large European cities : Difficult to organize[END_REF]. Initiatives, such as the "Chappelle International logistic hub" in Paris, show that cities are considering more urban logistics issues in their urban planning. However, there is still room for improvement: the first version of the "Métropole du Grand Paris" urbanization plan (administrative structure for cooperation covering the City of Paris and its nearest suburbs departments) did not consider freight [START_REF] Debrie | La question logistique dans l'aménagement de l'Ile-de-France : formulation d'un enjeu métropolitain versus absence de concrétisation dans les projets urbains ? Géographie[END_REF]. Benefits of raising awareness about city logistics seem obvious: planners are beginning to take into account urban logistics constraints; and voters are understanding the benefits of an efficient transportation system for freight. Although the non-expert public has a growing interest in urban logistics, there is still a significant knowledge gap between them (inhabitants, city planners) and the research community. As a result, non-experts are not always aware of the research contributions. For example, when implementing access regulation based on vehicle size, cities do not expect to see an increase in the number of vehicles. This relationship between vehicle size (i.e. capacity) and number of vehicles for a fixed demand has been largely explored in research related to vehicle routing problems. Interactive simulation could be used to disseminate this type of results in an accessible manner.

To the best knowledge of the authors, there are no interactive tools to engage non-expert public with the constraints of city logistics. It seems important to establish a formal distinction between interactive simulation and Decision Support Systems "DSS" [START_REF] Comi | CLASS: A DSS for the analysis and the simulation of urban freight systems[END_REF][START_REF] Grzybowska | Decision Support System for Real-Time Urban Freight Management[END_REF][START_REF] Perboli | Decision support system for collaborative freight transportation management : a tool for mixing traditional and green logistics[END_REF]. DSS has an operational objective [START_REF] Gabriel | Planning models for freight transportation 0[END_REF], whereas interactive simulation has a teaching objective [START_REF] Vogel | Computer Gaming and Interactive Simulations for Learning: A Meta-Analysis[END_REF].

Studies show that learners have a preference for interactive simulations compared to traditional teaching methods and cognitive gain can be higher [START_REF] Vogel | Computer Gaming and Interactive Simulations for Learning: A Meta-Analysis[END_REF]. Moreover, it has been shown that interactive simulation can be an efficient way to tackle difficult problems. For example, the Human-based computation game "phylo" is able to leverage the human capacity to solve NP-complete pattern-matching problems [START_REF] Kawrykow | Phylo: A citizen science approach for improving multiple sequence alignment[END_REF] for comparative genomics. In addition, in the context of transportation, humans are able to propose reasonably good solutions without computer assistance to the "Traveling Salesman Problem" for small instances [START_REF] Best | A model of fast human performance on a computationally hard problem[END_REF].

In this paper, an interactive simulator has been developed to raise awareness about city logistics towards nonexperts. The simulator is accessed through an interface allowing users to modify certain parameters of an urban logistics system and quickly see the impact of their choices. The main objective of this research is to assess if interactivity allows the users to deeply understand the model -by discovering its limits -and therefore the problem it represents.

This paper presents an application case that performs a comparison of an emerging and the traditional distribution scheme for urban distribution. The idea is to use interactive simulation in order to help non-experts to explore and understand these models that might seem trivial to researchers in urban freight, but that are often unknown to inhabitants. To this end, the distribution schemes are model and simulated. The relevance of the tool is discussed after a trial on a group of 28 people.

Method

Use of the interactive simulator

The proposed interactive simulator can be used following four main steps as shown in Fig. 1. In the first step, an urban logistic problem is identified by the expert. In the second step, the expert develops one or several models to represent the problem. In the third step, the interactive simulator acts as an interface between the model(s) and the users. This is the interactive part of the process where users are able to easily change parameters to explore a model, test hypothesis and get familiar with the problem. The fourth and final step consists in collecting feedback from users. The comprehension of users is assessed via a survey.

Use case

In this paper, interactive simulation is used to compare two distribution schemes for express deliveries of 200 points in the city of Paris. The interactive simulator allows users to quickly test numerous combinations of the parameters. First, the interaction with the simulation aims at teaching the relationship between parameters (e.g. capacity vs. number of vehicles). Secondly, the interactivity puts the user in a better position to criticize the model. As users explore the model, they also understand its limitations (missing parameters, wrong hypothesis, etc.). In other words, the user starts to have a better understanding of the problem as it appears in real life.

The first scheme is a traditional parcel delivery operation (cf. Fig. 2 left). Vans leave a depot in the city center to the assigned points of delivery. The second scheme is multi-modal (cf. Fig. 2 right), one truck (capacity of 200 deliveries) leaves the depot and stops several times in the city. At each stop, parcels are transshipped to bicycle couriers performing the last mile. From an optimization perspective, the first scheme is a vehicle routing problem. And the second scheme can be seen as a Two-echelon Vehicle Routing Problem [START_REF] Crainic | Two-Echelon Vehicle Routing Problem: A satellite location analysis[END_REF][START_REF] Drexl | A survey of variants and extensions of the location-routing problem[END_REF][START_REF] Gonzalez-Feliu | The two-echelon capacitated vehicle routing problem To cite this version : The Two-Echelon Capacitated Vehicle Routing Problem[END_REF]. The optimization was carried out using Gurobi 7.5 (MIP solver) and scikit-learn (Python machine learning package), all within a Python 3.5 environment. The distance between the points has been considered as Euclidian. The optimization computes the routing of the vehicles, where the objective is minimizing the total distance traveled. Only one parameter can change the routing of the vehicle: the capacity of the vans or the bikes. All the other parameters have no influence on the optimization results. For example, the fuel consumption or the cost of a vehicle will have no impact on the routing solution. Indeed, the objective is to minimize the total distance. It does not consider the minimization of the operational cost. The KPI (presented in Section 2.3) are computed from the routing solution and the parameters (presented in Section 2.5).

In order to ensure a good level of responsiveness, the interactive simulator contains a database with the resulting routing solutions for all the possible combinations of input parameters. Actually, every possible routing solution is obtained by only changing the vehicle capacity (i.e. van capacity for the VRP and bike capacity for 2EVRP). Then, the KPI are computed from the routing solutions depending on the parameters chosen by the user.

As shown in Fig. 2, the depot and the delivery points are the same for both distribution schemes. It is the actual location of a depot from the French company La Poste. The delivery points are random points in the city of Paris.

Key Performance Indicators

Three Key Performances Indicators are computed in the interactive simulation model in order to compare the two solutions on the grounds of economic, operational and environmental performance.

• The first KPI is the total cost of delivering 200 points.

• The second KPI is the average time of the delivery tours. This represents the lead-time performance of the distribution, as the tour is shorter (or faster), it is possible to deliver more during peak hours. For example, if all the customers want to receive their deliveries after work hours (between 6pm to 8pm). • The third KPI aims at representing the environmental performance by estimating the quantity of fine particles generated by the distribution. The different KPI are used to represent the different interests of the stakeholders.

Stakeholders

City logistics involve multiple stakeholders, their different interests as described in [START_REF] Abel | Best Practice Handbook (Year[END_REF] illustrates well the complexity of the system. The shipper aims at minimizing overall cost while meeting the needs of the client. The carrier is also looking for low cost, but additionally wants high-quality transportation operations to satisfy both the shipper and the client. The client desires to be delivered on time and on a short notice. The city wants to be attractive for inhabitants by minimizing negative externalities while having efficient transport operations.

Parameters

The simulator is initially set up with the parameters presented in Table 1 andTable 2. These estimations are based on information about the city of Paris.

Interactivity

Interaction with the model is made possible thanks to the use of the python module "ipwidget" within jupyter notebooks. The module allows to easily creating sliders and buttons. In this case, the user is able to change the simulation parameters (cf. Fig. 3) and directly display the KPIs (cf. Fig. 4). The jupyter notebooks can also take advantage of the "binder" online service. The notebook is available for anyone to test from a web browser. A significant advantage of using jupyter notebooks relies on the fact that they are run on a remote server; as a result, there is no need for local environment configurations by the users. The interactive simulator used in this paper can be accessed at the following link: https://github.com/arthurgaudron/interactive_sim.

Results and discussion

Expected feedback

Two comprehension levels of the models have been considered. The first level is the identification of the parameters and their relationships. From this perspective, the user uncovers the basic foundations of the model. The second level is the identification of possible inconsistencies in the model. The user acknowledges the limitations of the tool and can ponder the conclusions drawn from it.

In the first level, users should identify the relationships between parameters:

• Capacity, number of vehicles and distance: vehicles with lower capacity will induce more travels in order to satisfy the same demand. As a result, there are more vehicles and total distance is increased. • Lead-time and cost: more vehicles will lower the delivery lead-time but increase the cost.

• Some parameters (e.g. external costs) have limited impact on the total cost. In the second level, users should be able to criticize the model with arguments:

• Understand that parameters should have an influence on each other. For example, fuel consumption and particle emissions, speed and fuel consumption, price of delivery and increase of capacity, public health cost depends also on the PM emission of the motorization. • Identify missing parameters: choice of hubs' localization, information system cost, transshipment cost.

• Assess the limits of the chosen parameters. For example, average speed can be used to represent fluidity of the traffic. However, this may be an oversimplication of the traffic phenomena.

Table 3. Questions and scores from the experiment on 28 students. The answers in the first part were True/False (T/F) or redacted (R). In the second part, users had to choose a value between 1 (strongly disagree with the statement) to 5 (strongly agree with the statement).

Questions Score (%) Level

Understanding of the models and the problem

In real life, the capacity of the vehicles impacts the size of the fleet. (T/F) 100% 1

In the VRP simulation, the capacity of the vehicles impacts the size of the fleet. (T/F) 79% 1

In the 2EVRP simulation, the capacity of the bikes impacts the cost. (T/F) 76% 1

Why 2EVRP is more expensive than VRP? What do you think is missing in the model? (R) 26% 2

What would be the main parameter to change to make the VRP as fast as the 2EVRP? (R) 76% 1

Identify two independent parameters in the simulation that are dependent in real life. (T/F) 66% 1

Identify two missing parameters. (R) 60% 2

Identify the limitation of one parameter. Why is it not representing the reality accurately? (R) 28% 2

Identify two parameters with a limited impact in decision. (R) 43% 2

Delivering a parcel with a bike costs in average 5€. How could companies decrease their average price? (R) In the trial performed for this research, self-evaluation and direct observation were used in order to test the understanding of the users. For the self-evaluation a survey was performed. The direct observation is performed with a supervision of the trial. The expert is supervising the experiment, asks and answers questions, and follows the progress of the users. Table 3 presents the questions and the results of the survey. The survey was developed in a brain storming session after identification of the key components of these distribution problems. The score indicates the average accuracy of the answers.

Feedback from the trial

The interactive simulation tool was tested with 28 participants from 11 different nationalities. They all had an engineering background with diverse specializations: architecture, civil engineering, technology management, electronics, mechanical engineering. The trial took place in the middle of a week dedicated to urban logistics, as a result the students were fairly aware of the relative problematics.

The participants had 45 minutes to test the interactive simulator. They were guided by the questionnaire shown in Table 3, which intended to test their understanding of the model. Results were not graded, so it is considered a sincere feedback. However, results have to be considered cautiously as validation test of the survey has not been carried out due to time constraint.

The objective was to test the capacity of participants to the two proposed models. To this end, minimal information was given at the beginning. More information was given to everyone upon request. The correctness of their answers is difficult to evaluate as nobody can claim to know the truth. Redundant questions in the survey show the absence of contradiction in the users' reasoning. The questions and scores are displayed in the Table 3. The first ten questions aimed at assessing the comprehension of the models. The last six questions aimed at obtaining feedback about the tool and the opportunity for using it as a communication support.

First, the results are encouraging. Most of the users got involved in the exercise. Although they were not deeply familiar with the urban logistics issues, they were able to tackle most of the questions in a short time.

Second, there is a clear difference of performance between the level 1 and level 2 questions. Users had lower performance on the questions relative to the limits of the tool (level 2). There is a great disparity in the results. Direct discussion during the test showed that some participants were quick at finding flaws in the model. On the contrary, some users seemed to be confused with the concept of model, and the fact they would have to study a wrong model. This confusion may have affected the quality of their answers.

Third, the answers and discussions showed that the inexperience about optimization and the lack of information prevented the users to fully apprehend the proposed models.

Conclusion

This paper proposed a tool that explored two distribution schemes using interactive simulation. The tool was tested on a trial group of 28 participants that did not know these schemes beforehand.

The results of the trial highlight that interactivity helps users to understand the models and their limits. It showed that users were able to get familiar with a new problem through the use of interactive simulation. Users can easily identify what are the significant parameters (or combinations) impacting the KPI. Users can also imagine their own hypothesis, and check if the model took them into account. They have appreciated the possibility of changing parameters. The fast response from this simulation was an important feature for users.

The assessment of the tool's performance is an important perspective for comparison with other learning material. As this tool has a teaching purpose, we considered the understanding of the participants as an evaluation metric. However, there are many methods to test the understanding of a subject: self-evaluation, one-to-one interview, dissertation, etc. As a first step, it would be valuable to test two different groups (e.g. interactive simulator vs. text book) with different evaluations (e.g. self-evaluation vs. one-to-one interview). It also opens the question of the complexity the model used: how to measure complexity? what is the maximum complexity one can use with an interactive simulator?

In urban logistics, interactive simulation can be a dissemination tool to involve non-experts in the decision process. Once the public is involved, they can give their opinion on the choices made in their city. One fundamental difficulty in city logistics is the multiplicity of objectives. A promising opportunity for interactive simulation is that it could be used to consult people on their priorities and to find out how they perceive choices of others, more precisely it could be used to validate regulations, new logistics schemes and offers.

In addition, being a dissemination tool, the proposed interactive simulator can be used as a support for animating discussions between several stakeholders. From the expert point of view, animating a discussion through interactive simulation allows three main advantages: [START_REF] Perboli | Decision support system for collaborative freight transportation management : a tool for mixing traditional and green logistics[END_REF] integrating the opinions of actors with different backgrounds; (2) identifying the main priorities of the participants (what KPI are relevant for which actors); (3) converging to a configuration of parameters that embodies an acceptable trade-off.

As a conclusion, the global feedback supports the perspective to consider further the interactive simulator. The proposed tool has shown its capacity to involve users in the learning process. The accessibility of this type of tool is questionable for a model with a higher level of details. As the model becomes more and more complex, interaction between the parameters will be harder to track for a non-expert.

Fig. 1 .

 1 Fig. 1. Overview of the interactive simulator

Fig. 2 .

 2 Fig. 2. Models implemented in the interactive simulator

Fig. 3 .

 3 Fig. 3. Parameters of the interactive simulation

  the tool's limitations, although it is a useful tool for communication. (1

Table 1 .

 1 Parameters for the Vehicle Routing Problem (Version HBEFA 3.3 2017 -Handbook Emission Factors for Road Transport; Korzhenevych et al., 2014) Van driver cost: 83 €/driver Van climate change cost: 2.80 cents €/km Van cost: 40 €/vehicle Van air pollution cost: 1.10 cents €/km

	Van fuel consumption: 7 L/100 km	Van capacity: 80 deliveries
	Van PM emission: 0.049 g/km	Van drop-off duration: 5 min
	Van speed: 30 km/h
	Table 2. Parameters for the MDVRP (Version HBEFA 3.3 2017 -Handbook Emission Factors for Road Transport;
	Korzhenevych et al., 2014)
	Truck driver cost: 83 €/driver	Truck climate change cost: 2.80 cents €/km
	Truck cost: 50 €/vehicle	Truck air pollution cost: 1.70 cents €/km
	Truck fuel consumption: 25 L/100 km	Truck drop-off duration: 5 min
	Truck PM emission: 0.057 g/km	Bike drop-off duration: 5 min
	Truck speed: 30 km/h	Bike speed: 20 km/h
	Truck capacity: 200 deliveries	Bike cost: 5 €/delivery

Acknowledgment

This work is supported by ADEME (Agency for the Environment and Energy Management), La Poste Group, Mairie de Paris (Paris City Hall), Pomona Group and RENAULT through the Urban Logistics Chair at MINES ParisTech.