François Torregrossa
email: ftorregrossa@solocal.com

Robin Allesiardo

Vincent Claveau
email: vincent.claveau@irisa.fr

Nihel Kooli

Guillaume Gravier

A Survey on Training and Evaluation of Word Embeddings

Keywords: Embeddings, Word Embedding Evaluation, Survey, Contextualised Embeddings, Non-Euclidean Embeddings

Word Embeddings have proven to be effective for many Natural Language Processing tasks by providing word representations integrating prior knowledge. In this article, we focus on the algorithms and models used to compute those representations and on their methods of evaluation. Many new techniques were developed in a short amount of time and there is no unified terminology to emphasise strengths and weaknesses of those methods. Based on the state of the art, we propose a thorough terminology to help with the classification of these various models and their evaluations. We also provide comparisons of those algorithms and methods, highlighting open problems and research paths, as well as a compilation of popular evaluation metrics and datasets. This survey gives: 1) an exhaustive description and terminology of currently investigated word embeddings, 2) a clear segmentation of evaluation methods and their associated datasets, and 3) high-level properties to indicate pros and cons of each solution.

embeddings are models for words answering to the particular use-case of providing meaningful representations for words. In other words, word embeddings are projections of words in a continuous space preserving the semantic relationships between them. Words are mapped to vectors sharing semantic properties through geometrical relations.

The strength of word embeddings relies on their ability to represent natural language with geometric relations. They enable efficient end-to-end architectures by transposing a discrete world representation into a continuous space. That is why they are widely used in Natural Language Processing (NLP) problems: they are easy to plug in Deep Learning architectures. Sentiment analysis, Named Entity Recognition [START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF][START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF][START_REF] Pennington | Glove: Global vectors for word representation[END_REF] and many other tasks outperformed their standard counterparts with these models.

We are mainly interested in word embeddings learnt over a training corpus «from scratch» [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF]. This family of representations is trying to compile a plain text dataset into a continuous vector representation without any expert knowledge. All of them rely on a simple statement of [START_REF] Harris | Distributional structure[END_REF] assuming that words appearing in similar context must have similar meanings. Therefore, in this work we assume this «distributional hypothesis» claiming that the sense of a word highly depends on the words surrounding it. Other word embedding techniques refine this assumption and use a language model to build contextualised word representations, such as BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Finally, a very different technique is to build vectors using knowledge bases or other sources of expert knowledge, an example being the TransE approach proposed by [START_REF] Bordes | Translating embeddings for modeling multi-relational data[END_REF]. We do not consider them on their own, but we will expose embedding techniques using the «distributional hypothesis» in combination with others, as done by [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF].

Different corpora, optimisation processes or basic geometries yield different sets of properties in the resulting projection. In this work, we are trying to emphasise these properties in order to provide a detailed analysis of these various embedding methods. Along with this analysis we also detail evaluation procedures.

The remainder of this paper is organised as follows. In Section 2, we explain elements of the underlying theory of embeddings. In Section 3, we discuss the different embedding evaluation metrics and methods. In Section 4, we present high-level features that characterise embeddings. In the final section, we conclude on suggestions for future research in the area.

2 Embeddings: from discrete variables to continuous semantic space Word embeddings, in their original formulation, are Lookup Tables mapping words to vectors of real numbers [START_REF] Collobert | Natural language processing (almost) from scratch[END_REF]. They are functions giving continuous vector representations in R D for elements of a set V (e.g. a set of words or tokens). In the following, v e w ∈ R D is the representation of an element w ∈ V in an embedding e. Formally a word embedding e can be represented as:

e : V → R D w → e(w) = v e w .
(1)

In order to estimate the word embedding function e, an optimisation process is performed on a large sample of language data on an arbitrary task. During this optimisation process, word vectors are coalesced given a criterion in charge of, for instance, extracting the similarity between words from a corpus. At the end, the embedding has accumulated information from the corpus for each word vector leading to noticeable geometric relationships. An example is shown on Figure 1: countries and their main cities are separated by distance and a direction almost identical for every pair. Automatic systems benefit from these relations since they explore numerical features and hyperplanes.

Fig. 1: Main cities and countries in a FastText [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] embedding of dimension 300, projected onto a 2d plane using PCA. Dashed lines join each country with its main city.

In the following we separate static from contextualized embeddings:

-Static word embeddings use the «distributional hypothesis» to learn global and constant vectors. In other words, they represent a word by a unique vector condensing every local usage of the word in the training corpus. We distinguish Euclidean embeddings from others.

-Euclidean word embeddings use the Euclidean geometry as mathematical support to embed vectors. Yet, due to intrinsic properties of this geometry [START_REF] Sun | Space-time local embeddings[END_REF], it may be difficult to embed asymmetric information. In fact, these kinds of embeddings [START_REF] Pennington | Glove: Global vectors for word representation[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF][START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] preferably incorporate symmetric knowledge such as semantic synonymy or analogy. -Non-Euclidean geometries provide a solution to embed asymmetric relations such as entailments or logical thinking. -Contextualised word embeddings are models providing variable vectors. In this work, we only review contextualised word embeddings using a language model to compute adaptative word representations. In this particular case, word representations are multiple and are directly computed from their context. The context of a word is usually composed by the words surrounding it.

For contextualised embeddings, we only consider the Euclidean space as non-Euclidean contextualised word embeddings have not yet been investigated.

On Figure 2, a graph gives the main classes of word embedding techniques explained by basic properties of those. Let a static word embedding be a word embedding where each word representation does not vary among contexts. For instance, the word «bank» in «I am going to the bank» and «I want to open a bank account» has two different meanings: a location or a financial organisation. However, a static word embedding gives the same word representation in both sentences without considering the context. Thus, the term «static» refers to this particularity: a word vector is always the same and can be statically pre-computed and stored in a matrix.

Static Word Embeddings

Static word embeddings are commonly represented with a matrix M = (m i j ) ∈ R |V |×D . The i-th row of M is then a vector of R D representing the i-th word of the vocabulary V . In other words, v M w i = (m i j ) j∈ 1,D , with w i being the i-th word of V .

Euclidean Static Word Embeddings

Euclidean Static Word Embeddings are using the Euclidean geometry to build word representations. For instance, both Word2Vec Skip-Gram (SG) or Continuous Bag of Word (CBoW) proposed by [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] and FastText SG or CBoW from [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] belong to this class of models. et al. (2013b) proposed a static word embedding algorithm named Word2Vec producing static word embeddings. It incorporates paradigmatic and syntagmatic relations between words in the vocabulary into word vectors, using two different word embedding models: SG or CBoW. Paradigmatic refers to individual concepts represented by the different words composing the vocabulary, while syntagmatic corresponds to the meaning inferred by the structure or syntax of language. Typically, idiomatic expressions are syntagmatic elements since their sense is derived from the whole sentence and not from the individual words in it. The Vocabulary V is composed of |V | words extracted from a set of sentences in a corpus T (see Figure 3). In the end, an embedding vector is associated with each word. This process relies on the «distributional hypothesis» from [START_REF] Harris | Distributional structure[END_REF], claiming that word sharing similar context has similar meanings. Following [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] and [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF], the context of a word can be defined by the k words before and the k words after this word in a sentence. k is called the window size and bounds the 2k words surrounding a target word (see Figure 3). Let C w 1 T = ... w i-(k+1) w i-k w i-(k-1) ... w i-1 w i w i+1 ... w i+(k-1) w i+k w i+(k+1) ... Fig. 3: Window of size k around a target word w i in a corpus T . The context words are then {w i-k , w i-(k+1) , ..., w i-1 , w i+1 , ..., w i+(k-1) , w i+k }. (resp. C w 2 ) be the set of words appearing in the context of a word w 1 (resp. w 2 ). The «distributional hypothesis» states that the semantic proximity between w 1 and w 2 is positively correlated to |C w 1 ∩C w 2 |. Two word embedding models were proposed to proceed the Word2Vec algorithm: the Skip-Gram (SG) and the Continuous Bag-of-Words (CBOW), presented on Figure 4.

Word2Vec

Mikolov

Word2Vec uses two embeddings: a target word embedding (or input matrix) W ∈ R |V |×D and a context word embedding (or output matrix) C ∈ R |V |×D . The word embedding C serves only for training and is mostly discarded after this process. One could argue that we could use a single matrix instead of two. However, context and target matrices play different roles. The context matrix C embed w.r.t. target words while the target matrix W embed w.r.t. context words. The aim of these two matrices is to reflect the difference between being a context and target word.

The Word2Vec Skip-Gram model tries to predict context words given a target word w t . Therefore, for all w t in the training corpus T , it maximises p(w c | w t , T, W, C) (w c , w t respectively being a context and a target word), i.e. the probability to observe w c in the window of w t . Increasing this probability is equivalent to integrating significant information on w c in the representation of w t . Formally, from Mikolov et al. (2013a), the criterion to maximise is:

1 |T | |T | ∑ t=1 t+k ∑ c=t-k c =t log p(w c | w t , T, W, C), (2)
where

p(w c | w t , T, W, C) = e v C wc ,v W w t
∑ w∈V e v C w ,v W w t .

(3)

The term

∑ w∈V e v C w ,v W w t
is not tractable in practice because the vocabulary contains millions of elements [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. Actually, at each optimisation step it requires the computation of millions of dot products in a high dimensional space, which is not feasible in reasonable time. An alternative is to use Negative Sampling (NS) to turn the Skip-Gram objective into a computable one [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. So far, we described Skip-Gram as a model trying to predict a context word given a target word. The idea of NS is to add random context words and detect them. Therefore, instead of predicting a context word given a target word, it classifies whether a context-target pair belongs to the corpus or was generated. In the end, it creates a representation where real pairs are geometrically close and noisy ones are scattered. Real pairs are called positives while noisy ones negatives, hence the name of the trick. Mathematically, NS maximises the dot product among positive pairs and minimises the dot product for negative pairs and is formulated as:

1 |T | |T | ∑ t=1 t+k ∑ c=t-k c =t log σ ( v C w c , v W w t )+ E w neg ∼P T (w) log σ (-v C w neg , v W w t ) , (4)
where σ (x) = 1 + e -x and P T (w) = |w| |T | . The Continuous Bag-of-Words [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] version of Word2Vec is similar to the Skip-Gram version but instead of predicting context words given a target word, it does the opposite: predicting a target word given the context words. Thus, the task is simpler since a larger set of words is known to infer the target word. In a sense, it might gather less semantic and more syntactic features, as shown by the experiment of Mikolov et al. (2013a). Yet for some languages, the CBOW version of Word2Vec leads to higher results [START_REF] Grave | Learning word vectors for 157 languages[END_REF]. Therefore, choosing the version is still a parameter to optimise.

Depending on the window size, different aspects of language are represented. For instance, a short window is best suited for syntagmatic representations whereas a long window catches paradigmatic notions [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. [START_REF] Levy | Neural word embedding as implicit matrix factorization[END_REF] have proven that Word2Vec relies on the fact that it is equivalent to factorising the pointwise mutual information matrix, i.e. the amount of shared information between a context and a target word.

GloVe

The Glove method [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] reduces also this co-occurrences matrix in order to get vector representations. This procedure compiles the whole corpus into a large sparse matrix, each coordinate being the co-occurrences count between two words. Two words co-occur if they are in the context window of each other. This approach has two main advantages. First, the process is divided into two steps being the compression of the corpus into a sparse matrix followed by its factorisation by word vectors. The compression step is convenient since it has to be carried out once and produce a matrix summarising global information for each word in the corpus. Therefore, the optimisation step will benefit from global knowledge. The second point relies on the context window extracting local information on words. Consequently, GloVe cumulates both global and local information into its vectors leading to effective semantic representations, as proven by their experiments. In comparison, Word2Vec embeddings perform slightly less. An explanation is that Word2Vec focuses on local information and embed less easily global information.

Formally, let X i j be the number of times word j appears in the context of word i, as done by [START_REF] Pennington | Glove: Global vectors for word representation[END_REF]. With these counters, authors defined

P i j = X i j ∑ k X ik (5)
as the probability that the word j appears in the context of word i. This probability helps at quantifying the relevance of a word with regard to others. For instance, given three words i, j and k, the quantity

Q = P ik P jk (6)
acts as a discriminative feature:

-Q 1 means that word k is more related to i than j, -Q 1 means that word k is less related to i than j, -Q 1 means that word k is equally related to i and j.

The idea behind GloVe is to embed word probabilities into word vectors v W w i , v W w j and v C w k all in R D by trying to approximate Q using those vectors leveraging a function F, formally:

F(v W w i , v W w j , v C w k ) = Q = P ik P jk , (7)
where F must satisfy several criteria. First, F has to treat word vectors linearly without ambiguity over dimensions, thus, F is written

F((v W w i -v W w j ) T v C w k
). This property preserves the cosine similarity between vectors since dimensions are treated consistently. The second property is the symmetry between context word and target word which is artificial and implies that F must be able to exchange both roles. Therefore, authors [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] added the constraint

F((v W w i -v W w j ) T v C w k ) = F((v W w i ) T v C w k ) F((v W w j ) T v C w k ) (8)
which entails

F = exp (9)
and

P ik = F((v W w i ) T v C w k ) ⇐⇒ (v W w i ) T v C w k = log X ik -log X i . (10)
log X i is considered as a bias parameter b W i and another bias b C k is added to the formula to have symmetry. In addition, as most of X i j are zero value, the log is not well defined. To alleviate this problem, a weight function f will be used in the loss function with the following properties:

f (0) = 0 and lim x→0 f (x)log 2 (x) is finite, f should increase, f should be small for large values.

Those properties ensure that f do not overweight small or high values and smoothly cancel the logarithm. Finally, the GloVe loss function is given by

J = ∑ i, j f (X i j )((v W w i ) T v C w j + b W i + b C j -log X i j ) 2 . ( 11
)
GloVe is another way to produce word embedding vectors from plain text source using contextual information as SG and CBoW. Its main advantages are that it does not use NS for training, and as we mentioned earlier, that it globally compiles the corpus (less computation).

FastText

The main drawback of Word2Vec and GloVe (see the two preceding sections) is their inability to compute representations for words not seen during the training phase, called Out Of Vocabulary (OOV) words. To address this issue, the FastText method [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] proposes to extend the embedding matrix with character n-grams. This is relevant since often the morphology of words influences their meaning. For instance, bi-gram «ed» carries a temporal notion: past. Other examples include: «er» for comparison, «est» for superlatives and «ly» for adverbs. FastText includes these subword information in the representation in order to provide pertinent vectors for unseen words. To do so, the

|V | × D Word2Vec matrix is turned into a matrix W FT of size (|V | + H) × D,
(x) = λ 1 -λ 2 e -x/λ 3 , λ = (λ 1 , λ 2 , λ 3
) being parameters for f λ . This model was chosen because we claim that, with fixed parameters, when the dimension is high enough the accuracy can not increase further a threshold λ 1 . λ 2 controls x-axis offset and λ 3 the convergence speed to λ 1 . Red points represent observations. For FastText, a character n-gram is a subset of n consecutive characters in a word w i . Every n-gram is hashed 1 in order to get the index of the row of W FT representing it. In the implementation of FastText [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF], vector representations are computed as follows:

v W FT w i FastText representation = W i, * FT word representation + maxn ∑ n=minn ∑ z∈G n (w i ) W N+h(z), * FT , sum of n-gram representations (12)
where:

minn and maxn are respectively the minimum and maximum size of character n-grams.

-G n (w i ) is the set of n-grams in w i , G n ( * ) being the set of all possible n-grams, such that G n (w i ) ⊂ G n ( * ). -W k, * FT is the k-th row of a FastText storage matrix W FT . -h : G n ( * ) → 1, H the hashing function: mapping a n- gram to an index.
FastText enables the computation of a word embedding via the sum of its n-gram embeddings. For instance, 3-grams of the word « <HELLO> » are: «<HE», «HEL», «ELL», «LLO», and «LO> ». Thus, the representation for « <HELLO> » is:

v <HELLO> = v HELLO + v <HE + v HEL + v ELL + v LLO + v LO> .

Knowledge Enhanced Word Embeddings

Word embedding techniques presented so far are not able to deal with word polysemy. Indeed, static word embeddings map each word to a vector which is unable to change for different word usage. Some researchers [START_REF] Iacobacci | SensEmbed: Learning sense embeddings for word and relational similarity[END_REF] 1 Using the FNV algorithm [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]) [START_REF] Lu | Concept representation by learning explicit and implicit concept couplings[END_REF] proposed to use external tools and knowledge bases to alleviate this problem and embed words with regard to their different meanings. SensEmbed [START_REF] Iacobacci | SensEmbed: Learning sense embeddings for word and relational similarity[END_REF] and CoupledCR [START_REF] Lu | Concept representation by learning explicit and implicit concept couplings[END_REF] are examples of such solutions and both use the following workflow.

First, dealing with polysemy is achieved by a slight modification of the word embedding model. A vector is no longer mapped to a word but to a word sense. For instance, the word spring has two word senses: the season and the metallic object, which leads to two word vectors, one for each meaning. In the literature, those vectors are called concept vectors because they represent the concept of the word.

Then, the textual data is preprocessed by replacing each word by the corresponding concept when possible (words are left if no matching concept is found): this is obtained by using a word disambiguation tool (such as Babelfy used in [START_REF] Iacobacci | SensEmbed: Learning sense embeddings for word and relational similarity[END_REF]) or annotations in Wikipedia [START_REF] Lu | Concept representation by learning explicit and implicit concept couplings[END_REF]. The training of concept/word vectors is performed similarly to word2vec SG, CBoW or GloVe. In addition to that, CoupledCR embeds the Wikipedia hyperlink graph and adds it to the concept vector through concatenation followed by an SVD. Once the concept/word vectors are optimised, it is possible to recover the similarity between two words by taking the most similar concept vectors (w.r.t. cosine similarity) attached to those words [START_REF] Iacobacci | SensEmbed: Learning sense embeddings for word and relational similarity[END_REF].

This technique is convenient because it enables word polysemy with word embeddings and it has been shown more effective than previous methods at recovering humanlike similarities between words. The main disadvantage is that it relies on third-party tools and good annotations which breaks the end-to-end fashion of word embeddings.

Non-Euclidean Static Word Embeddings

Entailment is a term commonly used to refer to asymmetric relations such as is_a or type_o f . For instance a dog is a mammal but a mammal is not always a dog. As mentioned ear- lier, projecting asymmetric relations using Euclidean embedding is complex. Actually, the Euclidean distance and most of Euclidean metrics are symmetric, i.e. f (w i , w j ) = f (w j , w i ). For example, the cosine similarity in Euclidean space does not consider the order of the input vectors. Thus, it is ambiguous to distinguish words as hyponyms or hypernyms of others [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF]. Furthermore, representing complex data structures (such as language) is successfully achieved in Euclidean spaces with a large number of dimensions. To corroborate this statement, we evaluated the performances of Euclidean embeddings on three different tasks w.r.t. the embedding dimension, as shown in Figure 5. This behaviour is due to the underlying properties in Euclidean spaces as shown in [START_REF] Sun | Space-time local embeddings[END_REF] and [START_REF] Laub | Feature discovery in nonmetric pairwise data[END_REF]. Some of them are represented in Figure 6. More degrees of freedom are required to emancipate from those properties that constrain the representation power of Euclidean embeddings. Besides, high dimensional spaces require more parameters and thus complicate the optimisation, as pointed by [START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF].

2 Source code for the tetrahedron by Ignasi https: //tex.stackexchange.com/questions/174317/ creating-a-labeled-tetrahedron-with-tikzpicture.

The point is not to discredit Euclidean embedding, but to claim that these properties are adapted for certain tasks (such as Analogy or Similarity) but fail to represent complex relations such as entailment (Nickel andKiela, 2017, 2018). In the next section, we detail non-Euclidean geometries more adapted to this class of relations. Those geometries are often used to embed Knowledge Graph, but this usage is out of the scope of this survey. Instead, we mostly review works where these geometries are used with plain-text data.

Probabilistic Word Embeddings (Word2Gauss)

Probabilistic Word Embeddings are word embeddings that map each word to a distribution probability (e.g. Gaussian). Instead of modelling each word by a vector and then infer a probability for p(w c |w t ) as done with Word2Vec or GloVe, probabilistic embeddings [START_REF] Curran Associates | Word representations via gaussian embedding[END_REF] directly parameterise a distribution probability and estimate p(w c |w t ) in the probabilistic embedding space. Then distance between words is estimated by metrics between distributions as Kullback-Leibler (KL) divergence, Expected Likelihood (EL) or Wasserstein [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF]. The first two are defined as

EL(p, q) = p(x) • q(x)dx. KL(p, q) = p(x) • log p(x) q(x) dx.
where p, q are distributions.

(13) Specifically, EL is used to represent symmetric relations as synonymy or analogy while KL is used to build asymmetric ones (as entailment, or hierarchies).

Gaussian Word Embeddings or Word2Gauss [START_REF] Curran Associates | Word representations via gaussian embedding[END_REF] is an implementation of this approach. Considering a Word2Gauss Embedding W G, each word is associated with a normal distribution:

v W G w i (x) = N (x, µ i , σ i )
, where µ i ∈ R D and σ i ∈ R D×D are respectively the mean and the covariance matrix of the normal law corresponding to w i . The mean interpretation is comparable to Word2Vec vectors. However, the variance encodes the uncertainty of a word. Here, uncertainty refers to how generic the word is. In other terms, words with high uncertainty (high variance) are very generic while words with low uncertainty (low variance) are very specific. For instance, the distribution of fruit must be very uncertain since there is a large variety of fruit hyponyms. Thus, we are not sure of what kind of fruit this fruit is. Now, an apple is more certain because it has less hyponyms. Therefore, the variance is an important addition of Word2Gauss w.r.t. Word2Vec as it enables hierarchical representations. Figure 7 shows how hierarchies can be represented in the probabilistic embedding space, using the uncertainty.

The optimisation process is similar to Word2Vec and relies on distances to embed asymmetric (KL) or symmetric (EL, Wasserstein) relations. Some experiments show that this advantageous ability to combine asymmetric and symmetric metrics in the same space is beneficial. Actually, it can incorporate information from taxonomies / knowledge graphs and plain text data simultaneously and outperforms the standard approach [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF].

Hyperbolic Word Embeddings (Poincaré / Lorentz)

Hyperbolic geometry is anisotropic: the distance depends on the direction of the movement. Euclidean shortest paths are straight lines where hyperbolic ones are curved: the curvature intensity depending on the location in hyperbolic space. Hyperbolic Word Embeddings are representations for words using the anisotropic attitude of the hyperbolic distance to embed structured data, as shown in [START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF] with a tree. Figure 8 (taken from their work) shows how this can be done: root elements are placed in the centre and most specific elements are positioned on the edges. Hierarchies are preserved using the hyperbolic distance in order to create tree-like structure. There are two main hyperbolic models used to embed data in the literature [START_REF] Nickel | Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry[END_REF][START_REF] Leimeister | Skip-gram word embeddings in hyperbolic space[END_REF][START_REF] Tifrea | Poincaré glove: Hyperbolic word embeddings[END_REF][START_REF] Nickel | Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry[END_REF][START_REF] Sun | Space-time local embeddings[END_REF]: Poincaré and Lorentz models. The major difference relies on the fact that the Poincaré model is bounded by the unit sphere while the Lorentz model is infinite. Therefore, the Poincare model is harder to implement than the Lorentz one since it generates numerical instabilities as 1 represents infinity. Yet, its geometry is easier to comprehend, as confirmed by [START_REF] Nickel | Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry[END_REF], because concepts of a similar hierarchy tend to coalesce along a single radius. Also, the specificity of concepts grows as the norm increases. In other words, central area should contain generic concepts whereas edges should have the most specific ones as shown in [START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF]. Such characteristics are harder to observe on the Lorentz model. However, numerical stability of Lorentz and visualisation simplicity of Poincaré can be used simultaneously since there is a diffeomorphism between both models, as presented by [START_REF] Nickel | Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry[END_REF].

Complex hierarchy such as WordNet [START_REF] Miller | Wordnet: A lexical database for english[END_REF] was projected into hyperbolic spaces (both Lorentz and Poincaré, Nickel andKiela (2018, 2017)). In all experiments, hyperbolic embeddings lead to a higher reconstruction accuracy even with small dimensionality. In comparison, best reconstruction performance for Euclidean embeddings is reached with high dimensionality and is low, as shown in Table 1.

The optimisation process is the following. Considering a vocabulary of concepts V , a set of edges E = {(w a , w b ), w a ∈ V, w b ∈ V } extracted from a hierarchy and a hyperbolic embedding H, the objective is to minimise the hyperbolic distance d H between existing edges while maximising d H for non-existent ones. Then, following [START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF], the objective is to maximise:

∑ (w a ,w b )∈E log e -d H (v H wa ,v H w b ) ∑ w b ∈E (w a ) e -d H (v H wa ,v H w b ) (14)
where: E (w a ) = {w b , (w a , w b ) / ∈ E } ∪ {w a }, the set of concepts not connected to w a including w a .

Another possible use-case for hyperbolic embeddings is to discover hierarchies from raw pair-wise similarities [START_REF] Sun | Space-time local embeddings[END_REF][START_REF] Nickel | Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry[END_REF]. The idea is to derive global structures from local similarity relations between concepts. Then the distance in the embedded space should encode the compatibility between concept and entailment information.

We are more interested into Hyperbolic space version of SG or GloVe [START_REF] Tifrea | Poincaré glove: Hyperbolic word embeddings[END_REF][START_REF] Leimeister | Skip-gram word embeddings in hyperbolic space[END_REF], replacing the dot product by the hyperbolic distance. For similarity and analogy tasks, it did not improve performance compared to high dimensional Euclidean equivalent, except for Glove with a noticeable difference. This confirms our point on the fact that Euclidean embeddings are more adapted for analogy and similarity tasks. Yet, hyperbolic embeddings still exhibit striking performances with lower dimensions.

Contextualised Word Embeddings

Contextualised Word Embeddings are directly opposed to static word embeddings since they produce variable word representations. Here, contextualised word embeddings propose to represent a given word based on the sentence in which it occurs. Therefore, a context-independent representation for a given word is not compiled from a dataset. Instead, local information is extracted using a language model and a contextualised word embedding vector is computed depending on the local usage.

Contextualised word embeddings handle three major problems of their static counterparts. Firstly, in the original modelling, static Euclidean embeddings cannot cope with polysemy since they use a unique global vector for each word. In the best case, knowledge enhanced word embeddings (see section 2.1.1.4) are able to cope with polysemy using concept but they require third-party tools and are not end-to-end. On the contrary, contextualised word embeddings adapt representations to current situations, theoretically leading to more relevant vectors. Indeed, the meanings of words highly depend on the context in which they occur. Secondly, it enables representations for n-gram of words. In some cases, the unigram language model is not enough to infer the meaning of a group of words. For instance, the sense of the English expression 'Speak of the devil' is very different from the individual meaning of the words composing it. Static word embeddings are theoretically adapted to unigram representation, thus, they are not convenient in this situation. The third and last point concerns the distributional hypothesis stating that the meaning of a word is distributed in its context. In practice, static Euclidean embeddings assume this distribution to be uniform. In other words, each context word equally participates in the meaning of target words. However, this is not totally right for each context window, as highlighted by the work of [START_REF] Ling | Not all contexts are created equal: Better word representations with variable attention[END_REF]: some context words do not participate (as stop words), some others participate more, or worse, have an opposite meaning.

An important remark is that contextualised word embeddings still employ static word embeddings to project the text to a continuous space. Then the projection is fed to sequence processing model that combines static features or enrich those to mutually embed contextual information.

According to [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF], there are two main approaches: autoregressive or autoencoding. For each of them, we will dedicate a section and examples. In the following, we consider a sentence (i.e. sequence of words) S = w 1 , ..., w T . Fig. 9: Autoregressive modelling for a sequence of tokens. The first model is the forward factorisation and the second one is the backward factorisation.

Then, the final goal of contextualised word embedding is to compute the representation of w i given the context of the sentence: (w j ) j∈ 1,i-1 for the autoregressive model, and (w j ) j∈ 1,T \{i} for the autoencoding model.

Autoregressive

Autoregressive models are models inspired from n-gram Language Modelling [START_REF] Józefowicz | Exploring the limits of language modeling[END_REF]. Indeed, n-gram Language Models (LM) try to estimate p(w t |w t-1 , ..., w t-n ) (see Figure 9), i.e. anticipate the upcoming word in a sentence given the n preceding words. Autoregressive models extend the limit of n-gram models by representing previous words by context vectors. Thus, they only consider relevant words in the preceding context. They seek to maximise [START_REF] Gu | Non-autoregressive neural machine translation[END_REF])

p(S) = p(w 1 , ..., w T ) = T ∏ i=1 p(w i |w j<i ) (15)
for all sentences S in the corpus, hence the name of these models. Actually, each word is exclusively explained by words previously seen in the sequence.

The first autoregressive models often use the combo pretrained static embeddings + RNN + softmax [START_REF] Akbik | Contextual string embeddings for sequence labeling[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Mccann | Learned in translation: Contextualized word vectors[END_REF][START_REF] Józefowicz | Exploring the limits of language modeling[END_REF] to represent the temporal sequence with a hidden context vector and then create an output distribution for words. Also, the sequence is reversed to perform bidirectional language understanding. However, as mentioned by [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]; [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF], the resulting contextualised embeddings is divided, in the sense that the past and post context are not merged but considered independently. If it were not the case, then the model would see upcoming words, thus leading to a useless or trivial representation. As we will show later, autoencoding architectures achieve inference with a unified context which can lead to better understanding of natural language, as proven by [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] and [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF].

More recent autoregressive model widely use transformer decoder, as GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF]. Using transformer instead of LSTM or recurrent network has shown to be beneficial. Even when the number of parameters is increased, transformer-based models tend to be more effective.

The OOV issue can arise in autoregressive embeddings as well, since they use a preset vocabulary. This is why, character CNN, proposed in [START_REF] Józefowicz | Exploring the limits of language modeling[END_REF][START_REF] Mccann | Learned in translation: Contextualized word vectors[END_REF], as well as byte pair encoding vocabulary, in Radford (2018), are used in complement with the contextualised representation to comprehend the morphology of words and extend representation to unknown words. Another solution is to train bidirectional systems with characters instead of words and then concatenate static pre-trained representations, as FLAIR method proposed in [START_REF] Akbik | Contextual string embeddings for sequence labeling[END_REF]. The two following paragraphs detail ElMo and FLAIR approaches.

ELMo

ELMo (Embedding for Language Modelling), proposed by [START_REF] Peters | Deep contextualized word representations[END_REF], is an autoregressive contextualised word embedding based on words, using a RNN. The idea is to create contextualised features by taking the hidden values of a multilayer bidirectional RNN learning a language model: predicting the upcoming word knowing the previous ones in a sentence. In the end, stacked hidden layers of the model should encode information of this language model. [START_REF] Akbik | Contextual string embeddings for sequence labeling[END_REF], is an autoregressive contextualised word embedding based upon characters, using a RNN. It considers sequence of characters instead of sequence of words. Word representations are finally extracted using the space character to determine boundaries of words. For instance, the forward (resp. backward) FLAIR model uses hidden state of the last (resp. first) letter of a word. Both hidden states (forward and backward) are concatenated in order to unify the context into a single vector. It is beneficial to add to this representation a static word embedding, according to [START_REF] Akbik | Contextual string embeddings for sequence labeling[END_REF]. An explanation for this behaviour is that static representations add word-level information to the character-based FLAIR representations.

FLAIR FLAIR, proposed by

GPT

GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Brown | Language models are few-shot learners[END_REF]) is a model that has been improved over time. It is an autoregressive contextualised word embedding using a transformer decoder to comprehend word sequences. The third version GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF] the way to future contextualised word embedding, since it is able to perform NLP tasks with limited efforts spent on fine-tuning. Instead, the model is trained on a very large corpus and is then directly used to answer NLP tasks. Actually, any NLP task can be formulated as a question, and GPT-3, as a generative autoregressive model, is ideally able to generate the correct answer (for instance «Translate to French: cheese.»). When NLP task is proposed as sentences, then, the NLP task objective is close to the unsupervised autoregressive objective. Hopefully, with a sufficiently large amount of data and enough training, we will be able to reconcile both.

Autoencoding

Autoencoding-like approaches such as BERT approach, proposed by [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], try to encode bidirectional context to predict a target word. Of course, no information is given to the model on the word to predict. This is achieved by using a [MASK] token hiding a target word in a sentence. Therefore the objective is to reconstruct the original token based on this masked sentence. Figure 10 pictures this task. Generally, after this step, the model is fine-tuned on a downstream task.

Autoencoding models, particularly BERT, are suspected not to capture language complexity (Niven and Kao, 2019). Instead, it would overfit spurious statistics in the training set. Furthermore, the [MASK] token is supposed to introduce noise in the model because downstream tasks do not contain this one, as pointed by [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF]. Likewise, the pre-training step is a reconstruction exercise where input sentences are corrupted by the [MASK] token. Therefore, the autoencoding embedding never observes real sentences and has to create representations based on a corrupted proxy. A solution to thwart those phenomena is to introduce new hyper-parameters controlling the amount of corruption or to replace the [MASK] token with randomly selected real words. However, those are not straightforward to choose nor to optimise. Finally, [MASK] tokens are supposed independent. This point, arose by [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF], consists of saying that the reconstruction of each masked token is done without considering the reconstruction of others. In the following, we explain BERT and XLNET embeddings.

BERT

BERT architecture proposed by Devlin et al. ( 2019) is an autoencoding contextualised word embedding using Transformer Encoder from [START_REF] Vaswani | Attention is all you need[END_REF]. It is trained in two steps. The first one involves two pre-training tasks: Masked Language Modelling (MLM) and Next Sentence Prediction (NSP). The second one is a fine-tuning phase. MLM consists of (as said above) masking random token and try to predict the most probable word given the whole context. NSP is a classification problem where the model has to tell whether a sentence can follow another. After the training phase, the representation for words is taken from the output of the encoder. We do not detail here more recent version of BERT, such as ROBERTA Liu et al. (2019), which mainly focus on reducing the computational costs of training as well as its stability and global performance.

XLNET

XLNET from [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF] is a method trying to make the most of both autoregressive and autoencoding approaches. It reduces noise and approximations introduced by autoencoding models while benefiting from unified context encoding. Therefore, it benefits from a complete representation of the context without having to mask inputs and does not require masked tokens.

Training

Today, two training schemes are opposing. The first one is a two step training with a pre-training phase and a fine-tuning phase (FLAIR, ELMo, BERT, XLNET). The model is first pre-trained on a large corpus with unsupervised tasks (MLM, NSP, maximise Eq. 15) and then fine-tuned on a specific NLP task for real-world usage. The second scheme (GPT-3) is a Global Intrinsic Extrinsic Involve external data Involve external models one-step training where an autoregressive model is massively trained using a huge amount of data and an enormous model. Then, answers of NLP tasks are generated by feeding the model with correct queries. As autoencoding models are not necessarily designed to be generative model, it is hard to use them in the second scheme since they will be unable to generate the answer to a query. Yet, the second scheme seems more advantageous because it does not require a large amount task-specific data or further training costs.

Evaluation

In this section, we propose to explore various evaluation solutions and metrics on Word Embeddings. As we mentioned in the previous section, different methods lead to different structures having pros and cons for solving specific tasks. In Table 2, we propose a partition , based on the work of [START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF], of word embeddings evaluation measures that we will detail in the following.

Global

Based on [START_REF] Amsaleg | Estimating local intrinsic dimensionality[END_REF] and [START_REF] Torregrossa | On the correlation of word embedding evaluation metrics[END_REF], let global metrics be the set of measures quantifying the distribution of vectors in an embedding space. In other words, any metric that does not need external labelled dataset and gives information on the representation capacity of the embedding.

For instance, the Intrinsic Dimensionality (ID), coming from Information Retrieval and introduced by E. [START_REF] Houle | Generalized expansion dimension[END_REF], aims to be a local metric correlated with the ideal number of dimension required to project datapoints in an embedding space [START_REF] Amsaleg | Estimating local intrinsic dimensionality[END_REF][START_REF] Houle | Generalized expansion dimension[END_REF]. One could create a global metric by studying the distribution of these local metrics in the embedding space. Another example of a global measure would be the δ avg -hyperbolicity, introduced by [START_REF] Tifrea | Poincaré glove: Hyperbolic word embeddings[END_REF], that measures whether the embedded objects have a tree-like structure. [START_REF] Torregrossa | On the correlation of word embedding evaluation metrics[END_REF] introduced other metrics used in different areas than NLP such as computer vision. They expect them to be simple and fast indicators on the representation capacity of embeddings. Those are particularly adapted to count significant dimensions in matrices. In other words, they estimate the minimal dimensionality required to compress matrices with a decent reconstruction error. Such metrics are also usable on word embedding matrices in order to discover the number of significative dimension of the representation: a high number of those being correlated with a good usage of the space dimensionality and potentially to effective representations.

Those two metrics are: effective rank (erank) [START_REF] Roy | The effective rank: A measure of effective dimensionality[END_REF] from signal processing and empirical dimension (edim) [START_REF] Poling | A new approach to twoview motion segmentation using global dimension minimization[END_REF] from computer vision. Their mathematic formulations are given by erank

(W ) = exp - D ∑ i=1 s i ∑ D j=1 s j • log s i ∑ D j=1 s j , (16) edim
(W, p) = ||s|| p ||s|| p 1-p , (17)
with s = (s i ) i∈ 1,D being the singular values of a matrix W

and ||x|| p = (∑ i x p i ) 1 p , p ∈ [0, 1]
. They both use singular values because those are related to the principal axis of variance of the matrix row vectors, the singular value itself being the value of this variance. The goal of erank and edim is to study the contribution of each singular value in the decomposition of the matrix row vectors. In the end, erank and edim return continuous values between [1, D], corresponding to the number of singular values that matter in this decomposition. A value close to D means that singular values are equally important whereas a value close to 1 exposes a preponderant singular value that concentrate almost all information. Other values indicate that the embedding is compressible into a space with a lower dimensionality (almost equals to the erank or edim value).

Intrinsic

Intrinsic evaluations [START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF] reflect the coherence between word vectors and human judgement. They need external human-labelled data in order to compare the vector structures with a ground truth. The intrinsic term refers to the fact that it measures the structure of the vectors in the embedding space without adding external information to the model. Indeed, those evaluations assess the global quality of the language representation. Such evaluations commonly rely on similarity metrics in order to compare, retrieve or coalesce word vectors. The Cosine Similarity:

cos(v w A , v w B ) = v w A , v w B ||v w A || • ||v w B || , (18)
is the most used metric for intrinsic evaluations. Actually, the dot product is not chosen since word vector norms capture Table 3: Similarity datasets [START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF]).

Name Size Pairwise Score based on WordSim353 [START_REF] Finkelstein | Placing search in context: The concept revisited[END_REF] 353 Synonymy (common words)

MEN [START_REF] Bruni | Multimodal distributional semantics[END_REF] 3000 Synonymy (common words) RG [START_REF] Rubenstein | Contextual correlates of synonymy[END_REF] 65 Synonymy (common words) SimLex-999 [START_REF] Hill | SimLex-999: Evaluating semantic models with (genuine) similarity estimation[END_REF] 999 Synonymy (common words) SimVerb [START_REF] Gerz | Simverb-3500: A large-scale evaluation set of verb similarity[END_REF] 3500 Synonymy (essentially verbs) RareWords (RW) [START_REF] Luong | Better word representations with recursive neural networks for morphology[END_REF] 2034 Synonymy (low frequency words) HyperLex [START_REF] Vulić | HyperLex: A large-scale evaluation of graded lexical entailment[END_REF] 2616 Entailment (common words) the frequency of words, as proven by [START_REF] Schakel | Measuring word significance using distributed representations of words[END_REF]. We consider three types of intrinsic measures, commonly used in the state of the art (as shown in Figure 11 3.2.1 Similarity (Figure 11a -Table 3)

It quantifies agreement between human word similarity judgement with the word vectors of an embedding. High agreement rate means that word vectors represent well words and language aspects of the dataset. This rate is obtained using a correlation measure. Spearman is the most used correlation for this purpose. This choice is quite relevant because it summarises all kinds of correlation (linear or more complex) into a single score. Specifically, it considers a matrix of pairwise similarities (S i j ) i, j∈ 1,N obtained after compilation of several human judgements. Then, Spearman correlation is performed on the set of points, using the cosine similarity (18):

{(x = S i j , y = cos(v w i , v w j )), i ∈ 1, N , j ∈ i + 1, N }.
The cosine similarity can be replaced by other measures capturing different kinds of relations (as entailment as done by [START_REF] Tifrea | Poincaré glove: Hyperbolic word embeddings[END_REF]; [START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF]). is to find a word D in the embedding so that R(C, D) holds.

Analogy (Figure 11b -

A convenient way to write the problem is A : B -C : ?. For instance, we should find Spain as a solution of the problem Paris : France -Madrid : ?, because Paris is the capital of France as Madrid is the capital of Spain. D is found by solving the following problems using the cosine similarity (18) between vectors, following [START_REF] Curran Associates | Improving distributional similarity with lessons learned from word embeddings[END_REF]:

-3CosAdd (1): argmax

D∈V \{A,B,C} cos(v B -v A + v C , v D ), -3CosAdd (2): argmax D∈V \{A,B,C} cos(v D , v B ) -cos(v D , v A ) +cos(v D , v C ), -3CosMul: argmax D∈V \{A,B,C} cos(v D v B )•cos(v D ,v C ) cos(v D ,v A )+ε , ε = 0.001.
The 3CosMul problem is known to be the most effective so far.

3.2.3 Categorisation (Figure 11c -Table 5)

It characterises the capacity of the embedding to distinguish semantic clusters among a set of words. A categorisation dataset D is defined by a set of N classes C = (C i ) i∈ 1,N , and a set of K words W = (w j ) j∈ 1,K . Each word belongs to a specific class, thus D = {(w,C(w)), w ∈ W } indicates the class C(w) ∈ C of a word w ∈ W . Then, the goal is to create N reconstructed clusters C r = (C r i ) i∈ 1,N with the K word vectors. CLUTO toolkit from [START_REF] Karypis | Cluto: A clustering toolkit[END_REF], with default parameters, is generally [START_REF] Baroni | Don't count, predict! a systematic comparison of contextcounting vs. context-predicting semantic vectors[END_REF]; [START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF]) used to complete this operation. At the end, comparison between C and C r is performed with the purity measure as explained by [START_REF] Manning | Introduction to Information Retrieval[END_REF]:

Purity(C , C r ) = 1 K N ∑ i=1 max j∈ 1,N #(C i ∩C r j ). (19)
A thorough presentation of notorious datasets for intrinsic tasks is given by [START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF]. We stored commonly used datasets in Tables 3 (similarity), 4 (analogy) and 5 (categorisation). Figure 11 pictures the three tasks. The major issue concerning the intrinsic evaluations is that they are adapted to static word embeddings and do not deal with contextualised ones. In addition, they evaluate on an artificial pre-defined task not necessarily in relation to a real need. (a) Similarity task on WordSim [START_REF] Finkelstein | Placing search in context: The concept revisited[END_REF]. Red dots represent word pairs, x-value being the cosine similarity for this pair in a word embedding and the y-value being the labelled value in the dataset. The spearman correlation is computed on this chart, giving 0.75 in this case.

v Paris v France v Madrid v Spain v France -v Paris (b)
Analogy task, the relation IsCapitalOf is assumed to be linear and is extracted from the example (France, Paris). On the figure, v Spain is perfectly fitting v Madrid + v Francev Paris . In reality, this is different, therefore the nearest neighbour of v Madrid +v France -v Paris is found in the embedded vocabulary. In practice, cosine similarities between vectors is used.

(c) Categorisation task on Bless dataset (Baroni andLenci, 2011) dataset (following Bakarov (2018)) with a 300d embeddings projected on a 2d-plane with t-SNE: each color represents a golden category, each mark represents a predicted category. The purity measures the mixture in the predicted categories w.r.t. the golden categories. For instance, predicted bowtie category in the bottom-left corner are dispatched in two golden categories (open-blue: weapon, and open-pink: vehicle).

Extrinsic

Extrinsic evaluations [START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF] need higher level and more complex representations than word embeddings to be solved. Therefore, external information is added via training and additional modelling in order to solve these complex tasks when word vectors may not be enough to solve the extrinsic problem. For instance, static embeddings are fed into a RNN that will name entity over a sentence. The RNN will comprehend sequential information. Another training phase is needed in order to optimise the RNN for the task. Therefore, external information is added to the word vectors through the RNN. Similarly, contextualised word embedding has to be fed into task-specific layers and fine-tuned to be compatible with the task. In both static and contextualised models, the task-specific layers are used to extract the rele- A large number of other extrinsic tasks are detailed in the survey of [START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF]. As stated by the authors, it is arguable that extrinsic evaluations provide good assessment of word embeddings quality, due to the fact that there are correlations across a wide range of extrinsic tasks. However, this assumption is controversial among the NLP community, since some embeddings can be constructed in order to achieve specific tasks while neglecting others, as shown by [START_REF] Claveau | Direct vs. indirect evaluation of distributional thesauri[END_REF]. Actually, the global quality of an embedding does not necessarily exist, instead, maximising the quality of the representation with regard to a specific task has more sense, as proven by the same authors and [START_REF] Schnabel | Evaluation methods for unsupervised word embeddings[END_REF]. Overall, evaluation of embeddings remains significantly complex as interaction mechanisms across different kinds of tasks and different methods are not well understood.

To overcome the need of proper and high-level evaluation, the famous benchmark GLUE Wang et al. (2018) was proposed to evaluate word embedding models and is now widely used. This benchmark provides nine different natural language understanding tasks aiming at enhancing the simple input-output correspondence evaluation. These nine tasks are supposed to give a wide description of the inner understanding of the word embedding model.

As an example we provide a small benchmark gathering some of the famous extrinsic task in NLP. We consider the three following tasks:

-Named Entity Recognition (NER) is a sequence labelling task, where a sentence is given and the model has to assign each word of the sentence to a particular entity class (a person, an organisation or other). CoNLL2003 [START_REF] Sang | Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition[END_REF] As mentioned earlier, we have to train new models when solving extrinsic tasks:

-For NER, we feed the sequence of static vectors or contextualised vectors into a one-layer bidirectional GRU followed by a CRF layer that predict the sequence of labels. Fine-tuning of some of the weights is allowed for contextualised models. -For TC and SA, we used the sequence of static vectors or contextualised vectors in combination with a one-layer bidirectional GRU followed by a linear mapping fitting the number of classes.

The implementation and pre-trained models are from the Python library transformers, presented by [START_REF] Wolf | Huggingface's transformers: State-of-the-art natural language processing[END_REF]. Table 6 reports the results we obtained using these three tasks and 4 word embeddings (1 is contextualised and 3 are static).

GloVe, Word2Vec Skip-Gram and FastText Skip-Gram are trained on a Wikipedia Dump following [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF]. All methods are used with default parameters. The dimension is set to 768 to match the dimensionality of BERT and perform a fair comparison. For each task, we train the bidirectional GRU and CRF model for 5 epochs and allow fine-tuning for BERT embeddings only. We noticed that finetuning static embeddings reduces performance gap between them.

BERT is the best of all tested embeddings, and this is explained by the fact that it uses contextual information and therefore can create a word representation adapted to its usage in a sentence. We notice that FastText is the best among static word embedding and this is mainly due to the way it handles OOV. These are simple observations on these tasks and different could be observed on different datasets.

Discussion

In this part we discuss properties characterising methods of Section 2. -Contextualised: when the representation of a word is calculated online using its context, i.e. when the word embedding is a contextualised word embedding. -Self-supervised [START_REF] Zhai | S 4 l: Self-supervised semi-supervised learning[END_REF]: commonly the task used to train an embedding is self-supervised. This means that a surrogate supervised task is automatically extracted from an unlabelled dataset. We do not consider WordNet [START_REF] Miller | Wordnet: A lexical database for english[END_REF] as an unlabelled dataset, because highlevel human knowledge was required to build this graph. Hence techniques that involve WordNet do not satisfy this property. -Non-Euclidean: when the representation is using a non-Euclidean geometry. -Multisources: embeddings use a main optimisation criterion. However, it is possible to enrich word vectors using various kinds of datasets and thus different criteria. For instance in [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF], authors use the KL-divergence and the Wasserstein distance to embed simultaneously a plain text dataset and a knowledge base. Contextualised embeddings are particularly multitask because they often require a final fine-tuning step.

Presentation of properties

Analysis

Training data sources are a major leverage for learning effective word embeddings. Plain text data is the most common sort of datasource considered by techniques listed in Table 7.

Word embedding is performed with a self-supervised training task based on the «distributional hypothesis». Word2Vec, GloVe and FastText have shown to be helpful representations for a wide range of NLP tasks. Indeed, they are often chosen as input prior knowledge of extrinsic problems [START_REF] Lample | Neural architectures for named entity recognition[END_REF][START_REF] Zhou | Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling[END_REF].

Sometimes, this hypothesis is locally unsatisfied [START_REF] Ling | Not all contexts are created equal: Better word representations with variable attention[END_REF]. Indeed, those models are particularly convenient for building unigram word vectors but not for n-gram ones. Contextualised propositions, such as FLAIR or BERT, are word embedding alternatives and solutions to these issues. Their surrogate self-supervised training task is based on a language model and therefore try to increase the theoretical reliability of the whole model. In the end, those new architectures globally improved the performance on extrinsic tasks. However, this gain is not free, and this because of the cost of the training phase. Today, on standard computers, Fast-Text tool4 complete a training on Wikipedia (2.5B tokens) in about a day. Conversely, according to [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], on the same corpus, BERT requires not less than 4 days on 16 TPUv3 chips, a massive and expensive physical architecture. A solution to time-consuming step is to download pre-trained multilingual or monolingual BERT models (when available), and perform a fine-tuning step to suit the downstream task. Another one is to do large-batch optimisation as done by [START_REF] You | Reducing BERT pre-training time from 3 days to 76 minutes[END_REF].

Instead of changing the hypothesis, it is possible to work with non-Euclidean methods. While Euclidean embeddings produce particularly efficient representations on semantic or analogy tasks, they still seem unable to extract logical thinking or entailment scheme from natural sentences. Hyperbolic or probabilistic embeddings try to reconfigure the representation (introducing asymmetric metrics) in order to integrate these different aspects of language. Hyperbolic spaces are known to encode more information with a lower dimensionality and show performance challenging high-dimensional Euclidean embeddings with low dimension [START_REF] Leimeister | Skip-gram word embeddings in hyperbolic space[END_REF]. On entailment task such as HyperLex [START_REF] Tifrea | Poincaré glove: Hyperbolic word embeddings[END_REF], hyperbolic word embeddings trained on wikipedia outperform their euclidean counterparts. However, on paradigmatic and syntagmatic tasks, Euclidean word embeddings remain the state of the art. Concerning probabilistic word embeddings, they also introduce uncertainty encoded in the variance of the distributions. Thanks to that, they significantly improved the performance on intrinsic tasks [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF].

Using multiple sources (multisources) of data also helps enhancing word vectors. So far we only evoked plain text data source, but there are many structured sources on the web such as WordNet [START_REF] Miller | Wordnet: A lexical database for english[END_REF]. Those could be embedded into vectors along with plain text data source as done in [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF].

Non-Euclidean

Multisources

GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] Word2Vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] FastText [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] SensEmbed [START_REF] Iacobacci | SensEmbed: Learning sense embeddings for word and relational similarity[END_REF] CoupledCR [START_REF] Lu | Concept representation by learning explicit and implicit concept couplings[END_REF] Word2Gauss EL [START_REF] Curran Associates | Word representations via gaussian embedding[END_REF] Word2Gauss Wasserstein [START_REF] Sun | Gaussian Word Embedding with a Wasserstein Distance Loss[END_REF] Poincaré GloVe [START_REF] Tifrea | Poincaré glove: Hyperbolic word embeddings[END_REF] Lorentz Skip-Gram [START_REF] Leimeister | Skip-gram word embeddings in hyperbolic space[END_REF] ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] -FLAIR [START_REF] Akbik | Contextual string embeddings for sequence labeling[END_REF] BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] -GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF] -XLNET [START_REF] Yang | XLNet: Generalized autoregressive pretraining for language understanding[END_REF] -

Conclusion

In this survey, we explained the main categories of widely used word embeddings. Terminologies and definitions are given to properly segment each technique as well as their advantages. Similarly, major metric tools have been introduced. Intrinsic metrics were thoroughly presented in order to give a clear procedure to compute them. Finally, high-level properties are proposed to qualitatively assess word embeddings and highlight uninvestigated approaches.

To conclude, we propose three main research paths.

First, contextualised word embeddings can only be assessed using extrinsic tasks. Therefore it is hard to understand what kind of semantics or syntactics they intrinsically capture. It would be beneficial to see whether the promise of context-dependent representation is fulfilled, and assess the extent of this potential accomplishment.

Hyperbolic deep neural network [START_REF] Ganea | Hyperbolic neural networks[END_REF] is a recent kind of neural network able to comprehend more naturally entailment and logical thinking. Many libraries as geoopt [START_REF] Kochurov | Geoopt: Adaptive riemannian optimization in pytorch[END_REF] offer the possibility to build hyperbolic networks. Therefore, it is possible to build contextualised word embeddings using the hyperbolic geometry and compare it with their Euclidean nemesis whether they are better able to capture grammar or thinking.

Finally, correlations between different kinds of measures (global, intrinsic, extrinsic) is not clear. In other word, it is not obvious to tell the quality of an embedding on a specific extrinsic task regarding its global performance on several intrinsic or global tasks. Investigations on this path would leverage two points. On the one hand, it would give new optimisation criterion for the training phase. On the other hand, it would propose a quick way to select an embedding for an extrinsic task.

Fig. 4 :

 4 Fig.4: The Skip-Gram (SG) and the Continuous Bag-of-Words (CBOW) models. On the left, the Skip-Gram model which predicts context words in the window given a target words. On the right, the CBOW model which predicts a target word given the context words of the window (adapted fromMikolov et al. (2013a)).

  Fig. 5: Evaluation metrics explained by the dimension FastText Embeddings. Dashed lines are trend lines calculated with the model f λ(x) = λ 1λ 2 e -x/λ 3 , λ = (λ 1 , λ 2 , λ 3) being parameters for f λ . This model was chosen because we claim that, with fixed parameters, when the dimension is high enough the accuracy can not increase further a threshold λ 1 . λ 2 controls x-axis offset and λ 3 the convergence speed to λ 1 . Red points represent observations.

  Fig. 6: Example of properties in R d

µ

  Fig. 7: Example of embedded hierarchy in the probabilistic Embedding space, as shown by Vilnis and McCallum (2015).

  Geodesics in the Poincaré disk represented by black lines. It is noticeable that distances between points depends on the location of those. (b) Tree embedding in the Poincaré disc. Shortest paths between points are represented by black lines.

Fig. 8 :

 8 Fig.8: Representations of the Poincaré disc, taken from[START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF]

Fig. 10 :

 10 Fig. 10: Masked Language Modelling (MLM) task, used to train autoencoding contextualised word embeddings. Tokens are randomly replaced in sentence by the [MASK] token. At the end, the contextualised representation of [MASK] is fed to a classification layer over the vocabulary. The objective is to predict the original token.

  , Mikolov et al. (2013b); Schnabel et al. (2015); Pennington et al. (2014); Bojanowski et al. (2017); Tifrea et al. (2019)).

Fig. 11 :

 11 Fig. 11: Intrinsic task samples.

Contextualised Word Embedding (Sec. 2.2) Static Word Embedding (Sec. 2.1)

					Start
		Euclidean	YES	Static Vectors	NO	Generative
		YES	NO			YES	NO
	Handles	Static Non-Euclidean Word Embedding		Autoregressive (Sec. 2.2.1)	Autoencoding (Sec. 2.2.2)
	OOV	(Sec. 2.1.1)		ElMo
			Word2Gauss		FLAIR	BERT
		NO	Poincaré GloVe		GPT
	YES		Hyperbolic SG
	FastText (Sec. 2.1.1.3)	Use External Knowledge
		YES
	Knowledge-Enhanced Embedding (Sec. 2.1.1.4)	Word2Vec (Sec. 2.1.1.1) GloVe (Sec. 2.1.1.2)
	SensEmbed
	CoupledCR

NO

Fig.

2

: Relations between word embeddings based on some basic properties.

Table 1 :

 1 Mean Average Precision (MAP) of Hyperbolic and Euclidean word embeddings on two WordNet tasks (reconstruction and link prediction). Reconstruction consists of learning every WordNet links and predict them based on distance. Link prediction evaluates generalisation: WordNet links are randomly separated in a train and test sets. Word embeddings are trained on the train set and try to predict links from the test set. Experiments and results are taken from the work of[START_REF] Nickel | Poincaré embeddings for learning hierarchical representations[END_REF].

	WordNet Task	Embedding kind	5	Embedding dimensionality 10 20 50 100	200
	Reconstruction	Euclidean Hyperbolic	0.024 0.059 0.087 0.140 0.162 0.168 0.823 0.851 0.855 0.86 0.857 0.87
	Link Prediction	Euclidean Hyperbolic	0.024 0.059 0.176 0.286 0.428 0.490 0.825 0.852 0.861 0.863 0.856 0.855

  is probably paving

				Softmax probabilities
				on the vocabulary
				maximised w.r.t. the masked token t3
				Output softmax
	Contextualised	vector	vector	vector	vector	vector
	token vectors	t1	t2	[MASK]	t4	t5
			Bidirectional Encoder Representation Transformers (BERT)
	Masked tokenized sentence	t1	t2	[MASK]
				Random Masking
	tokenized sentence	t1	t2	t3	t4	t5

Table 2 :

 2 Global, Intrinsic and Extrinsic Evaluations in term of properties.

Table 4 :

 4 Analogy datasets[START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF]).

	Name	Size	Relation Types
	Google Analogy Mikolov et al. (2013a)	19,000	Capital, Country, Family, Currency, Cities, Morphology
	MSR Mikolov et al. (2013c)	8,000	Morphology

Table 4 )

 4 It consists of solving a relationship problem. A pair of words is given (A, B) satisfying a relation R, i.e. A is something to B or R(A, B) holds. Then, a word C is given, and the objective

Table 5 :

 5 Categorisation datasets[START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF]).

	Name	Size	Number of Clusters
	Battig Baroni et al. (2010)	5330	56
	AP Almuhareb (2006)	402	21
	BLESS Baroni and Lenci (2011)	200	17

Table 6 :

 6 Extrinsic Evaluation

	Embedding Type	P	NER R	F1	P	SA R	F1	P	TC R	F1
	GloVe (768d)	0.833 0.809 0.820 0.381 0.358 0.346 0.908 0.908 0.908
	Word2Vec SG (768d) 0.882 0.837 0.858 0.447 0.436 0.415 0.920	0.92	0.919
	FastText (768d)	0.881 0.852 0.866 0.440 0.434 0.391 0.918 0.918 0.917
	BERT	0.921 0.928 0.925 0.508 0.500 0.471 0.937 0.938 0.937
	vant features prior learnt by the embedding model on natural
	language text.

Table 7

 7

	describes different embedding methods using five
	properties:
	-OOV (Out Of Vocabulary): a word embedding will han-
	dle OOV if it is able to give representations for unseen
	words. Basically, every embedding technique is able to
	do so by introducing a <UNK> token. However, this is not
	a proper solution for the OOV issue since each unknown
	word shares an identical vector. In the table, we indicate
	with a tick mark ( ) embeddings able to generalise the
	representations to unknown words using, for instance,
	subword information. The -symbol highlights methods
	that do not handle OOV completely. In fact, ELMo, BERT
	and XLNET work at the token level, and the tokeniser
	might incorporate subword information. Therefore, they
	handle OOV only if context words around the unknown
	word are well known.

Table 7 :

 7 High-level properties of various embeddings techniques.

	Embedding technique	OOV	Contex-tualised	Self-supervised training

http://groups.di.unipi.it/~gulli/AG_corpus_ of_news_articles.html

https://github.com/facebookresearch/ fastText

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.


Andreas Savin
email: andreas.savin@lct.jussieu.fr

Concluding remarks

density functional theory Faraday Discussion

WHO CARES ABOUT DFT 50 years ago, in a conference on the numerical simulation of plasmas(!) 1 Arnold C. Wahl said Some time ago, being fresh from the enthusiasm of computing Hartree-Fock (best molecular orbital) wave functions for non-trivial molecules, I gave a talk 2 entitled "Hartree-Fock is Here: What Next?" and I received, from an experimentalist, a reprint request for the article "Hartree-Fock is Here -Who Cares?" These days we discussed about "Density Functional Theory (DFT) is Here -What Next".

Would somebody dare to ask today for an article "Density Functional Theory is Here -Who Cares?"? I don't think so. The position towards computational chemistry has changed in the last 50 years. A few years ago, I met an experimentalist, member of the Comité National de la Recherche Scientifique. This is the steering committee of the CNRS, the largest fundamental science agency in Europe. I asked him what he thinks about theoretical chemistry. He answered Since we have DFT, theory competes with experiment. It became predictive.

Only DFT? Certainly not. Wave function methods also can be predictive. However, DFT computations provide good mix of accuracy and feasibility for systems of interest.

How far can we go with DFT calculations? In the nineties Walter Kohn 3 made a bet. He predicted that, in the year 2000, DFT calculations for systems with 1000 atoms will be possible.

He won the bet.

We heard from B. Civalleri 4 that nowadays calculations can be performed on systems with 3000 atoms, and this routinely (on workstations with less than 100 cores, available to labs, not in huge computing centers).

We also got reinforcement from mathematicians. We used to have them tell us that we were right, when we "knew" we were right. But now, they give us new tools. M. Herbst told us how to produce bounds in our calculations 5 , and promises also to give us bounds on our results. I will come back to this point at the end of this manuscript.

There is also competition. From machine learning (ML). Why not? D. Cole 6 showed us that models produced by ML can be (very!) fast and reliable. However, there are still aspects of machine learning that need attention. Let me enumerate some of them.

1. ML needs a heavy overhead of computations. Of course, as L. Reining pointed out 7 , we are used to it: in DFT we rely on uniform electron gas calculations that were done many years

ago. However, we may ask if the input we provide to a ML model is sufficient. What if we need new information? Let me try to clarify what I mean. If we use only the electron gas calculation in DFT we have the local density approximation. However, in a solid or in a molecule, there are points having the same density contributing differently to the density functional. For example, we don't expect that the region around the center of a bond should have the same role as one the molecular surface having the same density. How can a ML model be improved? With DFT approximations introducing gradient corrections was a natural step already advocated for (it must be said, in an unsuccessful form) in the Hohenberg and Kohn paper (see, e.g., equation (48) in their paper [START_REF] Hohenberg | [END_REF] ). The gradient corrections could be implemented with a conceptual, but not an important computational overhead.

2. It is not clear what data to choose to train a ML model. This, of course, is not the fault of ML, but of the existent quantum chemistry methods. With wave function methods, we don't work at the basis set limit (neither the one-or the N-particle one). If a density functional approximation should be chosen, which one should be selected? 3. How sensitive is the ML model to the class of systems used for learning? Density functional approximations are quite robust to the change of system. It was one of features that imposed them in chemistry. For example, they could treat well not only molecules with main group elements, but also those containing transition metals. Density functionals were also used for properties for which they are not supposed to work. A typical example is using orbital energies differences to estimate band gaps, although we know that this is not a property of exact Kohn-Sham (see, e.g., 9 ). Normally, ML gives out one number as an output.

Are we doomed to accept these limitations of ML? We can be certain that much can be done. L.

Reining stressed that the connector approach does not prohibit the use of vectors [START_REF] Aouina | Faraday Discuss[END_REF] . J. Wetherell showed us that more flexible physics can enter into ML by using data compression to search for constraints. [START_REF] Wetherell | Faraday Discuss[END_REF] . M. Gatti showed us that one can use quantities that are innovative [START_REF] Vanzini | Faraday Discuss[END_REF] .

CHALLENGES

Density functional approximations have still problems. Let us discuss a few of them, as they showed up during the discussions.

Potentials

As a rule, approximate density functionals are given as integrals over some local quantity (energy density). However, this quantity is not unique, as adding to it any function that integrates to zero does not change the value of the density functional. A truly local quantity is the Kohn-Sham potential, v KS . The Hohenberg-Kohn theorem tells us that there is a one-to-one correspondence between it and the density. However, the density-density response function tells us that there can be many potentials that give nearly the same density (see, e.g., [START_REF] Savin | [END_REF] ). This might be an explanation why so many approximations work: we can have many potentials that give a similar density.

An interesting viewpoint has been presented by D. Kooi and P. Gori-Giorgi. [START_REF] Kooi | Faraday Discuss[END_REF] She showed that one can describe exactly the leading terms of dispersion interaction by taking into account only the fragment densities, and explained that this does not violate the Hohneberg-Kohn theorem.

Nevertheless, we may want good potentials, e.g., as a starting point for the calculation of spectra. What path should we choose? W. Yang gave arguments for a potential functional theory. [START_REF] Yang | Faraday Discuss[END_REF] N. Gidopoulos showed that potentials can be improved by imposing physical constraints on them. [START_REF] Callow | Faraday Discuss[END_REF] N. Maitra showed how difficult it is to construct time-dependent potentials (nevertheless making progress with it). [START_REF] Lacombe | Faraday Discuss[END_REF] M. Gatti constructed models to obtain directly the spectral function by constructing models that are not based upon local, static potentials. [START_REF] Vanzini | Faraday Discuss[END_REF] A possible way to summarize is it is to say that there are other one-particle models that can be interesting (see also [START_REF] Aouina | Faraday Discuss[END_REF] ).

Spin

Density functional theory would have never had success in thermochemistry without using spin-densities (unrestricted Kohn-Sham). However, it is not clear why they are needed in DFT.

Finally, the Hohenberg-Kohn theorem tells us that for computing the energy we need only the density and the external potential. When I entered DFT, I heard people say It is easier to get approximations if we apply a weak magnetic field. For example, for the dissociated H 2 molecule, a weak magnetic field breaks the symmetry of the molecule, and stabilizes a spin-polarized solution.

Just to be clear: such a field should not be a uniform magnetic field, because this would stabilize the triplet, and this is not what we want. I asked myself whether the magnetic field to produce a fully polarized electron gas (the basis of almost all our approximations) is weak. For r s = (3/4πρ) 1/3 = 1 bohr, it is around one atomic unit, two orders of magnitude stronger than the strongest magnetic field ever produced on Earth. [START_REF] Bykov | [END_REF] It should not really be a surprise, as electronic excitations are needed to overcome the Pauli principle.

Different paths are being explored. C. Jacob favors using spin-state dependent exchangecorrelation functionals. [START_REF] Brüggemann | Faraday Discuss[END_REF] J. Neugebauer points out an alternative route to broken-symmetry calculations based on subsystems. [START_REF] Massolle | Faraday Discuss[END_REF] Maybe one should better treat properly magnetic fields. [START_REF] Irons | [END_REF] Or should one give up spin-densities and concentrate on pair-densities, as done by K. Pernal [START_REF] Pernal | Faraday Discuss[END_REF] and D.

Truhlar [START_REF] Bao | Faraday Discuss[END_REF] ?

Excited states

Initially, DFT was seen as a ground state theory. With time, it was shown that there are extensions that allow providing excitation energies, and this became one of the successes of DFT.

Nevertheless, there still are questions related to it. I once discussed with a mathematician who said that from the mathematical viewpoint there is no rigorous proof for it. One day, I proposed that we have a discussion about this topic. His answer was You want to discuss with me about a second derivative [ f xc ] of a quantity [E xc ] for which the first derivative [v xc ] is not well defined?

We know that we need more to have a "clean" theory. But progress is made. P. Romaniello showed us how to get beautiful results with improved f xc . [START_REF] Sabatino | Faraday Discuss[END_REF] G. Levi developed algorithms to find saddle points (associated with some excited states). [START_REF] Levi | Faraday Discuss[END_REF] P.-F. Loos presented us ensembles as a way to follow. [START_REF] Marut | Faraday Discuss[END_REF] W. Yang explained that we can go even further, and consider even changes in the number of electrons. [START_REF] Yang | Faraday Discuss[END_REF][START_REF] Yang | [END_REF]27 D. Truhlar calculated excited states even in regions of strong state interaction, such as near conical intersections. [START_REF] Bao | Faraday Discuss[END_REF] There is more to understand in DFT In 1988 T. Ziegler gave a talk in Banff. He said We know that density functionals work. The question is why.

He favored the explanation that density functional approximations model exchange-correlation holes. Is the Random Phase Approximation not producing models for it? J. Erhard showed us holes produced this way. [START_REF] Erhard | Faraday Discuss[END_REF] C. Jacob decomposed them into symmetric and antisymmetric (singlet and triplet [START_REF] Mcweeny | [END_REF] ) components and analyzed them. [START_REF] Brüggemann | Faraday Discuss[END_REF] N. Maitra finds a relation between why adiabatic TDDFT sometimes gives good results for dynamics in strong fields and the time-evolution of the natural occupation numbers. [START_REF] Lacombe | Faraday Discuss[END_REF] A fundamentally new approach is that proposed by K. Burke. He finds that the approximations can be justified by analyzing an asymptotic behavior (h → 0 keeping the chemical potential fixed). [START_REF] Burke | Faraday Discuss[END_REF] L. Reining explained how connector approximations generalize the path used to construct density functional approximations. [START_REF] Aouina | Faraday Discuss[END_REF] Going beyond Kohn-Sham Kohn-Sham had an overwhelming success. But we don't have to stay with it. J. Erhard discussed an extension of the Random Phase Approximation. [START_REF] Erhard | Faraday Discuss[END_REF] One can consider taking into account the pair density, in particular the on-top pair density. K. Pernal combined CAS wave function with dynamic correlation obtained with density functionals; she also drew our attention to the fact that in CASSCF calculations correlation can bring electrons together. [START_REF] Pernal | Faraday Discuss[END_REF] D. Truhlar showed us a way to combine multi-reference with DFT calculations. [START_REF] Bao | Faraday Discuss[END_REF]

OUTLOOK

We have seen that DFT is not only successful, but also has a high potential to develop. However, I would like to stress something that we are still missing in it.

We all know about J. Perdew's classification of DFT approximation according to his Jacob's ladder [START_REF] Perdew | AIP Conference Proceedings[END_REF] . Another well-known ladder in mystical tradition is the one of Guigo II. He wrote a book, The Ladder of Monks. Let me make an analogy between Guigo's rungs to reach God, and our steps to judge the quality of our computational results.

1. Lectio, i.e., reading. We have to be informed, we judge our results by comparing them with reference data, we use statistics. This is the method used for density functional approximations.

2. Meditatio, i.e., thinking. We know we can go further in a formal way. This is the method used in perturbation theory.

3. Oratio, i.e., prayer. We would like to approach the exact result. This is what we do in the variational method. 4. Contemplatio. We would like to be close to the exact result. This is why we need rigorous bounds, to know how far we are from the exact solution (what M. Herbst called errors of the model [START_REF] Herbst | Faraday Discuss[END_REF] ).

We should try to go beyond the first step, and the different contributions at this meeting have shown that we are on the good path.

CONFLICTS OF INTEREST

There are no conflicts to declare.

ACKNOWLEDGEMENTS

I would like to thank the speakers and Andrew Teale (University of Nottingham, UK) for going through the first draft of this manuscript and commenting it.