Benjamin Spector
email: benjamin.spector@ens.fr

The pragmatics of plural predication: Homogeneity and Non-Maximality within the Rational Speech Act Model *

This paper offers an account of two puzzling properties of the intepretation of plural definites, known as Homogeneity and Non-Maximality, within the framework of the Rational Speech Act model.

Introduction

Plural definites (PDs) display two well-known properties: Non-maximality and Homogeneity.

Non-maximality refers to the fact the the quantificational force of plural definites is variable across contexts. While they tend to have universal quantificational force, they easily 'allow for exceptions'. For instance, if there are many windows in a building and if I was asked to make sure that some fresh air enters the building, (1a) below can be judged true if I opened all of the windows but two or three. If the quantifier all is added, as in (1b), this is no longer the case.

(1) a. I opened the windows. b. I opened all the windows.

Homogeneity refers to the fact that even in contexts where a sentence such as (1a) has universal or near-universal force, its negation does not amount to the negation of a universally quantified statement. That is, (2a) does not merely mean that I didn't open all of the windows, but rather suggests that I didn't open any of the windows (or at most very few). Again, this effect is removed when all is added, as in (2b).

( Križ and Spector offer an account ( [START_REF] Križ | Interpreting plural predication: Homogeneity and non-maximality[END_REF], henceforth KS) of both properties based on two components: a semantic component according to which plural predication generates multiple interpretations, and a pragmatic component that regulates how the resulting underspecified meanings are used and interpreted in different contexts. The goal of this paper is to reconstruct the pragmatic component within a general framework for pragmatic reasoning, namely the Rational Speech Act model ( [START_REF] Goodman | Knowledge and implicature: Modeling language understanding as social cognition[END_REF]). In so doing, I hope to strengthen the conceptual motivations for our pragmatic component, by showing them to be derivable from general principles of rational conversation, and to make additional predictions regarding how context influences the interpretation of sentences that contain plural definites.

KS's account

KS's account relies on a specific semantic rule for predicates, and two interpretative principles.

Semantic component

KS offers a complete compositional semantics whereby, whenever a predicate is applied to a plural object, the resulting denotation consists of a set of propositions. Simplifying somewhat, the set of propositions associated with (3a) is given informally in (3b) [START_REF] Goodman | Knowledge and implicature: Modeling language understanding as social cognition[END_REF] Assume that there are exactly three This set is obtained by 'plugging' in the position of the books all the generalized quantifiers that can be obtained by applying disjunction in all possible ways to all the sums that can be obtained from {a, b, c}. When negation is applied, as in Mary didn't read the books, it applies pointwise to all members of the denotation of the negated sentence or predicate, giving rise to the set {¬(Mary

read a ∨ b ∨ c), . . . , ¬(Mary read a ∧ b ∧ c)}.
The members of the denotation of a given sentence are called its candidate meanings.

Interpretative principles

KS's account relies on two interpretative principles

• Ban on overinformative meanings

The first principle states that, among the candidate meanings for S (i.e. all the propositions in the denotation of S), those that are overinformative relative to a given Question Under Discussion (QUD) are ruled out. This is what governs the following definition of relevant candidate meanings.

1. Overinformativity relative to QUD Given an underlying QUD Q, modeled as an equivalence relation over possible worlds notated ∼ Q , a proposition φ is overinformative if it makes distinctions between some possible worlds that are equivalent given the QUD. In other words, φ is overinformative relative to Q if: ∃w 1 , w 2 (w 1 ∼ q w 2 ∧ φ(w 1 ) = φ(w 2 )) (where φ(w) denotes the truth-value of φ in w).

Relevant candidate meanings given a QUD

A candidate meaning for S is a relevant candidate meaning for S relative to a QUD Q if and only if it is not overinformative relative to Q.

Importantly, the QUD with respect to which an utterrance is interpreted need not correspond to an actual interrogative utterance in the prior discourse. Rather, a QUD is simply an equivalence class over worlds that represents what speakers care about at a certain point of a conversation. In fact, there might be uncertainty about what the QUD is, and KS's proposal is based on an idealization. The RSA model that we will develop will not need this idealization.

• Truth on all intepretations.

The second principle states that a sentence S is judged true in the context of a QUD Q only if all its relevant candidate meanings relative to Q are true. This principle entails as a special case the Stronger Meaning Hypothesis ( [START_REF] Dalrymple | Reciprocal expressions and the concept of reciprocity[END_REF]). It is possible for both a sentence S and its negation not to be judged true, and so I have implicitly defined a trivalent semantics. In fact, the principle of 'Truth on all interpretations' is similar in spirit to the guiding intuition of supervaluationism. Consider first a context where the underlying QUD is something like Which books did Mary read?, which corresponds to the equivalence class such that two worlds are equivalent just in case Mary read exactly the same books in both. In this case, none of the candidate meanings for (4a) is overinformative relative to the QUD, because the various candidate meanings differ from each other only with respect to which books Mary needs to have read for them to be true. 'Truth on all readings' then entails that (4a) is judged true if all its candidate meanings are true. One candidate meaning, namely the proposition that Mary read all the books on her reading list, entails all the others, and so (4a) predicted to be judged true if and only if Mary read all the books on her reading list. As to (4b), again no candidate meaning is overinformative relative to the QUD. (4b) is judged true if all candidate meanings are judged true. There is again a candidate meaning that entails all the others, namely the proposition that Mary didn't read any book on the reading list (this proposition is obtained by 'plugging' in the position of the books on her reading list the disjunction of all the individual books on the reading list). (4b) is therefore judged true just in case Mary didn't read any books on her reading list.

Applying of the theory

Non-monotonic contexts

As discussed in KS, 'Truth on all interpretations' captures the intuitive truth-conditions of sentences in which a plural definite is under the scope of a non-monotonic quantifier, as in:

(5) Only one student read the books on the reading list.

In a context where we care about which books each student read, ( 5) is predicted to be judged true if, for every disjunction of D of pluralities of books on the reading list, only one student read D. These truth-conditions are equivalent to 'there is a student who read all of the books on the reading list and all other students read no books on the reading list' -the conjunction of the two most 'extreme' candidate meanings, namely 'Only one student read all the books' and 'Only one student read at least one of the books'. This appears to be a good result, given, in particular, the experimental results reported in [START_REF] Križ | Two methods to find truth-value gaps and their application to the projection problem of homogeneity[END_REF].

Capturing non-maximality

Suppose now that Mary has the following reading obligation: 'read at least at least half of the twenty books on her reading list'. Assume further that the underlying question is whether she in fact did that. In such a context, if we learn that Mary read, say, 15 of the 20 books, it seems that we might truthfully utter something like 'Mary read the books on her reading list, so she will probably pass' (in contrast with 'Mary read all the books on her reading list, so she will probably pass'). This connection between non-maximal readings and the underlying QUD, which is substantiated and discussed at length in [START_REF] Malamud | The meaning of plural definites: A decision-theoretic approach[END_REF], is captured as follows. Consider (4a), in the context of the QUD 'Did Mary read at least half of the books on her reading list'. First, note that no candidate meaning entails a negative answer to the question (even the weakest candidate meanings are compatible with Mary having read all the books). There is one candidate meaning that is equivalent to the positive answer, namely the proposition that Mary read at least half of the books (which is obtained by taking the disjunction of all pluralities of books that contain at least half of the books). This candidate meaning is not overinformative: it is true in all the worlds in which the answer is positive, false in all others. However, all other candidate meanings are either equivalent to this one, or are overinformative, in the sense that they draw distinctions between worlds in which the answer to the underlying polar question is the same. For instance, the proposition that Mary read, say, at least a specific list of 12 books entails that the answer to the question is positive, but is false in some worlds where the answer is positive. As a result, there is only one relevant candidate meaning, namely the proposition that Mary read at least half of the books, and so (4a) is predicted to convey, in this context, that Mary read at least half of the books on her reading list. In practice this might be too precise a prediction, in that what proposition is exactly conveyed might not be completely clear.

In our RSA account, this will be explained by the fact that there might be uncertainty about what the QUD is (cf. section 3.5).

The negative case, (4b), works in the same way. Because the property of being overinformative relative to a QUD is closed under negation, there is again only one relevant candidate meaning for (4b) relative to the QUD 'Did Mary read at least half of the books on her reading list', namely the proposition that Mary didn't read half of the books on her reading list, and so (4b) is predicted, in this context, to convey precisely this proposition.

Motivation

KS argues that their interpretative principles are natural principle of language use. Ruling out overinformative candidate meanings is rational because it is assumed that speakers respect Grice's maxim of relevance, and so do not provide irrelevant information. A proposition that is overinformative relative to a QUD Q does more than just addressing Q, and is in this sense not relevant. As to 'Truth on all interpretations', it can be viewed as a rational solution to a coordination problem. Faced with a sentence that can have multiple interpretations that are all relevant and equally plausible in a given context, one has no way to decide which reading is intended. The speaker who uses such a sentence to convey one of its possible meanings to the exclusion of others cannot be sure that the intended reading will be the one that the listener will converge on, and in fact incurs a serious risk that the listener will end up believing something that she herself does not believe, in violation of Grice's maxim of quality. Only if the speaker believes that all the possible meanings are true is this risk eliminated, and so the sentence will typically be used in such cases. The listener can herself reason that the speaker will use the sentence only in such situations, and so the requirement that the sentence be true on all its relevant interpretations for it to be used becomes a convention between speakers and listeners.

One of my goals here is to provide a formally explicit foundation for this reasoning.

The Rational Speech Act account

In this section, I attempt to reconstruct the account I have just presented by building on a game-theoretic model of pragmatic inference and message choice, namely the Rational Speech Act model ( [START_REF] Goodman | Knowledge and implicature: Modeling language understanding as social cognition[END_REF]). For simplicity, I will now assume that sentences with a plural definite are underspecified between just two candidate meanings, i.e. the one where the plural definite has universal force, and the one where it has existential force.

The basic RSA model.

In the basic RSA model, we start from a literal listener L 0 who has a prior probability distribution over worlds and knows the literal meanings of sentences. When hearing an utterance u, L 0 updates her prior distribution by conditionalizing it with the proposition expressed by the literal meaning of u. Then we define a speaker S 1 who wants to communicate her beliefs to L 0 and knows how L 0 interprets sentences (i.e. knows L 0 's prior probability distribution and knows that L 0 interprets messages by conditionalization). S 1 is characterized by a utility function U 1 such that the utility of a message u if S 1 believes w is increasing with the probability that L 0 assigns to w after updating her distribution with u, and decreasing with the cost of u. Importantly, if the literal meaning of u is incompatible with w, the utility of u is infinitely negative. S 1 's probability of choosing a message u when she wants to communicate w is defined by a function that is increasing with the utility of u relative to w. A parameter λ determines the extent to which S 1 maximizes her utility. Next, we define a more sophisticated listener, L 1 , who, when receiving a message u, uses Bayes's rule to update her prior distribution on worlds, under the assumption that the author of u is S 1 . A speaker S 2 is then defined exactly like S 1 , except that now S 2 assumes that she talks to L 1 , not L 0 . And so on:

1. The interpretation function L, when applied to a message u and a world w, returns 1 if u is true in w, 0 otherwise.

2. L 0 (w|u) ∝ P (w)L(u, w).

3. U n+1 (u|w) = log(L n (w|u)) -c(u)
4. S n+1 (u|w) ∝ e λUn+1(u|w)

5. L n+1 (w|u) ∝ P (w)S n+1 (u|w).

Truth on all readings.

Assume now that some sentences are ambiguous, i.e. we start with a set of multiple interpretation functions I. There are as many different literal listeners as there are interpretation functions. We now define S 1 as believing that she talks to a literal listener L 0 but as being uncertain about which interpretation function L 0 is using. This uncertainty is represented by a probability distribution over I. The rational utility function for such an S 1 is such that the utility of a message u is the expected utility of u across interpretation functions. The next listener, L 1 , assumes that she is receiving a message from S 1 and does not need to care about which interpretation S 1 is using, since there is no uncertainty about S 1 's strategy. Then for the higher levels nothing changes. This leads to:

1. The interpretation function L, when applied to a message u and a world w, returns 1 if u is true in w, 0 otherwise.

Update rule for a literal listener who uses an interpretation function L:

L 0 (w|u, L) ∝ P (w)L(u, w).

3. U 1 (u|w) = L∈I P(L).U 1 (u|w, L) = L∈I P(L i )[log(L 0 (w|u, L)) -c(u)]
4. For n ≥ 1, S n (u|w) ∝ e λUn(u|w)

5. For n ≥ 1, L n (w|u) ∝ P (w)S n (u|w).

For

n ≥ 1, U n+1 (u|w) = log(L n (w|u)) -c(u)
Suppose now that, for some interpretation function, the literal meaning of a sentence u is false in w. Then at least one term in the sum in Equation 3 is -∞, and as a result the whole sum is -∞. The utility of u for an S 1 who believes w is thus -∞, and so the probability of choosing u to convey w is 0. 'Truth on all interpretations' is thus derived, and, thereby, homogeneity as well as the behavior of plural definites in non-monotonic contexts.

An important remark is in order. This model significantly departs from the use of lexical uncertainty in the RSA literature ([9, 1]), where there are as many S 1 's as there are interpretation functions, and it's only at L 1 that reasoning about lexical uncertainty takes place. In these models, the first pragmatic speaker can be viewed as picking an interpretation at random, and is thus less sophisticated than in the model we propose here. The first pragmatic listener then performs Bayesian inference jointly about both the speaker's beliefs about the world and the particular reading that the speaker picked. For instance, if one of the reading is very implausible due to the prior probability distribution, the first-level pragmatic listener will assign a low probability to the possibility that the speaker picked that reading. This type of model probably captures important facts about how prior probabilities play a role in disambiguation. The current proposal, on the contrary, cannot capture the role played by prior probabilities in disambiguation. In the next section, I introduce a model that includes questions under discussions, and where the listener, at each level, performs Bayesian inference about the identity of the QUD. This might in principle allow for an indirect influence of prior probabilities over worlds on disambiguation, in the following way. If the conveyed meaning of a sentence given a certain QUD is very unlikely (given the priors), the listener will infer that the underlying QUD is a different one, which will in turn have an effect on the interpretation of the sentence. I will not investigate, here, however, the extent to which this indirect mechanism can provide a satisfying theory of disambiguation.

Non-maximality, QUDs and Overinformativity.

In various works in the RSA framework, speaker utility is relativized to a QUD, which helps account for certain cases of non-literal readings ( [START_REF] Kao | Nonliteral understanding of number words[END_REF][START_REF] Kao | Formalizing the pragmatics of metaphor understanding[END_REF][START_REF] Lassiter | Adjectival vagueness in a bayesian model of interpretation[END_REF]). E.g., a speaker who is answering 'Are some windows open?' and believes that some but not all are, will receive a utility that is not infinitely negative from using the message All windows are open, because this message, though false, entails the correct answer to the QUD. Specifically, the utility of a a message u for a speaker who believes that the actual world is w is no longer determined by the probability that the listener will assign to w after interpreting u, but rather by the probability that the listener will assign to Q(w) after interpreting u, where Q(w) is the true answer to Q in w, i.e. Q(w) = {v|v ∼ Q w}. This move should thus help us account for the context-sensitivity of plural definites. However, it also carries the risk that all will too easily receive a non-universal construal, blurring the contrast between plural definites and quantifiers. In the relevant RSA works, there is uncertainty about the QUD (hence a probability distribution over QUDs), on the part of the listener. The speaker takes into account the listener's uncertainty about QUDs, and this restricts, to a certain degree, the extent of non-literal readings. The basic RSA-model with QUDs is as follows. It assumes that the prior probability distributions over worlds and QUDs are independent. 11. The interpretation function L, when applied to a message u and a world w, returns 1 if u is true in w, 0 otherwise.

2. Update rule for a literal listener who uses an interpretation function L: L 0 (w|u, L) ∝ P (w)L(u, w).

3. U n+1 (u|w, Q) = log(L n (Q(w)|u)) -c(u) = log( v∼ Q w L n (w|u)) -c(u) 4. S n+1 (u|w, Q) ∝ e λUn+1(u|w,Q) 5. L n+1 (w|u) ∝ P (w) Q P (Q).S n+1 (u|w, Q)
Importantly, in this model, the speaker does not care if the listener forms a false belief if this belief is orthogonal to the QUD. Various simulations I have run suggest that this model licenses too easily non-literal readings, in many different cases. For instance, I applied the model to the following case.

1. 3 worlds (no student came, just some of them came, all of them came).

2. Messages: Some (students came), All, Just some, No, Not all.

3. Possible QUDs: 'Did some of the students come?', 'Did all of the students come?', 'What's the case?' (= 'Did no, some but not all, or all of the students come?').

4. Message costs: 0 for Some, No, All, 1 for the others.

5. Flat priors on worlds.

6. Priors on QUDs: 0.8 for Did some of the students come?, 0.1 for the two others.

With λ = 5, The first-level listener (i.e. the first pragmatic listener) assigns a probability of about 0.4 to 'some but not all' after having interpreted all, and this probability increases with higher recursion-depth, converging to about 0.47. This appears to be a pretty bad result, since it does not seem to be the case that a sentence such as All the students came can be interpreted as being compatible with the possibility that some but not all students came, unless maybe the prior probability distribution over worlds makes it extremely unlikely that all the students came (in which case the sentence might receive a kind of hyperbolic interpretation, suggesting that many students came). 2 Increasing the value of λ does not change the general pattern (the probability of 'some but not all' for a listener who hears All remains unacceptably high).

Now, where could a pragmatic ban on overinformative sentences come from? Using All the students came to answer Did some of the students come? might be misleading in suggesting that the QUD one is answering is actually Did all students come?. In the above model, the speaker cares about this only instrumentally, i.e. only to the extent that the listener's wrong belief about what the QUD is might prevent her from identifying the intended meaning of the sentence. I propose to modify the QUD-model so that that the speaker cares not only about communicating

  ) a. I didn't open the windows. b. I didn't open all the windows.

  books a, b and c. a. Mary read the books b. {Mary read a ∨ b ∨ c, Mary read a ∨ b, Mary read a ∨ c, Mary read b ∨ c, Mary read a ∨ (b ∧ c), . . . , Mary read a, Mary read b, Mary read c, Mary read a ∧ b, Mary read a ∧ c, Mary read b ∧ c, Mary read a ∧ b ∧ c }

The independence assumption is simplistic and in general false, as it is quite clear that the prior distribution over worlds has an influence on which questions conversational participants might be interested in (cf.[START_REF] Van Rooij | Questioning to resolve decision problems[END_REF][START_REF] Hawkins | Questions and answers in dialogue[END_REF]).

We also need to take care of the potential confound of domain restrictions, which we can by considering sentences such as 'All the students who take my class came'.

* Thanks to Leon Bergen for useful discussions in relation with this work. The research reported in this work received support from the Agence Nationale de la Recherche (Grants ANR-10-LABX-0087 IEC, ANR-10-IDEX-0001-02 PSL, and ANR-14-CE30-0010-01 TriLogMean).

the true answer to the QUD, but also about communicating what QUD she is using. This amounts to replacing Eq. 3. above with U n+1 (u|w, Q) = log(L n (Q(w), Q|u)) -c(u) (where L n (Q(w), Q|u) is the joint probability assigned by the level-n listener to the proposition Q(w) and to the QUD being Q, having heard u). The level-n + 1 speaker is now modeled as wanting to maximize this joint probability. This gives rise to the following revised model:

1. The interpretation function L, when applied to a message u and a world w, returns 1 if u is true in w, 0 otherwise.

2. Update rule for a literal listener who uses an interpretation function L: L 0 (w, Q|u, L) ∝ P (Q)P (w)L(u, w).

Applied to the previous case, with the same parameters, this model yields a reasonable result with enough recursion-depth. The following table shows the probabilities (rounded to 2 decimal placed) assigned by the level-5 listener to worlds (=rows) depending on the message being interpreted (=columns).

Some

All Just Some No Not All No 0.00 0.00 0.00 1.00 0.50 Just Some 0.50 0.03 1.00 0.00 0.50 All 0.50 0.97 0.00 0.00 0.00

Even with a strong bias in favor of the question 'Did some of the students come?', the sentence All the students came ends up assigning a very high probability to the world where all the students came. With further iterations, or higher values for λ, the probability assigned to the 'some-but-not-all' worlds converges to 0.

This new model explains the ban on overinformative sentence in the following way: overinformative sentences can be misleading, even when they are literally true: they can mislead the listener's beliefs about which QUD the speaker is trying to answer.

Full proposal

Now that we have a mechanism to capture 'Truth on all interpretations' and the ban on overinformative sentences, we can put them together, which gives rise to the following model:

To apply the model to the pair in (4), we need to specify alternative messages. These alternative messages play no role for 'Truth on all interpretations', but they are necessary in order to make sure that overinformative meanings are not chosen, since the reason why an informative message is not chosen is that is has lower utility than other messages that would be less misleading regarding the nature of the QUD being answered. We also want to check that while the model allows for a flexible interpretation of plural definites, this is not so for quantifiers such as some or all. I assume that the books is less costly than some (of the) books and all (of the) books because, on top of being less complex when the quantifiers are partitive (some of the/all of the), it also has a less complex semantic type than quantifiers. I applied the above model to the following case:

1. 3 worlds as before 2. Messages: No, Some, All, Not All, The, Not The

2 interpretation functions, depending on whether

The is existential or universal 4. 3 QUDs: 'Is Some?', 'Is All?', 'What's the case?' 5. Cost: c(The) = 0<c(Not The)=c(Some)=c(All )=1<c(No)=1.5<c(Not All )=2.

6. Flat priors on worlds and interpretation functions.

7. 2 types of priors on QUDs: either 'What's the case' or 'Is Some?' has a probability of 0.8, and the two others 0.1. With a recursion depth of 5 (but for lower values as well) and λ = 5, the results appear to closely match intuitions. When 'What's the case?' has a prior probability of 0.8, the level-5 listener interprets messages as follows (values are rounded to the second digit):

Some

All No Not All The Not The No 0.00 0.00 1.00 0.50 0.00 0.98 Just Some 1.00 0.00 0.00 0.50 0.09 0.02 All 0.00 1.00 0.00 0.00 0.91 0.00 When 'Is some?' has probability 0.8, we get: Some All No Not All The Not The No 0.00 0.00 1.00 0.50 0.00 1.00 Just Some 1.00 0.00 0.00 0.50 0.50 0.00 All 0.00 1.00 0.00 0.00 0.50 0.00 So we capture the context sensitivity of sentences with plural definites, as well as homogeneity. When the salient question is 'What's the case?', the positive sentence in the pair (4) ('Mary read the books on the reading list') receives a universal interpretation, and its negative counterpart means that Mary read no books on the reading list. When the salient question is 'Did Mary read at least some of the books on the reading list?', the plural definite ends up equivalent to an existential quantifier (both for the positive and the negative sentence).

As to quantified sentences, they are not interpreted in such a flexible way, which was the desired outcome -somewhat surprisingly, some ends up meaning some but not all even when the salient QUD is 'Is Some?', possibly because, as discussed below, it also raises the posterior probability of the question 'What's the case?'.

Added Benefits

As an added benefit, this model is able to capture the fact that the interpretation of plural definites might appear somewhat vague. This is the case for instance if the priors over QUDs are flat. Keeping all other parameters constant, the level-5-listener then interprets (4a) as conveying that Mary read some of the books, and probably all of them (with probability 0.7).

This model also captures the intuition that utterances can 'raise new questions'. When the underlying question 'Is Some?' has a high prior probability, the messages Some, All and Not All result in the listener now believing that the speaker is answering another question. This is illustrated by the following table, which represents the posterior probability assigned by the level-5 listener to each QUD depending on which message was received, in the case where the QUD with the highest prior probability (0.8) is 'Is Some?'. Whether these specific predictions are empirically adequate is a matter for further research.

Some

All No Not All The Not The Is Some? 0.00 0.00 0.89 0.00 1.00 0.89 Is All? 0.00 0.50 0.00 1.00 0.00 0.00 What's the case?

1.00 0.50 0.11 0.00 0.00 0.11


Gilbert Badaro
email: gilbert.badaro@eurecom.fr

Mohammed Saeed
email: mohammed.saeed@eurecom.fr

Paolo Papotti
email: paolo.papotti@eurecom.fr

Transformers for Tabular Data Representation: A Survey of Models and Applications

In the last few years, the natural language processing community has witnessed advances in neural representations of free texts with transformer-based language models (LMs). Given the importance of knowledge available in tabular data, recent research efforts extend LMs by developing neural representations for structured data. In this work, we present a survey that analyzes these efforts. We first abstract the different systems according to a traditional machine learning pipeline in terms of training data, input representation, model training, and supported downstream tasks. For each aspect, we characterize and compare the proposed solutions. Finally, we discuss future work directions.

Introduction

Many researchers are studying how to represent tabular data with neural models for traditional and new natural language processing (NLP) and data management tasks. These models enable effective data-driven systems that go beyond the limits of traditional declarative specifications built around first order logic and SQL. Examples of tasks include answering queries expressed in natural language [START_REF] Katsogiannis | A deep dive into deep learning approaches for text-to-SQL systems[END_REF][START_REF] Herzig | TaPas: Weakly supervised table parsing via pre-training[END_REF]Liu et al., 2021a), performing fact-checking [START_REF] Chen | Tab-Fact: A large-scale dataset for table-based fact verification[END_REF][START_REF] Yang | Exploring decomposition for table-based fact verification[END_REF][START_REF] Aly | FEVER-OUS: Fact extraction and VERification over unstructured and structured information[END_REF], doing semantic parsing [START_REF] Yin | TaBERT: Pretraining for joint understanding of textual and tabular data[END_REF][START_REF] Gong | AST: Audio spectrogram transformer[END_REF], retrieving relevant tables [START_REF] Pan | CLTR: An endto-end, transformer-based system for cell-level table retrieval and table question answering[END_REF][START_REF] Kostić | Multi-modal retrieval of tables and texts using tri-encoder models[END_REF][START_REF] Glass | Capturing row and column semantics in transformer based question answering over tables[END_REF], understanding tables [START_REF] Suhara | Annotating columns with pre-trained language models[END_REF][START_REF] Du | TabularNet: A neural network architecture for understanding semantic structures of tabular data[END_REF], and predicting table content [START_REF] Deng | TURL: Table understanding through representation learning[END_REF][START_REF] Iida | TABBIE: Pretrained representations of tabular data[END_REF]. Indeed, tabular data contain an extensive amount of knowledge, necessary in a multitude of tasks, such as business [START_REF] Chabot | A framework for automatically interpreting tabular data at orange[END_REF] and medical operations [START_REF] Raghupathi | Big data analytics in healthcare: promise and potential[END_REF][START_REF] Dash | Big data in healthcare: management, analysis and future prospects[END_REF], hence, the importance of developing table representations. Given the success of transformers in developing pre-trained language models (LMs) [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | RoBERTa: A robustly optimized BERT pretraining approach[END_REF], we focus our analysis on the extension of this architecture for producing representations of tabular data. Such architectures, based on the attention mechanism, have proven to be successful as well on visual [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Khan | Transformers in vision: A survey[END_REF], audio [START_REF] Gong | AST: Audio spectrogram transformer[END_REF] and time series data [START_REF] Cholakov | Transformers predicting the future[END_REF]. Indeed, pre-trained LMs are very versatile, as demonstrated by the large number of tasks that practitioners solve by using these models with fine-tuning, such as improving Arabic opinion and emotion mining [START_REF] Wissam Antoun | AraBERT: Transformer-based model for arabic language understanding[END_REF][START_REF] Badaro | A large scale arabic sentiment lexicon for arabic opinion mining[END_REF]Badaro et al., , 2018cBadaro et al., ,a,b, 2019[START_REF] Badaro | A link prediction approach for accurately mapping a large-scale arabic lexical resource to english wordnet[END_REF]. Recent works show that a similar pre-training strategy leads to successful results when language models are developed for tabular data. 1As depicted in Figure 1, this survey covers both (1) the transformer-based encoder for pre-training neural representations of tabular data and (2) the target models that use the resulting LM to address downstream tasks. For (1), the training data consist of a large corpus of tables. Once the representation for this corpus has been learnt, it can be used in (2) for a target task on a given (unseen at pretraining) table, such as the population table, along with its context, i.e., relevant text information such as table header and captions. In most cases, the LM obtained from pre-training with large datasets in (1) is used in (2) by fine-tuning the LM with a labeled downstream dataset, e.g., a table with cells annotated as the answer for the question in the context. Extensions on the typical transformer architecture are applied to account for the tabular structure, which is different and richer in some as-Figure 1: The overall framework for developing and consuming neural representations for tabular data. Wikipedia Tables or WDC Web Table Corpus are typically used in the pre-training step (1). In (2), a table along with its context (header, the additional question and the highlighted answer) are used for a Question Answering downstream task. Both processes combine the serialized table data with natural language text in the context. pects than traditional free text.

While all proposed solutions make contributions to the neural representation of tabular data, there is still no systematic study to compare those representations given the different assumptions and target tasks. In this work, we aim to bring clarity in this space and to provide suitable axes for a classification that highlights the main trends and enables future work to clearly position new results. Our contributions are threefold:

1. We characterize the tasks of producing and consuming neural representations of tabular data in an abstract machine learning (ML) pipeline. This abstraction relies on five aspects that are applicable to all proposals and that are agnostic to methods' assumptions and final application (Section 2). 2. We describe and compare the relevant proposals according to the characteristics of the datasets (Section 3), the processing of the input (Section 4), and the adaptations of the transformer architecture to handle tabular data (Section 5). 3. To show the impact of the proposed solutions, we present six downstream tasks where they have achieved promising results (Section 6) and discuss how tabular language models are used by systems in practice (Section 7). We conclude the survey with a discussion of the limitations of existing works and future research directions (Section 8). Related surveys. Some recent surveys cover the use of deep learning on tabular data [START_REF] Borisov | Deep neural networks and tabular data: A survey[END_REF][START_REF] Gorishniy | Revisiting deep learning models for tabular data[END_REF][START_REF] Shwartz | Tabular data: Deep learning is not all you need[END_REF]. These surveys are different from ours in multiple aspects. First, they investigate how deep learning models (including non-transformer-based) compare against classical ML models (mainly gradient boosted trees) by providing empirical results with lack of details over the changes to the model, while our work taxonomizes transformer-based models in depth. Second, other works focus on standard classification and regression tasks where the term "tabular data" is used for labeled training data, representing data points with columns consisting of features. Every row is an input to the model together with a label (e.g., predicting house prices) [START_REF] Somepalli | SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training[END_REF]. In contrast, we do not consider tabular data necessarily as a training dataset with features and labels, but rather as an input containing information needed for prediction by a target model, such as question answering on tables. For this reason, while they focus on data generation and classification, we relate design choices and contributions to a large set of downstream tasks. A recent survey [START_REF] Dong | Table pre-training: A survey on model architectures, pre-training objectives, and downstream tasks[END_REF] covers approaches for table pre-training and usage of LMs in downstream tasks with an overlap in terms of surveyed works. In contrast with their work, we provide a more detailed study of the extensions to the transformer architecture and analyze in detail the relevant datasets' properties and pre-processing steps.

Terminology and Overview

We focus on tabular data that comes in the form of a table, as depicted in the two examples in Table 1. . A label 'Identity' on a top row spanning 'Player' and 'Team' would make the relational table a spreadsheet, similarly for the entity table with the same label spanning the first two rows.

tributes. This can be seen as a vertical orientation, where all cells in a column share the same atomic type, e.g., the relational table in Table 1. An entity table has the same properties, but with a horizontal structure with only one entity whose properties are organized as rows. Spreadsheets, or matrix tables, are the most general kind with information that can be organized both horizontally and vertically, possibly with hierarchies in the header and formatting metadata, such as in financial tables.

Tables can have rich metadata, such as attribute types (e.g., DATE), domain constraints, functional dependencies across columns and integrity constraints such as primary keys. In the table sample in Figure 1, column Country is a primary key. Most systems focus on a single table, with or without metadata. However, a few systems, such as GTR (Wang et al., 2021a), GRAPPA [START_REF] Gong | AST: Audio spectrogram transformer[END_REF], and DTR [START_REF] Herzig | Open domain question answering over tables via dense retrieval[END_REF], consume databases, which are collections of relational tables, possibly under referential constraints. We identify as input the table(s) and its context. The context is a text associated to the table. Depending on the dataset and task at hand, it varies from table metadata, the text surrounding the tables or their captions up to questions, expressed in natural language, that can be answered with the tabular data [START_REF] Badaro | Transformers for tabular data representation: a tutorial on models and applications[END_REF].

The main advantage of the transformer architecture is its ability to generate a language model (LM), a large neural network, with self-supervised pre-training. This pre-trained LM is then usually followed by supervised fine-tuning to adapt it to the target task with a small amount of training data. While transformers have proven to be effective in modeling textual content, tables have a rich structure that comes with its own relationships, such as those across values in the rows and attributes. New solutions are therefore needed to jointly model the characteristics of the table, its text content, and the text in the table context. As shown in Figure 1, we distinguish two main phases to spell out these contributions. First, we focus on the development of tabular LMs by using transformer-based deep neural networks (1). Given a table and its context, the goal is to learn a pre-trained representation of the structured data (cell values, rows, attributes) in a continuous vector space. We then discuss the use of those representations in the downstream tasks (2).

Figure 1 shows the reference pipeline and the aspects that we propose to model existing systems.

• Training Datasets (Sec. 3): the datasets used for pre-training and fine-tuning the models toward specific tasks; datasets for the latter case usually come with annotations and/or labels. • Input Processing (Sec. 4): the steps to prepare the data for the model processing, such as the transformation from the two dimensional tabular space to one dimensional input.

Training Datasets

We present both the datasets used for pre-training and for fine-tuning in the downstream tasks. Pretraining tables are not annotated, in some cases scraped from the web, while data used for finetuning have task-dependent annotation labels. The datasets consist of tables and their context, such as table metadata, surrounding texts, claims or questions. To construct large pre-training datasets and in an attempt to reduce bias, multiple sources can be used, independently of the target task at hand. For instance, unlike TAPAS [START_REF] Herzig | TaPas: Weakly supervised table parsing via pre-training[END_REF], that only uses Wikipedia Tables for QA, and TAB-ULARNET [START_REF] Du | TabularNet: A neural network architecture for understanding semantic structures of tabular data[END_REF] that only uses Spreadsheets for TMP, TABERT [START_REF] Yin | TaBERT: Pretraining for joint understanding of textual and tabular data[END_REF] uses Wikipedia Tables and WDC for SP; GRAPPA uses Wikipedia Tables, Spider and WikiSQL for SP; MMR [START_REF] Kostić | Multi-modal retrieval of tables and texts using tri-encoder models[END_REF] uses NQ, OTT-QA, and WikiSQL for TR; MATE [START_REF] Eisenschlos | MATE: Multi-view attention for table transformer efficiency[END_REF] uses Wikipedia Tables and HybridQA for QA; and TUTA (Wang et al., 2021b) uses Wikipedia Tables, WDC and spreadsheets for TMP. In general, it is recommended to utilize different data sources for pre-training to ensure covering different kinds and content, and thus, improve the scope of representations. For instance, Wikipedia tables has a large number of relational tables [START_REF] Sekhar Bhagavatula | TabEL: Entity linking in web tables[END_REF], while WDC and Spreadsheets include also entity tables and spreadsheets with complex structure.

Table 2 summarizes the main characteristics for the most common datasets. We mark the tasks for which the dataset has been used by ✔ under the column "Task". We note that the top four datasets are mostly used for pre-training, while the others can be used for fine-tuning as well since they include annotations for the target task, e.g., questions/answers for QA. The column "Large Tables" is a binary indicator, where ✔ and ✘ indicate whether or not the tabular corpus include large tables and hence whether or not some preprocessing is needed to reduce table content to meet the limits of the transformer architecture (512 input tokens in most cases). Some works, such as TABERT, TAPEX (Liu et al., 2021a) and CLTR [START_REF] Pan | CLTR: An endto-end, transformer-based system for cell-level table retrieval and table question answering[END_REF], apply filtering in any case to reduce noisy input. Finally, the "Context" column describes additional text that come with the tables. This can be text describing the

Input Processing

As for the original text setting, transformers for tabular data require as input a sequence of tokens. However, in addition to the typical tokenization executed before feeding the text to the neural network [START_REF] Lan | ALBERT: A lite BERT for self-supervised learning of language representations[END_REF], tabular data requires some steps to be processed correctly. Some requirements come from the nature of the transformer, such as the limitation on the input data size (Section 4.1). Other requirements are due to the nature of the tabular data, with structural information expressed in two dimensions that need to be converted into a one dimensional space (Section 4.2).

Finally, given a table, its data and its context must be jointly fed to the transformer (Section 4.3).

Data Retrieval and Filtering

Filtering methods on the table content are applied to stay within the size limits of the transformer architecture, to reduce the model training time, and to eliminate potential noise in the representation.

TABERT uses content snapshot to keep the topk most relevant rows in the table. Such content is identified with the tuples with highest n-gram overlap with respect to the given context (question or utterance). TAPEX and the retrieval model in FEVEROUS [START_REF] Aly | FEVER-OUS: Fact extraction and VERification over unstructured and structured information[END_REF] randomly select rows to limit the input size, while RCI [START_REF] Glass | Capturing row and column semantics in transformer based question answering over tables[END_REF] down-samples rows using term frequency inverse document frequency (TF-IDF) scores; the frequency can also be used to summarize cells with long text [START_REF] Li | Deep entity matching with pre-trained language models[END_REF]. In addition to keeping tables with number of columns below a fixed threshold, TUTA and TABULARNET split large tables into non-overlapping horizontal partitions. Every partition contains the same header row and it is processed separately by the model. While effective, splitting is more demanding in terms of computation cost.

In terms of table selection, while for systems like GTR and DTR the objective is to retrieve tables that contain the answer to a given question, others, such as CLTR, use a ranking function, such as BM25 [START_REF] Stephen E Robertson | Okapi at TREC-3[END_REF], to retrieve relevant tables prior to training, or to generate negative examples, as in MMR. Regardless of the downstream task, most systems filter and reduce the size of the input data to meet the limits of transformers technology. However, frequency-driven sampling, such as in RCI, is more effective than random as it reduces noise as well in data representations.

To summarize, one can group the different content selection strategies based on the targeted downstream task. For TFC, QA and SP, a selection strategy relying on n-gram overlap such as content snapshot or TF-IDF is recommended, while for TR, BM25 is endorsed. In the cases of TMP and TCP, where having the full content of the table is required, such as relation extraction or cell filling, splitting the tabular data without any selection is adopted.

Table Serialization

A crucial step is the transformation from the two dimensions of a table to its serialized version con-

FEVEROUS

Table 2: Datasets for the development and evaluation of neural representation models of tabular data. The top four datasets are mostly used for pre-training models, the rest of the datasets also come with a context and labels that are used in the downstream task for training and evaluation. For Large Tables, ✔/✘ denotes whether or not the dataset includes large tables and thus requiring pre-processing to meet transformers' limits. Application example refers to sample systems using the respective dataset. For the task, we use the acronyms as follows. TFC: or just the row of column headers (GRAPPA).

The second linearization type scans the table by column, again either by simple concatenation of column values or by using special tokens as separators, as in DODUO [START_REF] Suhara | Annotating columns with pre-trained language models[END_REF]. A column serialization for the country population table is the second example in Figure 2.

The third linearization type consists of combining the output from both types of serialization by using element-wise product (RCI, CLTR), average pooling and concatenation (TABULARNET), or the average of row and column embeddings (TAB-BIE [START_REF] Iida | TABBIE: Pretrained representations of tabular data[END_REF]). In a context outside of our framework, which focuses on transformers, it has also been proposed to transform the input relation table in a graph and perform random walks on the latter to ultimately produce node embeddings [START_REF] Cappuzzo | Creating embeddings of heterogeneous relational datasets for data integration tasks[END_REF].

The fourth type consists of using a text template to represent the tabular data as sentences [START_REF] Chen | Tab-Fact: A large-scale dataset for table-based fact verification[END_REF][START_REF] Hulliyyatus Suadaa | Towards table-to-text generation with numerical reasoning[END_REF]Chen et al., 2020a). Instead of using prede-fined templates, natural sentences are generated out of the tabular data, by fine-tuning sequence to sequence language models such as T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF] or GPT2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], as in DRT [START_REF] Thorne | Database reasoning over text[END_REF] and [START_REF] Neeraja | Incorporating external knowledge to enhance tabular reasoning[END_REF].

The generation can rely on models for this table-to-text task, such as TOTTO [START_REF] Parikh | ToTTo: A controlled table-to-text generation dataset[END_REF], PYTHIA [START_REF] Veltri | Pythia: Unsupervised generation of ambiguous textual claims from relational data[END_REF][START_REF] Veltri | Data ambiguity profiling for the generation of training examples[END_REF], TABLEGPT [START_REF] Gong | TableGPT: Few-shot table-to-text generation with table structure reconstruction and content matching[END_REF], LOGIC2TEXT (Chen et al., 2020d), UNIFIED-SKG [START_REF] Xie | UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models[END_REF], or other efforts [START_REF] Hulliyyatus Suadaa | Towards table-to-text generation with numerical reasoning[END_REF]Chen et al., 2020a,e). However, most of these methods show limitations for tables with prevalence of numerical attributes and with missing table context. Within the efforts for table-to-text, graph traversal algorithms have been explored for linearization such as relation-biased breadth first search [START_REF] Li | Few-shot knowledge graph-to-text generation with pretrained language models[END_REF]. While tableto-text generation is an important and challenging task, most methods rely on fine-tuning existing pre-trained language models.

While it is still not clear which table serialization method should be used for a given task, few papers performed ablation studies. For instance, TABFACT [START_REF] Chen | Tab-Fact: A large-scale dataset for table-based fact verification[END_REF] does not report any significant difference in performance when comparing row and column encoding. TABERT reports that both (i) adding type information for cell values and (ii) phrasing the input as a sentence improve results. The most promising approach is to incorporate several aspects by appending column headers to cell content, combining row and column encoding, or by adding structure aware indicators as the positional embeddings discussed in Section 5. Finally, there is evidence that textual templates to represent table content is a valid solution when one directly fine-tunes existing pretrained language models, without performing pretraining on tabular data [START_REF] Hulliyyatus Suadaa | Towards table-to-text generation with numerical reasoning[END_REF].

Context and Table Concatenation

When available, the table context is concatenated with the table content. Most systems (TABERT, TAPAS, DTR, GRAPPA, and DECO) combine the context by concatenating it in the serialization before the table data, while TAPEX appends it. The authors of UNIFIEDSKG found that placing context before the tabular knowledge yields to better results in their setting. In some cases, including CLTR, MMR, and RCI, the table and the context are encoded separately and then are combined at a later stage in the system. Some works do not include context in their pretraining input besides the column headers (TAB-BIE, TABULARNET, DODUO, GRAPPA). Typically, the decision is based on the target downstream task. A richer context is used when the tasks are closer to the corresponding NLP task applied on free text. For instance, all models for QA use table captions or descriptions as context. To summarize, including context is crucial for TFC, QA, SP, TR, while it can be excluded for TMP and TCP. In case the context is needed, prepending or appending it to the table content does not modify the performance of the model.

Adaptation of Transformers

To account for structured tables in the input, several pre-trained transformer-based language models (LMs) and systems have been developed. Vanilla LMs are customized to make the model more "data structure-aware", thus rendering a modified transformer-based encoder to be utilized on other tasks. These encoders, depicted in part (1) of Figure 1, capture both structure and semantic information. To exploit such resources and build applications, as in part (2) of Figure 1, several systems build on top of the encoder, usually with more modules and fine-tuning. In a different approach, other systems use the encoder as part of a bigger architecture in a more task-oriented fashion rather than encoder-oriented. We first briefly revise the vanilla transformer architecture, then discuss customizations to LMs.

Vanilla Transformer

The vanilla transformer [START_REF] Vaswani | Attention is all you need[END_REF]) is a seq2seq model [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] consisting of an encoder and a decoder, each of which is a stack of N identical modules. The encoder block is composed of a multi-head self-attention module and a position-wise feed-forward network. Multihead attention allows the model to jointly attend to information from different representation subspaces at different positions. Residual connections and layer-normalization modules are also used. Decoder blocks consist of cross-attention modules between the multi-head self-attention modules and the position-wise feed-forward networks, where masking is used to prevent each position from attending to subsequent positions.

The transformer architecture can be used as an encoder-decoder [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], an encoder-only [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | RoBERTa: A robustly optimized BERT pretraining approach[END_REF], or decoder-only [START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Tom | Language models are few-shot learners[END_REF] model. The choice of the architecture depends on the final task. Encoder-only models are mainly used for classification and are the most popular choice for extensions for tabular data. In this case, pretraining is done with a masked language modeling (MLM) task, whose goal is to predict masked token(s) of an altered input. The encoder-decoder architecture is used for models that focus on sequence generation tasks (RPT, TAPEX).

LM Extensions for Tabular Data

To properly model the structure of data in tables, the vanilla transformers are extended and updated by modifying components at the (i) input, (ii) internal, (iii) output, and (iv) training-procedure levels. We discuss each of them in the following. A summary of the extensions is provided as a taxonomy in Figure 3. We note that since the encoder and decoder modules of a transformer share similar structure, most of these modifications can be applied to both.

Input Level

Modifications on the input level are usually designated with additional positional embeddings to explicitly model the table structure. TABERT and TAPAS show how such embeddings improve performance on structure-related tasks. For example, embeddings that represent the position of the cell, indicated by its row and column IDs, are common for relational tables, e.g., in TABERT, TAPAS, and TABBIE. TABLEFORMER [START_REF] Yang | TableFormer: Robust transformer modeling for table-text encoding[END_REF] drops row and column ids to avoid any potential row and column biases and they instead use per cell positional embeddings similar to MATE. For tables without a relational structure, such as entity tables and spreadsheets, including complex financial tables, TUTA introduces tree-based positional embeddings to encode the position of a cell using top and left embeddings of a bi-dimensional coordinate tree. Other supplementary embeddings include those that provide relative positional information for a token within a caption/header (TURL) or a cell (TUTA). For tasks such as QA, segment embeddings are used to differentiate between the different input types, question and table, e.g., in RPT [START_REF] Tang | RPT: Relational pretrained transformer is almost all you need towards democratizing data preparation[END_REF] and TAPAS. Finally, TUTA introduces embeddings for numbers when discrete features are used. While row/column positional embeddings can better map context and table content, such as in QA or TFC tasks, it cannot overcome the challenge of empty cells, nested row headers, or descriptive cells in complex spreadsheets such as financial tables. In this case, tree-based positional embeddings are required. Aside from new positional embeddings encoding the structure of a token in a table, original, vanilla positional embeddings can also be modified for a better representation of tokens within table cells. For example, TABERT employs a pre-training task that explicitly uses positional embeddings, alongside a cell representation, for the goal of recovering a cell value. In this case, the original positional embeddings help better model multiple tokens in a cell.

Internal Level

Most of the modifications on the internal level are applied to make the system more "structureaware". Specifically, the attention module is updated to integrate the structure of the input table. For example, in TABERT vertical selfattention layers are produced to capture cross-row dependencies on cell values by performing the attention module in a vertical fashion. Empirical results in TUTA show that employing rowwise and column-wise attention, instead of having additional positional embeddings for rows and columns, hurts model performance for cell-type classification tasks, but it is not the case for tabletype classification tasks.

Other systems, such as TURL, employ a masked self-attention module, which attends to structurally related elements such as those in the same row or column, thus ignoring the other elements, unlike the traditional transformer where each element attends to all other elements in the sequence. Moreover, for categorical data, named entities, such as city names, can be identified in the cell values. In TURL and TUTA, masking such entities helps the models capture the factual knowledge embedded in the table content as well as the associations between table metadata and table content. Ablation studies showed the positive impact of these two modifications on model performance.

Other modifications address the input size constraint of attention modules, where large tables are often neglected. Sparse attention methods are proposed to cope with this issue [START_REF] Tay | Efficient transformers: A survey[END_REF]. For instance, MATE sparsifies the attention matrix to allow transformer heads to efficiently attend to either rows or columns.

Modifications to the input level, by additional input embeddings to encode cell positions (TAPAS), help in tasks requiring information at the cell level, such as QA and cell type classification. However, tasks requiring understanding of table structure, such as table type classification, benefit more from modifying the attention module (TABERT). Systems modifying both, such as TUTA, seem to be the best option. However, modifying the internal level is more effective, as removing such modification leads to the larger decrease in performance.

Output Level

Additional layers can be added on top of the feedforward networks (FFNs) of the LM depending on the task at hand. Tasks such as cell selection (TAPAS), TFC (TABFACT), TMP (DODUO), and QA (TAPAS) require to train one or more additional layers. Classification layers for aggregation operations and cell-selection are used to support aggregating cell values (GRAPPA, MATE, DODUO, DECO, RCI). In TAPEX, aggregation operations are also "learned" end-to-end in a seq2seq task.

Training-Procedure Level

Modifications on the training-procedure level can be attributed to the pre-training task and objective. Pre-training Tasks. Systems are either trained end-to-end or fine-tuned after some pre-training procedure. Almost all pre-training tasks fall under the category of reconstruction tasks, where the idea is to reconstruct the correct input from a corrupted one. Rather than applying traditional token-level MLM on the serialized tabular data, effectively treating it as natural language sequences, most pre-training tasks are designed to consider the information about the table structure.

Typically, traditional masking of the tokens is used for both, the textual context surrounding the table and the table cells (e.g., TAPAS 15%, TURL 20%) with some novelties for table cells such as masked columns (TABERT) and masked entities (TURL). Specifically, TAPAS follows the strategy to mask the whole cell table whenever a token in that cell is randomly masked. Inspired by TAPAS, TUTA masks 15% of the cells by randomly selecting those that consist mainly of text, rather than numerical values. Among these selected cells, 30% are masked completely, while for 70% only a single token within the cell is masked. Masking column names and data types for relational tables encourages the model to recover the metadata, while masking tokens and whole cell values ensure that data information is retained across the different vertical self-attention layers. Indeed, masking at the entity level enables the model to integrate the factual knowledge embedded in the table content and its context.

In TABBIE and RPT, the pre-training tasks detect if a cell/tuple is corrupted or not. While corruption has been shown to outperform MLM for standard textual LMs [START_REF] Clark | ELEC-TRA: pre-training text encoders as discriminators rather than generators[END_REF], it is not clear if the benefit generalizes to tabular setting. In TUTA, text headers of tables are used to learn representations of tables in a self-supervised manner, i.e., a binary classification task where a positive example is the table and its associated header, and a negative example is any other header.

While most systems pre-train for the purpose of learning initial tabular representations, performing aggregations and numerical operations is usually adapted by fine-tuning a classifier for predicting cells and operations, e.g., in TAPAS. However, systems such as GRAPPA and TAPEX, pre-train with SQL queries. In GRAPPA, the pre-training task enables the discovery of suitable fragments for SP by pre-training on a large number of queries, with SQL logic grounding to each table, which might help generalize to other similar queries. Empirical results show a decrease in performance without this pre-training task. TAPEX enforces the LM to emulate a SQL engine by pre-training on sampled tables and synthesized queries. This pre-training task gives (i) the LM a deeper understanding of the tables, as the system encounters operations such as selection and aggregation rendering its representations 'aware' of such operations, and (ii) improves the performance in QA.

According to [START_REF] Chang | Pretraining tasks for embedding-based large-scale retrieval[END_REF], a pre-training task should be cost-efficient, ideally not requiring additional human supervision. As manually annotating queries with tables is costly, using surrounding information, such as the table header or caption, as a query acts as a self-supervised signal, helping the system learn better representations for the tables. It is the case for TR systems, such as DTR and GTR, where performance improves when using a pre-training task rather than when training from scratch.

To summarize, token-level masking is the base for all the systems and it can be sufficient for TFC and TR tasks, while cell or entity masking are also recommended for QA, TMP, such as cell role classification, and TCP, such as cell filling. Column masking is additionally suggested for SP since it helps in identifying the columns to formulate the logical queries. Pre-training Objectives. The objective of the majority of the systems is to minimize crossentropy loss for a certain classification task. GTR utilizes a point-wise ranking objective for endto-end training after pre-training, where multiple tables are ranked according to a relevance score given a certain query.

Table 3 reports a summary of the transformer customizations adopted by every system. Overall, a tabular LM requires modifications to the input level, through additional embeddings, and to the internal level, through adjustments of the attention module. Pre-training on table-related tasks, such as masking or corrupting table cells, also enhances encoding capabilities for structured data on finetuned tasks. Modifications on the output level are more task specific and have less impact on making the LM "understand" the data structure.

Downstream Tasks

Using neural representations for tabular data show improvements in performance in several downstream tasks. In this section, we describe the tasks and define their input and output. While they all consume tables, settings can be quite heterogeneous, with systems exploiting different information, even for the same task. A summary of the covered tasks along their input, output, and some representative systems addressing them is shown in Table 4. We detail next the mandatory input elements and the different contexts.

Table based Fact-Checking (TFC):

Similar to text-based textual entailment [START_REF] Dagan | Recognizing Textual Entailment: Models and Applications[END_REF][START_REF] Daniel Z Korman | Defining textual entailment[END_REF], checking facts with tables consists of verifying if a textual input claim is true or false against a trusted database (TAPEX, DECO, TABFACT), also provided as input. Some factchecking systems, such as FEVEROUS, also output the cells used for verification as evidence [START_REF] Nakov | Automated factchecking for assisting human fact-checkers[END_REF][START_REF] Karagiannis | Scrutinizer: A mixed-initiative approach to largescale, data-driven claim verification[END_REF].

Question Answering (QA): In free text setting, QA aims at retrieving passages that include the answer to a given question. In the tabular data set- ting, it consists of returning as output the cells that answer an input consisting of a question and a table. One can distinguish two levels of complexity. Simple QA involves lookup queries on tables (DTR, CLTR), while a more complex QA task involves aggregation operations and numerical reasoning (TAPAS, MATE, RCI, DRT, TAPEX). Most of the systems in this survey aim at improving accuracy in QA w.r.t. hand-crafted embeddings.

Semantic Parsing (SP): In the tabular data setting, given a question and a table as input, SP generates a declarative query in SQL over the table's schema to retrieve the answer to the question. While in QA the interest is in directly getting the answer, SP produces the (interpretable) query to obtain it (TABERT, GRAPPA, TAPEX).

Table Retrieval (TR):

Given a question and a set of tables as inputs, TR identifies the table that can be used to answer the question. TR is helpful when trying to reduce the search space for a QA task (GTR, DTR, MMR). It is a challenging task given the limited input size of transformers, i.e., their constraint to sequences of 512 tokens. We observe that most tasks can be seen as traditional NLP problems where structured data replace free text, such as the case of QA where answers are located in tabular data instead of documents [START_REF] Gupta | A survey of text question answering techniques[END_REF]. TFC involves retrieving cells that entail or refute a given statement, whereas on free text the corresponding objective is to select sentences as evidence [START_REF] Thorne | FEVER: a large-scale dataset for fact extraction and verification[END_REF]. SP is the task of converting natural language utterance into a logical form [START_REF] Berant | Semantic parsing via paraphrasing[END_REF], which in this setting is expressed as a declarative query over a relational table. TR on tabular data corresponds to passage retrieval on free text [START_REF] Kaszkiel | Passage retrieval revisited[END_REF]. TCP is analogous to predicting missing words or values in a sentence [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF]. Finally, TMP can be related to syntactic parsing in NLP [START_REF] Roger | Syntactic parsing[END_REF], where relationships between different tokens are depicted.

We conclude this section with an analysis of the performance of the systems over the different downstream tasks. For every task, we selected datasets for which at least two systems have reported results. All datasets reported in Table 5 are described in Table 2, with the exception of WCC [START_REF] Ghasemi | TabVec: Table vectors for classification of web tables[END_REF], which contains Web tables annotated with their type (relational, entity, matrix, list and non-data), and EntiTab [START_REF] Zhang | EntiTables: Smart assistance for entity-focused tables[END_REF], which contains Web tables annotated with possible header labels for the column population task. Table 5 also contains the size, expressed as number of parameters, of the largest model used by every system. As some systems are not comparable on any shared datasets, we report here their size: TAB-FACT (110M), MMR (87M), TURL (314M), RPT (139M) and CLTR (235M). Bigger models do not correlate with better performance across different systems for the same task, but a larger model always brings higher scores for the same system, as expected. Execution times for training and testing depend on the size of the model and the computing architecture.

The results in the table show that some tasks, such as TFC and TMP, can already be handled successfully by the systems, while some tasks, such as TCP, TR and SP, are harder. QA is the task supported by most systems and the quality of the results vary depending on the dataset at hand. Differences in performance can be explained with different improvements across the systems. For example, MATE has better performance w.r.t. TAPAS in two tasks because of its mechanism to deal with larger input tables. Similarly, TUTA improves over TABERT because it handles tabular data beyond relational tables. Finally, TABERT and TAPAS are the systems that show most coverage in terms of tasks, with multiple papers using them as baselines in their experiments. For the systems that are not reported in Table 5, we notice that TURL obtains similar F1 results for column type prediction, but on a dataset different from VizNet; TURL, TABBIE and TABERT also report comparable MAP results for the row population task (not in Table 5) over different datasets.

Using the Language Model

The initial neural representations of tabular data are the result of the pre-training. Systems use the output LM in different ways. It can be fine-tuned or used "as-is", for example by using its representations as features in traditional ML algorithms (Section 7.1). However, as pre-trained LMs can act as encoders of the input, they are also used as a module in bigger systems (Section 7.2).

Encoder Output and its Usage

The alternative systems expose different granularity of the output table representation as embeddings. Almost all systems provide token and cell output embeddings. Most of them also expose column and row embeddings, while few works provide table and pairwise column (DODUO) or pairwise table (DECO) embeddings. When a special separator token separates the context and the table content, a representation of the context is also provided (TABFACT, CLTR, DTR, MMR).

These representations can be used in an arbitrary ML program simply as features or directly by the systems creating them to tackle a given task (Section 6). Indeed, as we discussed in Section 5.2.3, most systems use the pre-trained embeddings for further fine-tuning to tackle the tasks. We observe a relationship between the granularity of the output representations and the target downstream tasks. For instance, table representations are used for the TR task, while column-pairs representations are used for the TMP task to support columns relation annotation. Cell representations are used for the QA task, since the cells including the answer should be returned. Finally, column representations are used for the SP task, as columns are needed to formulate the output query.

Most pre-trained models, such as TAPAS, are usually available out-of-the-box, while in some cases, such as in TURL, MATE and CLTR, the users have to retrain the models. For such systems, the code and the dataset are available, but the users need to run the training on their side to generate the representations (no checkpoints available).

Encoder as a Module

Several systems, such as TUTA and TURL, add layers on top of the LMs, which are then finetuned for a task. While this is a common use of pre-trained LMs, other works employ LMs as components in a larger system. In these cases, the system not only needs to learn how to encode properly, but also to adjust its representations to a certain task by training end-to-end together multiple components, such as the ones that (individually) generate the embeddings for tables and text with the one for scoring similarity of textual and tabular embeddings.

Most of these larger systems focus on the retrieval of tables from an input natural language query. DTR answers a natural language question from a corpus of tables in two steps. First, it retrieves a small set of candidate tables (TR), where encoding of questions and tables are learned through similarity learning. The similarity score is obtained through an inner product between questions and table embeddings. Then, it performs the standard answer prediction (QA) with each candidate table in the input. A recent study [START_REF] Wang | Table retrieval may not necessitate table-specific model design[END_REF] claims that table-specific models may not be needed for accurate table retrieval and that fine-tuning a general text-based retriever leads to superior results compared to DTR. MMR studies a multi-modal version of DTR using both tables and text passages by proposing several biencoders and tri-encoders. Similarly, CLTR introduces an end-to-end system for QA over a table corpus, where the retrieval of candidate tables is performed by a coarse-grained BM25 module, followed by a transformer-based model that concatenates the question with the row/column and classifies whether the associated row/column contains the answer. Other systems, as GTR, support retrieval of tables, where tables are represented by graphs. In this setting, stacked layers of a variant of Graph Transformers [START_REF] Koncel-Kedziorski | Text generation from knowledge graphs with graph transformers[END_REF] are employed for obtaining node features that are combined with query embeddings. These combined embeddings are then aggregated with the BERT embeddings of the table context and query, and a relevance score is finally obtained.

Future Directions

Tabular LMs effectively address some of the challenges that arise with classical ML models [START_REF] Borisov | Deep neural networks and tabular data: A survey[END_REF], such as transfer learning and self-supervision. However, several challenges remain unaddressed.

Interpretability. Only few systems expose a justification of their model output, e.g., TAPAS, CLTR, MATE, thus model usage remains a black box. One direction is to use the attention mechanism to derive interpretations [START_REF] Serrano | Is attention interpretable[END_REF][START_REF] Dong | Attention is not all you need: Pure attention loses rank doubly exponentially with depth[END_REF]. Looking at self-attention weights of particular layers and layer-wise propagation w.r.t. the input tokens, we can capture the influence of each cell value/tuple on the output through back-propagation [START_REF] Huang | ClinicalBERT: Modeling clinical notes and predicting hospital readmission[END_REF]. For instance, in TFC, providing explanation is crucial when the decision is derived from aggregating several cells, such as sum or average operation. In this case, a basic explanation would be to show all the cells that led to the final true/false decision.

Error Analysis. Most works focus on the downstream evaluation scores rather than going through manual evaluation of errors. In the downstream task level, this analysis could trace back misclassified examples to get evidence of issues in the tabular data representation. For example, in a QA task with wrong answers, there could be a pattern that explains how these errors are all due to the sys-tem confusing two columns with similar meaning (e.g., max value and last value for a stock), thus returning the wrong cell value in several cases.

Complex Queries & Rich Tables. Several works, such as TAPEX and MATE, handle queries with aggregations by adding classification layers. However, such methods fail short with queries that join tables. As most works assume a single table as input, a much needed direction is to develop models that handle multiple tables, e.g., with classification layers predicting when a join is required. Also, as tables might contain heterogeneous types and content, such as non-uniform units (e.g., kg and lbs), systems should be able to handle such differences, which are abundant in practice. Moreover, an interesting direction is to conduct a study to show where tabular LMs can be successfully applied and where they fail in querying data.

Model

Efficiency. Transformer-based approaches are computationally expensive and some approaches try to approximate the costly attention mechanism by using locality-sensitive hashing to replace it [START_REF] Kitaev | Reformer: The efficient transformer[END_REF], approximating it by a low-rank matrix [START_REF] Wang | Linformer: Selfattention with linear complexity[END_REF], or applying kernels to avoid its computational complexity [START_REF] Katharopoulos | Transformers are RNNs: Fast autoregressive transformers with linear attention[END_REF][START_REF] Choromanski | Masked language modeling for proteins via linearly scalable long-context transformers[END_REF]. While there exist methods to make transformers more efficient for long context [START_REF] Tay | Efficient transformers: A survey[END_REF], all optimizations consider an unstructured textual input. We believe more traction is needed for efficient transformers on structured data, with ideas from the textual counterpart, such as limiting attention heads to rows/columns, which can be obtained naturally from the structured input [START_REF] Eisenschlos | MATE: Multi-view attention for table transformer efficiency[END_REF], or exploring prompt learning and delta tuning [START_REF] Ding | Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models[END_REF].

Benchmarking Data Representations. There are no common benchmark datasets where researchers can assess and compare the quality of their data representations in a level playing field. Current evaluation is extrinsic, i.e., at the downstream task level, where each work has its own assumptions. Intrinsic methods to evaluate the quality of the representations, such as those for word embeddings [START_REF] Bakarov | A survey of word embeddings evaluation methods[END_REF], can include predicting table caption given table representation or identifying functional dependencies. A set of precise tests can be designed to assess data-specific properties, such as the ability of the transformer-based models, designed to model sequences, to capture that row and attributes are sets in tables. Also, it is not clear whether the model representations are consistent with the table structure, effectively capturing their relationships. For example, given two cell values in the same row/column, are their embeddings closer than values coming from different rows/columns? For this, following the lines of CheckList for text LMs (Ribeiro et al., 2020), basic tests should be designed to measure the consistency of the data representation.

Data Bias. It is recognized that LMs incorporate bias in the model parameters in terms of stereotypes, race and gender [START_REF] Nadeem | StereoSet: Measuring stereotypical bias in pretrained language models[END_REF][START_REF] Vig | Investigating gender bias in language models using causal mediation analysis[END_REF]. The bias is implicitly derived from the training corpora used to develop LMs. Therefore, there is a need to develop methods to overcome this drawback by, for instance, pre-filtering the training data or by correcting the tabular LMs, similarly to the ongoing efforts for text LMs (Liu et al., 2021b;[START_REF] Bordia | Identifying and reducing gender bias in wordlevel language models[END_REF].

Green LMs. The use of large-scale transformers for learning LMs requires a lot of computation, which contributes to global warming [START_REF] Strubell | Energy and policy considerations for modern deep learning research[END_REF][START_REF] Schwartz | Green AI[END_REF]. Therefore, it is important to consider enhancements or potentially new techniques that limit the carbon footprint of tabular language models without a significant decrease in the performance in the downstream tasks. One enhancement can be at the level of the size of the training data by removing redundant, or less informative, tuples and tables. How to identify such data is a key challenge.

Conclusion

We conducted a survey on the efforts in developing transformer-based representations for tabular data. We introduced a high level framework to categorize those efforts and characterized each step in terms of solutions to model structured data, with special attention to the extensions to the transformer architecture. As future work, we envision a generic system to perform an experimental study based on our framework. The first part of the system would develop tabular data representations with alternative design choices, while the second part would evaluate them in downstream tasks. This work would help identifying the impact of alternative techniques on the performance in the final applications.

  Table kinds differ in terms of horizontal/vertical orientation and w.r.t. the presence of hierarchies. A relational table, which is the most common kind, consists of rows, or records, and columns that together identify cell values, or cells. Columns represent attributes for a given table, and each row represents an instance having those at-

Table 1 :

 1 Examples of (vertical) relational table and (horizontal) entity table

table, such as caption or title of the document containing the table; table metadata, such as table orientation, header row, and keys; or questions and claims that can be addressed with the table.

Table 4 :

 4 List of tasks utilizing neural representations for tables.

	Task	Task Label	Tasks Coverage	Input	Output	System Ex-
	ID						amples
	TFC Table-based Fact-Checking	Fact-Checking Text Refusal/Entailment	Table + Claim	True/False Refused/Entailed (Data Evidence)	DECO TABFACT TAPEX
							TAPAS
	QA Question Answer-	Retrieving the Cells for the	Table + Question	Answer Cells	MATE
		ing		Answer			DTR
							TABERT
	SP	Semantic Parsing Text-to-SQL	Table + NL Query	Formal QL	GRAPPA
							TAPEX
	TR	Table Retrieval	Retrieving Table that Con-tains the Answer	Tables + Question	Relevant Table(s)	GTR CLTR
				Column Type Prediction		Column Types
				Table Type Classification		Table Types	DODUO
	TMP Table Prediction Metadata	Header Detection Cell Role Classification Column Relation Annotation	Table	Header Row Cell Role Relation between Two Cols	TABULARNET TURL TUTA
				Column Name Prediction		Column Name
							TURL
	TCP Table	Content	Cell Content Population	Table with Corrupted	Table with Complete Cell	TABBIE
		Population			Cell Values	Values	RPT

Table Metadata

 Metadata

Prediction (TMP): Given an input table with corrupted or missing metadata, the TMP objective is to predict inter-table metadata, such as column types and headers, cell types, table types, and intra-tables relationships, such as equivalence between columns and entity link-ing/resolution. Relevant efforts focus both on spreadsheets (TUTA, TABULARNET) and relational tables (TURL, DODUO).

Table 5 :

 5 Performance table of results reported from the original system papers for every downstream task and different datasets. Metrics (accuracy, F1 measure, MAP) change across datasets. Results reproduced by follow-up papers are denoted with *. Reported sizes are for the largest model of every system.

		TFC		QA		SP	TR	TMP	TCP
	System (size)	TabFact HybridQA WikiTQ WikiSQL Spider WQT WCC VizNet EntiTab
		accu.	accu./F1	accu.	accu.	MAP MAP	F1	F1	MAP
	TABERT (367M)			53.5*	70.9*	65.2	63.4 83.6* 97.2*	33.1*
	TAPAS (340M)	81.2*	62.7/70.0	48.8	86.4
	MATE (340M)	81.4	62.8/70.2	51.5
	TAPEX (406M)	84.2		57.5	89.5
	TABBIE (170M)							96.9	37.9
	RCI (235M)				89.8
	GRAPPA (355M)			52.1*		69.6
	DODUO (110M)							94.3
	DECO (1454 M)	82.7
	TUTA (134M)							87.6
	GTR (117M)						73.7

We refer to tabular "language" models with a slight abuse of the name, as the LM captures properties and relationships of the structured data rather than those of the language.

Ro Co Tr Nu Fr Vrt Sp Tr Vi Ro Co Tb Ce To CP NliSp Ag To Ce Co TC Tu Ce Nse CE PR TUTA X X X X X X X X X TURL X X X X X X TABERT X X X X X X X X TABBIE X X X X X X X MATE X X X X X X X X X X X RCI X X GRAPPA X X X TAPEX X X X X DODUO X X X TABFACT X X X X RPT X X X X X X TAPAS X X X X X X X X GTR X X XTable3: Transformer customizations for every system. The acronyms from left to right follow the same order of the leaves of the taxonomy tree in Figure3from top to bottom.

Acknowledgment

We would like to thank the action editors and the reviewers for their valuable inputs that helped in further improving the content and the presentation of our article. This work has been partially supported by the ANR project ATTENTION (ANR-21-CE23-0037) and by gifts from Google.