Hélène Ledouble
email: ledouble@univ-tln.fr

Emmanuel Marty Abtract

GRESEC Emmanuel Marty
email: emmanuel.marty@univ-grenoble-alpes.fr

The 2016 presidential primaries in the United States: a quantitative and qualitative approach to media coverage

Keywords: American presidential election, press coverage, media frames, lexicometrics, lexical cooccurrence, themes, semantics, morphosyntax, Trump

. Several semi-automated operations, based on lexical co-occurrence (notably hierarchical descendant classification and factorial analysis) enable a re-construction of semantic cues (Mayaffre 2014), revealing ideological postures, argumentative strategies and thematic disparities between the different newspapers. In the paper, we address the subjective interpretation schemes inherent to the way each media outlet presents the different candidates depending on their communication contract, with a final focus on the candidate Donald Trump.

Introduction

This article falls within the framework of a research project based on the study of presidential elections in the United States and France as seen by the media. Using the theories of media frames and their application to discourse analysis [START_REF] Gitlin | The whole world is watching[END_REF][START_REF] Entman | Media framing biases and political power: Explaining slant in news of campaign 2008[END_REF], our objective is to characterise editorial strategies through discursive contents in the coverage of primary elections.

In this study, we focus on the 2016 primary elections in the United States, covered by six national papers between January and March 2016. Our corpus of almost three million words is analysed using lexical statistics: several semi-automated operations based on lexical co-occurrence are implemented with a view to revealing ideological postures and thematic differences between different newspapers. We simultaneously address the subjective interpretation schemes used by each media outlet, depending on their editorial identities, as they introduce the different candidates.

Following the presentation of the contextual and theoretical background in the second part of our study, we will delineate the characteristics of our corpus and the methodological approach used in the investigation. The statistical instruments presented in the third section will precede the corpus analysis itself. In the fourth section, an introductory perspective on the different sources composing our corpus will lead us towards a more comprehensive approach of its thematic content.

The study finally applies a complementary semantic and morphosyntactic method for a deeper investigation of thematic issues, culminating in a focus on the candidate Donald Trump.

Context and research questions

The 2016 presidential elections in the United States received intense media coverage. This coverage started long before the candidate Donald Trump was elected: during the primary period, from January to June 2016, the media began to reveal the agendas and personalities of both the Democrat and Republican candidates.

The role of the media in presidential elections has been an inexhaustible source of controversy for academics worldwide. The question of media influence on people's choices, addressed by Lazarsfeld et al.'s pioneering work [START_REF] Lazarsfeld | The people's choice: How the voter makes up his mind in a presidential campaign[END_REF], is the core of this controversy. To tackle the issue, we now have strong indicators, inherited from media frame theories [START_REF] Entman | Framing: Toward clarification of a fractured paradigm[END_REF], to determine some of the processes by which the press coverage of an election may highlight certain issues or aspects related to the campaign or to the different candidates.

Our objective in this article is to identify the discursive and lexical content of media frames, as this constitutes the raw material in the perception of election issues, actors and positions. In 'media frame theories' and in their methodological application (frame mapping), discursive content and strategies (even ideologies) are grasped by identifying and measuring lexical co-occurrence. We are therefore interested in lexis as an indicator of discourse strategies (see section 3.2. for methodological explanations). By setting this goal, we also address the following questions: are those frames spread homogeneously across different newspapers, or can we identify differences between them? If so, what kind of political or editorial logic might structure the apparent disparities? And finally, are there specific discursive characteristics in news reports concerning the presidential candidate Donald Trump? To answer these questions, we base our corpus analysis on media frame theories, considered as a heuristic and operational concept for Critical Discourse Analysis (CDA) of media coverage ( [START_REF] Dijk | Principles of critical discourse analysis[END_REF].

Media frames constitute both a theoretical paradigm of social co-construction of reality, in the wake of Goffman's concept of frame (1991), and a heuristic set of methods and tools for media discourse analysis (see section 3.2 for the specific method, and section 4 for its application to our corpus). As far as the theoretical aspects are concerned, the transposition of Goffman's frame analysis to media discourse has been performed by Gitlin, for whom 'Media frames, largely unspoken and unacknowledged, organise the world both for journalists who report it and, to a large degree, for those who rely on their reports' (Gitlin 1980: 8). This principle corresponds to what Gamson and Modigliani (1989: 4) call 'interpretive packages', which 'have the task of constructing meaning over time, incorporating news events into their interpretive frames'.

Since their work, Entman has developed a definition of framing as a process of -necessarily biased -meaning construction, which seems to be widely accepted by academics. According to him, [t]o frame is to select some aspects of a perceived reality and make them more salient in a communicating text, in such a way as to promote a particular problem definition, causal interpretation, moral evaluation, and/or treatment recommendation for the item described. (Entman 1993: 52) Furthermore, a dichotomous definition of frames has been developed by Iyengar (1991: 2), opposing episodic and thematic framing: episodic framing focuses on 'specific events or particular cases' to illustrate an issue, while thematic framing 'places political issues and events in some general context', in order to question them. In the coverage of an election, Iyengar highlights the fact that the media strongly favour episodic frames over thematic ones, by overusing what he calls 'horse race' framing, that is, devoting most of their reports to the succession of polls, alliances and strategies, to the detriment of political and social issues (Iyengar 1991: 133-134).

However, not all newspapers use the same frames and interpretive packages. Indeed, their editorial strategies in the coverage of different issues are also determined by their political values, socio-economic and editorial constraints, all of these being linguistically encoded in media discourse in order to be appropriately decoded by targeted audiences. Consequently, we may observe more or less significant differences in the coverage of primary elections by the media, including their use of frames: linguistic and semantic routines that characterise their discourse.

Corpus material and methodology

Corpus presentation

Our corpus comprises 3,151 articles published by six national newspapers in the United States (The New York Times, The Washington Post, USA Today, New York Daily News, The Wall Street Journal, New York Post), from January 25 to March 25, 2016. For that period, we extracted all articles containing the words primary, primaries, caucus, caucuses, totalling 2,898,035 words from 3,151 articles.1 Then, we proceeded to a Descending Hierarchical Classification (DHC, see 3.2.2 for further explanation) on the whole corpus. This exploratory analysis produced results which were lexically very heterogeneous, due to the choice of keywords used for corpus extraction (notably primary/primaries). Some articles included lexical material related to other contexts (mostly sports-related), their lexicon being confined to one specific class by the DHC. Text segments constituting this class were consequently removed from the analysis, to focus on the proper context of primary, that is, the American presidential election, in a new and more focused sub-corpus. The new sub-corpus totals 1,980,221 words from 3,117 articles. This new corpus can be broken down into the following parts: Each of these newspapers focuses on a specific audience and has its own editorial identity. They all relate in different ways to the Conservative or Liberal perspective, depending on this identity and political stance. We aim to investigate this position further in an unbiased manner by studying their divergences or convergences in terms of lexical content, using statistical methods, which we present below.

Methods for textual analysis

Statistical methods are here applied within the CDA theoretical framework. Therefore statistical indexes of word frequency give information on language use in a given communicative situation. As far as the methodological aspects are concerned, two successive operations are central to textual data analysis carried out by any textual statistics application: tokenisation and partition. Tokenisation consists of cutting the text into minimal lexical units, tokens. Identical tokens are then gathered as 'word types' in an index and their occurrences are counted. Partition consists of splitting and gathering texts (here, newspaper articles) according to extra-textual characteristics (in this case, the newspaper that published the article), enabling contrastive analyses based on the different parts of a whole corpus. Thus, textual statistics applications generate a double-entry table which associates lexical forms (rows) with partitioned texts (columns). Most statistical processes are then implemented based on this table, to determine whether the potential imbalances between the different parts are statistically significant [START_REF] Lebart | Statistique textuelle[END_REF].

In this section, we present different approaches to CDA, most using statistical tools -lexical specificity, Correspondence Factor Analysis (CFA) and Descending Hierarchical Classification (DHC) -with a view to revealing how the use of these different methods can assist semantic interpretation of a dense corpus.

Lexical specificity and Correspondence Factor Analysis

According to [START_REF] Lebart | Statistique textuelle[END_REF] the field of textual statistics is perfectly adapted to discourse analysis and the study of argumentative strategies, ideological postures being potentially conveyed by the repetition of words and phrases. A statistical index named 'lexical specificity index', based on the hypergeometric model [START_REF] Lafon | Sur la variabilité de la fréquence des formes dans un corpus[END_REF], identifies significant imbalances in the distribution of words between parts of a whole corpus, by measuring the difference between a theoretical (balanced) distribution of lexical units (as if the extra-textual characteristics had no impact on their position) and their actual distribution. 3 A heuristic way to represent these lexical disparities, then, is the Correspondence Factor Analysis developed by Benzecri (1973). 'Correspondence analysis is a method that gives a geometrical representation of the associations between two sets of elements in correspondence as they appear in a table' (Beaudouin 2016: 8), here a contingency table crossing lexis (in row) and partitions (in columns). 'Correspondence analysis uses a Euclidean space and a distributional distance, or the chi-square distance, which is a distinctive feature of correspondence analysis' (Beaudouin 2016: 9). The CFA graph thus displays lexical distance or proximity between both lexical forms and corpus partitions, spatially.

Descending Hierarchical Classification

The Descending Hierarchical Classification (DHC), developed by [START_REF] Reinert | Une méthode de classification descendante hiérarchique: Application à l'analyse lexicale par contexte[END_REF] and implemented in the Iramuteq software [START_REF] Ratinaud | IRaMuTeQ: Implémentation de la méthode ALCESTE d'analyse de texte dans un logiciel libre[END_REF] splits the text into segments of equal length (about 40 words)4 and codifies the presence or absence of each form within each segment so as to gather those containing the same words. The common lexical units that co-occur within segments form different lexical classes, and the significance of their affiliations to those classes are expressed by a χ 2 score. [START_REF] Reinert | Mondes lexicaux stabilisés et analyse statistique de discours. 9es Journées internationales d'Analyse statistique des Données Textuelles[END_REF] calls those classes 'lexical universes': they function as mental and linguistic spaces, closely linked to social representations [START_REF] Ratinaud | Des mondes lexicaux aux représentations sociales. Une première approche des thématiques dans les débats à l'Assemblée nationale (1998-2014)[END_REF], within which speakers situate their discourse. This unsupervised approach enables the semantic reconstruction of themes: according to [START_REF] Mayaffre | De l'occurrence à l'isotopie. Les co-occurrences en lexicométrie[END_REF] co-occurential pairs can be considered as 'semantic molecules', a statistically recurring co-presence of forms involving their semantic correlation. Therefore multiple co-occurrences help to disambiguate potentially polysemous words by constructing the previously mentioned lexical universe around them. Identifying the semantic correlation between forms on the basis of lexical co-occurrence constitutes the operational side of media frames analysis, often referred to as 'frame mapping' [START_REF] Miller | Frame mapping and analysis of news coverage of contentious issues[END_REF].

Additionally, in order to apprehend the linguistic complexity in the expression of emerging themes through textual analysis, we used the TXM platform [START_REF] Heiden | TXM: Une plateforme logicielle open-source pour la textométrieconception et développement[END_REF]. Its morphosyntactic approach facilitates the identification of the different patterns of information related to a specific element (see 3.3). This syntagmatic and paradigmatic method will prove to be highly relevant for the in-depth study of what we formally characterise as themes (Ben Hamed & Mayaffre 2015).

Similarity analysis

The goal of similarity analysis is to identify co-occurrence between words, providing information on their individual connections within the larger textual structure. Such an approach is conventionally used to describe and visualise social representations [START_REF] Ratinaud | Des mondes lexicaux aux représentations sociales. Une première approche des thématiques dans les débats à l'Assemblée nationale (1998-2014)[END_REF]. After calculation of the contingency coefficient, which is a classic similarity index [START_REF] Flament | L'analyse de similitude, une technique pour les recherches sur les représentations sociales[END_REF], the method helps to reveal the organization of textual data, in the form of a 'maximum tree',5 presenting a fine outline of how to go from one element to the other.

By way of a synthesis, Lexical specificity and Correspondence Factor Analysis are used for the contrastive analysis between newspapers, whereas DHC and similarity analysis are devoted to the process of 'frame mapping' [START_REF] Matthes | The content analysis of media frames: Toward improving reliability and validity[END_REF]. Based on this material and these multiple methods for our study, we now turn to the results of our analyses.

Corpus analysis

General lexical perspective

Using the statistical index of lexical specificity on this sub-corpus, the Correspondence Factor Analysis graph (Figure 1) represents the lexical proximity (or distance) between sources. Significant imbalances can be observed in the distribution of words between the different newspapers as the graph below shows: This can be seen as materialising general differences between the different papers, based on the contribution of each lexical form to a specific source of information. In order to explore this question more deeply and identify specific discursive or lexical components, we now turn to Descending Hierarchical Classification (DHC). Based on lexical co-occurrence, the following class distribution is the outcome of a DHC on our sub-corpus. As illustrated in Figure 2, we can distinguish three clusters (three different branches) at the top of the graph, gathering classes presenting a similar lexical content: the first cluster (classes 1, 3, 4), the second cluster (classes 2, 5, 6) and the final one (classes 7, 8, 9). The previously mentioned dissimilarity between episodic and thematic framing is noticeable in this perspective. The first two clusters, on the left hand side of the dendrogram, mostly involve episodic framing, favoring the coverage of polls, candidates' strategies and results before and after each state primary or caucus (Rubio, Bush, Cruz, rally, attend, event, etc.). This is referred to as a 'horse-race' framing type of information [START_REF] Iyengar | Is Anyone Responsible? How Television Frames Political Issues[END_REF]). In the last cluster (on the right), we can identify a very general collection of political and social topics (health, tax, security, etc.) tackled by the different candidates in their programs, corresponding to what we previously designated as thematic framing. But further investigation of these classes is required in order to produce a fine-grained description of the different 'interpretive packages' of these framing models, and a more robust frame mapping analysis.

Global frame mapping

Figure 3 (Factorial Analysis) shows the relations between the classes previously identified in Figure 2, and their overlapping sections. We can distinguish the three class-based sections mentioned above: The above representation highlights the proximity between classes (due to their similar lexicon) and the need to focus on their content better to understand the structuration of information in our corpus. We will now focus on each class using similarity analysis, and attempt to define their lexical connections more precisely. The interpretation of this lexical perspective is based on the centrality of a lexical form as well as the width (thickness) of the link between forms (and not their spatial position). Surrounding the central form Trump (due to a very high frequency), the related lexicon helps us to understand the semantics of class 3: Megyn Kelly, real estate, question/answer, Mr Trump, etc. Trump is presented as a real estate magnate and front runner in the presidential race, and this class is fully devoted to the illustration of his character and actions (with words like insult, mock, tweet, etc.), notably in his relations with journalists.

Excerpt (1) exemplifies the lexical universe contained in this class:8

( (2) Donald Trump came up early in the Republican presidential debate Thursday, but not often.

The seven Republican candidates participating in the Fox News-hosted event from Des Moines quickly addressed the elephant in the room -the absent front-runner, who was boycotting the event in favor of his own rally just 3 miles away. (New York Daily News, 29/01/2016)

Representative of the episodic framing, this class is structured around one event, the absence of the main candidate in the debate. This is significant in the sense that much media coverage in the primary period depends upon the political agenda and tackles one-time topics -e.g. his absence in extract (2), a tweet in extract (1) -rather than actively questioning the different political issues. Extract (2) can also account for the importance of the form Trump in the class, his repeated name (and wordplay with his name) illustrating his overwhelming media presence: In this first cluster, class 1 contains all the other candidates' names in the primary, as we can see in the similarity analysis displayed in Figure 5. The following classes are also part of an episodic cluster, but this time centered more on the Democrats. Class 5 focuses on primary elections among liberal candidates, and contains discourses aiming to define the sociotypes of Democrat supporters (women, black, young, white, liberal, etc.) and highlighting a duel between Hillary Clinton and Bernie Sanders (the former being more central to the class): Conversely, class 8 focuses on social and societal issues, such as health care reform and health policy, linked to the question of immigration, notably (but not only) from Mexico. Excerpt ( 9) is representative of this type of political content, focusing on programs and proposals for measures:

(9) That's the theory, anyway, and it's deeply embedded in Mr. Sanders's approach. His proposals for single-payer health care, free college tuition and paid family leave financed through a small payroll tax reflect the view that successful programs should be universal and create a connection between individuals and government. (The New York Times, 10/02/2016)

The last class (class 9) in this thematic framing cluster includes several events or affairs that seem to go from anecdotal events to real political issues, both being debated or tackled, at the top of which appear Hillary Clinton's email issue, but also local issues such as poisoned tap water from the Flint River in Michigan or the spread of the Zika virus. Thematic classes gather elements that show concern about the political content of ongoing debates, rather than sticking to shorter time-frame events, declarations from candidates, voting or polling results.

We will now try to determine if there are significant links between lexical classes or framing types and the editorial identity of the different newspapers, as detailed above (see 4.1), by observing the correlation between each newspaper and the different lexical classes. The New York Times and The Wall Street Journal can be characterised by their predominance in classes 8 and 7 respectively, which are the main two thematic classes (social for one, economic for the other), i.e. placing the primaries within a larger spatiotemporal and conceptual framework.

Conversely, episodic framing seems to characterise the New York Post (classes 1, 2 and 3) and the New York Daily News (classes 3 and 4).

The Washington Post and USA Today adopt an intermediate position. Class 6 (episodic from a general perspective) as well as thematic class 9 (dealing with political affairs) both originate from the former. Class 7 on economic debates and class 2 on polling results contain lexicon from the latter.

This result partly echoes some of the positions (i.e. the distance between the New York Times and the New York Post /the New York Daily News) on the first horizontal axis of the Factorial analysis (cf. Figure 1), but also indicates the editorial dynamics by materialising (from a lexical viewpoint) major acknowledged differences between sources of information: on the one hand, The New York Times, which, according to Entman, exemplifies a center-left approach to politics, and a source of information committed to the respect of 'objectivity norms' (2010: 390) 11 ; whilst on the other hand, the graph displays the distant and opposing position of the New York Daily News and the New York Post on this horizontal axis. Stylistic factors can probably explain this antagonism, shorter articles and a somewhat sensationalistic approach being specific to the tabloid press. 12 USA Today can be considered as having a centrist approach to national politics, which would account for its intermediate position on the graph, matching its 'middle-market newspaper' status, as described by the Oxford Index: '[n]ewspapers which are neither upmarket (primarily hard news) nor downmarket (primarily sensationalist), and which combine entertainment with more serious news' (Oxford Index, 2018).

Conversely, the vertical axis of Figure 1 displays the isolated position of the Wall Street Journal, in the upper part of this graph, distant and opposed to The Washington Post, located in the lower part. This can possibly be explained by their position on the political scale. The Wall Street Journal is considered to be more conservative, the target readers often being seen as neocons, while The Washington Post is thought to lean towards the liberal side of American politics (together with the New York Times). On this same vertical axis, the New York Daily News and the New York Post would then incline towards a more liberal perspective on American politics, as illustrated by their position at the bottom of the graph. However, their position on the horizontal line -as it is often the case when observing and interpreting CFAs -is the most significant factor here. Their (lexical) proximity to more liberal papers would need additional investigation as well as their respective distinct position on this graph.

These interpretations could be visualised by naming the continuum illustrated by the two axes as shown in Figure 9: Figure 9. An interpretation of factorial analysis based on newspapers' lexical specificities 11 He applied the same comment to The Washington Post. 12 'Newspapers with pages about 30 cm (12 inches ) by 40 cm (16 inches), usually characterised by an emphasis on photographs and a concise and often sensational style.' (Collins English Dictionary, 2019) We can thus observe a correspondence between a content that can be qualified as quality content and lexical universes centered on political issues at stake within the candidates' manifestos.

Linguistic approach to themes

As presented earlier, the DHC enables us to go from words to themes based on lexical cooccurrence. Co-occurring words become indicators of isotopy or semantic correlates, based on a purely inductive -also called non-supervised -perspective. The following method aims at redefining themes using a meaning-based approach so as to examine their distribution from a contrastive point of view.

Redefining themes

In order to (re)define themes from a more semantic perspective, we used two methodological approaches. We first identified the lexicon in the different classes related to each theme. For instance, for the Immigration theme, we kept the lexical units that relate to this theme in this particular political period (mainly class 8), but which are context-dependent (and not semantically related to Immigration as such) : border, wall, deport, deportation, rapist, undocumented, Mexican, etc. Secondly, to ensure a more semantic approach to themes, we completed the list by adding a limited set of metonyms or semantically-related terms to the main subject: 13 illegal(ly), immigrant(s), citizenship, migrant(s), visa(s), etc., thus forming a limited but consistent lexicon encompassing a specific theme. 14 We applied the same methodology to the following thematic issues: Health, Religion and Gun control (Health: Social Security, health care, medicare, Obamacare, insurance, Medicaid, etc. Religion: religious, religiousness, evangelist, evangelism, evangelical, evangelists, Christian-Baptists, Presbyterian, Pope, pontiff, Christianity, etc. and Gun control: gun(s), arm(s), weapon(s), gun control, etc.) (The New York Times, 10/02/2016) (11) His [Sanders] most expensive proposal, about $1.4 trillion annually, is for a "Medicare for all" government-financed health care system, which he said was needed to ensure all Americans have affordable health coverage and to control costs. (The Wall Street Journal, 08/02/2016) 13 We used the Collins and Merriam-Webster thesauri of the English Language for this purpose.

14 From a general perspective, these units are not always related to the main theme either. However, from a specific point of view (in these 3,117 articles dealing with the US Primary and extracted during that period), the semantic link between these units and the notion of immigration was extremely strong. We used the TXM platform to identify and extract lexical patterns based on part-of-speech sequences: adverbial phrases, complex noun groups, long verbal phrases, etc.). For instance, looking for all the adjectives associated with Trump and style within 20 words15 yielded passages such as shown in extract ( 14 Other complex patterns that can be pointed out include structures such as Trump + adverb + verbfuture (with potential lexical insertions), exemplified by extract (15):

(15) Donald Trump will very likely be the Republican nominee for president, and there is a nonzero chance he could win in November. (The Washington Post, 18/03/2016).

Without quoting all the potential patterns and queries, we now present a synthetic vision of two very different approaches to the candidate Trump, exemplified by The Wall Street Journal and the New York Post.

Trump in The Wall Street Journal

In this newspaper, the candidate Trump is mostly referred to with the following qualifiers: businessman, billionaire, real-estate magnate, estate mogul / tycoon, New York persona.

The key elements of his speeches are usually reported indirectly, in order not to promote his very provocative language style while still mentioning his favourite issues: 'Trump is light on policy substance, and his supporters have mainly responded to his stick-it-to-the-man style.'; 'The push to the anti-establishment fringe is so strong by Mr. Trump'.

The candidate is presented as atypical and provocative (unconventional, freewheeling, blunt, phony, provocative, controversial, etc.), but the paper remains rather neutral in its presentation of his actions, the candidate being constantly associated with factual but controversial achievements: Trump Tower, Trump University, Trump magazine, Trump Golf Club, Trump wine, etc.

Trump in the New York Post

The New York Post highlights his omnipresence in the media (Trump told CNN, Trump said, he warned, he replied, etc.) and provides evidence of his constant social media activity (Trump tweeted, warned on Twitter, accusatory tweets, etc.). The candidate's name is the subject and object of criticism (depicted by the media), as the following co-occurring items reveal: blame, mock, blasted, trashing, caustic barbs, etc. Similarly, a semantic field associated with his name is clearly related to warfare: attack, defend, blows, assaults, hostilities, feud, shoot, etc. Finally, lexical items that can be categorised as insults regularly co-occur with the name Trump, as, in that respect, the initiator of those insults : egghead, liar, pussy, corrupt, sick, crook, pawn, fat pig, etc. As opposed to the other papers, the New York Post often quotes his provocative style in direct discourse in order to emphasise the powerful speech acts of this candidate in the presidential race -see extract ( 16): ( 16) "I could stand in the middle of Fifth Avenue and shoot somebody and I wouldn't lose voters". (The New York Post, 06/06/2016) Direct speech also highlights his dialectic approach, based on simplicity and self-evidence: 'we either have a country or we don't. We need a border. We need a wall.'; 'I think we are weak. We cannot beat ISIS. We should beat ISIS very quickly', etc.

Conclusion and perspectives

Several conclusions can be drawn from this analysis. If we focus on the object of this study, we have demonstrated the existence of different media frames and links between types of press and types of frames. The main similarity among sources is the hegemonic position of the candidate Trump (even in Democrat universes). The major difference between sources is the way information on this candidate is presented and, above all, the editorial strategies adopted by the different newspapers, leading them to cover the presidential campaign in an episodic or, conversely, thematic framing.

If we now concentrate on the method, three major conclusions summarise the different points:

• General contexts can be visualised (based on specificity and Factorial Analysis), and opposed to local contexts (based on co-occurrences and concordances). • The study of non-supervised co-occurrences16 (based on a textual statistics approach) is relevant in the elaboration of a thematic description of a diversified corpus. • A quantitative approach informs the analysis, a qualitative approach determines the interpretation of texts, constituting a complementary method for meaning construction.

Finally, if we consider the object and the method, lexical statistics enable both a fine-grained corpus segmentation [START_REF] Ratinaud | Des mondes lexicaux aux représentations sociales. Une première approche des thématiques dans les débats à l'Assemblée nationale (1998-2014)[END_REF], and a dual (macro and micro) perspective on themes.

Figure 1 .

 1 Figure 1. Factorial analysis based on lexical specificities

Figure 2 .

 2 Figure 2. Dendrogram obtained after DHC analysis of the sub-corpus

Figure 3 .

 3 Figure 3. Factorial analysis obtained as the outcome of DHC upon the sub-corpus

  4.1.2 First cluster (classes 1, 4, 3): the Republicans' episodic framing On the basis of similarity analysis (see section 3.2.3 for methodological explanations), we can identify a specific class (class 3) structured mainly around the Republican candidate Donald Trump -see Figure4.

Figure 4 .

 4 Figure 4. Similarity analysis of class 3 7

( 3 )

 3 Trump, Trump, Trump, Trump. Can any conversation of more than 45 seconds' duration fail to turn to le sujet Trump? […] Sunrise tomorrow will be Trump o'clock, the weatherman is calling for a partly Trumpy day, you can read about it in the Trumpington Post on the way home. Wake up and smell the coffee: It's morning in Trumpmerica. (New York Post, 06/03/2016)

Figure 5 .

 5 Figure 5. Similarity analysis of class 1

Figure 6 .

 6 Figure 6. Similarity analysis of class 5 9

  Figure 8, based on the χ 2 score, illustrates the correlation between papers and lexical classes: Figure 8. Correlation between newspapers and lexical classes 10

Figure 10 .

 10 Figure 10. Correlation between newspapers and themes (based on the specificity index)

  ): (14) Trump is belligerent and hyperbolic, with an authoritarian style. (The Washington Post, 02/02/2016).

Table 1 . Corpus information 2 Media Word-tokens Percentage of the whole corpus Number of articles per newspaper

 1

	The New York Times	789,431	39.7%	957
	The Washington Post	575,481	29.0%	815
	The Wall Street Journal	351,916	17.7%	623
	USA Today	110,275	5.5%	234

  1) Fox News blasted Donald Trump Friday night for his "crude and sexist" tweet trashing anchor Megyn Kelly and for telling his supporters to boycott her program. The network released a statement charging that Trump has a "sick obsession" with the cable news anchor and it's proud of Kelly's work as a journalist.(New York Post, 19/03/2016) This class is linked to another class, class 4 in the dendrogram (see Figure2), which focuses on a televised Republican debate held in Des Moines, Iowa. The main feature of the Republican debate was the fact that Donald Trump skipped it, as excerpt (2) explains perfectly:

  This class is linked to the previous two. Together, they constitute a first cluster of episodic framing. This cluster reveals the predominant focus on the candidate Donald Trump, insofar as he is the structuring element of two classes (including one class linked to his absence during a specific event), and a lesser media coverage of Trump's opponents as all the other candidates end up together in one single class, as opposed to Trump (present in all three classes).

	This is typical of what Iyengar calls 'horse-race framing', reporting daily polls, as the following
	characteristic segment -extract (4) -illustrates:
	(4) [L]atest poll of likely new Hampshire voters (500 Republican and 500 Democratic) in this
	Tuesday's primary: John Kasich 13%, Jeb Bush 10%, Ted Cruz 7%, Chris Christie 5%, Ben
	Carson 4%, Carly Fiorina 4%, Bernie Sanders 50%, Marco Rubio 19%, Donald Trump
	29% Source: Suffolk University/Boston Globe telephone poll taken Feb. 2-4. (New York
	Post, 07/03/2016)
	4.1.3 Second cluster (classes 2, 6, 5): the Democrats' episodic framing

  After observing the structuring themes in the different classes, our aim was to further investigate what we qualified as discursive objects. Given the lexical predominance of the Trump theme (see 4.1.2.), we decided to examine what was said about this candidate in two different newspapers and also more generally in all sources.

	Religious issues, as presented in extract (12), are clearly specific to The Washington Post's
	approach to the campaign:
	(12) [B]usinessman Donald Trump […] continued to try to blunt Cruz's appeal among Iowa's
	powerful evangelical voters (The Washington Post, 01/02/2016)
	Conversely, none of these issues is statistically significant in the New York Post, and only barely
	addressed in the New York Daily News, which is mostly characterised by the handling of gun
	control issues -see extract (13) -or immigration:
	(13) Broadly speaking, the GOP field is wrongheaded in supporting absolutist positions on gun
	control (The New York Daily News, 21/02/2016)
	4.2.2 Further investigation of discursive objects

The international news database Factiva (produced by Dow Jones) was used for this extraction.

The difference in the number of articles and their size is measured (and balanced out) by textual statistics (see section

3.2.1 for more information).

As opposed to keyness analysis which identifies keywords using a reference corpus, these methods aims at contrasting word distribution among different papers (partitions) taking into account different parameters such as the length (size) of the article and the partition within the global corpus. Specificity will indicate the over-representation or the under-representation of the occurrences of a word (or a sequence of words) in each partition.

This word count is an average, adjusted according to the presence of punctuation marks.

Generic term: 'Maximum likelihood phylogenetic tree'

See section 3.2 for the method brief description, and AFC related literature mentioned in the text for further explanations.

The colours are used to separate and highlight the different lexical clusters but do not have a specific meaning

Excerpts are selected on a statistical basis by the textual analysis software: they are representative of lexical classes when they contain a large number of class-specific words.

Note that Sanders has been stemmed to Sander on this representation (due to general stemming rules).

The size of each bar is determined by the value of the χ 2 score, indicating the over-and under-representativeness of variable modalities (here, the source) in each thematic class (see 3.2 for additional information about the χ 2 score).

The corresponding CQL (Corpus Query Language) query is:[word="Trump"][]{1,10}[enpos="JJ"][word="style"]|[enpos="JJ"][word="style"][]{1,10}[word="Trump"]

This can also be referred to by the generic term 'Unsupervised hierarchical clustering'.


Mariia Vladimirova
email: mariia.vladimirova@inria.fr

Julyan Arbel
email: julyan.arbel@inria.fr

Stéphane Girard
email: stephane.girard@inria.fr

Bayesian neural network unit priors and generalized Weibull-tail property

Keywords: generalized Weibull-tail, sub-Weibull, Bayesian neural networks

The connection between Bayesian neural networks and Gaussian processes gained a lot of attention in the last few years. Hidden units are proven to follow a Gaussian process limit when the layer width tends to infinity. Recent work has suggested that finite Bayesian neural networks may outperform their infinite counterparts because they adapt their internal representations flexibly. To establish solid ground for future research on finite-width neural networks, our goal is to study the prior induced on hidden units. Our main result is an accurate description of hidden units tails which shows that unit priors become heavier-tailed going deeper, thanks to the introduced notion of generalized Weibull-tail. This finding sheds light on the behavior of hidden units of finite Bayesian neural networks.

Introduction

Theoretical insights and the development of a comprehensive theory are often the driving force underlying the development of new and improved methods. Neural networks are powerful models, but they still lack comprehensive theoretical support. The Bayesian approach applied to neural networks is considered one of the best-suited frameworks to obtain theoretical explanations and improve the models.

Infinite-width Bayesian neural networks are well-studied. Induced priors in Bayesian neural networks with Gaussian weights are Gaussian processes when the number of hidden units per layer tends to infinity [START_REF] Radford | Bayesian learning for neural networks[END_REF][START_REF] Alexander G De G Matthews | Gaussian process behaviour in wide deep neural networks[END_REF][START_REF] Lee | Deep neural networks as Gaussian processes[END_REF][START_REF] Garriga-Alonso | Deep convolutional networks as shallow Gaussian processes[END_REF]. Stable distributions also lead to stable processes, which are generalizations of Gaussian ones [START_REF] Favaro | Stable behaviour of infinitely wide deep neural networks[END_REF].

Since in some cases finite models perform better [START_REF] Lee | Deep neural networks as Gaussian processes[END_REF][START_REF] Garriga-Alonso | Deep convolutional networks as shallow Gaussian processes[END_REF][START_REF] Lee | Finite versus infinite neural networks: an empirical study[END_REF], there is a need for theoretical justifications. One of the ways is to study induced priors. By analyzing the priors over representations, [START_REF] Aitchison | Why bigger is not always better: on finite and infinite neural networks[END_REF] suggests that finite Bayesian neural networks may generalize better than their infinite counterparts because of their ability to learn representations. Another idea is to find the induced priors in the functional space [START_REF] Gordon | Bayesian deep learning and a probabilistic perspective of generalization[END_REF].

By bounding moments of distributions induced on units, [START_REF] Vladimirova | Understanding priors in Bayesian neural networks at the unit level[END_REF] proved that hidden units have heavier-tailed upper bounds and follow sub-Weibull distributions with an increasing tail parameter depending on the layer depth. Those bounds are optimal as they are achieved for shallow Bayesian neural networks; however, they are not accurate.

Recently, Zavatone-Veth and [START_REF] Pehlevan | Exact priors of finite neural networks[END_REF]; [START_REF] Noci | Precise characterization of the prior predictive distribution of deep ReLU networks[END_REF] showed that there exists a precise description of induced unit priors through Meijer G-functions. These are full descriptions of priors at the unit level. The results are in accordance with the heavy-tailed nature and asymptotic expansions in a wide regime, but it has restrictions. First, the setting is simplified: linear or ReLU activation functions and Gaussian priors on weights. While [START_REF] Gordon | Bayesian deep learning and a probabilistic perspective of generalization[END_REF] argue that vague Gaussian priors in the parameter space induce useful function-space priors, in some cases heavier-tailed priors can perform better (Fortuin et al., 2021). Second, it is hard to work with Meijer G-functions due to their complexity. Our goal is to obtain more general characterizations for hidden units.

We introduce a new concept for describing distributional properties of tails by extending the existing Weibull-tail characterization. A random variable X is called Weibulltail [START_REF] Gardes | Weibull tail-distributions revisited: a new look at some tail estimators[END_REF] with tail parameter β > 0, which is denoted by

X ∼ WT R + (β), if its cumulative distribution function F X satisfies F X (x) = 1 -F X (x) = e -x β l(x) , for x > 0, (1)
where l(x) is a slowly-varying function, i.e. it is a positive function such that for all t > 0 lim x→∞ l(tx) l(x) = 1. We note that the Weibull-tail property only considers the right tail of distributions. Here we adapt the Weibull-tail characterization to the whole space R by taking into consideration the left tail as well. Additionally, we introduce generalized Weibull-tail random variables with tail parameter β > 0 which have Weibull-tail upper and lower bounds for both tails (Definition 1). Such a characterization is easily interpretable and stable under basic operations, even for dependent random variables. The family of generalized Weibull-tail distributions covers a large variety of fundamental distributions such as Gaussian (β = 2), gamma (β = 1), Weibull (β > 0), to name a few, and turns out to be a key tool to describe distributional tails.

Contributions. We make the following contributions:

• We introduce a new notion of tail characteristics called generalized Weibull-tail, a version of Weibull-tail characteristics on R + extended to variables on R (Section 2). The additional advantage of this notion is stability under basic operations such as multiplication by a constant and summation. • With the results on generalized Weibull-tail characterization and dependence, we obtain an accurate characterization of the heavy-tailed nature of hidden units in Bayesian neural networks (Section 3). We establish these results under possibly heavy-tailed priors and fairly mild assumptions on the non-linearity.

Generalized Weibull-tail random variables

The study of the distributional tail behavior arises in many applied probability models of different areas, such as hydrology [START_REF] Strupczewski | On the tails of distributions of annual peak flow[END_REF]), finance (Rachev, 2003) and insurance risk theory [START_REF] Alexander | Quantitative risk management: concepts, techniques and tools[END_REF]. Since in most cases exact distributions are not available, deriving asymptotic relationships for their tail probabilities becomes essential. In this context, an important role is played by so-called Weibull-tail distributions satisfying Equation (1) [START_REF] Gardes | Weibull tail-distributions revisited: a new look at some tail estimators[END_REF][START_REF] Gardes | On the estimation of the functional Weibull tailcoefficient[END_REF].

A majority of works focuses on right tails of distributions and develops a theory only applicable to right tails, while it is important to study both right and left tails of distributions. We extend the notion of Weibull-tail on R + to R and introduce generalized Weibull-tail random variables on R in the following definition. They are stable (under basic operations) extensions of Weibull-tail random variables on R (Appendix B).

Definition 1 (Generalized Weibull-tail on R) A random variable X is generalized Weibulltail on R with tail parameter β > 0 if both its right and left tails are upper and lower bounded by some Weibull-tail functions with tail parameter β:

e -x β l r 1 (x) ≤F X (x) ≤ e -x β l r 2 (x)
, for x > 0 and x large enough, e -|x| β l l 1 (|x|) ≤F X (x) ≤ e -|x| β l l 2 (|x|) , for x < 0 and -x large enough, where l r 1 , l r 2 , l l 1 and l l 2 are slowly-varying functions. We note X ∼ GWT R (β).

This family includes widely used distributions such as Gaussian (β = 2), Laplace (β = 1) and generalized Gaussian distributions. These distributions are also symmetric (around 0), so their left and right tails are equal. Thus, it naturally leads to obtain tail characteristics for symmetric distributions by considering random variables whose absolute value is Weibulltail on R + . See Appendix B for details. While Weibull-tail random variables are also generalized Weibull-tail, the opposite is not always true. Consider slowly-varying functions l 1 ≡ 1 and l 2 ≡ 2. Then function l(•) = 1 + cos 2 (ln(•)) satisfies l 1 ≤ l ≤ l 2 but is not slowly varying. However, for any β ≥ 2, function F X (x) = e -x β l(x) is the survival function (it is decreasing) of some random variable X and it satisfies e -x β l 2 (x) ≤ F X (x) ≤ e -x β l 1 (x) . Therefore, X is GWT(β) but not WT(β).

Next we aim to obtain a tail characterization for the sum of generalized Weibull-tail variables on R, where we allow random variables to be dependent. Further, we show that under the following assumption of positive dependence, the sum has tail parameter equal to the minimum among the considered ones (Theorem 2.1).

Definition 2 (Positive dependence condition) Random variables X 1 , . . . , X N satisfy the positive dependence (PD) condition if the following inequalities hold for all z ∈ R and some constant C > 0:

P (X 1 ≥ 0, . . . , X N -1 ≥ 0|X N ≥ z) ≥ C, P (X 1 ≤ 0, . . . , X N -1 ≤ 0|X N ≤ z) ≥ C.
Remark 2.1 The choice of zeros and z in the PD condition is arbitrary: one can choose any z 1 , . . . , z N instead such that z 1 + • • • + z N = z. The choice of the N -th variable within X 1 , . . . , X N is also arbitrary. Besides, if random variables X 1 , . . . , X N are independent with non-zero right and left tails, then they satisfy the PD condition and the constant C is equal to the minimum between P(X 1 ≤ 0) . . . P(X N -1 ≤ 0) and P(X 1 ≥ 0) . . . P(X N -1 ≥ 0).

There is a great variety of dependent distributions that obey this dependence property including pre-activations in Bayesian neural networks (see Lemma 3.1 for details). Note that the positive orthant dependence condition (POD, see Nelsen, 2007) implies the PD condition.

Theorem 2.1 (Sum of GWT R variables) Let X 1 , . . . , X N ∈ GWT R with tail parame- ters β 1 , . . . , β N . If X 1 , . . . , X N satisfy the PD condition of Definition 2, then, X 1 + • • • + X N ∈ GWT R (β) with β = min{β 1 , . . . , β N }.
All proofs are deferred in Appendix. The intuition of the PD condition is to prevent the tail of a sum to become lighter due to negative connection. The simplest example of such negative dependence comes with counter-monotonicity where (X, Y ) is such that Y = -X.

Another less trivial example is Y = XI(|X| ≤ m) -XI(|X| > m) for some m > 0: the sum X + Y = 2XI(|X| ≤ m) is a version of X truncated to the compact set [-m, m].
In both cases, it is easy to see that the PD condition does not hold. We conclude this section with a result on the product of independent generalized Weibull-tail random variables.

Theorem 2.2 (Product of GWT R variables) Let X 1 , . . . , X N ∈ GWT R be indepen- dent symmetric with tail parameters β 1 , . . . , β N . Then, the product X 1 . . . X N ∈ GWT R (β) with β such that 1 β = 1 β 1 + • • • + 1 β N .
Now the obtained results can be applied to Bayesian neural networks, showing that hidden units are generalized Weibull-tail on R.

Bayesian neural networks induced priors

Neural networks are hierarchical models made of layers: an input, several hidden layers and an output. Each layer following the input layer consists of units which are linear combinations of previous layer units transformed by a nonlinear function, often referred to as the nonlinearity or activation function denoted by φ. Given an input x ∈ R H 0 , the -th hidden layer consists of a vector whose size is called the width of layer, denoted by H . The coefficients of linear combinations are called weights, denoted by w ( ) ij for i = 1, . . . , H -1 , j = 1, . . . , H . In Bayesian neural networks, weights are assigned some prior distribution [START_REF] Radford | Bayesian learning for neural networks[END_REF]. The pre-activations and post-activations of layer are respectively defined as

g ( ) j = H -1 i=1 w ( ) ij h ( -1) i , h ( ) j = φ(g ( ) j ) (2)
where h

i are elements of input vector x, so h

i are deterministic numerical object features. The main ingredient for Theorem 2.1 is the positive dependence condition of Definition 2. The product of hidden units and weights satisfies the positive dependence condition: Lemma 3.1 Let X 1 , . . . , X N be some possibly dependent random variables and W 1 , . . . , W N be symmetric, mutually independent and independent from X 1 , . . . , X N , then random variables X 1 W 1 , . . . , X N W N satisfy the PD condition.

Along with Theorem 2.1, the previous lemma implies that neural network hidden units are generalized Weibull-tail with tail parameter depending on those of the weights.

Theorem 3.1 (Hidden units are GWT) Consider a Bayesian neural network as described in Equation (2) with ReLU activation function. Let -th layer weights be independent symmetric generalized Weibull-tail on R with tail parameter β ( ) w . Then, -th layer pre-activations are generalized Weibull-tail on R with tail parameter β ( ) such that

1 β ( ) = 1 β (1) w + • • • + 1 β ( ) w .
Note that the most popular case of weight prior, iid Gaussian [START_REF] Radford | Bayesian learning for neural networks[END_REF], corresponds to GWT R (2) weights. This leads to units of layer which are GWT R ( 2 ).

To illustrate this theorem, we have built neural networks of 4 hidden layers, with 4 hidden units on each layer. We used a fixed input x of size 10 4 , which can be thought of as an image of dimension 100 × 100. This input was sampled once for all with standard Gaussian entries. In order to obtain samples from the prior distribution of the neural network units, we have sampled the weights from independent centered Gaussians from which units were obtained by forward evaluation with the ReLU non-linearity. This process was iterated n = 10 6 times. Note that for a GWT R random variable, P(X ≥ x) = e -x β l(x) so the tail parameter can be expressed as:

β = log(-log P(X ≥ x)) log x - log l(x) log x . (3)
In Figure 1, we plot log(-log P(X ≥ x)) as a function of log x. We see that the obtained tail parameters approximations are increasing for the increasing layer number and visually correspond to the theoretical tail parameter.

Figure 1: Solid lines: approximations of tail parameters β ( ) based on Equation (3) where X are hidden units of layers = 1, 2, 3, 4 corresponding theoretically to generalized Weibull-tail with tail parameters β ( ) = 2, 1, 2 /3, 1 /2, under the independent Gaussian weights assumption. Dashed lines: linear regressions with coefficients equal to the theoretical tail parameters β ( ) = 2 / and manually selected biases to approach the solid lines for visual comparison.

β (4) = 1/2 β (3) = 2/3 β (2) = 1 β (1) = 2

Comparison of different characterizations

Generalized Weibull-tail vs sub-Weibull

Some of the commonly used techniques to study the tail behavior is to consider probability tail bounds such as sub-Gaussian, sub-exponential, or their generalization to sub-Weibull distributions [START_REF] Vladimirova | Sub-Weibull distributions: Generalizing sub-Gaussian and sub-exponential properties to heavier tailed distributions[END_REF][START_REF] Kumar | Moving beyond sub-Gaussianity in high-dimensional statistics: Applications in covariance estimation and linear regression[END_REF]. A nonnegative random variable X is called sub-Weibull with tail parameter θ > 0 if its survival function is upper-bounded by that of a Weibull distribution:

F X (x) ≤ ae -bx 1/θ , for x > 0 and some a, b > 0.

(4)

This property ensures the existence of the moment generating function as well as bounds on moments. In contrast, the Weibull-tail property characterizes the survival or density functions without a hand on moments. While tail parameters in Equation ( 1) and ( 4) of generalized Weibull-tail and sub-Weibull properties respectively are different, there exist connections. Notice that for any constants a, b, β > 0, function l(x) = b -log a x β ≥ 0 is slowlyvarying for x large enough and ae -bx β = e -x β l(x) . It means that if a random variable X is sub-Weibull with parameter θ = 1/β > 0, satisfying Equation ( 4), then the survival function of X is upper-bounded by a Weibull-tail function with tail parameter β and slowly-varying function l(x) = 1, satisfying Equation ( 1). If random variable X is generalized Weibull-tail with tail parameter β, then from the last item of Proposition A.1, for a 1 , a 2 > 0 we have 2019) that hidden units of Bayesian neural networks with iid Gaussian priors are sub-Weibull with tail parameter proportional to the hidden layer number, that is θ = 2 . It means that the unit distributions of hidden layer can be upper-bounded by some Weibull distributions ae -x 2/ for all . For larger tail parameter θ, Weibull distribution ae -x 1/θ is heavier-tailed but being sub-Weibull does not guarantee the heaviness of the tails. However, this upper bound is optimal in the sense that it is achieved for neural networks with one hidden unit per layer.

a 1 e -x β 1 ≤ F X (x) = e -x β l(x) ≤ a 2 e -x β 2 , or GWT R + (β) ⊂ SubW(1/β 2 ) and GWT R + (β) ⊂ SubW(1/β 1 ) for x large enough and ∀(β 1 , β 2 ) such that 0 < β 2 < β < β 1 , as illustrated on Figure 2.
From Theorem 3.1, for neural networks with independent Gaussian weights, hidden units of -th layer are generalized Weibull-tail with tail parameter β = 1/θ = 2/ so they have upper and lower bounds of the form e -x 2/ l(x) up to a constant where l is some slowlyvarying function. Therefore, it proves that hidden units are heavier-tailed as going deeper for any finite numbers of hidden units per layer.

Meijer G-functions description

In Springer and [START_REF] Springer | The distribution of products of beta, gamma and Gaussian random variables[END_REF] it was shown that the probability density function of product of independent normal variables can be expressed through a Meijer G-function. It resulted into a precise description of induced unit priors given Gaussian priors on weights and linear or ReLU activation function [START_REF] Pehlevan | Exact priors of finite neural networks[END_REF][START_REF] Noci | Precise characterization of the prior predictive distribution of deep ReLU networks[END_REF]. It is a full description of function-space priors but under strong assumptions, requiring Gaussian priors on weights and linear or ReLU activation functions, and with convoluted expressions. In contrast, we provide results for a large variety of distributions including heavy-tailed ones, and our results can be extended to smooth activation functions, such as PReLU, ELU, Softplus.

Future applications

Cold posterior effect and priors. It was recently empirically found that Gaussian priors lead to the cold posterior effect in which a tempered "cold" posterior, obtained by exponentiating the posterior to some power largely greater than one, performs better than an untempered one [START_REF] Wenzel | How good is the Bayes posterior in deep neural networks really[END_REF]. The performed Bayesian inference is considered sub-optimal due to the need for cold posteriors, and the model is deemed to be misspecified. From that angle, [START_REF] Wenzel | How good is the Bayes posterior in deep neural networks really[END_REF] suggested that Gaussian priors might not be a good choice for Bayesian neural networks. In some works, data augmentation is argued to be the main reason for this effect [START_REF] Izmailov | What are Bayesian neural network posteriors really like?[END_REF][START_REF] Nabarro | Data augmentation in Bayesian neural networks and the cold posterior effect[END_REF] as the increased amount of observed data naturally leads to higher posterior contraction [START_REF] Izmailov | What are Bayesian neural network posteriors really like?[END_REF]. At the same time, even taking into account the data augmentation for some models, cold posterior effect is still present. In addition, [START_REF] Aitchison | A statistical theory of cold posteriors in deep neural networks[END_REF] demonstrates that the problem might originate in the wrong likelihood of the models and that modifying only the likelihood based on data curation mitigates the cold posterior effect. [START_REF] Nabarro | Data augmentation in Bayesian neural networks and the cold posterior effect[END_REF] hypothesize that using an appropriate prior incorporating knowledge of data augmentation might provide a solution. Moreover, heavy-tailed priors have shown to mitigate the cold posterior effect (Fortuin et al., 2021). According to Theorem 3.1, heavier-tailed priors lead to even heavier-tailed induced priors in function-space. Thus, the heavy-tail property of distributions in function-space might be a highly beneficial feature. Fortuin et al. ( 2021) also proposed correlated priors for convolutional neural networks since trained weights are empirically strongly correlated. Correlated priors improve overall performance but do not alleviate the cold posterior effect. Our theory can be extended to correlated weight priors. This direction is promising for further uncovering the effect of weight prior on function-space prior.

Edge of Chaos. An active line of research studies the propagation of deterministic inputs in neural networks [START_REF] Poole | Exponential expressivity in deep neural networks through transient chaos[END_REF][START_REF] Samuel S Schoenholz | Deep information propagation[END_REF][START_REF] Hayou | On the impact of the activation function on deep neural networks training[END_REF]. The main idea is to explore the covariance between pre-activations for two given different data points. [START_REF] Poole | Exponential expressivity in deep neural networks through transient chaos[END_REF][START_REF] Samuel S Schoenholz | Deep information propagation[END_REF] obtained recurrence relations under the assumption of Gaussian initialization and Gaussian pre-activations. They conclude that there is a critical line, so-called Edge of Chaos, separating a signal propagation into two regions. The first one is an ordered phase in which all inputs end up asymptotically correlated, and the second region is a chaotic phase in which all inputs end up asymptotically independent. To propagate the information deeper in a neural network, one should choose Gaussian prior variances corresponding to the separating line. [START_REF] Hayou | On the impact of the activation function on deep neural networks training[END_REF] show that the smoothness of the activation function also plays an important role. Since this line of works considers Gaussian priors not only on the weights but also on the pre-activations, it is closely related to a wide regime where the number of hidden units per layer tends to infinity. Given the fact that hidden units are heavier-tailed with depth, we speculate that future research will focus on finding better approximations of the pre-activation functions in recurrence relations obtained for finite-width neural networks.

Conclusion

We extend the theory on induced distributions in Bayesian neural networks and establish an accurate and easily interpretable characterization of hidden units tails. The obtained results confirm the heavy-tailed nature of hidden units for different weight priors.

Definition 5 (Weibull-tail on R) A random variable X on R is Weibull-tail on R with tail parameter β > 0 if both its right and left tails are with tail parameter β:

F X (x) = 1 -F X (x) = e -x β l 1 (x) , x > 0 F X (x) = e -|x| β l 2 (|x|) , x < 0,
where l 1 and l 2 are slowly-varying functions. We note X ∼ WT R (β).

Lemma B.1 (i) If random variable X on R is WT R (β), then |X| is WT R + (β).
(ii) If X is asymmetric but both tails are Weibull-tail and |X| is WT R + (β), then one of the tails (right or left) is WT R + (β) and the other one is WT R + (β ) where β ≥ β.

(iii) For symmetric distributions, X is WT R (β) if and only if |X| is WT R + (β).
Proof (i) For x > 0, the cumulative distribution function of |X| is the following

F |X| (x) = F X (x) -F X (-x).
Then, F |X| (x) can be expressed as a sum of the right and left tails: (ii) Let |X| be WT R + (β) and X has Weibull left and right tails with different tail parameters. Without loss of generality, assume that F X (-x) is Weibull-tail with tail parameter β < β. According to Lemma A.3, the sum survival function F |X| (x) will be Weibull-tail with tail parameter β = min{β , β}. We obtained a contradiction and F X (-x) must have tail parameter greater or equal β. If both tail parameters are greater than β, then the tails sum have the tail parameter equal to the minimum tail parameter among them which is greater than β. It means that at least one tail must have tail parameter β.

F |X| (x) = F X (x) + F X (-x). If X is WT R (β), then |X| is WT R + (β)
(iii) For symmetric distributions F X (x) = F X (-x) for any x, then 1 2 F |X| (x) = F X (x) = F X (-x) and we have the equality. (ii) For symmetric distributions we have F X (x) = F X (-x) for all x, then 1 2 F |X| (x) = F X (x) = F X and we have the equality. Since |X| is generalized Weibull-tail on R + with tail parameter β, e -x β l 1 (x) ≤ P (|X| ≥ x) ≤ e -x β l 2 (x) , where l 1 and l 2 are slowly-varying functions, it implies e -y β/bl 1 (y) ≤ P a|X| b ≥ y ≤ e -y β/bl 2 (y) ,

Lemma B.2 (i) If a random variable X is GWT R (β), then |X| is GWT R + (β). (ii) For symmetric distributions, X is GWT R (β) if and only if |X| is GWT R + (β). Proof Similarly as in Lemma B.1, we obtain F |X| (x) = F X (x) + F X (-x) for x ≥ 0. (i) If X is GWT R (β),
where li (y) = l i ((y/a) 1/b ) a β/b , i = 1, 2 are slowly-varying functions by Lemma A.1. It leads to the statement of the lemma.

Theorem 2.1 (Sum of GWT R variables) Let X 1 , . . . , X N ∈ GWT R with tail parameters β 1 , . . . , β N . If X 1 , . . . , X N satisfy the PD condition of Definition 2, then, X 1 + • • • + X N ∈ GWT R (β) with β = min{β 1 , . . . , β N }.

Proof Let us start with N = 2. For any random variables X and Y , the following upper bound holds: P(X + Y ≥ z) ≤ P(X ≥ z /2) + P(Y ≥ z /2) ≤ 2 max{P(X ≥ z /2), P(Y ≥ z /2)}.

The PD condition leads to a lower bound for the sum:

P(X + Y ≥ z) ≥ P(X ≥ 0, Y ≥ z) = P(X ≥ 0|Y ≥ z)P(Y ≥ z) ≥ C P(Y ≥ z),
where constant C > 0. Thus, the sum survival function F Z (z) = P(X + Y ≥ z) has the following bounds for the right tail:

C F Y (z) ≤ F Z (z) ≤ 2 max{F X ( z /2), F Y ( z /2)},
where F X and F Y are the survival functions of X and Y .

Let X and Y be generalized Weibull-tail on R of parameters β x and β y , then for z > 0 large enough C e -z βy l r 1 (z) ≤ F Z (z) ≤ 2e -z min{βx,βy } l r 2 (z) , where l r 1 = l y 1 is the slowly-varying function appearing in the right tail lower bound of generalized Weibull-tail Y and l r 2 (z) = min{ 1 2 βx l x 2 ( z /2), 1 2 βy l y 2 ( z /2)} is the minimum among

Figure 2 :

 2 Figure 2: Relation between sub-Weibull and generalized Weibull-tail characteristics.

  as a consequence of Definition 5 and Lemma A.3.

  then right and left tails are upper and lower-bounded by some Weibull-tail functions. Then, the sum of the right and left tails is upper and lowerbounded by these Weibull-tail functions and |X| ∼ GWT R + (β).

  Lemma B.3 (Power and multiplication by a constant) If X ∼ GWT R (β) and the distribution of X is symmetric, then a|X| b ∼ GWT R + β b for a, b > 0. Proof According to Lemma B.1, |X| ∼ GWT R + (β). For a, b > 0, the tail of Y = a|X| b is P a|X| b ≥ y = P |X| b ≥

Appendix A. Slowly and regularly-varying functions theory

The set of regular-varying functions with index ρ ∈ R is denoted by RV ρ . Note that for ρ = 0, the set RV 0 boils down to the set of slowly-varying functions. In particular, any function r ∈ RV ρ can be written r(x) = x ρ l(x), where l is slowly-varying.

Definition 3 (Regularly-varying function) Let r be a positive function. Then r ∈ RV ρ if for all t > 0 lim x→∞ r(tx) r(x) = t ρ . Proposition A.1 (Bingham et al., 1989, Proposition 1.3.6) Let l, l 1 , . . . , l k be slowly-varying functions. Then:

1. (log l(x))/ log x → 0 as x → ∞.

2. l α varies slowly for every α ∈ R.

3. l 1 l 2 , l 1 + l 2 , and (if l 2 (x) → ∞ as x → ∞) l 1 • l 2 vary slowly.

4. If f (x 1 , . . . , x k ) is a rational function with positive coefficients, f (l 1 , . . . , l k ) varies slowly.

For any

) is slowly-varying for a > 0.

Lemma A.2 If l 1 , l 2 vary slowly, so does max{l 1 , l 2 } and min{l 1 , l 2 }.

Lemma A.3 Let r and r 2 be regularly-varying functions with parameters β 1 > 0 and β 2 > 0. Then, the function r such that e -r = e -r 1 +e -r 2 , is regularly-varying with parameter

Proof Let us express function r from the statement: r(x) =log ϕ(x) =log e -r 1 (x) + e -r 2 (x) .

If β 1 = β 2 , without loss of generality, let us assume that

.

→ 0.

Then ϕ(x) e -r 1 (x) and r(x) =log ϕ(x) r 1 (x). It means that for the case

Let us consider the case with equal parameters

Consider the logarithm of the latter expression

Since the function l is slowly-varying by Lemma A.2 and 0 ≤ e -x β |l 2 (x)-l 1 (x)| ≤ 1, then r(x) =log ϕ(x) x β l(x) ∈ RV β .

Appendix B. Weibull-tail properties on R

Let us firstly introduce a notion of generalized Weibull-tail random variable which has an additional property of stability:

Definition 4 (Generalized Weibull-tail on R + ) A random variable X is called generalized Weibull-tail with tail parameter β > 0 if its survival function F X is bounded by Weibull-tail functions of tail parameter β with possibly different slowly-varying functions l 1 and l 2 : x) , for x > 0.

(5)

We note X ∼ GWT R + (β).

Now we define a random variable whose both right and left tails are Weibull-tail on R + .

1. We say functions f g if and only if f /g → 1.

slowly-varying functions, where l x 2 and l y 2 are slowly-varying functions in the right tails upper bounds of X and Y . According to Lemma A.2, l r 2 (z) is also slowly-varying. Similarly, can get bounds for the left tail. Therefore, X + Y is generalized Weibull-tail on R with tail parameter β = min{β x , β y }.

Similarly as above when N = 2, bounds for the right tail of the sum

where F i is the survival functions of X i and constant C > 0. The rest of the proof is identical to the one of the case N = 2.

The case when distributions have only right tails (or only left tails), can be considered as a particular case of the last theorem: a sum of non-negative generalized Weibull-tail random variables is non-negative generalized Weibull-tail with tail parameter equal to the minimum among those of the terms.

and Y ∼ GWT R 1 θy . From Lemma B.1 and since random variables X and Y are symmetric, it is equivalent to

Our goal is to show that for some slowly-varying functions l 1 and l 2 , there exist upper and lower bounds for the survival function of |Z| and z large enough as follows:

(i) Upper bound. First, notice that from the concavity of the logarithm, we have ln(pu + (1p)v) ≥ p ln u + (1p) ln v for any u, v > 0 and p ∈ (0, 1). Then pu + (1p)v ≥ u p v 1-p . The change of variables x = u p , y = v 1-p implies px 1 /p + (1p)y 1 /(1-p) ≥ xy.

From the latter equation, an upper bound of the product tail is

θy . Taking p = θx θx+θy and 1p = θy θx+θy , yields a sum of two independent non-negative generalized Weibull-tail random variables with tail parameter 1 θx+θy on the right-hand side of Equation (7). By Theorem 2.1, this sum is generalized Weibull-tail with the same tail parameter 1 θx+θy . It means that there exists a slowly-varying function l 2 such that the tail of product absolute value |XY | is upper-bounded by Since |X| and | are generalized Weibull-tail on R + , we can define function l 1 (z) = l x 1 (z θx/(θx+θy) ) + l y 1 (z θy/(θx+θy) ) with l x 1 and l y 1 being slowly-varying functions in the lower bounds of generalized Weibull-tail |X| and |Y |. Then, l 1 is slowly-varying by Lemma A.1 and we have

Combining together Equations ( 8) and ( 9) and Definition 4 with Lemma B.1 implies the statement of the theorem.

Appendix C. Bayesian neural network properties

Proofs of Section 3. Lemma 3.1 Let X 1 , . . . , X N be some possibly dependent random variables and W 1 , . . . , W N be symmetric, mutually independent and independent from X 1 , . . . , X N , then random variables X 1 W 1 , . . . , X N W N satisfy the PD condition.

Proof The joint probability for the right tail P

Independence between W i and X j yields

where the last equality is due to the mutual independence of weights W 1 , . . . , W N . Let

then, due to the symmetry of W i , the probability P (W i x i ≥ 0) = 1 2 . Thus, the following lower bound holds:

Substituting the latter equations into Equation (10) leads to the lower bound:

By the conditional probability definition, we have

The proof for the left tail is identical.

Theorem 3.1 Consider a Bayesian neural network as described in Equation (2) with ReLU activation function. Let -th layer weights be independent symmetric generalized Weibull-tail on R with tail parameter β ( ) w . Then, -th layer pre-activations are generalized Weibull-tail on R with tail parameter β ( ) such that 1

.

Proof The goal is to show that h ( )

. We proceed by induction on the layer depth .

(i) First hidden layer For = 1, by Lemma 3.1, products w

(1) ij h

i for i = 1, . . . , H 1 satisfy the positive dependence condition of Definition 2, thus the sum g (1)

w . Since ReLU function does not change the right tail and post-activations lie completely on R + , we have h

(1) is symmetric generalized Weibulltail with the same tail parameter. The ReLU activation does not impact the right tail, therefore, h ( )