Frank Vega
email: vega.frank@gmail.com

[Corrigendum] Note on the Riemann Hypothesis: International Conference on Recent Developments in Mathematics (ICRDM 2022)

Keywords: Riemann hypothesis, Prime numbers, Sum-of-divisors function, Robin's inequality, Superabundant numbers MSC Classification: 11M26, 11A41, 11A25

Robin's criterion states that the Riemann hypothesis is true if and only if the inequality σ(n) < e γ • n • log log n holds for all natural numbers n > 5040, where σ(n) is the sum-of-divisors function of n and γ ≈ 0.57721 is the Euler-Mascheroni constant. We require the properties of superabundant numbers, that is to say left to right maxima of n → σ (n) n . In this note, using Robin's inequality on superabundant numbers, we prove that the Riemann hypothesis is true.

Introduction

This is a "Corrigendum" for a paper presentation at the ICRDM 2022 and an extension of the article published by The Ramanujan Journal [1, pp. 14-17], [2, pp. 750-753]. The Riemann hypothesis is a conjecture that the Riemann zeta function has its zeros only at the negative even integers and complex numbers with real part 1 2 . It is considered by many to be the most important unsolved problem in pure mathematics. It was proposed by Bernhard Riemann (1859). The Riemann hypothesis belongs to the Hilbert's eighth problem on David Hilbert's list of twenty-three unsolved problems. This is one of the Clay Mathematics Institute's Millennium Prize Problems. As usual σ(n) is the sumof-divisors function of n d|n d,

where d | n means the integer d divides n. Define f (n) as σ(n)
n . We say that Robin(n) holds provided that

f (n) < e γ • log log n,
where the constant γ ≈ 0.57721 is the Euler-Mascheroni constant and log is the natural logarithm. The Ramanujan's Theorem stated that if the Riemann hypothesis is true, then the previous inequality holds for large enough n. Next, we have the Robin's Theorem: In 1997, Ramanujan's old notes were published where he defined the generalized highly composite numbers, which include the superabundant and colossally abundant numbers [START_REF] Nicolas | Highly Composite Numbers by Srinivasa Ramanujan[END_REF]. Superabundant numbers were also defined by Leonidas Alaoglu and Paul Erdős (1944). Let q 1 = 2, q 2 = 3, . . . , q k denote the first k consecutive primes, then an integer of the form [5, pp. 367]. A natural number n is called superabundant precisely when, for all natural numbers m < n f (m) < f (n).

k i=1 q ai i with a 1 ≥ a 2 ≥ . . . ≥ a k ≥ 1 is called a Hardy-Ramanujan integer
We know the following property for the superabundant numbers:

Proposition 2 If n is superabundant, then n is a Hardy-Ramanujan integer [6, Theorem 1 pp. 450].
A number n is said to be colossally abundant if, for some ϵ > 0,

σ(n) n 1+ϵ ≥ σ(m) m 1+ϵ for (m > 1).
There is a close relation between the superabundant and colossally abundant numbers.

Proposition 3 Every colossally abundant number is superabundant [6, pp. 455].

Several analogues of the Riemann hypothesis have already been proved. Many authors expect (or at least hope) that it is true. However, there are some implications in case of the Riemann hypothesis might be false. Putting all together yields the proof of the Riemann hypothesis.

Main Results

The following is a key Lemma.

Lemma 1 If the Riemann hypothesis is false, then there are infinitely many superabundant numbers n such that Robin(n) fails.

Proof This is a direct consequence of Propositions 1, 3 and 4. □

For every prime number q k > 2, we define the sequence:

Y k = e 0.2 log 2 (q k ) (1 -1 log(q k ) )
.

As the prime number q k increases, the sequence Y k is strictly decreasing [2, Lemma 6.1 pp. 750]. We use the following Propositions:

Proposition 5 [2, Theorem 6.6 pp. 752]. Let k i=1 q ai i be the representation of a superabundant number n > 5040 as the product of the first k consecutive primes q 1 < . . . < q k with the natural numbers a 1 ≥ a 2 ≥ . . . ≥ a k ≥ 1 as exponents. Suppose that Robin(n) fails. Then,

αn < log log(N k ) Y k log log n ,
where N k = k i=1 q i is the primorial number of order k and αn = k i=1 q

a i +1 i q a i +1 i -1
.

Proposition This is the main insight.

Lemma 2 Let k i=1 q ai i be the representation of a superabundant number n > 5040 as the product of the first k consecutive primes q 1 < . . . < q k with the natural numbers a 1 ≥ a 2 ≥ . . . ≥ a k ≥ 1 as exponents. Suppose that Robin(n) fails. Then,

αn < (N k ) Y k n ,
where N k = k i=1 q i is the primorial number of order k and αn = k i=1 q This is the main theorem.

a i +1 i q a i +1 i -1 . Proof When n >
Theorem 1 The Riemann hypothesis is true.

Proof We know there are infinitely many superabundant numbers [6, Theorem 9 pp. 454]. In number theory, the p-adic order of an integer n is the exponent of the highest power of the prime number p that divides n. It is denoted νp(n).

Equivalently, νp(n) is the exponent to which p appears in the prime factorization of n. For every prime q, νq(n) goes to infinity as long as n goes to infinity when n is superabundant [7, Theorem 4.4 pp. 12], [6, Theorem 7 pp. 454]. Let n k > 5040 be a large enough superabundant number such that q k is the largest prime factor of n k where we know that νq k (n k ) = 1 for n k > 5040 [6, Theorem 3 pp. 450]. Suppose that Robin(n k ) fails and thus, we have that necessarily Y k < 1.03352795481 [2, Theorem 6.7 pp. 753]. In the same way, let n k ′ be another superabundant number such that q k ′ > n k , νq k (n k ′ ) > q k and Robin(n k ′ ) fails too. By Lemma 2, we have

αn k < (N k ) Y k n k and αn k ′ < (N k ′ ) Y k ′ n k ′ .
Hence,

αn k ′ • αn k < (N k ′ ) Y k ′ n k ′ • αn k .
Consequently,

αn k ′ • αn k < (N k ′ ) Y k ′ n k ′ • (N k ) Y k n k . So, (αn k ′ • αn k ) 2 < (N k ′ ) Y k ′ n k ′ • (N k ) Y k n k .
However, we know that

(αn k ′ • αn k ) 2 > 1. Moreover, we can see that (N k ′ ) Y k ′ n k ′ • (N k ) Y k n k ≤ 1
since the following inequalities

Y k ≤ log(n k ′ • n k ) log( Y k √ n k ′ • n k ) ≤ log(n k ′ • n k ) log((N k ′ ) Y k ′ Y k • N k ) could simultaneously hold for q k ′ > n k and νq k (n k ′ ) > q k such that log( Y k √ n k ′ • n k ) ≥ log ( k i=1 q νq i (n k ′ ) 1.03352795481 i ) • ( k ′ i=k+1 q Y k ′ Y k i ) ≥ log((N k ′ ) Y k ′ Y k •N k ), because of Y k ′ Y k < 1 and lim k→∞ Y k = 1.
In this way, we obtain the contradiction 1 < 1 under the assumption that Robin(n k ) fails. To sum up, the study of this arbitrary large enough superabundant number n k > 5040 reveals that Robin(n k ) holds on anyway. Accordingly, Robin(n) holds for all large enough superabundant numbers n. This contradicts the fact that there are infinite superabundant numbers n, such that Robin(n) fails when the Riemann hypothesis is false according to Lemma 1. By reductio ad absurdum, we prove that the Riemann hypothesis is true. □

Conclusions

Practical uses of the Riemann hypothesis include many propositions that are known to be true under the Riemann hypothesis and some that can be shown to be equivalent to the Riemann hypothesis. Indeed, the Riemann hypothesis is closely related to various mathematical topics such as the distribution of primes, the growth of arithmetic functions, the Lindelöf hypothesis, the Large Prime Gap Conjecture, etc. Certainly, a proof of the Riemann hypothesis could spur considerable advances in many mathematical areas, such as number theory and pure mathematics in general.

Proposition 4

 4 If the Riemann hypothesis is false, then there are infinitely many colossally abundant numbers n > 5040 such that Robin(n) fails (i.e. Robin(n) does not hold) [3, Proposition 1 pp. 204].

  Proposition 1 Robin(n) holds for all natural numbers n > 5040 if and only if the Riemann hypothesis is true [3, Theorem 1 pp. 188].

  6 [7, Lemma 3.3 pp. 8]. Let x ≥ 11. For y > x, we have

	log log y log log x	<	y x	.


Douglas J Cattani
email: doug.cattani@umanitoba.ca

O Duchene

F Celette

C David

What are the critical phenological periods for sustainable perennial grain production in the annual development of intermediate wheatgrass?

Introduction:

We previously suggested an annual growth cycle for perennial grasses with respect to plant development and seed productivity (1). This cycle begins after harvest of the current year's seed crop and extends through to the next seed harvest, crossing calendar years. Others have shown that environmental conditions from a previous growing season have correlations to the current years seed production potential (2). Species will be different (2,3,4,5) and therefore this will require patience as species individual needs must be determined. Intermediate wheatgrass (Thinopyrum intermedium) is approaching the market in North America. A first study has been published with respect to phenological development (7). Understanding the phenological development will aid in developing effective agronomic practices for sustained grain productivity.

Results to date: Plant growth and development are impacted by growing conditions and are dominated by two measures; precipitation (ppt) and gdd (Table 1). Precipitation may have a greater influence on plant development in an IWG crop. Post-harvest regrowth (Figure 2) is important and is related to gdd and ppt. It appears that while temperature is the primary driver, as long as moisture is adequate, gdd can be adequate to set-up yield potential (reproductive tiller density). In years with short fall regrowth periods, added nutrients can aid in providing yield potential (data not shown (Figure 3). Vernalisation is required (Figure 4). In Manitoba, spring initiation is variable (Table 2), however growth rarely is great prior to early May (days to 100 accumulated gdd base T4°C). Due to a dual induction requirement (6), a specific nightlength needs to be reached for RTE (Figure 5). GDD from the beginning of RTE to first anthesis (Figure 6) appears to be in the 12 to 14 hours of daylight range (Table 3). Observations in France have indicated that RTE is most likely initiated at the 13-hour day-length (Figure 5).

Therefore, timing of RTE initiation and gdd accumulation will influence agronomic practices. In Manitoba, the onset of RTE conditions generally coincides with the onset of spring (Table 2), limiting response to spring applied nutrients but providing for rapid weed competitiveness.

Growth through anthesis appears to be dependent on gdd accumulation with a base temperature of 3°or 4°C. After anthesis, seed development and maturation takes place. The duration to maturity and the size of seed (Figure 7) will be dependent upon two main factors, number of seeds set inflorescence -1 and climate (gdd and ppt) (Figure 8). The interaction of these factors provides the realization of seed yield at harvest (Figure 9). At this point the growth and development cycle begins anew and harvest timing and ensuing growth conditions determine inflorescence density for the following growing season. As the stand ages, nutrient status appears to increase in importance. Impacts at any stage may influence succeeding stages and therefore grain yield.

Conclusions: Reproductive tiller initiation day-length is most likely 13 hours and gdd base T is possibly 3°or 4°C (for Manitoba). This will be refined as we accumulate more data on adapted germplasm and with additional year-to-year variability. Data from other growth environments will likely provide the conclusive evidence. A gdd x precipitation interaction in the fall regrowth period in Manitoba requires a much greater volume of work to allow for resolution.

Acknowledgements: Funding from Manitoba Wheat and Barley Growers Assoc., Growing Forward II, NSERC Strategic, Canadian Agricultural Partnership

FigureFigure 5 :

 5 Figure 4: Pre-elongation of reproductive apical meristems.

Figure 6 .

 6 Figure 6. Flowering and pollination in intermediate wheatgrass.

Figure

  Figure 7. Seed development within a spikelet.

Table 2 for Manitoba spring dates). Means and standard deviations were used to calculate coefficients of variation for the onset of anthesis.

1. Cattani and Asselin, 2018, CJPS 98: 235-246. 2. Khanal, 2017, CPS-CSA meeting, personal communication. 3. Jonsdottir (1992) J. of Veg. Sci. 2:89-94. 4. Thompson and Clark 1991, CJPS 73:569-575. 5. Cattani et al., 2004. CJPS 84:117-124. 6. Heide (1994) New Phytol. 128:347-362. 7. Jungers et al. 2018, Annals Appl. Biol. 172:346-354.

Table 1 . Fall gdd and ppt and impact on harvest at Carman and Winnipeg, MB. Carman ppt (mm) kg ha -1 year date gdd fall harvest harvest yield 2016-17 Nov 16 681 127 361 Aug 08 950 2017-18 Oct 25 709 115 357 July 31 842 Winnipeg 2016-17 Nov 16 724 128 351 Aug 07 866 2017-18 Oct 25 715 85 242 Aug 01 387Table 3 . Selected growing degree-day analysis from Carman 2012-2019.

 13

		ytd5 1	h125	h135	h145	h155
	mean	733.60	729.21	724.35	671.40	525.74
	s.d.	34.89	34.91	35.29	38.53	124.12
	c.v.	4.76	4.79	4.87	5.74	23.61
		ytd4	h124	h134	h144	h154
	mean	810.74	804.59	797.21	736.10	571.13
	s.d.	39.54	38.36	37.88	45.95	139.33
	c.v.	4.88	4.77	4.75	6.24	24.40
		ytd3	h123	h133	h143	h153
	mean	890.79	882.68	872.18	797.35	616.80
	s.d.	45.55	42.72	41.25	55.61	154.85
	c.v.	5.11	4.84	4.73	6.97	25.11
		ytd0	h120	h130	h140	h150
	mean	1152.02	1130.55	1105.76	992.24	754.30
	s.d.	78.09	67.89	61.56	89.10	201.87
	c.v.	6.78	6.00	5.57	8.98	26.76

ytd5 -year to date -5°C based temperature, h12 -12hour day-length accumulation, s.d. standard deviation of the mean, c.v. coefficient of variation.


Raphaël Bailly
email: rbailly@univ-paris1.fr

Laurent Leblond
email: laurent.leblond1@stellantis.com

Kata Gábor
email: kata.gabor@inalco.fr

Syntax and Geometry of Information

This paper presents an information-theoretical model of syntactic generalization. We study syntactic generalization from the perspective of the capacity to disentangle semantic and structural information, emulating the human capacity to assign a grammaticality judgment to semantically nonsensical sentences. In order to isolate the structure, we propose to represent the probability distribution behind a corpus as the product of the probability of a semantic context and the probability of a structure, the latter being independent of the former. We further elaborate the notion of abstraction as a relaxation of the property of independence. It is based on the measure of structural and contextual information for a given representation. We test abstraction as an optimization objective on the task of inducing syntactic categories from natural language data and show that it significantly outperforms alternative methods. Furthermore, we find that when syntax-unaware optimization objectives succeed in the task, their success is mainly due to an implicit disentanglement process rather than to the model structure. On the other hand, syntactic categories can be deduced in a principled way from the independence between structure and context.

Introduction

In the context of both human learning and statistical machine learning, what distinguishes generalization from memorization is the process of deliberately ignoring a part of the input information. In machine learning, the generalization will be guided towards some specific direction by the learning hypothesis: the model structure and the choice of regularization impacts the nature of the information loss. We can talk about syntactic generalization from textual data when the information pertaining to sentence structure tend to be preserved by the model and the information pertaining to other aspects of the text tend to be ignored. Syntactic generalization is of great interest because the human capacity to assign an abstract structure to utterances is a prerequisite to creatively combine constituents and understand novel sentences [START_REF] Frege | Über Sinn und Bedeitung[END_REF]. Knowledge of syntax can boost the robustness of NLP applications with respect to unseen data, in particular when there is a distribution shift [START_REF] He | Enhancing generalization in natural language inference by syntax[END_REF][START_REF] Wu | Improving the robustness of deep reading comprehension models by leveraging syntax prior[END_REF]. In a broader perspective, understanding syntactic generalization informs the discussion on the learnability of syntax from unlabelled text without any built-in grammatical knowledge or inductive bias [START_REF] Gold | Language identification in the limit[END_REF][START_REF] Clark | Unsupervised learning and grammar induction[END_REF][START_REF] Bailly | Emergence of syntax needs minimal supervision[END_REF]. Finally, studying syntactic generalization in large language models (LLMs) sheds light on whether and to what extent these models emulate human functioning with respect to linguistic competence.

The prevailing formalization of syntax is by means of algebraic compositional rules operating on a finite set of discrete categories (parts of speech). Language models can acquire syntactic knowledge when they are given specific supervision or bias [START_REF] Dyer | Recurrent neural network grammars[END_REF][START_REF] Shen | Explicitly modeling syntax in language model improves generalization[END_REF][START_REF] Sartran | Transformer grammars: Augmenting transformer language models with syntactic inductive biases at scale[END_REF]. Whether unsupervised settings can lead to syntactic generalization and under which conditions is still unknown. Current LLMs use distributed representations that cannot be unequivocally mapped to a set of categories, let alone syntactically meaningful categories. The question whether their representations encode syntactic information and how to uncover it is actively investigated today [START_REF] Hu | A systematic assessment of syntactic generalization in neural language models[END_REF][START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF]. The majority of works in the topic of syntactic generalization in language models adopt an empirical approach, such as probing or analysis of a model with a comparison to actual or expected human performance on linguistically motivated tasks. In contrast, we present a theoretical approach to formalize syntactic generalization in an information theoretical framework.

Statistical learning can be formulated as the minimization of KL-divergence -a measure of information loss -subject to constraints. The constraints on model expressivity ensure that generalization takes place by eliminating the information resulting from sampling noise. We claim that the training objective of maximum likelihood estimation by nature does not incentivize models to syntactic generalization. In the case of syntactic generalization, the information loss needs to be directed to non-structural information, which is only remotely related to the elimination of sampling noise. First, a corpus is not randomly sampled from the set of grammatical sentences. Word co-occurrences in a corpus are indeed influenced by different factors such as semantics and pragmatics. The process of abstracting away from these factors is arguably different from the concept of generalization in machine learning, as the acquisition of syntactic knowledge always involves a shift of distribution [START_REF] Hupkes | State-of-the-art generalisation research in NLP: a taxonomy and review[END_REF]. Second, the target of generalization is the capacity to recognize the set of grammatical sentences: well-formedness is inherently a binary notion rather than a probabilistic one. These considerations motivate our proposition to decompose a corpus distribution as a factor of semantic/pragmatic context and a factor of structure representing well-formedness. In what follows, we reinterpret syntactic generalization based on the separation of structural and semantic information, and we show that our approach outperforms concurrent methods on unsupervised POS induction. We also define the notion of abstraction, an optimization objective specifically conceived for disentangling semantic information and syntactic well-formedness.

Related work

Generative linguists agree on the nativist argument that learners cannot converge on the same syntax unless some of their linguistic knowledge is innate [START_REF] Baker | Syntactic theory and the projection problem[END_REF][START_REF] Chomsky | Aspects of the Theory of Syntax[END_REF][START_REF] Chomsky | Reflections on language[END_REF], which makes the complete unsupervised learning of syntax impossible. Therefore, theoretical linguistics demonstrated little interest in machine learning and the interaction between the two fields is limited [START_REF] Lappin | Machine learning theory and practice as a source of insight into universal grammar[END_REF][START_REF] Linzen | Syntactic structure from deep learning[END_REF]. 1With the recent advent of large language models [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF] it has become relevant to test their linguistic competence [START_REF] Linzen | Assessing the ability of LSTMs to learn syntax-sensitive dependencies[END_REF][START_REF] Belinkov | Analysis methods in neural language processing: A survey[END_REF][START_REF] Baroni | Linguistic generalization and compositionality in modern artificial neural networks[END_REF]. Researchers in NLP thus turned to linguistic theory to create probing tasks [START_REF] Alain | Understanding intermediate layers using linear classifier probes[END_REF][START_REF] Giulianelli | Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information[END_REF] or test sets targeted at specific linguistic knowledge [START_REF] Linzen | Assessing the ability of LSTMs to learn syntax-sensitive dependencies[END_REF]. Linguistic challenges like long-distance agreement [START_REF] Linzen | Assessing the ability of LSTMs to learn syntax-sensitive dependencies[END_REF], hierarchical syntax [START_REF] Lin | Open sesame: Getting inside BERT's linguistic knowledge[END_REF][START_REF] Dyer | Recurrent neural network grammars[END_REF][START_REF] Conneau | What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties[END_REF][START_REF] Hupkes | Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure[END_REF], parts of speech [START_REF] Saphra | Language models learn POS first[END_REF][START_REF] Kim | Testing for grammatical category abstraction in neural language models[END_REF], or morphology [START_REF] Belinkov | What do neural machine translation models learn about morphology[END_REF][START_REF] Peters | Deep contextualized word representations[END_REF] have been applied to probe the latest language models with contrasting results. Recently, probing classifiers have also been subjected to methodological criticism. Models can succeed on some test tasks by learning shallow heuristics [START_REF] Mccoy | Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference[END_REF][START_REF] Poliak | Hypothesis only baselines in natural language inference[END_REF]. It was also argued that the presence of sufficient information to learn a given task does not entail alone that models rely on it [START_REF] Ravichander | Probing the probing paradigm: Does probing accuracy entail task relevance[END_REF]; [START_REF] Hewitt | Designing and interpreting probes with control tasks[END_REF]; [START_REF] Xu | A theory of usable information under computational constraints[END_REF]. On rarer occasions, studies aimed to test the capacity of language models to predict grammaticality judgments. Out of distribution testing systematically shows that the performance drops when the test data contains natural or artificial examples which are deliberately different from the training examples (Lake and [START_REF] Brenden | Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks[END_REF][START_REF] Marvin | Targeted syntactic evaluation of language models[END_REF][START_REF] Absar | RNN simulations of grammaticality judgments on long-distance dependencies[END_REF][START_REF] Van Schijndel | Quantity doesn't buy quality syntax with neural language models[END_REF][START_REF] Hall | Do syntactic probes probe syntax? Experiments with Jabberwocky Probing[END_REF].

Another branch of model analysis and interpretation studies are concerned with the nature of the generalization that takes place, with a particular accent on the notion of compositionality [START_REF] Loula | Rearranging the familiar: Testing compositional generalization in recurrent networks[END_REF][START_REF] Baroni | Linguistic generalization and compositionality in modern artificial neural networks[END_REF][START_REF] Valvoda | Benchmarking compositionality with formal languages[END_REF]. Among others, [START_REF] Fodor | Compositionality Papers[END_REF] and [START_REF] Kottur | Natural language does not emerge 'naturally' in multi-agent dialog[END_REF] claim that syntactic compositionality [START_REF] Chomsky | Syntactic Structures[END_REF][START_REF] Chomsky | Aspects of the Theory of Syntax[END_REF] is a prerequisite to learn to generalize to complex unseen input. In empirical studies, [START_REF] Gulordava | Colorless green recurrent networks dream hierarchically[END_REF] and [START_REF] Lakretz | The emergence of number and syntax units in LSTM language models[END_REF] report a lack of compositionality in the models they analyse, despite their impressive performance. In contrast, [START_REF] Bastings | Jump to better conclusions: SCAN both left and right[END_REF] and [START_REF] Valvoda | Benchmarking compositionality with formal languages[END_REF] find that some compositional relations can be learned by neural sequenceto-sequence models. [START_REF] Chaabouni | Compositionality and generalization in emergent languages[END_REF] argue that there is no correlation between the compositionality of an emergent language and its ability to generalize.

The problem of conflation between semantic and syntactic information in language models has been identified (Maudslay and Cotterell, 2021) as a factor hindering syntactic generalization. A new line of research is concerned with disentangling syntactic and semantic information in representations [START_REF] Felhi | Disentangling semantics in language through vaes and a certain architectural choice[END_REF][START_REF] Huang | Disentangling semantics and syntax in sentence embeddings with pre-trained language models[END_REF] by adversarial training or syntactic supervision. In order to incite syntactic generalization in models, [START_REF] Shen | Explicitly modeling syntax in language model improves generalization[END_REF] and [START_REF] Dyer | Recurrent neural network grammars[END_REF] propose to integrate explicit syntactic information for language modelling. [START_REF] Hu | A systematic assessment of syntactic generalization in neural language models[END_REF] show that there is a trade-off between the general language modelling objective and syntax-specific performance.

Some recent work relies on information theory to improve our understanding of the syntactic knowledge in LMs. [START_REF] Pimentel | Information-theoretic probing for linguistic structure[END_REF] reformulates probing as approximating mutual information between a linguistic property and a contextual representation. Subsequently, [START_REF] Pimentel | A bayesian framework for information-theoretic probing[END_REF] introduced Bayesian Mutual Information, a definition that allows information gain through processing. [START_REF] Voita | Information-theoretic probing with minimum description length[END_REF] use Minimum Description Length to measure regularity in LM representations with respect to the labels to be predicted in a linguistic probe. Our work builds on the propositions formulated in [START_REF] Bailly | Emergence of syntax needs minimal supervision[END_REF] who address the problem of the learnability of grammar by separating syntactic and semantic information in a corpus.

Syntactic Representation

Autonomy of Syntax

The concept of generalization we introduce is based on the autonomy of syntax [START_REF] Chomsky | Syntactic Structures[END_REF][START_REF] Chomsky | Some concepts and consequences of the theory of government and binding[END_REF][START_REF] Adger | The autonomy of syntax[END_REF] reinterpreted in terms of statistical independence. In the process of linguistic generalization, learners need to abstract away from semantic, pragmatic and idiosyncratic lexical information in the input they are exposed to. With a string prediction task and likelihood maximization as a training objective, models have no incentive to abstract away from these features. One can expect a statistical learner to ignore sampling noise, but the above features are relevant to learn the distribution behind a corpus. This insight motivates our proposition of statistical abstraction, a training objective that focuses on certain aspects of the input while deliberately ignoring others.

We want our learner to concentrate on the structure and ignore the factors we call context, i.e. all the aspects that are unrelated to well-formedness. We do so by creating two representations of the input: one of them structured, the other having structural information removed but co-occurrence relations conserved.

Let us consider a small artificial example for illustration. Our observation is a corpus with the two sentences below: cats eat mice men build houses A valid syntactic generalization would recognize the sentence cats build mice as grammatical. In order to do so, we consider p(cats eat mice) as a factor of the probability of the co-occurrence of its words in the same context : p({cats, eat, mice}) and a factor of the probability of the words to appear in a given structure : p(cats eat mice|{cats, eat, mice})

A syntactic representation with a desirable degree of generalization would identify the distributional classes {cats, men}, {build, eat}, {mice, houses}.

This set of distributional classes can be seen as a function f that associates a word (e.g cats) with its class ({cats, men}). Our goal is to study the properties of such a function so that it can be considered as achieving syntactic generalization, for instance: p(cats eat mice|{cats, eat, mice}) can be deduced from

p(f(cats) f(eat) f(mice)|{f(cats), f(eat), f(mice)})

Properties of a Syntactic Partition

We define the probability distribution that predicts the grammaticality of sequences, learned from observation. In order to do so, we first define a partition of words into abstract categories. This mapping, together with the category sequences found in the corpus, will allow us to induce the grammar.

Behind the corpus data there is a probability distribution p(w 1 w 2 . . . w n ). This distribution can be written as a product of two factors. First, the unstructured data, i.e. the probability of the elements of the vocabulary to occur in the same sequence without considering their order. Second, the probability of these elements to be observed in a particular structure. The contextual information is related to the former, and the structural information to the latter.

Let us see a probabilistic interpretation. Let A be the vocabulary, one defines the set A + = A * \ε where ε is the empty sequence. w = w 1 . . . w n ∈ A n a sequence (of words) of length n and p({w 1 , . . . , w n }) the probability of observing these elements in the same sequence, in any order. A trivial decomposition of p(w 1 w 2 w 3 ) would be

p({w 1 , w 2 , w 3 })p(w 1 w 2 w 3 |{w 1 , w 2 , w 3 })
However, we want structural information to be independent of the context. The decomposition above does not suppose the autonomy of structure. We propose to transform the above distribution with a mapping f , which will induce a partition over the elements of the vocabulary. In what follows, we examine which properties of this mapping will ensure that the categories of the resulting partition do not contain contextual information, while still preserving the information necessary to predict grammaticality. Let A be the vocabulary, one defines the set A + = A * \ε where ε is the empty sequence.

w = w 1 . . . w n ∈ A n |w| = length of w f (w) = f (w 1 ) . . . f (w n ) f [w] = {w ′ ∈ A + | f (w ′ ) = f (w)} for σ ∈ S n , σ(w) = w σ(1) . . . w σ(n) ⟨⟨W ⟩⟩ = ∪ σ∈Sn,w∈W {σ(w)} µ(w) = card({σ ∈ S n | w = σ(w)})
Let f (w) denote the sequence of categories resulting from the mapping of a word sequence, and f [w] the set of sequences that map to f (w). W denotes a set of sequences w. In the case of a singleton we will denote ⟨⟨w⟩⟩ = ⟨⟨{w}⟩⟩. The contextual information will be modeled through the probability p(⟨⟨w⟩⟩), where one can see the object ⟨⟨w⟩⟩ as a bag of words, from which the information of the structure (order) has been erased. A syntactically relevant representation needs to meet two criteria: it has to allow to recover the structure, i.e. the ordering of the bag of words, and it needs to be independent of contextual information. The first criterion is defined below as factorization, the second as minimality.

Factorization. One will say that a mapping f factorises a distribution p if the order of a bagof-words {w i } drawn from p can be entirely deduced from the knowledge of the corresponding categories. Definition 1. Let p be a distribution over A + , and f : A → B be a mapping. The distribution p is factorised by f if there exists a mapping λ f (⟨⟨w⟩⟩)

such that ∀w ∈ A + p(w | ⟨⟨w⟩⟩) = λ f (⟨⟨w⟩⟩) p(f [w] | ⟨⟨f [w]⟩⟩) in that case, one has λ f (⟨⟨w⟩⟩) = µ(w)
µ(f (w)) . In the case where f factorises p, one will say that context and structure are independent conditionally to f . Independence. As the property of factorization does not guarantee the complete independence of structure and context (for instance the identity always factorises p), we need to limit the information carried by f to its minimal value in order to reach this independence. From f [w] one can deduce, at the minimum, the length of w. The purpose of minimality is to ensure that knowing f [w] provides no further information for finding w: Definition 2. Let p be a distribution over A + and let f : A → B be a mapping. We will say that f is (information)-minimal for p if

∀w ∈ A + , p(w | f [w]) = p(w | A |w| )
We will say that context and structure are independent in p if there exists an informationminimal factorization of p.

Induced grammar

From a probability distribution p and a mapping f , it is possible to induce a syntax based on the observed patterns: a sequence is structurally correct if its pattern corresponds to an observed pattern. Definition 3. Let p be a distribution over A + and let f : A → B be a mapping. One denotes the syntax induced by p and f by

w ∈ G(p, f ) ⇔ p(f [w]) > 0
One has for instance G(p, id) = supp(p): this representation is a memorization with no generalization.

Minimal syntax. A syntax induced by minimal factorization of p will be called minimal syntax. The set of all minimal syntaxes will be denoted G * (p).

It can be shown that the intersection of all minimal syntaxes of p is a minimal syntax of p:

G * (p) = ∩ f ∈G * (p) G(p, f ) ∈ G * (p)
Hence, if the independence between context and structure holds, there exists a canonical way to define the set of well-structured sequences which is different from the support of p.

Example 1. Let us consider the first example above: let p be the distribution defined by p(cats eat mice) = 1 2 p(men build houses) = 1 2 then the mapping f defined by

f (cats) = f (men) = b 0 f (eat) = f (build) = b 1 f (mice) = f (houses) = b 2
is a minimal factorization of p. The minimal syntax G * (p) is the set cats eat mice cats eat houses cats build mice cats build houses men eat mice men eat houses men build mice men build houses

Geometry of Information

Using information theoretical tools, we transform the criteria above into metrics and define an information space which allows to track the amount of contextual and structural information in a partition, as well as the direction of generalization during a training process.

The concept of minimal factorization provides the formal definition of minimal syntax; however, the conditions of factorization (Definition 1) and minimality (Definition 2) are restrictive. In natural language corpora, a perfect independence between semantic context and grammaticality cannot be expected. Syntax and semantics do interface in natural language, semantic acceptability interacts with grammaticality and depending on how one deals with this interface, either the assumption of perfect independence or the precise retrieval of the distribution underlying the corpus may not be met. This motivates our methodology for relaxing both conditions in a way that gives an equivalent but quantifiable formulation for each criterion in terms of information. We thus provide a method to measure the amount of structural information present in a partition, hence relaxing the factorization criterion. We also define contextual information, which relaxes the minimality requirement.

Structural information

Let

H(p ∥ q) = - w∈A + p(w) log(q(w))
be the cross entropy of the distribution q with respect to the distribution p.

For a distribution p over A + , we will consider the distance (in terms of cross-entropy) between p and the class of factorised distributions. Definition 4. Let p be a distribution over A + and let f be a mapping. One denotes:

F f = {q | q is factorised by f }
and one defines the projection of p conditionally to f by

p |f = arg min q∈F f H(p ∥ q)
The structural information of f with respect to p is given by

i s (p ∥ f ) = H(p ∥ p |z ) -H(p ∥ p |f )
where z is the null mapping.

The set F f represents the set of distributions for which the knowledge of f is sufficient to recover the order of a sequence. The structural information is minimal for z, and maximal for the identity (see Appendix):

i s (p ∥ z) = 0 ≤ i s (p ∥ f ) ≤ i s (p ∥ id)
The link between structural information and factorization is given by: Lemma 1. Let p be a distribution over A + and let f : A → B be a mapping. One has

i s (p ∥ f ) is maximal ⇔ f factorises p

Contextual information

An optimal syntactic representation is one that fulfills the independence requirement: the probability of a sequence of categories does not provide information about which actual words are likely to appear in the sentence. The contextual information will measure the amount of lexical or semantic information that is present in a representation.

Let

H(p) = H(p ∥ p)
be the Shannon entropy. Let p be a distribution over A + and let f : A → B be a mapping. One will denote p • f -1 the distribution on B + induced by f . One has p

• f -1 (f (w)) = p(f [w]).
Definition 5. The contextual information of f with respect to p is given by

i c (p ∥ f ) = H(p • f -1 ) -H(p • z -1 )
where z is the null mapping.

From standard properties of Shannon entropy, i c (p ∥ f ) is minimal for z, and maximal for the identity (see Appendix):

i c (p ∥ z) = 0 ≤ i c (p ∥ f ) ≤ i c (p ∥ id) The maximum value of i c (p ∥ f ) is reached for H(p • f -1 ) = H(p).
The link between contextual information and information-minimality is given by: Lemma 2.

i c (p ∥ f ) = 0 ⇔ f is minimal for p

Representation of a mapping in the information space

Let us now consider how to represent geomtrically the two types of information in a partition. For a given distribution p, any mapping f will be represented in R 2 by its coordinates

x f = i s (p ∥ f ), y f = i c (p ∥ f )
Example 2. Let us consider the same distribution as in Example 1. Fig. 1 represents all possible mappings g by the point with coordinates

(i s (p ∥ g), i c (p ∥ g)).
Details are in the Appendix. One can check that the minimal factorization is in (1, 0), and the second closest mapping to a minimal factorization is {cats, mice, men, houses}{eat, build}.

Abstraction

Abstraction relaxes the definition of a minimal factorization of p in terms of a solution to an optimization problem. For a given probability distribution p and a mapping f , the abstraction measures the distance between f and the position of a minimal factorization of p in the information space: Definition 6. Let p be a distribution over A + and let f :

A → B be a mapping. Let d be a distance on R 2 . Let t f = (i s (p ∥ f ), i c (p ∥ f )) and t * = (i s (p ∥ id), 0)). The abstraction (w.r.t. d) is defined as α d (p ∥ f ) = e -d(t f ,t * )
One has α d (p ∥ f ) ≤ 1, with the maximum value reached iff f is a minimal factorization of p. For a mapping f , maximizing abstraction can be considered as a relaxation of the property of being a minimal factorization.

Minimal Syntax Identification

We prove here that abstraction can be used to identify the set of minimal syntaxes of p from a sample.

Consistency of the plug-in estimator of abstraction. In the case where the set of of possible sequences is infinite, it is not possible to ensure a convergence rate of the abstraction (cf. [START_REF] Antos | Convergence properties of functional estimates for discrete distributions[END_REF]). Nevertheless, it is possible to show the following consistency result: Proposition 1. Let p be a distribution over A + , and let d be a distance on R 2 . Let pN be the empirical distribution derived from an i.i.d. sample of size N drawn from p.

The plug-in estimator for the abstraction α d (p||f ) is consistent:

α d (p N ∥ f ) N →∞ -→ α d (p ∥ f ) a.s.
As a consequence, when the vocabulary A is finite, abstraction can be used to isolate the set of minimal factorizations of p.

Corollary 1. Let p be a distribution over A + , with |A| < ∞. Let d be a distance measure on R 2 . Let pN be the empirical distribution derived from an i.i.d. sample of size N drawn from p.

Then one has:

lim N →∞ P [G(p, f * d (p N )) ∈ G * (p)] = 1
where f * d (p N ) maximizes abstraction for pN .

Experiments

We test abstraction as an optimization objective for learning syntactic representations, when the representation takes the form of a mapping into discrete syntactic categories. The results are evaluated on an unsupervised POS induction task. While our understanding of a syntactic category may not perfectly overlap with actual parts of speech (the latter being defined on the basis of a mixture of criteria instead of pure syntax, and are usually more coarse-grained than real distributional categories), this task will allow a good comparison with concurrent models on a gold standard.

In NLP, part-of-speech categories are usually a part of a probabilistic model; typically a parameter which will be tuned during learning. For instance, if the model is an HMM, its hidden states correspond to POS categories. If the model is a PCFG, categories will correspond to non-terminals. We call this approach -when POS categories are deduced from a given model structure as a parameter -the model-specific approach. In the experiments, we compare the model-specific approach with our hypothesis: that POS categories can be deduced from the independence of structure and context. We consider the task of unsupervised POS induction, and compare the accuracy of the abstraction maximization criterion with model-specific crossentropy minimization.

The corpus we use comes from Wikipedia in simplified English, contains 430k sentences, 8M tokens, and was POS tagged by the Stanford POS tagger [START_REF] Toutanova | Feature-rich part-ofspeech tagging with a cyclic dependency network[END_REF]. To create the target partition, words (a vocabulary of 6044 elements) were assigned to their most frequent POS. There are 36 POS categories.

The target partition in the information space

We created (Fig. 2) the information space for the Wikipedia corpus with the coordinates indicating structural and contextual information. We represented the target partition (36 categories, correct mapping), and located randomly generated modifications of this partition obtained by changing 1) the assignment of words to the target POS categories (in red) and 2) the number of categories between 2 and 2000 (partitions with > 36 categories are in yellow, partitions with < 36 categories in green) by merging or splitting existing categories. First, it can be observed that any random modification of the target partition (whether it increases or decreases the information) comes at the expense of the abstraction objective. This distinctive position of the syntactic partition could not be visualized in one dimension, suggesting the relevance of the coordinates in the information space in identifying it.

Second, with a strict constraint on the number of categories, the representation of the noisy target (in red) indicates a negative correlation between contextual information and structural information: a trade-off induced by the limitation of information capacity. The choice of normalised d || 2 distance for abstraction is driven by the shape of random partitions in the information space (in blue).

POS induction

We compare abstraction and likelihood maximization as training objectives for unsupervised POS induction. The most efficient POS induction methods at present are mainly -if not exclusively -based on models derived from HMM [START_REF] Peter | Class-based n-gram models of natural language[END_REF][START_REF] Merialdo | Tagging English text with a probabilistic model[END_REF][START_REF] Lin | Unsupervised POS induction with word embeddings[END_REF][START_REF] Stratos | Unsupervised part-of-speech tagging with anchor hidden Markov models[END_REF][START_REF] Ke | Unsupervised neural hidden Markov models[END_REF][START_REF] He | Unsupervised learning of syntactic structure with invertible neural projections[END_REF]. We experiment with different variations of the model by [START_REF] Peter | Class-based n-gram models of natural language[END_REF], because the method is purely distributional, involves discrete embeddings and is still competitive (cf. [START_REF] Stratos | Unsupervised part-of-speech tagging with anchor hidden Markov models[END_REF][START_REF] Christodoulopoulos | Two decades of unsupervised POS induction: How far have we come[END_REF]).

As we cannot perform a brute-force search for the best possible partitions for our criteria, we replaced it by a local measure of the performance : for every single word, provided that all other words are correctly classified, we checked whether the criterion would attribute the correct POS category. Accuracy indicates the rate of correctly classified words.

Tested models We will call plain model the general form of a distribution p factorised by a mapping f :

p(f [w])p(⟨⟨w⟩⟩ | ⟨⟨f [w]⟩⟩) µ(w) µ(f (w))
We can add model-specific constraints:

(MK): Markov constraint for p(f [w]) = p(f (w 1 )) n i=2 p(f (w i ) | f (w i-1 )) (CI): contextual independence constraint for p(⟨⟨w⟩⟩ | ⟨⟨f [w]⟩⟩) = n i=1 p(w i | f (w i )) µ(f (w)) µ(w)
We will consider the normalised α || 2 abstraction maximization objective, and the likelihood maximization objective (with a constraint on the number of categories) for the plain model alone, with contextual independence (CI) constraint, with Markov (MK) constraint, or with both constraints (MK) + (CI) (Brown clustering criterion).

The results are shown in Figure 3. They indicate that the abstraction criterion significantly outperforms likelihood maximization for any model considered. This reinforces our hypothesis that syntactic categories emerge naturally from the criterion of independence between structure and context, without any assumption about the structure of the model.

The second important finding concerns the phenomenon we call implicit disentanglement. By definition, if we estimate the parameters of a distribution q |f with likelihood maximization, we maximize structural information (i.e. the partition f tends towards the right-most solutions in the information space). However, contextual information will still be present. Syntactic generalization may occur when the encoding capacity of the model is bounded (e.g. by limiting the number of categories), inducing a trade-off between structural and contextual information. A way to estimate the role of implicit disentanglement is to consider the confusion matrix of correctly classified or misclassified words for abstraction maximization classifier (a) and a likelihood maximization classifier (b), and decompose (b) into a convex combination of (a) and an independent classification process (c).

With the confusion matrix for Brown clustering criterion: M = b b a 0.202 0.049 a 0.212 0.537 one obtains that a proportion F = 0.667 of the correct classification of (b) is imputable to (a), and at most 33.3% of correct classification by (b) can be considered as independent from implicit disentanglement. This factor F is known in literature as certainty factor (see [START_REF] Tan | Selecting the right interestingness measure for association patterns[END_REF], Appendix for details)

The accuracy of maximum likelihood with the plain model (no constraint) is a good example of implicit disentanglement : it can only be the result of the limitation on the number of categories. The hatched part in Figure 3 represents the fraction of correct classification due to implicit disentanglement in max-likelihood classifiers. These results indicate that the impact of model structure in the ability to infer syntactic categories (and, more broadly, in syntactic generalization capacity) is over-estimated: parameter tuning seems far less efficient than the application of the principle of independence between context and structure.

Conclusion

As to our current knowledge, language models do not have a convincing performance on modelling grammaticality: despite their impressive results on downstream tasks, they are not good at syntactic generalization unless syntactic knowledge is somehow injected in the system. Moreover, there is a trade-off in large LMs between syntactic generalization and language modelling performance.

We suggest a measurable interpretation of syntactic generalization and show results that align with the observations reported by many authors: training on a natural language corpus (e.g. using language models) results in memorization of semantics and entanglement with syntactic information. This motivates our proposition of abstraction, a new training objective for syntactic generalization without supervision. We prove the statistical consistency of abstraction in the task of grammar identification. Empirical results on an unsupervised POS induction task show that abstraction considerably outperforms concurrent models trained with a likelihood estimation objective, without making any assumptions about the structure of the model.

Limitations

The contribution of this paper is mainly theoretical. Like most of the POS identification algorithms, the optimization of a criterion among the space of all partitions requires the use of heuristics, and finding the optimum is never guaranteed. Additional work is required before a generalization model that is efficient in practice can be obtained.

Acknowledgement

We would like to thank Guillaume Wisniewski and the anonymous reviewers for their valuable comments.

Remark 1. One has

⟨⟨σ(w)⟩⟩ = ⟨⟨w⟩⟩ ⟨⟨f [σ(w])⟩⟩ = ⟨⟨f [w]⟩⟩ f (σ(w)) = σ(f (w)) p(f [w]) = p • f -1 (f (w)) p(⟨⟨f [w]⟩⟩) = p • f -1 (⟨⟨f (w)⟩⟩) Remark 2. One has p(⟨⟨w⟩⟩) = 1 µ(w) σ∈Sn p(σ(w)) Remark 3. One has p(⟨⟨f [w]⟩⟩) = 1 µ(f (w)) σ∈Sn p(f [σ(w)])
Proof. By Remark 2 applied to p • f -1 and Remark 1.

Remark 4. One has

⟨⟨⟨⟨w⟩⟩⟩⟩ = ⟨⟨w⟩⟩ f [f [w]] = f [w] ⟨⟨σ(w)⟩⟩ = ⟨⟨w⟩⟩ ⟨⟨f [σ(w)]⟩⟩ = ⟨⟨f [w]⟩⟩ Proof. The second equality comes from f -1 • f • f -1 • f = f -1 • f Lemma 3.
Let p be a distribution over A + , and f : A → B be a mapping. One supposes that there exists a mapping λ f (⟨⟨w⟩⟩) such that ∀w ∈ A

+ p(w | ⟨⟨w⟩⟩) = λ f (⟨⟨w⟩⟩) p(f [w] | ⟨⟨f [w]⟩⟩) then one has λ f (⟨⟨w⟩⟩) = µ(w) µ(f (w)) .
Proof. One has

∈ A + , p(w | f [w]) = p(w | A |w| ) ⇔ ∀w ∈ A + , p(f [w]) = p(A |w| ) or 0
Proof. Suppose the left hand side of the equivalence, then, for any w ∈ A + , either p(f [w]) = 0 or there exists

w ′ ∈ f [w] such that p(w ′ ) > 0 then p(f [w]) = p(f [w ′ ]) = p(A |w| ).
On the other way, either p(w) = 0 (and the equality holds) or p(w) > 0 implying p(f

[w]) > 0 hence p(f [w]) = p(A |w| ) implying in turn p(w | f [w]) = p(w | A |w| ).
Computing Example 1. Let w 1 be the sentence "cats eat rats" and let w 2 be the sentence "men build houses". one has ⟨⟨w 1 ⟩⟩ = {cats eat mice, cats mice eat, . . . , mice eat cats} ⟨⟨w 2 ⟩⟩ = {men build houses, men houses build, . . . , houses build men}

f (w 1 ) = b 0 b 1 b 2 f (w 2 ) = b 0 b 1 b 2 f [w 1 ] =
{cats eat mice, cats build mice, . . . , men eat mice, men build mice} ⟨⟨f [w 1 ]⟩⟩ = {cats eat mice, cats build mice, . . . , build mice men, men houses eat} One has

p |f (w 1 ) = p(⟨⟨w 1 ⟩⟩)p(f [w 1 ] | ⟨⟨(f [w 1 ]⟩⟩) µ(w 1 ) µ(f [w 1 ]) = 1 2 • 1 • 1 1 hence f factorises p. One has p(w 1 | f [w 1 ]) = 1 2 1 = p(w 1 | A 3 )
hence f is information-minimal for p.

The Lemma 5 gives a formula for the projection p |f of p conditionally to f . Lemma 5. Let p be a probability distribution, and let f be a mapping. Let us define π(p, f ) by

π(p, f )(w) = p(⟨⟨w⟩⟩)p(f [w] | ⟨⟨f [w]⟩⟩) µ(w) µ(f (w)) then one has p |f = π(p, f )
One needs a few steps in order to prove Lemma 5. Lemma 6. Let p be a distribution over A + and f : A → B be a mapping. Then

1. π(p, f )(⟨⟨w⟩⟩) = p(⟨⟨w⟩⟩) 2. π(p, f )(f [w]) = p(f [w]) Proof. 1: one has π(p, f )(⟨⟨w⟩⟩) = 1 µ(w) σ∈Sn π(p, f )(σ(w)) with π(p, f )(σ(w)) = p(⟨⟨w⟩⟩)p(f [σ(w)]) p(⟨⟨f [w]⟩⟩) µ(w) µ(f (w)) and σ∈Sn p(f [σ(w)]) p(⟨⟨f [w]⟩⟩)µ(f (w)) = 1 hence the result. 2: one has π(p, f )(f [w]) = p(⟨⟨f [w]⟩⟩)p(f [f [w]] | ⟨⟨f [f [w]]⟩⟩) µ(f (w)) µ(f (w)) = p(⟨⟨f [w]⟩⟩)p(f [w] | ⟨⟨f [w]⟩⟩) µ(f (w)) µ(f (w)) = p(f [w])
Corollary 3. Let p be a distribution over A + and f : A → B be a mapping. Then

1. π(p, f ) is a probability distribution. 2. π(π(p, f ), f ) = π(p, f ).
Proof.

(1): The ⟨⟨w⟩⟩ form a partition of A + hence

π(p, f )(A + ) = ⟨⟨w⟩⟩ π(p, f )(⟨⟨w⟩⟩) = ⟨⟨w⟩⟩ p(⟨⟨w⟩⟩) = 1
(2): Since π(p, f ) is only computed from p(⟨⟨w⟩⟩) and p(f [w]), with the Lemma 6 one has the conclusion.

Remark 5. In particular, one can give another definition of the set

F f = {q | q is factorised by f } as F f = {π(q, f ) | q is a distribution}
Lemma 7. Let p and q be two probability distributions over A + , and let f : A → B be a mapping. Then one has:

H(p||π(q, f )) ≥ H(p||π(p, f ))
with equality iff π(q, f ) = π(p, f ).

Proof. The inequality is equivalent to

w∈A + p(w) log( q(⟨⟨w⟩⟩) p(⟨⟨w⟩⟩) ) + log( q(f [w])p(⟨⟨f [w]⟩⟩) p(f [w])q(⟨⟨f [w]⟩⟩) ) ≤ 0
By Jensen's inequality and concavity of the log, and summing over ⟨⟨w⟩⟩ one has w∈A + p(w) log( q(⟨⟨w⟩⟩) p(⟨⟨w⟩⟩) ) = ⟨⟨w⟩⟩ p(⟨⟨w⟩⟩) log( q(⟨⟨w⟩⟩) p(⟨⟨w⟩⟩) ) ≤ 0 with equality iff ∀w, q(⟨⟨w⟩⟩) = p(⟨⟨w⟩⟩).

By summing over ⟨⟨f [w]⟩⟩, one has

f [w] q(f [w])p(⟨⟨f [w]⟩⟩) q(⟨⟨f [w]⟩⟩) = ⟨⟨f[w]⟩⟩ q(⟨⟨f [w]⟩⟩)p(⟨⟨f [w]⟩⟩) q(⟨⟨f [w]⟩⟩) = 1
and by Jensen's inequality and concavity of the log, and summing over f [w] one has

w∈A + p(w) log( q(f [w])p(⟨⟨f [w]⟩⟩) p(f [w])q(⟨⟨f [w]⟩⟩) ) = f [w] p(f [w]) log( q(f [w])p(⟨⟨f [w]⟩⟩) p(f [w])q(⟨⟨f [w]⟩⟩) ) ≤ 0 with equality iff ∀w, q(f [w] | ⟨⟨f [w]⟩⟩) = p(f [w] | ⟨⟨f [w]⟩⟩).
Because the value of π(p, f ) only depends on p(⟨⟨w⟩⟩) and p(f [w] | ⟨⟨f [w]⟩⟩), the equality holds in the statement iff π(q, f ) = π(p, f ).

Proof of Lemma 5. One applies Remark 5 and Lemma 7, and one gets the result.

Separating structure from data.

Our goal is to isolate structural information form contextual information for an observation (a 1 , . . . , a n ).

For any permutation σ ∈ S n , the tuple

(a σ(1) , . . . , a σ(n) ) = (a ′ 1 , . . . , a ′ n ) satisfies a relation (a 1 , . . . , a n ) = (a ′ σ -1 (1) , . . . , a ′ σ -1 (n) )
which will be denoted σ -1 .

Definition 7. For an observation X = (a 1 , . . . , a n ), and a permutation σ ∈ S n , let us denote σ(X) = (a σ(1) , . . . , a σ(n) )

For any probability distribution p over A + , one will define

p 2 (X = (a 1 , . . . , a n ), Y = σ) = 1 n! p(σ(X)) Lemma 8. One has p(w) = σ∈Sn p 2 (X = σ -1 (w), Y = σ) p(⟨⟨w⟩⟩) = |w|! p 2 (X = w) µ(w)
where |w| is the length of w.

Proof. The first statement is just straightforward from the definition of p 2 . In particular, one has

p(w) = |w|! p 2 (Y = id, X = w) One has p(⟨⟨w⟩⟩) = 1 µ(w) σ∈Sn p(σ(w)) = 1 µ(w) σ∈Sn,ρ∈Sn p 2 (X = ρ -1 (σ(w)), Y = ρ) = σ∈Sn p 2 (X = σ(w)) µ(w)
and, with the fact that

∀σ ∈ S n , p 2 (X = w) = p 2 (X = σ(w))
one has the result.

Definition 8. Let us define

p 2|f (X = w, Y = σ) = p 2 (X = w)p 2 (Y = σ|f (X) = f (w)) Lemma 9. One has 1. p 2|f = p |f 2 2. H(p 2 ||p |f 2 ) = H p 2 (Y |f (X)) + H p 2 (X) 3. H(p||p |f ) = H(p 2 ||p 2|f ) -E p (log(|w|!)) Proof. 1. One has p |f 2 (X = w, Y = σ) = 1 n! p |f (σ(w)) = 1 n! p |f (⟨⟨σ(w)⟩⟩)p |f (f [σ(w)]) p |f (⟨⟨f [σ(w)]⟩⟩) µ(σ(w)) µ(f (σ(w)))
with, by Lemma 6 and Lemma 8,

p |f (⟨⟨σ(w)⟩⟩) = p(⟨⟨w⟩⟩) = n! p 2 (X = w) µ(w) and p |f (f [σ(w)]) = p(f [σ(w)]) = n!p 2 (f (X) = f (w), Y = σ) and p |f (⟨⟨f [σ(w)]⟩⟩) = p(⟨⟨f [w]⟩⟩) = n! p 2 (f (X) = f (w)) µ(f (w)
and one has the result. The second statement is an application of the definition of p 2|f together with statement 1.

The third statement comes from the fact that

p(w) = σ∈Sn p 2 (X = σ -1 (w), Y = σ) p |f (w) = |w|!p |f 2 (X = σ -1 (w), Y = σ)
one has

H(p||p |f ) = - w p(w) log(p( |f (w)) = - w,σ p 2 (X = σ -1 (w), Y = σ) log(|w|!p |f 2 (X = σ -1 (w), Y = σ)) = H(p 2 ||p 2|f ) -E p (log(|w|!))
Lemma 10. Let p be a probability distribution, and let f and g be two mappings. One has

i s (p, g • f ) ≤ i s (p, f ) Proof. From Lemma 9, one has i s (p||f ) = H p 2 (Y |z(X)) -H p 2 (Y |f (X))
and one has

H p 2 (Y |f (X)) ≤ H p 2 (Y |g • f (X)) hence i s (p||f ) ≥ i s (p||g • f )
Lemma 11. Let p be a probability distribution, and let f and g be two mappings. One has

i s (p, g • f ) ≤ i s (p, f ) Proof. With the fact that H p (f (w)) ≥ H p (g • f (w))
one has the result.

Details of Example 2 . The two sentences are strictly equivalent, thus we will only compute the values for, say, u = cats eat mice . One has p(⟨⟨u⟩⟩) = p({cats, eat, mice} = 1 2 . Let z : A → {a} be a null mapping. By Lemma 5, one has One has p • h -1 (ccc) = 1 2 , thus

p |z (u) = p(⟨⟨u⟩⟩) p(z[u]) p(⟨⟨z[u]⟩⟩) µ(u) µ(z(u))
H(p • h -1 ) = log(2), i c (p ∥ h) = log(2)
One can check that the gain of information from z to g is purely structural, while the gain from z to h is purely contextual.

Proof of Proposition 1. From Lemma 9, one has

i s (p||f ) = H p 2 (Y |z(X)) -H p 2 (Y |f (X))
hence i s is a sum of entropies, as well as i c .

The Corollary 1 in [START_REF] Antos | Convergence properties of functional estimates for discrete distributions[END_REF] states that for a countable support target distribution p with entropy H and its M.L.E. p n with entropy Ĥn , the plugin estimator satisfies lim n→∞ Ĥn = H a.s. and this directly implies the conclusion.

Lemma 12. Let p be a probability distribution, having a minimal factorization f . Then the intersection of all minimal syntaxes of p is a minimal syntax of p: • an independent process of POS identification involving HMM constraints has a 40% success rate.

• the rate of correct classification by (b) is 0.586, decomposed in 0.195 of success from an independent process, and 0.391 due to implicit disentanglement.

• 66.7% of the overall success rate can be imputed to implicit disentanglement.

The fraction of success due to implicit disentanglement is exactly the certainty factor F = 0.667 (see [START_REF] Tan | Selecting the right interestingness measure for association patterns[END_REF] where M ⇒ represents a complete implication and M ⊥ ⊥ represents a complete independence, both with same marginals as M . (see [START_REF] Tan | Selecting the right interestingness measure for association patterns[END_REF] for a definition in the context of association rules)

Figure 1 :

 1 Figure 1: Information space: normalised representation of partitions for the distribution p in Example 1.

Figure 3 :

 3 Figure 3: Unsupervised POS induction: abstraction vs likelihood maximization

  ⟨⟨σ(w⟩⟩)) p(f [σ(w)] | ⟨⟨f [σ(w)]⟩⟩) = λ f (⟨⟨(w⟩⟩) p(⟨⟨f [w]⟩⟩) σ∈Sn p(f [σ(w)]) = λ f (⟨⟨(w⟩⟩)µ(f (w))from Remark 1 and Remark 3. With the fact that p(σ(w)) = p(⟨⟨w⟩⟩)p(σ(w) | ⟨⟨σ(w)⟩⟩) Let p be a distribution and f be a mapping. Then f factorises p iffp(w) = p(⟨⟨w⟩⟩(f [w] | ⟨⟨f [w]⟩⟩) µ(w) µ(f (w))Lemma 4. Let p be a distribution and f a mapping. Then one has ∀w

  with p(z[u]) = p(aaa) = 1 and p(⟨⟨z[u]⟩⟩) = p({a, a, a}) = 1, µ(u) = 1 and µ(aaa) = 6, one has p |z (u) = 1 12 and finallyH(p ∥ p |z ) = log(12)One has also p • z -1 (aaa) = 1 thusH(p • z -1 ) = 0 One has p |id (u) = 1 2 thus H(p ∥ p |id ) = log(2), i s (p ∥ id) = log(6) and p • id -1 = p thus H(p • z -1 ) = log(2), i c (p ∥ id) = log(2)Let g be the mappinga = {cats, mice, men, houses}, b = {eat, g[u]) = p(aba) = 1, p(⟨⟨g[u]⟩⟩) = p({a, b, a}) = 1, µ(u) = 1 and µ(g(u)) = µ(aba) = 2 one has p |g (u) = 1 4 and H(p ∥ p |g ) = log(4), i s (p ∥ g) = log(3) One has p • g -1 (aba) = 1, thus H(p • g -1 ) = 0, i c (p ∥ g) = 0Let h be the mapping c = {cats, eat, mice}, d = {men, build, houses} then one hasp |h (u) = p(⟨⟨u⟩⟩) p(h[u]) p(⟨⟨h[u]⟩⟩) µ(u) µ(h(u)) with p(h[u]) = p(ccc) = 1 2 , p(⟨⟨h[u]⟩⟩) = p({c, c, c}) = 1 2 , µ(u) = 1 and µ(h(u)) = µ(ccc) = 6 one has p |h (u) = 1 12 and H(p ∥ p |h ) = log(12), i s (p ∥ h) = 0

G

  * (p) = ∩ f ∈G * (p) G(p, f ) ∈ G * (p)Proof. Let f be an optimal factorizations of p. The property∀w ∈ supp(p), p(w | f [w]) = p(w | A |w| )means that there is only one observed pattern of categories of length |w|, say b 1 . . . b n .For any (i, n) ∈ N 2 , let us define the subsetA (i,n) = {a ∈ A | ∃w ∈ A n , p(w) > 0, w i = a}Each subset A (i,n) is entirely inside a class of the partition induced by f -otherwise there would be at least two patterns for sequences of size n.One builds a partition m of A by merging the subsets A (i,n) with non-empty intersection. Any class of m is entirely included in a class induced by f .The mapping m is minimal by construction, and it is a refinement of f : G(p, f ) ⊆ G(p, f ), and, by Lemma 10, m factorises p.Implicit disentanglement. We consider the following matrix of confusion 251(1 -µ) 0.251µ a 0.749(1 -µ) 0.749µand we write M as a convex combination of M ⇔ and M ⊥ ⊥ :M = λM ⇔ + (1 -λ)M ⊥ ⊥This gives λ = 0.522, µ = 0.408 which leads to the interpretations:

  ) which represents a convex decomposition of M M = F.M ⇒ + (1 -F ).M ⊥ ⊥ with:

Table 1 :

 1 Notations

The nativist claim has however been subjected to criticism from other fields of research[START_REF] Hsu | The logical problem of language acquisition: a probabilistic perspective[END_REF];[START_REF] Yang | One model for the learning of language[END_REF].

Appendix -Proofs and complements

Let p be a probability distribution, and f a mapping. One will denote p • f -1 the probability distribution induced by f .