Charlotte Caucheteux
email: charlottecaucheteux<ccaucheteux@fb.com>.

Alexandre Gramfort

Jean-Remi King

Disentangling Syntax and Semantics in the Brain with Deep Networks

Keywords:

HAL is a

Introduction

Within less than three years, transformers have enabled remarkable progress in natural language processing [START_REF] Devlin | Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. Pretraining these architectures on millions of texts to predict words from their context greatly facilitates translation, text synthesis and the retrieval of world-knowledge [START_REF] Lample | Cross-lingual Language Model Pretraining[END_REF][START_REF] Brown | Language Models are Few-Shot Learners[END_REF]. To understand the meaning of a phrase, one must combine the meaning of each word using the rules of syntax. For example, the meaning of the phrase NOT VERY HAPPY is (roughly) SAD, and can be found by recursively combining the two adverbs and the adjective. B. Here, we aim to decompose lexical features (what relates to the word level) from the compositional features (what relates to a combination of words) both for syntactic representations (e.g. part-of-speech versus syntactic tree) and for semantic representations (e.g. the set of word meaning versus the meaning of their combination).

Interestingly, the activations of language transformers tend to linearly map onto those of the human brain, when presented with the same sentences [START_REF] Jain | Incorporating context into language encoding models for fmri[END_REF][START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Abnar | Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains[END_REF][START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Goldstein | Thinking ahead: prediction in context as a keystone of language in humans and machines[END_REF]. This linear mapping suggests that, in spite of their vast learning1 and architectural differences2 , the brain and language transformers converge to similar linguistic representations [START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Caucheteux | Gpt-2's activations predict the degree of semantic comprehension in the human brain[END_REF].

However, the nature of these shared representations remains largely unknown. Three factors explain this gap-ofknowledge. First, linguistic theories are generally described and interpreted in terms of combinatorial symbols (discrete words, syntactic trees, etc). In contrast, brain and language transformers generate high-dimensional vectors (a.k.a "distributed" representations). While these formats are formally equivalent [START_REF] Smolensky | Tensor product variable binding and the representation of symbolic structures in connectionist systems[END_REF], interpreting vectorial representations in language models and in the brain is particularly challenging. . Method to isolate syntactic representations in GPT-2's word and compositional embeddings. To isolate the syntactic representations of a sequence of words e.g. w = NOT VERY HAPPY, we (1) synthesize sentences with the same syntactic structure as w (e.g DIMLY SO TRUE, etc.), then (2) extract the corresponding GPT-2 activations (from layer 9), and finally (3) average these activation vectors across the synthesized sentences. The resulting vector X is an approximation of the syntactic representations of X in GPT-2.

Second, the representations of deep learning models have been interpreted independently of brain imaging. For example, deep neural networks have been shown to encode lexical analogies in their word embeddings (Mikolov et al., 2013), as well as singular/plural relationships [START_REF] Lakretz | The emergence of number and syntax units in LSTM language models[END_REF], long-distance dependency information [START_REF] Jawahar | What Does BERT Learn about the Structure of Language[END_REF], and syntactic trees [START_REF] Manning | Emergent linguistic structure in artificial neural networks trained by self-supervision[END_REF].

Similarly, the brain responses to language have been decomposed into a cascade of representations, which maps speech and reading input into phonetic (or orthographic), morphemic, lexical, and syntactic representations [START_REF] Hickok | The cortical organization of speech processing[END_REF][START_REF] Dehaene | The unique role of the visual word form area in reading[END_REF][START_REF] Pallier | Cortical representation of the constituent structure of sentences[END_REF][START_REF] Friederici | The Brain Basis of Language Processing: From Structure to Function[END_REF][START_REF] Mesgarani | Phonetic feature encoding in human superior temporal gyrus[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Nelson | Neurophysiological dynamics of phrase-structure building during sentence processing[END_REF][START_REF] Brennan | Hierarchical structure guides rapid linguistic predictions during naturalistic listening[END_REF][START_REF] Gwilliams | Neural dynamics of phoneme sequencing in real speech jointly encode order and invariant content[END_REF]. However, we do not know whether all or any of these representations effectively drive the linear mapping between language models and the brain.

Third, the mapping between language transformers and the brain has been mainly investigated with speech and/or narratives [START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Abnar | Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains[END_REF][START_REF] Reddy | Syntactic representations in the human brain: beyond effort-based metrics[END_REF] (although see [START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF]). The resulting sentences are thus poorly controlled and potentially confound various features such as phonological variations, sentiment contours, semantic contents, and syntactic properties (e.g. stressful texts may tend to be read more quickly, and make use of smaller constituency trees). In sum, the linear correspondence observed between language models and the brain may be driven by a wide variety of factors.

Here, we aim to decompose the similarity between the brain and high-performance language transformers like GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], in light of four distinct linguistic classes, namely lexical, compositional, syntactic and semantic representations. To that end, we formalize a taxonomy that factorizes them into four distinct vector bases. We then describe a statistical procedure to extract syntactic representations from neural networks, decompose their lexical and compositional components, and separate them from semantic representations. Finally, we assess the linear mapping between i) the factorized activations of GPT-2 and ii) the brain signals of 345 subjects listening to the same narratives (4.6 hours of audio stimulus in total) as recorded with functional magnetic resonance imaging (fMRI) [START_REF] Nastase | Narratives: fMRI data for evaluating models of naturalistic language comprehension[END_REF].

Operational Taxonomy

The notions of lexicon, composition, syntax and semantics are notoriously debated in linguistics. Without pretending to resolve these debates, we propose five definitions that unambiguously decompose the distributed representations of artificial and biological neural networks.

First, we use the standard definition of a representation as the information that can be linearly extracted from a vector of activations, with the rationale that a single artificial or biological neuron can read-out this information [START_REF] Kriegeskorte | Representational similarity analysis -connecting the branches of systems neuroscience[END_REF][START_REF] King | Encoding and decoding neuronal dynamics: Methodological framework to uncover the algorithms of cognition[END_REF]. In this view, a system Ψ 1 is said to share the representation of a system Ψ 2 if there exists a linear mapping from X to Y , where X = Ψ 1 (w) and Y = Ψ 2 (w) are the activations elicited by the words w in each system.

Second, we define lexical representations as the representations that are context-invariant. This definition follows the standard notion of (non-contextualized) word-embeddings, which associate a unique vector to each word of a dictionary.

By contrast, we define compositional representations as the "contextualized" representations generated by a system combining multiples words: Ψ(w 1 . . . w M ). For clarity, we restrict the term "compositional" to its strict sense: i.e. to the set of representations that cannot be accounted for by lexical representations, and thus by a linear combination of word-embeddings.

Fourth, we define syntactic representations as the set of representations associated with the structure of sentences independently of their meaning. Linguistic theories have proposed symbolic representations of such structures (e.g partof-speech, dependency and constituency trees, see Figure 1). Furthermore, deep language models have been shown to linearly encode some of these features [START_REF] Jawahar | What Does BERT Learn about the Structure of Language[END_REF][START_REF] Manning | Emergent linguistic structure in artificial neural networks trained by self-supervision[END_REF][START_REF] Lakretz | The emergence of number and syntax units in LSTM language models[END_REF][START_REF] Baroni | Linguistic generalization and compositionality in modern artificial neural networks[END_REF][START_REF] Linzen | Syntactic structure from deep learning[END_REF]. Here, we introduce a versatile method to extract the distributed representations of syntax in a deep language model. Specifically, we extract these syntactic representations from the average activations elicited by a set of synthetic sentences that share the same syntactic properties (Section 3.1).

Finally, even though a variety of meaningful features are captured by both word embeddings (Mikolov et al., 2013) and contexualized embeddings [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], meaning and semantics are notoriously difficult to define formally [START_REF] Jackendoff | Foundations of Language: Brain, Meaning, Grammar, Evolution[END_REF]. To decompose syntax and semantics in distributed representations, we thus propose to define semantic representations as the lexical or supra-lexical representations of a language system that are not syntactic.

According to these five definitions, lexical and compositional classes fully decompose both syntax and semantics (and vice versa). For example, lexico-syntactic representations refer to the functional categories of words (partof-speech i.e. verb, noun, adjective, etc.). By contrast, compositional syntax refers to the representations that link words with one another, typically referred to as dependency (or constituency) trees. For example, in the phrase NOT VERY HAPPY (Figure 1), the set of lexical meaning can be distinguished from their compositional meaning. The representation of this composition need not contain syntactic information, because its outcome (≈SAD) can be similar across phrases following distinct syntactic structures (e.g. NOT VERY HAPPY = DOWN IN THE DUMPS = SOMEWHAT SAD, etc.). Note that, under this definition, the distributed representations of syntax need not have a symbolic counterpart in theoretical linguistics -e.g. temporary structures that allow building the syntactic tree of a sentence, represent multiple alternative and their respective probabilities etc.

Methods

Isolating Syntactic Representations

We introduce below a method to isolate distributed representations of syntax in neural networks. We assume that a system Ψ (Ψ : V M → R d×M , V a vocabulary of words), takes sequences of M words as inputs and generates activations that encode syntactic properties (among other properties).

Let w be a sentence of M words (w ∈ V M , e.g THE CAT IS ON THE MAT), and Ω w be the set of sentences that have the same syntax as w (e.g. A BOY GOES TO A POOL, THIS BOAT To check that the syntactic embeddings X only contain syntactic information, we train a 2-regularized linear model to predict three semantic features (frequency, word embeddings and semantic category of content words [START_REF] Binder | Toward a brain-based componential semantic representation[END_REF]) and two syntactic features (part-of-speech and depth of syntactic tree), given the syntactic embedding X (red), or the full GPT-2 activations X (grey) (Appendix C). On the y-axis, the decoding performance of the model on left-out data (adjusted accuracy for the categorical features marked with a star, R 2 for the other continuous features).

The chance level is zero. Semantic features (left) can be decoded from X (grey), but not from X (red), while syntactic features (right) can be decoded from both.

FLOATS NEAR THE SHORE, etc.). The syntactic representation of w is, by construction, also the syntactic representations of all sentences w ∈ Ω w . If this common syntactic representation is denoted ψ ∈ R d , we have:

∀w ∈ Ω w , Ψ(w ) = ψ + z w
with z w a random perturbation of distribution P w , that corresponds to the non-syntactic part of the randomized activations Ψ(w ). If the density of P w is well-defined and centered around 0, then:

E Ψ(w ) = ψ ,
where w is sampled uniformly in Ω w . Thus, ψ (the syntactic representation of w) can be approximated through:

Ψ k = 1 k k i=1 ψ + z wi l.l.n ----→ k→∞ ψ with (z w1 , . . . , z w k ) i.i.d samples from P w .
Overall, the syntactic component of the activations is the average of activations induced by random sentences of the same syntax (Figure 2).

Mapping Representations onto FMRI Signals

In the present section, we aim to map the activations of two systems Ψ 1 , a neural network, and Ψ 2 , the brain, input with the same sequence words w = (w 1 , . . . , w M ). Let

A. ℛ ( X ) = Corr ( f ∘ g ( X ) , Y ) Compositional Layers Lexical Layer B.
Figure 4. Method to decompose the language representations shared between brains and deep language models A. The human brain and modern language models like GPT-2 both generate distributed representations, which are thus difficult to link with the symbolic properties of linguistic theories. We introduce a method to decompose the representations of GPT-2, and the corresponding activations X onto the brain activations Y , elicited by the same sequence of words (e.g. NOT VERY HAPPY) with a spatio-temporal estimator f • g. This mapping is evaluated through cross-validation, with a Pearson correlation between the predicted and the actual brain signals R(X). B.

Comparison used to decompose the brain score R(X) into the four linguistic components. X (l) refers to the the l th layer's activations of GPT-2 input with the sentences heard by the subjects; X (l) refers to the average l th layer's activations of GPT-2 input with the synthetic sentences with a similar syntax (cf. Figure 2); ⊕ indicates a feature concatenation, and '-' indicates a subtraction between scores.

X = Ψ 1 (w) ∈ R M ×d be a vector of Ψ 1 activations elicited by w (M vectors of dimension d, one per input word), and Y = Ψ 2 (w) ∈ R N the observable brain response at each of the N fMRI recorded time sample (a.k.a TR). For simplicity, we consider the analysis for one particular fMRI voxel, the same analysis can be repeated to map X with every voxel in the brain.

To assess the mapping between X and Y , we use the standard model-based encoding analysis of fMRI signals [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Yamins | Using goal-driven deep learning models to understand sensory cortex[END_REF][START_REF] Naselaris | Encoding and decoding in fmri[END_REF], and evaluate a linear spatio-(f ) temporal (g) encoding model trained to predict the i th fMRI volume given the network's activations X, on a given interval

I ⊂ [1 . . . N ]: R(X) : f → L f • g(X) i∈I , (Y i ) i∈I (1)
Specifically, given a story w of M words (w = (w 1 , . . . , w M ) = (THE, CAT, IS, ON, THE, MAT, . . . END), we first extract the corresponding brain measurements Y of length N time samples. To maximize signal-to-noise ratio, we average the responses across the subjects that listened to that story, and apply the analysis to the average signal Y .

The sampling frequency of fMRI is typically lower than word rate. Furthermore, fMRI signals are associated with delayed time responses that can span several seconds. Following others [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Deniz | The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality[END_REF][START_REF] Shain | fMRI reveals language-specific predictive coding during naturalistic sentence comprehension[END_REF], we align the word-times features X, of length M , to the dynamics of the fMRI signals applying a finite impulse response (FIR) model g (cf. Appendix D).

Finally we learn a "spatial" mapping f ∈ R d from the zeromean unit-variance of X to the zero-mean unit-variance fMRI recordings Y with a 2 -regularized "ridge" regression:

argmin f i∈Itrain Y i -f T g(X) i 2 + λ||f || 2
with λ the regularization parameter. We summarize the mapping with a Pearson correlation score evaluated on left out data:

R = corr f • g(X), Y . (2)
This correlation score measures the linear mapping between the brain and the activation space X. Following others [START_REF] Yamins | Using goal-driven deep learning models to understand sensory cortex[END_REF], we will refer to this score as the brain score of the embedding X.

Decomposing Shared Activations between Brains and Neural Language Models

Here, we use the definitions and methods introduced in Section 2, 3.1 and 3.2 to decompose the shared representations of two systems: a deep neural network that encode linguistic properties, and the average brain of 345 subjects listening to narratives.

To that end, we (i) compute the activations of the neural language model elicited by the same narratives as the subjects (ii) factorize its activations into linguistic components, (iii) map with supervised learning the factorized components onto brain activity, and finally (iv) decompose the brain activations by evaluating this mapping.

Acoustic features GPT2 activations

A.

ℛ ( X ( 9)

) Lexical Syntactic Compositional Semantic ℛ ( X (0) ) -ℛ ( X (0) ) ℛ ( X (9) ) -ℛ ( X (9) ⊕ X (0) ) ℛ ( X (0) ) ℛ ( X (9) ) -ℛ ( X (0) ) ℛ ( X (0) ) ℛ ( X (9) ) ℛ ( X (9)
) -ℛ(X (0) ) semantic (E-G), lexical (B-H), compositional (C-I) components and their combinations (ten combinations in total). A Comparison between the brain scores of three phonological features (word rate, phone rate, and phone categories, on the top) and the brain scores of the activations extracted from the 9 th layer of GPT-2, when input with the same narratives (on the bottom). B-J. Brain scores decomposed into different sub-processes. To focus on language -and not low-level speech -processing, we display the gain in brain scores compared to the phonological features. For simplicity, the R values reported refers to this gain. Brain scores are computed for each fMRI voxel (averaged across subjects), on 100 splits of ≈ 2.5 min of audio stimulus. Non-significant brain regions are not displayed (.05 threshold), as assessed with a two-sided Wilcoxon test across splits, corrected for multiple comparison across the 75 regions of interest (cf. Section E).

Syntactic + Semantic Lexical + compositional ℛ(X (9) ) ℛ ( X (9) ) -ℛ ( X ( 9
Language transformers are composed of multiple layers (l ∈ [1 . . . L]), stacked over a (non contextualized) word embedding layer (l = 0). Each layer can be written as a non-linear system Ψ (l) that transforms a sequence of words w (e.g. NOT, VERY, HAPPY) into a vectorial representation of the same length,

Ψ (l) : V M → R M ×d w → Ψ (l) (w) = [Ψ (l) (w) 1 , . . . , Ψ (l) (w) M ]
with V the set of vocabulary words, M the length of the sequence, and d the dimensionality of the output representation taken at each word.

We denote X (l) the activations of Ψ (l) elicited by w, and X (l) the syntactic representations extracted from X (l) using the method introduced in Section 3.1. Following the definitions of Section 2, we can decompose the activations X of Ψ into their:

• lexical representations: X (0) , the word embedding of the network.

• compositional representations: X (l) , l > 0.

• syntactic representations: X (l) , that can be extracted for any layer l ∈ [0 . . . L]. The lexical syntactic representations X (0) isroughly equivalent to the part-ofspeech of the word. Compositional syntactic repre-sentations can be extracted from any layer l > 0 that encode syntactic information.

• semantic representations: X (l) -X (l) , as the residuals of syntactic representations. They can be defined at both the lexical X (0) -X (0) and compositional levels (l > 0).

In practice, to verify that our syntactic embedding (X) only contains syntax, we evaluate its ability to predict three semantic and two syntactic features (Figure 3, Appendix C).

The results confirm that semantic features can be decoded from X but not from X, whereas syntactic features can be decoded from both.

Finally, following Section 3.2, we can compute the brain scores of the network's representations to decompose brain activity into:

• lexical representations: R(X (0) )

• compositional representations: R(X (l) ), l > 0.

Strictly compositional representations are defined as the compositional representations that cannot be explained by lexical features: R(X (l) ) -R(X (0) ), with l > 0. For clarity, and except if stated otherwise, we will refer to strictly compositional representations as "compositional" representations.

• syntactic representations: R(X (l) ), l ∈ [0 . . . L]

• semantic representations: R(X (l) ) -R(X (l) ), i.e. the residual brain scores of syntactic representations, for any layer l ∈ [0 . . . L]

Experiments on the Narratives fMRI Dataset

Here, we apply the general method described in Section 3.1, 3.2 and 3.3 to decompose the activations of two nonlinear systems, GPT-2 (Ψ 1 ) and the brain activity of 345 subjects listening to narratives (Ψ 2 ).

Functional MRI dataset. We analyze the "Narratives" public dataset [START_REF] Nastase | Narratives: fMRI data for evaluating models of naturalistic language comprehension[END_REF], which contains the fMRI measurements of 345 unique subjects listening to narratives. The narratives consist of 27 English spoken stories, ranging from ≈ 3 minutes to ≈ 56 minutes, for a total of ≈ 4.6 hours of unique stimuli. The original paper included two fMRI preprocessing pipelines, one with spatial smoothing and the other without. All our analyses are tested on the unsmoothed fMRI. As suggested in the original paper, we exclude (story, subject) pairs because of noisy fMRI recordings or missing transcripts, resulting in 617 unique (story, subject) pairs in total and ≈ 4 hours of unique audio stimuli.

Phonological features. To focus on lexical and supralexical language processing -as opposed to low-level speech processing, we extract three potential sets of confounds: the phone rate (the number of phones between two fMRI measurements, of dimension 1), the word rate (the number of words between two fMRI measurements) and the concatenation of the phoneme, stress and tone of the words in the stimulus. For each story, a phoneme-level transcript was provided in the Narratives database thanks to Gentle 3 , a forced-alignment algorithm. Gentle annotations led to 117 unique categories (with unique phone, stress and tone), resulting in a one-hot encoded feature of the same dimension.

Language model features. GPT-2 is a high-performing causal (i.e. left to right) language model trained to predict a word given its previous context [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], and known to generate brain-like representations [START_REF] Goldstein | Thinking ahead: prediction in context as a keystone of language in humans and machines[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Affolter | Brain2word: Decoding brain activity for language generation[END_REF][START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Caucheteux | Gpt-2's activations predict the degree of semantic comprehension in the human brain[END_REF]. It is comprised of 12 Transformer (contextual) layers (l ∈ [1 . . . 12]) stacked over a (non-contextual) embedding layer (l = 0), each of dimensionality 768, with 1.5 billion parameters in total. We used the pretrained version of GPT-2 from Huggingface [START_REF] Wolf | Transformers: State-ofthe-art natural language processing[END_REF], trained on a dataset of 8 million web pages. In practice, the 27 stories are pre-processed, tokenized and input to the model (Appendix A). The activations 3 https://github.com/lowerquality/gentle of each GPT-2 layer are extracted, resulting in 12 vectors of 768 activations for each token of each story transcript. For comparison, we also study five other transformers: BERT [START_REF] Devlin | Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], XLnet [START_REF] Yang | XLNet: Generalized Autoregressive Pretraining for Language Understanding[END_REF], Roberta [START_REF] Liu | A Robustly Optimized BERT Pretraining Approach[END_REF], AlBert [START_REF] Lan | A Lite BERT for Self-supervised Learning of Language Representations[END_REF] and DistilGPT-2 (a smaller version of GPT-2) and recover similar -although lower -brain scores (Appendix A).

Extracting syntactic representations from GPT-2 . To isolate the syntactic representations of GPT-2 , we synthesize, for each sentence of each story, k = 10 sentences with the same syntactic structures (Figure 2). We ensure in supplementary analyses that (i) the k synthetic sentences do not include the target sentence and (ii) these syntactic embeddings (Ψ k ) lead to stable representations of syntax (Appendix B). To this end, we proceed as follows:

• The transcript is formatted, split into sentences and tokenized using the large English tokenizer provided by spaCy [START_REF] Honnibal | spaCy: Industrial-strength Natural Language Processing in Python[END_REF] (cf. Appendix A).

• Then, we use Supar, a state-of-the art dependency parser [START_REF] Zhang | Efficient secondorder TreeCRF for neural dependency parsing[END_REF] to extract the dependency structure of each sentence and the part-of-speech.

• For each target word of each sentence of the Narratives dataset, we sample, from a ≈ 58,000 word corpus, consisting of Wikipedia combined with Narratives' transcripts, up to to k = 1, 000 words that have the same part-of-speech and dependency tags (e.g. CAT: NOUN, SINGULAR, SUBJECT OF). At this stage, k versions of the target Narratives transcripts are synthesized.

• The synthesized sentences are not always grammatically correct. Thus, we automatically correct the sentences with Gector [START_REF] Omelianchuk | GECToR -Grammatical Error Correction: Tag, Not Rewrite[END_REF], and filter out the sentences that do not have the same length or part-of speech as the target sentence in the Narratives corpus.

• Some of the generated sentences may end up with a distinct syntactic tree than the original sentence, because semantics can disambiguate syntax (e.g. I SHOT AN ELEPHANT IN MY PYJAMAS). To assess the syntactic similarity between the original and the generated sentences, we compute, from their respective syntactic trees, the Pearson correlation between the words' pairwise distances, following [START_REF] Manning | Emergent linguistic structure in artificial neural networks trained by self-supervision[END_REF]'s method. Then, we select the sentences whose syntactic trees are the most similar. 95% of the generated sentences have a syntactic tree that correlates with the tree of the target sentence above R=90%.

Mapping embeddings onto onto the fMRI signals. As described in equation ( 1), we evaluate the mapping between a set of modeling features X and the fMRI signals Y ∈ R N ×dy by fitting a linear spatio-(f ) temporal (g) encoding model. f • g was fitted on I train = 99% of the dataset, and evaluated on I test = 1% of the left out-data (2.5 min of audio). We evaluate the quality of this mapping with a Pearson R correlation between predicted and actual brain signals on I test . Specifically, we use the linear ridge regression from scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF], with penalization parameters chosen among 10 values log-spaced between 10 -1 and 10 8 and g was a finite impulse response (FIR) model with 5 delays, following [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF].

X and Y are normalized (mean=0, std=1) across scans for each story, using a robust scaler clipping below and above the 0.01 st and 99.99 th percentiles, respectively. We repeat the procedure 100 times with a 100-fold cross-validation, using scikit-learn 'KFold' without shuffling [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF].

Statistical significance. We assess the significance of our results across test folds (k = 100). To this end, we first average the brain scores within each brain region, as defined by the Destrieux Atlas parcellation [START_REF] Destrieux | Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature[END_REF]. Then, we apply a Wilcoxon two-sided signed-rank test across folds to evaluate whether this average brain score is significantly different from zero. The p-values of the 75 brain regions were corrected for multiple comparison using a False Discovery Rate, (Benjamini/Hochberg) as implemented in MNE-Python [START_REF] Gramfort | MEG and EEG data analysis with MNE-Python[END_REF]. Nonsignificant p-values (p ≥ .05) are masked in Figure 5.

Experimental Results

Phonological features. To isolate the sublexical speech representations, we compute the brain scores using a concatenation of three sets of features, i.e., word rate, phone rate, and phone categories. These sublexical features lead to significant brain scores across the expected language networks and mainly peak within the bilateral superior temporal lobe, the temporo-parietal junction, the lateral intra-parietal sulcus, the infero-frontal cortex (IFG) as well as in the right motor cortex (Figure 5A and6).

To isolate lexical and compositional representations, we focus the next analyses on the gain in brain scores obtained over those of sublexical features (i.e. to the increase of brain scores obtained with each feature set, as compared to the scores obtained with phonological features). For simplicity, the R scores reported in Figure 5, 6 and in the text below refer to this gain.

The brain scores corresponding to the lexical (R(X (0) )), compositional (R(X (9) )), syntactic (R(X (9) ) and semantic representations (R(X (9) ) -R(X (9) )) of the ninth layer of GPT-2 are displayed in figures 5 and 6 (non-significant scores after correction for multiple comparisons across regions are masked). Lexical features. The lexical representations of the brain have been repeatedly investigated through the lens of a wordembedding [START_REF] Mitchell | Predicting human brain activity associated with the meanings of nouns[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF]. Here, we replicate these analyses: GPT-2's word embedding X (0) leads to lexical brain scores significantly higher than sublexical features' in most of the language network, i.e. in the bilateral superior temporal lobe and the infero-frontal cortex (Figure 5H).

Lexical syntax. Do these brain scores result from semantic and/or syntactic representations? To tackle this issue, we compute brain scores from the word embeddings (X (0) ) input with synthesized and syntactically-matched sentences: i.e. word sequences sharing the same syntax as the target sentence in the original Narratives corpus (Figure 5B). The results reveal significant brain scores (i.e. higher than sublexical ones) in a distributed network including the inferofrontal cortex, the angular gyrus and the posterior superior temporal gyrus (Figure 6).

Lexical semantics. To identify the representations of lexical semantics, we compare the brain score obtained with the word embedding to those obtained with the embedding of lexical syntax (R(X (0) ) -R(X (0) ) in Figure 5E). The resulting brain scores are significant mainly in the left hemisphere, and peak in the superior temporal gyrus, the infero-frontal cortex as well as in the precuneus and the tranverse temporal gyrus. These results are more modest than we anticipated given past work [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF].

Compositional representations. Recent studies have shown that the contextual (i.e. deep) layers of language models better predict brain activity than word embedding [START_REF] Jain | Incorporating context into language encoding models for fmri[END_REF][START_REF] Jat | Relating simple sentence representations in deep neural networks and the brain[END_REF][START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF]. We replicate this result with a representative contextual layer of GPT-2 (layer 9 out of 12, Figure 5J): R(X (9) ) almost doubles the brain scores obtained with the word embedding R(X (0) ) in the bilateral temporal, infero-frontal and infero-parietal cortices.

Compositional syntax. Do these gains in brain score reflect compositional semantics and/or compositional syntax?

To tackle this issue, we compare the brain scores obtained with the ninth layer of GPT-2 input with the syntax-matched synthesized sentences R(X (9) ), to the the brain scores obtained with the first layer of GPT-2, input with those same synthesized sentences R(X (0 ). The results show that the representations of compositional syntax are distributed over the bilateral temporal and infero-frontal cortices, and actually extend to a relatively large set of brain areas (Figure 5C-D). Overall, these results, although correlational, thus favor a distributed [START_REF] Fedorenko | Lexical and syntactic representations in the brain: an fmri investigation with multi-voxel pattern analyses[END_REF] rather than a modular [START_REF] Pallier | Cortical representation of the constituent structure of sentences[END_REF][START_REF] Friederici | Segregating semantic and syntactic aspects of processing in the human brain: an fmri investigation of different word types[END_REF] view of syntax: both lexical and compositional syntactic effects do not appear to be confined within a single brain area.

Compositional semantics. Finally, we estimate the brain representations of compositional semantics by comparing the brain scores obtained with the syntactic representations R(X (9 ) to those obtained with the "normal" activations R(X (9 ), i.e. GPT-2's activations obtained with the same sentences as subjects heard. Again, the resulting effects proved to be remarkably distributed, and peaked in the cingulate, supramarginal, and middle-frontal cortex (Figure 5G). These brain scores appear to result from strictly compositional semantics: these effects remain significant even when we subtract away the contribution of lexical semantics (Figure 5E and6).

Control 1: low-level linguistic properties. Do the syntactic representations evidenced above simply capture the length of sentences? To address this issue, we input the above analyses with i) random words sequences (i.e. non grammatical) and ii) random but well-formed sentences that have the same length as those of the Narratives corpus. The results show that neither of these two embeddings match the brain scores obtained with syntactic and/or semantic representations (Figure 7). Similarly, using the GPT-2 activations elicited by the sentences of the Narratives after a random word permutation leads to lower brain scores than our original analyses. Together, these results confirm that our decomposition of syntactic and semantic representations in the brain cannot be reduced to simplistic representations like bags of words and/or sentence length.

Control 2: generalisation to other layers and architectures. The above results are obtained using the ninth layer of GPT-2. We chose to study this model and this layer, because a) GPT-2, like the brain, processes words in a causal way, b) it is known to best predict brain responses [START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Caucheteux | Gpt-2's activations predict the degree of semantic comprehension in the human brain[END_REF], c) its middle layers best encode complex semantic and syntactic properties [START_REF] Jawahar | What Does BERT Learn about the Structure of Language[END_REF][START_REF] Manning | Emergent linguistic structure in artificial neural networks trained by self-supervision[END_REF]. To test the generality of our study, we apply the same analyses to five other language transformers as well as to all of the layers of GPT-2 (Figure 7). The results generalize to each layer of GPT-2, and peak around layer 9. The five other transformers (for their middle layer l = 2/3 × n layers ) result in similar, although significantly lower brain scores (Appendix A).

Discussion

In the present study, we introduce a simple taxonomy and its associated method to decompose the distributed representations of language in brains and deep language models.

Our taxonomy capitalizes on classic linguistic proposals [START_REF] Lycan | Philosophy of language: A contemporary introduction[END_REF][START_REF] Givón | Syntax: an introduction[END_REF][START_REF] Chomsky | The minimalist program[END_REF] to offer precise definitions of lexicality, compositionality, syntax and semantics, which operate on distributed representations.

Our results show that these four sets of linguistic features, typically theorized in terms of discrete symbols, can be, as long predicted [START_REF] Smolensky | Tensor product variable binding and the representation of symbolic structures in connectionist systems[END_REF], investigated in artificial and biological neural networks.

The present definitions remain imperfect. First, compositionality is often associated with specific properties that are not presently considered (e.g. systematicity and generalisation [START_REF] Szabó | [END_REF][START_REF] Hupkes | The compositionality of neural networks: integrating symbolism and connectionism[END_REF][START_REF] Baroni | Linguistic generalization and compositionality in modern artificial neural networks[END_REF]). Furthermore, we here define semantics as the residual representations of any text embedding once syntactic representations have been removed. This proposal is very coarse: semantics is generally defined as the study of meaning (which is itself not easy to define). Yet, some language features like emotional value and textual style may arguably not "mean" anything, in that they do not necessarily refer to a state of the world and yet would be categorized as semantics according to our proposed taxonomy. In spite of these limits, the advantage of our framework is that it makes simple, precise and quantifiable predictions to investigate distributed linguistic representations in the human brain. Furthermore, the present framework is particularly versatile in that i) it can, in principle accommodate any natural sentences and ii) its conclusions can be refined with the development of better and/or more biologically-plausible models of language.

The present study follows suit with past research on naturalistic and thus poorly-controlled linguistic stimuli [START_REF] Mesgarani | Phonetic feature encoding in human superior temporal gyrus[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Brennan | Naturalistic sentence comprehension in the brain[END_REF][START_REF] Brennan | Hierarchical structure guides rapid linguistic predictions during naturalistic listening[END_REF][START_REF] Stehwien | The little prince in 26 languages: Towards a multilingual neuro-cognitive corpus[END_REF][START_REF] Gwilliams | Neural dynamics of phoneme sequencing in real speech jointly encode order and invariant content[END_REF]. While we replicate previous neuroscientific findings regarding lexical semantics (Figure 5E) [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF] and lexical vs compositional processing in the brain (Figure 5.H,J) [START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Goldstein | Thinking ahead: prediction in context as a keystone of language in humans and machines[END_REF], our systematic decomposition of language representations brings new light on the brain bases of syntax (Figure 5.BCDFG). In addition, our approach diverges with and complements previous practices, consisting of carefully designed stimuli, typically matched for word length, word frequency [START_REF] Kutas | Reading senseless sentences: Brain potentials reflect semantic incongruity[END_REF] and/or constituent size [START_REF] Pallier | Cortical representation of the constituent structure of sentences[END_REF][START_REF] Ding | Cortical tracking of hierarchical linguistic structures in connected speech[END_REF], which becomes exponentially difficult when the number of variables to control increases [START_REF] Hamilton | The revolution will not be controlled: natural stimuli in speech neuroscience[END_REF]. This change of paradigm has been empowered by the rise of high-performing language models: previous research lacked a method to make single trial/single sentence predictions and could thus only compare the average activations across blocks of similarly constructed sentences. By contrast, modern language models offer the possibility to predict the representations of individual words and sentences [START_REF] Hale | Finding syntax in human encephalography with beam search[END_REF][START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Heilbron | A hierarchy of linguistic predictions during natural language comprehension[END_REF]. Consequently, carefully-controlled experimental designs can now be relaxed to naturalistic settings, and allow one to refine her tests and hypotheses without having to conduct new (and arguably artificial) experiments.

The main drawback of such an uncontrolled setting is undoubtedly signal-to-noise ratio: like any bias/variance tradeoff, relaxing the set of hypotheses that one can test in a given dataset reduces the probability of a successful finding. To accommodate this issue, we here opted to analyze the average brain signal across subjects. Even then, brain scores remain far from 100%. Given that the brain bases of languageare notoriously variable across individuals [START_REF] Fedorenko | New method for fmri investigations of language: defining rois functionally in individual subjects[END_REF] future works remain necessary to better account for the functional and anatomical variability across subjects.

Thanks to machine learning, our method sheds new light on the neural bases of language in general, and of syntactic processes in particular. First, it supplements previous work on the neural basis of lexical [START_REF] Friederici | Segregating semantic and syntactic aspects of processing in the human brain: an fmri investigation of different word types[END_REF][START_REF] Mitchell | Predicting human brain activity associated with the meanings of nouns[END_REF] and compositional representations of language [START_REF] Pallier | Cortical representation of the constituent structure of sentences[END_REF][START_REF] Nelson | Neurophysiological dynamics of phrase-structure building during sentence processing[END_REF][START_REF] Fedorenko | Lexical and syntactic representations in the brain: an fmri investigation with multi-voxel pattern analyses[END_REF][START_REF] Brennan | Meg evidence for incremental sentence composition in the anterior temporal lobe[END_REF]: syntactic processes, in particular, appear to be linked to a remarkably wide-spread distribution of activation in the language networks. This result favours a distributed [START_REF] Fedorenko | Lexical and syntactic representations in the brain: an fmri investigation with multi-voxel pattern analyses[END_REF] as opposed to a modular [START_REF] Pallier | Cortical representation of the constituent structure of sentences[END_REF][START_REF] Friederici | Segregating semantic and syntactic aspects of processing in the human brain: an fmri investigation of different word types[END_REF] view of syntactic processes. Second, our study highlights the remarkably-large recruitment of compositional semantics -an observation that strengthens and extends what had already been reported at the lexical level [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF]. Overall, these results thus reinforce the idea that speech comprehension results from the coordination of a huge cortical network. While its functional principles remain largely unexplored, the similarity between the human brain and deep language models offers a new and powerful mean to understand the laws of language.

for all the 2,800 Wikipedia words studied here. Thus, we first train a linear model (scikit-learn 'RidgeCV-Classifier') to predict the semantic category of the 535 labeled words used in [START_REF] Binder | Toward a brain-based componential semantic representation[END_REF], given their Spacy word embedding (300 dimensions). We then label the 2,800 Wikipedia words using the semantic category predicted by the classifier.

Linear decoder To evaluate the ability of a linear decoder to predict the five linguistic features from X, we:

• Build syntactic embeddings X for 100 Wikipedia sentences (≈ 2, 800 words), following Section 3.1, using the ninth layer of GPT-2.

• Build the three semantic and two syntactic features described above from the 2,800 Wikipedia words Wikipedia words.

• Fit a 2 -regularized linear model to predict the five features given the syntactic embeddings. We use the 'RidgeCV' regressor (resp. 'RidgeClassifierCV' classifier) from scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] to predict the continuous (resp. categorical) features, with ten possible penalization values log-spaced between 10 -3 and 10 6 .

• Evaluate the linear model on held out data, using a 10 cross-validation setting ('KFold' cross-validation from scikit-learn). Performance is assessed using adjusted accuracy ('balanced accuracy score' from scikit-learn) for the categorical features, and R 2 for the continuous features. Thus, the chance level is zero for both types of features, and the best score is one.

• Report the average decoding performance in Figure 3 (red bars), and the standard-error of the means across the ten test folds.

For comparison, we repeat the exact same procedure with the full GPT-2 activations X (instead of their syntactic component X), and report the results in Figure 3 (grey bars).

D. Temporal Alignment g between X and Y

In Section 3.2, we map the network's activations X (of length M , the number of words) and the brain response Y (of length N , the number of fMRI measurements) induced by the same story w (of M words). M is usually greater than N . To align the two spaces, we first sum the features between successive fMRI measurements, and then apply a finite impulse response (FIR) model. We denote g this laneous, multimodal, object, part, perceptual, period, physical, place, plant, property, social, somatosensory, sound, spatial, state, temporal, time, tool, vehicle, visual, weather transformation. Specifically, for each fMRI time sample i ∈ [1 . . . N ], g i combines word features within each acquisition interval as follows:

g i : R M ×d → R 5d u → u i , u i-

E. Brain Parcellation

In Figure 6, brain scores are averaged across voxels within regions of interest using the Brodmann's areas from the PALS parcellation of freesurfer 6 . To gain in precision, we split the superior temporal gyrus (BA22) into its anterior, middle and posterior parts. In Figure 6

Figure 1 .

 1 Figure 1. Taxonomy A. To understand the meaning of a phrase, one must combine the meaning of each word using the rules of syntax. For example, the meaning of the phrase NOT VERY HAPPY is (roughly) SAD, and can be found by recursively combining the two adverbs and the adjective. B. Here, we aim to decompose lexical features (what relates to the word level) from the compositional features (what relates to a combination of words) both for syntactic representations (e.g. part-of-speech versus syntactic tree) and for semantic representations (e.g. the set of word meaning versus the meaning of their combination).

  Figure2. Method to isolate syntactic representations in GPT-2's word and compositional embeddings. To isolate the syntactic representations of a sequence of words e.g. w = NOT VERY HAPPY, we (1) synthesize sentences with the same syntactic structure as w (e.g DIMLY SO TRUE, etc.), then (2) extract the corresponding GPT-2 activations (from layer 9), and finally (3) average these activation vectors across the synthesized sentences. The resulting vector X is an approximation of the syntactic representations of X in GPT-2.

Figure 3 .

 3 Figure 3. Semantic and syntactic information encoded in X.To check that the syntactic embeddings X only contain syntactic information, we train a 2-regularized linear model to predict three semantic features (frequency, word embeddings and semantic category of content words[START_REF] Binder | Toward a brain-based componential semantic representation[END_REF]) and two syntactic features (part-of-speech and depth of syntactic tree), given the syntactic embedding X (red), or the full GPT-2 activations X (grey) (Appendix C). On the y-axis, the decoding performance of the model on left-out data (adjusted accuracy for the categorical features marked with a star, R 2 for the other continuous features). The chance level is zero. Semantic features (left) can be decoded from X (grey), but not from X (red), while syntactic features (right) can be decoded from both.

Figure 5 .

 5 Figure 5. Results Decomposition of the brain scores of 345 subjects listening to narratives into their phonological (A) syntactic (B-D),semantic (E-G), lexical (B-H), compositional (C-I) components and their combinations (ten combinations in total). A Comparison between the brain scores of three phonological features (word rate, phone rate, and phone categories, on the top) and the brain scores of the activations extracted from the 9 th layer of GPT-2, when input with the same narratives (on the bottom). B-J. Brain scores decomposed into different sub-processes. To focus on language -and not low-level speech -processing, we display the gain in brain scores compared to the phonological features. For simplicity, the R values reported refers to this gain. Brain scores are computed for each fMRI voxel (averaged across subjects), on 100 splits of ≈ 2.5 min of audio stimulus. Non-significant brain regions are not displayed (.05 threshold), as assessed with a two-sided Wilcoxon test across splits, corrected for multiple comparison across the 75 regions of interest (cf. Section E).

Figure 6 .

 6 Figure 6. Same as Figure 5.BCEF, with voxel-averaged brain scores (after subtraction of phonological brain scores), for the top ten regions of interest of the left hemisphere (Appendix E). Error bars are the standard-errors of the mean across the 100 crossvalidation folds. Significance ('*') is assessed with a Wilcoxon test across folds, with p < .05 as a threshold.

Figure 7 .

 7 Figure7. Generalisation to other layers and architectures In red, the brain scores of the syntactic embeddings (R( X)) built out of GPT-2 layers (from the word embedding to layer 12), and the middle layer of five transformer architectures (top, cf. Appendix A, l = 2/3 × n layers ). In blue, the residuals of syntax (R(X) -R( X)) in the brain. Bottom, the brain scores of i) acoustic features (the concatenation of word rate, phoneme rate, phoneme stress and tone), GPT-2 activations induced ii) by random words sampled in the stimulus, iii) by sentences randomly sampled from Wikipedia, matching in length with the sentences of the stimulus, iv) by the actual sentences of stimulus, but with random word order in each sentence (Appendix F.)

  1 , . . . , u i-4 . . . M → 1 . . . N m → i / |t yi -t xj | = min k∈ 1...N |t y k -t xm |with ũ the summed activations of words between successive fMRI time samples, u the five lags of FIR features, (t x1 , . . . , t x M ) the timings of the M words onsets, and (t y1 , . . . , t y N ) the timings of the N fMRI measurements.

	u i =	u m
	m∈ 1...M
	T (m)=i
	with
	T : 1

  , we report the top ten areas of the left hemisphere in term of average brain score. Certain areas are renamed for clarity, as specified in the table below:

	Label	Corresponding Brodmann's areas
	A1	BA41 / BA42
	Fusiform	BA37
	Angular	BA39
	aSTG	BA22-anterior
	mSTG	BA22-middle
	pSTG	BA22-posterior
	M1	BA4
	Supramarginal BA40
	IFG (Op)	BA44
	IFG (Tri)	BA45
	IFG (Orb)	BA47
	Middle-frontal BA46
	V1	BA17
	Fronto-polar	BA10
	Temporo-polar BA38
	Precuneus	BA7
	Cingulate	BA23 / BA26 / BA29 / BA30 / BA31

6 https://surfer.nmr.mgh.harvard.edu/fswiki/PALS B12

The brain learns continuously from a small set of situated sentences, whereas transformers learn from large sets of pure texts.

The brain is a single-stream recurrent architecture, whereas the transformer is a multi-stream feedforward architecture.

https://pypi.org/project/wordfreq/

Categories are: abstract, action, animal, auditory, body, building, cognitive, construct, creative, device, distant, document, electronic, emotion, emotional, entity, event, food, furniture, general, geological, group, human, instrument, locative, mental, miscel-

Acknowledgement

This work was supported by ANR-17-EURE-0017, the Fyssen Foundation and the Bettencourt Foundation to JRK for his work at PSL.

Appendix A. Deep Neural Networks' Activations

Pre-trained tansformers In Section 4, we extract the activations of GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] and five transformer architectures: BERT [START_REF] Devlin | Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], XLnet [START_REF] Yang | XLNet: Generalized Autoregressive Pretraining for Language Understanding[END_REF], Roberta [START_REF] Liu | A Robustly Optimized BERT Pretraining Approach[END_REF], AlBert [START_REF] Lan | A Lite BERT for Self-supervised Learning of Language Representations[END_REF] and DistilGPT-2. We use the pre-trained models from Huggingface [START_REF] Wolf | Transformers: State-ofthe-art natural language processing[END_REF]: 'bert-base-cased', 'xlnetbase-cased', 'roberta-base', 'albert-base-v1', and 'distilGPT-2' respectively. In Figure 7, we focus on one middle layer of these transformers (l = n layers × 2/3), because it has shown to best encode brain activity [START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF] and to encode relevant linguistic properties [START_REF] Manning | Emergent linguistic structure in artificial neural networks trained by self-supervision[END_REF][START_REF] Jawahar | What Does BERT Learn about the Structure of Language[END_REF].

Text formatting and tokenization To extract the activations elicited by one story, we proceed as follows: we first format and lower case the text (replacing special punctuation marks such as "-" and duplicated marks "?." by dots), then apply the tokenizer provided by Huggingface [START_REF] Wolf | Transformers: State-ofthe-art natural language processing[END_REF] to convert the transcript into either word-level or sub-word-level tokens called "Byte Pair Encoding" (BPE) [START_REF] Sennrich | Neural Machine Translation of Rare Words with Subword Units[END_REF]. Here, more than 99.5% of BPE-level tokens were complete words. The tokens are then split into sections of 256 tokens (this length is constrained by GPT-2's architecture) and input to the deep network one story at a time. The activations of each layer are finally extracted, resulting in n layers vectors of 768 activations for each token of each story transcript. In the 0.5% case where BPE are not complete words, BPE-features are summed between successive words, to obtain n layers vectors per word per story.

B. Convergence of the Method to Build X

In Section 3.1 and 3.3, we compute the syntactic component X of GPT-2 activations X elicited by a sentence w. X is approximated by X k , the average activations across k sentences with the same syntax as w. Here, we sample k = 10 sentences. We check in Figure 8 that the method has converged before k = 10. We compute the cosine similarity between X k and X k-1 for k between 1 and 15. The syntactic embeddings stabilize with at least eight sampled sentences.

C. Evaluating the Level of Semantic and

Syntactic Information in X

In Section 3.3 and Figure 3, we check that the syntactic embedding X extracted from GPT-2 only contains syntax.

To this aim, we evaluate the ability of a linear decoder to predict two syntactic features and three semantic features from X.

Semantic and syntactic features

The two syntactic features derived from the stimulus are:

• The part-of-speech of the words (categorical feature), as defined by Spacy tags [START_REF] Honnibal | spaCy: Industrial-strength Natural Language Processing in Python[END_REF].

• The depth of the syntactic tree (continuous feature).

The syntactic tree is extracted with the state-of-the-art Supar dependency parser [START_REF] Zhang | Efficient secondorder TreeCRF for neural dependency parsing[END_REF].

The three semantic features are only computed for verbs, nouns and adjectives (as defined by Spacy part-of-speech tags) and are the followings:

• Word frequency (labeled as 'Word freq' in Figure 3, continuous feature). We use the 'zipf frequency' from the wordfreq 4 python library.

• Word embedding (continuous feature), computed using the pre-trained model from Spacy [START_REF] Honnibal | spaCy: Industrial-strength Natural Language Processing in Python[END_REF] ('en core web lg', 300 dimensions).

• Semantic category (categorical feature). We used the 47 semantic categories 5 . Categories are not available

F. Control for Low-level Linguistic Features

In Section 5 and Figure 7, we check that the brain scores are not driven by low-level linguistic features. Thus, we compute the R scores of GPT-2 activations (ninth layer) induced by modified versions of the stimulus:

• Random words sampled from the same story. Words are uniformly sampled from the words of the story, tokenized using Spacy [START_REF] Honnibal | spaCy: Industrial-strength Natural Language Processing in Python[END_REF]. Punctuation marks are considered as words. Upper-cases are kept.

• Random sentences from Wikipedia, of the same length as the sentences of the stimulus. We first build a dictionary of (length, list of match-length sentences) pairs out of 10K sentences from Wikipedia (≈ 577K words).

Then, for each sentence of the stimulus, a sentence is uniformly sampled from the set of Wikipedia matchlength sentences.

• The sentences of the stimulus, but with random word order. Words are shuffled within each sentence.

Then, we extract the corresponding GPT-2 activations and compute the R scores following Section 4. R scores are evaluated for each subject and reported in Figure 7.


Charlotte Caucheteux

Alexandre Gramfort

Jean-Rémi King

GPT-2's activations predict the degree of semantic comprehension in the human brain

Language transformers, like GPT-2, have demonstrated remarkable abilities to process text, and now constitute the backbone of deep translation, summarization and dialogue algorithms. However, whether these models encode information that relates to human comprehension remains controversial. Here, we show that the representations of GPT-2 not only map onto the brain responses to spoken stories, but also predict the extent to which subjects understand narratives. To this end, we analyze 101 subjects recorded with functional Magnetic Resonance Imaging while listening to 70 min of short stories. We then fit a linear model to predict brain activity from GPT-2's activations, and correlate this mapping with subjects' comprehension scores as assessed for each story. The results show that GPT-2's brain predictions significantly correlate with semantic comprehension. These effects are bilaterally distributed in the language network and peak with a correlation of R=0.50 in the angular gyrus. Overall, this study paves the way to model narrative comprehension in the brain through the lens of modern language algorithms.

predictions directly depends on the models' ability to complete 13 text [START_REF] Schrimpf | Artificial Neural Networks Accurately Predict Language Processing in the Brain[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF]. [START_REF] Cohen | How does 332 inattention affect written and spoken language processing[END_REF] In spite of these achievements, strong doubts subsist on 15 whether language transformers actually encode meaningful 16 constructs [START_REF] Nie | Adversarial nli: A new benchmark for natural language understanding[END_REF]. When asked to complete "I had $20 and gave 17 $10 away. Now, I thus have $", GPT-2 predicts "20" * . Simi-18 lar trivial errors can be observed for geographical locations, 19 temporal ordering, pronoun attribution and causal reasoning. [START_REF] Gauthier | Does the brain represent words? An evaluation of brain 349 decoding studies of language understanding[END_REF] These results have thus led some to argue that such "system 21 has no idea what it is talking about" [START_REF] Marcus | Gpt-2 and the nature of intelligence[END_REF]. Thus, how the rep- Here, we propose to evaluate how the similarity between the 25 brain and GPT-2 vary with semantic comprehension. Specifi-26 cally, we first compare GPT-2's activations to the functional 27 Magnetic Resonance Imaging of 101 subjects listening to 28 70 min of seven short stories, and we quantify this similar-29 ity with a "brain score" (M) [START_REF] Yamins | 311 Performance-optimized hierarchical models predict neural responses in higher visual cortex[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF]. Second, we evaluate how 30 * as assessed using Huggingface interface (https://github.com/huggingface/transformers) and GPT-2 pretrained model with temperature=0.

the brain scores systematically vary with semantic comprehension, as individually assessed by a questionnaire at the end of each story.

GPT-2's activations linearly map onto fMRI responses to spoken narratives.

To assess whether GPT-2 generates similar representations to those of the brain, we first evaluate, for each voxel, subject and narrative independently, whether the fMRI responses can be predicted from a linear combination of GPT-2's activations (Figure 1A). We summarize the precision of this mapping with a brain score M: i.e. the correlation between the true fMRI responses and the fMRI responses linearly predicted, with cross-validation, from GPT-2's responses to the same narratives (cf. Methods). To mitigate fMRI spatial resolution and the necessity to correct each observation by the number of statistical comparisons, we here report either 1) the average brain scores across voxels or 2) the average score within each region of interest (n = 314, following an automatic subdivision of Destrieux atlas [START_REF] Destrieux | Automatic parcellation 318 of human cortical gyri and sulci using standard anatomical nomenclature[END_REF], cf. SI.1). Consistent with previous findings [START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Jain | Incorporating Context into Language Encoding Models 321 for fMRI[END_REF]12), these brain scores are significant over a distributed and bilateral cortical network, and peak in middle-and superior-temporal gyri and sulci, as well as in the supra-marginal and the infero-frontal cortex [START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Jain | Incorporating Context into Language Encoding Models 321 for fMRI[END_REF] (Figure 1B). By extracting GPT-2 activations from multiple layers (from layer one to layer twelve), we confirm that middle layers best map onto the brain (Figure 1C), as seen in previous studies [START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Jain | Incorporating Context into Language Encoding Models 321 for fMRI[END_REF]. For clarity, the following analyses focus on the activations extracted from the eighth layer, i.e. GPT-2's most "brain-like" layer (Figure 1B).

GPT-2's brain predictions correlate with semantic comprehension.

Does the linear mapping between GPT-2 and the brain reflect a fortunate correspondence (4)? Or, on the contrary, does it reflect similar representations of high-level semantics? To address this issue, we correlate these brain scores to the level of comprehension of the subjects, assessed for each subject-story pair. On average across all voxels, this correlation reaches R = 0.50 (p < 10 -15 , Figure 1D, as assessed across subjectstory pairs with the Pearson's test provided by SciPy). This correlation is significant across a wide variety of the bilateral temporal, parietal and prefrontal cortices typically linked to language processing (Figure 1E). Together, these results suggest that the shared representations between GPT-2 and the brain reliably vary with semantic comprehension.

Low-level processing only partially accounts for the correlation between comprehension and GPT-2's mapping

Low-level speech representations typically vary with attention [START_REF] Mesgarani | Selective cortical representation of attended speaker 327 in multi-talker speech perception[END_REF][START_REF] Cohen | How does 332 inattention affect written and spoken language processing[END_REF], and could thus, in turn, influence down-stream comprehension processes. Consequently, one can legitimately wonder whether D R A F T Fig. 1. A. 101 subjects listen to narratives (70 min of unique audio stimulus in total) while their brain signal is recorded using functional MRI. At the end of each story, a questionnaire is submitted to each subject to assess their understanding, and the answers are summarized into a comprehension score specific to each (narrative, subject) pair (grey box). In parallel (blue box on the left), we measure the mapping between the subject's brain activations and the activations of GPT-2, a deep network trained to predict a word given its past context, both elicited by the same narrative. To this end, a linear spatio-temporal model (f • g) is fitted to predict the brain activity of one voxel Y , given GPT-2 activations X as input. The degree of mapping, called "brain score" is defined for each voxel as the Pearson correlation between predicted and actual brain activity on held-out data (blue equation, cf. Methods). Finally, we test the correlation between the comprehension scores of the subjects and their corresponding brain scores using Pearson's correlation (red equation). A positive correlation means that the representations shared across the brain and GPT-2 are key for the subjects to understand a narrative. B. Brain scores (fMRI predictability) of the activations of the eighth layer of GPT-2. Scores are averaged across subjects, narratives, and voxels within brain regions (142 regions in each hemisphere, following a subdivision of Destrieux Atlas (10), cf. SI.1). Only significant regions are displayed, as assessed with a two-sided Wilcoxon test across (subject, narrative) pairs, testing whether the brain score is significantly different from zero (threshold: .05). C. Brain scores, averaged across fMRI voxels, for different activation spaces: phonological features (word rate, phoneme rate, phonemes, tone and stress, in green), the non-contextualized word embedding of GPT-2 ("Word", light blue) and the activations of the contextualized layers of GPT-2 (from layer one to layer twelve, in blue). The error bars refer to the standard error of the mean across (subject, narrative) pairs (n=237). D. Comprehension and GPT-2 brain scores, averaged across voxels, for each (subject, narrative) pair. In red, Pearson's correlation between the two (denoted R), the corresponding regression line and the 95% confidence interval of the regression coefficient. E. Correlations (R) between comprehension and brain scores over regions of interest. Brain scores are first averaged across voxels within brain regions (similar to B.), then correlated to the subjects' comprehension scores. Only significant correlations are displayed (threshold: .05). F. Correlation scores (R) between comprehension and the subjects' brain mapping with phonological features (M(Phonemic) (i), the share of the word-embedding mapping that is not accounted by phonological features M(Word) -M(Phonemic) (ii) and the share of the GPT-2 eighth layer's mapping not accounted by the word-embedding M(GPT2) -M(Word) (iii). G. Relationship between the average GPT-2-to-brain mapping (eighth layer) per region of interest (similar to B.), and the corresponding correlation with comprehension (R, similar to D.). Only regions of the left hemisphere, significant in both B. and E. are displayed. In black, the top ten regions in terms of brain and correlation scores (cf. SI.1 for the acronyms). Significance in D, E and F is assessed with Pearson's p-value provided by SciPy † . In B, E and F, p-values are corrected for multiple comparison using a False Discovery Rate (Benjamin/Hochberg) over the 2 × 142 regions of interest.

the correlation between comprehension and GPT-2's brain mapping is simply driven by variations in low-level auditory 80 processing. To address this issue, we evaluate the predictabil-81 ity of fMRI given low-level phonological features: the word 82 rate, phoneme rate, phonemes, stress and tone of the narrative 83 (cf. Methods). The corresponding brain scores correlate with 84 the subjects' understanding (R = 0.17, p < 10 -2 ) but less so 85 than the brain scores of GPT-2 (∆R = 0.32). These low-level 86 correlations with comprehension peak in the left superior tem-87 poral cortex (Figure 1F). Overall, this result suggests that the 88 link between comprehension and GPT-2's brain mapping may layer 0) to those obtained from GPT-2's eighth layer, i.e. a 97 contextual embedding. On average across voxels, the corre-98 lation with comprehension is 0.12 lower with GPT-2's word 99 embedding than with its contextual embedding. An analogous 100 analysis, comparing word embedding to phonological features 101 is displayed in 1F. Strictly lexical effects (word-embedding versus phonological) peak in the superior-temporal lobe and in pars triangularis. By contrast, higher-level effects (GPT-2 eighth layer versus word-embedding) peak in the superiorfrontal, posterior superior-temporal gyrus, in the precuneus and in both the triangular and opercular parts of the inferior frontal gyrus -a network typically associated with high-level language comprehension [START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Lerner | Topographic Mapping of a Hierarchy 334 of Temporal Receptive Windows Using a Narrated Story[END_REF][START_REF] Pallier | Cortical representation of the constituent 337 structure of sentences[END_REF][START_REF] Fedorenko | Neural correlate of the construction of sentence meaning[END_REF][START_REF] Friederici | The Brain Basis of Language Processing: From Structure to Function[END_REF][START_REF] Hickok | The cortical organization of speech processing[END_REF].

Comprehension effects are mainly driven by individuals' variability

The variability in comprehension scores could result from exogeneous factors (e.g. some stories may be harder to comprehend than others for GPT-2) and/or from endogeneous factors (e.g. some subjects may better understand specific texts because of their prior knowledge). To address this issue, we fit a linear mixed model to predict comprehension scores given brain scores, specifying the narrative as a random effect (cf. SI.1). The fixed effect of brain score (shared across narratives) is highly significant: β = 0.04, p < 10 -29 , cf. SI.1). However, the random effect (slope specific to each single narrative) is not (β < 10 -2 , p > 0.11). We also replicate the main analysis (Figure 1D) within each single narrative: the correlation with comprehension reaches 0.76 for the 'sherlock' story and is above 0.40 for every story (cf. SI.1). Overall, D R A F T these analyses confirm that the link between GPT-2 and semantic comprehension is mainly driven by subjects' individual differences in their ability to make sense of the narratives.

Discussion

Our analyses reveal a positive correlation between semantic comprehension and the degree to which GPT-2 maps onto brain responses to spoken narratives.

These results strengthen and complete prior work on the brain bases of semantic comprehension. In particular, previous studies have used inter-subject brain correlation to reveal the brain regions associated with understanding [START_REF] Fedorenko | Neural correlate of the construction of sentence meaning[END_REF] The relationship between GPT-2's representations and human comprehension remains to be qualified. First, although highly significant, our brain scores are relatively low [START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Fedorenko | Neural correlate of the construction of sentence meaning[END_REF]. This phenomenon likely results from a mixture of different elements: i) we ran our analyses across all voxels to avoid selection biases, which automatically reduces the average effect sizes and ii) we report the results without correcting for a noise ceiling (cf. SI.1), as our pilot analyses suggest that such noise-ceiling can greatly vary depending on how it is implemented (i.e. fit from mean across subjects, from all or on voxels etc). Second, the correlation between semantic comprehension and GPT-2's mapping is robust (p < 10 -15 ) but far from perfect (R = 0.50). Such correlation thus indicates that the modeling of brain responses with GPT-2 does not fully account for the variation in comprehension. While this result is expected [START_REF] Marcus | Gpt-2 and the nature of intelligence[END_REF], our study provides a promising framework to evaluate the extent to which deep language models represent and understand texts like we do. Finally, our results suggest that the neural bases of comprehension relate to the high-level representations of deep language models. While the mapping of phonological features and word embeddings do correlate with comprehension, GPT-2's contextual embeddings provides brain maps that more reliably predict comprehension (Figure 1F). The superiority of contextual-embedding in predicting comprehension suggests that i) GPT-2 encodes features supporting comprehension and ii) our finding are not solely driven by low-or mid-level processing [START_REF] Mesgarani | Selective cortical representation of attended speaker 327 in multi-talker speech perception[END_REF][START_REF] Cohen | How does 332 inattention affect written and spoken language processing[END_REF]. These elements remain solely based on correlations, however. The factors that causally influence comprehension, ranging from prior knowledge, attention and language complexity should be explicitly manipulated in future work.

Overall, the present study strengthens and clarifies the similarity between the brain and deep language models, repeatedly observed in the past three years (2-4, 11, 20). Together, these findings reinforce the relevance of deep language models in unraveling the neural bases of narrative comprehension. For each of the seven narratives: number of subjects (n), distribution of comprehension scores across subjects and length of the narrative.

Materials and Methods

Brain activations

The brain activations of the 101 subject who 203 listened to the selected narratives were recorded using fMRI, as de-204 scribed in [START_REF] Samuel | Narratives: fMRI 356 data for evaluating models of naturalistic language comprehension[END_REF]. As suggested in the original paper, pairs of (subject, 205 narrative) were excluded because of noisy recordings, resulting in 206 237 pairs in total. word embedding (a look-up table that outputs a single vector per 213 vocabulary word) [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF]. Each layer l can be seen as a nonlinear 214 system that takes a sequence of w words as input, and outputs 215 a contextual vector of dimension (w, d), called the "activations" 216 of layer l (d = 768). Intermediate layers were shown to better 217 encode syntactic and semantic information than input and output 218 layers [START_REF] Jawahar | What Does BERT Learn about the 359 Structure of Language[END_REF], and to better map onto brain activity [START_REF] Toneva | Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)[END_REF][START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF]. Here, we

219

show that the eighth layer of GPT-2 best predicts brain activity 220 1C. We thus select the eighth layer of GPT-2 for our analyses.

D R A F T

Linear mapping between GPT-2 and the brain For each (subject, narrative) pair, we measure the mapping between i) the fMRI activations elicited by the narrative and ii) the activations of GPT-2 (layer nine) elicited by the same narrative. To this end, a linear spatiotemporal model is fitted on a train set to predict the fMRI scans given the GPT-2 activations as input. Then, the mapping is evaluated by computing the Pearson correlation between predicted and actual fMRI scans on a held out set I:

M (s,w) : I → L f • g(X (w) ) i∈I , (Y (s,w) i
) i∈I [START_REF] Radford | Language Models are Unsupervised Multitask Learners[END_REF] With f • g the fitted estimator (g: temporal and f: spatial mappings), L Pearson's correlation, X (w) the activations of GPT-2 and Y (s,w) the fMRI scans of subjects s, both elicited by the narrative w.

In practice, f is a 2 -penalized linear regression. We follow scikitlearn implementation § with ten possible regularization parameters log-spaced between 10 -1 and 10 8 , one optimal parameter per voxel and leave-one-out cross-validation. g is a finite impulse response (FIR) model with 5 delays, where each delay sums the activations of GPT-2 input with the words presented between two TRs. For each (subject, narrative) pair, we split the corresponding fMRI time series into five contiguous chunks using scikit-learn cross-validation.

The procedure is repeated across the five train (80% of the fMRI scans) and disjoint test folds (20% of the fMRI scans). Pearson correlations are averaged across folds to obtain a single score per (subject, narrative) pair. This score, denoted M(X) in Figure 1A, measures the mapping between the activations space X and the brain of one subject, elicited by one narrative.

Phonological features

To account for low-level speech processing, we computed the alignment (Eq. ( 1)) between the fMRI brain record- In Figure 1G, we use the original parcellation for simplicity, and the following acronyms: were assessed with a t-test, as implemented in statsmodels.

Replication across single narratives. To further support that the R were not driven by the narratives' variability, we replicate the analysis of Figure 1D within single narratives. In Figure 3, we show that correlation scores between brain scores and comprehension scores are positive for each of the seven narratives.

Noise Ceiling Estimates. fMRI recordings are inherently noisy.

Thus, we estimate an upper bound of the best brain score that can be obtained given the level of noise in the Narrative dataset.

To this end, for each (subject, narrative) pair, we linearly map the fMRI recordings, not with the GPT-2 activations, but with the average fMRI recordings of the other subjects who listened to that narrative. More precisely, we use the † † https://www.statsmodels.org/ exact same setting as in Eq. ( 1), but we predict Y (s) , not 402 from g(X) (GPT-2's features after temporal alignment, of size 403 ntimes × n dim ), but from the mean of the other subject's brains 404 Y = 1 |S| s =s Y (s ) (of size ntimes × n voxels ). This score is 405 called the noise ceiling for the (subject, narrative) pair. The 406 noise ceilings for each brain region are displayed in Figure 4, 407 and correspond to upper bounds of the brain scores displayed 408 in Figure 1B.

22 resentations

 22 of GPT-2 relate to a human-like understanding 23 remains largely unknown.

  24

  89 be partially explained by -but not reduced to -the variations 90 of low-level auditory processing. 91 The reliability of high-level representations best predict comprehen-92 sion Is the correlation between comprehension and GPT-2's 93 mapping driven by a lexical process and/or by an ability to 94 meaningfully combine words? To tackle this issue, we compare 95 the correlations obtained from GPT-2's word embedding (i.e.

  96

  For example, Lerner et al. recorded subjects' fMRI while they listened to normal texts or texts scrambled at the word, sentence or paragraph level, in order to parametrically manipulate their level of comprehension (15). The corresponding fMRI signals correlated across subjects in the primary and secondary auditory areas even when the input was scrambled below the lexical level. By contrast, fMRI signals also became correlated in the bilateral infero-frontal and temporo-parietal cortex when the scrambling was either not performed, or performed at the level of sentences and paragraphs. Our results are consistent with this hierarchical organization, and thus make an important step towards the development of a cerebral model of narrative comprehension.

185 186Fig. 2 .

 1852 Fig. 2.For each of the seven narratives: number of subjects (n), distribution of comprehension scores across subjects and length of the narrative.

207GPT- 2

 2 activations GPT-2 (1) is a high-performing neural language 208 model trained to predict a word given its previous context (it does 209 not have access to succeeding words), given millions of examples 210 (e.g Wikipedia texts). It consists of multiple Transformer modules 211 (twelve, each of them called "layer") stacked on a non-contextual 212

  221 Our conclusions remain unchanged with other intermediate-to-deep 222 layers of GPT-2 (from 6 th to 12 th layers). 223 In practice, the narratives' transcripts were formatted (replacing 224 special punctuation marks such as "-" and duplicated marks "?." by 225 dots), tokenized using GPT-2 tokenizer and input to the GPT-2 226 pretrained model provided by Huggingface ‡ . The representation of 227 each token is computed separately using a context window a 1024. 228 For instance, to compute the representation of the third token of 229 the story, we input GPT-2 with the third, second and first token, 230 and then extract the activations corresponding to the third token. 231 To compute the representation of a token w k at the end of the 232 story, GPT-2 is input with this token combined with the 1,023 233 preceding tokens. Then, we extract the activations corresponding 234 to w k . The procedure results in a vector of activations of size (w, d) 235 with w the number of tokens in the story and d the dimensionality 236 of the model. There are fewer fMRI scans than words. Thus, 237 the activation vectors between successive fMRI measurements are 238 summed to obtain one vector of size d per measurement. To match 239 the fMRI measurements and the GPT-2 vectors over time, we used 240 the speech-to-text correspondences provided in the fMRI dataset 241 (21).

  ings Y and phonological features X: the word rate (of dimension d = 1, the number of words per fMRI scan), the phoneme rate (d = 1, the number of phonemes per fMRI scan) and the concatenation of phonemes, stresses and tones of the words in the stimuli (categorical feature, d = 117). The latter features are provided in the original Narratives database[START_REF] Samuel | Narratives: fMRI 356 data for evaluating models of naturalistic language comprehension[END_REF], and computed using Gentle ¶ forced-alignment algorithm.SignificanceSignificance was either assessed by using either (i) a second-level Wilcoxon test (two-sided) across subject-narrative pairs, testing whether the mapping (one value per pair) was significantly different from zero (Figure1B), or (ii) by using the first-level Pearson p-value provided by SciPy ‖ (Figure 1D-G). In Figure 1B, E, F, pvalues were corrected for multiple comparison (2 × 142 ROIs) using False Discovery Rate (Benjamin/Hochberg) * * . D R A F T Supporting Information (SI) Brain parcellation. In Figure 1B, E, and F, we used a subdivision of the parcellation from Destrieux Atlas (10). Regions with more than 400 vertices were split into smaller regions (so that each regions contains less than 400 vertices). The original parcellation consists of 75 regions per hemisphere. Our custom parcellation consists in 142 regions per hemisphere.

Mixed-effect

  model. Not all subjects listened to the same stories. To check that the R scores (correlation between comprehension and brain mapping) were not driven by the narratives and questionnaires' variability, a linear mixed-effect model was fit to predict the comprehension of a subject given its brain mapping scores, specifying the narrative as a random effect. More precisely, if w i ∈ R corresponds to the mapping scores of the i th subject that listened to the story w, and Cw i ∈ R refers to the comprehension scores, we estimate the fixed effect parameters β ∈ R and η ∈ R (shared across narratives), and the random effect parameter βw ∈ R and ηw ∈ R (specific to the narrative w) such that:Cw i = ( β + βw) ×w i +( η + ηw) + w iwith w i a vector of i.i.d normal errors with mean 0 and variance σ 2 . In practice, we use the statsmodels † † implementation of linear mixed-effect models. Significance of the coefficients

Fig. 3 .

 3 Fig. 3. Replication within single narratives. Same as Figure 1D for each single narrative.

| Caucheteux et al.

‡ https://github.com/huggingface/transformers Caucheteux et al.

Computational Linguistics, pages 3651-3657, Florence, Italy, 2019. Association for Compu-

tational Linguistics. .

| Caucheteux et al.

Caucheteux et al.

D R A F T