Subba Reddy Oota
email: subba-reddy.oota@inria.fr

Manish Gupta

Mariya Toneva
email: mtoneva@mpi-sws.org

Joint processing of linguistic properties in brains and language models

Keywords:

Language models have been shown to be very effective in predicting brain recordings of subjects experiencing complex language stimuli. For a deeper understanding of this alignment, it is important to understand the correspondence between the detailed processing of linguistic information by the human brain versus language models. We investigate this correspondence via a direct approach, in which we eliminate information related to specific linguistic properties in the language model representations and observe how this intervention affects the alignment with fMRI brain recordings obtained while participants listened to a story. We investigate a range of linguistic properties (surface, syntactic, and semantic) and find that the elimination of each one results in a significant decrease in brain alignment. Specifically, we find that syntactic properties (i.e. Top Constituents and Tree Depth) have the largest effect on the trend of brain alignment across model layers. These findings provide clear evidence for the role of specific linguistic information in the alignment between brain and language models, and open new avenues for mapping the joint information processing in both systems. We make the code publicly available 1 .

Introduction

Language models that have been pretrained for the next word prediction task using millions of text documents can significantly predict brain recordings of people comprehending language (Wehbe et al., 2014;[START_REF] Jain | Incorporating context into language encoding models for fMRI[END_REF]Toneva & Wehbe, 2019;[START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF][START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF][START_REF] Goldstein | Shared computational principles for language processing in humans and deep language models[END_REF]. Understanding the reasons behind the observed similarities between language comprehension in machines and brains can lead to more insight into both systems. While such similarities have been observed at a coarse level, it is not yet clear whether and how the two systems align in their information processing pipeline. This pipeline has been studied separately in both systems. In natural language processing (NLP), researchers use probing tasks to uncover the parts of the model that encode specific linguistic properties (e.g. sentence length, tree depth, top constituents, tense, bigram shift, subject number, object number) [START_REF] Adi | Fine-grained analysis of sentence embeddings using auxiliary prediction tasks[END_REF][START_REF] Hupkes | Visualisation and'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure[END_REF][START_REF] Conneau | What you can cram into a single! vector: Probing sentence embeddings for linguistic properties[END_REF][START_REF] Jawahar | What does bert learn about the structure of language?[END_REF][START_REF] Rogers | A primer in bertology: What we know about how bert works[END_REF]. These techniques have revealed a hierarchy of information processing in multi-layered language models that progresses from simple to complex with increased depth. In cognitive neuroscience, traditional experiments study the processing of specific linguistic properties by carefully controlling the experimental stimulus and observing the locations or time points of processing in the brain that are affected the most by the controlled stimulus [START_REF] Hauk | Effects of word length and frequency on the human event-related potential[END_REF][START_REF] Pallier | Cortical representation of the constituent structure of sentences[END_REF].

More recently, researchers have begun to study the alignment of these brain language regions with the layers of language models, and found that the best alignment was achieved in the middle layers of these models [START_REF] Jain | Incorporating context into language encoding models for fMRI[END_REF]Toneva & Wehbe, 2019;[START_REF] Caucheteux | Language processing in brains and deep neural networks: computational convergence and its limits[END_REF]. This has been Figure 1: Approach to directly test for the effect of a linguistic property on the alignment between a language model and brain recordings. First, we remove the linguistic property from the language model representations by learning a simple linear function f that maps the linguistic property to the language model representations, and use this estimated function to obtain the residual language model representation without the contribution of the linguistic property. Next, we compare the brain alignment (i.e. encoding model performance) before and after the removal of the linguistic property by learning simple linear functions g and h that map the full and residual language model representations to the brain recordings elicited by the corresponding words. Finally, we test whether the differences in brain alignment before and after the removal of the linguistic property are significant and, if so, conclude that the respective linguistic property affects the alignment between the language model and brain recordings.

hypothesized to be because the middle layers may contain the most high-level language information as they are farthest from the input and output layers, which contain word-level information due to the self-supervised training objective. However, this hypothesis is difficult to reconcile with the results from more recent NLP probing tasks [START_REF] Conneau | What you can cram into a single! vector: Probing sentence embeddings for linguistic properties[END_REF]; [START_REF] Jawahar | What does bert learn about the structure of language?[END_REF], which suggest that the deepest layers in the model should represent the highest-level language information. Taken together, these findings open up the question of what linguistic information underlies the observed alignment between brains and language models.

Our work aims to examine this question via a direct approach (see Figure 1 for a schematic). For a number of linguistic properties, we analyze how the alignment between brain recordings and language model representations is affected by the elimination of information related to each linguistic property. For the purposes of this work, we focus on one popular language model-BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF])-which has both been studied extensively in the NLP interpretability literature (i.e. BERTology [START_REF] Jawahar | What does bert learn about the structure of language?[END_REF]) and has been previously shown to significantly predict fMRI recordings of people processing language (Toneva & Wehbe, 2019;[START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF]. We test the effect of a range of linguistic properties that have been previously shown to be represented in pretrained BERT [START_REF] Jawahar | What does bert learn about the structure of language?[END_REF]. We use a dataset of fMRI recordings that are openly available [START_REF] Samuel A Nastase | The "narratives" fmri dataset for evaluating models of naturalistic language comprehension[END_REF] and correspond to 18 participants listening to a natural story.

Using this direct approach, we find that the elimination of each linguistic property results in a significant decrease in brain alignment across all layers of BERT. We additionally find that the syntactic properties (Top Constituents and Tree Depth) have the highest effect on the trend of brain alignment across model layers. Specifically, Top Constituents is responsible for the bump in brain alignment in the middle layers for all language regions whereas Tree Depth has an impact for temporal (ATL and PTL) and frontal language regions (IFG and MFG). Performing the same analyses with a second popular language model, GPT2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], yielded similar results (see Appendix section F).

Our main contributions are as follows:

1. We propose a direct approach to evaluate the joint processing of linguistic properties in brains and language models. 2. We show that removing specific linguistic properties leads to a significant decrease in brain alignment. We find that the tested syntactic properties are the most responsible for the trend of brain alignment across BERT layers. 3. Detailed region and sub-region analysis reveal that properties that may not impact the whole brain alignment trend, may play a significant role in local trends (e.g., Object Number for ATL and IFGOrb regions, Tense for PCC regions, Sentence Length and Subject Number for PFm sub-region).

We make the code publicly available 1 .

Related Work

Our work is most closely related to that of [START_REF] Merlin | Language models and brain alignment: beyond word-level semantics and prediction[END_REF], who employ a similar residual approach to study the supra-word meaning of language by removing the contribution of individual words to brain alignment. We build on this approach to study the effect of specific linguistic properties on brain alignment across layers of a language model. Other recent work has examined individual attention heads in BERT and shown that their performance on syntactic tasks correlate with their brain encoding performance in several brain regions [START_REF] Kumar | Reconstructing the cascade of language processing in the brain using the internal computations of a transformer-based language model[END_REF]. Our work presents a complementary approach that gives direct evidence for the importance of a linguistic property to brain alignment.

This work also relates to previous works that investigated the linguistic properties encoded across the layer hierarchy of language models [START_REF] Adi | Fine-grained analysis of sentence embeddings using auxiliary prediction tasks[END_REF][START_REF] Hupkes | Visualisation and'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure[END_REF][START_REF] Conneau | What you can cram into a single! vector: Probing sentence embeddings for linguistic properties[END_REF][START_REF] Jawahar | What does bert learn about the structure of language?[END_REF][START_REF] Rogers | A primer in bertology: What we know about how bert works[END_REF]. Unlike these studies in NLP, we focus on understanding the degree to which various linguistic properties impact the performance of language models in predicting brain recordings.

Our work also relates to a growing literature that relates representations of words in language models to those in the brain. A number of studies have related brain responses to word embedding methods [START_REF] Pereira | Decoding of generic mental representations from functional mri data using word embeddings[END_REF][START_REF] Andrew J Anderson | Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns[END_REF][START_REF] Pereira | Toward a universal decoder of linguistic meaning from brain activation[END_REF]Toneva & Wehbe, 2019;[START_REF] Hollenstein | Cognival: A framework for cognitive word embedding evaluation[END_REF][START_REF] Sun | Neural encoding and decoding with distributed sentence representations[END_REF], sentence representation models [START_REF] Sun | Towards sentence-level brain decoding with distributed representations[END_REF]Toneva & Wehbe, 2019;[START_REF] Sun | Neural encoding and decoding with distributed sentence representations[END_REF], recurrent neural networks [START_REF] Jain | Incorporating context into language encoding models for fMRI[END_REF][START_REF] Reddy Oota | Stepencog: A convolutional lstm autoencoder for near-perfect fmri encoding[END_REF], and Transformer methods [START_REF] Gauthier | Linking artificial and human neural representations of language[END_REF]Toneva & Wehbe, 2019;[START_REF] Schwartz | Inducing brain-relevant bias in natural language processing models[END_REF][START_REF] Jat | Relating simple sentence representations in deep neural networks and the brain[END_REF][START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF][START_REF] Goldstein | Shared computational principles for language processing in humans and deep language models[END_REF]Oota et al., 2022b,a;[START_REF] Merlin | Language models and brain alignment: beyond word-level semantics and prediction[END_REF][START_REF] Loong | Training language models to summarize narratives improves brain alignment[END_REF][START_REF] Reddy Oota | Syntactic structure processing in the brain while listening[END_REF]. Our approach is complementary to these previous works and can be used to further understand the reasons behind the observed brain alignment.

Dataset Curation

Brain Imaging Dataset The "Narratives" collection aggregates a variety of fMRI datasets collected while human subjects listened to naturalistic spoken stories. We analyze the "Narratives-21 st year" dataset [START_REF] Samuel A Nastase | The "narratives" fmri dataset for evaluating models of naturalistic language comprehension[END_REF], which is one of the largest publicly available fMRI datasets (in terms of number of samples per participant). The dataset contains data from 18 subjects who listened to the story titled "21 st year". The dataset for each subject contains 2226 samples (TRs-Time Repetition).

The dataset was already preprocessed and projected on the surface space ("fsaverage6"). We use the multi-modal parcellation of the human cerebral cortex (Glasser Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps [START_REF] Matthew F Glasser | A multi-modal parcellation of human cerebral cortex[END_REF], since the Narratives dataset contains annotations that correspond to this atlas. et al., 2018;[START_REF] Camille K Milton | Parcellationbased anatomic model of the semantic network[END_REF][START_REF] Desai | Proper and common names in the semantic system[END_REF].

Extracting Word Features from BERT We investigate the alignment with the base pretrained BERT model, provided by Hugging Face (Wolf et al., 2020) (12 layers, 768 dimensions). The primary rationale behind our choice to analyze the BERT model comes from the extensive body of research that aims to delve into the extent to which diverse English language structures are embedded within Transformer-based encoders, a field often referred to as "BERTology" [START_REF] Jawahar | What does bert learn about the structure of language?[END_REF][START_REF] Mohebbi | Exploring the role of bert token representations to explain sentence probing results[END_REF]. We follow previous work to extract the hidden-state representations from each layer of pretrained BERT, given a fixed-length input (Toneva & Wehbe, 2019). To extract the stimulus features from pretrained BERT, we constrained the tokenizer to use a maximum context of previous Downsampling Since the rate of fMRI data acquisition (TR = 1.5sec) was lower than the rate at which the stimulus was presented to the subjects, several words fall under the same TR in a single acquisition. Hence, we match the stimulus acquisition rate to fMRI data recording by downsampling the stimulus features using a 3-lobed Lanczos filter. After downsampling, we obtain the chunkembedding corresponding to each TR.

TR Alignment To account for the delay of the hemodynamic response, we model the hemodynamic response function using a finite response filter (FIR) per voxel and for each subject separately with 8 temporal delays corresponding to 12 seconds [START_REF] Nishimoto | Reconstructing visual experiences from brain activity evoked by natural movies[END_REF].

Probing Tasks Probing tasks [START_REF] Adi | Fine-grained analysis of sentence embeddings using auxiliary prediction tasks[END_REF][START_REF] Hupkes | Visualisation and'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure[END_REF][START_REF] Jawahar | What does bert learn about the structure of language?[END_REF] help in unpacking the linguistic features possibly encoded in neural language models. In this paper, we use six popular probing tasks grouped into three categories-surface (sentence length), syntactic (tree depth and top constituents), and semantic (tense, subject number, and object number)-to assess the capability of BERT layers in encoding each linguistic property. Since other popular probing tasks such as Bigram Shift, Odd-Man-Out, and Coordination Inversion (as discussed in [START_REF] Conneau | What you can cram into a single! vector: Probing sentence embeddings for linguistic properties[END_REF]) have constant labels for our brain dataset, we focused on the remaining six probing tasks mentioned above. Details of each probing task are as follows. Sentence Length: This surface task tests for the length of a sentence. TreeDepth: This syntactic task tests for classification where the goal is to predict depth of the sentence's syntactic tree (with values ranging from 5 to 12). TopConstituents: This syntactic task tests for the sequence of top-level constituents in the syntactic tree. This is a 20-class classification task (e.g. ADVP_NP_NP_, CC_ADVP_NP_VP_). Tense: This semantic task tests for the binary classification, based on whether the main verb of the sentence is marked as being in the present (PRES class) or past (PAST class) tense. Subject Number: This semantic task tests for the binary classification focusing on the number of the subject of the main clause of a sentence. The classes are NN (singular) and NNS (plural or mass: "colors", "waves", etc). Object Number: This semantic task tests for the binary classification focusing on the object number in the main clause of a sentence. Details of these tasks are mentioned in [START_REF] Conneau | What you can cram into a single! vector: Probing sentence embeddings for linguistic properties[END_REF].

Balancing Classes As discussed in Section 4, our method involves removing one linguistic property p at a time. Ideally, removing p should not impact the performance for any other property p ′ . However, if some classes of p co-occur very frequently with a few classes of p ′ , this undesired behavior may happen. The Left column of Appendix Figure 6 shows the presence of such frequent class cooccurrences across properties. To avoid skewed class co-occurrences, we regroup the class labels for three tasks: Sentence Length, TreeDepth and TopConstituents, as shown in Appendix Figure 5.

Probing Task Annotations To annotate the linguistic property labels for each probing task for our dataset, we use Stanford core-NLP stanza library [START_REF] Christopher D Manning | The stanford corenlp natural language processing toolkit[END_REF] for sentence-level annotations. Further, for word-level annotations, we assign the same label for all the words present in a sentence except for the sentence length task.

Probing Tasks Similarity Pearson correlation values between task labels for each pair of tasks were used to construct the similarity matrix for the "21st year" stimulus text, as shown in Figure 2. We observe a high correlation only between Subject Number and Object Number. There are a number of small positive and negative correlations among the remaining properties.

Methodology

Removal of Linguistic Properties To remove a linguistic property from the pretrained BERT representations, we use a ridge regression method in which the probing task label is considered as input and the word features are the target. We compute the residuals by subtracting the predicted feature representations from the actual features resulting in the (linear) removal of a linguistic property from pretrained features (see Figure 1 for a schematic). Because the brain prediction method is also a linear function (see next paragraph), this linear removal limits the contribution of the linguistic property to the eventual brain prediction performance.

Specifically, given an input matrix T i with dimension N × 1 for probing task i, and target word representations W ∈ R N×d , where N denotes the number of words (8267) and d denotes the dimensionality of each word (768 dimension), the ridge regression objective function is f

(T i ) = min θi ∥W -T i θ i ∥ 2 F + λ∥θ i ∥ 2
F where θ i denotes the learned weight coefficient for embedding dimension d for the input task i, ∥.∥ 2 F denotes the Frobenius norm, and λ > 0 is a tunable hyper-parameter representing the regularization weight for each feature dimension. Using the learned weight coefficients, we compute the residuals as follows:

r(T i ) = W -T i θ i .
We verified that another popular method of removing properties from representations-Iterative Null Space Projection (INLP) [START_REF] Ravfogel | Null it out: Guarding protected attributes by iterative nullspace projection[END_REF]-leads to similar results (see Appendix Table 4). We present the results from the ridge regression removal in the main paper due to its simplicity.

Similar to the probing experiments with pretrained BERT, we also remove linguistic properties from GPT-2 representations and observe similar results (see Appendix Table 11).

Voxelwise Encoding Model To explore how linguistic properties are encoded in the brain when listening to stories, we use layerwise pretrained BERT features as well as residuals by removing each linguistic property and using them in a voxelwise encoding model to predict brain responses. If a linguistic property is a good predictor of a specific brain region, information about that property is likely encoded in that region. In this paper, we train fMRI encoding models using Banded ridge regression [START_REF] Nikolaevich Tikhonov | Solutions of ill-posed problems[END_REF] on stimulus representations from the feature spaces mentioned above. Before doing regression, we first z-scored each feature channel separately for training and testing. This was done to match the features to the fMRI responses, which were also z-scored for training and testing. The solution to the banded regression approach is given by f

( β) = argmin β ∥Y -Xβ∥ 2 F + λ∥β∥ 2 F
, where Y denotes the voxels matrix across TRs, β denotes the learned regression coefficients, and X denotes stimulus or residual representations. To find the optimal regularization parameter for each feature space, we use a range of regularization parameters that is explored using cross-validation. The main goal of each fMRI encoding model is to predict brain responses associated with each brain voxel given a stimulus. Cross-Validation The ridge regression parameters were fit using 4-fold cross-validation. All the data samples from K-1 folds were used for training, and the generalization was tested on samples from the left-out fold.

Evaluation Metrics We evaluate our models using Pearson Correlation (PC) which is a popular metric for evaluating brain alignment [START_REF] Jain | Incorporating context into language encoding models for fMRI[END_REF][START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF][START_REF] Goldstein | Shared computational principles for language processing in humans and deep language models[END_REF]. Let TR be the number of time repetitions. Let Y = {Y i } T R i=1 and Ŷ = { Ŷi } T R i=1 denote the actual and predicted value vectors for a single voxel. Thus, Y ∈ R T R and also Ŷ ∈ R T R . We use Pearson Correlation (PC) which is computed as corr(Y, Ŷ ) where corr is the correlation function.

Implementation Details for Reproducibility All experiments were conducted on a machine with 1 NVIDIA GEFORCE-GTX GPU with 16GB GPU RAM. We used banded ridge-regression with the following parameters: MSE loss function, and L2-decay (λ) varied from 10 -1 to 10 -3 ; the best λ was chosen by tuning on validation data; the number of cross-validation runs was 4.

Results

Successful removal of linguistic properties from pretrained BERT

To assess the degree to which different layers of pretrained BERT representation encode labels for each probing task, a probing classifier is trained over it with the embeddings of each layer. For all probing tasks, we use logistic regression as the probing classifier. We train different logistic regression classifiers per layer of BERT using six probing sentence-level datasets created by [START_REF] Conneau | What you can cram into a single! vector: Probing sentence embeddings for linguistic properties[END_REF]. At test time, for each sample from 21 st year, we extract stimuli representations from each layer of BERT, and perform classification for each of the six probing tasks using the learned logistic regression models. To investigate whether a linguistic property was successfully removed from the pretrained representations, we further test whether we can decode the task labels from the residuals for each probing task of the 21 st year stimuli.

Table 1 reports the result for each probing task, before and after removal of the linguistic property from pretrained BERT. We also report similar results for GPT2 in Table 11 in the Appendix. Similarly to earlier works [START_REF] Jawahar | What does bert learn about the structure of language?[END_REF], we find that BERT embeds a rich hierarchy of linguistic signals also for our 21 st year stimuli: surface information at the initial to middle layers, syntactic information in the middle, semantic information at the middle to top layers. We verify that the removal of each linguistic property from BERT leads to reduced task performance across all layers, as expected.

We further test whether removing information about one property affects the decoding performance for another property. We observed that most properties are not substantially affected by removing other task properties (see Appendix Tables 5, 6, 7, 8, 9 and 10).

Removal of linguistic properties significantly decreases brain alignment across all layers

In Figure 3 (left), we present the average brain alignment across all layers of pretrained BERT before and after the removal of each linguistic property. Removing each linguistic property leads A red * at a particular layer indicates that the alignment from the pretrained model is significantly reduced by the removal of this linguistic property at this particular layer. The layer-wise results for removing other linguistic properties differ across properties, but are significant at the same layers as TopConstituents and are presented in Appendix Figure 7.

to a significant reduction in brain alignment. In contrast, removing a randomly generated linguistic property vector (for this baseline, think about a new task which has no linguistic meaning. The values for each row is then set to a random value chosen uniformly from across all class labels) does not significantly affect the brain alignment. This validates that the reduction in brain alignment observed after removing the linguistic information is indeed due to the elimination of linguistic information, rather than to the removal method itself. Further, we observe that the brain alignment of a random vector is significantly lower than the residual brain alignment after removal of each linguistic property. This suggests that there are additional important properties for the alignment between pretrained BERT and the fMRI recordings, beyond the ones considered in this work.

In Figure 3 (right), we also report the layer-wise performance for pretrained BERT before and after the removal of one representative linguistic property-TopConstituents. We present the results for the remaining properties in Figure 7 in the Appendix. We also report similar results for GPT2 in Figure11 in the Appendix. Similarly to previous work (Toneva & Wehbe, 2019), we observe that pretrained BERT has the best brain alignment in the middle layers. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure). We observe the most substantial reductions in brain alignment in middle to late layers. This pattern holds across all tested linguistic properties. These results provide direct evidence that these linguistic properties in fact significantly affect the alignment between fMRI recordings and pretrained BERT.

Statistical Significance To estimate the statistical significance of the performance differences (across all tasks), we performed a two-tailed paired-sample t-test on the mean correlation values for the subjects. In all cases, we report p-values across layers. For all layers, the main effect of the two-tailed test was significant for all the tasks with p≤ 0.05 with confidence 95% . Finally, the Benjamni-Hochberg False Discovery Rate (FDR) correction for multiple comparisons [START_REF] Benjamini | Controlling the false discovery rate: a practical and powerful approach to multiple testing[END_REF] is used for all tests (appropriate because fMRI data is considered to have positive dependence [START_REF] Christopher R Genovese | A bayesian time-course model for functional magnetic resonance imaging data[END_REF]). Overall, the p-values confirmed that removing each linguistic property from BERT significantly reduced brain alignment.

Figure 10 in the Appendix displays p-values for the brain alignment of pretrained BERT before and after the removal of each linguistic property across all the layers. We observe that the alignment with the whole brain is significantly reduced across all the layers.

ROI-Level Analysis

We further examine the effect on the alignment specifically in a set of regions of interest (ROI) that are thought to underlie language comprehension [START_REF] Fedorenko | New method for fMRI investigations of language: Defining ROIs functionally in individual subjects[END_REF] Table 2: ROI-level (left) and whole brain (right) correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. [START_REF] Fedorenko | Reworking the language network[END_REF]) and word semantics [START_REF] Jeffrey R Binder | Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies[END_REF]. We find that across language regions, the alignment is significantly decreased by the removal of the linguistic properties across all layers (see Appendix Figure 8). We further investigate the language sub-regions, such as 44, 45, IFJa, IFSp, STGa, STSdp, STSda, A5, PSL, and STV and find that they also align significantly worse after the removal of the linguistic properties from pretrained BERT (see Appendix Figure 9).

Decoding task performance vs brain alignment

While the previous analyses revealed the importance of linguistic properties for brain alignment at individual layers, we would also like to understand the contribution of each linguistic property to the trend of brain alignment across layers. For this purpose, we perform both quantitative and qualitative analyses by measuring the correlation across layers between the differences in decoding task performance from pretrained and residual BERT and the differences in brain alignment of pretrained BERT and residual BERT. A high correlation for a specific linguistic property suggests a strong relationship between the presence of this property across layers in the model and its corresponding effect on the layer-wise brain alignment. Note that this analysis can provide a more direct and stronger claim for the effect of a linguistic property on the brain alignment across layers than alternate analyses, such as computing the correlation between the decoding task performance and the brain alignment across layers only at the level of the pretrained model. We present the results of these analyses for the whole brain and language regions, and language sub-regions in Tables 2 and3, respectively. 2 indicate that the syntactic NLP tasks TopConstituents and TreeDepth have the strongest effect on the alignment between BERT and the whole brain. At the level of the whole brain, surface level and semantic properties have a moderate effect on the trend in brain alignment across layers.

Whole Brain Analysis High correlations in the last column of Table

ROI-Level Analysis Further, we analyze these correlations for each language brain region separately, and report the results in the remaining columns of Table 2. We make the following observations:

(1) Both TreeDepth and TopConstituents are responsible for the bump in brain alignment not just for the entire brain but also for several language ROIs. TopConstituents is important across all language ROIs, but most important in IFG which is related to syntax. This result aligns with previous work that has found the inferior frontal regions to be sensitive to syntax [START_REF] Friederici | The role of left inferior frontal and superior temporal cortex in sentence comprehension: localizing syntactic and semantic processes[END_REF][START_REF] Friederici | The cortical language circuit: from auditory perception to sentence comprehension[END_REF]. ( 2) Unlike TopConstituents which has a strong effect on the alignment with all language ROI, TreeDepth has a strong effect only on alignment with the temporal (ATL and PTL) and frontal regions (IFG and MFG). ( 3) Although semantic properties are not shown to be responsible for the shape in brain alignment at the whole-brain level, semantic properties such as Tense, Subject Number and Object Number affect the alignment more specifically in PCC. This finding agrees with previous work which suggests the PCC supports word semantics [START_REF] Jeffrey R Binder | Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies[END_REF]. ( 4 Sentence Length has a strong effect on the alignment with the IFG and MFG, but not as high as the syntactic properties in these language ROIs.

Sub-ROI-Level Analysis Each language brain region is not necessarily homogeneous in function across all voxels it contains. Therefore, an aggregate analysis across an entire language region may mask some nuanced effects. Thus, we further analyze several important language sub-regions that are thought to exemplify the variety of functionality across some of the broader language regions [START_REF] Rolls | The human language effective connectome[END_REF]: AG sub-regions (PFm, PGi, PGs), ATL sub-regions (STGa, STSda), PTL sub-regions (STSdp, A5, TPOJ1, PSL, STV, SFL), and IFG sub-regions (44,45,IFJa,IFSp). We present the correlations for each of these sub-regions separately in Table 3. We make the following observations: (1) Although TopConstituents strongly affect the trend in the brain alignment across layers for the IFG, it strongly affects the trend for only three sub-regions (44,45,and IFSp). It is possible that other factors affect the trend in alignment with IFJa across layers, as this IFG region is involved in many cognitive processes, including working memory for maintaining and updating ongoing information [START_REF] Rainer | Memory fields of neurons in the primate prefrontal cortex[END_REF]Zanto et al., 2011;[START_REF] Gazzaley | Top-down modulation: bridging selective attention and working memory[END_REF]. ( 2) While TopConstituents was shown to have the strongest effect on the brain alignment with the whole IFG, the alignment with two of the IFG sub-regions (IFSp and IFJa) is more strongly affected by Tree Depth.

(3) Subject Number has a medium to strong effect on the alignment with AG sub-region PFm. This implies that the semantic property Subject Number is important in at least one sub-region of language ROI. ( 4) Similarly, we find Object Number to be important for the trend of alignment across layers with most language sub-regions of ATL and PTL. Taken together, these results show that the semantic properties are most responsible for the trend in brain alignment across layers for many language sub-regions of ATL, PTL and AG, while syntactic properties have the highest effect on the alignment with the IFG.

Qualitative Analysis To present the correlations above at an even finer grain, we show them now at the voxel-wise level in Figure 4. We make the following qualitative observations: (1) These results confirm that Sentence Length and Subject Number are the least related to the pattern of the brain alignment across layers.

(2) The effect of Tree Depth, Top Constituents, and Object Number is more localized to the canonical language regions in the left hemisphere and is more distributed in the right hemisphere.

(3) The semantic property Object Number has a much larger effect on the alignment with the left hemisphere. Furthermore, Appendix Figure 12 shows the voxel-wise correlations across layers when removing all linguistic properties, instead of each individual one. We observe that removing all linguistic properties leads to very different region-level brain maps compared to removing individual linguistic properties (Figure 4), and that the remaining correlation across layers after removing all linguistic properties is not substantial in the key language regions.

Discussion and Conclusion

We propose a direct approach for evaluating the joint processing of linguistic properties in brains and language models. We show that the removal of a range of linguistic properties from both language models (pretrained BERT and GPT2) leads to a significant decrease in brain alignment across all layers in the language model.

To understand the contribution of each linguistic property to the trend of brain alignment across layers, we leverage an additional analysis for the whole brain, language regions and language sub-regions: computing the correlations across layers between 1) differences in decoding task performance before and after removing each linguistic property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. For the whole brain, we find that Tree Depth and Top Constituents are the most responsible for the trend in brain alignment across BERT layers. Specifically, TopConstituents has the largest effect on the trend in brain alignment across BERT layers for all language regions, whereas TreeDepth has a strong effect on a smaller subset of regions that include temporal (ATL and PTL) and frontal language regions (IFG and MFG).

Further, although other properties may not have a large impact at the whole brain level, they are important locally. We find that semantic properties, such as Tense, Subject Number, and Object Number affect the alignment with more semantic regions, such as PCC [START_REF] Jeffrey R Binder | Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies[END_REF].

One limitation of our removal approach is that any information that is correlated with the removed linguistic property in our dataset will also be removed. This limitation can be alleviated by increasing the dataset size, which is becoming increasingly possible with new releases of publicly available datasets. It is also important to note that while we find that several linguistic properties affect the alignment between fMRI recordings and a pretrained language model, we also observed that there is substantial remaining brain alignment after removal of all linguistic properties. This suggests that our analysis has not accounted for all linguistic properties that are jointly processed. Future work can build on our approach to incorporate additional linguistic properties to fully characterize the joint processing of information between brains and language models. The current study was performed using experimental stimulus in English and language models trained on English text. While we expect our results to be generalizable at least to languages that are syntactically close to English, this should be explored by future work. Additionally, investigating larger language models beyond the two models used in our study, BERT and GPT-2, can contribute to an even more comprehensive understanding of the reasons for alignment between language models and human brain activity.

The insights gained from our work have implications for AI engineering, neuroscience and model interpretability-some in the short-term, others in the long-term. AI engineering: Our work most immediately fits in with the neuro-AI research direction that specifically investigates the relationship between representations in the brain and representations learned by powerful neural network models. This direction has gained recent traction, especially in the domain of language, thanks to advancements in language models [START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF][START_REF] Goldstein | Shared computational principles for language processing in humans and deep language models[END_REF]. Our work most immediately contributes to this line of research by understanding the reasons for the observed similarity in more depth. Specifically, our work can guide linguistic feature selection, facilitate improved transfer learning, and help in the development of cognitively plausible AI architectures. Computational Modeling in Neuroscience: Researchers have started viewing language models as useful model organisms for human language processing (Toneva, 2021) since they implement a language system in a way that may be very different from the human brain, but may nonetheless offer insights into the linguistic tasks and computational processes that are sufficient or insufficient to solve them [START_REF] Mccloskey | Networks and theories: The place of connectionism in cognitive science[END_REF][START_REF] Baroni | Linguistic generalization and compositionality in modern artificial neural networks[END_REF]. Our work enables cognitive neuroscientists to have more control over using language models as model organisms of language processing. Model Interpretability: In the long-term, we hope that our approach can contribute to another line of work that uses brain signals to interpret the information contained by neural network models (Toneva & Wehbe, 2019;[START_REF] Loong | Training language models to summarize narratives improves brain alignment[END_REF]. We believe that the addition of linguistic features by our approach can further increase the model interpretability enabled by this line of work. Overall, we hope that our approach can be used to better understand the necessary and sufficient properties that lead to significant alignment between brain recordings and language models. 4 12 0 1 0 3 0 2 1 0 3 0 0 0 0 1 1 1 1 0 0 3 0 1 0 2 0 0 1 0 1 0 0 0 0 0 0 0 2 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 150 3 1 4 1 0 4 12 27 0 0 5 4 2 0 2 42 0

B INLP Projection vs. Our Method

We also implemented the Iterative Null-Space Projection (INLP) method [START_REF] Ravfogel | Null it out: Guarding protected attributes by iterative nullspace projection[END_REF] to verify whether our removal method performance is similar to previously proposed method. We found the results to be similar. Results using our method are in Table 4. Results using the INLP method are below. D Layer-wise Language ROIs Analysis before/after removal of linguistic properties.

Here, we examine the effect on the alignment specifically in a set of regions of interest (ROI) that are thought to underlie language comprehension [START_REF] Fedorenko | New method for fMRI investigations of language: Defining ROIs functionally in individual subjects[END_REF][START_REF] Fedorenko | Reworking the language network[END_REF] and word semantics [START_REF] Jeffrey R Binder | Where is the semantic system? a critical review and meta-analysis of 120 functional neuroimaging studies[END_REF]. We find that across language regions, the alignment is significantly decreased by the removal of the linguistic properties across all layers, as shown in Figure 8.

E Layer-wise Language sub-ROIs Analysis before/after removal of linguistic properties.

We further investigate the language sub-regions, such as 44, 45, IFJa, IFSp, STGa, STSdp, STSda, A5, PSL, and STV and find that they also align significantly worse after the removal of the linguistic properties from pretrained BERT (see Figure 9).

Each language brain region is not necessarily homogeneous in function across all voxels it contains. Therefore, an aggregate analysis across an entire language region may mask some nuanced effects. We thus further analyze several important language sub-regions that are thought to exemplify the variety of functionality across the broader language regions [START_REF] Rolls | The human language effective connectome[END_REF]: STGa, STSda, 44, 45, 55b, STV, SFL, PFm, PGi, PGs, IFJa and IFSp in the main paper. Here we extend this analysis to more sub-regions: PSL, STV, SFL, PFm, PGs, PGi. We demonstrate the correlations for each language brain sub-region separately in Table 3 in the main paper.

F Probing Results: GPT2

Like the probing experiments with BERT in the main paper, we also perform experiments with GPT2. We find the results to be similar to BERT, i.e., a rich hierarchy of linguistic signals: initial to middle layers encode surface information, middle layers encode syntax, middle to top layers encode semantics. Table 11 reports the result for each probing task, before and after removal of the linguistic property from pretrained GPT2. We verify that the removal of each linguistic property from GPT2 leads to reduced task performance across all layers, as expected. We also report the layer-wise performance for pretrained GPT2 before and after the removal of one representative linguistic property (TopConstituents) in Fig. 11. We observe that the brain alignment is reduced significantly across all layers after the removal of the linguistic property (indicated with red cross in the figure).

Removal of Sentence Length

Removal of All Tasks

Figure 3 :

 3 Figure 3: Brain alignment of pretrained BERT before (blue) and after (all other bars) removal of different linguistic properties. Left plot compares the average Pearson correlation across all layers of pretrained BERT and all voxels, and the same quantity after removal of each linguistic property. The error bars indicate the standard error of the mean across participants. The right plot compares the layer-wise performance of pretrained BERT and removal of one linguistic property-TopConstituents.A red * at a particular layer indicates that the alignment from the pretrained model is significantly reduced by the removal of this linguistic property at this particular layer. The layer-wise results for removing other linguistic properties differ across properties, but are significant at the same layers as TopConstituents and are presented in Appendix Figure7.

Figure 4 :

 4 Figure4: Voxel-wise correlations across layers between 1) differences in decoding task performance before and after removing each property and 2) differences in brain alignment of pretrained BERT before and after removing the corresponding linguistic property. Here L/R denotes the left and right hemispheres.

AppendixAFigure 5 :

 5 Figure5reports the distribution of class labels across each linguistic property. For the probing tasks such as Sentence Length, Tree Depth, and Top Constituents, we balance the number of classes to overcome the imbalance problem. We balance the classes for these three probing tasks as follows: (i) Sentence Length: 3-classes (≤5, 5-8 and ≥9), (ii) TreeDepth: 3-classes (5, 6-7 and ≥8), and TopConstituents: 2-classes (1, ≥2).

Figure 6

 6 Figure6displays the common samples between class labels of pair of probing tasks. This reports whether the cells are balanced across class labels of different probing tasks.

Figure 6 :

 6 Figure 6: Number of common samples between pair of class labels across probing tasks.

Figure 7 :

 7 Figure7: Balanced Classes: Model trained on pretrained BERT features and removal of different linguistic properties. Each plot compares the layer-wise performance of pretrained BERT and removal of each probing task. Bottom plot displays the pretrained BERT vs. removal of all tasks. A red * at a particular layer indicates that the alignment from the pretrained model is significantly reduced by the removal of this linguistic property at this particular layer.

Figure 8 :

 8 Figure 8: Avg Pearson correlations for language regions AG, ATL, PTL, IFG, MFG, IFGOrb, PCC and dmPFC. Model trained on pretrained BERT features and removal of different linguistic properties (shown only for important properties for clarity). Each plot compares the layer-wise performance of pretrained BERT and removal of each probing task.

Figure 10 :

 10 Figure10: For whole brain, paired t-test was performed to identify the layers and removal of linguistic properties where the pretrained BERT model has greater brain alignment than the removal of each linguistic property, with statistical signifi-cance. p-values obtained from paired t-test, after false discovery rate (FDR) correction using the Benjamini-Hochberg (BH) pro-cedure.

Figure 11 :

 11 Figure 11: Comparison of layer-wise performance of pretrained GPT2 and removal of one linguistic property-TopConstituents.

Figure 12 :

 12 Figure 12: Voxel-wise correlations across layers between brain alignment of pretrained BERT before and after removing all linguistic properties.

Table 1 :

 1 Word-Level Probing task performance for each BERT layer before and after removal of each linguistic property using the 21 st year stimuli.

	Layers Sentence Length	TreeDepth	TopConstituents	Tense	Subject Number Object Number
		3-classes	3-classes	2-classes	2-classes	2-classes	2-classes
		(Surface)	(Syntactic)	(Syntactic)	(Semantic)	(Semantic)	(Semantic)
		before	after	before after before	after	before after before after before after
	1	74.67	43.28	76.30 42.93 77.15 47.28 87.00 59.25 92.10 49.95 93.28 47.31
	2	69.83	42.44	76.72 38.88 78.60 42.75 87.18 48.25 92.32 55.50 93.47 54.59
	3	72.31	46.19	75.76 40.33 77.81 48.85 87.42 44.26 93.04 48.55 93.80 49.76
	4	71.34	46.43	75.94 38.63 78.36 48.00 88.09 42.56 93.50 50.12 94.90 50.06
	5	72.67	46.97	76.00 40.88 78.60 45.28 88.39 44.26 94.05 49.88 93.59 51.45
	6	70.38	44.37	79.02 41.89 80.23 43.47 87.17 44.44 94.98 55.08 94.50 54.17
	7	72.98	46.55	77.93 41.23 80.23 46.43 88.69 42.62 95.88 50.24 94.62 47.58
	8	72.67	44.67	76.07 40.08 78.90 46.86 87.42 44.56 96.10 50.24 95.10 50.18
	9	70.50	45.28	77.15 42.62 79.87 44.55 88.27 47.22 96.38 52.78 94.56 49.27
	10	72.91	47.93	76.90 41.78 78.17 47.76 88.94 45.47 96.06 53.68 94.50 50.30
	11	70.07	46.67	77.27 45.47 77.69 45.77 87.24 48.43 96.94 53.44 94.92 49.52
	12	71.77	42.93	76.39 46.61 78.29 48.67 86.88 45.10 94.03 51.45 93.95 48.73

  Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745-757, 2022. Shaonan Wang, Jiajun Zhang, Haiyan Wang, Nan Lin, and Chengqing Zong. Fine-grained neural decoding with distributed word representations. Information Sciences, 507:256-272, 2020.

	Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-based statistical
	models of language with brain activity during reading. In Proceedings of the 2014 Conference on
	Empirical Methods in Natural Language Processing, pp. 233-243, 2014.
	Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
	Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art
	natural language processing. In Proceedings of the 2020 conference on empirical methods in
	natural language processing: system demonstrations, pp. 38-45, 2020.
	Theodore P Zanto, Michael T Rubens, Arul Thangavel, and Adam Gazzaley. Causal role of the
	prefrontal cortex in top-down modulation of visual processing and working memory. Nature
	neuroscience, 14(5):656-661, 2011.

Table 4 :

 4 Word-Level: INLP Projection Method vs Our Removal Method: Probing task accuracy for each BERT layer after removal of each linguistic property using the 21 st year stimuli.

Table 7 :

 7 Layer-wise Probing task performance for TopConstituents task. Note in tables below AR=After Removal and BR=before removal.

https://github.com/subbareddy248/linguistic-properties-brain-alignment 37th Conference on Neural Information Processing Systems (NeurIPS

2023).

C Layer-wise Whole Brain Analysis before/after removal of linguistic properties.

In Figure 7, we report the layer-wise performance for pretrained BERT before and after the removal of each of the linguistic properties. Similar to previous work (Toneva & Wehbe, 2019), we observe that pretrained BERT has best brain alignment in the middle layers across all properties. We further observe that the alignment after removing the linguistic property is significantly worse mainly for middle to late layers. Note that the layers at which there is a significant difference are indicated with a red dot. This pattern holds across all of the linguistic properties that we tested. These results provide 1 2 3 4 5 6 7 8 9 10 11 12 0


Tiberi

F Goffinet

J-L Pham

Semantic analysis of Self-Defining Memories (SDM) emotional content among antisocial forensic inpatients without (ASPD) or with psychopathic personality disorder (PPD)

Introduction

Self-Defining Memories (SDM) are autobiographical memories, associated with an intense affective charge which enables the memory persistence over time and plays a role in the Self identity establishment (self-consistency and self-coherence). SDM exhibit four main characteristics: a) specificity (memory with a time window of less than 24 hours and sensory details), b) affect (memory with positive or negative, or mixed emotions), c) content (memory with a central theme such as relationship, physical threat, etc.) and d) integration (memory from which the person can learn, and which will build his identity). Self-consistency along with the ability to learn from an experience were found to be impaired among antisocial offenders, suggesting to be a key-factor in the perpetration of new offences. ASPD offenders are characterized as highly impulsive and irritable, transgressive of social and legal norms, and disregarding others and their security. A severe form of this PD appears to be the PPD, with interpersonal-affective features as narcissistic traits (loquacity, superficial charm or over-estimation of Self, etc.), lack of remorse and empathy or the inability to assume responsibility, in addition to antisocial traits. Although still rarely studied, a growing body of literature is now focusing on SDM among ASPD/PPD offenders. Previous research highlighted that ASPD exhibit deficit in the integration and recall of integrated and specific SDM. PPD exhibit deficit in their autobiographical memory, consisting in a distancing of memory recall, using the 3 rd person during the SDM recall. Thereby, little is known about the words used by ASPD and PPD forensic inpatients during the recall of SDM, whereas the use of words is significant and can reveal clues about the personality, the motivation, the cognitive and social resources of individuals. and PDD: M = 7,15; SD = 1,852) than NCP (M = 14,49; SD = 2,45). Forensic samples exhibited a least a Major Mental Disorders diagnosis (56,70%) mainly with an Addiction Disorders (43,30%), on substance (23,30%) and on alcohol (10,00%). All forensic inpatients were diagnosed with ASPD (Cluster B) whom a small prevalence exhibited comorbidity with Borderline PD (30,00%) and Narcissistic PD (16,17%). On a criminal level, the most prevalent non-sexual infraction was 'violent robbery' (33%) and the least prevalent was 'threat' (16,70%). For the sexual infraction, the most prevalent sexual infraction was 'rape' (33%) and the least prevalent was 'indecent exposure' (23%).

Instruments and Procedure

Data Analysis

Each participant had to recall 5 SDM, leading to a sample size of 335 SDM (n ASPD = 83; n PPD = 60; n NCP = 192) for the emotional semantic cues analysis and 336 SDM (n ASPD = 81; n PPD = 62; n NCP = 193) for the SDM characteristics analysis. In the absence of normality of our variables (Kolmogorov-Smirnov), we conducted non-parametric groups comparison analyses (Kruskal-Wallis H), followed with Mann-Whitney U post hoc tests (Dunn-Bonferroni correction). We compared the three groups in term of absolute values and proportions of positive, negative, and not specified words recalled (p-value corrected at .017). High versus low-activation emotions were considered, with a p-value corrected (.025).

We also compared the groups on the SDM recalled characteristics (specificity, integration, valence and theme) using Chi-Square (𝜒 2 ) and Fischer Exact test analyses.

Results

Discussion

SDM Task

Recall 5 SDM (recorded speech and video) • Semantic analyses indicated that both ASPD and PPD forensic inpatients use more low activation emotional words when recalling SDM. This suggest that both forensic inpatients groups distance themselves from their emotions. Nevertheless, and specifically for PPD forensic inpatients, semantic results tend to disagree with the global emotional deficit hypothesis. At the contrary, it suggests that PPD forensic inpatients exhibited specific deficits in the semantic emotional sphere. • As previously found, forensic inpatients (ASPD and PPD) exhibit difficulties to integrate SDM. Moreover, PPD forensic inpatients recall mainly SDM with a neutral valence. This result supports Baumeister's theory concerning the Self coherence deficit among antisocial offenders.

EMOTAIX Text Analysis

Comparison analyses (𝜒

Future perspectives

• In-depth analyses using the four PPD profiles: primary, secondary, manipulative and explosive • Analyze SDM recalling (structure and coherence) using an interview analyze instrument and reflexive functions

View publication stats

  Main objective : Comparison of semantic cues between forensic inpatients (ASPD and PPD), and non-clinical participants (NCP) for naturalistic emotions production through the recall of SDM Method Participants Sample is composed of 69 male participants: ASPD forensic inpatients (n = 17), PPD forensic inpatients (n = 13) and non-clinical participants (n = 39). Forensic inpatients were interned in a High-Security Forensic Hospital because considered 'not guilty for reason of insanity' (NGRI) according to the Law on the internment of persons (2014). ASPD (M = 46,59; SD = 15,04) and PPD (M = 42,38; SD = 12,14) forensic inpatients were significantly older (U = 58,50 ; p < .001) than NCP (M = 26,82; SD = 2,64). Moreover, both forensic samples were significantly less educated (ASPD: M = 8,12; SD = 2,89

  3 levels:Global (6): anxiety, well-being, etc. Intermediate (18): hate, affection, etc. Specific (56): disgust, love, etc. 2 valences: positive and negative + unspecified emotions, surprise and impassibility

2.

  Comparison analyses (Mann-Whitney U) of SDM emotional semantic cues between groups: With regard to the absolute values of words pronounced was considered, ASPD forensic inpatients pronounced significantly less: a) tNEW than NCP (U = 5702,50; p ≤ .001), b) tNHEAW than NCP (U = 5544,50; p ≤ .001) and PPD (U = 1951,00; p ≤ .05), c) PEW than NCP (U = 6334,50; p ≤ .01), d) NeEW than NCP (U = 6615,00 ; p ≤ .05) and NSEW than NCP (U = 5860,50; p ≤ .001).However, when the proportion (%) of these words on the total number of words pronounced was considered, results are slightly different. Besides the confirmation of ASPD pronouncing less tNHEAW than NCP (U = 5973,00; p ≤ .001) and PPD (U = 1994,00; p ≤ .05), and less NSEW than NCP (U = 5309,50; p ≤ .001), they also pronounced more tNLEAW than NCP (U = 5262,00; p ≤ .001) and less NSEW than PPD (U = 1976,50; p ≤ .05). Finally, PPD pronounced more tNLEAW than NCP (U = 4262,00; p ≤ .001).

  2 and Fischer Exact test) of SDM characteristics between groups : 1. Descriptive Statistics on SDM emotional semantic cues for each group: forensic inpatients recalled less specific SDM than NCP (p ≤ .01). No difference was found between the two forensic inpatients groups and between PPD forensic inpatients and NCP. ≤ .001) and PPD (p ≤ .001) forensic inpatients recalled less integrated SDM than NCP.No difference was found between the two forensic inpatients groups.

		ASPD (n = 81)	PPD (n = 62)	NCP (n = 193)	𝝌 𝟐
		n	%	n	%	n	%
	Specific Non-Specific	40 41	49,40 50,60	35 27	56,50 43,50	130 63	67,40 32,60	8,41*
	Note: *p ≤ .05
		ASPD (n = 81)	PPD (n = 62)	NCP (n = 193)	𝝌 𝟐
		n	%	n	%	n	%
	Integrated Non-Integrated	12 69	14,80 86,20	7 55	11,30 88,70	87 106	45,10 54,90	38,64**
	Note: **p ≤ .001
	Positive Valence Negative Valence Neutral Valence Mixed Valence ASPD (p Note: **p ≤ .001	ASPD (n = 81) n % 27 33,30 19 23,50 21 25,90 14 17,30	PPD (n = 62) 10 n 16,10 % 17 27,50 26 41,90 9 14,50	NCP (n = 193) n % 43 22,30 57 29,50 34 17,60 59 30,60	𝝌 𝟐 24,41**
	PPD forensic inpatients recalled less positive valenced SDM than ASPD forensic inpatients
	(p ≤ .05). However, they recalled more neutral valenced SDM than both ASPD forensic inpatients
	(p ≤ .05) and NCP (p ≤ .001).
	Finally, both forensic inpatients recalled less mixed valenced SDM than NCP (p ≤ .05).
		ASPD (n = 81)	PPD (n = 62)	NCP (n = 193)	𝝌 𝟐
		n	%	n	%	n	%
	TLE	17	21,00	13	21,00	46	23,80
	P-RLE	11	13,60	4	6,50	38	19,70
	IR	16	19,80	23	37,10	48	24,90
	S	12	14,80	5	8,10	49	25,40	64,61**
	MC/Sh/G	0	0,00	2	3,20	7	3,60
	SC	1	1,20	2	3,20	1	0,50
	Other	24	29,60	13	21,00	4	2,10
	Note: TLE = Threatening Life Events; P-RLE = Pleasant/Recreational Life Events; IR = Interpersonal Relationships; S = Success;
	MC/Sh/G = Moral Choice, Shame, Guilt; SC = Substance Consumption; ; **p ≤ .001

ASPD

ASPD forensic inpatients recalled significantly more SDM whose theme was 'Other' than NCP (p ≤ .001) but significantly less SDM whose theme was 'Interpersonal Relationships' than PPD forensic inpatients (p ≤ .05). Furthermore, PPD forensic inpatients recalled less SDM whose theme were 'Pleasant/Recreational Events' (p ≤ .05) and 'Success' (p ≤ .01) than NCP. Conversely, PPD forensic inpatients recalled more SDM whose theme was 'Other' than NCP (p ≤ .001).