Subba Reddy Oota
email: subba-reddy.oota@inria.fr

Frederic Alexandre
email: frederic.alexandre@inria.fr

Long Short-Term Memory of Language Models for Predicting Brain Activation During Listening to Stories

Keywords: Brain Encoding, Linear Mapping, fMRI, LSTM, ELMo, Longformer, Transformer

Several popular sequence-based and pretrained language models have been found to be successful for text-driven prediction of brain activations. However, these models still lack longterm memory plausibility (i.e. how they deal with long-term dependencies and contextual information) as well as insights on the underlying neural substrate mechanisms. This paper studies the influence of context representations of different language models such as sequence-based models: Long Short-Term Memory networks (LSTMs) and ELMo, and a pretrained Transformer language model (Longformer). In particular, we study how the internal hidden representations align with the brain activity observed via fMRI when the subjects listen to several narrative stories. We use brain imaging recordings of subjects listening to narrative stories to interpret word and sequence embeddings. We further investigate how the representations of language models layers reveal better semantic context during listening. Experiments across all language model representations provide the following cognitive insights: (i) the representations of LSTM cell states are better aligned with brain recordings than LSTM hidden states, the cell state activity can represent more long-term information, (ii) the representations of ELMo and Longformer display a good predictive performance across brain regions for listening stimuli; (iii) Posterior Medial Cortex (PMC), Temporo-Parieto-Occipital junction (TPOJ), and Dorsal Frontal Lobe (DFL) have higher correlation versus Early Auditory (EAC) and Auditory Association Cortex (AAC).

Introduction

In the past decade, artificial neural networks have witnessed a remarkable insights in the computational neuroscience community in understanding how the brain performs stimulus perception (1) given various forms of sensory inputs like visual processing in object recognition tasks [START_REF] Yamins | Performance-optimized hierarchical models predict neural responses in higher visual cortex[END_REF][START_REF] Cadieu | Deep neural networks rival the representation of primate it cortex for core visual object recognition[END_REF][START_REF] Eickenberg | Seeing it all: Convolutional network layers map the function of the human visual system[END_REF], or (ii) by studying higher-level cognition like language processing [START_REF] Gauthier | Linking artificial and human neural representations of language[END_REF][START_REF] Schrimpf | The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing[END_REF][START_REF] Schwartz | Inducing brain-relevant bias in natural language processing models[END_REF]. This line of work, namely brain encoding, aims at constructing neural brain activity given an input stimulus.

Sentence comprehension is studied using fMRI since a while [START_REF] Constable | Sentence complexity and input modality effects in sentence comprehension: an fmri study[END_REF], Some studies have looked into the modelling of language comprehension: e.g. how sequence-based language models such as echo-state networks (ESN) [START_REF] Hinaut | Real-time parallel processing of grammatical structure in the fronto-striatal system: A recurrent network simulation study using reservoir computing[END_REF] or long short-term memory networks (LSTM) [START_REF] Jain | Incorporating context into language encoding models for fmri[END_REF][START_REF] Variengien | A journey in esn and lstm visualisations on a language task[END_REF] encode syntactic structures and contextual information.

Morevoer, [START_REF] Jain | Incorporating context into language encoding models for fmri[END_REF] used LSTMs to get the context representation of sentences (with a next word prediction task) and then used this representation to predict fMRI data.

Some works studied the LSTM capacity of representing long-term information [START_REF] Karpathy | Visualizing and understanding recurrent networks[END_REF] and its ability to model working memory [START_REF] O'reilly | Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia[END_REF], there still lacks investigation of the long-term memory cognitive plausibility of LSTM and its link to fMRI data. In this paper, we open the black box of LSTM to look at particular LSTM activations: the cell state and the hidden state. This can give more insights on longer-term and shorter-term information. Indeed, the cell state mechanism has been introduced in the original LSTM paper [START_REF] Hochreiter | Long short-term memory[END_REF] in order to keep the error gradient of backpropagation constant over long-time scales. Thus, its activity can represent more long-term information than the hidden state of the LSTM. We also investigate how the pretrained bi-directional sequence embedding language model ELMo [START_REF] Peters | Deep contextualized word representations[END_REF] handles the longer context and interprets the LSTM layers representations that better predict brain activity.

Recently, the researchers studied how the representations from Transformer [START_REF] Vaswani | Attention is all you need[END_REF] based language models such as BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] and RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] could directly predict fMRI data. Interestingly, such Transformer-based neural representations have been found to be very effective for brain encoding as well [START_REF] Schrimpf | The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing[END_REF]. On the other hand, [START_REF] Gauthier | Linking artificial and human neural representations of language[END_REF] fine-tune a pretrained BERT model on multiple natural language processing tasks to find tasks best correlated with high decoding performance. In recent works, (Caucheteux, Gramfort, & King, 2021a;[START_REF] Antonello | Lowdimensional structure in the space of language representations is reflected in brain responses[END_REF] interpret the representations of the Transformer model (GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF]) by disentangling the high-dimensional Transformer representations of language models into four combinatorial classes: lexical, compositional, syntactic, and semantic representations to explore which class is highly associated with language cortical ROIs. However, these models are unable to handle the long-term dependencies (sequence length is fixed to 512 words) due to their self-attention operation. To overcome this limitation, recently, [START_REF] Beltagy | Longformer: The long-document transformer[END_REF] introduced Longformer making it easy to process documents of thousands of tokens or longer and combining local windowed attention with global attention.

In this paper, we uncover insights about the association between fMRI voxel activations and representations of diverse language models: LSTM, ELMo, and Longformer. The predictive power of language model specific representations with brain activation is ascertained by (1) using ridge regression on such representations and predicting activations and (2) computing popular metrics like 2V2 accuracy and Pearson correlation between actual and predicted activations.

Specifically, we make the following contributions in this paper. (1) Given a language model pretrained on corpora by handling long-term dependencies, we propose the problem of finding which of these are the most predictive of fMRI brain activity for listening tasks. (2) The investigation of longterm context of language model results reveals that ELMo and Longformer representations display better correlation during narrative story listening. (3) We also investigate the internal memory representations of LSTM (cell state and hidden state) and derive interesting insights that the cell state representations yield better performance than hidden state representations.

Methodology Brain Imaging Dataset

Narratives-Pieman (Listening to Stories) The "Narratives" collection aggregates a variety of fMRI datasets collected while human subjects listened to naturalistic spoken stories. The Narratives dataset that includes 345 subjects, 891 functional scans, and 27 diverse stories of varying duration totaling ∼4.6 hours of unique stimuli (∼43,000 words) was proposed in [START_REF] Nastase | Narratives: fmri data for evaluating models of naturalistic language comprehension[END_REF]. Similar to earlier works [START_REF] Caucheteux | Modelbased analysis of brain activity reveals the hierarchy of language in 305 subjects[END_REF], we analyze data from 82 subjects listening to the story titled 'PieMan' with 259 TRs (repetition time) 1 . A TR is the length of time between corresponding consecutive points in fMRI: here it is 1.5 sec. We list number of voxels per ROI (Region of Interest) in this dataset in Table 1. We use the multi-modal parcellation of the human cerebral cortex (Glasser Atlas: consists of 180 ROIs in each hemisphere) to display the brain maps [START_REF] Glasser | A multi-modal parcellation of human cerebral cortex[END_REF], since the Narratives dataset contains annotations tied to this atlas. The data covers ten brain ROIs, i.e., Left hemisphere (L), and Right hemisphere (R) for each of the following: (i) early auditory cortex (EAC: A1, LBelt, MBelt, PBelt, and R1) which plays a key role for sound perception since it represents one of the first cortical processing stations for sounds; (ii) auditory association cortex (AAC: A4, A5, STSdp, STSda, STSvp, STSva, STGa, and TA2) which is concerned with the memory and classification of sounds; (iii) posterior medial cortex (PMC: POS1, POS2, v23ab, d23ab, 31pv, 31pd, 7m) which has been implicated in tasks as diverse as attention, memory, spatial navigation, emotion, self-relevance detection, and reward evaluation; (iv) the temporo parieto occipital junction (TPOJ: TPOJ1, TPOJ2, 1 282 TRs (before preprocessing) and 259 TRs (after preprocessing).

TPOJ3, STV, PSL) which is a complex brain territory heavily involved in several high-level neurological functions, such as language, visuo-spatial recognition, writing, reading, symbol processing, calculation, self-processing, working memory, musical memory, and face and object recognition; and (v) the dorsal frontal lobe (DFL: L 55b, SFL, L 44, L 45, IFJA, IFSP) which covers the aspects of pragmatic processing such as discourse management, integration of prosody, interpretation of nonliteral meanings, inference making, ambiguity resolution, and error repair. These five brain ROIs (EAC, AAC, TPOJ, DFL, and PMC) span a cortical hierarchy supporting language and narrative comprehension [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Baldassano | Discovering event structure in continuous narrative perception and memory[END_REF].

Encoding Models

To explore how and where contextual language features are represented in the brain when listening to stories, we extract internal hidden representations from two sequencebased models: Random LSTM and LSTM, ELMo (obtaining context-dependent word embeddings), and popular pretrained Transformer language model (Longformer) used for describing each stimulus sentence and use them in an encoding model to predict brain responses. Our main objective is to compare the correlation between each model dense hidden representations and human cognitive process. In this paper, we train fMRI encoding models using Ridge regression on stimuli representations obtained using four models: Random LSTM, LSTM, ELMo, and Longformer. The main goal of each fMRI encoder model is to predict brain responses associated with each brain region given stimuli. In all cases, we train a ridge regression model per subject separately. Following the literature on brain encoding [START_REF] Caucheteux | Modelbased analysis of brain activity reveals the hierarchy of language in 305 subjects[END_REF][START_REF] Toneva | Modeling task effects on meaning representation in the brain via zero-shot meg prediction[END_REF], we choose to use a ridge regression model instead of more complex models. For instance, [START_REF] Affolter | Brain2word: Decoding brain activity for language generation[END_REF]) used a neural network-based model that directly maps fMRI-to-word in a decoding setup. We plan to explore more such models as part of future work in brain encoding. Here, our main is objective to investigate the influence of context representations of different language models such as sequence-based models: LSTM and ELMo, and popular pretrained Transformer language model (Longformer).

LSTM

First, we train an LSTM [START_REF] Hochreiter | Long short-term memory[END_REF] network to predict the probability of the next word as a function of the history of previous words. The weights of LSTMs are learned using the error back-propagation through time (BPTT). To make the association between encoded stimuli from LSTM's internal components and fMRI brain activity, we do the folllowing: (i) At the time step t, we use vector a t to represent the internal neurons of encoded stimuli in LSTM. In this paper, a t may be hidden state vector (h t ) and cell state vector (c t ). (ii) In order to map the stimuli encoded vector a t of LSTM and brain activity at the t-th time step (Y t ), we define a simple linear model, ridge regression, to predict the brain activity ( Ŷt ) from a t , as discussed in the ridge regression section.

Pretrained text Transformer: Longformer

Longformer [START_REF] Beltagy | Longformer: The long-document transformer[END_REF] builds on BERT's language masking strategy and supports long document generative sequence-to-sequence tasks. We use the pretrained Longformer model with a local attention mechanism, where the default window size is set to 5. To obtain the stimuli representation, we use the last layer token representations where each token dimension is 768.

Linear Probing of Language Models

Here, we do not train Random LSTM, LSTM, ELMo, and Longformer networks to directly predict brain activities, for two reasons. First, the dimension of the fMRI voxels varies among different subjects and across different ROIs. Therefore, it is not convenient to design a universal neural network architecture for generating outputs of different dimensions. Second, the goal of this research is not to improve the performance of language models in predicting fMRI. We want to explore linear mappings between particular features of language models states and neural activities in the auditory and language brain ROIs. Namely, we look at (i) the characteristics of hidden state vectors (h t ) and artificial memory vectors (c t ) in both LSTMs and Random LSTMs, and (ii) local context vectors obtained from performance-optimized deep neural network models (ELMo and Longformer). Therefore, we avoid any possible supervision from the fMRI data when training LSTM and Random LSTM language models.

Ridge Regression

We trained a ridge regression based encoding model to predict the fMRI brain activity associated with the semantic vector representation obtained from each language model: ran-dLSTM (hidden state, cell state), LSTM (hidden state, cell state), GloVe, ELMo, and Longformer. Each voxel value is predicted using a separate ridge regression model. Formally, at the time step (t), we encode the stimuli as X t ∈ R N×D and brain region voxels Y t ∈ R N×V , where N denotes the number of training examples, D denotes the dimension of input stimuli representation, and V denotes the number of voxels in a particular region. Hyper-parameter Setting: We used sklearn's ridgeregression with default parameters, 5-fold cross-validation, Stochastic-Average-Gradient Descent Optimizer, Huggingface for Longformer, MSE loss function, and L2-decay (λ):1.0. We used Word-Piece tokenizer for the Longformer model and Spacy-tokenizer for the GloVe and ELMo models.

Model Prediction Across Whole Brain

To determine the significant voxel predictions across the whole-brain, we ran the permutation tests where we shuffled the true responses 5000 times, computed the Pearson correlation scores, and finally obtained the FDR corrected p-values for the whole brain results using both Longformer and ELMo. We set the correlation score of voxels to zero if the p-value of the correlation obtained from the permutation test is above the significance threshold (p> 0.05, FDR corrected).

Evaluation Metrics

We evaluate our models using popular brain encoding evaluation metrics described in the following. Given a subject and a brain region, let N be the number of samples. Let {Y i } N i=1 and { Ŷi } N i=1 denote the actual and predicted voxel value vectors for the i th sample. Thus, Y ∈ R N×V and Ŷ ∈ R N×V where V is the number of voxels in that region. 2V2 Accuracy is computed as follows.

2V2Acc = 1 N C 2 N-1 ∑ i=1 N ∑ j=i+1 I[{cosD(Y i , Ŷi ) + cosD(Y j , Ŷj )} < {cosD(Y i , Ŷj ) + cosD(Y j , Ŷi )}]
where cosD is the cosine distance function. I[c] is an indicator function such that I[c] = 1 if c is true, else it is 0. The higher the 2V2 accuracy, the better. Ŷi ] where corr is the correlation function.

Pearson Correlation (PC) is computed as PC= 1 N ∑ n i=1 corr[Y i ,

Comparison to Other Language Models

We compare the LSTM model with the Longformer and several other pretrained language models: Random LSTM, ELMo, and GloVe. We encoded the number of words same for both ELMo and Longformer to match the performance. ELMo: ELMo (Embeddings from Language Models) is a successful NLP framework developed by the Al-lenNLP [START_REF] Peters | Deep contextualized word representations[END_REF] group. Unlike earlier embeddings, ELMo embeddings represent words in a contextual fashion using a bidirectional LSTM model. We perform the average of word embeddings at each TR (1.5sec.) to obtain the contextual representation. GloVe: based word vectors (each word is a 300-dimension vector) [START_REF] Pennington | Glove: Global vectors for word representation[END_REF], and the average of word embeddings results in context vector in each time step.

Results

In order to assess the performance of the fMRI encoder models learned using the representations from a variety of language models, we computed the 2V2 accuracy and Pearson correlation coefficient between the predicted and true responses across various ROIs for the listening (Narratives-Pieman) dataset (Fig. 1).

Encoding performance of Language Models

From Fig. 1, we observe that the profiles of performance show low scores in the early auditory cortex (EAC) and auditory association cortex (AAC); average scores in TPOJ and DFL; and superior scores in PMC. This aligns with the known language hierarchy for spoken language understanding [START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF][START_REF] Baldassano | Discovering event structure in continuous narrative perception and memory[END_REF][START_REF] Nastase | Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space[END_REF]. Language models ELMo, and Longformer yield better performance in predicting the brain responses than the LSTM model across all the ROIs. These Pearson correlation (ρ) results are comparatively much higher compared to those obtained using the pretrained GPT2 model in (Caucheteux et al., 2021a) (ρ ranging from 0.02 -0.06).

As shown in Fig. 1, our method obtains more than 3 times higher correlations (ρ ranging from 0.02 -0.19)2 . The main reason is that the Longformer is designed to process documents of thousands of tokens or longer sequences while GPT-2 models are unable to handle the long-term dependencies (sequence length is fixed to 512 words). Also, the narrative dataset consists of longer documents (more than 2000 words in one story); the traditional transformer models consider the context up to 512 words, whereas Longformer handles even longer documents. Further, from Fig. 1, we see that the bilateral posterior medial cortex (PMC) associated with higher language function exhibits a higher correlation among all the brain ROIs. ROIs, including bilateral TPOJ and bilateral DFL, yield higher correlations with the ELMo and Longformer, which is in line with the language processing hierarchy in the human brain. Finally, across all regions, Rand Noise vector and Rand LSTM models have worse correlation compared to LSTM and other language models. In summary, different and dis- tinct language model features seem to be related to the encoding performance in listening tasks.

In order to estimate the statistical significance of the performance differences, we performed one-way ANOVA on the mean correlation values for the subjects across the language models (GloVe, LSTM (cell state), LSTM (hidden state), ELMo, and Longformer) for the five brain ROIs. The main effect of the ANOVA test was significant for all the ROIs with p≤ 10 -2 with confidence 95%. Further, post hoc pairwise comparisons [START_REF] Ruxton | Time for some a priori thinking about post hoc testing[END_REF] confirmed the visual observations that on both 2V2 accuracy and Pearson correlation measures, tasks such as ELMo and Longformer performed significantly better compared to other models, as shown in Table 2.

LSTM: Effects of Hidden State vs Cell State Vectors

In order to explore how LSTM hidden units learn to encode the long-term and short-term memory information and the interaction between the two types of working memories, we compare the encoding performance between representations of hidden state and cell state vectors. Fig. 1 showcases the fMRI encoding performance of both RandLSTM and LSTM models where the cell state representations (long term-memory vector) yield better performance than hidden state representations (short-term memory). This supports the cognitive plausibility of the LSTM cell architecture. Besides, the performance of GloVe and RandLSTM models have significantly equal performance, indicating that semantic context is missing in these models.

Which ELMo layers perform better encoding?

We investigate how the performance of ELMo changes at different layers (Embedding layer, LSTM layer-1, and LSTM layer-2), as they are provided in different contexts. The results are shown in Fig. 2. From Fig. 2, we observe that layer-2 displays better 2v2 accuracy and Pearson correlation score compared to other layers. We further observe that the layer 1 show a sharp increase in performance compared to embedding layer in the context of narrative story listening.

Which Longformer layers perform better encoding?

Given the hierarchical processing of language information across the Transformer layers, we further examine how these Transformer layers encode fMRI brain activity using encoder layers of Longformer. We present the layer-wise encoding performance results across brain ROIs in Fig. 3. We observe that in all the layers, intermediate layers (6 to 8) perform the

Cognitive Insights

We further analyse in more detail the prediction performance of the encoder model trained on sub ROIs for the ELMo and Longformer in Fig. 4. In the EAC, the sub ROI pbelt (parabelt) display higher Pearson correlation among other sub ROIs, and it is adjacent to the lateral belt on the exposed surface of the superior temporal gyrus (STG). Also, the pbelt area represent the next level in the auditory hierarchy and mainly concerned with memory or decision-making. Similarly in the AAC, the sub ROIs such as A4, A5, stsvp, and stsda yield better correlation, and these ROIs shares the primary medial and posterior borders with TPOJ (Trumpp, Kliese, Hoenig, Haarmeier, & Kiefer, 2013). Further, there is evidence that these sub ROIs of the AAC process perceptual and conceptual acoustic sounds during auditory stories and social interaction tasks [START_REF] Glasser | A multi-modal parcellation of human cerebral cortex[END_REF]. It can be observed that sub ROIs such as Pos1 and Pos2 have a higher Pearson correlation than other sub ROIs of the PMC region. Both sfl and l55b display a higher correlation among all the sub ROIs for the DFL ROI. However, all the sub ROIs in the TPOJ yield higher correlation, as shown in Fig. 4. The control and attention ROIs in the posterior cingulate cortex (for ex., POS1 in PMC), together with the superior frontal language region (sfl in DFL) and TPOJ, are part of the language network associated with narrative comprehension (Nastase et al., 2020): it is encouraging to see that both ELMo and Longformer also relate to semantic analysis of the ongoing narrative because they obtain best performance, showing that capturing longerterm context is important.

Brain maps for whole brain predictions

The whole-brain prediction Pearson correlation for all the voxels using ELMo and Longformer is shown in Fig. 5. In the listening task, we observe from Fig. 5 that Longformer displays higher correlation values for many voxels than ELMo. From Fig. 5, we see that ROIs such as EAC and AAC have a lower percentage of voxels with a higher correlation compared to PMC and TPOJ brain ROIs (higher percentage of voxels with higher correlation).

Limitations

We believe that more complex models instead of simple ridge regression models can lead to further exciting insights: (i.e.

we may be able to link more directly internal model mechanisms to brain activations). However, to achieve this, one would need more data (more subjects and longer stories) to train such complex models.

Conclusion

This paper studied the long-term memory plausibility of language models for brain encoding. We observe that building individual encoding models and interpreting the internal representations among models can provide a more in-depth understanding of the neural representation of language information. Our experiments on the Narrative listening stories dataset lead to interesting cognitive insights.

Figure 1 :

 1 Figure 1: Pearson correlation coefficient (top figure) and 2V2 Accuracy (bottom figure) between predicted and true responses across different brain regions using a variety of language models (for Narratives-Pieman dataset). Results are averaged across all participants. ELMo and Longformer are the best. Rand Noise stands for a "Random noise vector". (hidden state) stands for the "short-term memory of internal state of the LSTM".

Figure 3 :

 3 Figure 3: Longformer layers: Pearson correlation coefficient between predicted and true responses across different brain regions (different color lines) using Longformer. Results are averaged across all participants. Middle layers (6 and 8) show best correlation.

Figure 4 :

 4 Figure 4: Pearson correlation coefficient between predicted and true responses across different sub ROIs of the Language Network using ELMo and Longformer. Results are averaged across all participants.

Figure 5 :

 5 Figure 5: BrainMaps: Whole-brain prediction correlation using representations of Longformer (left) and ELMo (right) in one sample subject (subject 1) of Narratives-Pieman dataset.Discussion(1) We used a ridge regression model instead of more complicated models for encoding; (2) We experimented with several language models where the pretrained context representations (such as in ELMo and Longformer) are better predictors of voxel activations. (3) In LSTM, the cell state representations (long term memory vector) yield better encoding performance than hidden state representations; thus, internal dynamics of LSTMs seem to have more cognitively plausible activations than classically studied LSTM (4) We used different layers of ELMo and Longformer, where higher layers display better correlation for ELMo while intermediate layers show superior performance for Longformer. (5) The control and attention ROIs in the posterior cingulate cortex, together with the superior frontal language region (sfl in DFL) and TPOJ, are part of the language network associated with narrative comprehension. (6) The posterior medial cortex (PMC), temporo-parieto-occipital junction (TPOJ), dorsal frontal lobe (DFL) have higher correlation versus early auditory and auditory association cortex.

Table 1 :

 1 # Voxels in each ROI in the Narratives Dataset. LH -Left Hemisphere. RH -Right Hemisphere. Pieman has 82 subjects.

	ROIs→	EAC LH RH LH RH LH RH LH RH LH RH AAC PMC TPOJ DFL
	# Voxels 808 638 1420 1493 1198 1204 847 1188 1061 875

Table 2 :

 2 p-values obtained using post hoc pairwise comparisons for the three best models (+ LSTM hidden state).

	Models compared	EAC	AAC	PMC	TPOJ	DFL
	Longformer vs ELMo	0.521	0.271	0.168	0.054	0.356
	LSTM (Cell state vs hid-	0.991	0.177	0.0357* 0.0038*	0.158
	den state)
	Longformer vs LSTM	0.048* 0.0008* 0.00002* 0.00003* 0.0015*
	(cell state)
	ELMo vs LSTM (cell	0.372 0.003* 0.00004* 0.00006* 0.0049*
	state)

we do not apply on the same number of subjects and/or same amount of stories than in(Caucheteux et al., 2021a). However, we tested with few other stories such as Lucy and Slumlord, and our results (higher correlations) show similar trends.

Acknowledgments

We thank the anonymous reviewers for their very helpful comments on previous version of this paper.


Karine Amis Cavalec
email: karine.cavalec@enst-bretagne.fr

Ramesh Pyndiah

K Cavalec-Amis

Block turbo codes for space-time systems

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

Capacity of wireless communication systems undergoes lim itations due to limited bandwith, propagation loss, noise, mul tipath effects of the physical channel .. . On the other hand the development of mobile applications requires higher and higher capacities. Recent research in the field of Information Theory showed that systems using multiple transmitters and multiple receivers can achieve very high spectral efficiencies on wire less channels. Several Space-Time architectures [START_REF] Foschini | Layered Space Time Architecture for Wireless Communication in a Fading Environment when using Multi-Element Antennas[END_REF], [START_REF] Ta Rokh | Combined Array Processing and Space Time Coding[END_REF] have since been proposed and Space-Time codes have been devel oped [START_REF] Tarokh | Space Time codes for high data rate wireless communication : performance criterion and code construction[END_REF]. More recently Stefanov and Duman [START_REF] Stefanov | Turbo Coded Modulation for Systems with Transmit and Receive Antenna Diversity[END_REF] proposed Convolutional Turbo Codes for achieving Transmit and Re ceive diversity. This forward Error Correction technique in troduced in 1993 by C. Berrou[ 1] was a real breakthrough in the field of Information Theory as it achieves very low BERs at SNRs close to the theoretical Shannon's limit. In 1994, R. Pyndiah extended the principle to Product codes [START_REF] Pyndiah | Near-Optimum Decoding of Product Codes: Block Turbo Codes[END_REF], [START_REF] Pyndiah | Near Optimum Decoding of Product Codes[END_REF]. Block Turbo Codes (BTC) perform as well as Convolutional Turbo Codes (CTC). In this article, we propose to apply BTC to Space-Time systems. In Section II we present the considered wireless system and the associated notations. In Section III, after a quick overview of Product Codes, we deal with the interleaving device, the Soft Input Output Decoder computations and the Turbo Decoding algorithm. Finally in Section IV, we give performance of BTC

I � b t I r I e • j e [ hij] r t m e • b t
Figure 1: The System Model.

used with a QPSK and a 16-QAM.

II. THE SYSTEM MODEL

We consider a communication system consisting of n trans mitters and m receivers. Information data are first encoded, then interleaved with other frames if necessary before passing through a serial to parallel converter. Each converter output is mapped to a given signal constellation. Let us note e� the sig nal transmitted from the ith antenna at time t. The Space-Time system is shown in Fig. The signal at time t at the l h receiver is the sum of the noise with the superposi�ion of the n attenuated signals e: . The noise samples (bf, ... , bf, .. , bfl) are supposed to be indepen dent.complex, gaussian, of zero mean and variance No/2 per dimension. The channel attenuation coefficients h:; are con sidered complex, gaussian, of zero mean and variance 1/../2 stand for code length, number of information bits and mini mum Hamming distance respectively. So as to obtain the prod uct code P = C1 x C2, we first place information bits in a k1 x k2 matrix (cf. Fig. 2). Then we encode the ki rows using C2 and finally we encode the n2 columns using C1. The code parameters of P are :

One can show that all the n1 rows of the matrix in Fig. 2 are code words of C2 just as the n2 columns are code words of C t •

B. Interleaving

After encoding, the encoded frame Ek is interleaved diag onalwise giving h. Then, if inter frames interleaving is re quired, we proceed according to the device illustrated in Fig. 3.

(Ikh<k<K are the frames that are to be interleaved and D the n x ( L x q x T) resulting matrix with T the symbol number over which the channel coefficients are considered constant, L x T the number of symbols transmitted from a given antenna and 2q the modulation constellation size.

C. Decoder input computation

Let us assume that the modulation constellation size is 2q.

We note & this constellation, (d�, � •. .. , d�) the q-uple mapped to the constellation signal e ; . At time t, we have to extract n x q reliability values x� from r1, .. , rm. They will be used as inputs of the SISO decoder. As proposed by Stefanov and Duman [START_REF] Stefanov | Turbo Coded Modulation for Systems with Transmit and Receive Antenna Diversity[END_REF], we have taken the weighted Log Likelihood Ratio jn (LLR) values for x� (corresponding to d�). The LLR of d� is defined as:

I I I I •• ••• ••••••••••••• -I Lxgxi:
( Ji ) P(d� = ljr1, .. , rm)

r ap = In ( . I 1 ) P d� = 0 r , .. , rm (2)
We note r = [r1r 2 ... rm]T and e = [e 1 e 2 ... enjT. Eq.( 2) can be written in the following way :

(3)

By applying the Bayes' rule and taking into account the fact that the noise is AWGN per dimension, we can write r ( �) as follows :

The main drawback of this approach is its computational com plexity. A first simplification can be obtained by the approxi mation given in Eq.( 5). Simulations show that this approxima tion does not introduce any significant degradation. After the Log Likelihood computations, data are deinter leaved before being decoded. We note v; the deinterleaved data. The received matrix (see Fig. 2) is formed by correctly placing the data v;. The iterative decoding consists in decod ing first the rows of the matrix then its columns. This procedure is iterated x times. Each elementary decoder uses soft inputs provided by the channel and by the previous decoder which de livers soft outputs called "extrinsic information" [START_REF] Pyndiah | Near-Optimum Decoding of Product Codes: Block Turbo Codes[END_REF], [START_REF] Pyndiah | Near Optimum Decoding of Product Codes[END_REF]. This extrinsic information is added to the channel observation and becomes the next decoder soft input. Each elementary decoder uses the Chase algorithm [START_REF] Chase | A class of algorithms for decoding block codes with channel measurement information[END_REF] to find the most likely code word D = (di)1<i<n and evaluates the Log Likelihood Ratio (LLR) of each deeoded bit d; given by :

f(d•) =In P{cj = +llV} 3 P{cJ = -ljV} LetSe(j) = {C E Clcj =€}. Then (6) becomes : f(dJ) =In L cEs+•o> P{V IC} L CES -l(j) P{V I C} (6) (7)
Let us assume that the noise is AWGN of variance u2• One can write [START_REF] Pyndiah | Near Optimum Decoding of Product Codes[END_REF] in the following way :

f(d • ) -I L..,, CES+•O> exp 2a > J -n -llV-Cll2 L cEs-•o> exp 2a2
One can approximate [START_REF] Pyndiah | Near-Optimum Decoding of Product Codes: Block Turbo Codes[END_REF], [START_REF] Pyndiah | Near Optimum Decoding of Product Codes[END_REF] f(dj) by the following:

llV -c+1ul112 2u2 (8)
performance in terms of BE R are given; ideal interleaving be cause the channel coefficients are changed for each symbol. By increasing K, we get closer to these optimal performance. Ta king into account that this system is dedicated to high bit rate applications, we have only used 4 iterations. We assume a [START_REF] Chase | A class of algorithms for decoding block codes with channel measurement information[END_REF] perfect knowledge of the channel at the receiver.

with ce(j) = argmin{llV -ce(j)ll , C E seU) }. Let us note D = argmin{llV -c+i<J>ll, llV -c-i(;)ll}. Di s the most probable code word and C = argmax{llV -c+i(j)ll, llVc-1(j)ll} is called the competing code word of D. Eq.( 9) can be rewritten as:

Eq.( 10) can be also expressed as:

(11

)
where Vj is the soft input and W j the soft output called "extrin sic information" . If C is not available, then W j = (Jdj, where (J is a predefined constant.

IV. SIMULATION RESULTS

In all simulations, our Space Time system consists of 2 trans mitters and 2 receivers. We consider the (32, 26, 4) 2 extended BCH product code which bears a code rate close to 2/3. We compare several interleaved frame numbers K. Therefore the total transmitted bit number is equal to K x 1024. The channel coefficients are supposed to be constant over T = 128 con stellation symbols. For each modulation, the ideal interleaving

A. 4-PSK modulation

The spectral efficiency is close to 3. We compare different interleaved frame numbers. One of the main features of a Phase Modulation is that the amplitude of the constellation signals is constant. We have considered a Gray mapping. Simulation results are shown in Fig. 4. If we compare the curve with 2 interleaved blocks with the optimal curve, we observe a gap of 2.5 dB for a BER of 10-5• But we improve the SNR by 1.5 dB with 4 interleaved blocks and by 2 dB with 8 interleaved blocks. By increasing the number of blocks, we would get closer to the optimal curve. Our results confirm Stefanov and Duman's conclusions [START_REF] Stefanov | Turbo Coded Modulation for Systems with Transmit and Receive Antenna Diversity[END_REF] : they announced a gain of up to 6 dB with a one half rate Convolutional Turbo Code over the 16 state space-time treillis code [START_REF] Tarokh | Space Time codes for high data rate wireless communication : performance criterion and code construction[END_REF] at a BER of 10-5, with a QPSK (For an interleaving over 10400 bits, they achieve a BER of 10-5 at a SNR of 5 dB). Let us recall that our code rate is nearly 2/3 and we use only 4 iterations (instead of 10),which accounts for the 2 dB difference with their results at a BER of 10-5, while our spectral efficiency is higher ( 3 bits/s/Hz instead of 2 bits/s/Hz).

B.

16-QAM modulation

The spectral efficiency is greater than 5 bits/s/Hz. We have used a Gray mapping (see Fig. 5). If we note abcd the qua- Time systems perform better than space-time treillis codes.

C. Petformance of simplified receiver

In this part we compare previous results with performance obtained when using the approximation given by Eq.( 5). When looking at Fig. 7, one can notice no significant degradation be tween performance with the exact computation and those with its approximation. However this approximation bears a lower computational complexity .

V. CONCLUSION AND PERSPECTIVES

In [START_REF] Stefanov | Turbo Coded Modulation for Systems with Transmit and Receive Antenna Diversity[END_REF], the authors showed that Convolutional Turbo Codes performed much better than Space-Time Trellis Codes. Our purpose in this paper was to extend this study to Block Turbo Codes. We showed that BTC with multiple transmitters and multiple receivers perform as well as CTC while achieving higher spectral efficiencies within less iterations.

The main weak point of the proposed solution is the LLR com putation before SISO decoding. Indeed it requires (2q)" metric computations for each bit, which brings a (2q)n order complex ity. This complexity becomes prohibitive as q or n increases.

Therefore future work should consider lower complexity solu tions even at the expense of some performance degradation. A simplified solution has been investigated and its performance are equivalent to the optimal solution ones. One interesting feature of Transmit and Receive diversity is the fact that we can use PSK modulations (constant amplitude) with multiple trans mitters so as to reach the same spectral efficiency as systems based on QAM modulations. For instance, systems using 16-QAM with n transmitters could be replaced by 4-QPSK with 2 x n transmitters. Advantages of such a solution are numer ous. First, transmitted power over each transmitter is halved and two half power transmitters cost less than a full power one.

Furthermore, QAMs are very sensitive to AM-AM and AM PM distorsions. They require linear amplification. So as to en sure linear operation, a significant back-off on the output power is to be taken into account, increasing the power requirements.

On the other hand QPSK are more robust to non-linearity ef fects as it is a constant amplitude modulation.

  I

  .

Figure 2 :

 2 Figure 2: Construction of product code P .

Figure 3 :

 3 Figure 3: Interleaving.

P

  r( cf.) = mi11 ll r -hell 2

Figure 4 :

 4 Figure 4: QPSK, T/ = 2.66 bits/s/Hz, two transmit and two receive antennas.

Figure 5 :

 5 Figure 5: Mapping for the 16-QAM.

8 nFigure 6 : 5 SNRFigure 7 :

 8657 Figure 6: 16-QAM, '1 � 5.33 bits/s/Hz,two transmit and two receive antennas.