Alexandre Pasquiou
email: pasquiou<alexandre.pasquiou@inria.fr>.

Yair Lakretz

John Hale

Bertrand Thirion

Christophe Pallier

Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps

d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps Alexandre Pasquiou * 1 2 Yair Lakretz * 1 John Hale 3 Bertrand Thirion 2 Christophe Pallier 1

Abstract

Neural Language Models (NLMs) have made tremendous advances during the last years, achieving impressive performance on various linguistic tasks. Capitalizing on this, studies in neuroscience have started to use NLMs to study neural activity in the human brain during language processing. However, many questions remain unanswered regarding which factors determine the ability of a neural language model to capture brain activity (aka its 'brain score'). Here, we make first steps in this direction and examine the impact of test loss, training corpus and model architecture (comparing GloVe, LSTM, GPT-2 and BERT), on the prediction of functional Magnetic Resonance Imaging timecourses of participants listening to an audiobook. We find that (1) untrained versions of each model already explain significant amount of signal in the brain by capturing similarity in brain responses across identical words, with the untrained LSTM outperforming the transformerbased models, being less impacted by the effect of context; (2) that training NLP models improves brain scores in the same brain regions irrespective of the model's architecture; (3) that Perplexity (test loss) is not a good predictor of brain score;

(4) that training data have a strong influence on the outcome and, notably, that off-the-shelf models may lack statistical power to detect brain activations. Overall, we outline the impact of modeltraining choices, and suggest good practices for future studies aiming at explaining the human language system using neural language models.

Introduction

In the last few years, Transformer-based language models have revolutionized the field of natural language processing in virtually all areas. Although these models were developed for applications in language technology, their impressive success has raised interest in whether these models could also shed light on language processing in the human brain. Promising results in this direction suggest that brain activations and transformer-based models converge to similar linguistic representations [START_REF] Caucheteux | Disentangling Syntax and Semantics in the Brain with Deep Networks[END_REF] showing that brain activity can be significantly predicted from linear combinations of model activations, as was shown for fMRI (Toneva et al., 2020;[START_REF] Caucheteux | Disentangling Syntax and Semantics in the Brain with Deep Networks[END_REF]a), MEG (Caucheteux & King, 2021), and intracranial data [START_REF] Goldstein | Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines[END_REF].

However, several differences between Transformer-based models and the human brain raise questions about how far we can advance our understanding of brain function using these models. First, the architecture of Transformers is based on multi-head self-attention modules, which does not clearly map on neural computations in biological networks (e.g., [START_REF] Dayan | Theoretical neuroscience: computational and mathematical modeling of neural systems[END_REF]. Does this architecture contribute to or hinder the ability of the model to predict brain activity compared to other, possibly more brain-like, architectures (e.g., recurrent neural networks)? Second, the data used to train Transformer-based models is often different from that available for children, both in type and size. Training a Transformer-based model requires massive corpora, on the order of billions of words, whereas children require orders of magnitudes less words to achieve comparable or better linguistic performance. How does the training corpus (type and size) affect the model's ability to fit brain activity? Finally, the learning and evaluation objective commonly used with these models, such as masked or next-word prediction, is at most a rough approximation of the computational problem the human brain solves during language acquisition and processing. Can one consider that a well-trained model (according to perplexity loss) is a good model for brain activity in language tasks?

We investigate these questions by contrasting several types of language models in their ability to fit functional Magnetic Resonance Imaging (fMRI) timecourses of participants lis-tening to the 'The Little Prince' audiobook. Importantly, we conduct the model comparison while controlling for various aspects of the architecture of the models and the type and size of the corpus on which they are trained. To address the first question about the architecture of the models, we study the ability of untrained models to fit brain activity. We obtain significant differences across architectures, with that of recurrent neural networks achieving highest scores.

Next, we study brain-score gains brought by training across models, and find a network of brain regions, in which brain activity is consistently better fitted by various types of models. Moreover, running a comprehensive comparison of neural language models, we find that the effect of training is stronger in the case of Transformer-based models. We next question the relationship between perplexity and brain score, and study it across models and across training epochs during convergence. In contrast to previous studies, we find that perplexity is not a reliable predictor of model's brain score. Finally, we show the impact of training data on the model ability to fit brain data, notably, that off-the-shelf models, such as ones trained only on Wikipedia, may lack statistical power to fit brain activation.

Taken together, we conclude that while the starting point of Transformer-based models is less advantageous compared to that of recurrent neural networks, due to differences in their architectures, training leads to them outperforming all other models in predicting brain data.

Related Literature

Current knowledge about the cerebral basis of language mostly comes from brain imaging studies that have used tightly controlled stimuli, typically isolated words or sentences out of context (see [START_REF] Price | A review and synthesis of the first 20years of PET and fMRI studies of heard speech, spoken language and reading[END_REF][START_REF] Hickok | Neurobiology of language[END_REF], for reviews). As conclusions from such studies may be bounded to the peculiarity of the task and setup used in the experiment [START_REF] Varoquaux | Predictive models avoid excessive reductionism in cognitive neuroimaging[END_REF], researchers have become more and more interested in data using "Ecological Paradigms", in which participants are engaged in more natural tasks, such as conversation or story listening (e.g. [START_REF] Regev | Selective and Invariant Neural Responses to Spoken and Written Narratives[END_REF][START_REF] Lerner | Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story[END_REF][START_REF] Wehbe | Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses[END_REF].

Ecological paradigms commonly require methodologies of machine learning based on predictive modeling, to account for the high number of degrees of freedom in the complex system that is the brain. It has been shown that representations extracted from computational models can explain part of the signal acquired in brain neuroimaging. This was shown in early studies by using non-contextualized semantic representations [START_REF] Mitchell | Predicting human brain activity associated with the meanings of nouns[END_REF][START_REF] Huth | Natural speech reveals the semantic maps that tile human cerebral cortex[END_REF], moving in later studies to recurrent neural networks to extract context-based word representations (Jain & Huth, 2018;[START_REF] Jain | Interpretable multi-timescale models for predicting fmri responses to continuous natural speech[END_REF], and more recently to Transformer-based language models (e.g., Toneva et al., 2020;[START_REF] Caucheteux | Disentangling Syntax and Semantics in the Brain with Deep Networks[END_REF]a;[START_REF] Goldstein | Thinking ahead: spontaneous prediction in context as a keystone of language in humans and machines[END_REF] -see [START_REF] Hale | Neurocomputational models of language processing[END_REF] for a review.

Interestingly, the architecture of neural language models has been shown to substantially contribute to the ability of the model to fit brain data. Untrained neural language models fitted human brain activity surprisingly well [START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF]. Training was shown to improve brain scores by around 50% on average, across different architectures. This was suggested as evidence that the human cortex might already provide a sufficiently rich structure for relatively rapid language acquisition. However, conclusions in [START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF] were based on relatively small datasets, from no more than 9 participants. Also, different models in the comparison had different number of units, layers, and were trained on different datasets with varying vocabulary sizes.

In our work, we suggest a more controlled study of the effect of architecture and training on brain score, comparing different types of models, while controlling for the aforementioned factors, using a larger brain-imaging dataset, from 51 participants.

The performance of neural language models on a nextword prediction task, but not on other linguistic tasks, was shown to correlate with their ability to fit human brain data [START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF]. This was suggested as evidence that predictive processing shapes language-comprehension mechanisms in the brain. Here, we question this conclusion and study the relation between next-word prediction and brain score in various types of models, training corpora and training steps.

Analysis Setting: Fitting Brain Data with Modern NLP Models

Investigating the ability of neural language model to capture brain activity, we (1) first defined the three families of model architectures that we explored: non-contextualized word embeddings (GloVe; [START_REF] Eckert-Boulet | The Little Prince/Le Petit Prince[END_REF] to both human participants and artificial neural language models. Their activation in response to each word and punctuation sign of The Little Prince was extracted and (4) used to fit participants fMRI brain activations thanks to regularized linear encoding models. (5) Finally, at the end of the analysis pipeline, we had for each model: a test loss evaluated on our test set and a volumic R maps containing, for each brain voxel, the cross-validated correlation between the encoding model prediction and the observed fMRI response.

Datasets

Brain-imaging data. The brain data consisted of functional Magnetic Resonance Imaging (fMRI) scans from 51 participants who listened to the entire audiobook of The Little Prince1 . For each participant, there were 9 runs of fMRI acquisition lasting about 10 minutes. Whole brain volumes were acquired every 2 seconds. A global brain mask was computed to only keep voxels containing useful signal across all runs for at least 50% of all participants (26,164 voxels). Finally, linear detrending and standardization (mean removal and scaling to unit variance) were applied to each voxel's time-series. The analysis pipeline relies on Nilearn (v.0.8.1) for data access and visualization.

Encoding and subsequent statistical analyses were run with custom Python code using sklearn.

The acoustic onsets and offsets of the 15,435 spoken words were marked to align the audio recording with the text of The Little Prince. In addition to the words, the token streams fed to the neural language models included punctuation signs (commas, dots, ...).

Text Corpora. We designed several datasets on which we trained and evaluated our models. In total, we created 6 training datasets from Wikipedia (425M) and Project Guten-

Pipeline

Models. Given common training, validation and test datasets, we trained several instances of GloVe, LSTM, GPT-2 and BERT.

• GloVe was trained using the open-source code made available by Pennington and al. 3 ,

• GPT-2 and LSTM were trained on a Language Modeling task,

• while BERT was trained on a Masked-Language Modeling task with a 15% masking-rate.

Each model had a vocabulary of 50,001 tokens. GloVe and LSTM were trained for 23 epochs, while GPT-2 and BERT were trained during 5 epochs. Convergence assessment and comparisons with off-the-shelf models are provided in Supplementary Material. For computational cost reasons, we limited our analysis to 1, 2 and 4-layers models. In the following, we denoted MODEL.X a MODEL with X-layers.

Activation generation. (See Fig. 3.2) We presented the transcription of the audio book used to acquired fMRI brain data (The Little Prince, TLP; 15.435 words) to both trained and untrained versions of the selected artificial neural language models.

For each model, we extracted the model hidden-states while it processed its input, and defined from it what we call the activation matrix (one for each run). More precisely, if we note d the dimensionality of a neural model, which corresponds to the total number of units in the model, and w the total number of words in the text. We obtained an activation matrix A ∈ R w×d after the presentation of the entire text to the model. This means that each word of TLP is represented by a d-dimensional vector. Then, model activations were transformed into time-series matched to the fMRI acquisition times, using the following procedure:

1. Normalization: To match the scale of activations across layers (for multi-layers models), the activations of each layer were normalized by dividing them by the average L2-norm over words.

2. Convolution: each column of the resulting activation matrix, mapped onto words' offsets times, was convolved with SPM's canonical Haemodynamic Response Function(HRF; Friston 2007) sampled at 0.2s. The outcome was resampled at 2s to match the repetition time of fMRI acquisition and then mean-centered.

Fitting brain data. The latter stage resulted for each model and each run into a design-matrix of size #scans × d. Given a neural language model, we gave the associated nine designmatrices to a nested cross-validated L2-regularized univariate linear encoding model to fit the fMRI brain data (of size #scans × #voxels).

The encoding model is a function that maps a vector of stimulus features onto brain responses activity [START_REF] Naselaris | Encoding and decoding in fMRI[END_REF]. We denote by x t the vector of features at time t, such as predicted time-courses derived from a language model, and by y v t the corresponding brain responses measured at voxel v. We learnt a linear voxel-level encoding model using Ridge regression, whose general solution is given by:

βv Ridge = arg min β n t=1 (y v t -β T x t ) 2 + λ∥β∥ 2 2
To evaluate model performance and the optimal regularization parameter λ * , we used a nested cross-validation procedure: we split each participant's dataset into training, validation and test sets, such that the training set included 7 out of the 9 experiment runs, and the validation and test sets contained one of the two remaining sessions. We evaluated model performance using the Pearson correlation coefficient R, which is a measure of the linear correlation between models' predicted time-courses and the actual time-courses. It is defined as:

R(y, ŷ) v,test = (ŷ v t -ȳv )(y v t -ȳv ) (ŷ v t -ȳv ) 2 (y v t -ȳv ) 2
,

where ȳv = 1 T T t=1 ŷv t , ȳv = 1 T T t=1 y v t
For each subject and each voxel, we first determined λ * by comparing R valid for 10 different values of λ, linearly spaced in log-scale between 10 and 10 5 . We then calculated R test for λ * . Finally, we repeated this procedure 9 times, using cross-validation. This resulted in 9 R test values that we then averaged to produce a single R test map for the participant.

Results of the analysis pipeline. Finally, at the end of the analysis pipeline we had for each model: a test loss evaluated on our test set derived from the Gutenberg Project, and a volume-based R map displaying for each brain voxel the correlation between the encoding model prediction and the observed time series. Volume maps are rendered on cortical surfaces by projection.

Methods and Experimental Setup

4.1. Assessing model fitness to brain data Whole-brain, voxel-based, group analyses were performed, using one-sample t-tests applied to the individuals' R test maps spatially smoothed with an isotropic Gaussian kernel with 6mm FWHM. In each voxel, the test assessed whether the distribution of R test values across participants was significantly larger than zero. To control for multiple comparisons, all the maps displayed in this document were corrected using a Bonferroni correction of 0.1 [START_REF] Bonferroni | Teoria statistica delle classi e calcolo delle probabilita[END_REF], that is, values reported on the maps (e.g. R scores) are shown only for voxels that survived this threshold.

We derived in a model-agnostic manner from a Shared-Response Model (SRM, Chen et al. 2015) the most "responsive" voxels, that is, the voxels whose R values were the 25% highest ones. This set of 6541 voxels, which we will refer to as "SRM25" is displayed on a brain surface at the bottom of Fig. 2. It is used to compute the distributions of brain scores.

Comparison of untrained models and baselines

In our first analysis, we assessed whether the model class and number of layers bias its ability to fit fMRI brain data. We instantiated several untrained versions of each model class, varying the number of layers, and generated activation matrices from these models before fitting them to brain data.

For each model, the activation matrices were built using all the hidden-states of all layers, including the embedding layer. We also defined a Baseline model whose activation matrices are obtained by associating a fixed embedding vector of size 768 (size of each model's layer) to each word of the text. It is equivalent to an untrained GloVe model (and will be referred to as such). For each, we obtained 3D brain maps displaying the average R test values in each voxel. Finally we displayed boxplots of the R test values distributions in the previously SRM-defined voxel selection.

Comparison of trained and untrained neural language models

We then investigated how training models impacts their ability to fit fMRI brain data. We first generated activation matrices from the trained language models before fitting them to brain data and finally displaying the group-level difference between each model's map and its untrained ver- sion's map. To study the overlap between the different contrast maps, we extracted for LSTM, GPT-2 and BERT the set of voxels whose R values were the 10% highest.

Then we computed the intersection of these three sets and the percentage of overlap for pairs of models and for the three models. We additionally identified the set of voxels whose R values were the 10% highest in the untrained models maps, and computed the intersection of the three sets and the percentage of overlap for the three models. Finally, we studied the intersection between the 2 overlaps of the three models and synthesized differences and similarities in Fig. 4.

Mapping the brain scores of NLP models

The next step was to run a comprehensive comparison of the selected models to understand models similarities and specificities. We contrasted individual maps between pairs of models to test in each voxel: (1) the effect of incorporating context into target-word representations, by contrasting LSTM and GloVe;

(2) the effect of attention vs. recurrence mechanisms on prediction, by contrasting the maps of transformers and LSTM.

(3) the interaction between model architecture and training between transformers and LSTM.

(4) and finally the effect of bi-directionality vs. incrementality (BERT vs. GPT-2). Compared models always have the same number of hidden-states.

Perplexity and Brain score

Finally, we investigated the relation between perplexity and brain score. Using the set of trained LSTM, GPT-2 and BERT models, we evaluated them using the standard loss, that is, the average logarithm of model perplexity, on the test set. For each model, we also computed the brain score, defined as the average R-value within the SRM25 voxelset.

We also investigated the importance of the training data on the model ability to fit brain data, comparing models trained on Wikipedia and on the Full dataset (Wikipedia + Gutenberg xlarge). These analyses were performed on LSTM and GloVe.

Results

Figure 2 shows the distributions of R test -values across voxels for trained and untrained models of various architectures and number of layers.

Performance of untrained models

Remarkably, all untrained models, regardless of their architecture, explain signal better than chance. Untrained LSTM and untrained GloVe (that is, Random Embeddings) perform equally well with an average score around 6.3% (SE=0.02%), and significantly better than Transformers as attested by direct comparisons between 4 layers models: LSTM.4-GPT-2.4 (1.6% SE=0.02%); LSTM.4-BERT.4 (0.7% SE=0.004%). Overall, untrained GPT-2.4 had the worst performance (BERT.4-GPT-2.4 (0.9% SE=0.01%)).

The brain regions where LSTM untrained performs significantly better than GPT-2 are displayed on Fig. 3. They are located within the left hemispheric language network and its right counterpart (Superior Temporal Gyrus/Superior Temporal Sulcus and Inferior Frontal Gyrus (pars opercularis)).

Effect of number of layers

Next, we looked at the effect of number of layers for LSTM, GPT-2 and BERT models.

As can be seen on Fig. 2 the change in performance for untrained models is either flat (for LSTM and BERT) or negative (for GPT-2). Comparing 4-layer models to 1-layer models yields the following: LSTM (-0.02% SE=0.002%); GPT-2 (-0.6% SE=0.004%), BERT (-0.02% SE=0.003%).

For trained models, performance improves with the number of layers. The increase in performance (4-layer model's performance -1-layer model's performance) is more marked for Transformers -GPT-2 (2% SE=0.006%) and BERT (1% SE=0.006%) -than for LSTM (0.4% SE=0.005%).

Effect of Training

Visual inspection of Fig. 2 shows, unsurprisingly, that training helps: trained model fit brain data better than models initialized with random weights. To quantify this improvement for each model type, we computed, in each voxel, the difference in R test between the trained model and the untrained model. All differences were statistically significant: GloVe (1.5% SE=0.02%); LSTM (3.1% SE=0.02%) ; GPT-2 (4.5% SE=0.02%); BERT (4.4% SE=0.02%); in Student T-tests, all ps < 10 -16 ). Fig. 4A shows the distributions of these training effects.

The maps on Fig. 4B show the locations of voxels where the R test increases are significant. The effect of training is spatially consistent across models, that is, displays similar topographies across models; and the R-score improvements are comparable in high-order language networks across models.

To assess the similarity between the hotspots on these maps, we thresholded them, keeping the 10% of voxels (2617 voxels) showing the highest gains with training. We then computed the percentage of overlap across the resulting binarized maps. Results are presented in Table 1. The overlap between the three maps is 75% across all 3 models. We -0.06 0 0.06 then thresholded the untrained models brain maps, keeping again the 10% (2617 voxels) of voxels showing the highest r score, and computed the percentage of overlap across the resulting binarized maps. Results are presented in Appendix Table 1. The overlap between the three maps is 79% across all 3 models. The differences and similarities of these two overlaps are displayed in Fig. 4C.

Comparisons between models

Then, we ran a comprehensive comparison of the models described in Section 3. Firstly, we contrasted a model that takes context into account (LSTM) to a model that does not (Glove). Importantly, to conduct a fair comparison, both models were trained on exactly the same corpus, and had vocabulary of same size (see Section 3). The contrast map displayed in Fig. 5A highlights a set of regions located in the temporo-parietal junction, similar to the trained models overlap of Fig. 4 (in green), with bilateral effects in the medial and lateral Superior Frontal Gyri and Posterior Cyngulate gyri, as well as left-lateralized effects in the Temporal Pole, the Inferior Frontal and Middle Temporal gyri.

Note that this map is quite similar to the green network showing an effect of training across models (Fig. 4C). This suggests the role of these regions in context processing, given that in both cases, the model benefits from using context for predicting activity in these regions.

Next, we compared a model using attention (BERT) to a recurrence-based model (LSTM). Fig. 5B1) shows that BERT outperforms LSTM mostly in the Superior Temporal Gyri, and in the auditory cortex.

Comparing the bidirectional BERT with the incremental GPT-2 in Fig. 5 B2) shows that BERT outperforms GPT-2 in the entire language network. Finally, to test which model architecture best benefit from training, we investigated the interaction between model architecture (BERT vs. LSTM) and training (trained vs. untrained). The interaction map is shown on Fig. 5C. The effects are more spread than the direct difference between the trained BERT and LSTM, as expected from the fact that LSTM untrained has higher performance than BERT untrained. Transformer-based models gain more with training relative to LSTM (this is also the case for GPT2, see Appendix Fig. S6C), and explorations show that this relative gain increases with the number of

Model GPT-2 BERT LSTM 79% 86% GPT-2 . 85%
Table 1. Overlap between training effect brain maps The percentage of common voxels when the maps were thresholded at their 10% upper percentile.

layers.

In summary, i) LSTM model outperforms the noncontextual model GloVe in core regions of the language network; ii) Transformer-based models benefit more from training than the LSTM model, and achieve higher brain scores (compared to LSTM) mostly around the auditory cortex.

Relationship between Perplexity and Brain score

Fig. 6A shows the relationship between perplexities (model loss) and brain scores derived from various models, architectures, training sets and training stages. Unlike previous reports [START_REF] Schrimpf | The neural architecture of language: Integrative modeling converges on predictive processing[END_REF], we did not observe a clear monotonic relationship between the two variables. For example, the average LSTMs perplexity is worse than that of GPT-2, but the average brain score is higher.

We

Discussion

Previous work has shown that brain activity during visual or language processing can be significantly predicted from artificial neural-network activations. From a neuro-scientific point of view, the interest lies in the possibility to study language processing in the brain by manipulating the information provided to an artificial model and then to assess the impact on the model's ability to fit brain data. In the present research, we examined the impact of several factors on the fitting performance of artificial models. We studied the impact of model architecture (assessing GloVe and One discovery is that all architectures are not equal, but training them consistently increases brain scores in the same set of brain areas. Moreover, these very brain regions are also better predicted by models that take the context into account, such as LSTMs, compared to static models, represented here by GloVe. This provides converging evidence that these regions perform context-dependent computations.

From a neuratomical point of view, they are located on the border of the core language regions (IFG and STS) and partly overlap with regions assigned to the Default mode network (Angular Gyri, Dorso mesial prefrontal cortex). These results are coherent with previous work investigating the processing of contextual information by either using LSTM models (Jain & Huth, 2018) or by scrambling the stimuli at different levels [START_REF] Lerner | Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated Story[END_REF], confirming that this network (in green/blue in Fig. 4) is at the center of combinatorial language processing in the human brain.

We observed that even if transformers start with a disadvantage regarding the ability to fit fMRI brain data, they benefit more from training than LSTMs, and they are able to take advantage of stacks of layers to improve their fitting performance. The comparison of untrained LSTM and untrained GloVe (i.e., random embeddings) showed no significant differences, whereas the comparison of untrained GloVe and untrained transformers showed significant R-score differences in some regions. The difference between untrained LSTMs and untrained Transformers might be due to their different architectures. However, there might be an alternative explanation. Note that for untrained GloVe (random embeddings), each word in the corpus is assigned a fixed vector, whereas for untrained Transformers, each word is mapped to a variable vector, depending on the context that surrounds the word. Therefore, untrained Glove might better predict brain responses to words that occur frequently both in the training and in the test data (e.g., function words).

In contrast, untrained transformers might generate very different embeddings to the same word (e.g., 'the' in the train and test sets), due to their context sensitivity. This variability could therefore reduce the brain score of untrained Transformers compared to that of untrained GloVe. Finally, our results suggest that untrained LSTM are more similar to untrained Glove, having less context sensitivity compared to Transformers. Taken together, this suggests that most of what the untrained baselines capture is similarity in brain responses to words that appear in both the train and test sets.

The discrepancy between brain score and perplexity indicates that training is not a guarantee of convergence towards brain-like representations (see also [START_REF] Hale | Text genre and training data size in human-like parsing[END_REF]. Relatedly, other research also seems to militate against perplexity as a royal road to cognitive models (see, e.g., Clark 2000 , chapter 11, 2 nd edition).

A final methodological word of caution stem from our results: data used for training have a strong influence on the outcome, showing that off-the-shelf models trained on small datasets like Wikipedia lack statistical power to capture brain activations and should be avoided to probe brain representations.

C. Improvement in

  berg 2 , using up to 2 thirds of the entire Project Gutenberg in the xlarge version and splitting the remaining 1/3 left into equal size validation (1.1G) and test sets (1.1G). The datasets created from the Gutenberg Project's data are nested (small(240M ) ⊂ medium(737M ) ⊂ large(2.2G) ⊂ xlarge(4.4G) ⊂ f ull(4.8G; xlarge + W ikipedia)).

Figure 1 .

 1 Figure 1. Overview of the pipeline: (A) Human participants were presented with an auditory version of The Little Prince story while their brain activity was recorded with fMRI. Neural models were presented with a text transcription and the entire state of the network was recorded for each word and punctuation sign. Pre-processing steps were applied to both brain and model activations before aligning the two signals using a nested cross-validated Ridge-regression model. Finally, brain maps of correlation coefficients between models' predictions and fMRI time-series were computed. (B) In the case of models with several layers, model activations were extracted from each layer and were concatenated into a single activation matrix.

Figure 3 .Figure 4 .

 34 Figure 3. LSTM vs. GPT-2 architecture: Brain regions in which an untrained LSTM outperforms an untrained GPT-2 model. The comparison is for 2-layer models.

Figure 5 .

 5 Figure 5. Localisation of differences between models A) Comparison between a contextual model (LSTM) and a non-contextual model (GloVe). LSTM outperforms the non-contextual model GloVe in core regions of the language network, with significant effects that are bilateral and included in the core of the previously highlighted network of regions that are better fitted with training 4C. B1) Comparison of a transformer-based model (BERT) with a recurrent neural network (LSTM). Trained BERT better fits fMRI brain data than LSTM model around Heschl's Gyri bilaterally. B2) Comparison of a bidirectional transformer-based model (BERT) with an incremental transformer-based model (GPT-2). Trained BERT better fits fMRI brain data than GPT-2 in the entire language network. C) Interaction between model architecture (BERT vs. LSTM) and Training (trained vs. untrained) for 1-layer (top), 2layer (middle) and 4-layer (bottom) models. BERT benefits more from training than LSTM. The more layers, the more it learns.

Figure 7 .

 7 Figure 7. Influence of Training dataset on Rtest. LSTM and GloVe better fit brain data when learning on more training data. This shows the dependence of models contrast on training data. Our full dataset comprised Gutenberg + Wikipedia (4.8 GB) while Wikipedia represented 425 MB.

Figure 12 .

 12 Figure 11. A consistent increase in R scores due to training across various types of neural language models. Contrasts between R scores of trained vs. untrained models. Contrast maps are shown for 2-layer models (panel A) and 4-layer models (panel B). In each panel, from top to bottom: LSTM, GPT-2, BERT.

  Perplexity does not predict Brain score. Among all instances of LSTM, GPT-2 and BERT that are plotted, we found no monotonic relationships, showing that Perplexity cannot serve as a simple proxy to determine Brain Score, as the relation between the two is impacted by the model class, its architecture and training. We represented the brain scores and perplexities of BERT (blue), of several instances of GTP-2 at different training stages (green) and of various instances of LSTM trained on different datasets (red).

	0.1	LSTM GPT-2 BERT	1 layer model 2 layers model 4 layers model
	0.09 0.095 Brain Score
	0.085
	0.08	Perplexity 10 15 20 25 30 35 40 45
	Figure 6.

Available fromhttps://openneuro.org/ datasets/ds003643/versions/1.0.2

Project Gutenberg. (n.d.). Retrieved February 21, 2016, from www.gutenberg.org.

https://nlp.stanford.edu/projects/glove/.

Acknowledgements

This project/research has received funding from the American National Science Foundation under Grant Number 1607441 (USA), the French National Research Agency (ANR) under grant ANR-14-CERA-0001, the European Union's Horizon 2020 Framework Programme for Research and Innovation under the Specific Grant Agreement No. 945539 (Human Brain Project SGA3), and the KARAIB AI chair (ANR-20-CHIA-0025-01).

A. Convergence of the Language Models During Training

To assess the performance of our trained models, we compared their ability to predict brain data with that of off-the-shelf models ( https://github.com/google-research/bert).

The 2 and 4-layers BERT models either significantly outperform the baseline or are on a par. The 12-layers baseline, which is 3-times bigger than the 4-layers model, outperforms the latter in core regions of the language network, but only to a small extent.

B. Evaluation of Brain-Fit Performance of the Untrained Models

Model LSTM GPT-2 BERT LSTM . 81% 92% GPT-2 . . 86%

Table 2. Overlap between untrained brain maps The percentage of common voxels when the maps were thresholded at their 10% upper percentile. E. Relation Between Brain Score and Perplexity


Amanda Edmonds

Jon Clenton

Hosam Elmetaher

Exploring the Construct Validity of Tests Used to Assess L2 Productive Vocabulary Knowledge

Keywords: vocabulary, productive, receptive, exploratory factor analysis

Vocabulary knowledge in a second language (L2) is thought to be a complex construct and, accordingly, there exist numerous ways to evaluate a L2 learner's vocabulary knowledge.

These assessments are generally billed as tests of vocabulary size (i.e., number of words known) or depth (i.e., how well words are known) of either receptive or productive vocabulary knowledge. However, inconsistencies persist in how existing assessments are characterized, leading to sometimes contradictory claims over what these tests are measuring. This state of affairs leads to concerns with the construct validity of the tests in question. In the current study, we contribute to these ongoing discussions with a focus on tests that target productive vocabulary knowledge in L2 English. Four vocabulary tests (three productive, one receptive) were administered to a group of Francophone learners of English, and results were analyzed using an exploratory factor analysis. This analytic approach led to the identification of two underlying constructs, which we labeled receptive vocabulary knowledge and productive vocabulary knowledge, respectively. These results highlight the importance of the crucial distinction between receptive and productive knowledge in the conceptualization of the overall construct of vocabulary knowledge.

Introduction

Within the field of second language acquisition (SLA), vocabulary knowledge has received sustained theoretical and empirical attention over the past three decades (e.g., [START_REF] Yanagisawa | Measuring depth of vocabulary knowledge[END_REF], with second language (L2) vocabulary researchers consistently highlighting the complex nature of vocabulary knowledge. In particular, two main conceptual distinctions cut across and structure relevant research. First, vocabulary knowledge is commonly conceived of either in terms of the number of words that a learner knows (i.e., their vocabulary size), or in terms of how well words are known (i.e., vocabulary depth). A second important distinction contrasts receptive (or passive) knowledge of words with productive (or active) knowledge.

Despite continuing debates over their operationalization, the concepts of vocabulary size versus depth and receptive versus productive knowledge have proven useful in L2 vocabulary research, particularly in assessments of vocabulary knowledge. Indeed, taking account of these two concepts suggests that vocabulary knowledge may rely on four separate constructsreceptive vocabulary size, receptive vocabulary depth, productive vocabulary size, and productive vocabulary depth -, each of which may presumably be assessed using specific tests. However, the potential for a clean mapping of assessment measures onto these four potential constructs is belied by the characterizations and actual uses of many tests. Two examples are illustrative: (i) although there is agreement that Lex30 [START_REF] Meara | Lex30: An improved method of assessing productive vocabulary in an L2[END_REF] elicits productive vocabulary knowledge, disagreements arise as to whether it provides a measure of depth of vocabulary knowledge (see Read, 2004, p. 220) or vocabulary size (i.e., [START_REF] Williams | Estimating second language productive vocabulary size: A capture-recapture approach[END_REF]; and, (ii) although the Productive Vocabulary Levels Test (PVLT, [START_REF] Laufer | A vocabulary -size test of controlled productive ability[END_REF] was designed to reveal the size of an individual's productive vocabulary, given that participants must complete words presented in carrier sentences, Webb (2005, p. 82) suggests that it might "actually test receptive knowledge."

These examples highlight ongoing discussions about the construct validity of vocabulary tests, namely discussions of whether they measure what they purport to. Accordingly, in the current study, we contribute to these discussions with a project whose point of departure was tests that target productive vocabulary knowledge in L2 English. This focus is justified by the relative paucity of research on the assessment of productive (versus receptive) vocabulary knowledge. We thus administered four vocabulary tests (three productive and one receptive) to a group of 100 Francophone learners of L2 English. While previous research (see, e.g., [START_REF] Clenton | Investigating the construct of productive vocabulary knowledge with Lex30[END_REF][START_REF] Fitzpatrick | Productive vocabulary tests and the search for concurrent validity[END_REF][START_REF] Fitzpatrick | Making sense of learner performance on tests of productive vocabulary knowledge[END_REF][START_REF] Walters | Aspects of validity of a test of productive vocabulary: Lex30[END_REF] has reflected on the construct validity of multiple vocabulary tests using correlational analyses, we opt for a different analytic approach. More specifically, we analyzed our results using exploratory factor analysis, an approach that allows the researcher to explore complex correlational patterns in parallel. This allowed us to uncover two constructs that underpin the performances on the four tests, thus offering novel insights into the ongoing debates over the construct validity of different L2 vocabulary tests.

Background

Conceptualizing Vocabulary Knowledge in an L2

Two distinctions are central in conceptualizing vocabulary knowledge in a L2: vocabulary size/depth and receptive/productive knowledge. Vocabulary size refers to the number of words 1 that an individual knows. In most research, knowing a word has been synonymous with demonstrating receptive knowledge about its form-meaning mapping, and it has been assumed that more frequent words are learned earlier and better than less frequent ones (see Milton, 2007, for discussion). As a result, most assessments of vocabulary size target a sample of words from various frequency bands and extrapolate size scores from performance on the sample. Vocabulary depth, on the other hand, is a more unruly concept to define, with numerous approaches having been proposed (see [START_REF] Read | Plumbing the depths: How should the construct of vocabulary knowledge be defined[END_REF][START_REF] Schmitt | Size and depth of vocabulary knowledge: What the research shows[END_REF][START_REF] Yanagisawa | Measuring depth of vocabulary knowledge[END_REF]. In simple terms, vocabulary depth refers to how well one knows a word; however, the operationalization of this concept often proves challenging. One common way of conceiving of vocabulary depth has come to be known as the components (or dimensions) approach. Although this approach to vocabulary depth can take many forms, they all agree that vocabulary knowledge covers a variety of components. [START_REF] Nation | Learning vocabulary in another language[END_REF] list of nine components of vocabulary knowledge is a well-known example of such an approach:

(1)

Component 1: spoken form In a components approach, knowledge with respect to more dimensions is considered to reflect greater depth of knowledge. A second way to conceive of vocabulary depth is what [START_REF] Read | Plumbing the depths: How should the construct of vocabulary knowledge be defined[END_REF] referred to as network knowledge. In this case, greater vocabulary depth is equated with words boasting richer associative networks. More elaborated connective networks are interpreted as reflecting greater lexical organization [START_REF] Meara | V_LINKS: Beyond vocabulary depth[END_REF].

Considering vocabulary size and depth together, numerous studies have reported evidence of a strong positive relationship between the two (see [START_REF] Schmitt | Size and depth of vocabulary knowledge: What the research shows[END_REF], for an overview). This is not particularly surprising, given that "size by definition is the number of lexical items known to some criterion level of mastery. But the criterion will always be some measure of depth" (Schmitt, 2014, p. 942). Schmitt goes further, stating that "there can be no clear distinction between size and depth." Following this reasoning, it is unsurprising that inconsistencies may exist concerning whether certain tests tap vocabulary size or depth.

The second central conceptual distinction in L2 vocabulary research concerns receptive versus productive knowledge. Generally speaking, receptive abilities correspond to the recognition and understanding of words, whereas productive abilities include the capacity to recall or produce a word. If this distinction is generally accepted by vocabulary researchers, there continue to be discussions about the underlying nature of the distinction. On the one hand, [START_REF] Meara | Towards a new approach to modelling vocabulary acquisition[END_REF] discusses the possibility of receptive and productive abilities being qualitatively different. This qualitative difference depends, according to Meara, on the associative links attached to a word in the mental lexicon. Words for which a speaker has receptive knowledge have fewer links and need external stimuli to be activated. Words for which one has productive knowledge can be activated within the mental lexicon (independent of external stimuli), by dint of their richer associative links. Other researchers conceive of receptive-productive (or passive-active) knowledge as a continuum, and this continuum has formed the foundation for many conceptualizations of vocabulary knowledge in a L2 (e.g., [START_REF] Henriksen | Three dimensions of vocabulary development[END_REF]Laufer & Goldestein, 2004;[START_REF] Melka | Receptive versus productive aspects of vocabulary[END_REF][START_REF] Palmberg | Patterns of vocabulary cevelopment in foreign-language learners[END_REF][START_REF] Schmitt | Researching vocabulary: A vocabulary research manual[END_REF].

This view suggests that language users need to develop their receptive vocabulary knowledge to the extent that it becomes productive, and an actively used component of the learner lexicon. Although the exact threshold at which receptive knowledge becomes productive has not as yet been identified [START_REF] Laufer | Testing vocabulary knowledge: Size, strength, and computer adaptiveness[END_REF][START_REF] Schmitt | Researching vocabulary: A vocabulary research manual[END_REF], Schmitt (2019, p. 264) suggests that "learning most words to receptive mastery is relatively easy; it is enhancing that knowledge to productive mastery which is the real challenge." Empirical results concerning the acquisition of receptive versus productive vocabulary knowledge indicate that "receptive vocabulary knowledge develops before and at a faster rate than productive vocabulary, is larger than productive vocabulary and, importantly, is easier and more straightforward to measure or quantify than its productive counterpart" (Williams et al., 2014, p. 24).

Concerning Williams et al.'s observation that receptive knowledge "is easier and more straightforward to measure or quantify than its productive counterpart", there indeed currently exist more assessments of receptive knowledge than there do of productive knowledge (see Miralpeix, 2020, p. 192).

Assessing Productive Vocabulary Knowledge

A small number of tools have been developed in order to assess productive vocabulary knowledge in a L2. For the current project, we focused on three such measures: (i) Lex30 [START_REF] Meara | Lex30: An improved method of assessing productive vocabulary in an L2[END_REF] based on word association, using single word cues, (ii) G_Lex [START_REF] Fitzpatrick | Making sense of learner performance on tests of productive vocabulary knowledge[END_REF] based on word association, using sentence cues, and (iii) the PVLT [START_REF] Laufer | A vocabulary -size test of controlled productive ability[END_REF], a sentence completion test. Example items for each of the tests, along with instructions, are provided in Table 1. 1. She loved to _________ over the phone.

2. When I feel sad I always go to the _________.

3. They think car-racing is _________.

PVLT

Complete the underlined words.

1. I am glad we had this opp_________ to talk.

2. There are a doz_________ eggs in the basket.

3. Every working person must pay income t_________.

On the basis of the example items, it is clear that each of these three measures takes a rather different approach to eliciting productive vocabulary knowledge. Beginning with Lex30, [START_REF] Meara | Lex30: An improved method of assessing productive vocabulary in an L2[END_REF] opted for a word association format. Respondents are asked to provide up to four words in response to 30 single-word stimuli (see Figure 1A [START_REF] Clenton | Investigating the construct of productive vocabulary knowledge with Lex30[END_REF][START_REF] Fitzpatrick | Exploring the validity of a test of productive vocabulary[END_REF][START_REF] Walters | Aspects of validity of a test of productive vocabulary: Lex30[END_REF].

G_Lex was devised by [START_REF] Fitzpatrick | Making sense of learner performance on tests of productive vocabulary knowledge[END_REF] as a point of comparison to investigate the construct underlying Lex30. G_Lex is a sentence completion test in which participants are asked to provide up to five words to complete each of 24 sentence gaps (an example is provided in the Appendix, Figure 1B). Like Lex30, no specific responses are targeted, and any infrequent word provided by a respondent (i.e., any word that is not within the first 1,000 most frequent words in English) receives one point. The main difference between G_Lex and Lex30 is that the former encourages test takers to consider context (both in terms of part of speech and appropriate meaning) in providing their responses. Unlike the other tests used in this study, G_Lex has not been as extensively tested, and thus our study serves to provide initial evidence of the construct validity of this relatively new test.

The PVLT [START_REF] Laufer | A vocabulary -size test of controlled productive ability[END_REF] -was designed to provide an estimate of productive vocabulary size using a word completion format (see Appendix, Figure 1C, for an example). On the PVLT, 18 test sentences for each of five frequency levels (2k, 3k, 5k, UWL

[the University Word List], and 10k) are presented for a total of 90 items, and in each sentence, participants are required to provide a target word (belonging to the frequency band in question). The first few letters of each target item are provided to restrict responses to a specific target word. A point is awarded for each correctly provided target item. The PVLT is a widely used measure of productive vocabulary knowledge (e.g., [START_REF] De Jong | Facets of speaking proficiency[END_REF][START_REF] Eguchi | Lexical competence underlying second language word association tasks: Examining the construct validity of response type and response time measures[END_REF], and has been the subject of validation tests [START_REF] Laufer | The development of passive and active vocabulary in a second language: Same or different?[END_REF][START_REF] Laufer | A vocabulary -size test of controlled productive ability[END_REF].

Each of these tests aims to assess productive vocabulary knowledge. However, whereas [START_REF] Laufer | A vocabulary -size test of controlled productive ability[END_REF] clearly state that the PVLT is intended as a test of productive vocabulary size, it is less clear whether the creators of Lex30 and especially G_Lex intended their tests as measures of size, depth, or perhaps as a general measure of productive vocabulary (for Lex30, see Meara & Fitzpatrick, 2000, p. 22 and 27). Looking beyond the size-depth distinction, authors such as [START_REF] Fitzpatrick | Productive vocabulary tests and the search for concurrent validity[END_REF]Meara (2004), Fitzpatrick (2007), and [START_REF] Clenton | Investigating the construct of productive vocabulary knowledge with Lex30[END_REF] have argued that existing productive vocabulary measures have the potential to mobilize -and, thus, may measure -quite different aspects of vocabulary knowledge. Using [START_REF] Nation | Learning vocabulary in another language[END_REF] nine aspects of vocabulary knowledge, these researchers argue that whereas all tests can tap into the productive form-meaning connection (which is generally how vocabulary size is operationalized), they vary considerably in the other types of vocabulary knowledge (i.e., other dimensions of vocabulary depth) that can be demonstrated. Fitzpatrick and Meara (2004, p. 72) go so far as to suggest that the assessments of productive vocabulary knowledge they explored "do not measure the same things at all; productive vocabulary is a misleadingly simplistic label for an extremely complex construct."

With respect to the three productive vocabulary measures included in the present study, a similar observation can be made. In Table 2, we build on work by Fitzpatrick (2007, p. 129) and Clenton (2010, p. 178) in order to provide an overview of the aspects of vocabulary knowledge that each measure may arguably tap. Although all three tests presumably allow participants to demonstrate productive knowledge of how a word is written and the productive form-meaning connection for a given word, only Lex30 and G_Lex allow learners to showcase productive knowledge of associations and only G_Lex and PVLT potentially tap into receptive collocational knowledge (because of the presence of carrier sentences). The PVLT arguably taps three additional vocabulary knowledge dimensions. Due to the constrained nature of this test and the necessity to be able to understand the carrier sentence in order to provide the targeted response, participants must demonstrate knowledge of grammatical functions and constraints on use. Finally, because the PVLT provides between 1 and 5 letters of the target word, receptive recognition of what the target words look like is also tested (see Read, 2020, pp. 549-550). The profile provided in Table 2 thus suggests that the PVLT may tap three aspects of receptive knowledge, echoing Webb's (2005, p. 82) concern about the construct validity of this test as an assessment of productive vocabulary knowledge. Given that the aim of the current project is to contribute to discussions on the construct validity of productive vocabulary assessments, the fact that PVLT may also be

Assessing Assessments of L2 Vocabulary Knowledge

Given the multi-faceted nature of vocabulary knowledge and the growing number of measures proposed to measure that knowledge, researchers have turned their attention to the important endeavor of exploring the reliability and validity of these different tests (for a given population and for a given context). There are many ways to contribute to this line of research, and in what follows, we briefly review the use of correlations and structural equation modeling (SEM). We end by explaining what exploratory factor analysis, the approach adopted in this study, has to offer.

One approach to determining whether a given test measures the knowledge or skill it was intended to measure is to compare results on the test in question with results obtained from another test that has been shown to measure the construct of interest. This is what is referred to as concurrent validity. According to Langsford et al. (2018, p. 5), "measure agreement is an important check of validity for diverse measures claiming to reflect the same underlying construct." This approach has been frequently adopted in L2 vocabulary measurement studies and has involved exploring the correlations obtained between the results from two vocabulary tests (Bachman, 1990, p. 248). Limiting ourselves to those studies having explored concurrent validity involving at least two of the tests included in the current study, several conclusions can be drawn. First, studies having assessed vocabulary knowledge using the VLT and PVLT report significant positive correlations between the scores [START_REF] Laufer | The relationship between passive and active vocabularies: Effects of language learning context[END_REF][START_REF] Nemati | Active and passive vocabulary knowledge: The effect of years of instruction[END_REF]. Second, investigations that have administered the PVLT and Lex30 to the same participants have also reported significantly positive correlations [START_REF] Elmetaher | Investigating productive vocabulary knowledge development: A taskbased approach[END_REF][START_REF] Fitzpatrick | Exploring the validity of a test of productive vocabulary[END_REF][START_REF] Walters | Aspects of validity of a test of productive vocabulary: Lex30[END_REF]. Third, in the two studies to explore the concurrent validity of Lex30 and G_Lex [START_REF] Elmetaher | Investigating productive vocabulary knowledge development: A taskbased approach[END_REF][START_REF] Fitzpatrick | Making sense of learner performance on tests of productive vocabulary knowledge[END_REF], a moderate positive correlation was reported. Taken together, these results provide evidence that the constructs measured by the different tests are highly related. These results are moreover consistent with the possibility that the tests tap the same underlying construct.

However, separate bivariate correlations cannot reveal more complex correlational patterns and, as such, may provide an overly simplistic perspective on latent constructs. Other types of analysis -namely, SEM and factor analysis -are designed to accommodate and identify such complex patterns.

SEM is designed to allow the researcher to test hypothesized relationships among sets of measured and latent variables. In other words, this type of analysis is confirmatory, insofar as researchers propose a model (or a set of models) that they then test against a given dataset (see [START_REF] Schoonen | Structural equation modeling in L2 research[END_REF], for an overview of SEM applied to SLA). In the realm of L2 vocabulary acquisition, a small number of researchers have availed themselves of SEM for various aims:

to offer a model of motivated vocabulary learning [START_REF] Tseng | Toward a model of motivated vocabulary learning: A structural equation modeling approach[END_REF], to model the impact of vocabulary and grammatical knowledge on reading comprehension [START_REF] Zhang | Vocabulary and grammatical knowledge in second language reading comprehension: A structural equation modeling study[END_REF], or -and of particular relevance to the present study -"to examine the nature of the overall SEM and other confirmatory analyses (e.g., confirmatory factor analysis) are appropriate when the researcher wishes to test a predetermined model. Exploratory factor analysis, on the other hand, is a type of analysis that allows scholars to "uncover the latent constructs underlying the variables, in an attempt to better understand the nature of such constructs" (Bandalos & Boehm-Kaufman, 2009, p. 63). Exploratory factor analysis is appropriate in cases "when researchers do not have any particular expectations regarding the number and nature of the underlying factors (i.e., latent variables) that exist in the data" (Loewen & Gonulal, 2015, p. 183). Factor analyses have been widely used to address questions of construct validity, with exploratory factor analysis often used to develop a model that can subsequently be tested using SEM or confirmatory factor analysis. To take one example from SLA, factor analyses have been central in the debate over the construct validity of different measures of implicit and explicit knowledge (see, among others, [START_REF] Ellis | Measuring implicit and explicit knowledge of a second language: A psychometric study[END_REF][START_REF] Ellis | Confirming the operational definitions of explicit and implicit knowledge in Ellis (2005): Responding to Isemonger[END_REF][START_REF] Gutiérrez | The construct validity of grammaticality judgment tests as measures of implicit and explicit knowledge[END_REF]. In the context of the present study, we are interested in exploring the construct validity of four measures of L2 vocabulary knowledge. Exploratory factor analysis (as opposed to a confirmatory approach) was deemed most appropriate considering current discussions and debates regarding the nature of vocabulary knowledge and the measurement of that knowledge within the broader context of the L2. In particular, we note that the characterization of assessments of vocabulary knowledge continues to be the subject of debate. Not only are there disagreements in classification according to the two most widely cited dimensions (i.e., vocabulary size versus depth and receptive versus productive knowledge), but researchers such as Fitzpatrick and Meara (2004, p. 72) have explicitly stated that tests of productive vocabulary knowledge may not measure the same construct, while others [START_REF] Fitzpatrick | Making sense of learner performance on tests of productive vocabulary knowledge[END_REF] have highlighted that test constructs are to some extent overlapping. While attempts have been made to identify different aspects of vocabulary knowledge potentially tapped by different tests (see Table 2), there is as yet no clear model to test.

Method

Participants

In selecting our participant population, we opted for highly proficient learners, given that some of the words targeted by two of the tests (PVLT and VLT) are infrequent and, thus, unlikely to be known by less proficient learners. Data were collected from all students enrolled in the fourth semester of an English degree program at a French university. Overall, these participants were highly proficient in the L2, as they were able to function in an academic context where most classes and classwork were conducted in English. Each semester, students enrolled in this program take obligatory courses on translation, English linguistics, Anglophone history and culture, and English literature, for a total of 12 to 15 hours a week of instruction in English. Our analysis is based on data from the 100 participants who completed all measures. On average, participants were 20.47 years old (SD = 2.04, range = 18-28), and 79 were women. Participants reported speaking a wide variety of languages within their families: 63 reported speaking only French, 29 reported speaking French and at least one other language, 7 reported growing up speaking a language other than French (e.g., Berber, Wolof), and one person did not respond to this question. When asked what foreign languages they had studied, six participants responded only English, 38 reported learning English and one other foreign language, and all remaining participants stated that they had learned English in addition to two or more foreign languages.

Data Collection

Participants completed four English vocabulary measures: Lex30, G_Lex, the PVLT, and the VLT. Data collection took place during two 1-hour class periods separated by one week, with task order counterbalanced within each testing session. During the first datacollection session, participants completed paper-and-pencil versions of the VLT and G_Lex, before completing a background questionnaire at the end of the session. For the second session, participants responded to the PVLT and Lex30. Tests were divided between the two class sessions in such a way to allow for testing sessions of the same length.

Data Coding and Analysis

To allow for comparability across tests, we used one single frequency benchmark for scoring. For this purpose, we opted to use the online BNC-COCA corpora [START_REF] Nation | The BNC/COCA Level 6 word family lists (Version 1.0.0[END_REF]. On Lex30 and G_Lex, any response provided that was not a function word, proper noun, or number and did not belong to the first 1,000 most frequent words in the reference corpora received one point. Although the VLT and the PVLT are also intended to assess knowledge of words that lie outside of the first 1,000 frequency band, these tests target specific words which were chosen based on frequency information available when they were designed. This means that the frequency of certain target items may have changed and, importantly, some may now fall within the first 1,000 most frequent word families according to more recent frequency information. We thus verified the frequency of each target item in the BNC-COCA, observing that 11 of the original 150 items on the VLT and that 5 of the original 90 items on the PVLT now belong to the 1,000 most frequent word families. Given that these tests are intended to target less frequent words (i.e., words beyond the 1,000 frequency band), we excluded those items.

This dataset was analyzed with an exploratory factor analysis using R (RStudio Team, 2020). To begin, we verified that the four scores from the four vocabulary tests were normally distributed. One set of scores -from the VLT -was found to be non-normally distributed, because of a significantly negative skew. For this reason, we subjected these scores to a reverse score transformation (see Field, Miles & Field, 2012, p. 192). They were then logtransformed and reversed back. The final set of scores showed no problems with normality. In running the factor analysis, we followed general recommendations from [START_REF] Field | Discovering statistics using R[END_REF] and field-specific recommendations from [START_REF] Loewen | Exploratory factor analysis and principal components analysis[END_REF] and [START_REF] Plonsky | Methodological synthesis in quantitative L2 research: A review of reviews and a case study of exploratory factor analysis[END_REF]. The results section is organized following the step-by-step process described by Loewen and Gonulal. To begin, we assess the appropriateness of conducting a factor analysis on our dataset. We then move on to decisions regarding the factor extraction method, the factor retention criteria, and the factor rotation method. Once these steps have been presented, we present the results from the factor analysis.

Factor Analysis

Factorability of this Dataset

Descriptive statistics for this dataset are presented in Table 3, whereas the full correlation matrix is provided in Table 4. The appropriateness of conducting a factor analysis on this dataset was assessed in three ways, namely by running Bartlett's test, the Kaiser-Meyer-Olkin (KMO) measure, and by checking the determinant of the correlation matrix.

Beginning with Bartlett's test of sphericity, the Chi-square value was 155.245 (df = 6), which was significant at the p > .001 level. This indicates that the correlations in this dataset are large enough for factor analysis. KMO is a measure of sampling adequacy. In the case of this dataset, the overall KMO value was .74, and ranged from .7 to .79 for individual variables.

According to Field et al. (2012, p. 770), values between .7 and .8 are considered good, which further supports the conclusion that factor analysis is appropriate for this dataset. Finally, we explored whether there may be issues of multicollinearity in this dataset by calculating the determinant of the correlation matrix. The determinant for this matrix was 0.2012485, which suggests that there are no problems with multicollinearity.

Factor Analysis Decisions

The first decision to be made concerned the type of factor analysis to carry out.

Because we were interested in uncovering potential latent constructs underlying the four variables measured (see Bandalos & Boehm-Kaufman, 2009, p. 63), exploratory factor analysis was considered more suitable than principal component analysis. To conduct the analysis in RStudio, we used the fa() command from the psych package with the factor extraction method of minimum residual. 2 In order to determine the number of factors to extract from the dataset, we followed Loewen and Gonulal's (2015, pp. 196-197) recommendation and explored several factor retention criteria. We began by using parallel analysis (with the fa.parallel() command in the psych package). In this approach to factor retention, the eigenvalues of the actual variables are compared with randomly generated eigenvalues based on a data matrix of the same size as the original dataset. Only actual eigenvalues that are larger than the generated ones are recommended to be retained, regardless of their absolute value. In the case of this dataset, parallel analysis showed a twofactor solution, with the first factor having an eigenvalue of 2.30 and the second of .33. Thus, although the absolute eigenvalue associated with the second factor is below the recommended values of 1.0 (Kaiser's cut-off) and 0.7 (Joliffe's cut-off), parallel analysis suggests a twofactor solution. It should be noted that across-the-board cut-offs like Kaiser's and Joliffe's have come under criticism, with most empirical assessments of retention criteria showing parallel analysis to perform better than both (see [START_REF] Bandalos | Four common misconceptions in exploratory factor analysis[END_REF]Field et al., 2012, p. 764). We next looked at cumulative variance. When using a two-factor solution, as suggested by parallel analysis, the first factor accounts for 57% and the second factor for 8% of variance, for a total cumulative variance of 65%. If we follow Loewen and Gonulal (2015, p. 194), who suggested that "it may be appropriate to continue factor extraction until at least 60% of the total variance is accounted for," a two-factor solution is justified. Finally, inspection of the scree plot revealed inflection points compatible with the retention of one or two factors. Taken together, we retained a two-factor solution for the final analysis. The final decision that we made at this stage concerned the method of factor rotation. As we expected that the two factors may correlate (see discussion in Plonsky & Gonulal, 2015, p. 22), we opted for oblique rotation (more specifically, oblimin).

Factor Loadings

The pattern matrix for the rotated factor loadings is provided in Table 5. Each of the four tests loaded strongly (> 0.3, see Loewen & Gonulal, 2015, p. 199) on only one factor. On the one hand, Lex30 and G_Lex loaded strongly on Factor 2, whereas the PVLT and VLT

showed high loading values on Factor 1.

Interpretation and Discussion

The interpretation of factor loadings requires that the analyst identify the core content for each factor (Loewen & Gonulal, 2015, p. 203). To aid in the identification of this core content for the two factors identified in our exploratory factor analysis, we returned to Table 2, in which we identified the different components of vocabulary knowledge called upon by each of the four tests under study (using Nation's, 2013, components approach to vocabulary knowledge). In Table 6, we have grouped together the tests as a function of their factor loadings (the PVLT and VLT on Factor 1, Lex30 and G_Lex on Factor 2). Under each test, we have reproduced the aspects of vocabulary knowledge potentially tapped by each (taken from Table 2). In addition, we have bolded aspects of vocabulary knowledge that are shared by tests loading onto the same factor. In terms of core content for Factor 1, the PVLT and VLT overlap with regards to one aspect of vocabulary knowledge, namely receptive knowledge of the written form of a word. In addition, both of these tests are predicted to tap two other aspects of receptive vocabulary knowledge. For Factor 2, three cases of overlap are visible, all three of which involve productive vocabulary knowledge: productive knowledge of written form, productive knowledge of form and meaning, and productive knowledge of word associations. By considering the aspects of vocabulary knowledge solicited by each of the four tests, it appears that at the core of Factor 1 lies the assessment of receptive knowledge, whereas the thematic core of Factor 2 is the measurement of productive vocabulary knowledge. We thus suggest that Factor 1 corresponds to receptive vocabulary knowledge, whereas Factor 2 reflects productive vocabulary knowledge. In what follows, we discuss in turn each of the two factors identified in this analysis before then reflecting more generally on what this study brings to the discussion of the constructs underlying vocabulary knowledge in a L2. We conclude our discussion by identifying the limitations of the current project and offering suggestions for future research.

At first blush, the results from Factor 1 -what we have labeled "receptive vocabulary knowledge" -might appear surprising. This is because the findings for this factor reveal that performance on the PVLT patterned with performance on the VLT, even though the former is billed as a test of productive vocabulary knowledge, whereas the latter is intended to tap receptive knowledge. However, as already mentioned, despite its widespread use as a measure of productive vocabulary knowledge, certain researchers (i.e., [START_REF] Webb | Receptive and productive vocabulary learning: The effects of reading and writing on word knowledge[END_REF] have expressed reservations about the PVLT as a measure of productive knowledge, highlighting the fact that certain of its design features may also lead to the assessment of receptive knowledge. The results from the present study provide empirical evidence in support of these reservations. The fact that performance on the PVLT patterned significantly with performance on the VLT is particularly important, given that the PVLT has been used as a benchmark in past validation studies for new tests of productive vocabulary knowledge (like Lex30; see [START_REF] Fitzpatrick | Exploring the validity of a test of productive vocabulary[END_REF][START_REF] Walters | Aspects of validity of a test of productive vocabulary: Lex30[END_REF]. More specifically, the PVLT has often been used as a point of comparison for assessments of concurrent validity in correlational studies. Indeed, Walters (2012) reported high correlations (p = .772) between results obtained on PVLT and Lex30, whereas [START_REF] Fitzpatrick | Exploring the validity of a test of productive vocabulary[END_REF] reported a more moderate correlation (p = .504). In our own results, the PVLT and Lex30 results showed a correlation at p = .616 (see Table 4).

Although these relatively high bivariate correlations are consistent with the conclusion that the two tests are measuring the same or an overlapping construct (and, thus, these results may be -and have been -taken as evidence of concurrent validity), the results from our exploratory factor analysis suggest a different story, namely one in which the PVLT patterns with the measure of receptive vocabulary knowledge (the VLT). With respect to the PVLT, these results thus call into question whether this test may be the best choice for concurrent validity studies concerning the assessment of productive vocabulary knowledge. At a more general level, the divergence seen between validity arguments based on bivariate correlations and the factor analysis we have presented highlights the importance of varied and complementary approaches to assessing construct validity, as well as demonstrating the need for ongoing and sustained attention to questions of validity.

Turning now to Factor 2, Lex30 and G_Lex both loaded strongly onto this factor, suggesting that these two tests are tapping into the same construct. [START_REF] Fitzpatrick | Making sense of learner performance on tests of productive vocabulary knowledge[END_REF] created G_Lex to closely match Lex30 insofar as on both tests participants respond to several activation events (meaning elements that solicit vocabulary knowledge) and scores are based on the number of infrequent words provided. G_Lex crucially differs from Lex30 with respect to the presence of context (in the form of carrier sentences), a difference which Fitzpatrick and Clenton hypothesize will result in differences in the vocabulary knowledge reflected in performance (see their vocabulary test capture model). However, the fact that performance on these two tests patterns together in our analysis suggests that the hypothesized difference is more conceptual than empirical, at least for the current dataset. Considering the core content covered by these two tests, we named this factor "productive vocabulary knowledge." The fact that Lex30 and G_Lex load strongly onto this factor is consistent with the intended aim of these tests, which was to provide a general assessment of productive vocabulary knowledge. Moreover, whereas Lex30 has been the focus of numerous validation attempts, the more recently developed G_Lex has yet to receive such attention. Thus, one of the notable contributions of the present study is to provide initial evidence for the construct validity for G_Lex. The results with respect to this test are promising and will hopefully lead to additional research involving this measure.

The results of our exploratory factor analysis and our interpretation of the factor loadings thus suggest that the receptive versus productive knowledge distinction underlies the primary constructs assessed by the four vocabulary tests under study. Our findings echo those of [START_REF] González-Fernández | Word knowledge: Exploring the relationships and order of acquisition of vocabulary knowledge components[END_REF], who reported that the most parsimonious SEM model of their data identified as crucial the distinction between recall (i.e., the ability to retrieve word knowledge) and recognition (i.e., the ability to recognize and select word knowledge, see p. 486). These results have potentially important implications for theory. In [START_REF] Nation | Learning vocabulary in another language[END_REF] widely cited list of vocabulary knowledge components (see [1] and Table 2), vocabulary knowledge is first organized into nine components, with each component then covering receptive and productive knowledge. For example, the component "spoken form" encompasses the receptive knowledge of what the word sounds like and the productive knowledge of how to pronounce the word. As González-Fernández and Schmitt (2020, pp. 500-501) highlight, this presentation implies a hierarchical organization, with vocabulary knowledge components on the first tier and the receptive-productive knowledge distinction conceptualized within each component. However, both the results from the present study and those reported in González-Fernández and Schmitt (2020) offer evidence of the potentially greater importance of the receptive-productive distinction. Additional research is necessary, but these initial findings suggest that a distinction between the receptive and the productive knowledge constructs might prove to be the most fundamental division along which vocabulary knowledge in a L2 is organized.

To end our discussion, we identify two limitations of our study, which also serve to highlight potential directions for future research. The first limitation concerns the participant population who took part in this experiment. As English majors well on their way to a university degree in English studies, these participants were proficient in their L2. In the context of the present study, the relative homogeneity in L2 proficiency constitutes a potential limitation insofar as it has been suggested that certain assessments of vocabulary knowledge may tap different types of knowledge as a function of the learners' overall proficiency level.

For example, [START_REF] Walters | Aspects of validity of a test of productive vocabulary: Lex30[END_REF] administered Lex30 to Turkish learners of English at three proficiency levels. At a subsequent testing time, the participants also completed a sentence elicitation test to determine the extent to which "test takers could also use the words they were able to recall in association with the [Lex30] stimulus words" (p. 181). Based on her elicitation task responses, Walters' suggested that Lex30 might elicit different types of productive vocabulary knowledge as a function of a learner's level of L2 competence, leading her to conclude "Lex30 may be a valid test of productive vocabulary use for higher proficiency students, [whereas] it is more valid as a test of productive recall at the lower levels" (p. 183). In light of these results, it is unclear whether our findings can be generalized to other proficiency levels, a possibility that should be explored in future research. A second limitation concerns the number of different tests used in this study. The main focus of our project was on the assessment of productive vocabulary knowledge. However, there currently exist few reliable tools to assess this construct (Miralpeix, 2020, p. 192). In addition to three published assessment tools of productive vocabulary knowledge (PVLT, Lex30, and G_Lex), we also decided to include one test of receptive vocabulary knowledge (the VLT), given doubts about the construct validity of the PVLT. This small number of measures meant that only two tests loaded on each factor, which is, according to Loewen and Gonulal (2015, p. 203), the minimum number of variables needed for a meaningful interpretation. Future research would thus do well to include a larger number of tests, which may allow to potentially confirm and strengthen the current results.

Conclusion

The impetus for this study was a desire to explore the construct validity of measures used to assess productive vocabulary knowledge in a L2. Vocabulary researchers have shown strong interest in questions of construct validity, which has been addressed using a variety of analytic approaches. We contributed to this discussion with an exploratory factor analysis.

Our results showed that for a group of 100 Francophone learners of L2 English, performance on the PVLT and the VLT patterned together (Factor 1), while performance on Lex30 and G_Lex loaded strongly onto the same factor (Factor 2). We interpreted Factor 1 as representing receptive vocabulary knowledge and Factor 2 as reflecting productive vocabulary knowledge. These results allowed us to offer insights into the nature of the four tests under study and additional evidence of the importance of the receptive-productive knowledge distinction in the conceptualization of L2 vocabulary.

  vocabulary knowledge construct"(González-Fernández & Schmitt, 2020, p. 491). In their study, González-Fernández and Schmitt administered nine vocabulary tests to 144 Spanishspeaking learners of English. These tests included one measure of receptive vocabulary size (the VLT) and eight measures of vocabulary depth (for 20 target words), covering both recall (an aspect of productive knowledge) and recognition (an aspect of receptive knowledge) for four components of vocabulary knowledge: form-meaning connection, derivations, polysemy, and collocations. Spearman correlations between all tests were high and positive. Results from the SEM analysis, however, revealed more complex patterns. Using the results from the eight measures of vocabulary depth, two alternate models of vocabulary knowledge were tested. Model 1 hypothesized that the four components (knowledge of form-meaning connections, derivations, polysemy, and collocations) would be direct contributors to vocabulary knowledge, with the recall and recognition measures for each contributing to the component in question. In Model 2, on the other hand, recall and recognition measures for each component contributed directly to the vocabulary knowledge construct. The authors found that Model 2 was supported by the data, which they suggest indicates "that recognition vs. recall knowledge was the key distinction in our study" (p. 501). Concretely, this means that, for example, performance on measures of receptive knowledge of different components appeared to behave more similarly than did receptive and productive knowledge for the same component.

Figure 1D .

 1D Figure 1D. Example of completed VLT (5K band only)

Table 1

 1 Instructions and Example Items for Three Tests of Productive Vocabulary Knowledge

	Measures	Instructions	Example items
	Lex30	Write down the first four	1. attack
		(English) words you think of	2. board
		when you read each word in	3. close
		the list
	G_Lex	Write down five different
		words that might fit into each
		gap. The gaps are suitable for
		nouns, adjectives, and verbs
		in equal measure (eight
		sentences each).

  test for assessing receptive vocabulary size (but see[START_REF] Webb | The updated Vocabulary Levels Test: Developing and validating two new forms of the VLT[END_REF], for a discussion of the appropriateness of this use; an example from the test is provided in the Appendix, Figure1D). As is shown in Table2, the types of knowledge potentially assessed by the VLT overlap with the knowledge types mobilized by the PVLT, insofar as both tests require the recognition of the written form of target words. This overlap, combined with the fact that the PVLT was developed as a productive counterpart to this test, makes the VLT a relevant point of comparison for the PVLT (i.e., these two tests may measure similar aspects of vocabulary knowledge).

tapping into receptive knowledge led us to include a fourth measure -a receptive vocabulary knowledge test -in the present study. The inclusion of a receptive test importantly allows us to verify whether performance on the PVLT patterns more closely with performance on other productive measures of vocabulary or with the Vocabulary Levels Test (VLT,

[START_REF] Nation | Testing and teaching vocabulary[END_REF][START_REF] Schmitt | Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test[END_REF]

, the receptive measure included in this study. The VLT is a widely used

Table 2

 2 Dimensions of Vocabulary Knowledge Tapped by Four Vocabulary Tests

	Lex30 G_Lex PVLT VLT

Table 3

 3

	Descriptive Statistics
	Measure	Mean	Median	SD	range
	Lex30	40.12	42	13.80	3-77
	G_Lex	17.96	17.5	9.68	0-41
	PVLT	47.01	49	11.39	18-71
	VLT	104.31	112	23.87	28-135
	Table 4
	Correlation Matrix
		Lex30	G_Lex	PVLT	VLT
	Lex30 1	0.569	.616	.482
	G_Lex	1		.527	.356
	PVLT			1	.689
	VLT				1

Table 5

 5 Rotated Factor Loadings (Pattern Matrix)

	Factors

Table 6

 6 Core Content for Factor Loadings

	Factor 1
	Receptive knowledge
	PVLT	VLT
	Receptive written form	Receptive written form
	Receptive grammatical functions	Receptive form and meaning
	Receptive collocations	Receptive associations
	Productive written form
	Productive grammatical functions
	Productive constraints on use
	Productive form and meaning
	Factor 2
	Productive knowledge
	Lex30	G_Lex
	Productive written form	Productive written form
	Productive form and meaning	Productive form and meaning
	Productive associations	Productive associations
		Receptive collocations

In this paper we refer to the generic term 'word'. While we acknowledge the complexity and appropriacy of different lexical units (e.g.,[START_REF] Webb | The lemma dilemma: How should words be operationalized in research and pedagogy?[END_REF], we use word types as our measurement unit.

We also tested other extraction methods, such as maximum likelihood; these changes had no impact on the results.