Z Maazaz

O De La Barrière

O Hubert

Influence of plastic shear strain on the magnetic behaviour of pure iron

Keywords: Pure iron, plastic shear strain, magnetic hysteresis, magneto-plastic modeling

The effect of plastic strain on magnetic behaviour was studied earlier by many authors. These studies mainly focused on plastic straining obtained by tensile testing and on magnetic measurement along the tensile axis. We present in this paper some new results showing the influence of a plastic shear strain obtained thanks to a torsion test on the magnetic behaviour of a pure iron. These results highlight the dominant role of the cumulative plastic strain on the magnetic behaviour.

Introduction

The study and modeling of the effects of the mechanical state on the magnetic properties of materials remains an ongoing challenge [1,2]. A better understanding and modeling of the effect of plasticity (permanent deformation) is particularly sought. Applications are numerous: improving the predictivity of electromagnetic non-destructive testing [3]; better control the influence of cutting on the performance of electrical steels [4]. Previous research has mainly focused on the influence of tensile plastic deformation on magnetic behavior [5,6,7,8] or magnetic domains structure [9] thus leading to different interpretations. In this work, plasticity is obtained through a torsion test. We then study the influence of this plastic shear strain on the magnetic behavior of pure iron. Results are discussed in terms of the influence of defects and residual stresses.

Protocol

The material employed in this investigation is a pure iron (ARMCO grade). This material exhibits a single ferrite untexturized phase as illustrated in figure 1. The torsion test is probably the most suitable test for carrying out plastic shear deformation. It consists of applying a torque C along the z axis of a tubular test piece (figure 2a). The torque produces a twist angle α. Under these conditions stress and deformation tensors σ and are homogeneous and given in the cylindrical frame (rθz) by :

σ =           0 0 0 0 0 τ 0 τ 0           =           0 0 0 0 0 γ/2 0 γ/2 0           (1)
where γ is the distortion (corresponding to twice the shear strain term -see figure 2a). The shear behaviour of a material is obtained by plotting the shear stress τ as function of the distortion γ. τ is linked to the torque C by:

τ = C(R o + R i ) π(R 4 o -R 4 i ) (2)
R o and R i are the outer and inner radii of the cylinder. The test specimen has been meticulously sized using a nonlinear mechanical finite element formulation (and a constitutive behaviour obtained from a previous tensile test).

100µm This design has to prevent plastic buckling while ensuring the presence of a sufficiently homogeneous region for accurate magnetic measurements. An illustration of the simulated geometry (half specimen), demonstrating the homogeneity of stress across the specimen, is provided in the figure 2b. It is a dogbone specimen where heads (in blue) allow for a suitable application of a torque. Inner and outer diameter of final geometry are 13mm and 14.9mm respectively. It defines a homogeneous gauge length of about 71mm. The average fluctuation of the plastic shear strain through the thickness is assessed to be approximately 4%. Mechanical tests were carried out using the MTS hydraulic twisting testing machine with angular control allowing for γ = 2 × 10 -3 s -1 . Figure 3a and 3b illustrate the specimen and the mechanical setup employed. A strain gauge positioned at 45 degrees from z-axis has been used to measure the local shear strain = γ/2. A measurement error of ±0.001 on γ can be estimated. As illustrated in figure 3 c), a magnetic measurements set-up based on Helmholtz coils has been used. Indeed, this system allows a homogeneous magnetic field to be applied in the central zone where the sample is placed while providing access to the center of the setup to position a Hall effect sensor for the measurement of the local magnetic field H. The homogeneity of the field in the Hall probe measurement zone has been verified by finite elements calculations. A B-coil wound around the sample in the middle (with a length less than 10 mm ) allows for the measurement of the magnetic induction B after analog integration. Power generation is carried out through a programmable generator (used in DC mode), and the power is amplified by a linear amplifier. The measurement principles are given in figure 3 d). A ballistic method [START_REF] Fiorillo | Measurement and characterization of magnetic materials[END_REF] has been chosen for the acquisition of the magnetic behaviour removing any dynamic effect since the time delay between each point is large.

Results

Specimen #1 is non-deformed and used to get reference magnetic behaviour. Figure 4 reports the shear stress-distortion (τγ) curves of the three other deformed specimens. Specimen #2 has been subjected to monotonous torsion up to a plastic distortion (remanent distortion) of about γ p =+4%. Specimen #3 has been subjected to a cyclic testing beginning by a positive plastic distortion first up to γ p =+4% then followed by a negative twist reaching a plastic distortion of about -4%. Specimen #4 has been subjected to high magnitude plastic distortion up γ p =12%. Following Levy-Mises description, the cumulative plastic strain is defined as :

p = T 0 2 3 ˙ p : ˙ p dt (3)
with p the plastic strain tensor and T the end time of mechanical test. Considering pure plastic shear strain, the cumulative plastic strain writes :

p = T 0 γp √ 3 dt (4)
This parameter is usually related to the defects quantity. Specimen #3 and #4 are characterized by the same level of cumulative plastic strain different plastic strain levels (figure 4). Figure 5a shows the experimental major B(H) loops for the tested specimens. The main effects that can be observed relate to the saturation knee region: the undeformed specimen presents the highest induction level. Plasticity leads to a decreasing of magnetic induction consistent with the observations made by many authors. It is remarkable to note that the evolution of the induction is correlated with the level of cumulative plasticity but is not correlated with the plastic strain level: the behavior of specimen #2 is intermediate between that of specimen #1 and #4. Specimen #3 and #4, characterized by a same level of cumulative plastic deformation, exhibit similar magnetic behaviours (specimen #4 is slightly more deformed than specimen #3 as observed in figure 4 explaining the hierarchy), Specimen #2 and #3, which have been subjected to the same absolute level of plastic deformation |γ p |, exhibit different behaviours.

Figure 5b shows the variation of coercive field H c extracted from all magnetic data as a function of maximum magnetic field level H max of various loops. The field increases nonlinearly with respect to the maximum applied magnetic field as expected. It increases too as function of the cumulative plastic strain. Values reached are significantly higher than values for reference specimen. Again, coercive field curves of specimen #3 and #4 are very close to each other. The same conclusion applies when looking at the energy density losses W (area of hysteresis loops) as function of maximal magnetic induction B max in figure 6. Energy losses look however slightly higher for specimen #4 than for specimen #3. This may be explained by small differences of true cumulative plastic strain or other factors discussed in the last section. Figures 7a to 7c represent finally maximum relative permeability µ mr (highest average slope of B(H) curves), maximum coercive field (evaluated at H max =8000A/m), magnetic energy density losses (evaluated at B max =1.5T) as function of cumulative plastic strain. Conclusions join the previous comments: cumulative plastic strain seems to be a relevant mechanical term that correlates with the variations of magnetic quantities.

Discussion

It can be observed first that results obtained in this study join the stablished knowledge in the literature: plastic strain "degrades" the magnetic behaviour. This study shows that the mode of deformation appears to have minimal or possibly negligible involvement in the process: Shear plastic strain acts qualitatively similarly to tensile plastic strain [6]. This study shows on the other hand that cumulative plastic strain is relevant unlike plastic strain level. It is not possible to observe such effect by a simple tensile testing. Cumulative plastic strain is well known to be related to microstructural defects such as dislocations [START_REF] Anderson | Theory of dislocations[END_REF]: indeed accumulation of dislocations serves as pinning sites for magnetic domain walls [START_REF] Hilzinger | [END_REF]. That leads several researchers to propose a relationship between coercive field and dislocation density [13].

In some recent works, it has been proposed to link the variation of magnetic behaviour to internal stresses that occur with plastic straining [14]. This approach allows for the understanding why a plastic compression can improve the magnetic behaviour of a pipeline steel unlike a plastic tension [START_REF] Maazaz | Influence of plastic compression on the magnetic behaviour of a pipeline steel[END_REF]. Indeed the same cumulative plastic strain can be reached by tension and compression. Since behaviours are different, interpretation using internal stresses may be relevant. Figure 4 shows the experimental τ(γ) results. Experiment with specimen #2 involves a cyclic torsion that allows for an observation of the new elastic domain (ED) after the first loading. This domain is not symmetric with respect to τ = 0 axis, meaning that internal stresses have been developed [START_REF] François | Mechanical Behaviour of Materials -Volume I: Elasticity and Plasticity[END_REF]. Following the proposition of [8], a material can be divided in soft and hard (h) phases in a single phase material). Residual stresses in each phase are given by :

σ s = -X and σ h = f s f h X (5)
f s and f h are the volume fractions of the soft and hard phases and X the macroscopic kinematic hardening tensor. In case of pure shear stress-strain loading, the kinematic hardening tensor follows the plastic strain tensor and writes:

X =           0 0 0 0 0 X θz 0 X θz 0           (6)
The stress state in each of phase is therefore deviatoric. We then employ the principle of equivalent stress proposed in [START_REF] Hubert | [END_REF] to assess the effect of this particular residual stress state on the magnetic behaviour. The equivalent stress writes:

σ eq = 3 2 t n.σ D . n (7)
where n indicates the magnetic measurement direction and σ D the stress deviator. Applied to the present situation, n is corresponding to direction z, σ D = σ s for the soft phase and σ D = σ h for the hard phase. Calculation shows that the equivalent stress is null for both phases. This means that residual stresses, even existing, have no effect on the magnetic behaviour according to this model. The torsion test therefore has the advantage to allow for an observation of the effect of defects on the magnetic behaviour, without being masked by the effect of internal stresses. This result can explain why specimen #3 and #4 exhibit similar magnetic behaviour despite a probably different residual stress state (since plastic strain level and sign are different).

Conclusion

To the best knowledge of authors, it is the first time that effect of plastic shear strain on magnetic behaviour is reported. This work highlights the very significant role of cumulative plastic strain on the magnetic behaviour, leading to a global degradation of magnetic performances of a soft magnetic material: decreasing of permeability; increasing of coercive field and static power losses. On the contrary, the deformation level is not the relevant parameter. The torsion test also leads to a deviatoric residual stress state whose influence is negligible when magnetic measurement is carried out along the torsion axis. The experimental protocol therefore makes it possible to observe the sole influence of defects generated by plasticity on magnetic behavior.

Figure 1 :

 1 Figure 1: Microstructure of pure iron (ARMCO grade) illustrated by inverse pole figure from electron back-scattered diffraction.

Figure 2 :Figure 3 :

 23 Figure 2: (a) principle of torsion test; (b) optimized (half) geometry and shear stress distribution for C=32Nm.

Figure 4 :

 4 Figure 4: shear stress vs shear strain (τ-γ) history of conducted tests. ED: elastic domain after deformation of specimen #2.

Figure 5 :

 5 Figure 5: a) Experimental hysteresis cycles for the tested specimens. b) coercive field variation according to maximum magnetic field

Figure 6 :

 6 Figure 6: Static magnetic energy density losses for samples #1 to #4

Figure 7 :

 7 Figure 7: Maximal relative permeability (a), coercive field at H max =8000A/m (b) and static magnetic energy density losses at B=1.5T (c) as function of cumulative plastic strain.


M.D Eric Van Belle

Julien Labreuche

M.D Flavien Vincent

M.D Fabio Barili
email: fabarili@libero.it

Nicholas Freemantle

 of the article) and the wide differences between the hazard ratio at 0 to 2 years (0.89; 95% confidence interval [CI], 0.73 to 1.09) and that at 2 to 5 years (1.27; 95% CI, 1.06 to 1.53) suggest that the hazards were not constant over time. Therefore, the reported 5-year hazard ratio is not an accurate reflection of the findings, which require landmark analysis or time-varying modeling. 3,4 The key message of the trial is also overshadowed by the disadvantage of TAVR in terms of reoperations and rehospitalization, which underscores concerns about the durability of TAVR devices. In summary, the 5-year results from the PARTNER 2 cohort A trial are not a swan song for surgery.

R E S P O N S E

The authors reply: We agree with Barili et al. that the higher risk of death or disabling stroke between 2 and 5 years after TAVR than after surgery in the landmark analysis arouses concerns regarding the long-term effectiveness of TAVR. Although there were wide differences between hazard ratios at 0 to 2 years and at 2 to 5 years, the hazard ratios that were calculated for later time periods include only survivors of earlier time periods and thus are not truly randomized comparisons. Furthermore, a similar 5-year restricted mean survival time (event-free time for the primary end point) in the TAVR and surgery groups supports the validity of similar effectiveness with respect to death or disabling stroke at 5 years. It is important to emphasize that these results were obtained with a previousgeneration valve limited by higher rates of paravalvular regurgitation than newer-generation valves and 25% usage of the transthoracic approach that has now largely been abandoned owing to worse outcomes. The comparisons between TAVR and surgery in the transfemoral cohort are more representative of current practice (Fig. S2A in the Supplementary Appendix of our article). Khokhar et al. point to the more favorable reductions in left ventricular volumes and mass regression in patients treated with surgery as compared with TAVR. Postprocedure moderate or severe aortic regurgitation has been independently associated with less left ventricular mass regression in TAVR. 1 In the current data, the differences in left ventricular mass regression between TAVR and surgery persisted after excluding patients with at least moderate aortic regurgitation. The effect of additional procedural outcomes including new pacemaker and conduction-system block on left ventricular regression needs further elucidation. A recent analysis of the PARTNER 3 trial 2 involving low-risk patients showed equivalent left ventricular mass regression both at 30 days and 1 year for TAVR and surgical aortic-valve replacement, which suggests that the hemodynamics of the SAPIEN 3 valve may have advantages over the SAPIEN XT valve used in our trial. The results of the recent trials showing superiority or noninferiority of TAVR over surgery in low-risk patients at 1 and 2 years 3,4 and noninferiority to surgery in intermediate-risk patients at 5 years justify careful expansion of TAVR independent of patients' risk of surgery. Nonetheless, given the lack of long-term data, our trial is not a swan song for surgery.

R E S P O N S E

The editorialist and colleagues reply: We thank Barili et al. for their methodologic point regarding the trial by Makkar et al. reporting the 5-year outcomes of the comparison of surgical aortic-valve replacement versus TAVR with the SAPIEN XT valve (PARTNER 2 cohort A trial). They noted that the proportional-hazards assumption was not verified over time, as easily spotted by the crossing of the Kaplan-Meier curves after 2 years of follow-up. Makkar et al. anticipated this possibility and prespecified the calculation of the restricted mean survival time and the restricted mean event-free time for the primary end point (death or disabling stroke). 1 These results, which are reported in Table S8 and in the text, show that the 5-year restricted mean survival time was similar for TAVR and surgery (46.3 months and 46.6 months, respectively), as was the restricted mean event-free time for the end point of death or disabling stroke (45.0 months and 44.8 months). They also performed a landmark analysis showing superiority of surgery beyond 2 years (hazard ratio with TAVR, 1.27; 95% CI, 1.06 to 1.53) (Table S7). The potential explanations for this late increase in risk with TAVR were extensively discussed in the editorial, including the potential role of paravalvular regurgitation, the lack of concomitant coronary or valve procedures, and durability concerns. Khokhar et al. commented that the magnitude of the left ventricular mass regression was less pronounced in the TAVR group than in the surgery group. We can reasonably speculate that the higher rate of mild or greater paravalvular regurgitation with the second-generation transcatheter bioprosthesis than with the surgical valve may have adversely influenced left ventricle remodeling. 2 The recently published echocardiographic outcomes of low-risk patients in the PARTNER 3 trial are supportive of this hypothesis, since the very low rate of aortic regurgitation that was achieved with the third-generation valve (SAPIEN 3) was associated with left ventricular mass regression similar to that with surgery at 1 year (-11.9±21.10 vs. -12.0±27.09 g per square meter, P=0.84).

  Raj R. Makkar, M.D. Cedars-Sinai Medical Center, Los Angeles, CA raj.makkar@cshs.org Michael J. Mack, M.D. Baylor Scott and White Health, Plano, TX Martin B. Leon, M.D. Columbia University Irving Medical Center, New York, NY Since publication of their article, the authors report no further potential conflict of interest. 4 References 1.Chau KH, Douglas PS, Pibarot P, et al. Regression of left ventricular mass after transcatheter aortic valve replacement: the PARTNER trials and registries. J Am Coll Cardiol 2020;75:2446-2458. 2.Pibarot P, Salaun E, Dahou A, et al. Echocardiographic results of transcatheter versus surgical aortic valve replacement in low-risk patients: the PARTNER 3 trial. Circulation 2020;141:1527-1537. 3.Mack MJ, Leon MB, Thourani VH, et al. Transcatheter aortic-valve replacement with a balloonexpandable valve in low-risk patients. N Engl J Med 2019;380:1695-1705. 4.Popma JJ, Deeb GM, Yakubov SJ, et al. Transcatheter aortic-valve replacement with a selfexpanding valve in low-risk patients. N Engl J Med 2019;380:1706-1715.

3

  Eric Van Belle, M.D., Ph.D.Institut Coeur Poumon, Lille, France Julien Labreuche, B.S.T.Université de Lille, Lille, France Flavien Vincent, M.D., Ph.D.Centre Hospitalier Universitaire de Lille, Lille, France Mr. Labreuche and Dr. Vincent report no potential conflict of interest. Since publication of his editorial, Dr. Van Belle reports no further potential conflict of interest. 3 References 1.Royston P, Parmar MKB. Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome. BMC Med Res Methodol 2013;13:152-152. 2.Van Belle E, Vincent F, Labreuche J, et al. Balloon-expandable versus self-expanding transcatheter aortic valve replacement: a propensity-matched comparison from the FRANCE-TAVI registry. Circulation 2020;141:243-259. 3.Pibarot P, Salaun E, Dahou A, et al. Echocardiographic results of transcatheter versus surgical aortic valve replacement in low-risk patients: the PARTNER 3 trial. Circulation 2020;141:1527-1537.

  No potential conflict of interest relevant to this letter was reported. 2 References 1.Ngo A, Hassager C, Thyregod HGH, et al. Differences in left ventricular remodelling in patients with aortic stenosis treated with transcatheter aortic valve replacement with corevalve prostheses compared to surgery with porcine or bovine biological prostheses. Eur Heart J Cardiovasc Imaging 2018;19:39-46. 2.Lindman BR, Stewart WJ, Pibarot P, et al. Early regression of severe left ventricular hypertrophy after transcatheter aortic valve replacement is associated with decreased hospitalizations. JACC Cardiovasc Interv 2014;7:662-673.

	Arif	A.	Khokhar,	B.M.,	B.Ch.
	Francesco		Giannini,		M.D.
	Antonio		Colombo,		M.D.
	Maria	Cecilia	Hospital,	Cotignola,	Italy
	arifkhokhar@doctors.org.uk


Yixiao Wang

Zied Bouraoui
email: zied.bouraoui@cril.fr

Luis Espinosa Anke

Steven Schockaert
email: schockaerts1@cardiff.ac.uk

Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection

Keywords: {research, professor, science, education, institute} beneficiary, creditor, investor {protein, disease, medical, cancer, cells} lymphocyte, macrophage, axon, astrocyte, organelle {food, plant, water, gas, power, oil} electrode, electrolyte, cathode, anode, substrate {physics, mathematics, space, ngc, theory} surface, torus, mesh, grid, cone port {station, building, railway, historic, church} harbor, software, data, forewings} link, gateway, router, line, socket {game, book, novel, story, reception} version, remake, compilation, patch, modification bulb {station, building, railway, historic, church} lamp, transformer, dynamo, projector, lighting {protein, disease, medical, cancer, cells} epithelium, ganglion, nucleus, gland, cortex {species, genus, described, description, flowers} rootstock, fern, vine, tuber, clover mail {station, building, railway, historic, church} cargo, grain, baggage, coal, livestock {game, book, novel, story, reception} paper, jewelry, telemarketing, spam, wiretap, internet, money fingerprint {radio, station, fm, software, data, forewings} signature, checksum, bitmap, texture, {game, book, novel, story, reception} cadaver, skull, wiretap, body, tooth {party, election, minister, elected, elections} wiretap, forensics, postmortem, polygraph, check sky {greek, ancient, castle, king, roman} underworld, sun, afterlife, zodiac, moon {river, lake, mountain, island, village} horizon, ocean, earth, sun, globe {physics, mathematics, space, ngc, theory} ionosphere, sun, globe, earth, heliosphere strength {food, plant, water, gas, power} stiffness, ductility, hardness, permeability, viscosity {game, book, novel, story, reception} intelligence, agility, charisma, power, telepathy {army, regiment, navy, ship, air} morale, firepower, resistance, force, garrison noon {physics, mathematics, space, ngc, theory} declination, night, equinox, perihelion, latitude {army, regiment, navy, ship, air} dawn, sunset, night, morning, shore galaxy {physics, mathematics, space, ngc, theory} nebula, quasar, pulsar, nova, star {game, book, novel, story, reception} globe, future, world, planet, nation

One of the long-standing challenges in lexical semantics consists in learning representations of words which reflect their semantic properties. The remarkable success of word embeddings for this purpose suggests that highquality representations can be obtained by summarizing the sentence contexts of word mentions. In this paper, we propose a method for learning word representations that follows this basic strategy, but differs from standard word embeddings in two important ways. First, we take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts. Second, rather than learning a word vector directly, we use a topic model to partition the contexts in which words appear, and then learn different topic-specific vectors for each word. Finally, we use a taskspecific supervision signal to make a soft selection of the resulting vectors. We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.

Introduction

In the last few years, contextualized language models (CLMs) such as BERT [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF] have largely replaced the use of static (i.e. noncontextualized) word vectors in many Natural Language Processing (NLP) tasks. However, static word vectors remain important in applications where word meaning has to be modelled in the absence of (sentence) context. For instance, static word vectors are needed for zero-shot image classification [START_REF] Socher | Zero-shot learning through cross-modal transfer[END_REF] and zero-shot entity typing [START_REF] Ma | Label embedding for zero-shot fine-grained named entity typing[END_REF], for ontology alignment [START_REF] Kolyvakis | Deepalignment: Unsupervised ontology matching with refined word vectors[END_REF] and completion [START_REF] Li | Ontology completion using graph convolutional networks[END_REF], taxonomy learning [START_REF] Bordea | SemEval-2015 task 17: Taxonomy extraction evaluation (TExEval)[END_REF][START_REF] Bordea | Semeval-2016 task 13: Taxonomy extraction evaluation (texeval-2)[END_REF], or for representing query terms in information retrieval systems [START_REF] Nikolaev | Joint word and entity embeddings for entity retrieval from a knowledge graph[END_REF]. Moreover, [START_REF] Liu | Towards better context-aware lexical semantics: Adjusting contextualized representations through static anchors[END_REF] recently found that static word vectors can complement CLMs, by serving as anchors for contextualized vectors, while [START_REF] Alghanmi | Combining bert with static word embeddings for categorizing social media[END_REF] found that incorporating static word vectors could improve the performance of BERT for social media classification.

Given the impressive performance of CLMs across many NLP tasks, a natural question is whether such models can be used to learn highquality static word vectors, and whether the resulting vectors have any advantages compared to those from standard word embedding models [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Pennington | GloVe: Global vectors for word representation[END_REF]. A number of recent works have begun to explore this question [START_REF] Ethayarajh | How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings[END_REF][START_REF] Bommasani | Interpreting pretrained contextualized representations via reductions to static embeddings[END_REF][START_REF] Vulic | Probing pretrained language models for lexical semantics[END_REF]. Broadly speaking, the idea is to construct a static word vector for a word w by randomly selecting sentences in which this word occurs, and then averaging the contextualized representations of w across these sentences.

Since it is not usually computationally feasible to run the CLM on all sentences mentioning w, a sample of such sentences has to be selected. This begs the question: how should these sentences be chosen? In the aforementioned works, sentences are selected at random, but this may not be optimal. If we want to use the resulting word vectors in downstream tasks such as zero-shot learning or ontology completion, we need vectors that capture the salient semantic properties of words. Intuitively, we should thus favor sentences that best reflect these properties. For instance, many of the mentions of the word banana on Wikipedia are about the cultivation and export of bananas, and about the specifics of particular banana cultivars. By learning a static word vector from such sentences, we may end up with a vector that does not reflect our commonsense understanding of bananas, e.g. the fact that they are curved, yellow and sweet.

The main aim of this paper is to analyze to what extent topic models such as Latent Dirichlet Allocation [START_REF] Blei | Latent dirichlet allocation[END_REF] can be used to address this issue. Continuing the previous example, we may find that the word banana occurs in Wikipedia articles on the following topics: economics, biology, food or popular culture. While most mentions might be in articles on economics and biology, it is the latter two topics that are most relevant for modelling the commonsense properties of bananas. Note that the optimal selection of topics is taskdependent, e.g. in an NLP system for analyzing financial news, the economics topic would clearly be more relevant. For this reason, we propose to learn a word vector for each topic separately. Since the optimal choice of topics is task-dependent, we then rely on a task-specific supervision signal to make a soft selection of these topic-specific vectors.

Another important question is how CLMs should be used to obtain contextualized word vectors. Given a sentence mentioning w, a model such as BERT-base constructs 12 vector representations of w, i.e. one for each layer of the transformer stack. Previous work has suggested to use the average of particular subsets of these vectors. In particular, [START_REF] Vulic | Probing pretrained language models for lexical semantics[END_REF] found that lexical semantics is most prevalent in the representations from the early layers, and that averaging vectors from the first few layers seems to give good results on many benchmarks. On the other hand, these early layers are least affected by the sentence context [START_REF] Ethayarajh | How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings[END_REF], hence such strategies might not be suitable for learning topic-specific vectors. We therefore also explore a different strategy, which is to mask the target word in the given sentence, i.e. to replace the entire word by a single [MASK] token, and to use the vector representation of this token at the final layer. The resulting vector representations thus specifically encode what the given sentence reveals about the target word, making this a natural strategy for learning topic-specific vectors.

Note that there is a clear relationship between this latter strategy and CBOW [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]: where in CBOW the vector representation of w is obtained by averaging the vector representations of the context words that co-occur with w, we similarly represent words by averaging context representations. The main advantage compared to CBOW thus comes from the higher-quality context encodings that can be obtained using CLMs. The main challenge, as already mentioned, is that we cannot consider all the mentions of w, whereas this is typically feasible for CBOW (and other standard word embedding models). Our contributions can be summarized as follows1 :

• We analyze different strategies for deriving word vectors from CLMs, which rely on sampling mentions of the target word from a text collection.

• We propose the use of topic models to improve how these mentions are sampled. In particular, rather than learning a single vector representation for the target word, we learn one vector for each sufficiently relevant topic.

• We propose to construct the final representation of a word w as a weighted average of different vectors. This allows us to combine multiple vectors without increasing the dimensionality of the final representations. We use this approach for combining different topicspecific vectors and for combining vectors from different transformer layers.

Related Work

A few recent works have already proposed strategies for computing static word vectors from CLMs. While Ethayarajh (2019) relied on principal components of individual transformer layers for this purpose, most approaches rely on averaging the contextualised representations of randomly selected mentions of the target word [START_REF] Bommasani | Interpreting pretrained contextualized representations via reductions to static embeddings[END_REF][START_REF] Vulic | Probing pretrained language models for lexical semantics[END_REF]. Several authors have pointed out that the representations obtained from early layers tend to perform better in lexical semantics probing tasks. However, [START_REF] Bommasani | Interpreting pretrained contextualized representations via reductions to static embeddings[END_REF] found that the optimal layer depends on the number of sampled mentions, with later layers performing better when a large number of mentions is used. Rather than fixing a single layer, [START_REF] Vulic | Probing pretrained language models for lexical semantics[END_REF] advocated averaging representations from several layers. Note that none of the aforementioned methods uses masking when computing contextualized vectors. This means that the final representations may have to be obtained by pooling different word-piece vectors, usually by averaging them.

As an alternative to using topic models, Chronis and Erk (2020) cluster the contextual word vectors, obtained from mentions of the same word. The resulting multi-prototype representation is then used to compute word similarity in an adaptive way. Along similar lines, Amrami and Goldberg (2019) cluster contextual word vectors for word sense induction. [START_REF] Thompson | Topic modeling with contextualized word representation clusters[END_REF] showed that clustering the contextual representations of a given set of words can produce clusters of semantically related words, which were found to be similar in spirit to LDA topics. The idea of learning topic-specific representations of words has been extensively studied in the context of standard word embeddings [START_REF] Liu | Topical word embeddings[END_REF][START_REF] Li | Generative topic embedding: a continuous representation of documents[END_REF][START_REF] Shi | Jointly learning word embeddings and latent topics[END_REF][START_REF] Zhu | A neural generative model for joint learning topics and topicspecific word embeddings[END_REF]. To the best of our knowledge, learning topic-specific word representations using CLMs has not yet been studied. More broadly, however, some recent methods have combined CLMs with topic models. For instance, [START_REF] Peinelt | tBERT: Topic models and BERT joining forces for semantic similarity detection[END_REF] use such a combination for predicting semantic similarity. In particular they use the LDA or GSDMM topic distribution of two sentences to supplement their BERT encoding. Finally, [START_REF] Bianchi | Pre-training is a hot topic: Contextualized document embeddings improve topic coherence[END_REF] suggested using sentence embeddings from SBERT [START_REF] Reimers | Sentence-BERT: Sentence embeddings using Siamese BERTnetworks[END_REF] as input to a neural topic model, with the aim of learning more coherent topics.

Constructing Word Vectors

In Section 3.1, we first describe different strategies for deriving static word vectors from CLMs. Section 3.2 subsequently describes how we choose the most relevant topics for each word, and how we sample topic-specific word mentions. Finally, in Section 3.3 we explain how the resulting topicspecific representations are combined to obtain task-specific word vectors.

Obtaining Contextualized Word Vectors

We first briefly recall the basics of the BERT contextualised language model. BERT represents a sentence s as a sequence of word-pieces w 1 ..., w n . Frequent words will typically be represented as a single word-piece, but in general, word-pieces may correspond to sub-word tokens. Each of these word-pieces w is represented as an input vector, which is constructed from a static word-piece embedding w 0 (together with vectors that encode at which position in the sentence the word appears, and in which sentence). The resulting sequence of word-piece vectors is then fed to a stack of 12 (for BERT-base) or 24 (for BERT-large) transformer layers. Let us write w s i for the representation of word-piece w in the i th transformer layer. We will refer to the representation in the last layer, i.e. w s 12 for BERT-base and w s 24 for BERT-large, as the output vector. When BERT is trained, some of the word-pieces are replaced by a special [MASK] token. The corresponding output vector then encodes a prediction of the masked word-piece. Given a sentence s in which the word w is mentioned, there are several ways in which BERT and related models can be used to obtain a vector representation of w. If w consists of a single word-piece, a natural strategy is to feed the sentence s as input and use the output vector as the representation of w. However, several authors have found that it can be beneficial to also take into account some or all of the earlier transformer layers, where finegrained word senses are mostly captured in the later layers [START_REF] Reif | Visualizing and measuring the geometry of BERT[END_REF] but word-level lexical semantic features are primarily found in the earlier layers [START_REF] Vulic | Probing pretrained language models for lexical semantics[END_REF]. For this reason, we will also experiment with models in which the vectors w s 1 , ..., w s 12 (or w s 1 , ..., w s 24 in the case of BERT-large) are all used. In particular, our model will construct a weighted average of these vectors, where the weights will be learned from training data (see Section 3.3). For words that consist of multiple word-pieces, following common practice, we compute the representation of w as the average of its word-piece vectors. For instance, this strategy was found to outperform other aggregation strategies in [START_REF] Bommasani | Interpreting pretrained contextualized representations via reductions to static embeddings[END_REF].

We will also experiment with a strategy that relies on masking. In this case, the word w is replaced by a single [MASK] token (even if w would normally be tokenized into more than one wordpiece). Let us write m s w for the output vector corresponding to this [MASK] token. Since this vector corresponds to BERT's prediction of what word is missing, this vector should intuitively capture the properties of w that are asserted in the given sentence. We can thus expect that these vectors m s w will be more sensitive to how the sentences mentioning w are chosen. Note that in this case, we only use the output layer, as the earlier layers are less likely to be informative.

To obtain a static representation of w, we first select a set of sentences s 1 , ..., s n in which w is mentioned. Then we compute vector representations w s 1 , ..., w sn of w from each of these sentences, using any of the aforementioned strategies. Our final representation w is then obtained by averaging these sentence-specific representations, i.e.:

w = n i=1 w s i n i=1 w s i

Selecting Topic-Specific Mentions

To construct a vector representation of w, we need to select some sentences s 1 , ..., s n mentioning w. While these sentences are normally selected randomly, our hypothesis in this paper is that purely random strategies may not be optimal. Intuitively, this is because the contexts in which a given word w is most frequently mentioned might not be the most informative ones, i.e. they may not be the contexts which best characterize the properties of w that matter for a given task. To test this hypothesis, we experiment with a strategy based on topic models. Our strategy relies on the following steps:

1. Identify the topics which are most relevant for the target word w;

2. For each of the selected topics t, select sentences s t 1 , ..., s t n mentioning w from documents that are closely related to this topic.

For each of the selected topics t, we can then use the sentences s t 1 , ..., s t n to construct a topic-specific vector w t , using any of the strategies from Section 3.1. The final representation of w will be computed as a weighted average of these topic-specific vectors, as will be explained in Section 3.3.

We now explain these two steps in more detail. First, we use Latent Dirichlet Allocation (LDA) [START_REF] Blei | Latent dirichlet allocation[END_REF] to obtain a representation of each document d in the considered corpus as a multinomial distribution over m topics. Let us write τ i (d) for the weight of topic i in the representation of document d, where m i=1 τ i (d) = 1. Suppose that the word w is mentioned N w times in the corpus, and let d w j be the document in which the j th mention of w occurs. Then we define the importance of topic i for word w as follows:

τ i (w) = 1 N w Nw j=1 τ i (d w j ) (1)
In other words, the importance of topic i for word w is defined as the average importance of topic i for the documents in which w occurs. To select the set of topics T w that are relevant to w, we rank the topics from most to least important and then select the smallest set of topics whose cumulative importance is at least 60%, i.e. T w is the smallest set of topics such that t i ∈Tw τ i (w) ≥ 0.6.

For each of the topics t i in T w we select the corresponding sentences s t 1 , ..., s t n as follows. We rank all the documents in which w is mentioned according to τ i (d). Then, starting with the document with the highest score (i.e. the document for which topic i is most important), we iterate over the ranked list of documents, selecting all sentences from these documents in which w is mentioned, until we have obtained a total of n sentences.

Combining Word Representations

Section 3.1 highlighted a number of strategies that could be used to construct a vector representation of a target word w. As mentioned before, it can be beneficial to combine vector representations from different transformer layers. To this end, we propose to learn a weighted average of the different input vectors, using a task specific supervision signal. In particular, let w 1 , ..., w k be the different vector representations we have available for word w (e.g. the vectors from different transformer layers). To combine these vectors, we compute a weighted average as follows:

λ i = exp(a i ) k j=1 exp(a i ) (2)
w = i λ i w i i λ i w i (3)
where the scalar parameters a 1 , ...a k ∈ R are jointly learned with the model in which w is used. Another possibility would be to concatenate the input vectors w 1 , ..., w k . However, this significantly increases the dimensionality of the word representations, which can be challenging in downstream applications. In initial experiments, we also confirmed that this concatenation strategy indeed under-performs the use of weighted averages. If topic-specific vectors are used, we also want to compute a weighted average of the available vectors. However, (2)-( 3) cannot be used in this case, because the set of topics for which topicspecific vectors are available differs from word to word. Let us write w i topic for the representation of word w that was obtained for topic t i , where we assume w i topic = 0 if t i / ∈ T w . We then define:

µ w i = exp(b i ) • 1[t i ∈ T w ] k j=1 exp(b i ) • 1[t j ∈ T w ]
(4)

w topic = i µ w i w i topic i µ w i w i topic (5)
where 1[t i ∈ T w ] = 1 if topic t i is considered to be relevant for word w (i.e. t i ∈ T w ), and 1[t i ∈ T w ] = 0 otherwise. Note that the softmax function in (4) relies on the scalar parameters b 1 , ..., b k ∈ R, which are independent of w. However, the softmax is selectively applied to those topics that are relevant to w, which is why the resulting weight µ w i is dependent on w, or more precisely, on the set of topics T w .

Evaluation

We compare the proposed strategy with standard word embeddings and existing CLM-based strategies. In Section 4.1 we first describe our experimental setup. Section 4.2 then provides an overview of the datasets we used for the experiments, where we focus on lexical classification benchmarks. These benchmarks in particular allow us to assess how well various semantic properties can be predicted from the word vectors. The experimental results are discussed in Section 4.3 and a qualitative analysis is presented in Section 4.4.

Experimental Setup

We experiment with a number of different strategies for obtaining word vectors:

C last We take the vector representation of w from the last transformer layer (i.e. w s 12 or w s 24 ).

C input We take the input embedding of w (i.e. w 0 ).

C avg We take the average of w 0 , w s 1 , ..., w s 12 for the base models and w 0 , w s 1 , ..., w s 24 for the large models.

C all

We use all of w 0 , w s 1 , ..., w s 12 as input for the base models, and all of w 0 , w s 1 , ..., w s 24 for the large models. These vectors are then aggregated using ( 2)-(3), i.e. we use a learned soft selection of the transformer layers.

C mask We replace the target word by [MASK] and use the corresponding output vector.

For words consisting of more than one word-piece, we average the corresponding vectors in all cases, except for C mask where we always end up with a single vector (i.e. we replace the entire word by a single [MASK] token). We also consider three variants that rely on topic-specific vectors:

T last We learn topic-specific vectors using the last transformer layers. These vectors are then used as input to (4)-( 5).

T avg Similar to the previous case but using the average of all transformer layers.

T mask Similar to the previous cases but using the output vector of the masked word mention.

Furthermore, we consider variants of T last , T avg and T mask in which a standard (i.e. unweighted) average of the available topic-specific vectors is computed, instead of relying on ( 4)-( 5). We will refer to these averaging-based variants as A last , A avg and A mask . As baselines, we also consider the two Word2vec models [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]:

SG 300-dimensional Skip-gram vectors trained on a May 2016 dump of the English Wikipedia, using a window size of 5 tokens, and minimum frequency threshold of 10.

CBOW 300-dimensional Continuous Bag-of-Words vectors trained on the same corpus and with the same hyperparameters as SG.

We show results for four pre-trained CLMs [START_REF] Devlin | BERT: pre-training of deep bidirectional transformers for language understanding[END_REF][START_REF] Liu | Roberta: A robustly optimized BERT pretraining approach[END_REF]: BERT-baseuncased, BERT-large-uncased, RoBERTa-baseuncased, RoBERTa-large-uncased2 . As the corpus for sampling word mentions, we used the same Wikipedia dump as for training the word embeddings models. For C mask , C last , C avg and C all we selected 500 mentions. For the topic-specific strategies (T last , T avg and T mask ) we selected 100 mentions per topic. To obtain the topic assignments, we used Latent Dirichlet Allocation [START_REF] Blei | Latent dirichlet allocation[END_REF] with 25 topics. We set α = 0.0001 to restrict the total number of topics attributed to a document, and use default values for the other hyper-parameters3 .

To select the relevant topics for a given word w, we find the smallest set of topics whose cumulative importance score τ i (w) is at least 60%, with a maximum of 6 topics. In the experiments, we restrict the vocabulary to those words with at least 100 occurrences in Wikipedia.

Datasets

For the experiments, we focus on a number of lexical classification tasks, where categories of individual words need to be predicted. In particular, we used two datasets which are focused on commonsense properties (e.g. and the CSLB Concept Property Norms5 . We furthermore used the WordNet supersenses dataset6 , which groups nouns into broad categories (e.g. human). Finally, we also used the BabelNet domains dataset7 [START_REF] Camacho | BabelDomains: Large-scale domain labeling of lexical resources[END_REF], which assigns lexical entities to thematic domains (e.g. music).

In our experiments, we have only considered properties/classes for which sufficient positive examples are available, i.e. at least 10 for McRae, 30 for CSLB, and 100 for WordNet supersenses and BabelNet domains. For the McRae dataset, we used the standard training-validation-test split. For the other datasets, we used random splits of 60% for training, 20% for tuning and 20% for testing. An overview of the datasets is shown in Table 2.

For all datasets, we consider a separate binary classification problem for each property and we report the (unweighted) average of the F1 scores for the different properties. To classify words, we feed their word vector directly to a sigmoid classification layer. We optimise the network using AdamW with a cross-entropy loss. The batch size and learning rate were tuned, with possible values chosen from 4,8,16 and 0.01, 0.005, 0.001, 0.0001 respectively. Note that for C all and the topic-specific variants, the classification network jointly learns the parameters of the classification layer and the attention weights in (2) and (4) for combining the input vectors.

Results

The results are shown in Table 1. We consistently see that the topic-specific variants outperform the different C-variants by a small margin. This confirms our main hypothesis, namely that using topic models to determine how context sentences are selected has a material effect on the quality of the resulting word representations. Among the Cvariants, the best results are obtained by C mask . None of the three T-variants consistently outperforms the others. Surprisingly, the A-variants outperform the corresponding T-variants in several cases. This suggests that the outperformance of the topic-specific vectors primarily comes from the fact that the context sentences for each word were sampled in a more balanced way (i.e. from documents covering a broader range of topics), rather than from the ability to adapt the topic weights based on the task. This is a clear benefit for applications, as the A-variants allow us to simply represent each word as a static word vector.

The performance of SG and CBOW is still strong. In particular, these traditional word embedding models outperform all of the C-variants, as well as the T and A variants in BabelNet Domains. This seems to be related, at least in part, to the lower dimensionality of these vectors. The classification network has to be learned from a rather small number of examples, especially for McRae and CSLB. Having 768 or 1024 dimensional input vectors can be problematic in such cases. To analyse this effect, we used Principal Component Analysis (PCA) to reduce the dimensionality of the CLM-derived vectors to 300. For this experiment, we focused in particular on C mask and T mask . The results are also shown in Table 1 as C mask -PCA and T mask -PCA. As can be seen, this dimensionality reduction step has a clearly beneficial effect, with T mask -PCA outperforming all baselines, except for the BabelNet domains benchmark. The latter benchmark is focused on thematic similarity rather than semantic properties, which the CLMbased representations seem to struggle with.

Qualitative analysis

Topic-specific vectors can be expected to focus on different properties, depending on the chosen topic. In this section, we present a qualitative analysis in support of this view. In Table 3 we list, for a sample of words from the WordNet supersenses dataset, the top 5 nearest neighbours per topic in terms of cosine similarity. For this analysis, we used the BERT-base masked embeddings. We can see that for the word 'partner', its topic-specific em- beddings correspond to its usage in the context of 'finance', 'stock market' and 'fiction'. These three embeddings roughly correspond to three different senses of the word 8 . This de-conflation or implicit disambiguation is also found for words such as 8 In fact, we can directly pinpoint these vectors to the following WordNet [START_REF] Miller | Wordnet: a lexical database for English[END_REF] senses: partner.n.03, collaborator.n.03 and spouse.n.01.

'cell', 'port', 'bulb' or 'mail', which shows a striking relevance of the role of mail in the election topic, being semantically similar in the corresponding vector space to words such as 'telemarketing', 'spam' or 'wiretap'. In the case of 'fingerprint', we can also see some implicit disambiguation (distinguishing between fingerprinting in computer science, as a form of hashing, and the more traditional sense). However, we also see a more topical distinction, revealing differences between the role played by fingerprints in fictional works and forensic research. This tendency of capturing different contexts is more evidently shown in the last four examples. First, for 'sky' and 'strength', the topicwise embeddings do not represent different senses Table 3: Nearest neighbours of topic-specific embeddings for a sample of words from the WordNet SuperSenses dataset, using BERT-base embeddings. The top 6 selected samples illustrate clear topic distributions per word sense, and the bottom 4 also show topical properties within the same sense. The most relevant words for each topic are shown under the TOPIC column.

of these words, but rather indicate different types of usage (possibly related to cultural or commonsense properties). Specifically, we see that the same sense of 'sky' is used in mythological, landscaping and geological contexts. Likewise, 'strength' is clustered into different mentions, but while this word also preserves the same sense, it is clearly used in different contexts: physical, as a human feature, and in military contexts. Finally, 'noon' and 'galaxy' (which only occur in two topics), also show this topicality. In both cases, we have representations that reflect their physics and everyday usages, for the same senses of these words.

As a final analysis, In Figure 1 we plot a twodimensional PCA-reduced visualization of selected words from the McRae dataset, using two versions of the topic-specific vectors: T mask and T last . In both cases, BERT-base was used to obtain the vectors. We select four pairs of concepts which are topically related, which we plot with the same datapoint marker (animals, plants, weapons and musical instruments). For T last , we can see that the different topic-specific representations of the same word are clustered together, which is in accordance with the findings from Ethayarajh (2019). For T mask , we can see that the representations of words with similar properties (e.g. cheetah and hyena) become more similar, suggesting that T mask is more tailored towards modelling the semantic properties of words, perhaps at the expense of a reduced ability to differentiate between closely related words. The case of turnip and peach is particularly striking, as the vectors are clearly separated in the T last plot, while being clustered together in the T mask plot.

Conclusions

We have proposed a strategy for learning static word vectors, in which topic models are used to help select diverse mentions of a given target word and a contextualized language model is subsequently used to infer vector representations from the selected mentions. We found that selecting an equal number of mentions per topic substantially outperforms purely random selection strategies. We also considered the possibility of learning a weighted average of topic-specific vector representations, which in principle should allow us to "tune" word representations to different tasks, by learning task-specific topic importance weights. However, in practice we found that a standard average of the topic specific vectors leads to a comparable performance, suggesting that the outperformance of our vectors comes from the fact that they are obtained from a more diverse set of contexts.

  dangerous): the extension of the McRae feature norms dataset (McRae et al., 2005) that was introduced by Forbes et al. (2019) 4

Figure 1 :

 1 Figure 1: BERT-base topic-specific vectors when using the output vectors without using masking (left) and with masking (right). Words have been selected from the McRae dataset.

Table 1 :

 1 Amask 63.1 53.9 59.2 41.4 63.2 56.8 60.6 41.5 64.0 55.3 60.6 40.8 63.4 57.3 62 42.3 Alast 62.8 52.4 59.6 44.4 61.4 55.5 60.6 46.7 55.7 36.8 56.5 39. 7 59.6 47.8 59.7 42.5 Aavg 61.3 49.7 57.9 44.4 63.3 52.2 59.4 43.8 57.6 40.6 56.4 39.8 59.4 47.3 58.5 42.4 Cmask-PCA 61.8 52.6 58.8 41.2 62.3 53.2 60.1 41.6 61.5 52.6 59.2 40.3 62.2 51.5 59.1 40.5 Tmask-PCA 63.3 56.2 62.6 46.9 64.4 57.3 62.6 48.0 61.6 55.8 62.5 46.0 65.4 56.3 64.1 46.4

		BERT-base	BERT-large	RoBERTa-base	RoBERTa-large
		MC CS SS BD	MC CS SS BD	MC CS SS BD	MC CS SS BD
	SG	59.6 54.5 55.6 49.1 59.6 54.5 55.6 49.1 59.6 54.5 55.6 49.1 59.6 54.5 55.6 49.1
	CBOW	61.1 50.6 48.4 45.0 61.1 50.6 48.4 45.0 61.1 50.6 48.4 45.0 61.1 50.6 48.4 45.0
	Cmask	60.8 51.7 59.7 42.6 61.8 53.4 59.5 42	62.5 51.8 58.5 40	61.3 53.2 59.2 42.8
	Clast	60.0 51.4 59 46.1 56.2 53.4 59 42.3 56.5 43.4 58.2 42.1 57.9 47.7 58.8 41.9
	C input	58.8 40.1 50.2 40.3 57.2 42 51.7 40.2 45.8 24.1 44.4 37.9 41.2 20.6 52.6 40.0
	Cavg	59.9 49.6 59.1 44.2 60.0 47.1 58.9 42.5 53.7 40.7 50.2 40.3 59.5 47.4 58.8 42.9
	Call	59.9 51.2 59.5 46.4 61.7 50.7 58.4 42.6 45.3 39.3 52.6 36.7 48.2 40.2 56.6 40.4
	Tmask	60.9 54.1 60.5 45.8 62.8 55 61.4 46.2 58.6 49.4 56.7 42.1 59.2 50.4 57.2 43.3
	Tlast	63.0 51.8 59.7 47.3 62.1 55.8 61.6 49.2 52.3 43.9 54.6 41.3 62.1 48.8 59.5 45.1
	Tavg	61.0 52.7 59.6 43.4 65.2 54.8 60.7 48.4 54.5 39.9 55.9 41.5 59.5 47.4 60.0 45.2

Results of lexical feature classification experiments for the extended McRae feature norms (MC), CSLB norms (CS), WordNet Supersenses (SS) and BabelNet domains (BD). Results are reported in terms of F1 (%).

Table 2 :

 2 Overview of the considered datasets.

	Dataset	Type	Words Properties
	McRae	Commonsense	475	49
	CSLB	Commonsense	570	54
	WN supersenses Taxonomic	24,324	24
	BN domains	Topical	43,319	34

All code and data to replicate our experiments is available at https://github.com/Activeyixiao/ topic-specific-vector/.

We used the implementations from https://github. com/huggingface/transformers.

We used the implementation from https: //radimrehurek.com/gensim/wiki.html.

https://github.com/mbforbes/ physical-commonsense

https://cslb.psychol.cam.ac.uk/ propnorms

https://wordnet.princeton.edu/ download

http://lcl.uniroma1.it/babeldomains/

Acknowledgments

This work was performed using the computational facilities of the Advanced Research Computing @ Cardiff (ARCCA) Division, Cardiff University and using HPC resources from GENCI-IDRIS (Grant 2021-[AD011012273]).