Viola C Orcid

D'mello

Gildas Goldsztejn

Venkateswara Rao Mundlapati

Valérie Brenner

Eric Gloaguen

Florence Charnay-Pouget

David J Aitken
email: david.aitken@universite-paris-saclay.fr

Michel Mons
email: michel.mons@cea.fr

Keywords: Conformation analysis, Peptides, Asx-Turn, Laser spectroscopy, Quantum chemistry

Models of asparagine-containing dipeptides, specifically designed to favor intrinsic folding into an Asx-turn, were characterized both theoretically, using quantum chemistry, and experimentally, using laser spectroscopy in the gas phase. Both approaches provided evidence for the spontaneous folding of both the Asn-Ala and Asn-Gly dipeptide models into the most stable Asx-turn, a conformation stabilized by a C10 H-bond which was very similar to a type II' -turn. In parallel, analysis of the Asx-turns implicating asparagine in crystallized protein structures in the Protein Data Bank revealed a sequence-dependent behavior. In Asn-Ala sequences, the Asx-turn was found in conjunction with a type I -turn for which the first of the four defining residues was Asn. The observation that the Asx turn in these structures is mostly of type II' (i.e. its most stable innate structure) suggests that this motif may foster the formation and/or enhance the stability of the backbone -turn. In contrast, the Asx turns observed in Asn-Gly sequences extensively adopted a type II Asx-turn structure, suggesting that their formation should be ascribed to other factors, such as hydration. The fact that the Asx-turn in a Asn-Gly sequence is also often found in combination with a hydrated -bulge supports the premise that a Asn-Gly sequence might efficiently promote the formation of the -bulge secondary structure. Details of organic synthesis, quantum chemistry, UV and IR spectroscopy and data mining are provided in a Supporting Information file available on the web. Additional data that support the findings of this study are available from the corresponding authors upon reasonable request.

Introduction

Among the secondary structures of proteins and peptides, turns occupy a prominent position. Their ubiquity stems from their unique capacity to enable the reversal of the protein chain direction. [1] Their conservation within protein families, together with their robustness vis-à-vis mutations, has countenanced the deduction that they play a key role in folding mechanisms. [1e, 2] A typology of turn secondary structures has been established, based on surveys of X-ray structures in the Protein Data Bank (PDB). [1,3] Ranging over four residues, labelled i to i+3, -turns (Figure 1, top) are characterized by specific H-bonding between backbone NH and CO groups four residues apart (referred to as a C10 H-bond, in accordance with the size of the pseudo-cycle formed by the H-bond). There are four different types of -turn, [4] whose occurrence depends on whether side chains are present (i.e. glycine vs. other amino acids) and the configurations of the chiral residues involved. [1] Early PDB structural surveys also reported the occurrence of an alternative H-bonding pattern, not involving a backbone CO but implicating instead the side chain of either an asparagine (Asn) or an aspartic acid (Asp) residue (with the latter in its carboxylate form under physiological conditions) which acted as the hydrogen bond acceptor. [1a, c, 5] In these H-bonds, the carboxamide oxygen atom of the Asn side chain (or one of the carboxylate oxygen atoms of the Asp side chain) interacts with the next-but-one NH site along the backbone, giving rise to a C10 H-bond bearing a marked resemblance to a -turn; these structures are referred to as Asxturns (Figure 1, bottom). Detailed analyses, however, have shown that Asx-turns are rarely isolated structures and are most often found in combination with another secondary structure, essentially a -turn, [1c, 5] with which they form an intertwined C10 H-bond network, labelled an Asx--turn. [6] The coexistence of these features, of course, raises the question of anteriority: which one precedes the other during the folding process and thereby serves as the folding nucleus of the structure. In this context, Asx-based secondary structures were suggested to account for the biased distribution of residues in the most frequently found -turns, viz. type I, [2b, 5c, 7] and in particular for the over-representation (by a factor of 2-3) of the polar residues Asp and Asn in position i of a -turn. [5c, 7b] It was thus surmised that if Asx-turns preexisted -turns, they could foster the formation of the latter; alternatively, if formed subsequently they could enhance -turn stability. The over-representation (to a similar extent) of an Asx residue in a second location, namely position i+2 of type I -turns, was also reported [5c, 7b] but was not discussed in detail. It is worth noting here that the type I -turn structures considered in these studies did not distinguish isolated turns from those embedded in larger secondary structures, such as -hairpins [8] or -bulges. [9] For these latter, the presence of a Gly residue in position i+3 was considered as strongly favorable, [9b, 10] suggesting that the Asx-Gly (i+2, i+3) sub-sequence of the bulge may adopt a specific motif, again favoring formation of the -turn or stabilizing it, depending on the folding chronology. Despite a detailed study of specific structures comprised of the most abundant residue at each of the four turn positions taken individually, [2c] such as the so-called consensus sequence Asx-Pro-Asx-Gly, no rationalisation has been proposed so far to the best of our knowledge.

The structural diversity of Asx-turns was described in detail, based on a PDB survey, [11] which demonstrated a marked similarity between the typology of -turns and Asx-turns; the latter closely resembled several types of the former, with a preference for type II', but again this study did not take account of the backbone or secondary structure environment of the Asx-turn considered.

In parallel with these investigations on crystallized proteins, the issue of the intrinsic formation of Asx-turn in two-or three-residue peptide chains has been studied in weakly polar solutions using IR and NMR spectroscopy, providing evidence for C10 H-bonding. [12] Crystal structures of these models were also obtained, providing evidence for the formation of Asx-turns, although very often in combination with other main chainmain chain or intermolecular interactions, which unfortunately blurred the inherent features; it transpired that the constitutive stability could only be demonstrated for the Asx-Pro sequence. [12] The authors also proposed, on the basis of theoretical considerations, that several types of Asx-turn might exist; [12] nonetheless, the assessment of the innate stability of these secondary features through molecular modelling using relevant methods ─ namely quantum chemistry ─ has not been addressed systematically to date. The formation of Asxturns has been probed theoretically in a few studies using model peptides containing an Asn residue; [13] the turns were indeed present in stable conformations of these models, albeit invariably combined with other structural motifs. However, no experimental evidence for such structures was obtained, whether in solution [13a] or in the gas phase. [13b] Over the last decade, gas phase isolation has been shown to provide useful insight into the folding properties of flexible molecules, including peptides, whether natural or synthetic. [14] Investigations conducted by a supersonic expansion followed by a laser study under isolated conditions provide invaluable information on the folding propensities of molecular systems. The secondary structures typical of protein chains ─ -turns, turns, -sheets, -hairpins, short 310 helices ─ have all been detected in the gas phase using this approach, demonstrating the fundamental "folding" character of these features in vacuo. Comparison of experimental data (H-bond sensitive IR spectroscopy of each conformer together with relative abundances) with their theoretical counterparts, obtained at the relevant quantum chemistry level, enables cross-checking between the techniques used. So far, however, studies on the gas phase conformational preferences of Asn-containing peptides investigated using laser spectroscopy [13b, 15] have not provided experimental evidence for the formation of low energy structures based solely on an Asx-turn, presumably because the model peptide sequences that were employed also bore H-bonding sites that enabled -turn formation at the expense of Asxturn structures.

The present work therefore aims at investigating the intrinsic folding behavior of an Asn residue by studying truncated Asn-containing peptide model sequences wherein the formation of -turns or -turns is specifically excluded. Since the presence of a peptide bond at the N α -position of the Asn residue would favor the formation of a C10 β-turn, [15] we reasoned that our objective could be achieved by replacing the Asn N α -fragment with a benzyl group, while retaining an H-bond donor at the C-terminus in the form of a methylamide group. Two such sequences are addressed in this work, namely Asn-Ala and Asn-Gly, giving rise to the corresponding model compounds N-(2-benzylsuccinyl)alanine methylamide and N-(2-benzylsuccinyl)glycine methylamide (respectively referred to hereafter as Bn-NA (1) and Bn-NG (2); Figure 2). The benzyl group conveniently serves as the UV chromophore required by the laser spectroscopic techniques used. The experimental characterization of compounds 1 and 2 was carried out in the gas phase using UV and IR/UV double resonance spectroscopies. In parallel, the structures of these compounds and those of simplified analogs bearing a methyl group instead of the benzyl group ─ namely N-(2-methylsuccinyl)alanine methylamide and N-(2-methylsuccinyl)glycine methylamide (respectively referred to hereafter as Me-NA and Me-NG; Figure 2) ─ were investigated using relevant quantum chemistry techniques, with the objective of characterizing low-energy side chain-backbone Asx-turn structures and identifying their typology by analogy with the corresponding -turns in a capped Ala-Ala or Ala-Gly sequence. These data were then used to interpret the experimental spectroscopic data and assign the experimentally-observed conformations. The H-bonds in the Asx-turn structures found using these procedures were also analyzed using a high-level theoretical characterization tool ─ namely Natural Bond Orbital (NBO) theory ─ which was recently applied successfully in an examination of amide-amide interactions in peptides. [16] Finally, the observed stable structures were compared to the Asx-turn structures found in proteins, especially those combined with -turns in which Asn occupies position i or i+2. The type of Asx-turn observed in proteins was found to depend upon the Asn environment, in particular the residue following Asn in the sequence, which leads us to propose different mechanistic roles for Asn-turns in the formation of the associated -turn structures.

Materials and Methods

Synthesis

The synthesis of Bn-NA (1) and Bn-NG (2) is shown in Scheme 1. Treatment of commercial (S)-2benzylsuccinic acid with benzyl bromide and DBU gave a ~2:1 mixture of the isomeric monoesters 3 and 4 which were separated chromatographically (38% and 16% isolated yields, respectively). Coupling of the former with the methylamide of either alanine or glycine (each prepared by deprotection of the corresponding Bocprotected derivative) using HATU/DIEA in DMF under microwave activation gave derivatives 5 and 6 (88% and 72% yields, respectively). Each of these derivatives was selectively deprotected by hydrogenolysis in the presence of a palladium catalyst and the resulting acid was converted into the corresponding carboxamide using di-tert-butyl dicarbonate and ammonium bicarbonate to furnish the target molecules 1 and 2. While the yield for the last two steps was nearly quantitative for the alanine derivative, the transformation was considerably less efficient for the glycine derivative (33% yield), with reverse phase HPLC purification being deemed appropriate. Scheme 1. Synthesis of the model peptides Bn-NA (1) and Bn-NG (2).

Quantum chemistry methods

A conformational search using the Macromodel Suite [START_REF] Macromodel | Schrödinger Release 2019-2013[END_REF] was carried out on the Bn-NA and Bn-NG systems, wherein conformations were sought with a relative stability of up to 20 kJ•mol -1 with respect to the global minimum. Those conformations that mimicked any of the four types of -turn (type I, type II, type I, and type II) were further submitted to a geometry optimization using quantum chemistry at the RI-B97-D3(BJ-abc)/def2-TZVPPD level of theory [18] using the Turbomole 7.3 suite. [START_REF]Turbomole V7.2 A Development of University of Karlsruhe and Forschungszentrum Karlsruhe GmbH[END_REF] The related compounds Me-NA and Me-NG were generated starting from any of the Bn-NA and Bn-NG conformations, substituting the phenyl ring by an H atom, then reoptimizing the structure. For all these conformations, harmonic vibrational frequencies were calculated at the same level of theory and were used to deduce the electronic energies corrected for zero-point vibrational energy ─ namely H at 0 K and Gibbs free energies at 298 K ─ as well as the theoretical vibrational spectra.

The NH stretch IR spectra were generated from a mode-dependent scaling procedure, using linear scaling factors (see details in Supporting Information Section 3) deduced from a previous fitting on a similar set of peptides, which provides a good agreement with experimental frequencies (within 20 cm -1 for NH stretches in NH•••O bonds). [14a] In the amide I and II regions, the scaling factors used (1.004 and 1.008, respectively) were chosen due to the good level of prediction they provided in previous studies. [20] [21] Detailed NBO analyses, [22] which are based on an expression of the electronic wavefunction in terms of a localized Lewis-type chemical bond depiction, were also performed for selected relevant conformations in order to provide a quantification of the stabilizing role of electron delocalization through H-bonding effects.

Analyses were carried out on the RI-B97-D3(BJ-abc)/def2-TZVPPD structures using the NBO module [23] of the Gaussian 16 software. [24] The NBO analysis was carried out at both HF/TZVPP and MP2/TZVPP levels. The donor and acceptor NBO occupancies considered in the following are those obtained at the MP2/TZVPP level whereas the HF level was used to calculate, for each donor NBO(i) and acceptor NBO(j), the E(2) stabilization energy through the second order perturbation theory:

𝐸(2) = ∆𝐸 = 𝑞 ( , ) | |
where qi is the donor orbital occupancy, F(i j) is the off-diagonal NBO Fock matrix element and 𝜀 (resp. 𝜀 ) is the diagonal element, i.e. the donor (acceptor) orbital energy; the threshold for considering the interactions as significant being equal to 0.2 kJ•mol -1 . Previous works have shown that the MP2/TZVPP level of theory used here provides a good compromise between accuracy and computation times. [16,25]

Gas phase laser spectroscopic methods

The UV and IR spectroscopy of each isolated compound was recorded using a setup and procedures that have been described earlier. [14] Briefly, each compound was vaporized by laser desorption from a pressed pellet containing the sample and graphite powder. The vapor was introduced into a supersonic jet of a He/Ne mixture (18 bars) generated by a pulsed valve (10 Hz). The jet was skimmed and the cooled molecules entered the source of a time-of-flight mass spectrometer (TOF-MS) where they interacted with the UV laser (frequencydoubled Continuum Nd:YAG-pumped Radiant dye laser, 1 mJ, operating at 10 Hz). The ions formed by resonant two-photon ionization were then accelerated towards the TOF-MS detector. The time-selected signal was measured and processed by an in-house LabView program. Conformation-specific IR spectra were recorded using the IR/UV double resonance technique, in which an IR laser sent before the UV laser depopulated the vibrational ground state level of the molecules. [START_REF] Rijs | Oomens in IR Spectroscopic Techniques to Study Isolated Biomolecules[END_REF] The UV laser being tuned on a transition of a well-defined conformer, scanning the IR laser and measuring the depletion of the UV-induced ion signal enabled us to record conformer-selective IR spectra. The IR beam was generated by a Continuum Nd:YAGpumped LaserVision OPO/OPA, operating between 3100 and 4000 cm -1 , whose idler beam was used for the NH stretch region. The amide I and II regions (1400-1800 cm -1 ) were covered by frequency difference mixing between signal and idler using a dedicated AgGaSe2 crystal module. For a better signal-to-noise ratio the IRinduced ion signal depletion was normalized to an IR-off ion signal, obtained from a second passage of the UV beam into the jet. [START_REF] Gloaguen | [END_REF] 3. Theoretical conformational landscape of Me-NA and Me-NG Quantum chemistry methods were first used to determine the Asx-turn structures adopted by the simplified Me-NA and Me-NG sequences together with their stability, in order to characterize their typology by comparison with relevant -turns, namely the capped peptides Ac-Ala-Ala-NHMe and Ac-Ala-Gly-NHMe.

Structures

For both Me-based sequences, geometry optimization at the quantum chemistry level revealed three types of Asx-turns, with limited differences between the two sequences. These turns were all characterized by a C10

H-bond bridging the Asn side chain carbonyl and the C-terminal peptide NH bond, but differed by the backbone torsions along the sequence, as indicated by the sets of dihedrals defined in Figure 2 and presented in Tables S1 andS2. Comparison of these structures with those of -turns in the capped dipeptides (Figure 1; Tables S1 andS2), obtained at the same level of theory, showed a striking resemblance, although less marked for type I (Figure 3), which led us to label the Asx-turns according to the -turns they mimic.

Figure 3: The three Asx-turns of Me-NA (left), compared to the four -turn types of Ac-Ala-Ala-NHMe (right), as obtained by quantum chemistry at the DFT-D level of theory (B97-D3(BJ-abc)/def2-TZVPPD, see details in Methods section). The same types of structures are also found for the Me-NG compound and Ac-Ala-Gly-NHMe, respectively. The type I Asx-turn was not found as a stable conformation.

Energetics

In both Me-based sequences the most stable conformation was the type II turn (Table 1). It was followed by the types II and I, which were significantly higher in energy (more than 13 kJ•mol -1 , except for Me-NG for which the type II lowest form was only at 7 kJ•mol -1 ). The energetic values were not drastically changed when temperature effects were taken into account due to the similar shapes of these forms. The type I' turn was not found to be a minimum in either sequence. Despite the structural resemblance, the energetic order of the structure types differs considerably between Asx-turns and -turns: in the Ala-Ala sequence, the most stable -turn is type I followed by type II', as expected from steric hindrance with -substituted L-residues, [1d] in contrast to the Asx-turn model Me-NA, where the type I appears distorted (Table S1), and thus destabilized, relative to its -turn counterpart due to a similar steric hindrance. The order is different with Ala-Gly where the most stable type II -turn is challenged by type I, which is only 4 kJ•mol -1 higher in energy.

Table 1: Dependence of the energetics with the type of turn in the Asx-turn conformations of Bn-NA and Bn-NG, of Me-NA and Me-NG and of the -turns formed by Ac-Ala-Ala-NHMe and Ac-Ala-Gly-NHMe. Zero-point energy corrected electronic relative energies (H at 0 K) and Gibbs free energies (G at 298 K), obtained at the B97-D3(BJ-abc)/def2-TZVPPD level of theory, are given in kJ•mol -1 . For each molecule, the values are given relative to the most stable conformation and types are ordered according to their relative stability at 298 K.

ΔH (0 K) (kJ•mol -1 ) ΔG (298 K) (kJ•mol -1 ) ΔH (0 K) (kJ•mol -1 ) ΔG (298 K) (kJ•mol -1 ) Asx-

Effect of the benzyl group

The presence of three rotamers of the benzyl group for each turn type (Figure 4) introduced a spread in the energetic distribution, due to the dependence of the interactions between the benzyl moiety and the backbone upon the orientation of the former. The energetic ordering between the types was respected, however, with the type II' rotamers being the most stable in most cases; the exception was the Bn-NG molecule where the highest type II' rotamer was challenged by the lowest type II form. These calculations qualified the Bn-NA and Bn-NG molecules as good models to observe isolated Asn-turns in the gas phase.

Gas phase characterization of Bn-NA and Bn-NG

The excitation spectra of the two peptide model systems were recorded using one-color R2PI (Figure 5). The sharp peaks attested to the efficient cooling of the molecules in the jet. The excitation spectra of Bn-NA and Bn-NG each showed one intense band in the red part of the spectrum, accompanied by several weaker bands which can be a priori assigned to either vibronic features of the same conformation or to other conformations populated in the jet. For further assignment, IR spectra were recorded using the ion depletion technique.

IR/UV spectra were recorded for Bn-NA and Bn-NG in both NH stretch, amide I, and amide II regions (Figure 6, Table S3). For each molecule, the IR NH stretch spectra recorded for the labelled UV bands of Figure 5 were all identical (See Figs S1 andS2), demonstrating the existence of a single predominant conformation populated in the jet. Moreover, when the IR spectra of Bn-NA and Bn-NG were compared, they were found to be qualitatively similar in the NH stretch region, although some differences were observed in the amide I and II ranges.

The assignments were first guided by general spectroscopic rules and confirmed by theoretical calculations. [14] The free amide NH stretch absorptions are expected to appear in the 3450-3500 cm -1 range, while weak interactions like a C5 or an NH- interaction can cause this feature to red-shift to 3420-3450 cm -1 . [28] A broad and intense peak that was significantly red-shifted to around 3400-3420 cm -1 was an indication of the presence of a hydrogen bond. The coupled NH oscillators of a free C-terminus carboxamide group were expected to give rise to a doublet with a splitting of 117 cm -1 , with the blue asymmetric stretch appearing around 3500-3560 cm -1 . [29] Figure 6: NH stretch (right) and amide II and II (left) regions of the IR spectrum of the Bn-NA (upper panel) and Bn-NG (lower panel) molecules, as obtained from IR/UV double resonance spectroscopy, using their most intense UV band as a probe. Absorption spectra appear as depletions, plotted as positive signals. In the lower part of each panel the theoretical spectrum of the most stable form calculated for each molecule is given, namely the type II' trans form. In the NH stretch region, the assignment of the NH stretch bands is color-coded: green for the symmetric and antisymmetric components of the Asn NH2 group, blue for NHAla/Gly and red for the C-terminus NHMe group.

On this basis, a primary assignment of the bands observed in the IR/UV spectra could be proposed. The broad and red-shifted feature at around 3420 cm -1 in the IR spectra of both compounds indicated the presence of a hydrogen-bonded NH group. The doublet formed by the peaks at 3445 cm -1 and 3562 cm -1 was assigned to the symmetric and asymmetric stretches respectively of a free C-terminus carboxamide group, in agreement with previously observed free or nearly carboxamide group. [15,30] The band at 3465 cm -1 (3485 cm -1 ) in Bn-NA (Bn-NG) indicates a free NH oscillator. All these features were in full agreement with expectations for an Asxturn.

A more refined assignment, both in terms of Bn rotamer and turn type, was carried out by comparison with calculated IR spectra, taking into account the calibrations carried out in previous studies. [31] In order to assign the type of turn in the observed conformer, we compared the experimental IR spectral data with the computed IR spectra of the various conformations (Figure 6 and Figures S4-S5). An overall consideration of the calculated spectra for the most stable types of turn showed that the amide I region was significantly dependent upon the turn type and was therefore able to provide a good indication as to the structure, when combined with energetic data. The same spectral considerations (Figures S4 andS5) showed that rotamers of the same turn type (which exhibit similar backbones) present similar IR spectra in the amide I and II regions, whereas the amide A region showed marked differences, especially in terms of red shift ─ and hence strength ─ of the C10 H-bond. This property illustrates the flexibility of the Asx-turn feature and the sensitivity of its strength to its environment, and it was used advantageously to support the assignments made from the amides I and II.

A careful inspection of Figs S4 and S5 shows that type II' conformations provide a good fit with the experimental data in the amide I region, with a two-component CO stretch motif for Bn-NA (Figure 6 top) and a broad band exhibiting a blue shoulder for Bn-NG (Figure 6 bottom). In both cases, the most stable rotamer (trans) was the one that best accounted for the amide A spectrum, in particular the C10 band position. For these reasons, we confidently assign the forms observed to type II' Asx-turns, with a trans Bn rotamer orientation.

Discussion of the isolated structures in the gas phase

The good agreement between gas phase experimental data and theory validated the theoretical methods used for both the structure and the energetics and provided evidence for the formation of an Asn(i) carboxamide C=O•••NH(i+2) C10 H-bond and hence for an Asx-turn structure in Bn-NA and Bn-NG. The H-bond strengths, as judged from spectral shifts (Figure 6) or intermolecular distances (Table 2), are similar to those found in comparable -turns, [32] albeit slightly diminished. The C10 H-bond strengths remain relatively modest within the H-bonding scale of peptides; the C7 H-bonds of -turns, for example, are much stronger. This was confirmed by NBO analysis of the related Me-NA molecule, which showed that the stabilization induced by the type II' Asx-turn formation is of the order of 16 kJ•mol -1 for an intermolecular distance of 208 pm, whereas the corresponding values for a regular type II' -turn are 21 kJ•mol -1 and 203 pm, respectively. Despite being characterized by a similar geometrical disposition, the H-bond in the Asx-turn is weaker and has a larger intermolecular distance, epitomizing the frustration of the H-bonding due to the constraints imposed by the covalent backbone along the Asn side chain and the main chain around Ala. Interestingly, the opposite conclusion is drawn for type II turns (Table 2), where the frustration is greater in the regular -turn (stabilization energy of 16 kJ•mol -1 vs. 21 kJ•mol -1 in the Asx-turn), demonstrating the dependence of the constraints on the turn type.

Besides C10 H-bonding, other (minor) interactions have been evoked previously that may further stabilize a -turn structure: i) an NH•••amide interaction, [START_REF] Imani | [END_REF]33] where the N 3 H bond not only provides the donor for the C10 H-bond but also interacts with the  system of the central amide of the turn, and ii) a carbonyl-carbonyl interaction [34] involving the N-terminus and the central amide of a turn. The same NBO analysis that was used to assess H-bonding was also employed to quantify these interactions in Asx-turns as well as in their -turn counterparts (Table 2). The first type of interaction, corresponding to the amide 2  *N 3 H NBO interaction, was found to be modest, at around 1 kJ•mol -1 ; the second was not detected at all and was therefore lower than the NBO analysis threshold value of 0.2 kJ•mol -1 .

N 3 H amide2 H-bond Eij(2) (kJ•mol -1 ) lp(1)N 2  σ*N 3 H 1.6 1.0 1.1 1.0 N 2 •••HN 3 distance (pm)
233.1 236.3 235.0 236.5

Comparison with crystallized protein data Abundance of Asx-turns

A statistical search within a set of 430 proteins was carried out using the Motivated Proteins on-line facility. [35] It revealed 1319 Asx-turns, leading to an average number of 3.1 Asx-turns per protein. The type distribution of these Asx-turns was found to be as follows: type II, 67 %; type I, 20 %, type II, 13 %, while type I was marginal at only 0.2%. This was in line with the statistical findings of Duddy et al. [11] When particularized to "Asn-turns" (wherein Asx is represented explicitly by Asn) these abundances remained similar, namely: type II, 64 %; type I, 26 %, type II, 12 %, and type I only 0.3 %. It is of note that these abundances were in accordance with the relative stability of the Asx-turns of the Asn-Ala peptide models (as listed in Table 1): the type II Asx-turn was found to be the most stable Asx-turn conformation, followed by type II and type I conformations; type I was not found as a stable conformation. This correlation between protein crystallographic data and the calculated gas phase stabilities clearly suggests that the abundance of the type II Asx-turn in proteins may be explained by its intrinsic stability.

Dependence upon the sequence

A detailed analysis of the nature (Ala or Gly) of the residue adjacent to Asn (i.e. following Asn in the sequence) in the Asn-turn revealed some significant differences. Figure 7 shows the type distribution of Asn-turns in Asn-Ala and Asn-Gly sequences resulting from the Motivated Proteins facility search. Whereas the Asn-Ala sequence was essentially associated to types II' and I, in line with the general trend mentioned above when the sequence was not particularized, the Asn-turns with a Asn-Gly sequence were predominantly of type II.

This suggests that the underlying mechanism at play in the appearance of Asn-turns in Asn-Gly sequences differs radically from the other Asn-Xaa sequences, where Xaa stands for any residue other than Gly.

Figure 7: Distribution of the Asn-turn types for two adjacent residues of Asn in the Asn-turn, namely Ala and Gly, as obtained from a direct examination of the turns detected by the Motivated Proteins facility.

The character of the Asn-turns (isolated vs. combined with turn/bulge structures) is indicated by a color code (orange vs. blue, resp.).

Overlap with other secondary structures

The same analysis showed that many of these Asn-turns actually partially overlapped with another turn structure, namely a β-turn or a -bulge (in blue in Figure 7). However, as noted above for the type of Asn-turn, the overlap with these β-turn or -bulge structures and the fraction of Asn-turns combined with these structures was also found to depend dramatically on the nature of the residue following Asn in the sequence.

When the residue following Asn in an Asn-turn is alanine, this gives rise mainly to type II' and I Asn-turns; most of these turns (83%) are combined with a type I -turn (Figure 7) and in these cases the Asn-Ala sequence occupies the i, i+1 positions of the -turn (Figure 8). This turn combination has been described as an Asx-turn by Milner-White and coworkers. [6] Our present studies now show that it comes with two structural variants, which differ by the orientation of the Asn side chain (and hence the type of Asx-turn). In contrast, when the residue following Asn in an Asn-turn is glycine, a radically different picture emerges. The majority (59 %) of the Asn-turns in these Asn-Gly sequences are found to be isolated, not combined with any nearby secondary structure, and type II is most prominent (Figs. 7 and9). When the Asn-turn is in combination with a -turn or a -bulge (i.e. in the remaining 41% of the Asn-turns), the Asn-Gly sequence is found to occupy positions i+2, i+3 of the -turn/-bulge and to adopt nearly exclusively a type II Asx-turn structure (Figure 10), which has never been detected before, to the best of our knowledge. It transpires that the type distribution in proteins of Asx-turns implicating Asn depends significantly on the nature of the adjacent residue (Figure 7).

The Asn-Ala sequence is representative of the general behavior observed for all Asn-Xaa sequences, as mentioned at the beginning of this section. The fact that the Asn-turns observed in proteins correspond to the most stable intrinsic structures found in our model studies, and that their abundance follows the energetic order, vivify two extreme-case localized mechanisms: i) Asn-turns may form spontaneously and play the role of stable intermediate structures serving as precursors of -turns, or ii) once a -turn is formed it can be stabilized by an Asn-turn, provided that an Asn residue occupies the position i of the -turn. This would simply arise from the fact that the Ramachandran dihedrals at position i+1 imposed by the formation of a -turn roughly match those corresponding to the second residue of an intrinsic Asn-turn. Table S1 provides a clear example: position i+1 occupied by an Ala residue in a type I -turn gives rise to (1, 1) = (-72°, -16°), not excessively different from the dihedrals of an Ala residue in position i+2 of an Asn-turn : (2, 2) = (-91°, 1°) and (-112°, 28°), for types II' and I Asn-turns respectively. Both mechanisms might be active in turn formation.

Asn residues may indeed be quite effective in backbone dehydration [36] since they can replace the hydrating water molecule attached to a backbone NH site and simultaneously accommodate it, due to the capacity of their side chain to serve as both an H-bond donor and acceptor.

Asx-turns implicating an Asn-Gly sequence present a quite different portfolio. Firstly, the ubiquity of isolated Asn-turns of all types (Figure 7) suggests that these structures can form spontaneously, presumably aided by flexibility in the absence of a side chain on the glycine residue, a postulate which was made previously to account for the large probability of finding Gly in the i+3 position of a -turn. [5c] Secondly, the most frequent type in the isolated turns, type II, is not the most intrinsically stable form (Table 1), which suggests that the formation mechanism does not rely on innate properties but instead depends on additional interactions, for example with the water molecules that solvate the Asn side chain. Thirdly, Asx-turns in Asn-Gly sequences are also observed in combination with a type I -turn or -bulge (which also contains a -turn). In this case, the Asx-turn is almost exclusively of type II, which fits well the structural arrangement of the -turn/bulge (Figure .   10) and of its network of hydration molecules.

Figure 10: Asn-turns in Asn-Gly sequences found in combination with type I -turns or -bulges, as found in the PDB survey carried out using Motivated Proteins. For both structures (-turns and -bulges) the central parts of the structures were overlaid (see Section S4.1 for separate plots of turns and bulges). When detected (in high resolution structures of the PDB), O atoms of the hydration waters (labelled w1 and w2) were also indicated (blue and cyan balls, resp.). The H-bonds present in the structure are identified by a color code: the C10 H-bond of the Asn-turn in yellow; bands show the -turn C10 Hbonds and C17 H-bonds in -bulges in grey, the hydration Hbond network in blue, and the elongated C6 This observation suggests that a type II Asx-turn in a Asn-Gly sequence may specifically foster the formation of a hydrated -turn/bulge structure, with a C10 H-bond located just before the Asn-Gly sequence in the peptide chain. This postulate is supported by the fact that the Asn  Ramachandran dihedral (~0°) in the sequences of Figure 10 and in the type II Bn-NA and Me-NA models of Table S1 fits the i+2 value required for the formation of a type I -turn (see for instance Ac-Ala-Ala-NHMe in Table S1). A previous study by us, [15a] in which Asncontaining -turns in both isolated model peptides and crystallized proteins were compared, has been reexamined (Section S4.2) and further supports this contention.

Interestingly, these peculiarities of the Asx-turns in Asn-Gly sequences echo the specific abundances of Asn (and Asp) residues in -turns, as collected by data mining studies, [5c, 7b] in particular in types I and I' -turns.

Regarding type I turns, Asx residues exhibit an overrepresentation (by a factor of 2-3) in positions i and i+2.

More precisely, correlations of residue presence in a type I -turn, detected using an on-line survey facility, [START_REF] Guruprasad | Database of Structural Motifs in Proteins[END_REF] showed that when Asn occupies an i+2 position in a type I -turn, Gly as the residue in position i+3 is overrepresented by a factor of 3.5 in comparison with its average abundance in proteins. Conversely, in the same type I -turns, when Asn is in position i, Gly in position i+1 is underrepresented by a factor of 4.2, well beyond the average penalty that represents a flexible Gly residue in position i+1 of a -turn (characterized by an underrepresentation of 2.3, taking the probability of having a Gly residue in position i+1, whatever the residues in the other position and normalizing it to the average abundance of Gly in proteins).

A similar specificity also seems to occur in type I' -turns. The Asn-Gly sequence is indeed known to be overrepresented in positions i+1, i+2 of these turns in proteins [5c, 7b] and to foster the folding of short peptides into type I' -turn structures in solution. [13a, 38] It is interesting to note that the Ramachandran dihedrals of a type I' -turn (for example see Table S1 in Supp. Info.: (1, 1) = (62°, 30°) and (2, 2) = (95°, -11°), for an Ala-Ala sequence Table S1) coincide to those of a type II Asx-turn ( Asn = 3° and (Gly, Gly) = (94°, -12°)), suggesting that interconversion from the latter to the former occurs mainly through a simple torsion of the Gly dihedral, presumably mediated by the water environment. [39] These observations are entirely consistent with the hypothesis made above whereby, once an Asn-Gly sequence folds into an Asn-turn, it has a great probability to locally induce a -turn, either of type I, where the Asn residue occupies the position i+2, or of type I', with Asn in the i+1 position.

Conclusion

The present work reports the synthesis of Asn-containing peptide models that were specifically designed to inhibit the formation of a -turn and therefore allow intrinsic folding into an Asx-turn. Two model systems were considered, which differed by the identity of the residue following Asn in the sequence, namely Ala and Gly. A theoretical analysis showed that the most stable Asx-turns mimic a type II' -turn, whatever the next neighbor residue of Asn. The gas phase experiments, carried out in a supersonic expansion, provided spectroscopic evidence for the spontaneous folding of the model compounds into the most stable form, namely a type II' Asxturn, confirming the validity of the theoretical approach.

A survey of Asn-turns in PDB proteins was also carried out. It confirmed earlier data, which showed that Asnturns are often associated with a -turn where Asn occupies the position i, the whole structure being referred to as a Asx--turn. In this case the Asx-turn is mainly of type II', corresponding to its most stable innate structure. When the next neighbor residue along the sequence is a Gly, however, the survey showed that i) a majority of Asx turns are found as isolated secondary structures, ii) most of them are of type II, i.e. differ from that observed in the gas phase, and iii) when associated to another structure, this latter is a type I -turn/bulge type, where Asn exclusively occupies the position i+2 in the turn. The structures observed in crystallized proteins suggest that the Asn-Gly sequence exhibits a propensity to form a type II Asn-turn, which is therefore not controlled by intrinsic features but presumably by other factors such as hydration. In all cases, Asn-turns might be efficient promotors of type I -turns and -bulges. Finally, the turn-controlling behavior of Asn is expected to be shared by the aspartic acid residue, whose side-chain carboxylate function (in physiological conditions) exhibits similar properties to the carboxamide group of asparagine, namely good H-bond accepting behavior, in line with the resemblance of their codons in the genetic code (which only differ by their first letter), which tends to minimize the harmful effects of Asp/Asn mutations.

Figure 1 :

 1 Figure 1: Geometrical representations of a -turn (top) and an Asxturn (bottom), with the latter illustrated for an Asn residue. The dihedrals describing the backbones are indicated by circular arrows (Ramachandran dihedrals in the -turn case).

Figure 2 :

 2 Figure 2: (left) Model peptides 1 and 2 used for gas phase spectroscopy. (right) Model peptides used for turn typology evaluation by theoretical methods. Also indicated is the amide group numbering, and hence atom numbering, used in this paper.

Figure 4 :

 4 Figure 4: Quantum chemistry optimized structures of the Asx-turns formed by Bn-NA (left) and Bn-NG (right) mimicking the respective -turn conformations. For each turn type, the three rotamers of the benzyl moiety are displayed simultaneously, with their backbone structures overlaid. Dotted lines indicate the C10 H-bond.

Figure 5 :

 5 Figure 5: Near-UV mass-selected resonant two-photon ionization spectrum of the Bn-NA and Bn-NG molecules, in the spectral region of the origin of the first * transition of the benzyl moiety.

Figure 8 :

 8 Figure 8: Asn-turns in Asn-Ala sequences combined with a type I -turn, as found in the PDB partial survey carried out using Motivated Proteins. O and N peptide atoms are colored in red and blue, respectively. Unless present in high resolution PDB structures, H atoms of the amide groups have been added on a force field basis. Two types of Asn-turns are observed, type II' (top) and type I (bottom). For each type the Asn-and -turn backbones have been overlaid. The yellow bands indicate the C10 H-bond of the Asn-turn and the grey ones indicate the C10 H-bond of the -turn. Errors in the Asn side chain orientation deduced from the X-ray data analysis cause a few Asn carbonyl O atoms to appear as colored in blue.

Figure 9 :

 9 Figure 9: Isolated Asn-turns in Asn-Gly sequences, as found in the PDB survey carried out using Motivated Proteins. O and N peptide atoms are colored in red and blue respectively. For each main type of Asn-turn observed, the backbones are overlaid. The yellow bands indicate the C10 Hbond of the Asn-turn, together with, in the type II case, an elongated intraresidue C6  H-bond (NH•••O(carboxamide) distance ~260 pm) bridging the NH bond of the previous residue and the Asn side chain carbonyl.

  Figure10: Asn-turns in Asn-Gly sequences found in combination with type I -turns or -bulges, as found in the PDB survey carried out using Motivated Proteins. For both structures (-turns and -bulges) the central parts of the structures were overlaid (see Section S4.1 for separate plots of turns and bulges). When detected (in high resolution structures of the PDB), O atoms of the hydration waters (labelled w1 and w2) were also indicated (blue and cyan balls, resp.). The H-bonds present in the structure are identified by a color code: the C10 H-bond of the Asn-turn in yellow; bands show the -turn C10 Hbonds and C17 H-bonds in -bulges in grey, the hydration Hbond network in blue, and the elongated C6  H-bond linking the central amide to the Asn side chain in cyan. The NH•••O distances of the C10 and C17 bonds are of the order of 200 pm, as well as the NH(i+1)•••w2 and NH(i+2)•••w1 bonds. The distance between water clusters is of the order of 300 pm, as well as the w1•••O(carboxamide) distance, providing evidence for hydration of the backbone NH(i+1) and NH(i+2) groups together with the O(carboxamide) atoms of the Asn side chain.

Table 2 :

 2 Interactions between donor (i) and acceptor (j) NBOs and corresponding stabilization energies (noted Eij(2) , given in kJ•mol -1 ) in Me-NA Asx-turn types II' and II and in their Ac-Ala-Ala-NHMe -turn counterparts. Several NBOs are found as active donors contributing to the *N 3 H NBO of the NH group for the C10 H-bond: the lone pairs (lp(1) and lp(2)) of the carbonyl O 1 atom and the carbonyl C 1 O 1 bonds ( and marginally  belonging to the Asn side chain in Asx-turns or to the acetyl group in the -turns. An interaction between the NBO located on the lp(1) of the N 2 atom of the central amide group and the *N 3 H NBO, made possible by the short N 3 H•••N 2 distance, is also found but remains negligible compared to the C10 H-bonding.

			MeNA	MeNA	Ac-Ala-Ala-NHMe	Ac-Ala-Ala-NHMe
			type II'	type II	type II'	type II
			N 3 H ••• O 3 C C10 H-bond
	Eij(2)	lp(1)O 1  σ*N 3 H	8.2	11.3	12.8	8.2
	(kJ•mol -1 )	lp(1)O 1  σ*N 3 H	6.6	2.0	4.3	5.9
		lp(2)O 1  σ*N 3 H	1.0	7.3	4.0	2.4
		σC 1 O 1  σ*N 3 H			0.3
	Sum Eij(2) (kJ•mol -1 )	15.8	20.6	21.4	16.5
	O 1 •••HN 3 distance (pm)	208.5	204.7	202.6	208.6

Acknowledgements

Support from the French National Research Agency (ANR; Grant ANR-17-CE29-0008 "TUNIFOLD-S") and from the "Investissements d'Avenir" Funding program (LabEx PALM; grant ANR-10-LABX-0039-PALM; DIRCOS) are acknowledged. This work was granted access to the HPC facility of [TGCC/CINES/IDRIS] under the Grants 2020-A0070807540 and 2021-A0090807540 awarded by GENCI (Grand Equipement National de Calcul Intensif) and to the CCRT High Performance Computing (HPC) facility at CEA under the Grants CCRT2020-p606bren and CCRT2021-p606bren.

Data availability statement

Conflict of Interest

The authors declare no conflict of interest.


Alexandre Défossez

Charlotte Caucheteux

Jeremy Rapin

Ori Kabeli

Jean-Rémi King

Jérémy Rapin

Decoding speech from non-invasive brain recordings

come

Introduction

Every year, thousands of patients suffer from brain or spinal cord injuries and suddenly lose their ability to communicate [START_REF] Carol | Demographics of rehabilitation robotics users[END_REF][START_REF] Elmar | Estimated prevalence of the target population for brain-computer interface neurotechnology in the netherlands[END_REF][START_REF] Kübler | Braincomputer communication: Unlocking the locked in[END_REF][START_REF] Elmar | Estimated prevalence of the target population for brain-computer interface neurotechnology in the netherlands[END_REF][START_REF] Claassen | Detection of brain activation in unresponsive patients with acute brain injury[END_REF][START_REF] Owen | Detecting awareness in the vegetative state[END_REF][START_REF] Cruse | Bedside detection of awareness in the vegetative state: a cohort study[END_REF]. Brain Computer Interface (BCI) has been raising high expectations to detect [START_REF] Owen | Detecting awareness in the vegetative state[END_REF][START_REF] Claassen | Detection of brain activation in unresponsive patients with acute brain injury[END_REF][START_REF] Birbaumer | A spelling device for the paralysed[END_REF][START_REF] King | Single-trial decoding of auditory novelty responses facilitates the detection of residual consciousness[END_REF] and restore language abilities in such patients [START_REF] Brumberg | Artificial speech synthesizer control by brain-computer interface[END_REF][START_REF] Herff | Brain-to-text: decoding spoken phrases from phone representations in the brain[END_REF][START_REF] Sergey D Stavisky | Decoding speech from intracortical multielectrode arrays in dorsal "arm/hand areas" of human motor cortex[END_REF][START_REF] Francis R Willett | High-performance brain-to-text communication via handwriting[END_REF][START_REF] David A Moses | Neuroprosthesis for decoding speech in a paralyzed person with anarthria[END_REF][START_REF] Kennedy | Slow firing single units are essential for optimal decoding of silent speech[END_REF]: Over the past decades, several teams used BCI to efficiently decode phonemes, speech sounds [START_REF] Pei | Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans[END_REF][START_REF] Akbari | Towards reconstructing intelligible speech from the human auditory cortex[END_REF], hand gestures [START_REF] Sergey D Stavisky | Decoding speech from intracortical multielectrode arrays in dorsal "arm/hand areas" of human motor cortex[END_REF][START_REF] Francis R Willett | High-performance brain-to-text communication via handwriting[END_REF] and articulatory movements [START_REF] Gopala K Anumanchipalli | Speech synthesis from neural decoding of spoken sentences[END_REF][START_REF] David A Moses | Neuroprosthesis for decoding speech in a paralyzed person with anarthria[END_REF] from electrodes implanted in the cortex or over its surface. For instance, [START_REF] Francis R Willett | High-performance brain-to-text communication via handwriting[END_REF] decoded 90 characters per minute (with a 94% accuracy, i.e. roughly ≈15-18 words per minute) from a spinal-cord injury patient recorded in the motor cortex during 10 hours of writing sessions. Similarly, [START_REF] David A Moses | Neuroprosthesis for decoding speech in a paralyzed person with anarthria[END_REF] decoded 15.2 words per minute (with 74.4% accuracy, and using a vocabulary of 50 words) in an anarthria patient implanted in the sensorimotor cortex and recorded over 48 sessions spanning over 22 hours. However, such invasive recordings face major practical challenges: these high-quality signals require

(1) brain surgery (2) long training sessions and (3) can be difficult to maintain chronically. Several laboratories have thus focused on decoding language from non-invasive recordings of brain activity like magneto-and electro-encephalography (M/EEG). MEG and EEG are sensitive to macroscopic changes of electric and magnetic signals elicited in the cortex, and can be acquired with a safe and Preprint. Under review. potentially wearable setup [START_REF] Boto | Moving magnetoencephalography towards real-world applications with a wearable system[END_REF]. However, these two devices produce notoriously noisy signals that vary greatly across sessions and across individuals [START_REF] Tibor Schirrmeister | Deep learning with convolutional neural networks for eeg decoding and visualization[END_REF][START_REF] King | Encoding and decoding neuronal dynamics: Methodological framework to uncover the algorithms of cognition[END_REF][START_REF] Hämäläinen | Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain[END_REF]. It is thus common to engineer pipelines that output hand-crafted features, which, in turn, can be learned by a decoder trained on a single participant [START_REF] Vernon J Lawhern | Eegnet: a compact convolutional neural network for eeg-based brain-computer interfaces[END_REF][START_REF] Lopopolo | Part-of-speech classification from magnetoencephalography data using 1-dimensional convolutional neural network[END_REF][START_REF] Alexander M Chan | Decoding word and category-specific spatiotemporal representations from meg and eeg[END_REF][START_REF] Chuong H Nguyen | Inferring imagined speech using eeg signals: a new approach using riemannian manifold features[END_REF].

In sum, decoding language from brain activity is, to date, either limited to invasive recordings or to impractical tasks. Interestingly, both of these approaches followed a similar method: i.e. (1) training a model on a single patient and (2) aiming to decode a limited set of interpretable features (MEL spectrogram, letters, phonemes, small set of words).

Instead, we here propose to decode speech from non-invasive brain recordings by using (1) a single architecture trained across a large cohort of participants and (2) deep representations of speech learnt with self-supervised learning on a large quantity of speech data. For this, we introduce a convolutional neural network stacked onto a "Subject Layer" and trained with a contrastive objective to predict the representations of the audio waveform learnt by wav2vec 2.0 pretrained on 56k hours of speech [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] (Figure 1). To validate our approach, we curate and integrate four public [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]) and learns the representations Z of the brain activity on the corresponding 3 s window (X) that maximally align with these speech representations with a contrastive loss (CLIP: [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]). The representation Z is given by a deep convolutional network. At evaluation, we input the model with left-out sentences and compute the probability of each 3 s speech segment given each brain representation. The resulting decoding can thus be "zero-shot" in that the audio snippets predicted by the model need not be present in the training set. This approach is thus more general than standard classification approaches where the decoder can only predict the categories learnt during training.

Method

We first formalize the general task of neural decoding and then describe and motivate the different components of our model, before describing the datasets, preprocessing, training and evaluation.

Problem formalization

We aim to decode speech from a time series of high-dimensional brain signals recorded with non-invasive magneto-encephalography (MEG) or electro-encephalography (EEG) while healthy volunteers passively listened to spoken sentences in their native language. How spoken words are represented in the brain is largely unknown [START_REF] Hickok | The cortical organization of speech processing[END_REF]. Thus, it is common to train decoders in a supervised manner to predict a latent representation of speech known to be relevant to the brain [START_REF] Akbari | Towards reconstructing intelligible speech from the human auditory cortex[END_REF], Angrick et al., 2019b[START_REF] Krishna | Speech synthesis using eeg[END_REF][START_REF] Komeiji | Transformer-based estimation of spoken sentences using electrocorticography[END_REF]. For example, the Mel spectrogram is often targeted for neural decoding because it representats sounds similarly to the cochlea [START_REF] Mermelstein | Distance measures for speech recognition, psychological and instrumental[END_REF]. Let X ∈ R C×T be a segment of a brain recording of a given subject while she listens to a speech segment of the same duration, with C the number of M/EEG sensors and T the number of time steps. Let Y ∈ R F ×T be the latent representation of speech, using the same sample rate as X for simplicity, here the Mel spectrogram with F frequency bands. Thus, supervised decoding consists of finding a decoding function: f reg : R C×T → R F ×T such that f reg predicts Y given X. We denote by Ŷ = f reg (X) the representation of speech decoded from the brain. When f reg belongs to a parameterized family of models like deep neural networks, it can be trained with a regression loss L reg (Y, Ŷ ) (e.g. the Mean Square Error),

min freg X,Y L reg (Y, f reg (X)). (1)
Empirically, we observed that this direct regression approach faces several challenges: decoding predictions appear to be dominated by a non-distinguishable broadband component when speech is present (Figure 2.A-B). This challenge motivates our three main contributions: the introduction of a contrastive loss, a pre-trained deep speech representation, and a dedicated brain decoder.

Model

Contrastive loss

First, we reasoned that regression may be an ineffective loss because it departs from our objective:

decoding speech from brain activity. Consequently, we replaced it with a contrastive loss, namely, the "CLIP" loss (originally for Contrastive Language-Image Pre-Training) by [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF],

which was originally designed to match latent representations in two modalities, text and images.

We implement the CLIP loss as follows: Let X be a brain recording segment and Y ∈ R F ×T the latent representation of its corresponding sound (a.k.a "positive sample"). We sample N -1 negative samples Ȳj∈{1,...,N-1} over our dataset and we add the positive sample as ȲN = Y . We want our model to predict the probabilities ∀j ∈ {1, . . . , N }, p j = P Ȳj = Y . We thus train a model f clip mapping the brain activity X to a latent representation Z = f clip (X) ∈ R F ×T . The estimated probability can then be approximated by the dot product of Z and the candidate speech latent representations Y j , followed by a softmax:

pj = e ⟨Z, Ȳj ⟩ N j ′ =1 e ⟨Z, Ȳj ′ ⟩ , (2)
with ⟨•,•⟩ the inner product over both dimensions of Z and Ŷ . We then train f clip with a cross-entropy between p j and pj . Note that for a large enough dataset, we can neglect the probability of sampling twice the same segment, so that we have p j = 1 j=N , and the cross-entropy simplifies to

L CLIP (p, p) = -log(p N ) = -⟨Z,Y ⟩ + log N j ′ =1 e ⟨Z, Ȳ ′ j ⟩ . (3)
Following [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], we use the other elements of the batch as negative samples at train time. At test time, the negative samples correspond to all of the segments of the test but the positive one. Replacing the regression loss with a CLIP loss [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] improves reconstruction in the same subject, still using the mel-spectrogram as the speech representation. D. Now replacing the mel-spectrogram with wav2vec 2.0 [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]. The probabilities given by Eq. ( 2) are used to rebuild a mel-spectrogram. E. Architecture of the brain module. Architecture used to process the brain recordings. For each layer, we note first the number of output channels, while the number of time steps is constant throughout the layers. The model is composed of a spatial attention layer, then a 1x1 convolution without activation. A "Subject Layer" is selected based on the subject index s, which consists in a 1x1 convolution learnt only for that subject with no activation. Then, we apply five convolutional blocks made of three convolutions. The first two use residual skip connection and increasing dilation, followed by a BatchNorm layer and a GELU activation. The third convolution is not residual, and uses a GLU activation (which halves the number of channels) and no normalization. Finally, we apply two 1x1 convolutions with a GELU in between.

Speech module

Second, the Mel spectrogram is a low-level representation of speech and is thus unlikely to match the rich variety of cortical representations [START_REF] Hickok | The cortical organization of speech processing[END_REF]. Consequently, we replaced the Mel spectrograms Y with latent representations of speech, that are either learned end-to-end ("Deep Mel" model) or learned with an independent self-supervised speech model ("wav2vec 2.0", [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]) As detailed in the result section, the "Deep Mel" model uses an architecture similar to the brain module, but proved less efficient than its pretrained counterpart. We will thus focus the decoding results obtained with wav2vec 2.0.

Wav2vec 2.0 is trained to transform the raw waveform with convolutional and transformer blocks to predict masked parts of its own latent representations. [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] showed that the resulting model can be efficiently fine-tuned to achieve state-of-the-art performance in speech recognition.

Besides, this model effectively encodes a wide variety of linguistic features [Millet andDunbar, 2022, Adolfi et al., 2022]. Finally, recent work shows the existence of linear correspondence between the activations of the brain and those of wav2vec 2.0 [Millet et al., 2022[START_REF] Vaidya | Self-supervised models of audio effectively explain human cortical responses to speech[END_REF].

Consequently, we here test whether this model effectively helps the present decoding task. In practice, we use the wav2vec2-large-xlsr-531 , which has been pre-trained on 56k hours of speech from 53 different languages.

Brain module

Finally, for the brain module, we use a deep neural network f clip , input with raw M/EEG times series X and a one-hot-encoding of the corresponding subject s, and outputs the latent brain representation Z, with the same sample rate as X. This architecture consists of (1) a spatial attention layer over the M/EEG sensors followed (2) by a subject-specific 1x1 convolution designed to leverage inter-subject variability, which input to (3) a stack of convolutional blocks. An overview of the model is given in Figure 2. In the following, given a tensor U , we will note U (i,...) access to specific entries in the tensor.

Spatial attention and subject layer. The brain data is first remapped onto D 1 = 270 channels with a spatial attention layer based on the location of the sensors. The 3D sensor locations are first projected on a 2D plane obtained with the MNE-Python function find_layout [START_REF] Gramfort | Meg and eeg data analysis with mne-python[END_REF], which uses a device-dependent surface designed to preserve the channel distances. Their 2D positions are finally normalized to [0, 1]. For each output channel, a function over [0, 1] 2 is learnt, parameterized in the Fourier space. The weights over the input sensors is then given by the softmax of the function evaluated at the sensor locations. Formally, each input channel i has a location (x i , y i )

and each output channel j is attached a function a j over [0, 1] 2 parameterized in the Fourier space as z j ∈ C K×K with K=32 harmonics along each axis, i.e.

a j (x, y) = K k=1 K l=1
Re(z

(k,l) j ) cos (2π(kx + ly)) + Im(z (k,l) j ) sin (2π(kx + ly)) . (4)
The output is given by a softmax attention based on the evaluation of a j at each input position (x i , y i ):

∀j ∈ [D 1 ], SA(X) (j) = 1 D1 i=1 e aj (xi,yi) C i=1 e aj (xi,yi) X (i) (5)
with SA the spatial attention. In practice, as a j is periodic, we scale down (x, y) to keep a margin of 0.1 on each side. We then apply a spatial dropout by sampling a location (x drop , y drop ) and removing from the softmax each sensor that is within a distance of d drop of the sampled location. We then add a 1x1 convolution (i.e. with a kernel size of 1) without activation and with the same number D 1 of output channels. Finally, to leverage inter-subject variability, we learn a matrix M s ∈ R D1,D1 for each subject s ∈ [S] and apply it after the spatial attention layer along the channel dimension. This is similar but more expressive than the subject embedding used by [START_REF] Chehab | Deep recurrent encoder: A scalable end-to-end network to model brain signals[END_REF] for MEG encoding, and follows decade of research on subject alignment [START_REF] Xu | Regularized hyperalignment of multi-set fmri data[END_REF][START_REF] James V Haxby | Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies[END_REF].

Residual dilated convolutions. We then apply a stack of five blocks of three convolutional layers. For the k-th block, the first two convolutions are applied with residual skip connections (except for the very first one where the number of dimension potentially doesn't match), outputs D 2 = 320 channels and are followed by batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]] and a GELU activation [START_REF] Hendrycks | Gaussian error linear units (gelus)[END_REF]. The two convolutions are also dilated to increase their receptive field, respectively by 2 2k mod 5 and 2 2k+1 mod 5 (with k zero indexed). The third layer in a block outputs 2D 2 channels and uses a GLU activation [START_REF] Yann | Language modeling with gated convolutional networks[END_REF] which halves the number of channels. All convolutions use a kernel size of 3 over the time axis, a stride of 1, and sufficient padding to keep the number of time steps constant across layers. The output of the model is obtained by applying two final 1x1 convolutions: first with 2D 2 outputs, followed by a GELU, and finally with F channels as output, thus matching the dimensionality of speech representations.

Given the expected delay between a stimulus and its corresponding brain responses, we further shift the input brain signal by 150 ms into the future to facilitate the alignment between Y and Z.

Datasets

We test our approach on four public datasets, two based on MEG recordings and two on EEG.

All datasets and their corresponding studies were approved by the relevant ethics committee and are publicly available for fundamental research purposes. Informed consent was obtained from all human research participants. We provide an overview of the main characteristics of the datasets on Table 1, including the number of train and test segments and vocabulary size over both splits.

For all datasets, healthy adult volunteers passively listened to speech sounds (accompanied with some memory or comprehension questions to ensure participants were attentive), while their brain activity was recorded with MEG or EEG. In [START_REF] Schoffelen | A 204-subject multimodal neuroimaging dataset to study language processing[END_REF], Dutch-speaking participants listened to decontextualized Dutch sentences and word lists (Dutch sentences for which the words are randomly shuffled). The study was approved by the local ethics committee (CMO -the local "Committee on Research Involving Human Subjects" in the Arnhem-Nijmegen region). In [START_REF] Gwilliams | Meg-masc: a high-quality magneto-encephalography dataset for evaluating natural speech processing[END_REF], English-speaking participants listened to four fictional stories from the Masc corpus [START_REF] Ide | The manually annotated sub-corpus: A community resource for and by the people[END_REF] in two identical sessions of one hour [START_REF] Gwilliams | Neural dynamics of phoneme sequencing in real speech jointly encode order and invariant content[END_REF]

Preprocessing

M/EEG is generally considered to capture neural signals from relatively low frequency ranges [START_REF] Hämäläinen | Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain[END_REF]. Consequently, we first resampled all brain recordings down to 120 Hz with Torchaudio [Yang et al., 2021] and then split the data into training, validation, and testing splits with a size roughly proportional to 70%, 20%, and 10%. We define a "sample" as a 3 s window of brain recording with its associated speech representation. A "segment" is a unique 3 s window of speech sound. As the same segment can be presented to multiple subjects (or even within the same subject in [START_REF] Gwilliams | Meg-masc: a high-quality magneto-encephalography dataset for evaluating natural speech processing[END_REF]), the splits are defined so that one segment is always assigned to the same split across repetitions. We ensure that there is no identical sentences across splits, and checked that each sentence was pronounced by a unique speaker. Furthermore, we exclude all segments overlapping over different splits. For clarity, we restrict the test segments to those that contain a word at a fixed location (here 500 ms into the sample).

M/EEG data can suffer from large artifacts, e.g. eye movements, or variations in the electro-magnetic environment [START_REF] Hämäläinen | Magnetoencephalography-theory, instrumentation, and applications to noninvasive studies of the working human brain[END_REF]. To limit their impact, we apply a "baseline correction" (i.e.

we subtract to each input channel its average over the first 0.5 s) and a robust scaler with scikit-learn [START_REF] Pedregosa | Scikit-learn: Machine learning in python[END_REF]. We clamp values greater than 20 after normalization to minimize the impact of large outlier samples. For the Mel spectrogram, we use 120 Mel bands (see Section A.2 in the Appendix) [START_REF] Young | The htk book[END_REF], with a normalized STFT with a frame size of 512 samples and hop length of 128 samples, using audio sampled at 16kHz. We apply log-compression, i.e. log(ϵ + mel), with ϵ=10 -5 . When using wav2vec 2.0, we average the activations of the last four layers of its transformer. We use standard normalization for both representations. To further assess the gains from using a self supervised representation, we also test a "Deep Mel" variant, where we train a deep transformation of the Mel, with the same architecture as the one applied to the brain recording, without the spatial attention and subject layer, and matching the output dimension of wav2vec 2.0.

This transformation is trained along with the brain decoder using the contrastive objective (3). By definition, the Deep Mel model only sees the audio from the each of the studied datasets (unlike wav2vec 2.0).

Training

One training epoch is defined as 1,200 updates using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] with a learning rate of 3•10 -4 and a batch size of 128. We stop training when no improvement is observed on the valid set for 10 epochs and keep the best model based on the valid loss. For the direct regression of the Mel spectrogram, we use the MSE loss. We use two V100 GPUs with 16GB of memory.

Evaluation

Segment-level evaluation. In Figure 2, we estimate the Mel spectrogram from the model output.

Given a segment and its matching audio (here the sentence "Thank you for coming Ed"), we retrieve the predicted distribution over the 1, 594 segments given by ( 2). We use this distribution to average whether the true segment is in the top-10 most likely segments according to the same probabilities.

Word-level evaluation. We also evaluate the model at the word level (Figure 4). For each word of the test set, we select a 3 s segment starting with this word. We input the model with the corresponding brain recordings, and output the probability distribution over all test segments including the true segment. To obtain the distribution over the vocabulary, we group the candidate segments by their first word and sum the probabilities within each group.

Code availability

The code to reproduce the present study will be made publicly available upon publication.

Results

Accurately decoding speech from M/EEG recordings

Our model predicts the proper segment, out of more than 1,000 possible ones, with a top-10 accuracy of 72% and 67% for MEG datasets (top-1 accuracy of 44% and 36%) (Table 2). For more than half of samples, the true audio segment is ranked first or second in the decoders' predictions. For comparison, a model that predicts a uniform distribution over the vocabulary ('random model') only achieves a 2% top-10 accuracy on the same MEG datasets. Decoding performance for EEG datasets is lower: our model reaches 19% and 31% top-10 accuracy. While modest, these scores are four times higher than the random baseline.

Effect of contrastive loss, deep speech representations, and number of participants

Our ablation highlights the importance of: (1) the contrastive loss, (2) the use of deep speech representations [START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF] and (3) the combination of a large number of participants. First, a model trained to predict the Mel spectrogram with a regression objective ('base model' in Table 2) achieves 10% top-10 accuracy on average across datasets -i.e. nearly five times lower than our model, when using the model output to rank the candidate segments by cosine similarity.

Second, predicting the Mel spectrogram with a contrastive loss leads to a 3X improvement over the base model, and gains another 16 points by using wav2vec 2.0 as the speech representation. We verified that the wav2vec 2.0's latent representations provide higher decoding performances than those learnt end-to-end with contrastive learning, as shown by the results of the Deep Mel model on Table 2.

Third, to test whether our model effectively leverage the inter-individual variability, we trained it on a variable number of subjects and computed its accuracy on the first 10% of subjects. As shown in Figure 3B, decoding performance increases as the model is trained with more subjects on the two MEG datasets. This ability to learn from multiple subjects is strengthened by another ablation experiment: training on all participants, but without the subject-specific layer, leads to a drop of 17% accuracy on average across the four datasets (Table 3). However, this last gain is relatively modest compared to the a subject embedding introduced recently [START_REF] Chehab | Deep recurrent encoder: A scalable end-to-end network to model brain signals[END_REF]. *: we used the subject embedding from [START_REF] Chehab | Deep recurrent encoder: A scalable end-to-end network to model brain signals[END_REF] instead of the subject layer.

Finally, other design choices modestly but significantly impact the performance of our model.

Performance systematically decreases when removing skip connections, the spatial attention module, the initial or final convolutional layers (Table 3). We also show how essential clamping is to train the model, except for the [START_REF] Gwilliams | Meg-masc: a high-quality magneto-encephalography dataset for evaluating natural speech processing[END_REF] dataset, which led to similar performances, although with a doubling of the training time. See Section A.2 in the Appendix for more ablations analyses.

Discussion

Here, we aimed to decode the perception of natural speech from non-invasive brain recordings.

Our results, based on the largest decoding study of M/EEG responses to speech to date, show that combining (1) a contrastive objective, (2) a convolutional architecture enhanced by a "Subject Layer", and (3) pretrained speech representations allows the decoding of new 3 s speech sounds with up to 44% top-1 accuracy out of more than 1,500 possibilities.

Our approach contributes to the rapid transformation of hand-crafted pipelines into their end-to-end counterparts. In particular, this study shows how self-supervised and contrastive learning can improve both (1) the analyses of brain signals and (2) the definition of the linguistic features that should be used for decoding. In particular, previous models were typically trained on individual subjects to categorize a very small number of highly-repeated categories and/or hand-crafted features [START_REF] Ali | Enhancing the decoding accuracy of eeg signals by the introduction of anchored-stft and adversarial data augmentation method[END_REF][START_REF] Jayaram | Moabb: trustworthy algorithm benchmarking for bcis[END_REF][START_REF] Vernon J Lawhern | Eegnet: a compact convolutional neural network for eeg-based brain-computer interfaces[END_REF] 6 distinct part-of-speech (48% accuracy), 10 words (83% accuracy) and 3 words (70% accuracy), from MEG signals. Finally, both [START_REF] Dash | Decoding imagined and spoken phrases from noninvasive neural (meg) signals[END_REF] and [START_REF] Wang | Towards decoding speech production from single-trial magnetoencephalography (meg) signals[END_REF] trained a classifier to decode 5 distinct sentences from MEG activity (both around 94% accuracy). Given the combinatorics of language, such single-subject / limited-vocabulary approach necessarily limits the possibility to decode natural speech. By contrast, our model effectively achieves "zero-shot" decoding by matching a large number of brain recordings to the deep representations of their corresponding speech sounds.

One remarkable phenomenon revealed by the present study is the difference of performance obtained between with EEG and MEG: while EEG is known to be less precise than MEG, we did not expect such a strong difference. This result thus holds great promises for the development of a safe and scalable system based on the analysis of magnetic -rather than electric -fields. It should be stressed that while the scientific community should remain vigilant that this approach will not be adapted to decode brain signals without the consent of the participants, this possibility appears unlikely at this stage: Unlike other biomarkers, such as fingerprints, DNA and facial features, electro-magnetic signals cannot be acquired unbeknownst to the participants. Furthermore, teeth clenching, eye blinks and other muscle movements are known to massively corrupt these signals, and thus presumably provide a simple way to counter downstream analyses. In any case, we believe that open science remains the best way to responsibly assess risks and benefits in this domain.

The present non-invasive study is limited to speech perception. It thus differs from the recent achievements obtained in a small set of heavily-trained patients implanted for clinical purposes and tasked to produce language [START_REF] Herff | Brain-to-text: decoding spoken phrases from phone representations in the brain[END_REF][START_REF] Martin | Word pair classification during imagined speech using direct brain recordings[END_REF], Angrick et al., 2019b[START_REF] Francis R Willett | High-performance brain-to-text communication via handwriting[END_REF][START_REF] David A Moses | Neuroprosthesis for decoding speech in a paralyzed person with anarthria[END_REF], Angrick et al., 2021[START_REF] Kohler | Synthesizing speech from intracranial depth electrodes using an encoder-decoder framework[END_REF]. In particular, Willett et al. [2021] showed that a 1 s time window of neuronal activity in the motor cortex suffices to decode one of 26 characters with 94.1% top-1 accuracy during a spelling task. Similarly, Moses et al. [2021] showed that 4 s of neuronal activity recorded in the sensory-motor cortices is sufficient to decode the intention to communicate one of 50 words with a median word error rate of 25.6%. We deliberately chose to focus on speech perception here, because speech production generates muscle activity, which can be easily read with EEG and MEG. Consequently, decoding speech production could be trivial in healthy participant, without ensuring any kind of utility for patients with an inability to control facial muscles.

While our approach remains to be adapted to language production, the possibility of leveraging data from (i) multiple subjects and (ii) large natural language datasets, together with (iii) the multiplication of public neuroimaging datasets, makes us hopeful about possibility of decoding intended communication from non-invasive recordings of brain activity. This possibility could also be accelerated by the development of new MEG hardwares: the MEG used in the present study makes use of superconducting quantum interference device (SQUID) and necessitates to cool the sensors down to ≈ 4 • K with a very large tank of liquid helium. However, several room-temperature sensors are now available, and already show signal-to-noise ratio comparable to SQUIDs [START_REF] Boto | Moving magnetoencephalography towards real-world applications with a wearable system[END_REF].

Combined with A.I. systems, these new devices will thus likely contribute to improve the diagnosis, prognosis and restoration of language processing in non-or poorly-communicating patients without putting them at risks for brain surgery.

  M/EEG datasets, encompassing the brain activity of 169 participants passively listening to sentences of short stories. With a sample of 3 seconds of M/EEG signals, our model identifies the matching audio segment (i.e. zero-shot decoding) with up to 72.5% top-10 accuracy (out of 1,594 segments) for MEG and up to 19.1% top-10 accuracy (out of 2,604 segments) for EEG.

Figure 1 :

 1 Figure 1: Method We aim to decode speech from the brain activity of healthy participants recorded with magnetoencephalography (MEG) or electroencephalography (EEG) while they listen to stories and/or sentences. For this, our model extracts the deep contextual representations of 3 s speech signals (Y ) from a pretrained self-supervised model (wav2vec 2.0: Baevski et al. [2020]) and learns the representations Z of the brain activity on the corresponding 3 s window (X) that maximally align with these speech representations with a contrastive loss (CLIP:[START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]). The representation Z is given by a deep convolutional network. At evaluation, we input the model with left-out sentences and compute the probability of each 3 s speech segment given each brain representation. The resulting decoding can thus be "zero-shot" in that the audio snippets predicted by the model need not be present in the training set. This approach is thus more general than standard classification approaches where the decoder can only predict the categories learnt during training.

Figure 2 :

 2 Figure 2: Design choices. A. Illustration of a 3 s speech sound segment (bottom) and its corresponding Mel spectrogram (top). B. Mel-spectrogram predicted with a direct regression loss L reg of a brain decoder (orange). C. Replacing the regression loss with a CLIP loss[START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] improves reconstruction in the same subject, still using the mel-spectrogram as the speech representation. D. Now replacing the mel-spectrogram with wav2vec 2.0[START_REF] Baevski | wav2vec 2.0: A framework for self-supervised learning of speech representations[END_REF]. The probabilities given by Eq. (2) are used to rebuild a mel-spectrogram. E. Architecture of the brain module. Architecture used to process the brain recordings. For each layer, we note first the number of output channels, while the number of time steps is constant throughout the layers. The model is composed of a spatial attention layer, then a 1x1 convolution without activation. A "Subject Layer" is selected based on the subject index s, which consists in a 1x1 convolution learnt only for that subject with no activation. Then, we apply five convolutional blocks made of three convolutions. The first two use residual skip connection and increasing dilation, followed by a BatchNorm layer and a GELU activation. The third convolution is not residual, and uses a GLU activation (which halves the number of channels) and no normalization. Finally, we apply two 1x1 convolutions with a GELU in between.

Figure 3 :

 3 Figure 3: Segment-level decoding. A. Probability distribution of the decoded rank for each segment (lower is better) for each dataset. The gray dotted line indicates the number of segments in the test set. B. Top-10 accuracy obtained for the first 10% of subjects (y-axis) as a function of the number of subjects seen during training (x-axis). The line and confidence intervals represent the mean and standard error of the mean (SEM) across participants, respectively.

Figure 4 :

 4 Figure 4: A. Single-word prediction for the first three subjects of Gwilliams et al. [2022] listening to the sentence "Thank you for coming, Ed". Text color indicates whether the decoded word is accurate. Text size is proportional to the log-probability output by our model. B. Top-1 accuracy at the word level (as explained in Section 2.6) as a function of the number of negatives during inference. C. The R 2 summarize how word frequency, part-of-speech tag, word embedding, and contextual embedding respectively predict the accuracy of single-word and single-segment decoding (Appendix A.4). Error bars are the SEM across participants.

Figure A. 1 :

 1 Figure A.1: Top-1 accuracy as a function of vocabulary size for word presented during random word lists in Schoffelen et al. [2019]. Error bar indicate SEM across participants.

Table 1 :

 1 Datasets, noting chs. for channels and subj. for subjects.

	Train set	Test set

  . The study was approved by the Institution Review Board (IRB) ethics committee of New York University Abu Dhabi. In Broderick et al. [2018], English-speaking participants listened to extracts of "The old man and the see". The study was approved by the Ethics Committees of the School of Psychology at Trinity College Dublin, and the Health Sciences Faculty at Trinity College Dublin. In Brennan and Hale [2019], English-speaking participants listened to a chapter of "Alice in Wonderlands". See Section A.1 in the Appendix for more details. The study was approved by the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board (HUM00081060).

Table 2 :

 2 Results. Top-10 segment-level accuracy (%) for a random baseline model that predicts a uniform distribution over the segments ('random'), a convolutional network trained to predict the Mel spectrograms with a regression loss ('base'), the same model trained with a contrastive loss ('+ clip') and our model, i.e. trained to predict the features of wav2vec 2.0 with a contrastive loss ('+ wav2vec 2.0'). ± indicates the standard deviation across three random initializations of the model's weights.

	Method	Schoffelen2019 Gwilliams2022 Broderick2019 Brennan2019 Mean
	Random model	1.5 ± 0.18	2.2 ± 0.16	4.1 ± 0.09	7.6 ± 0.13	3.8
	Base model	19.3 ± 0.83	14.9 ± 0.56	1.3 ± 0.19	6.6 ± 0.53	10.5
	+ CLIP	51.5 ± 0.47	58.6 ± 0.28	13.3 ± 0.54	14.5 ± 1.33	34.5
	+ Deep Mel	57.7 ± 0.16	64.4 ± 1.67	16.5 ± 0.26	23.7 ± 0.90	40.6
	+ wav2vec 2.0	67.2 ± 0.09	72.5 ± 0.22	19.1 ± 1.15	31.4 ± 1.59	47.5
	the Mel spectrogram of each candidate segment. Similarly, the top-10 segment accuracy indicates

Table 3 :

 3 Ablations. Top-10 segment-level accuracy (%) for our model and its ablated versions. Delta refers to the average decrease in accuracy of each ablated version compared to our model.

	Arch. change	Schoffelen2019 Gwilliams2022 Broderick2019 Brennan2019 Mean	Delta
	Our model	67.2 ± 0.09	72.5 ± 0.22	19.1 ± 1.15	31.4 ± 1.59	47.5	0.00
	\wo spatial attention dropout	61.6 ± 0.14	71.2 ± 0.93	19.0 ± 1.07	30.2 ± 1.70	45.5	-2.00
	\w subj. embedding*	59.5 ± 0.24	72.0 ± 0.77	20.2 ± 1.24	30.2 ± 0.77	45.4	-2.08
	\wo GELU, \w ReLU	61.4 ± 0.67	72.2 ± 0.05	19.2 ± 0.79	26.4 ± 1.03	44.8	-2.72
	\wo spatial attention	60.0 ± 1.32	69.5 ± 0.44	17.9 ± 0.34	26.0 ± 0.61	43.3	-4.18
	\wo final convs	62.3 ± 0.07	71.0 ± 0.47	15.7 ± 1.13	22.7 ± 2.05	43.0	-4.57
	\wo initial 1x1 conv.	57.8 ± 1.12	69.6 ± 0.37	17.9 ± 0.31	26.3 ± 1.43	42.9	-4.62
	\wo skip connections	59.2 ± 0.71	68.0 ± 0.60	16.7 ± 0.29	25.7 ± 4.32	42.4	-5.13
	\wo non-residual GLU conv.	63.5 ± 0.68	73.0 ± 0.67	17.0 ± 0.03	6.70 ± 0.57	40.0	-7.50
	\wo subject-specific layer	38.3 ± 0.77	49.2 ± 0.23	11.8 ± 0.14	21.5 ± 0.59	30.2 -17.30
	\wo clamping brain signal	1.1 ± 0.26	57.6 ± 13.4	4.0 ± 0.14	11.0 ± 1.92	18.4 -29.10

https://github.com/pytorch/fairseq/blob/main/examples/wav2vec

https://pypi.org/project/wordfreq/

https://spacy.io

https://github.com/facebookresearch/laser

https://scikit-learn.org

A Appendix

A.1 Datasets

The data from [START_REF] Schoffelen | A 204-subject multimodal neuroimaging dataset to study language processing[END_REF] was provided (in part) by the Donders Institute for Brain, Cognition and Behaviour with a "RU-DI-HD-1.0" licence 2 . The data for [START_REF] Gwilliams | Meg-masc: a high-quality magneto-encephalography dataset for evaluating natural speech processing[END_REF] is available under CC0 1.0 Universal 3 . The data for [START_REF] Michael P Broderick | Electrophysiological correlates of semantic dissimilarity reflect the comprehension of natural, narrative speech[END_REF] is available under the same licence 4 Finally, the data from [START_REF] Brennan | Hierarchical structure guides rapid linguistic predictions during naturalistic listening[END_REF] is available under the CC BY 4.0 licence 5 All audio files were provided by the authors of each dataset.

A.2 Extra Results

In this Section, we provide extra analysis with regard to the number of MEL band used, and the clamping value.

A.2.1 Effect of clamping

Clamping is essential due to the sensitivity of electro-magnetic recordings to perturbations. As explained in Section 2.4, we first use a quantile based robust scaler such that the range [-1, 1] maps to the [0.25, 0.75] quantile range. The scaling is computed independently for each recording. Thus it is expected most values for M/EEG recordings would have a scale of the order of 1. In the following table, we provide the top-10 accuracy for the Wav2Vec2.0 based model from Table 2. We observe that extending the clamping range from 20 to 100 doesn't allow the model to extract more information, which would be expected if large scale values are outliers without useful information on the underlying brain dynamics. On the other hand, when removing entirely clamping, we observe a collapse of the performance. This is expected, as extreme outliers will impact for instance the BatchNorm mean and standard deviation estimate, and one outlier can impact the entire batch. Outliers can also cause extreme gradients and throw off the optimization process. Interestingly, on Gwilliams2022, the drop is limited, potentially due to builtin preprocessing. We now study the impact of the number of Mel bands. 120 bands is usually considered high enough for most practical use [START_REF] Young | The htk book[END_REF], which we selected for the main evaluation in Table 2. We study the impact of the numer of Mel bands for different versions of the model. For clarity, we only provide the average top-10 accuracy overall datasets. We observe a small increase of the accuracy when using more Mel bands. Interestingly, when using the Deep Mel model, 20 bands is sufficient to achieve the best performance. Does our model predict all words similarly? To address this question, we evaluate whether our model's ability to decode individual words depends on their properties, namely their zipf frequency as provided by Wordfreq 6 , as well as their part-of-speech tag and their word embedding as provided by spaCy 7 . Similarly, we evaluate whether the decoding of the entire 3 s speech segment varies with its linguistic properties, as assessed by its average word embedding as well as its sentence embedding, as computed with Laser 8 . For this, we trained a regularized ridge regression with scikit-learn 9 's default parameters to predict the softmax probability of the true word output by the decoder, given a feature. We then estimate the R 2 with a 5-split cross-validation: i.e. how well the feature predicts the probability of being selected by the decoder. The results, displayed in Figure 4-C, show that the word and segment embedding effectively explain the single-trial decoding accuracy. These results thus suggest that our decoder uses semantic and contextual information to make its predictions.

Clamping value

# Mel bands

A.5 Decoding of isolated words

To what extent can our approach be used to decode words presented in isolation? To explore this issue, we evaluated our model using a subset from [START_REF] Schoffelen | A 204-subject multimodal neuroimaging dataset to study language processing[END_REF], where subjects are presented with random word lists. We use a segment ranging from -300 ms to +500 ms relative to word onset.

The results, displayed in Supplementary