Cécile Labarre
email: cecile.labarre@i2bc.paris-saclay.fr

Nathalie Dautin
email: dautin@ibpc.fr

Anna Grzegorzewicz
email: anna.grezegorzewic@colostate.edu

Mary Jackson
email: mary.jackson@colostate.edu

Michael Mcneil
email: m.mcneil@colostate.edu

Niloofar Mohiman

Laila Sago
email: laila.sago@i2bc.paris-saclay.fr

Nicolas Bayan
email: nicolas.bayan@i2bc.paris-saclay.fr

S16 and T18 mannosylation sites of LppX are not essential for its activity in phthiocerol dimycocerosates localization at the surface of Mycobacterium tuberculosis

Keywords: Corynebacterium glutamicum, Mycobacterium, glycosylation, phthiocerol dimycocerosates

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Among post-translational modifications of proteins, glycosylation is the most common modification occurring in nature. These modifications which were initially thought to be restricted to eukaryotic cells are now better characterized and known to be widely distributed in bacteria and archaea. All these organisms share similar N-glycosylation and O-glycosylation process. While N-glycosylation corresponds to the addition of glycosyl residues on asparagine, O-glycosylation targets serine, threonine or tyrosine. A large variety of glycans (differing in chain length, type of ose, chemical bonds and/or chemical modifications) might be added and thus creating a vast repertoire of modifications contributing to protein diversity [START_REF] Schäffer | Emerging facets of prokaryotic glycosylation[END_REF][START_REF] Corfield | Eukaryotic protein glycosylation: a primer for histochemists and cell biologists[END_REF].

In eukaryotes numerous functions have been assigned to protein glycosylation such as protein folding and stability, targeting to distinct cellular compartments and cell to cell recognition [START_REF] Corfield | Eukaryotic protein glycosylation: a primer for histochemists and cell biologists[END_REF]. Glycans can affect protein stability and accessibility as well as enable specific interaction. Moreover the nature of the glycan is important as protein dynamics differs depending on glycan chemistry [START_REF] Prates | The impact of O -glycan chemistry on the stability of intrinsically disordered proteins[END_REF]. As an example Prates et al. [START_REF] Prates | The impact of O -glycan chemistry on the stability of intrinsically disordered proteins[END_REF] studied the role of αmannosylation of a fungal Intrinsically Disordered Protein (IDP) by comparing different glycosylated variants of its linker. Compared to the other tested glycans, α-mannose was shown to increase proteolytic resistance, restrict protein flexibility and lead to more extended conformations of this linker. According to the authors this would explain why, in fungi and yeast, most of the secreted IDP and proteins exhibiting Intrinsically Disordered Regions (IDR) are O-mannosylated. In bacteria, the specific need for α-mannose in such structure does not seem essential since their intrinsic high content of proline favors extended conformation of IDR. Still, elongation and further stiffening are observed upon glycosylation of these structures [START_REF] Prates | The impact of O -glycan chemistry on the stability of intrinsically disordered proteins[END_REF].

In recent years there has been a growing interest towards glycosylation in bacteria.

Although glycosylation is not restricted to pathogenic bacteria, most studies were performed in these bacteria and conclude to an important role of glycosylation on pathogenicity [START_REF] Schäffer | Emerging facets of prokaryotic glycosylation[END_REF]. Indeed, this phenomenon was described in Mycobacterium tuberculosis [START_REF] Liu | Bacterial protein-Omannosylating enzyme is crucial for virulence of Mycobacterium tuberculosis[END_REF] where the authors demonstrated that disruption of the M. tuberculosis Rv1002c gene encoding the membraneassociated protein O-mannosyl transferase (PMT) resulted in strong attenuation of the strain.

In M. tuberculosis, PMT is unique and shown to be responsible for the transfer of mannose residues from polyprenol monophosphate mannose (PPM) to dedicated proteins [START_REF] Espitia | New insights into protein O-mannosylation in actinomycetes[END_REF]. In fact, various putatively mannosylated proteins were identified in M. tuberculosis by Concanavalin A affinity capture experiments [START_REF] González-Zamorano | Mycobacterium tuberculosis glycoproteomics based on ConA-lectin affinity capture of mannosylated proteins[END_REF]. Several lipoproteins including lipoproteins belonging to the Llp family (Lipid binding lipoprotein) were identified among these proteins including LppX-tb.

The gene encoding LppX-tb was initially identified in a transposition mutant of M. tuberculosis with an attenuated phenotype [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF]. In this mutant, the subcellular distribution of phthiocerol dimycocerosates (DIM) was highly altered as compared to the wild type strain. DIMs are one of the major lipidic virulence factor in pathogenic mycobacteria and their translocation at the surface of the bacterial envelope, essential for the virulence of the strain, is proposed to be mediated by LppX-tb. This hypothesis is quite convincingly corroborated by the structure of LppX-tb which ressemble a U-shaped β-half-barrel delimitating a large hydrophobic cavity able to accommodate a DIM molecule [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF]. Although supposed to be glycosylated, the complete characterization of LppX-tb modifications has not been so far reported. Indeed, glycosylation of LppX-tb was previously reported during heterologous expression in slow and fast-growing mycobacteria M. smegmatis, M. bovis and M. abscessus [START_REF] Tschumi | Identification of apolipoprotein N-acyltransferase (Lnt) in mycobacteria[END_REF][START_REF] Brülle | Lipoproteins of slow-growing Mycobacteria carry three fatty acids and are N-acylated by Apolipoprotein N-Acyltransferase BCG-2070c[END_REF][START_REF] Becker | Lipoprotein glycosylation by protein-O-mannosyltransferase (MAB_1122c) contributes to low cell envelope permeability and antibiotic resistance of Mycobacterium abscessus[END_REF]. In M. smegmatis glycosylation was only suggested and not demonstrated. Whereas further results were reported for the two other species for which 1 to 2 hexoses were found associated to the N-terminal peptide of the protein. However the exact glycosylation site(s) was/were never reported. Similar results have been reported in Corynebacterium glutamicum, a model host in which LppX-tb was found to be heterogeneously glycosylated with 1 to 4 hexose(s) units in its N-terminal peptide (aa6 to aa29) and where α-mannosidase deglycosylation experiments suggested that LppX-tb is indeed mannosylated [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF]. Interestingly LppX-tb glycosylation was found to be differently affected by mutations in the lipoprotein post-translational modification pathway in C. glutamicum [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF][START_REF] Dautin | Role of the unique, nonessential phosphatidylglycerol::prolipoprotein diacylglyceryl transferase (Lgt) in Corynebacterium glutamicum[END_REF]. These results enabled the authors to propose that the N-acylation enzyme Lnt associates with PMT but that acylation (S-and N-acylation) is not a prerequisite for glycosylation of LppX-tb in C. glutamicum.

As mannosylated LppX-tb is engaged in the virulence process of M. tuberculosis we wondered if the attenuation phenotype observed in the PMT mutant [START_REF] Liu | Bacterial protein-Omannosylating enzyme is crucial for virulence of Mycobacterium tuberculosis[END_REF] resulted from the absence of glycosylation of LppX-tb. For this, we first identified S16 and T18 as the two sites of LppX-tb mannosylation. We then undertook the construction of a variant where both amino acids were modified in order to test the functionality of this LppX-tb variant. Both the glycosylated and non-glycosylated forms of LppX-tb were characterized and found associated to the capsule layer of M. smegmatis. In M. tuberculosis Mt103, complementation of the knock-out mutant of LppX-tb either with a wild or mutated copy of the LppX-tb gene fully restored the release of DIM into the culture medium suggesting that glycosylation of S16 and T18 of LppX-tb is not necessary for DIM translocation.

Materials and methods

Bacterial strains and growth conditions

C. glutamicum ATCC13032 RES167 [START_REF] Dusch | Expression of the Corynebacterium glutamicum panD gene encoding L-aspartate-α-decarboxylase leads to pantothenate overproduction in Escherichia coli[END_REF] and its derivative pmt::Km r , were routinely cultured on BHI medium (Difco) at 30°C. Mycobacterium smegmatis MC 2 155 and Mycobacterium tuberculosis Mt103 were grown in Middlebrook 7H9 broth (Fluka) supplemented with Middlebrook ADC enrichment and 0.5% glycerol or BHI. Tween 80 (0.05 %) was added to liquid broth to avoid clumping except for capsule analysis.

Plasmids

Plasmids pCGL482 [START_REF] Schäfer | Small mobilizable multipurpose cloning vectors derived from the Escherichia coli plasmids pK18 and pK19: selection of defined deletions in the chromosome of Corynebacterium glutamicum[END_REF], pVWEx2 [START_REF] Eikmanns | A family of Corynebacterium glutamicum/Escherichia coli shuttle vectors for cloning, controlled gene expression, and promoter probing[END_REF], pMV261-Gm (derivative of pMV261) [START_REF] Stover | New use of BCG for recombinant vaccines[END_REF] and pVV16 [START_REF] Korduláková | Definition of the first mannosylation step in phosphatidylinositol mannoside synthesis: PimA is essential for growth of mycobacteria[END_REF] were used as expression vectors in C. glutamicum, M. smegmatis and M. tuberculosis respectively. When appropriate, antibiotics were added at the following concentrations: kanamycin (Km) 25 µg. ml -1 , chloramphenicol (Cm) 6 µg. ml -1 for C. glutamicum or 30 µg. ml -1 for E. coli, gentamycin (Gm) 10 µg. ml -1 , hygromycin (Hyg) 25 µg.

ml -1 and tetracyclin 10 µg. ml -1 .

Cloning of Cg-pmt and Ms-pmt in C. glutamicum : The pmt genes of C. glutamicum Cgpmt (Cg1014 or NCgl0854) and M.smegmatis Ms-pmt (Msmeg_5447) [START_REF] Liu | Bacterial protein-Omannosylating enzyme is crucial for virulence of Mycobacterium tuberculosis[END_REF] were cloned under the tac promoter into the expression vector pVWEx2 [START_REF] Eikmanns | A family of Corynebacterium glutamicum/Escherichia coli shuttle vectors for cloning, controlled gene expression, and promoter probing[END_REF]. The pmt genes were amplified from the genomic DNA of C. glutamicum ATCC13032 or M. smegmatis mc 2 155 by PCR using the primers DirCgPmt-XbaI and RevCgPmt6His-Bgl2 or DirMsmPmt-XbaI and RevMsmPmt6His-BamHI respectively (Table 1). The PCR products were digested with XbaI and Bgl2 or XbaI and BamHI and cloned into the XbaIand BamHI-digested pVWEx2 resulting in the pmt-expressing plasmids pVWEx2-CgPmt-His and pVWEx2-MsPmt-His.

Cloning lppX-tb in M. tuberculosis: Cloning of lppX-tb and lppX-tb ∆S16 T18A in the mycobacterial expression vector pVV16 was performed by PCR as described in Sulzenbacher et al. [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF] except for the DNA template. Here we used plasmid pMV261 LppX-tb and pMV261

LppX-tb ∆S16 T18A as templates.

Transformation of the bacterial strains was performed by electrotransformation as described for E. coli [START_REF] Dower | Nucleic Acids Research erythocytes Nucleic Acids Research[END_REF], C. glutamicum [START_REF] Bonamy | Interspecies electro-transformation in Corynebacteria[END_REF], M. smegmatis and M. tuberculosis [START_REF] Goude | Electroporation of Mycobacteria[END_REF].

Pmt gene disruption

The pmt gene (Cg1014 or NCgl0854) of C. glutamicum ATCC13032 was interrupted by integration of non-replicative plasmid with a kanamycin resistance cassette into the 5' part of the gene. Briefly, a 532 bp-DNA fragment of the pmt gene was amplified by PCR using primers PMT1-3 and PMT1-4 (Table 1). The DNA fragment was cloned into the pPrime cloning vector (5PRIME) and used to transform E. coli as described by the manufacturer. The recombinant plasmid was transferred into C. glutamicum by electroporation and transformants were selected on Km-containing plates. As the plasmid is unable to replicate into C. glutamicum, only the bacteria with the plasmid integrated into the genome were able to grow in presence of kanamycin. Correct integration of the plasmid into the pmt gene was checked by PCR analysis on the genomic DNA using combinations of primers localized upstream and downstream of the cloned 532 bp-DNA fragment (PMT1-1 and PMT1-6) and into the vector (M13R and M13F(-20)). After sequencing of the PCR products, one clone (pmt::Km r ) was used for further experiment.

Site directed mutagenesis of LppX-tb

Site-directed mutagenesis of the lppX-tb gene was performed by PCR-driven overlap extension [START_REF] Heckman | Gene splicing and mutagenesis by PCR-driven overlap extension[END_REF]. The mutagenic primers and flanking primers used to generate the mutated-lppX-tb genes are listed below (Table 1). Plasmid pCGL482 [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF] and pMV261 [START_REF] Tschumi | Identification of apolipoprotein N-acyltransferase (Lnt) in mycobacteria[END_REF] both carrying the native LppX-tb gene with a hemagglutinin (HA) and a His6 epitope fused to the 3' part of the gene were used as templates for PCR. The generated PCR fragments were digested with EcoRV-XhoI or EcoRI and cloned either into the EcoRV-XhoI digested pCGL482-AmyE [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF] for C. glutamicum or the expression vector pMV261 for M. smegmatis. All the mutated genes were sequenced to confirm the mutations. All mutated lppX-tb genes contained the hemagglutinin and a His6 epitope fused to their 3' part.

Purification of native LppX-tb and derivatives

Purification of LppX-tb and mutated LppX-tb from C. glutamicum was performed as described in Mohiman et al. [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF].

For purification from M. smegmatis, bacteria from liquid cultures were harvested and resuspended in a 50 mM phosphate buffer pH 8.0 containing 4-(2-Aminoethyl)benzenesulfonyl fluoride hydrochloride (AEBSF, Sigma) as a protease inhibitor. Cells were broken using a one shot cell disrupter (CellD) at 30kPsi. Unbroken cells were removed by low speed centrifugation (4000 g for 15 min) and the cleared supernatant was centrifuged at 106,000 g for 60 min (rotor Ti 50 Beckman). Total membranes were homogenized in a 50 mM phosphate buffer pH 8.0 containing 4% LDAO (lauryl dimethylamine oxide) and stirred 1h in order to efficiently solubilize membrane proteins. Insoluble material was removed by centrifugation at 106,000 g for 60 min. The soluble membrane proteins were diluted twice and NaCl and imidazole were added to final concentrations of 50 mM and 10 mM respectively in 25 mM phosphate buffer, 2% LDAO, pH 8.0. Ni-NTA agarose previously equilibrated with 25 mM phosphate buffer pH 8.0 with 2% LDAO, 50 mM NaCl and 10 mM imidazole was added to the soluble membrane proteins. The mixture was gently agitated for 1h at 4°C and loaded in an empty column. The agarose resin was washed with ten volumes of 25 mM phosphate buffer pH 8.0 with 1 % LDAO, 50 mM NaCl and 20 mM imidazole to remove weakly associated proteins. LppX-tb was eluted with the same buffer containing 50 mM imidazole. The eluted samples were precipitated with trichloroacetic acid (TCA) 10% to concentrate the proteins. If necessary further purification of LppX-tb was performed with anti HA agarose (Pierce) instead of TCA precipitation.

Extraction and analysis of proteins from different cellular localisation in C. glutamicum and M. smegmatis

The proteins from the envelope and culture supernatant of C. glutamicum were prepared as follows. After growth, cells were separated from the culture supernatant by centrifugation at 7000 g for 10 min. Proteins from the culture supernatant (S) were precipitated with 10% TCA (Sigma) at 4°C for 30 min. The pellet, recovered by centrifugation (13 000 g, 10 min), was washed with cold acetone and resuspended in TE buffer pH8 (Tris-HCl 10 mM EDTA 1 mM). Extraction of envelope proteins (E) of C. glutamicum was performed with SDS at high temperature. Briefly, the centrifuged cells were resuspended in Laemmli buffer and after heating 5 min at 100°C, cells were discarded after centrifugation. The supernatant containing the envelope proteins (E) and the proteins from the culture supernatant (S) were analyzed on SDS PAGE.

Extraction and analysis of the proteins from the capsule of M. smegmatis was performed as described by Sani et al. [START_REF] Sani | Direct visualization by Cryo-EM of the mycobacterial capsular layer: A labile structure containing ESX-1-secreted proteins[END_REF] using 1% Tween 80 and the proteins were precipitated with TCA before analysis on SDS PAGE.

Western blotting

Protein extracts (5 to 10 µg of total protein) were separated by SDS-PAGE (12%) and transferred to nitrocellulose membrane. Immunoblotting was performed using either anti-His6, anti-HA-or Concanavalin A peroxidase-conjugates (Roche) as recommended by the manufacturer. The blots were developed using clarity western ECL substrate (Bio-Rad). The ImageQuant LAS 500 was used for chemioluminescence detection.

Mass spectrometry analysis -Sample preparation-.

LppX SDS-PAGE bands, corresponding to different strains, were excised from the gel after Coomassie Blue staining. In-gel enzymatic digestion was performed using standard conditions. Briefly, protein bands were extensively washed with acetonitrile (ACN) and 100 mM ammonium carbonate. The excised bands were treated with 10 mM dithiothreitol (DTT) at 56 °C for 30 min. After DTT removal, cysteine carbamidomethylation was performed at room temperature for 30 min by addition of 55 mM iodoacetamide. After removal of the supernatant, the washing procedure was repeated, and gel slices were dried. Tryptic digestion was performed overnight at room temperature by addition of 20 µl of 10 ng/µl Porcine Gold Trypsin (Promega) diluted in 50 mM ammonium carbonate. Proteolytic peptides were extracted first by addition of 20 µl of 50% ACN and 0.1% formic acid and second by addition of 20 µl of 100% ACN. Tryptic peptides extracted were vacuum dried and resuspended in 0.05% ACN and 0.1% trifluoroacetic acid (TFA) prior to nano LC-MS/MS mass spectrometry analysis.

-Nano LC-MS/MS analysis-Nano LC-MS/MS analysis were performed with the triple-TOF 4600 mass spectrometer (Absciex, Framingham, MA, USA) coupled to the Nano-RSLC system (Thermoscientific). Briefly, peptides were desalted on a C18 reverse phase pre-column (C18 Acclaim Pepmap100, 3 μm, 100 Å, 75 μm i.d., 2 cm length) using a loading buffer containing H2O/ACN/TFA (98%/2%/0.05%) at 5μl/min and were then eluted at a flow rate of 300 nl/min from the reverse phase analytical C18 column (C18 Acclaim Pepmap100, 2 μm, 100 Å, 75 μm i.d., 50 cm length) using a 5-35% solvent B gradient for 40 min. Solvent B was 0.1% formic acid in 100% ACN and solvent A was 0.1% formic acid in water. Nano LC-MS/MS experiments were conducted in Data Dependent acquisition method by selecting the 20 most intenses precursors for CID fragmentation with Q1 quadrupole set at low resolution for better sensitivity and with a collision energy ramp (±15%) set to 35 V.

-Data analysis-Raw data were processed using MS data converter software (Absciex) and protein identification was performed using the MASCOT search engine (Matrix science, London, UK) against LppX sequences with carbamidomethylation of cysteines set as fixed modification.

Oxydation of methionines and glycosylation of serine and threonine were set as variable modifications. Peptide and fragment tolerance were respectively set at 25 ppm and 0.05 Da.

Only peptides with mascot ions score higher than the identity threshold [START_REF] Smith | O-linked glycosylation sites profiling in Mycobacterium tuberculosis culture filtrate proteins[END_REF] at less than 1% of false positive discovery rate are considered. For relative quantification of the LppX P6-29 glycoforms, MS extracted ion chromatogram (XIC) were realized on LppX P6-29 harboring glycosylation using peakview software 1.2 (Absciex).

Lipids extraction from culture supernatants, cell surface and cells of M. tuberculosis

Culture supernatants (50 ml) were sterilized by filtration, lyophilized and lipids were extracted following Bligh and Dyer protocol [START_REF] Bligh | Canadian Journal of Biochemistry and Physiology[END_REF] and analyzed by LC/MS. Cells obtained from 100 ml cultures were extracted with 3 ml of water-saturated butanol for 16h. The butanol extracts were collected and filter sterilized. The remaining cell pellets were extracted with chloroform: methanol (1:2, v:v) once and chloroform: methanol (2:1, v:v) twice. The combined extracts were subjected to Folch wash, dried and analyzed by LC/MS. LC/MS was performed using an electrospray ionization/atmospheric pressure chemical ionization source in the positive mode on an Agilent 6220 time-of-flight mass spectrometer equipped with a MultiMode Source.

Results

Identification of the glycosylation sites of LppX-tb in C. glutamicum

In a previous study, we reported that LppX-tb was O-glycosylated in C. glutamicum [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF]. The glycosylation sites were located within the N-terminal part of LppX-tb between amino acids 6 and 29 (Fig. 1). MS analysis identified heterogeneously glycosylated forms of LppX-tb carrying from 1 to 4 hexose units. In this study, we tried to identify the glycosylation site(s) within the peptide using conventional CID MS/MS techniques but unfortunately this was not successful. To determine the residues targeted by glycosylation in LppX-tb we thus performed a PCR-mediated approach to generate site-directed mutagenesis of putative glycosylation sites within the N-terminal part of LppX-tb. In this part of the polypeptidic chain, 3 hydroxyl-bearing amino acids were present (serine S16 and S20, threonine T18). All were predicted as putative glycosylation sites (Fig. 1) by the NetOGlyc 4.0 software [START_REF] Steentoft | Precision mapping of the human O-GalNAc glycoproteome through SimpleCell technology[END_REF]. Threonine T18 and serine S20 were individually substituted by an alanine leading to the mutated-genes LppX-tb T18A and LppX-tb S20A. As we failed in replacing serine S16 by alanine, this serine was deleted leading to the mutated-gene LppX-tb ∆S16. All mutated-genes were generated by PCR and cloned into the C. glutamicum expression vector pCGL482 under the control of the amyE promoter [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF]. Recombinant strains were grown in BHI at 30°C overnight. After growth, cells were separated from the media by centrifugation and broken. Unbroken cells were removed and total membranes (inner and outer membranes) were recovered by high speed centrifugation. The membranes were solubilized in LDAO and solubilized membrane protein extracts were loaded on Ni-NTA column to purify the native and mutated His tagged LppX-tb.

The purified proteins were analyzed by MS.

Different glycoforms of the N-terminal peptide were detected after proteolysis of the native LppX-tb protein (Fig. 2A): the peptide with no modification (P6-29), the peptide with 1, 2, 3 and 4 hexose units (P6-29 Hex1, P6-29 Hex2, P6-29 Hex3 and P6-29 Hex4). The P6-29 Hex2 form was the most abundant form detected (Fig. 2B). Analysis of the mutants shows no difference in the glycosylated pattern between the native LppX-tb and LppX-tb S20A. For the mutants LppX-tb ∆S16 and T18A, only P6-29 and P6-29 Hex2 were detected. A double-mutated LppX-tb gene ∆S16-T18A was finally constructed and did not reveal any glycosylated peptide (Fig. 2C).

All these results demonstrate that S16 and T18 are both native glycosylation sites and that each site is diglycosylated in the native protein. It is interesting to note that the glycosylation of S16 and T18 was not depending on each other. Moreover, despite modification of the glycosylation pattern, LppX-tb variants ∆S16, T18A and ∆S16-T18A were still membraneassociated.

Cg_PMT is essential for glycosylation of LppX-tb in C. glutamicum

As seen previously, LppX-tb is O-glycosylated with hexoses but the exact chemical nature of the modification was not determined. Nevertheless alpha mannosidase deglycosylation experiment suggests that LppX-tb is indeed mannosylated in C. glutamicum [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF]. A similar result is suggested in M. tuberculosis since LppX-tb was detected among the ConA affinity capture glycoproteome [START_REF] González-Zamorano | Mycobacterium tuberculosis glycoproteomics based on ConA-lectin affinity capture of mannosylated proteins[END_REF].

In C. glutamicum, different proteins were identified as glycosyltransferases. Among these proteins, Pmt (encoding gene NCgl0854) was identified as a protein O-mannosyl transferase which transfers mannose residues from lipid carriers to proteins [START_REF] Mahne | The Corynebacterium glutamicum gene pmt encoding a glycosyltransferase related to eukaryotic protein-O-mannosyltransferases is essential for glycosylation of the resuscitation promoting factor (Rpf2) and other secreted proteins[END_REF]. To investigate the possible role of Pmt in the glycosylation process of LppX-tb, a mutant of pmt was constructed in C. glutamicum. The pmt mutant (pmt::Km r ) was generated by integration of a kanamycin resistance cassette into the 5' part of the gene and the broad glycosylation protein pattern of the pmt mutant was tested. The wild type and pmt::Km r C. glutamicum strains were grown in BHI. The cells and culture supernatant were separated by centrifugation.

Treatment of whole cells with high concentrations of SDS is known to release cell wall components without affecting cell integrity in Corynebacterium. This method was used to extract proteins from the envelope. Proteins from the culture supernatant were recovered by TCA precipitation. Both proteins from the envelope and culture supernatant were transferred on SDS-PAGE and analyzed by immunoblotting with Concanavalin A peroxidase conjugate. As seen in Fig. 3A, several glycoproteins were detected both in the envelope and supernatant culture of the wild type strain. As expected no ConA reactive glycoproteins were detected in the pmt::Km r strain. The pmt gene of C. glutamicum (Cg-pmt) was cloned in the expressing vector pVWEx2 under the control of the tac promoter and the recombinant plasmid was transferred in the pmt mutant for complementation studies. When the wild type pmt gene is expressed in pmt::Km r , glycoproteins are again detected with a similar pattern to the wild type strain whether IPTG was added or not. As described in Mahne et al [START_REF] Mahne | The Corynebacterium glutamicum gene pmt encoding a glycosyltransferase related to eukaryotic protein-O-mannosyltransferases is essential for glycosylation of the resuscitation promoting factor (Rpf2) and other secreted proteins[END_REF], our results confirm that Pmt is responsible for the glycosylation of secreted proteins in the culture supernatants.

In addition, in this study we detected other pmt-dependent glycosylated proteins associated to the cell envelope of C. glutamicum.

As the pmt::Km r mutant was confirmed, we specifically checked if glycosylation of LppX-tb was dependent of this gene by expressing LppX-tb in the mutant pmt. The LppX-tb gene was cloned in the expression vector pCGL482 and used to transform the wild type and pmt::Km r mutant of C. glutamicum. LppX-tb was partially purified from the membranes of these strains and analyzed by SDS PAGE and western immunoblotting (Fig. 3B). As seen in Fig. 3B, a major protein of 25kDa was detected by Concanavalin A in the wild type strain. In the mutant pmt::Km r , a major band was detected slightly below than in the wild type strain and was not detected by the Concanavalin A conjugate suggesting that LppX-tb is not glycosylated in the pmt::Km r mutant. To confirm that this is not related to a polar effect of the mutation, the pmt gene of C. glutamicum Cg-pmt cloned into the expression vector pVWEx2 was co expressed with LppX-tb in pmt::Km r . In these conditions the migration behavior of LppX-tb was restored. The corresponding band was detected by Concanavalin A whether IPTG was added or not. Glycosylation of LppX-tb in the complemented strain was definitively confirmed by MS analysis (Suppl. Fig. S1). As for the wild type, different glycoforms of the peptide were detected (P6-29 Hex2 and P6-29 Hex4) as well as the 262 Da modification previously seen in LppXtb by Mohiman et al. [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF]. In the mutant pmt::Km r MS analysis confirmed that the peptide P6-29 was not glycosylated (Suppl. Fig. S1) which is in accordance with the absence of ConA binding seen on western immunoblot. These results demonstrate unambiguously that glycosylation of LppX-tb is dependent on Cg-pmt in C. glutamicum.

Complementation studies were also performed with the M. smegmatis pmt gene (Msmeg_5447). The PMT proteins encoded by Ms-pmt from M. smegmatis or Cg-pmt from C. glutamicum are quite similar with a sequence identity of 42% (Suppl Fig. S2). Moreover some residues important for the catalytic activity [START_REF] Mahne | The Corynebacterium glutamicum gene pmt encoding a glycosyltransferase related to eukaryotic protein-O-mannosyltransferases is essential for glycosylation of the resuscitation promoting factor (Rpf2) and other secreted proteins[END_REF] are conserved in both Ms-pmt and Cg-pmt. Despite this similarity, the heterologous complementation of the C. glutamicum pmt::Km r mutant with Ms-pmt was unsuccessful. Neither the glycoproteins from the envelope and supernatant culture nor LppX-tb were detected by Concanavalin A in the complemented strain (Fig 3C and3D). The absence of glycosylation of LppX-tb was also confirmed by MS analysis.

As the absence of glycosylation might be related to an expression problem, we tried to purify the protein PMT expressed from Ms-pmt during heterologous expression. The same experiment was performed with Cg-pmt as a positive control. In both cases, we were not able to detect the PMT proteins. Although this result suggests that very low amounts of Cg-PMT, is sufficient to allow complementation, it does not allow us to conclude for the absence of complementation by Ms-pmt.

Glycosylation of LppX-tb in M. smegmatis

Here, before proceeding on the role of LppX glycosylation, it was important to check whether LppX-tb glycosylation sites and pattern in Mycobacteria are the same as those found heterogously in C. glutamicum. For this, we choose to work in M. smegmatis, a fast growing species which is easily manipulated and in which several glycosylation studies have been performed so far.

M. smegmatis strain MC 2 155 carrying pMV261-LppX-tb [START_REF] Tschumi | Identification of apolipoprotein N-acyltransferase (Lnt) in mycobacteria[END_REF] was grown in BHI and LppXtb was purified as described in materials and methods. A similar experiment was performed in parallel with C. glutamicum using pCgl482-LppX-tb. Both purified proteins were analyzed by western immunoblot using Concanavalin A peroxidase conjugate. As expected, LppX-tb was detected by the ConA conjugate when purified from C. glutamicum. Surprisingly, we were not able to visualize any signal when LppX-tb was purified from M. smegmatis suggesting that either LppX-tb is not mannosylated or that the glycosylation level of LppX-tb is not high enough to allow detection. In order to discriminate between these hypothesis, LppX-tb was analyzed by MS analysis. As shown in Fig. 4, LppX-tb purified from M. smegmatis was found to be glycosylated on the N-terminal part of the protein. Several forms of peptide P6-29 were detected. The non-glycosylated form was the most abundant but small amounts of the Hex1 and Hex2 glycoforms were also detected. This low amount of glycoforms detected in M. smegmatis may account for the absence of detection by Concanavalin A.

As LppX-tb was found glycosylated in M. smegmatis, we wanted to determine if the glycosylation sites of LppX-tb in M. smegmatis were the same as in C. glutamicum. Mutations of the putative glycosylation sites S16, T18 and S20 of LppX-tb were generated according to the same procedure as for C. glutamicum except that the mutated genes were cloned in pMV261 an expression vector for mycobacteria. The recombinant plasmids were introduced into M. smegmatis MC 2 155 and the strains were grown in Middlebrook 7H9 broth supplemented with Middlebrook ADC enrichment and 0.5% glycerol and Tween 80 (0.05 %).

The different LppX-tb proteins were purified from the membrane fraction as described in materials and methods and analyzed by MS (Fig. 5). When expressed in M. smegmatis, different glycoforms of the peptide (peptide with no modification P6-29, peptide with 1 and 2 hexose units respectively P6-29 Hex1 and P6-29 Hex2) were detected with native LppX-tb as well as with the variant LppX-tb S20A. For the variant LppX-tb ∆S16 and T18A, only P6-29 and P6-29 Hex1 were detected demonstrating that both S16 and T18 are glycosylation sites and that each site can be monoglycosylated. This was confirmed by analysis of the double mutant LppXtb ∆S16 T18A in which no glycosylated peptide could be found. As for C. glutamicum it is interesting to note that the glycosylation of S16 and T18 was not depending on each other. These results show that the residues targeted by glycosylation in LppX-tb are identical in both C. glutamicum and M. smegmatis. This result is in line with a recent work supporting similar process of glycosylation in M. smegmatis and M. tuberculosis [START_REF] Alonso | Protein O-mannosylation deficiency increases LprG-associated lipoarabinomannan release by Mycobacterium tuberculosis and enhances the TLR2-associated inflammatory response[END_REF].

Native and ∆S16 T18A variant of LppX-tb are associated to the outerlayer of M. smegmatis

In our work, when expressed in M. smegmatis, LppX-tb was detected in the membrane fraction. In M. tuberculosis and M. bovis LppX-tb and its ortholog were found associated to the outermost layers of the cells that is the cell wall [START_REF] Lefèvre | Cloning of the gene encoding a 22-kilodalton cell surface antigen of Mycobacterium bovis BCG and analysis of its potential for DNA vaccination against tuberculosis[END_REF][START_REF] Mawuenyega | Mycobacterium tuberculosis Functional Network Analysis by Global Subcellular Protein Profiling[END_REF][START_REF] Hermann | Cell Envelope Proteomics of Mycobacteria[END_REF] and the cell surface [START_REF] Lefèvre | Cloning of the gene encoding a 22-kilodalton cell surface antigen of Mycobacterium bovis BCG and analysis of its potential for DNA vaccination against tuberculosis[END_REF]. In these mycobacteria, the cell wall is surrounded by a capsular layer [START_REF] Sani | Direct visualization by Cryo-EM of the mycobacterial capsular layer: A labile structure containing ESX-1-secreted proteins[END_REF]). We decided to analyze the mycobacterial capsular layer of M. smegmatis to see if LppX could also reach this outerlayer and whether S16 and T18 glycosylation was necessary for this localization.

The mycobacterial capsular layers of the different M. smegmatis strains MC 2 155 carrying pMV261, pMV261-LppX-tb and pMV261 LppX-tb ∆S16 T18A were extracted from the cells as described by Sani et al. [START_REF] Sani | Direct visualization by Cryo-EM of the mycobacterial capsular layer: A labile structure containing ESX-1-secreted proteins[END_REF]. For this experiment Tween 80 was not added in the culture media since this detergent is known to shed the capsule into the medium. After extraction, the proteins of the mycobacterial capsular layer were precipitated with TCA and analyzed by SDS PAGE and immunoblotting (Fig. 6A). The non-recombinant strain was used as a negative control. As seen in Fig. 6A similar amounts of proteins were extracted from the different strains. According to the immunoblot, LppX was only detected in the outerlayer of both recombinant strains whether S16 and T18 residues of LppX were modified or not.

As we noticed minor differences on SDS PAGE patterns of the different strains, an additional capsular extraction was performed and the protein content of the mycobacterial capsular layers were further analyzed on 8% SDS-PAGE (Fig. 6B). As seen in Fig. 6B, an additional protein was specifically detected in the strain expressing the ∆S16 T18A variant of LppX-tb. This protein was identified by MS as the 60-kDa chaperonin GroEL1 from M. smegmatis (Suppl Fig. S3). Interestingly, the presence of GroEL1 in the capsule layer of M. smegmatis and M. tuberculosis was previously reported [START_REF] Sani | Direct visualization by Cryo-EM of the mycobacterial capsular layer: A labile structure containing ESX-1-secreted proteins[END_REF]. Here we show that this protein is specifically enriched in the capsule of M. smegmatis expressing the LppX-tb ∆S16 T18A variant and this may suggest a specific function related to the absence of LppX glycosylation.

The ∆S16 T18A variant of LppX-tb is still able to promote translocation of phthiocerol dimycocerosates to the surface of M. tuberculosis

In M. tuberculosis phthiocerol dimycocerosates are required for virulence of the bacteria [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF]. Since LppX-tb is necessary for translocation of these molecules to the surface of M. tuberculosis and, because mannosylation was shown to contribute to M. tuberculosis virulence, we reasoned that glycosylation of LppX-tb might be important for LppX-tb function.

As disruption of LppX-tb in M. tuberculosis had no effect for growth in 7H9 broth [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF], this mutant was chosen for complementation studies with the native and the double mutated ∆S16 T18A LppX-tb gene. As previously described, in contrast to M. tuberculosis wild type strain, the LppX-tb null mutant fails to release DIM into the culture medium but retains the ability to translocate these lipids across the plasma membrane [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF]. We therefore decided to check the repartition of DIM in various mutants: the LppX-tb knock-out strain and the knock-out strains complemented with native or mutated LppX-tb cloned in the expression plasmid pVV16 (Fig. 7). For this experiment cells were grown in 7H9 medium containing ADC and 0.05% Tween 80 allowing the release of some outermost components of the cells in the culture media. In these conditions DIM molecules were present in the cells of all strains and all strains produced similar amounts of DIM (Fig. 7A). The chromatography profile of the DIM molecules was however slightly different in the LppX-tb knock-out cells as compared to the other cells. As expected and seen previously [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF] DIM molecules were absent from the culture filtrates of the knock-out strain. However DIM molecules were present in the culture filtrates of all complemented strains including the strain harboring the double mutation ∆S16 T18A (Fig. 7A). This result demonstrates that the presence of DIM in the culture filtrate of M. tuberculosis is not dependent on S16 and T18 LppX glycosylation of LppX-tb and therefore neither the localization of DIM in the outerlayer of M. tuberculosis.

Further analysis of the composition of DIM was performed by LC/MS. As shown in Fig. 7B, cells of the knock-out strain mainly contain DIM A molecules whereas cells (Suppl. Fig. S4) and culture filtrates of the two complemented mutants mainly contain DIM B molecules (Fig. 7B). Again the S16 and T18 LppX glycosylation state of LppX-tb has no impact in the composition of DIM molecules.

Discussion

In this article we used heterologous hosts to determine the glycosylation sites of the M. tuberculosis protein LppX-tb. A non glycosylated LppX-tb variant was then constructed and used to study the impact of glycosylation on LppX-tb function in M. tuberculosis. However O-glycosylation sites are most often localized within regions of low sequence complexity [START_REF] Schäffer | Emerging facets of prokaryotic glycosylation[END_REF]. Here in agreement with these studies we showed that LppX-tb glycosylation sites are localized in the N-terminal region of the lipoprotein LppX-tb in a region lacking any secondary structure [START_REF] Sulzenbacher | LppX is a lipoprotein required for the translocation of phthiocerol dimycocerosates to the surface of Mycobacterium tuberculosis[END_REF]. Moreover as seen for other mannosylated mycobacterial proteins [START_REF] Espitia | New insights into protein O-mannosylation in actinomycetes[END_REF][START_REF] Smith | O-linked glycosylation sites profiling in Mycobacterium tuberculosis culture filtrate proteins[END_REF] these modified sites are within a region enriched with amino acids proline and alanine. Both S16 and T18 residues are also present in the corresponding LppX-tb ortholog of M. bovis suggesting potential glycosylation at the same residues in this protein. In the M. leprae, M. marinum M. ulcerans and M. gordonae LppX-tb-orthologs these residues differ at position 16 and/or 18 but their N terminal regions are enriched with threonine and serine residues with some of them predicted as potential glycosylation target (Suppl. Fig. S5).

The glycosylation sites S16 and T18 of LppX-tb are identical in C. glutamicum and M. smegmatis but the glycosylated forms detected are different.

Whether LppX-tb is expressed in C. glutamicum or M. smegmatis, the lipoprotein is found to be modified by a hexose (probably mannose) at exactly the same glycosylation sites.

However the glycosylation pattern differs between both bacteria. In M. smegmatis up to 2 unknown modification of 262 Da occasionally detected in the N-terminal peptide of LppX-tb as seen previously [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF]. This modification was never detected when LppX-tb was expressed in M. smegmatis.

These results point out homologous sequence recognition in mycobacteria and corynebacteria but also minor differences reflecting their specific metabolism. In mycobacteria the highly glycosylated form of LppX-tb (Hex 3 and Hex 4) and the 262 Da modification detected in C. glutamicum may either be absent and/or the amounts synthetized not high enough to allow detection.

Glycosylation of LppX-tb is performed by Pmt

In C. glutamicum the gene pmt (Cg1014 or NCgl0854) encoding for a mannosyltransferase responsible for protein-mannosylation was previously identified [START_REF] Mahne | The Corynebacterium glutamicum gene pmt encoding a glycosyltransferase related to eukaryotic protein-O-mannosyltransferases is essential for glycosylation of the resuscitation promoting factor (Rpf2) and other secreted proteins[END_REF].

When expressed in the interrupted pmt mutant of C. glutamicum, LppX-tb is no longer glycosylated. Glycosylation is recovered when complementation with the Cg-pmt gene is performed. Thus our results confirm that Cg-pmt is involved in the glycosylation process of LppX-tb in addition to the polyprenol phosphate mannose synthase (cg-ppm1) and the apolipoprotein N-acyl transferase (cg-ppm2 or lnt) in C. glutamicum [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF] . Moreover as previously proposed [START_REF] Mohiman | The ppm Operon Is Essential for Acylation and Glycosylation of Lipoproteins in Corynebacterium glutamicum[END_REF] mannosylation of LppX-tb is confirmed. The exact role of Cg-pmt remains to be elucidated since Cg-pmt could be responsible both for the initial attachment of the mannosyl residue to the target residues and the elongation of the oligosaccharide chain. Alternatively other(s) glycosyltransferase(s) might be needed for the elongation step.

Previously identified glycosyltransferases in C. glutamicum [START_REF] Mishra | Lipoarabinomannan biosynthesis in Corynebacterineae: the interplay of two α(1→2)mannopyranosyltransferases MptC and MptD in mannan branching[END_REF] could be involved.

No complementation was noticed during heterologous expression of Ms-pmt of M. smegmatis in the interrupted pmt mutant of C. glutamicum. This result was surprising since we already observed successful complementation of C. glutamicum mutant with a M. smegmatis gene [START_REF] Dautin | Role of the unique, nonessential phosphatidylglycerol::prolipoprotein diacylglyceryl transferase (Lgt) in Corynebacterium glutamicum[END_REF]. However different hypothesis might explain this result. First during the mannosylation process Pmt transfers mannose from polyprenol phosphate mannose (PPM) to a target protein during its translocation across the membrane. In M. tuberculosis this process requires interaction with the Sec complex [START_REF] Vanderven | Export-mediated assembly of mycobacterial glycoproteins parallels eukaryotic pathways[END_REF]. Thus the interaction of Ms-pmt with the Sec translocon of C. glutamicum might not be functional. Second the polyprenols involved in mannose translocation in actinomycetales differ in chain length and saturation of the isoprene unit [START_REF] Espitia | New insights into protein O-mannosylation in actinomycetes[END_REF]. Only C55-polyprenolmannose was detected in C. glutamicum [START_REF] Gibson | Disruption of Cg-Ppm1, a Polyprenyl Monophosphomannose Synthase, and the Generation of Lipoglycan-less Mutants in Corynebacterium glutamicum[END_REF] whereas C35, C40 and C50-PPM were found in M. smegmatis [START_REF] Gurcha | Ppm1, a novel polyprenol monophosphomannose synthase from Mycobacterium tuberculosis[END_REF]. The protein Ms-pmt may not be able to recognize the C55-PPM of C. glutamicum and/or transfer the mannose. Finally the Ms-pmt may also be not correctly expressed or localized in the cell.

According to our results (that is evidence of mannosylation of LppX-tb in different actinobacteria and the role of Cg-Pmt) and considering that the proteins Pmt from C. glutamicum, M. smegmatis et M. tuberculosis are homologous [START_REF] Liu | Bacterial protein-Omannosylating enzyme is crucial for virulence of Mycobacterium tuberculosis[END_REF][START_REF] Mahne | The Corynebacterium glutamicum gene pmt encoding a glycosyltransferase related to eukaryotic protein-O-mannosyltransferases is essential for glycosylation of the resuscitation promoting factor (Rpf2) and other secreted proteins[END_REF] Sm-Pmt and Mt-Pmt are certainly responsible for LppX-tb mannosylation in the respective bacteria. Moreover as the glycosylated residues of LppX-tb are the same in C. glutamicum and M. smegmatis we can conclude that Cg-Pmt and Sm-Pmt are able to recognize identical glycosylation pattern in LppX-tb. Although very likely, more experimental results are needed to expand this conclusion to a larger group of proteins.

Distribution of DIM in the mycobacterial cell is not altered whether S16 and T18 residues of LppX-tb are glycosylated or not

In lipoproteins, an intrinsically disordered domain is generally found between the N terminal lipid anchor and the functional domain of the protein [START_REF] Zückert | Secretion of bacterial lipoproteins: through the cytoplasmic membrane, the periplasm and beyond[END_REF]. Various functions are associated with intrinsically disordered proteins or regions such as connecting ordered domains, regulation, molecular recognition and signaling and assisting in protein folding [START_REF] Prates | The impact of O -glycan chemistry on the stability of intrinsically disordered proteins[END_REF].

In lipoproteins this unfolded domain contains the lipoprotein sorting signal and is thought to properly position the lipoprotein [START_REF] Zückert | Secretion of bacterial lipoproteins: through the cytoplasmic membrane, the periplasm and beyond[END_REF]. Moreover Prates et al. [START_REF] Prates | The impact of O -glycan chemistry on the stability of intrinsically disordered proteins[END_REF] proposed that the elongation effect due to glycosylation of unfolded domains may be important to provide the optimum distance between the connected domains for protein function. As the LppX-tb glycosylated residues were positioned in this part of the lipoprotein, glycosylation could act as a signal for its export and/or function. These data prompted us to evaluate the possible impact of LppXtb glycosylation on its localization and function. Our results show that the outerlayer localization of WT and modified LppX-tb ∆S16T18A in M. smegmatis is identical. Moreover LppX-tb harboring mutated S16 and T18 residues is still able to promote DIM translocation across the cell envelope in M. tuberculosis suggesting that its function is not significantly impaired. A similar result was also described for LprG, a structurally homolog protein of LppXtb. Indeed, LprG, which is O-mannosylated with a unique hexose on a threonine residue [START_REF] Alonso | Protein O-mannosylation deficiency increases LprG-associated lipoarabinomannan release by Mycobacterium tuberculosis and enhances the TLR2-associated inflammatory response[END_REF] was also shown to be essential for correct surface localisation of complex lipids (mannosecapped lipoarabinomannan (LAM)) in M. tuberculosis [START_REF] Gaur | LprG-Mediated Surface Expression of Lipoarabinomannan Is Essential for Virulence of Mycobacterium tuberculosis[END_REF]. Interestingly, Alonso et al. showed that the surface localization of LAM in a pmt mutant of M. tuberculosis is not impaired [START_REF] Alonso | Protein O-mannosylation deficiency increases LprG-associated lipoarabinomannan release by Mycobacterium tuberculosis and enhances the TLR2-associated inflammatory response[END_REF].

Thus it seems that the distribution of complex lipids such as LAM and DIM in the mycobacterial cell is not dependent on the glycosylation state of their protein carriers LprG and LppX-tb respectively. Consequently these results suggest that the absence of LppX and LprG glycosylation does not account for the attenuation of virulence observed in the pmt mutant.

Is there a role for GroEL1 in DIM translocation?

In absence of mannosylation the function of LppX is not affected which is quite surprising as generally glycosylation is thought to affect protein folding and stability. However one can imagine that in the absence of glycosylation other factors may substitute for assisting LppX stability to maintain a sufficient level of DIM translocation at the cell surface. In that respect it is interesting to notice that Wang et al [START_REF] Wang | Biochemical and immunological characterization of a cpn60.1 knockout mutant of Mycobacterium bovis BCG[END_REF] previously discovered that the presence of phthiocerol dimycocerosates in M. bovis BCG is dependent on GroEL1. Because we show that the amount of GroEL1 is largely increased in the capsular layer of M. smegmatis only when the non-glycosylated variant of LppX-tb is expressed, it is thus tempting to propose that GroEL1 may be involved in the DIM translocation through an important role on LppX folding and/or stability. Indeed in the absence of glycosylation, the N-terminal non ordered chain of LppX may not be correctly extended and this effect could be reversed by GroEL1 overexpression. By extension, in M. tuberculosis, overexpression of GroEL1 could also counteract the absence of glycosylation of LppX-tb ∆S16-T18A and explain that DIM are still translocated at the cell surface. Further experiments are needed to validate this attracting hypothesis. For complementation studies cells were grown in presence (+) or absence (-) of IPTG. [START_REF] Sani | Direct visualization by Cryo-EM of the mycobacterial capsular layer: A labile structure containing ESX-1-secreted proteins[END_REF], separated on a 12% SDS-PAGE (A) stained with coomassie blue (Left) and analyzed by western-immunoblot using anti HA peroxidase conjugate (Right).

Legends to figures

Additional and refined analysis was performed on a 8% SDS-PAGE stained with coomassie blue (B). The additional protein identified by MS is indicated ( ). Tables Table 1. Primers used in this study for the cloning and disruption of pmt and obtention of the mutated lppX genes. The mutagenic and flanking primers used to generate site directed mutations in the lppX gene are indicated. The mutated lppX genes were cloned into the expression vector pCGL482 and pMV261.

Gene or plasmid Primers

Cg

4. 1 .

 1 The glycosylation sites S16 and T18 are localized in the unfolded N-terminal part of LppXtb No specific target sequon is known for O-glycosylation in contrast with N-glycosylation.

Fig. 1 .

 1 Fig. 1. Amino acid sequence of the mature lipoprotein LppX. Bold letters indicate the

Fig. 2 .

 2 Fig. 2. (A) Extracted ion chromatograms and (B) relative abundances of the LppX P6-29

Fig. 3 .

 3 Fig. 3. Pmt dependent protein glycosylation in C. glutamicum.

Fig. 4 .Fig. 5 .

 45 Fig. 4. (A) Relative abundance of the LppX P6-29 glycoforms observed during expression of the

Fig. 6 .

 6 Fig. 6. LppX is found in the outermost layer of M. smegmatis. M. smegmatis Mc 2 155 carrying

Fig. 7 .

 7 Fig. 7. LC-MS analysis of the distribution of DIM in the cells (cells) and culture filtrates

  5'-GGAATCACTTCGCAATGGCC-3'

  Fig. 1 730

Acknowledgments

This work has benefited from the facilities and expertise of the I2BC proteomic platform (Proteomic-Gif, SICaPS) supported by IBiSA, Ile de France Region, Plan Cancer, CNRS and Paris-Sud University.

Conflict of interest

Supplementary material

and M. smegmatis (PMT_Mycosmeg) using EMBOSS Needle https://www.ebi.ac.uk [START_REF] Madeira | The EMBL-EBI search and sequence analysis tools APIs in 2019[END_REF] (Identity : 228/539 (42.3%); Similarity : 310/539 (57.5%); Gaps : 42/539 (7.8%)). The conserved residues seen in various PMT proteins [START_REF] Mahne | The Corynebacterium glutamicum gene pmt encoding a glycosyltransferase related to eukaryotic protein-O-mannosyltransferases is essential for glycosylation of the resuscitation promoting factor (Rpf2) and other secreted proteins[END_REF] are framed in blue.

Suppl


Dhruv Agarwal

Tanay Agrawal

Laura M ‡1

Ferrari

Franc ¸ois Bremond

From Multimodal to Unimodal Attention in Transformers using Knowledge Distillation

Multimodal Deep Learning has garnered much interest, and transformers have triggered novel approaches, thanks to the cross-attention mechanism. Here we propose an approach to deal with two key existing challenges: the high computational resource demanded and the issue of missing modalities. We introduce for the first time the concept of knowledge distillation in transformers to use only one modality at inference time. We report a full study analyzing multiple student-teacher configurations, levels at which distillation is applied, and different methodologies. With the best configuration, we improved the state-of-the-art accuracy by 3%, we reduced the number of parameters by 2.5 times and the inference time by 22%. Such performancecomputation tradeoff can be exploited in many applications and we aim at opening a new research area where the deployment of complex models with limited resources is demanded

Introduction

New deep learning models are introduced at a rapid rate for applications in various areas. Among others, transformers are one of those that have sparked great interest in the computer vision community for vision and multimodal learning [START_REF] Khan | Transformers in vision: A survey[END_REF]. Numerous model variants of the paper "Attention is All You Need" [23] have been proposed and many works have centered around the attention mechanism. In multimodal learning the, cross-attention concept has been proven to be an efficient way to incorporate attention in one modality based on others. A key existing challenge is the high computational resource demanded, even at inference * These authors contributed equally to this work time, and therefore the need for GPUs with large memory.

In this paper, we study how to minimize the time and resources required using a transformer based model trained on multimodal data. We utilize the concept of knowledge distillation (KD) to use only one modality at inference time. Basically, KD transfers knowledge from one deep learning model (the teacher) to another (the student), and originally, it was introduced as a technique to reduce the distance between the probability distributions of the output classes of both networks [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF]. When it comes to applying KD to tranformers the first issue is related to the level on which to apply it. In a cross-attention transformer the modalities other than the primary one, the query, are taken as key and value. The primary modality has skip connections allowing it to preserve more information during back-propagation. Thus, when trying to distill information from the teacher (e.g. a cross-attention transformer), at the output level, the student does not have enough knowledge about the other modalities. Thereafter, distillation need to take place at lower feature levels. In order to do so, we propose here the use of Contrastive Representation Distillation (CRD) [21], a recent method for KD with state-of-the-art performances which enables transfer of knowledge about representations. As KD has been effectively applied in deep learning but little has been explored with transformers, a study is here presented comparing multiple student-teacher configurations, KD applied on multiple levels, and comparison of two methods for distillation.

We chose emotion recognition as our application since it is a complex problem where multimodality showed significant improvement [17]. Human emotions possess not only the behavioral component, expressed as visual attributes; they have as well the cognitive and the physiological aspects. This rich information allows us to emphasize and interact with others. For this reason, the use of multimodal inputs, as verbal (e.g. spoken words) and acoustic features (e.g. tone of voice), permitted to combine with the video data complementary salient information increasing the results accuracy. Nevertheless, the limits of multimodal analysis are various. Besides the high computational cost, one of the main issues is related to missing modalities. In the real world, we may not have all the modalities available and some parts of the data can go missing due to hardware, software or human faults. Moreover, the use of heterogeneous data causes synchronization issues, linked to the frequency at which the modalities are recorded. To address these aspects we propose a novel approach and our contributions are summarized below.

1. At the best of our knowledge, we introduce for the first time KD in a transformer architecture for multimodal machine learning. We reduce the computational cost at inference time and able to use just one modality (e.g. video), improving the state-of-the-art accuracy by 3%.

2. We develop a framework to study multiple configurations of student-teacher networks and distillation. We especially compare two methods to distill knowledge, the CRD and the use of cross-entropy loss in the attention module of transformers. We name this second method as Entropy Distillation in Attention Maps (EDAM).

The remaining of this paper is organized as follows. Section 2 reports the main findings in the state-of-the-art with respect to multimodal machine learning, transformers and KD. Section 3 describes the dataset exploited for the experiments. In Section 4, we present the developed detailed framework of the student-teacher network architectures and the KD methods applied. Section 5 discusses the results and section-6 concludes with a summary of our contributions and future perspectives.

Related Work

Multimodal machine learning aims at developing models that can process information from multiple input types. One of the earliest work was in audio-visual speech recognition [26], while more recently other fields have started exploiting multimodality as multimedia content indexing [20] and emotions [3,28] or personality [START_REF] Nojavanasghari | Deep multimodal fusion for persuasiveness prediction[END_REF] recognition. First works in the field proposed the use of early fusion [START_REF] Morency | Towards multimodal sentiment analysis: Harvesting opinions from the web[END_REF][START_REF] Pérez-Rosas | Utterancelevel multimodal sentiment analysis[END_REF] while late fusion has started to be explored more recently [24]. In the last years more complex architectures have moreover been proposed, as the Tensor Fusion Network for intra-and inter-modality dynamics [START_REF] Degottex | COVAREP -A collaborative voice analysis repository for speech technologies[END_REF] or the use of an attention gate to shift the words through visual and acoustic features [START_REF] Wang | Words can shift: Dynamically adjusting word representations using nonverbal behaviors[END_REF]. From the famous work by Vaswani et al. [23], in 2019, Tsai et al. [22] proposed the Multimodal Transformer (MulT) to learn representations directly from unaligned multimodal data. In this work cross-modal attention is used to latently adapt elements across modalities, in an end-to-end manner, without explicitly aligning the data and inferring long term dependencies across modalities. In our proposed framework we exploited the MulT architecture for developing the student and teacher network architectures.

The research field of Multimodal Machine Learning brings some unique challenges, given the heterogeneity of the data, that have been deeply explored [START_REF] Oviatt | The handbook of multimodal-multisensor interfaces[END_REF]. One of the main core challenges is the so called co-learning, that is related to the capability of transferring knowledge between modalities. Another core challenge is the representation of heterogeneous data, meaning learning how to summarize multimodal data exploiting their complementarity [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF]. This aspect is linked to the model robustness, that is a relevant theme in real world applications, which in turn is correlated to missing data/modalities. Finally, a common general issue of these models is their high computational cost due to their complex architecture, especially when dealing with transformers [10]. This is an emerging area of interest and just lately a multimodal end-to-end model have been proposed with a sparse cross-modal attention mechanism to reduce the computational overhead [6]. An unexplored method to reduce the inference time and dealing with missing modalities is the use of KD. The idea is to leverage on multimodality to build rich representational knowledge that can be transferred to a lighter network, able to infer through just one modality. The concept of KD was firstly introduced by Bucilua et al. [2] and Hinton et al. [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], where a large teacher network is distilled into a small and faster student network to improve the performance of the student at test time. From here on a wide variety of works have been proposed [START_REF] Dai | Learning an augmented rgb representation with cross-modal knowledge distillation for action detection[END_REF][START_REF] Crasto | Mars: Motion-augmented rgb stream for action recognition[END_REF] and recently many self-supervised methods leveraging on contrastive loss achieved good performance [START_REF] Oord | Representation learning with contrastive predictive coding[END_REF]4]. Among those works, the CRD method proposed by Tian et al. [21] reported state-of-the-art results proposing to use a contrastive learning strategy for transferring the knowledge in multiple scenarios, comprehending cross-modal transfer and ensemble distillation.

In this work we analyzed the use of CRD in multiple levels of the proposed framework and in a deeper one, in the attention maps of the transformers, we compared the CRD with what we called the EDAM method. We opted for the EDAM methodology as it is a simpler and runs with less computational cost. This comparison has been performed uniquely at the attention layer as this is the only level that outputs probability distribution, essential for the EDAM loss (i.e. cross-entropy loss).

Dataset

The dataset used to do experiments in the proposed framework is the CMU-MOSEI dataset [28], which is one The video and the audio data are directly extracted from the clips while the language data are transcribed and properly punctuated. Each sample from the dataset is labeled by human annotators with continuous values from -3 (strongly negative emotion) to 3 (strongly positive emotion), which we discretized in 7 emotion classes (-3, -2, -1, 0, 1, 2, 3) to model our emotion classification task. Due to the variable sampling rates of different modalities, there is an inherent non-alignment in the dataset. The dataset is moreover available in both the aligned and non-aligned mode and we used non-aligned one in our task. We used the extracted features by [22] for each modality from the dataset. For video, Facenet [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] was adopted to give 35 facial action units showing muscle movement for each from the video. For audio, COVAREP [7] is used to get low level acoustic features. As language is the transcript of the audio, the chosen audio features represent the non-semantic features like pitch tracking, loudness and maxima dispersion quotients. For language, Glove word embeddings (glove.840B.300d) [START_REF] Pennington | Glove: Global vectors for word representation[END_REF] is used to encode the transcript words in 300 dimensional vectors.

The Proposed Framework

The framework is a student-teacher network based on transformers. The student network exploits self-attention (uni-modal attention) while the teacher exploits crossattention (cross-modal attention) on the input sequences, as detailed in section 4.1. The outputs from the transformers are concatenated and passed through linear layers to make the final predictions on emotion classes (Fig 1). In section 4.2 we present the multiple student-teacher configurations experimented here. The capability of transferring knowledge from multiple modalities to one is performed through distillation, at four levels of the framework. As described in section 4.3 we employ KD in the linear layers and in the transformer layers (Fig 1). For each of these cases, we calculate the final loss (Loss) as the sum of the distillation loss(L KD ) and classification loss (cross-entropy loss, L c ), as stated in equation ( 1), which is used to train the student network. While training, we keep the trained weights of the teacher network fixed and only backpropagate gradients through the student network.

Loss = αL c + βL KD (1)
where α, β are hyperparameters in the range [0, 1]

Teacher and student Transformers

Teacher and student networks take multimodal sequences (Video, Audio, Language) as input and outputs the likelihood of the 7 emotion classes. The architecture of the networks are derived from [22]. and are detailed below.

Teacher network: The teacher network takes in input all the modalities and pass them through the respective Conv1D layers, along the temporal dimension, with kernel size of 3. This passage ensures each element in the sequence is aware of its neighboring elements. The outputs from the Conv1D layers are added with sinusodial positional embeddings before they enter into the transformers, to carry the temporal information [23]. Since we have 3 modalities in the teacher network, we use 6 transformers applying cross-attention in every combination of Query (Q), Key (K) and Value (V) pairs, coming from 2 different modalities. For example, the transformer V T ← A T (Fig 1 )  represents a transformer where Q is from modality A (Audio) and K and V are from modality V (Video). The transformers A T ← V T and L T ← V T form the Video Branch of the teacher network, as they both use video modality as Q and have the same output dimension at the end of all transformer layers. Similarly, V T ← A T and L T ← A T form the Audio Branch, and V T ← L T and A T ← L T form the Language branch. All three branches together form the Complete Teacher Network represented in Fig 1 . We experimented on four teacher configurations. One is the Complete Teacher Network and the other three are the individual branches: the Video, the Audio and the Language branches. The output sequences from each of the 2 transformers in a branch are concatenated along the last dimension to be further input respectively to another set of 3 transformers. This set of transformers applies temporal attention on the fused output of previous transformers. The last sequence elements from the output sequences of the 3 transformers are, individually passed (for individual branches), concatenated and passed (for Complete Teacher Network), through linear layers.

Student Network: In the student network the Video is the only input modality and it is passed to the model in parallel three times as shown in Fig 1 . Each of the inputs goes through a Conv1D layer with different sized kernels to downsample them. Outputs of the 2 nd and 3 rd Conv1D layers (from left in Fig 1) are used as proxy for missing audio and language modality sequences and are hence named A S and L S respectively. The naming scheme of the transformers in the student network is identical to that of teacher network with subscript "S" in place of "T" to distinguish between "Student" and "Teacher".

We experimented on four student configurations. One is developed same as the Complete Teacher Network, with 9 transformers. The other three configurations are a simplified version of the previous, with 3 transformers. These 3 transformers vary for each student configuration as detailed in the next section, one of those configuration (the 5th configuration in section 4.2) is reported in Fig 1 . The last sequence elements of output sequences from all 3 transformers are concatenated and passed through linear layers.

The following section elaborate the different configurations of student-teacher networks.

Student-Teacher Configurations

We study multiple pairs of student and teacher networks to see the effect of distillation on the different architectures.

1. KD from the Complete Teacher Network: The teacher is the Complete Teacher Network and the student is constructed exactly the same, except for the inputs. All the 9 cross-attention transformers from the teacher are used for distillation to the corresponding self-attention transformers in the student network.

KD from Video Branch:

The teacher is the Video Branch and the student network consists of 3 transformers: V S ← V S , A S ← V S , and L S ← V S . The transformers A T ← V T and L T ← V T are used for distillation to corresponding transformers in the student.

KD from Language Branch:

The teacher is the Language Branch and the student network consists of 3 transformers,

V S ← V S , V S ← L S ,

Knowledge Distillation

Since different layers in a deep network carry different information, we explore applying distillation to various stages of the network and we compare two methods for loss calculation, CRD and EDAM. We apply CRD at four stages, two at high-level features, in the final and penultimate linear layers, and the other two at the transformer level, where we compare the use of contrastive with cross-entropy loss.

Overview of Contrastive Representation Distillation (CRD):

The CRD method [21] provides a general framework to bring closer the representation of "positive pairs" from student and teacher networks while pushing apart the representations of "negative pairs". A pair is called positive when the same sample from a dataset is provided to both the networks, while it is negative when different samples from the dataset are input to the networks. In the following, we use the expression, CRDLoss(X, Y) to imply CRD being applied on 1D feature vectors X and Y.

1. CRD on final and penultimate layers: CRD is applied on the final and penultimate layers of the student and teacher networks. Here the L KD is calculated by applying the CRD loss at the output of the respective layers from the student and the teacher network, as stated in the following equation

L KD = CRDLoss(Student (l) , T eacher (l) ) (2)
where l indicates the l th layer of the networks.

2. CRD on post-attention linear layers: CRD is applied on the outputs of the attention module of the studentteacher transformer pairs (see Fig- 1 in Supplementary Materials). Here the L KD is calculated as stated in the equations (3) -(7) and we take the average of computed L KD for all the pairs when we have multiple pairs in an configuration.

Attention(Q X , K Y , V Y ) = sof tmax( Q X K T Y √ d k )V Y (3) DisLvl(Y ← X) = Attention(Q X , K Y , V Y ) (4)
where X and Y denote output sequences from Conv1D layers (i.e. V S , A S , L S , V T A T , and L T ), and Y ← X indicates that the sequence X serves as Q, while K, V of the transformer come from sequence Y.

Since our transformers have multiple attention layers, we store the DisLvl(Y ← X) computed from multiple layers in the variable, DislList as follows

DisList α S ←β S = [DisLvl(α S ← β S )] i=l i=1 (5)
DisList α T ←β T = [DisLvl(α T ← β T )] i=l i=1 (6)
where α and β are input modalities (Video, Audio, or Language), subscripts "S" and "T" refer to Student and Teacher respectively, and l is the number of layers in the student and teacher transformers. Finally, L KD is calculated using equation ( 7)

L α←β KD = l i=1 CRDLoss(DisList α S ←β S i , DisList α T ←β T i ) l (7)
3. CRD on Attention Maps:. CRD loss is applied on the attention map of each transformer pairs (see Fig- 1 in supplementary Materials). The attention map of a transformer layer is described in the following equation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF].

DisLvl(Y ← X) = sof tmax( Q X K T Y √ d k ) (8)
In our cross-attention transformers (the teacher network), Q comes from different modality than K and V while all are from the same modality in case of self-attention transformers (in student network). The teacher network benefits from multiple modalities as it has the opportunity of extracting features from all combinations of modalities put as Q and (K, V), which however is not the case with student networks. Distilling different attention maps from multiple transformers can account for the missing modalities in the student networks to an extent and can be conducive in learning richer representations. In an attention map, the sum of values in every row is 1 and every cell has value in the range 0 -1. We therefore see every row as a probability distribution over n classes where n is the number of columns in the attention map of dimension m × n. Through distillation we aim to bring closer the probability distribution of student network to the teacher network and at the same time push apart the distributions of positive samples away from negative samples. (L KD ) for one transformer pair is thus calculated using equations, ( 5), (6) (with DisLvl defined in equation ( 8)), and ( 9). In the configurations with multiple transformer pairs, average over L KD obtained from all pairs serves as the final L KD

L α←β KD = l i=1 m j=1 CRDLoss(DisList α S ←β S ij , DisList α T ←β T ij ) ml (9)
where DisList Y ←X ij denotes the j th row of the i th layer attention map from transformer Y ← X. m = number of rows in attention map and l = number of layers in the transformer 4. EDAM on Attention Maps: The cross-entropy loss is applied on the attention maps of each transformer pairs. For an attention map of m × n dimension, we view the rows as probability distribution over n classes and define the distillation loss as a modification to classical cross-entropy loss between student and teacher attention map rows. Similar to the case of CRD on attention maps, we obtain attention maps from all transformer layers using equations ( 5) and ( 6) (with Attention map / DisLvl defined using equation ( 10) and ( 11))

DisLvl(Y ← X, t) = temp sof tmax( Q X K T Y √ d k , t) (10) temp sof tmax(X, t) (i) = exp(X (i) /t) k j=1 exp(X (j) /t) ( 11
)
where t is the temperature parameter which adjusts the sharpening of distribution.

and calculate the distillation loss for a pair using the equations ( 12) and ( 13).

L α←β KD = l i=1 m j=1 F (DisList α S ←β S ij , DisList α T ←β T ij ) ml (12) F (a, b) = -alog(b) (13)
For the configuration with distillation on multiple transformer pairs, final L KD is calculated by averaging over L KD obtained from all pairs.

We employ the EDAM in two ways, named EDAM-S↓ and EDAM-T↑ to deal with diverse dimensions, due to the different inputs of the teacher and the student.

The teacher has multiple modalities as input while the student has only one modality, and each modality may have a different sequence length. Thereafter the attention map in the student and teacher networks can be of different dimensions. To handle this, in EDAM-S↓ we downsample the Video inputs to the student, A S and L S using Conv1D to match them with Audio and Language input in teacher network, A T and L T respectively.

Instead of downsampling inputs, in EDAM-T↑ we upsample the attention maps in the teacher transformer layers to match with the dimensions of attention maps in the student transformer layers using linear layers.

Results and Discussion

Table-1 reports the results obtained with the proposed framework by varying the teacher network design and the KD levels. The table is arranged with alternating student and teacher rows. The first row shows the result for just the student network, without KD. Following that, every teacher-student row pair has the teacher network performance and the performance for student, with different distillation methods (Section-4.3).

We achieved the best result with the simplified Student Network and the Complete Teacher Network (the 5th configuration in section 4.2), through the EDAM-S↓ method on attention maps. The accuracy in this case is 44.231% which improves the one-modality case (row-1 of the Table-1: 41.296 %), performed with state-of-the-art architecture [22], by approximately 3% in accuracy and 4 points in F1score. We further compared the number of parameters of our best performing student network (0.675 Million) to the complete teacher network (1.802 Million). The parameters are reduced of around 2.5 times, which leads to a minimized inference time of 29ms (22% less than the Complete Teacher Network). Note that, the results obtained for KD from Complete Teacher Network (row-2 in Table-1) were not optimized rigorously as the behavior of the smaller student (row-5 in Table-1) was similar to it.

Notably, the improvements, in accuracy and F1-score, occurred in the lower layers of the architecture, in the attention map. In the Fig- 3 of Supplementary Material we compare the first two layers of the attention maps of the teacher and of the students, with and without KD. Here we clearly show the improvement got through KD application. This proves our initial hypothesis, suggesting that higher layers do not provide enough information in cross-attention transformers to the student network. Moreover, by comparing CRD with EDAM methods we found out that EDAM, the lighter one, gives the best results. Therefore we were able to obtain the highest accuracy and F1-score with the less computational requirement. A challenge with applying distillation to attention maps is that the dimensions of the teacher and student are different as the student capitalizes on self attention, while the teacher on cross-attention. We explored two methodologies to overcome this issue, downsampling (EDAM-S↓) and upsampling (EDAM-T↑), showing that the former works better. This result is intuitive as the features are designed to be as orthogonal as possible and reducing the map size using a linear layer will lead to loss of information.

In this work we used unaligned modalities as input. Nevertheless, if aligned modalities are adopted instead, the downsampling method for which loss of information is linked to the reduced temporal resolution, would have shown even better performance.

Note that in Looking at the diverse teacher configurations, the use of the complete teacher network or of the individual teacher branches, does not have a significant impact on the results (see results of teacher networks in the Table -1). This can be due to the high temporal and semantic correlation among the modalities with respect to the output. On the other hand, individual branches teacher are smaller in size, therefore quicker to train.

Finally, looking at the results, we noted that in some cases (e.g. in the case of Teacher with Language Branch) accuracy decreases while the F1-score increases when applying KD in deeper layers. This can be due to the fact the data is skewed and the better accuracy reached in the higher layers is obtained by giving the majority class as output most of the times. When applying KD in the deeper layers the network better learns to take into account the other classes, so the accuracy decreases. Controversially the F1 score, which is a good metric for imbalanced data, is aligned with the expected trend, where the results improve applying KD in deeper layers.

Conclusions

In this work we explore KD at various levels and with multiple student-teachers configurations. We establish that going deeper in the transformer network is conducive to KD by getting best results with KD on attention maps.

In this first work we used as single modality, in the student network, the video; nevertheless, the same methodology can be adopted to other modalities. Studying the multiple teacher configurations we demonstrated that using different modalities as query, key and value does not have much impact on the results. This will help future studies, to reduce the effort in the evaluation of diverse permutations.

Since this is an unexplored field, multiple directions are envisioned as future work. First, others techniques of distillation can be explored. The EDAM-S↓ method, based on simple cross-entropy loss achieved the best accuracy and F1-score, and required fewer learning parameters compared to KD with CRD loss. This serves as a proof of concept and opens the possibility for experimenting different loss functions for KD on attention maps. The accuracy of onemodality based models can be moreover improved by exploiting other modalities, such as the physiological data. We believe that, while strong emotions can be better recognized by using one modality with distilled knowledge, subtle emotions are more complicated to be addressed and physiological cues could add relevant dependencies in the final one-modality. We believe our approach is robust and extending it to other datasets and fields of work is a pertinent next step.

We aim that this paper will pave the way for a novel research area in multimodal machine learning focusing on resources reduction. This will permit the use of complex algorithms when the inference time has to be minimized, exploiting a richer embedding space and increased accuracy.

An example of application is in the field of emotion recognition, where high computational cost can not be sustained and some modalities might be partly missing.

[28] A. B. Zadeh

A. Supplementary Material

This supplementary material provides more insights into our proposed methodology. In the section-A.1, we elaborate how KD is applied at the transformer level. In section-A.2 we visualize KD in action and finally in section-A.3 we disclose the training details of our experiments.

A.1. KD at transformer level

In section 4.3, we proposed two levels at which KD can be applied inside the transformer architecture, KD at attention map and KD at post-attention linear layers.

In the attention module of a transformer, Q, K, and V ∈ R n×d (where n = sequence length, and d = feature embedding length) are are divided into n h chunks where n h is the number of attention heads in a transformer. The n h chunks of Q, K, and V ∈ R n×d/n h undergo the attention mechanism (equation highlighted in the Fig- 2) to give n h outputs of attention maps. The average of all these n h attention maps is then serves as the attention map which is used for distillation Similar to the attention maps, the output of the product of attention map with V gives a feature vector ∈ R n×d/n h . All these n h feature vectors are concatenated along the feature dimension to get back the feature vector ∈ R n×d (this operation is illustrated with concatenation layer in the Fig- 2). The resultant feature vector is then passed through linear layers to get what we call as post-attention linear layer, used for the distillation.

We extract these attention maps and post-attention linear layers for all the layers of the transformers and apply KD on all of them.

A.2. Visualizing KD

Here we present a way to visualize the effects of Knowledge Distillation on Attention Maps of the student transformers. In Fig- 3, we plot the attention maps of student and teacher transformers before and after applying EDAM-S↓ on attention maps. From the figure we can clearly see that before distillation the attention maps of teacher network and student network are very different and thus both the models learn disparate representations. However, after applying EDAM-S↓ on attention maps, the attention maps of student network begin to align themselves to that of teacher network, thus getting closer to the teacher network at lower levels. This we believe is conducive for learning richer rep-resentations, closer to teacher network, as the outputs from the attention maps are further passed to high-level layers which learn representations built upon low-level features. In Fig- 3 we only provide the attention maps of one of the transformer pair due to lack of space, however, with EDAM-S↓ on attention maps, all of the transformers from student closely resembled the corresponding teacher transformers.

A.3. Training Details

All of our experiments were performed on 1 Nvidia RTX-8000 with a batch size of 64. We used the same data splits (train, val, test) as used by [22] in their approach. We trained our models on training set for 100 epochs and used the weights which gave lowest validation loss to compute results on test set.

We used a learning rate of 1e-3 and decayed it with ReduceLROnP lateau with a patience value of 10 epochs and a factor of 0.5. In all our transformers we had 4 layers with 8 attention heads.

Figure 1 :

 1 Figure 1: The proposed framework, with the student and teacher networks. The knowledge distillation is performed at high level, in the linear layers, and at lower level, inside the transformers. The student network (left) here is the simplified 3-transformers based architecture, while the teacher (right) is the Complete Teacher Network with 9 transformers

5 .

 5 and A S ← L S . The transformers V T ← L T and A T ← L T are used for distillation to corresponding transformers in the student. 4. KD from Audio Branch: The teacher is the Audio Branch and the student network consists of 3 transformers, V S ← V S , V S ← A S , and L S ← A S . The transformers V T ← A T and L T ← A T are used for distillation to corresponding transformers in the student. KD from Language and Audio Branches of Complete Teacher Network: The teacher is the Complete Teacher Network and the student network consists of 3 transformers, V S ← V S , V S ← L S , and V S ← A S as depicted in the Fig 1. The transformers V T ← A T and V T ← L T are used for distillation to corresponding transformers in the student.

Figure 2 :

 2 Figure 2: Knowledge distillation in the student-teacher transformers. Representation of the two deeper layers, in the transformers architecture, where distillation is applied, at attention maps and at the post-attention linear layer.

Figure 3 :

 3 Figure 3: KD Visualization using EDAM-S↓ on attention maps: Top-row = Attention maps from 1 st and 2 nd transformer layers of V T ← A T respectively, Middle-row = Attention maps from 1 st and 2 nd layers of V S ← A S after KD respectively, and bottom-row = Attention maps from 1 st and 2 nd layers of V S ← A S before KD respectively

Table 1 :

 1 Table-1 we did not optimize the hyperpa-Results on CMU-MOSEI Dataset -Unaligned rameters for the CRD on attention map, as the outputs were closed to the EDAM method that do not required any additional parameters for training.

	Network Description	Accuracy	F1-
		(%)	score
	Student Without KD	41.296	32.142
	KD from the Complete Teacher Network
	Teacher Complete Teacher Net-	49.779	48.817
	work
	Student EDAM-S↓ on Atten-	44.137	36.286
	tion Maps
	KD from Video Branch
	Teacher Video Branch	49.621	48.542
	Student CRD on final linear	43.711	33.281
	layer
	CRD on penultimate	42.576	35.565
	linear layer
	CRD on attention maps 43.837	35.912
	KD from Language Branch
	Teacher Language Branch	49.217	47.250
	Student CRD on final linear	43.154	32.863
	layer
	CRD on penultimate	42.901	34.467
	linear layer
	KD from Audio Branch
	Teacher Audio Branch	49.217	47.232
	Student CRD on final linear	43.584	33.260
	layer
	CRD on penultimate	43.232	35.115
	linear layer
	KD from Language and Audio Branches
	Teacher Complete teacher net-	49.779	48.817
	work
	Student CRD on final linear	42.731	32.513
	layer
	CRD on penultimate	43.732	35.632
	linear layer
	CRD on post-attention	43.837	35.711
	linear layers
	CRD on attention maps 44.031	36.110
	EDAM-S↓ on attention	44.231	36.332
	maps
	EDAM-T↑ on attention	43.993	36.189
	maps

  , P. P. Liang, S. Poria, E. Cambria, and L.-P. Morency. Multimodal language analysis in the wild: Cmumosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246, 2018. 2

Acknowledgements

This work has been partially supported by Ville de Nice and French government through the UCAJEDI Investments in the Future project managed by the National Research Agency (ANR) (ANR-15-IDEX-01).