Yujia Liu

Kazuki Onodera

Nobuhiro Takeda

Armelle Ouali

Masafumi Unno

Synthesis and Characterization of Functionalizable Silsesquioxanes with Ladder-type Structures

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

ABSTRACT: Novel tetravinyl-and tetraallyl-substituted laddersiloxanes were synthesized and characterized. It was shown that peripheral olefins could successfully undergo hydrosilylation quantitatively, possibly leading to tetrahalide ladder structures. All reported functional laddersiloxanes LS (LS-Vinyl4, LS-Allyl4, LS-Cl4 and LS-I4) thus constitute promising building blocks for more complex inorganic-organic hybrid materials.

Silsesquioxanes, organosilicon compounds with the empirical formulas RSiO3/2 (R = H, alkyl, aryl), have been extensively studied due to their interesting properties such as thermal stability, low dielectric constant and water repellency. 1 Several types of well-defined silsesquioxanes have been reported and they include partially condensed, cages 2 (e. g. T8, T10 and T12) or ladder structures (so called "laddersiloxanes"). The latter, possessing highly ordered double-chain structures and possibly presenting polymeric forms are of particular interest. 3 For example, they appear as attractive candidates in the field of electronic devices where highly thermally stable and well-defined structures without defects are required to improve the physical properties without altering the electronic properties of the organic semiconducting fragment. [START_REF] Kamino | The use of siloxanes, silsesquioxanes, and silicones in organic semiconducting materials[END_REF] Since the preparation of the first bicyclic and tricyclic laddersiloxanes about 50 years ago, many efforts have been devoted to improve synthetic routes to these compounds as well as the characterization methods, 3 but also to reach pentacyclic, 5 heptacyclic 5c,6 and even nonacyclic 7 ladder structures from our group.

The current work concerns tricyclic laddersiloxanes. These compounds can be composed of three eight-membered rings (8-8-8-membered fused rings) or possess two external six-membered rings (6-8-6-membered fused rings) (Scheme 1, A). The first ones could be obtained e.g. by reaction of dihydroxysiloxanes with cyclotetrasiloxanes bearing isocyanate substituents 8 or by oxidation of the corresponding preformed tricyclic ladder oligosilanes. [START_REF] Kyushin | Selective Si-Si Bond Cleavage in Decaisopropylbicyclo[2.2.0hexasilane. A Route to Sterically Hindered 1,4-Dichlorocyclohexasilanes[END_REF] Ladder-type siloxanes with 6-8-6-membered fused rings can also be synthesized by reaction of cyclic tetrabromide and diphenylsilanediol. [START_REF] Unno | cis-trans-cis-Tetrabromotetramethylcyclotetrasiloxane: a Versatile Precursor of Ladder Silsesquioxanes[END_REF] Both 8-8-8 and 6-8-6 laddersiloxanes have also been prepared from cyclic tetrasilanols by using dichlorosiloxanes 5a or dichlorosilanes, [START_REF] Endo | Synthesis and Properties of Phenylsilsesquioxanes with Ladder and Double-Decker structures[END_REF] respectively. It is noteworthy that for all these tricyclic compounds, most of the peripheral groups present on the silicon atoms (Scheme 1, A, R 1 , R 2 and R 3 ) consist of non-reactive phenyl or alkyl groups. Interestingly, hydrolytic condensation of cycloheptyltrichlorosilane or opening of cage hexasilsesquioxanes under basic conditions allows generation of tricyclic 8-8-8 ladder structures having four reactive peripheral silanols (Scheme 1, B, R = cycloheptyl, X = OH). 12 Several of us reported the preparation of more stable phenyl counterparts (R = Ph, X = OH) from the corresponding tetraphenyl substituted cyclic 8-8-8 laddersiloxanes in two steps and high yields: i) cleavage of the Si-phenyl bond by HCl leading to tetrachloro ladder siloxanes (Scheme 1, B, X = Cl); ii) hydrolysis of the Si-Cl bonds to yield Si-OH bonds. 5a,c Therefore and to the best of our knowledge, the functionalizable laddersiloxanes reported to date possess either peripheral chlorosilanes or silanols (Scheme 1, B). The former ones were shown to react with silanols and the second with chlorosil(ox)anes, both via dehydrochlorinations, going pentacyclic ladder structures. 5a,c Scheme 1. Structures of ladder-type siloxanes.

The possibilities of using laddersiloxanes as building blocks in more complex structures are thus quite limited due to the lack of available peripheral functional groups. In an effort to enlarge the scope of functionalizable laddersiloxanes and broaden the number of pathways to introduce new functionalities around the tricyclic inorganic skeletons, a synthetic route to tetravinyl-and tetraallyl-decorated ladder structures is herein reported (Scheme 1, C). In this work, tetraallyl laddersiloxanes are shown to undergo hydrosilylation with dimethylphenylsilane and chloromethyldimethylsilane. In the latter case, the tetra-chloro-tricyclic ladder structure obtained successfully undergoes halogen exchange to form the corresponding tetra-iodoladdersiloxane (Scheme 1, C).

RESULTS AND DISCUSSION

The precursor, all-cis-tetraphenyl-cyclotetrasiloxanetetraol (T4-Ph-OH), was synthesized according to Kawakami et al. [START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] Noteworthy, the phenyl substituents are preferred to alkyl ones to ensure higher thermal stability of the targeted laddersiloxanes. [START_REF] Endo | Synthesis and Properties of Phenylsilsesquioxanes with Ladder and Double-Decker structures[END_REF] Therefore, the solution of divinyldichlorosilane in anhydrous THF was added dropwise into a solution of cyclic silanol T4-Ph-OH, distilled triethylamine and anhydrous THF at 0 °C under an argon atmosphere. The mixture was then warmed to room temperature (25 ˚C) and allowed to stir for one hour. The desired tricyclic laddersiloxane 1 was obtained in 52% isolated yield after washing with hexane and purification through a silica plug. The tricyclic laddersiloxane 2 was prepared using the same protocol from cyclic silanol T4-Ph-OH with diallyldichlorosilane and isolated with 72% yield (Scheme 2). It is worth noting that no polymers were observed at the end of reaction. Addition rates and reaction times of the dichlorosilane solution were found to be key factors for the success and selectivity of this synthesis. Scheme 2. Synthesis of laddersiloxanes 1 and 2 from T4-Ph-OH.

Multinuclear NMR spectroscopies were performed to characterize laddersiloxanes 1 and 2. Both 29 Si NMR spectra (Supporting Information Fig. S3 andS6) clearly displayed signals corresponding to the T-group silicon atoms (Si in blue) and to the D-group silicon atoms (Si in red). For the laddersiloxane 1, the 29 Si resonances at -67.8 and -34.5 ppm have respectively been assigned to T-and D-groups silicon atoms. For 2, the chemical shift of T-group silicon atoms  -67.7 was found to be almost the same as that of 1 while the signal of silicon atoms was significantly shifted to the left,  -15.2 (against 34.5 ppm in the case of 1). Such awkward observation, related to the enhanced electro-donating properties of the allyl substituents compared to the vinyl ones, has already been reported in the case of double-decker siloxanes bearing vinyl or allyl groups. [START_REF] Liu | Characterization, and Functionalization of Tetrafunctional Double-Decker Siloxanes[END_REF] It is worth noting that 1 and 13 C NMR spectra highlighted that vinyl-(resp. allyl-) moieties bound to each peripheral D-group silicon atom (Scheme 3, Si in red) were not magnetically equivalent, each expected signal for methylene or olefinic moieties giving rise to two distinct patterns (see Supporting Information Fig. S1, S2, S4 and S5). These data are thus in accordance with the expected all-cis configuration of the cyclotetrasiloxanetetraol T4-Ph-OH in laddersiloxanes 1 and 2 (Scheme 3).

Both compounds 1 and 2 were also analyzed by matrix-assisted laser desorption/ionization coupled time-of-flight (MALDI-TOF) mass spectroscopy and the experimental mass spectra were in good agreement with the calculated molecular weight (see SI). Along the same lines, elemental analyses results matched very well to the theoretical values. Scheme 3. All-cis-configuration of the laddersiloxanes: nonequivalence of the substituents beared by the peripheral Dgroup silicon atoms.

To our knowledge, the only previous example of tetrafunctionalizable DDSQ involved four reactive Si-Cl (or Si-OH) bonds (Scheme 1, B) likely to react with hydroxylated (or chlorosilane) compounds. The non-hydrolizable unsaturated C-C bonds reported here open the way to other possibilities in terms of functionalizations including hydrosilylations, an important and efficient reaction for olefin functionalization [START_REF] Nakajima | Hydrosilylation Reaction of Olefins: Recent Advances and Perspectives[END_REF] that was chosen for functionalizating alkenyl laddersiloxanes 1 and 2. Along these lines, 1 and 2 were allowed to react with dimethylphenylsilane as a model substrate in the presence of 2.5 mol% of Karstedt catalyst (Scheme 4). In the case of 1, several non-identified products were formed, obviously resulting from the polymerization of the C=C bond, and the isolation of the expected laddersiloxane by GPC (Gel Permeation Chromatography) was not possible (Scheme 4, top). To our delight, hydrosilylation occurred cleanly and quantitatively starting from 2 and the desired laddersiloxane 3 was successfully isolated in 99% yield after purification through a plug of silica (Scheme 4, bottom). Scheme 4. Hydrosilylation of laddersiloxanes 1 and 2 with dimethylphenylsilane in the presence of Karstedt's catalyst. 29 Si NMR spectrum of 3 displayed expected and previously described signals for T silicon atoms (Si in blue) corresponding to the T4 core (Figure 1). The chemical shift of D-group silicon atoms (Si in were shifted to  -6.6 as compared to  -15.2 in 2 which is in the favor of a successful reduction of the olefin. The appearance of two new singlets (Si in pink) at  -3.7 and -4.0 respectively, was also consistent with the formation of the carbosilane moiety and the successful hydrosilylation of allyl fragments (Figure 1). The 1 H and [START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] C NMR spectra of compound 3 clearly demonstrated the quantitative disappearance of olefinic groups at around 5-6 ppm and 131 ppm (in 1 H NMR and [START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] C NMR respectively) and the appearance of new patterns corresponding to methylene groups formed during hydrosilylation (see SI). As in the case of laddersiloxanes 1 and 2, both carbosilane groups (CH2-CH2-CH2-Si(Me)2Ph) bound to peripheral D-group silicon atoms (Scheme 4, Si in red) are not magnetically equivalent and gave rise to two distinct patterns. Noteworthy, the results of MALDI-TOF mass spectroscopy of 3 supported very well the structure. More challenging chloromethyl(dimethyl)silane possessing a reactive chloro group was next tested as the hydrosilylation reagent (Scheme 5). Hydrosilylation occurred in quite similar reaction conditions as those used with dimethylphenylsilane. The expected laddersiloxane 4 could thus be quantitatively formed within 16h at 40 °C and isolated in 95% yield after purification over a plug of silica. A halogen exchange was next performed under typical conditions for Finkelstein reaction to lead to tetra-iodo laddersiloxane 5 in 81% yield (Scheme 5). [START_REF] Ornelas | Efficient Mono-and Bifunctionalization of Polyolefin Dendrimers by Olefin Metathesis[END_REF] The latter is expected to be highly reactive in further nucleophilic substitutions, thus opening the way to readily functionalizable ladder-type structures.

As expected, on the 29 Si NMR spectra of 4 and 5, the chemical shifts of T-and D-group silicon atoms (Si in blue and Si in red respectively, Supporting Information, Fig. S12 andS15) were found to be very close to those of laddersiloxane 3. The signals of the carbosilane silicon atoms (Si in pink) were found respectively at 3.1, 2.8 ppm for 4 and 4.7, 4.5 ppm for 5. This observed shift to the left compared with laddersiloxane 3 is consistent with the stronger electro-withdrawing properties of phenyl groups in 3 (Figure 1). The slight shielding effect in the case of 5 relative to 4 is in accordance with the lower electronegativity of iodine. 1 H and [START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] C NMR spectra were consistent with a quantitative hydrosilylation and as expected, the characteristic carbon atoms adjacent to the halides (Cl or I for 4 and 5 resp.) were quite different, -4.66,-4.33 ppm for 4 and -13.14,-13.06 ppm for 5 (Supporting Information, Fig. S11 andS14). The experimental MALDI-TOF mass spectroscopy and elemental analysis results of these two compounds were also found in good accordance with their calculated values.

Scheme 5. Hydrosilylation of 2 with chloromethyl(dimethyl)silane to lead to laddersiloxane 4 and halide exchange to yield laddersiloxane 5.

The thermal stability of laddersiloxanes 1-5 was explored by thermogravimetric analysis (TGA) under a nitrogen stream.

The melting points of laddersiloxanes 1-3 which are solid compounds decreased from 196 to 81 °C with longer chains on the peripheral silicon atoms (See SI, Table S1). Besides, laddersiloxanes 1 and 2 displayed a Td5 (5 % weight loss) around 310 ˚C while the highest Td5 temperature (426 ˚C) was observed for laddersiloxane 3. This higher thermal stability is likely due to the presence of phenyl groups on the peripheral chains. [START_REF] Endo | Synthesis and Properties of Phenylsilsesquioxanes with Ladder and Double-Decker structures[END_REF] Ladder structures 4 and 5 decorated with 4 chloro-or iodo-substituents respectively are viscous liquids and 4 was found to display a higher Td5 (357 ˚C) than 5 (315 °C).

In summary, novel 6-8-6 laddersiloxanes involving four reactive substituents -vinyl, allyl, chloride or iodidewere prepared and characterized by multinuclear H, [START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] C and 29 Si NMR, mass spectroscopy and elemental analysis. The synthetic method is simple and the expected laddersiloxanes are formed in high yields (81 to 99%) and purities. Olefin groups of the allyl-moieties from 2 were shown to react through hydrosilylation with various silanes including functional chloromethyl(dimethyl)silane. The resulting tetrachloro-substituted laddersiloxane 4 could be successfully transformed into the corresponding tetraiodo compound 5. In summary, the various readily functionalizable tricyclic laddersiloxanes reported here for the first time constitute building blocks for preparing more complex organic-inorganic hybrid materials, the peripheral alkenes being moreover expected to react through various other reactions including metathesis, Heck or thiol-ene reactions. Potential areas of application range from the fields of electronics and optics to those of supported catalysis or nanocomposites. More generally, this work may open new opportunities to develop applications for ladder structures which remain underexplored to date compared to their double-decker or cage counterparts. Whatever the domain, original properties are expected due to the highly ordered double-chain structure that distinguishes laddersiloxanes from other polyhedral oligomeric silsesquioxanes.

Figure 1 .

 1 Figure 1. 29 Si NMR spectrum (CDCl3) for the laddersiloxane 3. Chemical shifts are in ppm unit.More challenging chloromethyl(dimethyl)silane possessing a reactive chloro group was next tested as the hydrosilylation reagent (Scheme 5). Hydrosilylation occurred in quite similar reaction conditions as those used with dimethylphenylsilane. The expected laddersiloxane 4 could thus be quantitatively formed within 16h at 40 °C and isolated in 95% yield after purification over a plug of silica. A halogen exchange was next performed under typical conditions for Finkelstein reaction to lead to tetra-iodo laddersiloxane 5 in 81% yield (Scheme 5).[START_REF] Ornelas | Efficient Mono-and Bifunctionalization of Polyolefin Dendrimers by Olefin Metathesis[END_REF] The latter is expected to be highly reactive in further nucleophilic substitutions, thus opening the way to readily functionalizable ladder-type structures.As expected, on the 29 Si NMR spectra of 4 and 5, the chemical shifts of T-and D-group silicon atoms (Si in blue and Si in red respectively, Supporting Information, Fig.S12 and S15) were found to be very close to those of laddersiloxane 3. The signals of the carbosilane silicon atoms (Si in pink) were found respectively at 3.1, 2.8 ppm for 4 and 4.7, 4.5 ppm for 5. This observed shift to the left compared with laddersiloxane 3 is consistent with the stronger electro-withdrawing properties of phenyl groups in 3 (Figure1). The slight shielding effect in the case of 5 relative to 4 is in accordance with the lower electronegativity of iodine. 1 H and[START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] C NMR spectra were consistent with a quantitative hydrosilylation and as expected, the characteristic carbon atoms adjacent to the halides (Cl or I for 4 and 5 resp.) were quite different, -4.66,-4.33 ppm for 4 and -13.14,-13.06 ppm for 5 (Supporting Information, Fig.S11 and S14). The experimental MALDI-TOF mass spectroscopy and elemental analysis results of these two compounds were also found in good accordance with their calculated values.Scheme 5. Hydrosilylation of 2 with chloromethyl(dimethyl)silane to lead to laddersiloxane 4 and halide exchange to yield laddersiloxane 5.

ASSOCIATED CONTENT Supporting Information

Experimental procedures and characterization data, 1 H, [START_REF] Ito | Cyclic Tetrasiloxanetetraols: Formation, Isolation, and Characterization[END_REF] C, and 29 Si NMR spectra, and Maldi-Tof MS spectra, and IR spectra, and TGA for all compounds 1-5

AUTHOR INFORMATION Corresponding Author

* E-mail: unno@gunma-u.ac.jp; armelle.ouali@enscm.fr.

ACKNOWLEDGMENT

The authors acknowledge the Gunma University Initiative for Advanced Research (GIAR) for funding the Open International Laboratory and the grant to Y. L.

Conflict of Interest

The authors declare no conflict of interest.


F Leroy

A El Barraj

F Cheynis

P Muller

S Curiotto

Electric forces on a confined ad vacancy island

The passage of an electric current in a material can cause a biased mass transport at its surface. This migration phenomenon is intimately related to the microscopic details of atomic processes of diffusion and attachment/detachment at step edges. Using low energy electron microscopy we have examined in operando under an electric current the migration of Si(111)-1×1 ad vacancy islands confined on Si(111)-7×7 terraces. The islands move opposite to the current direction, with velocity increasing with the radius. The effective valence of Si ad atoms is 2.8±0.5 and the kinetic length of attachment-detachment is about 500 nm. The analysis of the islands shape reveals that the electric current biases significantly the kinetic rate of mass transfers at step edges modifying the overall island shape.

Advances in the fabrication of nanostructures widely depend on the degree of knowledge of atomic processes at surfaces. In that respect atomic steps, as the most abundant structures at surfaces, play a key role in mass transfers. They are involved in complex atomic mechanisms such as the attachment-detachment of atoms or the atomic diffusion at the periphery of nanostructures [1 and 2]. To study the mass transfer mechanisms, different experimental strategies have been carried out based on the spatio-temporal fluctuations of the position of isolated/interacting steps [3][4][5] or on the step displacement velocity when a driving force intervenes using the fluctuation-dissipation theorem [6][7][8]. In particular the application of an electric current is known to bias the diffusion of mobile ad atoms. This effect, called electromigration [9][10][11][12][13][14][15], can cause substantial changes in the surface morphology such as step bunching for vicinal surfaces [START_REF] Latyshev | 431 Stenin[END_REF][17][18][19] or shape instabilities of 2D islands [20][21][22][23][24].

However it has been recently recognized that an electric current may not only impact ad atom diffusion but also atomic steps themselves by modifying their local properties such as the ad atom equilibrium concentration close to the step and/or the kinetic coefficients of attachmentdetachment at step edges [5, 25, and 26]. These effects arise since the force acting on atoms depends on their local environment that differs at step edge, kink site or on top of a terrace. These local modifications of step properties are, to date, largely unknown whereas they are suspected to be extremely strong [5 and 25]. Moreover a better understanding of the effects of the electric current on surface mass transport gives also indirect information about the electric resistance of surfaces [27].

Indeed the electric forces acting on atoms, kink sites and step edges are compensated by opposite forces acting on charge carriers caused by these surface structures. These forces change the surface electric resistivity and may play a major role in electrical conductors when down-scaling in size [28 and 29]. This calls for specific studies on the effect of an electric current on the step properties and mass transport phenomena at the nanoscale.

In this letter we analyze quantitatively the atomic mechanisms of mass transport and step properties on Si(111) under an electric bias by addressing precisely the 47 boundary conditions to disentangle all the contributions.

48

In that purpose we have met two essential conditions: the island drift velocity resulting from these processes is 156 [20]:

157 V isl = c eq v 1×1 R R + d (1)
where c eq is the equilibrium surface concentration of mo- typical crossover is about the attachment-detachment ki-264 netic length d). To describe the island shape, we use the 265 polar coordinates R(θ) = R 0 + ρ(θ) (R 0 is the mean ra-266 dius), and we apply the Fourier series expansion of ρ(θ):

267 ρ(θ) = ∞ n≥2 ρ n cos (nθ) + ν n sin (nθ) (2)
where ρ n and ν n are the Fourier coefficients (ν n = 0 for 268 symmetric islands and ρ 1 and ν 1 are not considered be-269 cause they correspond to a simple shape translation). In

309 ρ 2 R 0 = - 1 12Γ R 2 0 ξ 2 d < 0 (3)
where Γ = a 2 β kB T is the capillary length (Gibbs-Thomson island with this electrobias effect we obtain:

336 ρ 2 R 0 = - 1 12Γ R 2 0 ξ 2 d + d 2 2 -2 R 0 ξ (d 1 + d 3 ) (4)
The shape elongation

49 ( 1 )

 491 An ad vacancy island where atomic displacements oc-50 cur at the interior of a confined 2D space closed by a 51 step edge; and (2) a driving force induced by an elec-52 tric current to move the island. By adjusting the area 53 of the island and measuring its drift velocity induced 54 by an electric current we determine the mechanisms of 55 mass transfers. This study is based on an in operando 56 observation under an electric current of the Si(111) sur-57 face with low energy electron microscopy (LEEM). The 58 experimental set-up allows to study the spatio-temporal 59 dynamics of mass transfers at atomic steps [30]. We show 60 a transition from a kinetics of mass transfer limited by 61 attachment-detachment of atoms at step edges for small 62 islands to a kinetics limited by terrace diffusion for large 63 islands. We deduce that the kinetic length for attach-64 ment/detachment is d ∼ 500 nm and the effective va-65 lence Z* of the Si ad atoms at the surface is 2.8±0.5. Im-66 portantly our detailed analysis of the stationary shape of 67 the electromigrating ad vacancy islands is consistent with 68 a strong modification of the local properties of attach-69 ment/detachment at step edges induced by the electric 70 current. 71 The experiments were performed in an ultra high vac-72 uum (UHV) setup equipped with a low energy electron 73 microscope (LEEM III, Elmitec GmbH) [30]. Si(111) 74 substrates (n or p-doped, ρ=1 Ωcm) were cut into pieces 75 of 15×3×0.5 mm 3 , cleaned with acetone and ethanol be-76 fore introduction in UHV. An electric current is applied 77 through the sample via two Mo electrodes clamped to its 78 extremities. The samples were degassed in UHV for sev-79 eral hours at about 1100 K and then flashed above 1500 K 80 for a few seconds by direct current heating. Ad vacancy is-81 lands are created by Si sublimation in the middle of large 82 terraces [31]. The surface evolution under electromigra-83 tion is studied by LEEM in bright field mode, with an 84 electron beam energy of 3 eV. To change the ad vacancy 85 islands size, Si was deposited in situ by a homemade 86 direct current evaporator made of a piece of Si wafer 87 clamped between Mo electrodes.

  FIG. 1. (a) Sequence of LEEM images during the 1×1 → 7×7 phase transition and under electric heating (see complete movie S1 in the supplementary materials). (i) Nucleation of 7×7 surface reconstruction at the step edges on the upper terraces. (ii) Spreading of the 7×7 onto the terraces except in the ad vacancy island (black arrow) where the 7×7 nucleation is hindered. (iii) Formation of 1×1 out-of-phase boundaries at the 7×7 domain intersections (white arrows). (iv) Migration of the ad vacancy island in the 112 direction, opposite to the electric current. Out-of-phase boundaries merging at the rear side of the island. Electron energy E=3 eV. Scale bar 1µm. (b) Scheme of the surface evolution under slow cooling. (c) Time evolution of the displacement of the ad vacancy island (black square). The steady velocity is 13.1±0.1 nm.s -1 (velocity in (a)-i is 5.4±0.3 nm.s -1 , see top inset) and the area is 1.5±0.1 10 6 nm 2 (see bottom inset).

142 143

 142 To address the mass transport mechanisms that are 144 occurring inside the ad vacancy islands under electromi-145 gration we have studied the size-dependence of the island 146 velocity in the stationary regime. Figure 2(a) shows that 147 the velocity increases with the island effective radius R 148 (R = √ A/π where A is the island area). Pierre-Louis et al. [20] have analyzed the island velocity in the frame-150 work of the linear response theory with weak electromi-151 gration. Considering a kinetics of mass transport by at-152 tachment (A), detachment (D) and terrace diffusion (TD) 153 inside the 1×1 ad vacancy island (see Fig. 2(c)), and ne-154 glecting the ad atom flux from the upper terrace (7 × 7), 155

158 7 × 7

 77 bile ad atoms, v 1×1 is the ad atoms velocity on the (1 × 1) 159 terrace and d = D 1×1 /k is the kinetic length of attach-160 ment/detachment and is defined as the ratio of the sur-161 face diffusion coefficient D 1×1 to the rate k of ad atom 162 attachment to the step from the terrace. Let us note that 163 the mechanism of periphery diffusion of atoms along the 164 step edge has been neglected since the velocity should 165 decay as 1/R [20] and no evidence of this behavior is 166 measured even for the smallest radius. The fit of the ex-167 perimental plots give two key parameters c eq v 1×1 and d. 168 The first term is deduced from the asymptotic velocity at 169 large radius (15±1 nm.s -1 ) and is only related to terrace 170 diffusion of the electromigrating ad atoms. To estimate 171 the ad atom velocity v 1×1 we have to determine first c eq . 172 Since the step edge is hybrid, 1 × 1 reconstructed on the 173 lower terrace and 7 × 7 on the upper one, the equilib-174 rium concentration of ad atom close to the step edge is 175 unknown. We have measured the ad atom concentration 176 in the 1 × 1 ad vacancy island by decreasing the temper-177 ature to induce the 1 × 1 → 7 × 7 phase transition. The 178 excess of ad atoms expelled by the phase transition con-179 densates at step edges and shrinks the ad vacancy island 180 area (see supplementary materials S3). The area fraction 181 lost after the phase transition is 0.08 ± 0.02. Moreover 182 considering that the 7 × 7 and the bulk-terminated 1 × 1 183 structure have a difference of atomic density of 0.04 [35], 184 we can estimate that the density of mobile ad atoms of 185 the 1 × 1 is 0.12 ± 0.02. As the steps on Si(111) are bi-186 layers this corresponds to 0.24 ± 0.04 monolayer (ML) 187 of ad atoms on the 1 × 1 surface of the ad vacancy is-188 land. This result is similar to 0.2 ML as estimated by 189 [35] considering the 1 × 1 surface. This result is also 190 consistent with the fact that the equilibrium concentra-191 tion of ad atoms is a thermodynamic quantity. It is re-192 lated to a difference of energy between two states: an 193 atom attached at a step edge and on a terrace (ad atom). 194 As the chemical environments of an atom attached to a 195 or 1 × 1 step edge are similar and very distinct 196 from an ad atom on top of a 1 × 1 terrace, we expect 197 that the step edge reconstruction only slightly modifies 198 the equilibrium concentration. Using our experimental 199 result of c eq and correcting the velocity with ad vection 200 [37] (sweeping effect on the ad atoms due to the step mo-201 tion) we get finally the ad atom velocity v 1×1 = 110 ± 8 202 nm.s -1 on the 1×1 surface reconstruction at the phase 203 transition temperature. This velocity derives from the 204 Einstein relation v 1×1 = D1×1 kB T F where k B is the Boltz-205 mann constant, T the temperature and F = Z * eE the 206 electromigration force. Therefore the force and the ef-207 fective charge Z * of Si ad atoms can be obtained if the 208 diffusion coefficient D 1×1 is known. Hibino et al. have 209 found D 1×1 c eq = 3.0 10 7 s -1 [7 and 33] at the phase 210 transition temperature. Pang et al. have obtained by 211 different approaches D 1×1 c eq = 2.0 ± 0.2 10 7 s -1 [8] in a 212 slightly higher temperature regime (1163 K). Considering 213 an average value for D 1×1 we can deduce F = 1.4 ± 0.3 214 10 -6 eV.nm -1 and the only free parameter, i.e. the ef-215 fective charge of Si ad atoms Z * = 2.8 ± 0.5 (E = 490 216 V.m -1 , atomic area: 0.064 nm 2 ). The deduced value of 217 Z * is larger by one order of magnitude with earlier re-218 ports [38 and 39] except for [40] (Z * > 1.3). The model 219 hypothesis of a weak electromigration is confirmed since 220 the available thermal energy is much larger than the en-221 ergy to electromigrate F a kB T ∼ 10 -5 ≪ 1 where a = 0.384 222 nm is the atomic lattice parameter [20]. The second term 223 that is deduced from the fit is the kinetic length of at-224 tachment/detachment d. We obtain d 112 = 450 ± 100 225 nm and d 112 = 500 ± 30 nm respectively for an island 226 displacement in the 112 and the 112 directions. It 227 is interesting to note that, contrary to the equilibrium 228 concentration that is close to the 1×1 surface, the kinetic 229 length of attachment/detachment at the hybrid step edge 230 is similar to the one measured at the step edges of the 231 7×7 reconstructed surface [41]. To explain this kinetic 232 length we can note that, as for the 7×7 reconstructed 233 surface, the advance of the hybrid step edge needs also 234 to build 7×7 unit cells. This process is related to en-235 ergy barriers and probably to the occurrence of concerted 236 events that are necessary to achieve the complex mech-237 anisms involved in the formation of a 7×7 unit cell [42 238 and 43]. From the evaluation of d we can estimate the 239 rate of attachment/detachment at a step edge per atomic 240 site kc eq a = c eq D 1×1 a/d ∼ 1.9 × 10 4 s -1 . It is also in-241 structive to estimate the average macroscopic time for 242 ad atoms to detach from the front side, cross the island 243 and attach at the rear side. The traveling time across 244 the terrace by diffusion is about t d ∼ 2R/v 1×1 and for a 245 typical island of 1 µm radius t d ∼ 15 s. As a comparative 246 time scale, the delay time for an atom to make all attach-247 ment/detachment processes to cross the island is about 248 t AD ∼ 2d/v 1×1 ∼ 6 to 8 s. This indicates that many 249 events of (re)-attachment-detachment occur during this 250 traveling (kc eq a × t AD ∼ 10 5 , see Fig.2(c)).

FIG

  FIG. 3. (a) LEEM images of ad vacancy islands electromigrating in the 112 (top) and the 112 (bottom) directions (scale bar 5 µm). The shape is respectively a triangle and a lozenge. Unfaulted steps (Ustep) are shown as dotted lines and faulted steps (Fstep) as dashed lines [44]. (b) Fourier coefficients of the island shape as function of island radius for both direction. (c) Scheme of the U and F step edge structure. (d) LEEM image of two ad vacancy islands electromigrating in the 110 direction. The shape is asymmetric (scale bar 1 µm, see complete movie S4 in the supplementary materials).

270Fig. 3 (

 3 Fig. 3(b) are plotted the normalized Fourier coefficients 271

  310 effect), β is the step edge stiffness and ξ is a charac-311 teristic length associated with the electromigration force 312 (ξ = kB T F = 70 µm). This result is opposite to the ex-313 perimental shape since the elongation of the ad vacancy 314 islands is along the migration direction ( ρ2 R0 > 0). As this 315 result is observed whatever the direction of the electric 316 current we infer that even if the modeling could include 317 the anisotropy of the surface properties, it alone could 318 not explain that the shape is always elongated along the 319 migration axis. Therefore we propose that the electric 320 current modifies not only the ad atom displacement but 321 also the atomic step properties. As a minimum model, 322 the electric current breaks the threefold symmetry of the 323 kinetic rate of attachment-detachment at the step edges 324 [47]. To study this effect we expand the kinetic length d 325 as a Fourier series d = d + n d n cos (nθ) where d is the 326 mean kinetic length of attachment/detachment and d n 327 are the Fourier coefficients for a symmetric shape. The 328 main Fourier term acting as an electrobias, i.e. chang-329 ing the kinetics of attachment-detachment at the step 330 edge, is expected to be d 1 since it breaks the symmetry 331 between the island front where the current is step-down 332 and the island rear where the current is step-up. Let 333 us note that without electrobias only d 3n exists by sym-334 metry. Expanding linearly the shape of the ad vacancy 335

355

  to an intrinsic change of step properties induced by the 356 current but it may also arise from a change of kink den-357 sity at step edges. Indeed it has been shown[49] that an 358 electric current in the 112 direction along a step and as-359 cending the kinks favors the formation of an atomically 360 straight step edge. Therefore considering that the kinet-361 ics of mass transfers at step edges is mediated by kinks 362 then the rate of attachment/detachment could be indeed 363 strongly modified by the electric current. Such an electro-364 bias effect on step edges or kink sites has been suspected 365 to occur on semiconductor surfaces [50]. On metals a 366 similar electrobias effect has also been found. However 367 the studies on Ag metal [5 and 25] have addressed a dif-368 ferent regime of mass transport dominated by atomic dif-369 fusion along the island periphery. The electrobias effect 370 was studied considering a different methodology based on 371 the analysis of step fluctuations and island velocity but 372 not on the island shape whereas it is strongly sensitive to 373 the local modifications of the step edge properties [20]. 374 In conclusion we have shown on Si(111) surface that 375 an ad vacancy island in the 1×1 high temperature phase 376 and surrounded by the 7×7 low temperature phase can be 377 stabilized. This regime allows keeping the 2D island in a 378 confined state in terms of atomic exchanges. Then under 379 the influence of an electric current, the island is moving. 380 The analysis of the velocity and shape of the island as 381 function of its radius show that (i) Si ad atoms migration 382 on the terrace is biased and they have an effective valence 383 Z * of 2.8±0.5 (ii) the kinetic of attachment/detachment 384 of atoms at the step edges is very slow and we evaluate 385 the kinetic length as ∼ 500 nm. (iii) An electrobias effect 386 on the kinetics of attachment/detachment at step edges 387 elongates the island shape in the direction of the electric 388 current. We believe that a complete modeling including 389 all the effects of anisotropy, non-linearities and high den-390 sity of ad atoms would be necessary to describe precisely 391 the island shape.

39212

  We are grateful to Olivier Pierre-Louis for instructive 393 discussions and Plateform PLANETE (CNano PACA) 394 for technical support. This work has been supported by 395 the ANR grant HOLOLEEM (ANR-15-CE09-0012). 396 * leroy@cinam.univ-mrs.fr 397 Hyeong-Chai Jeong and Ellen D Williams. Surf. Sci. Rep., 398 34(6-8):171-294, 1999. 399 Chaouqi Misbah, Olivier Pierre-Louis, and Yukio Saito. 400 Reviews of Modern Physics, 82(1):981-1040, 2010. 401 3 N. C. Bartelt, J. L. Goldberg, T. L. Einstein, Ellen D. 402 Williams, J. C. Heyraud, and J. J. Métois. Physical Review 403 B, 48(20):15453-15456, 1993. 404 4 DB Dougherty, I Lyubinetsky, ED Williams, M Con-405 stantin, C Dasgupta, and S Das Sarma. Physical Review 406 Letters, 89(13):136102, 2002. 407 5 O. Bondarchuk, W. G. Cullen, M. Degawa, E. D. Williams, 408 T. Bole, and P. J. Rous. Physical Review Letters,

  ρ2 R0 shows a new contribution that 337 increases linearly with the island radius R 0 as in the mea-338 surements, and is along the migration axis if d 1 + d 3 > 0.

	339
	340 341	As R0 ξ ∼ 0.014 ≪ 1 we can neglect the second order con-tribution in eq. (4). To estimate only the electrobias
	342	effect d 1 we use the change of the current direction in
	343	the experiment. Assuming that d 3 is not significantly af-
	344	fected by the electrobias effect, since it is a three order
	345	term in the series expansion and do not coincide with
	346	the symmetry of the electric current, the inversion of
	347 348 349	the current direction changes d 3 by -d 3 [48]. There-fore the kinetic length of electrobias d 1 is obtained by averaging both shape elongation ρ2 R0 in the 112 and
	350 351	the 112 directions. We estimate that d 1 ∼ 83 ± 12 nm (Γ =1 nm [8]) and considering that this contribution
	352	is thermally activated, we extract the activation energy
	353	E 1 = k B T ln 1 + d1 d	=1.5 10 -2 eV. This electrobias ef-
		fect on the step edge is much larger than on ad atoms

354

(E = F a/2 ∼2.6 10 -7 eV). This result could be related


Yassine Ouali
email: yassine.ouali@centralesupelec.fr

Céline Hudelot
email: celine.hudelot@centralesupelec.fr

Myriam Tami
email: myriam.tami@centralesupelec.fr

Semi-Supervised Semantic Segmentation with Cross-Consistency Training

In this paper, we present a novel cross-consistency based semi-supervised approach for semantic segmentation. Consistency training has proven to be a powerful semisupervised learning framework for leveraging unlabeled data under the cluster assumption, in which the decision boundary should lie in low density regions. In this work, we first observe that for semantic segmentation, the low density regions are more apparent within the hidden representations than within the inputs. We thus propose crossconsistency training, where an invariance of the predictions is enforced over different perturbations applied to the outputs of the encoder. Concretely, a shared encoder and a main decoder are trained in a supervised manner using the available labeled examples. To leverage the unlabeled examples, we enforce a consistency between the main decoder predictions and those of the auxiliary decoders, taking as inputs different perturbed versions of the encoder's output, and consequently, improving the encoder's representations. The proposed method is simple and can easily be extended to use additional training signal, such as image-level labels or pixel-level labels across different domains. We perform an ablation study to tease apart the effectiveness of each component, and conduct extensive experiments to demonstrate that our method achieves stateof-the-art results in several datasets.

Introduction

In recent years, with the wide adoption of deep supervised learning within the computer vision community, significant strides were made across various visual tasks yielding impressive results. However, training deep learning models requires a large amount of labeled data which acquisition is often costly and time consuming. In semantic segmentation, given how expensive and laborious the acquisition of pixel-level labels is, with a cost that is 15 times and 60 times larger than that of region-level and image-level labels respectively [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], the need for data efficient semantic segmentation methods is even more evident. As a result, a growing attention is drown on deep Semi-Supervised learning (SSL) to take advantage of a large amount of unlabeled data and limit the need for labeled examples. The current dominant SSL methods in deep learning are consistency training [START_REF] Rasmus | Semi-supervised learning with ladder networks[END_REF][START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF][START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF][START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semi-supervised learning[END_REF], pseudo labeling [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF], entropy minimization [START_REF] Grandvalet | Semi-supervised learning by entropy minimization[END_REF] and bootstrapping [START_REF] Qiao | Deep co-training for semi-supervised image recognition[END_REF]. Some newly introduced techniques are based on generative modeling [START_REF] Kumar | Semi-supervised learning with gans: Manifold invariance with improved inference[END_REF][START_REF] Souly | Semi supervised semantic segmentation using generative adversarial network[END_REF].

However, the recent progress in SSL was confined to classification tasks, and its application in semantic segmentation is still limited. Dominant approaches [START_REF] Huang | Weakly-supervised semantic segmentation network with deep seeded region growing[END_REF][START_REF] Wei | Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation[END_REF][START_REF] Wei | Object region mining with adversarial erasing: A simple classification to semantic segmentation approach[END_REF][START_REF] Lee | Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference[END_REF] focus on weakly-supervised learning which principle is to generate pseudo pixel-level labels by leveraging the weak labels, that can then be used, together with the limited strongly labeled examples, to train a segmentation network in a supervised manner. Generative Adversarial Networks (GANs) were also adapted for SSL setting [START_REF] Souly | Semi supervised semantic segmentation using generative adversarial network[END_REF][START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF] by extending the generic GAN framework to pixel-level predictions. The discriminator is then jointly trained with an adversarial loss and a supervised loss over the labeled examples.

Nevertheless, these approaches suffer from some limitations. Weakly-supervised approaches require weakly labeled examples along with pixel-level labels, hence, they do not exploit the unlabeled data to extract additional training signal. Methods based on adversarial training exploit the unlabeled data, but can be harder to train.

To address these limitations, we propose a simple consistency based semi-supervised method for semantic segmentation. The objective in consistency training is to enforce an invariance of the model's predictions over small perturbations applied to the inputs. As a result, the learned model will be robust to such small changes. The effectiveness of consistency training depends heavily on the behavior of the data distribution, i.e., the cluster assumption, where the classes must be separated by low density regions. In semantic segmentation, we do not observe the presence of low density regions separating the classes within the inputs, but rather within the encoder's outputs. Based on this observation, we propose to enforce the consistency over different forms of perturbations applied to the encoder's output. Specifically, we consider a shared encoder and a main decoder that are trained using the labeled examples. To leverage unlabeled data, we then consider multiple auxiliary decoders whose inputs are perturbed versions of the output of the shared encoder. The consistency is imposed between the main decoder's predictions and that of the auxiliary decoders (see Fig. 1). This way, the shared encoder's representation is enhanced by using the additional training signal extracted from the unlabeled data. The added auxiliary decoders have a negligible amount of parameters compared to the encoder. Additionally, during inference, only the main decoder is used, reducing the computation overhead both in training and inference.

The proposed method is simple and efficient, it is also flexible since it can easily be extended to use additional weak labels and pixel-level labels across different domains in a semi-supervised domain adaption setting. With extensive experiments, we demonstrate the effectiveness of our approach on PASCAL VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] in a semi-supervised setting, and CityScapes, CamVid [START_REF] Gabriel J Brostow | Semantic object classes in video: A high-definition ground truth database[END_REF] and SUN [START_REF] Song | Sun rgb-d: A rgb-d scene understanding benchmark suite[END_REF] in a semisupervised domain adaption setting. We obtain competitive results across different datasets and training settings.

Concretely, our contributions are four-fold:

• We propose a cross-consistency training (CCT) method for semi-supervised semantic segmentation, where the invariance of the predictions is enforced over different perturbations injected into the encoder's output. • We propose and conduct an exhaustive study of various types of perturbations. • We extend our approach to use weakly-labeled data, and exploit pixel-level labels across different domains to jointly train the segmentation network.

• We demonstrate the effectiveness of our approach with with an extensive and detailed experimental results, including a comparison with the state-of-the-art, as well as an in-depth analysis of our approach with a detailed ablation study.

Related Work

Semi-Supervised Learning. Recently, many efforts have been made to adapt classic SSL methods to deep learning, such as pseudo labeling [START_REF] Lee | Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks[END_REF], entropy minimization [START_REF] Grandvalet | Semi-supervised learning by entropy minimization[END_REF] and graph based methods [START_REF] Liu | Deep metric transfer for label propagation with limited annotated data[END_REF][START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF] in order to overcome this weakness. In this work, we focus mainly on consistency training. We refer the reader to [START_REF] Chapelle | Semi-supervised learning[END_REF] for a detailed overview of the field. Consistency training methods are based on the assumption that, if a realistic form of perturbation was applied to the unlabeled examples, the predictions should not change significantly. Favoring models with decision boundaries that reside in low density regions, giving consistent predictions for similar inputs. For example, Π-Model [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF] enforces a consistency over two perturbed versions of the inputs under different data augmentations and dropout. A weighted moving average of either the previous predictions (i.e., Temporal Ensembling [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF]), or the model's parameters (i.e., Mean Teacher [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF]), can be used to obtain more stable predictions over the unlabeled examples. Instead of relying on random perturbations, Virtual Adversarial Training (VAT) [START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semi-supervised learning[END_REF] approximates the perturbations that will alter the model's predictions the most.

Similarly, the proposed method enforces a consistency of predictions between the main decoder and the auxiliary decoders over different perturbations, that are applied to the encoder's outputs rather than the inputs. Our work is also loosely related to Multi-View learning [START_REF] Zhao | Multiview learning overview: Recent progress and new challenges[END_REF] and Cross-View training [START_REF] Clark | Semi-supervised sequence modeling with cross-view training[END_REF], where each input to the auxiliary decoders can be view as an alternate, but corrupt representation of the unlabeled examples.

Semi-Supervised Semantic Segmentation. A significant number of approaches use a limited number pixellevel labels together with a larger number of inexact annotations, e.g., region-level [START_REF] Song | Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation[END_REF][START_REF] Dai | Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation[END_REF] or image-level labels [START_REF] Lee | Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference[END_REF][START_REF] Zhou | Learning deep features for discriminative localization[END_REF][START_REF] Wei | Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation[END_REF][START_REF] Li | Tell me where to look: Guided attention inference network[END_REF]. For image-level based weak-supervision, primary localization maps are generated using class activation mapping (CAM) [START_REF] Zhou | Learning deep features for discriminative localization[END_REF]. After refining the generated maps, they can then be used to train a segmentation network together with the available pixel-level labels in a SSL setting.

Generative modeling can also be used for semisupervised semantic segmentation [START_REF] Souly | Semi supervised semantic segmentation using generative adversarial network[END_REF][START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF] to take advantage of the unlabeled examples. Under a GAN framework, the discriminator's predictions are extended over pixel classes, and can then be jointly trained with a Cross-Entropy loss over the labeled examples and an adversarial loss over the whole dataset.

In comparison, the proposed method exploits the unlabeled examples by enforcing a consistency over multiple perturbations on the hidden representations level. Enhancing the encoder's representation and the overall performance, with a small additional cost in terms of computation and memory requirements.

Recently, CowMix [START_REF] French | Consistency regularization and cutmix for semi-supervised semantic segmentation[END_REF], a concurrent method was introduced. CowMix, using MixUp [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF], enforces a consistency between the mixed outputs and the prediction over the mixed inputs. In this context, CCT differs as follows: (1) CowMix, as traditional consistency regularization methods, applies the perturbations to the inputs, but uses MixUp as a high-dimensional perturbation to overcome the absence of the cluster assumption. (2) Requires multiple forward passes though the network for one training iteration. (3) Adapting CowMix to other settings (e.g., over multiple domains, using weak labels) may require significant changes. CCT is efficient and can easily be extended to other settings. Domain Adaptation. In many real world cases, the existing discrepancy between the distribution of training data and and that of testing data will often hinder the performances. Domain adaptation aims to rectify this mismatch and tune the models for a better generalization at test time [START_REF] Vishal | Visual domain adaptation: A survey of recent advances[END_REF]. Various generative and discriminative domain adaptation methods have been proposed for classification [START_REF] Geng | Daml: Domain adaptation metric learning[END_REF][START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF][START_REF] Ganin | Domain-adversarial training of neural networks[END_REF][START_REF] Cao | Partial adversarial domain adaptation[END_REF] and semantic segmentation [START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF][START_REF] Zhang | Curriculum domain adaptation for semantic segmentation of urban scenes[END_REF][START_REF] Fatemeh | Effective use of synthetic data for urban scene semantic segmentation[END_REF][START_REF] Kalluri | Universal semi-supervised semantic segmentation[END_REF] tasks.

In this work, we show that enforcing a consistency across different domains can push the model toward better generalization, even in the extreme case of non-overlapping label spaces.

Method

The cluster assumption in semantic segmentation

We start with our observation and analysis of the cluster assumption in semantic segmentation, motivating the proposal of our cross-consistency training approach. A simple way to examine it is to estimate the local smoothness by measuring the local variations between the value of each pixel and its local neighbors. To this end, we compute the average euclidean distance at each spatial location and its 8 intermediate neighbors, for both the inputs and the hidden representations (i.e., the ResNet's [START_REF] He | Deep residual learning for image recognition[END_REF] outputs of a DeepLab v3 [START_REF] Chen | Rethinking atrous convolution for semantic image segmentation[END_REF] trained on COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]). For the inputs, following [START_REF] French | Consistency regularization and cutmix for semi-supervised semantic segmentation[END_REF], we compute the average distance of a patch centered at a given spatial location and its neighbors to simulate a realistic receptive field. For the hidden representations, we first upsample the feature map to the input size, and then compute the average distance between the neighboring activations (2048-dimensional feature vectors). The results are illustrated in Fig. 2. We observe that the clus-ter assumption is violated at the input level, given that the low density regions do not align with the class boundaries. On the contrary, for the encoder's outputs, the cluster assumption is maintained where the class boundaries have high average distance, thus corresponding to low density regions. This observation motivates the following approach, in which the perturbations are applied to the encoder's outputs rather than the inputs. i as the i-th unlabeled input image, and x l i as the i-th labeled input image with spatial dimensions H × W and its corresponding pixel-level label y i ∈ R C×H×W , where C is the number of classes.

As discussed in the introduction, the objective is to exploit the larger number of unlabeled examples (m ≫ n) to train a segmentation network f , to perform well on the test data drawn from the same distribution as the training data. In this work, our architecture (see Fig. 3) is composed of a shared encoder h and a main decoder g, which constitute the segmentation network f = g •h. We also introduce a set of K auxiliary decoders g k a , with k ∈ [1, K]. While the segmentation network f is trained on the labeled set D l in a traditional supervised manner, the auxiliary networks g k a •h are trained on the unlabeled set D u by enforcing a consistency of predictions between the main decoder and the auxiliary decoders. Each auxiliary decoder takes as input a perturbed version of the encoder's output, and the main encoder is fed the uncorrupted intermediate representation. This way, the representation learning of the encoder h is further enhanced using the unlabeled examples, and subsequently, that of the segmentation network f .

Cross-Consistency Training

As stated above, to extract additional training signal from the unlabeled set D u , we rely on enforcing a consistency between the outputs of the main decoder g m and those of auxiliary decoders g k a . Formally, for a labeled training example x l i , and its pixel-level label y i , the segmentation network f is trained using a Cross-Entropy (CE) based supervised loss L s :

L s = 1 |D l | x l i ,yi∈D l H(y i , f (x l i )) (1)
with H(., .) as the CE. For an unlabeled example x u i , an intermediate representation of the input is computed using where one perturbation function can be assigned to multiple auxiliary decoders. With various perturbation settings, we generate K perturbed versions zk i of the intermediate representation z i , so that the k-th perturbed version is to be fed to the k-th auxiliary decoder. For consistency, we consider the perturbation function as part of the auxiliary decoder, (i.e., g k a can be seen as g k a • p r ). The training objective is then to minimize the unsupervised loss L u , which measures the discrepancy between the main decoder's output and that of the auxiliary decoders:

L u = 1 |D u | 1 K x u i ∈Du K k=1 d(g(z i ), g k a (z i )) (2)
with d(., .) as a distance measure between two output probability distributions (i.e., the outputs of a sof tmax function applied over the channel dimension). In this work, we choose to use mean squared error (MSE) as a distance measure.

The combined loss L for consistency based SSL is then computed as: For one training iteration, we sample a labeled input image x l and its pixel-level label y together with an unlabeled image x u . We pass both images through the encoder and main decoder, obtaining two main predictions ŷl and ŷu . We compute the supervised loss using the pixel-level label y and ŷl . We apply various perturbations to z, the output of the encoder for x u , and generate auxiliary predictions ŷ(i) a using the perturbed versions z(i) . The unsupervised loss is then computed between the outputs of the auxiliary decoders and that of the main decoder.

L = L s + ω u L u ( 3
where ω u is an unsupervised loss weighting function. Following [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF], to avoid using the initial noisy predictions of the main encoder, ω u ramps up starting from zero along a Gaussian curve up to a fixed weight λ u . Concretely, at each training iteration, an equal number of examples are sampled from the labeled D l and unlabeled D u sets. The supervised loss is computed using the main encoder's output and pixellevel labels. For the unlabeled examples, we compute the MSE between the prediction of each auxiliary decoder and that of the main decoder. The total loss is then compute and back-propagated to train the segmentation network f and the auxiliary networks g k a • h. Note that the unsupervised loss L u is not back-propagated through the main-decoder g, only the labeled examples are used to train g.

Perturbation functions

An important factor in consistency training is the perturbations to apply to the hidden representation, i.e., the encoder's output z. We propose three types of perturbation functions p r : feature based, prediction based and random.

Feature based perturbations. They consist of either injecting noise into or dropping some of the activations of encoder's output feature map z.

• F-Noise: we uniformly sample a noise tensor N ∼ U (-0.3, 0.3) of the same size as z. After adjusting its amplitude by multiplying it with z, the noise is then injected into the encoder's output z to get z = (z ⊙ N) + z. This way, the injected noise is proportional to each activation.

• F-Drop: we first uniformly sample a threshold γ ∼ U (0.6, 0.9). After summing over the channel dimension and normalizing the feature map z to get z ′ , we generate a mask M drop = {z ′ < γ} 1 † , which is then used to obtain the perturbed version z = z ⊙ M drop . This way, we mask 10% to 40% of the most active regions in the feature map.

Prediction based perturbations. They consist of adding perturbations based on the main decoder's prediction ŷ = g(z) or that of the auxiliary decoders. We consider masking based perturbations (Con-Msk, Obj-Msk and G-Cutout) in addition to adversarial perturbations (I-VAT).

• Guided Masking: Given the importance of context relationships for complex scene understanding [START_REF] Oliva | The role of context in object recognition[END_REF], the network might be too reliant on these relationships. To limit them, we create two perturbed versions of z by masking the detected objects (Obj-Msk) and the context (Con-Msk). Using ŷ, we generate an object mask M obj to mask the detected foreground objects and a context mask M con = 1 -M obj , which are then applied to z. • Guided Cutout (G-Cutout): in order to reduce the reliance on specific parts of the objects, and inspired by Cutout [START_REF] Devries | Improved regularization of convolutional neural networks with cutout[END_REF] that randomly masks some parts of the input image, we first find the possible spatial extent (i.e., bounding box) of each detected object using ŷ.

We then zero-out a random crop within each object's bounding box from the corresponding feature map z. • Intermediate VAT (I-VAT): to further push the output distribution to be isotropically smooth around each data point, we investigate using VAT [START_REF] Miyato | Virtual adversarial training: a regularization method for supervised and semi-supervised learning[END_REF] as a perturbation function to be applied to z instead of the unlabeled inputs. For a given auxiliary decoder, we find the adversarial perturbation r adv that will alter its prediction the most. The noise is then injected into z to obtain the perturbed version z = r adv + z.

Random perturbations. (DropOut) Spatial dropout [START_REF] Tompson | Efficient object localization using convolutional networks[END_REF] is also applied to z as a random perturbation.

Practical considerations

A each training iteration, we sample an equal number of labeled and unlabeled samples. As a consequence, we iterate on the set D l more times than on its unlabeled counterpart D u , thus risking an overfitting of the labeled set D l .

Avoiding Overfitting. Motivated by [START_REF] Pohlen | Full-resolution residual networks for semantic segmentation in street scenes[END_REF] who observed improved results by sampling only 6% of the hardest pixels, and [START_REF] Xie | Unsupervised data augmentation[END_REF] who showed an improvement when gradually † {condition} 1 is a boolean function outputting 1 if the condition is true, 0 otherwise. releasing the supervised training signal in a SSL setting, we propose an annealed version of the bootstrapped-CE (ab-CE) in [START_REF] Pohlen | Full-resolution residual networks for semantic segmentation in street scenes[END_REF]. With an output f (x l i ) ∈ R C×H×W in the form of a probability distribution over the pixels, we only compute the supervised loss over the pixels with a probability less than a threshold η:

L s = 1 |D l | x l i ,yi∈D l {f (x l i ) < η} 1 H(y i , f (x l i )) (4)
To release the supervised training signal, the threshold parameter η is gradually increased from 1 C to 0.9 during the beginning of training, with C as the number of output classes.

Exploiting weak-labels

In some cases, we might be provided with additional training data that is less expensive to acquire compared to pixel-level labels, e.g., image-level labels. Formally, instead of an unlabeled set D u , we are provided with a weakly labeled set D w = {(x w 1 , y w 1 ), . . . , (x w m , y w m )} alongside a pixel-level labeled set D l , with y w i is the i-th image-level label corresponding to the i-th weakly labeled input image x w i . The objective is to extract additional information from the weak labeled set D w to further enhance the representations of the encoder h. To this end, we add a classification branch g c consisting of a global average pooling layer followed by a classification layer, and pretrain the encoder for a classification task using binary CE loss.

Following previous works [START_REF] Ahn | Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation[END_REF][START_REF] Lee | Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference[END_REF][START_REF] Huang | Weakly-supervised semantic segmentation network with deep seeded region growing[END_REF], the pretrained encoder and the added classification branch can then be exploited to generate pseudo pixel-level labels y p . We start by generating the CAMs M as in [START_REF] Zhou | Learning deep features for discriminative localization[END_REF]. Using M ∈ R C×H×W , we can then generate pseudo labels y p , with a background θ bg and a foreground θ f g thresholds. The pixels with attention scores less than θ bg (e.g., 0.05) are considered as background. For the pixels with an attention score larger than θ f g (e.g., 0.30), they are assigned the class with the maximal attention score, and the rest of the pixels are ignored. After generating y p , we conduct a final refinement step using dense CRF [START_REF] Krähenbühl | Efficient inference in fully connected crfs with gaussian edge potentials[END_REF].

In addition to considering D w as an unlabeled set and imposing a consistency over its examples, the pseudo-labels are used to train the auxiliary networks g k a •h using a weakly supervised loss L w . In this case, the loss in Eq. (3) becomes:

L = L s + ω u L u + ω w L w (5)
With

L w = 1 |D w | 1 K x w i ∈Dw K k=1 H(y p , g k a (z i )) (6

Cross-Consistency Training on Multiple Domains

In this section, we extend the propose framework to a semi-supervised domain adaption setting. We consider the case of two datasets {D (1) , D (2) } with partially or fully non-overlapping label spaces, each one contains a set of labeled and unlabeled examples

D (i) = {D (i) l , D (i) u }.
The objective is to simultaneously train a segmentation network to do well on the test data of both datasets, which is drown from the different distributions.

Our assumption is that enforcing a consistency over both unlabeled sets D u might impose an invariance of the encoder's representations across the two domains. To this end, on top of the shared encoder h, we add domain specific main decoder g (i) and auxiliary decoders g k(i) a . Specifically, as illustrated in Fig. 4, we add two main decoders and 2K auxiliary decoders on top of the encoder h. During training, we alternate between the two datasets, at each iteration, sampling an equal number of labeled and unlabeled examples from each one, computing the loss in Eq. ( 3) and training the shared encoder and the corresponding main and auxiliary decoders.

Experiments

To evaluate the proposed method and investigate its effectiveness in different settings, we carry out detailed experiments. In Section 4.4, we present an extensive ablation study to highlight the contribution of each component within the proposed framework, and compare it to stateof-the-art methods in a semi-supervised setting. Additionally, in Section 4.5 we apply the proposed method in a semi-supervised domain adaptation setting and show performance above baseline methods.

Network Architecture

Encoder. For the following experiments, the encoder is based on a ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] pretrained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] provided by [START_REF] You | Torchcv: A pytorch-based framework for deep learning in computer vision[END_REF] and a PSP module [START_REF] Zhao | Pyramid scene parsing network[END_REF]. Following previous works [START_REF] Zhao | Pyramid scene parsing network[END_REF][START_REF] Huang | Weakly-supervised semantic segmentation network with deep seeded region growing[END_REF][START_REF] Ahn | Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation[END_REF], the last two strided convolutions of ResNet are replaced with dilated convolutions.

Decoders. For the decoders, taking the efficiency and the number of parameters into consideration, we choose to only use 1 × 1 convolutions. After an initial 1 × 1 convolution to adapt the depth to the number of classes C, we apply a series of three sub-pixel convolutions [START_REF] Shi | Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[END_REF] with ReLU non-linearities to upsample the outputs to original input size.

Datasets and Evaluation Metrics

Datasets. In a semi-supervised setting, we evaluate the proposed method on PASCAL VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF], consisting of 21 classes (with the background included) and three splits, training, validation and testing, with of 1464, 1449 and 1456 images respectively. Following the common practice [START_REF] Huang | Weakly-supervised semantic segmentation network with deep seeded region growing[END_REF][START_REF] Zhao | Pyramid scene parsing network[END_REF], we augment the training set with additional images from [START_REF] Hariharan | Semantic contours from inverse detectors[END_REF]. Note that the pixel-level labels are only extracted from the original training set.

For semi-supervised domain adaption, for partially overlapping label spaces, we train on both Cityscapes [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] and CamVid [START_REF] Gabriel J Brostow | Semantic object classes in video: A high-definition ground truth database[END_REF]. Cityscapes is a finely annotated autonomous driving dataset with 19 classes. We are provided with three splits, training, validation and testing with 2975, 500 and 1525 images respectively. CamVid contains 367 training, 101 validation and 233 testing images. Although originally the dataset is labeled with 38 classes, we use the 11 classes version [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF]. For experiments over non-overlapping labels spaces, we train on Cityscapes and SUN RGB-D [START_REF] Song | Sun rgb-d: A rgb-d scene understanding benchmark suite[END_REF]. SUN RGB-D is an indoor segmentation dataset with 38 classes containing two splits, training and validation, with 5285 and 5050 images respectively. Similar to [START_REF] Kalluri | Universal semi-supervised semantic segmentation[END_REF], we train on the 13 classes version [START_REF] Handa | Understanding real world indoor scenes with synthetic data[END_REF].

Evaluation Metrics. We report the results using mIoU (i.e., mean of class-wise intersection over union) for all the datasets.

Implementation Details

Training Settings. The implementation is based on the PyTorch 1.1 [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] framework. For optimization, we train for 50 epochs using SGD with a learning rate of 0.01 and a momentum of 0.9. During training, the learning rate is annealed following the poly learning rate policy, where at each iteration, the base learning rate is multiplied by 1 -( iter max iter ) power with power = 0.9.

For PASCAL VOC, we take crops of size 321 × 321 and apply random rescaling in the range of [0.5, 2.0] and random horizontal flip. For Cityscapes, Cam-Vid and SUN RGB-D, following [START_REF] Kalluri | Universal semi-supervised semantic segmentation[END_REF][START_REF] Hung | Adversarial learning for semi-supervised semantic segmentation[END_REF], we resize the input images to 512×1024, 360 × 480 and 480 × 640 respectively, without any dataaugmentation.

Reproducibility All the experiments are conducted on a V-100 GPUs. The implementation is available at https: //github.com/yassouali/CCT The proposed method consists of several types of perturbations and a variable number of auxiliary decoders. We thus start by studying the effect of the perturbation functions with different numbers of auxiliary decoders, in order to provide additional insight into their individual performance and their combined effectiveness. Specifically, we measure the effect of different numbers of auxiliary decoders K (i.e., K = 2, 4, 6 and 8) of a given perturbation type. We refer to this setting of our method as "CCT {perturbation type}", with seven possible perturbations. We also measure the combined effect of all perturbations resulting in K × 7 auxiliary decoders in total, and refer to it as "CCT full". Additionally, "CCT full+ab-CE" indicates the usage of the annealed-bootstrapped CE as a supervised loss function. We compare them to the baseline, in which the model is trained only using the labeled examples.

CamVid. We carried out the ablation on CamVid with 20, 50 and 100 labels; the results are shown in Fig. 5. We find that each perturbation outperforms the baseline, with the most dramatic differences in the 20-label setting with up to 21 points. We also surprisingly observe an insignificant overall performance gap among different perturbations, confirming the effectiveness of enforcing the consistency over the hidden representations for semantic segmentation, and highlighting the versatility of CCT and its success with numerous perturbations. Increasing K results in a modest improvement overall, with the smallest change for Con-Msk and Obj-Msk due to their lack of stochasticity. Interestingly, we also observe a slight improvement when combining all of the perturbations, indicating that the encoder is able to generate representations that are consistent over many perturbations, and subsequently, improving the overall performance. Additionally, gradually releasing the training signal using ab-CE helps increase the performance with up to 8%, which confirms that overfitting of the labeled examples can cause a significant drop in performance. PASCAL VOC. In order to investigate the success of CCT on larger datasets, we conduct additional ablation experiments on PASCAL VOC using 1000 labeled examples, The results are summarized in Fig. 6. We see similar re-sults, where the proposed method makes further improvement compared to the baseline with different perturbations, from 10 to 15 points. The combined perturbations yield a small increase in the performance, with the biggest difference with K = 6. Furthermore, similar to CamVid, when using the ab-CE loss, we see a significant gain with up to 7 points compared to CCT full.

Based on the conducted ablation studies, for the rest of the experiments, we use the setting of "CCT full" with K = 2 for Con-Msk and Obj-Msk due to their lack of stochasticity, K = 2 for I-VAT given its high computational cost, and K = 6 for the rest of the perturbations, and refer to it as "CCT".

Comparison to Previous Work

To further explore the effectiveness of our framework, we quantitatively compare it with previous semi-supervised semantic segmentation methods on PASCAL VOC. Table 1 compares CCT with other semi-supervised approaches. Our approach outperforms previous works relying on the same level of supervision and even methods which exploit imagelevel labels. We also observe an increase of 3.8 points when using additional image-level labels, affirming the flexibility of CCT, and the possibility of using it with different types of labels without any learning conflicts.

Semi-Supervised Domain Adaptation Setting

In real world applications, we are often provided with pixel-level labels collected from various sources, thus distinct data distributions. To examine the effectiveness of CCT when applied to multiple domains with a variable degree of labels overlap, we train our model simultaneously on two datasets, Cityscapes (CS) + CamVid (CVD) for partially overlapping labels, and Cityscapes + SUN RGB-D (SUN) for the disjoint case. Cityscapes + CamVid. The results for CCT on Cityscapes and CamVid datasets with 50 and 100 labeled examples are given in Table 2. Similar to the SSL setting, CCT outperforms the baseline significantly, where the model is iteratively trained using only on the labeled examples, with up to 12 points for n = 100, we even see a modest increase compared to previous work. This confirms our hypothesis that enforcing a consistency over different datasets does indeed push the encoder to produce invariant representation across different domains, and consequently, increases the performance over the baseline while delivering similar results on each domain individually. Cityscapes + SUN RGB-D. For cross domain experiments, where the two domains have distinct labels spaces, we train on both Cityscapes and SUN RGB-D to demonstrate the capability of CCT to extract useful visual relationships and perform knowledge transfer between dissimilar domains, even in completely different settings. The results are shown in Table 2. Interestingly, despite the distribution mismatch between the datasets, and the high number of labeled examples (n = 1500), CCT still provides a meaningful boost over the baseline with 5.9 points difference and 7.3 points compared to previous work. Showing that, by enforcing a consistency of predictions on the unlabeled sets of the two datasets over different perturbations, we can extract additional training signal and enhance the representation learning of the encoder, even in the extreme case with non-overlapping label spaces, without any performance drop when an invariance of representations across both datasets is enforced at the level of encoder's outputs.

Conclusion

In this work, we present cross-consistency training (CCT), a simple, efficient and flexible method for a consistency based semi-supervised semantic segmentation, yielding state-of-the-art results. For future works, a possible direction is exploring the usage of other perturbations to be applied at different levels within the segmentation network. It would also be interesting to adapt and examine the effectiveness of CCT in other visual tasks and learning settings, such as unsupervised domain adaptation.

Figure 1 .

 1 Figure 1. The proposed Cross-Consistency training (CCT). For the labeled examples, the encoder and the main decoder are trained in a supervised manner. For the unlabeled examples, a consistency between the main decoder's predictions and those of the auxiliary decoders is enforced, over different types of perturbations applied to the inputs of the auxiliary decoders. Best viewed in color.

3. 2 .

 2 Cross-Consistency Training for semantic segmentation 3.2.1 Problem Definition In SSL, we are provided with a small set of labeled training examples and a larger set of unlabeled training examples. Let D l = {(x l 1 , y 1 ), . . . , (x l n , y n )} represent the n labeled examples and D u = {x u 1 , . . . , x u m } represent the m unlabeled examples, with x u

Figure 2 .

 2 Figure 2. The cluster assumption in semantic segmentation. (a) Examples from PASCAL VOC 2012 train set. (b) Pixel-level labels. (c) Input level. The average euclidean distance between each patch of size 20 × 20 centered at a given spatial location extracted from the input images, and its 8 neighboring patches. (d) Hidden representations level. The average euclidean distance between a given 2048-dimensional activation at each spatial location and its 8 neighbors. Darkest regions indicate high average distance. the shared encoder z i = h(x u i ) * . Let us consider R stochastic perturbations functions, denoted as p r with r ∈ [1, R],where one perturbation function can be assigned to multiple auxiliary decoders. With various perturbation settings, we generate K perturbed versions zk i of the intermediate representation z i , so that the k-th perturbed version is to be fed to the k-th auxiliary decoder. For consistency, we consider the perturbation function as part of the auxiliary decoder, (i.e., g k a can be seen as g k a • p r ). The training objective is then to minimize the unsupervised loss L u , which measures the discrepancy between the main decoder's output and that of the auxiliary decoders:

Figure 3 .

 3 Figure3. Illustration of our approach. For one training iteration, we sample a labeled input image x l and its pixel-level label y together with an unlabeled image x u . We pass both images through the encoder and main decoder, obtaining two main predictions ŷl and ŷu . We compute the supervised loss using the pixel-level label y and ŷl . We apply various perturbations to z, the output of the encoder for x u , and generate auxiliary predictions ŷ(i) a using the perturbed versions z(i) . The unsupervised loss is then computed between the outputs of the auxiliary decoders and that of the main decoder.

Figure 4 .

 4 Figure 4. CCT on multiple domains. On top of a shared encoder, we add domain specific main decoder and K auxiliary decoders. During training, we alternate between the two domains, sampling labeled and unlabeled examples and training the corresponding decoders and the shared encoder at each iteration.

Figure 5 .

 5 Figure 5. Ablation Studies on CamVid with 20, 50 and 100 labeled images. With different types of perturbations and a variable number of auxiliary decoders K, we compare the individual and the combined effectiveness of the perturbations to the baseline in which the model is trained only on the labeled examples. CCT full refers to using all of the 7 perturbations, i.e. the number of auxiliary decoder is K × 7.

Figure 6 .

 6 Figure 6. Ablation study on PASCAL VOC. Ablation study results with 1000 labeled examples using different perturbations and various numbers of auxiliary decoders K.Inference Settings. For PASCAL VOC, Cityscapes and SUN RGB-D, we report the results obtained on the validation set, and on the test set of CamVid dataset.

4. 4 .

 4 Semi-Supervised Setting 4.4.1 Ablation Studies

Table 1 .

 1 Comparison with the-state-of-the-art. CCT performance on PASCAL VOC compared to other semi-supervised approaches.

	Method	Pixel-level	Image-level	Val
		Labeled	Labeled
		Examples	Examples
	WSSL [38]	1.5k	9k	64.6
	GAIN [32]	1.5k	9k	60.5
	MDC [53]	1.5k	9k	65.7
	DSRG [22]	1.5k	9k	64.3
	Souly et al. [49] 1.5k	9k	65.8
	FickleNet [31]	1.5k	9k	65.8
	Souly et al. [49] 1.5k	-	64.1
	Hung et al. [23] 1.5k	-	68.4
	CCT	1k	-	64.0
	CCT	1.5k	-	69.4
	CCT	1.5k	9k	73.2

Table 2 .

 2 CCT applied to CS+CVD. CCT performance when simultaneously trained on two datasets with overlapping label spaces, which are Cityscapes (CS) and CamVid (CVD).

	Method	n=50	n=100
		CS CVD Avg.	CS CVD Avg.
	Kalluri, et al. [24] 34.0 53.2 43.6	41.0 54.6 47.8
	Baseline	31.2 40.0 35.6	37.3 34.4 35.9
	CCT	35.0 53.7 44.4	40.1 55.7 47.9

Table 3 .

 3 CCT applied to CS+CVD. CCT performance when trained on both datasets Cityscapes (CS) and SUN RGB-D (SUN) datasets, for the case of non-overlapping label spaces.

	Method	Labeled	CS SUN Avg.
		Examples
	SceneNet [35]	Full (5.3k)	-	49.8	-
	Kalluri, et al. [24] 1.5k	58.0 31.5 44.8
	Baseline	1.5k	54.3 38.1 46.2
	CCT	1.5k	58.8 45.5 52.1

Acknowledgements. This work was supported by Ranstad corporate research chair. We would also like to thank Saclay-IA plateform of Université Paris-Saclay and Mésocentre computing center of CentraleSupélec and École Normale Supérieure Paris-Saclay for providing the computational resources.

Code is available at https://github.com/yassouali/CCT