Rachel Pflieger

Sergey I Nikitenko

Muthupandian Ashokkumar

Effect of NaCl salt on sonochemistry and sonoluminescence in aqueous solutions

Keywords: sonochemistry, sonoluminescence, salt, NaCl, active bubble population

The presence of salts in a solution is known to affect sonochemistry, but until now no consensus has been reached in the literature on how and why a salt influences sonochemistry. The present study focuses on the effect of NaCl on sonochemical activity and sonoluminescence at 362-kHz frequency in aqueous solutions saturated with He and Ar. It is shown that the presence of salt has a multiple impact: the global population of active bubbles decreases due to the decreasing gas solubility, new chemical reactions involving Na • and Cl • atoms occur that influence hydrogen and hydrogen peroxide yields and the standing wave component of the US wave is enhanced, favoring sonoluminescence emission. Interestingly, the effect of salt greatly depends on the nature of the saturating gas: for instance, strong acidification occurs under He, while it is limited under Ar.

Introduction

During the passage of ultrasound in a solution, chemical activity takes place, referred to as sonochemistry that is due to acoustic cavitation, the nucleation, growth and violent collapse of microbubbles in the solution. The sonochemistry of pure water in the presence of different gases is pretty well known. However, as soon as some solutes are added, many properties of the system are affected, as has been shown on salts, common "simple" solutes that can be considered as models to account for the composition of real sonochemical systems: the gas solubility decreases, the viscosity increases, [START_REF] Goncalves | Viscosity of NaCl and KCl Solutions in Range 25-50-Degrees-C[END_REF] bubbles get smaller, [START_REF] Brotchie | Acoustic Bubble Sizes, Coalescence, and Sonochemical Activity in Aqueous Electrolyte Solutions Saturated with Different Gases[END_REF] coalescence is reduced, 2a, 3 and probably the bubble dynamics is modified, too. Also the chemical activity has been shown to be impacted, though the reported trends show some discrepancy. Lepoint et al. [START_REF] Lepoint | What Exactly is Cavitation Chemistry[END_REF] studied I3 -yield (Weissler reaction) upon addition of MgCl2 (in the concentration range from 10 -4 M to 0.3 M) in CCl4-saturated solutions sonicated by 1.67 MHz ultrasound under air. They reported a sharp minimum for concentrations between 2 10 -3 M and 3 10 -3 M and explained it by a change in the electrokinetic potential of the bubbles. However, this result could not be reproduced by Gutiérrez et al. [START_REF] Gutierrez | Observations on the Role of MgCl2 in the Weissler Reaction[END_REF] at 1 MHz under air saturated conditions, who obtained, both in the presence and absence of CCl4 in solution, constant I3 -yield up to a MgCl2 concentration of 0.1 M or higher. At MgCl2 concentrations higher than 0.1 M in CCl4saturated solutions (above 2 M in the absence of CCl4) I3 -yield then strongly decreased, which the authors correlated to the concomitant increase in viscosity creating conditions less favorable for cavitation. It is however to be noted that the change in viscosity is relatively 2 low: at 20°C water viscosity is of 1 cP while that of 3 M NaCl is 1.2 cP. [START_REF] Goncalves | Viscosity of NaCl and KCl Solutions in Range 25-50-Degrees-C[END_REF] On the contrary, some other works reported an increase in H2O2 or I3 -formation rates upon addition of salts. For instance, Brotchie et al. 2a measured 3-10 times higher I3 -yields in concentrated solutions of NaClO4 or NaNO3 compared to water, during sonication at 515 kHz after saturation with He or Ar. They attributed it to lower dissolved gas concentrations leading to reduced gas and vapor contents in bubbles prior to collapse and to a lower clustering density. As for Katekhaye and Gogate, [START_REF] Gogate | A comparison of the degree of intensification due to the use of additives in ultrasonic horn and ultrasonic bath[END_REF] they reported an increase by almost a factor of 2 of I3 -yield during the sonication under air at 20 kHz or 204 kHz in the presence of 0.034-0.34 M NaCl or 0.029 -0.29 M NaNO2. The discrepancy that appears between the reported trends is probably to be accounted for by the lack of a detailed analysis of the various factors that can affect H2O2 or I3 -formation. It is to be noted here that these two yields can also be different, [START_REF] Pflieger | Characterization of Cavitation Bubbles and Sonoluminescence[END_REF] in particular in the presence of air, because KI dosimetry is not specific to OH radicals: the sonolysis of water in the presence of nitrogen yields nitrous acid [START_REF] Ouerhani | Spectroscopy of Sonoluminescence and Sonochemistry in Water Saturated with N2-Ar Mixtures[END_REF] that readily oxidizes iodide ion [START_REF] Abeledo | The reaction between nitrite and iodide and its application to the iodometric titration of these anions[END_REF] and this reaction is catalyzed by dissolved oxygen [START_REF] Couto | The catalytic cycle of oxidation of iodide ion in the oxygen/nitrous acid/nitric oxide system and its potential for analytical applications[END_REF] . In particular, while these studies underlined the various physical effects induced by salts in solution, like changes in the viscosity or in the gas solubility, they did not take into account the chemical reactions in which salts can be involved.

Another source of information on the way salts impact the conditions of cavitation is sonoluminescence (SL). In air-saturated aqueous solutions of salts, [START_REF] Wall | Multibubble sonoluminescence in aqueous salt solutions[END_REF] the SL spectra consist of a featureless continuum. On the contrary, in the presence of a rare gas, emissions arising from electronic transitions appear in the SL spectra: OH (A 2  + -X 2 ) that is typical of aqueous solutions and emission from excited Na atoms accompanied by a blue satellite. The latter corresponds to the transition of a Na-Ar exciplex, an alkali-metal / rare-gas van der Waals molecule formed within the cavitation bubbles. [START_REF] Lepoint-Mullie | Evidence for the emission of 'alkali-metal-noble-gas' van der Waals molecules from cavitation bubbles[END_REF] Lepoint-Mullie et al. [START_REF] Lepoint-Mullie | Evidence for the emission of 'alkali-metal-noble-gas' van der Waals molecules from cavitation bubbles[END_REF] proposed the following mechanism of formation: after mechanical addition of some volume of solution to a collapsing bubble, the salt molecules would be released in the gas (or, actually, plasma) phase and homolytically cleaved. The metal atoms would then be electronically excited by threebody reactions, leading to Na * . The formation of an electronically excited Na-Ar exciplex would occur following a three-body collision with two rare gas atoms.

Besides, recent literature showed that different bubble populations are responsible for the emissions of Na* and of the SL continuum. Abe and Choi 13 reported a difference in the timing distributions of both SL emissions during the sonication at 137 kHz of a NaCl 2 M aqueous solution saturated with Ar. Spectroscopy of SL indicated that the evolution of sonoluminescing bubble population and of Na-emitting bubbles with gas flow, power and frequency were different. [START_REF] Cairos | Effects of Argon Sparging Rate, Ultrasonic Power, and Frequency on Multibubble Sonoluminescence Spectra and Bubble Dynamics in NaCl Aqueous Solutions[END_REF] Moreover, visual observations of SL from both concentrated sulfuric acid solutions [START_REF] Hatanaka | Sonoluminescence of Alkali-Metal Atoms in Sulfuric Acid: Comparison with That in Water[END_REF] , phosphoric acid [START_REF] Thiemann | Sonoluminescence and dynamics of cavitation bubble populations in sulfuric acid[END_REF] and in aqueous solutions [START_REF] Abe | Spatiotemporal Separation of Na-Atom Emission from Continuum Emission in Sonoluminescence[END_REF][START_REF] Sunartio | Correlation between Na* emission and "Chemically active" acoustic cavitation bubbles[END_REF] evidenced a spatial separation of continuum emission (and emission from OH radicals) and alkali-metal emission (blue continuum and orange Na* emission). The former was attributed to higher-temperature bubbles and the latter to lower-temperature bubbles. [START_REF] Abe | Spatiotemporal Separation of Na-Atom Emission from Continuum Emission in Sonoluminescence[END_REF] It was also suggested, based on a bubble radius simulation that continuum emission would result from smaller bubbles than Naatom emission. The latter assumption was confirmed by high-speed imaging in sulfuric acid at 23 kHz under Xe: [START_REF] Thiemann | Sonoluminescence and dynamics of cavitation bubble populations in sulfuric acid[END_REF] different cavitation bubble populations were observed in the zones of different SL colors, namely slow and spherically collapsing bubbles in the blue-emitting zones, fast bubbles subject to liquid jetting during their collapse in the red-emitting zone. However, while it is clear that Na * emission arises only from jetting bubbles, it has also been reported in Xe saturated phosphoric acid that SL continuum emission can also be observed from large bubbles having a translational motion and therefore showing jetting. [START_REF] Cairos | Simultaneous High-Speed Recording of Sonoluminescence and Bubble Dynamics in Multibubble Fields[END_REF] Also, it should be kept in mind that not all sonochemically active bubbles do emit SL, whether continuum or Na*. For instance, the spatial distribution of sonochemically active bubbles is broader than that of sonoluminescing bubbles. [START_REF] Ashokkumar | Spatial Distribution of Acoustic Cavitation Bubbles at Different Ultrasound Frequencies[END_REF] Thus different kinds of bubbles (chemically active, SL-continuum emitting, SL-Na*-emitting, etc.) do coexist and they may be impacted in different ways by the presence of salts. So, the common "simple" salts do in fact impact many aspects of acoustic cavitation, and the way to a global understanding of the effect of salts in cavitating systems necessitates a multi-lateral approach combining physical and chemical considerations. This paper aims at combining diverse complementary approaches (measurement of sonochemical activity, of sonoluminescence spectra and visualization of cavitation activity) to better understand the influence of the presence of a salt, using NaCl as a representative, in water under a rare gas flow system (He or Ar), at 362 kHz.

Experimental Methods

All chemicals (in particular NaCl, 99+%, STREM chemicals) were of analytical grade, and deionized water (Milli-Q 18.2 MΩ cm) was used to prepare all solutions. Argon, helium (99.999%) and Ar-20%O2 gas mixture (99.9%) were provided by Air Liquide.

Experiments were performed at 362 kHz ultrasonic frequency and an absorbed acoustic power of 43 W in a thermostated glass cylinder fitted on the 362-kHz transducer (ELAC Nautik, 25 cm 2 ) which was connected to a multifrequency generator (T&C Power Conversion, Inc.). On top of the reactor, opposite to the high-frequency transducer was placed a 20-kHz titanium probe (750 W, 1 cm 2 , Sonics generator), although not used here, to keep conditions constant with other studies in our group.

In the sonoreactor, 250 mL of the solutions were continuously sparged with Ar, He or Ar-20%O2 at a flow rate of 70 mL/min for Ar and Ar-20%O2 (respectively 210 mL/min for He) about 30 min before and during sonication. The solution temperature was maintained around 10°C with a Huber Unistat Tango cryostat.

The SL spectra were collected using parabolic Al-coated mirrors through the quartz window of the sonoreactor in the spectral range 240-800 nm using a SP 2356i Roper Scientific spectrometer (gratings 300blz300 and 150blz500, slit width 0.25 mm) coupled to a liquid-nitrogen-cooled CCD camera (SPEC10-100BR, Roper Scientific). For each experiment, at least three spectra (measured after reaching a steady-state temperature) were averaged and corrected for the background noise and for the quantum efficiencies of gratings and CCD. Optical high-pass filters were used when needed, to avoid second-order light. Typical spectrum acquisition times were of 60 s under Ar and 300 s under He.

Images of SL were taken through the same quartz window of the sonoreactor as SL spectra, using a digital camera (Canon EOS 70D) equipped with a Canon zoom lens of 35 mm. Acquisition was performed with the software Digital Photo Professional. Sensitivity was in general ISO 4000 and typical acquisition times were 10-300 s.

The formation rate of the gaseous sonolysis product H2 was quantified for 1-h-sonolysis using a quadrupole mass spectrometer (PROLAB 300, Thermo Fisher) in multiple ion monitoring (MIM) mode, after trapping water vapor present in the outlet gas with molecular sieves (Aldrich, 3 Å). Mass spectrometer calibration was performed with gas mixtures provided by Messer. Hydrogen peroxide formation kinetics was monitored by UV-vis absorption spectrophotometry after the addition of 750 µL of a solution of 2 × 10 -2 M TiOSO4 in H2SO4 0.5 M to 750-µL samples of the sonicated solutions to form an orange Ti(IV) peroxide complex (max = 410 nm, ε = 814 cm -1 mol -1 L). The accuracy for the reaction rate measurements was statistically estimated to be 10-20% for H2 and 10% for H2O2, based on multiple frequent measurements of reference systems. In contrast to Weissler reaction, unless performed in the presence of a catalyst, this analytical method is specific to H2O2 and is not sensitive to secondary redox processes. To allow easier comparison with results published elsewhere, formation rates were converted into yields (in µmol/kJ) by multiplying them by the solution volume and dividing by the absorbed acoustic power.

The pH evolution during sonication was followed using a Cyberscan pH300 from Eutech Instruments. Presented H + yields were measured during approximately half an hour after reaching a steady-state temperature.

Results and discussion

Chemical yields

Fig. 1 presents sonolysis yields of H2 and H2O2 (normalized to water) as a function of NaCl concentration, under Ar and He continuous flows. In both gases, the yields of H2 and H2O2 decrease with an increase in NaCl concentration. A first possible explanation would consider a decrease in the number of active bubbles as a result of a decrease in gas solubility with an increase in salt concentration. This is in line with Okitsu et al.'s 21 observation that in pure water the chemical efficiency was proportional to the gas solubility of rare gases. Indeed, as shown in Fig. 1SI (in Supporting Information), H2 and H2O2 yields divided by the gas solubility give the same value for He and Ar. It has also been shown that when salt is added in solution, the gas solubility decreases [START_REF] Schumpe | The Estimation of Gas Solubilities in Salt-Solutions[END_REF] leading to a lesser amount of coalescence, 2a less active bubbles and lower sonochemical yields. However, this fact alone cannot account for the strong decrease in chemical yields: as exemplified in Fig. 1SI in Supporting Information, H2 and H2O2 yields divided by the gas solubility decrease with NaCl concentration (H2 seems to plateau above 2 M). Also, a decrease in water vapor pressure (of the order of 10% between 0 and 3 M) [START_REF] Sirkin | Vapor Pressure of Aqueous Solutions of Electrolytes Reproduced with Coarse-Grained Models without Electrostatics[END_REF] alone cannot explain this strong decrease. One can suggest that chemical reactions do strongly contribute to this decrease in addition to the changes in the number of bubbles and vapor pressure. When solution droplets enter the cavitation bubbles during bubble collapse, salt molecules are released in the plasma following water evaporation, where they are homolytically cleaved, leading to Na • and Cl • , as described by Lepoint-Mullie et al. [START_REF] Lepoint-Mullie | Evidence for the emission of 'alkali-metal-noble-gas' van der Waals molecules from cavitation bubbles[END_REF] NaCl ))

) → Na • + Cl • (1)
The so-formed Na • and Cl • atoms can then react with H (or H2), OH • and H2O species, hence impacting the formation yields of sonolysis products. Reactions with water molecules (3 and 8) are expected to be prevailing due to the higher concentration and reactive crosssection of these molecules, but many possible reactions do coexist, the reaction rates of which strongly depend on whether the considered species are in their ground or excited state. [START_REF] Lavvas | Electron Densities and Alkali Atoms in Exoplanet Atmospheres[END_REF] Some of the relevant reactions are (with M a third body):

Na • + OH • → NaOH (2) Na • + H2O → NaOH + H (3)
Na • + H + M → NaH + M (4) Cl • + OH • → HOCl (5) Cl • + H → HCl (6) Cl • + H2 → HCl + H (7)
Cl • + H2O → HCl + OH • (8)

Na • + Cl • + M → NaCl + M (9)
As underlined previously, no agreement is found in the literature concerning the influence of salts on H2O2 formation rate. The observed discrepancy may be traced back to chemical reactions linked to the nature of the used salts. For instance, contrary to the present work, Katekhaye and Gogate [START_REF] Gogate | A comparison of the degree of intensification due to the use of additives in ultrasonic horn and ultrasonic bath[END_REF] reported an increase in Weissler reaction rate in the presence of NaCl during the sonication of a KI solution at 20 kHz under air. They attributed it to a change in the physicochemical properties of the solution due to the salt, leading to increases in collapse pressure and temperature and thus to higher H2O2 formation rate. However, this increase in I3 - formation can most probably be attributed to the following interfering reaction: Cl • + I -→ Cl -+ I • (10) The so-formed I • atom can then lead to I3 -formation following the same reaction path as in standard Weissler reaction. In this case, no direct correlation can be considered between I3 - and H2O2 yields. Similarly, the presence of an interfering reaction can explain the increase in H2O2 formation rate observed by Brotchie et al. 2a upon addition of NaClO4 or NaNO3 during sonication under He or Ar at 515 kHz. Indeed, the sonolysis products of ClO4 -and NO3 - anions are oxidants that can lead to the formation of additional H2O2. For instance, it was shown [START_REF] Dalodiere | Effect of ultrasonic frequency on H2O2 sonochemical formation rate in aqueous nitric acid solutions in the presence of oxygen[END_REF] that the reaction of NO3 -with OH • radicals forms NO3 • radicals that yield H2O2 after hydrolysis, thus doubling the amount of H2O2 produced as soon as enough NO3 -ions are present.

The mechanism of scavenging of OH • radicals and H • atoms by Cl • and Na • proposed here explains the concomitant decrease in H2 and H2O2 yields. Due to the formation of HCl (reactions 6, 7, 8) and NaOH (reactions 2, 3), a change in the pH of NaCl solutions may be expected during sonication in a rare gas atmosphere (in contrary to water sonolysis under Ar or He, where no pH change is to be observed). Indeed, as shown in Fig. 2SI (in Supporting Information) that presents the concentration with sonication time of sonolytically produced H + in a 3 M NaCl solution under He, acidification of the solution occurs. The reason why more HCl than NaOH is produced may be due to the competition for reaction with OH radicals (reactions 2&5) and different reactive cross-sections of the different reactions. Yields of H + for the different NaCl concentrations and the two gases are presented in Fig. 2. The most striking feature in Fig. 2 is the very large H + yields obtained under He: much higher (an order of magnitude) than yields of water sonolysis products H2 and H2O2 in NaCl solutions saturated with He (Fig. 1) on the one hand, and on the other hand, much higher than H + yields under Ar. The H + can a priori be formed via reactions 6, 7 and 8. The first two involve H atoms or H2 molecules and directly lead to a decrease in H2 formation. Comparison of H + yields under He with a decrease in H2 yields (compared to the water case (Fig. 3)) shows that the decrease in H2 formation yield under He cannot account for the very high proton yields, thus underlining the importance of reaction 8 in the proton formation. This pH evolution shows one of the very rare examples (if any other exists) where a "sonochemical activity" would be higher under He. This surprising fact may be traced back to the different bubble contents induced by the different gas solubilities (the number of Ar atoms inside bubbles is higher than in the He case) and to the larger reactive cross-section of Ar compared to He due to the atom larger size. Thus, while under He, reactions 6-8 of radicals are favored, in the Ar case three-body reactions like reaction 9 have a high probability.

Another interesting feature in Fig. 2 is that the effect of NaCl concentration (in the range 0.5-5 M) is different for both gases. Under He, G(H + ) is nearly independent from NaCl concentration, while under Ar, it decreases. Actually, acidification occurs due to the presence of NaCl, thus the relevant parameter to consider would be H + yield relative to salt concentration (Fig. 3SI in Supporting Information) that decreases for both gases, though with a stronger decrease for Ar. As already stated, the origin of the changes in yields of sonochemical products in the presence of salt is not only based on reactions 2-8, but also linked to a decrease in the number of chemically active bubbles with NaCl concentration. The change in gas solubility between water and 5 M NaCl aqueous solution is almost double for Ar than for He (3.9 vs. 2.1) [START_REF] Schumpe | The Estimation of Gas Solubilities in Salt-Solutions[END_REF] , which may lead to a stronger decrease in the active bubble population for Ar, and thus participate (as confirmed by Fig. 1SI) in the more pronounced decrease in H2 and H2O2 yields with NaCl concentration (Fig. 1) and of the relative H + yield.

Fig. 3: Comparison of the yields of H + (G(H + )) formed during the sonolysis of NaCl solutions under Ar or He (362 kHz, 43 W, 13°C) with a decrease in H2 yields between water and NaCl solutions (G(H2) H2O -G(H2) NaCl ). Fig 4 shows SL spectra obtained under Ar, in water and in 1 M and 3 M NaCl solutions. They show the typical SL continuum, emission from excited OH radicals (in particular, OH (A 2  + -X 2 ) transition around 310 nm) and a very intense Na * line accompanied by a blue satellite corresponding to Na-Ar * exciplex. [START_REF] Lepoint-Mullie | Evidence for the emission of 'alkali-metal-noble-gas' van der Waals molecules from cavitation bubbles[END_REF] To increase light input inside the spectrophotometer, its slit was opened to 0.25 mm, which decreases the spectral resolution. Therefore, Na * doublet is not distinguished, neither the peak broadening due to interaction with the bubble content (broadening that was studied in details elsewhere [START_REF] Nakajima | Mechanism of two types of Na emission observed in sonoluminescence[END_REF] ). Similar spectra were obtained under He (Fig. 4SI), though with a lower total intensity and a less intense Na * peak relatively to the continuum. The lower SL intensity under He is well documented in the literature [START_REF] Young | Sonoluminescence[END_REF] and has been attributed to several factors: He lower solubility [START_REF] Hickling | Effects of Thermal Conduction in Sonoluminescence[END_REF] leading to fewer active bubbles, higher thermal conductivity [START_REF] Hickling | Effects of Thermal Conduction in Sonoluminescence[END_REF] and higher ionization potential. [START_REF] Yasui | Segregation of vapor and gas in a sonoluminescing bubble[END_REF] As for the intensity of Na * peak relatively to the continuum, it will be discussed below.

Sonoluminescence spectra

Three parameters were quantified in these spectra: the continuum intensity, measured at 500 nm, the intensity of OH (A 2  + -X 2 ) emission, defined as the difference between OH peak intensity at its maximum and continuum intensity at 360 nm, and the intensity of Na* emission, taken at 590 nm after subtraction of a linear baseline between 500 and 700 nm. Their evolution with NaCl concentration is presented in Fig. 5 Under Ar flow, these intensities show a fast increase up to approx. 1-2 M, then a slower increase between 2 and 4 M. They finally decrease above 4 M. Similarly, under an air flow and at 515 kHz, Wall et al. [START_REF] Wall | Multibubble sonoluminescence in aqueous salt solutions[END_REF] observed an increase of the SL continuum intensity up to 3 M NaCl, then a sudden decrease. Under He flow, while the intensities of SL continuum and of Na * line show a fast linear increase, that of OH emission is less pronounced. This fact may be explained by a change in bubble content: indeed, He solubility decreases with NaCl concentration, leading to a lower He concentration in bubbles. Moreover, according to Storey and Szeri, [START_REF] Storey | A reduced model of cavitation physics for use in sonochemistry[END_REF] a decrease in He content should lead to a decrease in water vapor mobility and an increase in the amount of water vapor trapped. The resulting change in composition must affect the conditions reached at collapse (on the one hand, more water molecules provide more material to produce OH radicals, on the other hand these molecules will cushion collapse and limit the energy concentration), conditions that control the extent of excited OH radical formation. The resulting effect is a slower increase in OH emission intensity.

The presence of a high salt concentration also decreases water vapour pressure, which would favor a higher SL intensity. However, this decrease is only about 10% between 0 & 3 M NaCl, i.e., equivalent to a decrease caused by a 2°C change in the bulk temperature. The latter leading to an increase in SL intensity of less than 20%, [START_REF] Young | Sonoluminescence[END_REF] it cannot account for the very large increase observed here.

It is usually considered that SL would follow the same trend as sonochemical activity. Though, it is observed here that upon addition of NaCl, the yields of H2 and H2O2 decrease, while the SL intensity increases. This can however be easily explained if one considers the different bubble populations that coexist. Of all chemically active bubbles, only some emit SL. [START_REF] Ashokkumar | Spatial Distribution of Acoustic Cavitation Bubbles at Different Ultrasound Frequencies[END_REF][START_REF] Kanthale | Sonoluminescence, sonochemistry (H2O2 yield) and bubble dynamics: Frequency and power effects[END_REF] Besides, among the former, only some are subject to jetting and solution droplet injection that leads to chemical reactions (2-9). Interestingly, one work [START_REF] Pflieger | Inverse effects of the gas feed positioning on sonochemistry and sonoluminescence[END_REF] already reported changes in experimental conditions leading to opposite trends in sonochemical activity and SL intensity. Thus, some changes in the experimental conditions, such as the salt concentration and subsequently the amount of dissolved gas, can lead to a global decrease in the active bubble population but at the same time lead to conditions more favorable to SL (e.g. less perturbations allowing a more symmetrical collapse), which strongly increases the proportion of SL bubbles within the active bubble population. Fig. 6 exemplifies the evolution of the SL continuum intensity (normalized to water to allow an easier comparison) with the gas solubility for Ar, He and air (for the latter, literature 11 values were taken). For all three gases, the lower the gas solubility (except at the lowest solubilities for air), the higher the SL continuum intensity, but the slope obviously depends on the gas nature and is much larger for He for which the intensity of the SL continuum increases by almost a factor 7 between water and NaCl 5 M. Other factors that do enhance the increase in the SL continuum intensity are the decreasing water vapor pressure and the increase in the acoustic pressure felt by the sonoluminescing bubbles (due to the lower attenuation of the acoustic wave). Fig. 6: Evolution of the SL continuum intensity (normalized to water) with the gas solubility for Ar, He and air (for air: literature values [START_REF] Wall | Multibubble sonoluminescence in aqueous salt solutions[END_REF] ).

Let us now consider jetting Na * -emitting bubbles. To quantify the efficiency of excitation, the intensity of Na * emission is normalized by NaCl concentration. As exemplified in Fig. 7, the observed trend depends on the rare gas nature. The comparison cannot be performed with air, since in this case no Na * emission is seen. Independent of the NaCl concentration, higher values of the intensity of Na * emission normalized by NaCl concentration are obtained for Ar, and correlate with the stronger decreases in H2 and H2O2 yields in NaCl solutions under Ar. These higher values under Ar cannot be explained by the higher solubility of Ar: indeed He solubility in NaCl 0.5 M is close to Ar solubility in NaCl 5 M, while the corresponding intensities of Na * emission normalized by NaCl concentration are very different. A plausible explanation is the larger bubble radius under Ar, 2 particularly for [NaCl] ≤ 2 M, larger size that may favor jetting and droplet injection. Direct observations with a high-speed camera would be needed to confirm this assumption but are pretty hard at high US frequency. Also, one may expect a higher probability of Na * formation for each bubble in the Ar case.

Another striking difference in Fig. 7 is that Na * emission intensity is proportional to NaCl concentration under He flow (which is the a priori expected behavior under the assumption of a constant jetting bubble population) but clearly not under Ar. This means that the number of jetting bubbles is approximately constant under He (in spite of He decreasing solubility) but not under Ar. In the Ar case, the higher the salt concentration is, the lower the relative Na * emission and this decrease is larger than would be expected from the evolution of the gas solubility (Fig. 7). This difference in the evolution of Na * emission normalized intensity between Ar and He correlates with the observed trends in H2 yields (Fig. 1): while it is roughly linear in He, it shows an inflexion in Ar for 2 M NaCl, i.e. at the concentration for which an inflexion point is observed in the intensity of Na* relative emission. This correlation between the evolutions of Na* relative emission intensity and of H2 yield decrease confirms the role of chemical reactions 2-8 in the quenching of H and H2.

Another hypothesis to investigate to explain the decrease in the relative Na * emission under Ar would be quenching of Na * excited state. The most probable way of quenching Na * would be to oxidize it back to Na + , for instance by reaction with OH radicals. To test this

Ar gas solubility

Ar gas solubility (mol/kg) possibility, SL spectra of a 1 M NaCl solution were measured under a gaseous mixture of Ar and 20% O2, and under Ar in the presence of 0.1 M H2O2 in solution. In the first case, oxidants are present mostly inside bubbles, in the latter in solution. As shown in Fig. 7, in both cases the intensity of Na* emission is hardly changed compared to the reference Ar case, meaning that no pronounced quenching occurs. The main differences observed are a large increase of the SL continuum intensity in the presence of O2, in full agreement with the literature [START_REF] Pflieger | Effect of operational conditions on sonoluminescence and kinetics of H2O2 formation during the sonolysis of water in the presence of Ar/O2 gas mixture[END_REF] , and absorption by H2O2 of the UV part of the SL spectrum. This comparison confirms that the decrease in the relative Na emission under Ar is most probably caused by a decrease in the number of Na * -emitting bubbles (i.e. jetting bubbles). Fig. 8: SL spectra of a 1 M NaCl solution under Ar, Ar-20% O2 and Ar after addition of 0.1M H2O2.

The spectra in Fig. 8 are also in agreement with the mechanism of formation of Na * by homolytical dissociation of NaCl proposed by Lepoint-Mullie et al. [START_REF] Lepoint-Mullie | Evidence for the emission of 'alkali-metal-noble-gas' van der Waals molecules from cavitation bubbles[END_REF] Indeed, this mechanism does not imply direct reduction of Na + so that formation of Na * is not sensitive to the presence of oxidants.

To further investigate the effect of salt for both rare gases, it is interesting to look at the relative evolutions of the different populations of bubbles: the bubbles emitting SL continuum and the jetting Na * -line emitting bubbles. In this view, Fig. 9 presents Na line intensity normalized by NaCl concentration, relative to the continuum intensity. Two striking features appear in Fig. 9. First, for NaCl concentrations ≤ 2 M this ratio takes higher values under Ar, in agreement with the literature, [START_REF] Cairos | Effects of Argon Sparging Rate, Ultrasonic Power, and Frequency on Multibubble Sonoluminescence Spectra and Bubble Dynamics in NaCl Aqueous Solutions[END_REF] indicating that for a given SL continuum emission, more bubbles are producing droplet injection and Na * emission under Ar. This fact may be explained by Ar higher solubility that leads to a higher number of bubbles and therefore more interactions between them, and to the bigger bubble radius under Ar, 2 that may favor jetting and droplet injection. Differences in bubble dynamics may also be conceivable, as was observed for instance between Ar and air: [START_REF] Cairos | Effects of Argon Sparging Rate, Ultrasonic Power, and Frequency on Multibubble Sonoluminescence Spectra and Bubble Dynamics in NaCl Aqueous Solutions[END_REF] larger Ar bubbles left a trace of smaller bubbles behind, which was not observed in air. This different behavior may account for the absence of Na * emission in air. [START_REF] Cairos | Effects of Argon Sparging Rate, Ultrasonic Power, and Frequency on Multibubble Sonoluminescence Spectra and Bubble Dynamics in NaCl Aqueous Solutions[END_REF] Second, this ratio strongly decreases upon addition of salt, with a more pronounced decrease under Ar, leading to similar values for both gases at concentrations ≥ 2 M.

Visualization: photographs of SL

Table 1 presents photographs of SL taken through the window of the reactor during the sonication at 362 kHz of NaCl aqueous solutions of different concentrations, under continuous Ar or He flow. Optics were focusing onto the central zone of the reactor, below the sonotrode. The tip of the latter can be distinguished in the last two rows of the table. In some pictures, a big bubble is present at the top. It has however no impact on cavitation and on observations made since it is stuck in the window holder, not in the zone of interest.

Table 1: Photographs of SL of NaCl solutions of different concentrations, under continuous Ar or He flow. Under He for water and 1 M NaCl, the light was too dim for a proper picture to be taken. The "shadow" sometimes present on the window is due to water condensation. A clear spatial separation can be observed between SL continuum (blue) and Na line (orange) emissions, in agreement with observations reported in the literature. 13, 15a, 17 In general, these pictures illustrate the tendencies observed in SL spectra: a much less intense light under He (therefore longer acquisition times were used), a strong increase of both continuum and Na line intensities with NaCl concentration under He, while these increases end at 4 M NaCl under Ar. For 5 M NaCl under Ar the photography even appears less red than blue, corresponding to the stronger decrease in Na line intensity compared to the continuum intensity.

At low (0-1 M) salt concentration under Ar, a standing wave pattern is visible only locally, below the sonotrode tip, and due to reflection of the ultrasonic wave on it. It starts appearing in the whole cavitation zone at 2-3 M NaCl and gets better defined with further increase in the salt concentration. This evolution is due to the strong decrease in gas solubility with NaCl concentration, that leads to the formation of much less (big) degassing bubbles, that otherwise dampen the propagation of the acoustic wave. [START_REF] Lee | Spatial Distribution Enhancement of Sonoluminescence Activity by Altering Sonication and Solution Conditions[END_REF] For both gases, the formation of a standing wave correlates with the evolution of Na * emission intensity normalized by NaCl concentration (Fig. 7). For He, the solubility of which is always low, the standing wave is expected to be formed also at 0-3 M NaCl where no photos could be taken. For Ar on the contrary, the standing wave is seen only above 2 M NaCl: between 0-2 M, only a low percentage of standing wave is present, and very high relative intensities of Na emission are observed; while this percentage strongly increases (up to 2 M) the decrease in the relative emission of Na * is fast; above 2 M a strong standing wave is formed, much lower relative intensities of Na emission are observed and the decrease of the latter slows down. Thus the presence of a standing wave strongly penalizes droplet injection and Na * emission. This observation is in agreement with the reported (opposite) effects of a gas flow: a high Ar gas flow rate favors collisions, merging and splitting of bubbles, hence Na * emission.

On the other hand, the standing wave, getting stronger as the dissolved gas concentration decreases, favors symmetrical collapse of bubbles and therefore SL continuum emission, which contributes to the decrease in the normalized Na * line intensity relative to the continuum intensity (Fig. 9). The presence of a high NaCl concentration in aqueous solutions strongly affects the yields of sonochemistry and sonoluminescence, as summarized in Fig. 10. A decrease in H2 and H2O2 yields was confirmed, and attributed to the combination of physical and chemical effects: the decrease in gas solubility combined to an increase in solution viscosity leads to changes in the number of chemically active bubbles; besides, in bubbles undergoing asymmetric collapse, Na • and Cl • radicals can form and react with H and OH • radicals. The opposite trends in H2O2 yield obtained in some previous works could be explained by chemical reactions between the salt and the water sonolysis products or between the salt and iodide ions. This remark underlines the fact that even in very simple systems, care should be taken that H2O2 or H2 yields cannot always be considered as an accurate measurement of the sonochemical activity. Indeed, these species can further react, or H and OH • radicals can be scavenged before forming them. It was also shown that Na • and Cl • radicals can react with water molecules in the collapsing bubbles, and that the balance of all reactions of these radicals leads to an acidification of the solution. Interestingly, H + yield gets higher under He than Ar for concentrated salt solutions showing one of the very rare examples of a higher sonochemical activity under He.

Conclusion

A good correlation is obtained between the sonochemical activity and Na * emission in SL spectra arising from 'jetting' bubbles. Indeed, a stronger Na * emission is observed under Ar compared to He, concomitant to a stronger decrease in yields of sonolytical products. Moreover, it was shown that the formation of a standing wave (for concentrations above 2 M under Ar) is unfavorable for droplet injection and Na • reactions (resp. Na* emission).

In general, sonoluminescence continuum intensity increases with salt concentration, due to the decrease in gas solubility that leads to less coalescence, less damping of the ultrasonic wave and to the formation of a standing wave. All these effects favor an increase in the number of sonoluminescing bubbles, at least up to a certain point. Indeed, above 4 M in Ar, a decrease in SL continuum intensity is observed, concomitant with a decrease in Na * emission and with very low yields in sonolytical products. A similar decrease in the SL continuum intensity was also observed under air above 3 M. [START_REF] Wall | Multibubble sonoluminescence in aqueous salt solutions[END_REF] This sudden decrease might be related to the dissolved gas concentration getting too low to allow the formation of enough active bubbles, although the threshold depends on the dissolved gas nature and isn't even detected up to 5 M NaCl under He, the gas with the lowest solubility among the studied three. This property of He may be linked to its very high diffusion coefficient that would favor bubble formation and growth. 2b This work underlines that three levels are necessary to contempt salt effect: the impact it has in each bubble (e.g. due to chemical reactions and changes in water vapour pressure), on the bubble number (e.g. because of changes in the gas solubility), and on the different bubble populations (e.g. due to the formation of a standing wave). While the total number of chemically active bubbles decreases upon addition of salt, the proportion of SL bubbles strongly increases due to more favorable conditions. As for the number of Na * -emitting bubbles, it seems to be constant under He but strongly decreases under Ar. Thus, the very simple NaCl salt appears to have a complex effect on sonicated solutions.

Fig. 1 :

 1 Fig.1: Normalized yields of the sonolytical products H2 and H2O2 as a function of NaCl concentration (362 kHz, 43 W, 13°C). Yields in pure water were of 0.40 µmol/kJ for Ar and 0.09 µmol/kJ for He.

Fig. 2 :

 2 Fig. 2: Yields of H + formed during the sonolysis of NaCl solutions under Ar or He (362 kHz, 43 W, 13°C).

Fig. 4 :

 4 Fig. 4: Measured SL spectra under Ar flow, in water and in 1 M and 3 M NaCl aqueous solutions (362 kHz, 43 W, 10°C).

Fig. 5 :

 5 Fig. 5: Evolution with NaCl concentration of the SL continuum intensity, taken at 500 nm, of OH (A 2  + -X 2 ) transition emission intensity and of Na * line intensity, for Ar (a) and He (b).

Fig. 7 :

 7 Fig. 7: Evolution with NaCl concentration of Na * line emission relative to NaCl concentration (A.U.), under He and Ar flows (values corrected to consider the same spectral acquisition time of 60 s), and of Ar solubility (He solubility is given in Fig. 5SI).

Fig. 9 :

 9 Fig. 9: Evolution with NaCl concentration of Na * line intensity normalized by NaCl concentration, relative to the continuum intensity, taken at 500 nm, under He and Ar flows.

Fig.

  Fig. Schematic representation of the different impacts of salt on sonochemistry and sonoluminescence.

The financial support from the University of Melbourne (joint research program CNRS/University of Melbourne, PRC0988 UCAB) is gratefully acknowledged. Gilles Guillot (ICSM) is thanked for technical help in the experiments.


Shell Xu Hu

Sergey Zagoruyko
email: sergey.zagoruyko@inria.fr

Nikos Komodakis
email: nikos.komodakis@enpc.fr

Exploring Weight Symmetry in Deep Neural Networks

We propose to impose symmetry in neural network parameters to improve parameter usage and make use of dedicated convolution and matrix multiplication routines. Due to the significant reduction in the number of parameters as a result of the symmetry constraints, one would expect a dramatic drop in accuracy. Surprisingly, we show that this is not the case, and, depending on network size, symmetry can have little or no negative effect on network accuracy, especially in deep overparameterized networks. We propose several ways to impose local symmetry in recurrent and convolutional neural networks and show that our symmetry parameterizations satisfy universal approximation property for single hidden layer networks. We extensively evaluate these parameterizations on CIFAR, ImageNet and language modeling datasets, showing significant benefits from the use of symmetry. For instance, our ResNet-101 with channel-wise symmetry has almost 25% fewer parameters and only 0.2% accuracy loss on ImageNet. Code for our experiments is available at https://github.com/hushell/deep-symmetry.

Introduction

For a long time, neural networks had a capacity problem: making them have too many parameters for a limited amount of data would dramatically affect their generalization capabilities. Thus, several regularization techniques were developed, for example, early stopping [START_REF] Weigend | Predicting the future: A connectionist approach[END_REF] and L 2 regularization. The advent of batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], skip-connections [START_REF] Hochreiter | Long short-term memory[END_REF][START_REF] Srivastava | Training very deep networks[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF], and overall architecture search helped to mitigate this problem very recently, so that increasing capacity no longer hurts the accuracy, but even before that it was known that "unimportant" connections between neurons can be removed, resulting in a network with significantly fewer parameters and little or no drop in performance. Optimal brain damage [START_REF] Lecun | Optimal brain damage[END_REF] proposed a second order method to prune such connections. Soft weight sharing [START_REF] Nowlan | Simplifying neural networks by soft weight-sharing[END_REF][START_REF] Ullrich | Soft weight-sharing for neural network compression[END_REF] can be used to very effectively reduce the number of parameters in a trained network. [START_REF] Denil | Predicting parameters in deep learning[END_REF] showed that it is possible to learn part of filters and predict the rest. In fact, parameter sharing is one of the most important features of convolutional neural networks, where sharing is built into convolutional layer and was thoroughly explored. Recurrent neural networks also heavily rely on sharing parameters over multiple time steps. It is also the key element in training siamese and triplet networks.

More recently, trained on massive amounts of data, image recognition neural networks were quickly increasing in the number of parameters, starting from AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. Network-In-Network [START_REF] Lin | Network in network[END_REF] proposed to stack MLPs which share local receptive fields, removing the need of massive fully-connected layers, and reducing the number of parameters to achieve the same accuracy as AlexNet by several times. This technique was further adopted by [START_REF] Szegedy | Going deeper with convolutions[END_REF] in their Inception architectures. Downsampling and upsampling can also be seen as a way of parameter sharing in SqueezeNet [START_REF] Iandola | Squeezenet: Alexnet-level accuracy with 50x fewer parameters and <1mb model size[END_REF]. More recently, HighwayNetwork [START_REF] Srivastava | Training very deep networks[END_REF] and later ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] proposed to add skip-connections, which allowed training of very deep networks with improved accuracy. It was later shown by Wide ResNet [START_REF] Zagoruyko | Wide residual networks[END_REF] that the number of parameters was key to their accuracy, and depth was complimentary. Either very deep, or very wide residual networks could be trained with a massive number of parameters, without suffering from decreased accuracy. After that, parameter sharing exploration continued on ResNet architectures. [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF] proposed to share some portion of convolutional filters in each group of the residual block -for example, every second convolution, achieving relatively small performance loss. Not all parameter sharing approaches brought both performance improvement and reduction in parameters, and there appears to be some trade-off between parameter sharing and computational efficiency. As an extreme case of parameter sharing with significant performance loss, HyperNetworks [START_REF] Ha | Hypernetworks[END_REF], proposed to have another network to generate filters. Among methods improving computational efficiency, a good example is MobileNet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF], which suggested to reparameterize each 3 × 3 convolution as a pair of depth-wise convolution and 1 × 1 convolution, can also be seen as a way of sharing parameters. Their work was later extended to grouped 1 × 1 convolution by ShuffleNet [START_REF] Zhang | Shufflenet: An extremely efficient convolutional neural network for mobile devices[END_REF], which further reduced the complexity of the residual block.

Another approach to reducing the number of parameters is post-processing. When a trained network is modified, either with or without additional fine-tuning. A number of low-rank approximation by tensor decomposition approaches were proposed, such as the works of [START_REF] Jaderberg | Speeding up convolutional neural networks with low rank expansions[END_REF] and [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cp-decomposition[END_REF]. For example, [START_REF] Denton | Exploiting linear structure within convolutional networks for efficient evaluation[END_REF] used the low-rank decomposition of the weight matrices to reduce the effective number of parameters in the network. Such approaches tend to be less applicable as more parameter-effective architectures are being invented. A very effective way of reducing parameters in a trained network was proposed by [START_REF] Han | Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding[END_REF], with a reduction of parameters of dozens of times, achieved by pruning, weight sharing, quantization and compression. Weight sharing locations are determined by weight values in a trained network, similar in value neurons share the same weight. Common approach today is to train a massive network and reduce it later. This can even improve results over a single-time trained network [START_REF] Han | DSD: regularizing deep neural networks with dense-sparse-dense training flow[END_REF].

Recurrent neural networks are known to be significantly overparameterized as well. For example, Kim and Rush (2016) show that it is possible to reduce the size of neural machine translation model up to 13× with an insignificant BLEU score drop by doing teacher-student knowledge distillation. Several works [START_REF] Merity | Pointer sentinel mixture models[END_REF][START_REF] Inan | Tying word vectors and word classifiers: A loss framework for language modeling[END_REF]Kim et al., 2016) focus on reducing the number of parameters in language modeling tasks.

All of the above suggest that architectures and methods used for training deep neural networks are suboptimal, and could be significantly improved by organizing parameter sharing from the start. It would be interesting to learn it, and a few attempts were made, e.g. in DCT space with hashing [START_REF] Chen | Compressing convolutional neural networks in the frequency domain[END_REF] and FFT space [START_REF] Mathieu | Fast training of convolutional networks through FFTs[END_REF]. Also, automatic architecture search approaches, such as [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF], do not handle sharing.

Despite so much work on reducing neural network parameters, surprisingly, very little attention has been given to weight symmetry. To the best of our knowledge, it has not been used in the context of parameter sharing so far. We propose a number of possible approaches to enforce symmetry on weights from scratch, i.e. not using a pretrained network. Imposed correctly, such symmetry can significantly reduce the number of parameters and computational complexity by sacrificing little or no accuracy, both at train and test time. Post-processing methods such as deep compression could be applied later. Also, our parameterizations are simple to implement in any modern framework with automatic differentiation. Besides, specialized routines for symmetric matrix multiplication and convolution could be used in both training and testing [START_REF] Goto | High-performance implementation of the level-3 blas[END_REF][START_REF] Nath | Optimizing symmetric dense matrix-vector multiplication on gpus[END_REF][START_REF] Igual | Level-3 blas on a GPU : Picking the low hanging fruit[END_REF].

We believe that our findings are surprising and uncover interesting properties of deep neural networks, which could lead to further advances in understanding and efficiency. Our contributions are summarized below.

• We propose an effective use of symmetry for model compression for the first time;

• Explore various ways of imposing symmetry constraints;

• Experimentally show that symmetry can successfully be imposed in various architectures and datasets, with no or little loss in accuracy;

• We show that our symmetric parameterization is generic and can be applied to both image classification and natural language processing tasks;

• We provide theoretical proof that neural networks with symmetric parameterization are universal approximators.

Symmetric reparameterizations

In this section we introduce several ideas to impose axial symmetry on the weights of convolutional / fully-connected layers, which prunes out a large fraction of redundant parameters and potentially gains computational speed-up for both training and testing. Throughout this section we consider all operations applied to matrices, which can be easily extended to multidimensional tensors by block symmetry.

Motivation

Real symmetric matrices have only real eigenvalues, while rotation matrices whose rotation angles are positive have at least one complex eigenvalue. Thus, restricting the weight matrix to be symmetric for some layers is equivalent to forcing these layers to learn non-rotational transformations. In a deep neural network, this restriction, which can be considered as an inductive bias, should not hurt the overall performance, since different layers are encouraged to focus on particular transformations, and then a good representation of the input is built up by compositing all of them. On the other hand, introducing symmetric weights for network compression have multiple benefits:

• Symmetric matrix multiplications could deliver potential speedup over generic BLAS routines.

• Theoretical guarantee in terms of universal approximation (see Section 2.5).

• Training from scratch: it avoids additional fine tuning compared with post-processing methods [START_REF] Jaderberg | Speeding up convolutional neural networks with low rank expansions[END_REF][START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cp-decomposition[END_REF][START_REF] Han | Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding[END_REF].

• Easy to combine with other compression methods (e.g. with ShaResNet [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF]).

Soft constraints

We first try to see if it is possible to enforce symmetry constraints in a soft manner during training, which has the advantage that training procedure remains very similar to standard supervised training. We start by adding a penalty term to the training loss, which leads to a soft constraint on W :

L(W ) + ρ vec(W ) -vec(W ) p , (1)
where L(W ) is a task-specific loss with respect to W ; ρ is a hyper-parameter to control the slackness of the constraints W = W ; vec(W ) is an operator that vectorizes the matrix into a column vector. For the norm of the penalty term, we use either p = 2 or p = 1. However, L 1 -norm turns out to be slightly more effective given that it promotes sparsity, which would possibly result in a larger number of exactly symmetric weights. Due to the soft constraints, the resulting matrix is not guaranteed to be symmetric and so at test time we use only the upper triangular part.

Hard constraints

Alternatively, we can make use of a specific parameterization by a linear operator T : W → Ŵ for some layer in the neural network that explicitly encode symmetry, where W is the actual weight and Ŵ is the constructed weight for that layer. We are interested in the case where W has fewer elements than Ŵ , which enables a reduction in weights while keeping exactly the same network architecture as if it is fully-parameterized. Note that we choose T to be nonparametric, thus the only additional burden of training is to forward and backward propagate through T ; while in testing, since Ŵ is symmetric, we only need to store the upper triangular of the learned Ŵ , which saves almost half of the space.

To this end, we propose different instantiations of T (W ) in terms of different ways to construct symmetric matrices.

Triangular parameterization One of the simplest ways to impose hard symmetry constraint is to define linear operator T as a sum of the upper triangular matrix of W ∈ R N ×N , its transpose W and the diagonal vector v ∈ R N :

Ŵ = T (triu(W ), v) := diag(v) + triu(W ) + triu(W ) , (2)
where triu(W ) returns the upper triangular part of the matrix W . Note that we use the full matrix W in the above expression just to simplify the notation. In fact, the triangular parameterization stores only the upper triangular of W and the main diagonal v, while elements of the lower triangular will never be touched. Thus, the number of parameters in this parameterization is 1 2 N (N + 1), reduced by almost 2 times compared to full parameterization. Note that (triu(W ), v) should be initialized from the same distribution as if W is learned directly. Due to strong sharing, the gradient with respect to W will be twice higher in magnitude, so the learning rate needs to be adjusted accordingly.

Average parameterization Let us also consider a redundant, but more straightforward formulation of the reparameterization T , in which we keep N × N matrix W as the actual weight, and define T as a sum of W and its transpose W divided by two:

Ŵ = T (W ) := 1 2 (W + W ). (3)
Although this is a redundant parameterization to obtain a symmetric matrix, we wanted to explore its performance given that it is known that overparameterized networks are often easier to optimize / train (e.g., directly training compact networks is much more challenging compared to first training overparameterized ones and then properly pruning their parameters). In this case, the learning rate remains the same since the gradient has been scaled by definition. As in triangular parameterization, W is initialized from the same distribution as basic non-symmetric parameterization. The number of parameters is the same as non-symmetric parameterization at training time, but1 2 N (N + 1) at testing time due to the fact that Ŵ is symmetric.

Eigen parameterization In addition, we consider a more generic eigen parameterization, inspired by the fact that any symmetric matrix has an eigen decomposition. Given a matrix V ∈ R N ×R and a vector λ ∈ R R as actual weights, T is defined by

Ŵ = T (V , λ) := V diag(λ)V . (4)
Ideally, V has to be an orthogonal matrix, but we relax this constraint due to a heavy computation of performing projected stochastic gradient descent. We still initialize V and λ from an eigen decomposition of the initial full weight matrix. The number of parameters in such relaxed parameterization is N (R + 1) at training time, and 1 2 N (N + 1) at testing time. We empirically choose R = N/2 as it yields similar performance as the case of R = N .

Note that a similar approach was suggested by [START_REF] Denil | Predicting parameters in deep learning[END_REF], where Ŵ is parameterized by matrices U and V with columns of U forming a dictionary of basis functions.

LDL parameterization There is a close relationship between the eigen decomposition of a matrix and its LDL decomposition. Recall that the LDL decomposition factorizes a matrix as a product of an unit lower triangular matrix L (meaning that all elements on the diagonal are 1's), a diagonal matrix D and the transpose of L. The advantage of using LDL decomposition over eigen decomposition is that it is much easier to maintain a valid decomposition of Ŵ during training. We thus also consider a LDL parameterization with additional assumptions 1 on Ŵ . Specifically, the reparameterization T with actual weights L and D is given by

Ŵ = T (L, D) := LDL , (5)
where L, D are restricted to be unit lower triangular matrix and diagonal matrix respectively.

N-way symmetry parameterization Inspired by triangular parameterization, which can be viewed as an axial symmetry about the main diagonal, we consider a more general N-way parameterization with respect to multiple axes of symmetry, where N denotes the number of repeated parts. Thus, previously introduced parameterizations are 2-way symmetries.

Given V as the actual weight, which is considered in general to be smaller than the required weight matrix W , we create a symmetrized version of W by a composition of linear transformations

Ŵ = T 1 • T 2 • . . . • T k (V ), (6)
where T j (•) is a basic linear operator which does one of the following things: translation, reflection, rotation, tessellation etc. In fact, the form of N-way symmetry is quite flexible. We consider only the cases where symmetry leads to efficient computations in testing. In this work, we focus on two N-way parameterizations: blocking and triangulizing.

The details are listed as follows. • Blocking: Given V as a M √ N × M √ N matrix, N -way blocking can be obtained by a series of reflections (denoted by reflect axis (•)) to include mirrors.

• Triangulizing: Given V as an isosceles right triangle with area M 2 N , Ŵ is obtained by a series of reflections about different axes.

We demonstrate several examples of N-way symmetries in Figure 1.

Combining with other methods

It is not surprising that hard-constrained symmetry parameterization can be viewed as a special weight sharing method. Nevertheless, symmetry parameterization is capable of taking advantage of special matrix computation routines to speed up both training and testing, which is not the case for unstructured weight sharing methods such as duplicating randomly picked elements to form Ŵ .

Symmetry parameterization can also be complementary to other parameter reduction or weight sharing methods. For example, we force weight sharing not only between residual blocks within the same stage [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF] but also within the same residual block using symmetry parameterizations. We show in Table 3 that triangular parameterization can be combined with ResNeXt [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] and MobileNet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF], which have already been designed to take advantage of weight sharing. Besides, post-processing methods [START_REF] Han | Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding[END_REF] can be applied on the trained symmetric weights with fine tuning, thus further reducing the number of parameters. We show experiments on combing channel-wise symmetry with other parameter reduction methods (e.g. ShaResNet [START_REF] Boulch | Sharesnet: reducing residual network parameter number by sharing weights[END_REF]) in Section 4.1.5.

Theoretical analysis

As a motivation of applying symmetry from a theoretical point of view, we show that a symmetric feedforward neural network with one fully-connected hidden layer admits the universal approximation property. The analysis is based on the result of [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]. We briefly explain here the high level ideas behind the proof of the universal approximation property.

The building block of a feedforward neural network is the sigmoid thresholded linear function of the form

σ w (x) := σ( w, x ), (7)
where σ(t) is a sigmoidal function such that σ(t) → 1 as t → +∞ and σ(t) → 0 as t → -∞. Then, a feedforward neural network g α,W with a single hidden layer is a finite linear combination of {σ wi } i with w i the i-th row of W :

g m α,W (x) := m i=1 α i σ wi (x) = α, σ(W x) . (8)
Define the sets Σ := {σ w :

w ∈ R n } and G := {g m α,W : m ∈ N, W ∈ R m×n , α ∈ R m }.
It is easy to see that G = span(Σ). The main result of [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF] shows that, on the domain [0, 1] n , G is a dense subset of the set of continuous functions (denoted by C([0, 1] n )).

Now, consider the set

S := {g m α,W ∈ G : m = n, W = W }.
We show that S is still a dense subset of C([0, 1] n ). Equivalently, we formally state the result as follows.

Lemma 1 [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]). The function σ w (x) satisfies the property that if [START_REF]5% accuracy drop compared to unconstrained ResNet-50, and even smaller for ResNet[END_REF]1] n σ w (x)dµ(x) = 0 holds for all w ∈ R n , then µ = 0.

Lemma 2. For any subset S of C([0, 1] n ), such that Σ ⊆ S, it follows that S is a dense subset of C([0, 1] n ).

Proof. Assume to the contrary that S is not dense, which means S is a closed proper subspace of C([0, 1] n ). By Hahn-Banach theorem, let f ∈ C([0, 1] n ) \ S, there exists a bounded linear functional

L ∈ C * ([0, 1] n ) such that L(g) = 0 ∀g ∈ S and L(f ) = dist(f, g) > 0.
By Riesz representation theorem, for all

L ∈ C * ([0, 1] n ), there is a unique µ ∈ C([0, 1] n ), such that L(f ) = f, µ for all f ∈ C([0, 1] n
) and L = µ . Note that ∀w : σ w ∈ S by definition. Thus, L(σ w ) = σ w , µ = 0 for all w, which implies µ = 0 by Lemma 1, and further implies that L = 0: a contradiction. Hence, S is a dense subset of

C([0, 1] n ). Theorem 1. Given any f ∈ C([0, 1] n ) and > 0, there is a function g n α,W ∈ S satisfying |f (x) -g n α,W (x)| < for all x ∈ [0, 1] n .
Proof. For any symmetric matrix W , we have

W = U ΛU -1 ,
where U is an orthonormal matrix (let u i be its ith row) and Λ is a diagonal matrix with diagonal elements being eigenvalues. Note that, for all x ∈ R n , there exists y ∈ R n , such that x = U y. Hence, W x = U Λy. Then, S can be rewritten as

S = {g m α,U Λ ∈ G : m = n, u i , u j = 0 ∀i = j, 1, u i = 1 ∀i}.
Note that for a n × n orthogonal real matrix, the degree of freedom is n 2 -n(n-1)

2

≥ n given the orthogonality constraints. Without loss of generality, we assume that u i Λ ∈ R n is the row of U Λ with the full degree of freedom. Then, for any function σ w ∈ Σ, there exists g n α,U Λ ∈ G satisfying g n α,U Λ = σ w by setting w = u i Λ and choosing α i = 1, α j = 0 ∀j = i. This implies Σ ⊆ S ⊆ G, and the result follows immediately from Lemma 2.

Implementations of block symmetry

The proposed symmetry parameterization is a generic parameter reduction method, which can be easily adapted to various network architectures. For example, convolutional / linear layers in feedforward networks (such as VGG (Simonyan and Zisserman, 2015), ResNeXt [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF], MobileNet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF] etc.) and in recurrent networks (such as LSTM [START_REF] Hochreiter | Long short-term memory[END_REF], GRU [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF] etc.) can be symmetrized. In the case of grouped convolution in ResNeXt, block symmetry can be imposed on each group kernel: for instance, suppose that filters are of shape g × N × N × k × k, where g is the number of groups, we can impose symmetry on dimensions of N × N . However, symmetry is not directly applicable to DenseNet [START_REF] Huang | Densely connected convolutional networks[END_REF], as the number of filters grows with every layer. We conduct some experiments along this direction in Section 4.1.

Imposing symmetry in convolutional neural networks

We denote by (W , W ) the whole set of parameters of the convolutional neural network, where W is the subset of parameters to be symmetrized, and W is the subset of free parameters. To be more specific, we assume there are totally L convolutional layers being reparameterized to equip symmetric parameters. That is, W := {W l } L l=1 with W l satisfying certain symmetric properties: W l ∈ R No×Ni×K h ×Kw is constructed so that the number of input channels is equal to the number of outputs channels (i.e. N i = N o = N ) and the spatial domain is a square (i.e. K h = K w = K). We propose to impose symmetry on slices of W l , namely, on the slice W l i , which is a square matrix.

Depending on which direction to slice the tensor, we have channel-wise symmetry and spatial symmetry. In general, we can write W l := {W l i } i∈I . For channel-wise symmetry, W l i is a N × N symmetric matrix, and

I := (k h , k w ) | k h , k w ∈ {1, .
. . , K} ; For spatial symmetry, W l i is a K × K symmetric matrix, and

I := (k i , k o ) | k i , k o ∈ {1, .
. . , N } . Since these two symmetries are not exclusive, we can indeed impose both at the same time.

In theory, both channel-wise and spatial symmetries will reduce the freedom of layers, and their ability to approximate functions. Enforcing them to a shallow network can be problematic, since it may significantly reduce the expressive power of the network. Deep highway and residual networks, on the other hand, are more robust to symmetric weights, since they have many layers that are capable to make relatively small changes and iteratively improve the representation of the input [START_REF] Greff | Highway and residual networks learn unrolled iterative estimation[END_REF]. In addition, spatial symmetry can be further motivated by the success of scattering networks [START_REF] Bruna | Invariant scattering convolution networks[END_REF], whose filters are fixed as wavelets and constructed to enjoy certain symmetric properties.

We show in experiments that the proposed symmetries can be applied to several modern deep neural networks without suffering a significant drop in both training and testing accuracy.

Imposing symmetry in recurrent neural network

We chose perhaps the most popular RNN variant, LSTM, which is known to be overparameterized, to experiment with symmetry. In a LSTM cell, the weight matrices between the hidden unit and gates (input, forget, output) as well as the weight matrix between the hidden unit and itself are square, so it is immediately valid to apply aforementioned symmetry parameterizations. This reduces about 25% parameters from the standard LSTM. We show in Section 4.3 that the symmetrized LSTM works as well as the standard LSTM in language modeling.

Experiments

This section is composed as follows. We start with CIFAR experiments, where we first test various symmetry parameterizations on wide residual networks (WRN) by [START_REF] Zagoruyko | Wide residual networks[END_REF]. After determining which parameterizations work best, we determine in which layers symmetry can be applied. We then test it with WRN of different widths and depths to determine the best configuration in terms of parameter reduction, computational complexity and simplicity. We also apply the proposed symmetrization to other architectures, and show that the conclusions drawn from CIFAR are able to transfer to larger datasets (ImageNet-1K dataset). Finally, we apply symmetrization to language modeling tasks.

We emphasize that our goal here is not to show state-of-the-art accuracy, but to show that very simple symmetry constraints can be used to significantly reduce the number of parameters in various network architectures.

There are two common ResNet variants are considered as baselines in the following experiments: basic blocks and bottleneck blocks [START_REF] He | Deep residual learning for image recognition[END_REF]. Basic blocks have two 3×3 convolutional layers and a parallel residual connection; bottleneck block is a combination of 1 × 1, 3 × 3 and 1 × 1 convolutional layers. Similar to WRN, we refer to WRNn-k-blocktype as the network of depth n, width k (number of channels multiplier), and blocktype meaning either basic or bottleneck.

Code for all our experiments is available at https://github.com/hushell/deep-symmetry.

CIFAR experiments

The results of various experiments on CIFAR-10/100 datasets are presented below.

Symmetry parameterizations

We first compare various symmetry parameterizations on CIFAR-10 with WRN-16-1-bottleneck, which is a relatively small network enabling us to perform quick experiments. The median validation errors over 5 runs are reported in Table 1 as well as a comparison in terms of the number of parameters needed in training and testing. All symmetry parameterizations have certain drop in performance comparing to the baseline. We observe that channel-wise triangular / average parameterizations have the lowest drop. Eigen parameterization does not work well in this experiment, which is possibly a consequence of V is not forced to be an orthogonal matrix. On the other hand, LDL parameterization as an alternative attains a better result. We also test soft channel-wise L 1 -norm (see eq. 1), where the number of parameters remains the same in training, and reduced at test time by using upper triangular weights only. The slackness is controlled by the coefficient ρ. For a large ρ, the soft-constrained symmetrization is slightly better than hard-constrained symmetrizations. Spatial symmetry parameterizations do not work as well as channel-wise symmetry. This is expected since the size of spatial dimensions is much smaller compared with channel dimensions (i.e. N i = N o > k h = k w ). Thus, the constraints imposed on spatial dimensions are much harsher making the learning much more difficult.

Based on the aforementioned analysis, we choose triangular parameterization (in terms of validation accuracy, parameter reduction and simplicity both at train and test time) as the main method to conduct all experiments further in this section.

Wide Residual Networks with symmetry

In this section we compare symmetrized WRN to its wider and thinner counterparts, and discuss the choice of network architecture. We use the terms conv0, conv1 and conv2 to refer to the first, the second and the third convolutional layers respectively (basic blocks have only conv0 and conv1). For our experiments we choose the bottleneck and constrain the mid-bottleneck 3 × 3 conv1 convolution to be symmetric, keeping 1 × 1 conv0 and conv2 unconstrained. This choice is because the approximating power of the residual block is least reduced, as real eigenvalues of conv1 are rotated by the surrounding convolutions, resulting in overall rich parameterization.

We present results for WRN-40-1-bottleneck and WRN-40-2-bottleneck trained on CIFAR in Table 2. We also train thinner networks with the same number of parameters to compare with their symmetric variants. On both datasets symmetric parameterization compares favorably to both wider and thinner non-symmetric counterparts in terms of accuracy and number of parameters. We further illustrate this in Fig. 2a, where we show training and validation accuracy of WRN with respect to various depths and widths. WRN has a lower accuracy in shallower networks when the training accuracy does not reach 100%, that is, the network struggles to fit into training data. In such cases symmetry constraints damage both training and validation accuracy significantly. Here we also notice that symmetry constraints cause much smaller accuracy drop in networks which are able to fit into training data perfectly, having almost 100% training accuracy. We hereafter refer to such networks as overparameterized, and we should note that overparameterization should not be confused with overfitting, that is, overparameterized network do not suffer from poor generalization. (a) WRN of various depth and width with bottleneck blocks and triangular symmetry. Dash (solid) lines denote train (val) accuracy respectively (medians over 5 runs). Table 3: Triangular symmetry applied to various architectures on CIFAR. Triangular channel-wise symmetry is imposed on every second convolution in residual blocks in WRN and ResNeXt, and in all square convolutions in Mo-bileNet and VGG. Median test accuracy of 5 runs is reported.

Other architectures

In this section we show that triangular symmetry works as well on other architectures and larger networks. We pick a residual network variant, ResNeXt, which reduces the number of parameters and computational complexity in ResNet by using grouped 3 × 3 convolution in bottleneck block. For large networks we use WRN-28-10, both basic and bottleneck variants, and a simple feedforward VGG. We also apply symmetry to MobileNets, a popular architecture for mobile devices. Even though being feedforward, it compares favorably to smaller residual networks such as ResNet-18 and ResNet-34, with significant reduction in parameters needed to achieve the same accuracy.

Results are presented in Table 3. We put triangular symmetry constraint on the second layer in each residual block of WRN-basic-28-10, and on 3 × 3 convolutional layers in WRN-bottleneck-28-10. In both cases, there are no drops in accuracy, as expected in overparameterized networks which easily achieve 100% training accuracy. Triangular parameterization works well even with ResNeXt, which has much less parameters in 3 × 3 layers. That is also the case for VGG, which, in contrast to others, does not have 1 × 1 or depth-wise convolutions between layers with symmetry. We believe the fact that there is no big reduction in accuracy is due to the existence of redundant parameters. In MobileNet, we parameterize all square 1× layers, and observe relatively small drop in accuracy.

Overall, we observe that overparameterized networks can easily benefit from symmetry parameterizations in terms of the number of parameters and potential speedup from more efficient implementation. It is surprising to see that triangular parameterization works well even with ResNeXt and MobileNet, which have already been designed to enjoy weight sharing. Among all these experiments, VGG with triangular parameterization reduces almost half of the parameters (i.e. 9.2 million), yet the accuracy still remains almost the same.

Importance of batch normalization

One might notice that batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] could potentially be a symmetry-breaking component when combined with symmetric convolutional or linear layers, as it can be viewed as an affine transform of each feature plane, or a diagonal fully connected layer. We, however, successfully apply symmetric parameterization to networks without batch normalization.

To test the influence of batch normalization on symmetric parameterization, we trained Network-In-Network on CIFAR with and without batch normalization, convergence plots are presented on Fig. 3. As can be seen, the difference in accuracies is very similar in both cases. If batch normalization played a significant role in improving symmetric parameterization, Network-In-Network without batch normalization (left) and triangular symmetry would have a higher accuracy drop. We use learning rate of 0.1 to train the networks with batch normalization, and of 0.01 without. Also, for triangular parameterization without batch normalization we reduce learning rate on upper triangular part by 2 to compensate for gradient magnitude increase due to sharing.

Combining symmetric parameterization with ShaResNet

We discussed the possibility of combining symmetry parameterizations with other parameter reduction methods in Section 2.4. Here, we conduct an experiment that combines 2-way channelwise symmetry with ShaResNet Boulch (2017), which is a weight sharing method that all conv1 of residual blocks (bottleneck block) within a group (i.e. layers between two dimensionality reduction convolutions) share the same weights. The results are shown in Table 4. It can be seen that the combination further reduces the number of parameters, while the testing performance is only slightly affected. In particular, with the bottleneck architecture, triangular symmetry pluses ShaResNet reduce about 33% of parameters and the performance drop is less than 1%.

N-way symmetries

As discussed in Section 2.3, it is possible to push forward the triangular parameterization to a more general N-way triangular parameterization. In addition to triangulizing and blocking, we also consider a chunking implementation (a naive N-way weight sharing: given V as a M × M N matrix. We define by tile N × (V ) a transforming function to tile/copy V N times to construct Ŵ = tile N × (V )) as an example to show that carefully designed N-way symmetries yield better performance than a naive N-way weight sharing. Recall that standard triangular parameterization is equivalent to 2way triangulizing. In this section, we test our proposals including chunking, blocking and triangulizing to achieve N-way channelwise symmetry. For chunking, we examine the cases of N = 2, 4, . . . , 64. If the number of channels in

ImageNet experiments

In this section we present ImageNet results for networks with triangular symmetry and without, to check if our conclusions from CIFAR transfer to larger dataset. We start with results for relatively small networks, MobileNet and ResNet-18. In MobileNet we impose triangular symmetry on all square 1×1 convolutional layers. ResNet-18 has basic block architecture and doesn't have enough parameters to fit into training data well, so we impose symmetry on every second 3 × 3 convolutional layer in block. Both MobileNet and ResNet-18 are relatively shallow networks and have 28% less parameters with triangular symmetry, so, as expected, the drop in accuracy is significant, see Table 5. Still, constrained MobileNet has only 3M parameters and achieves almost the same accuracy with ResNet-18.

As for large networks, we trained ResNet-50 and ResNet-101 with bottleneck block architecture and triangular symmetry on all 3 × 3 layers. As on CIFAR, drop in accuracy is much smaller for these: a reduction of 23% parameters

Language modeling

We test symmetry on a common language modeling Penn-Tree-Bank [START_REF] Marcus | Building a large annotated corpus of english: The penn treebank[END_REF] dataset. Our experimental setup reflects that of [START_REF] Zaremba | Recurrent neural network regularization[END_REF] 2 . We use a network with 2-layer LSTM and dropout, and test out a medium size configuration with 650 neurons (dropout 0.5), and a larger model with 1500 neurons (dropout 0.65). We try to symmetrize weights corresponding to input and hidden and gates separately and altogether, as well for each gate separately. Surprisingly, we find that adding symmetry on all hidden gates does not hurt, and even obtain slightly lower perplexity for both medium and large models. The results are presented in Table 6. We use the average parameterization 1 2 (W + W ) as we notice it gives slightly better results. Also, there is no L 2 -regularization applied during training, so the original W weights converge to be almost symmetric. There is also a large variation in final validation perplexity, so we train each network 5 times with different random seed and report mean±std results for all models.

For a fair comparison we also trained thinner networks with 565 and 1300 hidden neurons for medium and large networks correspondingly, so that the number of parameters is approximately equal to those of hidden-symmetrized networks. Symmetrized versions of large networks compare favourably to these networks.

We also include experimental results on a larger wikitext-2 [START_REF] Merity | Pointer sentinel mixture models[END_REF] dataset in Table 6, which is about 2 times larger than Penn-Tree-Bank, as well experiments with symmetry on each gate separately. We apply averaging symmetry to hidden gates of LSTM and compare to thinner networks with comparable number of parameters. In this case symmetry slightly hurts perplexity of medium model, and more significantly large model. This might be due to different dropout regularization in the networks (we use the same dropout rates as in Penn-Tree-Bank3 , which may not be the best choices for wikitext-2).

Conclusions

In this paper, we have shown that, quite surprisingly, deep neural network weights can be successfully parameterized to be symmetric without suffering a significant loss in accuracy. The proposed symmetry parameterizations could lead to potentially significant improvements for a wide range of mobile applications in terms of computational efficiency (dedicated routines for symmetric convolutions and matrix multiplication can be applied) and storage efficiency (memory requirements for storing network weights are dramatically reduced).

For future work, it would be interesting to compare with other structural weight matrices (e.g., Zhao et

  Figure 1: N-way parameterizations. (a) Original 4 × 4 weight matrix. (b) 4-way blocking: V is the bottom-right block; Ŵ = reflect -(reflect | (V )). (c) 4-way triangulizing: V is the top triangle; Ŵ = reflect / (reflect \ (V )). (d) 8-way triangulizing: V is the top-left triangle; Ŵ = reflect / (reflect \ (reflect | (V ))).

  20.0M parameters, triangular symmetry (b) Convergence curves of top-1 (top lines) and top-5 (bottom lines) validation errors of ResNet-50 and its triangularly symmetrized variant on ImageNet.

Figure 2 :

 2 Figure 2: Image classification results for ResNet with symmetric filters.

Figure 3 :

 3 Figure 3: Combining symmetric parameterization without batch normalization (left) and with (right), Network-In-Network. Training accuracy in shown by dashed lines, validation -solid. Accuracy drop is similar in both cases.

Figure 4 :

 4 Figure 4: N-way sharing (chunking) v.s. N-way symmetries (blocking, triangulizing) on CIFAR-10 with WRN-16-1bottleneck. x-axis represents N. y-axis represents the accuracy.

Table 1 :

 1 Various parameterizations on CIFAR-10 with WRN-16-1-bottleneck. We show median error over 5 runs and the numbers of parameters used in training and testing.

	symmetry parameterization	#parameters train test	CIFAR-10
	baseline (non-symmetric)	0.219M 0.219M	8.49
	L 1 soft constraints	0.219M 0.172M	8.61
	channelwise-triangular	0.172M 0.172M	8.84
	channelwise-average	0.219M 0.172M	8.83
	channelwise-eigen	0.173M 0.173M	10.23
	channelwise-LDL	0.172M 0.172M	9.15
	spatial-average	0.219M 0.187M	9.70
	spatial&channelwise-average 0.219M 0.156M	10.20

Table 2 :

 2 CIFAR test error (median of 5 runs) of triangular channel-wise parameterization on WRN-40 with bottleneck layers. conv0 and conv1 refer to the first and the second convolutional layers respectively.

	base network	symmetry location width #params CIFAR-10 CIFAR-100
		none	1	0.59M	6.12	26.86
	WRN-40-1-bottleneck	none	0.875	0.45M	6.29	28.36
		conv1	1	0.45M	6.24	27.66
		none	2.0	2.34M	4.95	22.51
	WRN-40-2-bottleneck	none	1.75	1.79M	5.12	23.18
		conv1	2.0	1.76M	4.96	22.98

Table 4 :

 4 Combining ShaResNet Boulch (2017) with 2-way channelwise symmetry. Test errors (mean/std/median over 5 runs) are compared for different symmetry parameterizations on CIFAR-10 using WRN-16-1-bottleneck with symmetric conv1. is less than N, we simply set N equal to the number of channels. For blocking and triangulizing, it is non-intuitive to come up with N-way symmetry for N > 4, so we only test the cases of N = 4, 16 and N = 2, 4, 8 respectively.In our experiments, as shown in Fig.4, chunking causes the steepest linear decrease in validation accuracy. Triangular parameterizations work fine even in the case of 8-way symmetry, which reduces almost 3 8 percentage of parameters from 2-way symmetry while only suffer about 1% decrease in accuracy.

	network	share symmetry #params mean std median
		none	0.219M	8.40 0.24	8.49
		none	0.171M	8.64 0.15	8.63
	WRN-16-1-bottleneck	triangular 0.172M	8.76 0.19	8.84
		triangular 0.147M	9.49 0.34	9.36
		average	0.171M	9.35 0.23	9.25
	a convolutional layer

  al. (2017)) with computational benefits and understand what kinds of inductive bias are implied. It should also be noted that our universal approximation analysis holds only for approximating single univariate continuous functions. It remains an open question what theoretical guarantees a symmetric deep neural network can provide for more general multivariate functions.

If Ŵ is symmetric and it factorizes as Ŵ = LDU = U DL , then by uniqueness, it follows that Ŵ = LDL . However, Ŵ has a LU decomposition if Ŵ satisfies a particular rank condition studied by[START_REF] Okunev | Necessary and sufficient conditions for existence of the lu factorization of an arbitrary matrix[END_REF]. Thus, we in fact assumes Ŵ is better conditioned.

https://github.com/pytorch/examples/tree/master/word_language_model

Suggested by https://github.com/pytorch/examples/tree/master/word_language_model

Acknowledgements

We thank Alexander Khanin and Soumik Sinharoy for providing us dedicated deep learning resources, without which this work would not be possible. S. Zagoruyko was also supported by the DGA RAPID project DRAAF.