Christophe Deroulers

Distribution of the resistance of nanowires with strong impurities

Keywords: numbers: 73.63.-b, 73.23.Hk, 72.20.Ee

Motivated by recent experiments on nanowires and carbon nanotubes, we study theoretically the effect of strong, point-like impurities on the linear electrical resistance R of finite length quantum wires. Charge transport is limited by Coulomb blockade and cotunneling. ln R is slowly self-averaging and non Gaussian. Its distribution is Gumbel with finite-size corrections which we compute. At low temperature, the distribution is similar to the variable range hopping (VRH) behaviour found long ago in doped semiconductors. We show that a result by Raikh and Ruzin does not apply. The finite-size corrections decay with the length L like 1/ ln L. At higher temperatures, this regime is replaced by new laws and the shape of the finite-size corrections changes strongly: if the electrons interact weakly, the corrections vanish already for wires with a few tens impurities.

Understanding the charge transport in materials with impurities has been a long standing problem which attracts new interest because of recent experiments on systems of nanoscopic size [1]. In the past, the case of doped semiconductors has been extensively studied [START_REF] Shklovskii | Electronic Properties of Doped Semiconductors[END_REF]. Because of impurities, the electrons are, in equilibrium, in localized states. These states are randomly spread in energy and space. At low temperature, conduction results of a sequence of thermally activated hops of the charge carriers, which tunnel from one state to another, borrowing the energy difference from the phonons bath if needed. The spatial range of these hops results from a balance between tunneling (favoring short ranges) and activation (finding a state close to the Fermi level is more likely if the range is long). As a consequence, the low-field resistance R of amorphous semiconductors follow Mott's VRH law, ln R ∼ T -1/(D+1) in dimension D, with peculiarities in the special case of 1D [START_REF] Kurkijärvi | [END_REF]4]. In presence of long-range Coulomb repulsion, as for crystalline semiconductors, the states close to the Fermi surface are rare and this law must be replaced at very low temperatures by the Shklovskii-Efros VRH law, ln R ∼ T -1/2 [START_REF] Shklovskii | Electronic Properties of Doped Semiconductors[END_REF].

Here we deal with 1D systems of nanoscopic crosssections (nanowires) with point-like, strong impurities, and repulsive interactions between the electrons, at low applied electric field. In a pure nanowire, or between two impurities, the physics is described by the Tomonaga-Luttinger liquids (LL) model rather than a Fermi liquid because, in a 1D fermionic system, the excitations have a bosonic character. The case of a few strong impurities, or of a uniform density of weak impurities, have been studied [5,[START_REF] Giamarchi | Quantum physics in one dimension[END_REF]. R was found to behave like powers of T at least in some regimes, with exponents which depend on the interaction parameter K of the LL. As was recently shown [START_REF] Fogler | [END_REF], in the case of many strong impurities, Coulomb blockade [1] and statistical effects yield different laws: at low temperature, a VRH-like regime is predicted, while at medium temperature a new, more complicated law (which can, on a small range of T , look like a power law) should replace it.

In this Letter, we first confirm numerically the existence of these two regimes for the average of ln R. But they might be hard to distinguish due to huge sample to sample fluctuations. Therefore, we study the full distribution of ln R. R and ln R are very slowly self-averaging and non Gaussian. Unless the wires are very long, the distribution of ln R is Gumbel with corrections that decay like 1/ ln L at low T . They are much weaker at medium T and have a different shape. We think this will help uncover experimentally the two regimes. Finally, we discuss the ergodicity hypothesis often used in experimental studies.

Our plan is the following: after introducing the model and the simulation technique, we show numerically that each sample has a high T regime where its resistance is proportional to the length and a "non extensive" regime where the voltage drop occurs essentially on a few breaks. We check the validity of a percolation-like approximation to R. Then we give our conclusions for the average of ln R and derive the distribution of ln R.

Model. Our model is similar to the one of Refs. [8,9]. We consider spinless electrons. Their interactions are short range because of screening by a nearby gate, which gives a finite capacitance per unit length C to the nanowires. Each wire is cut into a series of weakly coupled quantum dots by N impurities. They are placed randomly according to a Poisson process with mean spacing l large compared to the Fermi wavelength λ F . In the quasiclassical limit, which is relevant because the impurities are strong, each dot j can only have an integer number, q j , of electrons. Thus the charge transport proceeds by discrete hops, like in disordered semiconductors, and is limited by activation and tunneling. But unlike the localized states in the VRH model, here the energies of the states are correlated with their positions. Indeed, in the ground state, q j is the closest integer q (0) j to the mean charge Q j = l j /λ F + µ/∆ j , where µ is the chemical potential and ∆ j = 1/(Cl j ) is the charg-ing energy (we use e = k B = 1). The energy of dot j is E j (q j ) = (q j -Q j ) 2 ∆ j /2 [START_REF] Giamarchi | Quantum physics in one dimension[END_REF]. The typical charging energy ∆ := 1/(lC) is the natural energy scale of the problem. In addition, each charged excitation q j = q (0) j may itself be neutrally excitated by an energy difference ∼ K∆ j .

Having in mind experiments where a four-point measurement is performed [1], we do not pay attention to the contacts between the wire and the two leads. We assume for convenience that each impurity has the bare transparency 1/(eπ) ≈ 0.12. According to [9,11,12], because of the LL effects, its effective tunneling transparency may be found by an instanton calculation, yielding exp(-s j,j+1 ) with

s j,j+1 = 1 + (K -1 -1)(σ j-1 + σ j ) with σ j = ln[K/(Cλ F )/ max(K∆ j , T )].
The tunneling process is limited by neutral excitations at low T and by thermal fluctuations at high T . This extends readily to the probability exp(-s j,k ) of the cotunneling of k -j electrons, each through one of the impurities j, . . . , k -1 (which is formally equivalent to the hop of one electron from the dot j to the dot k). At low temperatures, cotunneling allows to shortcut the narrow dots with high charging energy -this is why ln R increases slower than 1/T as T goes to 0.

Putting things together, we find the probability that an electron hops from dot j at x j with initial charge q j to dot k at x k with initial charge q k under field F :

w j,qj ,k,q k = e -s j,k γ 0 E [exp(E/k B T ) -1] -1 (1)
where

E = E j (q j -1) -E j (q j ) + E k (q k + 1) -E k (q k ) - eF (x k -x j ).
The energy difference E ≷ 0 is provided/absorbed by the phonons bath; the field F favors one hopping direction. γ 0 depends on the coupling with phonons [13].

In a steady regime, the average occupation probability of each state should be constant, therefore we write down for each state a balance equation between incoming and outgoing charge flows. Solving this in a general out-of-equilibrium setting is difficult, but, following the idea of Miller and Abrahams [13], we restrict to the small F regime and simplify these equations to the first order in F . The resulting equations are the Kirchhoff's laws for a resistor network between nodes labeled by j, q j . In the following we are concerned with the low temperature regime T 10∆ and we assume that only the first two charged excitations on each dot play a role. If this may not be true for all dots, it will certainly be the case for the dots which contribute the most to the resistance by their high charging energy. Moreover, we further simplify [START_REF] Fogler | [END_REF] to a network with one node per dot j and resistances R j,k equal to the minimum over q j and q k of the resistances between the states j, q j and k, q k . In this network, each node is connected to all others, although connections between remote dots are likely to be very resistive.

Simulations. We performed simulations by drawing at random the positions of the impurities of 10000 wires for each value of K, impurities number N and mean spacing l. The low field resistance R of each wire is computed as the resistance of the equivalent network defined above; more precisely, we took care to manipulate ln R rather than R to prevent overflows at low T where R become huge; arbitrary precision arithmetics was used when needed to extract subdominant values. Simple linear solving of the Kirchhoff's equations fails at low temperatures because of numerical instabilities, but repeatedly eliminating one node of the network by applying a generalized star-triangle transformation [14] until only two nodes are left is efficient and stable.

For each wire, we also computed the resistance of the "best path", R bp , which is the less resistive of the subnetworks without loops connecting the nodes 0 and N +1 and going always in the same direction, in analogy with the percolation approach to VRH [START_REF] Shklovskii | Electronic Properties of Doped Semiconductors[END_REF]. A link in the best path is termed a hop. We found that ln R bp is a very good approximation of ln R up to T ∼ 10∆. Clearly, R bp > R because the full network has derivations around the best path. Empirically, ln R bp / ln R < 1.01 always, and this ratio goes to 1 as T decreases. On our curves the averages of ln R bp and ln R can't be distinguished. In the following we use R bp for our analytical discussion because it is a sum of individual resistances.

From the simulations we conclude first that each sample has two well separated regimes: at high T the voltage drop is almost uniform and the resistance R bp ≈ R is extensive (proportional to the length), but at low T the voltage drops mainly in one or a few regions, the breaks (Fig. 1), which makes R non extensive unless the wire is extremely long [15]. The positions of the breaks strongly depend on T and their width increases as T goes to zero and cotunneling through more and more impurities takes place. Fig. 1 shows this for one particular sample but it is true for all but pathological samples and for all values of K. The very low temperature regime T ≤ 0.01∆ is cut into sample-specific ranges where R(T ) is activated, as was observed experimentally [1]: in each range of temperature, a different break fixes the voltage drop.

The second conclusion is a confirmation of the results of [START_REF] Fogler | [END_REF] for the average over many samples ln R : a VRHlike law ln R ∼ 1/ √ T holds at low T but has to be replaced by a new regime at intermediate T where 1/ ln R is a affine in ln T (Fig. 2). However, unless the wire is very long, the temperature range where this new, non VRH-like law can be observed is small and it is in the cross-over between low and high temperatures (cotunneling and sequential tunneling), so that distinguishing the two laws is hard, all the more than the sample to sample fluctuations are large (Fig. 2). But we shall see that looking at the full distribution of ln R may help.

Analytical results for the distribution of ln R. From now on, we restrict to the low T regime, where breaks are observed. Taking for simplicity uniform s and λ F = 0, and neglecting correlations between hops, the probability FIG. 1: The gradient of the electrical potential V (x) along the best path in one random sample with N = 100 impurities, interaction parameter K = 0.3 and mean impurities spacing l = 10 3 λF . For each value of the temperature T , we apply a voltage unity to the wire and we compute the gradient V ′ of V on each hop along the best path (see text). Then we color this hop with a grey intensity depending on V ′ (the bar above the figure indicates the intensity coding: white is null gradient and black is the maximal gradient over the whole figure). Top inset: The sum of the squares of the voltage drop of each hop along the best path is 1 at low temperature (one hops dominates) and 1/N at high temperature (N hops with drop 1/N ). For each of 10 4 samples from the same distribution, we look for the temperature T * where it is equal to 1/ √ N . This separates the non-extensive and extensive R regimes (see text). Strong lines: median value of T * for several N values (solid: K = 0.1, dashed: K = 0.7). Dotted lines: boundaries of the 68% confidence interval on T * at a given N . Bottom inset: Plot of ln R w.r.t. 1/T . The curve is made of activated segments. In each one, the voltage drop occurs in one break which is identified by a letter on the main plot.

that a given hop on the best path has ln R < u is

P {R < e u } = 1- k j=1 f [ T ∆ (ln u-j s)] with k = ⌊u/s⌋ (2)
where [START_REF] Fogler | [END_REF]

f (x) := 1 -x [1 -exp(-1/x)].
The number n of hops in a sample is proportional to N (at low T , n ≈ sT /∆N ) and, because of the equivalence of statistical ensembles, we can replace n or N by L in our asymptotic estimations. The most resistive hop has typically a resistance R max such that P {R < R max } n ≈ 1/2. As long as R max is large compared to the average hop resistance times n [16], we are in the non extensive R regime, R bp ≈ R max -this regime extends at least to wires with several tens of thousands of impurities. If R bp /s 6, -ln P [R > e u ] is well approximated by an integral, which itself takes two forms [START_REF] Fogler | [END_REF]: (a) T u 2 /(s∆) if u T /∆ ≪ 1 and (b) (u/s) ln(u T /∆) if u T /∆ ≫ 1. The conditions R bp /s ≫ 1 and T /∆ ln R bp ≪ 1 are always fulfilled at (very) low temperatures, say T ≤ 0.001∆. Furthermore, we verified numerically that the cumulative distribution function (CDF) of ln R of the hops along the best path of the wires (with nonuniform s) is in very good agreement with the shape exp(-u 2 ) once rescaled. Then ln R ≈ ln R bp has for large n a Gumbel distribution because it is the maximum of n random variables with exponential decay of the probability density [START_REF] David | Order statistics[END_REF][START_REF] Huillet | [END_REF]. Its average behaves like s(ln n)∆/T and its standard deviation σ like a/2/ (ln n)T /(∆s) where a := π/ √ 6, which is similar to the results of Lee et al. for the VRH model [4] and agrees with the low T result of [START_REF] Fogler | [END_REF]. Both R and ln R are non gaussian but self-averaging. We prefer to study ln R because it is easier to sample numerically. We computed the CDF of the centered and reduced ln R with finite n corrections [19]: This expression is successfully compared to the numerical CDF of ln R in the inset of Fig. 3. Since 1/ ln n goes slowly to zero, the actual ln R of each sample will likely lie away from the average ln R even if n gets large. This is why activated segments are still visible on long wires. Pursuing the work of Lee et al., Raikh and Ruzin [15] evaluated the distribution of ln R in the case of 1D VRH. Interestingly, their result tends to a Gumbel law for very short wires or at very low T . But the finite-size effects are in 1/N 2 and do not match eq. (3), nor our numerical results (bottom inset of Fig. 3).

P { ln R -ln R σ < x} = exp[-g(x)]{1 - (3)
At not so low temperatures, say T > 0.01∆, the CDF of individual hops doesn't agree with form (a) above but, if interactions are not too strong, with form (b). The VRH-like law ln R ∼ 1/ √ T is replaced by ln R ≈ s ln n/ ln[(sT /∆) ln n] where the number of hops n is proportional to L as before. The difference between the averages in the two regimes is small compared to the fluctuations σ ∼ 1/ ln ln L. But a strong change in the finite-size effects of the CDF of ln R, which is Gumbellike, should be easy to detect: their amplitude is reduced by ln[(sT /∆) ln n] i.e. more than 3 in our data and their shape is changed. Fig. 3 shows that, for K = 0.7 and only N = 100 impurities one can't tell the difference between the CDF of ln R and a Gumbel CDF. For stronger interactions (smaller K), say K = 0.3, a typical break has only a few impurities, (b) is not valid, and the CDF is hard to characterize, but the deviations from a Gumbel law can't be confused with the formula (3) of the VRH case (top inset of Fig. 3) and might still be used as an evidence of strong impurities in a LL wire.

In an experimental study of the distribution of ln R, it is common to measure R for the same sample but several values of the gate potential: this changes µ, and the activation energies strongly depend on µ. This was argued to yield the same results as using many samples (see [20] and references therein). However, in addition to varying µ, we recommend to use as many samples as possible because we found numerically that this ergodicity hypothesis is only approximate and may lead to systematic errors (Fig. 3).

FIG. 2 :

 2 FIG. 2: Solid line: numerical results for the inverse of the average of ln R over 10 4 wires similar to the one of Fig.1as a function of T /∆. The error bars are smaller than the line's thickness. The thin, dotted lines above and below indicate the boundaries of the 68% confidence interval at each temperature. Strong short-dashed line: high T asymptote and upper bond, reached when the electrons tunnel sequentially through all impurities. Strong long-dashed line: low T asymptote where ln(R) follows the VRH law, as found from the inset. The thick straight line indicates the validity of the new law predicted in[START_REF] Fogler | [END_REF] in some range of T . Inset. Dots: plot of the average ln R w.r.t. ∆/T to show that, at low T , the VRH law is obeyed. The error bars are smaller than the symbols' size. Straight line: linear fit , which is used on the main plot as VRH asymptote.

g(x)[π 2

 2 /24(1 -x 2 ) + xζ(3)/(2p)]/ ln n + O[1/(ln n) 2 ]}where g(x) = exp(-γ -a x), ζ(3) ≈ 1.202 and γ ≈ 0.577.

FIG. 3 :

 3 FIG.3: Solid line: CDF of ln R for wires with N = 100 impurities, K = 0.7 and l/λF = 1000 at temperature T = 0.1∆ from our simulation. Dotted line: the same for one sample but µ varying uniformly in the rage -1..1 to show that the ergodic hypothesis is only approximate. Long/short dashed line: Gumbel/Gauss distributions having the same average and variance than this CDF. The Gumbel distribution is almost superimposed on the data. Insets. Solid line: finitesize effects for the same wires but K = 0.3 and at T = 0.1∆ resp. 0.001∆ (difference between the CDF of the centered and reduced ln R, and a Gumbel distribution). Long dashes: eq. (3) with n = 100. Dots: shape of the finite-size effects after Raikh and Ruzin's result.

* Electronic address: cde@thp.uni-koeln.de

Acknowledgments.

We acknowledge financial support from the SFB 608. It is a pleasure to thank M. Fogler, T. Nattermann and Z. Rácz for discussions and support.


Claire-Line Blanc

Julien Lemaire
email: julien.lemaire@centralesupelec.fr

Fanny Duval

Marc-André Théoleyre

Dominique Pareau Lgpm

Centralesupélec

Purification of pentoses from hemicellulosic hydrolysates without neutralization for sulfuric acid recovery

Keywords: pentoses, purification, acid recycling, electrodialysis, green chemistry. Highlights

come

Introduction

Chemical industry has shown a rapid growth in the 20th century thanks to petroleum as carbon source. Oil has indeed become the base for almost every product made by industrial chemistry. However this field is now facing a new challenge because of global warming mostly due to the accumulation of carbon dioxide (greenhouse gas) into the atmosphere.

Moreover oil is a fossil carbon resource whose world stocks are inevitably sinking while its price is strongly fluctuating. This is the reason why green chemistry is growing in many fields due to its advantages: carbon economy, more efficient syntheses, but alsonew renewable carbon sources.

Lignocellulosic biomass is considered as a very promising source of renewable carbon. A significant part is used by agro-industries, such as food-or forestry industries, generating large amounts of coproducts (wheat straw and bran, sugar beet pulp, wood waste …), which are often poorly valorized [START_REF] Hasunuma | A review of enzymes and microbes for lignocellulosic biorefinery and the possibility of their application to consolidated bioprocessing technology[END_REF]. Many chemical compounds and bio-based intermediates (sugars, paper pulp, surfactants, polymers or bioethanol) can however be produced from these lignocellulosic residues [START_REF] Cheng | Sugarcane bagasse hemicellulose hydrolysate for ethanol production by acid recovery process[END_REF][START_REF] Huang | A review of separation technologies in current and future biorefineries[END_REF].This valorization of every part of crop is the goal of the biorefinery concept.

Among those residues, bran is the major by-product of wheat milling [START_REF] Prückler | Wheat bran-based biorefinery 1: Composition of wheat bran and strategies of functionalization[END_REF]. Bran is produced in very large quantities but is mainly used as livestock feed. Wheat bran is mainly composed of hemicellulose (20-40%w), starch (10-20%w), protein (15-22%w),cellulose (9-15%w) and lignin (4-10%w) [START_REF] Maes | Alkaline Hydrogen Peroxide Extraction of Wheat Bran Non-starch Polysaccharides[END_REF][START_REF] Cripwell | Utilisation of wheat bran as a substrate for bioethanol production using recombinant cellulases and amylolytic yeast[END_REF][START_REF] Zeitoun | Procédés de fractionnement de la matière végétale: application à la production des polysaccharides du son et de la paille de blé[END_REF]. Arabinoxylans constitute the major part of wheat bran hemicellulose; therefore wheat bran is an attractive source of pentoseswhich have found growing applications, such as xylitol, surfactants and biopolymers productions [START_REF] Dumon | Progress and future prospects for pentose-specific biocatalysts in biorefining[END_REF].

Hydrolysis is necessary to extract monosaccharides from cellulose and hemicellulose. This chemical pretreatment is generally difficult and requires drastic conditions, such highly alkaline or acidic media or use of organic solvents, like ethanol [START_REF] Fitzpatrick | A biorefinery processing perspective: treatment of lignocellulosic materials for the production of value-added products[END_REF].A complex mixture of hexoses, pentoses, lignins and ashes is produced, needing further downstream separations. This work studies the purification of pentoses obtained by hydrolysis of wheat bran withdiluted sulfuric acid as a catalyst [START_REF] Fitzpatrick | A biorefinery processing perspective: treatment of lignocellulosic materials for the production of value-added products[END_REF][START_REF] Sarkar | Bioethanol production from agricultural wastes: An overview[END_REF].The classical separation methods involve ionexchange [START_REF] Asko | Process for making xylose[END_REF][START_REF] Ballerini | Procédé et unité de production en continu d'un mélange de sucres contenant au moins 80% de xylose à partir d'un substrat lignocellulosique[END_REF][START_REF] Ferreira | Process for the production of crystalline xylose from sugar cane bagasse, crystalline xylose obtained by said process, process for the production of xylitol from the said xylose and crystalline xylitol obtained thereby[END_REF], chromatography, adsorption or crystallization [START_REF] Heikkilä | Method for recovering xylose[END_REF][START_REF] Heikkilä | Method of producing xylose[END_REF][START_REF] Heikkilä | Process of producing xylose and dissolving pulp[END_REF], but need a partial or complete pre-neutralization of the inorganic acid. Those processes are not sustainable as they consumea lot of chemicalsand energy.Most notably large amounts of base are necessary to neutralize the acid catalyst, which is then lost for further hydrolysis. Besides, neutralization increases dramatically the amount of mineral salts which have to be removed in a further step. Electrodialysis (ED) was studied here as an alternative to neutralization, allowing direct acid recovery and reducing chemicals consumption. This process is a separation technique using an electrical potential as driving force to move ions through selective exchange membranes to separate them from uncharged species [START_REF] Vyas | Separation of inorganic and organic acids from glyoxal by electrodialysis[END_REF]. It is widely used for whey demineralization, in order to produce an ingredient for infant formula [START_REF] Batchelder | Electrodialysis method for demineralization of liquid, whey-based material[END_REF]. ED is also implemented in bioprocesses for organic acids purification, for example lactic acid [START_REF] Huang | Application of electrodialysis to the production of organic acids: State-of-the-art and recent developments[END_REF][START_REF] Lee | Recovery of lactic acid from fermentation broth by the two-stage process of nanofiltration and water-splitting electrodialysis[END_REF][START_REF] Lee | Lactic acid recovery using two-stage electrodialysis and its modelling[END_REF][START_REF] Lee | A feasibility study on the multistage process for the oxalic acid pretreatment of a lignocellulosic biomass using electrodialysis[END_REF].The major drawback of this technique is membrane clogging, leading to significant decreases in production. Indeed, during sulfuric acid extraction, pH rises and organic matters soluble at low pH (lignin, proteins…) precipitate [START_REF] Lee | Economic process for producing xylose from hydrolysate using electrodialysis and direct recovery method[END_REF]. This phenomenon strongly reducesboth process efficiency and membrane durability.Macromolecules must then be removed before ED and membrane separation seems appropriate for this purpose [START_REF] Lyu | Two-stage nanofiltration process for high-value chemical production from hydrolysates of lignocellulosic biomass through hydrothermal liquefaction[END_REF][START_REF] He | Recent advances in membrane technologies for biorefining and bioenergy production[END_REF].

A previous lab-scale study revealed the feasibility of sulfuric acid recovery from wheat bran hydrolysate using ultrafiltration followed by electrodialysis [START_REF] Lemaire | Purification of pentoses from hemicellulosic hydrolysates with sulfuric acid recovery by using electrodialysis[END_REF]. Operating conditions of both steps were optimized as well as the choice of ultrafiltration membranes. The retention of harmful macromolecules was effective only with organic membranes (10 to 50 kD), even with higher cut-off than ceramic membranes. Indeed, permeate flux and rejection rate are not only determined by pore size, but also by membrane nature, charge and hydrophobicity [START_REF] Liu | Application of inorganic membranes in the alkali recovery process[END_REF].

The present work aims atinvestigatingthe process feasibility at a larger scale.Indeed, it is necessary to estimate more accuratelyperformances (recovery rates and productivies) that could be obtained at the industrial scale. Moreover, the goal was to check the reliability and robustness of membranes for both processes. Furthermore, it was necessary to produce a significant amount of brine (15 g.L -1 H 2 SO 4 ) to carry on a new hydrolysis of wheat bran in order to verify the feasibility of acid recycling by our industrial partner.

Materials and methods

Wheat bran hydrolysate

The hydrolysate, which composition is given in table 1, was produced by an industrial partner by mixing destarched wheat bran with a sulfuric acid solution around 12 g.L -1 . The mixturehas been heated at 130 °C for 1 hour and roughly filtered after cooling.It was then splitted into three 23 L tanks. The hydrolysate contained suspended matter, which settled resulting in a roughly clear supernatant and a muddy tank bottom.When the withdrawn supernatant was not enough clear, a centrifugation was performed before further treatments to remove remaining suspended matter at 4000 g, 4 °C for 15 min.

Ultrafiltration

The filtration modulewas an Alfa Laval TestUnit M20, a crossflow membrane filtration unit equipped with a heat exchanger for temperature control, a recirculation pump and a 8 L stainless steel tank. A 2.5'' spiral organic membrane UFX10 pHt was furnished by Alfa Laval for ultrafiltration at a molecular weight cut-off of 10 kDa. This membrane was chosen owing to its retention performances and its specific resistance to very acid conditions (till pH 1). The total membrane area was 0.6 m 2 . The dead volume in the system was estimated to 2 L.

Before filtration, the water flux was measured at 40 °C with a transmembrane pressure of 5 bar, giving 340 L.h -1 .m -2 . This was the reference to evaluate membrane state after in situcleaning with an alkaline solution (Ultrasil 110, Ecolab). The permeate and retentate have been recirculated to the feed tank for 1 h (time necessary to heat the feed) before withdrawing permeate and concentrating retentate. A volume higher than the tank volume was filtered in one run, so the run was conducted in fed-batch mode (1 L permeate out, 1 L feed in) at 40 °C and a transmembrane pressure of 5 bar. Temperature, tank pH and conductivity and permeate flowrate were monitored.

Electrodialysis

The conventional electrodialysis plant was supplied by Eurodia. It was composed of three The electrolyte was a 20 mS.cm -1 Na 2 SO 4 solution (11 g.L -1 ). The initial brine was a 5 mS.cm -1 H 2 SO 4 solution (0.7 g.L -1 ). The operation was run in batch mode until the conductivity drop rate was lower than 0.25 mS.cm -1 .min -1 . Temperature, pH and conductivity were monitored.

This unit was optimized for demineralizationby the manufacturer. Operating conditions (flow rates, current, voltage, electrolyte and initial brine composition) were fixed according to manufacturer's recommendations. Conventional ED is performed with a constant tension so the limiting current density is not a constant and decreases during demineralization. Tension was fixed to work always under the limiting current density so as to optimize energy consumption and faradic yield.

During this operation, sulfuric acid was transferred from the feed to the brine through the electric field. Two successive batch operations were performed; acid enriched brine of the first batch was reused in the second batch in order to reachthe acid concentration required to recycleit for wheat branhydrolysis (10-15 g.L -1 ).

Polishing

The final product polishing was performed at lab-scale by ion-exchange and activated carbon because both techniques are commonly used and well understood at industrial scale.The goal was to check its final quality and to know in which extent the present process increased the capacity of ion-exchange resins compared to conventional method (neutralization with lime).

Firstly,the product (after ED) was further demineralized by ion-exchange to obtain a conductivity lower than 10 µS.cm -1 . Two vertical glass columns operating in series were filled respectively with 100 mL of a strong cationic resin (Lanxess S2528 H + form) and 100 mL of a weak anionic resin (Lanxess S4368 OH -form).The product was injected by a peristaltic pump at room temperature at 4.0 BV.h -1 (400 mL.h -1 ) until the breakthrough point of salts was reached. Its brix, pH, 420 nm absorbance and conductivity were monitored at the output.

Finally the demineralized productwasconcentrated at 40 °C under vacuum with a rotating evaporator (Büchi), thenpercolated through 50 mL activated carbon (Chemviron Carbon) by a peristaltic pump at 1.5 BV.h -1 (75 mL.h -1 ) at room temperature. Before use, carbon was dropped into boiling water and let for a night within the cooling water.

Analytic methods

Conductivity andpH were measured with a 340i conductimeter/pH-meter from WTW. Dry matter was estimated with a brixmeter (RFM732 Bellingham+Stanley). Sample color was evaluated by measuring absorbance at 420 nm with a spectrophotometer (UV-1800, Shimadzu).Total sample acidity was quantified by titration with 0.1 M NaOH solution.

Sugars and furfural were analysed with a HPLC system (Ultimate 3000, Dionex) equipped a RI-101 refractometer (Shodex). The analytical column was an Aminex HPX-87H (Biorad) and its temperature was maintained at 45 °C. The mobile phase was 2 mmol.L -1 H 2 SO 4

degassed by an on-line vacuum degasser. The flowrate was set to 0.5 mL.min -1 and the injection volume to 10 µL.

Anions and cations were analysed with anHPIC system (ICS-5000 + , Thermo Scientific Dionex) equippedwith two eluent generators, a column oven controller set to 35 °C and two suppressors (table 2). Detection is performed by conductivity. The last two tests (3 and 4) were performed to concentrate a complete tank volume (ca.

24 L) in one batch, achieving a higher VCF than the previous trials.The product was settled.The clear supernatant of each tank was recovered and the rest of the volume was centrifuged. Both liquid phases obtained after centrifugation were mixed to the clear supernatant to get a representative sample of a two phases industrial centrifugation. As seen on figure 1, the trends for both trials were rather similar, except for the initial flux values, and close to the UF2 curve. Indeed after UF3 and alkaline cleaning,it was not possible to reach the initial water flux of 300 L.h -1 .m -2 , but only 200 L.h -1 .m -2 .This highlights that membrane fouling was not totally reversible and some suspended matters was not removed by alkaline cleaning and progressively blocked membrane pores. However,the fouling by soluble macromolecules seems to be more reversible because the removal of suspended matter before ultrafiltration resulted in better and sustainable permeate flux.

The measured average fluxes were respectively of 6.4 L.h -1 .m -2 and 5.2 L.h -1 .m -2 for UF3 and UF4. Those values are probably too low for an industrial application, whereas fluxesobtained during UF1 are very interesting. So it is important to find a way to reach a feed quality equivalent to the UF1 supernatant. Centrifugation does not seem to be the right way as it segregates the liquid phase and produces light flocs. Consequently, before ultrafiltration, the hydrolysate has to be better filtered with membranes which largely reduce, even totally remove, suspended matters.

For the fourth ultrafiltration trial, analyses of the feed, the final retentate and the mean permeate are given in table 3.Due to the system dead volume, the initial retentate was slightly more diluted than the feed and its real volume was estimated to 25.2 L. So the final volumetric concentration factor was around 9.5.Only 88% of sugars (glucose, xylose and arabinose)were recovered in the permeate; such a sugars loss is not an option for the process viability. So a continuous diafiltration has to be performed after ultrafiltration to increase sugars recovery up to 99%. A 2.5 diafiltration volume should be enough to reach this aim, namely a water volume which corresponds to 2.5 fold the final retentate volume. As a conclusion it is possible to remove efficiently precipitating macromolecules from wheat bran hydrolysate by ultrafiltration with a 10 kDa organic membrane, without losing sugars. Hydrolysate treatment prior to ultrafiltration, for removal of suspended matter, has to be designed carefully as membrane clogging determines the permeate fluxes of ultrafiltration.The resulting permeate was afterwards demineralized by electrodialysis.

Electrodialysis

Electrodialysis was implemented to extract sulfuric acid from hydrolysatein order to recycle it to the hydrolysis step, provided the solution was concentrated enough in sulfuric acid (10-15 g/L). According to preliminary tests, when the acid enriched brine of the first batch was reused for a second batch, the second brine was suitable in terms of acid concentration.

So 4 L of ultrafiltered permeate (feed)were demineralized by conventional ED using an initial 2 L brine of 5 mS.cm -1 sulfuric acid. Around 22 L of permeate were treated in this way, producing 11 L of sulfuric acid solution. Results of four trial pairs are presented on figures 2 and 3.The first batches correspond to trials made with fresh feeds andfresh brines. In the second batchesfresh feeds and brines obtained after the first batches were used. The very good reproducibility proved that ion-exchange membrane performances remained quite constant. Consequently, biofouling is expected low and quite reversible. The ultrafiltration step must have removed the most part of macromolecules which can be responsible for biofouling, as well as macromolecules which can precipitate because of pH rise.

So as to keep productivity high enough, the product was always demineralized untilonly 2 mS.cm -1 (figure 2); its final pH was 2.7. Fromthis moment, the conductivity drop velocity became lower than 0.25 mS.cm -1 .min -1 . No precipitation was observed.The brine was enriched in acid from 5 to 40 mS.cm -1 (0.7 to 8.6 g.L -1 H 2 SO 4 ) during the first batches and up to 65 mS.cm -1 (14.7 g.L -1 H 2 SO 4 ) during the second batches.

Moreover, ED was performed until the product conductivity reached only 2 mS.cm -1 so as to recover most of acids and keep the faradic yield above 50%. In this case, the average faradic yield remained good (80%) and the specific energy consumption of the electrodialysis stack was quite interesting (ca.1.1 kWh per kg of H 2 SO 4 recovered and ca. 8.4 kWh per m 3 of hydrolysate). Better faradic yield was not expected because of high mobility of H + that are more difficult to keep in brine compartment than other cations.

It can be seen on figure 3 that the current intensity was also greatly reproducible. The initial intensity of the second batches was higher than that of the first batches due to a higherinitial brine conductivity. After 7 min, both intensities were equivalent due to similar product conductivities. Demineralized products of all the batches were gathered,as well as the final brines of the second batches. The sugars recovery rate in the products was very good (99%) (table 4); no sugar went through the ion exchange membranes into the brine. On the contrary 87% of the acid was recovered into the brine, reaching about 15 g.L -1 H 2 SO 4 .The resulting brine pH and concentration were appropriate to perform a new hydrolysis. The produced brine was then used to carry on a new hydrolysis by our industrial partner.

It was mixed with the required destarched wheat bran quantity and heated at 130 °C during 1 hour. No sulfuric acid was added.In order to avoid the centrifugation step, which had a dramatic impact on UF fluxes, the filtration method of suspended matter was improved. Its concentration was clearly reduced (table 5). Similar results must be got with an industrial operating press filter.

The composition of the hydrolysate obtained with this recycled brine was very close to the reference (table 5). Only salts concentrations were significantly higher in the case of the recycled brine. Indeed the salts were extracted by ED as well as H 2 SO 4 ; but as they are in minor concentrations, it must not be an inconvenient. However this accumulation has to be watched and might require an occasional purge. The feasibilityof recycling sulfuric acid was thendemonstrated. The whole process including suspended matter filtration, 10 kDa ultrafiltration with organic membrane, 2.5 diafiltration and conventional electrodialysis, seems to be reliable with a very good reproducibility. The recycled brine was successfully used for hydrolysis of destarched wheat bran.

Overall recovery ratesin the brine (sulfuric acid) and the product (sugars, furfural) were calculated and summed up in table 6.The sugars recovery rate was limited by the dead volume of the UF pilot device; a continuous operationandthe scale-up could easily improve the recovery rates, by limitinglosses due to the system dead volume. Sugars purity was of 84%; further purification steps would be necessary to reachhigherpurities.

Sugars recovery in product 94%

Furfural recovery in product 92%

Acid recovery in brine 79%

Sugars purity 84%

Polishing

After electrodialysis, the major impurities in the sugar solutionwere salts, furfural and organic elements (color); the remaining acid concentration was about 0.3 g/L. A polishing by ion-exchange and adsorption is necessary to remove salts and organic molecules. Previous electrodialysis step allowed to divide conductivity by fifteen and salts concentration by nine.

Thus ion-exchange capacity (expressed as treated volume per ion-exchange resin amount) was expected to be much higher thanobserved capacity with conventional purification methods including a first step of acid neutralization with soda or lime.

Figure 4 presents the evolution of the composition of demineralized fractions, at the output of the ion-exchange system,as a function of injected volume of electrodialyzed product (one bed volume (BV) corresponded to 100 mL). Up to15 BV of electrodialyzed hydrolysate were demineralized with a resulting conductivity lower than 10 µS.cm -1 during one batch. The product was also completely decolorized (420 nm absorbance = 0.00). Moreover the furfural concentration was reduced by 58%.As a comparison the same treatment was realized on a hydrolysate only neutralized with lime, as it is done in the conventional process. Only 2 BV weredemineralized before resin saturation. This clearly highlights the advantages of the present process, which greatly reduces chemicals and water consumption, in particular for resin regeneration. In order to eliminate remaining furfural, the product was treated after concentrationby activated carbon.The initial product dry weight(4.9%w) was increased by concentration up to 52.2%w, without losing sugars. A minor increase in coloration was also observed with 420 nm absorbance risingfrom 0.00 to 0.07. The product composition afterconcentration and activated carbon treatment was summed up in table 7. Furfural and color were totally removed. The sugars recovery rate for the global process, including rough filtration, ultrafiltration, 2.5 diafiltration, conventional electrodialysis and polishing, reached 90% with apurity close to 100%. Sugars can afterwards be separated classically by continuous chromatography or crystallization [START_REF] Heikkilä | Method for recovering xylose[END_REF][START_REF] Heikkilä | Method of producing xylose[END_REF][START_REF] Heikkilä | Process of producing xylose and dissolving pulp[END_REF].

Conclusion

This work is a significant contribution for the valorization of agricultural by-products as wheat bran. It presents a reliable process for the purification of hemicellulosic sugars, reducing chemicals and water consumption. Ultrafiltration is carried out with an organic membrane in a first time to remove precipitating macromolecules. Afterwards conventional electrodialysis allows the sulfuric acid recovery from hydrolysate in order to recycle it for the hydrolysis step. ED is a key step as it allows a doublechemical saving: less fresh sulfuric acid is needed for hydrolysis and no alkaline chemicals are consumed to neutralize acid and precipitate macromolecules. Thus the salts content is considerably decreased, facilitatingthe further demineralization by ion-exchange and also resulting in the reduction ofchemicals consumption. After activated carbon treatment,the product contains only sugars (glucose, xylose and arabinose). They can be separated from each other by continuous chromatography or crystallization.

The acid recovery rate in brine after electrodialysis is around 80%, whereas the overall sugars recovery rate after polishing is close to 90%. Both recovery rates could be increased by a scale-up operation or a continuous mode of the process, limiting losses due to the system dead volumes.

Attention has to be paid to the initial hydrolysate quality, as it influences greatly ultrafiltration fluxes; the improvement of suspended matter filtration method gave better and sustainable permeate flux (> 20 L.h -1 .m -2 )with the hydrolysate obtained with the reused brine.

Moreover during electrodialysis, not only sulfuric acid but also mineral salts are extracted into the brine; their initial concentrations are very low but their accumulation during recycling cycles may require an occasional purge.

2 L

 2 tanks containing the electrolyte, the product (ultrafiltered hydrolysate) and the brine (diluted sulfuric acid solution), three pumps for the circulation of each solution and a voltage generator. A conventional 10unit cells stack, with an alternation of anion and cation exchange membranes was used. The unit cell area was 0.02 m². It was operated at a constant voltage of 14 V and a free intensity, without temperature regulation. Pumps ensured a 180 L.h -1 flowrate in the brine and feed compartments and a 200 L.h -1 flowrate in the electrolyte compartment (supplier recommended values).

For

  the first trial (UF1) 14 L of the first tank supernatantwere introduced in the system resulting in 16 L of initial retentate because of the 2 L dead volume(centrifugation not required). It was concentrated till a volumetric concentration factor (VCF) 3.6. The initial flux was about 33 L.h -1 .m -2 and decreased to 16 L.h -1 .m -2 during the operation, as seen onfigure 1, resulting in an average flux of 27 L.h -1 .m -2 .Then the 9 L remaining of the first tank supernatanthad to be centrifuged because of the suspended matter amount. The pellet contained the major part of the suspended matter as a slurry. However, centrifugation splitted in two the liquid phase.The lighter phase also contained suspended matter in the form of light flocs. Nevertheless, both phases were filtered together and concentrated at VCF 2.7 during the second trial (UF2). A dramatic flux reduction was observed during this operation. The initial flux was only 11 L.h -1 .m -2 and dropped to 7 L.h -1 .m -2 (figure1), resulting in an average flux of 8 L.h -1 .m -2 . It must be explained by the increase of membrane fouling with suspended matters and the observed flocs. Moreover, the liquid demixing coulddecrease the transfer velocity at the membrane surface and consequently increase the concentration polarization. Indeed, in a recent work, the improvement of suspended matter removal without using centrifugationlead to better and sustainable permeate flux (> 20 L.h -1 .m -2 ).In situ cleaning was performed until recovering a satisfactory water flux (ca. 300 L.h -1 .m -2 ). The slight decrease of water flux showed that the fouling was quite reversible, so suspended matters had to remain mainly at the membrane surface.

Figure 1 -

 1 Figure 1 -Evolution of permeate flux as a function of volumetric concentration factor (VCF) for each ultrafiltration trial

Figure 2 -Figure 3 -

 23 Figure 2 -Evolution of brine (empty symbols) and product (full symbols) conductivities during the first (fresh brine) and second (reused brine) batches

Figure 4 -

 4 Figure 4 -Evolution of brix, pH, 420 nm absorbance and conductivity during ion-exchange treatment

Table 1 -Composition of the wheat bran hydrolysate

 1

	Total dry weight	%w			8.0
	Solubles dry weight	%w			6.9
	Dry suspended matter	g.kg -1			11
		pH	-			1.4
		Conductivity	mS.cm	-1	34.0
	420 nm absorbance	-			1.79
		Glucose				11.4
		Xylose Arabinose	-1 g.L			25.5 14.6
		Furfural				0.8
	H	+ (total acidity)	mmol.L	-1	240
		Na				6
		NH 4	mg.L -1	369
		K				240
	Total cations	meq.L -1	27
		SO 4	mg.L -1

Table 2 -Conditions of ionic chromatography

 2 Ultrafiltration was carried out to remove the macromolecules present in the hydrolysate, which could precipitate during electrodialysis, due to rising pH. Four batch operations were performed with the same membrane, which was cleaned in situbetween tests.This operation was very efficient since no precipitate was detected in permeate samplesafter rising their pH to ca. 7 with a 0.1 M NaOH solution.

		Column	Guard column	Eluent	Suppressor	Injection volume
	Anion analysis	Ionpack AS11HC 2x250 mm (Dionex)	AG11HC 2x50 mm, Dionex	30 mM 0.3 mL.min potassium -1 hydroxide	23 mA	2.5 µL

Table 3 -Experimental results of UF4 trial andrecovery rates estimation after 2.5 diafiltration

 3

					Feed	UF4 Mean permeate	UF4 Final retentate	Recovery rate in UF4 permeate (%)	Total recovery rate after UF4+ 2.5 diafiltration (%)
	Volume		L		23.2	22.5	2.7	90	-
	Dry weight		%w		6.9	5.4	11.7	80	97
	pH		-		1.4	1.4	1.3	-	-
	Conductivity mS.cm	-1	34.0	32.7	29.1	-	-
	Sugars				51.6	42.8	49.9	88	99
	Furfural		g.L -1		0.8	0.8	1.0	87	99
	Salts				12.0	9.6	23.7	78	98
	Total anions			247	198	501	77	96
	Total cations	meq.L -1	27	24	23	90	99
	Acidity (A --C	+ )			220	174	478	76	95

Table 4 -Electrodialysisresults of UF4 permeate

 4

					Feed	Fresh brine	Mean product	Mean brine	Recovery rate in product (%)
	Volume		L		21.6	10.6	21.4	11.2	66
	Dry weight		%w		5.4	0.1	5.0	2.4	80
	pH		-		1.4	2.1	2.4	1.3	-
	Conductivity	mS.cm	-1	32.7	5.0	2.2	70.8	-
	420 nm absorbance	-		1.79		1.07		-
	Sugars				42.8	0.0	43.1	0.7	99
	Furfural		g.L -1		0.8	0.0	0.7	0.0	97
	Salts				9.6	0.7	1.1	18.0	11
	Total anions			198	14	27	363	12
	Total cations	meq.L -1	24	0	1	46	6
	Acidity (A --C	+ )			174	14	25	318	13

Table 5 -Composition of both reference hydrolysate andhydrolysate obtained with recycled brine

 5

				Reference	Hydrolysate with
				hydrolysate	recycled brine
	Dry weight	%w		6.9	7.3
	pH	-		1.4	1.4
	Conductivity	mS.cm	-1	34.0	38.2

Table 6 -Performanceof the global process before polishing treatment

 6

Table 7 -Composition of wheat bran hydrolysate after activated carbon treatment

 7

		Dry weight	pH	Sugars	Conductivity Abs 420 nm	Furfural
		%w	-	g/L	µS.cm	-1	-	g/L
	Purified product	33.7	4.9	385.0	38.5		0.00	0.0
	Recovery rate	>99%	-	>99%	-		-	0%

Acknowledgements

The authors would like to thank Fondation Paris-Reims, Région Champagne-Ardenne, Département de la Marne and Reims Métropole for their financial support and ARD for the hydrolysate supply.


Hande Alemdar

Vincent Leroy

Adrien Prost-Boucle

Frédéric Pétrot

Ternary Neural Networks for Resource-Efficient AI Applications

The computation and storage requirements for Deep Neural Networks (DNNs) are usually high. This issue limits their deployability on ubiquitous computing devices such as smart phones, wearables and autonomous drones. In this paper, we propose ternary neural networks (TNNs) in order to make deep learning more resource-efficient. We train these TNNs using a teacher-student approach based on a novel, layer-wise greedy methodology. Thanks to our two-stage training procedure, the teacher network is still able to use state-of-the-art methods such as dropout and batch normalization to increase accuracy and reduce training time. Using only ternary weights and activations, the student ternary network learns to mimic the behavior of its teacher network without using any multiplication. Unlike its {-1,1} binary counterparts, a ternary neural network inherently prunes the smaller weights by setting them to zero during training. This makes them sparser and thus more energy-efficient. We design a purpose-built hardware architecture for TNNs and implement it on FPGA and ASIC. We evaluate TNNs on several benchmark datasets and demonstrate up to 3.1× better energy efficiency with respect to the state of the art while also improving accuracy.

I. INTRODUCTION

Deep neural networks (DNNs) have achieved state-of-theart results on a wide range of AI tasks including computer vision [START_REF] Hertel | Deep convolutional neural networks as generic feature extractors[END_REF], speech recognition [START_REF] Graves | Speech recognition with deep recurrent neural networks[END_REF] and natural language processing [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]. As DNNs become more complex, their number of layers, number of weights, and computational cost increase. While DNNs are generally trained on powerful servers with the support of GPUs, they can be used for classification tasks on a variety of hardware. However, as the networks get bigger, their deployability on autonomous mobile devices such as drones and self-driving cars and mobile phones diminishes due to the extreme hardware resource requirements imposed by high number of synaptic weights and floating point multiplications. Our goal in this paper is to obtain DNNs that are able to classify at a high throughput on low-power devices without compromising too much accuracy.

In recent years, two main directions of research have been explored to reduce the cost of DNNs classifications. The first one preserves the floating point precision of DNNs, but drastically increases sparsity and weights sharing for compression [START_REF] Han | Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding[END_REF], [START_REF] Han | EIE: Efficient Inference Engine on Compressed Deep Neural Network[END_REF]. This has the advantage of significantly diminishing memory and power consumption while preserving accuracy. However, the power savings are limited by the need for floating-point operation. The second direction reduces the need for floatingpoint operations using weight discretization [START_REF] Courbariaux | Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1[END_REF], [START_REF] Zhou | DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients[END_REF], [START_REF] Li | Ternary weight networks[END_REF], [START_REF] Lin | Neural networks with few multiplications[END_REF], with extreme cases such as binary neural networks completely eliminating the need for multiplications [START_REF] Kim | Bitwise neural networks[END_REF], [START_REF] Esser | Backpropagation for energy-efficient neuromorphic computing[END_REF], [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF]. The main drawback of these approaches is a significant degradation in the classification accuracy in return for a limited gain in resource efficiency.

This paper introduces ternary neural networks (TNNs) to address these issues and makes the following contributions:

• We propose a teacher-student approach for obtaining Ternary NNs with weights and activations constrained to {-1, 0, 1}. The teacher network is trained with stochastic firing using back-propagation, and can benefit from all techniques that exist in the literature such as dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF], batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], and convolutions, The student network has the same architecture and, for each neuron, mimics the behavior of the equivalent neuron in the teacher network without using any multiplications, • We design a specialized hardware that is able to process TNNs at up to 2.7× better throughput, 3.1× better energy efficiency and 635× better area efficiency than state-ofthe-art and with competitive accuracy, • We make the training code publicly available 1 and provide a demonstration hardware design for TNNs using FPGA. 2The rest of this paper is organized as follows. In the following section, we introduce our procedure for training the ternary NNs detailing our use of teacher-student paradigm to eliminate the need for multiplications altogether during test time, while still benefiting all state-of-the-art techniques such as batch normalization and dropout during training. In Section III, we provide a survey of related works that we compare our performance with. We present our experimental evaluation on ternarization and the classification performance on five different benchmark datasets in Section IV. In Section V, we describe our purpose-built hardware that is able to handle both fully connected multi-layer perceptrons (MLPs) and convolutional NNs (CNNs) with a high throughput and a low-energy budget. Finally, we conclude with a discussion and future studies in Section VI.

II. TRAINING TERNARY NEURAL NETWORKS

We use a two-stage teacher-student approach for obtaining TNNs. First, we train the teacher network with stochastically firing ternary neurons. Then, we let the student network learn how to imitate the teacher's behavior using a layer-wise greedy algorithm. Both the teacher and the student networks have the same architecture. The student network's weights are the ternarized version of the teacher network's weights. The

Weights Wi = [wj ], wj ∈ R W i = [w j ], w j ∈ {-1, 0, 1} Bias bi ∈ R b i lo ∈ Z b i hi ∈ Z Transfer yi = W i x + bi y i = W i x Function Act. Fun n t i =      -1 with prob. -ρ if ρ < 0 1 with prob. ρ if ρ > 0 0 otherwise n s i =      -1 if y i < b i lo 1 if y i > b i hi 0 otherwise where ρ = tanh(yi), ρ ∈ (-1, 1)
student network uses a step function with two thresholds as the activation function. In Table I, we provide our notations and descriptions. In order to emphasize the difference, we denote the discrete valued parameters and inputs with a bold font. Real-valued parameters are denoted by normal font. We use [.] to denote a matrix or a vector. We use subscripts for enumeration purposes and superscripts for differentiation. n t i is defined as the output of neuron i in teacher network and n s i is the output of neuron i in the student network. We detail the two stages in the following subsections.

A. The Teacher Network

The teacher network can have any architecture with any number of neurons, and can be trained using any of the standard training algorithms. We train the teacher network with a single constraint only: it has stochastically firing ternary neurons with output values of -1, 0, or 1. The benefit of this approach is that we can use any technique that already exists for efficient NN training, such as batch normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF], etc. In order to have a ternary output for teacher neuron i denoted as n t i , we add a stochastic firing step after the activation step. For achieving this stochastically, we use tanh (hyperbolic tangent), hard tanh, or soft-sign as the activation function of the teacher network so that the neuron output has (-1, 1) range before ternarization. We use this range to determine the ternary output of the neuron as described in Table I. Although we do not require any restrictions for the weights of the teacher network, several studies showed that it has a regularization effect and reduces over-fitting [START_REF] Li | Ternary weight networks[END_REF], [START_REF] Lin | Neural networks with few multiplications[END_REF]. Our approach is compatible with such a regularization technique as well.

B. The Student Network

After the teacher network is trained, we begin the training of the student network. The goal of the student network is to predict the output of the teacher real-valued network. Since we use the same architecture for both networks, there is a one-to-one correspondence between the neurons of both. Each student neuron denoted as n s i learns to mimic the behavior of the corresponding teacher neuron n t i individually and independently from the other neurons. In order to achieve this, a student neuron uses the corresponding teacher neuron's weights as a guide to determine its own ternary weights using two thresholds t lo i (for the lower threshold) and t hi i (for the higher one) on the teacher neuron's weights. This step is called The output ternarization step determines these. Figure 1 depicts the ternarization procedure for a sample neuron. In the top row, we plot the distributions of the weights, activations and ternary output of a sample neuron in the teacher network respectively. The student neuron's weight distribution that is determined by t lo i and t hi i is plotted below the teacher's weight distribution. We use the transfer function output of the student neuron, grouped according to the teacher neuron's output on the same input, to determine the thresholds for the step activation function. In this way, the resulting output distribution for both the teacher and the student neurons are similar. In the following subsections we detail each step.

1) Output Ternarization: The student network uses a twothresholded step activation function to have ternary output as described in Table I. Output ternarization finds the step activation function's thresholds b i lo and b i hi , for a ternary neuron i, for a given set of ternary weights W. In order to achieve this, we compute three different transfer function output distributions for the student neuron, using the teacher neuron's ternary output value on the same input. We use y -to denote the set of transfer function outputs of the student neuron for which the teacher neuron's output value is -1. y 0 and y + are defined in the same way for teacher neuron output values 0 and 1, respectively.

We use a simple classifier to find the boundaries between these three clusters of student neuron transfer function outputs, and use the boundaries as the two thresholds b i lo and b i hi of the step activation function. The classification is done using a linear discriminant on the kernel density estimates of the three distributions. The discriminant between y + and y 0 is selected as the b i hi , and the discriminant between y -and y 0 gives the b i lo . 2) Weight Ternarization: During weight ternarization, the order and the sign of the teacher network's weights are preserved. We ternarize the weights of the i th neuron of the teacher network using two thresholds t lo i and t hi i such that min(W i ) t lo i 0 and 0 t hi i max(W i ). The weights for the i th student neuron are obtained by weight ternarization as follows

ternarize(W i |t lo i , t hi i ) = W i = [w j ] (1)
where

w j =      -1 if w j < t lo i 0 if t lo i w j t hi i 1 if w j > t hi i (2)
We find the optimal threshold values for the weights by evaluating the ternarization quality with a score function. For a given neuron with p positive weights and n negative weights, the total number of possible ternarization schemes is np since we respect the original sign and order of weights. For a given configuration of the positive and negative threshold values t hi and t lo for a given neuron, we calculate the following score for assessing the performance of the ternary network, mimicking the original network.

S t lo ,t hi = d p(n t = ±1|x t d ) I(n s =±1|x s d ) p(n t = 0|x t d ) I(n s =0|x s d )
(3) where n t and n s denote the output of the teacher neuron and student neuron, respectively.

x t d is the d th input sample for the teacher neuron, and x s d is the input d th input sample for the student neuron. Note that x t d = x s d after the first layer. Since we ternarize the network in a feed-forward manner, in order to prevent ternarization errors from propagating to upper layers, we always use the teacher's original input to determine its output probability distribution. The output probability distribution for the teacher neuron for input d, p(n t |x t d ), is calculated using stochastic firing as described in Table I. The output probability distribution for the student neuron for input d, p(n s |x s d ) is calculated using the ternary weights W with the current configuration of t lo , t hi , and the step activation function thresholds. These thresholds, b hi and b lo are selected according to the current ternary weight configuration W.

The output probability values are accumulated as scores over all input samples only when the output of the student neuron matches the output of the teacher neuron. The optimal ternarization of weights is determined by selecting the configuration with the maximum score.

W * = arg max t lo ,t hi S t lo ,t hi (4)
The worst-case time complexity of the algorithm is O( W 2 ). We propose using a greedy dichotomic search strategy instead of a fully exhaustive one. We make a search grid over n candidate values for t lo by p values for t hi . We select two equally-spaced pivot points along one of the dimensions, n or p. Using these pivot points, we calculate the maximum score along the other axis. We reduce the search space by selecting the region in which the maximum point lies. Since we have two points, we reduce the search space to twothirds at each step. Then, we repeat the search procedure in the reduced search space. This faster strategy runs in O(log 2 W ), and when there are no local maxima it is guaranteed to find the optimal solution. When there are multiple local extremum, it may get stuck. Fortunately, we can detect the possible suboptimal solutions, using the score values obtained for the student neuron. By using a threshold on the output score for a student neuron, we can selectively use exhaustive search on a subset of neurons. Empirically, we find these cases to be rare. We provide a detailed analysis in Section IV-A.

The ternarization of the output layer is slightly different since it is a soft-max classifier. In the ternarization process, instead of using the teacher network's output, we use the actual labels in the training set. Again, we treat neurons independently but we make several iterations over each output neuron in a round-robin fashion. After each iteration we check against convergence. In our experiments, we observed that the method converges after a few passes over all neurons.

Our layer-wise approach allows us to update the weights of the teacher network before ternarization of any layer. For this optional weight update, we use a staggered retraining approach in which only the non-ternarized layers are modified. After the teacher network's weights are updated, input to a layer for both teacher and student networks become equal, x t d = x s d . We use early stopping during this optional retraining and we find that a few dozen of iterations suffice.

III. RELATED WORK

In this section, we give a brief survey on several related works in energy-efficient NNs. In Table III, we provide a comparison between our approach and the related works that use binary or ternary weights in the deployment phase by summarizing the constraints put on inputs, weights and activations during training and testing.

Courbariaux et al. [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF] propose the BinaryConnect (BC) method for binarizing only the weights, leaving the inputs and the activations as real-values. The same idea is also used as TernaryConnect (TC) in [START_REF] Lin | Neural networks with few multiplications[END_REF] and Ternary Weight Networks (TWN) in [START_REF] Li | Ternary weight networks[END_REF] with ternary {-1, 0, 1} weights instead of binary {-1, 1}. They use the back-propagation algorithm with an additional weight binarization step. In the forward pass, weights are binarized either deterministically using their sign, or stochastically. Stochastic binarization converts the realvalued weights to probabilities with the use of the hard-sigmoid function, and then decides the final value of the weights with this. In the back-propagation phase, a quantization mechanism is used so that the multiplication operations are converted to bitshift operations. While this binarization scheme helps reducing the number of multiplications during training and testing, it is not fully hardware-friendly since it only reduces the number of floating point multiplication operations. Recently, the same idea is extended to the activations of the neurons also [START_REF] Courbariaux | Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1[END_REF]. In Binarized NN, the sign activation function is used for obtaining binary neuron activations. Also, shift-based operations are used during both training and test time in order to gain energyefficiency. Although this method helps improving the efficiency in multiplications it does not eliminate them completely.

XNOR-Nets [START_REF] Rastegari | XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks[END_REF] provide a solution to convert convolution operations in CNNs to bitwise operations. The method first learns a discrete convolution together with a set of real-valued scaling factors (K, α ∈ R). After the convolution calculations are handled using bit-wise operations, the scaling factors are BC [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF], TC [START_REF] Lin | Neural networks with few multiplications[END_REF], TWN

[8] R {-1, 0, 1} R R {-1, 0, 1} R Binarized NN [6] R {-1, 1} {-1, 1} R {-1, 1} {-1, 1} XNOR-Net[16] R {-1, 1} {-1, 1} with K, α ∈ R R {-1, 1} {-1, 1} with K, α ∈ R EBP[17] R R R R {-1, 0, 1} {-1, 1} Bitwise NN [10] (-1, 1) (-1, 1) (-1, 1) {-1, 1} {-1, 0, 1} {-1, 1} [0, 1] (-1, 1) (-1, 1) {0, 1} {-1, 0, 1} {-1, 1} TrueNorth [11] [0, 1] [-1, 1] [0, 1] {0, 1} {-1, 0, 1} {0, 1} TNN (This Work) {-1, 0, 1} R {-1, 0, 1} {-1, 0, 1} {-1, 0, 1} {-1, 0, 1}
applied to obtain actual result. This approach is very similar to Binarized NN and helps reducing the number of floating point operations to some extent. Following the same goal, DoReFa-Net [START_REF] Zhou | DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients[END_REF] and Quantized Neural Networks (QNN) [START_REF] Hubara | Quantized neural networks: Training neural networks with low precision weights and activations[END_REF] propose using n-bit quantization for weights, activations as well as gradients. In this way, it is possible to gain speed and energy efficiency to some extent not only during training but also during inference time. Han et al. [START_REF] Han | Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding[END_REF] combine several techniques to achieve both quantization and compression of the weights by setting the relatively unimportant ones to 0. They also develop a dedicated hardware called Efficient Inference Engine (EIE) that exploits the quantization and sparsity to gain large speed-ups and energy savings, only on fully connected layers currently [START_REF] Han | EIE: Efficient Inference Engine on Compressed Deep Neural Network[END_REF].

Soudry et al. [START_REF] Soudry | Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights[END_REF] propose Expectation Backpropagation (EBP), an algorithm for learning the weights of a binary network using a variational Bayes technique. The algorithm can be used to train the network such that, each weight can be restricted to be binary or ternary values. The strength of this approach is that the training algorithm does not require any tuning of hyper-parameters, such as learning rate as in the standard back-propagation algorithm. Also, the neurons in the middle layers are binary, making it hardware-friendly. However, this approach assumes the bias is real and it is not currently applicable to CNNs.

All of the methods described above are only partially discretized, leading only to a reduction in the number of floating point multiplication operations. In order to completely eliminate the need for multiplications which will result in maximum resource efficiency, one has to discretize the network completely rather than partially. Under these extreme limitations, only a few studies exist in the literature.

Kim and Smaragdis propose Bitwise NN [START_REF] Kim | Bitwise neural networks[END_REF] which is a completely binary approach, where all the inputs, weights, and the outputs are binary. They use a straightforward extension of back-propagation to learn bitwise network's weights. First, a real-valued network is trained by constraining the weights of the network using tanh. Also tanh non-linearity is used for the activations to constrain the neuron output to (-1, 1). Then, in a second training step, the binary network is trained using the real-valued network together with a global sparsity parameter. In each epoch during forward propagation, the weights and the activations are binarized using the sign function on the original constrained real-valued parameters and activations. Currently, CNNs are not supported in Bitwise-NNs.

IBM announced an energy efficient TrueNorth chip, designed for spiking neural network architectures [START_REF] Merolla | A million spiking-neuron integrated circuit with a scalable communication network and interface[END_REF]. Esser et al. [START_REF] Esser | Backpropagation for energy-efficient neuromorphic computing[END_REF] propose an algorithm for training networks that are compatible with IBM TrueNorth chip. The algorithm is based on backpropagation with two modifications. First, Gaussian approximation is used for the summation of several Bernoulli neurons, and second, values are clipped to satisfy the boundary requirements of TrueNorth chip. The ternary weights are obtained by introducing a synaptic connection parameter that determines whether a connection exits. If the connection exists, the sign of the weight is used. Recently, the work has been extended to CNN architectures as well [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF].

IV. EXPERIMENTAL ASSESSMENT OF TERNARIZATION AND CLASSIFICATION

The main goals of our experiments are to demonstrate, (i) the performance of the ternarization procedure with respect to the real-valued teacher network, and (ii) the classification performance of fully discretized ternary networks.

We perform our experiments on several benchmarking datasets using both multi-layer perceptrons (MLP) in a permutation-invariant manner and convolutional neural networks (CNN) with varying sizes. For the MLPs, we experiment with different architectures in terms of depth and neuron count. We use 250, 500, 750, and 1000 neurons per layer for 2, 3, and 4 layer networks. For the CNNs, we use the following VGG-like architecture proposed by [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF]:

(2 × nC3 ) -M P 2 -(2 × 2nC3 ) -MP2 -(2 × 4nC3 ) - M P 2 -(2 × 8nFC ) -L2SVM
where C3 is a 3 × 3 convolutional layer, MP2 is a 2 × 2 maxpooling layer, FC is a fully connected layer. We use L2SVM as our output layer. We use two different network sizes with this architecture with n = 64 and n = 128. We call these networks CNN-Small and CNN-Big, respectively.

We perform our experiments on the following datasets: MNIST database of handwritten digits [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] is a well-studied database for benchmarking methods on real-world data. MNIST has a training set of 60K examples, and a test set of 10K examples of 28 × 28 gray-scale images. We use the last 10K samples of the training set as a validation set for early stopping and model selection.

CIFAR-10 and CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] are two color-image classification datasets that contain 32×32 RGB images. Each dataset consists of 50K images in training and 10K images in test sets. In CIFAR-10, the images come from a set of 10 classes that contain airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships and trucks. In CIFAR-100, the number of image classes is 100.

SVHN (Street View House Numbers) [START_REF] Netzer | Reading digits in natural images with unsupervised feature learning[END_REF] consists of 32× 32 RGB color images of digits cropped from Street View images. The total training set size is 604K examples (with 531K less difficult samples to be used as extra) and the test set contains 26K images.

GTSRB (German Traffic Sign Recognition Benchmark Dataset) [START_REF] Stallkamp | Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition[END_REF] is composed of 51839 images of German road signs in 43 classes. The images have great variability in terms of size and illumination conditions. Also, the dataset has unbalanced class frequencies. The images in the dataset are extracted from 1-second video sequences recorded at 30 fps. In order to have a representative validation set, we extract 1 track at random per traffic sign for validation. The number of images in train, validation and test set are 37919, 1290 and 12630 respectively.

In order to allow a fair comparison against related works, we perform our experiments in similar configurations. On MNIST, we only use MLPs. We minimize cross entropy loss using stochastic gradient descent with a mini-batch size of 100.

During training we use random rotations up to ±10 degrees. We report the test error rate associated with the best validation error rate after 1000 epochs. We do not perform any preprocessing on MNIST and we use a threshold-based binarization on the input.

For other datasets, we use two CNN architectures: CNN-Small and CNN-Big. As before, we train a teacher network before obtaining the student TNN. For the teacher network, we use a modified version of Binarized NN's algorithm [START_REF] Courbariaux | Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1[END_REF] and ternarize the weights during training. In this way, we obtain a better accuracy on the teacher network and we gain considerable speed-up while obtaining the student network. Since we already have the discretized weights during the teacher network training, we only mimic the output of the neurons using the step activation function with two thresholds for the student network. During teacher network training, we minimize squared Hinge loss with adam with mini-batches of size 200. We train at most 1000 epochs and report the test error rate associated with the best validation epoch. For input binarization, we use the approach described in [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF] with either 12 or 24 (on CIFAR100) transduction filters. We do not use any augmentation on the datasets.

A. Ternarization Performance

The ternarization performance is the ability of the student network to imitate the behavior of its teacher. We measure this by using the accuracy difference between the teacher network and the student network. Table III shows this difference between the teacher and student networks on training and test sets for three different exhaustive search threshold values. Since we always use the teacher network's original output as a reference, errors are not amplified in the network. On the contrary, deeper networks allow the student network to correct some of the mistakes in the upper layers, dampening the errors. Also, we perform a retraining step with early stopping before ternarizing a layer, since it slightly improves the performance. The ternarization performance generally decreases with lower ε threshold values, but the decrease is marginal. On occasion, performance even increases. This is due to teacher network's weight update, that allows the network to escape from local minima. In order to demonstrate the effect of ε in terms of run-time and classification performance, we conduct a detailed analysis without the optional staggered retraining. Figure 2 shows the distribution of the ratio of neurons that are ternarized exhaustively with different ε, together with the performance gaps on training and test datasets. The optimal trade-off is achieved with ε = 0.95. Exhaustive search is used for only 20% of the neurons, and the expected value of accuracy gaps is practically 0. For the largest layer with 1000 neurons, the ternarization operations take 2 min and 63 min for dichotomic and exhaustive search, respectively, on a 40-core Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz server with 128 GB RAM. For the output layer, the ternarization time is reduced to 21 min with exhaustive search.

B. Classification Performance

The classification performance in terms of error rate (%) on several benchmark datasets is provided in Table IV. We compare our results to several related methods that we described in the previous section. We make a distinction between the fully discretized methods and the partially discretized ones because only in the latter, the resulting network is completely discrete and requires no floating points and no multiplications, providing maximum energy efficiency.

Since the benchmark datasets we use are the most studied ones, there are several known techniques and tricks that give performance boosts. In order to eliminate unfair comparison among methods, we follow the majority's lead and we do not use any data augmentation in our experiments. Moreover, using an ensemble of classifiers is a common technique for performance boosting in almost all classifiers and is not unique to neural networks [START_REF] Ciresan | A committee of neural networks for traffic sign classification[END_REF]. For that reason, we do not use an ensemble of networks and we cite the compatible results in other related works.

MNIST is by far the most studied dataset in deep learning literature. The state-of-the-art is already down to 21 erroneous classifications (0.21%) which is extremely difficult to obtain without extensive data augmentation. TNN's error rate on MNIST is 1.67% with a single 3-layer MLP with 750 neurons [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF] 1.29 9.90 2.30 TC [START_REF] Lin | Neural networks with few multiplications[END_REF] 1.15 12.01 2.42 TWN [START_REF] Li | Ternary weight networks[END_REF] 0.65 7.44 EBP [START_REF] Cheng | Training binary multilayer neural networks for image classification using expectation backpropagation[END_REF] 2.20 XNOR-Net [START_REF] Rastegari | XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks[END_REF] 9.88 DoReFa-Net [START_REF] Zhou | DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients[END_REF] 2.40 in each layer. Bitwise NNs [START_REF] Kim | Bitwise neural networks[END_REF] with 1024 neurons in 3 layers achieves a slightly better performance. TNN with an architecture that has similar size to Bitwise NN is worse due to over-fitting. Since TNN selects a different sparsity level for each neuron, it can perform better on smaller networks, and larger networks cause over-fitting on MNIST. Bitwise NN's global sparsity parameter has a better regularization effect on MNIST for relatively bigger networks. Its performance with smaller networks or on other datasets is unknown. TrueNorth [START_REF] Esser | Backpropagation for energy-efficient neuromorphic computing[END_REF] with a single network achieves only 7.30% error rate. To alleviate the limitations of single network performance, a committee of networks can be used, reducing the error rate to 0.58% with 64 networks.

The error rate of TNN on CIFAR10 is 12.11%. When compared to partially discretized alternatives, a fully discretized TNN is obtained at the cost a few points in the accuracy and exceeds the performance of TrueNorth by more than 4%. On SVHN, it has a similar achievement with lower margins. For CIFAR100, on the other hand, it does not perform better than TrueNorth. Given the relatively lower number of related works that report results on CIFAR100 as opposed to CIFAR10, we can conclude that this is a more challenging dataset for resourceefficient deep learning with a lot of room for improvement. TNN has the most remarkable performance on GTSRB dataset. With 0.98% error rate, CNN-Big model exceeds the human performance which is at 1.16%.

Partially discretized approaches use real-valued input which contains more information. Therefore, it is expected that they are able to get higher classification accuracy. When compared to partially discretized studies, TNNs only lose a small percentage of accuracy and in return they provide better energy efficiency. Next, we describe the unique hardware design for TNNS and investigate to which extent TNNs are area and energy efficient.

V. PURPOSE-BUILT HARDWARE FOR TNN

We designed a hardware architecture for TNNs that is optimized for ternary neuron weights and activation values {-1, 0, +1}. In this section we first describe the purpose-built hardware we designed and evaluate its performance in terms of latency, throughput and energy and area efficiency. Ternary values are represented with 2 bits using usual two's complement encoding. That way, the compute part of each neuron is reduced to just one integer adder/subtractor and one register, both of width log 2 k + 1 bits, where k is the input size for the neuron. So each neuron is only a few tens of ASIC gates, which is very small. Inside each layer, all neurons work in parallel so that one new input item is processed per clock cycle. Layers are pipelined in order to simultaneously work on different sets of inputs, i.e. layer 2 processes image n while layer 1 processes image n + 1. The ternarization block processes the neuron outputs sequentially, so it consists of the memory of threshold values, two signed comparators and a multiplexer.

A. Hardware Architecture

We did a generic register transfer level (RTL) implementation that can be synthesized on both Field-Programmable Gate Array (FPGA) and Application-specific Integrated Circuit (ASIC) technologies. FPGAs are reprogrammable off-theshelf circuits and are ideal for general-purpose hardware acceleration. Typically, high-performance cloud solutions use high-end FPGA tightly coupled with general-purpose multicore processors [START_REF] Choi | A quantitative analysis on microarchitectures of modern CPU-FPGA platforms[END_REF], while ASIC is used for more throughput or in battery-powered embedded devices.

B. Hardware Performance

For the preliminary measurements, we used the dataset MNIST and the FPGA board Sakura-X [START_REF] Board | [END_REF] because it features precise power measurements capabilities. It can accommodate a 3-layer fully connected NN with 1024 neurons per layer (for a total of 3082 neurons), using 81% of the Kintex-7 160T FPGA.

The performance of our FPGA design in terms of latency, throughput and energy efficiency is given in Table V. With a 200 MHz clock frequency, the throughput (here limited by the number of neurons) is 195 K images/s with a power consumption of 3.8 W and a classification latency of 20.5 µs. We know that TrueNorth [START_REF] Esser | Backpropagation for energy-efficient neuromorphic computing[END_REF] can operate at the two extremes of power consumption and accuracy. It consumes 0.268 µJ with a network of low accuracy (92.7%), and consumes as high as 108 µJ with a committee of 64 networks that achieves 99.4%. Our hardware cannot operate at these two extremes, yet in the middle operating zone, we outperform TrueNorth both in terms of energy-efficiency -accuracy trade-off and speed. TrueNorth consumes 4 µJ per image with 95% accuracy with a throughput of 1000 images/s, and with 1 ms latency. Our TNN hardware, consuming 3.63 µJ per image achieves 98.14% accuracy at a rate of 255 102 images/s, and a latency of 8.09 µs.

For the rest of the FPGA experiments, the larger board VC709 equipped with the Virtex-7 690T FPGA is used because it can support much larger designs. We also synthesized the design as ASIC using STMicroelectronics 28 nm FDSOI manufacturing technology. The results are given in Table VI. We compare our FPGA and ASIC solutions with the state of the art: TrueNorth [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF] and EIE [START_REF] Han | EIE: Efficient Inference Engine on Compressed Deep Neural Network[END_REF].

The ASIC version compares very well with TrueNorth on throughput, area efficiency (fps/mm 2 ) and energy efficiency (fps/W). Even though EIE uses 16-bit precision, it achieves high throughput because it takes advantage of weight sparsity and skips many useless computations. However, we achieve better energy and area efficiencies since all our hardware elements (memories, functional units etc.) are significantly reduced thanks to ternarization. Our energy results would be even better if taking into account weight sparsity and zeroactivations (e.g. when input values are zero) like done in EIE works.

Finally, we implemented the CNN-Big and CNN-Small described in Section IV, on both FPGA and ASIC. Results are given in Table VII. We give worst-case FPGA results because this is important for users of general-purpose hardware accelerators. For ASIC technology, we took into account perdataset zero-activations to reduce power consumption, similar to what was done in EIE works. We compare with TrueNorth because only paper [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF] gives figures of merit related to CNNs on ASIC. The TrueNorth area is calculated according to the number of cores used. Using different CNN models than TrueNorth's, we achieve better accuracy on three datasets out of four, while having higher throughput, better energy efficiency and much better area efficiency.

VI. DISCUSSIONS AND FUTURE WORK

In this study, we propose TNNs for resource-efficient applications of deep learning. Energy efficiency and area efficiency are brought by not using any multiplication nor any floating-point operation. We develop a student-teacher approach to train the TNNs and devise a purpose-built hardware for making them available for embedded applications with resource constraints. Through experimental evaluation, we demonstrate the performance of TNNs both in terms of accuracy and resource-efficiency, with CNNs as well as MLPs. The only other related work that has these two features is TrueNorth [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF], since Bitwise NNs do not support CNNs [START_REF] Kim | Bitwise neural networks[END_REF]. In terms of accuracy, TNNs perform better than TrueNorth with relatively smaller networks in all of the benchmark datasets except one. Unlike TrueNorth and Bitwise NNs, TNNs use ternary neuron activations using a step function with two thresholds. This allows each neuron to choose a sparsity parameter for itself and gives an opportunity to remove the weights that have very little contribution. In that respect, TNNs inherently prune the unnecessary connections.

We also develop a purpose-built hardware for TNNs that offers significant throughput and area efficiency and highly competitive energy efficiency. As compared to TrueNorth, our TNN ASIC hardware offers improvements of 147× to 635× on area efficiency, 1.4× to 3.1× on energy efficiency and 2.1× to 2.7× on throughput. It also often has higher accuracy with our new training approach.

Fig. 1 .

 1 Fig. 1. Example ternarization for a single neuron

Fig. 2 .

 2 Fig. 2. The effect of threshold values on run-time and classification performance

Figure 3 2 Fig. 3 .

 323 Figure3outlines the hardware architecture of a fullyconnected layer in a multi-layer NN. The design forms a pipeline that corresponds to the sequence of NN processing steps. For efficiency reasons, the number of layers and the maximum layer dimensions (input size and number of neurons) are decided at synthesis time. For a given NN architecture, the

TABLE I TERNARY

 I NEURAL NETWORK DEFINITIONS FOR A SINGLE NEURON i

	Teacher network	Student Network

TABLE II COMPARISON

 II OF SEVERAL APPROACHES FOR RESOURCE-EFFICIENT NEURAL NETWORKS

				Training		Deployment
	Method	Inputs	Weights	Activations	Inputs	Weights	Activations

TABLE III ACCURACY

 III GAP DUE TO TERNARIZATION FOR DIFFERENT ε

			ε = 1	ε = 0.95	ε = 0
	Neurons	Layers	Train	Test	Train	Test	Train	Test
		1	0.63	0.75	0.69	0.76	0.72	0.98
	250	2	0.30	0.44	0.25	0.56	0.15	0.54
		3	0.08	0.47	0.10	0.62	0.03	0.52
		1	0.50	0.94	0.49	0.83	0.48	0.70
	500	2	0.23	0.36	0.29	0.39	0.26	0.49
		3	0.12	0.25	0.07	0.34	0.10	0.20
		1	0.27	0.90	0.27	0.90	0.29	1.05
	750	2	0.00	0.56	-0.01	0.65	0.00	0.67
		3	-0.02	0.43	-0.03	0.44	-0.03	0.26
		1	0.44	0.66	0.60	0.87	0.79	0.95
	1000	2	-0.02	0.59	-0.07	0.33	-0.05	0.37
		3	-0.16	0.29	-0.14	0.34	-0.14	0.34

TABLE V FPGA

 V HARDWARE PERFORMANCE OF MLPS ON SAKURA-X

	Neurons	Throughput	Layers	Energy µJ	Latency µs	Accuracy
		(fps)		(per image)	(per image)	(%)
			1	1.24	5.37	97.76
	250	255 102	2	2.44	6.73	98.13
			3	3.63	8.09	98.14
			1	2.44	6.63	97.75
	500	255 102	2	4.83	9.24	98.14
			3	7.22	11.9	98.29
			1	3.63	7.88	97.73
	750	255 102	2	7.22	11.8	98.10
			3	10.8	15.6	98.33
			1	6.22	10.2	97.63
	1000	198 019	2	12.4	15.3	98.09
			3	18.5	20.5	97.89

TABLE VI COMPARISON

 VI OF SEVERAL HARDWARE SOLUTIONS FOR MLP

	Platform	TrueNorth[12] EIE 64PE[5] EIE 256PE[5] TNN	TNN
	Year	2014	2016	2016	2016	2016
	Type	ASIC	ASIC	ASIC	FPGA	ASIC
	Technology	28 nm	45 nm	28 nm	Virtex-7 ST 28 nm
	Clock (MHz)	Async.	800	1200	250	500
	Quantization	1-bit	4-bit	4-bit	Ternary Ternary
	Throughput (fps)	1 989	81 967	426 230	61 035 122 070
	Power (W)	0.18	0.59	2.36	6.25	0.588
	Energy Eff. (fps/W)	10 839	138 927	180 606	9 771 207 567
	Area (mm 2 )	430	40.8	63.8		6.36
	Area Eff. (fps/mm 2 )	5	2 009	6 681		19 194

https://github.com/slide-lig/tnn-train

http://tima.imag.fr/sls/research-projects/tnn-fpga-implementation/

ACKNOWLEDGMENT

This project is being funded in part by Grenoble Alpes Métropole through the Nano2017 Esprit project. The authors would like to thank Olivier Menut from ST Microelectronics for his valuable inputs and continuous support.