Nadia Yacoubi Ayadi

Catherine Faron

Franck Michel
email: fmichel@i3s.unice.fr

Fabien Gandon

Olivier Corby
email: olivier.corby@inria.frfaron

WeKG-MF: a Knowledge Graph of Observational Weather Data

Keywords: Knowledge Graph, Semantic Modelling, Observational Data, Linked Data, Meteorology

In this paper, we present the WeKG-MF Knowledge Graph constructed from open weather observations published by Météo-France institution. WeKG-MF relies on a semantic model that formalizes knowledge about meteorological observational data. The model is generic enough to be adopted and extended by meteorological data providers to publish and integrate their sources while complying with Linked Data principles. WeKG-MF offers access to a large number of meteorological variables described through spatial and temporal dimensions and thus has the potential to serve several scientific case studies from different domains including agriculture, agronomy, environment, climate change and natural disasters.

Introduction

Meteorological data are crucial for many application domains. They typically include measurements of several weather parameters such as wind direction and speed, air pressure, rainfall, humidity and temperature. However, these data are most commonly collected and stored separately in different files using a tabular data format that lacks explicit semantics, which impedes their integration and sharing to serve researchers from different domains such as agriculture, climate change studies or natural disaster monitoring. A typical approach in integrating and publishing such data is to formalize a knowledge graph relying on linked data and semantic Web standard models and practices. To deal with the complexity of the knowledge domain to be modelled, we adopted the SAMOD agile methodology [START_REF] Peroni | SAMOD: an agile methodology for the development of ontologies[END_REF]. The SAMOD process is initiated by a motivating scenario that leads to a set of competency questions that, in turn, provide requirements on the knowledge graph model. As output, we designed a semantic model in which meteorological variables are semantically defined and described at a fine grained level, including aspects of time, location, units of measurement, etc. The WeKG-MF knowledge graph, constructed from open weather observations published by Météo-France, is compliant with the proposed semantic model. The first release of the WeKG-MF includes weather observations from January 2019 till December 2021. The paper is structured as follows. Section 2 presents an overview of WeKG semantic model and highlights its design principles. Section 3 presents the RDF-based knowl-edge graph WeKG-MF constructed from weather data archives of Météo-France and illustrates how it serves use cases identified in the context of the D2KAB French research project1.

Semantic Model for Weather Data

In order to propose a self-contained model for representing and publishing meteorological data, we extend the SOSA/SSN [START_REF] Janowicz | SOSA: A lightweight ontology for sensors, observations, samples, and actuators[END_REF][START_REF] Haller | The modular SSN ontology: A joint W3C and OGC standard specifying the semantics of sensors, observations, sampling, and actuation[END_REF] ontologies with three new classes. First, weo:Meterologi-calObservation is the core class of our model; it supports the description of a single, atomic observation that is related to a particular feature of interest, instance of the weo:MeteorologicalFeature class, and an observable property, instance of class weo:WeatherProperty. These three classes specialize classes from the SOSA/SSN ontologies [START_REF] Janowicz | SOSA: A lightweight ontology for sensors, observations, samples, and actuators[END_REF][START_REF] Haller | The modular SSN ontology: A joint W3C and OGC standard specifying the semantics of sensors, observations, sampling, and actuation[END_REF] as reflected by their formal definitions. These definitions express that only one weather property and one meteorological feature is used for a given meteorological observation: We reused the Value Sets2(VP) ontology design pattern and, because we can enumerate the values, this led us to define a SKOS vocabulary whose concepts are instances of weo:WeatherProperty or weo:MeteorologicalFeature and represent the possible values of observable properties and features of interest. Inline with Linked Data best practices, we aligned the weather properties of our SKOS vocabulary with terms from the NERC Climate and Forecast Standard Names vocabulary3. To avoid redundancies of measurements units among observations, we define for each SKOS weather property an applicable unit re-using QUDT Unit vocabulary4. Thus, observation results are modelled as literals and an observation is linked to its result by RDF property sosa:hasSimpleResult. Finally, since observable properties of our vocabulary are also defined as instances of the qudt:QuantityKind class, we aligned them with terms from the QUDT Quantity Kind vocabulary. Figure 1 presents an RDF graph of a meteorological observation relative to the wind feature of interest and reporting the average wind speed observable property. , by expressing OWL restrictions on their duration that may be declared in seconds, minutes or hours. The interest of doing this is that these time intervals are declared once in our semantic model and are reused for all observations, and thus avoid substantial redundancy. In Figure 1, the wevp:windAverageSpeed weather property is measured during a period of 10 minutes. This is denoted by sosa:phenomenonTime property whose value is an instance of weo:Interval10m class, while the end time of the interval is an instance of class time:Instant.

Meteorological observations are provided by weather stations which spatial information, such as longitude, latitude and altitude. We introduce the weo:WeatherStation class as a subclass of the geosparql:Feature class and sosa:Platform. According to GeoSPARQL vocabulary [START_REF] Battle | Enabling the geospatial semantic web with parliament and geosparql[END_REF], each instance of weo:WeatherStation has a geometry with specific coordinates which enables us to query weather observations based on spatial information. The OWL version of our semantic model as well as the related SKOS vocabularies are available in our Github repository6. The prefixes of ontologies and vocabularies reused or introduced in this paper are listed in the repository's README7.

5 https://public.wmo.int/en/ 6 https://github.com/Wimmics/d2kab/tree/main/meteo/ontology 7 https://github.com/Wimmics/d2kab/tree/main/meteo

WeKG-MF: Weather Knowledge Graph -Météo-France

We constructed the WeKG-MF knowledge graph according to the model presented in Section 2 from open weather observations published by Météo-France8. We first downloaded from Météo-France's portal the list of SYNOP weather stations in GeoJSON format, as well as the monthly observation reports generated by these stations as CSV files. Then, we implemented a reproducible pipeline to generate WeKG-MF in compliance with the proposed semantic model where the mapping is performed by the Morph-xR2RML tool [START_REF] Michel | Translation of relational and non-relational databases into RDF with xR2RML[END_REF]. The first version of WeKG-MF covers the period from January 2019 to the end of December 2021. Key statistics of the dataset are presented in Table 1. To illustrate how observations in the WeKG-MF can be queried and aggregated, we first developed a set of SPARQL queries available on the Github repository of the project9. Beyond this, we initially started this work to address the needs of the D2KAB French project whose primary objective is to create a framework to turn agronomy and biodiversity data into semantically described, interoperable, actionable, and open knowledge. A preliminary needs analysis pointed to competency questions that potential users may want to get answers.We only present some of them due to space constraints. Experts in agronomy investigate the correlations between the development rate of plants and weather parameters. They are especially interested in comparing aggregated values of a weather parameter for the same period of time in the same geographic location across years, e.g. the Growing Daily Degrees (GDD) calculated from the daily average air temperature minus a certain threshold called base temperature.

Category

We developed, together with the set of SPARQL queries, a Jupyter Notebook that demonstrate how the results of queries can be used to generate visualizations from the WeKG-MF knowledge graph. As an example, Figure 2 presents two plots. The first one shows daily cumulative precipitations measured at the 'Bordeaux-Merignac' station and the second one shows the evolution of daily average temperature collected from weather stations located in the French region of 'Nouvelle Aquitaine'. Each plot shows a comparison of aggregated values calculated based on two weather parameters (precipitation and air temperatures) available in the WeKG-MF knowledge graph. Several use cases in agronomy can benefit from pre-calculated spatio-temporal slices of observations. For this, our model supports the

Conclusion and Future Work

In this paper, we presented WeKG-MF, a knowledge graph of weather observational data generated from Météo-France's open data archives. It is based on a reusable, formal model exploiting and extending a network of well-known ontologies. In terms of sustainability, we released a fully automatic pipeline that enables anyone to generate and update the WeKG-MF graph over time with new data downloaded from Météo-France. In the short term, we will investigate identified use cases in the agronomy and agriculture domains in which our meteorological knowledge graph will be integrated with other knowledge sources in order to meet D2KAB project partners requirements.

Fig. 1 .

 1 Fig. 1. Example of a meteorological observation reporting the Wind Speed Average property

Fig. 2 .

 2 Fig. 2. Examples of Visualisation of Daily Precipitations and Average Temperature

Table 1 .

 1 key statistics of the WeKG-MF dataset

	Downloadable RDF dump	https://doi.org/10.5281/zenodo.5925413
	Total Nr. of triples	62.306.102
	Nr. of Observations	8.335.258
	Nr. of weather stations	62
	Nr. of weather properties	22
	Nr. of meteorological features	6
	Nr. of Observations per weather property	≈ 416.762
	Nr. of links to Wikidata	92

https://www.meteofrance.com/


J Acera Fernández

N Le Moigne

R El Hage

A Le Duc

M Lozachmeur

P Bono

A Bergeret
email: anne.bergeret@mines-ales.fr

INFLUENCE OF FLAX CELL WALL COMPONENTS ON THE INTERFACIAL BEHAVIOR OF FLAX WOVEN FABRIC / EPOXY BIOCOMPOSITES

In this work, the effect of the selective extraction of flax fibre components on the properties of flax woven fabric / epoxy biocomposites is investigated. Six treatments were applied to flax woven fabrics and their influence on fabric characteristics and the consequences on the fibre/matrix interface were studied. Components removal was evaluated by chemical composition and FT-IR analysis. Mechanical tests and SEM observations were used to study the composite interface. A defibrillation phenomenon within the fibre bundles with the extraction of some specific components was responsible for a significant decrease of the transverse strength of the biocomposites whereas the porosity was reduced for all the treatments used. Our results highlight the key role of cell wall components on the morphology and properties of flax woven fabric / epoxy biocomposites.

INTRODUCTION

Natural fibres based composites are increasingly used as alternatives to glass fibres based composites, mainly because of their low density and positive life cycle assessment. Nevertheless, these biocomposites present lower mechanical properties than those of glass fibres reinforced composites [START_REF] Liang | A comparative study of fatigue behaviour of flax/epoxy and glass/epoxy composites[END_REF]. The improvement of their mechanical properties requires a better understanding and control of the fibre/matrix interface and the intra-fibrillar interface within the fibre bundles which are composed of several cell wall components, i.e. cellulose micro-fibrils, hemicelluloses, lignins, pectins and lipophilic extractives [START_REF] Marques | Lipophilic extractives from several nonwoody lignocellulosic crops (flax, hemp, sisal, abaca) and their fate during alkaline pulping and TCF/ECF bleaching[END_REF]. The aim of this study is to investigate the influence of different treatments used to remove selectively flax fibre components, and their effect on flax fabrics properties and on the interfacial behaviour of flax fabrics / epoxy biocomposites.

In this study, flax woven fabrics supplied by Fibres Recherche Développement Co. (FRD, France) were used. Epoxy resin 1800/1805 was supplied by Resoltech Co. (France). Composites were made by thermo-compression technique. Fabrics were treated with six different solutions for 1h at 95ºC: 1) hot-water, 2) 70%w non-ionic and 30%w anionic surfactants solutions, 3) NaOH 1%, 4) NaOH 5%, and for 1h at 75ºC 5) ethanol and 6) toluene/ethanol (2:1w). After treatment, flax fabrics were washed for 10min at 50ºC then for10 min at room temperature in order to remove the residual solvents and extracted components, and then dried at 60ºC. Dimensional and gravimetric studies were applied to flax woven fabrics. Chemical composition of flax fibres was determined by chemical analysis according to several specific standards for the quantification of soluble extractives, hollocellulose, lignins, α-cellulose and ashes and by ATR-FTIR analysis. Surface characterizations were carried out by contact angle measurements and optical microscopy and SEM observations. Mechanical tests and SEM observations have been conducted to study the biocomposites interfacial behaviour and the effect of the different treatments.

RESULTS AND CONCLUSIONS

Lipophilic extractives and lignin removal were highlighted by ATR-FTIR spectrometry. Strong peaks at 2848 cm -1 and 2916 cm -1 , corresponding to C-CH3 stretching vibrations, disappear after ethanol and toluene/ethanol treatments. In contrast, a reduction of 1731cm -1 peak after NaOH 1% and 5% treatments can be attributed to the removal of lignins (C=O stretching in carboxylic groups). SEM section observations were carried out on untreated and treated flax fabrics/epoxy composites (Fig. 1). Pronounced defibrillation was observed for NaOH 5% (Fig. 1b) treated composite as well as a strong decrease of the transverse strength. At the same time, reduced porosity (black areas) was noticed for all treated composites. Treatments used to remove flax fibre components modify the physico-chemistry and the texturing of flax fabrics. These modifications affect the morphology and the fibre / matrix interface as well as the intra-fibrillar interface within the fibre bundles, influencing the transverse mechanical properties of the composites.

Fig. 1 .

 1 Fig. 1. SEM section images of a) untreated and b) NaOH 5% treated flax fabric/epoxy composite

ACKNOWLEDGMENTS

Authors gratefully thank the financial support by ADEME, N° TEZ12-26.


Maxime Christ
email: maxime.christ@insa-lyon.frflorent.de-dinechin@insa-lyon.frfrederic.petrot@univ-grenoble-alpes.fr

Florent De Dinechin

Frédéric Pétrot

Low-precision logarithmic arithmetic for neural network accelerators

Resource requirements for hardware acceleration of neural networks inference is notoriously high, both in terms of computation and storage. One way to mitigate this issue is to quantize parameters and activations. This is usually done by scaling and centering the distributions of weights and activations, on a kernel per kernel basis, so that a low-precision binary integer representation can be used. This work studies low-precision logarithmic number system (LNS) as an efficient alternative.

Firstly, LNS has more dynamic than fixed-point for the same number of bits. Thus, when quantizing MNIST and CIFAR reference networks without retraining, the smallest format size achieving top-1 accuracy comparable to floating-point is 1 to 3 bits smaller with LNS than with fixed-point. In addition, it is shown that the zero bit of classical LNS is not needed in this context, and that the sign bit can be saved for activations.

Secondly, low-precision LNS enables efficient inference architectures where 1/ multiplications reduce to additions; 2/ the weighted inputs are converted to classical linear domain, but the tables needed for this conversion remain very small thanks to the low precision; and 3/ the conversion of the output activation back to LNS can be merged with an arbitrary activation function.

The proposed LNS neuron is detailed and its implementation on FPGA is shown to be smaller and faster than a fixed-point one for comparable accuracy.

I. Introduction

Commodity hardware such as multi-core processors and GPUs, although intrinsically very powerful, is not very powerefficient in neural network inference. This is why the last decade has seen a boom in the number of hardware accelerators for inference tasks, applicable to both cloud and embedded solutions [START_REF] Lecun | Deep learning hardware: Past, present, and future[END_REF]. These accelerators can be either computation engines directly placed into processor pipelines, such as ARM Scalable Vector Extensions [START_REF] Stephens | The ARM scalable vector extension[END_REF] and Intel DL Boost [START_REF] Arafa | Cascade Lake: Next Generation Intel Xeon Scalable Processor[END_REF], or independent coprocessors such as Google TPU [START_REF] Norrie | The Design Process for Google's Training Chips: TPUv2 and TPUv3[END_REF], Qualcomm Cloud AI [START_REF] Chatha | Qualcomm Cloud Al 100: 12TOPS/W Scalable, High Performance and Low Latency Deep Learning Inference Accelerator[END_REF], or Kalray KaNN [START_REF] Dupont De Dinechin | A Qualitative Approach to Many-core Architecture[END_REF], to name a few. The chosen strategy is a trade-off between mapping tools complexity, target power efficiency, flexibility, ease of integration in existing frameworks, and so on.

These accelerators typically use 8-bit integers instead of 32bit floating-point numbers for inference tasks. It has indeed been demonstrated that 8-bit values entail only very small inference accuracy loss for most application domains [START_REF] Krishnamoorthi | Quantizing deep convolutional networks for efficient inference: A whitepaper[END_REF], hence the generalization of 8-bit quantization as a "one size fits all" solution. Furthermore, methods that efficiently map floating-point network parameters (weights and biases learned during training) onto the 256 available values do exist [START_REF] Jacob | Quantization and training of neural networks for efficient integer-arithmetic-only inference[END_REF], and are already available in the most popular frameworks for Machine Learning (ML). The gain in memory bandwidth and computational effort offered by 8-bit quantization is huge.

Still, the search for higher power efficiency has lead researchers to study even smaller bit-widths, from both the algorithmic and architectural point of view. At the extreme, binary ({-1, 1}) and ternary ({-1, 0, 1}) networks have drawn some attention. Their neuron computations can be implemented as a handful of logic gates, and the memory requirements can hardly be smaller [START_REF] Umuroglu | FINN: A framework for fast, scalable binarized neural network inference[END_REF][START_REF] Prost-Boucle | Scalable high-performance architecture for convolutional ternary neural networks on FPGA[END_REF]. Such networks can reach good accuracy [START_REF] Zhou | DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients[END_REF][START_REF] Bulat | XNOR-Net++: Improved binary neural networks[END_REF][START_REF] Alemdar | Ternary Neural Networks for Resource-Efficient AI Applications[END_REF], and research is still active in improving their training. However, to mitigate the accuracy loss due to such extreme quantizations, it is customary to increase the number of neurons and/or layers, at the cost of more parameters, hence a full retraining of the network.

This seems to leave some room for quantization lower than 8 bits, without changing the network structure, without retraining, and without accuracy loss.

However, for this to work, we need to drop fixed-point for a representation that offers a wider range for fewer bits. It the present study, we consider low-precision Logarithmic Number System (LNS) to that aim.

The paper is organized as follows. Section I gives some background on LNS with a focus on its former and current applications in ML. Using this representation and taking into account the constraints on the values induced by ML, Section III details the design and implementation of an "LNS neuron". Section IV explores the design space covered by this neuron, in particular the influence of its parameters on resource usage and top-1 accuracy. Finally Section V summarizes the results of this exploration, concluding that an LNS neuron is is more efficient than a fixed-point one for the same inference

X = (z X , s X , L X ) L X , L W , L P signed fixed-point logarithms, L X ≈ log b |X| s X a sign bit, s X = 1 when X < 0 z X a "isZero" bit, z X = 1 iff X = 0 m Most Significant Bit (MSB) of the fixed-point logarithm ℓ Least Significant Bit (LSB) of the fixed-point logarithm ℓ ′
LSB of the fixed-point sum in linear domain N number of terms to sum in the linear domain accuracy.

II. Background and State of the Art

A. Logarithmic Number Systems (LNS)

LNS is best viewed as an alternative to floating-point where the mantissa of a floating-point number is replaced with fractional bits of the exponent [START_REF] Detrey | A tool for unbiased comparison between logarithmic and floating-point arithmetic[END_REF]. A positive real X is represented by its logarithm L X ≈ log b X in some base b (usually b = 2 but other bases can be used [START_REF] Alam | Low-precision Logarithmic Number Systems: Beyond Base-2[END_REF]). L X is itself represented in signed fixed point. X = 1 is represented by

L X = 0, negative L X represent numbers X < 1, positive L X represent X > 1.
To represent negative values of X, an LNS system adds a sign bit s X . Finally, since log b (0) = -∞, the value X = 0 needs a special encoding in lg X. The mainstream approach is to use a "isZero" bit z X , but an encoding as a special value of L X can also be used. All these notations are summarized in Table I.

LNS is mathematically simpler and more elegant than floating-point. For base-2 LNS (b = 2), if L X has i integer bits and f fractional bits, the LNS system provides accuracy and dynamic range comparable to that of binary floating-point with i bits of exponent and f bits of significand fraction.

The main advantage of LNS arithmetic is that LNS multiplication of lg X by lg Y resumes to the fixed-point addition of L X and L Y . Being a fixed-point addition, it is exact, without any rounding error (contrary to floating-point multiplication that entails some rounding). It can over/underflow, though.

The huge drawback of LNS is the cost of addition and subtraction in this system. Without detailing it, let us just state that the hardware complexity of an LNS adder scales poorly with the precision [START_REF] Detrey | A tool for unbiased comparison between logarithmic and floating-point arithmetic[END_REF][START_REF] Vouzis | A Novel Cotransformation for LNS Subtraction[END_REF]. This has prevented its adoption as a mainstream format for general-purpose computing. A common option is to use approximate LNS adder/subtracters [START_REF] Mitchell | Computer Multiplication and Division Using Binary Logarithms[END_REF][START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF][START_REF] Kim | Low-power implementation of Mitchell's approximate logarithmic multiplication for convolutional neural networks[END_REF][START_REF] Arnold | Training neural nets using only an approximate tableless LNS ALU[END_REF]. The alternative approach proposed in this work is to use very low precisions (less than 8 bits) for which accurate LNS arithmetic is cheap. Here of course we mean: accurate with respect to the format, not accurate in absolute terms since the format precision itself is low. Indeed, for such precisions it is possible to define arbitrary functions in extension, then trust the synthesis tools to optimize their implementation. Thus the elegance of the format is not nullified by the architectural complexity [START_REF] Mitchell | Computer Multiplication and Division Using Binary Logarithms[END_REF][START_REF] Detrey | A tool for unbiased comparison between logarithmic and floating-point arithmetic[END_REF][START_REF] Arnold | On the cost effectiveness of logarithmic arithmetic for backpropagation training on SIMD processors[END_REF][START_REF] Vouzis | A Novel Cotransformation for LNS Subtraction[END_REF][START_REF] Miyashita | Convolutional neural networks using logarithmic data representation[END_REF][START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF] of implementing it in hardware.

More specifically, this work targets FPGAs whose fine-grain structure is that of a 4-to 6-bits Look-Up Table (LUT), and the present work aims at soft-spot formats that efficiently match these architectural LUTs. For this, a contribution of this work is to depart from off-the-shelf LNS formats, shaving as many bits as possible to define a purely application-specific format for machine learning inference.

B. LNS for machine learning

With the slide rule, logarithmic arithmetic predates electronic computers. It was studied for them very early [START_REF] Mitchell | Computer Multiplication and Division Using Binary Logarithms[END_REF][START_REF] Swartzlander | The Sign/Logarithm Number System[END_REF], with pioneering works [START_REF] Arnold | Implementing back propagation neural nets with logarithmic arithmetic[END_REF][START_REF] Arnold | On the cost effectiveness of logarithmic arithmetic for backpropagation training on SIMD processors[END_REF] In the era of deep learning, the potential of low precision logarithmic encoding was again demonstrated [START_REF] Miyashita | Convolutional neural networks using logarithmic data representation[END_REF] then put in practice [START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF] with an architecture (Figure 1b) that keeps the weights in the linear domain and implements multipliers as bit shifts. The final conversion to the logarithmic domain is approximated with a Leading One Detector (LOD). Another approach is to replace a standard multiplier with an approximate one based on LNS arithmetic [START_REF] Kim | Low-power implementation of Mitchell's approximate logarithmic multiplication for convolutional neural networks[END_REF][START_REF] Arnold | Training neural nets using only an approximate tableless LNS ALU[END_REF].

Figure 1 describes the various ways logarithmic encoding has been used in the recent literature. The approach we advocate (Figure 1d) is a simplified version of the "somewhat inaccurately named" (according to the author himself) Exact Log-Linear Multiply-Add (ELMA) approach [START_REF] Johnson | Rethinking floating point for deep learning[END_REF]. The term "exact" emphasizes the fact that both the addition of the logarithms and the summation of the product terms are exact, since both are performed as fixed-point additions. However there are still rounding errors in the log/linear conversions (in the b x and ReLU+log boxes of Figure 1d).

C. Originality and contributions

In the ELMA approach [START_REF] Johnson | Rethinking floating point for deep learning[END_REF], a product lg P may have integer and fractional bits: L P = I + F with I a binary integer and F ∈ [0, 1). In the present work, we make sure that 2 integer bits are enough (see Section III). In this case, we claim that with the wide LUTs of modern FPGAs, a plain tabulation of b x is more area-efficient than a table+shift approach: the shifter alone requires at least one FPGA LUT per output bit, just like the plain table. Plain tabulation also greatly improves the worst-case error, as in ELMA the b F rounding error may be amplified by the shift. Compared to previous work using just shifts [START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF] or a variant of Mitchell's approximation [START_REF] Mitchell | Computer Multiplication and Division Using Binary Logarithms[END_REF][START_REF] Arnold | On the cost effectiveness of logarithmic arithmetic for backpropagation training on SIMD processors[END_REF][START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF], plain tabulation even ensures correctly rounded logs and antilogs (i. e., as accurate as the format allows).

Finally, plain tabulation gives us the freedom to use the same architecture with b ̸ = 2, which has been shown to be beneficial [START_REF] Alam | Low-precision Logarithmic Number Systems: Beyond Base-2[END_REF], although the present study doesn't exploit this.

Similarly, in the trained data, the activation-weight products are consistently smaller than 1, therefore their logarithm is always negative, and the sign bit of logarithms can be saved: the fixed-point number stored in our approach for A ∈ {W, X, P } is actually -log b A. We also show later how we can dispense of the zero bit (and the MUXes that manage it) by making sure that some of the log values will be rounded to 0 for the summation by the b x box.

There are other complications in ELMA, such as tapered encoding and a few more architecture parameters. ELMA is also essentially a multiply-and-add operator that has to be used iteratively in an inference architecture. In our work the operator is a much coarser neuron, inputting in parallel N activation/weight pairs. Depending on the available resources and external memory bandwidth, the number of neurons and their number of inputs can be chosen so as to optimize the operational intensity [START_REF] Williams | Roofline: an insightful visual performance model for multicore architectures[END_REF] for a given FPGA. This allows supporting networks much larger than the considered FPGA can hold, at the price of external memory accesses and somewhat complex scheduling schemes [START_REF] Zhang | DNNBuilder: An automated tool for building highperformance DNN hardware accelerators for FPGAs[END_REF]. The value of N can also be chosen to directly match the needs of the various layers of small to medium neural networks and fits a mid-range FPGA -N = 784 allows for a fully parallel MNIST, N = 1152 allows for an efficient implementation of most layers in a reference CIFAR-100 implementation. This massive parallelism is available in current FPGAs (including the internal bandwidth needed to store intermediate activations and buffer the weights read from high-speed external DRAM), in particular when all values in use fit on few bits.

This approach enables a more area-efficient summation (the Σ box) thanks to the recent advances in bit array compression [START_REF] Kumm | Advanced Compressor Tree Synthesis for FPGAs[END_REF]. It is also more accurate thanks to a global error analysis.

Altogether, in ELMA the sizes of lg W and lg X 8 bits, and the sum size is 38 bits, whereas the formats we use are respectively 4 and 9 bits.

We have mentioned accuracy twice. It should be clear to the reader that the purpose of computing accurately is to reduce the size of the data format needed. It also helps avoiding the need for retraining.

Indeed, we do not address training in this work: we take networks pre-trained in floating-point [START_REF] Chen | Pytorch playground[END_REF] and quantize them to our application-specific format, building upon the observation by Lee et al. [START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF] that "Retraining of weights is necessary to enable a good model in linear 4 bits, but unnecessary for log 2 4 bits (...)". Of course, quantization-aware training can only improve the result, but this is left for future work.

III. Neuron design and implementation

A. LNS and sum of products

In both convolutional or fully connected layer, the mathematical operation performed by the neuron is the same: a sum of products. The main difference between these layer types

Σ B S in bias activation + log S out sfix(2, ℓ ′ ) L X ′ j ufix(m, ℓ) b -x + L X N L W N s W N b -x + L X 1 L W 1 s W 1 ufix(m, ℓ) ufix(m, ℓ) ufix(m + 1, ℓ) sfix(1, ℓ ′ ) • • • • • Fig. 2:
The proposed low-precision LNS neuron is the way they match inputs with weights, but in the end it always comes down to computing:

B + N ∑ i=1 X i × W i
For a linear layer, N is simply the number of neurons in the previous layer. For a convolutional layer, N is the number of weights in the kernel for this layer multiplied by the number of input channels (e.g.: 28×28 = 784 in the first layer of a Multi-Layer Perceptron (MLP) for MNIST, up to 3×3×512 = 4608 for a convolutional net on CIFAR). The scope of this work is focused on presenting a cheap hardware implementation of this operator (the so-called LNS neuron) that can be used in either layer type.

As mentioned before, focusing on FPGAs and low-bitwidth operands allows us to efficiently use the LUTs by tabulating complex functions. Three such functions are shown in figure 1d: b x , ReLU , log . They can be fairly costly to implement and the activation function (ReLU here) can change between networks (e.g.: sigmoid, tanh, ...). Tabulating them achieves a low implementation cost and flexibility and adaptability in the design. To minimize the cost of translating back and forth between log and linear domain, we fused the log block to the ReLU since they happen consecutively: the function tabulated is ⌊log b (ReLU (x))⌉.

Figure 2 shows an overview of a complete neuron architecture with its parameters. The next sections will explain the chosen encoding for each step of the process.

B. Distribution of weights and activations in deep learning

We consider networks which consist of a sequence of convolutional or fully connected layers, possibly with maxpooling layers between them. We ignore the latter as the encoding of activations is irrelevant in their operation (and they cost very little) and focus on the neural layers themselves (convolutional or fully connected). We evacuate the batch normalization layers, useful in training, as they can be removed for inference and their coefficients merged in the weights of the preceding layers (an approach suggested by the inventors of batch normalization [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF] and used in other studies, e.g. [START_REF] Johnson | Rethinking floating point for deep learning[END_REF]).

A few observations can be made on their trained weights and runtime activations.

Firstly, as already noted in [START_REF] Miyashita | Convolutional neural networks using logarithmic data representation[END_REF], the output of a ReLU is always positive. As all the activations (except in the input layer) come from a ReLU unit, they are positive (Figure 3). The activations in the input layers can be rescaled and shifted arbitrarily, in particular to ensure they are positive, too. Therefore there is no need to store a sign bit in lg X. Weights remain signed and need a sign bit (Figure 4).

We now show how to ensure that all the weights and activations remain smaller than 1 in magnitude, so that we can also spare the sign bits of their logarithmic encoding (the log of a value smaller than 1 is always negative). The constraints |X i | < 1 and |W i | < 1 would be relatively easy to integrate in the training, but we observe that it can also be ensured on trained networks by simple affine transformations of the inputs and weights, thanks to the fact that ReLU is piecewise affine.

Of course, any activation coming from a ReLU1 verifies 0 < X < 1 by design (Figure 3). If the network was trained with ReLU instead (currently more common case), the batch normalization is designed to force the mean and variance of each layer's output equal to 0 and 1 respectively, so large values of W i are very unlikely (see Figure 5). Now assume that a layer ensures X i < 1 (again this is easy to ensure for the input layer). The worst case run-time situation would be a neuron with all the activations equal to 1 for the positive W i , whereas activations for negative W i would all be zero (or the opposite case). In other words, the worst-case activation in absolute value is max(-

∑ Wi<0 W i , ∑ Wi>0 W i )
. This sum can be computed after training. In the networks we have studied, it does not exceed 25. It is thus possible to divide all the weights of the layer by a small power of two (in practice 8, 16 or 32) to ensure that the activations in the next layer will not exceed 1. In the logarithmic domain, this will consist in adding a small constant to all the L W . The biases

L W = -⌊log b |W |⌉ = w -1 w 0 w 1 w 2 s W L X = -⌊log b X⌉ = x -1 x 0 x 1 x 2 L P = -⌊log b |P |⌉ = p -1 p 0 p 1 p 2 p 3 s W binary weight 2 -1 2 0 2 1 2 2 2 3
Fig. 4: Logarithmic representation for inputs and product (here for MSB m = 2 and LSB ℓ = -1) have to be updated as well. All this preprocessing is performed in floating-point and only changes the weights and biases. It has no runtime cost, and no impact on classification accuracy since negative sums remain negative and positive ones remain positive, so the ReLU threshold is unchanged.

Another option would be to cap all the activations to 1 by replacing the ReLU with a ReLU1 (see [START_REF] Krizhevsky | Convolutional Deep Belief Networks on CIFAR-10[END_REF]Figure 3] for the effect of a ReLU1 on the activations of a layer without batch norm). This will require retraining, though.

C. Ad-hoc LNS formats for the weights and activations

With these observations, we have only negative logarithms for the weights and activations, and we may store the negated logarithm as an unsigned fixed-point format. The proposed logarithmic format for weights and activations is shown in Figure 4. It still has two parameters, the MSB m and the LSB ℓ of the parametric representation. L W is the correct rounding to the nearest of the exact logarithm of W (after the previous affine transformations). The ReLU+log component will also tabulate correctly rounded values.

With these formats, it should be clear on this figure that the product is computed exactly in the logarithmic domain as L P = L X + L W . The overflow bit is kept. Remember that we store negated logarithms: an overflow in this addition corresponds to the product of a very small X by a very small W that leads to an even smaller P . If this multiplication was performed in fixed point, it would be rounded to zero. If it was performed in floating-point, it would be rounded but the correct order of magnitude would be kept in the exponent of the result. In LNS, the encoding of the exact product of the input data costs only one bit more than these inputs.

It has been observed that weights can use smaller bitwidths than activations [START_REF] Mishra | WRPN: Wide reduced-precision networks[END_REF]. With the proposed approach, it will not save computation hardware (the addition hardware is the size of the wider format, and the smaller one must be padded with zero), but it may save memory and memory bandwidth.

D. Ad-hoc linear format for the exact sum

Some rounding will still be inflicted to our exact product, however, and this takes place in the b x box. Its output is again a fixed-point format with an MSB and an LSB. Since the product P is smaller than 1, the MSB is trivially 0. Actually, since the network has been preprocessed to guarantee that the output activations is smaller than 1 (in absolute value), the MSB of the sum itself is a sign bit at position 1 (weight 2 1 ).

The LSB, however, noted ℓ ′ in Figure 2, is kept open as a third parameter of the proposed approach.

E. Encoding of zeroes

LNS formats in the literature usually carry a zero bit, since log b (0) is not defined, all the more as 0 is an important value in neural networks, widespread across weights and activations. However, a good choice of the ℓ ′ parameter will allow us to discard that extra bit from our LNS encoding, for both lg X and lg W . Let us take an example to illustrate this: we will pick m = 2 and ℓ = -1, just as shown in Figure 4. The largest value that can be encoded in L X is then (111.1) 2 = (7.5) 10 . In base b = 2, it is the encoding of x = 2 -7.5 = (0,000 000 010 110...) 2 . The first significant bit happens at position -8. It means that for ℓ ′ ≥ -7, the b x function will round 2 -(111.1)2 to 0. Referring to Figure 2, we can also tell that if L X = (111.1) 2 or L W = (111.1) 2 , then L P ≥ (111.1) 2 ensuring that it will be rounded to 0 as well.

In a nutshell, the largest LNS value is in practice an encoding of 0, since when it is used, it will entail that the linear-domain product P is zero. Obviously, the higher ℓ ′ , the bigger the rounding errors in the b x conversion, negatively affecting the classification accuracy. However, picking ℓ ′ lower than or equal to the previous threshold also degrades the accuracy, since every 0 in the network now gets rounded to a small value, which leads to accumulation and classification errors, while costing more to implement.

F. Discussion on the impact of the format parameters

Figure 5 shows a histogram of the weights by layer of a (300, 100, 10)-MLP trained against MNIST. We can see that the |W | < 1 assumption is well met. The graph hints a normal distribution centered on 0. The vertical bars indicate where the LNS quantization would happen. We will later use the histogram of the first layer in other illustrations as it has the highest weight count and so the best visual representation of that distribution. We keep the parameters used in the examples so far because they are a good fit for this application. Notice that since X = 2 -L X , the quantization gets tighter around 0, but there is a gap with no value between 2 -(2 m+1 -2 ℓ ) (Figure 6 (left) is a zoom on the center).

Let us now take a closer look on the impact of m and ℓ on this quantization. Adding a bit of representation obviously doubles the number of available quantized values, but in a not so obvious way because of the LNS. Figure 6 highlights in red the additional values added to the quantization when adding one bit to:

• m (center): all the new values are located inside the gap previously mentioned. (Because L X = -⌊log b X⌉ =, so bigger L X means smaller X) • ℓ (right): similar to a standard linear quantization, it adds a value in between existing ones, with no influence on the central gap.

IV. Experiments

A. Setup

This work primarily relies on 2 frameworks:

• flopoco, an open-source VHDL generator written in C++, to produce the hardware design; • pytorch, an open-source Python machine-learning framework with a high-level interface. A neuron operator was added to flopoco. The classification accuracy of this neuron was evaluated with pytorch. A problem was that these frameworks are not fully available in the same language (pytorch offers some C++ helpers but only a portion of the interface is available). With the help of the pytorch's cpp extension, we wrote a wrapper to execute C++ code in a python environment. This way, the same C++ emulation function may be used by pytorch to compute the neuron output in a bit-accurate simulation, and by flopoco to build test benches for the VHDL neuron.

The first experiment was to implement a regular MLP to evaluate the proposed design on the MNIST benchmark (digit classification). This MLP consisted in 3 fully connected (FC) layers of 300, 100 and 10 neurons with no bias. The activation function for all hidden layers was ReLU1. The network was trained in standard 32 bits floating-point precision with pytorch against the MNIST dataset, split into a 60 000 samples training set and a 10 000 samples validation set. The learned parameters and input values were then converted to our custom LNS. We ran the bit-accurate simulation of the proposed LNS neuron against the validation set to evaluate the classification accuracy with pytorch. We also generated the VHDL code for the largest neurons of the network (in the first layer with 28 × 28 = 784 input pairs), and its implementation cost was obtained with Vivado 2021.2 for the target Kintex-7 7k70tfbv484-3 FPGA.

In a second experiment, we implemented a standard convolutional network for the CIFAR-10 image classification benchmark. This standard VGG-like network has been used in other works as baseline for quantization [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF], and pretrained weights for this architecture were found in the Pytorch Playground [START_REF] Chen | Pytorch playground[END_REF]. It consists in a succession of convolution layers with 3 × 3 kernels, batch norm and max-pool layers, and a final linear classifier. We merged the activation function in the neuron as previously described, and also merged the batchnorm layers into the weights (as suggested in [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF]). The modified architecture used here is detailed in table II.

There was one small issue: the original training used the ReLU activation function, which does not provide the hard cap at 1 that we expect for activations. To solve this without retraining, we statically computed the maximum activation a max achievable in the entire network with its set of weights. Assuming that the inputs are bounded by 1, a max is simply the maximum between the sum of the positive values of the weights and the sum of the negative values of the weights. Then we modified the floating-point weights and biases to be sure the output activations would be scaled down by a max . In the input layer, it required scaling both weights and bias:

a a max = B a max + ∑ i W i a max × X i . ( 1
)
In the hidden and output layers, only the bias requires scaling, since their inputs were already scaled down:

a a max = B a max + ∑ i W i × X i a max . ( 2
)
All this was performed on the pretrained weights, before conversion to LNS. We explored three different values for m. As m = 1 yields significantly lower accuracy, it was not considered further. This poor accuracy is understandable considering the associated representation space. Indeed, with ℓ being negative, the maximum value for L X or L W becomes 2 2 -2 ℓ ≈ 4, hence X ≈ 2 -4 = 0.0625. It means that the proposed encoding rounds every weight and activation in the ] -0.0625, 0.0625[ interval to 0. This is a poor match to the weight distribution shown in figure 5, especially in the first layer.

Figure 8 shows that m = 3 is not much better in terms of accuracy than m = 2, however its hardware cost is significantly higher (see Figure 9).

C. Hardware implementation

Figure 9 shows how the area evolves with the neuron parameters, using a simple model that estimates the cost of a 2 p × q table (as used for the b x and ReLU+log blocks) as 2 p-6 × q LUT, and counts 0.55 LUT per bit added in the summation (see [START_REF] Kumm | Advanced Compressor Tree Synthesis for FPGAs[END_REF] for the tree implementation details). This model is quite accurate, for instance Vivado synthesized the ((2, -1), (1, -6)) LNS neuron in 12,491 LUTs where our estimations were 12,890 LUTs. Further comparisons showed that the estimation is about 3% larger than the synthesis. Since the synthesis is computationally demanding, the plot shows the estimated area.

These figures can help us chose the best parameter configuration for our application. In our case for MNIST, we wanted to reduce the number of bits required for the weights and activations as much as possible to help with the memory bandwidth issue, while keeping an acceptable accuracy. This lead us to pick m = 2, ℓ = -1 and ℓ ′ = -6 for our previous examples.

Table III provides some actual synthesis results on Kintex7 using Vivado. We also merged here some of the best classification accuracies we could achieve and their configuration. The hardware cost of the lines referred to as "x-bit linear" relies on a standard integer multiplier that feeds its output to the same summation operator than the one used in our LNS neuron. Their associated classification accuracy is obtained with [START_REF] Chen | Pytorch playground[END_REF], and the number presented here is the top-1 accuracy ratio. This way we can compare to accuracy equivalent linear quantization. The results clearly shows that LNS neurons can achieve a similar accuracy with fewer bits, and a much cheaper hardware cost.

Since this work does not yet provide a complete neural network accelerator, the purpose of this table is merely to show that the proposed neuron fits the target FPGA.

V. Conclusion and future work

This work proposes a low-bitwidth LNS representation specifically designed for ML inference. With a simple weight conversion without retraining, the accuracy loss compared to the float32 reference can be negligible, using smaller bitwidths than when using fixed-point formats. This enables an efficient FPGA implementation: thanks to the small number of bits, the difficult computations can be tabulated and densely packed into FPGA lookup tables. Compared to a fixed-point neuron with comparable classification accuracy, the proposed LNS neuron features lower resource usage.

As a next step, we plan to build a complete accelerator and to study low-bitwidth LNS-aware training. A simple retraining approach giving encouraging results for a small number of bits has been proposed by [START_REF] Lee | LogNet: Energy-efficient neural networks using logarithmic computation[END_REF]. Several other authors have proposed full training strategies [START_REF] Miyashita | Convolutional neural networks using logarithmic data representation[END_REF][START_REF] Arnold | Training neural nets using only an approximate tableless LNS ALU[END_REF][START_REF] Sanyal | Neural network training with approximate logarithmic computations[END_REF][START_REF] Zhao | LNS-Madam: Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update[END_REF], but for bitcounts over 8. [START_REF] Chmiel | Logarithmic Unbiased Quantization: Practical 4-bit Training in Deep Learning[END_REF] proposes an approach using 4 bits, but fixes the representation, while our neuron is parametric, which will offer more flexibility. For such a study, the proposed approach also has the potential to use a different base b and an arbitrary activation function, provided that the training ensures that weights and activations remain below 1.

Fig. 1 :

 1 Fig. 1: Artificial neurons using log-encoded data (in red)

Fig. 3 :

 3 Fig. 3: ReLU (left) and ReLU1 (right) activation functions

Fig. 5 :

 5 Fig. 5: Quantization indices for m = 2 and ℓ = -1.

Fig. 6 :

 6 Fig. 6: Influence of parameters on quantization.

Fig. 7 :Fig. 8 :Figure 7

 787 Fig. 7: Influence of parameters on MNIST accuracy

Fig. 9 :

 9 Fig. 9: Parameters influence on hardware cost of a neuron for N = 784 (the smallest Virtex-7 FPGA contains more than 360,000 LUTs).

TABLE I :

 I Notations used in this article

	Symbol	Meaning
	b	the base of logarithms used, usually b = 2
	X, W , B, P	real values of input, weight, bias and W X product
	lg X, lg W , lg P	their LNS representation, e.g. lg

  demonstrating than 12-bit LNS could compete with floating-point in neural network training.

	W X		×	P		Σ	S	ReLU	X ′
				(a) All-linear or all-float
	W	≪	Σ		ReLU		LOD	lg X	′
	lg X
		(b) Log activations, linear weights [18]
				max
	( lg W , lg X)	≪ -	lg S	logReLU	lg X	′
				+
	(c) Log activations, log weights, approx log sum [18]
	lg W lg X	+	lg P	b x	P		Σ	S	ReLU+log	lg X	′
	(d) Proposed: log activations, log weights, accurate linear sum [25]

TABLE II :

 II VGG-like convolutional network

	layer index layer type
	(1) LN SConv(3, 128) + ReLU 1()
	(2) LN SConv(128, 128) + ReLU 1()
	(3) M axP ool2d(2, 2)
	(4) LN SConv(128, 256) + ReLU 1()
	(5) LN SConv(256, 256) + ReLU 1()
	(6) M axP ool2d(2, 2)
	(7) LN SConv(256, 512) + ReLU 1()
	(8) LN SConv(512, 512) + ReLU 1()
	(9) M axP ool2d(2, 2)
	(10) LN SConv(512, 1024) + ReLU 1()
	(11) M axP ool2d(2, 2)
	(12) LN SLinear(1024, 10)

TABLE III :

 III Synthesis results for accuracy-equivalent parallel neurons. (m, ℓ) = (2, -1) means that weights fit on 5 bits and activations fit on 4 bits (see Fig.4).

	benchmark	parameters	top-1 accuracy	cost	latency
		(m, ℓ), (1, ℓ ′ )	abs.	ratio	in LUTs	in ns
	MNIST	float32	98.03	100	-	-
	MNIST	(2, -1), (1, -6)	97.64	99.6	12491	10.3
	MNIST	(2, -1), (1, -7)	97.83	99.8	13790	10.9
	MNIST	6-bit linear	97.93	99.9	36658	10.2
	CIFAR10	float32	93.74	100	-	-
	CIFAR10	6-bit linear	90.83	96.9	51910	13.0
	CIFAR10	(3, -1), (1, -11) 91,40	97.5	30632	12.8
	CIFAR10	(2, -2), (1, -10) 92.33	98.5	28652	12.4
	CIFAR10	8-bit linear	93.55	99.8	83522	13.4

VI. Acknowledgements

The authors would like to acknowledge the financial support of the French Agence Nationale de la Recherche (ANR) through the MIAI@Grenoble Alpes ANR-19-P3IA-0003 grant and the Imprenum project.