Romain Orhand

Hiba Khodji

Amarin Hutt

Anne Jeannin-Girardon

Conference on Knowledge-Based and Intelligent Information & Engineering Systems Quantification of the transferability of features between deep neural networks

Keywords: transfer learning, feature transferability quantification, convolutional neural networks, deep learning

The computationally expensive nature of Deep Neural Networks, along with their significant hunger for labeled data, can impair the overall performance of these models. Among other techniques, this challenge can be tackled by Transfer Learning, which consists in re-using the knowledge previously learned by a model: this method is widely used and has proven e↵ective in enhancing the performance of models in low resources contexts. However, there are relatively few contributions regarding the actual transferability of features in a deep learning model. This paper presents QUANTA (QUANtitative TrAnsferability), a method for quantifying the transferability of features of a given model. A QUANTA is a 2-parameters layer added in a target model at the level at which one wants to study the transferability of the corresponding layer in a source model. Data from the target domain being fed to both the source and the target models, the parameters of the QUANTA layer are trained in such a way that a mutually exclusive quantification occurs between the source model (trained and frozen) and the (trainable) target model. The proposed approach is evaluated on a set of experiments on a visual recognition task using Convolutional Neural Networks. The results show that QUANTA is a promising tool for quantifying the transferability of features of a source model, as well as a new way of assessing the quality of a transfer.

Introduction

Deep Neural Networks are praised for their ability to extract hierarchical features from data and used for solving various recognition tasks such as image recognition [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF][START_REF] Ker | Deep learning applications in medical image analysis[END_REF], speech recognition [START_REF] Deng | New types of deep neural network learning for speech recognition and related applications: an overview[END_REF], natural language processing [START_REF] Young | Recent Trends in Deep Learning Based Natural Language Processing [Review Article[END_REF], bioinformatics [START_REF] Tang | Recent Advances of Deep Learning in Bioinformatics and Computational Biology[END_REF], etc. Such models are hungry for resources, either computation-wise or data-wise. Lack of computing resources induces an important training time for a model. Training time can be reduced by reducing the number of trainable parameters, possibly at the cost of no longer being able to capture subtle underlying complexity within the data. Obviously, acquiring more computing power also alleviates training time. Lack of data can be addressed by augmenting the training dataset [START_REF] Mikołajczyk | Data augmentation for improving deep learning in image classification problem[END_REF] or obtaining more training data. In both cases, and in some application domains, neither options are possible: an example is the context of studying rare disease genotypes for which, by definition, only few data are available. Either way, resource management is a major stake in machine learning and lacking either of these resources causes the performance of the model to drop.

An alternative answer to the problem of resource management is to re-use the knowledge previously learned by a model into another model, a method commonly known as transfer learning. Transfer learning is a machine learning paradigm focusing on applying knowledge gained from a source model to an untrained target model in order to [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF] reduce training time or [START_REF] Chowdhury | Instance-based Inductive Deep Transfer Learning by Cross-Dataset Querying with Locality Sensitive Hashing[END_REF] minimize the amount of data required for training, and (3) improve the performance of a model on a target task. If objectives [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF] or [START_REF] Chowdhury | Instance-based Inductive Deep Transfer Learning by Cross-Dataset Querying with Locality Sensitive Hashing[END_REF] and objective (3) are achieved, the transfer is considered a success. If the performance of a model, post-transfer, is lower than its performance without transfer, then the transfer is deemed unsuccessful: this phenomenon is known as negative transfer, and is usually caused by a dissimilarity between the source and the target domains that can negatively impact the performance on the target task [START_REF] Wang | Characterizing and avoiding negative transfer[END_REF]. A promising direction for avoiding this issue and to e↵ectively apply transfer learning is to address the following questions [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF]:

• "What to transfer": is it possible to determine which features or parameters should be transferred from a trained source model in order to improve the performance of a model on a target task? • "When to transfer": the similarity of the source and the target domains should be assessed in order to avoid a negative transfer. • "How to transfer": what actual methods should be used to carry out the transfer.

Given the availability of data and the (dis)similarity between both source and target domains and tasks, transfer learning problems can be categorized into three groups [START_REF] Chowdhury | Instance-based Inductive Deep Transfer Learning by Cross-Dataset Querying with Locality Sensitive Hashing[END_REF][START_REF] Tan | A Survey on Deep Transfer Learning[END_REF][START_REF] Pan | A Survey on Transfer Learning[END_REF]: Inductive Transfer Learning, where the target task is di↵erent-but-related-to the source task and the source and target domains may or may not be the same; Transductive Transfer Learning, where the target domain is di↵erent-but-related-to the source domain and the target task to learn is the same as the source task. Such transfer applies when the feature spaces between the source and target domains are di↵erent or when they are similar but the marginal probability distributions are di↵erent (Domain Adaptation). Lastly, Unsupervised Transfer Learning is an instance of Inductive Transfer Learning where no labelled data are provided in either the source or the target domain.

Within the context of Homogeneous Transfer Learning (that is, when the feature spaces of the source and the target domain are the same), di↵erent approaches can be considered in order to address the "what to transfer?" question [START_REF] Pan | A Survey on Transfer Learning[END_REF]: training instances from the source domain can be weighted to use only training data having a positive influence in the target domain; new representations can be learned in order to fill the gap between the source and the target domains; the commonalities between the source and the target domains can be learned and used to learn to solve the target task. The present study focuses on Parameter-based transfer: this approach, probably the most widely used, assumes that the parameters learned by the source model can be transferred to the target model. This process occurs at the layer level and the transferred parameters can either be frozen (i.e. un-altered during the target model training process) or fine-tuned (i.e. adjusted during the target model training process).

In the context of a Homogeneous, Inductive and Parameter-based transfer, this paper introduces QUANTA (QUANtitative TrAnsferability), a tool to evaluate the transferability of models at the layer level. The primary purpose of this tool is not to provide a methodology for transferring knowledge per se, but rather an analytical approach that would allow to further address the questions "what" and "when" to transfer.

This paper is organized as follows: the next section presents related works focusing on methodology around transfer learning. QUANTA are introduced in section 3, before experimental results are detailed and discussed in section 4. Finally, conclusions and future research directions are highlighted in section 5.

Related works

In the following, the works described were conducted in the context of Homogeneous, Inductive and Parameterbased transfer learning for visual recognition tasks.

The seminal paper [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF], that promoted the study of feature transferability in deep neural network, aimed at addressing the question of which learned features can be considered general (and thus, transferable), or specific to a given domain and task. The transferability of features is measured as the accuracy of the target model after the transfer of the first n layers. Both source and target domains are the same (D S = D T ), and two sorts of transfer were done: one where both source and target tasks are the same (T S = T T ) and one in which the target task di↵ers from the source task (T S , T T ). Several observations were made:

• Parameters from low-level layers are general and transferable.

• Transferring higher-level layers impairs the performance of the target model. These layers are more taskspecific. • Fine-tuning the transferred parameters results in better performance, especially when the transferred parameters are task-specific. • Features become less transferable as the distance between the source and target tasks increases.

• Transferring features, even from a model trained on an unrelated task, leads to better overall performance.

• The presence of learned co-adapted features in a model (that is, features which interact with each other over several layers and that cannot be re-learned by the upper layer alone) has a negative impact on transferability. Those features were highlighted by a performance drop followed by a recovery, within the network.

Most studies related to Transfer Learning aim to improve the performance of deep learning models, for instance by highlighting best practices in terms of model architecture, hyperparameters, etc., when applying transfer learning [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF]. More specific works focus on data selection for training, for instance in the case of source-target joint training such as [START_REF] Ge | Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-Tuning[END_REF]: target training instances are selected when their low-level features are highly similar to those of the data from the source domain, for which abundant data is available. The pre-trained and the target models are jointly fine-tuned using only the selected data (this process is referred to as selective joint fine-tuning).

The work presented in [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF] concerns the use of learned mid-level representation and how they can be applied for solving a target task di↵erent from the source task (i.e. whose label space is di↵erent from this of the source task). To do this, they transfer and freeze all mid-level representations (including those learned by the fully-connected layers usually found near the output of a Convolutional Neural Network) in the target model, and a fully-connected adaptation layer is added in order to adapt those representation to solve the target task. CactusNet [START_REF] Collier | Cactusnets: Layer applicability as a metric for transfer learning[END_REF] is an approach developed in order to apply learned features to a given piece of data, based on the hypothesis that a model has an individual response (or applicability) for each datapoint. The idea is to grow a pre-trained source model by creating branches from the most applicable features in order to specialize the model to the target task by learning higher-level features that are not yet known in the target data. This approach also has intrinsic applications regarding unsupervised learning and multi-task learning.

Cross-stitch networks [START_REF] Misra | Cross-stitch networks for multi-task learning[END_REF] are used in the context of multi-task learning as well. These models learn linear combinations of shared and task-specific representations between multiple models using cross-stich units (CSU) that combine activations maps from the models and use these combinations as inputs for the next layer of each model. Parameters Transfer Units (PTU) [START_REF] Zhang | Parameter Transfer Unit for Deep Neural Networks[END_REF] also apply combinations between activations maps, and those combinations are non-linear. Data from the target domain are fed to both the (frozen) source and the target model and the combinations learned by the PTU are fed only to the target network. The overall specification of a PTU exhibits more complexity than that of a CSU because a PTU is made of two distinct gates, used to (1) adapt source activations to the target domain and (2) combine target activations with the adapted source activations. It turns out that not only PTU are used for transferring parameters from the source to the target model, but their gates also provide some insights regarding the actual transferability of these parameters (i.e. to which extent parameters learned by the source model can be re-used by the target model): while this transferability is usually expressed in terms of the target model accuracy, PTU provides an "absolute" value that is not expressed as a function of the performance of the target model.

QUANTA is conceptually close to both CSU and PTU in the sense that it performs combinations of elements from the source and the target model. Beside the fact that the primary purpose of QUANTA is to evaluate the transferability of layers (rather than actually perform a transfer), it di↵ers from CSU et PTU in three ways:

1. QUANTA does not combine activations maps but the weighted inputs of layers. 2. This combination is less complex and it involves adding only two more parameters in the model.

3. Most importantly, the combination between the source and the target models is mutually exclusive (i.e. during training, a choice has to be made between the two models) and the expression of this combination provides a metrics that is not a function of the performance of the target model.

These three points are detailed in the next section.

3. Quantitative measure of layer transferability with QUANTA Before diving into the technical specification of QUANTA, it is useful, first, to detail the intuitions and ideas behind this proposition.

A source model S is trained on data drawn from D to solve a classification task T S . The objective is to train a model T on data also drawn from D, to solve a classification task T T (T T may or may not be the same as T S ), while re-using the features learned by S as much as possible (but not at the expense of the performance of T ).

Assessing to which extent the features of S can be used for solving T T comes down to feeding the data drawn for T to S , use an output layer adapted to task T T to compute the predictions, and check the accuracy of S : this assessment is basically an inductive paramater-based transfer of all the hidden layers of S .

In order to assess how a particular hidden layer n of S can be re-used for solving T T , both models S and T must be considered at the same time: models S and T have the same architecture, except for the output layer, which is specific to the task each model must solve; the parameters of T are initialized using a legacy initialization scheme such as Xavier's and they are trainable. At no point in the process are the parameters of the first n layers of S copied into T : S parameters are frozen and, while training T , the training instances are fed simultaneously to S and T .

In T , the activation of layer n is computed as a function of the weighted inputs of layers n of S and T : A n,T = f (Z n,S , Z n,T ) and the signal keeps propagating until T outputs a prediction: this process is depicted on figure 1. In the case of a "regular" inductive parameter-based transfer, the activation of layer n in T only uses Z n,S . Otherwise, Z n,S and Z n,T can be weighted: if Z n,S has an important weight and the prediction of T is wrong, the weight of Z n,S must decrease. At the same time, the parameters of T are also adjusted and, if the classification error of T is reduced, their weight increases. Consequently, T has two more parameters to learn: the weights v n,S and v n,T respectively applied to Z n,S and Z n,T . These two parameters express to which extent the parameters of S can be re-used or if T should use its own (learned) parameters.

" #,% & #,% " #,' QUANTA Layer - -+ 1 --1 … … … … Target data 1 2 % 1 2 ' Source Target & #,'
Both parameters are implemented in a layer (a QUANTA layer) at the n th layer within the target model T in order to compute A n,T = f (Z n,S , Z n,T ). Defining:

⇤ n = n,S n,T ! = softmax v n,S v n,T
! , the activation of layer n of the target model is now expressed as:

A n,T = f ( n,S Z n,S + n,T Z n,T ), (1)
where f () is an activation function. The weights v n,• are updated using the prediction error C T of T :

v n,• = v n,• ⌘ v @C T /@v n,• .
Being computed using a softmax function, the scalars n,S and n,T range within [0; 1] and n,S + n,T = 1. Considering the source model, the scalar n,S denotes the degree to which the source parameters can be used as-is so that the target model can solve its task. Thus, this value indicates how much the n th layer of the source model is transferable: a value close to 1 (resp. 0) means that the source parameters are transferable (resp. not transferable). The scalar n,T , associated to the target model, acts as a learning adjustment inversely proportional to the transferability n,S and it provides information about the degree of modification that should be applied to the transferred parameters. Using the softmax function to compute ⇤ n makes S and T mutually exclusive: if the features of S are not adapted then T must compensate by learning other features, and the parameters v n,• change according to how much of a layer n from the source model S can be expressed through the activated output A n,T of the target model T .

When applying transfer learning, the ideal situation is to maximize the use of the features of S to solve a target task T T . However, in the context of an inductive parameter-based transfer, the parameters of a set of layers are merely copied into the target model and possibly fine-tuned to adapt them to T T . Could the actual transferability of features, layer-wise, be evaluated? Or, to put it another way, is it possible to evaluate the quantity of features that can be re-used to solve a di↵erent task? QUANTA is an attempt at such an evaluation, through the combination computed between S and T . However, given the intrinsic complexity of deep neural networks, it would be excessive to claim that a QUANTA layer maximizes the use of features of S for solving T T .

That being said, in order to shift the system towards prioritizing the use of these features over the features that T would learn from scratch, the initial value of v n,S and v n,T are set such that n,S ⇡ 0.9 and n,T ⇡ 0.1. Since the fundamental idea is to reuse the features learned by S , the target model should be prevented from learning, de novo, features for solving T T if these are already known by the source model (and thus, transferable). The proposed initialization is expected to have the following e↵ect: if, initially, compelling T to use Z n,S gives rise to a significant error on a given input from the target dataset, n,S decreases because its associated error gradient is more important (in other words, the features that T attempted to use were not relevant for solving T T ). Since n,S and n,T are bound by a softmax, n,T will increase proportionally, meaning that T adjusts the features learned by S . Initially maximizing the use of Z n,S prioritizes S over T and, as the training process goes on, T compensates for unadapted features by adjusting them. The influence of the initial values of the QUANTA layer parameters is discussed in the next section, in which experimental results are presented after the protocol for running these experiments is described.

Experimental results

The experiments presented thereafter aim at providing inputs to the following questions:

(1) Does the inclusion of a QUANTA layer in a model impair the learning process of the model? (2) Does the parameter n,S provide some information regarding the transferability of the layer n, from the source model, to solve T T ? (3) What is the influence of the initial value of ⇤ n ? (4) How can n,S actually be interpreted?

Before discussing these questions, the experimental protocol is presented in the next section.

Experimental protocol

The data used to conduct the experiments with QUANTA are the CIFAR-10 and the CIFAR-100 datasets [START_REF] Krizhevsky | Learning Multiple Layers of Features from Tiny Images[END_REF] (both are built from the Tiny Image dataset). CIFAR-10 contains 60 000 32⇥32 color images split into 10 classes of 6000 images each. The dataset contains 50 000 training images and 10 000 testing images. CIFAR-100 is very similar to CIFAR-10, except that the images are split into 100 classes of 600 images each. Each class has 500 training images and 100 testing images. The labels of CIFAR-10 and CIFAR-100 do not overlap, so the tasks associated to each dataset are di↵erent (the label spaces are di↵erent).

CIFAR-10 is the dataset used to train the source model (source task A = T S ). B is the target task corresponding to the CIFAR-100 dataset. To test the QUANTA system, two sets of experiments have been defined: experiments AnB (resp. AnA) refer to the measure of the transferability of layer n in the source model when the target model attempts to solve task B (resp. A). Experiments AnA can be seen as control experiments, since it is expected that re-using the features learned by S to solve the same task it was trained on would lead to a successfully trained target model.

The architecture of the convolutional neural networks used is depicted on figure 2: both the source and the target model have the same architecture (except for the output layer, adapted to the task the target model should solve) made of three convolutional blocks (each block contains two convolutional layers) computing, respectively, 64, 256 and 512 3⇥3 feature maps. A fully connected layer with 2048 units serves as a classifier and a final layer computes the output for the task at hand. This last layer is not meant to be transferred, which means that there are 7 layers, in the source model, whose transferability can be assessed (the blue layers on figure 2). It has been empirically observed that complex Convolutional Neural Networks are prone to learning co-adapted features [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF][START_REF] Gumbira | Does fragile co-adaptation occur in small datasets?[END_REF], so a larger than necessary model has been used to test the ability of QUANTA to highlight learned co-adapted features.

The source model was trained during 60 epochs on CIFAR-10 using 32-instances batches. Training data was augmented with random rotations, pixel shifts, image crops, zooms and symmetries. The applied learning rate was 10 4 . The training process was optimized with the Adam optimizer (weight decay: 10 6 ) using a cross-entropy categorical loss. This configuration is the same for training the target model T .

An experiment is defined as the assessment of the transferability of layer n in the source model when the target model T attempts to solve task A or B. For a given target task, there was thus 7 experiments (that is, one experiment per transferable layer). Each experiment was repeated 30 times. During each run, and at the end of each training epoch, T is evaluated on the testing data and the following metrics are collected: the transferability n,S of layer n, the top-1-accuracy of the target model as well as its loss, its precision and recall and its top-3-accuracy. For a given run, the final metrics are computed as the average of the last 10 recorded metrics (in order to take into account the stochastic variability of the evaluation from one epoch to another). Then, the final metrics are averaged over the 30 runs of the experiment. Experiments were run for three di↵erent initial values of n,S : 0.90, 0.75 and 0.50.

A common way to check on the learning process of a model is to examine its loss and top-1-accuracy during training. Figure 3 depicts the evolution of those metrics, for init n,S = 0.90: for both task, the accuracy increases while the loss decreases, thus confirming that integrating a QUANTA layer within the target model did not impair the training process of this latter.

The results presented in the rest of this section are graphs depicting the measured transferability per layer (black dots) along with the top-1-accuracy associated to the layer (blue crosses -the dashed line is the top-1-accuracy of the source model on task A). Firstly, the overall dynamics of the measured transferability with init n,S = 0.90 is presented (section 4.2). More experiments are detailed in section 4.3 to try and characterize the influence of init n,S .

Dynamics of the measured transferability

Figure 4 presents the dynamics of the measured transferability (fig. 4a and4b) for tasks AnA and AnB, compared with the seminal results of [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] (fig. 4c) in which the transferability is expressed as the top-1-accuracy of the target model after transferring the first n layers. The transferability as expressed by QUANTA does not depend on a performance metric: this is why the dynamics of the transferability is looked at (that is, how it evolves according to the layer considered).

It should be noted that the experiments presented in [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] were conducted on the ImageNet dataset with a di↵erent CNN architecture. Since it is acknowledged that hierarchical features learned by CNN are more specific as the considered layer is deeper, it can be assumed that the dynamics showed in [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] can be generalized to most image datasets (PTU [START_REF] Zhang | Parameter Transfer Unit for Deep Neural Networks[END_REF] exhibited this dynamics when evaluating a transfer between CIFAR-10 and CIFAR-100).

For task AnA (fig. 4a), the dynamics of QUANTA over the layers is the same as the dynamics of model selfer BnB of fig. 4c: a drop of transferability, followed by a recovery, is observed at layer 3. This can be interpreted as the learning of co-adapted features by the source model. Regarding task AnB (fig. 4b), a significant transferability drop occurs at level 6: this was expected since the tasks are di↵erent: the classifier at the end of the source model cannot be re-used as-is to solve task T T = B. This dynamics is also the same as the dynamics of fig. 4c (transfer AnB).

In fig. 4c, transfer AnA+ and transfer AnB+ show the target accuracy when fine-tuning is applied to the transferred parameters. In both cases, the accuracy is maintained and constant. In the case of QUANTA (fig. 4a and4b), the target accuracy is also maintained and constant: for task AnA, the target accuracy (0.8488 -averaged over all layers) is at the level of the source accuracy (0.84). This was expected since the training data for T and S are the same and T S = T T . Regarding task AnB, the average target accuracy over all layers is 0.5743 (and, layer-wise, this accuracy drops slightly as n gets closer to 6). This is far below the source accuracy but this was also expected since (1) the source network is fed the target training data to pass Z n,S to the target model and (2) QUANTA does not apply fine-tuning.

Influence of the QUANTA parameters initial values

The experiments presented hereafter were conducted with two more values of init n,S : 0.75 and 0.50. Figure 5, top row, shows the measured transferability and the target accuracy for these initial values. For task AnA (resp. AnB), regardless of the initial value of init n,S , the target accuracy is ⇡0.84 (resp. ⇡0.57); this holds true for the other collected metrics (data available in the supplementary materials). This observation suggests that it is indeed possible to prioritize (if not maximize) the use of the features learned by S to solve a target task and a lower measured transferability would indicate that the target learned more features from scratch.

The bottom row of fig. 5 illustrates the evolution of n,S during training. The following comments can be made: using init n,S = 0.50 (fig. (f) and (h)) means that initially, no model is favored over the other. In this case, and regardless of the task, n,S increases during at least 10 epochs: the system attempts to use the features learned by S instead of relying on the randomly initialized parameters of T . This observation is in line with the experiments of [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] suggesting that transferring parameters from a not so related task leads to a better top-1-accuracy than starting training from random parameters. It is interesting to note that, for task AnB, the system first reinforces the use of the parameters of layer 6 before giving up because this layer is too specific to the source task.

When init n,S = 0.75 (fig. (e) and (g)), the system is initially compelled to apply the features learned by S . However, n,S quickly decreases, which means that the prediction error originates mostly from the source and that reducing n,S reinforces the adjustment made by the target.

To sum up, the results presented above provide some insights regarding the questions raised at the beginning of this section. Firstly, including a QUANTA layer in a model, whichever the layer, do not impair the learning process of the model: this latter minimizes its cost function and metrics such as the top-1-accuracy, the precision, the recall, etc. improve during training.

The dynamics exhibited by n,S over the n layers of the source model is coherent with the degree of genericity or specificity of the layer, as observed in previous studies such as [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF][START_REF] Zhang | Parameter Transfer Unit for Deep Neural Networks[END_REF][START_REF] Gumbira | Does fragile co-adaptation occur in small datasets?[END_REF]. Moreover, the results also highlighted the presence of learned co-adapted features. Those two points denote the ability of n,S to express a degree of transferability of a layer n.

However, this degree of transferability is not yet clear: indeed, while di↵erent initial values of n,S led to the same overall top-1-accuracy, the lower init n,S , the lower the final measured transferability. Nonetheless, this seems to indicate that a higher initial value of init n,S allows prioritizing the use of features learned by S to solve a target task while preventing T from learning these features from scratch, thanks to the mutually exclusive quantification computed by QUANTA. Now, the question is to interpret the final value of n,S for a given layer. Assuming the computed n,S of layer n = 0 for task AnB is 0.97, it seems incoherent to state that 0.97 ⇥ layer n can be transferred while the top-1-accuracy of this layer in the target model for this task barely reaches 0.60: the "actual" transferability may actually be a function of both the measured n,S and the accuracy of T when transferring n. Fruther investigations are required in order to better frame the interpretation of this measured transferability and to figure out how this metric can be exploited in practice.

Further considerations about this work also include the fact that, as it is, it is constrained to neural network architectures that are the same for both tasks.

Conclusion

While transfer learning is a well-established method to deal with the lack of labelled data and/or the lack of computing resource, it is still tricky to anticipate which parameters inferred on a source task are re-usable to solve a (related) target task and to which extent these parameters should be adjusted in order to solve this target task. The system QUANTA (QUANtitative TrAnferability) introduced in this paper is an attempt at providing a quantitative measure of transferability of the parameters of a trained deep neural network. To evaluate the transferability of the first n layers of a source model to a target model, QUANTA combines the weighted inputs of both the source and target models to computes the target activation of the considered layer. This combination is mutually exclusive, so if a layer is not transferable, the system indicates what amount of adjustment is required to compensate the lack of transferability. QUANTA was tested on a visual recognition task using Convolutional Neural Networks and the datasets CIFAR-10 (source task) and CIFAR-100 (target task). Not only is the learning process of the target model una↵ected by the inclusion of QUANTA, but the observed dynamics of the measured transferability is consistent with previous studies that measured the transferability of a source model as the accuracy of the target model post transfer. These encouraging results must now be reinforced by further study, firstly by conducting more experiments (using other data and tasks, possibly other than visual recognition tasks) in order to better frame the interpretation of this measured transferability. Other future works include studying the transferability of multiple source models or even to attempt tranferring knowledge learned on heterogenous data for solving a common task.

Fig. 1 :

 1 Fig. 1: Implementation of a QUANTA layer in the target model T : the activation of the n th layer in T is expressed as a weighted combination of the weighted inputs of layers n of both T and the source model S .

Fig. 2 :

 2 Fig. 2: Convolutional Neural Network architecture used for the source and the target models. The layers depicted in blue are the layers whose transferability can be assessed.

Fig. 3 :Fig. 4 :

 34 Fig. 3: Accuracy (a, c) and loss (b, d) of the target model during training; init n,S = 0.9

Fig. 5 :

 5 Fig. 5: Top-row: comparison of the final transferability for tasks AnA and AnB, layer-wise, for di↵erent values of init n,S . Bottom-row: evolution of n,S during training.


Yassin Kortli
email: yassin.kortli@isen-ouest.yncrea.fr

Souhir Gabsi
email: souhir.gabsi@fsm.rnu.tn

Maher Jridi
email: maher.jridi@isen-bretagne.fr

Ayman Alfalou
email: ayman.al-falou@isen-ouest.yncrea.fr

Mohamed Atri
email: matri@kku.edu.sa

Hw/Sw Co-Design Technique for 2D Fast Fourier Transform Algorithm on Zynq SoC

Keywords: 2D-FFT algorithm, Embedded systems, Hw/Sw Co-design, High-level synthesis, Complexity

The Two-Dimensional Fast Fourier Transform (2D-FFT) algorithm is used for the study of many modern systems applied for security and biometrics. The adoption of this algorithm, which is a compute intensive task, is limited due to its hardware design complexity. The first objective of this paper is to underline the effect of the hardware/software co-design (Hw/Sw co-design) for the reduction of the processing time and power consumption. Secondly, we propose an innovative architecture for the 2D-FFT algorithm tested on Zynq Soc, which requires less processing time and memory compared to the traditional algorithm. Three implementations (one software and two Hw/Sw co-designs) of the 2D-FFT algorithm using the Zynq SoC are presented in this paper. The first is based on ARM processor. A speedup of 29x is obtained compared to the original implementation thanks to many optimizations. The second is a Hw/Sw codesign solution of the traditional 2D-FFT algorithm introduced on a hybrid platform combining an ARM Cortex-A9 processor with an FPGA. The third is also a Hw/Sw codesign solution using our optimized 2D-FFT algorithm to reach the real-time contraints for high-resolution images(1920×1080). It provides a speedup of 1.13x, 3.31x and 96.21x faster than the Hw/Sw co-design implementation of the traditional RC algorithm, the pure software implementations with and without optimizations, respectively.

Introduction

The Fast Fourier Transform (FFT) algorithm was initially introduced by J. Cooley and J.Tukey [START_REF] Cooley | An algorithm for the machine calculation of complex Fourier series[END_REF] in 1965 as a useful algorithm for image processing. It has become the most effective algorithm used for computer vision to obtain the frequency or the spectrum content of an image for many applications such as correlation filters and discrete convolution [START_REF] Yan | An effective unconstrained correlation filter and its kernelization for face recognition[END_REF][3] [START_REF] Ouerhani | Fast face recognition approach using a graphical processing unit "GPU[END_REF]. It is necessary to improve their performance to respect real-time and memory consumption constraints. However, the FFT algorithm is costly in terms of processing time and memory consumption [START_REF] Ouerhani | Fast face recognition approach using a graphical processing unit "GPU[END_REF][5] [START_REF] Govindaraju | High performance discrete Fourier transforms on graphics processors[END_REF]. More specifically, the traditional 2D-FFT algorithm is achieved by performing two 1D-FFT algorithms along rows or columns and transpose operation [START_REF] Cooley | An algorithm for the machine calculation of complex Fourier series[END_REF]. This affects the overall latency and could be designed differently. The use of embedded systems such as Graphics Processing Units (GPUs) [START_REF] Ouerhani | Fast face recognition approach using a graphical processing unit "GPU[END_REF], a combination of Hw/Sw Co-Design, and Field Programmable Gate Arrays (FPGAs) [START_REF] Smach | An FPGA-based accelerator for Fourier Descriptors computing for color object recognition using SVM[END_REF] are expected to solve these challenges. In this work, we have used the Zynq SoC FPGA-based platform, which presents better performance in terms of energy efficiency compared to GPUs [START_REF] Ouerhani | Fast face recognition approach using a graphical processing unit "GPU[END_REF]. Zynq SoC FPGAs are a good candidate for computer vision system development for their ability to exploit parallelism.

Additionally, the design flow of current FPGAs synthesis tools supports high-level abstraction languages (C/C++) as input descriptions, in contrast to hardware description languages (HDL: Verilog and VHDL). High-level synthesis (HLS) is used to simplify the design process by transforming the algorithmic description into hardware while satisfying the design constraints.

Given the importance of using the 2D-FFT algorithm in many applications, it could be interesting to propose a different implementation of this algorithm. In this paper, three implementations (one software and two Hw/Sw co-designs) of the 2D-FFT algorithm using the Zynq SoC are proposed. The first is based on ARM Cortex-A9 processor to perform the software implementation using OpenCV functions. The second is based on a hybrid platform combining an ARM Cortex-A9 processor with an FPGA to perform a Hw/Sw co-design implementation. The Hw/Sw co-design implementation provides less processing time and less power, which is the first contribution of this paper. The second contribution is an innovative architecture for the 2D-FFT algorithm using the Zynq SoC. It requires less processing time and less dedicated hardware blocks (DSP, LUT, and FF) compared to the traditional algorithm. Particularly, we present two different Hw/Sw co-design implementations of the 2D-FFT algorithm using the Zynq SoC: (i) traditional Row-Column (RC) algorithm with transpose operation and (ii) optimized RC algorithm without transpose operation. Both 2D-FFT algorithms were compared in terms of processing time, memory, and dedicated hardware blocks.

Generally, the Radix-2 and Radix-4 architectures are the basic architectures used to implement the FFT IP core. The decimation-in-frequency and non-discrimination are often used for the pipelined streaming and burst I/O architectures, respectively. Moreover, it is well known that N-point FFT using Radix-2 introduces log2 (N) steps, each step containing N/2 Radix-2 butterflies. However, N-point FFT using Radix-4 introduces log4 (N) steps, each step containing N/4 Radix-4 butterflies. The butterfly architecture used and the number of inputs and outputs are the main difference between these architectures [START_REF] Ouerhani | Area-Delay Efficient FFT Architecture Using Parallel Processing and New Memory Sharing Technique[END_REF]. In this work, four architectures have been assessed in order to select the best one. These architectures include Radix-4 Burst I/O, Radix-2 Burst I/O, Pipelined Streaming I/O, and Radix-2 Lite Burst I/O. Vivado HLS tool is used to generate the hardware description (at RTL level) from software-coded functions and targeting Zynq SoC. This description serves to exchange data between the software part and the hardware part. Many optimizations are introduced in order to develop suitable hardware implementation. The contributions of this paper are as follows:

• Proposing a Hw/Sw co-design solution of the 2D-FFT algorithm, which provides less processing time, and less power consumption using the Zynq SoC.

• Innovative architecture for the 2D-FFT algorithm tested on Zynq Soc that requires less processing time, memory, and less number of computational blocks compared to the traditional algorithm.

• A comparative study between four existing architectures to select the best one.

• Study the impact of directives on resource usage and performance.

The rest of the paper is organized as follows: Section 2, presents the different applications of the 2D-FFT algorithm and platforms used for these implementations. Section 3 describes our innovative architecture for the 2D-FFT algorithm using the Zynq SoC. Section 4 describes the software implementation based on ARM Cortex-A9 processor. Section 5 presents the Hw/Sw co-design solution using the Zynq SoC and the different optimizations for both 2D-FFT algorithms performed on ARM Cortex-A9 processor and the FPGA. Section 6 presents and discusses all implementations. The last section concludes the work and gives future research directions.

RELATED WORK

Computer vision techniques are getting more and more popular in a variety of areas in scientific research, such as identification [START_REF] Yan | An effective unconstrained correlation filter and its kernelization for face recognition[END_REF][9] [START_REF] Smach | An FPGA-based accelerator for Fourier Descriptors computing for color object recognition using SVM[END_REF], classification [10][3][11] [START_REF] Napoléon | Local binary patterns preprocessing for face identification/verification using the VanderLugt correlator[END_REF], and marking images [START_REF] Yanqing | Remote Sensing Image Content Retrieval Based on Frequency Spectral Energy[END_REF]. Fast Fourier Transform (FFT) has been introduced in many of the above-listed applications. Several applications for image processing are presented and discussed to prove the importance of the 2D-FFT algorithm. For example, Lamas-Seco et al. [START_REF] Lamas-Seco | Vehicle classification using the discrete fourier transform with traffic inductive sensors[END_REF] implement a novel algorithm based on the Fourier Transform (FT) to extract some spectral features of inductive signatures used in traffic management systems. In addition, a variant of the 2D-FFT algorithm named Enhanced Partial Discrete Fourier Transform (EP-DFT) is used to implement a multi-biometric system to improve recognition accuracy and security [START_REF] Yang | A fingerprint and finger-vein based cancelable multi-biometric system[END_REF]. A simplified correlation method based on the 2D-FFT is implemented to simplify the correlation setup [START_REF] Jridi | One lens optical correlation: application to face recognition[END_REF]. Recognition with this method is done without resorting to inverse 2D-FFT compared to the traditional correlation application. In addition, some research work focuses on image preprocessing to improve performance. Zhang et al. [START_REF] Dehai | A PCA-based face recognition method by applying fast fourier transform in pre-processing[END_REF] proposes a system for face recognition with Principal Component Analysis (PCA) approach, which applied the Fast Fourier Transform (FFT) to combine the phase spectrum of one image with the amplitude spectrum of another image as a mixed image. In addition, Khan et al. [START_REF] Zhang | A novel way to improve facial expression recognition by applying fast fourier transform[END_REF] develops a computational method based on Particle Swarm Optimization (PSO) and the Discrete Fourier Transform algorithm.

The different applications mentioned previously prove that either a combination between 2D-FFT algorithm and other algorithms is used for objects recognition tasks.

The FFT algorithm is costly in terms of processing time and power consumption, many researchers have studied efficient acceleration based on FPGAs and GPUs by exploiting their high parallelism capabilities. Indeed, the heterogeneous multicore SoC includes CPU, GPU and FPGA are proposed in many types of research. An example of the heterogeneous multicore platform used to accelerate the vehicle detection process was implemented by Cheng et al. [START_REF] Cheng | Design of vehicle detection methods with opencl programming on multi-core systems[END_REF]. This platform includes an Intel i5-2400 processor and an AMD HD6670 GPU. Zhang et al. [START_REF] Zhang | Frequency domain acceleration of convolutional neural networks on CPU-FPGA shared memory system[END_REF], proposes a novel method based on the CPU-FPGA platform with coherent shared memory to accelerate state-of-art Convolutional Neural Networks (CNNs). Ouerhani et al. [START_REF] Ouerhani | Fast face recognition approach using a graphical processing unit "GPU[END_REF] proposes an implementation of the correlation technique based on the Nvidia Geforce 8400 GS GPU for facial recognition. This technique is based on the 2D-FFT filter and the phase-only filter (POF). A mixed radix FFT and hierarchical FFT algorithms for both power-of-two and non-power-oftwo sizes using GPUs are presented in [START_REF] Govindaraju | High performance discrete Fourier transforms on graphics processors[END_REF]. The work in [START_REF] Chen | Input-adaptive parallel sparse fast fourier transform for stream processing[END_REF] exploits the similarity between adjacent inputs and sparse input samples in stream processing to enhance the efficiency of sparse FFT algorithm. The sparse FFT implementation is evaluated based on Intel i7 CPU and three NVIDIA GPUs, i.e., NVIDIA GeForce GTX480, Tesla C2070, and Tesla C2075. Hyun et al. [START_REF] Hyun | FPGA based signal processing module design and implementation for FMCW vehicle radar systems[END_REF] developed a Virtex-5 FPGA implementation of signal processing of the vehicle FMCW radar systems with a two-step FFT. Smach et al. [START_REF] Smach | An FPGA-based accelerator for Fourier Descriptors computing for color object recognition using SVM[END_REF] presented a hardware implementation using FPGA technology in order to accelerate the computation of Fourier Descriptors (FD). The Support Vector Machine (SVM) is used for classification. To optimize the radix-2 FFT FPGA implementation, a novel algorithm is proposed to simplify the computation by Walid et al [START_REF] Walid | Design and implementation of a RADIX-4 FFT using FPGA technology[END_REF]. Raju et al. [START_REF] Raju | Hardware Implementation of Discrete Fourier Transform and its Inverse Using Floating Point Numbers[END_REF] present two implementations of DFT/IDFT architectures on the FPGA. These architectures are based on radix 2 butterfly because of its reduced computation time. Yu et al. [START_REF] Yu | Bandwidth-intensive FPGA architecture for multi-dimensional DFT[END_REF] develop a Multi-dimensional (MD) Discrete Fourier Transform (DFT) algorithm based on Xilinx Virtex-5 FPGA. The proposed algorithm can support 2D, 3D, and even higher dimensional DFT. Li and Wyrwicz [START_REF] Li | Parallel 2D FFT implementation on FPGA suitable for realtime MR image processing[END_REF] report the design and implementation of a parallel 2D-FFT) algorithm on FPGA for real-time MR image processing. The results indicate that the image-reconstruction acceleration is primarily limited by the speed of the data transfer between the FPGA device and external sensors.

Ouerhani et al. [START_REF] Ouerhani | Implementation techniques of high-order FFT into low-cost FPGA[END_REF] allow to optimize existing FFT algorithms for low-cost FPGA implementations.

Finally, this paper focuses on FPGA-based platforms, which presents a better performance in terms of energy efficiency compared to GPU platforms. The FPGA shows better results in terms of performance with low power consumption. GPU platforms, also, provide high-performance results but consume more power [START_REF] Ouerhani | Fast face recognition approach using a graphical processing unit "GPU[END_REF]. Unlike the different implementations introduced in relation to the FPGAs, this work contributes with a Hw/Sw co-design of the 2D-FFT algorithm using the Zynq SoC respecting realtime execution and memory consumption constraints. As a result, the FPGA is widely used as an accelerator for image processing due to the possibility of reducing execution time and extracting parallel computations. However, this accelerator requires a thorough knowledge of hardware description languages (HDL) such as VHDL and Verilog. These languages are used to describe and synthesize at the Register-Transfer Level (RTL) levels. Today, high-level synthesis tools (HLS) is used to synthesize hardware description (RTL) from a high-level language such as C/C ++ and SystemC. In addition, HLS tools can not directly transform high-level language into RTL levels. As a result, it needs to be optimized and restructured to be suitable and synthesizable for a specific hardware platform.

Vivado HLS reduces design time and makes complex algorithms much easier than to use HDL. However, Vivado HLS does not support the compilation of all C/C ++ command structures. So, Xilinx provides many popular libraries and model templates for programming users. The hls_fft.h and ap_fixed.h libraries are used to implement our proposed FFT IP core. The float data type is used for the default implementation of FFT IP core. Then, to reduce the processing time and the resources number, it is better to transfer the input data to fixed-point data. Once compiled into HDL, modules on Vivado HLS produces an FFT IP core, which can be instantiated and simulated with other software such as Matlab, ISE. Besides, this FFT IP core subject to various optimizations in order to improve performance and reduce memory requirements. The design process using Vivado HLS is shown in Fig. 1.

C/C++ Algorithm

Vivado HLS

Synthesis/Export RTL Design

hls_fft.h and ap_fixed.h Library

New IP 2D-FFT creating

Hardware Plaform

Directive

Fig. 1: Design Process on Vivado HLS

2D-FFT Algorithms

The 2D-FFT algorithm can be used to identify any object by extracting in the transform domain features that can be used for identification or recognition [START_REF] Jridi | One lens optical correlation: application to face recognition[END_REF]. It considered as a local descriptor to represent an object instead of the global descriptor.

The 2D-FFT and its inverse are defined as follows:

( ) ( ) N N vy ux u,v f x,y .W W N N x 0 y 0 2D FFT F å = å = = - (1) ( ) ( ) N N 1 vy ux F u,v .W W N N 2 N u 0 v 0 2D IFFT f x, y - - å = å = = - (2)
when N indicates the number of sample points, f(x, y) indicates the pixel value of the input image, F (μ, v) indicates coefficients of the output image, and = / are the twiddle factor, and x, y, u and v represent the coordinate of the input and transformed image. Hence, to compute N-point FFT, we need respectively 2( ) and ( /2) 2( ) complex additions and complex multiplications. The traditional RC algorithm is computed with three steps. Firstly, the input 2D frame is initially saved in the local memory. After that, the 1D-FFT transform is applied along the columns.

Secondly, the result is saved in the local memory and transposed (columns to rows).

Thirdly, the second 1D-FFT transform is applied to columns another time.

However, the traditional RC algorithm is not efficient and has poor performance for large image processing because the transpose operation requires more resources and time to transfer data to and from memories [START_REF] Lenart | A hardware acceleration platform for digital holographic imaging[END_REF]. To solve this problem, we introduce an optimized RC algorithm without transpose operation, which computes with two steps:

1. In the first step, we perform the 1D-FFT algorithm on columns of the input image. The intermediate results are written and stored inversely (i.e. stored in rows not in columns) in two memories for real and imaginary parts.

2. In the second step, the 1D-FFT algorithm is reactivated on columns of the 1D-FFT image obtained previously. The final results are stored as real and imaginary data parts in two separate memories.

The optimized RC algorithm needs two reading and writing operations, while the traditional RC algorithm requires three operations. Both implementations are depicted in Fig. 2.

Write

Transpose columns To rows column-by-column

Write inversely

Traditional RC algorithm

Optimized RC algorithm

Read

Read

Write

Write

Intermediate Storage Intermediate Storage

Read

1D-FFTs 1D-FFTs column-by-column

1D-FFTs 1D-FFTs column-by-column column-by-column

Intermediate Storage

Read

Intermediate Storage Intermediate Storage

Write for streaming access

Read for streaming access

Intermediate Storage Intermediate Storage

Read for streaming access

Write for streaming access Output [i][j]= Input[j][i]

Fig. 2. Traditional and optimized RC algorithms using Zynq SOC.

Software Implementation of the 2D-FFT Algorithm

The OpenCV library provides an implementation for many interesting applications dedicated to software platforms. In addition, the 2D-FFT algorithm applied to image reconstruction can be used for the validation of the software implementation based on ARM Cortex 9 processor using the Zynq device. A SIMD architecture integrated into the ARM Cortex 9 processor is used to accelerate the execution. The 2D-FFT algorithm provided by the OpenCV library applied on the input frames stored in the DDR memory. The data transfer between the ARM Cortex 9 processor and the DDR memory is controlled with a Video Direct Memory Access (VDMA) core based on the High Performance (AXI HP) port of the Zynq SoC [START_REF] Crockett | The Zynq Book: Embedded Processing with the Arm Cortex-A9 on the Xilinx Zynq-7000 All Programmable Soc[END_REF]. Finally, the output result is forwarded to the VGA output. The software implementation of the 2D-FFT algorithm is configured to forward direction. We should notice that for a single high-resolution frame 1920x1080, this implementation requires 229 ms, which is inefficient for realtime applications. Two pre-processing steps are introduced to reduce the computation time of the 2D-FFT algorithm based on OpenCV functions: grayscale conversion and resizing. Fig. 3 shows the design of the software implementation based on ARM Cortex 9 processor.

AXI Data FIFO AXI -S2MM AXI -MM2S

Test-bench

Processing System (PS) Fig. 3: Software implementation based on ARM Cortex 9 processor.

Pre-processing

The goal of the pre-processing step is to reduce the complexity and the amount of processed data. The input frame is first affected by a resizing up followed by gray-scale conversion. Generally, reducing the size of the frame does not substantially affect the important details. The spectrum Nearest-Neighbor interpolation algorithm is used to resize down/up a frame. After reducing the frame, the gray-scale conversion is applied [START_REF] Kortli | A novel illumination-invariant lane detection system[END_REF]. The above-mentioned optimizations are combined to obtain the best system performance and to decrease the computation time. The complete system steps are presented in Fig. 4.

Input Frame

Resizing Down

Gray-scale Conversion 2D-FFT Resizing Up Output Frame Fig. 4 A complete system steps.

Evaluation

The optimization used (resizing up / down, grayscale conversation) reduce the complexity and the computation time of the 2D-FFT algorithm due to the decreasing of the pixels to be treated. An image of resolution 1920x1080 is used as input. Fig. 5 presents the resize scale and the processing time in function with the PSNR of the software implementation of the 2D-FFT algorithm based on ARM Cortex-A9 processor. The output spectrum quality is analyzed for different scaling parameters by using the PSNR parameter. At the same time, the processing time is assessed for the same scaling parameters. The typical value of the PSNR used in signal processing to measure the quality is bigger than 35 dB. It is found that the processing time obtained with the resized image at 256x256 pixels is more than 29 times lower than that obtained with the size 1920x1080 pixels with a PSNR value equal to 40.12 dB. By these simulations, we emphasize the need to resize the images in order to dramatically decrease the processing time without a significant decrease in the quality. As a result, we have resized the input frame at 256x256 pixels to reduce the amount of processed data. Processing time(ms)

Resize scale (b)

Resize scale in function with the processing time

Hw/Sw Co-Design Implementation of the 2D-FFT Algorithms

This section presents 2D-FFT algorithm implementation tested on hybrid processing units: ARM Cortex-A9 processor and FPGA. The advantage of the Zynq SoC from Xilinx is the ability to generate any algorithm written with High-Level Language: C, C++, and SystemC. Some other tools can generate RTL description using python language. The HLS tool is used to reduce design time and greatly facilitate the deployment of complex algorithms in the FPGA. However, the functions written with C/C++ can be optimized and specified based on many directives in order to enhance the performance and reduce the processing time using the HLS tools. The Zynq SoC can automatically manage the exchange of data between the software part and the hardware part. Traditional and optimized RC algorithms will be accelerated on FPGA. Fig. 6 presents an overview of the Hw/Sw co-design of the traditional RC algorithm based on a hybrid platform combining an ARM processor with an FPGA.

Read for streaming access

Write for streaming access Conversion Matrix To Real and Imaginary Inputs Conversion Real and Imaginary Outputs

To Matrix

Streaming Data

Streaming Data Fig. 6 Overview of the Hw/Sw co-design of the traditional RC algorithm using Zynq SoC.

An FPGA-based implementation is used processed the traditional RC algorithm into hardware acceleration to achieve real-time implementation. As shown in Fig. 6, the algorithms that are processed as hardware part are gray-scale conversion, 1D-FFT algorithm on columns, transpose operation, and the second 1D-FFT algorithm on columns, while the read/write frame and resizing down/up are processed as software.

This choice is made because reading and writing processes do not require any arithmetic operations. Generally, the ARM processor is well optimized for this kind of operations. However, when saving the input frame in the local memory, the amount of BRAM resources is increased. An AXI bus interface can be used for communication between the PS and the PL. Vivado HLS provides three types of AXI communication: AXI4-Lite bus, AXI4 interface, and AXI4-Stream. The first type is used to transfers the mapped data in low-rate memory. The second type is suited for high-performance memorymapped operations, which is for sending and receiving bigger data. The third type is AXI4-Stream will be used for this project, it is suitable for streaming a large amount of data at high speed, such as video streaming. Row buffers save the input data in the local memory and transmitted it to AXI VDMA. These buffers introduced by the class HLS::

stream. The VDMA allows transferring data between the ARM and the FPGA or conversely. The interface of each buffer is controlled with the #pragma directives.

Optimization

The 2D-FFT algorithm needs a high amount of resources and should be optimized.

Using the Vivado HLS device, we can handle FPGA implementation constraints by several optimization techniques, known as synthesis pragmas or directives. The performance of the hardware implementation varies depending on the configurations of directives. Here, we can analyze the impact of some of them:

DATAFLOW: this directive is introduced to enhance the design throughput by allowing the data to be transformed in sequential order from one loop flow to the next loop. If a next loop uses the data generated in the previous loop, it is not necessary to wait for the end of the previous loop. They will immediately move to the next loop when the data is generated. This directive parallels the communication between different functions. In this paper, FIFO memory is the default port type of FFT IP core. Two sequential loops are required to transfer the data from BRAM to FIFO (Read) and to load the data from FIFO to BRAM (Write).

PIPELINE: while the DATAFLOW directive orients the parallel communication

between the different functions, the PIPELINE directive orients the parallel communication between the different operations of the same function. The DATAFLOW and the PIPELINE directives allow executing all operations simultaneously. In this paper, for each FFT IP core, the PIPELINE directive is used to map data from BRAM to FIFO, but it is necessary to use with the DATAFLOW directive, otherwise, it would not benefit. In addition, the pipeline directive in this project can be implemented using the #pragma HLS PIPELINE directive

[II = <N>]
where N is the number of clock cycles per pipeline. Fig. 8 presents how dataflow and pipeline optimizations works. UNROLL: this directive allows alliterations to run in parallel. This directive has the ability to perform each iteration in one clock cycle when the loop is complete unrolled, and in a separate clock cycle when the loop is rolled. More directive need to be used in order to target real-time applications and to cover more complex designs. Table 1 presents the key features of six widely-used synthesis directives provided by Vivado HLS [START_REF] Zhao | Performance Modeling and Directives Optimization for High Level Synthesis on FPGA[END_REF].

Read data FFT columns Write data

Read data FFT columns

Write data

Evaluation

Table 2 presents the resources utilization of the hardware implementation for the Traditional and the optimized RC algorithms based on Zynq 7z020 FPGA for frame at a resolution of 256x256 pixels. The maximum frequency that can be achieved for this design is 166.67 MHz. Vivado HLS was used to get these resources. In general, lower resources utilization means lower design cost. To determine the surface usage of an FPGA design, the use of resources is a critical metric. Therefore, a better design should use less resources. According to the comparison, the optimized RC algorithm uses less resource than the traditional RC algorithm. The most obvious difference between both algorithms is that the optimized RC algorithm is introduced without transpose operation, which requires a high number of BRAM and DSP.

Consequently, the utilization ratio is greater than 100%. After the first 1D-FFT algorithm, the data stored in the local memory are transposed and stored for another time. This caused an increase in the number of memories used by the transposition operation (256 BRAMs are used). DSPs show the complexity of each algorithm in terms of the number of arithmetic operations, logic operations, and shift operations. The optimized RC algorithm needs one reading/writing transaction between the local memories for each block 1D-FFT, while the traditional algorithm needs three reading/writing transaction between the local memories: two transactions for the two blocks 1D-FFT and one transaction for transposition operation. Thus, the traditional algorithm requires more resources, which increases complexity. Besides, Table 3 introduces the two performance parameters Latency and Timing. They used to measure the effectiveness of any vision application. These two parameters are used to verify the hardware acceleration performance based on the FPGA. Timing represents the processing time, while the latency represents the number of iterations necessary to process a frame. Therefore, it is possible to compute the processing time of each step for hardware implementation. As well as, the processing time is computed as follows: The results show a required clock period of 5.55ns and 5ns for the traditional and the optimized RC algorithms, respectively. In addition, the traditional and optimized algorithms require respectively 65556 and 8422 number of cycles to process a frame at a resolution of 256x256 pixels. Thus, the traditional RC algorithm needs high number of cycles and more resources caused by the transpose operation. According to the comparison, the optimized RC algorithm requires less number of cycles and less resources than the traditional RC algorithm. Therefore, it is better to choose an algorithm that needs less number of cycles and less resources to respect real-time execution and memory consumption constraints. As a result, the optimized RC algorithm should be made applicable to FPGA. 4 shows a comparison between these architectures in order to choose the best architecture in terms of resources used and processing time for the hardware implementation of the optimized RC algorithm to process frame at a resolution of 256x256 pixels. The original and reconstructed images obtained using the Hw/Sw co-design implementation for the optimized RC algorithm based on a hybrid platform combining an ARM Cortex 9 processor and an FPGA using Zynq SoC are shown in Fig. 11. It can be seen that the PSNR value between the Hw/Sw co-designs for the optimized RC algorithm using Zynq SoC and the 2D-FFT algorithm using Matlab is greater than 36 dB and ∞ when compared the inverse 2D-FFT.

( ) ( ) ( ) ( ) 2 fk k 2 2 fl fx k k db PSNR 10 log10 X k X k X k = ae ö ÷ ç ÷ ç ÷ ÷ ç è ø ae ö ÷ ç - ÷ ç ÷ ÷ ç è ø å å å (4)

Discussion

In However, the HW/SW co-design flow of the traditional or optimized RC algorithms needs an intermediate evaluation of sub-program, which would be very fastidious in terms of programmability. Then, we should mention that numerous directives have been used to get parallel processing. Consequently, the Hw/Sw co-design implementation provides a better compromise between the performance and the programmability.

Moreover, an RTL simulation for the Hw/Sw Co-design solution is evaluated to give detailed information about the number of resources used, Max Latency, and the timing.

In addition, many optimizations were introduced. The resizing down/up and the grayscale conversion functions from the OpenCV are used to reduce the number of pixels treated. Consequently, the complexity and the processing time of the 2D-FFT algorithm are reduced.

Since the traditional RC algorithm requires more memory and more dedicated hardware blocks due to the transpose operation. In this paper, an innovative architecture namely, "optimized RC algorithm" using the Zynq SoC is proposed. It implemented without transpose operation. As a result, the optimized RC algorithm uses less memory and less dedicated hardware blocks and requires less Latency, which is suitable for the implementation in FPGA. In addition, many directives such as DATAFLOW, PIPELINE, and LOOP UNROLL are used to improve the performance. Hence, these directives are used to make the optimized RC algorithm more suitable for real-time implementation. The combination of pipelined streaming architecture and the DATAFLOW and PIPELINE directives provides real-time constraints. Table 7 summarizes the results obtained from the different implementations.

Softawre implementation

Original implementation without optimizations(ARM)

1980x1020

Original implementation with optimizations (ARM)

1980x1020

Gray scale conversion + 2D-FFT algorithm (ARM)

256x256

First HW/SW Codesign implementation HW/SW Co-design implementation with optimizations (ARM+FPGA)

1980x1020

Gray scale conversion + traditional RC algorithm (FPGA)

256x256

Second HW/SW Codesign implementation HW/SW Co-design implementation with optimizations (ARM+FPGA)

1980x1020

Gray scale conversion + optimized RC algorithm (FPGA)

Conclusion

Fig. 5 (

 5 Fig. 5(a) presents the PSNR variation with the resize scale down for a high-resolution image, and Fig. 5(b) presents the processing time in function with resizing scale down for the whole 2D-FFT algorithm.

Fig. 5 :

 5 Fig. 5: Resize downscale in function with PSNR and processing time based on ARM Cortex-A9 processor of the 2D-FFT algorithm for different size of image.

TheFig. 7 .

 7 Fig. 7. (a) Diagram of the proposed 1D-FFT IP core, (b) hierarchical sequence of memories between the PS and the PL for the 1D-FFT accelerator.

Fig. 7 (

 7 Fig.7 (b) shows the hierarchical sequence of memories between the PS and the PL for the 1D-FFT IP core. The scatter-gather mode of AXI DMA is used to configure each interface and to transfer data to DDR memory. AXI DMA controls direct data transfers between AXI4-Stream target devices and system memory. From the DDR memory, the data is read and write via an AXI4 stream master. Memory-mapped to AXI4 stream (MM2S) master and AXI stream to memory-mapped (S2MM) slave are respectively two channels provided by AXI DMA for reading and writing data. The MM2S bus reads the data from DDR memory and transmits them to PL, while S2MM bus writes the data from PL and transmits them to DDR memory. As a result, AXI DMA core controls the data transfers, the address generation, and the transaction planning in memory between the DDR memory and PL.

Fig. 8 .

 8 Fig. 8. (a) Hierarchy of the dataflow optimization and (b) Hierarchy of the pipeline optimization.

Four

  architectures are used to implement the FFT IP: Radix-4 Burst I/O, Radix-2 Burst I/O, Radix-2 Lite Burst I/O, and Pipelined Streaming I/O. Table

Fig. 9 . 3 .

 93 Fig.9.3. show the effectiveness of our included multiple directives optimization solution in term of decreasing latency over than 356x.Table 5 presents the number of resources

Fig. 10 introduces

 10 Fig.10 introduces the processing time of each step for the software and the Hw/Sw Co-design implementations. The Hw/Sw co-design implementation of the optimized RC algorithm is accelerated by integrating the grayscale conversion function and the 2D-FFT algorithm into the FPGA.

Fig. 9 :

 9 Fig. 9: Analysis of the directives effects: Fig.(9.1) Directives effects on BRAM and DSP usage, Fig.(9.2) dircetives effects on FF and LUT usage, Fig.(9.3) Directives effects on latence.

Fig. 10

 10 Fig.10The processing time of each step for the 2D-FFT algorithm based on ARM Cortex-A9 processor, and a hybrid platform ARM combining an ARM processor with an FPGA.The hardware acceleration of the gray-scale conversion function and 2D-FFT algorithm achieve a speedup of 10.62x and 72.59x, respectively, compared to the software implementation based on the ARM Cortex 9 processor. The hardware implementation based on FPGA means higher performance than software implementation. This higher performance presents the interest of a high-level synthesis using Zynq SoC. In addition, the software implementation of the 2D-FFT function can take advantage of the SIMD engine.The results of the test bench obtained with HLS will be compared to the MATLAB implementation of each step of the optimized RC algorithm to verify that the Hw/Sw co-design implementation works correctly. A similar implementation of the 2D-FFT algorithm has been introduced in Matlab. Due to the simplicity and benefits of implementing multidimensional arrays, the implementation with Matlab was much shorter and easier in terms of development time. Note that the optimized RC algorithm has been computed is a pipeline which processes the input frame as a continuous stream of pixel values. When calculating PSNR between the results of the MATLAB implementation with the proposed architecture using Vivado HLS must be the same.Therefore, it is proven that the hardware implementation of the optimized RC algorithm works correctly. To compute the PSNR, we use an image of 256x256 pixels. The optimized RC algorithm has been introduced. The ( ( )) is the 2D-FFT exact with a

Fig. 11 (

 11 Fig. 11 (a) shows the Lena image, (b) shows the spectrum image obtained by the optimized RC algorithm using Zynq SoC, and (c) shows the reconstructed Lena image.

  this work, three implementations (one software and two Hw/Sw co-designs) are used to assess the performance of the 2D-FFT algorithm: software implementation based on ARM Cortex-A9 processor and two Hw/Sw Co-design implementations based on a hybrid platform combining an ARM processor and FPGA circuit. Many optimizations are used to improve the evaluation for each implementation. The software implementation based on ARM Cortex-A9 processor provides advantages in terms of flexibility of the programmability based on the well-known OpenCV libraries.

  demonstrate that the proposed Hw/Sw co-design implementation based on the optimized RC algorithm is 1.13x, 3.31x and 96.21 faster than the Hw/Sw co-design implementation based on the traditional RC algorithm, pure software implementations with and without optimizations, respectively. The hardware implementation of the grayscale conversion function and the Optimized RC algorithm performed on FPGA is 48.78x faster than the corresponding functions performed on ARM Cortex-A9 processor for a frame of 256x256 pixels. While the hardware implementation of the grayscale conversion function and the Traditional RC algorithm provide an acceleration of 8.38x. In addition, the hardware implementation of the Optimized and the Traditional RC algorithms produces an acceleration of 70.59x and 8.23x, respectively, compared to the corresponding functions performed on ARM processor. Therefore, the optimized architecture for the 2D-FFT algorithm significantly improves the resources uses (BRAM, DSP, FF, and LUT) and the processing time (Timing and Latency). As depicted in table 8, the proposed design of the 2D-FFT algorithm takes less number of clock cycles for execution and requires less processing time compared to existing FPGA-based implementations for an image at a resolution of 256x256 pixels.

  This work presents three implementations (one software and two Hw/Sw codesigns) of the 2D-FFT algorithm introduced by Cooley and Tukey. All of them have been implemented on ARM Cortex-A9 processor and FPGA using the Zynq SoC. The design was realized using the Vivado HLS. To begin, a software implementation of the 2D-FFT algorithm is introduced on ARM Cortex-A9 processor. After that, a Hw/Sw codesign solution of the traditional 2D-FFT algorithm is introduced on a hybrid platform Cortex-A9 processor with an FPGA. The traditional 2D-FFT algorithm uses more resources and more processing time due to the transpose operation.Therefore, to respect real-time execution and memory consumption constraints, an optimized architecture for the 2D-FFT algorithm using the Zynq SoC is proposed on FPGA. It provides an acceleration of 70.59x, while the traditional 2D-FFT algorithm provides an acceleration of 8.23x compared to the corresponding functions performed on ARM processor. Several directives are applied to improve the performance. About the processing time of the whole system, the proposed Hw/Sw co-design implementation based on the optimized RC algorithm is 1.13x, 3.31x and 96.21 faster than the Hw/Sw co-design implementation based on the traditional RC algorithm, pure software implementations with and without optimizations, respectively.

Frequency spectrum AXI VDMA Programmable Logic (PL)

	Frame 1920*1080	• Resize Down • Grayscale Conversion • 2D-FFT Algorithm • Resize Up ARM Cortex-A9	AXI Interface	DDR memory
		AXI HP

Table 1

 1 Configuration of synthesis directives.

	Directives	Target	Configuration
	Dataflow	Top loop/function	Enabled/Disabled
	Pipeline	Each loop level	Enabled/Disabled
	Loop unroll	Each loop level	Unrolling factors
	Function inline	Each function	Yes/No
	Loop flatten	Inter loop levels	Yes/No
	Function pipeline	Each function	Enabled/Disabled

Table 2

 2 Resource utilization of the hardware implementation for the Traditional RC and the optimized RC algorithms based on Zynq 7z020 FPGA with frequency 166.67 MHz.

	Functions/ Resources		BRAM	DSP	FF	LUT
	Traditional RC Algorithm	Total Available	290 280	79 220	33516 106400	27232 53200
		Utilization (%)	103	35	31	51
	Optimized RC Algorithm	Total Available	30 280	73 220	32805 106400	26514 53200
		Utilization (%)	10	33	30	49

Table 3

 3 Performance of the hardware implementation for Traditional RC and optimized RC algorithms based on Zynq 7z020 FPGA.

	Functions/ Resources		Latency	Max Timing
		min	max	(ns)
	Traditional RC Algorithm	3215	65556	5.55
	Optimized RC Algorithm	3550	8422	5

Table 4

 4 Comparison between the four architectures used for the hardware implementation of the optimized RC algorithm in terms of resource used and the number of cycles. Burst I/O architecture loads and processes the data separately. This architecture uses an iterative approach to process data with a large number of butterflies.

	Functions/ Resources	BRAM DSP	FF	LUT	Latency
							min	max
		Total	56	49	32803	26514
	Radix-4 Burst I/O	Available	280	220	106400	53200	3788	8660
		Utilization (%)	20	22	30	49
		Total	40	17	32803	26514
	Radix-2 Burst I/O	Available	280	220	106400	53200	7686	12558
		Utilization (%)	14	7	30	49
		Total	40	9	32803	26514
	Radix-2 Lite Burst	Available	280	220	106400	53200	12680	17552

  Table 5 presents the number of resources used of each step for the hardware implementation of the optimized RC algorithm based on FPGA to generate one frame at a resolution of 256x256 pixels. Also, it introduces the Timing and Max Latency.

	I/O	Utilization (%)	14	4	30	49
	Pipelined Streaming I/O	Total Available Utilization (%)	30 280 10	73 220 33	32805 106400 30	26514 53200	3550	8422

Table 5

 5 Resource utilization and timing of each step for the hardware implementation of the optimized RC algorithm based on FPGA.

Table 6

 6 Comparison results between Hw/Sw co-design implementation with Matlab for Lena image at 256*256 resolution.

	Functions	Optimized RC algorithm (PSNR)
	Abs(2D-FFT)	36,1576
	2D-IFFT	∞

Table 7

 7 Processing time for different implementations.

	Functions/ Resources	Optimizations	Processing Time	Frame Size
			(ms)

Table 8

 8 2D FFT performance comparison with existing FPGA-based implementations for an image at a resolution of 256x256 pixels.