Hadjer Benmeziane

Kaoutar El Maghraoui

Ieee Member

Hamza Ouarnoughi

IEEE Smail Niar

Martin Wistuba
email: martin.wistuba@ibm.com

IEEE Naigang Wang
email: nwang@us.ibm.com

A COMPREHENSIVE SURVEY ON HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH 1 A Comprehensive Survey on Hardware-Aware Neural Architecture Search

Keywords: AutoML, Hardware-aware Neural Architecture Search, Neural Network Acceleration, Edge Intelligence. I

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.

• The variety of data types and tasks that require different neural architectural designs and optimizations. • The vast amount of hardware platforms which makes it difficult to design one globally efficient architecture.

For instance, certain problems require task-specific models, e.g. EfficientNet [START_REF] Tan | EfficientNet: Rethinking model scaling for convolutional neural networks[END_REF] for image classification and ResNest [START_REF] Zhang | Resnest: Split-attention networks[END_REF] for semantic segmentation, instance segmentation and object detection. These networks differ on the proper configuration of their architectures and their hyperparameters. The hyperparameters here refer to the pre-defined properties related to the architecture or the training algorithm. In general, the neural network architecture can be formalized as a Directed Acyclic Graph (DAG) where each node corresponds to an operator applied to the set of its parent nodes [START_REF] Goodfellow | Deep Learning[END_REF]. Example operators are convolution, pooling, activation, and self-attention. Linking these operators together gives rise to different architectures. A key aspect of designing a well-performing deep neural network is deciding the type and number of nodes and how to compose and link them. Additionally, the architectural hyperparameters (e.g., stride and channel number in a convolution, etc.) and the training hyperparameters (e.g., learning rate, number of epochs, momentum, etc.) are also important contributors to the overall performance. Figure 1 shows an illustration of some architectural choices for the type of convolutional neural network.

According to this representation, DL models can contain hundreds of layers and millions or even billions of parameters. These models are either handcrafted by repetitive experimentation or modified from a handful of existing models. These models have also been growing in size and complexity. All of this renders handcrafting deep neural networks a complex task that is time-consuming, error-prone and requires deep expertise and mathematical intuitions. Thus, in recent years, it is not surprising that techniques to automatically design efficient architectures, or NAS for "Neural Architecture Search", for a given dataset or task, have surged in popularity.

In figure 2, we compare several deep learning models for the image classification task. Each dot in the plot corresponds to a given DL architecture that has been used for image classification. The dot size correlates with the size of the corresponding neural network in terms of the number of parameters. A quick look at the graph reveals the trend to design larger models to better Top 1 accuracy. However, the size is not necessarily correlated with better accuracy. There have been several efforts to conceive more efficient and smaller networks to achieve comparable Top 1 accuracy performance. We compare four classes of models: Handcrafted, Efficient handcrafted, NAS, and HW-NAS. Generally, throughout the years the handcrafted models rank high up in terms of accuracy but are much more complex in terms of the depth of the architecture or the large number of parameters. For instance, ViT-H [START_REF] Anonymous | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF], which is the state-of-the-art model as of December 2020, has over 600 million parameters and 32 layers. In the top right quadrant of the figure 2 (around the same region as most of the recently handcrafted models), we find some of the models that are automatically created by different NAS techniques. These latter techniques focus only on improving the model's accuracy without paying attention to the efficiency of the model in terms of its size and latency. Therefore, these NAS models are still large, with the number of parameters ranging between 100M and 500M.

Since 2015, we have noticed the rise of efficient handcrafted models. These models rely on compression methods (see section III-A1) to decrease the model's size while trying to maintain the same accuracy. MobileNet-V2 [START_REF] Sandler | Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation[END_REF] and Inception [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] are good examples where the number of parameters is between 20M and 80M. This paper focuses on the class of hardware or platform-aware NAS techniques: HW-NAS. This class encompasses work that aims to tweak NAS algorithms and adapt them to find efficient DL models optimized for a target hardware device. HW-NAS began to appear in 2017 and since then achieved state of the art (SOTA) results in resourceconstrained environments with Once-for-all (OFA) [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF] for example.

A conventional NAS process requires the definition of three main components: the search space, the search strategy, and the evaluation methodology, as illustrated in figure 3. First, the search space (1) draws from a set of neural network architectures to define the neural network operators and how they are connected to form a valid network. It determines how the architectures can be formed and what architectures are allowed. For example, NASNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF] introduced a fixed macro architecture where the search consists of finding the appropriate operators used within each block from a set of 12 specified operators. This search space is explored by a search strategy [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] which samples a population of network architectures' candidates. It evaluates the accuracy of the model using a specific evaluation methodology [START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF]. The mea-sured accuracy will then guide the search strategy to converge towards promising architectures in the search space. Generally, the evaluation component will train the architecture on the desired dataset, which often takes a considerable time. Many NAS algorithms have incorporated several techniques to speed up the training process described in Section VII-C.

NAS has proven its efficiency by proposing several SOTA models in Object Detection [START_REF] Ghiasi | NAS-FPN: learning scalable feature pyramid architecture for object detection[END_REF] and Image Classification [START_REF] Howard | Searching for mobilenetv3[END_REF]. However, these models are often composed of millions of parameters and require billions of floating-point operations (FLOP). This causes a large memory footprint and big FLOP preventing their usage in resource-constrained environments. Additionally, these models might require specific hardware (e.g. GPUs, TPUs, etc.) to allow their deployment in a reasonable time or real-time applications.

Since the inception of the very first NAS algorithm with reinforcement learning [START_REF] Zoph | Neural architecture search with reinforcement learning[END_REF], there has been tremendous work to study efficient NAS. More recently, integrating hardware awareness in the search loop (i.e. HW-NAS) has attracted several researchers and has opened up interesting new research directions. Some HW-NAS efforts have demonstrated SOTA results and have balanced the trade-off between accuracy and hardware efficiency. For example, FBNet [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF] has achieved SOTA results on ImageNet by using an objective function that minimizes both the cross-entropy error that leads to better accuracy and the latency that penalizes networks that are too slow.

This paper provides a detailed overview of existing HW-NAS research efforts and categorizes them according to their goals and problem formulation. There has been no survey dedicated to hardware-aware NAS in the literature to the best of our knowledge. With this survey, we provide a concise review of the NAS variants that focus on precision and hardware awareness. We hope that it will be beneficial to those with a basic understanding of NAS interested in (i) getting a comprehensible but accessible overview of how the multiobjective problem is solved and (ii) a succinct reference and guide of existing HW-NAS techniques.

Our review is divided into several sections. In section II, we highlight the relevant NAS surveys and explain the focus and importance of our survey compared to them. In section III, we present a brief overview of the efficient DL methods and where HW-NAS is situated among them. We also explain some of the concepts needed to understand the components of the NAS process. In section IV, we classify the different HW-NAS works based on their goals and target platforms. Based on the taxonomy, we introduce and define two search spaces: Architecture Search Space and Hardware Search Space in section V. In section VI, we formally define the HW-NAS optimization problem and how the multi-objective function is formulated. The following section Search Strategies is divided into three subsections. We first discuss, in section VII-A, the various algorithms used to explore the search space, including reinforcement learning and evolutionary algorithms. In section VII-B, we give insights on how NAS algorithms take care of the non-differentiable variables. And in section VII-C, we list the methods to speed up the search process and avoid training each sampled architecture. We dedicate section

Best Model

Performance of A VIII to the study of the hardware metrics used by HW-NAS and the methods used to measure them. In section IX, we introduce other considerations that either tackle the hardware efficiency objective, model compressions such as automatic quantization and pruning or add other objectives to the NAS like the robustness against adversarial attacks. We discuss the industrial adaptation of NAS by comparing several tools proposed by the commercial and research communities. We conclude in sections XI and XII by highlighting the challenges and limitations of current HW-NAS and discuss possible future directions. We summarized the referenced body of work in the table: https://tinyurl.com/y6458skt.

II. EXISTING SURVEYS ON NAS

In the past few years, a large number of research papers have surveyed the state of the art neural architecture search algorithms. Two of the surveys we have examined stood out. The first survey, published in 2018 [START_REF] Elsken | Neural architecture search: A survey[END_REF], describes the end-toend design of a NAS algorithm. It details the components (i.e., Search Space, Search Strategy and Performance Estimation Strategy) and gives a clear taxonomy of the optimization methods. In 2019, a similar survey was published by Wistuba et al. [START_REF] Wistuba | A survey on neural architecture search[END_REF], which describes the different components of NAS algorithms with a focus on the optimization methods. It also includes a section that describes a high-level multi-objective neural architecture search and how it can be formulated and solved. While existing NAS surveys have succeeded at giving a thorough understanding of how general NAS algorithms work and explaining the various concepts that allow NAS techniques to achieve SOTA results, they fell short in including enough details about HW-aware NAS algorithms and the emerging efforts that strive to render NAS-generated models practical and amenable to deployment in today's constrained hardware platforms. Zhang et al. [START_REF] Zhang | When neural architecture search meets hardware implementation: from hardware awareness to co-design[END_REF] surveyed a subset of hardwareaware NAS algorithms that only focus on FPGA platforms. There is a growing effort investigating the intersection of NAS and hardware platforms as depicted in Figure 4, where we can see the increasing number of research papers that are focusing on hardware-aware NAS. This work is a first attempt at filling the gap we have identified in existing NAS surveys. We aim to provide a detailed overview and analysis of existing HW-NAS research efforts. Table I shows the structure of the study and serves as a guide to the various sections and their content.

Focus of this paper In this survey, we provide a comprehensive overview of the architecture and hardware search spaces including how the HW-NAS problem is formulated, the search strategies, the hardware cost estimation methods, and the plethora of hardware platforms that have been targeted. We compare the used multi-objective search strategies such as reinforcement learning or evolutionary algorithm, along with some techniques that use non-differentiable parameters which

III. BACKGROUND

In this section, we define key terms needed to grasp the ideas surrounding HW-NAS and the concepts discussed in the following sections. Gray lines correspond to pruned vertices (i.e. weights) and white nodes correspond to pruned neurons. Source [START_REF] Deng | Model compression and hardware acceleration for neural networks: A comprehensive survey[END_REF] The significant advances and breakthroughs of deep learning, that have propelled from academic and industrial research labs' circles, are now electrifying the computing industry and transforming the world. Deep learning is now being largely used to solve real-world problems. As deep learning is computationally demanding, most of the deployments happen in the cloud or on-premises data centers. However, with the arrival of powerful and low-energy consumption Internet of Things (IoT) devices and the growing need to take action in real or near-real-time, deep learning computations are increasingly

A. Methodologies for Efficient Deep Learning

Hardware Agnostic

Non Hardware Agnostic Hardware-dependent & Hardware-agnostic Fig. 6. Overview of efficient deep learning techniques moving to the edge. Edge devices pose several challenges in this context, as they are constrained with limited energy and computational power. For example, autonomous driving cars depend on real-time object detection of the environment. They cannot tolerate the additional latency of sending the data to the cloud, processing the data and then sending it back to the edge device. Moreover, the compute capacity at the edge is significantly low, which does not match the increasing DL models complexity. This has motivated the research community to find innovative ways to reduce the DL models' size, their required number of floating operations, and their inference latency. This section presents an overview of the efficient deep learning techniques and where HW-NAS is situated among them.

Figure 6 illustrates the taxonomy of different techniques used to optimize deep learning models.

1) Model Compression: These methods aim to take one standard deep learning model such as ResNet or AlexNet and apply some optimizations that will decrease the model size and the number of FLOPs, i.e., increase the compression ratio, while trying to maintain the same accuracy. Relevant surveys [START_REF] Deng | Model compression and hardware acceleration for neural networks: A comprehensive survey[END_REF], [START_REF] Cheng | A survey of model compression and acceleration for deep neural networks[END_REF] on model compression classify the optimizations into these four classes:

• Compact Model: This technique modifies the standard operations used in DNNs. In a CNN, the standard convolution is replaced by more flexible convolution arithmetics that expand the number of feature maps and decrease the number of parameters such as dilated convolution [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] or separable depthwise convolution [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF]. In an RNN, cells like S-LSTM [START_REF] Zhu | Long short-term memory over tree structures[END_REF], or JANET [START_REF] Van Der Westhuizen | The unreasonable effectiveness of the forget gate[END_REF] simplify the gates and decrease the number of parameters compared to a regular LSTM.

• Tensor Decomposition: a tensor is the fundamental data structure used in machine learning. It can represent vectors, matrices and even n-dimensional arrays. Therefore, shrinking the tensors allows accelerating DNNs and reducing their size. Tensor decomposition is an extension of the matrix decomposition techniques used in mathematical settings. Equation 1 formulates the matrix decomposition system where the number of parameters of A and B combined is smaller than the number of parameters of M.

M = AB with M ∈ R m×n , A ∈ R m×r , B ∈ R r×n (1)
Note that tensor networks including hierarchical tensor representation (HT) [START_REF] Song | Hierarchical tensor decomposition of latent tree graphical models[END_REF] and tensor train decomposition (TT) [START_REF] Zniyed | A tt-based hierarchical framework for decomposing high-order tensors[END_REF] achieve higher compression rates in a fullyconnected network since they usually contain more redundancy.

• Quantization: In deep learning, quantization [START_REF] Gupta | Deep learning with limited numerical precision[END_REF] refers to converting data objects from 32-floating point to lower precision or a fixed point integer or even binary. These data objects can be the weights of the layer, the activations (the input data's internal representation), the error value, the gradient values and the weight update. Each method differs with the chosen number of bitwidth and the data objects that are quantized. • Network Sparsification or pruning [START_REF] Han | Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding[END_REF], [START_REF] Zhu | To prune, or not to prune: exploring the efficacy of pruning for model compression[END_REF] attempts to compress the model by pruning some weights (edges) or operations (nodes). Usually, the decision of pruning is taken based on its importance, which is directly the weight values or learned via an attention layer. 2) HW-NAS: Another efficient deep learning technique is HW-NAS. In HW-NAS, we search for the architecture that maximizes the accuracy and hardware usability among a set of architectures. Note that some HW-NAS can be considered under the model compression techniques as they search for the best bitwidth or the best way to prune. We further detail the search for hyperparameters, NN Block or full architectures in Section 4.

3) Code Transformations: An alternative approach that is gaining more attraction these recent years is to apply some code transformations that optimize the DNNs on the operator level [START_REF] Li | The deep learning compiler: A comprehensive survey[END_REF]. These transformations are hardware-specific and require a compiler to apply the right transformation for the right hardware platform automatically.

B. Search Algorithms

In this section, we give a high-level explanation of reinforcement learning and evolutionary algorithms; two of the most used search algorithms in NAS. The explanations are concise to ensure the core ideas are accessible to a broad audience, and so that the entire survey can be read end-to-end easily.

1) Reinforcement Learning: Reinforcement learning (RL) is a technique to allow an agent to learn by trial and error. The agent takes actions and interacts with an environment to maximize the accumulative reward. It is usually modelled as a Markov Decision Process (MDP).

"Reinforcement Learning is learning what to dohow to map situations to actions -to maximize a numerical reward signal. The learner is not told which actions to take, but instead must discover which actions to yield the most reward by trying them." [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] Fig. 7. The agent-environment interaction in a Markov Decision Process. Source: [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] As illustrated in figure 7, at each step, the agent observes the state of the environment sends and receives a reward for its previous action. It then selects its next action. The reward guides the agent to improve its policy such that better actions are chosen in the future. The policy of an agent is the algorithm that allows it to choose between multiple actions.

There are several variants and algorithms to train and update the policy of an agent. The basic one is the sample-eval-update method. This method is an iterative process where the agent samples one action, evaluates it using a Q-Learning [START_REF] Jang | Q-learning algorithms: A comprehensive classification and applications[END_REF] policy and updates the decision process accordingly. Another update method is REINFORCE [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF]. It is a Monte-Carlo variant of policy gradients. The agent collects a set of actions (called trajectory) τ using its current policy, and uses it to update the policy parameter. Since one full trajectory must be completed to construct a sample space, REINFORCE is updated in an off-policy way.

2) Evolutionary Algorithms: Evolutionary algorithms (EA) are optimization techniques that have three main characteristics:

• Population-based: EA maintain an entire set of candidate solutions. Each solution corresponds to a unique point in the search space of the problem. This set of solutions is called a population. • Fitness-oriented: EA assign a fitness score to each solution, reflecting the quality of a solution. A genetic algorithm is a type of evolutionary algorithms that encodes the individuals into numerical vectors, called chromosomes. The chromosome is represented as a set of parameters that defines a particular individual. A selection criterium is used to select a set of candidate individuals of which the fittest are mutated and recombined by crossover to create the next generation. This algorithm is the simplest version of the evolutionary methods as the chromosomes are numerical vectors and the mutations can be simple permutations.

IV. TAXONOMY OF HW-NAS

Unlike conventional NAS, where the goal is to find the best architecture that maximizes model accuracy, hardware-aware NAS (HW-NAS) has multiple goals and multiple views of the problem. We can classify these goals into three categories (See figure 9 from left to right) :

• Single Target, Fixed Configuration: Most of existing HW-NAS fall under this category. The goal is to find the best architecture in terms of accuracy and hardware efficiency for one single target hardware. Consequently, if a new hardware platform has to be used for the NAS, we need to rerun the whole process and feed it the right values to calculate the new hardware's cost. These methods generally define the problem as a constrained or multi-objective optimization problem [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF]. Within this category, two approaches are adopted: Category & References Single Target, Fixed Configuration Hardware-aware Search Strategy [START_REF] Smithson | Neural networks designing neural networks: Multi-objective hyper-parameter optimization[END_REF], [START_REF] Gordon | Morphnet: Fast & simple resource-constrained structure learning of deep networks[END_REF], [START_REF] Dong | Ppp-net: Platform-aware progressive search for pareto-optimal neural architectures[END_REF], [START_REF] Elsken | Efficient multi-objective neural architecture search via lamarckian evolution[END_REF], [START_REF] Zhou | Resource-efficient neural architect[END_REF], [START_REF] Hsu | MONAS: multi-objective neural architecture search using reinforcement learning[END_REF], [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Mitschke | Gradient based evolution to optimize the structure of convolutional neural networks[END_REF], [START_REF] Nekrasov | Fast neural architecture search of compact semantic segmentation models via auxiliary cells[END_REF], [START_REF] Lu | Nsga-net: Neural architecture search using multiobjective genetic algorithm[END_REF], [START_REF] Cheng | Instanas: Instance-aware neural architecture search[END_REF], [START_REF] Cai | TEA-DNN: the quest for timeenergy-accuracy co-optimized deep neural networks[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Dai | Chamnet: Towards efficient network design through platform-aware model adaptation[END_REF], [START_REF] Sood | Neunets: An automated synthesis engine for neural network design[END_REF], [START_REF] Li | Partial order pruning: For best speed/accuracy trade-off in neural architecture search[END_REF], [START_REF] Guo | Single path one-shot neural architecture search with uniform sampling[END_REF], [START_REF] Yu | Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers[END_REF], [START_REF] Stamoulis | Single-path nas: Designing hardwareefficient convnets in less than 4 hours[END_REF], [START_REF] Xiong | Resource constrained neural network architecture search: Will a submodularity assumption help?[END_REF], [START_REF] Baldeon-Calisto | Adaresu-net: Multiobjective adaptive convolutional neural network for medical image segmentation[END_REF], [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF], [START_REF] Fedorov | Sparse: Sparse architecture search for cnns on resource-constrained microcontrollers[END_REF], [START_REF] Tan | EfficientNet: Rethinking model scaling for convolutional neural networks[END_REF], [START_REF] Shaw | Squeezenas: Fast neural architecture search for faster semantic segmentation[END_REF], [START_REF] Liu | Memnas: Memory-efficient neural architecture search with grow-trim learning[END_REF], [START_REF] Guo | NAT: neural architecture transformer for accurate and compact architectures[END_REF], [START_REF] Bae | Resource optimized neural architecture search for 3d medical image segmentation[END_REF], [START_REF] Schorn | Automated design of error-resilient and hardware-efficient deep neural networks[END_REF], [START_REF] Wang | Energyaware neural architecture optimization with fast splitting steepest descent[END_REF], [START_REF] Cassimon | Using neural architecture search to optimize neural networks for embedded devices[END_REF], [START_REF] Wang | Evolving deep neural networks by multi-objective particle swarm optimization for image classification[END_REF], [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Loni | Deepmaker: A multi-objective optimization framework for deep neural networks in embedded systems[END_REF], [START_REF] Xu | Latency-aware differentiable neural architecture search[END_REF], [START_REF] Iqbal | Flexibo: Cost-aware multi-objective optimization of deep neural networks[END_REF], [START_REF] Bian | NASS: optimizing secure inference via neural architecture search[END_REF], [START_REF] Chen | Multi-objective neural architecture search via non-stationary policy gradient[END_REF], [START_REF] Yang | Fpnet: Customized convolutional neural network for fpga platforms[END_REF], [START_REF] Lu | Automating deep neural network model selection for edge inference[END_REF], [START_REF] Fernandez-Marques | Searching for winograd-aware quantized networks[END_REF], [START_REF] Hong | Memory-efficient models for scene text recognition via neural architecture search[END_REF], [START_REF] Gupta | Accelerator-aware neural network design using automl[END_REF], [START_REF] Huang | Ponas: Progressive one-shot neural architecture search for very efficient deployment[END_REF], [START_REF] Li | Neural architecture search for lightweight non-local networks[END_REF], [START_REF] Xiong | MobileDets: Searching for Object Detection Architectures for Mobile Accelerators[END_REF], [START_REF] Yu | Autofas: Searching lightweight networks for face anti-spoofing[END_REF], [START_REF] Xia | Hnas: Hierarchical neural architecture search on mobile devices[END_REF], [START_REF] Phan | Binarizing mobilenet via evolution-based searching[END_REF], [START_REF] Cassimon | Designing resource-constrained neural networks using neural architecture search targeting embedded devices[END_REF], [START_REF] Lee | Neuralscale: Efficient scaling of neurons for resource-constrained deep neural networks[END_REF], [START_REF] Wang | Apq: Joint search for network architecture, pruning and quantization policy[END_REF], [START_REF] Wang | Neural-architecturesearch-based multiobjective cognitive automation system[END_REF], [START_REF] Piergiovanni | Tiny video networks[END_REF], [START_REF] Lee | Journey towards tiny perceptual super-resolution[END_REF], [START_REF] Ferianc | Vinnas: Variational inferencebased neural network architecture search[END_REF], [START_REF] Lu | Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search[END_REF], [START_REF] Hu | Tf-nas: Rethinking three search freedoms of latency-constrained differentiable neural architecture search[END_REF], [START_REF] Wei | Mixed-precision quantization for cnn-based remote sensing scene classification[END_REF], [START_REF] Tsai | Finding fast transformers: One-shot neural architecture search by component composition[END_REF], [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF], [START_REF] Bruggemann | Automated search for resource-efficient branched multi-task networks[END_REF], [START_REF] Li | LC-NAS: latency constrained neural architecture search for point cloud networks[END_REF], [START_REF] Lee | S3NAS: fast npu-aware neural architecture search methodology[END_REF], [START_REF] Chen | Binarized neural architecture search for efficient object recognition[END_REF], [START_REF] Niu | Real-time execution of large-scale language models on mobile[END_REF], [START_REF] Wang | Efficientnet-elite: Extremely lightweight and efficient cnn models for edge devices by network candidate search[END_REF], [START_REF] Yuan | Enas4d: Efficient multi-stage cnn architecture search for dynamic inference[END_REF], [START_REF] Achararit | Apnas: Accuracy-and-performance-aware neural architecture search for neural hardware accelerators[END_REF], [START_REF] Lu | Multi-objective evolutionary design of deep convolutional neural networks for image classification[END_REF], [START_REF] López | E-dnas: Differentiable neural architecture search for embedded systems[END_REF], [111], [START_REF] Liberis | µnas: Constrained neural architecture search for microcontrollers[END_REF], [START_REF] Yang | Resource-aware pareto-optimal automated machine learning platform[END_REF] Hardware-aware Search Space [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Gupta | Accelerator-aware neural network design using automl[END_REF], [START_REF] Laube | Shufflenasnets: Efficient cnn models through modified efficient neural architecture search[END_REF], [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF], [START_REF] Li | Automated and lightweight network design via random search for remote sensing image scene classification[END_REF] Single Target, Multiple Configurations [START_REF] Jiang | Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search[END_REF], [START_REF] Yu | Softwaredefined design space exploration for an efficient AI accelerator architecture[END_REF], [START_REF] Jiang | Hardware/software co-exploration of neural architectures[END_REF], [START_REF] Lu | On neural architecture search for resource-constrained hardware platforms[END_REF], [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], [START_REF] Chen | You only search once: A fast automation framework for single-stage dnn/accelerator co-design[END_REF], [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF], [START_REF] Lin | Mcunet: Tiny deep learning on iot devices[END_REF], [START_REF] Colangelo | Automl for multilayer perceptron and FPGA co-design[END_REF], [START_REF] Choi | Dance: Differentiable accelerator/network co-exploration[END_REF], [START_REF] Zhang | Dna: Differentiable network-accelerator co-search[END_REF] Multiple Targets [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF], [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF] TABLE

II TAXONOMY OF HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH GOALS

-Hardware-aware search strategy where the search is defined as a multi-objective optimization problem. While searching for the best architecture, the search algorithm calls the traditional evaluator component to get the accuracy of the generated architecture but also a special evaluator that measures the hardware cost metric (e.g., latency, memory usage, energy consumption). Both model accuracy and hardware cost guide the search and enable the NAS to find the most efficient architecture. -On the other hand, the Hardware-aware Search Space approach uses a restricted pool of architectures. Before the search, we either measure the operators' performance on the target platform or we define a set of rules that will refine the search space; eliminate all the architectures' operators that do not perform well on the target hardware. For example, HURRICANE [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF] uses different operator choices for three types of mobile processors: Hexagon DSP, ARM CPU and Myriad Vision Processing Unit (VPU). Accumulated domain knowledge from prior experimentation on a given hardware platform help narrow down the search space. For instance, they do not to use depthwise convolutions for CPU, squeeze and excitation mechanisms for VPU and they do not lower the kernel sizes for a DSP. Such gathered empirical information helps to define three different search spaces according to the targeted hardware platform. Note that after defining the search space with these constraints, the search strategy is similar to the one used by conventional NAS, which means that the search is solely based on the accuracy of the architecture and no other hardware metric is incorporated.

• Single Target, Multiple Configurations: the goal of this category is not only to get the most optimal ar-chitecture that gets the best accuracy but also to get an optimal architecture with latency guranteed to meet the target hardware specification. For example, the authors of FNAS [START_REF] Jiang | Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search[END_REF] define a new hardware search space containing the different FPGA specifications (e.g., tiling configurations). They also use a performance abstraction model to measure the latency of the searched neural architectures without doing any training. This allows them to quickly prune architectures that do not meet the target hardware specifications. In [START_REF] Yang | Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks[END_REF], the authors use the same approach for ASICs and define a hardware search space that contains various ASIC templates. • Multiple Targets: In this third category, the goal is to find the best architecture when given a set of hardware platforms to optimize for. In other words, we try to find a single model that performs relatively well across different hardware platforms. This approach is the most favourable choice, especially in mobile development as it provides more portability. This problem was tackled by [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF], [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF] by defining a multi-hardware search space.

The search space contains the intersection of all the architectures that can be deployed in the different targets. Note that, targeting multiple hardware specifications at once is harder as the best model for a GPU, can be very different to the best model for a CPU (i.e., for GPUs wider models are more appropriate while for CPUs deeper models are).

V. SEARCH SPACES

Two different search spaces have been adopted in the literature to define the search strategies used in HW-NAS: the Architecture Search Space and the Hardware Search Space.

A. Architecture Search Space

The Architecture Search Space is a set of feasible architectures from which we want to find an architecture with high performance. Generally, it defines a set of basic network operators and how these operators can be connected to construct the computation graph of the model. We distinguish two approaches to design an architecture search space:

1) Hyperparameter Optimization for a fixed architecture:

In this approach a neural architecture is given including its operator choices. The objective is limited to optimizing the architecture hyperparameters (e.g., number of channels, stride, kernel size). 2) True Architecture Search Space: The search space allows the optimizer to choose connections between operations and to change the type of operation. Both approaches have their advantages and disadvantages but it is worth mentioning that although former approach reduces the search space size, it requires considerable human expertise to design the search space and introduces a strong bias. Whereas the latter approach decreases the human bias but considerably increases the search space size and hence the search time.

Generally, in the latter approach, we distinguish three types (See figure 10):

• Layer-wise Seach Space, where the whole model is generated from a pool of operators. FBNet Search Space [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], for example, consists of a layer-wise search space with a fixed macro architecture which determines the number of layers and dimensions of each layer where the first and last three layers have fixed operators. The remaining layers need to be optimized. • Cell-based Search Space, where the model is constructed from repeating fixed architecture patterns called blocks or cells. A cell is often a small acyclic graph that represents some feature transformation. The cell-based approach relies on the observation that many effective handcrafted architectures are designed by repeating a set of cells. These structures are typically stacked and repeated a number of time to form larger and deeper architectures. This search space focuses on discovering the architecture of specific cells that can be combined to assemble the entire neural network. Although cell-based search spaces are intuitively efficient to look for the best model in terms of accuracy, they lack flexibility when it comes to hardware specialization [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF]. • Hierarchical Search Space, works in 3 steps: First the cells are defined and then bigger blocks containing a defined number of cells are constructed. Finally the whole model is designed using the generated cells. MNASNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] is a good example of this category of search spaces.

The authors define a factorized hierarchical search space that allows more flexibility compared to a cell-based search space. This allows them to reduce the size of the total search space compared to the global search space. In existing NAS research works, the authors define a macroarchitecture that generally determines the type of networks considered in the search space. When considering CNNs, the macro architecture is usually identical to the one shown in figure 1. Therefore, many works [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF], [START_REF] Liu | Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation[END_REF] differ in the number of layers, the set of operations and the possible hyperparameters values. Recently, the scope of network type is changing. For instance, NASCaps [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF] changes their macro-architecture to allow the definition of capsules. Capsules network [START_REF] Sabour | Dynamic routing between capsules[END_REF] are basically cell-based CNNs where each cell (or capsule) can contain a different CNN architecture.

Other works like [START_REF] Guo | NAT: neural architecture transformer for accurate and compact architectures[END_REF], [START_REF] Tsai | Finding fast transformers: One-shot neural architecture search by component composition[END_REF] focus on transformers and define their macro-architecture as a transformer model. The search consists of finding the number of attention heads and their internal operations. When dealing with the hyperparameters only, the macro architecture can define a variety of network types. Authors in [START_REF] Iqbal | Flexibo: Cost-aware multi-objective optimization of deep neural networks[END_REF], [START_REF] Niu | Real-time execution of large-scale language models on mobile[END_REF] mix different definitions, transformers + CNN and transformers + RNN respectively. They define a set of hyperparameters that encompasses the predefined parameters for different network types at the same time.

Lately, more work [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Shaw | Squeezenas: Fast neural architecture search for faster semantic segmentation[END_REF] have been considering the use of over-parameterized networks (i.e. supernetworks) to speedup the NAS algorithms. These networks consist of adding architectural learnable weights that select the appropriate operator at the right place. Note that these techniques have been applied to transformers as well [START_REF] Wang | HAT: Hardware-aware transformers for efficient natural language processing[END_REF]. Finally, in some research efforts, the pool of operators/architectures is refined with only the models that are efficient in the targeted hardware [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF], [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF]. The search space size is considerably reduced by omitting all the architectures that cannot be deployed.

B. Hardware Search Space (HSS)

Some HW-NAS methods include a HSS component which generates different hardware specifications and optimizations by applying different algorithmic transformations to fit the hardware design. This operation is done before evaluating the model. Although the co-exploration is effective, it increases the search space time complexity significantly. If we take FPGAs as an example, their design space may include: IP instance categories, IP reuse strategies, quantization schemes, parallel factors, data transfer behaviours, tiling parameters, and buffer sizes. It is arguably impossible to consider all these options as part of the search space due to the added search computation cost. Therefore, many existing strategies limit themselves to only few options.

Hardware Search Space (HSS) can be further categorized as follows:

• Parameter-based: The search space is formalized by a set of different parameter configurations. Given a specific data set, FNAS [START_REF] Jiang | Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search[END_REF] finds the best performing model, along with the optimization parameters needed for it to be deployed in a typical FPGA chip for deep learning.

Their HSS consists of four tiling parameters for the convolutions. FNASs [START_REF] Yu | Softwaredefined design space exploration for an efficient AI accelerator architecture[END_REF] extends FNAS by adding more optimization parameters such as loop unrolling. The authors in [START_REF] Jiang | Hardware/software co-exploration of neural architectures[END_REF], [START_REF] Lu | On neural architecture search for resource-constrained hardware platforms[END_REF] used a multi-FPGA hardware search space. The search consists of dividing the architecture into pipeline stages that can be assigned to an FPGA according to its memory and DSP slices, in addition to applying an optimizer that adjusts the tiling parameters. Another example is presented in [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], where the adopted approach takes the global structure of an FPGA and adds all possible parameters to its hardware search space including the input buffer depth, memory interface width, filter size and ratio of the convolution engine. [START_REF] Chen | You only search once: A fast automation framework for single-stage dnn/accelerator co-design[END_REF] searches the internal configuration of an FPGA by generating simultaneously the architecture hyperparameters, the number of processing elements, and the size of the buffer. FPGA/DNN [START_REF] Hao | Fpga/dnn co-design: An efficient design methodology for iot intelligence on the edge[END_REF] proposes two components: Auto-DNN which performs hardware-aware DNN model search and Auto-HLS which generates a synthesizable C code of the FPGA accelerator for explored DNNs. Additional code optimizations such as buffer reallocation and loop fusion on the resulting C-code are added to automate the hardware selection. • Template-based: In this scenario, the search space is defined as a set of pre-configured templates. For example, NASAIC [START_REF] Yang | Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks[END_REF] integrates NAS with Application-Specific Integrated Circuits (ASIC). Their hardware search space includes templates of several existing successful designs.

The goal is to find the best model with the different possible parallelizations among all templates. In addition to the the tiling parameters and bandwidth allocation, the authors in [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF] define a set of FPGA platforms and the search finds a coupling of the architecture and FPGA platform that fits a set of pre-defined constraints (e.g., max latency 5ms)

In general, we can classify the targeted hardware platforms into 3 classes focusing on their memory and computation capabilities:

• Server Processors: this type of hardware can be found in cloud data centers, on premise data centers, edge servers, or supercomputers. They provide abundant computational resources and can vary from CPUs, GPUs, FPGAs and ASICs. When available, machine learning researchers focus on accuracy. Many NAS works consider looking for the best architecture in these devices without considering the hardware-constraints. Nevertheless, many HW-NAS works target server processors to speed up the training process and decrease the extensive resources needed to train a DL architecture and use it for inference. • Mobile Devices: With the rise of mobile devices, the focus has shifted to enable fast and efficient deep learning on smartphones. As these devices are heavily constrained with respect to their memory and computational capabilities, the objective of ML researchers shift to assessing the trade-off between accuracy and efficiency. Many HW-NAS algorithms target smartphones including FBNet [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF] and ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF] (refer to table III). Additionally, because smartphones usually contain system on chips with different types of processors, some research efforts [START_REF] Lu | Automating deep neural network model selection for edge inference[END_REF] have started to explore ways to take advantage of these heterogeneous systems. • Tiny Devices: The strong growth in use of microcontrollers and IoT applications gave rise to TinyML [134]. TinyML refers to all machine learning algorithms dedicated to tiny devices, i.e, capable of on-device inference at extremely low power. One relevant HW-NAS method that targets tiny devices is MCUNet [START_REF] Lin | Mcunet: Tiny deep learning on iot devices[END_REF], which includes an efficient neural architecture search called TinyNAS. TinyNAS optimizes the search space and handles a variety of different constraints (e.g., device, latency, energy, memory) under low search costs. Thanks to the efficient search, MCUNet is the first to achieves >70% ImageNet top-1 accuracy on an off-the-shelf commercial microcontroller.

Targeted HW References CPU [START_REF] Dai | Chamnet: Towards efficient network design through platform-aware model adaptation[END_REF], [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Xu | Latency-aware differentiable neural architecture search[END_REF], [START_REF] Bian | NASS: optimizing secure inference via neural architecture search[END_REF], [START_REF] Piergiovanni | Tiny video networks[END_REF], [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], [START_REF] Lu | Automating deep neural network model selection for edge inference[END_REF], [START_REF] Fernandez-Marques | Searching for winograd-aware quantized networks[END_REF], [START_REF] Fu | Enhancing model parallelism in neural architecture search for multidevice system[END_REF],

[73] GPU [START_REF] Dong | Ppp-net: Platform-aware progressive search for pareto-optimal neural architectures[END_REF], [START_REF] Hsu | MONAS: multi-objective neural architecture search using reinforcement learning[END_REF], [START_REF] Cheng | Instanas: Instance-aware neural architecture search[END_REF], [START_REF] Cai | TEA-DNN: the quest for timeenergy-accuracy co-optimized deep neural networks[END_REF], [START_REF] Mitschke | Gradient based evolution to optimize the structure of convolutional neural networks[END_REF], [START_REF] Nekrasov | Fast neural architecture search of compact semantic segmentation models via auxiliary cells[END_REF], [START_REF] Lu | Nsga-net: Neural architecture search using multiobjective genetic algorithm[END_REF], [START_REF] Sood | Neunets: An automated synthesis engine for neural network design[END_REF], [START_REF] Li | Partial order pruning: For best speed/accuracy trade-off in neural architecture search[END_REF], [START_REF] Stamoulis | Single-path nas: Designing hardwareefficient convnets in less than 4 hours[END_REF], [START_REF] Yu | Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers[END_REF], [START_REF] Hong | Memory-efficient models for scene text recognition via neural architecture search[END_REF], [START_REF] Loni | Deepmaker: A multi-objective optimization framework for deep neural networks in embedded systems[END_REF], [START_REF] Wang | Energyaware neural architecture optimization with fast splitting steepest descent[END_REF], [START_REF] Xu | Latency-aware differentiable neural architecture search[END_REF], [START_REF] Iqbal | Flexibo: Cost-aware multi-objective optimization of deep neural networks[END_REF], [START_REF] Huang | Ponas: Progressive one-shot neural architecture search for very efficient deployment[END_REF], [START_REF] Piergiovanni | Tiny video networks[END_REF], [START_REF] Lee | Neuralscale: Efficient scaling of neurons for resource-constrained deep neural networks[END_REF], [START_REF] Kim | Nemo : Neuro-evolution with multiobjective optimization of deep neural network for speed and accuracy[END_REF], [START_REF] Chen | You only search once: A fast automation framework for single-stage dnn/accelerator co-design[END_REF], [START_REF] Hu | Recurrent neural architecture search based on randomness-enhanced tabu algorithm[END_REF], [START_REF] Baldeon-Calisto | Adaresu-net: Multiobjective adaptive convolutional neural network for medical image segmentation[END_REF], [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF], [START_REF] Bae | Resource optimized neural architecture search for 3d medical image segmentation[END_REF], [START_REF] Schorn | Automated design of error-resilient and hardware-efficient deep neural networks[END_REF], [START_REF] Cassimon | Using neural architecture search to optimize neural networks for embedded devices[END_REF], [START_REF] Wang | Evolving deep neural networks by multi-objective particle swarm optimization for image classification[END_REF], [START_REF] Ferianc | Vinnas: Variational inferencebased neural network architecture search[END_REF], [START_REF] Lu | Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search[END_REF], [START_REF] Hu | Tf-nas: Rethinking three search freedoms of latency-constrained differentiable neural architecture search[END_REF], [START_REF] Li | LC-NAS: latency constrained neural architecture search for point cloud networks[END_REF], [START_REF] Tsai | Finding fast transformers: One-shot neural architecture search by component composition[END_REF], [START_REF] Yuan | Enas4d: Efficient multi-stage cnn architecture search for dynamic inference[END_REF], [START_REF] Achararit | Apnas: Accuracy-and-performance-aware neural architecture search for neural hardware accelerators[END_REF], [START_REF] Lu | Multi-objective evolutionary design of deep convolutional neural networks for image classification[END_REF], [113] MCU [START_REF] Lin | Mcunet: Tiny deep learning on iot devices[END_REF], [START_REF] Fedorov | Sparse: Sparse architecture search for cnns on resource-constrained microcontrollers[END_REF], [111], [START_REF] Weng | Nas-unet: Neural architecture search for medical image segmentation[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Cassimon | Designing resource-constrained neural networks using neural architecture search targeting embedded devices[END_REF] Mobile Devices [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Guo | Single path one-shot neural architecture search with uniform sampling[END_REF], [START_REF] Stamoulis | Single-path nas: Designing hardwareefficient convnets in less than 4 hours[END_REF], [START_REF] Guo | NAT: neural architecture transformer for accurate and compact architectures[END_REF], [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF], [START_REF] Xiong | Resource constrained neural network architecture search: Will a submodularity assumption help?[END_REF], [START_REF] Xia | Hnas: Hierarchical neural architecture search on mobile devices[END_REF], [START_REF] Xiong | MobileDets: Searching for Object Detection Architectures for Mobile Accelerators[END_REF], [START_REF] Yu | Autofas: Searching lightweight networks for face anti-spoofing[END_REF], [START_REF] López | E-dnas: Differentiable neural architecture search for embedded systems[END_REF], [START_REF] Li | Neural architecture search for lightweight non-local networks[END_REF], [START_REF] Fernandez-Marques | Searching for winograd-aware quantized networks[END_REF], [99] ASICs [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Jiang | Hardware/software co-exploration of neural architectures[END_REF], [START_REF] Lu | On neural architecture search for resource-constrained hardware platforms[END_REF], [START_REF] Wang | Apq: Joint search for network architecture, pruning and quantization policy[END_REF], [START_REF] Lee | S3NAS: fast npu-aware neural architecture search methodology[END_REF], [START_REF] Wang | Efficientnet-elite: Extremely lightweight and efficient cnn models for edge devices by network candidate search[END_REF], [START_REF] Zhang | Dna: Differentiable network-accelerator co-search[END_REF], [START_REF] Choi | Dance: Differentiable accelerator/network co-exploration[END_REF],

[100] Edge TPU [START_REF] Tsai | Finding fast transformers: One-shot neural architecture search by component composition[END_REF], [START_REF] Xiong | MobileDets: Searching for Object Detection Architectures for Mobile Accelerators[END_REF], [START_REF] Gupta | Accelerator-aware neural network design using automl[END_REF] TABLE III CLASSIFICATION OF HW-NAS BASED ON THEIR TARGETED HARDWARE.

C. Current Hardware-NAS Trends

Figure 11 shows the different types of platforms that have been targeted by HW-NAS in the literature. In total, we have studied 126 original hardware-aware NAS papers. By target, we mean the platform that the architecture is optimized for. Usually the search algorithm is executed in a powerful machine, but that is not the purpose of our study. "No Specific Target" means that the HW-NAS incorporates hardware agnostic constraints into the objective function such as the number of parameters or the number of FLOPs. In the figure, the tag "Multiple" means multiple types of processing elements have been used in the HW platform. Table III gives the list of references per targeted hardware.

In the figure, we note that the number of research papers targeting GPUs and CPUs has more or less remained constant. However, we can clearly see that FPGAs and ASICs are gaining popularity over the last 3 years. This is consistent with the increasing number of deep learning edge applications. Two recent interesting works are [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF], [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF] both of which target multiple hardware platforms at once.

In figure 12, we illustrates the different DNN operations that compose the architecture search space. First, we divide the CNN into two groups, standard CNN which only utilizes a standard convolution and extended CNN which involves special convolution operations such as the depthwise separable convolution or grouped convolutions. NAS has been mostly dominated by convolutional neural networks as shown in the figure. However, recent works have started explore more operators by incorporating capsule networks [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF], transformers [START_REF] Klyuchnikov | Nas-bench-nlp: Neural architecture search benchmark for natural language processing[END_REF], and GANs [START_REF] Lee | Journey towards tiny perceptual super-resolution[END_REF].

VI. HARDWARE-AWARE NAS PROBLEM FORMULATION

Neural Architecture Search (NAS) is the task of finding a well-performing architecture for a given dataset. It is cast as an optimization problem over a set of decisions that define different components of deep neural networks (i.e., layers, hyperparameters). This optimization problem can simply be seen as formulated in equation 2.

max α∈A f (α, δ) (2)
We denote the space of all feasible architectures as A (also called search space). The optimization method is looking for the architecture α that maximizes the performance metric denoted by f for a given dataset δ. In this context, f can simply be the accuracy of the model.

Although it is important to find networks that provide high accuracy, these NAS algorithms tend to give complex models that cannot be deployed on many hardware devices. To overcome this problem, practitioners consider other objectives, such as the number of model parameters, the number of floating-point operations, and device-specific statistics like the latency or the energy consumption of the model. Different formulations were used to incorporate the hardwareaware objectives within the optimization problem of neural architecture search. We classify these approaches into two classes, single and multi-objective optimization. The single objective optimization can be further classified as two-stage or constrained optimization. Similarly, the multi-objective optimization approach can be further classified as single or multi-objective optimizations. Please refer to figure 13 for a summary of these approaches. These 2 classes are further detailed with examples from the literature in the following sections:

A. Single-Objective Optimization

In this class, the search is realized considering only one objective to maximize, i.e the accuracy. Most of the existing work in the literature [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Yang | Fpnet: Customized convolutional neural network for fpga platforms[END_REF], [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF], [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], that tackle the hardware-aware neural architecture search, try to formulate the multi-objective optimization problem into a single objective to better apply strategies like reinforcement learning or gradient-based methods. We can divide this class into two different approaches: Two-stage optimization and constrained optimization.

1) Two-Stage optimization: In this first category, we retain the original formulation of the NAS problem and then specify the model for deployment. This approach is suboptimal as the final architecture proposed by the NAS is not always the one that gives the best performances on the hardware device. Two-stage optimization consists of applying NAS methods to obtain the best-performing architecture and then in a second stage, specialize this architecture for deployment on a target hardware platform. This specialization performs a series of optimization to fit the hardware requirements. [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF] applies a reinforcement learning agent to find the best quantization bitwidth and pruning level after selecting the most accurate model. [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF] searches over a pre-trained and selected architecture to find the most efficient one in terms of latency and energy consumption.

2) Constrained optimization: In this approach, the hardware-aware characterizations are considered as constraints in the original NAS formulation. The constraints take the form of thresholds to be respected. For example, inference time, energy consumption, memory occupation, etc. The conditions are added as constraints to the optimization problem to enforce requirements like fewer parameters or faster inference time. The threshold and the trade-off between different constraints can be adapted to practical requirements. For such cases, the single-objective optimization problem defined in Equation 2 turns into a constrained optimization problem defined by

max α∈A f (α, δ) subject to g i (α) ≤ T i ∀i ∈ I (3)
Here, g i corresponds to the different constraints taken into account (e.g., latency, memory, energy consumption) and T i denotes the respective threshold. As most of the optimization methods used by NAS (i.e. reinforcement learning and evolutionary algorithms) were designed for unconstrained optimization problems, this formulation is hard to be adopted directly. Therefore, many researchers turned to penalty methods to transform the equation into a single objective function that contains the hardware constraints as well as the accuracy measurement [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF]. For example, MNASNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], uses equation ( 4), where f is the accuracy measurement function, LAT is the latency of the model and T is the threshold. They use a learnable parameter w to control the effect of the hardware constraints on the global objective function.

max α∈A f ((α) • [LAT (α)/T ] w (4)
ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF] uses a loss function that comprises of the cross-entropy (CE) loss and hardware-aware constraints.

L = L CE + λ 1 ||w|| 2 + λ 2 E[latency] (5)
Equation 5 illustrates the loss calculated by the reinforcement learning agent used by ProxylessNAS. λ 1 and λ 2 are learnable parameters that adjust the effect of the efficiency of the overall loss. Specifically, a policy is learned that decides whether to add, remove or keep a layer as well as whether to alter its number of filters.

B. Multi-Objective Optimization

Another approach is to handle multiple fronts in the formalism of a multi-objective optimization problem defined as:

max α∈A f 1 (α, δ), f 2 (α, δ), ..., f n (α, δ) (6)
In this scenario, there is often no single optimal solution that simultaneously maximizes every objective function. Additionally, some objective functions can be conflicting. For instance, trying to minimize the number of parameters while aiming at maximize the accuracy. Therefore, in these situations, the task boils down to finding Pareto-optimal solutions. These techniques can: 1) transform the problem to a single-objective optimization using scalarization method, also called weighted sum method or, 2) solve the multi-objective optimization problem using dedicated heuristics or meta-heuristics such as genetic algorithms or tabu search [START_REF] Hu | Recurrent neural architecture search based on randomness-enhanced tabu algorithm[END_REF]. In general, this second approach provides not one optimal solution but a set of solutions that form the optimal Pareto front of the multi-objective optimization problem. 1) Scalarization Methods: One way to solve the multiobjective optimization problem is to use a scalarization approach. Equation 7 formulates this method. We use a parameterized aggregation function h to transform the multiobjective optimization problem into a single-objective optimization problem.

max α∈A h(f 1 (α, δ), f 2 (α, δ), ..., f n (α, δ)) (7)
The function h can be a weighted sum, a weighted exponential sum, a weighted min-max or a weighted product. However, in most situations, not all Pareto optimal solutions can be found in solving this problem with a fixed setting of the weights. Therefore, the problem is solved for multiple values of the vector w which requires multiple optimization runs. To mitigate the cost of having multiple run, researchers commonly use a set of fixed weights according to the desired tradeoff between the objectives and the practitioners' preferences. Thanks to the scalarization, the problem becomes a singleobjective optimization problem which can be solved by any optimizing methods discussed in Section VII. For example, [START_REF] Hsu | MONAS: multi-objective neural architecture search using reinforcement learning[END_REF] proposes to use the weighted sum as the objective function. The proposed formulation of this function is described by equation 8. ACC refers to the accuracy metric, E refers to the energy consumed by the architecture α and w is a learned parameter to adjust the effect of the energy on the reward function.

max α∈A w • ACC(α, δ) -(1 -w) • E(α) (8)
2) NSGA-II: An alternative approach is to use the elitist evolutionary algorithm NSGA-II [START_REF] Deb | A fast and elitist multiobjective genetic algorithm: Nsga-ii[END_REF] which claims that a linear combination of objectives is suboptimal. Thus, it is important to cast the problem as a multi-objective optimization problem, where a series of models is found along the Pareto front of multiple objectives such as accuracy, computational cost or inference time, and number of parameters. In this algorithm, the architectures are divided into fronts based on their dominance. The architecture in the i-th front is only dominated by all the architectures in the 1, . . . , i -1 fronts. Within each front, the architectures are prioritized by the crowding distance, which is computed by the sum of all the neighborhood distances across all the objectives. Hardwareaware NAS works [START_REF] Kim | Nemo : Neuro-evolution with multiobjective optimization of deep neural network for speed and accuracy[END_REF], [START_REF] Lu | Nsga-net: Neural architecture search using multiobjective genetic algorithm[END_REF], [START_REF] Chu | Multiobjective reinforced evolution in mobile neural architecture search[END_REF] have been using NSGA-II algorithm to ensure the exploration of diverse architectures in the search space. Moreover, NSGANet [START_REF] Lu | Nsga-net: Neural architecture search using multiobjective genetic algorithm[END_REF] uses Bayesian Optimization to profit from search history. MoreMNAS [START_REF] Chu | Multiobjective reinforced evolution in mobile neural architecture search[END_REF] uses a hybrid search strategy combining NSGA-II with reinforcement learning to regulate arbitrary mutations.

VII. SEARCH STRATEGIES

In this section, we will review all the search strategies used in hardware-aware NAS and how the hardware cost is integrated into each one of them. Following the algorithm presented in 1, the search strategy must define:

• An accuracy evaluation method: usually it is the actual training of the architecture on a subset of the dataset provided by the user D train . However, training each architecture while searching is computationally expensive and takes several GPU hours. Therefore, some speedup methods that try to decrease the training time and estimate the accuracy without effectively training an architecture are used and presented in section VII-C. • A hardware cost evaluation method: the hardware metrics need to be measured either by a real-time execution of each architecture on the targeted platform or an estimation method. The approaches used are presented in section VIII.

• A search algorithm: The search algorithm defines how the architectures are sampled from the search space, either with a generator or at random, and how the search explores this search space by updating the sampling strategy to better fit the best models according to an objective function. See the next section VII-A. Update best architecture < s, w(s)); Update sampling algorithm with f (acc, hw c ost); until convergence or time limit is reached;

Strategy References Evolutionary Algorithm [START_REF] Elsken | Efficient multi-objective neural architecture search via lamarckian evolution[END_REF], [START_REF] Lu | Nsga-net: Neural architecture search using multiobjective genetic algorithm[END_REF], [START_REF] Dai | Chamnet: Towards efficient network design through platform-aware model adaptation[END_REF], [START_REF] Sood | Neunets: An automated synthesis engine for neural network design[END_REF], [START_REF] Yu | Softwaredefined design space exploration for an efficient AI accelerator architecture[END_REF], [START_REF] Schorn | Automated design of error-resilient and hardware-efficient deep neural networks[END_REF], [START_REF] Baldeon-Calisto | Adaresu-net: Multiobjective adaptive convolutional neural network for medical image segmentation[END_REF], [START_REF] Loni | Deepmaker: A multi-objective optimization framework for deep neural networks in embedded systems[END_REF], [START_REF] Phan | Binarizing mobilenet via evolution-based searching[END_REF], [START_REF] Xia | Hnas: Hierarchical neural architecture search on mobile devices[END_REF], [START_REF] Wang | Apq: Joint search for network architecture, pruning and quantization policy[END_REF], [START_REF] Piergiovanni | Tiny video networks[END_REF], [START_REF] Lu | Nsganetv2: Evolutionary multi-objective surrogate-assisted neural architecture search[END_REF], [START_REF] Wang | Neural-architecturesearch-based multiobjective cognitive automation system[END_REF], [START_REF] Lin | Mcunet: Tiny deep learning on iot devices[END_REF], [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF], [START_REF] Colangelo | Automl for multilayer perceptron and FPGA co-design[END_REF], [START_REF] Wang | Efficientnet-elite: Extremely lightweight and efficient cnn models for edge devices by network candidate search[END_REF], [START_REF] Lu | Multi-objective evolutionary design of deep convolutional neural networks for image classification[END_REF] Reinforcement Learning [START_REF] Zhou | Resource-efficient neural architect[END_REF], [START_REF] Hsu | MONAS: multi-objective neural architecture search using reinforcement learning[END_REF], [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Nekrasov | Fast neural architecture search of compact semantic segmentation models via auxiliary cells[END_REF], [START_REF] Cheng | Instanas: Instance-aware neural architecture search[END_REF], [START_REF] Laube | Shufflenasnets: Efficient cnn models through modified efficient neural architecture search[END_REF], [START_REF] Jiang | Accuracy vs. efficiency: Achieving both through fpga-implementation aware neural architecture search[END_REF], [START_REF] Yu | Network slimming by slimmable networks: Towards one-shot architecture search for channel numbers[END_REF], [START_REF] Xiong | Resource constrained neural network architecture search: Will a submodularity assumption help?[END_REF], [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF], [START_REF] Jiang | Hardware/software co-exploration of neural architectures[END_REF], [START_REF] Bae | Resource optimized neural architecture search for 3d medical image segmentation[END_REF], [START_REF] Guo | NAT: neural architecture transformer for accurate and compact architectures[END_REF], [START_REF] Cassimon | Using neural architecture search to optimize neural networks for embedded devices[END_REF], [START_REF] Lu | On neural architecture search for resource-constrained hardware platforms[END_REF], [START_REF] Bian | NASS: optimizing secure inference via neural architecture search[END_REF], [START_REF] Chen | Multi-objective neural architecture search via non-stationary policy gradient[END_REF], [START_REF] Yang | Fpnet: Customized convolutional neural network for fpga platforms[END_REF], [START_REF] Yang | Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks[END_REF], [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], [START_REF] Gupta | Accelerator-aware neural network design using automl[END_REF], [START_REF] Xiong | MobileDets: Searching for Object Detection Architectures for Mobile Accelerators[END_REF], [START_REF] Chen | You only search once: A fast automation framework for single-stage dnn/accelerator co-design[END_REF], [START_REF] Cassimon | Designing resource-constrained neural networks using neural architecture search targeting embedded devices[END_REF], [START_REF] Lee | Neuralscale: Efficient scaling of neurons for resource-constrained deep neural networks[END_REF], [START_REF] Lee | Journey towards tiny perceptual super-resolution[END_REF], [START_REF] Fu | Enhancing model parallelism in neural architecture search for multidevice system[END_REF], [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF], [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF], [START_REF] Niu | Real-time execution of large-scale language models on mobile[END_REF], [START_REF] Achararit | Apnas: Accuracy-and-performance-aware neural architecture search for neural hardware accelerators[END_REF] Gradient-based Methods [START_REF] Zhang | You only search once: Single shot neural architecture search via direct sparse optimization[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Guo | Single path one-shot neural architecture search with uniform sampling[END_REF], [START_REF] Stamoulis | Single-path NAS: device-aware efficient convnet design[END_REF], [START_REF] Stamoulis | Single-path nas: Designing hardwareefficient convnets in less than 4 hours[END_REF], [START_REF] Shaw | Squeezenas: Fast neural architecture search for faster semantic segmentation[END_REF], [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF], [START_REF] Nekrasov | Fast neural architecture search of compact semantic segmentation models via auxiliary cells[END_REF], [START_REF] Xu | Latency-aware differentiable neural architecture search[END_REF], [START_REF] Hong | Memory-efficient models for scene text recognition via neural architecture search[END_REF], [START_REF] Huang | Ponas: Progressive one-shot neural architecture search for very efficient deployment[END_REF], [START_REF] Li | Neural architecture search for lightweight non-local networks[END_REF], [START_REF] Yu | Autofas: Searching lightweight networks for face anti-spoofing[END_REF], [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF], [START_REF] Ferianc | Vinnas: Variational inferencebased neural network architecture search[END_REF], [START_REF] Hu | Tf-nas: Rethinking three search freedoms of latency-constrained differentiable neural architecture search[END_REF], [START_REF] Tsai | Finding fast transformers: One-shot neural architecture search by component composition[END_REF], [START_REF] Bruggemann | Automated search for resource-efficient branched multi-task networks[END_REF], [START_REF] Li | LC-NAS: latency constrained neural architecture search for point cloud networks[END_REF], [START_REF] Lee | S3NAS: fast npu-aware neural architecture search methodology[END_REF], [START_REF] Chen | Binarized neural architecture search for efficient object recognition[END_REF], [START_REF] Choi | Dance: Differentiable accelerator/network co-exploration[END_REF], [START_REF] Yuan | Enas4d: Efficient multi-stage cnn architecture search for dynamic inference[END_REF], [START_REF] López | E-dnas: Differentiable neural architecture search for embedded systems[END_REF], [111], [START_REF] Zhang | Dna: Differentiable network-accelerator co-search[END_REF] Bayesian Optimization [START_REF] Cai | TEA-DNN: the quest for timeenergy-accuracy co-optimized deep neural networks[END_REF], [START_REF] Fedorov | Sparse: Sparse architecture search for cnns on resource-constrained microcontrollers[END_REF], [START_REF] Iqbal | Flexibo: Cost-aware multi-objective optimization of deep neural networks[END_REF], [START_REF] Liberis | µnas: Constrained neural architecture search for microcontrollers[END_REF], [START_REF] Gennari Do Nascimento | Finding Non-Uniform Quantization Schemes using Multi-Task Gaussian Processes[END_REF] Random Search [START_REF] Li | Automated and lightweight network design via random search for remote sensing image scene classification[END_REF] Hybrid EA + RL [START_REF] Chu | Multiobjective reinforced evolution in mobile neural architecture search[END_REF], [START_REF] Chu | Fast, accurate and lightweight super-resolution with neural architecture search[END_REF] EA + Bayesian Optimization [START_REF] Yang | Resource-aware pareto-optimal automated machine learning platform[END_REF] EA + Gradient-based [

69] TABLE IV SEARCH STRATEGIES OF HW-NAS

A. Search Algorithm 1) Reinforcement Learning: Most HW-NAS methods use reinforcement learning to search for the best architecture [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Zhang | When neural architecture search meets hardware implementation: from hardware awareness to co-design[END_REF], [START_REF] Yang | Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks[END_REF], [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF], [START_REF] Bender | Can weight sharing outperform random architecture search? an investigation with tunas[END_REF], [START_REF] Lu | On neural architecture search for resource-constrained hardware platforms[END_REF], [START_REF] Yang | Fpnet: Customized convolutional neural network for fpga platforms[END_REF] because the NAS problem is easily modeled as a Markov Decision Process. The RL controller samples an architecture from the search space and is rewarded according to its accuracy and hardware cost. The agent will then adjust its weights to generate better models. Different works differ on how they represent the agent's policy (set of actions) and how they optimize it.

Using reinforcement learning, MNASNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] tries to find the Pareto optimal solution of the objective function described in equation 4. It uses a sample-eval-update loop to train its RNN controller. To generate a block, the controller will first choose two hidden states (i.e., outputs of previous blocks) as inputs. Then, it will select an operation to apply to each one of them. Finally, it selects a combination method (e.g., addition or concatenation) to obtain the final output of the block. This implementation has been defined by NASNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF] before. Once a model is sampled, it is trained on the target task to get its accuracy and deployed on real phones to get its latency. The system computes the reward value and adjusts the controller parameters accordingly.

A similar approach was used by FPNet [START_REF] Yang | Fpnet: Customized convolutional neural network for fpga platforms[END_REF] but the RNN controller predicts only the architectural hyperparameters (i.e., number of filters, filter height, filter width, stride height, stride width. etc.) while keeping The macro architecture fixed.

Codesign-NAS [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF] proposes to use reinforcement learning to explore both architecture and hardware search spaces. The authors investigated three RL-based search strategies and use the REINFORCE algorithm to improve the accuracy and efficiency of image classification on FPGA. The first strategy consists of combining the two search spaces and updating the CNN and accelerator options at the same time. The second method uses two different controllers, one to learn the CNN architecture and another one to select the best FPGA options. The last one is a conventional NAS where they separately search for the best model in terms of accuracy then as a completely separate step look for the most efficient model. An expected result is that this latter strategy gives bad results when it comes to the constrained environment. An interesting result is that the second method (i.e., phase search) seems to be the most promising achieving higher rewards in most of their experiments.

2) Evolutionary Algorithm: Another popular strategy in conventional NAS [START_REF] Zoph | Neural architecture search with reinforcement learning[END_REF], [START_REF] Real | Large-scale evolution of image classifiers[END_REF] is the use of evolutionary algorithms. Generally, neuro-evolutionary NAS evolves a population of models, sample some models to generate offsprings by applying some mutations (recombination is not used in neuro-evolutionary NAS), and finally evaluate the fitness of the offsprings and update the new generation by adding the best ones to the population. When it comes to integrating the hardware constraints to the NAS algorithms, some research efforts have used evolutionary algorithms [START_REF] Wang | HAT: Hardware-aware transformers for efficient natural language processing[END_REF], [START_REF] Lin | Mcunet: Tiny deep learning on iot devices[END_REF], [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF], [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF].

TinyNAS [START_REF] Lin | Mcunet: Tiny deep learning on iot devices[END_REF] uses a hardware-aware search space approach. It first optimizes the search space to fit the tiny and diverse resource constraints and then performs an evolution search algorithm within the optimized search space to find the most accurate model. The evolution search is performed on a trained super network that contains all the possible subnetworks. First, they sample 100 satisfying networks that fit the resource constraints. Then, just like conventional NAS, they measure the validation accuracy of each model, mutate the offsprings and update the new generation. This process is repeated for 30 iterations.

Once-for-all [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF] proposes a special technique to train the supernetwork using gradient-based methods called progressive shrinking. A supernetwork is another name for the over-parameterized network. In their version, the supernetwork includes the highest values for the width, depth and channel number in each convolution. The process begins by training for larger width, depth and channel numbers and then finetune the network for the smaller sizes. Once the supernetwork is trained, they specialize the network using an evolutionary algorithm for different hardware settings with a combined cost of accuracy and latency.

HAT [START_REF] Wang | HAT: Hardware-aware transformers for efficient natural language processing[END_REF] is interesting because to our knowledge it is the only work that is trying to search for efficient transformers, targeting NLP tasks. The authors perform an evolutionary search with hardware latency constraints on a SuperTransformer. This means the engine only adds SubTransformers with latency smaller than the hardware constraint to the population.

NASCaps [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF] proposes a NAS framework that generates Capsule Networks along with CNN. The proposed framework uses a multi-objective Genetic Algorithm (based on NSGA-II, see section 4.2) to pick the Pareto optimal solutions. The two key operations are the crossover and mutation. In the crossover, they define the splitting point by ensuring that the generated DNN is made up of at least on initial convolution layer and a minimum of two capsules. No standard convolution is placed between two capsule layers. The mutation is performed by randomly choosing one of the layer descriptors from the candidate network and modifying one of the main parameters of the selected layer.

3) Gradient-Based Methods: Arguably the most promising search strategy promising in terms of results, Gradient-based methods are increasingly used by hardware-aware NAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF], [START_REF] Shaw | Squeezenas: Fast neural architecture search for faster semantic segmentation[END_REF], [START_REF] Nayman | XNAS: neural architecture search with expert advice[END_REF] and NAS generally. Running the search separated from the evaluation requires a lot of time and computation. Therefore, a common idea is to have a supernetwork that can emulate any child model in the search space. This means that different parts of the graph share weights between their common edges. This idea of weight sharing has the advantage of considerably reducing the search time. Gradientbased methods train the supernetwork to simultaneously get the architecture parameters and weights. This technique has been initiated by DARTS [150].

ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF] is one of the papers pioneering this method. It defines a supernetwork with binary architecture parameters (i.e., 1 implies that the operator is selected in the layer and 0 otherwise). The loss function combines the cross-entropy and the latency to better update the weights and architecture parameters. An approach similar to BinaryConnect [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF] is used to update the binary architecture parameters using an approximation of the gradient w.r.t architecture parameters.

FBNet [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF] proposes to use differentiable neural architecture search to discover hardware-aware efficient CNNs. They also use a combination of the cross-entropy and latency to train their supernetwork using stochastic gradient descent. Rather than using binary parameters, they relax the problem of finding the best architecture to finding a distribution that yields to the best model.

SqueezeNAS [START_REF] Shaw | Squeezenas: Fast neural architecture search for faster semantic segmentation[END_REF] focuses on semantic segmentation and uses a method very similar to FBNet [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF]. HTAS [START_REF] Jiang | Hardware-aware transformable architecture search with efficient search space[END_REF] uses a gradient-based method to find the best width and depth for their transformable CNNs.

XNAS [START_REF] Nayman | XNAS: neural architecture search with expert advice[END_REF] proposes to use the prediction with expert advice theory [START_REF] Cesa-Bianchi | Prediction, Learning, and Games[END_REF] for the selection. It leverages the Exponentiated-Gradient algorithm (EG) [START_REF] Kivinen | Exponentiated gradient versus gradient descent for linear predictors[END_REF] rather than the classical gradient descent which prevents the decay of architecture weights to promote the selection of arbitrary architectures.

4) Random Search & Bayesian Optimization: The most convenient and easiest search strategy to implement is the random search strategy. Generally, random search and Bayesian optimization are used for hyperparameter optimization. Therefore, most of the existing works that have adopted random search optimize the architectural hyperparameters within a fixed macro architecture. They argue that it is more important to design the architecture search space for the targeted hardware platform than to complicate the search strategy and incorporate the hardware constraints in the objective function. NASNet [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF], for example, tried both methods (i.e., reinforcement learning and random search). They found that with reinforcement learning the results are slightly better (Top-1 accuracy on CIFAR-10, 0,912 -0,925).

Li et al. [START_REF] Li | Random search and reproducibility for neural architecture search[END_REF] investigate the use of random search on two standard NAS benchmarks (i.e., PTB and CIFAR-10). They use an approach that is similar to ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF] which allows them to train a single network at a time and thus reduce the memory footprint with weight sharing. As a result, they show that random search with early-stopping is a competitive NAS baseline as it outperforms ENAS [START_REF] Pham | Efficient neural architecture search via parameter sharing[END_REF].

Furthermore, random search with weight-sharing outperforms random search with early stopping, achieving SOTA results on PTB. A more recent investigation [START_REF] Stamoulis | Single-path nas: Designing hardwareefficient convnets in less than 4 hours[END_REF] showed that random search cost time is not negligible and comparable to NAS methods.

B. Non-differentiable Hardware Constraints

In many of these search strategies, the authors use an over-parameterized network. Therefore, the loss function must be differentiable w.r.t the architectural parameters. Also, we need to check that the incorporated hardware cost is differentiable. Several methods have been used to make the gradient computation over discrete variables possible. In this section, we review some extensions to the differentiable architecture search space [150].

a) Gumbel Softmax: [START_REF] Jang | Categorical reparameterization with gumbel-softmax[END_REF] One way to relax the discrete variables is to use the Gumbel softmax function. It helps insert some random noise following the Gumbel distribution so that the gradient computation is possible. This technique was used by FBNet [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Wu | Mixed precision quantization of convnets via differentiable neural architecture search[END_REF].

b) Estimated Continuous Function: ProxylessNAS [53] mimics the concept of BinaryConnect [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF]. They approximately estimate the gradient w.r.t the architecture parameters using the gradient w.r.t the binary gates. To reduce the computational cost, they compare the gates two-by-two by factorizing the task of using one out of N paths into multiple binary selection tasks. c) REINFORCE algorithm: An alternative approach to BinaryConnect is also proposed by ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF]. They utilized REINFORCE to train the binarized weights. Furthermore, they combine the gradient-based update rule to the REINFORCE updates to form a new general update rule for the architecture parameters.

C. Runtime Performance Optimization Strategies

In this section, we discuss the various methods that are used to speed up the NAS algorithms. We are not going to mention weight sharing as it was described in section VII-A3, but we note that it is another way to accelerate the process tied to the search strategy used.

While NAS methods are efficient at finding state of the art architectures, their search cost is extremely high. Undoubtedly, the most time-consuming component of the NAS process is the training of each model to obtain the validation accuracy and thus making an evaluation. This training alone requires hours for just one architecture on a GPU. For example, training ResNet-50 requires 29 hours on 8 Tesla P100 GPUs [START_REF] Mikami | Imagenet/resnet-50 training in 224 seconds[END_REF]. For NAS, we want to estimate which architecture(s) is the most accurate without needing the exact accuracy. Therefore, we will review the accuracy estimation methods. a) Early Stopping: Early stopping is a technique that is used by several NAS works [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Li | Random search and reproducibility for neural architecture search[END_REF]. The idea is to train the models for few epochs (usually five epochs) and to take the validation accuracy of this premature model as an approximation of the performance of the fully trained model. b) Hot start: HotNAS [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF] proposes the idea of hot starting. The search heuristics does not start from a random model but it starts with an efficient model. Starting from ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF] and MNASNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] models, they obtained models with better latency and slightly lower accuracy: 91.47 and 92.2 for HotNAS and ProxylessNAS repectively. In addition to obtaining efficient models, the NAS search is faster as we start from a suboptimal model. c) Proxy datasets: Using proxy datasets like CIFAR-10 helps train the model with fewer data elements and then finetune it to be used for higher resolutions and more complex tasks.

d) Accuracy Prediction Models: Another approach to speed up the training process is to directly predict the accuracy based on the architecture specifications and dataset characteristics. The most used prediction models are the following.

Peephole [START_REF] Deng | Peephole: Predicting network performance before training[END_REF] is a framework to predict network performance before training based on its architecture. It encodes the architectures into vectors and inputs them onto an LSTM layer. The result is concatenated to the number of epochs and passed to an MLP which outputs the predicted accuracy. Similarly, PNAS [START_REF] Liu | Progressive neural architecture search[END_REF] uses an RNN model to handle variablesized inputs which consist of multiple LSTM layers. The final LSTM hidden state goes through a fully-connected layer and sigmoid layer to regress the validation accuracy.

Another approach by [START_REF] Baker | Accelerating neural architecture search using performance prediction[END_REF] uses a simple regressio model (SRM regressor) with features like the number of weights and the number of layers as well as the hyperparameters, to predict the learning curves during the training process.

NeuNetS [START_REF] Sood | Neunets: An automated synthesis engine for neural network design[END_REF] defines a special component called TAP, which stands for Train-less Accuracy Predictor. TAP is designed to perform fast and reliable CNN accuracy predictions.

Recently, [START_REF] Tang | A semi-supervised assessor of neural architectures[END_REF] uses a semi-supervised performance predictor. To locate latent feature representations, the architecture graph representation is used by an auto-encoder, which is then fine-tuned using a graph similarity calculation. Finally, a graph convolution neural network is used to output the final performance.

VIII. HARDWARE COST ESTIMATION MODELS

An important component in HW-NAS is the hardware cost measurements. First of all, many metrics have been used in order to evaluate the hardware efficiency of an architecture including the number of FLOPs, the number of parameters, the latency or execution time of the inference, the energy consumption, the memory footprint, the area of the hardware platform, etc. a) FLOPs & Model Size: The first HW-NAS approaches that were published in 2016 and 2017 [START_REF] Smithson | Neural networks designing neural networks: Multi-objective hyper-parameter optimization[END_REF], [START_REF] Gordon | Morphnet: Fast & simple resource-constrained structure learning of deep networks[END_REF] use the number of paramaters and number of FLOPs as an objective function to minimize. These techniques assume that the number of operations is positively correlated to the execution time. However, recent works have proved that two models can have the same number of FLOPs but different latencies [START_REF] Bouzidi | Performance prediction for convolutional neural networks in edge devices[END_REF], [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Wang | HAT: Hardware-aware transformers for efficient natural language processing[END_REF]. For example, NASNet-A and MobileNetV1 have roughly similar number of FLOPs, yet, NASNet-A can have slower latency due to the hardware-unfriendly structure. Therefore, using FLOPs as the hardware cost metric is not efficient and may return suboptimal models. On the other hand, using the model size represented by the number of parameters allows to reduce the memory footprint and tends to be considered as an automatic method to search for compressed models [START_REF] Wang | Apq: Joint search for network architecture, pruning and quantization policy[END_REF]. b) Latency: Searching for low-latency architectures at inference time is crucial for real world application such as autonomous driving and traffic control. Moreover, resourcelimited devices have latency constraints. Thus, a lot of works consider the latency in their objective function and search for the trade-off between inference time and accuracy. c) Energy Consumption: Energy is usually profiled by the provided hardware platform profilers such as nvprof by NVIDIA. The energy can be formalized either as the peak power consumption or the average power, both metrics are used by different HW-NAS works including [START_REF] Hsu | MONAS: multi-objective neural architecture search using reinforcement learning[END_REF], [START_REF] Yang | Netadapt: Platform-aware neural network adaptation for mobile applications[END_REF], [START_REF] Gong | Mixed precision neural architecture search for energy efficient deep learning[END_REF].

d) Area: Another metric that interest chip manufacturers is its area. The goal is to get the smallest chip possible that could run the best model. [START_REF] Yang | Co-Exploration of Neural Architectures and Heterogeneous ASIC Accelerator Designs Targeting Multiple Tasks[END_REF] uses MAESTRO [START_REF] Kwon | Understanding reuse, performance, and hardware cost of dnn dataflow: A data-centric approach[END_REF] to explore the area and power consumption and search for the best model and best ASICs templates within a set of predefined ones. The area of the circuit is also a good indicator of the static power consumption. These two values are correlated.

e) Memory footprint: We can get the model size by calculating the number of parameters it needs to learn but a more efficient way is to profile how much memory it uses while running; this is the memory footprint. Having a low memory footprint is important for edge devices. These devices are not able to run models with high memory footprint. To reduce the memory footprint in edges devices, techniques like those presented in Section IX are applied.

Table VI summarizes all the methods used to estimate the latency, energy consumption in different NAS methods.

Real-world measurements provides a high accuracy in measuring the hardware efficiency of an architecture. MnasNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] uses this method in the exploration. It achieves 75.2% top-1 accuracy with 78ms latency on a Pixel phone platform, which is 1.8x faster than MobileNetV2 with 0.5% higher accuracy. However, this method considerably slows down the search algorithm by averaging hundreds of runs to get precise measurements. Additionally, this strategy is not scalable and requires that all the hardware platforms are available. This solution could be costly and needs a lot of mobile devices and software engineering work. That's why many works tend to use a prediction model [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Zhang | Fast hardware-aware neural architecture search[END_REF], [START_REF] Jiang | Hardware/software co-exploration of neural architectures[END_REF], [START_REF] Lu | On neural architecture search for resource-constrained hardware platforms[END_REF], [START_REF] Zhang | When neural architecture search meets hardware implementation: from hardware awareness to co-design[END_REF], [START_REF] Hao | Fpga/dnn co-design: An efficient design methodology for iot intelligence on the edge[END_REF] Method How the method is achieved ? Hardware Cost Metric References

Real-time measurements

The sampled model is executed on the hardware target while searching.

Latency MNASNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] NetAdapt [START_REF] Yang | Netadapt: Platform-aware neural network adaptation for mobile applications[END_REF] Z. Guo or a pre-collected lookup table [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF], [START_REF] Abdelfattah | Best of both worlds: Automl codesign of a cnn and its hardware accelerator[END_REF], [START_REF] Jiang | Standing on the shoulders of giants: Hardware and neural architecture co-search with hot start[END_REF] or computing an analytical estimation [START_REF] Zhang | When neural architecture search meets hardware implementation: from hardware awareness to co-design[END_REF], [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF].

In ProxylessNAS [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], the authors have developed three latency prediction models for three different platforms: a mobile phone (Google Pixel 1), a GPU (NVIDIA V100) and a CPU (Intel Xeon). To build their mobile latency predictors, they use the type of operators, input and output feature map sizes and other architectural hyperparameters as features. The real values for Pixel 1 phone have been measured with Tensorflow-Lite as software. On ImageNet, their model achieves 3.1% better top-1 accuracy than MobileNetV2, while being 1.2x faster with measured GPU latency.

In NASCaps [START_REF] Marchisio | Nascaps: A framework for neural architecture search to optimize the accuracy and hardware efficiency of convolutional capsule networks[END_REF], the functional behavior of a given specialized CNN and CapsNet hardware accelerator is modeled at a high level, to quickly estimate the memory usage, energy consumption, and latency. The HW platform, the ASIC CapsAcc in the paper, is described at the RTL-level. Using a VLSI CAD tool and the RTL specifications, energy, memory, latency costs of elementary operations are measured.

Elementary operations' cost correspond for example to the number of cycles and energy required to execute a layer. These values are then multiplied by the number of occurrences of each operation in the architecture and accumulated to obtain the total cost for the latency and energy.

Although these techniques are efficient, they require hardware experts to build the models. For the lookup table method, for example, the researcher needs to dedicate a lot of time to optimize the code of each operator/architecture in the targeted hardware, which requires compilation knowledge. Similarly, to build the best model predictor, the researcher needs expert knowledge to select the best features and verify the results. Therefore, these methods impose a barrier to non-hardware experts.

In order to fairly compare the accuracy of each method, we computed the latency of each architecture in NAS-Bench-101 [START_REF] Ying | Nas-bench-101: Towards reproducible neural architecture search[END_REF] and compare it to the real-time measurements, according to three collecting methods: lookup table, prediction model and analytical estimation. For the lookup table, we calculated the latency of each operator used in the cell of the benchmark including Identity, Conv3x3BnRelu, Conv1x1BnRelu, MaxPool3x3, BottleneckConv3x3, BottleneckConv5x5, and MaxPool3x3Conv1x1. When a cell is generated, we sum the latency of the constructing operators and we get the latency of the whole cell. For the prediction model, we used two different models: a simple MLP and XGBoost, both trained on the real-time measurements of a portion of the benchmark (training set). We choose these two methods because they are both used by popular HW-NAS in [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF] and [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF] respectively. Lastly, for the analytical estimation we computed the number of MAC for the cell and multiply that by the latency of one multiply-add tensor instruction.

This experiment was run on a Tesla K80 GPU, the prediction model MLP had to run for 50 epochs with early stopping. NAS-Bench-101 defines more than 400k cells, we have done our test on 165,580 cells. The search algorithm used to calculate the search time is an evolutionary algorithm based on the validation accuracy given by the benchmark and the latency measured by the different methods. Table V presents the results of the accuracy values of different methods. As expected, the analytical estimation does not produce good results compared to the prediction models or the lookup table method. The prediction models even with a simple XGBoost give the best results and accelerate the search more than 5 times compared to the real-time measurements.

IX. OTHER CONSIDERATIONS FOR HARDWARE-AWARE NAS

Prior and Parallel to the hardware-aware NAS efforts, researchers have been working on reducing the memory footprint of deep learning models and execution time to facilitate the efficient deployment and design hardware-friendly models. Two main methods have been used, namely handcrafting new operators that are more efficient such as separable convolutions [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF], grouped convolutions [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], and applying deep learning optimizations such as quantization [START_REF] Zhang | Adaptive precision training: Quantify back propagation in neural networks with fixed-point numbers[END_REF] and pruning [START_REF] Asif | Ensemble knowledge distillation for learning improved and efficient networks[END_REF]. This latter method is automated by several NAS works to compress the model and make it possible to execute on different hardware accelerators. Moreover, reducing the number of learnable parameters makes the training faster. For instance, SAL [START_REF] Hacene | Processing and learning deep neural networks on chip[END_REF] reduces the number of parameters of ResNet-56 from 1.22M to 0.36M without a big degradation of the model's accuracy; 0.6% decrease.

In this section, we are going to define the two most used deep learning optimizations and review the NAS works that focus on searching the right optimization parameters.

A. Automatic Mixed-Precision Quantization

In the deep learning compression field, quantization is one of the most important methods. Starting with BinaryConnect [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF] in 2015 to binarize CNNs weights, it is now implemented in many deep learning software such as PyTorch or Tensorflow. The idea is that the weights of the deep learning model do not have to be represented in full 32-bits precision and can be represented in 8-bits precision, or even binary precision in some cases, without significantly decreasing the model's accuracy. This idea was extended to the activations and weights [172]. Recently, mixed-precision quantization that applies different bitwidth values for different layers in the same network is more commonly used.

HAQ [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF] implemented a dedicated reinforcement learning agent that learns to assign the right bitwidth to each layer. Their goal is to specialize the architecture to a specific hardware platform by incorporating hardware constraints and accuracy into the reward function. For each layer, the agent takes two decisions: one for the weights and another one for the activation.

Single-Path [START_REF] Guo | Single path one-shot neural architecture search with uniform sampling[END_REF] proposes an evolutionary algorithm that searches for the mixed-precision quantization policy. But the search costs a huge amount of time and processing as the space for mixed-precision is enormous.

BP-NAS [173] cast the mixed-precision quantization problem as a constrained optimization. In addition, it proposes a new constraint that encourages the model to focus on valid architectures while imposing a large punishment to the quantization outside the valid domain.

Although, these works present interesting solutions to the mixed-precision problem, the huge search space and computational cost of the learning process still represent a real challenge. Furthermore, mixed-precision representation requires specific MAC architectures with scalable-precision. This places a cap on the power efficiency, according to this study [START_REF] Camus | Review and benchmarking of precision-scalable multiply-accumulate unit architectures for embedded neural-network processing[END_REF]. With this in mind, [START_REF] Anonymous | Uniform-precision neural network quantization via neural channel expansion[END_REF] proposes a uniform quantization search algorithm called neural channel expansion (NCE). NCE expands the number of channel of a layer when it is more sensitive to the quantization error while maintaining the same precision level.

B. Automatic Pruning

The second prominent compression method is pruning. Pruning methods eliminate some neurons or connections according to a defined criterion to reduce the number of parameters in the architecture. Generally, we evaluate the importance of the neurons, we prune the least important ones, and finally, we fine-tune the network.

AMC [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF] proposes an automatic model compression that looks for the optimal sparsity for each layer during pruning. The authors trained a reinforcement learning agent to predict the best sparsity for given hardware. The reward function includes accuracy and FLOPs after pruning the architecture.

X. Dong and Y. Yang in [START_REF] Dong | Network pruning via transformable architecture search[END_REF] propose to prune the overparameterized network without performance damage. They directly search within their NAS process for a network with a flexible channel and layer sizes. ABCPruner [START_REF] Lin | Channel pruning via automatic structure search[END_REF] uses artificial bee colony algorithm to efficiently find the optimal pruned structure. Another worth mentioning paper is Partial Order Pruning [START_REF] Li | Partial order pruning: For best speed/accuracy trade-off in neural architecture search[END_REF] which proposes a hardware-aware NAS that prunes the search space with a partial order assumption to look for the best speed and accuracy trade-off.

C. Security and Reliability Considerations

Other NAS methods try to address safety-critical issues by discovering architectures that are robust against adversarial attacks [START_REF] Vargas | Evolving robust neural architectures to defend from adversarial attacks[END_REF], [START_REF] Cubuk | Intriguing properties of adversarial examples[END_REF]. RAS [START_REF] Vargas | Evolving robust neural architectures to defend from adversarial attacks[END_REF] formulates the robustness as the sum of the accuracies on a bunch of adversarial samples. This robust evaluation makes it easier for the evolutionary algorithm to select better architecture in the population and apply different mutations (e.g., add a block, remove a block, add a connection...). Along with [START_REF] Guo | When NAS meets robustness: In search of robust architectures against adversarial attacks[END_REF], they reveal these observations: first, the more dense the architecture is the more robust it is, second, under computational budget, adding convolution operations to direct connection edge is effective, and finally, flow of solution procedure (FSP) matrix is a good indicator of network robustness. Please note here, that none of the NAS methods that consider security and reliability in the search place are hardware-aware.

X. INDUSTRIAL ADOPTION OF NAS

One salient aspect that is not discussed in most AutoML surveys is its applications in the industrial domain. In this section, we compare different tools that apply NAS principles in order to let users build a specific model for their datasets. 1) Auto-Keras is a framework that is based on the deep learning framework keras. By far, auto-keras is the most used autoML tool with features allowing to explore a wide range of neural network types (CNN, RNN and MLP). The search runs in parallel on CPUs and GPUs, with an adaptive search strategy for different GPU memory limits. Auto-keras builds its search space by applying a set of network morphism operations which keep the functionality of a neural network while changing its neural architecture. 2) Microsoft NNI (Neural Network Intelligence) is a general AutoML toolkit to help users tune their machine learning models (e.g., hyperparameters), design neural network architecture. One property it focuses on is the efficiency of their automation. For example, it leverages early feedback to speedup the tuning procedure. The proposed NAS framework helps the user define their own super network. For example, one can specify multiple operators for one single layer including depthwise convolution, dilated convolution, maxpooling, and the NNI will automatically find the best candidate based on a gradient-based optimization. On the other hand, the toolkit is also customizable for NAS researchers and enable them to modify the search algorithm and/or objective function. 3) Google AutoML The idea with Google AutoML is to create a meta-model capable of learning a way to design, generate and propose architectures for a given dataset. The model will eventually achieve an accuracy on the data set that will be used as a reward to guide the reinforcement learning agent to explore and find the best architecture from the search space. Google AutoML generalize the use of machine learning to everyone by offering an easy-to-use toolkit to upload the data set and let the system find the appropriate model. The idea behind AutoNAC is to construct small and fast models such that each model specializes in a small portion of the dataset, the basic strategy uses divide-and-conquer paradigm, where a complex problem is decomposed into a set of simpler sub-problems. For example, rather than building one complex model that classifies the data into the three classes defined by the data set, three simpler models that separate the classes two-by-two are built. 8) Darwin is a cloud-based service that utilizes genetic programming to iteratively select the best architectural hyperparameters and pass them to the next generation. Given a target dataset, Darwin tries to find a previously trained dataset that shares the same characteristics and initiate the search with the best model for that dataset. The framework also includes automation for other machine learning components such as the feature extraction by automatically detecting the data types and data preprocessing.

Each of these tools have their own advantages and disadvantages as well as different spaces to compete with. Microsoft NNI, Archiai are designed to help NAS researchers compare, reproduce and experiment with multiple NAS search strategies and speedup methods. AutoKeras and TPOT are designed for the general machine learning researcher or engineer. Although they restrict their search space to a small set of architectural hyperparameters, the performance optimizations of their search are really attractive as an exchange for accuracy in this settings. In addition, both are open source which allow the user to run locally their search for privacy reasons. On the other hand, IBM Neunets, Google AutoML and deci.ai AutoNAC try to demcratize make machine learning and make it available and accessible to everyone regardless of their skill level and without writing any piece of code. It is a more ambitious goal which is the reason why they are using more limited architecture search spaces. However, their search algorithms consume a lot of computing power and time that's why they are usually offered within a cloud service. The majority of the surveyed industrial NAS frameworks are not hardwareaware except for AutoNAC which given a defined model and its accuracy tries to find a faster architecture that doesn't reduce the specified precision. Nonetheless, Microsoft NNI and Google AutoML apply some compression techniques to the found architecture to specialize it in the target hardware.

XI. CHALLENGES AND LIMITATIONS

In this section, we lay down the main challenges and barriers that prevent us to unfold hardware-aware NAS's full potential. Most of the challenges are also applicable to general NAS methods.

A. Benchmarking and Reproducibility

An important challenge while working on NAS and HW-NAS is the reproducibility of the methods, which allows comparisons and concrete improvements. Unfortunately, due to the use of different search spaces, various training methods, and the required significant computational resources, reproducibility is a difficult step. This difficulty is even higher when it comes to HW-NAS considering the numerous possibilities of targeted hardware devices. To address this issue, many works [START_REF] Ying | Nas-bench-101: Towards reproducible neural architecture search[END_REF], [START_REF] Dong | Nas-bench-201: Extending the scope of reproducible neural architecture search[END_REF], [START_REF] Zela | Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search[END_REF], [START_REF] Klyuchnikov | Nas-bench-nlp: Neural architecture search benchmark for natural language processing[END_REF] have proposed different benchmarks and data sets that allow NAS researchers to:

• Eliminate the cost of generating a search space by querying directly a tabular data set. • Evaluate different search strategies on the same search space which allows a fair comparison between them. • Provide data sets to be used by accuracy predictor models and hardware cost models. • Open HW-NAS research to non-hardware experts by proposing datasets that contain the hardware-related metrics. These metrics are usually obtained after optimization of the operators and software that require specific hardware knowledge.

In this section, we will review each NAS benchmark and highlight their strengths and weaknesses.

NAS-Bench-101 [START_REF] Ying | Nas-bench-101: Towards reproducible neural architecture search[END_REF] 1 is a tabular dataset that maps 432k unique architectures to their relative training accuracy, validation accuracy, testing accuracy as well as training time and the number of trained parameters. Each architecture is trained for various numbers of epochs 4, 12, 36, 108. The architectures follow a fixed macro architecture, with searchable cells stacked (See section V-A). Each cell is constructed with up to 7 layers that can include 3 types of operations: 3x3 conv, 1x1 conv, and 3x3 max-pooling. NAS-Bench-101 allows flexibility by allowing different layers to be used in the stacked cells. The search space is constrained by limiting the number of edges to 9 and the number of nodes to 7 including the input and output nodes for each cell. However, not including operations such as separable convolution or dilated convolutions makes the resulting models parameter-heavy, which is not hardware-friendly. Another downside of this benchmark is the inability to use one-shot optimizer NAS, this is due to the tabular format. To enable the evaluation of weight-sharing NAS methods, two benchmarks have been released NAS-Bench-201 [START_REF] Dong | Nas-bench-201: Extending the scope of reproducible neural architecture search[END_REF] containing 6240, 29160, and 363648 architectures. In order to use the expensive experimentation done by NAS-Bench-101, the authors created a one-shot architecture that contains all the discrete cell architectures defined by NAS-Bench-101. This allows the search algorithm to only train the supernetwork and get the performance of each path from the NAS-Bench-101. Therefore, their benchmark construction does not need any additional cost.

Alternatively, NAS-Bench-NLP defines a tabular benchmark for NLP tasks. Their cell-based search space is constructed based on an LSTM Macro-architecture borrowed from AWD-LSTM [START_REF] Merity | Regularizing and optimizing LSTM language models[END_REF]. Each cell can contain up to 24 nodes with 3 hidden states and 3 linear input vectors. With these constraints and a set of operators composed by the most used activations in recurrent cells, they are able to build LSTM, GRU cells and many more. The architectures are trained on Penn Tree Bank (PTB) dataset [START_REF] Marcus | Building a large annotated corpus of english: The penn treebank[END_REF] and a sub-sample of the best performing ones are also trained on WikiText-2 [START_REF] Merity | Pointer sentinel mixture models[END_REF], which is a more realistic dataset for real-world NLP problems.

A major disadvantage of these benchmarks is the size and diversity of their search spaces. Indeed, as presented by the experimentation results we obtained in figure 16, on the small dataset NAS-Bench-201, a local search, which is the simplest optimization strategy, achieves state of the art results without a significant search time compared to other search strategies, except NAS without training [START_REF] Mellor | Neural architecture search without training[END_REF]. In this experiment, we compare the different results obtained by different NAS approaches on the NAS-Bench-201. Most of them use the metrics provided by the benchmark along with fine-tuning the architecture to obtain a more accurate validation metric, except NAS without training [START_REF] Mellor | Neural architecture search without training[END_REF]. The NAS searche without any further training achieves decent results within 17sec of search. By dividing the benchmarks into N mini-batches, they increase their training efficacy. The higher this number is (N ), the higher the over-fitting probability on the benchmark. Therefore, using small datasets with complex search algorithms does not yield any good results in terms of accuracy of the final architecture or efficiency of the search. NAS-Bench-301 is a much bigger benchmark that was designed to overcome the over-fitting problem on the architectures. It is based on DARTS [150] search space. Since DARTS search space is far too large to be exhaustively evaluated by real-world measurements, the authors built a surrogate model capable of predicting the various performance metrics. This model is trained on a subsample of the benchmark, 60k architectures whose performance have been measured on CPUs.

A more recent paper introduced the first hardware-aware NAS benchmark, HW-NAS-Bench [START_REF] Li | {HW}-{nas}-bench: Hardwareaware neural architecture search benchmark[END_REF]. This work extends the number of hardware metrics and records the latency, the energy consumed on 6 hardware devices including commercial edge devices, FPGA, and ASIC . Its search space is a combination of FBNet [START_REF] Wu | Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search[END_REF] search space and NAS-Bench-201.

Although these datasets provide a good start to test different search strategy, they lack a lot of important operators that can significantly change the resulting architecture for hardwareaware NAS. In addition, we note a growing number of different applications of NAS in various tasks such as image restoration [START_REF] Van Wyk | Evolutionary neural architecture search for image restoration[END_REF], [START_REF] Zhang | IR-NAS: neural architecture search for image restoration[END_REF], [START_REF] Gong | Autogan: Neural architecture search for generative adversarial networks[END_REF], [START_REF] Ho | Neural architecture search for deep image prior[END_REF], semantic segmentation [START_REF] Liu | Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation[END_REF], [START_REF] Shaw | Squeezenas: Fast neural architecture search for faster semantic segmentation[END_REF], and medical segmentation [START_REF] Weng | Nas-unet: Neural architecture search for medical image segmentation[END_REF], [START_REF] Zhu | V-nas: Neural architecture search for volumetric medical image segmentation[END_REF]. Therefore, there is a growing need for proper benchmarks for each of the diverse tasks.

B. Transferability of the AI Models

A concept that is quite popular in deep learning is transferability. We train the model on a proxy dataset; that is usually smaller and simpler, then we fine-tune the trained weights on the targeted dataset to get a more efficient model. For example, we can train our image classifier on CIFAR-10 [START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF], which contains 10 classes of 32x32 pixels images, and then fine-tune our model on ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], which is made of 14 million images and 20,000 classes. Another benefit of transferability is when the target dataset is not large enough to train a model on.

To enhance transferability, previous NAS works used cellbased search spaces. To transfer a model obtained from a cellbased search spaces, we just need to adjust the input sizes of the cells and deepen the network by adding more cells. However, stacking the same blocks seems to be not efficient when incorporating hardware constraints. As MNASNet [START_REF] Tan | Mnasnet: Platform-aware neural architecture search for mobile[END_REF] argued, restricting the cell diversity is critical for achieving high accuracy and low latency on mobile settings. Therefore, MNASNet uses a hierarchical search space that diversify the cells in the architecture as well as the operators within that cell.

Many NAS works [START_REF] Liu | Progressive neural architecture search[END_REF], [START_REF] Nayman | XNAS: neural architecture search with expert advice[END_REF] have included dedicated evaluations of the transferability of their final model. PNAS [START_REF] Liu | Progressive neural architecture search[END_REF] proved that CIFAR-10 classification is highly correlated to ImageNet classification, hence transferring a cell or an architecture from CIFAR-10 to ImageNet would produce good results. XNAS [START_REF] Nayman | XNAS: neural architecture search with expert advice[END_REF] transferred their final cell structure on 6 popular classification benchmarks surpassing other conventional NAS methods while taking account of the hardware constraints.

Another interesting concept is presented by NAT [START_REF] Lu | Neural architecture transfer[END_REF]. They leverage NAS process to directly find transferable weights (i.e., get rid of the fine-tuning stage). The key idea behind NAT is that they start from a supernetwork and adaptively modify it to obtain a task-specific super network. This latter approach can then be used directly to search for architectures within one task without the additional training cost. Under mobile scenarios, they demonstrated the efficacy of NAT on 11 benchmarks including ImageNet.

Overall, taking the transferability of the model into account remains a difficult challenge. Most solutions modify the search space to fit a certain task by leveraging a task-specific macroarchitecture. Hardware-awareness adds another level to this challenge as the architecture needs to be flexible to adapt to multiple platforms. Note that, transfer learning generalizes the ability to use a model from one task to another. In this section, we only talked about NAS that transfers their models from one dataset to another within the same task, which is a sub definition of transfer learning. Unfortunately, as NAS tasks evolve around image classification mainly, the whole search needs to be executed all over again if we want an architecture for another task.

C. Transferability of the Hardware-aware NAS across multiple Platforms

HW-NAS suffers from the conditional optimality due to the variety of existing devices. Ideally, we should design different architectures for different platforms. However in real situations, given the prohibitive cost of the search and the cost of training on multiple architectures, we often resort to designing one architecture and deploying it anywhere. Transferring a model from one platform to another or being able to produce hardware transferable models via the NAS process is an interesting challenge for HW-NAS. The main difficulty lies in the variety and complexity of the existing platforms. For example, different platforms might perform well on different types of convolutions. In the following section, we discuss two approaches. The first transfers a single-target HW-NAS to another target by modifying the measurement values. The second takes the final architecture of the NAS process and transforms it to fit another platform. Each approach has its pros and cons as discussed below.

a) Transfer the entire NAS process: this means that the whole NAS process needs to be re-executed to suit the new targeted hardware. In a single target HW-NAS, it is quite a challenge to transfer the entire process to another platform. Without mentioning the huge computational cost of retraining the entire process, the collection of the hardware constraints can be costly as well. When using real-world measurements, [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF] ran the NAS search for three hardware platforms: CPU (Xeon E5-2640 v4), GPU (Tesla V100) and mobile phone (Google Pixel-1). However, using real-world measurements considerably slows down the search algorithm and requires the availability of the targeted hardware during the search time. On the other hand, using an analytical estimation requires expert knowledge for the different targeted platforms. When using other collection methods such as the lookup table or the prediction model, we'll need to collect data from the new platforms by running again the entire set of operators. To this end, HW-NAS tries to create a general measurement method. For example, Once-for-all [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF] created a lookup table with the reported inference latency on each tested hardware platform (i.e., Samsung S7 Edge, Note8, Note10, Google Pixel1, Pixel2,LG G8, NVIDIA 1080Ti, V100 GPUs, Jetson TX2, Intel Xeon CPU, Xilinx ZU9EG, and ZU3EGFPGAs). According to the used measurement method, transferring the NAS process to target another platform is increasingly difficult and not scalable.

b) Transfer the final model: An alternative approach is to find the best model for one hardware platform and then try to specialize it for another one. This was done by [START_REF] Cai | Once-for-all: Train one network and specialize it for efficient deployment[END_REF], [START_REF] Cai | Proxylessnas: Direct neural architecture search on target task and hardware[END_REF], [START_REF] Nayman | XNAS: neural architecture search with expert advice[END_REF]. Usually, the specialization is done by compressing the model using quantization which enables the model to fit in tiny devices. However, the specialization is challenging because of the following reasons:

• An operator may be efficient for one platform and less efficient in another: In [START_REF] Chu | Discovering multihardware mobile models via architecture search[END_REF], the authors argued that separable convolutions give great results when ran on GPUs but perform badly on CPUs. Additionally, it is common that deeper networks perform well on CPUs while wider ones perform well on GPUs because of the possible parallelization. Table VIII demonstrates the comparison between the execution times of different operators on an Intel i7 CPU and NVIDIA TX2 GPU . The results reinforce our assumption that different operators's efficiency vary from one platform to another. Therefore, the best model is highly correlated to the platform we choose which makes design a HW-NAS that targets multiple platforms very challenging. • Limits of the compression methods: We consider here the quantization and pruning. For these two methods, we know that theoretically the compression ratio have a threshold that cannot be surpassed. For example, quantizing a model implies encoding its activations and weights into the minimum possible bit length. Theoretically this length is 2 (one bit). Even without considering the tradeoff between accuracy and model size, these methods have limits. This is why we need to start from a model that is already optimized to be able to deploy the model on tiny devices. For this end, [START_REF] Han | Design Automation for Efficient Deep Learning Computing[END_REF] is composed of three components. The first components is a multiobjective NAS that searches for the best model in terms of accuracy and latency. Given the resultant model of the first component, the second component searches for pruning possibilities that would preserve the accuracy and decrease the model size. Finally, the last component takes care of quantizing the model with a mixed precision. [START_REF] Nascimento | Finding non-uniform quantization schemes using multi-task gaussian processes[END_REF] starts from a standard architectures such as VGG, ResNet and GoogleNet and casts the quantization as a neural architecture search problem. This work achieves a minimal loss of accuracy with appreciable memory savings. In addition to the limit for the model size, we also have a limit for the accuracy. The compressed model typically does not have a better accuracy than the pretrained model we started from.

D. Outlook and Future Directions

In addition to the benchmarking, reproducibility and transferability challenges presented in the previous three subsections, HW-NAS suffers from the same limitations as NAS. The main one is the cost of the search algorithm. The current popular way to speed up the process is to use differentiable [START_REF] Maas | Learning word vectors for sentiment analysis[END_REF]. Results were obtained using PyTorch with number of samples 1000.

neural architecture search and create a supernetwork representing the whole search space. This supernetwork is trained once to get the weights of all the sub-networks, thus avoiding training each sampled architecture. This technique reduces the search time from several days to hours. However, a major disadvantage of this network is its restriction on the targeted task and domain. More research is needed to explore efficient ways and tricks to make HW-NAS more practical and more amenable to a diverse set of tasks and domains. This would especially be useful in commercial settings. Given the increasingly fragmented hardware landscape, HW-NAS should investigate the possibilities to reduce the conditional optimality problem. Most existing NAS approaches design once and deploy to all. We have seen this clearly through the industrial adoption of NAS techniques which lacks hardware-awareness. Ideally, we should be able to design different architectures for different devices as no-one-model fits all. This problem is significantly exacerbated by the increasing hardware heterogeneity in AI-powered devices.

There is no doubt that the future of mobile and handheld devices is AI. AI-focused mobile chips from top manufacturing companies like Apple, Samsung, Huwaei and others make their ways into the mainstream. These devices use SoCs that take advantage of multiple platforms (e.g., GPUs, CPUs and NPU in the same chip). However, this heterogeneity of platforms needs to be well understood to speed up the inference time of deep neural networks [START_REF] Wang | Neural network inference on mobile socs[END_REF]. This motivates further the research community to simultaneously explore both the architecture search space and the hardware design space to identify the best neural architecture and hardware pairs that maximize both test accuracy and hardware efficiency. Such co-exploration will be critical to allow designing architectures that can be deployed efficiently on a variety of platforms: data center, edge, mobile, and embedded.

HW-NAS also suffers from a lack of benchmarks and public datasets that can help accelerate HW-NAS research and make it reproducible and accessible. HW-NAS-Bench [START_REF] Li | {HW}-{nas}-bench: Hardwareaware neural architecture search benchmark[END_REF] is a recent first attempt towards this direction. It tries to offer computed metrics on multiple commercial platforms including edge GPUs, FPGA and ASICs and make these measurements available to the research community. One of their main goals is to democratize HW-NAS research to non-hardware experts.

More work is also needed to further co-explore neural architectures with quantization, pruning, and hardware design. Most of the existing efforts, such as mixed-precisions have been studied in the context of fixed architectures.

HW-NAS should also look at exploring neural architecture search with emerging computing paradigms such as in-memory-computing [START_REF] Jiang | Device-circuit-architecture co-exploration for computing-in-memory neural accelerators[END_REF]. These new non-von-Neumann paradigms present novel solutions to AI computing based on emerging nano-devices called Phase-Change Memory (PCM) [START_REF] Sebastian | Computational memorybased inference and training of deep neural networks[END_REF]. Existing HW-NAS solutions are dedicated to the conventional von-Neumann computing architecture, where the memory wall heavily limits performance. The optimization space spans multiple design points that range from device types, to circuit topologies, to device non-idealities and variations, to neural architectures.

While deep neural networks have clear commercial use cases, the next AI breakthrough may require an entirely different combination of algorithm, hardware and software. HW-NAS offers a paradigm that opens up the design space and pushes forward the Pareto frontier between hardware efficiency and model accuracy for efficient and improved hardware/software co-design, hence pushing AI to its next frontiers.

XII. CONCLUSION

We have provided a comprehensive survey and systematic analysis of state-of-the-art hardware-aware neural architectural search. We reviewed several multi-objective strategies that aim to find the optimal architecture with the highest accuracy while reducing memory and computations costs. We provide a HW-NAS taxonomy and categorize existing approaches along four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategy. We also discussed other considerations in Hardwareaware NAS that include optimizations such as quantization and pruning. Finally, we presented a discussion on future directions that would benefit existing and future HW-NAS researchers. We believe that HW-NAS is an important direction that would enable hardware-software co-design and make deep learning solutions not only accessible to everyone but also sustainable.

Fig. 1 .

 1 Fig.1. Generic CNN architecture. For each layer an operator is chosen among a pre-defined list (convolution, dilated convolution, depthwise convolution, maxpooling, batch normalization...)

Fig. 2 .

 2 Fig.2. Accuracy of various CNN models on ImageNet for Image Classification task with the number of parameters. Inspired by[START_REF] Deng | Model compression and hardware acceleration for neural networks: A comprehensive survey[END_REF]

1 2 3 Fig. 3 .

 33 Fig. 3. Overview of conventional NAS components

Fig. 4 .

 4 Fig. 4. Number of papers describing HW-NAS by Dec 2020. The top 5 conferences and journals are: NeurIPS, ECCV, IEEE Transactions on Pattern Analysis and Machine Intelligence, IJCNN, and MICCAI.

Fig. 5 .

 5 Fig. 5. Illustration of sparsification. (a) Weight Pruning. (b) Neuron Pruning.Gray lines correspond to pruned vertices (i.e. weights) and white nodes correspond to pruned neurons. Source[START_REF] Deng | Model compression and hardware acceleration for neural networks: A comprehensive survey[END_REF]

Fig. 8 .

 8 Fig. 8. Overview of the evolutionary algorithm steps.

Fig. 9 .

 9 Fig. 9. Overview of different hardware-aware NAS designs.

Fig. 10 .

 10 Fig. 10. Architecture search spaces types. (a) Global search space, (b) Cell-based search space, and (c) Hierarchical search space. In orange the operators considered during the search.

Fig. 11 .

 11 Fig. 11. Statistics on targeted platforms

Fig. 13 .

 13 Fig. 13. HW-NAS problem formulations.

Algorithm 1 :

 1 A generalized pipeline of HW-NAS Input: The Search Space S, a Dataset D, Accuracy Evaluation Methods E acc (S, w(S), D), Hardware Evaluation Methods E hw (S, w(s), D), an Aggregation function f Result: Optimal Deep Learning Architecture repeat Sample an architecture s from the search space S; Train s on D to obtain the trained weights w(s); Compute acc = E acc (s, w(S), D); Compute hw cost = E hw (s, w(s), D); if f (acc, hw c ost) surpass the best seen value then

  2 and NAS-Bench-1Shot1[START_REF] Zela | Nas-bench-1shot1: Benchmarking and dissecting one-shot neural architecture search[END_REF] 3 .NAS-Bench-201 represents 15,625 architectures using a fixed cell-based macro architecture. Similar to NAS-Bench-101, it uses a predefined set of operations including conv 3x3, conv 1x1, Avg pooling, skip connection and no operation label. Each architecture is trained on three different datasets with different complexities namely, CIFAR-10, CIFAR-100 and imagenet-16. The authors extended this benchmark and presented a year later NATS-Bench. NATS-Bench [185] increased the search space size by varying the sizes of their architectures.Similarly, NAS-Bench-1Shot1 presented a new reformulation to reuse the even much more extensive computation of the NAS-Bench-101 dataset (120 TPU years) to create three new one-shot search spaces with growing complexity

Fig. 16 .

 16 Fig. 16. Results of different search algorithms on NAS-Bench-201.

COMPARISON

  BETWEEN DIFFERENT OPERATORS ON INTEL I7 CPU AND NVIDIA TX2 GPU. THE CONVOLUTION OPERATORS WERE USED TO CREATE A CNN MODEL THAT WAS TRAINED FOR IMAGE CLASSIFICATION ON IMAGENET. THE RNN CELLS WERE TRAINED ON TEXT CLASSIFICATION ON IMDB DATASET

	Operator	Avg Latency on	Avg Latency on	Accuracy
		CPU (ms)	GPU (ms)
	Conv2d	1.33	0.85	0.67
	Separable Depth-	2.05	0.54	0.64
	wise Convolution
	Dilated convolu-	1.36	0.835	0.56
	tion
	Grouped Convo-	2.27	1.94	0.62
	lution
	LSTM cell	14.93	2.45	0.57
	GRU cell	9.32	2.53	0.65
		TABLE VIII

Open sourced at https://github.com/google-research/nasbench

Open sourced at https://github.com/D-X-Y/AutoDL-Projects

Open sourced at https://github.com/automl/nasbench-1shot1