LCPC instrumentation division 1 has developed a generic wireless platform that can be considered as the a result of redundant needs in wireless monitoring especially applied to civil engineering monitoring applications. This platform includes software and hardware bricks and aims at being generic by its native implementation of sober components, the worldwide TCP/IP protocole (802.11g), a signal processor, a small GPS receiver, and a micro embedded operating system (uClinux).

Since 2009, this platform -named PEGASE -is subject of an industrial transfer that has generated some tens of indivual sales. A set of pluggable daugthers boards (that integrate the application specific sensing operation) offers a ready-to-use panel of wireless sensing solutions for developping specific applications as well as they can be seen as prototyping boards for further electronic developments.

As PEGASE platform reached a mature level of dissemination, LCPC recent efforts are now leaded with the goal of improving its wireless capacities. Those works concern energy saving while keeping a high level of embedded processing, of sampling rate or time-synchronization.

After a quick summary of PEGASE properties, this article aims at presenting the last algorithms and evolutions that have been developped and added to the system. As software layers are mainly written in standard C language under Linux OS, those pragmatic solutions could easily be re-used by even radically different systems. The focus will specifically be pointed on: an algorithm that allows PEGASE wireless boards to be synchronized up to some uS using a GPS technique while keeping the GPS receiver OFF most of the time; a description of how the use of an operating system such as uClinux allows a full and remotely uptade of wireless sensors; the hardware and software strategies that have been developped to make PEGASE fully autonomous using solar cells.

INTRODUCTION

Considering the exponential technological developments especially in electronic domain, a question could be: how to pose the concept of a generic wireless platform in a world in constant evolution ? Moreover, electronic components lifespan appears as an epsilon part of the expected structures lifespan (bridges, railway, industrial machines, etc.). So how to get pragmatic solutions that could reach the challenges that SHM ask ?

The ideas and the concepts defended in this article try to answer those questions by separating the solution in two part. One part of the answer resides in the most intelligent choice possible of an electronic solution. This means sober and resisting components, a relevant processor, etc. But no eternal electronic devices such as processors, memories, etc. have been designed for good yet. Consequently, thinking new hardware solutions while keeping an active technological survey in electronic and WSN domains will remain necessary for years. But a part of the answer is to extract, as much as possible, the generic concepts that can be reused independently from hardware solutions.

In a previous article [START_REF] Le | Design of a generic smart and wireless sensors network -benefits of emerging technologies[END_REF] -as other laboratories or companies did [START_REF] Culler; Estrin | Overview of wireless sensor networks[END_REF][START_REF] Lewis | Wireless sensor networks[END_REF][START_REF] Marcy | Wireless sensor networks for area monitoring and integrated vehicle health management applications[END_REF]-LCPC strongly supported the idea of generic wireless sensor platform using the best of the available technology. Considering the heavy cost of any electronic development (especially in the wireless world), developping generic and easily reusable hardware platforms is considered as relevant. From the first successes that LCPC wireless platform (PEGASE) met, the idea has been to factorize and to capitalize improvement strategies that can be applied to PEGASE and that could also be applied to other wireless platforms.

LCPC WIRELESS PLATFORM PROPERTIES

First of all, as many other entities deciced [START_REF] Vecchio | Wireless sensor developments for structural health monitoring[END_REF][START_REF] Swartz | Dsitributed data processing architectures for structural health monitoring systems implemented on wireless sensor networks[END_REF], LCPC wireless platform have been made modular to fit the need of beeing reusable without any expensive and long developments. This modularity resides in a sensor made of a "mother board" that integrates redundant needs (computation, communication, data storage) and a set of pluggable "daughter board" that integrates application specific needs such as the sensitive element (accelerometer, temperature, pressure,. . . ), its specific conditioning design including hardware filters, analog to digital converters. As justified in [START_REF] Le | Design of a generic smart and wireless sensors network -benefits of emerging technologies[END_REF], the main characteristics of PEGASE feature are the following:

• Use of TCP/IP/WiFi as the wireless protocol [START_REF] Dunkels | Making tcp/ip viable for wireless sensor networks[END_REF][START_REF] Scottsdale | Wifi moves into the sensor networking[END_REF]: reliable, low-cost, scalable (IP is the worldwide protocol). Turned OFF when PEGASE doesn't communicate.

• Use of the Analog Device low-power Blackfin BF537 as core processor (Digital Signal Processor): 16 bits processor able of complex operations.

• Implementation of a small and low-power GPS receiver to ensure localization and, first of all, absolute time synchronization up to few µS GMT.

• uClinux as the embedded operating system: allows high level of abstraction while PEGASE algorithms are then programmed using standard ANSI C language.

According to application needs, PEGASE power consumption could vary between 400 mW (full mode ON) to some 80 mW (in sleep mode). As described below, even if this energy consumption is not the lowest in WSN world, this level allows viable configurations based on batteries coupled to solar cells.

PEGASE APPLICATIONS

Since its first version on january 2008, PEGASE has been used in various configurations where its properties fitted specific needs. To illustrate that dissemination, three applications are described below.

Bridge inspection by UAV In order to simplify structures inspections and to make them less expensive, one of LCPC projects refers to the instrumentation of an Unmanned Aerial Vehicle. A military U.A.V. has been bought and its instrumentation by PEGASE platform has the goal to interface many devices that could help human subjectivity. Those devices are: GPS receivers, a camera controled by a 3-axial robot, a laser-meter. Cable health monitoring As already described [START_REF] Le | Design of a generic smart and wireless sensors network -benefits of emerging technologies[END_REF], cable health monitoring is a LCPC major preoccupation. One of the favorite method used to detect and to localize wirebreaks in cables (due to corrosion or mechanical fatigue) is based on the acoustic emission of those breaks and the elastic wave thus generated. In that case, PEGASE is customized in a sensor whose mission consists in permanently listening using an accelerometer. For each detected wave (comparison to a threshold), an asbolute time-stamping is made and information is wirelessly transmitted to a remote supervisor. Knowing wireless sensors location, waves spectrum and dating, supervisor is then able to localize the wire-break and to warn an operator via a GSM modem. Continous meteorologic monitoring PEGASE has been used and customized in order to implement specific needs relative to meteorologic measurements close to structures such as bridges or dams. In that application, the goal is the validation of models measuring the influence of wind gradient on noise propagation. Real-time monitoring processes associated to accurate time stamping and DSP capacities of PEGASE are involved. As figure 4 shows, a specific daugther board emulating 5 RS232 ports have been designed in order to be connected to 5 anemometers. The final product takes place into a small box.

PLATFORM IMPROVEMENTS

Using GPS for time-synchronisation and quartz drift correction Wireless Sensor Network time synchronization is a real challenge [START_REF] Le | Synchronization of wireless sensors : review of methodologies. experience feedback of the very precise gps solution[END_REF]. Because of quartzes drift (natural error, aging, t 0 c), very few determinist methods propose an accuracy better than mS and, if any, they often suppose numerous and energy-expensive communications with a supervisor sending beacons. A solution implemented in PEGASE concept, uses the propensity of any GPS receiver to deliver a digital signal named PPS (Pulse Per Second). Each PPS from each GPS receiver is synchronised with an accuracy of 1 S +/-20 nS (i.e. 20*10 -9 S). The PPS output of the GPS receiver is routed to the RESET entry of DSP time-counter. In parallel, the standard RS232 output of the GPS receiver that contains absolute GPS informations (localization, date, hour, minutes, seconds) is routed to a RS232 input port of the DSP. A real-time algorithm continuously decode those informations. Then, on PPS rising edge, a Linux real-time interruption resets microseconds counter and updates latest GPS information decoded (refreshed until next second). The result is that, between two PPS edges, any function, any process, any event such as a digital to analog conversion can be time-stamped by a C code instruction that gets the actual value of the time-counter. Testing campaigns and measurements gave, for a same event, a time stamping difference between two PEGASE wireless sensors doesnt exceed 15 µS, while keeping a GMT referential thanks to GPS standard.

Increasing synchronization accuracy by correcting quartz drift. Once previous PPS synchronization method was perfectly under LCPC control and knowledge, a second level of synchronization has been implemented in PEGASE algorithms. Considering that PPS accuracy (about 20 nS) is much more sufficient than whised accuracy in typical SHM applications (where synchronization needs vary from mS to some µS), time-counter values (cf. figure 5) have been analysed on PPS rising edge. As far as its value on PPS edge is supposed to be equal to 1000000 µS a linear difference can be observed from the begining of a second (theorical time-counter = 0µS) until the end of that second (theorical time-counter = 10 6 µS). This linear difference nammed δ is due to PEGASE quartzes drift (natural factory error) and, consequently, very constant. Then a specific algorithm has been developped and added to the previous one to complete PEGASE synchronization principle: that algorithm is called by the uClinux low-level interrupt mechanism. On each PPS interrupt, δ is evaluated as the difference between the theorical value of 10 6 µS and time-counter real value (influenced by quartz error). This value is added to a 16 places table that is averaged, this gives the ∆16 average drift of the quartz. On each PPS event (i.e. each second):

δ = 1000000 -time_counter ∆16 = ∆16+δ 16
Then, any event that occurs in PEGASE system is time-stamped reading the timecounter and applying a linear correction that takes ∆16 drift into account: Testing campaigns measurements gave, for a same event, including the previous time correction an absolute time stamping differencebetween two PEGASE wireless sensors doesnt exceed 4 µS, while keeping a GMT referential thanks to GPS standard.

Event_uS = time_counter * 10 6

Turning GPS OFF for power saving while keeping synchronised

The GPS based synchronization principle could be subject of a criticism: even a low-power GPS receiver needs some 25 mW. The present improvement proposes to periodically turn OFF the GPS receiver in order to cancel its power consumption. The postulate resides in the fact that, after short periods (around 50 seconds), the ∆16 average drift of the quartz is well known by the system, and, aging and temperature influences on quartz are null during short periods.

The idea is, once ∆16 is evaluated, a new time-counter is immediately start; its period is fixed to { 10 6 -∆16} µS. This period is the real value of what represent the delay of 1s view by PEGASE board using a specific quartz at a specific moment. Then the GPS receiver could be stopped and time-stamping operations are then made by the system using this new timer that acts as a relay while GPS receiver is OFF. This algorithm has also been implemented and works. First results give a linear response in the possibility to stop the GPS receiver while keeping a quite good accuracy in sensor synchronization (the final drift is due to the uncertainty of the OS and the residual errors in ∆16 evaluation). While keeping the GPS ON during 60 S and then OFF during 600 S, the sensor has drifted of 270 µS when GPS has been rearmed. Other developments in that way are under works.

A collection of reusable software bricks

As mentionned, uClinux implementation as the embedded operating system in PEGASE allows high level of abstraction while PEGASE algorithms are then programmed using standard ANSI C language. No specific knowledge about the BF537 processor registries is necessary since all its peripherals can be driven through uClinux libraries. More than the simple use of standard C language, LCPC engineers have developped a collection of classes (i.e. software bricks) written in C-object language. This library named UFC (for Uclinux Foundation Classes) offers a usefull dozen of directly reusable classes (.c & .h files) to easily manage the main PEGASE peripherals: CGPS, CAbsoltuTime, CSPI (to interface the SPI port and then Analog-to-Digital converters), CTcpIPClient, CTcpIPServer, CWiFiMngt, CRS232port, etc.

Those generic classes associated to the PEGASE hardware natural genericity accelerate the customization of PEGASE to develop a quick answer to a specific application. Moreover, the C knowledge is quite universally diffused and makes the concept easily adopted by designers.

Using solar cells for energy harvesting

According to the application needs, a strategy must be defined in terms of energy. For short term applications (from few hours to few weeks) effectives solutions could be found by means of batteries (Lead or NiMh technologies). In those cases, for uncontinuous monitoring (low sampling and processing), hibernate modes can be applied. PEGASE platform can reduce its power consumption to 80 mW. On the other hand, for long term monitoring (months, years) or applications that require continous processing or numerous wireless communications, solutions using solar cells could be defined For PEGASE wireless platform, a specific smart solar cell solution has been defined. Many solutions that use solar cell as an auxiliar source: while solar cell voltage is less or equal than battery voltage, wireless system uses energy from battery, when battery voltage is exceeded, solar cell energy is used. The optimization consists in the integration of a boost circuit. If the boost circuit is correctly designed, each voltage value that output from the solar cell is widely amplified. All the energy that the solar cell is able to deliver is amplified, the battery threshold is exceeded without time loss waiting for greater voltage. Such a boost circuit is integrated to PEGASE concept. Figure 7: A boost circuit added to the PEGASE concept for an optimal solar cells source This hardware optimization is driven by a software that implements a Maximum Power Point Tracking algorithm (MPPT [START_REF] Brad | Single sensor mppt algorithm for multiple solar panels configurations[END_REF]). Continuously and automaticaly, PEGASE finds and maintain the optimum level of energy that can be extracted from the solar cell source. First results gave an acceptable size of 30cm * 30cm solar cell (cf figure 7) for an application of continuous monitoring that requires 250 mW.

CONCLUSION AND ROADMAP

Since a third-party partner (A3IP company) has been licencied by LCPC, PEGASE has been sold in hundreds of specimens and implemented in various configurations. This dissemination prooved the capacity of wireless systems to really answer a large spectrum of applications. Developments in progress have the goal to increase this panoply. Even if uClinux and WiFi integration could be considered as heavy, the result is a great ability for developpers or customers to acheive their own applications. The genericity of C language and the worlwide IP protocol make them ubiquitous.

A quite expert job has been leaded to develop specific embedded drivers under uClinux OS in order to get specific behaviors for time synchronization, quartz drift auto-training and correction. This specific and dynamic correction takes temperature effects into account and the result is an absolute time synchronisation better that 4 µS.

Even if technologies evolve (components, processor, batteries...), generic principle could be extracted independently from technological choices. Those main principles are: daughter/mother boards, Linux integration, a ready to use c-object library, a boost circuit linked to a MPPT algorithm, GPS synchronization and quartz correction. Most of the improvements can be reused and applied to other wireless platforms even using drastically different electronic implementations.

Figure 1 :

 1 Figure 1: PEGASE mother board, a daughter board, a specific solution

Figure 2 :

 2 Figure 2: PEGASE platform for U.A.V. instrumentation

Figure 3 :

 3 Figure 3: PEGASE platform customized for acoustic monitoring of cables

Figure 4 :

 4 Figure 4: PEGASE involved in meteorological measures

Figure 5 :

 5 Figure 5: GPS synchronization principle, test-bench and results.

Figure 6 :

 6 Figure 6: GPS second test-bench and improved results.


Van Tien Pham

Yassine Zniyed

Thanh Phuong Nguyen

Enhanced network compression through tensor decompositions and pruning

Keywords: tensor decompositions, structured pruning, hybrid compression, efficient inference

Network compression techniques that combine tensor decompositions and pruning have shown promise in leveraging the advantages of both strategies. In this work, we propose NORTON (enhanced Network cOmpRession through TensOr decompositions and pruNing), a novel method for network compression. NORTON introduces the concept of filter decomposition, enabling a more detailed decomposition of the network while preserving the weight's multidimensional properties. Our method incorporates a novel structured pruning approach, effectively integrating the decomposed model. Through extensive experiments on various architectures, benchmark datasets, and representative vision tasks, we demonstrate the usefulness of our method. NORTON achieves superior results compared to stateof-the-art techniques in terms of complexity and accuracy. Our code is also available for research purposes.

I. INTRODUCTION

N ETWORK compression aims to reduce the computa- tional and memory requirements of an existing model, enabling its deployment in resource-constrained environments without compromising its ability to generalize, i.e., perform well on unseen data.

Among the existing paradigms for model compression, tensor decompositions [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF] and structured pruning [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF] have demonstrated their effectiveness and practicality. Both approaches are built upon the hypothesis that the original model is over-parameterized, and thus, the redundant information can be eliminated either by representing the weights more efficiently using low-rank representations or by directly removing a part of them through filter pruning. In addition to their capabilities of high compression rate, these methods share the advantage of enabling the deployment of compressed models on resource-constrained devices without requiring any specialized support. However, it is noteworthy that these two approaches have mostly been developed independently in the literature, with few efforts [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], [START_REF] Dai | Compressing deep model with pruning and tucker decomposition for smart embedded systems[END_REF] to explore their combined potential and leverage their individual strengths in an orthogonal manner. This highlights the need for an integrated approach that harnesses the benefits of both tensor decompositions and structured pruning.

The motivation for combining tensor decompositions and structured pruning arises from the observation that the weights All authors are with Université de Toulon, Aix Marseille Université, CNRS, LIS, UMR 7020, France (e-mails: van-tien-pham@etud.univ-tln.fr; zniyed@univ-tln.fr; tpnguyen@univ-tln.fr).

This work was granted access to the HPC resources of IDRIS under the allocation 2023-103147 made by GENCI. The work of T.P. Nguyen is partially supported by ANR ASTRID ROV-Chasseur. of CNNs possess both low-rank and sparse properties, as demonstrated in prior works [START_REF] Li | Towards compact cnns via collaborative compression[END_REF]- [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF]. These attributes are considered partially complementary [START_REF] Yu | On compressing deep models by low rank and sparse decomposition[END_REF]. Decomposition approaches are not designed to eliminate all redundant channels [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], while pruning approaches typically focus on removing redundant elements but may overlook the low-rank structure within the retained weights post-pruning. The natural inclination, therefore, is to integrate these two compression strategies to push the limits of network compression [START_REF] Li | Towards compact cnns via collaborative compression[END_REF]. For instance, considering the VGG-16 architecture, RGP [START_REF] Chen | Rgp: Neural network pruning through regular graph with edges swapping[END_REF], a state-of-theart (SOTA) filter pruning method, achieved a maximal pruning of 90.5% of MACs, while HALOC [START_REF] Xiao | Haloc: Hardware-aware automatic low-rank compression for compact neural networks[END_REF], representing SOTA in tensor decompositions, compressed a maximum of 86% of MACs. In other words, no reported work has effectively compressed more than 91% of MACs for this architecture. In contrast, by combining the two approaches, NORTON achieves a simultaneous compression of 99% of MACs and parameters with a modest drop, as depicted in Fig. 2. Specifically, applying only the decomposition part in NORTON can compress up to 88% of MACs (with the minimum rank 1, see Tab. VI). Conversely, applying only the pruning part, with a pruning ratio of around 90%, results in a significant accuracy drop, and the pruned model fails to recover the accuracy. However, by combining these two approaches, with rank 1 and a pruning ratio of 80%, NORTON achieves a 99% compression rate, evidently pushing the boundaries of network compression. Our work acknowledges a similar approach observed in image compression, akin to the widely used JPEG algorithm [START_REF] Wallace | The jpeg still picture compression standard[END_REF], where a combination of techniques enhances overall compression efficiency. Importantly, NORTON is not merely a mechanical combination of these two approaches; it also capitalizes on their intrinsic advantages. It is noteworthy that in our approach, both stages share a common representation-the factor matrices-which are concurrently employed for both the decomposition and pruning phases.

Considering tensor decompositions, the typical pattern involves selecting a decomposition method and determining the rank for the weight tensor of the original layer. The decomposition is then applied to obtain a low-rank representation, which is used to construct the weights of the new layer. Previous works in the literature have primarily focused on the decomposition method [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF] and rank selection [START_REF] Hawkins | Towards compact neural networks via end-to-end training: A bayesian tensor approach with automatic rank determination[END_REF]- [START_REF] Eo | An effective low-rank compression with a joint rank selection followed by a compression-friendly training[END_REF]. However, an unexplored aspect in the existing literature revolves around the decomposition of the weight tensor itself. Prior studies have predominantly focused on decomposing the entire layer as a 4-order tensor or reshaping it into a 3-order tensor before performing the decomposition. However, there is limited discussion on determining the most effective approach for decomposing the weight tensor. This aspect warrants further investigation to enhance and optimize network compression techniques.

The weight of a convolution layer is a 4-order tensor, which can be processed in different formats, each yielding distinct consequences. In Fig. 1, we illustrate three possible methods for handling this weight tensor, including layer decomposition [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF] and reshaped-based decomposition [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Eo | An effective low-rank compression with a joint rank selection followed by a compression-friendly training[END_REF]- [START_REF] Liebenwein | Compressing neural networks: Towards determining the optimal layer-wise decomposition[END_REF], while the proposed approach is referred to as filters decomposition. As a representative example, we consider the Canonical Polyadic decomposition (CPD) [START_REF] Kolda | Tensor decompositions and applications[END_REF] to compare the differences among these approaches. The approach presented by [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF] involves decomposing the entire layer, resulting in 4 factor matrices that correspond to 4 sublayers. Although this method preserves the multidimensional nature of the weight tensor, it processes the tensor at a coarse level. This opens up the potential for a more fine-grained treatment. In contrast, in [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF] the authors suggested reshaping the 4-order weight tensor into a 3-order tensor, followed by CPD to obtain 3 factor matrices and 3 sublayers. The authors argued that the kernel size is relatively small so they can be ignored. However, this resizing process compromises the multidimensionality of the weight tensor, resulting in information loss during approximation. Moreover, this argument does not seem reasonable with recent modern architectures where the kernel size increases significantly [START_REF] Liu | More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity[END_REF].

Our proposed method operates on the original weight tensor in a filter-by-filter manner. The key insight lies in the fact that in a convolution layer, the input undergoes convolution separately with each filter, and the outputs are then aggregated to generate the final feature map. Therefore, it is intuitive to decompose each 3-order filter tensor individually. The filters decomposition approach offers a higher level of granularity compared to its counterpart, layer decomposition. With filters decomposition, not only the multidimensional property is strictly preserved, but also the layer's 4-order weight is spontaneously interpreted as a set of 3-order filters. Additionally, an interesting side effect of filters decomposition is that it leads to a narrower range of ranks, which simplifies the rank selection process, as demonstrated later

MACs reduction (%)

Top-1 accuracy (%) NORTON (2023) ours ASTER (2023) [START_REF] Zhang | Adaptive filter pruning via sensitivity feedback[END_REF] APIB (2023) [START_REF] Guo | Automatic network pruning via hilbert-schmidt independence criterion lasso under information bottleneck principle[END_REF] AutoBot (2023) [START_REF] Castells | Automatic neural network pruning that efficiently preserves the model accuracy[END_REF] HALOC (2023) [START_REF] Xiao | Haloc: Hardware-aware automatic low-rank compression for compact neural networks[END_REF] WhiteBox (2022) [START_REF] Zhang | Carrying out cnn channel pruning in a white box[END_REF] DECORE (2022) [START_REF] Alwani | Decore: Deep compression with reinforcement learning[END_REF] EZCrop (2022) [START_REF] Lin | Ezcrop: Energy-zoned channels for robust output pruning[END_REF] CHIP (2021) [START_REF] Sui | Chip: Channel independence-based pruning for compact neural networks[END_REF] GFBS (2021) [START_REF] Liu | Exploring gradient flow based saliency for dnn model compression[END_REF] HRank (2020) [START_REF] Lin | Hrank: Filter pruning using high-rank feature map[END_REF] Fig. 2: The accuracy-MACs reduction Pareto curves of compressed VGG-16 models are compared on CIFAR-10.

in subsection III-B. Another notable difference is that filters decomposition replaces the original layer with 3 sublayers, while layer decomposition requires 4 sublayers, potentially increasing network depth unnecessarily and introducing the possibility of gradient vanishing issues. It is worth mentioning that the distinction between layer decomposition and filters decomposition aligns with a similar categorization seen in network pruning, specifically filter pruning [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF] and weight pruning [START_REF] Han | Learning both weights and connections for efficient neural networks[END_REF], emphasizing the fine-grained granularity aspect.

The second consideration of this work is to combine filters decomposition and filter pruning to leverage the independent advantages of each method. While previous studies [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], [START_REF] Li | Group sparsity: The hinge between filter pruning and decomposition for network compression[END_REF], [START_REF] Yeom | Toward compact deep neural networks via energy-aware pruning[END_REF] have used low-rank representations and network pruning for compression purposes, they have not explored an orthogonal combination of these techniques. In these works, low-rank representations have been used only in the pruning step without directly contributing to the reduction of model size. In contrast, our approach takes a different direction by sequentially applying low-rank representations and pruning in two separate phases, enabling a dual compression process. To the best of our knowledge, this direction has not been extensively investigated in the literature.

There are two possible arrangements for combination: decomposing then pruning, and pruning then decomposing. Previous works [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], [START_REF] Dai | Compressing deep model with pruning and tucker decomposition for smart embedded systems[END_REF] have adopted the pruning then decomposing scheme, utilizing a Taylor expansion-based pruning criterion [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF] and Tucker decomposition (TD) [START_REF] Kolda | Tensor decompositions and applications[END_REF]. However, this work [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF] lacks comprehensive analysis and experiments, leaving room for further investigation. To address this gap, our study explores the decomposing then pruning scheme, where the model is first decomposed using CPD and then subjected to filter pruning, as illustrated in Fig. 3. We argue that this order presents greater challenges compared to its counterpart, as the model's architecture becomes more complex after decomposition, with certain constraints imposed on the sublayers. To adapt to the decomposed components, we propose to use the Principal Angles Between Subspaces (PABS) [START_REF] Björck | Numerical methods for computing angles between linear subspaces[END_REF] as a suitable filter pruning metric which will be further discussed in subsection III-C.

Briefly, the contributions of this work are three-fold:

• Firstly, we introduce a novel filters decomposition method, promoting its differentiation with existing layer decomposition and reshaped decomposition methods. • Secondly, we investigate the sequential combination of filters decomposition and filter pruning. We propose a novel filter pruning algorithm designed to address the challenges associated with this integration scheme. • Thirdly, we evaluate the proposed framework on representative vision tasks including image classification, object detection, instance segmentation, and keypoint detection. We compare NORTON with SOTA methods in both low-rank representations and structured pruning domains to demonstrate the superiority of our method. Notably, our model with different compression levels can consistently outperform prior arts (see Fig. 2).

II. RELATED WORKS

A. Low-rank Representations

Existing works can be categorized into two branches: matrix decompositions [START_REF] Idelbayev | Low-rank compression of neural nets: Learning the rank of each layer[END_REF], [START_REF] Liebenwein | Compressing neural networks: Towards determining the optimal layer-wise decomposition[END_REF], [START_REF] Denton | Exploiting linear structure within convolutional networks for efficient evaluation[END_REF]- [START_REF] Yang | Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification[END_REF] and tensor decompositions [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF], [START_REF] Zhang | Accelerating very deep convolutional networks for classification and detection[END_REF], [START_REF] Astrid | Rank selection of cp-decomposed convolutional layers with variational bayesian matrix factorization[END_REF]. Matrix-based approaches consider the weight of the fully connected layer [START_REF] Denton | Exploiting linear structure within convolutional networks for efficient evaluation[END_REF] or the convolution layer [START_REF] Idelbayev | Low-rank compression of neural nets: Learning the rank of each layer[END_REF], [START_REF] Liebenwein | Compressing neural networks: Towards determining the optimal layer-wise decomposition[END_REF], [START_REF] Tai | Convolutional neural networks with low-rank regularization[END_REF]- [START_REF] Yang | Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification[END_REF] as a 2-order matrix on which the SVD is performed to obtain two smaller matrices. A required step of this work line is that the 4-order weight tensor is flattened into a 2-order matrix [START_REF] Liebenwein | Compressing neural networks: Towards determining the optimal layer-wise decomposition[END_REF]. On the other hand, tensor decompositions approaches [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF], [START_REF] Zhang | Accelerating very deep convolutional networks for classification and detection[END_REF], [START_REF] Astrid | Rank selection of cp-decomposed convolutional layers with variational bayesian matrix factorization[END_REF] execute directly with higher-order weight tensor of the convolution layer.

Based on the adopted decomposition, various factorizations have been examined, such as SVD [START_REF] Eo | An effective low-rank compression with a joint rank selection followed by a compression-friendly training[END_REF], [START_REF] Liebenwein | Compressing neural networks: Towards determining the optimal layer-wise decomposition[END_REF], [START_REF] Denton | Exploiting linear structure within convolutional networks for efficient evaluation[END_REF], CPD [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Wang | Kronecker cp decomposition with fast multiplication for compressing rnns[END_REF], TD [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], hierarchical TD (HTD) [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF], tensor train (TT) [START_REF] Garipov | Ultimate tensorization: compressing convolutional and fc layers alike[END_REF], tensor ring (TR) [START_REF] Wang | Wide compression: Tensor ring nets[END_REF] or a mixture of them [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Wu | Hybrid tensor decomposition in neural network compression[END_REF]. The pioneering work [START_REF] Denton | Exploiting linear structure within convolutional networks for efficient evaluation[END_REF] employed the truncated SVD to approximate matrix weights of fully connected layers and the monochromatic approximation to compress the two first convolution layers. In the first work of layer decomposition style [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], CPD was used to exploit the redundancy of the 4order weight tensor of convolution layers. However, this work was restricted to compressing a single layer. The authors of [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF] applied TD along with the variational Bayesian matrix factorization [START_REF] Nakajima | Global analytic solution of fully-observed variational bayesian matrix factorization[END_REF] for rank selection to make a one-shot whole network compression. Similarly, the hierarchical TD was used in [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF] with an energy-based algorithm for rank selection. Further, [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF] combines CPD and TD to deal with the problem of degeneracy (i.e., the difficulty of fine-tuning the factors). TT and TR were used in [START_REF] Garipov | Ultimate tensorization: compressing convolutional and fc layers alike[END_REF], [START_REF] Wang | Wide compression: Tensor ring nets[END_REF] to obtain the sparse representation of the redundant weights. In their empirical investigation of dimension balance preferences, the authors of [START_REF] Wu | Hybrid tensor decomposition in neural network compression[END_REF] proposed a hybrid consideration of TT for convolution layers and HTD for fully connected layers.

B. Filter Pruning

The criterion to measure the importance of filters is the foundation of filter pruning. Various approaches have been proposed to tackle this problem and they can be roughly categorized as follows: magnitude-based approaches [START_REF] Li | Pruning filters for efficient convnets[END_REF]- [START_REF] Chen | Whc: Weighted hybrid criterion for filter pruning on convolutional neural networks[END_REF], correlation-based approaches [START_REF] Lin | Pruning networks with cross-layer ranking & k-reciprocal nearest filters[END_REF], activation-based approaches [START_REF] Guo | Automatic network pruning via hilbert-schmidt independence criterion lasso under information bottleneck principle[END_REF], [START_REF] Sui | Chip: Channel independence-based pruning for compact neural networks[END_REF], [START_REF] Lin | Hrank: Filter pruning using high-rank feature map[END_REF], regularization [START_REF] Zhang | Adaptive filter pruning via sensitivity feedback[END_REF], [START_REF] Ding | Resrep: Lossless cnn pruning via decoupling remembering and forgetting[END_REF]- [START_REF] Tang | Scop: Scientific control for reliable neural network pruning[END_REF], optimization [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF], [START_REF] Hu | Catro: Channel pruning via class-aware trace ratio optimization[END_REF]- [START_REF] Gao | Efficient neural network compression inspired by compressive sensing[END_REF], and neural architecture search [START_REF] Chen | Rgp: Neural network pruning through regular graph with edges swapping[END_REF], [START_REF] Alwani | Decore: Deep compression with reinforcement learning[END_REF], [START_REF] Lin | Channel pruning via automatic structure search[END_REF]. In [START_REF] Li | Pruning filters for efficient convnets[END_REF], the L1 norm was used as a measure of saliency, assuming that filters with smaller magnitudes are less important. The authors of [START_REF] Sui | Chip: Channel independence-based pruning for compact neural networks[END_REF], [START_REF] Lin | Hrank: Filter pruning using high-rank feature map[END_REF] argued that the activation maps represent both the structure and data so the pruning criterion should be based on the feature map other than the filters. Regularization was used for learning structured sparse networks by applying different sparsity regularizers to different parameters [START_REF] Ding | Resrep: Lossless cnn pruning via decoupling remembering and forgetting[END_REF]- [START_REF] Tang | Scop: Scientific control for reliable neural network pruning[END_REF], [START_REF] Lin | Towards optimal structured cnn pruning via generative adversarial learning[END_REF]. The first-order information of Taylor expansion is used to optimize the change in the loss of channel pruning in [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF]. From the perspective of architecture search, [START_REF] Alwani | Decore: Deep compression with reinforcement learning[END_REF] used reinforcement learning to find optimal compact structures. However, some drawbacks of these methods are the complex design, large search space, and high computational resource requirement which may be impractical for users.

C. Hybrid Approach

The notion of combining low-rank representations and network pruning has been suggested but lacks in-depth exploration in the existing literature [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF]- [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], [START_REF] Li | Group sparsity: The hinge between filter pruning and decomposition for network compression[END_REF], [START_REF] Eo | A differentiable framework for end-to-end learning of hybrid structured compression[END_REF]. In a previous work [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], a compressed-aware block was incorporated to acquire a low-rank weight basis and sparse channels; however, this method is data-dependent, requiring an additional training stage. Another approach [START_REF] Li | Group sparsity: The hinge between filter pruning and decomposition for network compression[END_REF] employed group sparsity constraints on sparsity-inducing matrices after reshaping the weight tensor to connect filter pruning and decomposition. A collaborative compression scheme was introduced in [START_REF] Li | Towards compact cnns via collaborative compression[END_REF] to simultaneously learn model sparsity and low-rankness. In contrast, NORTON takes a distinctive approach by addressing tensor decompositions and filter pruning in two separate phases. This ensures that the advantages of both lowrank structure and filter removal contribute synergistically to the final compression by addressing sparse and redundant structures. The most closely related work to ours is [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], which employed Taylor pruning [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF] and TD [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF]. However, NORTON introduces the novel concept of filter decomposition and incorporates it with an adaptive filter pruning technique in the inverse order, a more challenging but promising approach. In contrast to [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], we provide comprehensive experiments and analysis to shed light on this hybrid approach. III. NORTON APPROACH Fig. 3 illustrates the overall pipeline of our approach, which comprises two main phases: decomposition and pruning. Firstly, the original model is decomposed into CPDBlocks (as explained in Subsection III-B). Next, the decomposed CPDBlocks undergo the filter pruning algorithm (as described in III-C). This algorithm selectively removes filters from the CPDBlocks based on certain criteria, effectively reducing the model's computational and memory requirements. Finally, a fine-tuning process is conducted to refine the compact model. Empirically, only one process of fine-tuning as presented in our proposal is sufficient instead of considering it after every phase of decomposition and pruning. This choice simplifies the methodology, reduces computational requirements, and maintains satisfactory performance. This scheme is applied simultaneously to all convolution layers of the original model. However, for the sake of simplicity, subsequent sections will focus on discussing one layer, as shown in Fig. 4.

A. Preliminaries

This paper uses lower-case letters (e.g., a) to denote scalars, bold lowercase letters (e.g., a) to represent vectors, bold uppercase letters (e.g., A) to represent matrices, and bold calligraphic letters (e.g., A) to denote tensors. A tensor is a generalization of a matrix to a multi-way data array. A d-order tensor is a multi-way data array A ∈ R I1×I2ו••×I d , where I n is the size of mode n. ∥.∥ F stands for the Frobenius norm. We now introduce some definitions that will be useful in the sequel.

Definition 1: A rank-1 d-order tensor A ∈ R I1ו••×I d can be expressed as a sequence of single outer product • of d vectors as:

A = u (1) • • • • • u (d) , (1)
where

u (n) ∈ R In for n = 1, . . . , d. Each entry of A is given by a i1...i d = d n=1 u (n) in , where u (n) in is the i n -th entry of the I n -length vector u (n) .
Definition 2: The CP decomposition [START_REF] Kolda | Tensor decompositions and applications[END_REF] expresses a tensor A ∈ R I1ו••×I d as the sum of multiple rank-1 tensors:

A = R j=1 u (1) j • u (2) j • • • • • u (d) j .
(

) 2
The minimal integer R that ensures equality is called the canonical rank of A. An equivalent and compact representation of ( 2) is A = [[U (1) ,

• • • , U (d) ]], where U (n) :,j = u (n) j
and is of size Definition 3: For two linear subspaces U , V ⊂ C I , the smallest principal angle between them denoted by θ is defined as [START_REF] Björck | Numerical methods for computing angles between linear subspaces[END_REF]:

I n × R.
cos(θ) = max u∈U ,v∈V u H v ∥u∥ 2 ∥v∥ 2 . (3)

B. Filters Decomposition Using the CP Decomposition

Consider a convolutional layer with O filters of size K h × K w × I, where I represents the number of input channels, and K h and K w represent the height and width of the kernel, respectively. The weight tensor W can be represented as a 4-order tensor of size K h × K w × I × O. Alternatively, it can be viewed as a set of O individual 3-order filters denoted as W 1 , W 2 , . . . , W O . It is worth noting that the k-th filter can be obtained by extracting the sub-tensor W :,:,:,k , which is equivalent to W k . These weights map an input tensor I of size H in ×W in ×I into an output tensor O of size H out ×W out ×O, where H in , W in , H out , and W out are the height and width of the input and output tensors, respectively. For simplicity, the dimension of the batch is disregarded.

In convolutional neural networks, the mapping of the input tensor I to the output tensor O is achieved through the convolution operation. This convolution is given by the following expression: 4) and the CPD definition in (2), we can apply the CPD to each individual filter W k in order to obtain a compact representation. By decomposing W k using CPD, we have [START_REF] Dai | Compressing deep model with pruning and tucker decomposition for smart embedded systems[END_REF] where A k , B k and C k are 3 factor matrices of size K h × R, K w × R and I × R, respectively. The complexity of this decomposition algorithm, using the alternating least squares algorithm, is O R 2 • I • max(K h , K w ) [START_REF] Kolda | Tensor decompositions and applications[END_REF]. This approximation is graphically represented in the left half of Fig. 4.

O k (i, j) = K h -1 m=0 Kw-1 n=0 I-1 p=0 I(i + m, j + n, p) • W k (m, n, p), (4) where, for 0 ≤ k ≤ O -1, O k = O :,:,k ,

and is of size

H out × W out . Based on (
W k (m, n, p) = R-1 r=0 A k (m, r) • B k (n, r) • C k (p, r),
By substituting ( 5) into (4), we obtain a new CPD-based approach to compute the convolution. This approach involves a sequence of mappings using the factor matrices instead of high-order tensors. The resulting equation for this CPD-based convolution is:

O k (i, j) = K h -1 m=0 Kw-1 n=0 I-1 p=0 R-1 r=0 I(i + m, j + n, p)• A k (m, r) • B k (n, r) • C k (p, r). (6)
Starting from [START_REF] Li | Towards compact cnns via collaborative compression[END_REF], we observe that the CPD-based convolution involves element-wise multiplications between the input tensor I and the factor matrices A k , B k , and C k . It is important to note that the order of the convolutions can be rearranged without affecting the final result. This flexibility allows us to describe the computation as a sequential block of convolutions with smaller kernels, followed by a summation. First, the input tensor I can be convolved with the kernel C k along the input channel dimension. The resulting tensor is then convolved with the kernel B k along the spatial dimensions. Finally, the output of the second convolution is convolved with the kernel A k along the spatial dimensions. The outputs of these convolutions are summed up to compute the final output tensor O k . The following set of equations captures these operations in a concise manner:

O C k (i + m, j + n, r) = I-1 p=0 I(i + m, j + n, p) • C k (p, r), (7)
where

O C k is of size H in × W in × R. O B k (i + m, j, r) = Kw-1 n=0 O C k (i + m, j + n, r) • B k (n, r), (8)
where

O B k is of size H in × W out × R. O A k (i, j, r) = K h -1 m=0 O B k (i + m, j, r) • A k (m, r), (9)
where

O A k is of size H out × W out × R. Finally, we have O k (i, j) = R-1 r=0 O A k (i, j, r). (10)
One should note that equations ( 7), [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], and ( 9) can be seen as convolutions and can be easily implemented using common deep learning frameworks. Specifically, equation ( 7) can be computed using a classical 2D convolution operation, while equations ( 8) and ( 9) can be computed via group convolutions. One can refer to Fig. 5, which illustrates the structure of the CPDBlock. To ensure compatibility with classical frameworks, certain basic preprocessing operations including reshaping and mode permutations are required for adapting the kernel. Specifically, for the O factors C k of dimensions I ×R, they need to be reshaped into a kernel of size 1×1×I ×(R•O). This enables the application of a classical 2D convolution for equation [START_REF] Yu | On compressing deep models by low rank and sparse decomposition[END_REF]. Additionally, for the group convolutions in equations ( 8) and ( 9), the kernels should be remodeled as About rank and the compression. The choice of rank R in CPD plays a crucial role in balancing model compression and accuracy. Kruskal's theory provides a weak upper bound on the maximum rank [START_REF] Kolda | Tensor decompositions and applications[END_REF], expressed as:

1×K w ×1×(R•O) and K h ×1×1×(R•O),
R ≤ min I • K h , I • K w , K h • K w . (11)
This upper bound offers a guideline for selecting an appropriate rank, ensuring a reasonable trade-off between model compression and preservation of critical features. By fixing a rank R, the CPDBlock achieves a significant reduction in the number of parameters compared to the original layer, which consists of

O • I • K h • K w parameters.
Through the CPD-based approach, NORTON achieves a compact representation of the filters, resulting in a reduced parameter count on the order of

O • R • (I + K h + K w ). The computational complexity of the original layer is O (I • O • K h • K w • H • W ). In contrast, the three sub-layers exhibit complexities of O(I • O • R • H • W ), O (O • R • K w • H • W ), and O (O • R • K h • H • W ), respec- tively.
As a result, the CPDBlock significantly reduces the computational load to

O (R • O • (I + K h + K w ) • H • W ).
It's important to note that the rank constraint introduced in (11) guarantees a favorable reduction in both the number of parameters and computational complexity.

C. CPDBlock Pruning

After demonstrating the use of filter decomposition and its integration into the new architecture, we will now delve into the methodology of factor matrices pruning. This technique focuses on reducing the number of parameters by pruning the factor matrices obtained through the decomposition process.

First, it is important to note that each output O k is computed based on three factor matrices: A k , B k , and C k . Therefore, when pruning the kernel related to a specific output, all three matrices must be considered for removal. This necessitates the use of a pruning criterion that takes into account the interdependencies among the three matrices. Second, it is worth mentioning that when the CP decomposition is unique, it is unique up to scaling and permutation ambiguities. In other words, if two 3-order filters, W i and W j , are strictly similar and satisfy the uniqueness conditions of the CP decomposition [START_REF] Kolda | Tensor decompositions and applications[END_REF], then

   W i = [[A i , B i , C i ]], W j = [[A j , B j , C j ]], W i = W j . ⇏    A i = A j , B i = B j , C i = C j . (12)
Instead, we have

A i = A j ΠΛ A , B i = B j ΠΛ B , and
C i = C j ΠΛ C ,
where Π is a permutation matrix, and the diagonal scaling matrices verify Λ A Λ B Λ C = I. For all the aforementioned reasons, we have decided to choose the PABS [START_REF] Björck | Numerical methods for computing angles between linear subspaces[END_REF] as a metric to measure the distance between two CP decompositions. The use of the PABS distance measure is justified in both the unique and non-unique cases of CP decomposition. In the unique case, PABS allows for capturing distances between factor matrices, enabling the identification of redundant filters based on their distance patterns, while handling scaling and permutation ambiguities. Let ϕ(., .) be a function that computes the PABS between two factor matrices, as defined in (3). If we reconsider the example in [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF] in the case of unique CPDs, we have

   W i = [[A i , B i , C i ]], W j = [[A j , B j , C j ]], W i = W j . ⇒    ϕ(A i , A j ) = 0, ϕ(B i , B j ) = 0, ϕ(C i , C j ) = 0. (13)
Even in the non-unique cases, PABS can still be effective in identifying redundancies, as it captures the distance between different sets of factor matrices representing the same tensor as will be confirmed in the simulations. This enables the pruning process to remove filters that contribute minimally to model performance or exhibit high similarity to other filters, resulting in a more compact model while preserving critical features and maintaining performance. The core idea of CPDBlock pruning is to construct a distance matrix D, where each element D ij corresponds to the distance between the factor matrices

[[A i , B i , C i ]] and [[A j , B j , Cj]].
The pruning process involves iteratively identifying the pair of decompositions, i and j, corresponding to the minimum value of D ij , and removing one of them. The removed decomposition in our strategy between i and j is the one that most closely resembles the rest of the decompositions, ensuring that the pruned one retains the representation that is most similar to the remaining ones. The distance matrix D ∈ R O×O can be expressed as

D ij = αD A ij + βD B ij + γD C ij , (14)
where D A ij = ϕ(A i , A j ) (similarly for D B ij and D C ij ), and α, β and γ are weight parameters, whose sum is equal to 1. The pruning strategy is outlined in Alg. 1, which is complemented by a visual representation in the right half of Delete the row and column of the deleted decomposition from D. 10: end for

IV. EXPERIMENTS

A. Experimental Settings

Architectures and Datasets. In order to demonstrate the adaptability of NORTON, we assess three representative architectures: VGG-16-BN [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], ResNet-56/110 with residual blocks [START_REF] He | Deep residual learning for image recognition[END_REF], and DenseNet-40 with dense blocks [START_REF] Huang | Densely connected convolutional networks[END_REF]. These models are tested on the CIFAR-10 dataset [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF]. Additionally, to validate the scalability of NORTON, experiments are conducted on the challenging ImageNet dataset [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] using the ResNet-50 architecture. Furthermore, the compressed ResNet-50 model is employed as the backbone network for FasterRCNN-FPN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], MaskRCNN, and KeypointRCNN [START_REF] He | Mask r-cnn[END_REF] on the COCO-2017 dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF].

Baselines. NORTON is compared with 52 SOTAs in the fields of low-rank decompositions (D) [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Xiao | Haloc: Hardware-aware automatic low-rank compression for compact neural networks[END_REF], [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], [START_REF] Gabor | Compressing convolutional neural networks with hierarchical tucker-2 decomposition[END_REF], [START_REF] Eo | An effective low-rank compression with a joint rank selection followed by a compression-friendly training[END_REF], [START_REF] Liebenwein | Compressing neural networks: Towards determining the optimal layer-wise decomposition[END_REF], [START_REF] Xu | Trp: Trained rank pruning for efficient deep neural networks[END_REF], [START_REF] Yang | Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification[END_REF], structured pruning (P) [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF], [START_REF] Chen | Rgp: Neural network pruning through regular graph with edges swapping[END_REF], [START_REF] Yeom | Toward compact deep neural networks via energy-aware pruning[END_REF]- [START_REF] Lin | Hrank: Filter pruning using high-rank feature map[END_REF], [START_REF] Li | Pruning filters for efficient convnets[END_REF]- [START_REF] Lin | Towards optimal structured cnn pruning via generative adversarial learning[END_REF], [START_REF] Qian | Hierarchical threshold pruning based on uniform response criterion[END_REF]- [START_REF] Luo | Neural network pruning with residual-connections and limited-data[END_REF] and hybrid methods (H) [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF]- [START_REF] Li | Towards compact cnns via collaborative compression[END_REF], [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], [START_REF] Li | Group sparsity: The hinge between filter pruning and decomposition for network compression[END_REF], [START_REF] Eo | A differentiable framework for end-to-end learning of hybrid structured compression[END_REF]. However, for certain methods where corresponding experiments were unavailable, we reproduced the approach of [START_REF] Lebedev | Speeding-up convolutional neural networks using fine-tuned cpdecomposition[END_REF], [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF], [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF] to ensure a more comprehensive comparison. To ensure fairness in the comparison, all original models are identical.

Evaluation Protocols. The model is assessed via three dimensions: accuracy, required Multiply Accumulate Operations (MACs), and the number of parameters (Params). The compression ratio (CR) is defined as the percentage reduction in MACs/Params when compared to the original model. Concerning the performance, top-1/top-5 accuracy is used on classification tasks while mean average precision (AP) and recall (AR) are used on detection/segmentation tasks.

Configurations. On CIFAR, fine-tuning was carried out for 400 epochs with a batch size of 256, momentum of 0.9, weight decay of 0.0005, and an initial learning rate of 0.01. On ImageNet, fine-tuning was conducted for 200 epochs with a batch size of 128, a momentum of 0.99, a weight decay of 0.0001, and a cosine learning rate scheduler with an initial learning rate of 0.1. On COCO, models are finetuned following the default recipe of torchvision [START_REF]Torchvision: Pytorch's computer vision library[END_REF]. In this work, we choose α = β = γ = 1 3 .

B. Results and Analysis

In the ensuing simulations, we adjusted the rank and pruning ratios to match other methods for a comparable compression rate. This enables us to examine the trade-off between CR and accuracy, as well as vice versa. For further transparency and accessibility, our accompanying code includes detailed information regarding the rank and pruning rates used in each case, ensuring the reproducibility and comprehensiveness of our results. We delve deeper into the impact of varying rank and pruning ratio selections in Section V.

VGG-16-BN. Tab. I shows compression results of VGG on CIFAR-10. In all compression levels, compared with other methods, NORTON consistently achieves the highest accuracy while reducing much more computation costs and enjoying a similar number of parameters. In particular, NORTON proves to be very robust with high compression rates. It can reduce 88% of FLOPs and 87% parameters with almost no loss, just 0.12%; or 96% MACs and 98% parameters with a slight loss of 1.42%. Finally, to evaluate the resistance of NORTON, we subject it to an ultra-high compression rate of 99% reduction in both MACs and parameters, and the proposed method remains firm, yielding a modest loss.

ResNet-56/110. Tab. II presents results on the ResNet-56/110 architectures. Our approach has the capacity to generalize the original model as it boosts 1.11% and 1.35% of the accuracy while eliminating 37% and 38% of parameters for ResNet-56 and ResNet-110, respectively. NORTON proves to be efficient with high-level compression (≈ 90%) where it outperforms [START_REF] Lin | Filter sketch for network pruning[END_REF] in all aspects. For ResNet-110, NORTON is better than DECORE [START_REF] Alwani | Decore: Deep compression with reinforcement learning[END_REF] in every way. DenseNet-40. Tab. III shows the compression results of DenseNet-40 on CIFAR-10. With similar performance, NORTON usually gains more compression than other methods. For example, with an accuracy of about 94.86%, our method reduces more than 7% MACs and 9% parameters in comparison with [START_REF] Alwani | Decore: Deep compression with reinforcement learning[END_REF].

ResNet-50. To evaluate the scalability of NORTON, we perform experiments on the extensive ImageNet dataset using the ResNet-50 architecture, as enumerated in Tab. IV. Across all evaluated scenarios, NORTON consistently outperforms other approaches in terms of both performance and complexity reduction. Our method can reduce 50% MACs while still enjoying an accuracy increment of 1.88% compared to Hinge [START_REF] Li | Group sparsity: The hinge between filter pruning and decomposition for network compression[END_REF], a hybrid method. In comparison to other hybrid methods [START_REF] Li | Towards compact cnns via collaborative compression[END_REF], [START_REF] Ruan | Edp: An efficient decomposition and pruning scheme for convolutional neural network compression[END_REF], NORTON yields higher accuracy while requiring lower computational cost and fewer parameters. Faster/Mask/Keypoint-RCNN. To assess NORTON's efficacy in downstream tasks, we employed our compressed ResNet-50/Imagenet as the backbone for training Faster/Mask/Keypoint-RCNN on COCO, as detailed in Tab. V. Our method enhances both precision and recall relative to the baseline, achieving a remarkable reduction of 17% MACs and at least 19% parameters. Notably, NORTON substantially elevates inference throughput, delivering over a 2× FPS im- provement compared to the baseline models. For instance, FasterRCNN sees the end-to-end latency drop from 85 ms to 41 ms, reaching a real-time framerate of 25 FPS. It's essential to highlight that these performance assessments were conducted on an RTX 3060 GPU, providing robust evidence of our approach's real-world utility in complex tasks of computer vision based on NORTON's network compression.

V. DISCUSSIONS A. The Role of the Rank

As the rank directly relates to the approximation error and the gain of compression rate, we conduct additional experiments on VGG16/CIFAR10 to investigate the effect of the rank selection. First, we only apply the CPD testing all possible ranks (see Tab. VI). To measure the approximation error between the original model and the decomposed model, we employ the normalized mean square error (NMSE) defined as:

NMSE = 1 N i=1 O i • N i=1 Oi j=1 W j i - Ŵj i 2 F W j i 2 F , (15)
where W j i is the j-th 3-order filter in the i-th layer composed of O i filters, Ŵj i is its approximation, and N is the number of layers. Tab. VI shows the complexity reduction, approximation error, and accuracy before and after fine-tuning. Obviously, without fine-tuning, the higher the rank is, the better the weights are approximated but with less compression obtained. With 5 ≤ R, our filter decomposition method produces comparable accuracies (maximum 1.46% drop) with the original model without fine-tuning. This suggests that our decomposition step behaves well without fine-tuning. In addition, we demonstrate that our decomposition step can effectively work when followed by a fine-tuning step. The results show that after fine-tuning, the accuracy is completely restored for all cases. However, it should be noted that the achieved compression ratios are not as favorable as those obtained with the proposed hybrid strategy.

Notably, our proposed method has not met the degeneracy problem (i.e., instability issue when training a CNN with decomposed layers in the CP format) as in a reshaped decomposition approach [START_REF] Phan | Stable low-rank tensor decomposition for compression of convolutional neural network[END_REF]. We suspect that the filter decomposition method is more fine-grained than the reshaped one. The trend of NMSE demonstrates a strong correlation between the approximation error and the accuracy (without fine-tuning). This observation aligns with recent work [START_REF] Schuurmans | How informative is the approximation error from tensor decomposition for neural network compression?[END_REF].

B. The Effect of Rank and Pruning Ratio Selection

To evaluate the effect of pruning ratios and rank on compression and accuracy, we conducted additional simulations depicted in Fig. 6. These simulations involve three compression levels, namely None (N), Soft (S), Moderate (M), and Hard (H) pruning ratios. Each level represents a different degree of pruning, allowing us to examine the impact of different compression settings on the decomposed models.

These experiments provide a comprehensive understanding of how the selection of rank and pruning ratio impacts complexity reduction and model performance. With a predefined complexity reduction goal, there are multiple choices for the rank and pruning ratio, resulting in different outcomes. For example, to eliminate about 65% of MACs, as shown in Fig. 6, possible choices are 3N, 4S, 6M, and 8H which produce 94.18%, 93.67%, 92.77%, and 93.15% accuracy after fine-tuning, respectively. Notably, these combinations, despite similar MACs reductions, lead to different levels of parameter reduction: 64.91%, 68.33%, 78.34%, and 87.49%, respectively. This observation raises questions about the optimal combination of rank and pruning ratio, warranting further investigation, particularly for hybrid compression approaches.

C. Component Analysis

Given that NORTON operates as a hybrid method, a detailed examination of the efficacy of its individual components becomes imperative. We conducted an experiment on the VGG-16-BN/CIFAR-10, focusing on the accuracy vs. MACs reduction Pareto curves of six approaches. These approaches were organized into three direct comparisons: our decomposition part versus Tucker decomposition [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], our pruning part versus Taylor pruning [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF], and our hybrid approach versus the combination of Taylor pruning and TD [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF]. It's worth noting that we selected these approaches based on their relevance to our work, especially [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF], as detailed in subsection II-C, and their components (TD and Taylor pruning) being representative in network compression. Due to the absence of results for this architecture in the referenced papers, we reproduced the results for Taylor pruning [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF], TD [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF], and their combination [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF]. The outcomes are presented in Fig. 7.

Effectiveness: In terms of pairwise comparisons, our components consistently outperform their counterparts from [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF]. For instance, our CPD consistently achieves higher performance than TD at similar compression ratios. Additionally, CPD demonstrates greater stability than TD, meaning its accuracy does not drop significantly as the CR increases (up to 88%, see Tab. VI). TD, a layer decomposition approach, experiences a rapid loss of performance beyond a 50% compression ratio, suggesting potential information loss due to its neglect of spatial dimensions of the kernel (3rd and 4th modes). This comparison underscores the effectiveness of our proposed filter decomposition method compared to layer decomposition and reshaped-based decomposition approaches. The subspace-based pruning method proves more effective and efficient than the Taylor pruning method. However, both pruning methods experience a significant performance drop at a 70% compression ratio threshold. Notably, the combination of Taylor pruning and TD exhibits suboptimal performance, even worse than its individual components at many points. In contrast, NORTON effectively synthesizes the strengths of its components, yielding higher accuracy at various points and achieving higher compression (beyond 90%). This experiment not only strengthens the motivation of our hybrid approach but also reaffirms that our decomposition/pruning proposal can stand alone as a promising candidate for network compression. TD [START_REF] Kim | Compression of deep convolutional neural networks for fast and low power mobile applications[END_REF] CPD (ours) Tayler pruning [START_REF] Molchanov | Pruning convolutional neural networks for resource efficient inference[END_REF] Subspace pruning (ours) Tayler pruning+TD [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF] CPD+Subspace pruning(ours) Fig. 7: The accuracy-MACs reduction Pareto curves of individual components in hybrid approaches.

Efficiency: Theoretical computational complexities for the proposed CPD and subspace pruning were detailed in Section III. Notably, the overheads associated with both the decomposition and pruning stages are deemed insignificant, each requiring less than 10 seconds in our simulation. This is notably less than the time spent on the fine-tuning step, which can extend up to 100 hours when using ResNet50/Imagenet. Additionally, our choice of a single-shot compression scheme ensures that the compression process occurs only once. In contrast, the hybrid method in [START_REF] Goyal | Compression of deep neural networks by combining pruning and low rank decomposition[END_REF] employs an iterative scheme, incurring higher costs as the decomposition and filters' saliency estimation need recalculation in each round. It's essential to emphasize that the critical metric lies in the inference speed during the deployment of the compressed model, aptly represented by the reduction in MACs, where our approach has proven to be particularly efficient.

This comprehensive evaluation shows that NORTON, as a hybrid approach, excels not only in synthesizing the strengths of its individual components but also in outperforming comparable methods in terms of accuracy and compression.

D. Visualizing Feature Preservation

We qualitatively assess feature preservation in NORTON, complementing the established efficiency from numerical results in Section IV. We randomly selected 5 images from the ImageNet validation dataset and evaluated three compression levels for the original ResNet-50 model: 50%, 64%, and 78% (see Table IV). Using GradCAM [START_REF] Selvaraju | Grad-cam: Visual explanations from deep networks via gradient-based localization[END_REF] for interpretation, we visualize and analyze feature maps in the original and compressed models in Fig. 8. It can be observed that at different CRs, NORTON consistently proves robust in capturing and preserving essential features across diverse classes. This robustness implies sustained effectiveness and reliability across varying scenarios and CR, establishing NORTON as a versatile choice for network compression.

VI. CONCLUSION

In this work, we have introduced NORTON, a novel hybrid network compression method that combines the strengths of tensor decompositions and structured pruning. Through extensive experiments and analyses, we have demonstrated the effectiveness and superiority of NORTON in achieving significant model compression while preserving critical features and maintaining performance. By leveraging filter decomposition and the integration of structured pruning, NORTON offers a powerful approach for reducing model complexity and the number of parameters, pushing the limits of network compression. For instance, our method achieved a simultaneous MACs/Params CR of 99% since current SOTAs only report to 91% to the best of our knowledge. Our results have shown that NORTON outperforms existing SOTA compression techniques in terms of compression ratios and accuracy retention. The combination of tensor decompositions and structured pruning in NORTON provides a versatile framework that allows for fine-grained control over the compression process. Moreover, NORTON's ability to handle scaling and permutation ambiguities further enhances its practicality and flexibility. The evaluation of NORTON on various architectures and datasets has demonstrated its scalability and generalizability. Furthermore, we have analyzed the impact of rank and pruning ratios on the compression and performance trade-offs, providing valuable insights for selecting optimal configurations.

Fig. 1 :

 1 Fig. 1: Comparison of tensor decompositions approaches.

Fig. 3 :

 3 Fig. 3: Graphic illustration of the NORTON approach.

Fig. 4 :

 4 Fig.4: The decomposition then pruning process for one layer.

  respectively. These modifications ensure the appropriate computation of O B and O A . By performing these preprocessing steps, the CPD-based convolutions can be seamlessly integrated into existing deep learning frameworks. Note that in Fig.5, the batch size B is depicted, and the O tensors O C k , O B k , and O A k are computed in a single operation, resulting in the corresponding output tensors O C , O B , and O A .

Fig. 5 :

 5 Fig. 5: Illustration of the CPDBlock structure for classical deep learning frameworks.

Fig. 4 .Algorithm 1

 41 Fig. 4. The complexity of the CPDBlock Pruning algorithm is O O2• (O -O ′ ) , resulting from pairwise comparisons in the calculation of the distance matrix and subsequent iterations for factor deletion. Algorithm 1 CPDBlock Pruning Require: The decompositions of O filters A 1 , B 1 , C 1 , . . . , A O , B O , C O and the number of filters after pruning O ′ . Ensure: Selected factors A p1 , B p1 , C p1 , . . ., A p O ′ , B p O ′ , C p O ′ . 1: Compute distance matrix D following (14). 2: for t = 1 to O -O ′ do 3:

Fig. 6 :

 6 Fig.6: Comparison of complexity reduction and accuracy with respect to the rank and the pruning ratio.

Fig. 8 :

 8 Fig. 8: Qualitative assessment of feature preservation in compressed models.

TABLE I :

 I Compression of VGG-16-BN on CIFAR

	Method	Type Top-1	MACs (CR) Params (CR)
	VGG-16-BN [59]		93.96	313.73M (00)	14.98M (00)
	HRank-1 [34]	P	93.43	145.61M (54)	2.51M (83)
	CHIP [32]	P	93.86	131.17M (58)	2.76M (82)
	EZCrop [31]	P	93.01	131.17M (58)	2.76M (82)
	DECORE-500 [30]	P	94.02	203.08M (35)	5.54M (63)
	APIB [27]	P	94.08	127.03M (60)	3.60M (76)
	AutoBot [28]	P	94.19	145.61M (54)	7.53M (50)
	NORTON (Ours)	H	94.45	126.49M (60)	2.58M (83)
	HRank-2 [34]	P	92.34	108.61M (65)	2.64M (82)
	DECORE-200 [30]	P	93.56	110.51M (65)	1.66M (89)
	Yeom et al. [25]	P	93.48	104.67M (66)	2.86M (81)
	EZCrop [31]	P	93.70	104.78M (67)	2.50M (83)
	CHIP [32]	P	93.72	104.78M (67)	2.50M (83)
	APIB [27]	P	94.00	106.67M (66)	3.30M (78)
	AutoBot [28]	P	94.01	108.71M (65)	6.44M (57)
	NORTON (Ours)	H	94.16	101.91M (68)	2.34M (84)
	RGP-64 16 [9]	P	92.76	78.78M (75)	3.81M (75)
	WhiteBox [29]	P	93.47	75.80M (76)	N/A
	AutoBot [28]	P	93.62	72.60M (77)	5.51M (63)
	NORTON (Ours)	H	94.11	74.14M (77)	3.60M (76)
	QSFM [76]	P	92.17	79.00M (75)	3.68M (75)
	DECORE-100 [30]	P	92.44	51.20M (82)	0.51M (96)
	FSM [74]	P	92.86	59.61M (81)	1.50M (90)
	ALDS [20]	D	92.67	66.95M (86)	1.90M (96)
	Dai et al. [5]	H	93.03	37.76M (87)	0.43M (97)
	Lebedev et al. [1]	D	93.07	68.53M (78)	3.22M (78)
	EPruner-0.73 [75]	P	93.08	74.42M (76)	1.65M (89)
	HALOC [10]	D	93.16	43.92M (86)	0.30M (98)
	CHIP [32]	P	93.18	66.95M (79)	1.90M (87)
	ASTER [26]	P	93.45	60.00M (81)	N/A
	EDP [8]	H	93.52	62.40M (80)	0.66M (96)
	HTP-URC-2 [67]	P	93.62	73.32M (77)	0.87M (94)
	DF [58]	H	93.64	94.00M (70)	1.47M (90)
	FSM [74]	P	93.73	106.67M (66)	2.10M (86)
	NORTON (Ours)	H	93.84	37.68M (88)	1.94M (87)
	HRank-3 [34]	P	91.23	73.70M (77)	1.78M( 92)
	RGP-64 6 [9]	P	91.45	31.37M (90)	1.43M (90)
	DECORE-50 [30]	P	91.68	36.85M (88)	0.26M (98)
	NORTON (Ours)	H	92.54	13.54M (96)	0.24M (98)
	NORTON (Ours)	H	90.32	4.58M (99)	0.14M (99)

TABLE II :

 II Compression results of ResNet-56/110 on CIFAR

	Method	Type Top-1	MACs (CR) Params (CR)
	ResNet-56 [60]		93.26	127.09M (00)	0.85M (00)
	HRank-1 [34]	P	93.52	88.72M (29)	0.71M (17)
	DECORE-450 [30]	P	93.34	92.48M (26)	0.64M (24)
	FilterSketch [73]	P	93.65	88.05M (30)	0.68M (21)
	TPP [46]	P	93.81	86.59M (31)	N/A
	WHC [47]	P	93.91	90.35M (28)	N/A
	NORTON (Ours)	H	94.37	83.11M (35)	0.54M (37)
	TRP [39]	D	92.77	55.60M (56)	N/A
	HRank-2 [34]	P	93.17	62.72M (50)	0.49M (42)
	FilterSketch [73]	P	93.19	73.36M (41)	0.50M (41)
	SOKS-40% [71]	P	93.22	81.40M (36)	0.51M (40)
	DECORE-200 [30]	P	93.26	62.93M (50)	0.43M (49)
	TPP [46]	P	93.46	62.75M (50)	N/A
	BSR [17]	D	93.53	55.70M (56)	0.37M (56)
	MFP [72]	P	93.56	59.40M (53)	N/A
	EDP [8]	H	93.61	53.07M (58)	0.39M (54)
	FSM [74]	P	93.63	61.49M (51)	0.48M (44)
	CC-0.5 [6]	H	93.64	60.00M (52)	0.44M (48)
	ResRep [49]	P	93.71	59.30M (53)	N/A
	DCP [77]	P	93.72	56.47M (55)	0.43M (50)
	AutoBot [28]	P	93.76	55.82M (56)	0.46M (46)
	DeGraph [70]	P	93.77	58.98M (53)	N/A
	WHC [47]	P	93.80	74.04M (41)	N/A
	EZCrop [31]	P	93.80	65.94M (47)	0.48M (43)
	NORTON (Ours)	H	94.00	73.22M (42)	0.44M (48)
	QSFM [76]	P	91.88	50.62M (60)	0.25M (71)
	CHIP [32]	P	92.05	34.79M (72)	0.24M (72)
	TPP [46]	P	92.35	36.39M (71)	N/A
	BSR [17]	D	92.51	32.10M (74)	0.21M (75)
	SOKS-55% [71]	P	93.08	61.30M (52)	0.39M (54)
	CLR-RNF-0.56 [48]	P	93.27	54.00M (57)	0.38M (56)
	Yang et .al [40]	D	93.27	33.88M (73)	N/A
	APIB [27]	P	93.29	41.41M (67)	0.29M (66)
	DF [58]	H	93.44	32.20M (75)	0.22M (74)
	NORTON (Ours)	H	93.81	37.52M (71)	0.21M (75)
	HRank-3 [34]	P	90.72	32.52M (74)	0.27M (68)
	DECORE-55 [30]	P	90.85	23.22M (81)	0.13M (85)
	FilterSketch [73]	P	91.20	32.47M (74)	0.24M (72)
	APIB [27]	P	91.53	23.84M (81)	0.15M (83)
	NORTON (Ours)	H	91.62	14.47M (89)	0.08M (91)
	ResNet-110 [60]		93.50	256.04M (00)	1.73M (00)
	DECORE-500 [30]	P	93.88	163.30M (35)	1.11M (36)
	NORTON (Ours)	H	94.85	163.00M (35)	1.08M (38)
	NNCS-61.7 [55]	P	93.41	99.86M (61)	0.66M (62)
	DECORE-300 [30]	P	93.50	96.66M (62)	0.61M (65)
	NORTON (Ours)	H	94.11	92.99M (64)	0.59M (65)
	DECORE-175 [30]	P	92.71	58.37M (77)	0.35M (80)
	NORTON (Ours)	H	92.77	47.34M (82)	0.30M (83)

TABLE III :

 III Compression results of DenseNet-40 on CIFAR

	Method	Type Top-1	MACs (CR) Params (CR)
	DenseNet-40 [61]		94.81	282.92M (00)	1.04M (00)
	DECORE-175 [30]	P	94.85	228.96M (19)	0.83M (21)
	NORTON (Ours)	H	94.86	213.58M (26)	0.74M (30)
	HRank-1 [34]	P	94.24	167.41M (41)	0.66M (37)
	DECORE-115 [30]	P	94.59	171.36M (39)	0.56M (46)
	Yeom et al. [25]	P	94.62	167.41M (41)	0.66M (37)
	AutoBot [28]	P	94.67	167.64M (42)	0.76M (28)
	NORTON (Ours)	H	94.67	168.23M (42)	0.58M (45)
	HRank-2 [34]	P	93.68	110.15M (61)	0.48M (54)
	EZCrop [31]	P	93.76	113.08M (60)	0.39M (62)
	DECORE-70 [30]	P	94.04	128.13M (55)	0.37M (65)
	HT2-0.8 [13]	D	94.06	104.31M (63)	0.34M (67)
	NORTON (Ours)	H	94.17	103.68M (64)	0.33M (69)

TABLE IV :

 IV Compression results of ResNet-50 on ImageNet

	Method	Type Top-1 Top-5 MACs (CR)	Params (CR)
	ResNet-50 [60]		76.15	92.87	4.09G (00)	25.50M (00)
	ABCPruner-100% [56]	P	72.84	92.97	2.56G (37)	18.02M (29)
	CLR-RNF-0.2 [48]	P	74.85	92.31	2.45G (40)	16.92M (34)
	EPruner-0.81 [75]	P	74.95	92.36	2.37G (42)	N/A
	FilterSketch-0.7 [73]	P	75.22	92.41	2.64G (36)	16.95M (33)
	Kim et al. [12]	D	75.34	92.68	N/A	17.60M (31)
	PFP [53]	P	75.91	92.81	3.65G (11)	20.88M (18)
	C-SGD-70 [68]	P	75.94	92.88	2.62G (36)	17.09M (33)
	LeGR [50]	P	76.20	93.00	2.99G (27)	N/A
	DECORE-8 [30]	P	76.31	93.02	3.54G (13)	22.69M (11)
	CHIP [32]	P	76.30	93.02	2.26G (44)	15.10M (41)
	TPP [46]	P	76.44	N/A	2.74G (33)	N/A
	NORTON (Ours)	H	76.91	93.57	2.32G (43)	14.51M (43)
	FilterSketch-0.6 [73]	P	74.68	92.17	2.23G (46)	14.53M (43)
	Hinge [24]	H	74.70	N/A	2.17G (47)	N/A
	HRank-1 [34]	P	74.98	92.33	2.30G (44)	16.15M (37)
	DECORE-6 [30]	P	74.58	92.18	2.36G (42)	14.10M (45)
	PFP [53]	P	75.21	92.43	2.29G (44)	17.82M (30)
	RGP-64 36 [9]	P	75.30	92.55	2.30G (44)	14.34M (44)
	WhiteBox [29]	P	75.32	92.43	2.22G (46)	N/A
	MFP [72]	P	75.67	92.81	2.37G (42)	N/A
	EZCrop [31]	P	75.68	92.70	2.26G (45)	15.09M (41)
	LeGR [50]	P	75.70	92.70	2.37G (42)	N/A
	C-SGD-60 [68]	P	75.80	92.65	2.19G (47)	14.58M (43)
	DepGraph [70]	P	75.83	N/A	2.09G (49)	N/A
	SCOP [51]	P	75.95	92.79	2.24G (45)	14.59M (43)
	CATRO [52]	P	75.98	92.79	2.21G (46)	N/A
	WHC [47]	P	76.06	92.86	2.37G (42)	N/A
	CHIP [32]	P	76.15	92.91	2.10G (49)	14.23M (44)
	DNCP [54]	P	76.30	N/A	2.20G (46)	N/A
	NORTON (Ours)	H	76.58	93.43	2.08G (50)	13.51M (47)
	HRank-2 [34]	P	71.98	91.01	1.55G (62)	13.77M (46)
	TRP [39]	D	72.86	91.58	1.78G (56)	N/A
	FilterSketch-0.4 [73]	P	73.04	91.18	1.51G (63)	10.40M (59)
	WhiteBox [29]	P	74.21	92.01	1.50G (63)	N/A
	EZCrop [31]	P	74.33	92.00	1.52G (63)	11.05M (57)
	DAIS [69]	P	74.45	92.21	1.83G (55)	N/A
	CC-0.6 [6]	H	74.54	92.25	1.53G (63)	10.58M (59)
	RGP-64 30 [9]	P	74.58	92.09	1.92G (53)	11.99M (53)
	Phan et al. [2]	D	74.68	92.16	1.56G (62)	N/A
	MFP [72]	P	74.86	92.43	1.88G (54)	N/A
	TPP [46]	P	75.12	N/A	1.60G (61)	N/A
	Yeom et al. [25]	P	75.25	92.49	1.52G (63)	11.05M (57)
	SCOP [51]	P	75.26	92.53	1.86G (55)	12.29M (52)
	CHIP [32]	P	75.26	92.53	1.52G (63)	11.04M (57)
	ASTER [26]	P	75.27	92.47	1.51G (63)	N/A
	C-SGD-60 [68]	P	75.29	92.39	1.82G (55)	12.37M (52)
	LeGR [50]	P	75.30	92.40	1.92G (53)	N/A
	ResRep [49]	P	75.30	92.47	1.52G (62)	N/A
	WHC [47]	P	75.33	92.52	1.88G (54)	N/A
	EDP [8]	H	75.34	92.43	1.92G (53)	14.28M (44)
	APIB [27]	P	75.37	N/A	1.56G (62)	10.71M (58)
	HTP-URC [67]	P	75.81	N/A	1.88G (54)	15.81M (38)
	NORTON (Ours)	H	75.95	92.91	1.49G (64)	10.52M (59)
	HRank-3 [34]	P	69.10	89.58	0.98G (76)	8.27M (68)
	DECORE-5 [30]	P	72.06	90.82	1.60G (61)	8.87M (65)
	Yeom et al. [25]	P	72.28	90.93	0.95G (77)	8.02M (67)
	ABCPruner-50% [56]	P	72.58	90.91	1.30G (68)	9.10M (64)
	CHIP [32]	P	72.30	90.74	0.95G (77)	8.01M (69)
	CLR-RNF-0.44 [48]	P	72.67	91.09	1.23G (70)	9.00M (65)
	EPruner-0.81 [75]	P	72.73	91.01	1.29G (68)	N/A
	NORTON (Ours)	H	74.00	92.00	0.96G (77)	7.96M (69)
	FilterSketch-0.2 [73]	P	69.43	89.23	0.93G (77)	7.18M (72)
	DECORE-4 [30]	P	69.71	89.37	1.19G (71)	6.12M (76)
	RGP-64 16 [9]	P	72.68	91.06	1.02G (75)	6.38M (75)
	CURL [78]	P	73.39	91.46	1.11G (73)	6.67M (74)
	NORTON (Ours)	H	73.65	91.64	0.92G (78)	5.88M (77)

TABLE V :

 V Compression results of Faster/Mask/Keypoint-RCNN-ResNet50-FPN on COCO-2017

	Model	AP 0.5:0.95	AP 0.5	AP 0.75	AR 1	AR 10	AR 100	MACs (CR) Params (CR) FPS Latency(ms)
	FasterRCNN [64], [79]	0.37	0.58	0.39	0.31	0.48	0.51	134.85G (00)	41.81M (00)	12	85
	NORTON (Ours)	0.38	0.59	0.42	0.32	0.50	0.52	111.47G (17)	30.72M (27)	19	53
	NORTON (Ours)	0.32	0.52	0.34	0.29	0.46	0.48	93.39G (31)	22.01M (47)	25	41
	MaskRCNN [65], [79]	0.34	0.55	0.36	0.29	0.45	0.47	134.85G (00)	44.46M (00)	9	111
	NORTON (Ours)	0.35	0.57	0.37	0.30	0.46	0.48	111.47G (17)	33.36M (25)	14	73
	NORTON (Ours)	0.32	0.52	0.33	0.28	0.44	0.46	93.39G (31)	24.65M (45)	20	50
					AR 0.5:0.95	AR 0.5	AR 0.75
	KeypointRCNN [65], [79] 0.65	0.86	0.71	0.71	0.90	0.77	137.42G (00)	59.19M (00)	8	125
	NORTON (Ours)	0.65	0.86	0.71	0.72	0.91	0.77	114.04G (17)	48.10M (19)	13	76
	NORTON (Ours)	0.63	0.85	0.69	0.69	0.90	0.75	95.97G (30)	39.39M (34)	17	59

TABLE VI :

 VI Complexity reduction, approximation error, and accuracy with and without fine-tuning with respect to the rank

	Rank MACs Params	NMSE	Accuracy (%)
		CR	CR		Without FT With FT
	1	88.03	87.06	0.6265	10.00	93.84
	2	76.44	75.98	0.4114	10.00	94.11
	3	64.85	64.91	0.2760	68.37	94.18
	4	53.27	53.84	0.1837	88.30	94.07
	5	41.69	42.76	0.1173	92.44	94.22
	6	30.10	31.69	0.0698	93.45	94.15
	7	18.51	20.61	0.0372	93.90	94.11
	8	6.93	9.54	0.0137	94.03	94.03