Haozhe Sun
email: haozhe.sun@universite-paris-saclay.fr

Isabelle Guyon

Modularity in Deep Learning: A Survey

Modularity is a general principle present in many fields. It offers attractive advantages, including, among others, ease of conceptualization, interpretability, scalability, module combinability, and module reusability. The deep learning community has long sought to take inspiration from the modularity principle, either implicitly or explicitly. This interest has been increasing over recent years. We review the notion of modularity in deep learning around three axes: data, task, and model, which characterize the life cycle of deep learning. Data modularity refers to the observation or creation of data groups for various purposes. Task modularity refers to the decomposition of tasks into sub-tasks. Model modularity means that the architecture of a neural network system can be decomposed into identifiable modules. We describe different instantiations of the modularity principle, and we contextualize their advantages in different deep learning sub-fields. Finally, we conclude the paper with a discussion of the definition of modularity and directions for future research.

Introduction

Modularity is a general principle present in many fields such as biology [START_REF] Barrett | Modularity in cognition: Framing the debate[END_REF][START_REF] Bongard | Evolving modular genetic regulatory networks[END_REF][START_REF] Clune | The evolutionary origins of modularity[END_REF][START_REF] Cosmides | Cognitive Adaptations for Social Exchange[END_REF][START_REF] Cosmides | Origins of domain specificity: The evolution of functional organization[END_REF][START_REF] Fodor | The Modularity of Mind[END_REF][START_REF] Fodor | The Mind Doesn't Work That Way: The Scope and Limits of Computational Psychology[END_REF][START_REF] Frankenhuis | Evolutionary Psychology Versus Fodor: Arguments For and Against the Massive Modularity Hypothesis[END_REF][START_REF] Michel A Hofman | Evolution of the human brain: When bigger is better[END_REF][START_REF] Kurzweil | How to Create a Mind: The Secret of Human Thought Revealed[END_REF][START_REF] Jose B Pereira-Leal | The origins and evolution of functional modules: Lessons from protein complexes[END_REF][START_REF] Pylyshyn | Is vision continuous with cognition?: The case for cognitive impenetrability of visual perception[END_REF][START_REF] Robbins | Modularity of Mind[END_REF][START_REF] Wagner | Perspective: Complex Adaptations and the Evolution of Evolvability[END_REF], complex systems [START_REF] Simon | The Architecture of Complexity[END_REF][START_REF] Simon | Aggregation of variables in dynamic systems[END_REF], mathematics [START_REF] Avigad | Modularity in mathematics[END_REF][START_REF] Bourbaki | The Architecture of Mathematics[END_REF], system design [START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF][START_REF] Gentile | Theory of Modularity, a Hypothesis[END_REF][START_REF] Modrak | Development of the Modularity Measure for Assembly Process Structures[END_REF][START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF][START_REF] Shao | Modularity measures: Concepts, computation, and applications to manufacturing systems[END_REF], computer science [START_REF] Carliss | Design Rules: The Power of Modularity[END_REF][START_REF] Ford | Architects of Intelligence: The Truth about AI from the People Building It[END_REF], graph theory [START_REF] Daniel Gómez | A new modularity measure for Fuzzy Community detection problems based on overlap and grouping functions[END_REF][START_REF] Muff | Local modularity measure for network clusterizations[END_REF][START_REF] Newman | Modularity and community structure in networks[END_REF][START_REF] Poisot | An a posteriori measure of network modularity[END_REF]. While sharing the same name, there is no universally agreed upon definition of modularity [START_REF] Béna | Extreme sparsity gives rise to functional specialization[END_REF]. However, we can identify a shared definition [START_REF] Amer | A Review of Modularization Techniques in Artificial Neural Networks[END_REF][START_REF] Schmidt | Modularity -A Concept For New Neural Network Architectures[END_REF]: in general, modularity is the property of an entity whereby it can be broken down into a number of sub-entities (referred to as modules). This definition has different instantiations in different fields with their nuances [START_REF] Schilling | Toward a General Modular Systems Theory and Its Application to Interfirm Product Modularity[END_REF] from which various properties may arise. Such field-specific properties include autonomy of modules (limited interaction or limited interdependence between modules) [START_REF] Goyal | Neural production systems[END_REF][START_REF] Avigad | Modularity in mathematics[END_REF][START_REF] Azam | Biologically Inspired Modular Neural Networks[END_REF][START_REF] Carliss | Design Rules: The Power of Modularity[END_REF][START_REF] Clune | The evolutionary origins of modularity[END_REF][START_REF] Galanti | On the Modularity of Hypernetworks[END_REF][START_REF] Goyal | Recurrent independent mechanisms[END_REF][START_REF] Huizinga | Evolving neural networks that are both modular and regular: Hyperneat plus the connection cost technique[END_REF][START_REF] Jacobs | Adaptive Mixtures of Local Experts[END_REF][START_REF] Modrak | Development of the Modularity Measure for Assembly Process Structures[END_REF][START_REF] Newman | Modularity and community structure in networks[END_REF][START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF][START_REF] Shao | Modularity measures: Concepts, computation, and applications to manufacturing systems[END_REF][START_REF] Yu | MAttNet: Modular attention network for referring expression comprehension[END_REF], functional specialization of modules [START_REF] Clune | The evolutionary origins of modularity[END_REF][START_REF] Cosmides | Origins of domain specificity: The evolution of functional organization[END_REF][START_REF] Fodor | The Modularity of Mind[END_REF][START_REF] Frankenhuis | Evolutionary Psychology Versus Fodor: Arguments For and Against the Massive Modularity Hypothesis[END_REF][START_REF] Gentile | Theory of Modularity, a Hypothesis[END_REF][START_REF] Kurzweil | How to Create a Mind: The Secret of Human Thought Revealed[END_REF][START_REF] Robbins | Modularity of Mind[END_REF], reusability of modules [6, 12, 14, 43, 51, 55, 61, 142, 174-176, 184, 188, 192, 207], combinability of modules [START_REF] Alet | Modular meta-learning[END_REF][START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Brenden | Compositional generalization through meta sequence-to-sequence learning[END_REF][START_REF] Li | Learning the compositional visual coherence for complementary recommendations[END_REF][START_REF] Mittal | Compositional attention: Disentangling search and retrieval[END_REF][START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF][START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF][START_REF] Rahaman | Dynamic inference with neural interpreters[END_REF][START_REF] Veniat | Efficient Continual Learning with Modular Networks and Task-Driven Priors[END_REF], replaceability of modules [START_REF] Pan | On Decomposing a Deep Neural Network into Modules[END_REF][START_REF] Pan | Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules[END_REF][START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF].

As a general principle, modularity is a descriptive property and an organizational scheme. It is a means of representing entities (data, tasks, models) to be able to manipulate them, conceptually or practically [START_REF] Carliss | Design Rules: The Power of Modularity[END_REF][START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF][START_REF] Ghazi | Recursive Sketches for Modular Deep Learning[END_REF][START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF]. Though modular entities are not necessarily hierarchical [START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF], many modular entities have a hierarchical structure [START_REF] Simon | The Architecture of Complexity[END_REF] in the sense that multiple modules of a lower hierarchy level can form one module of a higher hierarchy level. The modules of the lower hierarchy level are of finer granularity than those of the higher hierarchy level. At the same level of the hierarchy, modules can refer to an exclusive division of the overall entity (hard division) or overlapping parts of the overall entity (soft division). The decomposed modules can be homogeneous (similar modules) or heterogeneous (dissimilar modules).

Back to the very beginning of neural network research in the last century, the community started to be interested in bringing the notion of modularity to neural networks [START_REF] Auda | Modular neural networks a survey[END_REF][START_REF] Azam | Biologically Inspired Modular Neural Networks[END_REF][START_REF] Jacobs | Adaptive Mixtures of Local Experts[END_REF][START_REF] Reisinger | Evolving Reusable Neural Modules[END_REF], this interest has been revived recently [START_REF] Alet | Modular meta-learning[END_REF][START_REF] Amer | A Review of Modularization Techniques in Artificial Neural Networks[END_REF][START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Chang | Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignment[END_REF][START_REF] Csordás | Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks[END_REF][START_REF] Fernando | PathNet: Evolution Channels Gradient Descent in Super Neural Networks[END_REF][START_REF] Daniel Filan | Clusterability in Neural Networks[END_REF][START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF][START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF][START_REF] Veniat | Efficient Continual Learning with Modular Networks and Task-Driven Priors[END_REF]. The publication trend (Figure 1) shows an increasing interest in the modularity principle within deep learning over recent years. This survey investigates the notion of modularity in deep learning around three axes: data, task, and model. The organization of the survey is shown in Figure 2. The ratio of the count of publications containing "modular deep learning" and "modular neural network" among publications containing "deep learning" and "neural network", indexed by Google Scholar. The horizontal axis is the publication year.

Figure 2: Organization of this survey. The first three sections discuss how the modularity principle is instantiated in the three axes: data, task, and model architecture. We then cover other modularity notions for completeness. Finally, we discuss the definition of modularity and directions for future research. The introduction and conclusion are ignored in this figure.

Data modularity

Data is an entity used to represent knowledge and information. In the context of machine learning and deep learning, it can take various forms e.g., image, audio sound, and text. Data samples can be interpreted as points in a high dimensional space (fixed-length dense vectors) [START_REF] Almeida | Word Embeddings: A Survey[END_REF][START_REF] Koh | Comparison and Analysis of Deep Audio Embeddings for Music Emotion Recognition[END_REF][START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF]. A collection of data samples is a dataset. Datasets can be used to train or test deep learning models, referred to as training or test datasets. In these scenarios, data is the input of deep learning models (neural networks) [START_REF] Goodfellow | Deep Learning[END_REF].

Data modularity is the observation or creation of data groups; it refers to how a dataset can be divided into different modules for various purposes. The division of the dataset into modules facilitates conception and data manipulation. Data modularization can influence the training of learning machines [START_REF] Baz | Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification[END_REF][START_REF] Hacohen | On The Power of Curriculum Learning in Training Deep Networks[END_REF][START_REF] Sun | OmniPrint: A Configurable Printed Character Synthesizer[END_REF]. Some algorithms leverage data modularity so that each data module is processed by a different solver [START_REF] Qiao | A novel modular RBF neural network based on a brain-like partition method[END_REF].

We identify two types of data modularity: intrinsic data modularity and imposed data modularity. Intrinsic data modularity means identifiable dataset divisions naturally in data, which a human practitioner does not introduce. Imposed data modularity means identifiable dataset divisions that a human practitioner introduces. The rationale of this taxonomy is that when the dataset reaches the practitioner who analyses it, it already contains some form of intrinsic modularity, including that stemming from the class labels. The people who collect the data are not considered practitioners.

Intrinsic data modularity

Intrinsic data modularity means identifiable dataset divisions naturally in data, which are not introduced by a human practitioner.

Any supervised learning datasets can be divided according to the classes (labels); data points belonging to the same class are supposed to be close to each other in a hidden space, which allows for solutions of classification algorithms. Classes sharing common semantics can be further grouped to form super-classes. For example, Ima-geNet [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF] has a class hierarchy (see Figure 3 (a)) which is used by Meta-Dataset [START_REF] Triantafillou | Meta-dataset: A dataset of datasets for learning to learn from few examples[END_REF]. Omniglot dataset [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF] and OmniPrint datasets [START_REF] Sun | OmniPrint: A Configurable Printed Character Synthesizer[END_REF] contain character images organized in scripts, each script (super-class) contains several characters (classes); Meta-Album dataset [START_REF] Ullah | Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification[END_REF] is a meta-dataset including 40 datasets, where each dataset can be considered as a super-class. The super-classes provide information about class similarity, allowing splitting datasets according to the semantics [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF].

In addition to the classes or super-classes, data points can also be grouped by one or several metadata such as time, location, and gender. Such metadata is available with the Exif data of photos. The OmniPrint data synthesizer generates data together with a comprehensive set of metadata, including font, background, foreground, margin size, shear angle, rotation angle, etc. [START_REF] Sun | OmniPrint: A Configurable Printed Character Synthesizer[END_REF] (see Figure 3 (b)). The NORB dataset collected stereo image pairs of 50 uniform-colored toys under 36 angles, 9 azimuths, and 6 lighting conditions, where the angles, azimuths, and lighting conditions serve as the metadata [START_REF] Lecun | Learning methods for generic object recognition with invariance to pose and lighting[END_REF].

Some datasets contain intrinsic clusters in the high-dimensional feature space. from the underlying data generative process, where latent categorical variables determine the natural groups of data.

An illustrative example is a Gaussian Mixture distribution where data points are assumed to be generated from a mixture of a finite number of Gaussian distributions with unknown parameters [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference and Prediction[END_REF]. Some datasets have intrinsic manifolds; an illustrative example is the moons dataset as shown in Figure 3 (c), where the two manifolds interlace while preserving an identifiable division, each manifold can be considered as a module. Both of the above examples fall into the category of data clustering. When data samples are interconnected in the form of a graph [START_REF] Ma | Disentangled graph convolutional networks[END_REF][START_REF] Wu | Learning the implicit semantic representation on graph-structured data[END_REF], this is called graph partitioning. One question which arises is how to determine the optimal clustering of a dataset. Luxburg et al. [START_REF] Ulrike Von Luxburg | Clustering: Science or Art?[END_REF] argue that there are no optimal domain-independent clustering algorithms and that clustering should always be studied in the context of its end-use. Multi-modal deep learning aims to build models that can process and relate information from multiple modalities. Here the modality refers to the way in which something happens or is experienced e.g., data in the form of image, text, audio [START_REF] Baltrušaitis | Multimodal machine learning: A survey and taxonomy[END_REF]. Multi-modal datasets fall into the category of intrinsic data modularity in the sense that the data in each modality can be considered a module. For example, VQA v2.0 dataset [START_REF] Goyal | Making the v in vqa matter: Elevating the role of image understanding in visual question answering[END_REF] consists of open-ended questions about images; SpeakingFaces dataset [START_REF] Abdrakhmanova | Speakingfaces: A large-scale multimodal dataset of voice commands with visual and thermal video streams[END_REF] consists of aligned thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking.

Imposed data modularity

Imposed data modularity means identifiable dataset divisions which are introduced by a human practitioner.

When training deep learning models [START_REF] Goodfellow | Deep Learning[END_REF], human practitioners usually divide the whole training dataset into mini-batches, which can be seen as a kind of imposed data modularity. The gradient is computed using one minibatch of data for each parameter update; one training epoch means passing through all the mini-batches. This iterative learning regime is called stochastic gradient descent [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF]. Mini-batches reduce the memory requirement for backpropagation, which makes training large deep learning models possible. On the other hand, batch size also influences learning behavior. Smith et al. [START_REF] Samuel | Don't decay the learning rate, increase the batch size[END_REF] showed that the benefits of decaying the learning rate could be obtained by instead increasing the training batch size. Keskar et al. [START_REF] Shirish Keskar | On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima[END_REF] showed that learning with large batch sizes usually gives worse generalization performance.

Instead of using a sequence of mini-batches sampled uniformly at random from the entire training dataset, curriculum learning [START_REF] Hacohen | On The Power of Curriculum Learning in Training Deep Networks[END_REF] uses non-uniform sampling of mini-batches such that the mini-batch sequence exhibits an increasing level of difficulty. A related concept is active learning [START_REF] Ren | A survey of deep active learning[END_REF], which assumes that different data points in a dataset have different values for the current model update; it tries to select the data points with the highest value to construct the actual training set.

The model performance is usually tested on few-shot episodes in few-shot learning and meta-learning. Few-shot episodes are typically formed by drawing several classes N from the class pool and several examples K for each selected class, called N -way-K-shot episodes [START_REF] Finn | Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks[END_REF][START_REF] Snell | Prototypical Networks for Few-shot Learning[END_REF] (Figure 3 (d)). For such scenarios, the meta-training phase can employ the same episodic learning regime or not [START_REF] Triantafillou | Meta-dataset: A dataset of datasets for learning to learn from few examples[END_REF], recent studies [START_REF] Laenen | On episodes, prototypical networks, and few-shot learning[END_REF][START_REF] Wang | Bridging multi-task learning and meta-learning: Towards efficient training and effective adaptation[END_REF][START_REF] Wang | The role of global labels in few-shot classification and how to infer them[END_REF] and competition results [START_REF] Baz | Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification[END_REF] suggest that episodic meta-training is not more effective than vanilla pretraining with access to the global class pool.

Data augmentation is a way to generate more training data by applying transformations to existing data [START_REF] Simard | Best practices for convolutional neural networks applied to visual document analysis[END_REF]. The transformed versions of the same data point can be seen as a module. Some transformations, such as rotation and translation, form a group structure [START_REF] John S Rose | A Course on Group Theory[END_REF]. The effect of such data augmentation can be understood as averaging over the orbits of the group that keeps the data distribution approximately invariant and leads to variance reduction [START_REF] Chen | A group-theoretic framework for data augmentation[END_REF].

In addition to splitting the dataset into subsets of samples, each data sample can be split into subdivisions of features, referred to as feature partitioning. A dataset can be represented as a matrix where each row represents one data sample; each column represents one feature dimension. It can then be divided along the sample and feature dimensions. Schmidt et al. [START_REF] Schmidt | Modularity -A Concept For New Neural Network Architectures[END_REF] process each feature partition with a different model. For image classification tasks, input images can be split into small patches that can be processed in parallel [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF][START_REF] Jin | Split-CNN: Splitting Window-Based Operations in Convolutional Neural Networks for Memory System Optimization[END_REF].

Conclusion of data modularity

We argue that data without structure contains no useful information for learning dependencies (e.g., between feature and label). Some dependencies boil down to the emergence or the creation of groups. Intrinsic data modularity relates to the semantic relationship between samples and how data samples are similar or dissimilar. Imposed data modularity, on the other hand, relates to the way that practitioners organize data at hand to better train learning machines.

Future research for data-centric deep learning may investigate the relationship between intrinsic and imposed data modularity. For example, does intrinsic data modularity promote imposed data modularity? How does this interplay affect model training?

Data modularity describes how the input of deep learning models can be modularized. On the other hand, the end goal (the output) of deep learning models can also be modularized, which is the topic of the next section. Deep learning models are tools to solve tasks e.g., from the classification of entities to the generation of realistic photos. Solving a task is equal to achieving a corresponding objective. In deep learning, we usually model an objective by an explicit differentiable objective function (also known as a loss function), allowing end-to-end training. This perspective can be generalized to any task, even if the objective function is implicit and does not entail a differentiable form. For example, the task of "purchasing a cup of tea" can be characterized by an indicator function that returns a penalty if no tea can be purchased or a bonus otherwise. In deep learning, tasks are often related to data; but they are different. Given the same dataset, one can define various tasks on top of it. For example, the MNIST dataset can be used either for an image classification benchmarking task [START_REF] Mazzia | Efficient-capsnet: Capsule network with self-attention routing[END_REF] or for a pixel sequence classification benchmarking task [START_REF] Goyal | Recurrent independent mechanisms[END_REF][START_REF] Krueger | Zoneout: Regularizing RNNs by randomly preserving hidden activations[END_REF], the OmniPrint-meta [START_REF]Accelerate Fast Math with Intel® oneAPI Math Kernel Library[END_REF][START_REF] Abbas | DeTraC: Transfer Learning of Class Decomposed Medical Images in Convolutional Neural Networks[END_REF][START_REF] Abdrakhmanova | Speakingfaces: A large-scale multimodal dataset of voice commands with visual and thermal video streams[END_REF][START_REF] Wickliffe | Memory retention -the synaptic stability versus plasticity dilemma[END_REF][START_REF] Alain | Understanding intermediate layers using linear classifier probes[END_REF] datasets [START_REF] Sun | OmniPrint: A Configurable Printed Character Synthesizer[END_REF] can be used either for a few-shot learning benchmarking task or for domain adaptation benchmarking task. Tasks define the objective; they are orthogonal to how the end goal should be achieved. This section presents task modularity i.e., sub-task decomposition. Sub-task decomposition means that a task could be factorized or decomposed into sub-tasks. Sub-task decomposition facilitates conceptualization and problem-solving. The divide-and-conquer principle breaks down a complex problem into easier sub-problems [START_REF] Azam | Biologically Inspired Modular Neural Networks[END_REF][START_REF] Thomas H Cormen | Introduction to Algorithms[END_REF][START_REF] Jacobs | Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks[END_REF][START_REF] Qiao | A novel modular RBF neural network based on a brain-like partition method[END_REF]. By solving each individual sub-problem and combining the solutions, the complex problem can be solved more efficiently. The sub-task decomposition facilitates the integration of expert knowledge, and the a priori knowledge can further facilitate problem-solving. Sub-task decomposition can also promote reuse if the overall task is compositional; the solution to sub-tasks may be reused in other tasks [START_REF] Devin | Learning modular neural network policies for multi-task and multi-robot transfer[END_REF][START_REF] Meyerson | Modular universal reparameterization: Deep multi-task learning across diverse domains[END_REF][START_REF] Ostapenko | Attention for compositional modularity[END_REF][START_REF] Purushwalkam | Task-driven modular networks for zero-shot compositional learning[END_REF][START_REF] Simpkins | Composable modular reinforcement learning[END_REF].

Task modularity

The sub-task decomposition can be categorized into two regimes: parallel decomposition and sequential decomposition (Figure 4). Parallel decomposition means that the sub-tasks can be executed in parallel. Sequential decomposition means that the sub-tasks need to be executed in order; certain sub-tasks cannot be executed before the previous sub-task is finished. In practice, these two regimes can be mixed. For example, a sub-task from a sequential decomposition can be further decomposed parallelly, which leads to a directed acyclic graph workflow.

Parallel sub-task decomposition

A parallel sub-task decomposition is called homogeneous if the decomposed sub-tasks are similar. One typical example is dividing a multi-class classification problem into multiple smaller classification problems [START_REF] Ghazi | Recursive Sketches for Modular Deep Learning[END_REF]. Given a neural network trained to perform a multi-class classification problem, Csordás et al. [START_REF] Csordás | Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks[END_REF] identify subsets of parameters solely responsible for individual classes on their own. Kim et al. [START_REF] Kim | SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization[END_REF] learn to split a neural network into a tree structure to handle different subsets of classes. They assume that different classes use different features, the tree-structured neural network ensuring that the later layers do not share features across different subsets of classes. Pan et al. [START_REF] Pan | On Decomposing a Deep Neural Network into Modules[END_REF][START_REF] Pan | Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules[END_REF] and Kingetsu et al. [START_REF] Kingetsu | Neural Network Module Decomposition and Recomposition[END_REF] decompose a multi-class classification model into reusable, replaceable and combinable modules, where each module is a binary classifier. Such modules can be recombined without retraining to obtain a new multi-class classifier. These methods can be useful in situations where the classes to be classified frequently change. Abbas et al. [START_REF] Abbas | DeTraC: Transfer Learning of Class Decomposed Medical Images in Convolutional Neural Networks[END_REF] use transfer learning and class decomposition to improve the performance of medical image classification. Such sub-task decomposability is an implicit prerequisite of the model editing problem [START_REF] Kassner | BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief[END_REF][START_REF] Meng | Locating and Editing Factual Associations in GPT[END_REF][START_REF] Mitchell | Fast Model Editing at Scale[END_REF][START_REF] Mitchell | Memory-Based Model Editing at Scale[END_REF][START_REF] Sinitsin | Editable Neural Networks[END_REF]. Model editing aims to modify a specific sub-task learned by a trained neural network without damaging model performance on other inputs, e.g., it aims to patch the mistake of the model for a particular sample. If the task cannot be decomposed into disentangled sub-tasks, then model editing cannot be achieved.

A parallel sub-task decomposition is termed heterogeneous if the decomposed sub-tasks are dissimilar; such decomposition is usually problem-dependent and requires expert knowledge of the task at hand. Belay et al. [START_REF] Belay | Factored Convolutional Neural Network for Amharic Character Image Recognition[END_REF] decompose the recognition task of Amharic characters into a vowel recognition task and a consonant recognition task to reduce overall task complexity. Cao et al. [START_REF] Cao | DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering[END_REF] decompose the full self-attention into question-wide and passage-wide self-attentions to speed up inference for question answering tasks. Ding et al. [START_REF] Ding | Trunk-branch ensemble convolutional neural networks for video-based face recognition[END_REF] decompose the facial recognition task into multiple facial component recognition tasks. Zhou et al. [START_REF] Zhou | Meta-Learning Symmetries by Reparameterization[END_REF] decompose the neural network learning task into structure learning and parameter learning to learn equivariance from data automatically. Gatys et al. [START_REF] Gatys | Image Style Transfer Using Convolutional Neural Networks[END_REF] decompose the natural image synthesis task into a content component and a style component, which allows recombining the content and the style in a combinatorial way to generate new images.

Sequential sub-task decomposition

Sequential sub-task decomposition reflects the sequential pipeline of the task. A simple example is the division of a machine learning task into a preprocessing stage (data cleaning and normalization) and a model inference stage [START_REF] Ranganathan | A study to find facts behind preprocessing on deep learning algorithms[END_REF].

In reinforcement learning, a complex task can usually be decomposed [START_REF] Simpkins | Composable modular reinforcement learning[END_REF] into a sequence of sub-tasks or steps. An illustrative example is to imagine that the task of manufacturing an artifact Z requires purchasing the raw material X, forging X to produce parts Y , and then assembling the parts Y into the end product Z. Both X and Y can take different values independently (X ∈ {x 1 , x 2 , x 3 , ...}, Y ∈ {y 1 , y 2 , y 3 , ...}). Different values of X and Y can be recombined, which forms a combinatorial number of possible scenarios to learn. This pipeline can be factorized into three stages: (1) raw material purchase, (2) forging to produce parts, and (3) assembling of parts. Reinforcement learning agents would learn more efficiently if the learning happens at the granularity of the factorized stages instead of the overall task [START_REF] Colas | Curious: Intrinsically motivated modular multi-goal reinforcement learning[END_REF]. Furthermore, such a factorization enables the independence of credit assignment [START_REF] Peters | Elements of Causal Inference: Foundations and Learning Algorithms[END_REF]; the failure of the overall task can be traced back to the problematic stages, while the other stages can remain untouched. For example, if the raw material is of bad quality, then the purchase sub-task needs to be improved; the forging sub-task and the assembling sub-task do not need to be changed [START_REF] Chang | Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignment[END_REF].

The sequential pipeline is omnipresent in practical applications e.g., optical character recognition (OCR), natural language processing (NLP). When facing a multi-script (multi-language) recognition task, the pipeline can consist of a script identification stage and a script-specific recognition stage [START_REF] Huang | A multiplexed network for end-to-end, multilingual OCR[END_REF][START_REF] Shi | Script identification in the wild via discriminative convolutional neural network[END_REF], which decouples the domain classifier and the domain-specific solver. The text-in-the-wild recognition task [START_REF] Chen | Text Recognition in the Wild: A Survey[END_REF] usually consists of decoupled text detector (to localize the bounding box of the text) and recognizer (recognize the text from the bounding box) [START_REF] Chen | Text Recognition in the Wild: A Survey[END_REF]. Traditional OCR methods also decompose the word recognition task into a character segmentation task and a character recognition task [START_REF] Richard | A survey of methods and strategies in character segmentation[END_REF][START_REF] Choudhary | A new character segmentation approach for off-line cursive handwritten words[END_REF][START_REF] Kaur | Study of various character segmentation techniques for handwritten off-line cursive words: A review[END_REF][START_REF] Schenkel | Recognition-based segmentation of on-line handprinted words[END_REF]. Traditional NLP pipeline includes sentence segmentation, word tokenization, part-of-speech tagging, lemmatization, filtering stop words, and dependency parsing [START_REF] Jurafsky | Speech and language processing[END_REF]. In bioinformatics, the scientific workflow (data manipulations and transformations) groups similar or strongly coupled workflow steps into modules to facilitate understanding and reuse [START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF].

Conclusion of task modularity

The sub-task decomposition can be parallel, sequential, or mixed (directed acyclic graph). We provided examples from the literature that leverage sub-task decomposition to reduce task complexity or promote the reuse of sub-task solutions. with model modularity, as will be discussed in the next section.

Future research may focus on how to automate the process of sub-task decomposition or make the problemdependent sub-task decomposition techniques transferable to other tasks, which is an important step for AutoML. It would reduce the demand for highly qualified deep learning engineers, which can reduce expert bias and entry barriers to deep learning.

Model modularity

This section presents model modularity. It means that the architecture of the neural network system (one neural network or a system of neural networks) consists of identifiable sub-entities (modules).

Model modularity is different from task modularity. A task define an objective, task modularity focuses on decomposing the objective into sub-objectives. Model modularity focuses on the architecture of the neural network system, it decomposes the solution into sub-solutions.

Advantages of model modularity

Model modularity provides ease of conceptual design and implementation. For example, modern neural networks consist of repeated layer/block patterns (modules). Examples include fully-connected neural networks [START_REF] Goodfellow | Deep Learning[END_REF], vanilla convolutional neural networks, ResNet [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Xie | Aggregated Residual Transformations for Deep Neural Networks[END_REF], Inception [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] and models searched by Neural Architecture Search (NAS) [START_REF] Elsken | Neural Architecture Search[END_REF][START_REF] Zoph | Learning Transferable Architectures for Scalable Image Recognition[END_REF]. The design with homogeneous modules allows for a more concise description of the model architecture in the sense of Kolmogorov complexity (short description length) [START_REF] Lecun | Optimal brain damage[END_REF][START_REF] Li | An Introduction to Kolmogorov Complexity and Its Applications[END_REF]. For example, instead of specifying how each primitive operation (e.g., sum, product, concatenation) interacts in a computational graph, the model can be described as a collection of modules that interact with each other [START_REF] Ghazi | Recursive Sketches for Modular Deep Learning[END_REF]. The standardization of such neural network building blocks (fully-connected layers, convolutional layers) also enabled the development of highly optimized hardware and software ecosystems for fast computation [START_REF]Accelerate Fast Math with Intel® oneAPI Math Kernel Library[END_REF][START_REF] Gavali | Chapter 6 -deep convolutional neural network for image classification on CUDA platform[END_REF][START_REF] Gray | GPU Kernels for Block-Sparse Weights[END_REF][START_REF] Abadi | TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems[END_REF][START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF].

Together with sub-task decomposition (task modularity), model modularity offers ease of expert knowledge integration [START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Belay | Factored Convolutional Neural Network for Amharic Character Image Recognition[END_REF][START_REF] Goodfellow | Generative Adversarial Networks[END_REF][START_REF] Shin | Continual Learning with Deep Generative Replay[END_REF][START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF] and interpretability [START_REF] Jacobs | Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks[END_REF][START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF][START_REF] Krishnamurthy | Interpretability in gated modular neural networks[END_REF][START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF]. Interpretability can have different forms. For example, each neural network module could be assigned a specific interpretable sub-task. On the other hand, selective module evaluation provides insights on how different samples/tasks are related [START_REF] Alet | Modular meta-learning[END_REF][START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Jacobs | Adaptive Mixtures of Local Experts[END_REF][START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF] in the context of conditional computation [START_REF] Bengio | Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation[END_REF].

The model decomposition into modules promotes reusability and knowledge transfer [START_REF] Braylan | Reuse of neural modules for general video game playing[END_REF]. Though each neural network is typically trained to perform a specific task, its (decomposed) modules could be shared across tasks if appropriate mechanisms promote such reusability. The simplest example would be the classical fine-tuning paradigm of large pretrained models [START_REF] Chu | Best practices for fine-tuning visual classifiers to new domains[END_REF][START_REF] He | Rethinking imagenet pre-training[END_REF][START_REF] Hinton | Reducing the Dimensionality of Data with Neural Networks[END_REF][START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF]. This paradigm typically freezes the pretrained model and only retrains its last classification layer to adapt it to the downstream task. Pretrained models are typically pretrained on large datasets [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF][START_REF] Sun | Revisiting unreasonable effectiveness of data in deep learning era[END_REF][START_REF] Yalniz | Billion-scale semi-supervised learning for image classification[END_REF]. The large amount and diversity of training data make pretrained models' intermediate features reusable for other downstream tasks. More recently, the finer-grained reusability of neural network systems has attracted the attention of researchers. Such methods assume that the tasks share underlying patterns and keep an inventory of reusable modules (each module is a small neural network) [START_REF] Alet | Modular meta-learning[END_REF][START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF][START_REF] Veniat | Efficient Continual Learning with Modular Networks and Task-Driven Priors[END_REF]. Each module learns different facets (latent factors or atomic skills) of the knowledge required to solve each task. The selective/sparse use and dynamic reassembling/recombination of these modules can promote sample efficiency [START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF] and combinatorial generalization [START_REF] Alet | Modular meta-learning[END_REF][START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Vanessa | How Modular should Neural Module Networks Be for Systematic Generalization?[END_REF][START_REF] Islam | Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learning[END_REF].

Combinatorial generalization is also known as compositional generalization, "infinite use of finite means" [START_REF] Chomsky | Aspects of the Theory of Syntax[END_REF], and systematic generalization. It aims to generalize to unseen compositions of known functions/factors/words [START_REF] Chang | Automatically composing representation transformations as a means for generalization[END_REF][START_REF] Csordás | CTL++: Evaluating Generalization on Never-Seen Compositional Patterns of Known Functions, and Compatibility of Neural Representations[END_REF][START_REF] Fodor | Connectionism and cognitive architecture: A critical analysis[END_REF][START_REF] Keysers | Measuring compositional generalization: A comprehensive method on realistic data[END_REF][START_REF] Lake | Generalization without systematicity: On the compositional skills of sequence-tosequence recurrent networks[END_REF][START_REF] Ostapenko | Attention for compositional modularity[END_REF][START_REF] Purushwalkam | Task-driven modular networks for zero-shot compositional learning[END_REF], it is the ability to systematically recombine previously learned elements to map new inputs made up from these elements to their correct output [START_REF] Schmidhuber | Towards compositional learning in dynamic networks[END_REF]. For example, new sentences consist of new compositions of a known set of words. Combinatorial generalization is argued to be important to achieve humanlike generalization [START_REF] Battaglia | Relational inductive biases, deep learning, and graph networks[END_REF][START_REF] Hupkes | Compositionality decomposed: How do neural networks generalise[END_REF][START_REF] Hupkes | State-of-the-art generalisation research in NLP: A taxonomy and review[END_REF][START_REF] Kingetsu | Neural Network Module Decomposition and Recomposition[END_REF][START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF][START_REF] Lecun | A path towards autonomous machine intelligence version 0[END_REF][START_REF] Loula | Rearranging the familiar: Testing compositional generalization in recurrent networks[END_REF][START_REF] Mittal | Is a Modular Architecture Enough?[END_REF][START_REF] Ponti | Inductive Bias and Modular Design for Sample-Efficient Neural Language Learning[END_REF][START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF][START_REF] Ivan | Training neural networks to encode symbols enables combinatorial generalization[END_REF][START_REF] Wang | A Combinatorial Perspective on Transfer Learning[END_REF]. Learning different facets of knowledge with different modules in a reusable way could be one solution to combinatorial generalization. Modular systems have been shown effective for combinatorial generalization [START_REF] Rosenbaum | Routing Networks and the Challenges of Modular and Compositional Computation[END_REF] in various fields e.g., natural language processing [START_REF] Hupkes | Compositionality decomposed: How do neural networks generalise[END_REF][START_REF] Brenden | Compositional generalization through meta sequence-to-sequence learning[END_REF][START_REF] Murty | Characterizing Intrinsic Compositionality in Transformers with Tree Projections[END_REF][START_REF] Ponti | Inductive Bias and Modular Design for Sample-Efficient Neural Language Learning[END_REF][START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF], visual question answering [START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Bahdanau | Systematic generalization: What is required and can it be learned?[END_REF][START_REF] Vanessa | How Modular should Neural Module Networks Be for Systematic Generalization?[END_REF], object recognition [START_REF] Chang | Automatically composing representation transformations as a means for generalization[END_REF][START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF][START_REF] Parascandolo | Learning independent causal mechanisms[END_REF][START_REF] Purushwalkam | Task-driven modular networks for zero-shot compositional learning[END_REF], and robotics [START_REF] Alet | Modular meta-learning[END_REF][START_REF] Clavera | Policy transfer via modularity and reward guiding[END_REF][START_REF] Devin | Learning modular neural network policies for multi-task and multi-robot transfer[END_REF][START_REF] Pathak | Learning to control self-assembling morphologies: A study generalization via modularity[END_REF].

The modularization of neural network systems promotes knowledge retention. If different knowledge is localized into different modules, targeted knowledge updates and troubleshooting [START_REF] Kingetsu | Neural Network Module Decomposition and Recomposition[END_REF][START_REF] Pan | On Decomposing a Deep Neural Network into Modules[END_REF][START_REF] Pan | Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules[END_REF] alleviate gradient interference of different tasks [START_REF] Kanakis | Reparameterizing convolutions for incremental multi-task learning without task interference[END_REF][START_REF] Maninis | Attentive Single-Tasking of Multiple Tasks[END_REF][START_REF] Yu | Gradient Surgery for Multi-Task Learning[END_REF] and catastrophic forgetting [START_REF] Wickliffe | Memory retention -the synaptic stability versus plasticity dilemma[END_REF][START_REF] Alet | Modular meta-learning[END_REF][START_REF] Anderson | Beyond Fine Tuning: A Modular Approach to Learning on Small Data[END_REF][START_REF] Olav Ellefsen | Neural Modularity Helps Organisms Evolve to Learn New Skills without Forgetting Old Skills[END_REF][START_REF] French | Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionist Networks[END_REF][START_REF] Javed | Meta-Learning Representations for Continual Learning[END_REF][START_REF] Ke | Achieving forgetting prevention and knowledge transfer in continual learning[END_REF][START_REF] Ostapenko | Continual learning via local module composition[END_REF][START_REF] Rusu | Progressive Neural Networks[END_REF][START_REF] Terekhov | Knowledge Transfer in Deep Block-Modular Neural Networks[END_REF][START_REF] Veniat | Efficient Continual Learning with Modular Networks and Task-Driven Priors[END_REF].

Modular neural network systems facilitate model scaling in two ways. (1) Modular models like fully-connected models and ResNet can be scaled up (or down) by simply stacking more (or less) modules to increase (or decrease) the model capacity to fit larger (or smaller) datasets [START_REF] He | Deep residual learning for image recognition[END_REF]. (2) Modular methods based on sparsely activated Mixture-of-Experts [START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF] decouple computation cost from model size. They allow drastically increasing the model capacity without increasing compute cost because only a small fraction of the model is evaluated on each forward pass [START_REF] Barham | Pathways: Asynchronous Distributed Dataflow for ML[END_REF][START_REF] Chowdhery | PaLM: Scaling Language Modeling with Pathways[END_REF][START_REF] Du | Glam: Efficient scaling of language models with mixture-of-experts[END_REF][START_REF] Fedus | A Review of Sparse Expert Models in Deep Learning[END_REF][START_REF] He | FasterMoE: Modeling and optimizing training of large-scale dynamic pre-trained models[END_REF][START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF]. The extreme example of these sparsely activated models is Switch Transformer [START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF] which contains 1.6 trillion parameters, pushing the competition of large model sizes [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Smith | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model[END_REF] to the next level.

Typical modules in deep learning models

This section reviews some typical modules in the deep learning literature.

Almost all systems are modular to some degree [START_REF] Schilling | Toward a General Modular Systems Theory and Its Application to Interfirm Product Modularity[END_REF], neural network systems can almost always be decomposed into subsystems (modules) [START_REF] Balestriero | POLICE: Provably Optimal Linear Constraint Enforcement for Deep Neural Networks[END_REF] following different points of view. More specifically, they usually consist of a hierarchical structure in which a module of a higher hierarchy level is made of modules of a lower hierarchy level. The elementary layer of modern neural networks (e.g., fully-connected layer, convolutional layer) can be seen as a module on its own. On the other hand, any neural network as a whole can also be considered as a module e.g., in the context of ensemble [START_REF] Zhou | Ensemble Methods: Foundations and Algorithms[END_REF], Mixture-of-Experts [START_REF] Jacobs | Adaptive Mixtures of Local Experts[END_REF], and Generative Adversarial Networks (GAN) [START_REF] Goodfellow | Generative Adversarial Networks[END_REF]. Some literature [START_REF] Béna | Extreme sparsity gives rise to functional specialization[END_REF][START_REF] Csordás | Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks[END_REF][START_REF] Kingetsu | Neural Network Module Decomposition and Recomposition[END_REF][START_REF] Sun | Task Switching Network for Multi-task Learning[END_REF] define modules as sub-neural networks where part of the parameters are masked out (set to 0). In these cases, overlapping modules can be obtained when the masks overlap.

Modules for non-sequential data

Fully-connected layers (Figure 5 (a)) imitate the connections between neurons in biological neural networks but connect every input neuron to every output neuron [START_REF] Goodfellow | Deep Learning[END_REF]. In practice, a fully-connected layer is implemented as a matrix multiplication between input data and learnable parameters. Convolutional layers introduce the inductive bias of translation equivariance. Conceptually, a convolutional layer (with a single output channel) can be obtained from a fully-connected layer by enforcing local connectivity and parameter sharing [START_REF] Goodfellow | Deep Learning[END_REF]. Local connectivity means that each neuron only connects to a subset of neurons of the previous layer; parameter sharing means that the same learnable parameters are used across receptive fields. In practice, a convolutional layer is implemented as a collection of kernels/filters shifted over the input data [START_REF] Abadi | TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems[END_REF][START_REF] Paszke | PyTorch: An Imperative Style, High-Performance Deep Learning Library[END_REF]. Each kernel performs a dot product between input data and learnable parameters. Depending on the number of dimensions over which kernels are shifted, a convolutional layer is termed e.g., 1D, 2D, 3D. 2D convolutional layers are widely used in computer vision tasks [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. Locally connected layers are similar to convolutional layers except that they remove the constraint of parameter sharing (across kernels). It helps if one wants to impose local receptive fields while there is no reason to think each local kernel should be the same [START_REF] Goodfellow | Deep Learning[END_REF]. Low-rank locally connected layers relax spatial equivariance and provide a tradeoff between locally connected layers and convolutional layers. The kernel applied at each position is constructed as a linear combination of a basis set of kernels with spatially varying combining weights. Varying the number of basis kernels allows controlling the degree of relaxation of spatial equivariance [START_REF] Gamaleldin | Revisiting Spatial Invariance with Low-Rank Local Connectivity[END_REF] reflection), referred to as group convolutional layers [START_REF] Bekkers | Roto-Translation Covariant Convolutional Networks for Medical Image Analysis[END_REF][START_REF] Taco | Group Equivariant Convolutional Networks[END_REF][START_REF] Taco | Steerable CNNs[END_REF][START_REF] Sander Dieleman | Exploiting Cyclic Symmetry in Convolutional Neural Networks[END_REF][START_REF] Gao | Efficient and Invariant Convolutional Neural Networks for Dense Prediction[END_REF][START_REF] Weiler | General $E(2)$-Equivariant Steerable CNNs[END_REF][START_REF] Weiler | Learning Steerable Filters for Rotation Equivariant CNNs[END_REF][START_REF] Daniel | Harmonic Networks: Deep Translation and Rotation Equivariance[END_REF]. On the other hand, depthwise separable convolutional layers [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF][START_REF] Andrew G Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF][START_REF] Sifre | Rigid-Motion Scattering for Image Classification[END_REF] factorize a standard convolutional layer into a depthwise convolutional layer and a pointwise convolutional layer, which reduces model size and computation.

Multiple layers can be grouped into a building block (a module of a higher hierarchy level). Such examples include the building blocks of ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], Inception [START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF][START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], ResNeXt [START_REF] Xie | Aggregated Residual Transformations for Deep Neural Networks[END_REF], Wide ResNet [START_REF] Zagoruyko | Wide residual networks[END_REF]. Inception [START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF][START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] has parallel kernels of multiple sizes within each block and merge their results to extract information at varying scales. Inception also includes several techniques to reduce computation cost e.g., factorizing large kernels into smaller kernels and using 1 × 1 convolution to reduce dimensionality. A ResNet block [START_REF] He | Deep residual learning for image recognition[END_REF] (Figure 5 (b)) contains a sequence of convolutional layers; it adds a skip-connection (also known as residual connection, identity mapping) from the beginning to the end of the block to alleviate vanishing gradients. Many variants of the ResNet block have been proposed. For example, Wide ResNet [START_REF] Zagoruyko | Wide residual networks[END_REF] increases the block width; ResNeXt [START_REF] Xie | Aggregated Residual Transformations for Deep Neural Networks[END_REF] aggregates parallel paths within each block.

The block design could be automatically searched instead of handcrafted. In order to narrow down the model search space, some Neural Architecture Search methods [START_REF] Elsken | Neural Architecture Search[END_REF][START_REF]Automatic Machine Learning: Methods, Systems, Challenges[END_REF][START_REF] Ying | Nas-bench-101: Towards reproducible neural architecture search[END_REF][START_REF] Zoph | Learning Transferable Architectures for Scalable Image Recognition[END_REF] automatically search the optimal design pattern for a block (also known as a cell) while fixing the block composition scheme (also known as metaarchitecture). Once the block design patterns are searched, the full model is instantiated by repeating the searched blocks following the predefined block composition scheme. For example, NAS-Bench-101 [START_REF] Ying | Nas-bench-101: Towards reproducible neural architecture search[END_REF] defines the block search space as all possible directed acyclic graphs on V nodes (V 7) while limiting the maximum number of edges to 9.

McNeely-White et al. [START_REF] Mcneely-White | Inception and ResNet features are (almost) equivalent[END_REF] report that the features learned by Inception and ResNet are almost linear transformations of each other, even though these two architectures have a remarkable difference in the architectural design philosophy. This result explains why the two architectures usually perform similarly and highlights the importance of training data. This result is corroborated by Bouchacourt et al. [START_REF] Bouchacourt | Grounding inductive biases in natural images: Invariance stems from variations in data[END_REF], who argue that invariance generally stems from the data itself rather than from architectural bias.

Modules for sequential data

When the input data is sequential e.g., time series, text, audio, video, Recurrent Neural Networks (RNN) [START_REF] David E Rumelhart | Learning internal representations by error propagation[END_REF] come into play. The RNN module processes the sequential data one at a time; the output (also known as the hidden state) of the RNN module at the previous time step is recursively fed back to the RNN module, which allows it to aggregate information across different time steps. The vanilla RNN module suffers from short-term memory issues; it cannot effectively preserve information over long sequences. To overcome this issue, gated recurrent unit (GRU) [START_REF] Cho | On the properties of neural machine translation: Encoder-Decoder approaches[END_REF] and long short-term memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF] module use gates to control which information should be stored or forgotten in the memory, which allows better preservation of long-term information. In GRU and LSTM modules, gates are neural networks with trainable parameters. While GRU modules are faster to train than LSTM modules, their performance comparison varies depending on the scenario. GRU surpasses LSTM in long text and small dataset scenarios while LSTM outperforms GRU in other scenarios [START_REF] Yang | LSTM and GRU neural network performance comparison study: Taking yelp review dataset as an example[END_REF].

Contrary to RNN, GRU, and LSTM, which process sequential data one at a time, self-attention layers [START_REF] Vaswani | Attention is all you need[END_REF] process the data sequence in parallel. For each data point in a data sequence (e.g., each time step of a time series), a self-attention layer creates three transformed versions, referred to as query vector, key vector, and value vector, through linear transformations. Between each pair of data points, the dot product between the query vector and the key vector of the pair reflects how much those two data points are related within the sequence. These dot products are then normalized and combined with the corresponding value vectors to get the new representation of each data point in the sequence. An enhanced version of self-attention layers is multi-head self-attention layers, which extract different versions of query vector, key vector, and value vector for each data point. Multi-head self-attention layers improve performance by capturing more diverse representations. A transformer block combines multi-head selfattention layers, fully-connected layers, normalization layers, and skip-connections. Models built upon transformer blocks have achieved state-of-the-art performance in a wide range of tasks such as natural language processing [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] and speech synthesis [START_REF] Li | Neural speech synthesis with transformer network[END_REF]. Transformer models can be applied to image modality by transforming each input image into a sequence of small image patches [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF]. Despite the lack of image-specific inductive bias (translation equivariance, locality), vision transformers can achieve state-of-the-art performance when combined with a large amount of training data [START_REF] Bao | BEiT: BERT Pre-Training of Image Transformers[END_REF][START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF][START_REF] He | Masked autoencoders are scalable vision learners[END_REF].

Composition of modules

Section 4.2 presents typical modules in the literature. Section 4.3 discusses how to organize these modules to form a model (or a module of a higher hierarchy level).

Static composition of modules

Static composition means that the composed structure does not vary with input; the same structure is used for all input samples or tasks.

One straightforward way to compose modules is sequential concatenation (Figure 6 (a)). It implies that multiple (typically homogeneous) modules are sequentially concatenated into a chain to form a model, where a module's output is the next module's input. Examples of sequential concatenation include fully-connected models [START_REF] Goodfellow | Deep Learning[END_REF] and ResNet models [START_REF] He | Deep residual learning for image recognition[END_REF]. This composition scheme typically does not assume an explicit sub-task decomposition; the chain of concatenated modules can instead be seen as a series of information extraction steps [START_REF] Alain | Understanding intermediate layers using linear classifier probes[END_REF][START_REF] Tishby | Deep learning and the information bottleneck principle[END_REF][START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF], extracted features transition from low-level to high-level.

Ensembling composition [START_REF] Ju | The relative performance of ensemble methods with deep convolutional neural networks for image classification[END_REF][START_REF] Opitz | Efficient model averaging for deep neural networks[END_REF][START_REF] Zhou | Ensemble Methods: Foundations and Algorithms[END_REF], on the other hand, organizes modules in a parallel manner (Figure 6 (b)). The principle of ensembling is to aggregate (e.g., averaging) the results of multiple modules (weaker learners) to obtain a more robust prediction. The rationale is that different modules are expected to provide complementary and diverse views of input data. Each module's data is processed independently without relying on the other modules at inference time. The regularization method Dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF], which randomly deactivates neurons during training, can be seen as an implicit ensemble method of overlapping modules.

Sequential composition and parallel composition can be combined, e.g., in the form of a tree structure (Figure 6 (c)). A typical scenario of tree-structure composition is a model with a shared feature extractor and multiple task-specific heads [START_REF] Silver | Mastering the game of Go without human knowledge[END_REF][START_REF] Zhang | A survey on multi-task learning[END_REF]. All the above composition schemes are special cases of DAG (Directed Acyclic Graph, Figure 6 (d)). The general DAG composition scheme is typically found in models searched by Neural Architecture Search [START_REF] Liu | Hierarchical representations for efficient architecture search[END_REF][START_REF] Reisinger | Evolving Reusable Neural Modules[END_REF][START_REF] Xie | Exploring Randomly Wired Neural Networks for Image Recognition[END_REF].

Cooperation composition (Figure 6 (f)) assumes that each module is a standalone neural network with specific functionality and that these neural networks cooperate during training or inference; it is a neural network system that consists of multiple separate neural networks. Different from ensembling composition, modules in cooperation composition are typically heterogeneous and interact with each other more diversely. For example, siamese networks [START_REF] Bromley | Signature Verification using a "Siamese" Time Delay Neural Network[END_REF][START_REF] Chen | Exploring Simple Siamese Representation Learning[END_REF][START_REF] Jing | Masked Siamese ConvNets[END_REF] consists of two neural networks (module) which work together to produce different versions of the input data. Generative Adversarial Networks (GAN) [START_REF] Goodfellow | Generative Adversarial Networks[END_REF][START_REF] Zhu | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[END_REF] inator. The same spirit applies to teacher and student neural networks [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF]. Some deep reinforcement learning methods implement the Actor-Critic [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] with two separate new networks, such as AlphaGo [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF], A3C [START_REF] Mnih | Asynchronous Methods for Deep Reinforcement Learning[END_REF], ACKTR [START_REF] Wu | Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation[END_REF]. Continual learning with deep replay buffer [START_REF] Shin | Continual Learning with Deep Generative Replay[END_REF] consists of a continual neural network learner and a generative neural network serving as the replay buffer. Some other continual learning methods [START_REF] Anderson | Beyond Fine Tuning: A Modular Approach to Learning on Small Data[END_REF][START_REF] Rusu | Progressive Neural Networks[END_REF][START_REF] Terekhov | Knowledge Transfer in Deep Block-Modular Neural Networks[END_REF][START_REF] Veniat | Efficient Continual Learning with Modular Networks and Task-Driven Priors[END_REF] continuously expanding model capacity for new tasks by adding new modules which work in cooperation with old modules.

Conditional composition of modules

Conditional composition (Figure 6 (e)) is complementary to static composition in the sense that the composed modules are selectively (conditionally, sparsely, or dynamically) activated (used or evaluated) for each particular input. The input conditioning can happen at the granularity of individual sample [START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Jacobs | Adaptive Mixtures of Local Experts[END_REF][START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF] as well as task [START_REF] Jacobs | Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks[END_REF][START_REF] Maninis | Attentive Single-Tasking of Multiple Tasks[END_REF][START_REF] Masse | Alleviating catastrophic forgetting using contextdependent gating and synaptic stabilization[END_REF][START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF][START_REF] Sun | Task Switching Network for Multi-task Learning[END_REF]. In the literature, this paradigm is also termed conditional computation [START_REF] Bengio | Conditional Computation in Neural Networks for faster models[END_REF][START_REF] Bengio | Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation[END_REF].

The idea of conditional computation can be traced back to Mixture-of-Experts (MoE) introduced in the last century. An MoE is a system composed of multiple separate neural networks (modules), each of which learns to handle a sub-task of the overall task [START_REF] Jacobs | Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks[END_REF][START_REF] Zhou | Diverse ensemble evolution: Curriculum data-model marriage[END_REF] e.g., a subset of the complete training dataset. A gating network computes the probability of assigning each example to each module [START_REF] Jacobs | Adaptive Mixtures of Local Experts[END_REF][START_REF] Michael | Hierarchical mixtures of experts and the EM algorithm[END_REF] or a sparse weighted combination of modules [START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF]. Two issues of MoE are module collapse [START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF][START_REF] Mittal | Is a Modular Architecture Enough?[END_REF][START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF] and shrinking batch size [START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF], both of which are related to the balance of module utilization. Module collapse means under-utilization of modules or lack of module diversity. Due to the self-reinforcing behavior of the gating network during training, premature modules may be selected and thus trained even more. The gating network may end up converging to always selecting a small subset of modules while the other modules are never used. Shrinking batch size means the batch size is reduced for each conditionally activated module. Large batch sizes are necessary for modern hardware to make efficient inferences because they alleviate the cost of data transfers [START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF].

MoE can be generalized to e.g., stacked MoE [START_REF] Eigen | Learning factored representations in a deep mixture of experts[END_REF][START_REF] Fernando | PathNet: Evolution Channels Gradient Descent in Super Neural Networks[END_REF][START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF][START_REF] Ramachandran | Diversity and depth in per-example routing models[END_REF][START_REF] Rosenbaum | Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning[END_REF] or hierarchical MoE [START_REF] Shazeer | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer[END_REF][START_REF] Yao | Hierarchical mixture of classification experts uncovers interactions between brain regions[END_REF] (Figure 7). Eigen et al. [START_REF] Eigen | Learning factored representations in a deep mixture of experts[END_REF] first explored stacked MoE; they introduced the idea of using multiple MoE with their own gating networks. In order to train stacked MoE, Kirsch et al. [START_REF] Kirsch | Modular Networks: Learning to Decompose Neural Computation[END_REF] use generalized Viterbi Expectation-Maximization algorithm, Rosenbaum et al. [START_REF] Rosenbaum | Routing Networks: Adaptive Selection of Non-Linear Functions for Multi-Task Learning[END_REF] employ a multi-agent reinforcement learning algorithm, Fernando et al. [START_REF] Fernando | PathNet: Evolution Channels Gradient Descent in Super Neural Networks[END_REF] use a genetic algorithm. MoE systems do not always have explicit gating networks; for instance, Fernando et al. [START_REF] Fernando | PathNet: Evolution Channels Gradient Descent in Super Neural Networks[END_REF] rely on the results of the genetic algorithm to decide the module routing scheme. be conditionally reassembled for each input. This approach has been advocated to promote knowledge transfer, sample efficiency, and generalization. For example, in visual question answering, Neural Module Networks [START_REF] Andreas | Neural Module Networks[END_REF][START_REF] Vanessa | How Modular should Neural Module Networks Be for Systematic Generalization?[END_REF][START_REF] Hu | Learning to Reason: End-to-End Module Networks for Visual Question Answering[END_REF] dynamically reassemble modules into a neural network to locate the attention (region of interest) on the questioned image. The question's parsing guides the reassembling process so that the reassembled model reflects the structure and semantics of the question. For this particular task, the compositionality of modules comes from the compositionality of visual attention. Following the question's syntax, the reassembled modules sequentially modify the attention onto the questioned image. For example, the module associated with the word "cat" locates the image region containing a cat, and the module associated with the word "above" shifts up the attention. Zhang et al. [START_REF] Zhang | Network Transplanting[END_REF] investigated adding new abilities to a generic network by directly transplanting the module corresponding to the new ability, dubbed network transplanting.

Some work relies on the hypothesis that the tasks at hand share some commonalities i.e., hidden factors are shared across tasks. Each hidden factor can be learned by a separate module from the module inventory for transfer learning and meta-learning. For example, Alet et al. [START_REF] Alet | Modular meta-learning[END_REF] use simulated annealing to meta-learn an inventory of modules reusable across tasks to achieve combinatorial generalization. The parameters of an inventory of modules are optimized during meta-training; the trained modules are reassembled during the meta-test with an optional parameter fine-tuning process. They demonstrated the utility of their method for robotics tasks. Ponti et al. [START_REF] Edoardo | Combining Modular Skills in Multitask Learning[END_REF] assume that each task is associated with a subset of latent discrete skills from a skill inventory. They try to generalize more systematically to new tasks by disentangling and recombining different facets of knowledge. More precisely, they jointly learn a skill-specific parameter vector for each latent skill and a binary task-skill allocation matrix. For each new task, the new model's parameter vector is created as the average of the skill-specific parameter vectors corresponding to the skills present in the new task (in addition to a shared base parameter vector).

The conditional composition scheme also has other forms. For example, Teerapittayanon et al. [START_REF] Surat Teerapittayanon | BranchyNet: Fast inference via early exiting from deep neural networks[END_REF] save computation on easy input data via early exiting; later layers will be skipped if the intermediate feature's prediction confidence passes a predefined threshold. Fuengfusin et al. [START_REF] Fuengfusin | Network with Sub-networks: Layer-wise Detachable Neural Network[END_REF] train models whose layers can be removed at inference time without significantly reducing the performance to allow adaptive accuracy-latency trade-off. Similarly, Yu et al. [START_REF] Yu | Slimmable Neural Networks[END_REF] train models which are executable at customizable widths (the number of channels in a convolutional layer). Xiong et al. [START_REF] Xiong | Conditional convolutional neural network for modality-aware face recognition[END_REF] sparsely activate convolutional kernels within each layer for each particular input sample, which provides an example of the conditional composition of overlapping modules.

Conclusion of model modularity

Section 4 presents how the notion of modularity is instantiated in the architecture of neural network systems. The structure of neural network modules (Section 4.2) and the way to organize the modules (Section 4.3) provide a complementary view of model modularity.

While all modern neural networks are modular to some extent, different instantiations of the modularity principle offer different advantages (Section 4.1). The advantages include ease of conceptual design and implementation, ease of expert knowledge integration, better interpretability, ease of knowledge transfer and reuse, better generalization and sample efficiency, ease of knowledge retention, ease of troubleshooting, and better scalability.

Other notions of modularity

There remain some other notions of modularity in the deep learning literature.

In graph theory, the term "modularity" refers to a measure commonly used in community detection. It measures the density of connections within a community (module) compared to between modules communities [START_REF] Newman | Modularity and community structure in networks[END_REF]. This measure can be applied to graph clustering problems in the form of modularity optimization [START_REF] Brandes | On modularity clustering[END_REF][START_REF] Hu | Deep Stock Representation Learning: From Candlestick Charts to Investment Decisions[END_REF][START_REF] Salha-Galvan | Modularity-Aware Graph Autoencoders for Joint Community Detection and Link Prediction[END_REF][START_REF] Shiokawa | Fast algorithm for modularity-based graph clustering[END_REF]. Inspired by this measure, Filan et al. [START_REF] Daniel Filan | Clusterability in Neural Networks[END_REF] investigate the parameter clustering pattern that emerged from the training of a neural network. They view a neural network as an undirected weighted graph (edge weights are the absolute value of network parameters) and apply spectral clustering on the obtained graph. They observe that some neural networks trained on image classification tasks have some clustering properties of their parameters: edge weights are stronger within one cluster than between clusters. Watanabe et al. [START_REF] Watanabe | Modular representation of layered neural networks[END_REF] have obtained similar results. Béna et al. [START_REF] Béna | Extreme sparsity gives rise to functional specialization[END_REF] adapted the graph-theoretic modularity measure to define structural modularity and define functional specialization through three heuristic measures. The functional specialization can be intuitively understood as the extent to which a sub-network can do a sub-task independently. To investigate the relationship between structural and functional modularity, they design a scenario where a model with two parallel modules (with an adjustable number of interconnections) is used to predict whether the parity of the two digits is the same or different. They show that enforcing structural modularity via sparse connectivity between two communicating modules does lead to functional specialization of the modules. However, this phenomenon only happens at extreme levels of sparsity. With even a moderate number of interconnections, the modules become functionally entangled. Mittal et al. [START_REF] Mittal | Is a Modular Architecture Enough?[END_REF] observed that modular systems (weighted combination of parallel modules) with a good module specialization are good in terms of the overall system performance, however end-to-end training itself is not enough to achieve a good module specialization. The term "modularity" is related to the notion of independence in some literature. For example, Galanti et al. [START_REF] Galanti | On the Modularity of Hypernetworks[END_REF] use modularity to refer to the ability of hypernetworks [START_REF] Ha | [END_REF] to learn a different function for each input instance. A line of research has been carried out on learning disentangled representation. Intuitively, disentangled representation aims to reverse the underlying data generating process and retrieve its latent factors into the learned representation (Figure 8). One of the desirable properties of a disentangled representation [START_REF] Eastwood | A Framework for the Quantitative Evaluation of Disentangled Representations[END_REF][START_REF] Ridgeway | Learning Deep Disentangled Embeddings With the F-Statistic Loss[END_REF][START_REF] Zaidi | Measuring Disentanglement: A Review of Metrics[END_REF] is "modularity". In this context, a modular representation is a representation where each dimension of the representation conveys information about at most one latent generative factor.

Discussion

Defining modularity is, in itself, a challenging problem. The notion of modularity is present in literature across many different fields [START_REF] Avigad | Modularity in mathematics[END_REF][START_REF] Carliss | Design Rules: The Power of Modularity[END_REF][START_REF] Barrett | Modularity in cognition: Framing the debate[END_REF][START_REF] Bongard | Evolving modular genetic regulatory networks[END_REF][START_REF] Bourbaki | The Architecture of Mathematics[END_REF][START_REF] Cohen-Boulakia | Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities[END_REF][START_REF] Cosmides | Cognitive Adaptations for Social Exchange[END_REF][START_REF] Cosmides | Origins of domain specificity: The evolution of functional organization[END_REF][START_REF] Fodor | The Modularity of Mind[END_REF][START_REF] Fodor | The Mind Doesn't Work That Way: The Scope and Limits of Computational Psychology[END_REF][START_REF] Ford | Architects of Intelligence: The Truth about AI from the People Building It[END_REF][START_REF] Frankenhuis | Evolutionary Psychology Versus Fodor: Arguments For and Against the Massive Modularity Hypothesis[END_REF][START_REF] Gentile | Theory of Modularity, a Hypothesis[END_REF][START_REF] Daniel Gómez | A new modularity measure for Fuzzy Community detection problems based on overlap and grouping functions[END_REF][START_REF] Kurzweil | How to Create a Mind: The Secret of Human Thought Revealed[END_REF][START_REF] Modrak | Development of the Modularity Measure for Assembly Process Structures[END_REF][START_REF] Muff | Local modularity measure for network clusterizations[END_REF][START_REF] Newman | Modularity and community structure in networks[END_REF][START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF][START_REF] Jose B Pereira-Leal | The origins and evolution of functional modules: Lessons from protein complexes[END_REF][START_REF] Poisot | An a posteriori measure of network modularity[END_REF][START_REF] Pylyshyn | Is vision continuous with cognition?: The case for cognitive impenetrability of visual perception[END_REF][START_REF] Robbins | Modularity of Mind[END_REF][START_REF] Shao | Modularity measures: Concepts, computation, and applications to manufacturing systems[END_REF][START_REF] Simon | The Architecture of Complexity[END_REF][START_REF] Simon | Aggregation of variables in dynamic systems[END_REF][START_REF] Wagner | Perspective: Complex Adaptations and the Evolution of Evolvability[END_REF]. While many researchers have a strong intuition about what it means for an entity to be modular, there has yet to be a universal agreement on what defines modularity. The same is true even within the field of deep learning. As rightly said by Béna et al. [START_REF] Béna | Extreme sparsity gives rise to functional specialization[END_REF]: "Modularity of neural networks is a bit like the notion of beauty in art: everyone agrees that it's important, but nobody can say exactly what it means". We argue that the difficulty of defining modularity stems from the fact that the notion of modularity usually comes with many different properties: replaceability of modules, combinability of modules, reusability of modules, autonomy of modules (limited interaction or limited interdependence between modules), functional specialization of modules. Authors from different fields typically only retain one or two of the above properties to claim an entity to be modular.

In this survey, we define modularity as the property of an entity whereby it can be broken down into a number of sub-entities (referred to as modules). This definition is the prerequisite of the properties mentioned above; it is the greatest common definition of the notion of modularity. By recursively applying the definition of modularity, a modular entity is an entity that can be broken down into sub-entities, where each sub-entity can be further broken down into sub-sub-entities. This recursion can be repeated for discrete entities until the atomic elements (minimum indivisible modules) are reached. In that case, a set of atomic elements {a ∈ D} can formally characterize a discrete entity D; a subset of atomic elements can then characterize a module M ⊆ D. The above framework applies to data modularity (Section 2) and model modularity (Section 4). The reason is that data and models are both discrete: data samples and model parameters are stored in physical computers where everything is represented quantitatively. On the other hand, we need to use a different framework for task modularity because tasks are usually not discrete. As discussed in Section 3, each task can be characterized by an objective function F . In this sense, task modularity can be formally characterized by (objective) function compositions. A task is decomposable if there exists a set of functions {f 1 , f 2 , ...} that, when composed together, retrieve the form of the original objective function F .

For discrete entities, one needs to choose the atomic elements. Naively, one could choose each data sample in a dataset and each neuron in a neural network as the atomic elements. However, both choices remain to be discussed because they are indeed not the smallest indivisible modules. Regarding data modularity, the dataset division can happen both at the sample dimension and the feature dimension, which means that each data sample can be divided into smaller elements e.g., feature vectors of reduced length or image patches. Regarding model modularity, the modularization can happen at the granularity of parameters e.g., modules can be obtained by masking out parameters [START_REF] Béna | Extreme sparsity gives rise to functional specialization[END_REF][START_REF] Csordás | Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks[END_REF][START_REF] Kingetsu | Neural Network Module Decomposition and Recomposition[END_REF][START_REF] Sun | Task Switching Network for Multi-task Learning[END_REF]. Consequently, one can choose the scalar numbers stored in physical computers (often represented by floating-point numbers) as the atomic elements. The atomic elements for data are every single dimension of data samples; the atomic elements for models are every single scalar parameters in the neural network. It entails that, in some cases, there needs to be some relationship R among atomic elements {a ∈ D} because any arbitrary subsets of atomic elements do not necessarily form a valid module if the relationship R is broken. In the above example, the relationship R indicates which scalar numbers should come together to form data samples or how to use each scalar parameter along the feedforward computation in the computational graph of neural networks. In consequence, an entity can be a set or a system; a system is a set equipped with relationships R among atomic elements {a ∈ D}.

Future research

As modularity is a general principle, this survey covered many elements from different sub-fields of deep learning; each sub-field can provide a lot of future avenues of research on its own. To name a few, McNeely-White et al. [START_REF] Mcneely-White | Inception and ResNet features are (almost) equivalent[END_REF] and Bouchacourt et al. [START_REF] Bouchacourt | Grounding inductive biases in natural images: Invariance stems from variations in data[END_REF] showed that given the same training data, learned features exhibit similar properties across models with markedly different architectural inductive biases. Is it still worth improving neural network architectures if data dominate learning results? Future research may validate the results of McNeely-White et al. [START_REF] Mcneely-White | Inception and ResNet features are (almost) equivalent[END_REF] and Bouchacourt et al. [START_REF] Bouchacourt | Grounding inductive biases in natural images: Invariance stems from variations in data[END_REF] by extending their research to more kinds of models and training datasets in a more systematic way. If these results still hold, one may need to ground these results theoretically. On the other hand, whether neural networks can learn and behave compositionally is still an open question [START_REF] Andreas | Measuring compositionality in representation learning[END_REF][START_REF] Hupkes | Compositionality decomposed: How do neural networks generalise[END_REF]. It entails that we need a domain-agnostic way to test the compositionality of neural networks.

Different aspects of the modularity principle can be further investigated to improve deep learning models. It boils down to designing new deep learning methods that provide e.g., better interpretability, reusability, scalability, and efficiency. While model modularity may, to some extent, reflect task modularity, it is still unclear whether data modularity directly corresponds with model modularity. One research avenue is to automate imposed data modularization regarding specific models in the spirit of AutoML. Similarly, automating task modularization can facilitate problem-solving and reduce human-introduced bias.

Conclusion

Deep learning is becoming dominant in many applications, such as computer vision and natural language processing. It is natural to ask ourselves whether there are guidelines for designing deep learning algorithms. Modularity is one guiding principle that has been put forward in the literature. This survey reveals that modularity is pervasive in three related yet distinct axes of deep learning: data, task, and model architecture. We observed that some modularity concepts come in the form of a prior, while others come in the form of a posterior.

The efforts of bringing the modularity principle into deep learning are not new; however, reviewing deep learning literature using the point of view of modularity is relatively new. This survey provides a step towards clarifying and investigating the notion of modularity in deep learning and elsewhere.

  Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

Figure 1 :

 1 Figure 1: Publication trend of "modular deep learning" from 1990 to 2021.The ratio of the count of publications containing "modular deep learning" and "modular neural network" among publications containing "deep learning" and "neural network", indexed by Google Scholar. The horizontal axis is the publication year.

Figure 3 :

 3 Figure 3: Illustration of modularity in data. (a) intrinsic data modularity based on super-classes, images, and class hierarchy in ImageNet [63]; (b) intrinsic data modularity based on styles characterized by a set of metadata, the upper-left circle contains black-on-white characters, the upper-right circle contains white-on-black characters, the lower circle contains characters with natural foreground and background, all characters are drawn from the same set of classes (small-case Latin characters), these three circles illustrate the division of a character dataset based on its metadata; (c) intrinsic manifolds in the form of a moon dataset, where each data manifold can be considered as a module; (d) few-shot learning episodes, reprinted from [191]. (a), (b) and (c) are examples of intrinsic data modularity, (d) is an example of imposed data modularity.

Figure 4 :

 4 Figure 4: Illustration of sub-task decomposition. The upper figure illustrates the parallel decomposition of a task. The lower figure illustrates the sequential decomposition of a task.

Figure 5 :

 5 Figure 5: Examples of a module. (a) a fully-connected layer; (b) a basic ResNet module, reprinted from [105]; (c) an LSTM module, reprinted from [44].

  . Standard convolutional layers offer translation equivariance; a line of research focuses on generalizing this to other equivariances (rotation, Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

  Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

Figure 6 :

 6 Figure 6: Illustration of module composition. (a) Sequential concatenation. (b) Ensembling. (c) Tree-structure composition. (d) General Directed Acyclic Graph. (e) Conditional composition. (f) Cooperation composition.

  trains a generator under the guidance of a discrim-Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

Figure 7 :

 7 Figure 7: Extension of Mixture-of-Experts (MoE). (a) A stacked MoE, which stacks multiple MoE layers into a chain. (b) A hierarchical MoE, where a primary gating network chooses a sparse weighted combination of "modules", each of which is an MoE with its own gating network.

  Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

Figure 8 :

 8 Figure 8: Illustration of a disentangled representation.

  Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

  Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

  use parameter masks to Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

  Task modularity can help integrate expert knowledge and promote model interpretability when paired Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

  will be possible. This can Draft of the paper Sun, Haozhe, and Isabelle Guyon. 2023. "Modularity in Deep Learning: A Survey." Pp. 561-95 in Intelligent computing, edited by K. Arai. Cham: Springer Nature Switzerland.

Acknowledgments

We gratefully acknowledge constructive feedback and suggestions from Birhanu Hailu Belay, Romain Egele, Felix Mohr, Hedi Tabia, and the reviewers. This work was supported by ChaLearn and the ANR (Agence Nationale de la Recherche, National Agency for Research) under AI chair of excellence HUMANIA, grant number ANR-19-CHIA-0022.