Julien Pourcel

Ngoc-Son Vu

Robert M French

Online Task-free Continual Learning with Dynamic Sparse Distributed Memory

Keywords:

This paper addresses the very challenging problem of online task-free continual learning in which a sequence of new tasks is learned from non-stationary data using each sample only once for training and without knowledge of task boundaries. We propose in this paper an efficient semi-distributed associative memory algorithm called Dynamic Sparse Distributed Memory (DSDM) where learning and evaluating can be carried out at any point of time. DSDM evolves dynamically and continually modeling the distribution of any non-stationary data stream. DSDM relies on locally distributed, but only partially overlapping clusters of representations to effectively eliminate catastrophic forgetting, while at the same time, maintaining the generalization capacities of distributed networks. In addition, a local density-based pruning technique is used to control the network's memory footprint. DSDM significantly outperforms state-of-the-art continual learning methods on different image classification baselines, even in a low data regime.

Introduction

Although having obtained impressive performance on many different individual tasks, current static deep neural networks (DNNs) must be trained on batches of independently and identically distributed (i.i.d) data samples. These batch learning algorithms distinguish between the processes of knowledge training and knowledge inference, and require restarting the full training process each time new data becomes available. Indeed, they need to access all of the training data within multiple epochs (offline training).

Continual learning (CL) considers the scenario of learning from a non i.i.d data stream where different data and tasks are presented to the model in a sequential manner. While humans can continually observe and learn new knowledge, DNNs have been shown to suffer from catastrophic forgetting (CF) -they are not able to perform well on previously seen data after being updated with recent data. Because their learned knowledge is stored in a single set of weights and learning new information modifies that set of weights, sometimes drastically, new learning frequently causes the networks to forget all, or most of, what they have previously learned.

Catastrophic forgetting was actively studied in the late 1980s and 1990s, mainly in neuro-scientific literature, and numerous solutions were adopted in an attempt to overcome the problem [START_REF] Mccloskey | Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem[END_REF]. A more general problem, referred to as the stability-plasticity dilemma, where the trade-off between the ability of retaining previous knowledge and fast learning a new task, was studied as well [START_REF] Carpenter | ART 2: self-organization of stable category recognition codes for analog input patterns[END_REF]. CF is a direct consequence of the overlap of distributed representations. Indeed, highly distributed representations with significant interaction among them are able to generalize but suffer from CF while very local representations with little or no interaction among them do not suffer from CF but lack generalization ability or storage capacity.

Many early algorithms involved reducing the distributed nature of the network's internal representations, thereby creating relatively sparse representations to reduce the representational overlap. This was done in several ways, either by gradually nudging the network's internal representations into largely nonoverlapping clusters [START_REF]Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks[END_REF] or by orthogonalizing the input representations [START_REF] French | Dynamically Constraining Connectionist Networks to Produce Distributed, Orthogonal Representations to Reduce Catastrophic Interference[END_REF][START_REF] Kruschke | Human Category Learning: Implications for Backpropagation Models[END_REF][START_REF] Lewandowsky | Gradual unlearning and catastrophic interference: A comparison of distributed architectures[END_REF]. Other solutions have involved the network learning "pseudo-patterns" that reflected the previously learned patterns [START_REF] Robins | Catastrophic Forgetting, Rehearsal and Pseudorehearsal[END_REF]; using dual networks in which the old information was transferred to a second, independent network and then interleaved with the new information being learned [START_REF] French | Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma[END_REF][START_REF] Ans | Self-refreshing memory in artificial neural networks: learning temporal sequences without catastrophic forgetting[END_REF], creating an approximation of the error surface of the previously learned patterns and integrating that error surface with the error surface associated with the new patterns [START_REF] French | Using Noise to Compute Error Surfaces in Connectionist Networks: A Novel Means of Reducing Catastrophic Forgetting[END_REF], and so on. See [START_REF] French | Catastrophic forgetting in connectionist networks[END_REF] for a review of early work on CF, dating from the early 1990's through the mid-2000's. A large number of recent continual learning methods have been proposed in the literature and many of them are built on those early solutions.

Sparse Distributed Memory: the standard Kanerva Sparse Distributed Memory (SDM) is a content-addressable memory model developed in the 1980's [START_REF] Kanerva | Sparse distributed memory and related models[END_REF][START_REF] Kanerva | A cerebellar-model associative memory as a generalized random-access memory[END_REF]. In its development, SDM respected fundamental biological constraints and is considered to be a generalization of the original Hopfield network [START_REF] Keeler | Capacity for Patterns and Sequences in Kanerva' s SDM as Compared to Other Associative Memory Models[END_REF][START_REF] Ramsauer | Hopfield Networks is All You Need[END_REF]. Recent work [START_REF] Bricken | Attention Approximates Sparse Distributed Memory[END_REF] has shown mathematically that SDM closely approximates Attention, the core of widely used Transformer models. SDM was used as semi-distributed representations that allowed it to be robust to CF. In SDM, input content is stored in hyperspheres in a very high-dimensional space around hard locations that are largely, but not completely, independent and, thus, do not significantly interfere with one another. At the same time, the size of the hyperspheres is large enough that generalization is not significantly impaired. In this way, the representational overlap causing CF is avoided, while, at the same time, the generalization capacities of the system are not overly affected.

Taking into account these advantages of SDM, we propose a novel algorithm called Dynamic Sparse Distributed Memory (DSDM) that performs excellently on the most challenging setting of continual learning, namely, online task-free scenario. Figure 1 gives a brief overview and the difference between SDM and DSDM. Experiment results of DSDM on a number of image classification benchmarks under different scenarios demonstrate its excellent ability to eliminate catastrophic forgetting.

. . . . . .

𝑆𝑜𝑓𝑡𝑀𝑖𝑛

Content Weight

Address Weight . . . . . .

Fixed

Content Weight

Address Weight

Step function Output Output Variable Fig. 1: SDM (left) vs. DSDM (right). While in SDM, the number of neurons is fixed, their addresses are randomly distributed and fixed, and the radius of reading and writing patterns are predefined, in DSDM all these factors are datadriven and dynamically learnt. The dashed/dotted lines/states mean that these connections and nodes will be added or removed as needed.

Related work

We first briefly discuss different scenarios of CL in Section 2.1 and then present several existing CL methods in Section 2.2. We provide an overview of the standard SDM on which our algorithm is based in Section 2.3.

Continual learning settings

Continual learning has been studied under different scenarios that are often divided into three categories of increasing difficulty [START_REF] Van De Ven | Three scenarios for continual learning[END_REF]: task, domain, and class-incremental. In task-incremental learning, there is a single classification head assigned to each task, whereas in class-incremental learning, there is a single head for all tasks. Models doing domain-incremental learning use a single classification head and do not have to infer the identity of the task.

Online learning requires each image or item to be seen only once during learning, whereas in offline learning, the input data may be seen repeatedly, as is necessary. Additionally, continual learning can be task-based or task-free, depending on whether boundaries between different tasks are known or not. Recently, the data-incremental learning paradigm has been introduced [START_REF] De Lange | Continual Prototype Evolution: Learning Online From Non-Stationary Data Streams[END_REF] to facilitate learning from any data stream, without any assumption regarding task identity, task boundaries or the order of observing the data.

The work presented here studies the most challenging scenario of continual learning in an online, completely task-free and class-incremental (data incremental) setting where learning and evaluating can be carried out at any point of time.

Continual learning algorithms

Many continual learning methods have been developed in three major streams: regularization-based, architecture-based, and memory-based (or rehearsal-based) approaches.

Regularization This approach aims to preserve the knowledge acquired from previous tasks, while maintaining the plasticity of every node. Elastic Weight Consolidation (EWC) [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF] estimates the importance of each of the weights on previous tasks using the Fisher information matrix. Weight changes are then based on the degree of importance of particular weights. Another regularization method penalizes each weight change according to a measure of synaptic intelligence (SI) [START_REF] Zenke | Continual Learning Through Synaptic Intelligence[END_REF]. An attempt is made to estimate the task specificity for each parameter and penalize changes of parameters with high task specificity. This ensures that the parameter stays close to its present value, a value that ensured good performance on learning the initial task.

Knowledge Distillation (KD) is an efficient manner of transferring the knowledge between networks. It was initially introduced for network compression by using the teacher-student mechanism [START_REF] Hinton | Distilling the Knowledge in a Neural Network[END_REF] and widely adopted for CL methods [START_REF] Li | Learning without forgetting: 14th European Conference on Computer Vision[END_REF][START_REF] Schwarz | Progress & Compress: A scalable framework for continual learning[END_REF] and is considered to be one of regularization techniques.

However, it has been shown [START_REF] Van De Ven | Three scenarios for continual learning[END_REF] that methods based on only regularization such as EWC and SI completely fail in a class-IL setting.

Architecture These methods are often based on architectural modifications. They introduce task-specific parameters that dynamically increase the capacity of the model by adding a new layer to the network, as in [START_REF] Wang | Growing a Brain: Fine-Tuning by Increasing Model Capacity[END_REF]. Other methods of this kind dynamically add new layers to the network [START_REF] Roy | Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning[END_REF]. Other strategies developed by [START_REF] Mallya | PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning[END_REF]; [START_REF] Serrà | Overcoming catastrophic forgetting with hard attention to the task[END_REF] attempt to freeze weights that are judged by the system to be the most important to a particular task and that will, therefore, not be updated by future back-propagation. Similar strategies use a mask for each previous task to protect parameters during the backward pass or to choose which parameters to use during the forward pass.

Memory-based approaches

The key idea is to choose a strategy to store specific exemplars. It can be a strategy as simple as random selection or a more complex method, as in [START_REF] Rebuffi | iCaRL: Incremental Classifier and Representation Learning[END_REF]. Some models can replay the exemplars stored in memory with the stream of data from the current task or they can be used as a regularization term, as in GEM [START_REF] Lopez-Paz | Gradient Episodic Memory for Continual Learning[END_REF], or in A-GEM [START_REF] Chaudhry | Efficient Lifelong Learning with A-GEM[END_REF], by computing the scalar product of the loss gradient vectors of previous examples stored in memory and the current data gradient vector. The parameters of the model are then only updated if the scalar product is positive.

To overcome the necessity of an external memory to store data from previous tasks, as in memory-based approaches, certain architectures integrate a generative model able to generate exemplars from previous tasks. These models include the use of a Generative Adversarial Network (GAN) [START_REF] Goodfellow | Generative Adversarial Networks[END_REF] as in [START_REF] Wu | Memory Replay GANs: Learning to Generate New Categories without Forgetting[END_REF][START_REF] Ostapenko | Learning to remember: A synaptic plasticity driven framework for continual learning[END_REF] or a Variational AutoEncoder [START_REF] Kingma | Auto-Encoding Variational Bayes[END_REF][START_REF] Kemker | FearNet: Brain-Inspired Model for Incremental Learning[END_REF] as in [START_REF] Van De Ven | Brain-inspired replay for continual learning with artificial neural networks[END_REF]. These methods are also called Generative Replay methods. The main drawback is that generative models are themselves prone to catastrophic forgetting. Moreover, methods which use a GAN to generate new samples, such as GDM [START_REF] Ostapenko | Learning to Remember: A Synaptic Plasticity Driven Framework for Continual Learning[END_REF] and ILUGAN [START_REF] Xiang | Incremental Learning Using Conditional Adversarial Networks[END_REF], are both offline and task-based.

The proposed DSDM, a dynamic memory, can be considered to be a hybrid model that combines architecture-based and memory-based approaches.

Sparse Distributed Memory (SDM)

SDM is a content-addressable memory capable of both auto-associative and hetero-associative operation. It typically consisting of one million 1000-bit memory locations which are randomly distributed (i.e., fixed addresses) [START_REF] Kanerva | A cerebellar-model associative memory as a generalized random-access memory[END_REF]. Using only a million locations out of possible 2 1000 locations (10 6 ≪ 2 1000 ), it is 'sparse'. SDM is mapped to a two-layer feed-forward neural network (without counting the input layer). The input units convey data patterns to the memory units (i.e., the address nodes). A pattern has two components: the pattern address is the vector representation of a memory point; and the pattern content. The weights to the address nodes represent a 'hard locations' and the weights to the output units store the memory contents.

Storing patterns When storing a pattern, the network computes the Hamming distance between the input address vector and each hard address. Address nodes within a preset Hamming distance will be activated (i.e., set to 1). In other words, if the fixed fan-in weight-vector associated with an address node is similar to the input vector, the address node will be activated; otherwise, it will remain inactive. The content weights that will be modified are those that fan-out from an active address nodes.

Retrieving patterns Reading is the operation of retrieving a data pattern from the memory at a particular address pattern. The input pattern is used to select a certain number of hard memory locations. The retrieved content is the average of the contents (i.e. fan-out weights) at the selected locations, followed by a fixed thresholding step.

Dynamic Sparse Distributed Memory

Conventional SDM networks have a fixed number of memory nodes whose 'hard' addresses are randomly distributed in a binary memory space. This tends to lower performance when dealing with correlated, i.e., nonrandom, input data [START_REF] Hely | A new approach to Kanerva's sparse distributed memory[END_REF]. In our algorithm, we start with an empty memory space, and new address nodes are incrementally added depending on the considered input patterns and the current state of memory space. In other words, we dynamically grow new address nodes as a function of the input. DSDM models the distribution of the input patterns much better than SDM where certain fixed neurons are too far from the Writing the first pattern that is close enough to its best matching unit with respect to a dynamic write radius; B. Neurons in space will move toward the pattern address while their content is sharpened with the pattern content; C. Writing the second pattern being close enough to its BMU; D. The third pattern is far from its BMU with respect to a dynamic radius, so a new neuron in memory is created. Bottom row, Read operation: A. SDM read: the query content reading from nearby neurons within a fixed radius using a majority vote; B. DSDM read: the query content is retrieved from multiple neurons according to its distance to the query address using a softmin function.

current input distribution and therefore are never activated, meaning that our memory has a higher storage capacity. We show experimentally in Section 4.2 that DSDM considerably outperforms SDM with far fewer neurons. Moreover, while a predefined Hamming distance is crucial in SDM, we propose to use a dynamic threshold, a so-called recursive temperature (RT) mechanism as a means of dynamically deciding whether a new memory unit is created.

In our work on image classification, we use DSDM network almost as a backend to a deep encoder and each address-content association consists of a feature vector, extracted from inputs by the encoder and a category label vector, coded in a one-hot manner. A ← A ∪ (xt) 8:

C ← C ∪ (yt) 9:

K ← K + 1 10: else 11:

A ← A + λ × sof tmin(∥A -xt∥/β) × (A -xt) 12: C ← C + λ × sof tmin(∥A -xt∥/β) × (C -yt) 13: end if 14: RT ← λRT × RT + (1 -λRT ) × ∥xt -aBMU ∥ 15: if K > Q then # if memory is full 16:
LOF-based pruning 17: end if

Training phase

As can be seen in Figure 2, DSDM carries out two different types of learning, competitive and supervised learning, within its two layers respectively. The pseudo-code of DSDM training phase is described in Algorithm 1.

Our memory M is represented by two matrices A and C that are initialized with the first input pattern, denoted by (x 1 , y 1 ), presented to the model. Our algorithm is online, each data sample is considered only once. It is also completely task-free, no assumption on task boundaries or the order of data presentation is required. Consider the pattern {(x t , y t )} in the never-ending data stream {(x 2 , y 2 ), (x 3 , y 3 ), ...} progressively presented to the model, we first find its nearest memory node, i.e., the best matching unit -BMU, in the memory space with respect to the location (address). For simplicity, we denote the index of BMU as "BMU":

BMU = argmin i∈[[1,K]] ∥x t -a i ∥ (1)
where ∥a i -a j ∥ is distance between two memory locations; K is the number of neurons created so far; a BM U is then the nearest neuron of x t ; ∥x t -a BM U ∥ is hereafter called "to-BMU distance" (d BM U ).

In the literature, there exist several expandable architectures [START_REF] Yoon | Lifelong Learning with Dynamically Expandable Networks[END_REF][START_REF] Marsland | A self-organising network that grows when required[END_REF][START_REF] Fahlman | The cascade-correlation learning architecture[END_REF] and they compare the to-BMU distance or similar metric to a pre-defined threshold to decide whether a new node or layer is added. While other methods can add a new layer, our DSDM by design has only two layers (excluding the input layer) and we only add/remove nodes. Since the to-BMU distance changes depending on the order of observing data, using a static threshold cannot be optimal and this often leads to an inadequate trade-off between the memory performance and size. To overcome this problem, we define a dynamic threshold as a means of dynamically deciding on whether a new node is created. This is achieved by using an exponential moving average of the last to-BMU distances. We call this mechanism recursive temperature (RT) since its value tends to increase when DSDM starts to learn a new task and then gradually decreases (see section 4.6). The recursive temperature is dynamically updated as:

RT = λ RT × RT + (1 -λ RT ) × ∥x t -a BM U ∥ (2)
where λ RT is the learning rate. At this step, two possible cases can occur:

• If the to-BMU distance is lower than RT, meaning that the input sample is "close enough" to its BMU, the information for the input sample (address and content) will be distributedly stored across multiple nodes already created in memory. The address and content weights of these existing nodes are sharpened proportionally with respect to their distance to the considered input pattern's address, as follows:

A = A + λ × sof tmin(∥A -x t ∥/β) × (A -x t ) (3)
C = C + λ × sof tmin(∥A -x t ∥/β) × (C -y t ) (4)
where λ is the learning rate for sharpening the address and content. This update strategy is similar to that of competitive learning proposed by [START_REF] Keeler | Capacity for Patterns and Sequences in Kanerva' s SDM as Compared to Other Associative Memory Models[END_REF] to adapt models to the current data distribution.

• If the to-BMU distance is higher than RT, a new memory node is created and added into the memory space:

A ← A ∪ (x t ), C ← C ∪ (y t ), K ← K + 1.
Memory pruning Besides using a recursive temperature mechanism to dynamically expand the memory space, we also propose using a density-based algorithm as a means of controlling memory size. In order to estimate the local density of patterns, we use in this work the Local Outlier Factor LOF algorithm [START_REF] Breunig | LOF: Identifying Density-Based Local Outliers[END_REF], which is widely used for anomaly detection. However, we did not remove boundary samples as is done in LOF-based anomaly detection. Instead, we removed address nodes with high density, along with their associated content weights, until the desired number of address nodes has been reached. The intuition is that the removed patterns can still be retrieved by DSDM. We will continue to study different pruning techniques and threshold criteria.

Inference phase

Consider the input query with address x q ∈ R m , to obtain the model output, we first compute its L 2 distance to every patterns in the memory space: d(A, x q ) = [∥a 1 -x q ∥, ∥a 2 -x q ∥, ..., ∥a K -x q ∥].

We then compute the activation vector of the first layer, denoted by W = [w 1 , w 2 , ..., w K ], using a softmin function:

∀j ∈ [[1, K]], w j = e -dj /β K i=1 e -di/β (5)
where β is a temperature parameter which has the same role as the fixed threshold in the original SDM. If we increase β, nodes being far from the query pattern will have higher contribution in the final decision. If β decreases, these nodes being far from the query pattern will have less contribution. In the extreme case with β close to zero, only the content of the nearest address to the query is considered. The final output y q is finally obtained by:

q = K i=1 w i * c i (6)
where c i is the content of i th node, and * is the element-wise multiplication.

Experiments

Experiment setting

Evaluation metrics. We consider the commonly used evaluation metricsnamely, average accuracy (Avg) and last step accuracy (Last) where Avg is the average of accuracy obtained after learning of each task, and Last is the final performance once all tasks/classes have been learned one after the other. We ran each experiment five times on a computer with one NVIDIA RTX 2060 GPU and report the average Top-1 classification results.

Datasets. We evaluated our algorithm under an online class-incremental setting on five commonly used image classification datasets: MNIST dataset [START_REF] Deng | The MNIST Database of Handwritten Digit Images for Machine Learning Research[END_REF] is a handwritten digit recognition dataset with 60k training samples and 10k testing samples.

CIFAR-10 dataset [START_REF] Krizhevsky | Learning Multiple Layers of Features from Tiny Images[END_REF] contains 10 classes and each class is composed of 6k images. They are divided into 50k images for training and 10k for testing.

CIFAR-100 dataset [START_REF] Krizhevsky | Learning Multiple Layers of Features from Tiny Images[END_REF] consists of 100 mutually exclusive classes. Within each class, there are 500 training images and 100 testing images.

CUB-200 dataset [START_REF] Welinder | [END_REF] includes 200 fine-grained bird species. There are 5994 images for training and 5794 images for testing. For each category, there are about 29-30 training images and 11-30 testing images.

CORE-50 dataset [START_REF] Lomonaco | CORe50: a New Dataset and Benchmark for Continuous Object Recognition[END_REF] consists of 164,866 128×128 color images. It has for each class around 2400 training images and 900 testing images. For the class-incremental setting, it is divided into nine tasks and has a total of 50 classes with 10 classes in the first task and 5 classes in the other eight tasks.

In our experiments on all five datasets, we randomly arrange the class order. For each dataset (except CORE-50), other authors often report their results on different incremental step values. For a dataset of 10 classes in total (MNIST or CIFAR-10), when the incremental step is set to 2, the dataset itself is divided into five tasks with two class labels each (with step = 2, we will call Split MNIST and Split CIFAR-10 respectively). Those algorithms have different performances for different step sizes since they are not completely task-free. By contrast, DSDM is data-incremental learning, i.e., online (training with batch size = 1) and completely task-free. In this data configuration, we invariably obtained excellent performance, meaning that our results do not depend on the order of observing data.

Our model has three hyperparameters, i.e., β, λ RT , and λ, where β is the temperature parameter of the softmin function being used during inference phase and has the same role as the fixed threshold in the original SDM, λ RT and λ are the different learning rates, one for updating the dynamic threshold, the other for sharpening the addresses and contents. The model performed similarly for a wide range of parameter values: β ∈ [0.5, 6], λ, λ RT ∈ [0.001, 0.01] when an encoder is used (see details in Sections 4.2, 4.3). Without an encoder (for the MNIST dataset only), we report the accuracies with β = 0.5, λ = 0.0022, λ RT = 0.0025. In order words, for the moment these values are fixed. Future work will involve updating these values dynamically. One idea is to dynamically update β as K increases. We also consider having several BMUs, rather than only one BMU.

Advantage of DSDM over SDM

This section mainly aims at showing the advantage of DSDM over SDM. To this end, we consider the split MNIST benchmark and we do not use any encoder. The pattern address component of our memory is the original MNIST image vector, x t = I t . We modified the SDM to handle vectors of integers for multiple gray levels by replacing the Hamming distance with the city-block and L p distances. While it is theoretically shown in [START_REF] Aggarwal | On the Surprising Behavior of Distance Metrics in High Dimensional Space[END_REF] that for SDM, the city-block distance can slightly outperform the L 2 distance (by 1-2%), it is not the same for our DSDM. Indeed, we evaluated DSDM with these two distances and found that DSDM performs better with the L 2 distance.

As can be seen in Table 1, DSDM significantly outperforms SDM (we set the DSDM model size to 5k that is the half of SDM). Their last accuracies of 94.0 ± 0.2 and 74.2 ± 3.5, respectively, are surprisingly good. DSDM achieved the SOTA performance without any encoder. Our associative memory model is much simpler than other competing algorithms and no gradient flow was even considered. By comparison, other methods in Table 1 (except the Ensemble method) used a multi-layer perceptron with two hidden layers of 400 nodes each and ReLU non-linearities were used in all hidden layers. To achieve 91.0 ± 0.4, the Ensemble algorithm used a variational autoencoder where the encoder half comprises two convolutional layers followed by two linear layers and the decoder has two linear layers followed by two transpose convolutional layers. The results of CURL [START_REF] Rao | Continual unsupervised representation learning[END_REF], CN-DPM [START_REF] Lee | A neural dirichlet process mixture model for task-free continual learning[END_REF] and Ensemble [START_REF] Shanahan | Encoders and Ensembles for Task-Free Continual Learning[END_REF] methods are taken from their original work while other ones are from [START_REF] De Lange | Continual Prototype Evolution: Learning Online From Non-Stationary Data Streams[END_REF]. Table 1: Last accuracy on split MNIST (step=2). Without any encoder and gradient-based update, the shallow DSDM model outperforms SOTA deep methods.

Comparison with online SOTA methods

We compare our method with existing online approaches including A-GEM [START_REF] Chaudhry | Efficient Lifelong Learning with A-GEM[END_REF], GSS [START_REF] Aljundi | Gradient based sample selection for online continual learning[END_REF], MIR [START_REF] Aljundi | Online Continual Learning with Maximal Interfered Retrieval[END_REF], ASER [START_REF] Shim | Online Class-Incremental Continual Learning with Adversarial Shapley Value[END_REF], GDUMB [START_REF] Prabhu | GDumb: A Simple Approach that Questions Our Progress in Continual Learning[END_REF] and Candidates Voting (CV) [START_REF] He | Online Continual Learning Via Candidates Voting[END_REF]. Similar to the very recent work of CV [START_REF] He | Online Continual Learning Via Candidates Voting[END_REF], we report the average and last accuracies on Split CIFAR-10 and CORE-50. A small version of ResNet-18 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] pretrained on ImageNet [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF] is used as the backbone for all of the methods compared. In DSDM, the address component of a pattern is the feature extracted from the image produced by ResNet-18: x t = ResN et18(I t ). CV and DSDM freeze the parameters of the backbone network, while others do not. We vary the buffer size Q ∈ {1k, 2k, 5k} for comparisons. On CORE-50, the dataset that is specifically designed for continual learning, DSDM significantly outperforms existing online approaches on all considered buffer sizes Q with the single exception of CV with buffer sizes of 1k when measured by the average accuracy. As for the last accuracies, DSDM is on average 50% better than the second-best results. On the split CIFAR-10 benchmark, DSDM outperforms all other methods with a limited storage capacity Q = 1k and its performance is essentially the same as the best algorithm (CV). Moreover, only our algorithm is completely task-free, the learning and evaluating can be carried out at any point in time. The CV algorithm is online but it is not task-free during training with the use of task boundaries.

Several methods like MIR and GSS in Table 2 can perform reasonably well in both task-based and task-free settings. Their performance in the latter is worse than in the former. Their performance on the CIFAR10 dataset under these settings are respectively 52.2% and 42.8% (-9.6% for MIR) and 56.7% and 38.4% (-18.3% for GSS).

Comparison with offline SOTA methods

We also compare our method with offline continual learning approaches that update the model in multiple epochs on the CIFAR-100 and CUB-100 benchmarks with different incremental steps. Similar to FearNet ( [START_REF] Kemker | FearNet: Brain-Inspired Model for Incremental Learning[END_REF]) and ILUGAN ( [START_REF] Xiang | Incremental Learning Using Conditional Adversarial Networks[END_REF]), Resnet-50 backbone pre-trained on ImageNet is used as the feature extractor in DSDM: x t = ResN et50(I t ). On these benchmarks, we set Q = 3k. We consider SOTA offline approaches including EWC [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF], LWF [START_REF] Li | Learning without forgetting: 14th European Conference on Computer Vision[END_REF], iCaRL [START_REF] Rebuffi | iCaRL: Incremental Classifier and Representation Learning[END_REF], End-to-End [START_REF] Castro | End-to-End Incremental Learning[END_REF], FearNet [START_REF] Kemker | FearNet: Brain-Inspired Model for Incremental Learning[END_REF] and ILUGAN [START_REF] Xiang | Incremental Learning Using Conditional Adversarial Networks[END_REF]. The reported results are from [START_REF] Xiang | Incremental Learning Using Conditional Adversarial Networks[END_REF]. Although it is widely acknowledged that performance in the online scenario is worse than offline, our DSDM still outperforms those SOTA offline approaches Several methods like MIR and GSS can perform reasonably well under both task-based and task-free settings. We report here their best performance when task boundaries are known. The lines from "A-GEM" to "CV" are from [START_REF] He | Online Continual Learning Via Candidates Voting[END_REF].

with important margins, as can be seen in Table 3(a). phase. By contrast, our model operates in the completely task-free setting (DSDM achieved similar performances as can be seen in Tables 1,2). We demonstrate this capability by evaluating DSDM under the Gaussian schedule benchmarks, recently proposed in [START_REF] Shanahan | Encoders and Ensembles for Task-Free Continual Learning[END_REF]. A schedule defines how a data distribution evolves over training. In the Gaussian schedule, there are no task boundaries. Each label appears in the data with a probability that follows a Gaussian distribution, and each label's probability peaks at a different time.

Few-shot CL: We further study the data efficiency of our model. We report the results on CIFAR-10 and CIFAR-100 when using only 10% and 30% of the training samples from these datasets respectively. We use here two different feature extractors. We use two different feature extractors, ResNet50 pretrained on ImageNet in a supervised manner and a vision transformer encoder pretrained with self-supervised learning (SSL) [START_REF] Caron | Emerging Properties in Self-Supervised Vision Transformers[END_REF]). (The simplest version ViT8 was used). We denote these methods as 'DSDM + CNN' and 'DSDM + ViT' respectively. For 'DSDM + CNN', we used a buffer size Q = 2k, 6k for CIFAR-10 and CIFAR-100, respectively. For 'Ours ViT', we used a buffer size Q = 1k, 3k for CIFAR-10 and CIFAR-100, respectively. Table 4 compares the last accuracies under different scenarios. To the best of our knowledge, only the Ensemble architecture using a fixed ResNet50 pretrained in a SSL manner on ImageNet reports its performance with a Gaussian schedule (other results of Table 4 come from [START_REF] Shanahan | Encoders and Ensembles for Task-Free Continual Learning[END_REF]). DSDM outperforms Ensemble while requiring much less data. Under the challenging scenario of a Gaussian schedule, DSDM's gain is highly significant. The performance accuracies of 'DSDM + ViT' are respectively 69% and 57% better than those of Ensemble on CIFAR-10 and CIFAR-100, respectively.

Impact of a dynamic threshold

This section shows the importance of using a dynamic threshold. Figure 3 shows the evolution of recursive temperature and to-BMU distances on the Core-50 training dataset. The to-BMU distances change when new data is seen. Modifying or eliminating the use of DSDM's fixed encoder is currently under study. [START_REF] Goyal | Self-supervised Pretraining of Visual Features in the Wild[END_REF] has shown that an SSL encoder can be trained in online learning with random images. Moreover, [START_REF] Orhan | Self-supervised learning through the eyes of a child[END_REF] showed that features learned by an SSL encoder on a subset of the visual experiences of developing children led to high accuracy in non-trivial downstream categorization tasks. Further, the capacity to generalize under these circumstances was shown in [START_REF] Caron | Emerging Properties in Self-Supervised Vision Transformers[END_REF]. This potentially means that our algorithm could generalize to new data that is very different from the initial pre-training data. In future work we will explore training a combination of an encoder and our DSDM architecture in an online and continual manner with a scenario corresponding to that of developing children. Indeed, we could first learn good representation features with SSL, followed by the continual learning phase with supervised learning. This scenario would arguably simulate real human learning more realistically than most current CL scenarior that start directly with incremental learning without any pretraining. We will, in addition, study how to apply DSDM to other tasks, such as regression or anomaly detection, as in [START_REF] Jezequel | Efficient anomaly detection using self-supervised multi-cue tasks[END_REF].

Fig. 2 :

 2 Fig.2: The read and write operations of SDM and DSDM. Top row, SDM write: A+B+C. Two patterns are stored inside nearby fixed neurons within a fixed write radius; D. Writing a third pattern and neurons storing a superposition of multiple patterns. Middle row, DSDM write: A. Writing the first pattern that is close enough to its best matching unit with respect to a dynamic write radius; B. Neurons in space will move toward the pattern address while their content is sharpened with the pattern content; C. Writing the second pattern being close enough to its BMU; D. The third pattern is far from its BMU with respect to a dynamic radius, so a new neuron in memory is created. Bottom row, Read operation: A. SDM read: the query content reading from nearby neurons within a fixed radius using a majority vote; B. DSDM read: the query content is retrieved from multiple neurons according to its distance to the query address using a softmin function.

Algorithm 1 DSDM training 1 :

 1 Init: Address matrix A = a1 = x1 , content matrix C = c1 = y1 with capacity Q, RT = 0; K = 1; 2: Require: β, λRT , λ. λRT and λ are the different learning rates, one for updating the dynamic threshold, one for sharpening the address and content. 3: Input: Consider each sample (xt, yt) from the data stream {(x2, y2), (x3, y3), ...}. 4: BM U = argmin i∈[[1,K]] ∥xt -ai∥ 5: dBMU ← ∥xt -aBMU ∥ 6: if dBMU > RT then 7:

  47] CN-DPM[START_REF] Lee | A neural dirichlet process mixture model for task-free continual learning[END_REF] Ensemble[START_REF] Shanahan | Encoders and Ensembles for Task-Free Continual Learning[END_REF] MIR[START_REF] Aljundi | Online Continual Learning with Maximal Interfered Retrieval[END_REF] CoPE[START_REF] De Lange | Continual Prototype Evolution: Learning Online From Non-Stationary Data Streams[END_REF] 74.2 ± 3.5 94.0 ± 0.2 19.7 ± 0.05 92.6 ± 0.7 93.2 ± 0.1 91.0 ± 0.4 93.2 ± 0.4 93.9 ± 0.2

Fig. 3 :

 3 Fig. 3: The evolution of recursive temperature and to-BMU distance on the training set of Core-50

Table 2 :

 2 Average and last accuracy on CIFAR-10 (step=2) and CORE-50. Best results marked in bold; second best results are underlined. Only our algorithm is completely task-free.

	Datasets	CIFAR-10, step = 2		CORE-50
	Buffer size Q	1k	2k	5k	1k	2k	5k
	Accuracy	Avg Last Avg Last Avg Last Avg Last Avg Last Avg Last
	A-GEM [10]	43.0 17.5 59.1 38.3 74.0 59.0 20.7 8.4 21.9 10.3 22.9 11.5
	MIR [2]	67.3 52.2 80.2 66.2 83.4 74.8 33.9 21.1 37.1 24.5 38.1 27.7
	GSS [3]	70.3 56.7 73.6 56.3 79.3 64.4 27.8 17.8 31.0 18.9 31.8 21.1
	ASER [55]	63.4 46.4 78.2 59.3 83.3 73.1 24.3 12.2 30.8 17.4 32.5 18.5
	GDUMB [45]	73.8 57.7 83.8 72.4 85.3 75.9 41.2 23.6 48.4 32.7 54.3 41.6
	CV [21]	76.0 62.9 84.9 74.1 86.1 77.0 45.1 26.5 50.7 34.5 56.3 43.1
	DSDM	80.2 67.0 83.8 72.5 85.6 76.0 43.9 43.3 53.2 50.8 66.3 57.1
	DSDM/CV (%)	107	98	99	163	147	132

Table 3

 3 (b) presents the training batch size and epoch number of different algorithms and clearly shows the advantages of DSDM.

	Datasets Step	CIFAR-100 CUB-200 2 5 2 5	Datasets Training	CIFAR-100 CUB-200 BS Ep BS Ep
	EWC [31] LwF [36] iCaRL [48] End-to-End [9] 50.3 49.7 44.5 49.9 15.4 17.9 15.9 18.2 32.8 37.9 29.8 36.1 52.8 57.2 44.3 49.6 FearNet [29] 56.9 62.5 47.8 52.7 ILUGAN [61] 58.0 63.1 49.7 54.9 DSDM 63.3 63.2 55.5 55.2	iCaRL End-to-End 128 450 FearNet 450 ILUGAN CV 10 DSDM 1	70 70 250 90 1 1	450 70 128 70 200 100 90 10 1 1 1
		(a)			(b)

Table 3

 3

	: (a) Comparison of average accuracy on CIFAR-100 and CUB-200. (b)
	Comparison of training batch size (BS) and epochs (Ep). Only DSDM can
	carry out learning and evaluating at any point in time.
	4.5 Completely task-free and few-shot settings
	Completely task-free: As mentioned above, other authors often reported
	different results with different incremental step sizes since their algorithms
	distinguish tasks that have clear task boundaries, at least during the training

Table 4 :

 4 Comparison of last accuracy of different methods under the challenging scenarios. Best results marked in bold; second best results are underlined. Our algorithms works well on limited-data regimes.

https://github.com/


Allal Barroug
email: a.barroug@ucam.ac.ma

Ahmed Legrouri
email: blegrouri@aui.ma

Christian Rey
email: cchristian.rey@ensiacet.fr

Exchange reactions at calcium phosphates surface and applications to biomaterials

Keywords: Adsorption, Biomaterials, Calcium phosphates, Exchange reactions, Release

The interaction of calcium phosphates with biological molecules under controlled conditions permits the formulation of meaningful conclusions concerning the driving forces. The uptake and release at the material-solution interface is the result of the various interactions between and within the system components which include the solid surface, the adsorbate, the solvent and other solutes present. The understanding of adsorption and desorption mechanisms with respect to active molecules can be exploited for the development of drug delivery applications.

Open Archive Toulouse Archive Ouverte (OATAO)

OATAO is an open access repository that collects the work of Toulouse researchers and makes it freely available over the web where possible. This is an author-deposited version published in: http://oatao.univ-toulouse.fr/ Eprints ID : 2377

To link to this article : URL : http://dx.doi.org/10.4028/www.scientific.net/KEM. 361-363.79 To cite this version : Barroug, Allal and Legrouri, Ahmed and Rey, Christian ( 2008) Exchange Reactions at Calcium Phosphates Surface and Applications to Biomaterials. Key Engineering Materials, vol. 361 -363 . pp. 79-82. ISSN 1013-9826 Any correspondence concerning this service should be sent to the repository administrator: staff-oatao@inp-toulouse.fr

Introduction

Calcium phosphate apatites have received a recurrent attention for the last twenty years mainly for their use as biomaterials related to compositional similarity with bone mineral and biological properties. These materials have found various applications especially as bone substitutes and have been used as drug delivery systems [1][2][3][4]. The binding and release of active molecules and ions at the interface between calcium phosphate minerals and biological environments are crucial to determine how they perform in vivo. We report here on the study of calcium phosphates interaction with phosphoserine (P-ser) and bovine serum albumin (BSA), two important actors in the biomineralization processes [5]. The composition of the adsorption medium was considered as an experimental variable in order to establish a quantitative relationship between the sorbent uptakerelease and the surface characteristics of apatite crystals.

Materials and methods

Several types of apatites were used: a well crystallized hydroxyapatite (HA) and biomimetic apatites corresponding to different stages of bone mineral evolution (maturation) with age. They were prepared by double decomposition according to methods previously described [3]. Different maturation stages were obtained by ageing of the precipitates at room temperature in mother solutions. The samples were removed by filtration, washed with de-ionized water and lyophilized overnight. The powders were characterized by X-ray diffraction, Fourier transform infrared spectroscopy, and chemical analyses. The surface area of the crystals was measured by nitrogen adsorption according to the BET method.

Adsorption experiments were performed using P-Ser and BSA in the presence of different electrolytes (de-ionized water, potassium nitrate and potassium nitrate containing calcium or phosphate). The solid was dispersed in the adsorption medium at room temperature. After an incubation time, the suspension was centrifuged and the supernatant obtained was assayed for BSA and P-Ser [6,7].

Desorption was examined considering changes in the bulk solution concentration. The supernatants obtained after centrifugation were diluted with a solution with the same electrolyte content as in adsorption tests but without adsorbate or replaced by the same volume of a solution containing phosphate or carbonate. The pellets were re-dispersed and centrifuged again and the supernatants were thereafter examined for the presence of desorbed molecules.

Results

The solids prepared exhibit XRD patterns and FTIR spectra characteristic of poorly-crystalline (PCA) and well-crystalline (HA) apatitic calcium phosphates. Chemical analyses showed that all the specimens were calcium deficient (Ca/P ratio lower than 1.67). FTIR spectra Deconvolution of the PCA samples showed the presence of phosphates groups assigned to non-apatitic environments [8]. Maturation of the crystals was observed to cause a moderate improvement of the crystallinity index, a gradual decrease of the content of non-apatitic phosphates and cation vacancies in the lattice.

Despite the different characteristics exhibited by the adsorbates and the various solution compositions examined, the adsorption isotherms obtained were Langmuirian in shape (Fig. 1). The adsorption parameters, maximum amount loaded and affinity constant, are conditioned by the content of potential determining ions in solution (hydrogen, phosphate and calcium) and the surface composition of the nanocrystals. The influence of the solution composition on the adsorption of anionic molecules (amino acids, acidic proteins) on apatitic calcium phosphates shows a common behaviour. The isotherms obtained showed a distinct initial increase related to the affinity and adsorption plateau generally considered to represent the saturation of the adsorption sites. The maximum amount adsorbed decreases as the solution pH rises. Besides, the uptake in the range of concentration investigated was inhibited in presence of excess phosphate or carbonate, while a greater adsorption capacity occurred when calcium ions were added to the medium. However, an opposite tendency was observed with basic molecules such as lysozyme [9].

The P-Ser and BSA molecules bound to apatite crystals were not removable by dilution of the equilibrating solution. This phenomenon is well illustrated in Fig. 1 which revealed a difference between the adsorption and desorption branches for P-Ser. Similar trend was noted for the interaction of hydroxyapatite with acidic proteins such as catalase and succinylated lyzozyme [10,11]. Desorption can occur by changing the different solution parameters (pH, ionic strength, concentration of potential determining ions, etc). Indeed, the removal of P-Ser and BSA molecules was promoted by other surface active species such as phosphate and carbonate. Fig. 2 indicates that the extent of phosphoserine released markedly increased with increasing phosphate and carbonate concentrations in solution. These observations suggest that these species compete, to a certain extent, with the adsorbate molecules for the same location on the mineral surface. The hypothesis of an adsorption reaction involving ion replacement is further supported by the adsorption tests of P-Ser and BSA carried out on poorly crystalline samples at different stages of maturation. These samples exhibit on the crystals surface a hydrated layer with mobile bivalent ions (mainly Ca 2+ and HPO 4

2-

) in non-apatitic environments. During maturation, the hydrated layer is progressively thinning as the more stable apatite domain increases. The freshly precipitated crystals, which are rich in non-apatitic environments, offer more ion exchange capabilities than the matured samples and the maximum amount adsorbed is larger than in more mature samples. This ionexchange driven adsorption process would mainly be determined by the mobility of the ionic species at the mineral surface. Similar behaviour was obtained with heparin [12]. An example of the relationship between the surface content of non-apatitic, labile phosphate groups and the maximum amount adsorbed is presented in Fig. 3.

Discussion

The results reported in this study indicate that uptake and release of P-Ser and BSA molecules are conditioned by the changes in the content of different mineral ions in solution. The adsorption of these anionic molecules was inhibited in presence of excess phosphate or carbonate and favoured when calcium ions were added to the medium. Furthermore, the process was not reversible with respect to dilution. The adsorbed molecules can only be displaced by a reverse ion-exchange reaction. The main driving forces at the mineral interface seem to be an ion-exchange process involving the functional groups of the molecules and the ionic groups at the apatite surface. This is in line with the results reported in our previous studies [10,11], which revealed that interaction of proteins and apatite crystals, under conditions where they bear the same or opposite charges as evidenced by electrophoretic measurements, involves an exchange (incorporation or expulsion) with compensating low molecular weight ions. The correlation between the change of P-Ser adsorption behavior and the modification of surface composition of the nanocrystals upon maturation supports the hypothesis of an adsorption process driven by exchange reactions. The present study stresses the fact that adsorption properties of calcium phosphates are essentially dependent on their surface characteristics, especially the development of the hydrated layer, and on the mineral ion composition of the adsorbate solution. Adsorption properties of apatites can strongly vary and data obtained on well crystallized samples cannot be compared to those obtained on nanocrystalline samples. The flexibility of surface and biological properties of nanocrystalline apatites suggests that these could be used as potential substrates for drug delivery. Adsorption may be used to load the apatite crystals with selected active molecules in optimised and controlled ways. Moreover, the release process in vivo may occur by displacement of the agent by mineral ions or proteins from the body fluid, or alteration of the apatite.

Conclusion

The adsorption and release properties of apatite nanocrystals appear strongly dependent on the composition of the hydrated surface layer and the surrounding environments. The understanding of the interaction mechanisms and the control of the driving forces may provide fundamental tools for the development and application of delivery systems based on the adsorption of pharmaceuticals onto calcium phosphates.

Fig. 1 :

 1 Fig. 1: Adsorption and desorption isotherms of phosphoserine on calcium phosphate

Fig. 3 :Fig. 2 :

 32 Fig. 3: Phosphoserine adsorbed versus content of non-apatitic, labile phosphate groups

Exchange Reactions at Calcium Phosphates Surface and Applications to Biomaterials

  Bioceramics 20 doi:10.4028/www.scientific.net/KEM.361-363 doi:10.4028/www.scientific.net/KEM.361-363.79