Sabine Gorovitz

Brazilians in French Guiana: Types of bilingual talk in family interactions 1

Keywords: bilingual talk, family interaction, mobilities, codeswitching, language mixing

This article is the result of research conducted in French Guiana on linguistic integration of Brazilians living in Cayenne. We have observed family interactions of some families who have settled there, focusing on how its members speak different languages that make up their repertoire and the type of bilingual talk they produce.

Introduction

This study sought to understand the phenomenon of language contact resulting from migration dynamics between Brazil and French Guiana, which are marked by strong diversity of population movements. In order to do so, we have observed ordinary discourses produced by speakers in family interactions with the aim of understanding how its members speak, the kind of bilingual conversation or bilingual talk (Auer 1984) they produce and which linguistic and extralinguistic categories are relevant to grasp the phenomenon of bilingual talk in all its complexity.

Field Description: Brazilians in French Guiana

The population of French Guiana has doubled in less than thirty years, both because of its increasing fertility and decreasing mortality, and especially because of migratory waves occurring at different periods. According to data collected by Piantoni (2011), out of 66,700 foreigners (37% of the population), 25% (18,800) are Brazilian, who now represent the second largest immigrant community in the country. From initial findings about the wide diversity of bilingual talks, we chose to base our analysis on the concepts defined by Auer (1998).

His description model raises a central question about how speakers use and perceive the languages involved in bilingual talks. The author analyzes the phenomena of juxtaposition of languages and presents a typology, we adopted to distinguish the various types of bilingual talks observed in our corpora: codeswitching (CS) and language mixing (LM). Alternation can be labeled as CS when it makes sense for the participants of the interaction, i.e., when it appears as a contextual strategy, a meta-practice, revealing something that is possible to interpret, both because of previous events (linguistic and extralinguistic) and because of characteristics of the situation 2 . On the other hand, language mixing (LM) occurs not when alternation produces a specific and locally defined sense, but when it is the fact itself of mixing that makes sense, maybe corresponding to a personal or group style 3 .

Data Analysis

Our goal was to relate types of bilingual talks with a set of contextual and social factors, in order to identify speaker and possibly family profiles. To trace these profiles, it was necessary to select relevant socio-demographic categories 4 from repeated observations and interviews.

From these first categories, we developed 2 Meaning and function attributed to alternations can be highly variable: change of subject, speaker, tone, exemplification, quotation, translation, relation of utterance, emphasis, contrast, etc.

3 Auer describes another sub-categorization to distinguish the insertional type of LM (loans) and the alternational type of LM. 4 Among others, gender, country of birth, neighbourhood (homogeneous or diversified), frequency of trips to Brazil, social class, etc.

a preliminary quantitative analysis of data. We sought at first to illustrate the fact that the speeches in this context show more or less language alternation. To this end, a word counting produced in each language was realized.

Although artificial, this quantification provided a graphical representation of the percentage of language alternation produced by each speaker. It showed that generally certain speakers alternate much more than others and that some resemble others in their linguistic behavior. These similarities, which were established from the linguistic categories defined to describe the data, led to the identification of clusters of speakers whose language productions present comparable language alternation. For instance, it was possible to assert that the speakers who alternate the most were born in French Guiana, and conversely those who alternate less had recently arrived in French Guiana or arrived as adults already.

A relevant narrower quantification was then established from the categories defined by Auer (CS, LM, Insertional vs. Alternational LM).

In fact, it was not enough to show that some speakers alternate more than others, it was In conclusion, speakers mobilize the languages that make up their repertoires and make their choices depending on the challenges and on the situations of interaction. In fact, the results reaffirm that language alternation is a feature "carrying a symbolism, co-built by the agents, and the referent of their sociolinguistic personal or collective universe, constantly recreated" [START_REF] Dreyfus | Le jeu de l'alternance dans la vie quotidienne des jeunes scolarisés à Dakar et à Ziguinchor (Sénégal)[END_REF]: 9, our translation5 ). In fact, bilingual talks refer to certain knowledge, which is shared by the interlocutors because it underlies the interaction.

It is a pact of cooperation, which regulates the discursive positions and relationships between interlocutors.

Bibliography AUER, Peter, 1984, Bilingual conversation, Amsterdam: Benjamins.

AUER, Peter, 1998, "From codeswitching via language mixing to fused lects: Toward a dynamic typology of bilingual speech". The International Journal of Bilingualism, 3, 309-332. GARAFANGA, Joseph, 2001, Linguistic identities in talk-in-interaction ; Order in bilingual conversation. Journal of Pragmatics 33, 1901Pragmatics 33, -1925. . LEGLISE, Isabelle, & ALBY, Sophie,

  Brazilians arrived at different times and for different reasons, mainly from the regions of Amapá and Pará, focusing mostly the capital, Cayenne. Today Brazilian immigration is characterized by its diversified origin, both regionally and socially, with a second or third generation that was born and educated in French Guiana. The proximity to Brazil favors the intensity of border movements and maintenance of linguistic ties and identity of those immigrants. Object Search In order to understand how Brazilians living in Cayenne speak, we have searched about which languages they speak and convey; what kind of bilingual talks are produced and by whom. To this end, we crossed types of bilingual talks with socio-demographic criteria to determine profiles of speakers and families. It is noteworthy that all observed speakers produce bior multilingual discourses, in varying degrees and ways.To understand this diversity, we have observed the types of relationships built during interactions, referring to identity processes(Zimmermann 1998, Garafanga 2001), which can be expressed more or less explicitly. Thus we seek to describe the linguistic practices, taking into consideration the characteristics of the context in which speakers interact.Methodology and categorization of mixing categorizationWe analyzed a corpus of linguistic data collected during a lengthy immersion from 2008 to 2011, in which interactions in ten households (32 speakers) were observed, recorded, with the requirements of the research, in this case, identify alternations of language (codeswitching and language mixing) in the speech of speakers during the interaction.

  necessary to describe how they alternate from one language to another. Subcategories were used, such as CS1 (change of speaker), CS2 (quotation, translation, explanation, reference to local experience, etc), CS3 (change of subject), CS4 (introduction of a new fact), CS5 accommodation, CS6 (complicity, change of tone), ALM (alternational type) and ILM (insertional type). Thus we accounted the number of times each speaker switches from one language to another, during the interaction, sometimes on the insertional mode, sometimes on the alternational mode. Then we obtained the average percentage of types of juxtapositions in relation to the total of words produced by each speaker, in each utterance. With these values, it was possible to describe the prototypical attitude of each speaker in terms of language mixing. Some speakers tend to mix preferably using the insertional mode while others use the alternational mode. The results show that speakers who mix more tend to use the alternational mode. For many speakers alternations work as quotes, translations, clarifications or references to a local experience. For others, alternation is much more related to the listener. In addition, others alternate in a more systematic manner, as a kind of personal style, precluding a clear association of their alternations to a specific function. Thus we conclude that while some speakers mix languages as a matter of personal style, others do it out of necessity or as a strategic additional resource. Crossing types of bilingual talks and sociodemographic factors The next step was the examination of the correlation between the quantity and types of bilingual talks and some of the relevant sociodemographic factors. Although many factors had an insufficient correlation rate to be taken into account, others (such as age, place of birth and time living in French Guiana) revealed interesting results that suggest, for example, that speakers who interact the most on a bilingual mode are between 36 and 45 years old. Another significant result indicates that speakers who were born in French Guiana interact on a bilingual mode more than those who were born in Brazil. Conclusion The present study revealed a strong presence of bilingual discourse in most observed families and a systematic intergenerational transmission of Portuguese. It has also indicated a wide range of sociolinguistic profiles and individual pathways, making it impossible to generalize the results. The observed variations also indicate different types of bilingual talk, recurrences and idiosyncrasies of each speaker according to the interactional situation. It is necessary to highlight the extreme variability of the analyzed data, due to multiple individual, collective and social factors. Our challenge was therefore to overcome these particularities to find recognizable convergence elements in the set of bilingual talks produced. In order to associate family or speaker profiles to prototypical ways of bilingual speaking, several questions were raised: are there regularities personal or group styles? What kinds of pressure interfere with alternations? Are there moments when alternations are intensified? And which meanings emerge from them? To answer these questions, we identified tendencies characterizing certain behaviors, types of declared postures and ways of bilingual talk for each speaker in their relationship with others. Data interpretation has been therefore established on a case-by-case basis, depending on the interactional event, from a qualitative approach. At the end of this exploratory journey, and even though the quantitative results have not enabled many generalizations, we can affirm that part of the observed speakers present a clear tendency to produce a particular type of bilingual talk, and those who preferably mix in the alternational mode reveal a prototypical way of performing these alternations. It was also possible to establish links with social characteristics and assert, for example, that speakers that alternate the most in CS1 mode live in more open neighborhoods, occupied by diversified communities. This kind of result suggests, for instance, that heterogeneity inside neighborhoods acts as a conservative force, capable of slowing down the production of mixing and hence forthcoming linguistic changes (Léglise & Chamoreau, 2013).Another result is that talks on a bilingual mode occur in our data only under certain conditions. Two of these conditions are the regularity of contact between the languages and the frequency of stay in both countries.Moreover, three main profiles of speakers have been revealed. The first category regards speakers who preferably produce LM and who perceive their bilingualism as a factor of social mobility, because of their socioeconomic situation. The second regards speakers whose competence in both languages is asymmetric, causing the production of typical insertional mixing. The third category regards those whose languages usage is compartmentalized (home language, administration language, school language, work language, etc).It is noteworthy that, generally, the boundaries between the segments uttered in French and Portuguese (whether lexical or syntactic) are not always clear as showed in other language situation (Léglise & Alby, 2013), both languages often appearing closely intertwined.

"Le mélange linguistique observé au sein des interactions est un element emergent porteur d'un symbolisme coconstruit par les agents, et le référent de leur univers sociolinguistique, propre autant que collectif, constamment en mouvement et recréé".

1 This work is the result of research developed since 2008 as an institutional partnership between the University of Brasilia and laboratory SEDYL-CELIA (CNRS) in France. This research was financed by the ANR DC2MT "Les Suds Aujourd 'hui" (2008-2011) and the AIRD, through PEERS MOBILANG (2012)(2013) program.


This vibrant celebration of the legacy of Richard Allsopp, a pioneering lexicographer and cultural researcher, explores various aspects of language, culture, and identity in the Caribbean from the perspective of leading Caribbean scholars, some of whom were Allsopp's students and many,

The importance of studying Caribbean English is demonstrated by Pauline Christie's essay on the varieties and statuses of English in the Caribbean. Creole linguistics in the Caribbean is now recognized as a major field of investigation. Three linguistic contributions on Guyanese are included: Alim Hosein treats reduplication and shows that the classic decreolization hypothesis does not hold; Walter F. Edwards revisits the use of future markers go and gun and shows that they derive from different sources; and John R. Rickford investigates variability among personal pronouns. Rickford argues that in our excitement about the social and identity elements in sociolinguistic variation we easily forget the anchoring effect of linguistic structure. He shows the importance of internal constraints on variation and the structure of variability in the linguistic system.

Lexicography is also well represented with contributions by Jeannette Allsopp, John Simpson, and Lise Winer. Winer makes an interesting methodological point about the use of (old/colonial) photographs as a means of word elicitation, even if European photographers have reinforced and perpetuated stereotypes by the way they "selected, posed and framed images and exoticized" [START_REF] Bravman | Using Old Photographs in Interviews: Some Cautionary Notes About Silences in Fieldwork[END_REF]) large parts of the population in the world. Photographs, she argues, transcend time and space, and might help to elicit words and bridge "cultural gaps."

Two sections are devoted to culture in the Caribbean: one on folklore and religion and another on literature, music, and dance. Hazel Simmons-McDonald shows the afrogenesis of French Creole proverbs, while Ian Robertson links a Guyanese folkloric character, Mamma Mingi, to cultural and spiritual (African, Amerindian, and Caribbean) practices. Kean Gibson treats the adaptation of African Comfa to the Guyanese context. In "Narrative as autobiography of the folk," Velma Pollard analyzes short stories by Olive Senior in the light of race, religion, and class. Literary devices are particularly vivid in Caribbean calypso, as Claudith Thompson shows.

Caribbean identity is addressed by many contributions, but specifically by Hélène Zarmor and Mervyn Alleyne in its relation to creole identity. Although appealing, Alleyne argues that "creole identity" is an ill-defined concept, variable, and complex. Its first meaning, "person of European descent born in the New World," reminds us that the European prerogative to give names was a major instrument of control for setting norms. The term "creole" was never used in the English colonial Caribbean, in either popular or scientific discourse, whereas in the French colonial Caribbean, the term belongs to popular discourse and has gained literary, cultural, and identitarian meaning under the term créolité. For Alleyne, in the French Antilles, the term "creole" rejects essentialist notions such as origin. But, like any term, it is polyphonic and may be redefined in discourse on identity. See the evolution of what it means to be "French Guianese" and/or "Creole" in French Guiana [START_REF] Hidair | L'immigration étrangère en Guyane: entre stigmatisation et stratégie de récupération[END_REF][START_REF] Migge | Exploring Language in a Multilingual Context: Variation, Interaction and Ideology in language documentation[END_REF]. Hélène Zarmor's essay departs from the term creolization, reminding us how difficult the 1946 departmentalization was in the history of the French Caribbean, and highlighting the contradictory positions that emerged at that time in various movements-including within négritude. It was such a hostile political environment that identity became a challenge for French Caribbean people. Although language is vital in constructing identity, she argues that music and dance-because they employ French Creole language-have sustained this language and changed people's attitudes to it.

Finally, two contributions address language issues in Caribbean schools-one by Martha Isaac and another by Karen Carpenter and Hubert Devonish. Carpenter and Devonish remind us that bilingual education projects in Jamaica developed from a monoliterate transitional bilingual approach (using the home language, Jamaican Creole, as a bridge toward English, the language of education) to a pilot project of fully bilingual education or dual language education (using both L1 and L2 as subjects and medium of instruction). They discuss some results of a fully bilingual pilot project implemented over the last ten years, showing better literacy results to national tests for pupils within this project and the greater preference by boys for Jamaican Creole, leading them to certain (negative) attitudes toward English and literacy in English. Although they show evidence of the fact that the use of Jamaican Creole in education yields educational dividends, policy makers are still difficult to convince, and it is a good thing that "the struggle for language education policies based on evidence rather than opinions continues" in Jamaica and in the Caribbean, as well as all around the world. This book is inspiring for further research. It is recommended for graduate students and scholars interested in linguistics and Caribbean Studies.

Isabelle Léglise CNRS, UMR SeDyL-CELIA Paris, France leglise@vjf.cnrs.fr


Alexander Gepperth

Thomas Hecht

Mathieu Lefort

Biologically inspired incremental learning for high-dimensional spaces

We propose an incremental, highly parallelizable, and constant-time complexity neural learning architecture for multi-class classification (and regression) problems that remains resource-efficient even when the number of input dimensions is very high (≥ 1000). This so-called projection-prediction (PROPRE) architecture is strongly inspired by biological information processing in that it uses a prototype-based, topologically organized hidden layer trained with the SOM learning rule that updates hidden layer weights whenever an error occurs. The SOM learning adapts only the weights of localized neural sub-populations that are similar to the input, which explicitly avoids the catastrophic forgetting effect of MLPs in case new input statistics are presented. The readout layer applies linear regression to hidden layer activities subjected to a transfer function, making the whole system capable of representing strongly non-linear decision boundaries. The resource-efficiency of the algorithm stems from the fact of approximating similarity in the input space by proximity in the SOM layer due to the topological SOM projection property. This avoids the storage of inter-cluster distances (quadratic in number of hidden layer) or input space covariance matrices (quadratic in input dimensions) as K-means, RBF or LWPR would have to do. Tests on the popular MNIST handwritten digit benchmark show that the algorithm compares favorably to state-of-the-art results, and parallelizability is demonstrated by analyzing the efficiency of a parallel GPU implementation of the architecture.

I. INTRODUCTION

Incremental learning remains a challenging issue in machine learning. While it is almost self-evident to biologists that learning should be incremental, the technical realization presents baffling difficulties. First of all, incremental learning is inherently sub-optimal when it comes to optimizing an objective (or loss) function. As one can never assume to have seen all training samples at any single point during training, optimization can only take into account the examples seen up to the present moment. Furthermore, the statistics of inputoutput relations are usually not homogeneous for any finite dataset, so incremental learning must essentially assume nonstationary input statistics at some time scale, which raises the question of how to fuse already learned aspects of a task, without destroying them, with new ones. The latter issue is a real problem for connectionist models of learning [START_REF] Goodfellow | An empirical investigation of catastrophic forgeting in gradientbased neural networks[END_REF] and has been termed "catastrophic forgetting", and it is clear that any feasible incremental learning algorithm needs to avoid this issue.

a) Biological foundations and computational modeling:

As biological incremental learning has reached a high degree of perfection, we explicitly investigated the biological literature for hints as how to this might be achieved. Basing ourselves on observations from the basic sensory cortices, we noted that sensory representations seem to be prototype-based, where prototype-sensitive neurons are topologically arranged by similarity [START_REF] Tanaka | Inferotemporal cortex and object vision[END_REF], [START_REF] David A Leopold | Norm-based face encoding by single neurons in the monkey inferotemporal cortex[END_REF], [START_REF] David A Ross | Not just the norm: Exemplar-based models also predict face aftereffects[END_REF], [START_REF] Erickson | Clustering of perirhinal neurons with similar properties following visual experience in adult monkeys[END_REF]. Learning seems to act on these representations in a task-specific way, where more prototypes are allocated to sensory regions where finer discrimination is necessary [START_REF] Daniel B Polley | Perceptual learning directs auditory cortical map reorganization through top-down influences[END_REF], i.e., where more errors occur during learning. Learning is conceivably enhanced through acetylcholine release in case of task failures [START_REF] Norman | The nucleus basalis and memory codes: Auditory cortical plasticity and the induction of specific, associative behavioral memory[END_REF], [START_REF] Michael | The role of acetylcholine in learning and memory[END_REF], leading to higher "prototype density" in difficult regions of the sensory space. In particular, learning seems to respect and even generate topological layout of prototypes by changing only a small subset of neural selectivities [START_REF] Rolls | The effect of learning on the face selective responses of neurons in the cortex in the superior temporal sulcus of the monkey[END_REF] at each learning event, namely around those neurons that best matched the presented stimulus [START_REF] Erickson | Clustering of perirhinal neurons with similar properties following visual experience in adult monkeys[END_REF].

We model these findings by using a self-organizing map (SOM) learning to shape the feature preferences of hidden layer neurons in our architecture. SOM is a prototype-based algorithm in the sense that the hidden layer weight vectors "live" in the space of inputs in the sense that they are as close as possible to actually occurring inputs according to the SOM energy function (we use a slightly modified SOM model, see [START_REF] Heskes | Energy functions for self-organizing maps[END_REF] that has such a globally decreasing energy function, in contrast to the original model). We model the global, taskrelated error signal by the current classification error that activates SOM learning in case of mismatch. As SOM learning attributes more prototypes to regions where many learning events occur, this will ensure that prototype density increases in difficult regions of the input space. Furthermore, SOM adaptation is stably self-terminating since no more learning will occur when no more errors are made. Inversely, when error rates increase due to the presentation of new input statistics, the hidden layer representation will become plastic until error rates subside again, when a sufficient re-adaptation has been achieved. Thus, the hidden layer represents no longer a pure data distribution but a data distribution modulated by task demands. Lastly, SOM produces a topologically organized representation of the input space, which was the reason to formulate the model in the first place, and modifies weights Fig. 1. Schema of the three-layer PROPRE architecture composed of input, induced and output representations. Initially there is a forward transmission step, propagating the information to the top-level of the hierarchy where it is decided whether a correct result was obtained. In case it was not, SOM weights are updated in the feedback step, thus leading to a representation of difficult samples in N . only locally in case of learning (as observed in biology).

Summarizing, we have tried to incorporate as many facts about incremental learning in biology as possible while keeping the model as simple and efficient as possible. Our modeling takes place at the architectural level, leaving aside the finer details of neural modeling (rate/spike code, dynamic neuron models etc.).

b) Model properties: We propose a three-layer neural model for incremental learning that contains a topologically organized representation of prototypes in its hidden layer (termed "induced representation"), trained by the selforganized map (SOM) algorithm [START_REF] Kohonen | Self-organized formation of topologically correct feature maps[END_REF]. Due to the properties of SOMs, learning is always strictly local in the sense that only the prototypes that are similar to the best-matching one are adapted, thus avoiding catastrophic forgetting. SOM learning is activated by adverse task performance, which conversely means that learning stops once the task is acquired, thus maintaining long-term stability. Classification is performed by simple linear regression from the hidden layer towards a population-coded target vector after first applying a non-linear transfer function to all hidden layer activities.

c) Related work: Incremental learning algorithms are especially interesting for robotics applications [START_REF] Sigaud | On-line regression algorithms for learning mechanical models of robots: a survey[END_REF], and in fact several very interesting proposals have already been made in this context [START_REF] Sigaud | On-line regression algorithms for learning mechanical models of robots: a survey[END_REF]. An especially popular algorithm in robotics is LWPR [START_REF] Vijayakumar | Incremental online learning in high dimensions[END_REF], which partitions the input spaces into receptive fields (RFs), volumes that are defined by a centroid and a covariance matrix, to which separate linear models are applied. Many other incremental algorithms, reviewed in [START_REF] Sigaud | On-line regression algorithms for learning mechanical models of robots: a survey[END_REF] partition the input space in a similar way. Although LWPR (and similar approaches that partition the input space) has been shown to be very powerful, it runs into memory problems when the product of input and output dimensionalities K, P becomes large (e.g., ≥ 10000). This is because each RF requires the storage of a covariance matrix of dimensionality K 2 , and RFs are created independently for each output dimension, thus giving a total memory complexity of K 2 P . This makes LWPR rather unsuited for problems with high input and output dimensionalities. On the other hand, RBF approaches are conceptually very close to ours in that they perform a projection onto a set of prototypes, followed by linear regression. However, incremental learning capacity is lacking on most of these models because the cluster centers ("prototypes" in our architecture) are fixed in advance, based on training data. Furthermore, RBF approaches have no notion of topology preservation when creating cluster centers; whereas our approach essentially amounts to clustering as well, the topology preserving properties of the self-organizing map model are used to approximate input space distances by distances in the projected space, which allows for efficient incremental learning. d) Contribution of this article: In this article, we will propose a model for incremental learning that can cope with scenarios where KM ∼ 10000 (K, M denoting input/output dimensionality) and beyond, and evaluate its performance on the well-known MNIST benchmark [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. We explicitly evaluates the incremental aspect of learning by training on a subset of MNIST classes and subsequently adding the remaining classes. The key idea of our approach is to approximate distances in the high-dimensional input space by grid distances in the projected input space (the hidden layer of our architecture) which is a key property of the SOM algorithm we employ for this purpose. In this way, learning can be fully incremental by restricting weight changes to a small hypervolume around the current input without having to store RF centers and covariance matrices. Furthermore, the architecture we present has constant time complexity and is fully and naturally parallelizable, which we demonstrate by execution speed measurements using a separate GPU implementation 1 .

II. METHODS

A. The PROPRE architecture

PROPRE is an architecture composed of different algorithmic modules, rather than an algorithm in itself. One PROPRE iteration consists of the following steps, as described in [START_REF] Gepperth | Efficient online bootstrapping of representations[END_REF], where only the computation of the predictability measure λ is changed to represent the current binary classification error: input: new data is fed into the input representation I and provided to the SOM, and a new target representation T is provided projection: activity is formed in the induced representation N (see Fig. 1) by projection of I onto the SOM prototypes prediction: based on activity in N , a linear regression step is performed to produce representation P which predicts class membership evaluation: a mismatch measure is computed between P and T update: linear regression weights are updated. SOM weights are updated only if mismatch was detected. In mathematical terms, the whole model is governed by the following equations, where we denote neural activity at position y = (a, b) in a 2D representation X by z X ( y, t) and weight matrices for SOM and LR, represented by their line vectors attached to target position y = (a, b) by w SOM y :

zN ( y, t) = w SOM y (t) -z I (t) (1) zN ( y, t) = g κ zN z N ( y, t) = TF zN ( y, t) z P ( y, t) = w LR y (t) • TF z N (t) λ(t) = 0 if argmax y z P ( y, t) = argmax y z T ( y, t) 1 else w LR y (t + 1) = w LR y (t) + 2 LR z I (t) z P (t) -z T (t) w SOM y (t + 1) = w SOM y (t) + λ(t) SOM g σ ( y -y * )(z I -w SOM y ) κ(t + 1) = 0.999 κ(t) + 0.001 max y zN ( y, t)
where g s (x) is a zero-mean Gaussian function with standard deviation s and y * denotes the position of the best-matching unit (the one with the highest activity) in N . In accordance with standard SOM training practices, the SOM learning rate and radius, SOM and σ, start at 0 , σ 0 and are exponentially decreased in order to attain their long-term values ∞ , σ ∞ at t = T conv . In order to convert euclidean distance between a prototype vector and the input into a similarity score in the interval [0, 1], we pass the "naked" distances zN through a zero-mean Gaussian function with a standard deviation κ(t) that adapts to the average maximal distance found in the whole induced representation. This amounts to determining, over time, the expected maximal distance to which a small score should be assigned, whereas the highest score would always be assigned to a distance of 0. TF represents a monotonous non-linear transfer function, TF : [0, 1] → [0, 1] which we model as follows with the goal of maintaining the BMU value unchanged while gradually and nonlinearly suppressing all other values:

m 0 = max y zN ( y, t) m 1 = max y zN ( y, t) 20 TF (z N ( y) = m 0 z N ( y) 20 m 1 (2)
A softmax function would have done the trick as well, but we avoid this in order not to calculate too many exponentials.

B. The MNIST handwritten digit database

For all experiments, we use he publicly available MNIST classification benchmark as described in [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. It contains 10 classes, corresponding to the 10 handwritten digits from "0" to "9", see also Fig. 1, and comes separated into a well-defined train set and a smaller test set. Each sample has a dimensionality of K = 28 × 28 = 784. From the MNIST benchmark, we extract several subsets of classes: D0 containing the digits from 1 to 9, and D 0 containing just the digit "0". Analogously, we create D1, D2, D

III. EXPERIMENTS

We use the following fixed parameters for PROPRE: LR = 0.001, 0 = 0.5, σ 0 = 0.3n, T 1 = 50000, T conv = 100000,

∞ = 0.

A. Baseline performance measurement

In order to establish a baseline performance that demonstrates the principal capability of the PROPRE architecture to solve the classification problem posed by MNIST, we first train and evaluate the PROPRE architecture for 10 6 iterations on D 0-9 . Modulation of SOM learning is turned off by setting λ(t) ≡ 1 in eqns.(1) as this is not in incremental learning task. The performance thus obtained is to be compared to offline, batch-type algorithms. In the case of PROPRE, this would be other three-layer architectures such as multilayer perceptron or RBF networks. In particular, a goal of this experiment is to find a value of ∞ that will give maximal performance in this non-incremental setting. This is an important point as the capacity of the hidden layer to represent inputs as closely as possible is intimately tied to this parameter: the smaller it is, the smaller will be the average prototype-input distance expressed by the similarity score of the BMU, and it can be reasonably speculated that this is in turn related to classification performance of the linear regression readout. We will at the same time vary the hidden layer size n 2 , again with the goal of maximizing classification performance, where it is again reasonable to suppose that bigger hidden layers will give better classification as the inputs can be approximated with a higher overall resolution.

The results of this set of experiments are summarized in Table I. The show mainly two things:

• Smaller ∞ leads to better classification performance • Bigger hidden layer sizes lead to better classification performance Especially the latter result, while not really surprising, is interesting, as it suggests that in the case of the PROPRE architecture, we do not suffer from the problem of choosing a correct hidden layer size as in the case of multilayer perceptrons (MLPs). As the hidden layer projections are shaped by an energy-based variant of the generative SOM learning

h h h h h h h h h h h h h

B. Incremental learning performance

We conduct several experiments designed to measure the capability to perform incremental learning. To this effect, we let the architecture converge on the datasets D train 0 , D train 1 or D train 2 for 300.000 iterations, keeping all timing parameters like T 1 and T conv unchanged from the baseline experiment, and performing the same reduction of learning rates and neighbourhood radius. The modulation factor is kept at λ(t) ≡ 1 for t < 300.000 in order to have a defined starting point, and is determined according to eqns.(1) for t ≥ 300.000. From t = 300.000 onwards, we present one of the complementary datasets D train 0 , D train 1 or D train 2 for 20000 iterations, followed by a phase of 30.000 iterations where SOM plasticity is turned off (λ(t) ≡ 0) in order to let linear regression weights "catch up" with the changes to the hidden layer selectivities. At all times, we can measure model performance on any of the corresponding test sets.

e) Results: The numerical results of these experiments, conducted for σ inf = 0.05, are given in Tab. II. They clearly show that incremental learning is successful, as the newly added class is well learned while performance on the "old" classes is retained with little change, which is also reflected in the fact that the overall error on D test 0-9 increases only insignificantly. As all classes are present in equal frequency in the MNIST benchmark, an inferior performance on the newly added class could raise error rates by up to 10 percent which is not observed.

We observe what happens when the modulation factor λ(t) in eqns.( 1) is kept fixed at lambda ≡ 1 meaning that the SOM weights are updated at every iteration, regardless of errors in classification. Although errors on the new class drop very quickly, errors on the "old" classes rise much more quickly and unpleasantly than in the case where λ(t) is determined from current classification accuracy according to eqns. [START_REF] Goodfellow | An empirical investigation of catastrophic forgeting in gradientbased neural networks[END_REF]. This is because eqns.(1) update SOM weights only when misclassifications occur, or conversely, do not adapt anything when classifications are accurate. When presenting a new class, initially all classifications will be incorrect and strong adaptation occurs. After having learned a sufficiently good representation in the hidden layer, adaptation of SOM weights largely stops, which protects the old classes regardless of how long the new class is actually presented. Briefly put, the learning architecture we presents adapts its hidden layer selectivities only as much as necessary and no more.

We furthermore observe that the value of σ ∞ seems to control the incremental learning capacity of the architecture: if it is too large, adaptation is too fast and the old classes will be completely overwritten before all samples of the new class have even been fully presented. conversely, if it is too small, changing input statistics are incorporated too slowly to play a role within the considered 20.000 retraining iterations. This is natural since the SOM algorithm guarantees a graceful decay whose time scale is however controlled by σ ∞ . Fig. 2 shows the effect of large, small and just correct values for σ ∞ .

C. Parallelization and complexity issues

Apart from the very favorable time and memory complexity of the PROPRE architecture, all of its component parts can be very efficiently parallelized. Here we focus predominantly on the SOM layer as it produces the highest computational burden. We can deduce from eqn. (1) that both the projection as well as the weight adaptation part can be performed in parallel for each output neuron. This is evident for the projection step that produces z N , as well as for the weight adaptation step that just makes use of the activities z N computed during the projection step but not of the weight vectors (SOM prototypes) of other hidden layer neurons. In fact, in a parallel implementation we can go even further and stop weight adaptation for a particular hidden layer neuron when it is too far away from the best-matching unit (BMU), i.e., when the neighbourhood function g σ falls below a certain threshold for which we have taken a very conservative value of 10 -2 . Since the weight change becomes negligible by multiplying with the current learning rate (<< 1) this is a very justified approximation. For a purely CPU-based implementation without parallelism, this enormously speeds up computations when the neighbourhood radius has converged to its long-term level of σ ∞ , as effectively only a very small fraction of the total set of prototypes is adapted at each time step, namely those that are very close to the BMU. For a parallel GPU implementation, the benefit of this approximation depends Fig. 2. Effect of incremental learning on SOM prototypes. Shown is a 30x30 grid of prototypes in the form of little images of size 28x28, the size of one MNIST sample. Left: SOM converged on D train 0 at t = 300.000, σ∞ = 0.05. The similarity-based ordering of prototypes in the typical fashion of a SOM is well discernible. Middle: SOM with addition of the new class "0" at t = 320.000. We can clearly observe that the new class is inserted into the SOM at positions where it most closely matches existing prototypes (e.g., bottom left corner), and that prototypes dissimilar to the new "0" class (e.g., class "1" in the upper right corner) are not affected at all. Right: SOM with addition of new class at t = 320.000, with a different value of σ∞ = 0.75. We observe that insertions are now much more spatially organized by similarity, and that the prototypes representing the new "0" class are much more frequent, showing that more of the "old" knowledge has been overwritten. strongly on the actual way of distributing computations on the available multiprocessors, but is less pronounced for modern GPUs with high parallelism, because then each parallel process computes very little (typically 1-2 additions) and the difference of aborting or executing it is small.

The GPU implementation was done using CUDA on a GeForce GTX 570 graphics card (a rather old model) on a rather old computer (Pentium Quad-Core of 2GHz running Linux). The actual implementation was performed in Python using the pyCUDA library [START_REF] Klöckner | PyCUDA and PyOpenCL: A Scripting-Based Approach to GPU Run-Time Code Generation[END_REF]. The reference implementation was done in Python as well using the numpy library [START_REF] Van Der Walt | The numpy array: A structure for efficient numerical computation[END_REF] which is optimized for vectorization, so the approximation mentioned above was not employed. Obtained execution times are summarized in Tab. III. They show a very large performance difference between CPU and GPU implementations, on the one hand in absolute terms and on the other hand in terms of scaling: whereas CPU execution times scale exactly with the change in hidden layer size, the GPU execution times grow much more slowly and allow therefore to simulate much larger architectures. It is clear that this behavior will saturate at some point, but as the whole point of this experiments was to show to what extent and with what facility PROPRE can be parallelized, this is not a crucial issue.

IV. DISCUSSION f) Complexity: Incremental learning using the PROPRE architecture comes at constant time complexity; this is in contrast to conventional incremental learning algorithms such as LWPR which allocate "receptive fields" at runtime as needed, and whose time complexity is linear in the number of these structures. In a sense, PROPRE "pre-allocates" a certain number of "receptive fields" and uses them as well as it can, where having more receptive fields means better classification accuracy. Denoting input dimensionality, hidden layer size and output layer size (number of classes) by K,N and P, the memory complexity of PROPRE is roughly (KN + N P ) = N (K + P ) ≈ N K. For a pure RBF classifier with N cluster centers, the memory complexity would be N (K + P ) ≈ N K as well. However, if cluster centers should need to be updated in an efficient fashion, it will be necessary to store a matrix of inter-cluster distances so that each new sample can update the clusters to which it is nearest in the input space. This matrix will have N 2 entries, making the total memory complexity N (N + K + P ≈ N (N + K) which can be formidable for a large number of clusters. For the LWPR algorithm, the storage of receptive fields that are defined in the input space requires approximately P N (K+4K 2 ) ≈ 4P N K 2 which becomes prohibitive for large input dimensionalities K. The factor 4 in the last expression comes from the storage of sufficient data statistics along with each receptive field as detailed by [START_REF] Vijayakumar | A library for locally weighted projection regression[END_REF]. g) Is this really incremental learning?: in this article, we show how we can, in an additional training step, teach new things to our architecture without forgetting too much of previously learned knowledge. Where forgetting happens, it has a certain graceful decay property that is characteristic for the SOM model. However, in order to teach "new tricks" to the architecture, we perform a dedicated incremental learning procedure that is different from the initial learning procedure: fist, we present the new concept in the form of examples, and perform a subsequent linear regression retraining where SOM learning is deactivated. Each time something new (e.g., a class) is added, this step has to be repeated. This does not pose problems in practice, but from a conceptual point of view it would be much more elegant to perform incremental learning identically to the initial learning step. Already, the term "incremental learning" is not well defined in the literature, but if we stick to the terms defined in [START_REF] Sigaud | On-line regression algorithms for learning mechanical models of robots: a survey[END_REF], we can say that our method is incremental but not fully online. h) Incremental vs. non-incremental performance: We observe in all experiments that incremental learning incurs a slight cost in the form of a departure from the non-incremental version of the PROPRE architecture. This is not very surprising, first of all as the online linear regression we use to read out hidden layer activities may not yet be fully converged, but mainly because the addition of a new class to an already learned model is inherently sub-optimal (initial learning does not know at all about the new classes).

i) Influence of σ ∞ : We found a strong impact of σ ∞ on map formation in the hidden layer which in turn strongly influences classification accuracy. As suggested by Fig. 2, this parameter also governs the way incremental learning adapts SOM prototypes. We believe that further work will be necessary to elucidate the precise role of this parameter.

V. CONCLUSION

We have presented an algorithm for resource-efficient incremental learning that draws its efficiency from principles of biological information processing and showed that it can easily handle data dimensionalities of 750 entries. We showed that it compares favorably with the state-of-the-art on a standard machine learning benchmark (MNIST) in its non-incremental form, and that good classification accuracy persists when training it incrementally by adding classes to a trained model using the same benchmark. The algorithm is self-limiting and destroys old knowledge only "on demand", due to misclassifications. Due to neural design principles, the whole architecture can be very easily parallelized, obtaining performance gains up to a factor of 8, along with a much nicer scaling behavior when the hidden layer size is increased. Future work will include "deep" PROPRE architectures, namely investigating how the incremental learning capacity can be maintained in such an architecture, as well as efforts to make the PROPRE architecture fully online, which means that training and retraining steps should be conducted in the exact same fashion, thus assuring maximal simplicity and applicability in many different applied scenarios where autonomous, uncontrolled learning is required.

  001 and σ ∞ = 0.5. Both SOM and LR weight matrices are initialized to random uniform values between -0.001 and 0.001. No preprocessing is performed on the 28x28-dimensional MNIST input vectors. Training examples are always randomly and uniformly drawn from the current training set.

TABLE I BASELINE

 I 1 and D 2 . Each set D x is again split into training and test sets D train NON-INCREMENTAL PERFORMANCE EVALUATION OF PROPRE ON MNIST DATA WHILE VARYING THE MINIMAL SOM NEIGHBOURHOOD RADIUS AND THE HIDDEN LAYER SIZE. GIVEN ARE THE ERRORS ON THE MNIST TEST SET IN PERCENT.

	P n P	P P 10	∞ P P	2 32.9 17.9 10.3 8.5 1 0.5 0.1 0.05 8.6
		20		17.0 10.0	6.3	5.5	5.4
		30		12.8	7.2	5.5	4.8	4.8
		50		7.9	5.4	4.8	4.4	4.6

x , D test x to measure generalization performance, the split being made according to whether a certain sample belongs to the MNIST train or test set. For training and evaluating performance on all digits, we also create the sets D train 0-9 , D test 0-9 which correspond to the original MNIST train and test sets.

TABLE III EXECUTION

 III TIME MEASUREMENTS IN SECONDS PER 10 4 ITERATIONS OF PROPRE FOR GPU AND CPU IMPLEMENTATIONS. IT CAN BE OBSERVE THAT GPU EXECUTION TIMES SCALE (MUCH) LESS THAN LINEARLY IN THE CONSIDERED RANGE OF HIDDEN LAYER SIZES, WHEREAS THE CPU IMPLEMENTATION SCALES ALMOST EXACTLY IN A LINEAR FASHION.

	P time P P GPU P	N P P	10x10 20x20 30x30 50x50 15 19 27 53
	CPU		115	457	1037	2903

All simulation code will be made available for download


Thomas Hecht
email: thomas.hecht@ensta-paristech.fr

Alexander Gepperth
email: alexander.gepperth@ensta-paristech.fr

A generative-discriminative learning model for noisy information fusion

This article is concerned with the acquisition of multimodal integration capacities by learning algorithms. Humans seem to perform statistically optimal fusion, and this ability may be gradually learned from experience. In order to stress the advantage of learning approaches in contrast to hand-coded models, we propose a generative-discriminative learning architecture that avoids simplifying assumptions on prior distributions and copes with realistic relationships between observations and underlying values. We base our investigation on a simple selforganized approach, for which we show statistical near-optimality properties by explicit comparison to an equivalent Bayesian model on a realistic artificial dataset.

I. INTRODUCTION

Autonomous agents must possess the ability to acquire knowledge from multiple sensors in an unsupervised way. Similarly, proper multi-sensor fusion is a necessity for any biological organism [START_REF] Stein | The merging of the senses[END_REF], and it seems that, under certain conditions, humans and other animals can perform statistically optimal multisensory integration [START_REF] Calvert | Multisensory integration: methodological approaches and emerging principles in the human brain[END_REF]. As to how this is achieved, many questions remain : mainly, one can speculate whether there is a generic, sensor-independent fusion mechanism taking into account the basic statistical laws such as Bayes' rule at some neural level, or whether optimal fusion, where it occurs, is something that can be learned.

In this article we present an argument for the latter case since it stands to reason that multisensory integration in biological systems is not generally innate but learned [START_REF] Bauer | Attention modeled as information in learning multisensory integration[END_REF]. This ability seems to be gradually acquired in the course of development, and then refined and maintained throughout a life-span which would obviously be desirable for intelligent agents to reproduce. Thus not only the question of how multisensory integration is carried out is of importance, but also of how it is acquired. As we believe that the acquisition and maintenance of fusion skills is a life-long task, we propose a neural model capable of performing stable multisensor fusion without assumptions about noise variances. Our approach uses a hybrid architecture combining an unsupervised self-organizing map (SOM) layer with a supervised linear regression layer.

This article wishes to demonstrate that a SOM can learn a viable representation of a given data distribution given by Thomas Hecht gratefully acknowledges funding support by the Direction Générale de l'Armement (DGA-MRIS scholarship) and École Polytechnique. examples thus laying the basis for an optimal fusion decision taken by a subsequent linear regression module. We put emphasis of advantages of such a generative-discriminative model by showing that the discriminative ability to focus on a given-task best performance can be modulated by underlying statistics of data distribution captured by the unsupervised generative model. We compare our system final modulated decision with the well-known Bayesian fusion optimum.

II. RELATED WORK

Also called "multisensory data fusion", multimodal integration aims at providing a robust and unified representation of the environment based on multiple sensors inputs [START_REF] Ernst | Merging the senses into a robust percept[END_REF]. In mammalian brains, this process seems to be implied in maximizing information gathering and reliability by the effective use of a set of available observations from different sensors [START_REF] Angelaki | Multisensory integration: psychophysics, neurophysiology, and computation[END_REF]. It has been widely studied at different levels (i.e. from particular cortical cells to individual behaviour) and into different scientific fields (e.g. neurophysiology, neuroimagery, psychophysics or neurobiology).

There are many reasons for believing that self-organization is a fundamental mechanism used in the brain [START_REF] Kelso | Dynamic patterns: The self-organization of brain and behavior[END_REF]. Studies dealing with self-organizing artificial neural networks and multimodal fusion are usually based on properties one can find in biology : continuous unsupervised processes, adaptation and plasticity, topological preservation of the input space relationship or dimensionality reduction. Self-organized approaches have the potential to establish a transition from high-dimensional, noisy and modality-specific sensor readings to abstract, multimodal and symbolic concepts, whereas they are considered less appropriate for reproducing statistical optimality which should be respected by any integration process. Some recent papers used SOM or SOM-like bio-imitating architectures in order to reproduce individual behaviours or biological phenomena by imitating, more or less clumsily, hierarchical layered interconnected cortical areas, especially superior colliculus ( [START_REF] Martin | Modeling multisensory enhancement with self-organizing maps[END_REF] and [START_REF] Anastasio | A two-stage unsupervised learning algorithm reproduces multisensory enhancement in a neural network model of the corticotectal system[END_REF] which place emphasis on the positive impact of a non-linear transfer function applied to neural maps and [START_REF] Pavlou | Simulating the effects of cortical feedback in the superior colliculus with topographic maps[END_REF], [START_REF] Mayor | A neurocomputational account of taxonomic responding and fast mapping in early word learning[END_REF] and [START_REF] Bauer | A som-based model for multisensory integration in the superior colliculus[END_REF] which deal with imitating SC multisensory integration). Several studies aim at designing models inspired by recent neurophysiological findings without fully copying nature architectures or processes, focusing on well-defined applications like information retrieval [START_REF] Georgakis | An ensemble of SOM networks for document organization and retrieval[END_REF], visual categorisation [START_REF] Gliozzi | The impact of labels on visual categorisation: A neural network model[END_REF], data visualisation [START_REF] Baruque | A bio-inspired fusion method for data visualization[END_REF] or audio-visual multimodal integration [START_REF] Jantvik | A self-organized artificial neural network architecture for sensory integration with applications to letter-phoneme integration[END_REF].

III. METHODS

A. Hybrid model

In this section, we detail components and notations of the proposed neural architecture. Our bio-inspired hybrid learning method, depicted in Fig. 1, is presented noisy observations s i , i = 0 . . . N -1, obtained one after the other from N sensors, where the dependence of observations on the underlying "true" sensor value r is described by a probabilistic noise model (throughout the article, we will use N = 2 sensors to keep things simple). In order to increase the computational capacities of the architecture, observations and the "true" underlying value r are represented using a technique variously termed "population encoding" or "basis function representation", described in Sec. III-B. From the set of population-coded observations, the SOM algorithm III-C creates a combinatorial internal representation capturing key statistical properties of observations. Each neuron-like unit in this representation preferentially responds to certain input stimuli, the generative aspect of the SOM algorithm ensuring that common combinations cause strong responses and uncommon ones weak responses. Subsequently, a decoder module maps the internal representation to the populationcoded "true value" r by using a multiple linear regression (MLR) algorithm (see Sec. III-D) resulting in an estimate r * nn . Learning the decoder function happens in a supervised fashion using the known underlying values r. To verify that the internal representation indeed captures important statistical properties relevant to the fusion process, an alternative fusion mechanism that linearly maps sensory inputs to true values is investigated as well, producing estimates denotes r * alt . The comparison of population-coded estimates and true values is performed after a decoding step using the mean squared error (MSE) measure.

B. Population encoding for raw sensor values

Population coding or "basis function encoding" allows a common representational format based on biological observations [START_REF] Gepperth | The contribution of context information: a case study of object recognition in an intelligent car[END_REF] : this kind of discrete information coding occurs at the sensory input areas of the brain [START_REF] Pouget | A computational perspective on the neural basis of multisensory spatial representations[END_REF], [START_REF] Pouget | Inference and computation with population codes[END_REF]. A quantity is encoded as an "activation" on a two-dimensional surface, where the position and amplitude of the activation "blob" respectively code for value and confidence [START_REF] Ortiz | Autonomous generation of internal representations for associative learning[END_REF].

We use a simpler version of the population coding idea in order to enrich the representation of the raw unimodal sensor measurement by projecting scalars from a specific range into a unified two-dimensional grid of l × c of pixels : values are still encoded by placing a 2D Gaussian centered on a determined position but we do not encode the confidence part of the quantity into the block. The main reason why we do so is because the very point of our system is to identify the variability in data, whatever distribution it comes from. The encoding process translates a continuous variable from its own continuous range to a discrete position into a discrete space. Then a 2D Gaussian blob with given x-and y-axis variance is placed at this position. To prevent the center of the blob from exceeding the discrete border (e.g. because of the addition of noise to the value to encode) we usually set a margin δ which is used in the discretization process : with a and b being the continuous borders and c and d the discrete borders, the translation

[a, b] → [c; d] is replaced by [a, b] → [c + δ; d -δ].

C. Kohonen's Self-Organizing Map

The Kohonen's Self-Organizing Map (SOM) algorithm is a topology-based ("nearby patterns should be mapped by nearby neurons") unsupervised clustering method which combines biological inspiration and a solid mathematical foundation. The algorithm assumes the existence of p × q competing units receiving m-dimensional input vectors x i ∈ X and implements a winner-takes-most learning strategy. Each unit u j has an associated codebook vector or prototype w j which is adapted over time. For each iteration t, an input sample x t is randomly picked from X and the learning algorithm determines the best-matching unit (BMU). Usually, the BMU is found by applying a distance function. In this article, we use the so-called dot-product maps [START_REF] Kohonen | Essentials of the self-organizing map[END_REF] in which codebook vectors w j are compared to the current input sample x t through the cosine similarity. The current BMU corresponds to the unit for which w t = argmax j sim(x t , w j ) (x t and w j are kept normalized to constant length). The grid then moves its prototype a certain proportion η t of this similarity closer to x t . A set of adjoining codebook vectors around the BMU are also moved closer to the current training case. This set of "adjoining" units is determined by a kernel function φ(w a , w b , σ t ) called neighborhood function which tells for a codebook w a vector whether it stays in the influence area of another codebook vector w b , with respect to an influence radius σ t . Every w i ∈ W is updated with the following rule :

w t+1 i ← w t i + η t φ(w t i , w , σ t )(x t -w ).
The result of the algorithm is the mapping of the grid of codebook vectors to the underlying structure of the input samples in a low-dimensional projection. At each step t, the similarity of a unit's codebook vector to the input sample defines that unit's activation a t i . The SOM output activity o t i is the activity seen from modules following the SOM in the architecture, here the MLR module. It is defined by applying an output function Φ(•) to the map activations (here, Φ and its parameters are the same for all units). The importance of this output function has been studied in [START_REF] Hecht | Using self-organizing maps for regression: the importance of the output function[END_REF]. In this article, because we use a cosine similarity, Φ is a power function for which each unit activity and defined as :

o t i = Φ(a t i , p) = {a t i } p D.

Linear regression model

Applied to to real-valued target functions, the linear regression model explains a random continuous dependent target variable y by a linear combination of K explanatory random independent known variables x 1 , x 2 , . . . , x K . This model can be formally defined by:

y = β 0 x 0 + β 1 x 1 + β 2 x 2 + . . . + β K x K +
with β 0 , β 1 , . . . , β K some parameters and the normally distributed disturbance term and x 0 = 1.

If the data set consists of M pairs (x i , y i ) with x i = (x 0 , x 1 , . . . , x K ) T , the model looks for the optimal weight vector w = (β 0 , β 1 , . . . , β K ) in terms of the squared error (SE). Because we wish to work in an on-line fashion, we choose an iterative method to minimize the cost function SE(•). The Widrow-Hoff algorithm applies stochastic gradient descent techniques to the linear regression objective function. The error is not computed for all data points any more but for each individual example. It becomes, for the i-th update of the linear model weights:

SE on-line i (w) = 1 2 (h w (x i ) -y i ) 2
with h w (x n ) = w T x n the value of the hypothesis function attached to a certain w.

E. Data generation

During experiments, sensor values, target values and relations between them belong to two "families" of data generation : in one case (data generation Type 1) the system receives two values built by adding noise directly to a single random target value; in the other case (data generation Type 2), the system generates two independent random values, which are corrupted by noise, and the target value is a linear or non-linear combination of these two values.

1) Data generation Type 1: The first "family" of input data, so-called data generation Type 1 follows the "classic" Bayesian framework : a single "true" value r that gives rise to several noisy sensor readings s i . We suppose that the sensor readings s i are generated independently from each other from a unique r value by additive Gaussian noise ˜ .

r ∼ p U a,b (x) s i = r + ˜ i ˜ ∼ P N µ,σ (x)
2) Data generation Type 2: A more realistic setting is where the sensory readings s i and the underlying true value r are more tightly coupled. In this second "family" of input data, so-called data generation type 2, we suppose that the s i are no longer class-conditionally independent and depend on r as well as each other. In other words, all of the s i have to be considered simultaneously for inferring r. Formally, we express this by drawing the si independently from a bounded uniform distribution, and then making r a function of the si : r = f (s 1 , s2 , . . . ). Afterwards, the si are subjected to additive Gaussian noise in order to produce the measured values s i :

si ∼ p U a,b (x) r = f (s 1 , s2 , . . . ) ˜ i ∼ P N µ,σ (x) s i = si + ˜ i

F. Extending the classic Bayesian integration framework

In this section, we will present Bayesian multisensory fusion in a setting that is close to real-life operation. In particular, we will drop the common assumption of r being unbounded and introduce an interval r ∈ [a, b] into the Bayesian calculations. In the most generic form, Bayesian sensor fusion aims to find the most probable value of r given the observations s:

r * = arg max r p(r|s) ∼ arg max r p(s|r)p(r) (1)
This amounts to a maximization problem, putting the first derivative with respect to r to 0 yields the necessary condition:

∂ r p(s|r)p(r) = 0 ⇔ ∂ r (p(s|r)) p(r) + p(s|r)∂ s p(r) = 0 (2)
Eqn.

(2) has trivial solutions outside the interval [a, b] where both p(s) and ∂ s p(r) vanish. However they minimize p(s|r)p(r) (inserting an appropriate r always gives a value of 0), and are thus excluded from our considerations.

If, however, a solution exists inside [a, b], it must obey the simplified equation

∂ s (p(s|r)) = 0 (3)
On the other hand, if eqn.( 3) has a non-trivial solution outside the interval [a, b] then it must be either s = a or s = b, depending on which is closer, because the infinities in the derivatives of p(r) achieve a "clamping" of obtained fusion results to the known interval [a, b]. This can be implemented very efficiently, without solving any equations at all, as a postprocessing step of fusion.

Please bear in mind that we do not make any assumptions about the conditional distributions p(s|r) which may be defined by additive noise or other models, both of which we will discuss in next paragraphs.

1) Dealing with Data generation Type 2: The only tricky point consists here in computing the quantity p(s|r) required by eqn. [START_REF] Calvert | Multisensory integration: methodological approaches and emerging principles in the human brain[END_REF]. As the sensor measurements s i no longer directly depend on r but on si , the calculation is a little more cumbersome, especially since the factorization p(s|r) = Π i p(s i |r) no longer holds. For a simplified setting of two sensors we obtain:

p(s 1 s 2 |r) = ds 1 ds 2 p(s 1 s 2 |s 1 s2 r)p(s 1 s2 |r) = ds 1 ds 2 p(s 1 s 2 |s 1 s2 )p(s 1 s2 |r) = ds 1 ds 2 p(s 1 |s 1 )p(s 2 |s 2 )δ(f (s 1 , s2 ) -r) = ds 1 p(s 1 |s 1 )p(s 2 |f -1 s2 (s 1 , r)) ( 4
)
where the first transformation follows from the law of total probability: we insert a complete set of disjoint states s1 s2 . In the second line, the factor r has been removed from the conditional probability p(s 1 s 2 |s 1 s2 r) as it can be deduced from s1 and s2 . Later, the conditional probability has been split as s i depends only on si . The function f -1 s2 (s 1 , r) is the function obtained by solving f (s 1 , s 2 , . . . ) = r for s 2 . The optimal fused value of r in the interval [a, b] is obtained as before by maximizing eqn. ( 4). As the resulting expression is in general intractable analytically, we resort to numerical methods to solve it for s. It should be stressed that eqn. ( 4) can be derived for any number of sensors although its numerical solution will get more and more costly.

IV. EXPERIMENTS For all the experiments described in this section, the case two virtual sensors is considered. We assume that they both observe the same real phenomenon in the environment, which is represented by a single real number. We demonstrate the abilities of our generative-discriminative system by comparing its performances with optimal Bayesian fusion which is not learned but uses prior knowledge of noise type and properties [START_REF] Ernst | A bayesian view on multimodal cue integration[END_REF] : the Bayesian optimal fusion weights are statistically relevant but, on the other hand, specifically need prior estimate of sensor variances. Our common representation space is able to represent the statistical properties of all signals sufficiently well to allow for correct fusion.

A. Set up

Whatever the type of data generation, each experiment we conduct in this section varies the variance of each artificial sensor and measures how efficient fusion can be for a given combination. For simplicity, we use two simulated sensors with five possible standard deviations : for the k-th sensor, σ k ∈ [0.01, 0.0575, 0.105, 0.1525, 0.2].

Each target and noisy sensors values are respectively spatially encoded in shape of a Gaussian blob located in a 1 × 200 bunch of pixels (this bunch represents the discrete range [0; 100]) with an integrated margin δ = 3 as described in Sec. III-B.

The common representation space takes the shape of a 20 × 20 grid. The 400 corresponding codebook vectors are initialized uniformly at random and then updated following Kohonen's learning algorithm with a Gaussian neighborhood function and iteration-indexed decreasing neighborhood radius and learning rate (both with exponential decay), as described in Sec. III-C. For a given data generation type, for each of the 25 standard deviations couples and each of the 3 noise models, a SOM is trained during 100000 iterations on available spatially encoded training samples : at each step the SOM tries to map an input vector ξ t which is the concatenation of noisy s t 1 and s t 2 , with 1 ≤ t ≤ 100000.

After the first 100000 first steps, a MLR iteratively brings face to face the SOM output (with p = 25 for the power output function) with the spatially encoded target value. This interval of 100000 steps is not mandatory but allows a faster convergence of the MLR because it receives output from a quasi-stable learned organisation in the SOM. The MLR uses a learning rate α = 1 × 10 -4 .

At the end, the MLR prediction abilities are tested on 10000 never-seen sensor and target values with mean-squared error and compared with Bayesian (or extended Bayesian) predictions and MLR predictions based directly on encoded sensors couples.

B. Results

For data generated by "data generation type 1", with additive Gaussian noise for bounded target value, results show that the proposed architecture reaches classic and extended Bayesian frameworks results (Fig. 2). Because "data generation type 1" deals with linear combinations of sensor values weighted by sensors noise variance, a single MLR module can achieve such performances, on condition that there are enough available training observations (we verified that MSE predictions are not significantly different from Bayesian predictions). The added SOM module with a non-linear output function does not damage the performances of the MLR module. Since results are near-optimal, we do not show the effect of a fusion decision modulation here.

For data generated by "data generation type 2", with strongly coupled sensors with additive Gaussian noise, results show that the proposed architecture works far better than only with the MLR module (note that, in these cases, MSE performances are out of range in presented figures). Target value and sensors values are not class-conditionally independent and there is no direct linear link between the latter and the former anymore. The learning phase provides a self-organized space which preserves encoded noisy sensors couples topological relationships and allows reduced representation of observations. The MLR module then receives a non-linear projection of the map activation -comparable to a kernel trick over cosine similarities -which allows a better prediction of the real target value.

Because the generative part of our system tries to represent data distribution statistics, it can calculate how probable an observation of encoded sensors values actually is. We apply the following simple heuristic to exclude outliers which do not permit sensible fusion results : the regression fusion decision is taken into account only if the current BMU score is greater than the median of past BMU scores. More complex heuristics could be used as thresholds to improve performance. But this kind of filter explicitly shows that the SOM module autonomously catches intrinsic variances of noisy sensors values, and that we can use this information to remove outliers. This very point is a huge advantage of the proposed architecture and improves results especially in case of realistic relationships between sensors values and true values (Fig. 3a and3b).

V. DISCUSSION

For a typical Bayesian approach, the distribution of the true stimulus, the noise models and the variances of the problem must be known beforehand in order to perform inference. In our approach, such parameters are implicitly estimated from data using a generative learning algorithm (SOM). Then, in related modeling approaches on multisensory fusion [START_REF] Bauer | Attention modeled as information in learning multisensory integration[END_REF], it is assumed that the distribution of the underlying "true" stimulus r, p(r) is uniform and unbounded : p(r) ∼ U(∞, ∞), and that observations s i for each sensor are obtained from r by adding Gaussian noise with a variance that is known for each sensor. Individual observations are therefore class-conditionally independent. In this article, we treat Bayesian multisensory integration in a way much closer to real experiments. We introduce a finite interval [a, b] to which r is constrained which introduces additional complexity into the Bayesian formulas. Lastly, the models according to which observations s i are generated from the true stimulus r are considerably more complex, and above all individual observations need no longer be class-conditionally independent.

When a dependant linear link between target value and sensors values exists ("data generation type 1"), our system is able to reach good performances -especially thanks to the MLR module -which are comparable to Bayesian optima. When this link is more realistic and potentially non-linear ("data generation type 2"), the MLR module needs to rely on the non-linear SOM output to achieve performance close to extended Bayesian framework.

We can significantly improve our system results by filtering fusion decisions with available quality measures based on BMU scores. Without this fusion decision modulation, the MLR module always combines sensors values, as the classic Bayesian fusion formula does. Without knowing explicit information about data distributions or noise variances, the proposed architecture can perform near-optimal performances on realistic fusion tasks.

VI. FUTURE WORK

The hybrid architecture presented here still needs to have ground truths available for the discriminative part of it. A step towards more autonomy lies in using one cue as ground truth (a) Self-organizin map BMU test scores (cosine distances) (b) Online regression test errors Fig. 4: Measuring incremental abilities (with exponential smoothing) of the proprosed system. Entering the non-gray zone, we provoked a sudden drift in input samples' statistics : the system continues learning but now over a new input data distribution. Dash-dot lines and dotted lines represent test samples results, respectively over the first and the second data distribution.

: the system could try to predict a sensor cue based on other sensors. Such a system also implies information feedback from the discriminative module to the generative one.

The median threshold used to detect outliers is currently offline and global : it consistently rejects half of the decisions making performance go better in average by combining BMU scores over all units of the SOM and must be computed between train and test phases. We need to investigate on-line and adaptive thresholding techniques and rely on topological localisation to refine this selection.

Finally, we want our approach to deal with incremental learning tasks and life-long learning, hopefully on "real world" robotics datasets : Fig. 4 displays that our system is able to produce indicators concerning the non-stationarity of the underlying data properties while avoiding catastrophic forgetting.

Fig. 1 :

 1 Fig. 1: Detailed illustrative view of the proposed system.

Fig. 2 :

 2 Fig. 2: Comparison of mean squared errors between our architecture (solid line) and Bayesian (dotted line) fusion decisions with data generation type 1 and additive Gaussian noise

Fig. 3 :

 3 Fig. 3: Comparison of mean squared errors between our architecture (solid line) and Bayesian (dotted line) fusion decisions with data generation type 2 and additive Gaussian noise. Dash-dot lines represent cases where a fusion decision is taken only if BMU scores are greater than the median of BMU scores of the trained grid. (a) : targets values are linear combinations of sensors values ; (b) : targets values are non-linear combinations of sensors values.

VII. ACKNOWLEDGEMENTS

We gratefully thank Mathieu Lefort for technical assistance and reviewing this article.


Thomas Kopinski

Alexander Gepperth

Uwe Handmann

A simple technique for improving multi-class classification with neural networks

We present a novel method to perform multi-class pattern classification with neural networks and test it on a challenging 3D hand gesture recognition problem. Our method consists of a standard oneagainst-all (OAA) classification, followed by another network layer classifying the resulting class scores, possibly augmented by the original raw input vector. This allows the network to disambiguate hard-to-separate classes as the distribution of class scores carries considerable information as well, and is in fact often used for assessing the confidence of a decision. We show that by this approach we are able to significantly boost our results, overall as well as for particular difficult cases, on the hard 10-class gesture classification task.

Introduction

The problem of multi-class pattern classification can be defined as follows: For a sample X i taken from a set of n samples X 1 ,...,X n where each sample is characterized by m features a 1 ,...,a m ∈ A find a function f : X → L which maps the sample to the correct corresponding label l i ∈ L : { l 1 ,...,l n }. This is contrasted to the binary classification problem where l belongs to either class 0 or 1 (or -1,1) respectively. The generalization performance of such a function measures how well it is able to predict the correct label for a previously unseen sample. Various approaches exist for this problem and one generally distinguishes oneagainst-one (OAO), one-against-all(OAA) and P-against-Q (PAQ) methods (see [START_REF] Ou | Multi-class pattern classification using neural networks[END_REF]). Most of these methods are rather empirical in nature, and a precise mathematical theory to guide our choice of algorithms, as it exists for binary classification, seems unavailable. We present a novel approach which is somewhere in the middle between OAA and OAO, applied to the problem of hand pose recognition with a single ToF [START_REF] Kopinski | Neural network based data fusion for hand pose recognition with multiple tof sensors[END_REF]. The basic idea is to build a two-stage architecture that first performs a normal OAA classification, and subsequently fuses the obtained set of class scores with the original network input for a second layer of classification. To verify our approach, we create a very large a database of training and testing samples which is divided accordingly to the needs of the problem, and are able to show that the proposed approach can improve the overall generalization performance, as well as significantly boosting performance on specific difficult classes.

Related work

The problem of multi-class classification has been thoroughly researched in [START_REF] Ou | Multi-class pattern classification using neural networks[END_REF] with special emphasis of modelling pattern classes with one-against-all, oneagainst-one and P-against-Q. This study compares the advantages and disadvantages of each of the approaches by employing either a single or a series of neural networks supported by various decision functions. The results of this analysis show the benefit and drawbacks of splitting the task into multiple independently trained neural networks of which some are the uncovered feature space regions and ambiguity issues. The authors of [START_REF] Lu Murphey | Oaho: an effective algorithm for multi-class learning from imbalanced data[END_REF] show the increased difficulty in the classification problem with an imbalanced class distribution. In order to solve the multi-class classification problem with support vector machines without creating independent classifiers a single optimisation routine has been proposed in [START_REF] Weston | Support vector machines for multi-class pattern recognition[END_REF]. Whether to rely on an OAO, OAA or an PAQ approach is a non-trivial question and a more in-depth analysis in favor of the OAA approach can be found in [START_REF] Rifkin | In defense of one-vs-all classification[END_REF]. The most important statement made is that once the best binary classifier has been chosen the choice of the multiclass scheme becomes secondary and the simpler scheme should be preferrable. In [START_REF] Chris | Multi-class protein fold recognition using support vector machines and neural networks[END_REF] the authors successfully show how a multi-class classifier can be efficiently applied to the problem of protein folding with a special focus on the comparison of various discriminative methods. They achieve impressive results by employing support vector machines as classifiers. In [START_REF] Stallkamp | The german traffic sign recognition benchmark: a multi-class classification competition[END_REF] the performance of different neural network topologies is tested on the multi-class problem of traffic sign recognition.

Database and Descriptors

We recorded a database containing 3000 samples for each of 10 gestures collected from 15 persons, see Fig. 1. To increase variance in the data the snapshots were taken in three regions in a range from 20cm-50cm and the participants were asked to translate and rotate their hands during the recording. Overall, the total sample count is 450000 samples. Each sample originates from raw point cloud data which is transformed via a global Point Cloud Descriptor into a histogram of fixed size. The descriptor depicts the global shape of the cloud via the relationship of the angles calculated from a sample subset of point pairs. For a more in-depth specification of this feature transformation please refer to [START_REF] Kopinski | Neural network based data fusion for hand pose recognition with multiple tof sensors[END_REF]. Since we are interested in the multi-class classification task of recognizing shapes, we automatically remove the background from the recording by simple distance thresholding. The gestures chosen for this classification task are relevant to HMI applications and contain a significant amount of ambiguity, thereby creating a hard problem for a meaningful benchmark. We aim for a OAA classification approach and thus design our neural networks accordingly. We implement a 2-step approach involving two distinct multilayer perceptron (MLP) networks. Each neural network has three layers -input, hidden and output layer with the input and hidden layers fully connected to their subsequent layers. The input for the first net is formed by the point cloud descriptor of the current sample, the output layer consists of 10 output neurons, one for each class. The input for the second net differs as it not only takes into account the descriptor of the current sample but also the output values of the first net which classifies the same sample. The determination of the number of hidden neurons for each net is described in the following chapter. The output layer of the second network also consists of 10 neurons.

Early fusion technique

In order to train the two-stage architecture described in the previous chapter, the available data samples need to be divided into three disjunct sets. We distinguish between the training (D1 and D2) and generalization (D3) sets, the latter consisting of all the samples (about 30000) coming from person 1 ≤ i ≤ 15. The remaining samples are divided into the two equally sized sets D1 and D2. In order to monitor training performance we furthermore randomly split each D1 and D2 in a ratio 9:1 into the sets D1.train, D1.test, D2.train and D2.test. These sets are used during the training of each network to achieve a more independent measurement of the current generalization performance. For the first network, D1.train is used for training the network while the mean squared error measured on the D1.test set is used for selecting the best-performing net among all conducted iterations. As soon as the first net is trained, the second net is trained on D2 in an analogous manner. The main difference is that the input for the second net is formed as follows: For a sample taken from D2.train, feed this sample into the first net and memorize the output of all 10 output neurons. These values are concatenated with the descriptor (size n) of the sample, thus having length n + 10, and forms the new training sample fed into net 2, with supervision coming again from the known class of the sample. The rest of the procedure remains the same, D2.test is used as the performance measure for the decision process of the second phase of training.

Experiments and Results

All experiments were conducted with the FANN library and implemented under Ubuntu in C++. We have tested several configurations in order to determine the hidden layer size and optimal training parameters. Initial test runs have shown that the best generalization results can be achieved when setting the size of the hidden layer between 25-40 neurons for both networks. The remaining parameters were chosen as standard values and testing various configurations provided no meaningful insight regarding our problem. The training algorithm is RPROP, the activation function is the symmetric sigmoid function for both hidden and output layers. On average we were able to improve the generalization results by between 2-4% considering all classes, as Table 1 shows, when employing two nets each having 40 neurons in the hidden layer. The upper row shows the results for net 1 alone, being fed the feature vector only while the lower row displays the fused feature and neuron values. In 3 cases we see a slight drop in generalization performance, in one single case no change is notable while all the remaining cases show partly significant improvements when fusing the data with output activities. In some selected cases we have improvement rates of 5-9% averaged over all the gestures of one person.

As a comparison Table 2 displays the performance of both nets each with 25 and 20 hidden neurons respectively. The main difference in this case is the fact that the nets with smaller hidden layer seem to perform slightly more poorly while we are still able to show an improvement with our fusion technique when individually comparing each generalization case. Moreover the most significant improvements can be found in both experiments. approach by randomly splitting the whole dataset (i.e., not taking into account individual persons) into two equally sized datasets, one for training and one for evaluation. Since the sets were not completely independent, the overall classification rate was 86% for net 1 and 87% for net 2 which is not surprising but still shows we are able to improve the results with our fusion approach. A comparison of our approach with the MNIST dataset yielded, for 15 different test runs conducted in the same manner (as there is no person information available), neither a significant drop nor an improvement since the classification error remained around 5.5% for net 1 and net 2 with a slight variance. This variance can be ascribed to the random initialization of the weights. It should be noted that we split our dataset into three parts namely 40% for training each net and 20% for testing, which is a significantly reduced training data set. This contribution presents a novel fusion technique for the multi-class pattern classification problem of recognizing hand poses with neural networks. The described approach is a 2-stage deep neural network with a total of 5 layers. Our fusion approach exceeds the performance of a regular neural network by about 2-3% measured as the classification errors averaged over 15 persons, each of which was left out from the training set, amounting to a sort of cross-validation procedure. More specifically we are able to significantly boost the results in a number of cases, especially for poses that are more difficult to disambiguate. Nevertheless in two cases we see a decrease in classification performance which has also to be taken into consideration when designing a real-time applicable system. In these cases however one can simply rely on the first net when designing a decision function and use the second net for the other instances. In terms of applicability we have implemented this fusion technique into our realtime hand gesture recognition system and it has shown to stabilize the results when designing the decision process as described above [START_REF] Kopinski | A real-time applicable 3d gesture recognition system for automobile hmi[END_REF]. We are confident to be able to improve our performance even further by employing confidence measures for rejecting false positives and thus stabilizing the recognition performance.

Fig. 1 :

 1 Fig. 1: The 10 different gestures recorded from 15 different persons

Table 1 :

 1 Comparison of the generalization performance of net 1 (upper row, 40 hidden neurons) vs. net 2 (lower row, 40 hidden neurons). The indexed columns indicate the generalization data coming from person i.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
	.81 .43 .68 .43 .83 .49 .53 .68 .81 .70 .88 .73 .95 .66 .76
	.83 .44 .67 .54 .84 .53 .55 .67 .78 .70 .91 .74 .97 .73 .90

Table 2 :

 2 Comparison of the generalization performance of net 1 (upper row, 25 hidden neurons) vs. net 2 (lower row, 20 hidden neurons) Table3shows the change in recognition rate in reference to the individual hand gesture classes averaged over all persons. In 8 of 10 cases we achieve a boosting of the recognition rate except for gesture classes c and d (-2.1% and -2.8%). The best improvement amounts to 4.7% and 4.8% which is, for this problem, of special interest since these are two of the more difficult cases to disambiguate (cf. Fig.1).

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
	.81 .40 .68 .39 .81 .48 .52 .68 .79 .68 .88 .71 .94 .64 .74
	.81 .45 .67 .50 .83 .53 .52 .67 .77 .67 .90 .71 .96 .68 .89
	a	b		c	d	e	f		g	h		i	j
	+1.3 +4.8 -2.1 -2.8 +3.8 +2.0 +3.1 +3.2 +4.7 +2.5

Table 3 :

 3 The average increase/decrease in recognition rate for all 10 gestures averaged over all the persons for net 1 compared to net 2 (both 40 hidden neurons).Table4shows the corresponding improvements for the recognition rate averaged over all persons for all individual gesture classes. We equally test our

	a	b	c	d	e	f	g	h	i	j
	+.01 +.02 +.01 -.03 +.07 +.01 +.05 +.04 +.01 +.04

Table 4 :

 4 The average increase/decrease in recognition rate for all 10 gestures averaged over all the persons for net 1 compared to net 2 (25 and 20 hidden neurons respectively).