Imran N Junejo
email: ijunejo@sharjah.ac.ae

Emilie Dexter
email: emilie.dexter@inria.fr

Ivan Laptev
email: ivan.laptev@inria.fr

Patrick Pérez
email: patrick.perez@thomson.net

View-Independent Action Recognition from Temporal Self-Similarities

Imran N. Junejo, Member, IEEE, Emilie Dexter, Ivan Laptev, and Patrick Pérez

Abstract-This paper addresses recognition of human actions under view changes. We explore self-similarities of action sequences over time and observe the striking stability of such measures across views. Building upon this key observation, we develop an action descriptor that captures the structure of temporal similarities and dissimilarities within an action sequence. Despite this temporal self-similarity descriptor not being strictly view-invariant, we provide intuition and experimental validation demonstrating its high stability under view changes. Self-similarity descriptors are also shown stable under performance variations within a class of actions, when individual speed fluctuations are ignored. If required, such fluctuations between two different instances of the same action class can be explicitly recovered with dynamic time warping, as will be demonstrated, to achieve cross-view action synchronization. More central to present work, temporal ordering of local selfsimilarity descriptors can simply be ignored within a bag-offeatures type of approach. Sufficient action discrimination is still retained this way to build a view-independent action recognition system. Interestingly, self-similarities computed from different image features possess similar properties and can be used in a complementary fashion. Our method is simple and requires neither structure recovery nor multi-view correspondence estimation. Instead, it relies on weak geometric properties and combines them with machine learning for efficient cross-view action recognition. The method is validated on three public datasets. It has similar or superior performance compared to related methods and it performs well even in extreme conditions such as when recognizing actions from top views while using side views only for training.

Index Terms-Human Action Recognition, Human Action Synchronization, View Invariance, Temporal Self-Similarities, Local Temporal Descriptors

I. INTRODUCTION

Visual recognition and understanding of human actions have attracted much attention over the past three decades [START_REF] Moeslund | A survey of advances in visionbased human motion capture and analysis[END_REF], [START_REF] Wang | Recent developments in human motion analysis[END_REF] and remain an active research area of computer vision. A good solution to the problem holds a yet unexplored potential for many applications such as the search and the structuring of large video archives, video surveillance, human-computer interaction, gesture recognition and video editing. Recent work has demonstrated the difficulty of the problem associated with the large variation of human action data due to the individual variations of people in expression, posture, motion and clothing; perspective effects and camera motions; illumination variations; occlusions and disocclusions; and distracting effects of scenes surroundings. Also, actions frequently involve and depend on manipulated objects, which adds another layer of variability. As a consequence, current methods often resort to restricted and simplified scenarios with simple backgrounds, simpler kinematic action classes, static cameras or limited view variations.

Various approaches using different constructs have been proposed over the years for action recognition. These approaches can be roughly categorized on the basis of representation used by the authors. Time evolution of human silhouettes was frequently used as action description. For example, [START_REF] Bobick | The recognition of human movement using temporal templates[END_REF] proposed to capture the history of shape changes using temporal templates and [START_REF] Weinland | Free viewpoint action recognition using motion history volumes[END_REF] extends these 2D templates to 3D action templates. Similarly, the notions of action cylinders [START_REF] Syeda-Mahmood | action events from multiple viewpoints[END_REF], and space-time shapes [START_REF] Yilmaz | Actions sketch: A novel action representation[END_REF]- [START_REF] Grundmann | 3d shape context and distance transform for action recognition[END_REF] have been introduced based on silhouettes. Recently, space-time approaches analyzing the structure of local 3D patches in the video have been shown promising in [START_REF] Laptev | On space-time interest points[END_REF]- [START_REF] Gilbert | Scale invariant action recognition using compound features mined from dense spatio-temporal corners[END_REF]. Using space-time or other types of local features, the modeling and recognition of human motion have been addressed with a variety of machine learning techniques such as Support Vector Machines (SVM) [START_REF] Schüldt | Recognizing human actions: a local SVM approach[END_REF], [START_REF] Laptev | Learning realistic human actions from movies[END_REF], Hidden Markov Models (HMM) [START_REF] Ali | Chaotic invariants for human action recognition[END_REF]- [START_REF] Jia | Human action recognition using local spatiotemporal discriminant embedding[END_REF] and Conditional Random Fields (CRF) [START_REF] Sminchisescu | Conditional models for contextual recognition[END_REF]- [START_REF] Natarajan | View and scale invariant action recognition using multiview shape-flow models[END_REF].

Most of the current methods for action recognition are designed for limited view variations. A reliable and a generic action recognition system, however, has to be robust to camera parameters and different view points while observing an action sequence. View variations originate from the changing and frequently unknown positions of the camera. Similar to the multi-view appearance of static objects, the appearance of actions may drastically vary from one viewpoint to another. Differently to the static case, however, the appearance of actions may also be affected by the dynamic view changes of the moving camera.

Multi-view variations of actions have been previously addressed using epipolar geometry such as in [START_REF] Syeda-Mahmood | action events from multiple viewpoints[END_REF], [START_REF] Yilmaz | Recognizing human actions in videos acquired by uncalibrated moving cameras[END_REF]- [START_REF] Parameswaran | View invariance for human action recognition[END_REF], by learning poses seen from different viewpoints [START_REF] Ogale | View-invariant modeling and recognition of human actions using grammars[END_REF]- [START_REF] Farhadi | Learning to recognize activities from the wrong view point[END_REF] or by a full 3D reconstruction [START_REF] Yan | Learning 4d action feature models for arbitrary view action recognition[END_REF], [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF]. Such methods rely either on existing point correspondences between image sequences or/and on many videos representing actions in multiple views. Both of these assumptions, however, are limiting in practice due to (i) the difficulty of estimating non-rigid correspondences in videos and (ii) the difficulty of obtaining sufficient video data spanning view variations for many action classes.

In this work we address multi-view action recognition from a different perspective and avoid many assumptions of previous methods. In contrast to the geometry-based methods above we require neither the identification of body parts nor the estimation of corresponding points between video sequences. Differently to the previous view-based methods we do not assume multi-view action samples neither for training nor for testing.

Our approach builds upon self-similarities of action sequences over time. For a given action sequence and a given type of low level features, we compute distances between extracted features for all pairs of time frames and store results in a Self-Similarity Matrix (SSM). We claim SSMs to be stable under view changes of an action. Fig. 1 illustrates our idea with an example of a golf swing action seen from two different views. For this example we compute SSMs as pair-wise distances between all 2D points on the projected hand trajectories illustrated in Fig. 1(a,c). Despite the view variation, close trajectory points A and B remain close in both views while the distant trajectory points A and C have large distances in both projections. The visualizations of SSMs computed for both sequences in Fig. 1(b,d) have a striking similarity despite the different projections of the action. More generally, if body poses of an action are similar at moments t 1 , t 2 , the value of SSM(t 1 , t 2 ) i.e., the distance between some action descriptors at t 1 , t 2 will be low for any view of an action. On the contrary, if the body poses are different at t 1 , t 2 , the value of SSM(t 1 , t 2 ) is likely to be large for most of the views and non-trivial action descriptors.

In the rest of the paper we operationalize SSMs for human action sequences and deploy them for view-independent action recognition. In particular, we observe similar properties of SSMs computed for different image features and use such SSMs in a complementary fashion.

The paper is organized as follows. In the next section we review related work, with special emphasis on the relationship between SSMs and so-called Recurrence Plots (RPs) which SSM can be seen as an extension of. Section III gives a formal definition of SSM using alternative image features and reports first experiments on mocap data demonstrating its structural stability across views. Section IV describes the proposed representation of action sequences based on local temporal SSM descriptors and demonstrate how this representation is at the same time precise, specific to a class of action and largely view-independent, by using it to synchronize (align temporally) different performances of similar actions. In Section V, we introduce our view-independent action recognition system based on such descriptions and test it on three public datasets. These experiments demonstrate the practicality and the potential of the proposed method. Section VI concludes the paper.

II. RELATED WORK

This paper concerns view-independent action recognition, a topic which has received a considerable attention from researchers recently. To address this problem, [START_REF] Syeda-Mahmood | action events from multiple viewpoints[END_REF], [START_REF] Yilmaz | Recognizing human actions in videos acquired by uncalibrated moving cameras[END_REF], [START_REF] Carlsson | Recognizing walking people[END_REF] employ epipolar geometry. Point correspondences between actions are assumed to be known for imposing fundamental matrix constraints and performing view-invariant action recognition. Rao et al. [START_REF] Rao | View-invariant representation and recognition of actions[END_REF] show that the maxima in space-time curvature of a 3D trajectory are preserved in 2D image trajectories, and are also view-invariant. [START_REF] Parameswaran | View invariance for human action recognition[END_REF] proposes a quasi view-invariant approach, requiring at least 5 body points lying on a 3D plane or that the limbs trace a planar area during the course of an action. Recently [START_REF] Shen | View invariant action recognition using fundamental ratios[END_REF] showed that for a moving plane the fundamental ratios, i.e. the ratios among the elements in the upper left 2 × 2 submatrix of the fundamental matrix, are invariant to the camera parameters as well as its orientation and can be used for action recognition. However, obtaining automatic and reliable point correspondences for daily videos with natural human actions is a very challenging and currently unsolved problem, which limits the application of above mentioned methods in practice.

One alternative to the geometric approach is to represent actions by samples recorded for different views. [START_REF] Ogale | View-invariant modeling and recognition of human actions using grammars[END_REF]- [START_REF] Lv | Single view human action recognition using key pose matching and viterbi path searching[END_REF] create a database of poses seen from multiple viewpoints. Extracted silhouettes from a test action are matched to this database to recognize the action being performed. The drawback of these methods is that each action needs to be represented by many training samples recorded for a large and representative set of views. Other methods [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF] and [START_REF] Yan | Learning 4d action feature models for arbitrary view action recognition[END_REF] perform a full 3D reconstruction from silhouettes seen from multiple deployed cameras. This approach requires a setup of multiple cameras or training on poses obtained from multiple views, which again restricts the applicability of methods in practice.

The approach in [START_REF] Farhadi | Learning to recognize activities from the wrong view point[END_REF] exploits transfer learning for constructing view-stable and discriminative features for view-independent action recognition. For a pair of given views, the features are learned from a separate set of actions observed in both views. Given a new action class observed and learned in one view only, transfer learning enables recognition of instances of that class in the second view. While the use of transfer learning in [START_REF] Farhadi | Learning to recognize activities from the wrong view point[END_REF] is interesting, the method is limited to a set of pre-defined views and requires training to be done separately for each pair of views. It also requires (non-target) actions to be observed and viewtagged for multiple views. Our method avoids these limitations. We compare our results with [START_REF] Farhadi | Learning to recognize activities from the wrong view point[END_REF] on the common benchmark in Section V-C.

The methods most closely related to our approach are that of [START_REF] Shechtman | Matching local self-similarities across images and videos[END_REF]- [START_REF] People | Proc. ECCV[END_REF]. For image and video matching, [START_REF] Shechtman | Matching local self-similarities across images and videos[END_REF] recently explored local self-similarity descriptors. The descriptors are constructed by correlating the image (or video) patch centered at a pixel to its surrounding area by the sum of squared differences. The correlation surface is transformed into a binned log-polar representation to form a local descriptor used for image and video matching. Differently to this method, we explore the structure of similarities between all pairs of time-frames in a sequence. The main focus of our work is on the use of self-similarities for view-invariant action recognition which was not addressed in [START_REF] Shechtman | Matching local self-similarities across images and videos[END_REF].

Our approach has a close relation to the notion of video selfsimilarity used by [START_REF] Benabdelkader | Gait recognition using image self-similarity[END_REF], [START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF]. In the domain of periodic motion detection, Cutler and Davis [START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF] track moving objects and extract silhouettes (or their bounding boxes). This is followed by building a 2D matrix for the given video sequence, where each entry of the matrix contains the absolute correlation score between the two frames i and j. Their observation is that for a periodic motion, this similarity matrix will also be periodic. To detect and characterize the periodic motion, they resort to Time-Frequency analysis. Following this, [START_REF] Benabdelkader | Gait recognition using image self-similarity[END_REF] uses the same construct of the selfsimilarity matrix for gait recognition in videos of walking people. The periodicity of the gait creates diagonals in the matrix and the temporal symmetry of the gait cycles are represented by the crossdiagonals. In order to compare sequences of different length, the self-similarity matrix is subdivided into small units. Both of these works focus primarily on videos of walking people for periodic motion detection and gait analysis. The method in [START_REF] People | Proc. ECCV[END_REF] also concerns gait recognition using temporal similarities between frames of different image sequences. None of the methods above explores the notion of self-similarity for view-invariant action recognition.

SSM as a Recurrence Plot: Recurrence is a fundamental phenomenon of many dynamical systems. The study of such systems is typically based on recorded data time-series, {x t , t = 1 • • • T }, from which one wants to learn as much information about observed system as possible. Traditional techniques for understanding a dynamical system involve embedding this timeseries into an E-dimensional reconstruction phase space using delay coordinates [START_REF] Ali | Chaotic invariants for human action recognition[END_REF]- [START_REF] Jia | Human action recognition using local spatiotemporal discriminant embedding[END_REF]. This process involves estimation of two parameters, i.e., (i) the embedding dimension E and (ii) the delay, which is a difficult task.

In order to visualize the geometry of a dynamical system's behavior, Eckman et al. [START_REF] Eckmann | Recurrence plots of dynamical systems[END_REF] first proposed the Recurrence Plot (RP), defined as:

RP (i, j) = Θ(ε -x i -x j 2 ) (1)
where Θ(.) is the Heaviside function. Once a suitable threshold ε is determined, the RP is then a binary image displaying a black [START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF]. (e) represents the SSM obtained for this sequence by [START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF] using the absolute correlation score between frames of the sequence. Time-Frequency analysis is performed on this matrix to detect periodicity in a motion sequence.

dot where the values are within the threshold corridor. As we shall see, the proposed self-similarity matrix is a variant of the RP: instead of capturing the system's behavior using dots and lines by thresholding, we aim for plots with richer textures, in terms of distinct peaks and valleys, which are hopefully distinctive for different dynamical systems. These patterns on the RPs, and a fortiori on SSMs, contain a wealth of information about the dynamics of a system and capture specific behaviors of the system. Researchers have attempted to classify dynamic systems into different categories based on these textures. Part of this categorization [START_REF] Marwan | Recurrence plots for the analysis of complex systems[END_REF] is reproduced in Table I.

McGuire et al. [START_REF] Mcguire | Recurrence matrices and the preservation of dynamical properties[END_REF] have shown that RPs not only preserve invariants of a dynamical system (such as the Lyapunov exponents [START_REF] Bradley | Recurrence plots and unstable periodic orbits[END_REF]) but are also to some extent independent of the embedding dimension [START_REF] Mcguire | Recurrence matrices and the preservation of dynamical properties[END_REF], which naturally raises the question of whether embedding is necessary at all for understanding the underlying dynamics of a system [START_REF] Iwanski | Recurrence plots of experimental data: To embed or not to embed?[END_REF]. In addition, [START_REF] Mcguire | Recurrence matrices and the preservation of dynamical properties[END_REF] have shown that RPs for different systems are identical as long as the transformation is isometric. This conclusion also apply to proposed SSMs. As we shall see, SSMs are not strictly invariant under projective or affine transformations, but are experimentally found stable under 3D view changes.

III. SELF-SIMILARITY MATRIX (SSM)

Self-similarity matrices have already appeared in the past under various specific forms, including binary recurrence plots associated to time series, as mentioned above. In this section we define such matrices for different image features, with examples for several action classes, and start investigating their stability across views.

For a sequence of images [START_REF] Marwan | Recurrence plots for the analysis of complex systems[END_REF])

I = {I 1 , I 2 , . . . I T } in discrete (x, y, t)-space, a SSM of I is a square symmetric matrix of size

SSM Pattern Meaning (1) Homogeneity

The process is stationary

(2) Fading in the corners Non-stationary data; the process contains a trend or a drift

(3) Periodic/quasi-periodic patterns Cyclicities in the process; the time distance between periodic patterns (e.g. lines) corresponds to the period (4) Single isolated points (or structures) Strong fluctuation in the process; if only single isolated points occur, the process may be an uncorrelated random or even anti-correlated process [START_REF] Syeda-Mahmood | action events from multiple viewpoints[END_REF] Diagonal lines (parallel to the main diagonal)

The evolution of states is similar at different epochs; the process could be deterministic; if these diagonal lines occur beside single isolated points, the process could be chaotic (if these diagonal lines are periodic, unstable periodic orbits can be observed) [START_REF] Yilmaz | Actions sketch: A novel action representation[END_REF] Diagonal lines (orthogonal to the main diagonal)

The evolution of states is similar at different times but with reverse time; sometimes this is an indication for an insufficient embedding [START_REF] Gorelick | Actions as space-time shapes[END_REF] Long bowed line structures The evolution of states is similar at different epochs but with different velocity; the dynamics of the system could be changing

T × T , d ij i,j=1,2,...,T =       0 d 12 d 13 . . . d 1T d 21 0 d 23 . . . d 2T . . . . . . . . . . . . d T 1 d T 2 d T 3 . . . 0       (2)
where d ij is the distance between certain low-level features extracted in frames I i and I j respectively. The diagonal corresponds to comparing a frame to itself (no dissimilarity), hence is composed of zeros. The exact structures or the patterns of this matrix depend on the features and the distance measure used for computing the entries d ij . For example, after tracking walking people in a video sequence, [START_REF] Benabdelkader | Gait recognition using image self-similarity[END_REF] and [START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF] compute a particular instance of SSM where d ij is the absolute correlation between two frames, as depicted in Fig. 3. The computed matrix patterns (cf. Fig. 3(e)) have a significant meaning for their applicationthe diagonals in the matrix indicate periodicity of the motion.

In this work, we define d ij as the Euclidean distance between the different features that we extract from an action sequence. This form of SSM is known in the literature as the Euclidean Distance Matrix (EDM) [START_REF] Lele | Euclidean distance matrix analysis (EDMA): Estimation of mean form and mean form difference[END_REF].

To get a fist insight into the representation power of SSMs, a comparison with the notion of "dynamic instances" proposed by Rao et al. [START_REF] Rao | View-invariant representation and recognition of actions[END_REF] is illustrated in Fig. 2. The authors of [START_REF] Rao | View-invariant representation and recognition of actions[END_REF] argue that continuities and discontinuities in position, velocity and acceleration of a 3D trajectory of an object are preserved under 2D projections. For an action of opening a cabinet door, performed by two different actors from considerably different viewpoints, these points are depicted in Fig. 2. Fig. 2(c)(e) shows the SSMs computed for these two actions based only on one hand trajectory, where red color indicates higher values and dark blue color indicates lower values. The dynamic instances, red stars in Fig. 2(b)(d), correspond to valleys of different area/spread in our plot of SSM (cf. Fig. 2(c)(e)), marked by magenta circles along the diagonal of the matrix. The exact spread of these valleys depend on the width of the peaks in the spatio-temporal curvature of the actions, as shown in Fig. 2(b)(d). However, whereas [START_REF] Rao | View-invariant representation and recognition of actions[END_REF] capture only the local discontinuities in the spatio-temporal curvature, the SSM captures more information about other dynamics of the actions present in the off-diagonal parts of the matrix.

Note also that the proposed notion of self-similarity, unlike [START_REF] Syeda-Mahmood | action events from multiple viewpoints[END_REF] or [START_REF] Rao | View-invariant representation and recognition of actions[END_REF], does not require estimation of point correspondences or time-alignment between different actions.

A. Trajectory-based Self-Similarities

If a set of M points x m , m = 1 • • • M ,
distributed over a person is "tracked" (in sense to be specified later) over the duration of an action performance, the mean Euclidean distance between each of the k pairs of corresponding points at any two instants i and j of the sequence can be computed as

d ij = 1 M M m=1 x m i -x m j 2 (3)
where x k i and x k j indicate positions of points on the track k at time instants i and j. We denote the self-similarity matrix computed from (3) by SSM-pos.

In a first set of experiments aimed at investigating SSM properties in a controlled set-up, such point trajectories are directly obtained via motion capture, rather than from video sequences. In this case, a "view" corresponds to the projection of 3D point tracks onto a given 2D plane. In these experiments, we track M = 13 joints on a person performing different actions [START_REF] Ali | Chaotic invariants for human action recognition[END_REF], as shown in the Fig. 4(a). In order to remove the effect of global person translation, without loss of generality, the points are centered to their centroid so that their first moment is zero.

The overall goal of proposed work being the recognition of actions in videos irrespective to view points, the actual computation of SSM-pos requires that points are extracted and tracked in the input video. We assume that this task is handled automatically by an external module such as KLT [START_REF] Tomasi | Good features to track[END_REF] point tracker. Note that our method is not restricted to any particular subset of points as far as the points are distributed over moving body parts. The definition of SSM-pos in 3 needs however to be adapted to a set of tracks with arbitrary length and starting time:

d ij = 1 |S ij | m∈Sij x m i -x m j 2 , (4)
where

S ij ⊂ {1, • • • , M }
is the set with indices of point trajectories that are alive between frames i and j. In addition to the SSM-pos, we also compute similarities based on the first and the second-order derivatives of the 2D positions, i.e., velocities and accelerations. Similarities computed base on these features are denoted by SSM-vel and SSM-acc, respectively.

B. Image-based Self-Similarities

Beside point trajectories, alternative image features can be used to construct other SSMs for the same image sequence. To describe spatial appearance of a person at each image frame, we compute Histograms of Oriented Gradients (HoG) features [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]. This descriptor, originally used to perform human detection, characterizes the local shape by capturing the gradient structure. In our implementation, we use 4 bin histograms for each of 5 × 7 blocks defined on a bounding box around the person in each frame. Feature distance d ij between time instants i and j is then computed as the Euclidean distance between two HoG vectors extracted from frames I i and I j . We denote SSMs computed using HoG features by SSM-hog.

In addition to HoG features, we also test the proposed method by considering optical flow vectors as another input feature. The corresponding SSMs are denoted by SSM-of. More precisely, we assume, as for point trajectories, that optical flow is provided by another module, e.g., Lucas and Kanade algorithm [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF] based on two consecutive frames. We concatenate the components of optical flow vectors computed for all n pixels in a bounding box around a person into a flow vector of size 2n. Entry d ij of SSM-of matrix then amounts to the Euclidean distance between the flow vectors corresponding to the two frames I i and I j . In practice, we enlarge and resize bounding boxes in order to avoid border effects on the flow computation and to ensure the same size of the flow vectors along an action sequence. We resize the height to a value equal to 150 pixels and the width is set to the greatest value for the considered sequence.

Examples of SSMs computed for different image features are shown in Fig. 4. Fig. 4(a) contains example actions from the CMU motion capture (mocap) dataset projected onto different views. Column 1 and 5 of Fig. 4(a) represent two different actors while columns 2 and 4 represent their computed SSM-pos, respectively. The first two rows represent a bending action performed by two actors and projected onto two considerably different views. The last two rows, similarly, represent a football kick action for two actors and two different views. Note the similarity of SSMs computed for actions of the same class despite the changes of the actor and the considerable changes of views. Note also the visual difference of SSMs between two action classes. Computing SSMs on real image features instead of mocap data leads to similar conclusions. Fig. 4(b) illustrates SSMs obtained for the bending action from the video dataset [START_REF] Gorelick | Actions as space-time shapes[END_REF]. Row 2 shows SSMpos computed using point tracks overlaid on images in first row. Rows 3 and 4 show SSM-hog and SSM-of for the same sequences respectively. For a given type of features, note the similarity of SSMs over the different instances of the same action class. SSMs for different feature types do not look similar since different features capture different properties of the action. This suggests the use of SSMs computed for different features in a complementary manner.

C. Structural Stability of SSM across Views

As noted above, the patterns of proposed SSMs are promisingly stable through changes of viewpoints. In order to assess more thoroughly this stability we conducted the following experiments using the CMU mocap dataset. We deployed a total of K = 684 synthetic affine cameras (at distinct latitudes and longitudes) on a sphere surrounding the person performing an action, as shown in Fig. 5(a). For each of these cameras, we compute the SSM matrix, as described in Section III-A, and aim to assess qualitatively and quantitatively the stability of the patterns contained in these SSMs. To this end, we consider SSMs as being discrete "images" of size T × T which allows us to resort to classic tools for image structure analysis. We consider in particular orientation of bi-dimensional gradient as it is known to capture image structures independently of various changes, including changes in the dynamics of intensity levels. We will further rely on this philosophy when building SSM descriptors in next section. For the time being we consider a simpler structure analysis based on so-called circular statistics [START_REF] De Sa | Applied Statistics Using SPSS, STATISTICA, MATLAB and R[END_REF].

At each "pixel" (i, j) of the SSM associated to k-th view, we compute the orientation θ (k) ij of the bi-dimensional gradient vector. In order to ascertain the effect of different viewing directions on the computed SSMs, we then compute at this point the circular mean and standard deviation, θ ij and σ ij of the orientation over the K = 684 SSMs. Let r ij = c ij s ij , where

c ij = Σ K k=1 cos θ (k) ij /K s ij = Σ K k=1 sin θ (k) ij /K. (5)
Then the circular mean direction is given as:

θ ij = arctan(s ij /c ij ) if c ij ≥ 0 arctan(s ij /c ij ) + πsign(s ij ) if c ij < 0
The mean resultant length,

r ij = c 2 ij + s 2 ij , is used to compute the circular standard deviation as σ ij = -2 ln r ij . We computed [σ ij ] i,j=1•••T
for some sample action sequences from golf swing, kick and jumping jack action classes, as shown in Fig. 5(b)(c)(d), respectively. One can notice that, for each action, the standard deviations are low over most parts of the SSM support, which is a good indicator of SSM structure stability across views. Highest values delineate what can be seen as the strong contours of the average SSM structure for concerned action.

IV. SSM-BASED ACTION DESCRIPTION AND ALIGNMENT

As discussed in the previous section, SSMs have view-stable and action-specific structure. Here we aim to capture this structure and to construct SSM-based descriptors for subsequent view independent action analysis such as alignment and recognition. We note the following properties of SSM: (i) absolute values of SSM may depend on the varying properties of the data such as the projected size of a person in the case of SSM-pos; (ii) fluctuations in the individual performances of a type of actions and temporal de-synchronization of the views may effect the global structure of SSM; (iii) the uncertainty of values in SSM increases with the distance from the diagonal due to the increasing difficulty of measuring self-similarity over long time intervals. These properties led us to the SSM description that follows.

As already mentioned in previous section, we avoid dependency to varying absolute SSM values by resorting to gradient orientations computed from neighbouring elements of the matrix seen as an image. We also avoid global descriptors and, in a manner reminiscent to popular local image descriptors used for object detection and recognition, we accumulate histograms of gradient orientations in local patches. These patches however are only centered on the diagonal of SSM. Our patch descriptor has a log-polar block structure as illustrated in Fig. 6. The diameter of the circular regions under consideration should be seen as temporal window extent. For log-polar block a at time i, we compute the normalized 8-bin histogram h a i = [h a i,b ] b=1:8 of SSM gradient orientations within the block. We then concatenate the histograms of the 11 blocks of the analysis support into a descriptor vector h i = [h a i ] a=1:11 . For descriptors at boundaries with blocks falling outside SSM we set h a i to a zero vector. Choosing a temporal extent of the descriptor involves a tradeoff between the amount of captured temporal information and its variability, which is delicate to tune. In addition, using a single descriptor size may be suboptimal when representing events of varying lengths and with irregularly changing speed. We address this issue by considering local SSM descriptors of multiple sizes and demonstrate the impact of this approach on action recognition in Section V-C.

When constructing a joint local descriptor for multiple SSMs computed for F different features, we concatenate F corresponding local descriptors h f i from each SSM into a single vector

h i = [h f i ] f =1:F .
In such a way we obtain for instance SSMhog-of descriptors by concatenating image-based SSM-hog and SSM-of descriptors. When temporal ordering is required, the representation for a video sequence can finally be defined by the sequence of local descriptors H(I) = (h 1 , ..., h T ) computed over all diagonal elements of SSMs associated to all feature types.

Temporal cross-view action synchronization: Before moving to action recognition based on representation previously defined, we first test this representation on the problem of temporal alignment, or synchronization, of video sequences representing the same action from different viewpoints. The problem amounts to finding the monotonic mapping between the time-line of the first sequence and the time-line of the second one. Consider for instance two videos I 1 and I 2 recorded simultaneously for the side and the top views of a person in action, as shown in Fig. 7(a). To further challenge the alignment, we apply a nonlinear time transformation to one of the sequences. To solve the alignment, we (i) compute optical flow based SSM-of for both image sequences, (ii) represent both videos by a sequence of local SSM descriptors, H(I 1 ) and H(I 2 ) respectively, computed for a single temporal scale as described above and (iii) align the two descriptor sequences using Dynamic Programming (DP). The estimated time transformation is illustrated by the red curve in Fig. 7(b) which closely follows the ground truth transformation (blue curve) despite the drastic change of viewpoint between sequences.

Using the same method, we next address alignment of different instances of similar actions. Fig. 8 demonstrates alignment of pairs of videos representing actions throwing a ball, drinking and smoking performed by different people in varying views. The automatically estimated alignment recovers the manual alignment at key-frames as illustrated in Fig. 8(right) despite large variations in appearance and viewpoints across videos. Note, that in all alignment experiments we have used known person bounding boxes for computing SSM-of descriptors.

The successful alignment of actions illustrated above indicates the strength of SSM-based descriptors and their ability to cope with video variations in terms of viewpoints, subject appearance and movement speed. This suggests that SSM-based descriptors can be used for action recognition as will be investigated in the next section.

V. SSM-BASED ACTION RECOGNITION

In this section we evaluate SSM-based video descriptors for the task of view-invariant action recognition. To recognize action sequences we follow recently successful bag-of-features (BoF) approaches [START_REF] Niebles | Unsupervised learning of human action categories using spatial-temporal words[END_REF], [START_REF] Marszałek | Learning object representations for visual object class recognition[END_REF], [START_REF] Laptev | Local velocityadapted motion events for spatio-temporal recognition[END_REF] and represent each video as a set of quantized local SSM descriptors with their temporal positioning in the sequence being discarded. Taking this view that global temporal ordering is not taken into action (as opposed to its use for synchronization where it is a crucial information) permits to filter out fluctuations between actions from the same class while retaining sufficient action discrimination to build a viewindependent action recognition system, as demonstrated below. In the following we consider two different types of classifiers: the Nearest Neighbour Classifier (NNC) and a Support Vector Machine (SVM) classifier. In the case of NNC, we simply assign to test sequence H(I) the action label of the training sequence I * which minimizes distance D N N (H(I), H(I * )) over all training sequences. The distance D N N is defined by the greedy matching of local descriptors described in [START_REF] Laptev | Local velocityadapted motion events for spatio-temporal recognition[END_REF]. We apply NNC only to datasets with a limited number of samples. For SVM classification, we train non-linear SVMs using χ 2 kernel and adopt one-against-all approach for multi-class classification.

We evaluate SSM-based action recognition on three public datasets. For all recognition experiments we report results for nfold cross-validation and make sure the actions of the same person do not appear in the training and in the test sets simultaneously. In Section V-A we validate the approach in controlled multi-view settings using motion capture data. In Section V-B we demonstrate and compare the discriminative power of our method on a standard single-view action dataset [START_REF] Gorelick | Actions as space-time shapes[END_REF]. We finally evaluate the performance of the method on a comprehensive multi-view action dataset [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF] in Section V-C. We demonstrate the advantage of combining SSM descriptors computed for different types of image features and multiple temporal scales. Multi-view recognition results are compared with results of other methods on the same datasets.

A. Experiments with CMU MoCap dataset

To simulate multiple and controlled view settings we have used 3D motion capture data from CMU dataset (http://mocap.cs.cmu.edu). Trajectories of 13 points on the human body were projected to six cameras with pre-defined orientations with respect to the human body (see Fig. 9(a)). We have used 164 sequences in total corresponding to 12 action classes (bend, cartwheel, drink, fjump, flystroke, golf, jjack, jump, kick, run, walk, walkturn). To simulate potential failures of the visual tracker, we distracted trajectories by randomly breaking them into parts with the average length of 2 seconds. Fig. 9(b) demonstrates results of NNC action recognition when training and testing on different views using SSM-pos, SSM-vel and SSMacc. As observed from the diagonal, the recognition accuracy is the highest when training and testing on the same views while the best accuracy (95.7%) is achieved for cam5 (frontal view). Interestingly, the recognition accuracy degrades only moderately with substantial view changes and remains still high across top view (camera 6) and side views (camera 1 to 5). When training and testing on all views, the average accuracy is 90.5%.

B. Experiments with Weizman actions dataset

To asses the discriminative power of our method on real video sequences, we apply it to a standard single-view video dataset with nine classes of human actions performed by nine subjects [START_REF] Gorelick | Actions as space-time shapes[END_REF](see Fig. 10(top)). On this dataset we compute NNC recognition accuracy when using either image-based or trajectorybased self-similarity descriptors according to Section III. Given the low resolution of image sequences in this dataset, the trajectories were acquired by [START_REF] Ali | Chaotic invariants for human action recognition[END_REF] via semi-automatic tracking of body joints. Recognition accuracy achieved by our method for optical flow-based and trajectory-based self-similarities is 94.6% and 95.3% respectively and the corresponding confusion matrices are illustrated in Fig. 10(a)-(b). The recognition results are high for both types of self-similarity descriptors and outperforms the accuracy of 92.6% achieved by a recent trajectory-based method [START_REF] Ali | Chaotic invariants for human action recognition[END_REF]. Whereas higher recognition rates on this singleview dataset have been reported, e.g., in [START_REF] Ikizler | Human action recognition using distribution of oriented rectangular patches[END_REF], the main strength of our method will be demonstrated for action recognition across multiple views, as described in the next section.

C. Experiments with IXMAS dataset

IXMAS dataset is publicly available and numerous researchers have reported their results on this dataset. Without resorting to engineering a different experimental setup to test view invariance, using this dataset allows for a quick and a fair comparison of our method to the other methods. Thus we present results for IXMAS video dataset [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF] "punch" action "turn around" action of SSM-hog, SSM-of and their combination SSM-hog-of. We also consider local SSM descriptors computed at multiple temporal scales. For each SSM diagonal point, three local descriptors are computed corresponding to three different diameters for the logpolar domain (respectively 28, 42 and 56 frames in diameter). The number of descriptors assigned to a given sequence is thus multiplied accordingly. All descriptors are quantized independently of their scale using a single visual vocabulary and are used to compute a single histogram associated to the sequence.

Fig. 12(a-c of action recognition over view changes, now using visual data only. The method achieves reasonable recognition accuracy even for extreme variations in views such as for testing on top views (Test Cam4) when using side-views only for training. Also, these tables indicate that using jointly HoG-based and optical flowbased SSMs yields better recognition than using either of the two types of feature individually. The class confusion matrix in Fig. 13, computed using SSM-hog-of, illustrates good per-class recognition performance for all classes when averaged over all cross camera setups in Fig. 12(a), i.e., using camera-X for training and camera-Y for testing for X = Y .

Comparison to alternative methods: We compare recognition performance of SSM-based features to space-time interest points (STIPs) [START_REF] Laptev | On space-time interest points[END_REF], [START_REF] Laptev | Learning realistic human actions from movies[END_REF] representing videos by sets of descriptors computed from local space-time patches. STIP descriptors have been recently demonstrated to achieve competitive performance on several action recognition benchmarks [START_REF] Wang | Evaluation of local spatio-temporal features for action recognition[END_REF]. STIP features, however, are not designed to handle large view variations. The recognition performance of STIP features on IXMAS dataset using the same classification method as for SSM-based features is illustrated in Fig. 12(d). It is interesting to observe that STIP features outperform SSM-based features in recognition setups where the same or similar views are used for training and testing. For large variation between training and test views, however, SSM-based descriptors considerably outperform STIP features, especially when testing on top views after learning on side views, and vice-versa. This behavior is consistent with the intuition that SSM-based descriptors gain view independence at the cost of somewhat reduced discriminative power. The comparison of SSM-based and STIP features is summarized in Table II for different recognition setups.

We also compare our approach to the two alternative methods in the literature that were evaluated on the same dataset. Action recognition in IXMAS dataset is addressed by means of 3D reconstruction in [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF]. Results of this method reported for the same training/test camera setup are lower compared to our SSMbased recognition scheme as illustrated in Table II. Our SSMbased descriptors also outperform results of the transfer-learning approach reported in [START_REF] Farhadi | Learning to recognize activities from the wrong view point[END_REF] both for cross-camera and samecamera setups (cf. Table II). Apart from the superior recognition performance, our method does not require any knowledge about actions in test views which is not the case for [START_REF] Farhadi | Learning to recognize activities from the wrong view point[END_REF], [START_REF] Weinland | Action recognition from arbitrary views using 3D exemplars[END_REF].

Impact of multiple temporal scales: Table III presents recognition results for SSM-hog-of descriptors computed at multiple and single temporal scales. Comparing single-scale descriptors, we observe that accuracy increases with the temporal extent of the descriptor tested for descriptor sizes 28 frames (1,1sec.), 42 frames (1,9sec.) and 56 frames (2,2sec.). Combining different scales, however, results in the considerable increase of performance compared to single-scale results for all camera setups.

VI. CONCLUSION

We propose a self-similarity based descriptor for viewindependent video analysis, with human action recognition as a central application. Self-similarity being possibly defined over a variety of image features, either static (histograms of intensity gradient directions) or dynamic (optical flows or point trajectories), these descriptors can take different form and can be combined for increased descriptive power. Experimental validation on action recognition, as well as for the different problem of action synchronization, clearly confirms the stability of this type of description with respect to view variations. Results on public multi-view action recognition datasets demonstrate superior performance of our method compared to alternative methods in the literature.

Such encouraging results are simply obtained by exploiting the stability across views of SSM patterns, with no need to rely on the delicate recovery of 3D nor on the estimation of correspondences across views. Our method only makes mild assumptions about the rough localization of a person in the frame. This lack of strong assumptions is likely to make this approach applicable to action recognition beyond controlled datasets when combined with modern techniques for person detection and tracking.

Fig. 1 .

 1 Fig. 1. Cross-view stability of trajectory-based self-similarity matrices (SSMs) on a simple example. (a) and (c) demonstrate, based on motion capture (MOCAP) data, a golf swing action seen from two different views. (b) and (d) represent their respective SMMs for the trajectory of one hand projected in corresponding view. Even though the two views are different, the structures or the patterns of the computed SSMs are very similar.

Fig. 2 .

 2 Fig. 2. Relationship between proposed SSM representation and dynamic instances introduced in [26]. Two actors perform the action of opening a cabinet door, where the hand trajectory is shown in (b) and (d). The SSMs computed for these two actions based only on one hand trajectory are shown in (c) and (e), respectively. The "dynamic instances" (as proposed by [26]), marked in red stars in (b) and (d), represent valleys in the corresponding SSM, depicted by magenta circles in (c) and (e), respectively. The spread of each valley depends on the peak-width of the corresponding dynamic instance.

Fig. 3 .

 3 Fig. 3. Earlier example of SSM for motion periodicity analysis. (a)-(d) are frames from a sequence of a walking person[START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF]. (e) represents the SSM obtained for this sequence by[START_REF] Cutler | Robust real-time periodic motion detection, analysis, and applications[END_REF] using the absolute correlation score between frames of the sequence. Time-Frequency analysis is performed on this matrix to detect periodicity in a motion sequence.

Fig. 4 .

 4 Fig. 4. Examples of SSMs for different types of features and for different actions. (a) Examples from CMU mocap dataset. Columns 1 and 5 represent two actors while columns 2 and 4 represent corresponding SSM-pos computed with 13 projected point trajectories, respectively. Different rows represent different actions and viewing angles. Note the stability of SSMs over different views and persons performing the same action. (b) Examples from Weizman video dataset [7]. Row 1: four bending actions along with manually extracted point trajectories used for computing SSM-pos; Rows 2, 3, 4 represent SSM-pos, SSM-hog and SSM-of respectively for these four bending actions. Note the similarity column-wise.

Fig. 5 .

 5 Fig. 5. Stability of SSM-pos structures across viewpoints for mocap data sequence (a) Synthetic cameras around a person performing an action.Self-Similarity matrices (SSMs) are generated for each of these synthetic cameras and for each of these computed SSMs, a gradient angle is computed at each matrix point. From these orientations, circular standard deviations are computed for a golf swing (b), kick (c) and a jumping jack (d) action sequence (code provide by[START_REF] Shen | View invariant action recognition using fundamental ratios[END_REF] for (a)).

Fig. 6 .

 6 Fig. 6. Local descriptors for SSM: each individual descriptor is centered at a diagonal point i ∈ {1 • • • T } of the SSM and has a log-polar block structure. Histograms h a i of 8 gradient directions are computed separately for each of the 11 blocks of the analysis support and are concatenated into a descriptor vector h i .

Fig. 7 .

 7 Fig. 7. Temporal alignment of same action performances in videos from different viewpoints and with synthetic de-synchronization. (a): Two de-synchronized sequences with the side and the top views of the same action are represented with a set of matching key-frames. The second sequence has been time warped according to t = a cos(bt) transformation. (b) Distance matrix between sequences H(I 1 ) and H(I 2 ) of SSM-pos descriptors (bright colors represent large distance values). Dynamic Programming (red curve) finds the minimum cost monotonic path from (0, 0) to (T, T ) in this matrix. This path coincides almost perfectly with the original warping (blue curve) despite drastic view variations.

Fig. 8 .

 8 Fig. 8. Temporal alignment of video sequences representing different performances of actions throwing a ball, drinking and smoking. (Left): Pairs of aligned video sequences are illustrated with a few frames and the links between corresponding frames estimated by our algorithm. (Right): Distance matrices between sequential descriptors of both videos used as input for aligning video sequences by Dynamic Programming. The estimated temporal alignment is illustrated by red curves. The successful alignment achieved by our method on these sequences is confirmed when comparing red curves with yellow dots illustrating sparse manual alignment for a few key-frames of videos (best viewed in color).

  cam5

Fig. 9 .

 9 Fig. 9. SSM-based cross-view action recognition on CMU mocap data. (a) A person figure animated from the motion capture data and six virtual cameras used to simulate projections in our experiments. (b) Accuracy of the cross-view action recognition using SSM-pos-vel-acc descriptors to build the bag of features used by nearest-neighbor classifier.

Fig. 10 .

 10 Fig. 10. SSM-based action recognition on Weizman single-view action dataset [7]. (Top) Example frames for nine classes of actions. (Bottom) Confusion matrices corresponding to NNC action recognition using image-based self-similarities SSM-of (a) and trajectory-based self-similarities SSM-pos (b).

Fig. 11 .

 11 Fig. 11. Example frames from IXMAS multiview action dataset: for four classes of action, the five views at a given instant, of one performance of the action is shown.

Fig. 13 .

 13 Fig.[START_REF] Gilbert | Scale invariant action recognition using compound features mined from dense spatio-temporal corners[END_REF]. Class-confusion matrix for action recognition in IXMAS dataset: this confusion matrix is obtained using SSM-hog-of multi-scale SSM local descriptors. It corresponds to the average confusion computed for all crosscamera recognition setups in Fig.12(a).

TABLE I TYPICAL

 I PATTERNS OF RECURRENCE PLOTS AND THEIR MEANING (REPRODUCED FROM

  ) illustrate recognition accuracy of cross-views action recognition for different combinations of training and test cameras and for different types of SSMs. The results are averaged over all classes and test subjects. Similar to results on CMU dataset in Section V-A, here we observe high stability

		T e s t C a m 0 T e s t C a m 1 T e s t C a m 2 T e s t C a m 3 T e s t C a m 4 T e s t A l l	T e s t C a m 0 T e s t C a m 1 T e s t C a m 2 T e s t C a m 3 T e s t C a m 4 T e s t A l l
	Train Cam0	77.0 75.2 69.7 71.8 49.4 68.6	Train Cam0	66.4 73.3 63.9 60.3 41.5 61.1
	Train Cam1	78.5 77.3 67.9 71.5 48.0 68.6	Train Cam1	67.3 70.6 62.1 62.4 41.5 60.8
	Train Cam2	70.0 73.0 75.8 68.5 55.2 68.5	Train Cam2	63.9 65.8 71.8 57.6 52.1 62.2
	Train Cam3	73.6 72.4 67.3 71.2 45.9 66.1	Train Cam3	62.1 68.2 62.4 61.2 35.9 58.0
	Train Cam4	44.5 41.5 55.2 37.9 68.8 49.6	Train Cam4	34.8 36.4 55.8 33.3 63.0 44.7
	Train All	77.0 78.8 80.0 73.9 63.3 74.6	Train All	67.9 73.6 70.6 66.4 59.1 67.5
	cross-camera training/testing		same camera training/testing	cross-camera training/testing	same camera training/testing
	(a): Recognition results for SSM-hog-of multi-scale features	Recognition results for SSM-of multi-scale features
		T e s t C a m 0	T e s t C a m 1	T e s t C a m 2	T e s t C a m 3	T e s t C a m 4	T e s t A l l	T e s t C a m 0 T e s t C a m 1 T e s t C a m 2 T e s t C a m 3 T e s t C a m 4 T e s t A l l
	Train Cam0	67.6 66.1 54.8 57.3 44.4 58.0	Train Cam0	80.0 75.9 42.3 55.6 21.8 55.6
	Train Cam1	73.6 63.6 57.9 59.5 45.3 60.0	Train Cam1	74.8 83.9 36.5 58.3 23.6 56.0
	Train Cam2	58.2 54.5 63.3 54.2 49.4 55.9	Train Cam2	43.6 46.1 80.5 64.7 34.2 53.7
	Train Cam3	60.0 58.2 55.8 60.6 42.1 55.3	Train Cam3	47.0 50.0 45.8 85.5 18.8 49.5
	Train Cam4	46.7 44.2 51.5 43.9 60.0 49.3	Train Cam4	19.7 19.4 43.5 26.1 73.3 36.0
	Train All	69.7 63.3 64.8 62.7 52.7 62.7	Train All	80.3 84.5 79.4 84.8 68.5 79.6
	cross-camera training/testing		same camera training/testing	cross-camera training/testing	same camera training/testing
	73.7	8.1	0.0		1.1	1.5	1.7	11.3		0.5	0.1	0.0	1.9
	3.8	72.6	1.0		1.8	0.4	0.2	15.7		0.6	0.3	0.0	3.6
	0.5	0.6	72.8	3.6	4.1	1.4	0.2		8.8	7.7	0.4	0.0
	1.7	1.3	3.9		57.7	1.0	15.4	0.9		1.4	14.4	0.9	1.4
	0.9	0.1	1.7		0.4	84.5	1.9	0.7		6.5	1.1	2.0	0.3
	3.3	1.0	0.9		15.1	2.2	70.5	0.0		1.4	2.6	0.0	3.0
	13.5	11.9	1.2		0.6	0.4	0.8	61.1		0.3	0.1	0.7	9.3
	0.6	0.1	9.6		1.1	2.3	1.2	0.1	81.1	3.5	0.2	0.0
	0.0	0.1	4.0		5.2	0.8	0.8	0.0		1.8

(c): Recognition results for SSM-hog multi-scale features (d): Recognition results for STIP-hog-hof multi-scale features

Fig. 12. Comparative action recognition results for IXMAS multiview action dataset: results are averaged over 11 action classes and 10 subjects. Results in (a)-(c) are shown for different types of SSMs and the same bag-of-features SVM classification method. Results in (d) are obtained with the same bag-of-feature SVM approach, but using quantized descriptors of spatio-temporal interest points (STIP) instead of quantized local SSM descriptors. Recognition scores are illustrated for different combination of training and test cameras.

TABLE II COMPARISON

 II OF RECOGNITION RESULTS ON IXMAS DATASET BY ALTERNATIVE METHODS. RESULTS ARE PRESENTED FOR DIFFERENT COMBINATIONS OF TRAINING CAMERA-X AND TEST CAMERA-Y SETUPS WHERE "CROSS CAMERA" INDICATES SETUPS WITH X = Y , "SAME CAMERA" INDICATES SETUPS WITH X = Y AND "ANY-TO-ANY" INDICATES ALL COMBINATIONS OF X AND Y .

		cross camera same camera	any-to-any
	SSM-hog-of	61.8	74.0	64.3
	SSM-of	55.0	66.6	57.4
	SSM-hog	53.9	63.0	55.7
	STIP-hog-hof	42.4	80.6	50.0
	Farhadi [33]	58.1	68.8	60.3
	Weinland [35]	-	57.9	-
		cross camera	same camera any-to-any
	multi-scale	61.8	74.0	64.3
	56 frames	59.9	70.9	61.8
	42 frames	59.3	69.4	61.6
	28 frames	54.0	65.3	56.2

TABLE III COMPARISON

 III OF THE IMPACT OF SSM DESCRIPTOR SIZE ON THE RECOGNITION PERFORMANCE IN IXMAS DATASET.

ACKNOWLEDGEMENTS.

This work was partially funded by the QUAERO project and the MSR/INRIA joint laboratory.


Céline Casenave
email: celine.casenave@supagro.inra.fr

Denis Dochain
email: denis.dochain@uclouvain.be

Jérôme Harmand
email: jerome.harmand@supagro.inra.fr

Marc Perez
email: marc.perez@supagro.inra.fr

Alain Rapaport
email: alain.rapaport@supagro.inra.fr

Jean-Marie Sablayrolles

J-M Sablayrolles

Control of a Multi-Stage Continuous Fermentor for the study of the wine fermentation

Keywords: wine fermentation, cascade of reactors, control design, varying input constraints

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

The alcoholic fermentation is a fundamental step of the wine-making process. During the fermentation, which is performed in a batch reactor, the sugar contained in the grape juice is converted in ethanol and other compounds which give the wine all its aroma. The control of the fermentation process is therefore of importance; in particular, it requires a good knowledge of the yeasts and their physiology. However, studying the yeasts during the batch fermentation process is not easy because the physiological state of the yeasts change all along the fermentation: in particular, the yeasts go through a growth phase and a stationary phase. In [START_REF] Clement | Use of a continuous multistage bioreactor to mimic winemaking fermentation[END_REF] the use of a Multi-Stage Continuous Fermentor (MSCF) has been proposed to mimic the batch wine fermentation process. It is composed of 4 reactors connected in series; at each stage, the input of the reactor corresponds to a fraction of the output of the preceding reactor. With such a device, the yeasts obtained at equilibrium in each stage are in a stable physiological state which corresponds to a key transient state of the batch fermentation. Indeed, the MSCF enables to pass from a time scale (the transient states of the batch fermentation) to a space scale (the equilibrium states of the 4 reactors of the MCSF). Thus, it is possible to get simultaneously (in the 4 reactors) some yeasts in the growth phase and others in the stationary ⋆ The work is supported by the 7th Framework Program of the European Union (CAFE Project -Large Collaborative Project KBBE-2007-2-3-01). phase, in a (time) stable environment and physiological state. Some measurements and experimental tests can then be performed on the yeasts; compared to results obtained from batch fermentations, the obtained results are here completely independent of the reaction kinetics.

The equilibrium points reached in each of the 4 reactors of the MSCF can be characterized in term of the sugar concentration. The problem considered in this paper is the control of the sugar concentration of the 4 reactors. The values of the sugar concentration at equilibrium depends on the value of the input flow rates of the reactors, which are therefore considered as the control inputs.

The control of a continuous bioreactor has been widely studied in automatic control literature [START_REF] Bastin | On-line estimation and adaptive control of bioreactors (Process measurement and control[END_REF]]. This problem is currently encountered in practice, in particular for bioengineering processes used, for example, for the waste water treatment [START_REF] Katebi | Control and instrumentation for wastewater treatment plants[END_REF][START_REF] Olsson | Wastewater treatment systems: modelling, diagnosis and control[END_REF][START_REF] Dochain | Dynamical Modelling and Estimation in Wastewater treatment Processes[END_REF]. In comparison, only a few papers deal with the problem of the control of a cascade of bioreactors [START_REF] Vigie | Adaptive predictive control of a multistage fermentation process[END_REF][START_REF] Simeonov | Modelling and extremum seeking control of a cascade of two anaerobic bioreactors[END_REF]], whereas such kind of devices is often used in industrial engineering processes [START_REF] Chen | Modelling, identifiability and control of complex biotechnological systems[END_REF][START_REF] Hill | Minimum tank volumes for CFST bioreactors in series[END_REF][START_REF] De Gooijer | Bioreactors in series: An overview of design procedures and practical applications[END_REF][START_REF] Harmand | Optimal design of two interconnected bioreactors-some new results[END_REF]. Moreover, the control problems on multistage continuous reactors which have already been studied deal most of the time with the control of a unique quantity. In [START_REF] Vigie | Adaptive predictive control of a multistage fermentation process[END_REF]] for example, the authors indeed design a controller for the glucose concentration in the process outlet. In [START_REF] Simeonov | Modelling and extremum seeking control of a cascade of two anaerobic bioreactors[END_REF]], the objective is the maximization of the biogas production in a two-stage anaerobic bioreactor. However, the authors only propose a strategy that maximizes the biogas flow rate of the first reactor, the second one being not controlled.

In this paper, the goal is to control the sugar concentration in each stage of the reactor with, as control inputs, the four input flow rates. Because of the cascade structure of the process, the values of the control inputs are constrained: indeed, the input flow rate of a stage is necessarily smaller than the output flow rate of the preceding stage. In this paper, a control strategy is proposed, based on a linearizing control law coupled with a state observer and an anti-windup loop. The closed loop system stabilizes faster than when a constant input is applied 1 . It has been validated both in numerical simulation, and on the experimental process.

The paper is organized as follows. In Section 2 the experimental set up is described. The model used for the design of the control law is presented and validated in Section 3. Section 4 deals with the design of the control strategy. Finally in section 5, the results obtained on the experimental process are presented.

PROCESS DESCRIPTION

The experimental set up has been developed by the research unit SPO (Sciences for Oenology) of INRA (Montpellier). It is composed of 4 reactors connected in series (see Figure 1). The output flow rate of each reactor is equal to its input flow rate, in such way that the volumes V i , i = 1 : 4 of the reactors remain constant (as in classical chemostats). The first reactor is fed with a synthetic medium which simulates a grape juice. The other reactors are fed by the outlet medium of the preceding reactor. The input flow rate Q i , i = 1 : 4 of the reactors are controlled independently by some piston pumps, the only constraint (coming from the cascade structure of the device) being that the input flow rate Q i of the i th reactor has to be lower than the output flow rate Q i-1 of the (i -1) th reactor:

0 ≤ Q 4 ≤ Q 3 ≤ Q 2 ≤ Q 1 ≤ Q max , (1)
with Q max the maximal flow rate which can be applied (i.e. each tank is fed from the input of the previous one without any external addition, excepted for the first tank). The temperature of the medium in each reactor is regulated at 28 • C. Only the CO 2 production rate (in each of the 4 reactors) is measured on line. A scheme of the MSCF is given in Figure 1.

MODEL DESCRIPTION AND VALIDATION

Model of the batch fermentation

The model of the batch fermentation considered in this paper is the one described in [START_REF] David | Dynamical modeling of alcoholic fermentation and its link with nitrogen consumption[END_REF]. It is based on the following reaction scheme:

• the yeasts X grow on the nitrogen N , with a yield coefficient

k 1 : N -→ k 1 X (2)
1 The minimal time synthesis presents bang-bang and singular arcs (see [START_REF] Bayen | Optimal synthesis for a minimum time problem in the plane with a triangle control set[END_REF]). It is consequently sensitive to error measurements and therefore difficult to implement in practice • the sugar S is enzymatically degraded in ethanol E and CO 2 , and inhibited by the ethanol, with a yield coefficient k 2 :

R 1 V 1 R 2 V 2 R 3 V 3 R 4 V 4 8°C dCO2/dt (ml/min) 28°C Q 1 Q 2 Q 3 Q 4 Q 4 Q 1 Synthetic Q 1 -Q 2 Q 2 -Q 3 Q 3 -Q 4 Q 2 Q 3 media
S X -→ k 2 E + k 2 CO 2 (3)
The mass balance model is written as follows :

     dX dt = µ 1 (N )X; dN dt = -k 1 dX dt ; dE dt = µ 2 (E, S)X; dS dt = -k 2 dE dt ;
(4) with:

µ 1 (N ) = µ max 1 N K N + N , µ 2 (E, S) = µ max 2 S K S + S K E K E + E , (5)
where X, N, E and S are the concentrations of the yeast, the nitrogen, the ethanol and the sugar (in g.L -1 ) respectively; and

k 1 , k 2 , µ max 1 , µ max 2
, K E and K S are some positive constants. To complete the model, the following initial conditions are considered:

X(0) = X in , N (0) = N in , E(0) = 0, S(0) = S in . (6)
The CO 2 production rate is indeed here equal to µ 2 (E, S)X.

Model of the MSCF

The model of the MSCF is obtained from the model of the batch fermentation by addition of terms related to the flow rates: ∀i = 1 : 4, (P)

                     dX i dt = µ 1 (N i )X i +D i (X i-1 -X i ) dN i dt = -k 1 µ 1 (N i )X i +D i (N i-1 -N i ) dE i dt = µ 2 (E i , S i )X i +D i (E i-1 -E i ) dS i dt = -k 2 µ 2 (E i , S i )X i +D i (S i-1 -S i ) (7)
where

D i = Qi Vi is the dilution rate of the i th reactor, (X 0 , N 0 , E 0 , S 0 ) = (0, N in , 0, S in ) with N in , S in > 0, and µ 1 , µ 2 , k 1 , k 2 , µ max 1 , µ max 2
, K E and K S are the same functions and parameters than the ones of the batch fermentation model [START_REF] Clement | Use of a continuous multistage bioreactor to mimic winemaking fermentation[END_REF]]. In the sequel, we denote C i the CO 2 production rate of the i th reactor:

C i := µ 2 (E i , S i )X i . ( 8
)

Comparison between batch fermentation and multistage continuous fermentation

The design of the experimental is indeed based on the notion of equivalence between batch reactors and plug flow reactors in steady state, and the related notion of space time, widely used in reactor design (e.g. [START_REF] Levenspiel | Chemical Reaction Engineering[END_REF]).

If the input medium of the plug flow reactor is the same than the initial medium of the batch fermentation, then the equilibrium state reached inside the plug flow reactor has the same space evolution than the time evolution of the state variable during the batch fermentation.

The cascade of reactors is indeed aimed at approximating the ideal plug flow reactor behavior (see also e.g. [START_REF] Levenspiel | Chemical Reaction Engineering[END_REF]). In [START_REF] Clement | Use of a continuous multistage bioreactor to mimic winemaking fermentation[END_REF], the concentrations of yeast and nitrogen, and the CO 2 production rate obtained at equilibrium in the 4 reactors of the MSCF for different values of the dilution rates D i are compared to the time evolution of the same quantities obtained with batch fermentation. The obtained results make the authors conclude that the MSCF enables to mimic, at equilibrium, the transient states of the batch fermentation.

From a modeling viewpoint, the MSCF corresponds to a finite difference approximation of the plug flow reactor. This discretization necessarily induces errors; the quality of the approximation will in particular depend on the number of reactors considered. Many numerical studies have shown that in practice a small number of tanks in series, typically 3, with possibly different volumes, is sufficient to well approximate the behavior of a plug flow reactor [START_REF] Hill | Minimum tank volumes for CFST bioreactors in series[END_REF]]. In [START_REF] Clement | Use of a continuous multistage bioreactor to mimic winemaking fermentation[END_REF], they consider that 4 reactors are enough to obtain a good approximation (according to their needs).

There is indeed another source of errors in the approximation : the input medium of the MSCF is not exactly the same than the initial medium of the batch fermentation.

They have the same nitrogen and sugar concentrations, but the yeast concentrations are not equal. In fact there is no yeast in the inlet of the MSCF: the yeasts are inoculated in each reactor of the MSCF, at the starting of the process, as for the batch fermentation. Fortunately this difference does not decrease the quality of the approximation. This is due to the small value of the initial yeast concentration (around 10 6 cell/ml), and to the fact that the values of the concentrations are compared in terms of functions of the fermentation progress 1 -S(t) S in instead of the time. The same comparison as the one made in [START_REF] Clement | Use of a continuous multistage bioreactor to mimic winemaking fermentation[END_REF] with experimental data has been performed with numerical simulations. In Figure 2, the yeast concentration X and the CO 2 production rate µ 2 (E, S)X are plotted versus the fermentation progress 1 -S S in . The plotted values correspond for one part to the transient states of a batch fermentation and for the other part to the equilibrium states reached in the 4 reactors of the MSCF for dilution rates D i ∈ {0.05, 0.1, 0.15, 0.2, 0.25} ∀i = 1 : 4, such that2 D 1 D 2 D 3 D 4 . The results are similar to the ones obtained on the experimental process.

Model parameter identification and comparison with experimental data

The values of the parameters of model ( 7), except the one of3 k 2 , have been identified from experimental data: a simplex method was used to minimize the sum of the squares of the distances between the experimental measurements of the sugar concentration at equilibrium in the 4 stages of the MSCF and the values obtained by numerical simulation of the model. We finally get :

k n = 1.57 [g.L -1 ], k 1 =0.0606[-], µ max 1 =1.34 [h -1 ], k s = 0.0154 [g.L -1 ], k 2 =2.17 [-], µ max 2 =1.45 [h -1 ], k e = 14.1 [g.L -1 ].
The comparison between the experimental data used for the identification process, and the values obtained by simulation with the above set of parameter values is given in Figure 3 (top). For the cross validation, we also compared simulated trajectories of a batch fermentation with experimental data: the results are given in Figure 3 (bottom).

DESIGN OF THE CONTROL LAW

First recall that the objective is to control the sugar concentration in each stage of the MSCF, the control inputs being the input flow rates Q i , i = 1 : 4, of the 4 reactors. In the sequel, we denote S * i the value of the sugar concentration setpoint in the i th reactor (for i = 1 : 4). We only consider setpoint values such that there exist constant input control values (Q

* 1 , Q * 2 , Q * 3 , Q * 4 ) ∈ [0, Q max ]
4 verifying the constraint (1) and which, when applied to system (7), stabilize the sugar concentration in the 4 reactors at the values S * i , i = 1 : 4. Note that this implies that S in > S * 1 > S * 2 > S * 3 > S * 4 > 0. The control design is facing two main difficulties: on one hand, no on-line measurement of the sugar concentration is available, and on the other hand the control inputs Q i have to verify the constraint (1). The proposed control strategy is based on a linearizing control law, coupled with a state observer and an anti-windup component. A scheme of the control strategy is given in Figure 4.

In order to account for the system nonlinearities, a linearizing control law is first considered. The equation of S i has a relative degree equal to 1 (with respect to the control input Q i ); the control law is therefore written as follows :

Q i = V i k 2 C i + v i S i-1 -S i := Ψ i (v i , S, C), (9)
where

C = (C 1 , C 2 , C 3 , C 4 ) T , S = (S 1 , S 2 , S 3 , S 4 )
T and v i is the expression of the desired closed loop dynamic of S i . Here we consider a simple proportional-integral linear dynamics:

v i = a i,1 (S * i -S i ) + a i,2 t 0 (S * i (τ ) -S i (τ )) dτ, (10)
with a i,1 and a i,2 some positive constants. The integral term is introduced to compensate modeling and measurements errors.

The linearizing control law (9) assumes a full knowledge of both the sugar concentrations S i , S i-1 and the CO 2 production rate C i . However, only the CO 2 production rate is measured on-line. To get an on-line estimate of the sugar concentration, we use the following asymptotic observer [START_REF] Bastin | On-line estimation and adaptive control of bioreactors (Process measurement and control[END_REF]] : ∀i = 1 : 4,

(O)    d Ŝi dt (t) = -k 2 C m i (t) + D i (t) Ŝi-1 (t) -Ŝi (t) Ŝi (0) = Ŝ0
i , with Ŝ0 (t) = S in , ∀t > 0 and C m i (t) the measurement of C i (t). The rate of convergence of such an observer can not be adjusted: it depends on the value of D i . However, several other observation strategies (high gain observer, extended Kalman Filter, etc) have been tested, and no ones give better results: the convergence rate is always limited by the value of D i . We therefore have chosen to consider the asymptotic observer (O) which has the advantage to be very simple and robust because it requires the only knowledge of the parameter k 2 of the model, the value of which is reliable. Note that the observation problem with production rate measurement in the chemostat is difficult because of the existence of "bad" inputs. When an input D i (•) is set to zero, the estimator does not converge asymptotically. One may then possibly switch to another observer specifically designed for the zero input (see [START_REF] Rapaport | Design of exponential observers for nonlinear systems by embedding[END_REF] for a similar situation), but one can check that even in this simpler case, it is not possible to write the system in the observability form [START_REF] Gauthier | Deterministic Observation Theory and Applications[END_REF]. Theoretically, there should exists an observer of Kazantsis-Kravaris form (or an approximation) [START_REF] Andrieu | On the Existence of a Kazantzis-Kravaris/Luenberger Observer[END_REF] but it remains an open problem to write it explicitly.

Finally, the quality of the estimation of S will essentially depend on the quality of the initialization of the observer. In practice, the control law will be used to go from one equilibrium point to another. As a consequence, we assume that, at the observer initialization time t = 0, the system is at equilibrium. Under this assumption, we get, from (7), the following estimation Ŝ0

i of S i (0):

Ŝ0 i = -k 2 C 0 i,m D i (0) + Ŝ0 i-1 , i = 1 : 4. ( 11
)
where C 0 i,m is the measurement of C i (0) and Ŝ0 0 = S in . In practice, some off-line measurements of the sugar concentration are available. The information given by these measurements, even if they are available only a few hours after the sampling, will be used to adjust the estimation of the observer (see section 5).

The values Q i of Q i given by the control law (9) do not necessarily meet the constraint (1). To take this constraint into account, we use the following saturation operator :

sat(u; u m , u M ) := u M if u u M u if u m < u < u M u m if u u m . (12)
Compared to problems often considered in the literature, the constraints on the control inputs are coupled and time-varying. We chose to apply the saturation operator (12) to one control input after the other. The choice of the saturation order of the 4 control inputs Q i is not obvious: it can lead to very different control performances, depending on the setpoints, the initial conditions, and the experimental conditions. The 14 possible saturation orders are given in Table 1, n i = j meaning that Q i will be the j th control input to be saturated. For example, the order 1 2 3 4 5 6 7 8 9 10 11 12 13 14

n 1 1 1 1 1 1 2 2 2 3 2 2 3 4 3 n 2 2 2 3 3 4 1 1 3 2 3 4 2 3 4 n 3 3 4 2 4 3 2 3 1 1 4 3 3 2 2 n 4 4 3 3 2 2 3 2 2 2 1 1 1 1 1
Table 1. The 14 possible saturation orders.

n • 8 (which is the one used for experiments presented in Section 2) has to be understood as follows:

1. n 3 = 1 =⇒ Q 3 = sat( Q 3 ; 0, Q max ) 2. n 1 = n 4 = 2 =⇒ Q 1 = Q 4 = sat( Q 1 ; Q 3 , Q max ) sat( Q 4 ; 0, Q 3 ) 3. n 2 = 3 =⇒ Q 2 = sat( Q 2 ; Q 3 , Q 1 )
The saturation strategy can be expressed as a recurrence of the form:

Q i = sat Q i ; Q i,m (Q j,nj <ni ), Q i,M (Q j,nj <ni ) ,
where Q i,m and Q i,M depend on the chosen order.

It is well known that enforcing input control constraints by application of a saturation operator can result in poor closed-loop performance and overshooting of the integral term. To minimize the performance loss, some anti-windup control schemes have been developed. However, the problems studied in the literature are most of the time linear. In [START_REF] Doyle | An anti-windup input-output linearization scheme for SISO systems[END_REF] and [START_REF] Kendi | An anti-windup scheme for multivariable nonlinear systems[END_REF], the authors propose some anti-windup schemes for SISO and MIMO control affine nonlinear systems with constrained inputs. The anti-windup scheme we considered here comes from these works. It is based on one hand on the transformation of the constraint on Q i in a constraint on v i . Indeed, as the function Ψ i defined in ( 9) is linear with respect to v i , we have:

Q i+1 <Q i < Q i-1 ⇔ Φ i,i+1 (v i+1 , S, C) <v i < Φ i,i-1 (v i-1 , S, C), (13)
where Φ i,j depends on Ψ i and Ψ j . In our case, we have:

Φ i,j (v j , S, C) = S i-1 -S i V i Ψ j (v j , S, C) -k 2 C i . (14)
Consider some values (ṽ 1 , ṽ2 , ṽ3 , ṽ4 ) which do not verify the transformed constraint (13). By applying the same saturation order than for the Q i , i = 1 : 4 values, we can express the transformed saturation strategy as follows : v i = sat ṽi ; ṽi,m (v j,nj <ni , C, S), ṽi,M (v j,nj <ni , C, S) , where ṽi,m and ṽi,M depend on the chosen order.

The values of ṽi are then given by (see Figure 4) : 15) where L stands for the Laplace transform and with

L(ṽ i )(p) = K 1 (p)L(S * i -Ŝi )(p) -K 2 (p)L(v i )(p) (
K 1 (p) = a i,1 and K 2 (p) = - a i,2 a i,1 p + a i,2 , (16)
a i,1 and a i,2 being chosen in such way that the control law obtained with v i given by ( 10) without saturation constraint gives good performances.

S * i + + ṽi v i Q i Q i C i + - C m i Ŝ Ŝi - Ŝj,j =i C m j,j =i v j,n j <n i Q j,n j <n i C m noise + (t) (t)

EXPERIMENTAL VALIDATION

The proposed control strategy has been validated first on numerical simulations, then on the experimental set up.

The synthetic media used for the experiments contained S in = 200 g.L -1 of glucose, and N in = 0.465 g.L -1 of assimilable nitrogen. At the beginning of the experiment, the yeasts were inoculated in each reactor, with an initial concentration X in around 10 6 cell/ml which corresponds to 0.04g.L -1 for the yeast strain used (the commercial strain EC1118, Lallemand SA). The values of the setpoints S * i , the volumes V i and the initial dilution rates D i (0) are given in Table 2.

For the control law, the following parameters values were considered: k 2 = 2.17 [-],

Q max = 0.25 [L.h -1 ], and ∀i = 1 : 4, a i,1 = 1.2 [h -1 ], a i,2 = 1 [h -2 ].
For technical reasons (not detailed here), the constraint (1) has been replaced by the stronger following one:

Q 1 < Q max and ∀i = 2 : 4, 0 < Q i < 0.9 Q i-1 . ( 17
)
The saturation strategy presented in section 4 has therefore been modified consequently. The experimental results are presented in Figure 5. The estimated sugar concentrations given by the observer are first plotted in Figure 5 experiment, two off-line measurements of the sugar concentration were performed: one at the beginning of the experiment, and the other one at the end. The measurements of the initial sugar concentrations were available only at time t 1 = 4.33 h, and have been taken into account in the control strategy at this time, to adjust the estimation of the observer. The effects of such an adjustment are visible in Figure 5.a. The measurements of the final sugar concentrations are given in Table 2 for comparison with the setpoint values.

[g.L -1 ] [L] rate [h -1 ] i S * i S i S * i -S * i-1 S i -S i-1 V i D i (
As we can see, the qualitative behavior of the control law is good and similar to the one obtained in numerical simulation. However, without any on-line sugar concentration measurement, it is not possible to completely cancel the control error, which is even though smaller than 8.8 g.L -1 in all the reactors. The control strategy proposed in this paper does not really control the sugar concentration in each reactor, but rather the differences of sugar concentration between 2 consecutive reactors. The comparison between the values of the difference between consecutive setpoints and measurements (see Table 2) shows that the position error essentially comes from the first reactor. Indeed, for i = 2 : 4, the absolute value of the difference between S i -S i-1 and S * i -S * i-1 is smaller than 2.1 g.L -1 , whereas it is equal to 6.1 g.L -1 for i = 1. Two main reasons can be given to explain the huge value of the error obtained in the first reactor. Firstly, this error depends on the value of S in which can be ill-known. And secondly, the first reactor of the MSCF corresponds to the starting of the batch fermentation which is a complex and the most ill-known stage of the fermentation. During this phase, a part of the CO 2 produced is dissolved in the medium and can therefore not be measured.

CONCLUSION

In this paper, the problem of the control of the sugar concentration in each of the reactors of a Multi-Stage Continuous Fermenter (MSCF) is studied. Because of the structure of the system (cascade), the control inputs (that is the input flow rates of each reactor) are constrained: the input flow rate of a reactor is necessarily smaller than the one of the preceding reactor. To solve this problem, a control strategy has been proposed: it is based on a linearizing control law coupled with an observer and an antiwindup component. The control law has been validated first on numerical simulation and then on experiments. The obtained results are convincing. However, we still have a position error due to the lack of on-line sugar concentration measurement. This error essentially comes from the first reactor of the MSCF. To mitigate this problem, a solution would consist in controlling the nitrogen or yeast concentration instead of the sugar one in the first reactor. Indeed, in the first reactor, the equilibrium state can indifferently be characterized by the nitrogen, the yeast or the sugar concentration, which is not the case of the other reactors, in which the nitrogen concentration can be too small and the yeast concentration at its maximum value.

Fig. 1 .

 1 Fig. 1. Scheme of the Multi Stage Continuous Fermenter (MSCF).

Fig. 2 .

 2 Fig. 2. Numerical comparison between batch fermentation and equilibrium states of the MSCF.

Fig. 3 .

 3 Fig. 3. Comparison between experimental data and simulations of the MSCF model. Top: sugar concentration values at equilibrium. The experimental data are those have been used for the identification process. Bottom: yeast, nitrogen, ethanol and sugar concentrations of a batch fermentation (cross validation).

Fig. 4 .

 4 Fig. 4. Control strategy diagram of the i th reactor.

Fig. 5 .

 5 Fig. 5. Experimental results of the control of the sugar concentrations in the 4 reactors of the MSCF.

We consider here 4 reactors with the same volume V := V i , i = 1 : 4.

The value of parameter k 2 is considered as known; we used the value given in[START_REF] David | Dynamical modeling of alcoholic fermentation and its link with nitrogen consumption[END_REF].

ACKNOWLEDGEMENTS

Special thanks to our former PhD student J. Fernandez, enrolled in the European Project CAFE, but that unfortunately could not have carried on his first investigations.