Nicholas Michael
email: nicholas.michael@cut.ac.cy

Andreas Lanitis
email: bandreas.lanitis@cut.ac.cy

Model-Based Generation of Realistic 3D Full Body Avatars from Uncalibrated Multi-view Photographs

Keywords: personalized avatars, 3D body shape modelling, multi-view ASM, image segmentation

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

In today's world of rapid technological advancement, we find an increasing demand for low-cost systems that are capable of fast and easy generation of realistic avatars with minimal user intervention. Such systems have applicability in many Virtual Reality (VR) applications. For example, they can be used in cultural heritage visualizations and to populate virtual worlds in multi-player computer games, enhancing a user's immersion experience. In addition, they can facilitate communication and education when incorporated in chat applications and in virtual classrooms. They can even assist users to make shopping decisions by allowing them to dress their avatars accordingly. Existing technologies make it possible to create 3D avatars, e.g., using 3D scanning devices such as Microsoft's Kinect, using 3D modelling software, etc. However, this kind of technologies tend to require dedicated hardware, calibration of imaging equipment, creative skills and considerable amount of post-processing manual intervention.

Motivated by this multitude of applications and the limitations of existing technologies, we develop a model-based method for the generation of realistic personalized fullbody 3D avatars [START_REF] Michael | Increasing engagement in re-designed classic video games[END_REF]. The novelty of our work is that it is automatic and it works using photographs taken by any uncalibrated low cost camera. Our method extends the work of Hilton et. al. [START_REF] Hilton | Whole-body modelling of people from multiview images to populate virtual worlds[END_REF], however ours does not require the user to mark the location of 3D landmarks in the 2D image, in order to extract model-silhouette correspondences, for the purpose of improving accuracy. Furthermore, the proposed method does not require the user to measure the camera's field of view and the subject's distance to the camera, which serve as camera calibration. Instead, and as our main contribution, we train and use a multi-view Active Shape Model (ASM) [START_REF] Cootes | Active shape models-their training and application[END_REF] of human silhouettes (as viewed from the front, left, right and back), in order to refine the extracted silhouettes obtained from the segmentation result and to register them to the projection of the 3D shape model in each of the orthogonal views. Our only assumption is that when taking the multi-view input images the user does not significantly change their depth relative to the camera. This means that the avatars can be generated quickly and easily, as no time is wasted for calibrating the camera nor for marking correspondences. Additionally, in order to increase the recognizability and hence the realism of the generated avatar, together with the full-body model we use a face-only 3D model, which has a significantly higher resolution aimed at capturing the more detailed facial characteristics.

An outline of our method is illustrated in Figure 1. First we train a statistical model of 3D human shape using a dataset of dense full-body 3D scans [START_REF] Hasler | A statistical model of human pose and body shape[END_REF]. We project the 3D scans of the dataset to orthogonal views and extract the corresponding silhouettes, which we use to train our multi-view ASM [START_REF] Cootes | Active shape models-their training and application[END_REF], thus concluding the training phase. Once these two models are trained, we can deploy the proposed technique live as follows. Orthogonal input images are captured using an uncalibrated low cost camera and we extract 2D silhouettes using background subtraction and other image processing techniques. The extracted silhouettes are refined and registered to the 3D model's projection using the trained multi-view ASM. For improved accuracy around the face, we detect facial features (eyes, nose and mouth) using Viola/Jones detectors [START_REF] Viola | Robust real-time object detection[END_REF] and combine the detection result with the registered silhouettes, while performing an optimization over the space of permissible 3D shape parameter. Once the shape is reconstructed, the two 3D models (full-body and face-only) are aligned with each other, using a rigid transformation and then texture from input images is mapped to both models assuming a cylindrical model [START_REF] Hilton | Whole-body modelling of people from multiview images to populate virtual worlds[END_REF].

The remainder of our paper is organized as follows. Section 2 covers previous work on the problem of avatar generation. Section 3 discusses in detail each phase of our proposed method, such that Sect. 3.1 covers the components of the off-line model training phase and Sect. 3.2 describes the steps involved in the avatar generation phase. Section 4 presents experimentation for the evaluation of our work. We conclude with Sect. 5 where we also mention a few thoughts on possible future work.

Related Work

Previous efforts on the generation of realistic human-like avatars can be categorized into two groups. In one group are model-based methods (such as [START_REF] Hilton | Whole-body modelling of people from multiview images to populate virtual worlds[END_REF], [START_REF] Anguelov | Scape: Shape completion and animation of people[END_REF], [START_REF] Lee | Generating animatable 3d virtual humans from photographs[END_REF], [START_REF] Ahmed | Automatic generation of personalized human avatars from multi-view video[END_REF], [START_REF] Starck | Model-based multiple view reconstruction of people[END_REF] and [START_REF] Weiss | Home 3d body scans from noisy image and range data[END_REF]) that rely on a model of human body shape to reconstruct the subject's geometry from a multi-view camera setup. In the other group are model-free methods (such as [START_REF] Cui | Kinectavatar: Fully automatic body capture using a single kinect[END_REF], [START_REF] Furukawa | Accurate, dense, and robust multiview stereopsis[END_REF], [START_REF] Starck | Video-based character animation[END_REF], [START_REF] Tong | Scanning 3d full human bodies using kinects[END_REF] and [START_REF] Vlasic | Dynamic shape capture using multi-view photometric stereo[END_REF]), which do not rely on a human body shape model and perform multi-view reconstruction, using for example, multi-view photometric stereo [START_REF] Furukawa | Accurate, dense, and robust multiview stereopsis[END_REF], [START_REF] Vlasic | Dynamic shape capture using multi-view photometric stereo[END_REF] or even a setup of inexpensive range scanners like those found in the recently popularized Kinect device [START_REF] Cui | Kinectavatar: Fully automatic body capture using a single kinect[END_REF], [START_REF] Tong | Scanning 3d full human bodies using kinects[END_REF], [START_REF] Mashalkar | Personalized animatable avatars from depth data[END_REF]. Model-based approaches use a prior model the human body shape, which allows for consistent reconstruction across frames but the reconstructed shapes are limited by the utilized model, in that they may not accurately reconstruct a previously unseen human body shape that is not consistent with the training set. Model-free methods are more dynamic in nature and can reconstruct shape from any scene but because of their lack of a prior model, their reconstructed shapes may not be consistent even across neighboring frames and cannot handle ambiguities. An important advantage of modelbased approaches is the uniformity of the avatars created in terms of the number and positioning of polygons, that facilitates the animation of the resulting avatars in virtual environments. In our approach we adopt a PCA model-based approach, since it naturally enforces consistency in the reconstructed shape across the four orthogonal views.

Methodology

The proposed avatar generation method involves two main phases: (i) the off-line model training phase and (ii) the live avatar generation phase. Figure 1 illustrates the overall work flow of our proposed technique. In the following subsections we describe the various steps in detail.

Off-line Model Training

In the off-line phase, we use a dataset of full-body range scans [START_REF] Hasler | A statistical model of human pose and body shape[END_REF] (selecting only the 111 samples with a neutral pose -see Fig.

3D Model

In the training phase we utilize a Principal Component Analysis (PCA) model trained on range scans from a 3D body scan dataset [START_REF] Hasler | A statistical model of human pose and body shape[END_REF] to extend the work of [START_REF] Hilton | Whole-body modelling of people from multiview images to populate virtual worlds[END_REF], [START_REF] Starck | Model-based multiple view reconstruction of people[END_REF] and [START_REF] Starck | Video-based character animation[END_REF]. The purpose of the PCA model is to learn the permissible modes of human body 3D shape variations reflected in the training set, so that the avatars we generate will have a realistic human shape. Each range scan is represented as a column vector, x, of vertex coordinates such that x = [x 1 , ..., x N , y 1 , ..., y N , z 1 , ..., z N ] , where N represents the number of vertices in the 3D mesh (in our case N = 6449). The range scans are aligned using Procrustes Alignment [START_REF] Kendall | A survey of the statistical theory of shape[END_REF].

We apply eigen-decomposition on the covariance matrix of the aligned shapes, while keeping only the first m out of the resulting n eigenvectors, sorted in order of decreasing eigenvalue, λ i , such that:

arg max 1≤m≤n m i=1 λ i / n i=1 λ i < v , (1)
where v ∈ [0, 1] is the amount of variance that we want the PCA model to capture (we have set v = 0.98 in our experiments, which resulted in m = 13). In this way, any new

x i , can be represented as:

x i ≈ x + Cb i , (2)
where x is the mean shape of the model, C are the principal m eigenvectors of the learned shape manifold and b i is a column vector of shape parameters, also known as the encoding of shape x i . This encoding vector b is typically truncated, so that each element satisfies b

i ∈ [-2 √ λ i , +2 √ λ i ],
where usually k = 2. This ensures that any shapes generated by (2) remain plausible with respect to the training set.

Silhouette Multi-view ASM We project the 3D range scans of the training set to four orthogonal views (front, left, right and back) extracting the corresponding 2D silhouette contour in each view. Each silhouette contour is represented as a column vector, y i , of 2D coordinates such that y i = [x 1 , ..., x C , y 1 , ..., y C ] , where C represents the number of points in the 2D contour and i ∈ {front, left, right, back}. For each training range scan, the set of four silhouette vectors, representing its multi-view orthogonal 2D projection, is stacked vertically, resulting in the augmented column vector: y = [y front , y left , y right , y back ] .

The set of augmented column vectors is aligned using Procrustes Alignment and subsequently used to train by application of PCA a multi-view ASM model (we set v = 0.98, yielding m = 33), instead of training a separate ASM model per view. In this way, during the ASM search algorithm [START_REF] Cootes | Active shape models-their training and application[END_REF] the shape parameters are optimized across all four views simultaneously, instead of independently, providing robustness to outliers, as it naturally enforces shape consistency across all views, yielding a more accurate result. Figure 3 illustrates the modes of variation learned by the first few eigenvectors of the trained multi-view ASM.

Avatar Generation

In the live phase, we extract the subject's silhouettes and then we use the trained models to reconstruct the 3D shape of the subject, register the face model and perform texture mapping (see Fig. 1 (c-h)).

Image Acquisition Image acquisition is done using a low cost camera. First we photograph the background and then the subject is photographed from the front, left, right and back, using a tripod to keep the camera stationary. We assume that the subject maintains a constant distance from the camera during the acquisition.

Segmentation The segmentation step is needed to extract the silhouettes in each view. First the input images are converted to the CIELAB colorspace and we obtain their difference image with respect to the background. The difference image is then thresholded and we apply erosion and dilation to merge foreground regions, selecting the largest connected component as the foreground. We extract the contour by following 8connected adjacent pixels on the silhouette boundary. Figure 5 illustrates the procedure. Multi-view ASM Fitting The procedure for fitting the trained multi-view ASM to a set of four silhouette images follows the ASM search algorithm presented in [START_REF] Cootes | Active shape models-their training and application[END_REF].

For simplicity, we chose the search function in such a way that the model drives itself towards regions of high gradient, i.e. strong edges, which tend to coincide with foreground-background boundaries. In each iteration, each landmark searches a window in its neighborhood for the point of strongest gradient and moves towards it. Once all points have moved to new locations, the new shape vector is projected to the ASM shape manifold to regularize it and the resulting shape parameter vector is truncated to maintain shape plausibility. The ASM search terminates once the 2-norm of the shape parameter vector stabilizes within an -value between successive iterations (see Fig. 6).

3D Model Fitting Silhouette points on the reconstructed 3D shape when projected to each of the four views, should minimize the RMS error with the corresponding silhouette points in the input images. Therefore, we seek to find the shape parameter vector, b * , which minimizes the constrained objective given below:

b * = arg min b i 1 N i P i (x + Cb) -Q i Y i 2 , (3)
such that. b j ∈ [-2 λ j , +2 λ j ] , (4)
where i ∈ {front, left, right, back}, P i is the projection matrix, Y i is the matrix of 2D coordinates of the silhouette contour in the input image, Q i is the alignment transformation matrix and N i is the number of silhouette contour points in the i th view.

We solve the problem in (3) using CVX, a package for specifying solving convex programs [START_REF] Grant | Graph implementations for nonsmooth convex programs[END_REF], [START_REF] Grant | CVX: Matlab software for disciplined convex programming[END_REF].

Texture Mapping Once we have the shape of the avatar we fill-in the realistic appearance by mapping texture information from the input images to the projected model vertices using the cylindrical model described in [START_REF] Hilton | Whole-body modelling of people from multiview images to populate virtual worlds[END_REF]. This involves converting the coordinates of each reconstructed 3D vertex to polar coordinates to determine which image needs to be looked up to get the texture. Then the 3D coordinates are projected to 2D image coordinates (using P i from (3) above) to get the color value at the projected image pixel.

In order to obtain an avatar with a more realistic appearance, we then register the facial landmarks (nose, eyes, mouth) of the higher resolution face model (11655 vertices) to corresponding landmarks on the full-body 3D model. This allows us to determine the rigid transformation that aligns the two models. Hence, we transform the higher resolution face model and apply the texture mapping process again, transferring the realistic appearance to the high resolution face model (sample results shown in Fig. 4).

Evaluation

In order to evaluate the ability of the trained model to reconstruct a subject's 3D body shape when given four orthogonal images of the body's 2D projection, we conducted the following experiment. Using the trained PCA model of 3D body shape, we trained regression functions [START_REF] Hasler | A statistical model of human pose and body shape[END_REF] for various physical shape-controlling parameters e.g., height. In this way, we were able to synthesize 100 new shape models (50 male, 50 female) having desired physical measurements: From each synthetic shape we constructed four orthogonal images of its 2D projection (front, back, left, right). Landmark correspondences between image and model points were manually marked. The goal was then to recover the 3D synthetic shape when presented with only its four projection images and the manually marked landmark correspondences, and assess the 3D shape reconstruction accuracy. The shape parameters were estimated by minimizing a cost function of the average error between the image landmarks and the corresponding projected landmarks of the model. We performed the experiment twice, once using on average only 17 landmarks per view (selected similarly to the feature points in [START_REF] Hilton | Whole-body modelling of people from multiview images to populate virtual worlds[END_REF]) and once using 144 landmarks per view (sampled regularly around the silhouette outline). For each shape we then calculated the average relative point-to-point reconstruction error (calculated as the absolute difference between the reconstructed projection and the ground truth projection of each landmark, averaged over all 6449 landmarks; this was then divided by the maximum point to point distance between model vertices to yield the average relative point-topoint reconstruction error). The results were as follows: (i) 144 landmarks per view: µ = 0, 81%, σ = 0, 1803, (ii) 17 landmarks per view: µ = 0, 90%, σ = 0, 2167.

Fig. 1 .

 1 Fig. 1. Method overview (dotted line separates training and live phases): (a) A 3D shape model is trained by applying PCA on a dataset of dense full-body 3D scans, (b) A multi-view Active Shape Model is trained from the 2D silhouettes of the full-body 3D dataset, (c) Orthogonal input images are captured using an uncalibrated low cost camera, (d) Extract silhouettes using background subtraction, (e) Refine and register the silhouettes to the 3D model's projection using the multi-view ASM, (f) Detect facial features using Viola/Jones detectors, (g) Given the registered silhouettes and detected facial features optimize over the 3D shape parameters and map the texture from the input images, (h) Frontal and side views of generated 3D avatar

Fig. 2 .

 2 Fig. 2. (Left) Sample scans in used dataset [4]. Only 111 (in neutral pose) were used for training, (Right) The model can be used to generate random populations that follow the distribution of the training set by varying the shape parameters to vary e.g., height, weight, muscle/fat ratio, etc.

2

 2

  (right)) to train a 3D full-body shape model. Then the training samples are projected to four orthogonal views (front, left, right, back) to generate a training set of corresponding multi-view silhouettes, which we use to train a multi-view ASM model (see Fig. 1 (a-b)).

Fig. 3 .Fig. 4 .Fig. 5 .

 345 Fig. 3. Illustration of the modes variation in the first 6 eigenvectors of our trained silhouette multi-view ASM for the frontal (top) and left-side (bottom) views. Blue plots represent the mean shape, while the green and red plots represent a variation from this mean by an amount of -2 √ λi and +2 √ λi, respectively, where λi represents the i th eigenvalue and i ∈ {1, ...6}

-

  Weight: varied uniformly in the range [50kgs, 120kgs] -Height: varied uniformly in the range [150cm, 200cm] -% Muscle: varied uniformly in the range [0%, 100%]

Acknowledgements

This work was partially supported by the Cyprus Research Promotion Foundation and the European Union Structural Funds (project VR-CAVE: IPE/NEKYP/0311/02). We would like to thank Dr Nils Hasler for providing the 3D human body database [4].

Finally, we calculated the sample correlations and p-values to determine tions between the reconstruction error and the physical parameters. We found that the most significant correlations of reconstruction error were between weight and % muscle. More specifically, for weight we got ρ = 0.2685 and p-value=0.0069, indicating that error increases with increasing weight. For % muscle we got ρ = 0.3503 and p-value=0.0004, indicating that error increases with increasing % muscle because higher % muscle cause more bulging of the body thus greater deviation from the mean shape.

In another experiment, we evaluated the accuracy of the multi-view ASM against the accuracy of four separate single-view ASM models. The results, which are shown in Fig. 6 illustrate the superiority of the multi-view ASM. We are in the process of performing evaluation simulations of the proposed method against a large synthetic dataset as well as quantitative evaluation on real data. See Fig. 4 for a visual evaluation. Fig. 6. Comparative evaluation of our silhouette registration method by a multi-view ASM on a synthetic dataset of 100 subjects. (Left) Plot of average relative silhouette registration error for each view, using single-view ASM models (98% variance). (Right) Corresponding plot evaluating silhouette registration error using a multi-view ASM model.

Conclusions and Future Work

We presented our ongoing work towards the generation of realistic animation ready avatars that will allow users and their friends to participate and collaborate in VR game adaptations. While this is work in progress and additional evaluation experiments are under way, early results obtained prove the potential of our overall approach. In the future we plan to increase the accuracy of model fitting and texture mapping so that the end result is cleaner and more realistic. Furthermore, we plan to add automatic rigging and animation on the personalized avatars, so that they can be easily incorporated in collaborative VR applications.


Christos Mousas
email: c.mousas@sussex.ac.uk

Paul Newbury
email: p.newbury@sussex.ac.uk

Christos-Nikolaos Anagnostopoulos

Data-Driven Motion Reconstruction Using Local Regression Models

Keywords: character animation, local regressions, motion reconstruction

Reconstructing human motion data using a few input signals or trajectories is always challenging problem. This is due to the difficulty of reconstructing natural human motion since the low-dimensional control parameters cannot be directly used to reconstruct the high-dimensional human motion. Because of this limitation, a novel methodology is introduced in this paper that takes benefit of local dimensionality reduction techniques to reconstruct accurate and naturallooking full-body motion sequences using fewer number of input. In the proposed methodology, a group of local dynamic regression models is formed from pre-captured motion data to support the prior learning process that reconstructs the full-body motion of the character. The evaluation that held out has shown that such a methodology can reconstruct more accurate motion sequences than possible with other statistical models.

Introduction

Full-body motion reconstruction is a process that is quite important in cases in which the ability to animate virtual characters while using a reduced number of sensors or user defined trajectories is necessary. Such techniques, especially those that are developed to reconstruct the motion of the character during the performance capture process can be quite beneficial in various areas that are related to virtual reality, such as rehabilitation and sports training, as well as in video games. Although various motion capture systems for capture of the user's performance, such as Vicon [START_REF]Vicon Motion Capture Solution[END_REF] and XSens [START_REF]Xsens Motion Capture Solution[END_REF], can provide desirable results, the basic limitation is the high cost for general family use. Recently, low-cost commercial products, such as those provided by Microsoft, Sony, and Nintendo, have developed next generation hardware devices to capture the online performances of individual players. However, a reduced number of input signals retrieved from those devices cause the motion reconstruction process to be challenging.

The reason is that human motion has many degrees of freedom (DOF) and, therefore, motion models are required that can reconstruct the natural and realistic motion of a character while using few parameters.

In the proposed methodology, the motion reconstruction process is formulated in a maximum a posteriori (MAP) framework that is responsible for producing a naturallooking motion sequence that best matches the user-defined inputs. Specifically, the proposed methodology learns a group of local regression models in order to constrain the prior learning of the pre-captured motion data. Then, by searching within the motion database, by using K nearest motion examples that are similar to the previously reconstructed poses q t-1 , ..., q t-m , and the motion sequences q t k-1 , ..., q t k-m along with their subsequent poses q t k for k = 1, ..., K as the training data, it learns a predictive model for the reconstruction of the current character's posture q t .

The proposed methodology can reconstruct a variety of motion sequences by using a reduced number of input trajectories, such as walking, running, jumping, and punching, as well as golf swings. Further, by evaluating the presented methodology with previous solutions for reconstructing the character's motion, the proposed approach can reconstruct motion sequences by reducing the reconstruction error. The remainder of this paper is organized as follows: Section 2 presents related work in data-driven character animation and motion reconstruction. Section 3 provides an overview of the proposed methodology. The proposed reconstruction methodology is explained in Section 4. Section 5, presents the results obtained from the evaluations of the proposed methodology versus those oft previously examined techniques. Finally, conclusions are drawn and the potential future work is discussed in Section 6.

Related Work

In data-driven motion synthesis techniques [START_REF] Liu | Performance-Bbased Control Interfaces Using Mixture of Factor Analyzers[END_REF], the low-dimensional control signals that are obtained from the motion capture device are used to retrieve suitable motion sequences from a database that contains high-dimensional motion capture sequences. The reuse of pre-recorded human motion data requires efficient retrieval of similar motions from databases [START_REF] Keogh | Indexing Large Human-Motion Databases[END_REF] [START_REF] Müller | Efficient Content-Based Retrieval of Motion Capture Data[END_REF], as well as a good understanding of how motions must be parameterized in order to yield smooth transitions between several retrieved motion clips [START_REF] Kovar | Flexible Automatic Motion Blending with Registration Curves[END_REF].

On the other hand, statistical motion models are often described as several mathematical functions that represent human motion by a set of parameters that are associated with probability distributions [START_REF] Mousas | Evaluating the Covariance Matrix Constraints for Data-Driven Statistical Human Motion Reconstruction[END_REF]. So far, using pre-captured motion data to learn statistical motion models have been used for full-body character control [START_REF] Wei | Intuitive Interactive Human-Character Posing with Millions of Example Poses[END_REF], in key frames interpolation [START_REF] Li | Motion Texture: a Two-Level Statistical Model for Character Motion Synthesis[END_REF], motion styles synthesis [START_REF] Brand | Style Machines[END_REF] [START_REF] Mousas | Motion Style transfer in Correlated Motion Spaces[END_REF], facial animation [START_REF] Weise | Realtime Performance-Based Facial Animation[END_REF] and speechdriven facial expressions [START_REF] Bregler | Video Rewrite: Driving Visual Speech with Audio[END_REF][14], hands-over animation techniques [START_REF] Mousas | Efficient Hand-Over Motion Reconstruction[END_REF][16], interactive creation of a character's pose [START_REF] Wei | Intuitive Interactive Human-Character Posing with Millions of Example Poses[END_REF] [START_REF] Grochow | Style-Based Inverse Kinematics[END_REF] or control of human actions using vision-based tracking [START_REF] Chai | Performance Animation from Low-Dimensional Control Signals[END_REF], real-time human motion control with inertial sensors [START_REF] Liu | Realtime Human Motion Control with a Small Number of Inertial Sensors[END_REF] or accelerometer sensors [START_REF] Grochow | Style-Based Inverse Kinematics[END_REF], construction of physically-valid motion models for human motion synthesis [START_REF] Brand | Voice Puppetry[END_REF] and so forth.

During the past years, a number of researchers have developed approaches that use sparse constraints provided by sensors to control high-dimensional human motions. A single depth camera to track and reconstruct various human motions, their approaches acquired no makers attached on user's body, however, no less than 15 control points needed to be used to segment the human body [START_REF] Shotton | Real-Time Human Pose Recognition in Parts from a Single Depth Image[END_REF] [START_REF] Wei | Accurate Realtime Full-Body Motion Capture Using a Single Depth Camera[END_REF]. By combining inverse kinematics algorithms and a few constraints from eight magnetic sensors to provide an analytic solution for human motion control [START_REF] Semwal | Mapping Algorithms for Real-Time Control of an Avatar Using Eight Sensors[END_REF]. By using six to nine retro-reflective markers as the control points for online human motion reconstruction [START_REF] Chai | Performance Animation from Low-Dimensional Control Signals[END_REF], and by using five inertial sensors for real-time upper-body control [START_REF] Slyper | Action Capture with Accelerometers[END_REF]. Recently, full-body human motion control was achieved using the positional and orientational constraints from six inertial sensors [START_REF] Liu | Realtime Human Motion Control with a Small Number of Inertial Sensors[END_REF]. Finally, another solution [START_REF] Grochow | Style-Based Inverse Kinematics[END_REF] utilizing a few constraints provided by four accelerometer sensors achieved full-body motion control of the character.

Overview

In the proposed methodology a reduced number of input trajectories retrieved from a reference motion sequence are used to reconstruct the full-body motion of the character. Those input trajectories automatically transform the control inputs into realistic human motion by building a group of online local regression models during the runtime. An overview of the methodology appears in Figure 1. For the motion reconstruction process, it is assumed that the character's actions can be represented as an morder Markov chain, such as the current posture of the character at the tth frame q t can be considered to depend only on previous m reconstructed postures Q t,m = [q t-1 , ..., q t-m ]. Thus, the probabilistic model should be able to fulfill p(q t |q t-1 , ..., q 1 ) = p(q t |q t-1 , ..., q t-m ). In the proposed local regression modelling methodology, the spatiotemporal directions to the models to constrain the transformation of the character's postures in the configuration space are added. This approach predicts how the humans move in each region and constrains the reconstructed motion to remain in the natural-looking space. For that reason, it was considered to be an online learned model to generate the desired posture q t from various forms of kinematic constraints c t that are specified by the input trajectories. Finally, the motion reconstruction process is optimized in a MAP framework by estimating the posture q t that is satisfied by the input trajectories c t along with the previous reconstructed postured Q t,m such as: arg max

q t p(q t |c t , Q t,m ) ∝ arg max q t p(c t |q t )p(q t |Q t,m ) (1)
In this case, by applying the negative logarithm to the posteriori distribution function p(q t |c t , Q t,m ), the constrained MAP problem becomes an energy minimization problem, which is now represented as:

arg max q t -ln p(c t |q t ) E likelihood + -ln p(q t |Q t,m ) E prior (2)
where the likelihood term (E likelihood ) measures how well the reconstructed posture q t fits the user defined input trajectories c t , and the prior term (E prior ) measures the naturalness of the reconstructed posture. It should be noted that an optimal estimation of the reconstructed posture produced a natural motion that achieved the inputs that were specified by the user.

Motion Reconstruction

Using a reduced number of input trajectories to reconstruct the motion of the virtual character is quite challenging, since the control inputs cannot fully constrain the entire human motion to remain in the natural-looking space. Thus, in the proposed solution, a group of local regression models are used to solve this issue of motion reconstruction ambiguity. The methodology that used is presented in the following subsections.

Prior Motion Modelling

The prior motion modelling that was based on the local regression models is presented in this subsection. This model is responsible for adequately constraining the reconstructed posture of the character to remain in the natural-looking space. The novelty of the proposed model is that there is no need to find an appropriate structure for a global dynamic model, which would necessarily by high dimensional and non-linear. In the proposed methodology, a k-nearest neighbour (k -NN) searching algorithm is adopted to find the K motion examples that are contained in our motion capture database and are similar to the already reconstructed posture. The examples and their subsequent postures are used for our online learning process.

In order to estimate the current posture q t at the tth frame, a searching of the database of the motion data is employed in the first steps. This searching process finds the motion segments that are most similar to the recently reconstructed motion segment Q t,m = [q t-1 , ..., q t-m ]. Thus, the k nearest motion segments q t k for k = 1, ..., K are chosen as training data to learn a predictive model by means of a statistical learning method of current posture q t . Now, assume a linear relationship between an input angle vector x = [q t-1 , ..., q t-m ] and an output joint angle vector y = q t . In this case, one should note that the predictive function for each DOF in the output q t is learned separately. Then, by subtracting the means from the input and output training data, one assumes that the mean values of x and y are zeros. Therefore, the function of the proposed model is represented by using linear regression as follows:

y = a T x + β y (3)
where the input joint angle x is an m × D-dimensional vector. D represents the dimension of DOF for a virtual character and y is the joint angle value for the output motion. Vectors a and β y are regression coefficients that represent a homoscedastic noise variable, which is independent of vector x. Moreover, given the K motion examples (x k ; y k ) for k = 1, ..., K that are similar to the current reconstructed poses, and by minimizing the expected squared error E = min ∑ K k=1 y ka T x k 2 , the coefficient a is obtained by the least squares solution that follows:

a = (X T X) -1 X T y (4)
where the row of the matrix X stores the input joint angle vectors x k for k = 1, ..., K, and K output joint angle values are stacked in vector y.

The proposed methodology calculates the projections of the highest correlation between the input joint angle matrix X and the output vector y. These projections can be obtained by maximizing the squared relationship as follows:

correlation 2 (X u j , y) = (u T j X T y) (u T j X T X u j ) (5)
where u j denotes the one of the projection's directions. It should be noted that since each of the projections X u j is orthogonal to others and its length is unit, it is possible to get u T j X T X u j = 1. Hence, u j is one column of the matrix U that includes the eigenvectors of the covariance matrix C = (X T X) -1 X T yy T X. In the proposed model, X is projected onto U for only considering the projections. Thus, by minimizing y -XUγ 2 with respect to the reduced coefficient γ, one obtains:

a = Uγ = U(U T X T XU) -1 U T X T y (6)
Since each DOF is predicted separately in output q t , the model has only one projection direction u for each time. The weight for each data point x k can now be calculated by the Gaussian function, using its relative distance from the previously reconstructed postures Q t,m = [q t-1 , ..., q t-m ] as:

ω k = exp - 1 2 (x k -Q t,m ) T W (x k -Q t,m ) (7)
where W denotes the diagonal matrix that contains the weights for each DOF. It should be noted that in the proposed implementation used an identity matrix to represent W . The eigenvectors are extracted from the matrix as:

C ω = (DX) -1 X T Dyy T DX (8)
where D denotes a diagonal matrix that constrains ω k along its diagonal. Moreover, the weighted regression coefficients can be represented as:

a ω = U(U T X T DXU) -1 U T X T Dy (9)
In this case, assuming that there is a Gaussian distributed noise variable β y , its standard deviation σ can be estimated by y kβ x k for k = 1, ..., K. In our experiments, a predictive function for each DOF of the reconstructed posture is constructed. Therefore, to predict the dth DOF of the character's posture, the local regression model is described as:

q t,d = a T d,ω Q t,m + N(0, σ d ) (10

Likelihood Estimation

The likelihood term (E likelihood ) of the MAP framework measures how well the corresponding joint in the reconstructed character's postures fits the user-defined constraints. Therefore, the likelihood term is formulated as:

E likelihood = -lnp(c t |q t )
∝ f (q t ; s)c t 2 [START_REF] Mousas | Motion Style transfer in Correlated Motion Spaces[END_REF] where q t , c t , and s are vectors. q t denotes the joint angles of the reconstructed posture at frame t, s denotes the character's skeletal size, which is modelled according to the Acclaim Skeletal File format (ASF) as provided by [START_REF]Motion Capture Database[END_REF]. Finally, c t is the user-specified input trajectories that are retrieved from reference motion data. Finally, f denotes a forward kinematics function that calculates the global coordinates value of the current posture q t of the character.

Implementation and Results

The following subsections briefly present the implementation and the results obtained from the evaluation process of the proposed methodology.

Implementation

For the implementation of the proposed methodology a gradient-based optimization was used using the Levenberg-Marquardt method [START_REF] Lourakis | Levmar: Levenberg-Marquardt Nonlinear Least Squares Algorithms in C/C++[END_REF] for the objective function that is defined in Equation ( 2). This method uses the most similar motion examples that are already in the motion database to initialize the optimization. The computational efficiency of the proposed motion reconstruction process relies on the scope of the search in the motion database. Thus, the process to find the K nearest neighbour is accelerated using the neighbour graph approach as presented in [START_REF] Chai | Performance Animation from Low-Dimensional Control Signals[END_REF]. Finally, examples of posture reconstructed with the proposed methodology are shown in Figure 2.

Results

To evaluate the proposed methodology two different datasets were used. The first dataset contains 85,097 postures that are separated into five different actions that the character can perform: walking, running, jumping, punching, and swinging a golf club. The second dataset contains a total of 1.1 M poses that were downloaded from [START_REF]Motion Capture Database[END_REF]. All of the motion sequences were recorded by use of a Vicon motion capture system that has a framerate of 120 fps. In the proposed implementation, the motion data were downsampled to 60fps in order to achieve more natural-looking motion for visualization. The effectiveness of the proposed approach was verified on various behaviours and the reconstruction error was evaluated against the ground truth data.

Comparing to Other methods: The proposed methodology was evaluated against three popular approaches for reconstructing the character's motion. Specifically, the methodology was evaluated against the Gaussian Process Latent Variable Model [START_REF] Grochow | Style-Based Inverse Kinematics[END_REF], the local PCA model [START_REF] Chai | Performance Animation from Low-Dimensional Control Signals[END_REF] and the local PCR model [START_REF] Liu | Realtime Human Motion Control with a Small Number of Inertial Sensors[END_REF], while reconstructing different actions that the character can perform. The results of this evaluation process appear in Figure 3. Specifically, the mean error of five different actions for all of the aforementioned techniques is presented. In the evaluation, we also adopt six constraint points that were used in [START_REF] Liu | Realtime Human Motion Control with a Small Number of Inertial Sensors[END_REF]. The results show that our method achieved smaller mean errors than the other three techniques. In another aspect, while using only four control points (two wrists and two ankles), the three previous methods cannot reconstruct natural-looking human data. The results indicate that the proposed method is better than those of the two other local methods.

Using Different Number of Control Points: We tested a different number of control points for four methods. We chose from two to six positional control points. (1) Left wrist and right ankle; (2) left wrist and two ankles; (3) two wrists and two ankles; (4) root, two wrists and two ankles; (5) head, root, two wrists and two ankles. After testing with different motions, we concluded that the reconstruction errors usually decrease as the number of constraints increases. In addition, we also found that, in comparison to the six constraint points (head, center of torso, two wrists and two ankles), which was used in [START_REF] Liu | Realtime Human Motion Control with a Small Number of Inertial Sensors[END_REF] to obtain a natural-looking, reconstructed human motion, we can use as few constraint points as possible (4 constraint points: two wrists and two ankles) to achieve a comparable result with the motion capture data. Table 1 is the average reconstruction error comparison for various numbers of control points. Despite the use of fewer constraint points, our model is more powerful for accurate motion reconstruction than the three previous methods. Using Different Datasets: Table 2 presents the average reconstruction errors of five different actions from the three aforementioned techniques, while using the different training database. The reconstruction errors were calculated using 3D positional constraints from six control points. We found that, for the GPLVM method, the reconstruction error is large when using a large and heterogeneous database. When using a small database, the reconstruction error is also larger than that of local modeling approaches. For the other local modeling methods, the reconstruction error decreased when the size of training database increased. In addition, our proposed model can achieve a smaller reconstruction error than others. By testing on different databases, we also verified the proposed model's power.

Conclusion

In this paper, a new local regression model was presented for reconstructing natural full-body human motion based on as few user-specified constraints as possible. The proposed method, which uses a data-driven approach, utilizes several nearest motion examples to construct a group of online local regression models for online motion reconstruction. However, based on the same defined constraints and motion database, the proposed method has a better force of constraint than the previous local models and thus can reconstruct more realistic human motions. Therefore, our proposed model is suitable for the next generation of hardware devices to exploit the motion capture system for a common use.

On the other hand, the proposed method has three limitations. Firstly, like other data-driven approaches, the database is crucial for the quality of reconstructed motion. The system will not produce a desired motion if the training data does not contain any desired motion patterns. For example, if the walking motion pattern is not included in the database, our system cannot reconstruct desired walking data. Secondly, userspecified constraints are also crucial for the final results. In fact, if user-specified constraints are not natural or self-conflicting, the reconstruction result will not be a realistic human motion that satisfies the user's constraints. Finally, the motion data must be previously arranged for online search. Like most local modeling approaches, a specific data arrangement structure is applied for motion data to accelerate the searching process.

Fig. 1 .

 1 Fig. 1. An overview of the proposed system.

  ) where q t,d and σ d are scalars, q t,d represents the d -th DOF of the t -th frame posture, and σ d is the standard deviation of the dth predictive function. a T d,ω and Q t,m are vectors, where a T d,ω are the weighted regression coefficients for the d -th DOF, and Q t,m is the reconstructed motion segment of the previous m postures of the character. The complexity of such a model for reconstructing the character's posture is O(Km 2 D 2 ), where K, m, and D represents the number of training data, the previous m postures, and the dimension of DOF for the virtual character repsectivelly.

Fig. 2 .

 2 Fig. 2. Examples of postures synthesized with the proposed methodology.

Fig. 3 .

 3 Fig. 3. A comparison of the proposed methodology to three popular algorithms: GPLVM, local PCA, and local PCR. The percentage error while using six (up), and four joints (down) for reconstruction of the motion of the character.

Table 1 .

 1 Reconstruction errors based on different numbers of control points for different algorithms.

		2	3	4	5	6
	GPLVM	56.76 41.88 16.13 9.45 5.92
	LPCA	43.27 29.25 10.56 6.39 3.81
	LPCR	38.71 23.61 7.33 4.73 3.22
	Proposed Method 18.63 7.86 2.67 2.25 1.90

Table 2 .

 2 Average reconstruction errors for four methods on different databases.

		69888 poses 1.1 M poses
	GPLVM	5.86	26.57
	LPCA	3.92	3.02
	LPCR	3.38	2.75
	Proposed Method 2.01	1.56


K K Delibasis
email: kdelibasis@yahoo.com

S Georgakopoulos

V Plagianakos

I Maglogiannis

Calculation of Complex Zernike Moments with Geodesic Correction for Pose Recognition in Omni-directional Images

Keywords: Pattern recognition, computer vision, image descriptors, moment invariants, Omni-directional image/video, fish-eye camera, silhouette pose recognition

A number of Computer Vision and Artificial Intelligence applications are based on descriptors that are extracted from imaged objects. One widely used class of such descriptors are the invariant moments, with Zernike moments being reported as some of the most efficient descriptors. The calculation of image moments requires the definition of distance and angle of any pixel from the centroid pixel. While this is straightforward in images acquired by projective cameras, it is complicated and time consuming for omni-directional images obtained by fish-eye cameras. In this work, we provide an efficient way of calculating moment invariants in time domain from omni-directional images, using the calibration of the acquiring camera. The proposed implementation of the descriptors is assessed in the case of indoor video in terms of classification accuracy of the segmented human silhouettes. Numerical results are presented for different poses of human silhouettes and comparisons between the traditional and the proposed implementation of the Zernike moments are presented. The computational complexity for the proposed implementation is also provided.

Introduction

Pattern recognition in images is a very common task in artificial intelligence. Among other methods [START_REF] Mikolajczyk | Scale and affine invariant interest point detectors[END_REF], invariant moments have been used extensively for providing regional descriptors to be used in pattern recognition problems [START_REF] Hu | Visual pattern recognition by moment invariants[END_REF] - [START_REF] Schlemmer | Moment Invariants for the Analysis of 2D Flow Fields[END_REF]. The calculation of invariant moments of a region of an image requires a valid distance metric defined in the image domain (e.g. [START_REF] Teague | Image analysis via the general theory of moments[END_REF]). In omnidirectional images, the distance between two pixels cannot be defined in terms of their coordinates in the image frame, since the image is acquired not through a simple projection, but using a spherical element with 180 degrees field of view.

This need has been recently addressed in a few publications. In [START_REF] Cruz-Mota | Scale Invariant Feature Transform on the Sphere: Theory and Applications[END_REF] the implementation of the SIFT algorithm ( [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]) is presented for omni-directional images, using the convolution operators on a sphere in the Fourier domain, whereas it has also been studied for wide angle images [START_REF] Hansen | Scale invariant features on the sphere[END_REF]. In other approaches, the SIFT algorithm has been applied to the unwrapped omni-directional images [START_REF] Tamimi | Localization of mobile robots with omnidirectional vision using particle filter and iterative sift[END_REF]. Pixel distance was redefined in conic sensor images [11.] The contribution of this work focuses on an efficient way of computing Zernike moment invariant (or any other invariant moments) for calibrated omni-directional images in the time domain. We propose the measurement of geodesic distance and angles between image pixels, based on the calibrated model of the acquiring fish-eye camera. The geodesically corrected implementation of Zernike moments presents reduced variability when applied to known geometric shapes and it achieves increased accuracy when applied to pose classification of segmented human silhouettes. The proposed implementation is computationally quite efficient, since it allows the processing of a high number of frames per second.

2

Proposed Methodology

Zernike moment invariant

Zernike moments of order m, n are defined by a set of radial polynomials R nm (r), which are orthogonal inside the unit circle:

( ) ( ) ( ) ( )/2 2 0 ! 1 ! ! ! 2 2 n m s n s nm s n s R r r n m n m s s s - - = - = -  +   +  - +         ∑ , (1)
where n positive integer and m integer such that n-|m| is even and |m|<n. Orthogonality is preserved since ( ) ( ) ( )

1 0 1 2 1 nm km nm R r R r dr n δ = + ∫ .
Zernike polynomials are defined as ( ) ( )

, jm nm nm V r R r e θ θ = (2)
The Zernike moments of a bivariate function f(x,y) are defined as:

( ) ( )

1 2 0 0 1 , , nm nm n Z f r V r rdrd π θ θ θ π + = ∫ ∫ (3)
The above Eq. is easily adapted to be used in discrete images.

Omni-directional image formation and camera calibration

The formation of omni-directional image using a spherical camera, as presented in detail in [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF], is shown in Fig. 1, using only two dimensions to facilitate understanding. In Fig. 2, neighbouring pixels are shown in different positions on the image sen-sor. It is clear that the distance of two pixels is different when measured on the sensor and on the position of the image points on the spherical optical element. Therefore, in order to produce accurate results, image processing algorithms that use pixel distances have to be re-implemented for omni-directional images. In this work, we utilize the calibration of the specific fish-eye camera, proposed in [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF], according to which, for each pixel (i, j) of the image, the corresponding vector (θ, φ) on the spherical element is precalculated

( ) ( ) , , M j i θ ϕ = (4)

2.3

Calculation of Zernike moment invariants for calibrated omni-directional images -Geodesic correction

The calculation of Zernike moments requires the distance and orientation with respect to the centre of mass for each pixel of the segmented object / pattern to be classified. Let us assume that we need to calculate Zernike moments in the case of the segmented human silhouette of Fig. 2(b). If P 0 is the centroid pixel, the distance and angle to any other pixel P 1 needs to be calculated. Let us use the calibration of the camera to obtain the spherical coordinates ( )

0 0 , θ ϕ and ( ) 1 1
, θ ϕ on the unit sphere, of points P 0 and P 1 , respectively. Now, the distance and relative angle of P 1 with respect to P 0 can be measured on the unit sphere.

Parallel circle γ t1 t2 Fig. 3. The mapping of pixels P 0 and P 1 on the unit sphere and the calculation of their geodesic distance and angle. The great circle and the parallel circles through P 0 and P 1 are also shown.

It is well known that the geodesic curve of a sphere is a great circle. Thus, the distance between any two points on a sphere is the length of the arc that is defined by the two points and belongs to a circle that passes through the two points and has the same centre with the sphere. We will call this circle "great circle". Let v 0 and v 1 be the position vectors pointing to P 0 and P 1 . The distance r 01 of points P 0 and P 1 on the unit sphere is easily calculated, (assuming that cos -1 returns the result in radians):

( ) ( ) (
)

0 1 0 0 0 0 0 1 1 1 1 1 0 0 1 1 0 0 1 1 0 1
cos cos ,sin cos ,sin cos cos ,sin cos ,sin cos cos cos cos sin cos sin cos sin sin

θ ϕ θ ϕ ϕ θ ϕ θ ϕ ϕ θ ϕ θ ϕ θ ϕ θ ϕ ϕ ϕ ⋅ = ⋅ = + + v v (5)
( )

1 01 0 1 cos r - = ⋅ v v . ( 6
)
Zernike moments, like any other definition of moment invariants, require the distance between any two pixels as well as their angle γ. If we assume the parallel circle passing through P 0 and the great circle passing through P 0 and P 1 , then angle γ is calculated as the angle between the tangent vectors t 0 , t 1 of the two circles at P 0 . The great circle through P 0 , P 1 is calculated as following:

0 0 1 1 , with 1
λ λ = + = v v v v . ( 7
)
After some algebraic operations we obtain:

( )

2 2 0 0 1 0 1 1 2 cos 1 0 λ λ λ λ + ⋅ + -= v v . ( 8
)
After solving the above equation and requiring 0 1 , λ λ to be real numbers, we obtain:

( ) ( ) ( ) ( ) ( ) 2 2 0 0 1 2 2 2 1 0 1 0 0 1 0 cos 1 0, cos 1 cos 1 λ λ λ λ ⋅ -≥ = - ⋅ ± - ⋅ + v v v v v v (9)
Since execution speed is essential and taking into consideration that the circle is a very smooth curve (with constant curvature), the tangent vector is calculated by finite differences, as following. It is obvious that for 0 1 0 ( )

2 1 1
, , 0

y x v v = - t
. The angle γ is now easily obtained as

1 1 2 1 2 cos γ -  ⋅ =       t t t t . ( 10
)
The sign of angle γ will be set equal to the sign of (θ 1θ 0 ). Thus, the distance and angle (r, θ) in Eq. ( 3) are replaced by r 01 defined in [START_REF] Schlemmer | Moment Invariants for the Analysis of 2D Flow Fields[END_REF] and γ defined in Eq. [START_REF] Tamimi | Localization of mobile robots with omnidirectional vision using particle filter and iterative sift[END_REF]. This correction will be called "geodesic" in the rest of the paper.

Experimental Results

Experiments with a geometric shape

A rectangular shape of known constant RGB values was captured in a video sequence of approximately 500 frames with different distances and positions from the fish-eye camera.

Classification of the human silhouette pose

In this experiment, we acquired 2 different videos of a human with two different classes of poses: walking/standing and sitting. In each video the subjects sits in 2 or 3 different places. The human silhouette is segmented using an unsupervised segmentation algorithm presented in [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF] and the Zernike moments are calculated twice: without and with the geodesic correction. The simple k-Nearest Neighbour (k-NN) classifier with k=3 was employed to classify each frame into the two classes, using the Euclidean distance metric between the Zernike features. The classical Zernike moments have been used to classify the human action of each frame, utilizing moments of the first n orders, where n = 3, 4, 5. In Table 1 the results for each video are presented. The classification results for both videos are quite similar for moments of the same order. More specifically, in video 1 the classification accuracy increases for higher order Zernike moments. On the other hand, in the more complex video 2, the best results are obtained using N = 4. It is evident that higher order Zernike moments, results in an increase in computational complexity, as well as the CPU time needed to process each frame. The average number of frames per second (FPS) processed by applying the Zernike moments and the k-NN classifier is also reported. To improve the classification results, the subset of the Zernike moments with radial symmetry were used. Table 2 exhibits the classification results for n up to 30, while m=0 for the classical and the geodesically corrected Zernike moments, respectively. The rate of frames processed is also provided. It can be observed that the classification results are improved by the introduction of geodesic correction, with minimal increase to the measured execution time.

Discussion and Conclusions

As shown in the previous section the geodesic correction enhances the descriptive properties of Zernike moments, when applied to omni-directional images. The Zernike descriptors appear to be more stable (less variability for the same pattern). The accuracy of a classifier in terms of classifying two human postures was also increased when using Zernike moments with the proposed geodesic correction. Finally, the execution time was only marginally increased. The proposed methodology can significantly improve the accuracy of more complex activity recognition algorithms usually found in ambient assisted living environments.

Fig. 1 .

 1 Fig. 1. Model of image formation for spherical omni-directional camera.

Fig. 2 .

 2 Fig. 2. A typical omni-directional image (a) and the segmented human silhouette (b). The strong deformation imposed by the fish-eye spherical lens is evident.

  The tangent vector of the parallel circle at P 0 is trivially calculated:

Fig. 4 (Fig. 4 .

 44 Fig. 4. Zernike moments before and after geodesic correction for the rectangular geometric shape (see text for details).

Fig. 5 Fig. 5 .

 55 Fig.5graphically exhibits the classification results, using n≤20, m=0, without and with geodesic correction. Class 0 corresponds to sitting silhouette and class 1 corresponds to standing/walking silhouette. The algorithmic results are shown in blue and the ground truth in red. It can be seen that the geodesic correction produces less incorrectly classified poses in the video frames, thus improves classification accuracy.

Table 1 .

 1 The resulting classification accuracy of the segmented silhouette pose for different orders of the traditional Zernike implementation. The frame processing rate is also provided.

	Zernike	Classification Accuracy (%)
	order	Video 1	Video 2	FPS
	n = 3, m=-1,+1	84.22	85.77	16 fps
	n = 4, m=-2,0,+2	92.11	92.08	15 fps
	n = 5, m=-3,-1,+1,+3	94.88	91.82	12 fps

Table 2 .

 2 The resulting classification accuracy of the segmented silhouette pose for different orders of the traditional radial Zernike implementation. The rate of frame processing is also provided.

	Zernike	Geodesic	Classification Accuracy (%) Execution
	order	correction	Video 1	Video 2	speed
	n = 2,4,6,8,10, m=0	NO YES	93.39 94.09	94.53 96.13	17 fps 17 fps
	n = 2,4,6,8,…,20, m=0	NO YES	91.68 94.24	94.14 95.94	12 fps 11 fps
	n = 2,4,6,8,…,30, m=0	NO YES	92.11 92.24	94.01 95.75	8 fps 9 fps

Acknowledgment

The authors would like to thank the European Union (European Social Fund ESF) and Greek national funds for financially supporting this work through the Operational Program "Education and Lifelong Learning" of the National Strategic Reference Framework (NSRF) -Research Funding Program: \Thalis \ Interdisciplinary Research in Affective Computing for Biological Activity Recognition in Assistive Environments


K Kottari

K K Delibasis
email: kdelibasis@yahoo.com

V Plagianakos

I Maglogiannis

Fish-eye Camera Video Processing and Trajectory Estimation Using 3D Human Models

Keywords: fish-eye camera video processing, three-dimensional human modelling, posture recognition, minimization, generalized cylinders, and elliptical intersections

Video processing and analysis applications are part of Artificial Intelligence. Frequently, silhouettes in video frames lack depth information, especially in case of a single camera. In this work, we utilize a three-dimensional human body model, combined with a calibrated fish-eye camera, to obtain three-dimensional (3D) clues. More specifically, a generic 3D human model in various poses is derived from a novel mathematical formalization of a wellknown class of geometric primitives, namely the generalized cylinders, which exhibit advantages over the existing parametric definitions. The use of the fisheye camera allows the generation of rendered silhouettes, using these 3D models. Moreover, we present a very efficient algorithm for matching that 3D model with a real human figure in order to recognize the posture of a monitored person. Firstly, the silhouette is segmented in each frame and the calculation of the real human position is calculated. Subsequently, an optimization process adjusts the parameters of the 3D human model in an attempt to match the pose (position and orientation relatively to the camera) of real human. The experimental results are promising, since the pose, the trajectory and the orientation of the human can be accurately estimated.

Introduction

The field of automated human activity recognition utilizing fixed cameras of indoor environments has gained significant interest during the last years. It finds a variety of applications in diverse areas, such as assistive environments, smart homes, support for the elderly or the chronic ill, surveillance and security, traffic control, industrial processes, etc. This work focuses on fish-eye camera video processing for pose estimation of sitting or standing/walking humans. Therefore, human silhouette segmentation of the video sequence is a prerequisite. Recognizing a human pattern is often possible via volume intersection [START_REF] Bottino | A silhouette-based technique for the reconstruction of human movement[END_REF] or a voxel-based approach [START_REF] German | Shape-from silhouette of articulated objects and its use for human body kinematics estimation and motion capture[END_REF][START_REF] Mikic | Human body model acquisition and tracking using voxel data[END_REF]. Stereometry based models have also been constructed through calibrated camera pairs. Using triangulation, the depths of the points are calculated. This approach has been taken into account by Plänkers and Fua [START_REF] Plänkers | Tracking and modeling people in video sequences[END_REF] and Haritaoglu et al. in [START_REF] Haritaoglu | W4s: A real time system detecting and tracking people in 2 1/2D In: Proceedingsof the European Conference on Computer Vision[END_REF]. Stereo vision is also used by Jojic et al. [START_REF] Jojic | S: 3-Dreconstruction of multipart, selfoccluding objects[END_REF], with the optional aid of projected light patterns. The proposed algorithm is based on a parametric three-dimensional (3D) human model with limited degrees of freedom so that it allows efficient manipulation for standing/walking and sitting postures. Our aim is to estimate human position, trajectory and standing/sitting state, which would be useful towards human behavior recognition. The first step in applications dealing with human activity recognition from video is the foreground segmentation. Most video segmentation algorithms are based on background subtraction. The background has to be modelled, since it may change due to a number of reasons, including: motion of background objects, changes in light conditions, or video compression artifacts. In this work, we employ the forward and inverse camera model that was proposed in [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF]. We follow a "top-down" approach that matches the model rendered through the calibrated fish-eye camera, with the segmented frame of the video. Then, an optimization algorithm is utilised to find the model parameters and determine human orientation and pose. The rest of the paper is structured as follows: Section 2 discusses the technical details of the proposed algorithms, Section 3 presents some initial results, while Section 4 concludes the paper.

Proposed Methodology

Generalized Cylinders

For the generation of the human model, we utilized the concept of generalized cylinders (GC), as proposed in [START_REF] Delibasis | A novel tool for segmenting 3D medical images based on generalized cylinders and active surfaces[END_REF]. More specifically, let C 1 be a piecewise smooth curve defined in a Cartesian coordinate system OXYZ, as:

)) ( ), ( ), ( ( ) ( 1 t z t y t x t r = , R b] [a, t ⊂ ∈ (1)
and C 2 be a planar curve defined in an orthogonal local Cartesian system OXY. Letus now consider the surface S that is generated by moving the curve C 2 along C 1 , so that its plane is perpendicular to the tangent vector of C 1 , and the origin of OXY belongs to

) ( 2 ) ( 1
)) ( sin( )) ( ( ) ( ' ) ( ' ) ( ) ( 2
)) ( cos( )) ( ( ) ( ' ) ( ) ( ) , ( t P t P t u t u r t z t x t s t P t u t u r t y t s t x u t x φ φ φ φ + + + + + + = (2) ) ( ) ( )) ( sin( )) ( ( ) ( ) ( ) ( ) ( )) ( cos( )) ( ( ) ( ) ( ) ( ) , ( 2 1 '
+ + + + + - = (3) ) ( ) ( )) ( sin( )) ( ( )) ( ( )) ( )( ( ) ( ) , ( 2 1 2 ' 2 ' t P t P t u t u r t y t x t s t z u t z φ φ + + + - = (4) where, ] 2 , 0 [ ] , [ ) , ( π × ∈ b a u t , 2 ' 2 ' 2 ' 1 )) ( ( )) ( ( )) ( ( ) ( t z t y t x t P + + = and 2 ' 2 ' 2 )) ( )) ( ) ( t y t x t P + =
.The complete proof is given in [START_REF] Delibasis | A novel tool for segmenting 3D medical images based on generalized cylinders and active surfaces[END_REF].

3D Model Construction

In this work, a free triangulated model of a standing human (Fig. 2) is utilised, defined by the Cartesian coordinates of approximately 27,000 vertices [9]. Since we are interested in simulating the rendering of the human model through the fish-eye camera in real time, we discard the triangle information of the model and we treat it as a cloud of points. Therefore, we compute the intersections of the model in a number of horizontal planes, in distance of two centimetres along the Z axis (feet -head direction) as shown in Fig. 3 (a). The same process is repeated, along hands and legs -see Fig. Each intersection is estimated for approximating an ellipse with its semi-axes semi a , semi b parallel to X and Y axis of coordinate system, as shown in Fig. 4. the direction of the leading axis. In Fig. 5 we see the result of elliptical patterning of model intersections, through the insertion of ellipses to the GC Eq. ( 5). Note that a torsional inconstancy at the knees section is being observed. This phenomenon has been explained in [8, section 4] and does not affect the optimization process that matches the 3D model to the segmented human silhouette. The estimation of human posture (sitting or standing) is based on the construction of the human model. Having the 3D standing human model constructed as described above, its transformation to match the sitting position can be easily performed by changing the model parameter (angles) at waist and knees. The result of that transformation is shown in Fig. 5 right, while Fig. 6 depicts both models as they are utilised by the video-processing algorithm.

Video Processing Algorithm

In this work, we analyze videos captured by a fish-eye camera, fixed on the ceiling of a living environment. The recorded videos have been foreground segmented, while empty frames are being discarded. Then, the mask shown in Fig. 7(a) is applied to suppress noisy segmented pixels outside the field of view. The initial estimation of the real human position in the room coordinates is accomplished by the recently proposed algorithm in [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF] based on the segmented frame pixels. For this purpose, we employ the calibration of the acquiring fish-eye camera that provides the spherical coordinates (θ, φ) for each pixel of the current frame, as well as the frame pixel that corresponds to any real world point (x,y,z), according to [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF]:

( ) ( ) , , , j i M x y z = (6) ( ) ( ) 1 , , M i j θ ϕ = (7)
Let PHI(i,j) hold the value of φ for pixel (i,j), as obtained by [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF] and shown in Fig. 7(b). Thus, for any pose of the 3D parametric model, we can obtain the binary image-I M of the human model, rendered by the fish-eye model using [START_REF] Jojic | S: 3-Dreconstruction of multipart, selfoccluding objects[END_REF]. Fig. 8illustrates image I M combined for various standing and sitting models, as rendered by the calibrated fish-eye camera. Let I S be segmented binary frame, after using the binary mask in Fig. 7(a). The initial estimation of the person's position is obtained by locating the non-zero pixel (i 0 ,j 0 ) of I S that holds the minimum value of angle φ. The objective function, which quantifies the match between the model and the segmented human silhouette as a function of its real world position (x,y) and its orientation θ 0 , is defined as the intersection of the segmented silhouette I S and the rendered human modelI M :

( ) 0 , , M S image domain f x y I I θ = ∩ ∑ (8)
where I M (defined above) is shown in red, I S shown in green and their intersection I M ∩ I S is shown in yellow. Fig. 9 presents graphically the calculation of the objective function for one instance of each class (sitting and standing). Subsequently, the simplex [START_REF] Nelder | A simplex method for function minimization[END_REF] multidimensional unconstrained maximisation algorithm is utilised to optimise the objective function. The initial position (x,y) of the human is approximately computed from the first frame by finding the segmented human silhouette pixel (i,j) with maximum PHI, (as described in [START_REF] Delibasis | Fisheye Camera Modeling for Human Segmentation Refinement in Indoor Videos[END_REF]) and used to initialize the simplex method. Thus, the maximisation algorithm computes the human model parameters that best match the segmented figure and returns the coordinates x and y, as well as the orientation θ 0 of model.

Experimental Results

Classification results of two different videos are presented in Table 1, considering the sitting as "positive" and the standing as "negative" status. The ground truth was established by manually annotating the video with the human pose, as "sitting" or "standing". The proposed model-based algorithm is able to estimate the trajectory of the human silhouette, as well as its orientation.

Discussion and Conclusions

In this paper, an algorithm for estimating the trajectory of a human silhouette in indoor videos acquired by an omni-directional camera has been presented. The algorithm is based on a parametric 3D human model and it recovers the model parameters (translation and orientation) by optimizing a suitable defined objective function. Initial results show that the proposed algorithm can estimate the trajectory and orientation and discriminate between two different postures: sitting and standing. The proposed methodology may improve the accuracy of more complex activity recognition algorithms usually found in ambient assisted living environments. This methodology A B

C D E can be adopted for detecting higher level activity events and understand behavioural patterns.

C 1 .

 1 If we express the planar curve C 2 in polar coordinates 2 2 ( ), [0, 2 ] r r u u π = ∈ and introduce a scale factor s(t)and a rotation factor ( ) t φ along the tangent vector of C1 as function of position along C 1 , then the surface equation of the GC becomes:

Fig. 1 .

 1 Fig. 1.Two examples of surfaces derived from equation (2) -(4) from[START_REF] Delibasis | A novel tool for segmenting 3D medical images based on generalized cylinders and active surfaces[END_REF]

Fig. 2 .

 2 Fig. 2. Triangulatedmodel of a standing human [9].

  3(b).

Fig. 3 .

 3 Fig. 3. Elliptical intersections of torso and leg.

Fig. 4 .

 4 Fig. 4. Elliptical intersection of the human torsowith theXY plane.

Fig. 5 .

 5 Fig. 5.3D standing (left) and sitting human (right).

Fig. 6 .

 6 Fig. 6. 3D Human silhouettes

Fig. 7 .

 7 Fig. 7. (a) Binary mask used to exclude out of filed-of-view pixels in video frames (b) Visualization of the PHI angle for each frame pixel as resulted from the camera calibration [7].

Fig. 8 .Fig. 9 .

 89 Fig. 8. Rendered standing and sitting human model in various angles and positions in the room.

  Fig. 10 shows the estimated positions and orientation of the human silhouette for video 1, as recovered by the optimization described above. The person moves from left to right at the bottom of the frame (A to B), then vice versa at the top of the frame and finally sits down at the point designated by E, where it rotates. Between points B and C the segmentation fails temporarily, however, the model-based tracking algorithm successfully recovers the silhouette's new position. The experimental results indicate that the simplex optimization method is robust and efficient. It needs approximately 30 iterations for each frame in order to converge. Each objective function evaluation requires approximately 30msec on an Intel i5 laptop with 4 GB Ram using the Matlab environment. Running time approaches the 900 milliseconds per frame.

Fig. 10 .

 10 Fig. 10. Graphical representation of the results for model based trajectory and orientation estimation for video 1.

Table 1 .

 1 Pose classification results

	TP	TN	FN	FP	Accuracy	Sensitivity	Specificity
	Video 1 110/278 112/278	23/278	33/278	0.7986	0.8271	0.7724
	Video 2 98/164	54/164	5/164	7/164	0.9268	0.9515	0.8852

Acknowledgment

The authors would like to thank the European Union (European Social Fund ESF) and Greek national funds for financially supporting this work through the Operational Program "Education and Lifelong Learning" of the National Strategic Reference Framework (NSRF) -Research Funding Program: \Thalis \ Interdisciplinary Research in Affective Computing for Biological Activity Recognition in Assistive Environments.


Piotr Sobolewski
email: piotr.sobolewski@pwr.edu.pl

Micha L Woźniak
email: michal.wozniak@pwr.edu.pl

Identifying features with concept drift in multidimensional data using statistical tests

Keywords: Concept drift, detection, statistical test

Concept drift is a common problem in the data streams, which makes the classifiers no longer valid. In the multidimensional data, this problem becomes difficult to tackle. This paper examines the possibilities of identifying the specific features, in which concept drift occurs. This allows to limit the scope of the necessary update in the classification system. As a tool, we select a popular Kolmogorov-Smirnov test statistic.

Introduction

Due to the evolution of the internet and the expansion of the decision making technology, the systems designed for classifying the data streams [START_REF] Babcock | Models and issues in data stream systems[END_REF] recently became a popular area of research. In the field of machine learning, data streams are defined as sources of continuous data generation, examples of which can be found in real life e.g., shopping trends, stock market, weather control, surveillance systems or health care. Classification task in these areas is often hindered by various factors which cause undesirable changes in the data classification rules. Such phenomenon is called concept drift [START_REF] Schlimmer | Incremental learning from noisy data[END_REF] and it is a major problem in the classification systems.

There are various methods described in the machine learning literature for defending against concept drift, mostly deploying one of the two popular strategies [START_REF] Greiner | Learning cost-sensitive active classifiers[END_REF]:

-Adapting a learner at the regular intervals without considering whether the changes have really occurred or not, -First detecting the concept changes and then adapting a learner to them.

The idea presented in this article has a potential of improving the classifier adaptation process as well as enhancing the efficiency of the concept drift detection algorithms.

Problem description

We assume that in the multidimensional data, concept drift may influence only some specific features, leaving all other features in the same conceptual distribution model. Identifying these features may improve the adaptation of the classification systems, as well as provide useful information for the sophisticated concept drift detection algorithms, such as LDCnet [START_REF] Sobolewski | Ldcnet: minimizing the cost of supervision for various types of concept drift[END_REF].

Our previous experiments [START_REF] Sobolewski | Sequential Tests of Statistical Hypotheses[END_REF][10] have shown, that the popular test statistics, such as the Kolmogorov-Smirnov test [START_REF] Smirnov | Table for estimating the goodness of fit of empirical distributions[END_REF], Wilcoxon rank sum [START_REF] Wilcoxon | Individual Comparisons by Ranking Methods[END_REF] or Wald-Wolfowitz test [START_REF] Wolfowitz | On Wald's Proof of the Consistency of the Maximum Likelihood Estimate[END_REF] are capable of detecting concept drift with a similar efficiency as advanced methods, designed specifically for this purpose, such as the CNF test [START_REF] Dries | Adaptive concept drift detection[END_REF]. The most efficient in our experiments was the Kolomogorov-Smirnov test statistic, therefore we have selected it for further analysis in this article.

Kolomogorov-Smirnov test is a non-parametric statistic, as it makes no assumption about the distribution of data and therefore can be deployed on any data.

For the two-sample test, a Kolmogorov-Smirnov statistic is computed as

D n,m = sup x |F 1,n (x) -F 2,m (x)| (1)
where F 1,n and F 2,m are the empirical distribution functions of samples computed as:

F n (t) = 1 n n i=1 1{x i ≤ t}, (2)
where (x 1 , ..., x n ) are independent and identically distributed (i.i.d.) random variables laying in the real numbers domain with a common cumulative distribution function. The statistic is used to perform a KS-test to reject the null hypothesis at level α by computing:

nm n m D n,m > K α , (3)
where K α calculated from:

P r(K ≤ K α ) = 1 -α, (4)
and K is a Kolmogorov distribution computed as:

K = sup t∈[0,1] |B(t)|, (5)
B(t) being the Brownian bridge [START_REF] Revuz | Continuous Martingales and Brownian Motion (Grundlehren der mathematischen Wissenschaften)[END_REF].

In short, the Kolmogorov-Smirnov test compares the distributions of two samples by measuring a distance between the empirical distribution functions, taking into account both their location and shape.

In this paper we evaluate the possibilities of applying the Kolmogorov-Smirnov test statistic as a tool for identifying the fueatures, which are influenced by concept drift. For this purpose, the tool needs to accurately classify the true positives (sensitivity) as well as the true negatives (specificity).

Data

Due to limited availability of the real data with concept drift, the data used in experiments is taken from the UCI Repository of datasets [START_REF] Newman | UCI machine learning repository[END_REF] and concept drift is simulated by swapping the features with each other.

In mathematical notation, if a reference dataset DS is characterized by n features f then the concept drift is applied by swapping any two features i and j with each other. Data with swapped features forms a new dataset DS i,j and the role of the algorithm is to identify which features are influenced by concept drift (i.e. find the i and j).

Example swap of features 1 and 2:

DS = [f 1 , f 2 , ..., f n ] DS 1,2 = [f 2 , f 1 , ..., f n ] , (6)
The swaps are made for each possible pair of features, resulting in n 2 combinations, where n is the dimensionality of the dataset. Each of the dataset is described in general by the number of samples and number of features in Tab. 1. This method of simulating concept drift is relatively common in the machine learning literature [START_REF] Zliobaite | Determining the training window for small sample size classification with concept drift[END_REF].

Experiments

In the experiments, we use the original dataset D as the reference data and the drifted datasets D i,j (i and j indicate the features which are swapped), with the samples randomly drawn from the datasets D i,j and groupped into data windows DW of various sizes s.

The Kolomogorov-Smirnov statistic is evaluated on every feature f in the data window to reject the null hypothesis that the values arise from the same population as the values of features in the reference dataset D with confidence level of 5%. It means, that if the test returns the p-value lower than 0.05, then the analyzed feature is considered to be influenced by concept drift and the detection signal is noted and added to the scores.

In order to evaluate the specificity, i.e. the ability to identify the true negatives, the data windows which do not include any feature swap are evaluated and if in this test the statistic returns the p-value lower than 0.05, then the algorithm makes a mistake, as it results in a false positive concept drift detection.

A short description of the experimental process is described in the pseudocode in Fig. 1.

Algorithm 1 Pseudo-code of a single loop in experimental series

Notations:

DS -original dataset with n features, DSi,j -dataset with swapped features i and j, DW s i,j -data window of size s with features i and j swapped, Single loop of experiment seriers:

DW s = draw s random data samples from DS For i = 1 to f For j = 1 to f DW s i,j = swap features i and j in DW s In the presented way, the sensitivity and specificity of the Kolmogorov-Smirnov test statistic are evaluated for every possible feature swap and for various sizes of the data windows.

Results

All presented values are averaged from the series of 1000 trials.

Tables 2 and3 show the percentage of correctly detected concept drifts in certain features for the breast dataset (size of data window 20 and 50, respectively), where columns are the base features and rows are the features which swap them. The diagonals are the percentage of detected false positives, the lower the value the higher the specificity of the algorithm. Tables 4 and5 show how the window size influences the performance of algorithm for the breast and credit -australian datasets. tables store the results obtained by swapping the first feature with other features. The results obtained for the breast dataset are also presented on the Diagram 1 for a more clear view of the efficiency trend in the domain of window size. Finally, Tab. 6 shows the overall performance of the Kolmogorov-Smirnov statistic in identifying the features affected by concept drift, divided into the sensitivity and specificity scores for each of the datasets and for various window sizes. Specificity is presented only for window size 20, as the results were not significantly different for other window sizes. 0.00 1.00 0.89 0.91 1.00 0.99 0.46 0.02 0.06 0.26 0.06 1.00 0.96 0.63 10 0.00 1.00 0.98 0.98 1.00 1.00 0.62 0.04 0.13 0.32 0.09 1.00 1.00 0.78 20 0.00 1.00 1.00 1.00 1.00 1.00 1.00 0.09 0.31 0.90 0.24 1.00 1.00 1.00 50 0.00 1.00 1.00 1.00 1.00 1.00 1.00 0.31 0.83 1.00 0.70 1.00 1.00 1.00 100 0.00 1.00 1.00 1.00 1.00 1.00 1.00 0.63 0.99 1.00 0.97 1.00 1.00 1.00

Discussion

In this paper we have proposed an unsupervised tool for enhancing the methods coping with concept drift. We have evaluated the efficiency of the Kolmogorov-Smirnov test statistic in detecting the features affected by concept drift in the multidimensional data.

The most apparent conclusion is that the performance of algorithm depends on the data window size. Fig. 1 clearly shows this relation.

Regardless of the window size, algorithm achieves a very high specificity score, proving that the tool performs very well with true negatives, i.e. when there is no drift. It means, that the tool can be used for detecting features with concept drift without the need to worry about the false positive detections.

On the other hand, sensitivity i.e. the true positive detection rate, leaves a field for improvement. With increasing window size, sensitivity of the tool also increases, what suggests that the tool is more feasible for problems, which do not require a very limited window size.

Overall, the performance of the proposed tool is on a decent level, as e.g. in the optdigits dataset scenario, which has 2 64 possible feature swap combinations, algorithm correctly identifies on average 88% of them with only 8% variance. Pairing it with the fact that the method does not require any supervision, the Kolmogorov-Smirnov test statistic can be considered an efficient tool for detecting the features with concept drift in multidimensional data. This functionality may be used for supporting the adaptation of classifiers as well as improving algorithms designed for detecting concept drift, such as LDCnet [START_REF] Sobolewski | Ldcnet: minimizing the cost of supervision for various types of concept drift[END_REF].

Further research aims on expanding the functionality of the mentioned LD-Cnet algorithm using the presented technique to battle concept drift in the multidimensional data.

  For k = 1 to f Evaluate KS statistic on feature k of DW s i,j and feature k of D IF p-value < 0.05 Note concept drift for feature k END IF END FOR END FOR END FOR

Fig. 1 .

 1 Fig. 1. Concept drift detection ratio for various window sizes.

Table 1 .

 1 Datasets

	Dataset	# of features # of samples
	breast	9	683
	credit-australian	14	690
	haberman	3	306
	heart-c	13	297
	heart-statlog	13	270
	ionosphere	34	351
	kr-vs-kp	36	3196
	letter-recognition	16	20000
	mfeat-mor	6	2000
	nursery	8	12960
	optdigits	64	3823
	page-blocks	10	5473
	pendigits	16	7494
	pima-indians-diabetes	8	768
	segmentation	19	210
	tic-tac-toe	9	958
	vehicle	18	846
	vote	16	232
	waveform	21	5000
	yeast	8	1484

Table 2 .

 2 Concept drift detection ratio in breast dataset, window size = 20

	win size 20				Base feature
	Swap feature	1	2	3	4	5	6	7	8	9
	1	0.02	0.7	0.67	0.86	0.82	0.89	0.27	0.91	1
	2	0.65	0.02	0	0.02	0.93	0.01	0.5	0	0.47
	3	0.59	0.01	0.01	0.03	0.93	0.03	0.43	0.07	0.63
	4	0.84	0.02	0.02	0	0.95	0.01	0.71	0.03	0.34
	5	0.68	1	0.99	0.99	0	1	0.03	1	1
	6	0.73	0.02	0.08	0.02	1	0	0.8	0.04	0.33
	7	0.39	0.56	0.62	0.63	0.17	0.78	0.03	0.85	1
	8	0.89	0.03	0.03	0.01	1	0.02	0.83	0	0.16
	9	1	0.6	0.58	0.27	1	0.42	0.99	0.13	0

Table 3 .

 3 Concept drift detection ratio in breast dataset, window size = 20

	win size 20				Base feature
	Swap feature	1	2	3	4	5	6	7	8	9
	1	0.02	1	0.98	1	1	1	0.79	1	1
	2	1	0	0	0.02	1	0.07	1	0.03	0.92
	3	0.99	0.01	0	0.1	1	0.08	0.95	0.12	0.99
	4	1	0.03	0.05	0.01	1	0.05	0.99	0.03	0.83
	5	1	1	1	1	0	1	0.48	1	1
	6	1	0.1	0.11	0.14	1	0	1	0.11	0.89
	7	0.88	0.98	0.96	1	0.57	1	0	1	1
	8	1	0.07	0.09	0.01	1	0.03	1	0.01	0.59
	9	1	0.99	1	0.89	1	0.97	1	0.58	0

Table 4 .

 4 Window size influence, breast dataset.

	breast			Feature 1 swapped with feature..
	win size	1	2	3	4	5	6	7	8	9
	5	0.03	0.30	0.24	0.37	0.21	0.34	0.18	0.42	0.86
	10	0.02	0.41	0.37	0.55	0.43	0.47	0.20	0.60	0.97
	20	0.01	0.75	0.62	0.85	0.72	0.79	0.42	0.89	1
	50	0.01	0.99	0.99	1.00	0.99	1.00	0.85	1.00	1
	100	0.00	1.00	1.00	1.00	1.00	1.00	0.99	1.00	1

Table 5 .

 5 Window size influence, credit -australian dataset.

	credit-aus					Feature 1 swapped with feature..
	win size	1	2	3	4	5	6	7	8	9	10 11 12 13 14
	5

Table 6 .

 6 Overall sensitivity and specificity scores.

	Sensitivity	Specificity