Pablo Azagra

Yoan Mollard

Florian Golemo

Ana C Murillo

Manuel Lopes

Javier Civera

A Multimodal Dataset for Interactive and Incremental Learning of Object Models

This work presents an incremental object learning framework oriented to human-robot assistance and interaction. To learn new object models from interactions with a human user, the robot needs to be able to perform multiple recognition tasks: (a) recognize the type of interaction, (b) segment regions of interest from acquired data, and (c) learn and recognize object models. The contributions on this work are focused on the recognition modules of this human-robot interactive framework. First, we illustrate the advantages of multimodal data over camera-only datasets. We present an approach that recognizes the user interaction by combining simple image and language features. Second, we propose an incremental approach to learn visual object models, which is shown to achieve comparable performance to a typical offline-trained system. We utilize two public datasets, one of them presented and released in this work. This dataset contains synchronized recordings from user speech and three cameras mounted on a robot, which captured the user teaching object names to the robot.

I. INTRODUCTION

We are witnessing widespread adoption of service robotics in multiple aspects of our daily life and activities, such as household assistance devices or autonomous cars. One of the key aspects in service robotics is a comfortable and intuitive human-robot interaction. For such, it is essential to learn world models, affordances, and capabilities from the user's knowledge and behavior.

Learning systems can be trained offline, using all the data available at a specific moment, or incrementally, by augmenting and updating the learned model as new samples of data become available. Our work is focused on this latter option, for two reasons. Firstly, although offline learning has shown impressive performance, specially since the recent deep learning wave, it requires copious amounts of data. This amount of data is not available for all the relevant scenarios of human-robot interaction and other strategies are required. Secondly, after an initial learning phase, a service robot environment might change abruptly, which would require re-learning models from scratch. An incremental learning scheme is essential to deal with those changes.

From the robot perspective, a framework for interactive object learning from a human user should contain multiple modules: 1) recognizing the action/interaction that the user is performing; 2) segmenting the image regions relevant to the current object of interest; 3) recognizing the class of the object of interest and incrementally learning a visual model Fig. 1: Schematic for interactive incremental learning. Human and robot interact over objects on a table. Through the communication the robot learns to name and recognize new objects.

for each object class. Our current work is focused on the recognition steps (user interaction type and object classes).

This work presents several contributions towards a longterm goal of intelligent systems capable of assisting human users in daily tasks: (a) We present an interaction framework and setup based on language and vision for the incremental learning of object models. Fig. 1 shows a schematic of this framework. (b) We release a dataset, acquired on the presented setup, for interactive and incremental learning. It contains synchronized audio, multi-camera videos (three cameras), and depth information (two of the cameras are RGB-d sensors). To our knowledge, this is the first available dataset that offers user interaction data from the robot's perspective with multi-camera and microphone synchronized. (c) We present an illustrative example of the benefits of multimodal data to recognize types of user interaction with the robot, while the user is teaching different object classes to the robot. (d) We propose an incremental visual learning scheme for object models, that achieves results comparable to a common offline learning approach, within just a few iterations.

II. RELATED WORK

In recent years, significant advances have been made in the field of incremental learning. Incremental learning in general is perfectly suited to robotics, as the data arrives sequentially and the robot needs to keep the best model up to date at real time (e.g., inverse dynamics incremental learning in [START_REF] Gijsberts | Real-time model learning using incremental sparse spectrum gaussian process regression[END_REF]). More specifically, incremental learning is highly valuable in scenarios where human-machine interaction is required.

Our work is oriented towards this direction. There are plenty of applications where a service robot assists a human user to perform certain tasks [START_REF] Reiser | Care-o-bot R 3-creating a product vision for service robot applications by integrating design and technology[END_REF], [START_REF] Baraglia | Initiative in robot assistance during collaborative task execution[END_REF], [START_REF] Dumora | Robot assistance selection for large object manipulation with a human[END_REF]. Through the interaction, the robot incrementally learns and improves the models required for the assistance. Bohg et al. [START_REF] Bohg | Interactive perception: Leveraging action in perception and perception in action[END_REF] presented a very recent survey on interactive perception and how it can be leveraged for robotic actions, with specific references for interactive object modeling.

Frequently in related work, the robot interacts directly with the scene, e.g. grasping and moving an object, to build an incremental object model [START_REF] Kenney | Interactive segmentation for manipulation in unstructured environments[END_REF], [START_REF] Iravani | Visual object classification by robots, using on-line, self-supervised learning[END_REF], [START_REF] Krainin | Autonomous generation of complete 3d object models using next best view manipulation planning[END_REF], [START_REF] Sinapov | Learning relational object categories using behavioral exploration and multimodal perception[END_REF], [START_REF] Collet | Object recognition and full pose registration from a single image for robotic manipulation[END_REF]. Our approach is complementary to these works, as human interaction is needed in real scenarios, e.g. if the object is out of reach of the robot. Very related to our work, Pascuale et al. [START_REF] Pasquale | Teaching icub to recognize objects using deep convolutional neural networks[END_REF] uses Convolutional Neural Networkbased features and Support Vector Machine classification for visual recognition. Training data used there consists of egocentric images where a human presents an object in front of the robot. Camoriano et al. [START_REF] Camoriano | Incremental object recognition in robotics with extension to new classes in constant time[END_REF] harnessed that data (vision-only data and user interactions consisting always of a user showing the objects to the robot) and presents a variation of Regularized Least Squares for incremental object recognition. The contribution of our proposal, over these works is the use of multimodal data and different types of user interactions, such as pointing interactions, aiming at a more natural human-robot interaction.

In mobile robotics, we find multiple examples that propose how to incrementally adapt environment visual models as the robot moves. These approaches are often based on Gaussian mixture models that can be easily updated and maintained to recognize regions of interest for the robot [START_REF] Dahlkamp | Selfsupervised monocular road detection in desert terrain[END_REF], [START_REF] Rituerto | Label propagation in videos indoors with an incremental non-parametric model update[END_REF]. Other works, such as Angeli et al. [START_REF] Angeli | Fast and incremental method for loop-closure detection using bags of visual words[END_REF], present an incremental method to build a model to recognize visual loop-closures.

The relevance of interactive and incremental recognition extends beyond the field of robotics. Yao et al. [START_REF] Yao | Interactive object detection[END_REF] proposed an incremental learning method, that continually updates an object detector and detection threshold, as the user interactively corrects annotations proposed by the system. Kuznetsova et al. [START_REF] Kuznetsova | Expanding object detector's horizon: incremental learning framework for object detection in videos[END_REF] investigated incremental learning for object recognition in videos. Lee et al. [START_REF] Lee | Incremental learning from a single seed image for object detection[END_REF] presented a SIFTvocabulary that constructs an incremental graph from a single image.

The main problem we study in this work is how to learn an incremental model for object recognition from the robot. Object recognition is a traditional research area in computer vision and robotics, and the literature is densely populated with public datasets. For example [START_REF] Lai | A large-scale hierarchical multiview rgb-d object dataset[END_REF], [START_REF] Singh | Bigbird: A large-scale 3d database of object instances[END_REF] are two wellknown examples of datasets targeting object recognition from RGB-D images. However, most of the existing datasets focus on offline visual learning. Interactive, multi-sensor, and multimodal datasets are scarcer. There are multiple aspects that should be considered on a dataset targeted for interactive learning. In particular, we focus on the expected manner that the data will be presented and multimodal data available.

In a realistic human-robot interactive learning scenario, the training data is expected to be presented in a very different way than the previously mentioned datasets. The data is expected to be shown by a human actor through different ways of interaction. The data is expected to be seen from the point of view of the robot. Recognizing a pedestrian from a close-up view from service robot is immensely different from performing the same task with the raw video data from a distant wide-angle surveillance camera.

Vatakis et al. [START_REF] Vatakis | A multimodal dataset of spontaneous speech and movement production on object affordances[END_REF] shows multimodal recording approach similar to ours, but the purpose of their dataset was to capture the reactions of users to stimuli with objects or images in a screen. Datasets like [START_REF] Jaeyong | Human activity detection from rgbd images[END_REF] or [START_REF] Gong | A New Image Dataset on Human Interactions[END_REF] capture human-robot interaction from a third-person point of view (POV). This is useful in some cases, but since we are working with service robots, information must be taken from the onboard sensors. Additionally, many datasets lack additional sensor data that is easy to find in human-robot interactive scenarios like speech from the user. As detailed in next section, our released dataset is focused not only on capturing the user's expression, but also on capturing the scene information (hands, arms, desk, objects, etc.) related to the object classes being taught to the robot. The dataset1 contains recordings of several users teaching different object classes to the robot. It contains synchronized data from two Microsoft Kinect v1.0, a 1280 × 720 RGB camera, and a microphone. The cameras are mounted so that the user and the table on which the interaction occurs, are perfectly covered. The first, torso-mounted RGB-D Kinect is focused on the frontal interaction with the robot, the second one (head-mounted) gives an aerial view of the table and the HD-RGB camera is focused on the user's face. Figure 2 shows the placement of the cameras in the Baxter robot used for the acquisition.

III. MULTIMODAL HUMAN-ROBOT INTERACTION

The types of interactions captured in the dataset reflect the most common ways users show or teach objects to the robot: Point, Show, and Speak. The interaction Point captures a user pointing at an object and calling out its name to the robot. The interaction Show describes a user grabbing an object and bringing it approximately in front of the robot's torso camera while announcing the object's name. The interaction Speak captures a user describing to the robot where a certain object is placed. Figure 3 shows an example of each of these three types of interactions. Table I contains a summary of the contents of the dataset. We recorded 10 users, each of them performing 10 object interactions for each of the 3 tasks (point, show, speak) for a total of 300 recordings. The users were allowed to freely choose which objects they wanted to explain out of a pool of 22 objects on the table. They were given unspecific instructions on the interactions, meaning no exact phrasing. As a result, there is a natural variation in language usage between speakers. Figure 4 shows some examples of the dataset recordings. One of the main goals in our research is to exploit multimodal data for interactive learning, with a focus on natural exchange between robot and human. This section describes an illustrative experiment which shows the advantages of utilizing more than one modality of data. As we can see in [START_REF] Abdullah | Integrating audio visual data for human action detection[END_REF], combining language and image data can significantly boost the user action recognition. In our case action recognition is not the goal of the experiment, but only an initial step to facilitate the incremental object learning by the robot.

Our dataset includes three types of interaction: Point, Show, and Speak. A rough classification of the interaction can simplify and optimize the object segmentation step, in Fig. 4: Examples from the Human-robot interaction dataset. Each column shows the same event captured from the 4 sensors.

which the robot needs to extract the object's visual features. For example, if the user is pointing to an object, the object is very likely to be within a very restricted image region that we can estimate. If the user is grabbing an object and showing it to the robot, our system can assume that the object is held by the user's hand. In this case, it could predict that the hand is likely to occlude parts of the object.

A. Multimodal interaction recognition

This experiment aims to recognize the three types of interactions that occur in the dataset (Point, Show, and Speak) from language and visual cues. We are seeking a simple prefilter for the incremental object learning, in order to avoid costly spatio-temporal video analysis, and propose a lowcost individual frame classifier. Firstly, notice in Figure 6 the difficulty of the interaction recognition problem from a single frame. Classifying these 4 frames into Point, Show, or Speak can be tricky even for a human. It is hard to distinguish from a single frame if the user is pointing at an object, about to pick it up, or just narrating.

Visual data features: Our image descriptor is computed as follows. First, we segment the image into SLIC [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] superpixels. We classify each superpixel as skin/not skin using color and depth and fuse the adjacent skin superpixels into blobs. Afterwards we divide the image into a 5 × 5 grid, with the descriptor being a histogram summing the skin votes of the blobs per each image cell.

Language data features: Our language feature is the first word of each user sentence, which is 1) "that" when the user is pointing to something far; 2) "this" when the user is pointing to something close or showing something to the robot; 3) any other word if the user is just describing, saying, something to the robot. Figure 6 shows the language feature distribution for each user and interaction. Notice that the feature is not discriminative at all for the classes Point and Show, and therefore training a classifier with only this feature is not an option because the precision on those classes will be near random. Interaction classifier: We train a RBF-SVM classifier first using only visual data and then using the combination vision-language. We report the confusion matrix (10-fold validation) for both cases -Table II is vision-only and Table III is vision-language. Notice the boost in the performance for the multimodal descriptor, in particular for the class Speak that is ambiguous for the visual descriptor.

V. INCREMENTAL LEARNING OF OBJECT MODELS

We work towards a system where a robot can learn object models incrementally, while maintaining a limited amount of data stored. Our approach works with object views which are windows of images containing the objects. When the system gets a new object view, it attempts to assign a label from its database of currently know objects (if any). Based on the confidence of this assignment, the robot will either merge this information into the corresponding object cluster information or ask the user if the answer is correct. If the user confirms the label is correct, this new view of the object is merged to its model in the database. If the assigned label is wrong, the user will provide the label name of this new object and it will be incorporated into the database.

Our object model database consists of a set of representative descriptors for each object, described in sec. V-A. Each of these representative descriptors can be seen as the centroid of a cluster in the database. When a new object is added, the descriptor of that first object view is directly used as the seed centroid of a new cluster.

A. Object View Description

Our system is designed to run on robotic platforms, where computational performance is limited. Therefore, we propose to use descriptors that are reasonably small and fast to compute, but still able to discriminate among the common objects the robot will have to interact with. Our system uses two kind of descriptors to represent the image region enclosing the object we aim to learn:

BoW Histogram: This descriptor consists of a histogram of the frequency of occurrences of a set of visual words built from commonly used local image features. In particular, we use ORB [START_REF] Rublee | Orb: An efficient alternative to sift or surf[END_REF] features, since they provide a good compromise between efficiency and amount of key points detected.

The visual words, or vocabulary, are obtained as a result of clustering all the point features extracted on a large set of images from a public object dataset [START_REF] Lai | A large-scale hierarchical multiview rgb-d object dataset[END_REF]. We compute 1000 clusters from more than 2 million features extracted from around 12000 images. These images contain both clean images of all object classes in the dataset and test scene images, with objects and clutter.

To build the descriptor d BoW of a new image i, we extract point features, find the closest word to each of them and calculate d BoW as a 1000-bin histogram of the frequency of occurrence t w of each word in the image as:

d BoW = [t 1 , ..., t w , ...t 1000 ] t w = n wi n i , (1)
where n wi is the number of occurrences of word w in image i and n i is the total number of words in image i.

Color Histogram: This descriptor d RGB provides information about the color distribution in the image region described. We compute three normalized 8-bin histograms over the color pixel information on the region. There is one histogram, H c , for each RGB channel:

d RGB = [H r H g H b ] (2)
Our experiments evaluate the results, by using only one of these descriptors or by combining them. Since the image region representing an object can be small, there could be none or only very few point features present. Consequently not enough information would be obtain from them, as can be seen for example in Figure 7. Therefore, although point features are usually more discriminative than color histograms, as shown in section VI-C, the best option is the combination of both descriptors, which we will refer to as d ALL .

Fig. 7: Sample new objects views provided by the user to the robot. These objects segments occupy small image regions where there are often not enough point features found.

B. Classify and Learn from a New Object View.

Our incremental system is inspired by incremental clustering ideas. Each cluster in our model represents a representative subset of the descriptors seen so far for a certain class. As we get new samples, existing clusters evolve and update their centroids (representative descriptors) and new clusters are created for new objects. The total number of classes (N_Class) is not limited but, in order to avoid unlimited growing, the subset for each class is limited by a predefined size S. The algorithms proposed to build this model (training), and to classify the object (recognition) are shown in pseudo-code in algorithm 1 and algorithm 2 respectively.

Incremental training: Given a new view object v and a label l, first the descriptor d v is calculated. A new cluster is created with d v as centroid and l as label associated. If the label l doesn't exist in the database L, l is added to the database. If l has reached the maximum number of clusters associated S, the algorithm finds the clusters with minimum distance between them of this class and merges them. This method limits the amount of overlap that can occur in classes and maintain real data in the model. To take into account the confidence in the classification result, we establish a threshold, T h, on the distance to nearest % train. data 20% 30% 40% 50% 60% 70% 80% 90% neighbor found on the database, i.e., D B (v, C[x]). If it is above this threshold, the system asks the user about the label for this object. T h is experimentally set as we explained in section VI-A.

VI. EXPERIMENTAL RESULTS

The goal of the experiments described in this section is to show the performance of our incremental learning algorithm for object recognition.

A. Experimental Setup

The baseline considered for these experiments is an offline trained nearest-neighbor classifier that uses all the training data. We run 10-fold cross-validation for all of our experiments (both for the baseline and our incremental approach), making sure that there are at least two images of the same class in each fold. The evaluation metrics we consider are the accuracy (defined as the true positives over the total test samples) and the amount of data stored.

The parameter values for our approach, which are used in all the experiments, are as follows. The maximum number of clusters allowed for the model of one class is S = 20. The threshold, T h, set to decide when the descriptor distance is low enough for the robot to take a decision, i.e., assign a class to an object view, without asking the user (as explained in section V-B), is set to 0.1 for d RGB , 0.7 for d BoW and 0.7 for d ALL . These thresholds were set in such a way that they were providing a 99% of accuracy exceeded in the classification performed.

The algorithms were developed in Python using OpenCV and Sklearn libraries, and the experiments were run on a PC with Intel R Core TM i7-6700 CPU at 3.40GHz, 32GB RAM.

B. Results on the Washington Dataset

In this first experiment, we evaluate our approach, with respect to the offline baseline approach, using data from the Washington dataset [START_REF] Lai | A large-scale hierarchical multiview rgb-d object dataset[END_REF]. This dataset contains small and clean images of objects. This is in contrast to our dataset, which is noisier and prone to occlusions. We use the two descriptors proposed combined (d ALL ). Figure 8 and Table IV summarize the results. In Figure 8(a) we can see that our system's performance quickly reaches 90% accuracy, close to the baseline accuracy of 96%. Interestingly, Figure 8(b) shows that we require only to store one third of the data, compared to baseline, to achieve this result. Additionally, Table IV shows that our data usage converges to around 1000 samples, whereas the baseline linearly increases. This shows that our system maintains a stable precision without having an increase in the amount of data stored.

C. MHRI Dataset evaluation

Whereas the images in the Washington dataset are very clean, our MHRI dataset is better suited for an evaluation of a realistic scene and a natural human-robot interaction. We evaluated three descriptor options, as described in section V-A: RGB histogram (d RGB ), BOW histogram (d BoW ), and a combination of both (d ALL ).

Table V shows the accuracy of the baseline classifier for the three types of descriptor considered. The combination gives the best performance (75.50%). Notice that, as we advanced above, the performance of using only a BoW histogram descriptor is poor, as the low-resolution training views contains very few -if any-salient points.

Table VI shows the accuracy of our approach. The combination gives us the best score, it goes from 48% to 74%, very similar to top baseline performance. In Figure 9, we can see the progress of the performance for our incremental approach and the baseline. Our system's precision is a little lower than baseline, but, as we can see, the amount of data stored in our system is significantly less than baseline. Consequently, our system sacrifices a smidgen of precision for a considerable amount space, often costly in robotic and embedded systems.

Figure 10 shows the time expenditure for processing a single image while training and testing. In training, the time is in the range between 1 and 25 milliseconds. There is fluctuations depending on the amount of data for each class and if reorganization is required. In testing, the times range from 6 to 16 milliseconds. This range is smaller than the training range, and also depends on the amount of data in each class. As mentioned earlier, the fact that our object images, which come from the scenes with the user and the objects, influences the object segmentation. We designed an experimental setup to evaluate the magnitude of that influence. We separated our dataset into two groups. One contains all the Show interaction (in which the object can be occluded partially by the user holding it). The other group contains the remaining two interactions, Point and Speak (where the hand does not occlude the object). To perform the crossvalidation we divided each group into blocks, 3 blocks for the occlusion group and 7 for the "clean" group. As in the experiments before, we enforced that at least two images per class are in each block and we use d ALL as the descriptor.

The same experiment was run for both groups. We trained a model for each of the two groups, and performed three tests. For the first test, we trained each group's model with all the data from the opposing group and tested the model on each block. For the second test, we trained both group model on their own group data, and performed cross- In Table VII we can see that in the first test the accuracy is lower than previous experiments, around 26% with occlusion and 37% with the other. The second and the third test show that the occlusion group is more difficult to classify. The performance of the occlusion group is around 50% and the other group is around the 90%. Also we can see comparing the second and the third setup, that mixing the groups has little impact on the performance. This was to be expected for the clean images, but also illustrates that even with the occluded ones dividing the data can improve the precision.

D. Domain change

In this experiment we train the system with the Washington dataset images and use the test images of our dataset. Our aim is show experimentally the need for incremental learning, even for offline trained categories, due to the dataset bias.

As there is not a complete overlap between the objects in the Washington dataset and ours we cannot make a fair quantitative comparison in the same terms of the previous experiments. Figure 11 shows several qualitative examples. For the reduced test set that overlaps with the Washington dataset we observed that the accuracy was less than 20%.

VII. CONCLUSIONS

In this paper we have presented a framework for incremental and interactive learning of object models from multimodal robo-centric data. We also released an annotated multimodal dataset acquired in the context of the presented framework, but suitable for training and evaluation beyond this setting. We have presented and validated two recognition approaches for different stages of the whole interactive learning framework, both using the released dataset.

First, we have shown how even a simple task, classifying the type of user interaction from single frame global descriptors, can significantly benefit from multimodal data. The interaction recognition is a crucial step to facilitate future processing steps, such as automatic object and region of interest segmentation in images and videos. The contributions on this work are focused on the recognition modules of the human-robot interactive framework. In our incremental learning approach the object segmentation is done manually, Fig. 11: Examples of our system trained with the Washington dataset and tested with our dataset. The output label is written in the image and the view used is shown in a blue rectangle. therefore future lines of work will include the interaction recognition with automatic object segmentation.

Second, we have proposed an incremental approach to learn object models from the interaction with the user. Our experimental results show that our proposed system achieves results comparable to offline training, while operating on a much more limited amount of stored data. We evaluated our approach on both a standard public object recognition dataset, and in our multimodal dataset, which contains sample images of users interacting with objects and the robot. In addition to the interaction, our dataset presents other challenges like occlusions and low object resolution. A remarkable feature of the dataset is the synchronized recording of multimodal data, specifically with two RGB-D cameras, one high resolution RGB camera, and audio data. The incremental object learning approach presented here uses only visual data from a single camera. Future lines of work, taking advantage of the multimodal dataset released, will extend the presented approach by harnessing the higher modality.

Fig. 2 :

 2 Fig. 2: Image of the Baxter robot used to acquired the dataset. The three camera positions are highlighted.

Fig. 3 :

 3 Fig. 3: Three types of user interaction. In the respective cases the user would say for example (a) "This is a box", while pointing to the box. (b) "This is a box", while holding the box. (c) "The box is next to the chips and has a banana on top."

  type. 1 Object per interaction. Objects 22 Apple, Banana, Bottle, Bowl, Cereal Box, Coke, Diet Coke, Ketchup, Kleenex, Knife, Lemon, Lime, Mug, Noodles, Orange, Plate, Spoon, Tall Mug, Tea Box, Vase

Fig. 5 :

 5 Fig. 5: Sample frames from user interactions with the robot.It is hard to distinguish from a single frame if the user is pointing at an object, about to pick it up, or just narrating.

Fig. 6 :

 6 Fig. 6: Stacked graphic showing the language feature occurrences during the three types of interaction from all users.

4 )

 4 C l is the set of clusters corresponding to label l. The distance used to compare two clusters D B is the Bhattacharya distance between their centroid descriptors.(x, ŷ) = argminx,y{D B (C l [x], C l [y])} x = y (3) Clusters C[x] and C[ŷ] are merged into one cluster C[z]with L associated and the centroid is randomly chosen from one of them. Data: v, l d v =Calculate_descriptor(v); if N_Class == 0 then L.add_class(l); C.create_cluster(l,d v ); else C.create_cluster(l,d v ); if l is not in L then L.add_class(l); else if len(C_l) > S then distance_min = inf; for each x in C_l do for each y in C_l do if x = y then d = D B (C[x],C[y]); if d < distance_min then distance_min = d; x = x; ŷ = y; end end end end C.Merge(C[x],C[ŷ]); end end end Algorithm 1: Training Algorithm Data: v Output: l,Confidence d v =Calculate_descriptor(v); if N_Class == 0 then return None,False; else distance_min = inf; for each x in C do d=D B (d v ,C[x]); if d < distance_min then distance_min = d; x = x; end end if distance_min < T h then confidence = True; else confidence = False; end return L[x], Confidence; end Algorithm 2: Recognition Algorithm Recognition of a new view: Our recognition step classifies new object view v into the existing classes following a simple nearest neighbor approach. It computes a distance from the descriptor d v of this new view to the centroid of each model cluster. The distance D B is the Bhattacharya distance between v and the model centroids. C is the set of the current model clusters, and C[x] represents the centroid of cluster x. x = argmin x {D B (v, C[x])} (Since each existing cluster has an object label assigned, let us represent it as L[x], the new view will be classified as class L[x].

Fig. 8 :Fig. 9 :

 89 Fig. 8: Incremental object learning with different amounts of training data using the Washington dataset. IC is our approach and Offline is the offline baseline. (a) Accuracy vs training size. (b) Database size vs training size.

Fig. 10 :

 10 Fig. 10: Time used by our approach to process one image. The top plot is for training, the bottom one is for testing. The x-axis represents the number of samples trained or tested before processing this image. The y-axis displays the average time (seconds) for the 10 different executions from crossvalidation.

TABLE I :

 I

Overview over the parameters which were set to record the dataset.

IV. LEARNING FROM MULTIMODAL INTERACTIONS.

TABLE II :

 II Confusion matrix for interaction recognition using visual data only.

		Point	Show	Speak
	Point	29,39%	10,36%	0,00%
	Show	13,13%	6,63%	0,00%
	Speak	0,32%	0,00%	40,17%

TABLE III :

 III Confusion matrix for interaction recognition using visual and speech data.

TABLE IV :

 IV Database size for each method using different % of training data from the Washington dataset.

TABLE V :

 V Accuracy (%) for the offline trained baseline classifier using different descriptors and our MHRI dataset.

	% train. data 20% 30% 40% 50% 60% 70% 80% 90%
	d RGB	47.5 57.7 65.0 68.7 69.9 70.1 72.2 72.2
	d ALL	48.2 57.6 64.7 68.8 70.6 71.1 72.8 73.3

TABLE VI :

 VI Accuracy (%) of our approach using different descriptors and the presented MHRI dataset. Columns represent the amount of data used for training.

TABLE VII :

 VII Accuracy mean and standard deviation for training/test sets of different/same domains. validation within each group. For the third test, we trained both group model on their own group data, but performed cross-validation across all blocks of both groups. With this setup we intend to measure the influence of training on different interaction scenarios.

http://robots.unizar.es/IGLUdataset/


Grégoire Nieto

Frédéric Devernay

James L Crowley

Variational Image-Based Rendering with Gradient Constraints

Keywords: Image-Based Rendering, Computational Photography, Computer Graphics

  L'archive ouverte pluridisciplinaire

INTRODUCTION

Multi-view image-based rendering consists in generating a novel view of a scene from a set of source views. In general, this works by first doing a coarse 3D reconstruction of the scene, called a geometric proxy, then using this reconstruction to establish correspondences between source and target views(Figure 1 (a)), followed by blending the warped views to obtain the final image. Recent work by Pujades et al [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF] proposed a Bayesian formulation of the image-based rendering problem, building on previous work by Wanner and Goldluecke [START_REF] Goldluecke | Superresolution texture maps for multiview reconstruction[END_REF][START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF]. They showed that the weight of each source image in the novel view could be formally deduced from the camera properties, image content, and accuracy of the geometric proxy, leading to a formalization of the heuristic blending weights proposed initially by Buehler et al [START_REF] Buehler | Unstructured lumigraph rendering[END_REF]. Most of the "desirable properties that [...] an ideal image-based rendering algorithm should have" [START_REF] Buehler | Unstructured lumigraph rendering[END_REF] were thus given a formal explanation, except for the continuity property. Therefore, discontinuities in the source image weights, which are mainly Thanks to DGA for funding due to scene geometry or camera placement, result in artifacts in the target view.

In this work, we show that a way to avoid these artifacts is to impose additional constraints on the image gradients of the novel view. These constraints come from a simple observation: an image contour in the target image should also be present in source images where this part of the scene is visible. An energy functional similar to the one in Pujades et alis derived, which is composed of the usual data term and smoothness term, but the data term has an additional term which takes into account these gradient constraints. The resulting energy is optimized in a variational framework, by iteratively solving a linear system.

Our method is composed of two stages: a 3D reconstruction pipeline (section 4.1) followed by the target view rendering (section 3). The depth maps we obtain from multi-view stereo are used to compute a warp function from each input image to the novel view. The target view is computed via a novel variational formulation of the image-based rendering problem. We demonstrate the method on several unstructured multi-view datasets (section 4) and show that not only it numerically outperforms state-of-the-art methods (section 4.2), but also it eliminates artifacts that originated from visibility discontinuities. We conclude that taking into account both intensities and gradients in image-based rendering methods offers an elegant solution to enforcing the continuity property initially devised by Buehler et al.

PREVIOUS WORK

Image-Based Rendering (IBR) has been extensively reviewed by Shum et al [START_REF] Shum | Image-based rendering[END_REF]. Most state-of-the-art [START_REF] Kopf | Image-based rendering in the gradient domain[END_REF][START_REF] Sinha | Image-based rendering for scenes with reflections[END_REF][START_REF] Lipski | Correspondence and depth-image based rendering a hybrid approach for free-viewpoint video[END_REF][START_REF] Lipski | Virtual video camera: Image-based viewpoint navigation through space and time[END_REF][START_REF] Chaurasia | Depth synthesis and local warps for plausible image-based navigation[END_REF] approaches use a coarse 3D reconstruction of the scene, called a geometric proxy, which may have various degrees of accuracy. Ortiz-Cayon et al [START_REF] Ortiz-Cayon | A bayesian approach for selective image-based rendering using superpixels[END_REF] choose to over-segment the image and compute the quality of several IBR algorithms on each super-pixel. Then the output of the best IBR algorithm for each super-pixel is picked. These algorithms are largely inspired by the Unstructured Lumigraph [START_REF] Buehler | Unstructured lumigraph rendering[END_REF], that performs a blending of the k-nearest views, weighted by angles and distances to the target view, thus guarantying a smooth camera blending field. The continuity of the resulting blend in image domain is ensured by enforcing spatial smoothness on these weights, but temporal artifacts may still occur if the contributing cameras are too sparse. Davis et al [START_REF] Davis | Unstructured light fields[END_REF] propose a viewpoint-subdivision rendering technique to create a better blending field. Nevertheless, blending weights are still heuristics and the choice of cameras for rendering is arbitrary.

Disposing of heuristics and tunable parameters is a key objective in [START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF], who propose a physics-based Bayesian formulation of image-based rendering. The weight of each input view in the final blending is automatically deduced from the mathematical equations by deriving an energy functional. Pujades et al [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF] went further by integrating geometric uncertainty in the Bayesian formalism. They then obtain new weights that favor cameras satisfying both epipole consistency and minimal angular deviation, two principles stated by Buehler et al [START_REF] Buehler | Unstructured lumigraph rendering[END_REF] to describe the ideal IBR algorithm. They were unable, however, to provide a formal derivation that leads to the continuity principle, especially near the borders of each camera view field. We show that introducing an additional term in the energy functional, not only constrains the image intensities but also constrains the image gradients, providing an elegant solution to the continuity principle.

The key idea of high-quality rendering is that the quality of the image solution often relies on the constraints put on the search space. As a consequence, finding the right regularization or image prior has been widely researched in order to obtain high-quality images. The main contribution of Fitzgibbon et al [START_REF] Fitzgibbon | Imagebased rendering using image-based priors[END_REF] is the use of texture priors computed from a large database of patches to constrain the solution, independently from the input scene data. This idea was recently extended by Flynn et al [START_REF] Flynn | Deep-Stereo: Learning to predict new views from the world's imagery[END_REF] who perform new view synthesis via a deep network architecture, trained by a huge database of real-world image sets. In contrast our method does not rest upon any strong prior knowledge about the new view to synthesize, but rather makes better use of the data provided by the input views to add more constraints on the solution. There-fore our algorithm does not require large numbers of picture sets from the world's imagery to create a high-quality images.

Image fusion in the gradient domain has received much interest in recent years beginning with the seminal paper of Perez et al [START_REF] Pérez | Poisson image editing[END_REF], for applications such as image editing [START_REF] Mccann | Real-time gradientdomain painting[END_REF], inpainting [START_REF] Levin | Learning how to inpaint from global image statistics[END_REF] or image stitching [START_REF] Zomet | Seamless image stitching by minimizing false edges[END_REF]. The closest work to ours for image-based rendering is probably [START_REF] Kopf | Image-based rendering in the gradient domain[END_REF], who generate a new point of view by reconstructing the gradients in the target view, followed by an integration of these gradients on the GPU to recover the final image. However, their method is limited to interpolating between two views, and they neither address the generic case where the input views may have very different viewpoints, fields of view, and resolution, nor produce super-resolved images. Our method addresses these issues and proposes a more generic framework for multi-view image-based rendering.

VARIATIONAL IMAGE-BASED RENDERING

Our goal is to synthesize an optimal novel image u

: Γ ⊂ R 2 → R at the target viewpoint from the input images v k : Ω k ⊂ R 2 → R.
For the sake of simplicity, image values are taken as scalars, but this easily generalizes to color images v k : Ω k → R 3 . In all our experiments we process the images in the RGB color space. Source images are registered via the warps τ k that transform any point x m = (x m , y m ) of the input view k into the corresponding point x p = (x p , y p ) in the target view:

τ k : Ω k → Γ x m → x p (1)

Image Formation Model

As commonly assumed in the super-resolution literature [START_REF] Baker | Limits on super-resolution and how to break them[END_REF][START_REF] Hardie | Joint MAP registration and high-resolution image estimation using a sequence of undersampled images[END_REF], we consider the intensity value v k (x m ) at a point x m in the low-resolution observed image k to be the convolution of values in the super-resolved image with the point-spread function (PSF) b. Given an ideal super-resolved image u at the position of the target view defined over Γ, and a warp τ k that maps points from Ω k (low resolution domain) to Γ (high resolution), if we discard for now the visibility effects, the intensity in the observed image k can be written as:

v k (x m ) = Ω k u • τ k (x) b(x -x m ) dx, (2)
or simply

v k = b * (u • τ k ). The PSF b : Ω k → [0, 1] is a probability density function that can be turned into b k : Γ → [0, 1] by the change of variable x = τ k (x) so that v k (x m ) = Γ u(x )b k (x -τ k (x m )) dx (3)
There are several ways of computing the warped PSF b k , depending on how we model the initial PSF b. The more common assumption is to consider that b is a 2D Gaussian centered at position x m . Since Gaussian filters have infinite support, it is quite difficult to implement in practice. A simpler model of the PSF is to assume pixels are square and uniformly sensitive to light, so that the PSF is a uniform square density function. Noting A the area of a pixel centered on (0, 0) in a source view k, we get b(x, y) =

1 A 2 if -1 A ≤ x, y ≤ 1 A 0 elsewhere. ( 4
)
Under the assumption that the warp τ k is locally linear, the warped PSF is a uniformly distributed parallelogram. In this case, we can make an even stronger assumption by supposing that the warp preserve the pixels (their area and squared shape), which is actually untrue but largely simplify the implementation. From now on, we take a unit pixel area. Since the intensity is constant and equals u(p) over all the pixel area in the target view, the previous convolution can be written as in [START_REF] Hardie | Joint MAP registration and high-resolution image estimation using a sequence of undersampled images[END_REF]:

v k (x m ) = p∈ Γ u(p) p b k (x -τ k (x m )) dx , (5)
so the pixel intensity m in the source image is

v k (m) = p∈Γ B k,m,p u(p), (6)
where B k,m,p = p b k (x -τ k (x m )) dx is the area of intersection between the projection of the pixel in the target view and a pixel p of this view. It is equivalent to bilinearly interpolate the intensities of u.

Maximum A Posteriori (MAP) Estimation

The goal of the variational approach is to estimate the high resolution image u from the data

(v * k ) k∈[1..K] ,
where K is the number of inputs views. The estimator of u maximizes the posterior which is the probability of finding u given the input data. One can show that this is equivalent to minimizing the energy

E(u) = E intensity (u) + λE prior (u), (7)
E prior , often referred to as the smoothness term, comes from the image prior and prevents the emergence of high frequencies. λ is a parameter that controls the smoothness of the final solution. In this work, we use a total variation prior [START_REF] Goldluecke | An approach to vectorial total variation based on geometric measure theory[END_REF], E prior (u) = Γ |∇u|, which has several advantages over other more complicated image priors: this approach preserves strong edges and image contours, and is convex. A proof of convergence is given by Chambolle [START_REF] Chambolle | An algorithm for total variation minimization and applications[END_REF]. E intensity , often referred as the data term, is derived from the likelihood given the input image intensity [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. This accounts for how well the current solution fits the data in the intensity domain:

E intensity (u) = K k=1 1 2 Ω k ω k (u)((b * (u • τ k ) -v * k )) 2 dx. (8)
The terms ω k (u) are the per-pixel contribution of each input view k. They depend on the gradient of the current solution u and the geometric uncertainty of the 3D reconstruction.

Appending the Gradient Term

Since the geometry and the visibility may be discontinuous, the term ω k (u) in 8 may be discontinuous too, resulting in artifacts in the synthesized image that may appear as spurious edges or textures (Fig. 2). The IBR method should prevent these contours from appearing: actually, an image contour synthesized in the target image should also be present in source images where this part of the scene is visible.

To enforce this property, we add an extra term E gradient (u) to the previous energy (8) that forces the current solution to also fit the data in the gradient domain:

E gradient (u) = -log p(∇v 0 . . . ∇v K-1 |∇u) (9) = - K-1 k=0 log p(∇v k |∇u) (10)
= Ω k (∇v k -∇v * k ) 2 dx (11) = Ω k (∇(b * (u • τ k )) -∇v * k ) 2 dx.( 12
)
Finding u that minimizes this energy is equivalent to solving the Laplace equation:

∆((b * (u • τ k ) -v * k ) = 0, (13)
where ∆ = ∇.∇ denotes the Laplacian. We instantly deduce the derivative of the functional:

dE gradient (u) = (| ∂τ k ∂z | -1 b * (∆(b * (u • τ k )) -∆v * k )) • β k . ( 14
)
The β k are the backward warp that appear thank to the change of variable in the integral. b is the adjoint of the blur kernel b. The warps τ k are those which were estimated beforehand, and thus lack precision. This uncertainty has a drastic effect on the computation of ∆(b * (u•τ k )). As a consequence, we chose to compute the Laplacian of u first, then warp it in the Ω k domain. Under the assumption that the warps τ k can be locally linear, we neglect their second order derivatives and obtain:

∆(b * (u•τ k )) = b * ∂τ ∂x H u ∂τ k ∂x + ∂τ k ∂y H u ∂τ k ∂y . (15)
H u = ∂∇u ∂x is the Hessian matrix of u. Due to uncertain depth maps that cause strong discontinuities in warps, the Hessian matrix may be very unstable. For the computation, and in this case only, we assume τ k (x) ≈ x + d so that

∆(b * (u • τ k )) = b * (trace(H u ) • τ k ) = b * (∆u • τ k ). (16)
The final form of the energy to minimize is thus:

E(u) = αE intensity (u) + γE gradient (u) + λE prior (u).
(17) We minimize the functional (17) via Fast Iterative Shrinkage Thresholding Algorithm (FISTA) [START_REF] Beck | A fast iterative shrinkagethresholding algorithm with application to waveletbased image deblurring[END_REF].

Discretization

One can wonder how we go from a continuous model of the problem to a numerical solution. For each pixel m of each input view k we obtain an equation similar to [START_REF] Davis | Unstructured light fields[END_REF]. Let V * be the vector of all the pixels of every input view put in one column (v 0 (0) v 0 (1) ... v K-1 (M -1)) , U the column vector storing the current solution (u(0) ... u(N -1)) , and B the KM ×N matrix that stores the B k,m,p coefficients, we may naturally write V = BU. Consequently, we can express the energy (8) as a linear system:

E intensity (U) = (BU -V * ) W(BU -V * ), ( 18
)
where W is a KM × KM diagonal matrix that stores the weights |J x (β k )|ω k . To minimize this energy we derive the linear system, and obtain the normal equations that provide an estimator for the solution Û:

B WB Û = B WV * . ( 19
)
The matrix B WB is generally not invertible. This linear system can be solved by any linear least square solver.

Akin to the data term on color, the data term on image gradient is

E grad (U) = (B∇U -∇V * ) (B∇U -∇V * ) ( 20
)
and can be derived likewise.

EXPERIMENTS AND VALIDATION

3D Reconstruction

Our method takes as input an unstructured set of source views with no particular structure [START_REF] Buehler | Unstructured lumigraph rendering[END_REF], as opposed to view interpolation methods which usually take stereo pairs, or methods that are based on a structured light field [START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF]. To be as generic as possible, it only requires the warp functions τ k that we obtain via a classic 3D reconstruction pipeline [START_REF] Fuhrmann | MVE -a multiview reconstruction environment[END_REF]. It consists in camera calibration via bundle adjustment [START_REF] Moulon | La bibliothèque openMVG: open source multiple view geometry[END_REF], followed by an MVS reconstruction [START_REF] Goesele | Multi-view stereo for community photo collections[END_REF] to get a depth map for each input view (Fig. 3). Knowing the camera parameters and the depth of each input pixel, we can deduce per-pixels correspondences between the source images and the target Fig. 3: The depth map of an input view.

Fig. 4: Results on datasets fountain and herzjesu. The bottom row shows some of the input views. Note that the parts of the target view that are not visible by any of the inputs are filled by a push/pull inpainting algorithm.

view. We use the depth to derive the blending weights as it is demonstrated in [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF]. For further information about the reconstruction pipeline, more specifically occlusion handling and the derivation of the blending weights, see [START_REF] Nieto | Rendu basé image avec contraintes sur les gradients[END_REF].

Results

A set of experiments (Figure 5) is performed on real views taken from Strecha's dataset [START_REF] Strecha | On benchmarking camera calibration and multi-view stereo for high resolution imagery[END_REF], fountain and herzjesu. Geometric proxies are estimated according to the pipeline described in section 4.1. For both datasets we remove the central view, render it in the same dimensions as the input images and compare the result with the original for visual evaluation. All experiments were performed on GPU with an nVidia GTX Titan. Convergence is reached within 3 minutes for an input set of 11 input images of size 3072 × 2048. Since the convergence time strongly depends on the initialisation of the solution, the performance could be significantly increased. However, one drawback of our variational method is that it does not perform real-time rendering. To show that the quality of our results does not depend on the initialization, we start the optimization process from a null image. Since some parts of the target view are not visible from the input views due to self-occlusion, inpainting is needed to fill potential holes. To that end, we implemented the push/pull algorithm as it is described in [START_REF] Gortler | The lumigraph. SIGGRAPH '96[END_REF]. Firstly, the push stage decomposes the final image u into its Gaussian pyramid. At each level of the pyramid the image is filtered by a Gaussianlike 5 × 5 kernel and down-sampled by a factor 2. Only nonnull value pixels contribute to create the upper-level image, so that at the coarsest level of the pyramid the image has no hole. Note that we do not need to down-sample to the 1×1 pixel image; in our experiments the 6×4 pixel image does not contain any holes. Secondly the pull stage propagates missing information from the top of the pyramid downwards to the finest level. At each level, holes are filled with the corresponding pixels from the upper coarser image.

At first we tested for a null gradient term (γ = 0.0). Since the linear system to be solved is ill-constrained, high frequencies appear in areas that few cameras see. These artifacts are emphasized by a very noisy depth estimation near occlusion regions (around the fish on the fountain or the Jesus on the wall). To remove these artifacts and eliminate high frequencies, a stronger smoothness term is commonly used. Consequently we increased the λ parameter that controls our Total Variation regularizer to 0.003. However the results are not that convincing: although most high frequencies are removed, some image features are lost compared to originals. To fix this problem, we achieved a third rendering where λ is set to its initial value (0.002) and the gradient data term is appended to the equation (γ = 1.0). For the final solution to keep the original colors of the input image, we keep the intensity data term as a bias but choose a small controlling parameter (α = 0.1). One can notice that artifacts are completely removed, while preserving all of the image features. We conclude that appending the gradient data term prevents the emergence of spurious contours near visibility borders, hence guaranteeing the continuity property. Some numerical results are presented in table 1. To show that our rendering algorithm outperforms state-of-the-art methods when synthesizing a specific view only, we generate several input views that we preliminarily removed from the input α = 1.0, γ = 0.0, λ = 0.002 α = 1.0, γ = 0.0, λ = 0.003 α = 0.1, γ = 1.0, λ = 0.002 ( [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF][START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF]) ( [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF][START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF] 1: Numerical results on real-world datasets [START_REF] Strecha | On benchmarking camera calibration and multi-view stereo for high resolution imagery[END_REF]. Our method is compared against state-of-the-art methods [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF][START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF], for which there is no gradient constraints (γ = 0.0). For each result, the first value is the PSNR (bigger is better), the second value is DSSIM in units of 10 -4 . DSSIM = 10 4 (1 -SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] (smaller is better). The best value is highlighted in bold. See text for a detailed description of the experiments.

set and kept as reference views for quantitative comparison. Two numerical measures are computed with respect to the reference view to evaluate our results: PSNR (the higher the better) and DSSIM = 10 4 (1 -SSIM) (the lower the better). Although the increase of the PSNR score is noticeable but not conclusive, the strong improvement of the DSSIM demonstrates a higher structural similarity with the reference view, which accounts for the elimination of most visual artifacts.

CONCLUSION

We presented an image-based rendering method that renders a novel view from a generic and unstructured set of input views. This method is inspired by previous work by Pujades et al [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF], which proposed a formulation for most of the "desirable properties" that were listed in the seminal work by Buehler et al [START_REF] Buehler | Unstructured lumigraph rendering[END_REF], using a Bayesian formulation, and optimizing the target image in a variational framework. The only property that could not be formally derived was the continuity property, which states that the contribution of each input view to the pixels of the target image should be a continuous function of the pixel coordinates. We showed that an alternative approach for enforcing the continuity property is to state that edges, contours or textures should not be created in the target image if they are not present in the source images. This results in an additional data term, based on image gradients, which can be added to the energy functional. The energy can then be solve by iteratively solving a linear system devised from the energy functional. The results show an improvement over previous intensity-based unstructured IBR methods, both in terms of objective image quality measurements, and in terms of subjective quality.

Limitations: Despite the noticeable improvement of the image quality, our method does not fully eliminate all the visible artifacts. It could be reworked to optimize directly the target image gradients, rather than intensities, and the target intensity could then be reconstructed by solving the Poisson equation, as is done in Kopf et al [START_REF] Kopf | Image-based rendering in the gradient domain[END_REF]. This should totally remove any variations in the synthesized image that come from the discontinuity of the visibility functions, and are still visi-ble, although attenuated, in our results.

Future work: As shown in section 3, appending gradient constraints as a new energy data term is comparable to solving the Poisson equation. In the stitching literature [START_REF] Zomet | Seamless image stitching by minimizing false edges[END_REF], it is known as Laplacian blending. Considering that the Laplacian of an image behaves like a band-pass filter, blending the Laplacian of the input images is analogous to blending the images in the frequency domain for a specific band of frequencies that depends on the scale of the Laplacian. In our case the Laplacian is computed at the original scale of the image, level 0 of a Laplacian pyramid. Therefore we blend the images for the band of highest frequencies, and high-frequency artifacts due to naive intensity blending are prevented. Our future work will focus on an extension to all scales via the use of Laplacian pyramids, enabling a complete Laplacian blending to prevent not only high-frequency but also low-frequency artifacts.

Fig. 1 :

 1 Fig. 1: (a) We coarsely estimate the geometry of the scene to register input images. The warp functions τ k link the input images v k to the target view u. (b) The target view u is reconstructed by blending the source images in intensities. (c) Our method consists in appending constraints on the gradient of the solution; we show that it is equivalent to a Laplacian blending and that it removes high frequency blending artifacts.

Fig. 2 :

 2 Fig. 2: Warp and blending weight discontinuities cause artifacts (in red). Left: a close-up to a view rendered by minimizing the energy 7. Right: a warp that presents visibility discontinuities that caused the artifacts.

Proposed approach is for γ = 0. State-of-the-art approach [START_REF] Pujades | Bayesian view synthesis and image-based rendering principles[END_REF][START_REF] Wanner | Spatial and angular variational super-resolution of 4D light fields[END_REF] creates high-frequency artifacts due to blending intensities only. A higher smoothness parameter λ = 0.003 partially removes these artifacts at the cost of detail loss. Our approach preserves detail and show finer results by appending constraints of the gradients of the solution, forcing a Laplacian blending.


Saugat Bhattacharyya

Maureen Clerc

Mitsuhiro Hayashibe

A study on the effect of electrical stimulation as a user stimuli for motor imagery classification in Brain-Machine Interface

Keywords: Electrical Stimulation, Brain machine interfacing, Motor Imagery, Electroencephalography

Functional Electrical Stimulation (FES) provides a neuroprosthetic interface to non-recovered muscle groups by stimulating the affected region of the human body. FES in combination with Brain-machine interfacing (BMI) has a wide scope in rehabilitation because this system directly links the cerebral motor intention of the users with its corresponding peripheral muscle activations. In this paper, we examine the effect of FES on the electroencephalography (EEG) during motor imagery (left-and right-hand movement) training of the users. Results suggest a significant improvement in the classification accuracy when the subject was induced with FES stimuli as compared to the standard visual one.

The advent of Brain-Machine Interfacing (BMI) has opened a direct communication pathway between the brain and the environment in the form of robots, prosthesis, wheelchair or a computer without any muscle intervention. [START_REF]Toward Brain-computer Interfacing, ser. A Bradford book[END_REF] BMI has immense potential in facilitating rehabilitation for patients suffering from stroke, amyotrophic lateral sclerosis, spinal injury and physical disability1. A BMI system would drive a neuroprosthetic device to bridge the gap between the central nervous system and peripheric muscles and in some instance even a separate prosthesis. Electroencephalography (EEG) signals is the most commonly used BMI recording device and signals related to limb movement (known as motor imagery signals) are elicited as Event Related Desynchronization/Synchronization (ERD/ERS) response 2,3 . The BMI system decodes the ERD/ERS signals to determine the movement as intended by the user. Functional Electrical Stimulation (FES) [START_REF] Zhang | Evoked electromyography-based closedloop torque control in functional electrical stimulation[END_REF][START_REF] Popovic | Advances in functional electrical stimulation (FES)[END_REF] aims at directly stimulating the muscles on the affected side of the patient and it is reported that FES is capable of reconstructing certain daily life skills6. Till date, many researchers [START_REF] Yao | Using SSVEP based brain-computer interface to control functional electrical stimulation training system[END_REF][START_REF] Gollee | An SSVEP-based brain-computer interface for the control of functional electrical stimulation[END_REF][START_REF] Takahashi | Electroencephalogram (EEG) and functional electrical stimulation (FES) system for rehabilitation of stroke patients[END_REF] have combined BMI with FES to directly link the mental intention of the user with a muscular response. In such a system, the BMI detects the motor intentions of the user which in turn activates the FES device. FES activation may affect the sensory pathways, if intact, which may, in turn impact the cortical activity. Such phenomena may influence the performance of the BMI. Gollee et al. [START_REF] Gollee | An SSVEP-based brain-computer interface for the control of functional electrical stimulation[END_REF] found no major influence of FES on Steady-State Visually Evoked Potential (SSVEP) signal. Takahashi et al. [START_REF] Takahashi | Electroencephalogram (EEG) and functional electrical stimulation (FES) system for rehabilitation of stroke patients[END_REF] also found no effect of FES on leg stimulation. In this paper, we report the preliminary results of the effect of electrical stimulation during a motor imagery training task on healthy subjects and its comparison with visual stimuli. Through this study, we aim at employing FES as a proprioceptive feedback to the brain to improve the learning of the subject both in terms of accuracy and time. If successful, this system can accelerate the motor recovery process of the patient by enhancing the motor learning in the brain.

Materials and Methods

The EEG recording is conducted using a 14 channel Emotiv Epoc neuro-headset with a sampling rate of 128Hz and an in-built band-pass filter of 0.2-45Hz. The electrodes: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8 and AF4, are arranged on the basis of the standard 10-20 system10. Three healthy male subjects (2 right handed and one left handed), participated in this experiment. Two subjects participated in the BMI experiment as a first experience whereas one subject was an experienced BMI user. In this experiment, we abide by the norms of Helsinki Declaration of 1975, revised in 2000. Prior to the experiments, the subjects are informed about the purpose of the experiment and the tasks they have to perform. Surface electrical stimulation is transmitted to induce the hand extension tilting towards the back of the hand by applying the stimulation to the extensor digitorum muscle. It is applied to each side of the hand respectively, corresponding to right/left motor imagery task. The stimulus is delivered by a computer-controlled stimulator (ProStim, MXM, France) with PW modulation (PW max = 450 us) at a constant amplitude and frequency (30 Hz). Each stimulation sequence consists of a trapezoidal envelope train of PW (0.4 s ramp-up, 1.2 s plateau, 0.4 s ramp-down). The experiment designed for this work is divided into three sessions: only visual, only FES and both visual-FES stimuli. The sessions consist of instructing the subjects through a sequence of repetitive stimuli to execute the corresponding motor imagery task, which in our case, is left and right hand movement. One sequence of stimulus is known as trial. A trial for the visual stimulus is designed as follows: First, a blank screen is projected to the subject for 20s, which provides the baseline of the EEG. Then, a fixation '+' is displayed on the screen for 1s which is an indicator to the subject to get ready for the task. Next, the instructions are provided to the subject for 3 seconds in the form of arrows. According to the direction of the arrow, the subject imagines either left or right hand movement. Following the instructions, a blank screen is again displayed for 1.5s-3.5s. The FES session is similar to the visual one except in place of the arrows, stimulation is directly induced in the fore-arm of the hand of interest, without providing any visual information. In the Visual-FES session, both the combined stimulations are time-synchronized to each other. Each session is composed of 40 trials (20 for each movement). After acquisition, the incoming raw EEG signal is band-pass filtered at 8-30 Hz. Then, common spatial filters (CSP) is applied to extract features relevant to left-and right-motor hand movement EEG signals. CSP is a spatial filter widely used in BMI because the spatial patterns contain highly discriminative features between two classes. [START_REF] Lu | Regularized common spatial patterns with generic learning for EEG signal classification[END_REF] In this study, we prepare the feature vectors using 6 spatial filters which is then transferred as inputs to a linear discriminant analysis (LDA) classifier. [START_REF] Alpaydin | Introduction to Machine Learning (Adaptive Computation and Machine Learning[END_REF] Finally, the classifier detects the corresponding motor intention of the subject, i.e., left and right motor movement. A block diagram of our experimental setup during Visual-FES session is illustrated in Fig. 1.

Results

In this paper, we first attempt to observe the difference of the spatial patterns during the sessions: visual, FES and visual+FES. For this purpose, we determine the power spectral density (PSD)13 of each spatial filters. The PSD distribution for six spatial filters of subject 1 is shown in Fig. 2. As observed from the figure, the patterns are highly discriminable for the first four filters and they overlap in the last two. it is also noted that FES+Visual session (in black) has the highest power followed by Visual (in blue) session and lastly FES session (in red) for the first three filters. A similar trend is noted for the other two subjects too. Next, we employ 10 cross validation12 on the dataset, where at every iteration, one block of data is randomly selected for testing and the rest of the block are employed for training. Table I presents the average classification accuracy (over 10 runs) and it is noted that for Subject 01 and 02 (with first BMI experience) the accuracy during FES session increases by 6.28% and 5.51% as compared its visual counterpart. The difference between the FES and Visual+FES sessions are small. On the other hand, Subject 03 (with previous BMI experience) has a better result during the visual session than the FES session. Thus, it can be inferred from this result that subjects with no prior BMI training of motor imagery show a significant improvement in learning when induced with FES.

Discussion

In this paper, we aimed at studying the effect of electrical stimulation as a user stimuli for motor imagery classification (left and right hand movement) in BMI. Fig. 2 shows a significant difference of the spatial patterns during the three stimuli sessions: visual, FES and visual+FES. Results on Table I also shows a significant rise in accuracy for 2 (of 3) subjects which suggest a positive influence of FES during motor imagery training of the subjects. It was noted that both the subjects had no previous experience

Fig 1 .

 1 Fig 1. Block diagram of our experimental setup during motor imagery sessions where user stimuli is with conventional visual stimuli, electrical stimulation stimuli or the combined, respectively

Acknowledgement

The authors would like to thank the participants in this study. The paper is presented at IFESS Conference 2016.

-167 -on BMI, then they were not familiar with generating motor imagery with visual stimuli. Visual stimuli are the widely accepted form of motor training but the subject requires constant training to reach an optimal result. Based on the results of this study, we can infer that electrical stimulation can also be used for motor training and it can potentially provide better performance as it can make natural proprioceptive feedback related to motor performance than visual stimuli which requires user's recognition regarding the visual cue. Further studies on a larger group of subjects are required to validate this claim. Future studies in this research will include studying the effect of FES as proprioceptive neuro-feedback training to BMI. This will lead to an improvement in motor imagery classification which helps would aid in neuro-prosthetic or robot control [START_REF] Bhattacharyya | A synergetic brain-machine interfacing paradigm for multi-DOF robot control[END_REF] .

Contributions

Saugat Bhattacharyya: Experiment preparation, data acquisition and processing, publication writing Maureen Clerc: Experiment preparation, Data acquisition, supervision of data processing and publication drafting, publication review. Mitsuhiro Hayashibe: Experiment preparation, Data acquisition, supervision of data processing and publication drafting, publication review. -168 -

Conflict of Interest

The author declare no potential conflict of interests. France. E-mail: saugat.bhattacharyya@inria.fr E-mails of coAuthors Maureen Clerc: Maureen.clerc@inria.fr Mitsuhiro Hayashibe: Mitsuhiro;hayashibe@inria.fr

Corresponding Author


Anders Thorin

Pierre Delezoide

Mathias Legrand

NONSMOOTH MODAL ANALYSIS OF PIECEWISE-LINEAR IMPACT SYSTEMS

Periodic solutions of autonomous and conservative second-order dynamical systems of finite dimension n undergoing a single unilateral contact condition are investigated in continuous time. The unilateral constraint is complemented with a purely elastic impact law conserving total energy. The dynamics is linear away from impacts. It is proven that the phase-space is primarily populated by onedimensional continua of periodic solutions, generating an invariant manifold which can be understood as a nonsmooth mode of vibration in the context of vibration analysis. Additionally, it is shown that nonsmooth modes of vibration can be calculated by solving only k 1 equations where k is the number of impacts per period. Results are illustrated on a mass-spring chain whose last mass undergoes a contact condition with an obstacle.

MOTIVATION

Nonlinear modal analysis consists in calculating, in the phase space, two-dimensional manifolds which are continuous families of periodic trajectories [START_REF] Shaw | Non-linear normal modes and invariant manifolds[END_REF]. When dealing with nonsmooth systems, i.e. systems subjected to impact and friction [START_REF] Acary | Numerical methods for nonsmooth dynamical systems: applications in mechanics and electronics[END_REF], the induced discontinuities are usually regularized [START_REF] Chen | Normal modes for piecewise linear vibratory systems[END_REF]. The novelty of the proposed work here is to incorporate the discontinuities in the formulation to exhibit continuous families of periodic nonsmooth trajectories, thus leading to nonsmooth modal analysis. The results hold for all autonomous linear dynamical systems subjected to a single contact condition modeled through a unilateral linear sceleronomic constraint complemented with a perfectly elastic impact law.

FORMULATION

The vector of generalized coordinates x is an element of R n where n is the number of dofs of the system. During free flights, that is away from impacts, the dynamics is governed by MR x C Kx D 0 where M and K are the mass and stiffness matrices. Introducing x D OEx > ; P

x > > , this differential equation can be reformulated in the first-order form as P x D Ax hence if no impact occurs in time interval OEt; t 0 , x.t / D S.t t 0 /x.t 0 / with S.t / D exp.t A/. The signed contact distance (gap function) is g.x/ D w > x C g 0 . It is shown that the impact law, written in terms of normal contact velocities C D , reads in terms of as x C D Nx where N D M 1=2 .I 2rr > /M 1=2 and r is the unit vector given by r > D OE0 1;n ; M 1=2 w.w > M 1 w/ 1=2 > 2 R 2n . We introduce the times of impact t 0 ; : : : ; t k with the convention t 0 D 0 and period T D t k so that k is the number of impacts per period (ipp). The dynamics is a succession of free flights of duration t i t i 1 and impacts at t i , i 2 1; k , so for t 2 OEt i 1 ; t i /:

x.t / D S.t t i /NS.t i t i 1 /N : : :

NS.t t 0 /x.t 0 /: (1)
This expression shows that there exists a linear mapping from a vector of initial conditions to the current state at t . Let u be the endomorphism of matrix NS.t k t k 1 /N : : : NS.t 1 t 0 / in the canonical basis of R 2n . The problem of finding periodic solutions of the dynamics with impact times t 1 ; : : : ; t k is formulated in three necessary conditions (NC) to be satisfied by x 0 : Given T 2 R and k 2 N , find x 0 2 R 2n and t 1 ; : : : ; t k such that:

8 < : u.x 0 / D x 0 NC1 x determined by x 0 using (1) is such that: 8i 2 1; k ; g.x.t i // D 0 NC2 8t 2 OE0; T ; g.x.t // 0 NC3 (2a) (2b) (2c)
with g.x/ WD g.x/. Eq. (2a) enforces periodicity through a sequence of free flights over .t i 1 ; t i / punctuated by perfectly elastic impacts at t i , i 2 1; k . Eq. (2b) guarantees that the impact law only applies when the separating gap is closed. Ineq. (2c) is necessary to ensure that the unilateral contact condition is not violated during the free flights.

MATHEMATICAL RESULTS

We can show that there is an isomorphism ' between the invariant (maximal) subspace of u and the kernel of a known skew-symmetric matrix … of size k k and function of the impact times. This implies that NC1 is satisfied if and only if there exists a 2 R k such that ….t 1 ; : : : ; t k / D 0, and then x 0 can be retrieved using x 0 D '. /. We also know that NC2 is satisfied by x if and only if †.t 1 ; : : : ; t k /'.x 0 / D g 0 j where † is a symmetric k k matrix and j D OE1; : : : ; 1 > 2 R k .

Corresponding author. Email: anders.thorin@mcgill.ca

The expressions of …, † and ' are known explicitly but are not provided here; they depend only on M, K, g and t 1 ; : : : ; t k . Satisfying NC1 and NC2 reduces to finding t 1 ; : : : ; t k and such that … D 0 and † D g 0 j. Then, † is generically invertible so it suffices to find t 1 ; : : : ; t k such that … † 1 j D 0, and then recover x 0 D '. g 0 † 1 j/. The non-generic cases exhibit very specific properties which are not described here. If x 0 satisfies NC1 and NC2, it is a solution iff it satisfies NC3. This last condition has to be tested numerically.

It is then proved that if x 0 is a solution for g 0 ¤ 0 then, generically, it lies on a two-dimensional manifold of the phase space. As opposed to smooth nonlinear modes, here the manifolds feature discontinuities, corresponding to the discontinuities at impact times. The positions and velocities as a function of time emanating from the initial conditions x 0 can then be calculated. The results are summarized in Fig. 1.

s S t 1 t 2 t k s j ….s/ † 1 .s/j D 0 s belongs to a curve S D g 0 † 1 .s/j
x 0

x 0 D '. /

x. /

x n 1

x n P x n Nonsmooth periodic orbit Figure 1: Summary of the mathematical results in the generic case with g 0 ¤ 0. When s D .t 1 ; : : ; t k / travels along the red curve S, the periodic solution describes a nonsmooth mode with k impacts per period.

APPLICATION TO A MECHANICAL SYSTEM

The mathematical developments prove the existence of nonsmooth modes yet also give a constructive way of calculating them by solving k 1 equations (… † 1 j D 0) independently of the number of dofs. Results are illustrated on a simple spring-mass chain whose free mass is subjected to an elastic Newton impact law, see Fig. 2. A nonsmooth mode with 7-ipp was calculated following the methodology of Fig. 1 for this system with n D 5. The position of the last mass is represented for two initial conditions on the nonsmooth mode in Fig. 3, illustrating the possibilities given by the derivations.

m 1 k 1 m 2 k 2 m n 1 k n 1 m n k n

Figure 2 :

 2 Figure 2: Simple chain whose free mass is subjected to impacts.

Figure 3 :

 3 Figure 3: Two periodic trajectories with 7 ipp belonging to the same nonsmooth mode represented in . Px n ; t /. Dashed lines correspond to impact times.