The use of citizen science for marine biodiversity surveys : from species identification to ecologically relevant observations

T. Changeux, Christophe Blazy, Sandrine Ruitton

To cite this version:

T. Changeux, Christophe Blazy, Sandrine Ruitton. The use of citizen science for marine biodiversity surveys : from species identification to ecologically relevant observations. Hydrobiologia, In press, 10.1007/s10750-019-04070-7. hal-02304651


Dr Nicolas Louveton
email: nicolas.louveton@uca.fr

Mr Korok Sengupta
email: koroksengupta@uni-koblenz.de

Dr Rod Mccall
email: mccall@list.lu

Dr Raphael Frank
email: raphael.frank@uni.lu

Thomas Engel
email: thomas.engel@uni.lu

Testing a Commercial BCI Device for In-Vehicle Interfaces Evaluation: A Simulator and Real-World Driving Study

Keywords: Driving Simulator, Real Driving, Distraction, Dual-Task, Mental Workload, BCI

This study is assessing the sensitivity of an affordable BCI device in the context of driver distraction in both low-fidelity simulator and real-world driving environments. Twenty-three participants performed a car following task while using a smartphone application involving a range of generic smartphone widgets. On the first hand, the results demonstrated that secondary task completion time is a fairly robust metric as it is sensitive to user-interfaces style while being consistent between the two driving environments. On the second hand, while the BCI attention level metric was not sensitive to the different user-interfaces, we found it to be significantly higher in the real-driving environment than in the simulated one, which reproduces findings obtained with medical-grade sensors.

INTRODUCTION

Multitasking is a commonly observed behavior in everyday life [START_REF] Salvucci | The multitasking mind[END_REF]. In some circumstances, such as driving, executing concurrent tasks (such as interacting with displays), may impair driving safety [START_REF] Cooper | Turning gap acceptance decisionmaking: the impact of driver distraction[END_REF][START_REF] Horrey | Cell Phones and Driving Performance: A Meta Analysis[END_REF][START_REF] Rudin-Brown | Driver distraction in an unusual environment: Effects of text-messaging in tunnels[END_REF]Törnros & Bolling, 2005). Although the use of digital media in cars such as connected apps, navigation systems or music players can be beneficial, they raise issues concerning the design and evaluation of such innovative services.

One major challenge in the domain of in-vehicle infotainment systems concerns evaluation methods [START_REF] Green | Driver Distraction, Telematics Design, and Workload Managers: Safety Issues and Solutions[END_REF]. While a thorough evaluation is required for near-market innovations, early Human-Computer Interaction (HCI) studies need more agile means of evaluating new concepts. In these situations a low-fidelity simulator might be suitable [START_REF] Jamson | The validity of a low-cost simulator for the assessment of the effects of in-vehicle information systems[END_REF]. While driving simulator measurements can demonstrate adverse effect of a secondary task on driving performance, they will provide little insight into covert attentional phenomenon. For instance, estimating driver's covert attentional phenomenon may require very specific and expensive equipment [START_REF] Girouard | From brain signals to adaptive interfaces: using fnirs in hci[END_REF][START_REF] Mehler | Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task an on-road study across three age groups[END_REF][START_REF] Pomplun | Pupil dilation as an indicator of cognitive workload in humancomputer interaction[END_REF].

In this context, commercial Brain-Computer Interfaces (BCI) are particularly relevant (i.e., as opposed to medical-grade sensors). Indeed, they could allow for an affordable and easy-to-use way to assess driver attention while multitasking. However, there is a lack of knowledge concerning the potential added value of such devices in human factor research. This work aims to address the gap in knowledge concerning the use of a commercial BCI device as a reliable measurement tool for user mental workload, notably in different driving environments (i.e., driving simulator and real-driving testing).

For this reason, we tested the sensitivity of a commercial and affordable BCI (MindCap XL1 ) relative to two experimental factors: (i) the driving environment that could be either a low-fidelity simulator or a real-driving environment; and (ii) a range of standard Android user-interface widgets for a smartphone-based secondary task. We expect those experimental factors to have an impact respectively on (i) the mental workload and (ii) the visual-manual distraction. Participants performed the same car following task in both driving environments while they were interacting with the smartphone application. Application usage, driving speed and BCI metric (so-called attention level) were collected and analysed across the different conditions.

RELATED WORK

Comparison of simulated and real driving

The usage of driving simulators raises the question of transferring the results from simulated (whether of a low or a high quality) to real environments. Several studies found differences between those conditions: Indeed, [START_REF] Reymond | Role of lateral acceleration in curve driving: Driver model and experiments on a real vehicle and a driving simulator[END_REF] found that in driving simulator experiments curvilinear speed was underestimated when taking a curve. It has also been demonstrated [START_REF] Boer | Experiencing the same road twice: A driver-centred comparison between simulation and reality[END_REF] that participants braked later and stronger in driving simulator than in a real-driving environment. However, [START_REF] Panerai | Speed and safety distance control in truck driving: comparison of simulation and real-world environment[END_REF] showed that speed control did not vary significantly between the two types of environments. Finally, [START_REF] Engström | Effects of visual and cognitive load in real and simulated motorway driving[END_REF] found the estimated workload higher in real-driving condition than in simulated one.

Comparing low-and high-fidelity simulators

Driving simulators exist in a wide range of complexity and fidelity with regard to real-life driving (motion simulation, 3D engine, cockpit etc.) The fidelity of a driving simulator has been shown to have an impact on the way participants react to the virtual traffic events. Indeed, low-cost simulators decrease accuracy in the perception of ego-motion, speeds and distances, which in turn leads to under-estimated inter-vehicular judgements [START_REF] Kemeny | Evaluating perception in driving simulation experiments[END_REF]. The same authors also pointed out that while high-fidelity simulators are required for assessing complex driving situations, low-cost simulators could be used successfully for dashboard ergonomic and simple driving scenarios. Other authors also pointed to low-cost simulator being particularly useful in early prototyping stages of innovative infotainment services [START_REF] Green | Driver Distraction, Telematics Design, and Workload Managers: Safety Issues and Solutions[END_REF]. For instance, in [START_REF] Jamson | The validity of a low-cost simulator for the assessment of the effects of in-vehicle information systems[END_REF] authors found consistent measurements across simulator types at least for metrics concerning speed control and secondary task completion time.

Mobile devices and visual-manual distraction

Studies confirm that the increased use of mobile phones while driving degrades driving performance [START_REF] Cooper | Turning gap acceptance decisionmaking: the impact of driver distraction[END_REF][START_REF] Horrey | Cell Phones and Driving Performance: A Meta Analysis[END_REF][START_REF] Rudin-Brown | Driver distraction in an unusual environment: Effects of text-messaging in tunnels[END_REF]Törnros & Bolling, 2005). The reason being when one shifts their visual attention to a mobile phone, this leads to visual-manual distraction. Visual-manual distraction refers to any secondary activity that involves controlling hand gestures toward a visual interface. Engaging in such activity will lead to longer and more frequent glances off-the-road [START_REF] Burns | The importance of task duration and related measures in assessing the distraction potential of in-vehicle tasks[END_REF]. With a high penetration on mainstream market, touch-screen interactions such as those used on current smartphones are both familiar and easy-to-use due to the imprecise interactions required in finger pointing activity. However many studies showed that the type of widget used for a smartphone application impact differently driver's distraction [START_REF] Kim | Evaluation of the safety and usability of touch gestures in operating in-vehicle information systems with visual occlusion[END_REF][START_REF] Louveton | Driving while using a smartphone-based mobility application: Evaluating the impact of three multi-choice user interfaces on visual-manual distraction[END_REF]. Additionally, it has been shown that text-entry and kinetic scrolling are the two major sources of visual-manual distraction in the car [START_REF] Kujala | Visual-manual in-car tasks decomposed: text entry and kinetic scrolling as the main sources of visual distraction[END_REF].

Estimation of mental workload and BCI devices

Mental workload could be estimated by a variety of psycho-physiological measurements such as heart-rate, skin conductance or pupil dilation [START_REF] Collet | Physiological and behavioural changes associated to the management of secondary tasks while driving[END_REF][START_REF] Healey | Detecting stress during real-world driving tasks using physiological sensors[END_REF][START_REF] Mehler | Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task an on-road study across three age groups[END_REF][START_REF] Pomplun | Pupil dilation as an indicator of cognitive workload in humancomputer interaction[END_REF]2003;[START_REF] Solovey | Classifying driver workload using physiological and driving performance data: Two field studies[END_REF]. Another possibility is to use brain activity as input for estimation [START_REF] Fort | Attentional demand and processing of relevant visual information during simulated driving: A MEG study[END_REF][START_REF] Haufe | Electrophysiology-based detection of emergency braking intention in real-world driving[END_REF][START_REF] Kincses | Measuring drivers mental workload using eeg[END_REF][START_REF] Lei | Influence of task combination on EEG spectrum modulation for driver workload estimation[END_REF][START_REF] Liang | Monitoring driver's alertness based on the driving performance estimation and the eeg power spectrum analysis[END_REF]. However, those measurements can be expensive and difficult to setup or to analyze. With the evolution of several commercial and affordable BCI devices, understanding the signals of the brain on the move has become much easier, faster and cost effective. In this respect, several studies demonstrated successful use of simple BCI devices in the context of interactive applications and workload estimation [START_REF] Afergan | Dynamic difficulty using brain metrics of workload[END_REF][START_REF] Girouard | From brain signals to adaptive interfaces: using fnirs in hci[END_REF][START_REF] Herff | Mental workload during n-back taskquantified in the prefrontal cortex using fnirs[END_REF].

METHODOLOGY Participants

In total 23 participants took part in this study, including 15 for the simulator set-up and 8 for the real driving condition. The driving simulator sample was composed of 12 males and three females with a mean age of 28 years (sd = 4.08) and they had held their driving license for an average of 8.91 years (sd = 4.7). The real driving set-up was composed of seven males and one female with a mean age of 29 years (sd = 5.18) and they had held their driving license for an average of 7.13 years (sd = 2.8).

The population has been drawn from University staff members and students. Each of the drivers participated in the event had a valid European Union driving license (for at least four years) and a normal or corrected-to-normal vision. All participants agreed and signed an informed consent form before taking part.

Car following task

Participants were to perform a car following task on a test track (see the schema of the track in Figure 1, located in Colmar-Berg, Luxembourg2 ), either in the real-world test-track or in its 3D driving simulator version. The track was a closed course with no other traffic vehicles than those of the experiment. The task was the same for both environment. All the participants were instructed on the task they needed to perform prior to the experiment starting. Traffic was limited to one lead (i.e., preceding) vehicle going in the same direction driving at a constant speed. The participants were asked to follow the car in front of them at all times and never to overtake it and to maintain a reasonable gap and speed. The initial starting distance between the two cars was mentioned to each of the participants to be 30 meters. They were requested to keep a safe speed of 50 km/h (the lead car was driving within a range of 50 to 60 km/h). While driving behind the lead vehicle, they had to use the mobile phone attached inside the cockpit and interact with it depending on the activity that popped-up on the screen.

Driving environments set-up

For the simulator environment, we used OpenDS (version 2.5)3 as 3D engine and the DriveLab platform [START_REF] Louveton | Assessing in-vehicle information systems application in the car: a versatile tool and unified testing platform[END_REF][START_REF] Avanesov | Towards a Simple City Driving Simulator Based on Speed Dreams and OSM[END_REF] for triggering events and synchronising data. The test track used in the virtual environment was developed to mimic the geometry of the real test track, that has been used for the real driving condition.

For achieving this we have followed the procedure described in [START_REF] Avanesov | Towards a Simple City Driving Simulator Based on Speed Dreams and OSM[END_REF]: the real track geometry has been extracted from OpenStreetMap using OSM2World4 and Blender (version 2.49b)5 in order to make it a 3D model for OpenDS. We used a low-cost simulation setup: the display was handled by a video-projector, and controls by a Logitech gaming set including a steering wheel and pedals. The simulated car had an automatic transmission.

For the real-world environment, participants were to drive a Renault Twizy (electric quad-cycle, no gear change). Telemetry was accessed through a additional smartphone application making use of the On Board Diagnostic (OBD) port of the Renault Twizy and with the help of an OBD2 Bluetooth device.

Secondary task

The secondary task was displayed on a smartphone located on the right side of the driver (i.e., steering wheel assumed to be on the left-side of the cockpit). The smartphone used was a Galaxy S III mini running Android 4.1 with a 4 inches display size. The secondary task is implemented using generic Android widgets as they represent a realistic source of visual-manual distraction while driving.

The task displayed by the smartphone was a simple mental calculus challenge (i.e., of the type 5×2+3=?), then the user had to input the correct answer from a list of alternatives. This task was presented using five different types of interfaces: (i) Touch Button, (ii) Circular Dial, (iii) Input Data, (iv) Drop Down Menu, and (v) Radio Button (see Figure 2). Each trial was preceded by a visual and auditory notification then the secondary task was presented to the participant.

Brain-Computer Interface

The BCI device used to measure the brain signals was a MindCap XL headband equipped with a NeuroSky sensor6 . This device measures brain activity from sensors placed on the forehead and the proprietary algorithm automatically outputs the so-called attention level metric. Because the sensors are located on the forehead of the user, the attention level metric is supposed to be associated to focused attention and mental workload [START_REF] Norman | Working memory and mental workload[END_REF].

Experimental procedure

Participants were asked to drive on the test track for seven laps (lasting approximately 20-25 minutes). Prior to setting off the participants were instructed about the driving tasks. Each participant was given a chance to familiarise her/himself with the track by driving around it prior to starting the study, no data was recorded during the familiarisation phase.

The participants were instructed to keep an eye on the mobile phone attached to the cockpit and interact with it while continuing to drive. The secondary task and each of the interface options were explained to them. At the beginning of the experiment, the BCI device was attached to the forehead of the participant before initialising the smartphone application and the driving simulator environment.

The secondary task application was triggered on three fixed points located on the test track (cf. Figure 1). Thus, each participant had to perform 21 trials (three triggers on seven laps). The three points were located on a straight stretch of the track. Geo-fencing has been used with GPS coordinates in order to trigger trials in the real-world experiment while those coordinate have been translated in in the simulated environment. Participants had until the prompt for the next task to answer the current one.

Figure 3. Schema of the experimental design under the driving simulator perspective (left) and real-driving one with the Twizy (right). In the first case driving simulator position is triggering

secondary task on the smartphone while in the second case GPS coordinates are used. In all conditions the BCI headband is placed on the forehead of the participant.

Experimental design and data analysis

We used a mixed factorial design with Environment (simulator or real driving) as a betweensubject factor and user-interface styles, so-called UI as a within-subjects one. For each secondary task trial, the type of user-interface and the question/answer pair were selected at random.

For both simulated and real driving environments, the current speed of the car was collected. The secondary task usage was measured in terms of task completion time and success rate. Finally, we collected from the BCI device a metric called attention level. This metric is computed by the BCI using real-time measurements and a proprietary algorithm. The attention level metric was output every one second and was ranging from 0 to 100. All the different types of data were synchronised and averaged across experimental conditions. We did not include a baseline condition in statistical analysis: Indeed, because of the test-track characteristics it seemed arbitrary to compare drivingonly data samples with dual-task ones.

Parametric tests were used whenever the validity conditions were met, otherwise, non-parametric tests were used. Post-hoc tests were performed using pair-wise two-sample tests with a Bonferronni correction.

RESULTS

Success and completion time

Overall, results show that participants were successful in achieving the secondary task, both in simulated (86%) and real environment (90%). The highest success rate was achieved with the RadioButton (99%) and DropDown Menu (97%), followed by the Slider (87%), Text Insertion (80%) and Button (76%). Those results indicate that participants performed reasonably well with all the user-interfaces proposed.

We performed a two-way mixed-design ANOVA on the completion time measurement. This analysis did not reveal an effect of the Environment factor (p=.17) or of the Environment×UI interaction (p=.86). However, the analysis revealed an effect resulting from the UI factor (F(4,91)=30; p<.001). On average the duration required for completing the tasks was higher for the real driving environment (16.3, sd = 17.9) than in for the simulated one (12.4, sd = 12.9). The most important variations were due to the type of interface (see also Figure 3): the post-hoc analysis revealed significant differences (ps<.05) when comparing Button condition (8.2, sd=7.6) to DropDown (14.1, sd=14.6), InsertData (17.1,sd=18.6),and Slider (17.8,sd=14.3) conditions. Finally, we found that Button and RadioButton (11.8, sd=15.7) conditions did not differ significantly and were the interfaces which allowed for the fastest completion time.

Driving speed

The analysis evidenced an effect of the Environment factor (F(1,15)=626.16;p<.001) and of the UI one (F(4,89)=2.97; p<.05). We did not find an effect of Environment×UI interaction (p=.17). While interacting with the smartphone, participants clearly drove at slower speed when immersed in a real driving environment (26km/h, sd=5.8) compared to when they were in a simulated one (54.6km/h, sd=9.9). The interface conditions also impacted the driving speed (cf. Figure 4): speed was the highest in the Button condition (48.3km/h, sd=17.5) followed by RadioButton (46.8km/h, sd=14.9), Slider (45km/h, sd=15.9), InsertData (43.9km/h, sd=15.3), and DropDown (42.8km/h, sd=13.9) conditions. However, the post-hoc analysis with corrected p-values failed to find significant differences between these conditions.

Attention metric

The analysis revealed an effect of the Environment factor (F(1,15)=28.7; p<.001) and an effect of the Environment×UI interaction (F(4,91)=2.6; p<.05). We found no effect of the UI factor (p=.23).

Figure 6. Attention metric (represented with standard deviation) was much higher in the real environment condition than in the simulated one. The variability of this metric was also higher between the different user-interfaces in the real compared to the simulated environment.

Results showed (see also Figure 5) that attention metric level was higher in the real driving environment (47, sd=17.7) than in the simulated one (34, sd=12). Also, the attention metric varied more across interface styles in the real environment (ranging from 42.6 to 51.6) than in the simulated one (ranging from 33.9 to 34.9).

Using a post-hoc analysis, more specific differences between user-interface styles have been found when comparing real and simulated driving environments. For instance, the DropDown interface in the real environment was associated (p<.001) with higher attention level (52.9, sd=15.9) than Button (35.3,sd=10.2), DropDown (35.5, sd=8.7), InsertData (35.5, sd=8.9), RadioButton (33.9, sd=8.7) and Slider (33.2, sd=8.9) interfaces in the simulated one.

We also found (p<.05) that the RadioButton interface in the real driving environment was associated with higher attention level (55.5, sd=16) than the DropDown (35.5, sd=8.7), InsertData (35.5, sd=8.9) and RadioButton (33.9, sd=8.7) interfaces in the simulated one.

DISCUSSION

While commercial BCI devices could be useful for HCI research, little knowledge has been found in this context. In this work we assessed how sensitive and reliable a commercial, affordable and easy-to-use BCI device is when assessing driver mental workload. Using such a BCI we assessed the impact on driver distraction different interface styles for a smartphone application in both a low-fidelity and real-driving environments.

Our results point to estimated workload being significantly higher in the real-driving environment than in the simulator one. These results actually confirm former findings (Engström, Johansson, & Östlund, 2005), although we used a much simpler physiological estimation of workload. Indeed, these authors used skin conductance and electrocardiogram which are highly specialised measurement devices. Because the BCI device we used implements a proprietary algorithm it is difficult to know to which cognitive process the so-called attention level metric actually refers to. However, taken together with Engström et al. (2005) our results are compatible with a correlation between the attention level metric obtained from the MindCapXL and the increase of mental workload induced by the real-driving environment.

As expected by [START_REF] Jamson | The validity of a low-cost simulator for the assessment of the effects of in-vehicle information systems[END_REF] completion time of the secondary task was consistent across driving environment while it was sensitive to the different user-interface styles. Those results suggest that this metric is a good indicator of secondary task difficulty independently of the environment. Contrary to [START_REF] Panerai | Speed and safety distance control in truck driving: comparison of simulation and real-world environment[END_REF], we did not find speed control metric to be stable across the two environments: Instead, driving speed was significantly lower in the real-driving condition.

Finally, completion time indicated that the two worst interfaces were the text-entry and slider widget which is congruent with earlier findings [START_REF] Kujala | Visual-manual in-car tasks decomposed: text entry and kinetic scrolling as the main sources of visual distraction[END_REF][START_REF] Louveton | Driving while using a smartphone-based mobility application: Evaluating the impact of three multi-choice user interfaces on visual-manual distraction[END_REF]. Speed control and attention level were not sensitive to the different widgets.

Although, the attention level metric was shown as more variable in the real-driving condition indicating a possible interaction between the two factors. As said above, the BCI metric used is difficult to match with a specific cognitive process. One possible explanation is that the secondary task proposed was inducing visual-manual distraction more than cognitive distraction. Considering this and the location of the BCI sensors we can assume that the BCI device we used would have been more sensitive with cognitively more demanding tasks.

CONCLUSION

In this work, we assessed the sensitiveness of a commercial BCI device as an easy and affordable tool for estimating driver's mental workload. We used two driving environments (simulated and real) and a range of mainstream smartphone widgets as a test-bed for our measurements. We conclude that a commercial BCI device could be useful when assessing mental workload associated with large variations of task difficulty in terms of quantity of information to be processed. Other methods should be preferred in order to analyse specific distraction sources (e.g., visual-manual distraction, conversation etc.)

Figure 1 .

 1 Figure 1. Schema of the test track used in both driving environment conditions. The three triggers for secondary events are indicated (yellow crosses) as well as the path of circulation (arrows).

Figure 2 .

 2 Figure 2. The five graphical user-interfaces used in the study. We used generic Android widgets as they represent realistic sources of distraction.

Figure 4 .

 4 Figure 4. Completion time (represented with standard deviation) was relatively stable across environment conditions, although it varied noticeably for the different user-interfaces.

Figure 5 .

 5 Figure 5. When dual-tasking, participants drove at a much slower speed in the real environment than in the simulated one. Practically no differences are found between the different userinterfaces. Speed is represented with standard deviation.

http://www.mindtecstore.com/en/mindcap-xl

https://www.openstreetmap.org/#map=16/49.7955/6.1050

http://opends.de/

http://osm2world.org/

http://www.blender.org/

http://neurosky.com/

ACKNOWLEDMENTS

The i-Gear project was funded by the Fond National de la Recherche (FNR) , Luxembourg (Grant number: 11/IS/1204159). We also thank other colleagues and students from the i-Gear team.


Michaël Sdika
email: michael.sdika@creatis.insa-lyon.fr

Laure Alston

David Rousseau

Jacques Guyotat

Laurent Mahieu-Williame

Bruno Montcel

Repetitive Motion Compensation for Real Time Intraoperative Video Processing

Keywords: motion compensation, image registration, subspace learning, brain surgery, real time video processing, extended direct linear transform

In this paper, we present a motion compensation algorithm dedicated to video processing during neurosurgery. After craniotomy, the brain surface undergoes a repetitive motion due to the cardiac pulsation. This motion as well as potential video camera motion prevent accurate video analysis. We propose a dedicated motion model where the brain deformation is described using a linear basis learned from a few initial frames of the video. As opposed to other works using linear basis for the flow, the camera motion is explicitly accounted in the transformation model. Despite the nonlinear nature of our model, all the motion parameters are robustly estimated all at once, using only one singular value decomposition (SVD), making our procedure computationally efficient. A Lagrangian specification of the flow field ensures the stability of the method. Experiments on in vivo data are presented to evaluate the capacity of the method to cope with occlusion or camera motion. The method we propose satisfies the intraoperative constraints: it is robust to surgical tools occlusions, it works in real time, and it is able to handle large camera viewpoint changes.

Introduction

Optical imaging is a modality of choice for surgery assistance: it is inexpensive, it has high temporal and spatial resolution and, as opposed to magnetic resonance or computational tomography, it implies only limited constraints on the surgical material and room. This is why even simple devices such as true color camera are often used for assistance or monitoring during surgical intervention or radiotherapy for a variety of body parts: brain [START_REF] Pichette | Intraoperative video-rate hemodynamic response assessment in human cortex using snapshot hyperspectral optical imaging[END_REF], heart [START_REF] Richa | Towards robust 3d visual tracking for motion compensation in beating heart surgery[END_REF] or abdomen [START_REF] Spinczyk | Methods for abdominal respiratory motion tracking[END_REF] for example. Analysis of videos from these camera can be used for brain surface strain estimation [START_REF] Ji | Cortical surface strain estimation using stereovision[END_REF], brain surface 3D reconstruction [START_REF] Marreiros | Superficial vessel reconstruction with a multiview camera system[END_REF][START_REF] Paul | A surface registration method for quantification of intraoperative brain deformations in image-guided neurosurgery[END_REF], to investigate the deformation of the exposed cortical surface [START_REF] Ji | Tracking cortical surface deformation using stereovision[END_REF], to measure hemodynamic response [START_REF] Pichette | Intraoperative video-rate hemodynamic response assessment in human cortex using snapshot hyperspectral optical imaging[END_REF][START_REF] Villringer | Non-invasive optical spectroscopy and imaging of human brain function[END_REF][START_REF] Steimers | Imaging of cortical haemoglobin concentration with RGB reflectometry[END_REF][START_REF] Sobottka | Intraoperative optical imaging of intrinsic signals: a reliable method for visualizing stimulated functional brain areas during surgery: clinical article[END_REF][START_REF] Oelschlägel | Intraoperative identification of somato-sensory brain areas using optical imaging and standard RGB camera equipment -a feasibility study[END_REF]. In all these works, motion compensation is used during some stage of the processing workflow with general purpose routines such as Horn and Schunck [START_REF] Horn | Determining optical flow[END_REF] or Farnebäck [START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF] optical flow routines, demons registration [START_REF] Thirion | Image matching as a diffusion process: an analogy with maxwell's demons[END_REF] or the Matlab 1 Medical Image Registration Toolbox. It is indeed mandatory to have very stable image in the surgical microscope when pixel-wise postprocessing is needed. Even if the microscope is rather stable, this is not the case of the patient brain. Heart and respiratory pulsativities make the brain surface move. Furthermore awake surgery, used for example to investigate languages brain areas, implies strong instability of the patient head and larger motion should be accounted for. The motion compensation can be improved by exploiting the specificity of the data and, as opposed to standard nonrigid registration methods, it should also be able to account for occlusions (of surgical tools for example). In this work, a motion compensation method is proposed that satisfy three important properties required to work in surgical conditions: the method is real time, it is robust to occlusions and the object motion is disentangled from the camera motion.

State of the Art

Literature on image registration is vast, allowing, for instance, multimodality registration with statistical cost function [START_REF] Maes | Multimodality image registration by maximization of mutual in-formation[END_REF] or to guarantee that the deformation is invertible [START_REF] Miller | On the metrics and eulerlagrange equations of computational anatomy[END_REF][START_REF] Beg | Computing large deformation metric mappings via geodesic flows of diffeomorphisms[END_REF][START_REF] Sdika | A fast nonrigid image registration with constraints on the Jacobian using large scale constrained optimization[END_REF][START_REF] Sdika | A sharp sufficient condition for b-spline vector field invertibility. application to diffeomorphic registration and interslice interpolation[END_REF]. When it comes to motion estimation on video camera data, the high temporal resolution and the contrast invariance assumption simplify the problem: Methods used for this problem are mostly based on the optical flow equation (see [START_REF] Fortun | Optical flow modeling and computation: A survey[END_REF] a recent review).

In the problem of brain motion estimation during neurosurgery the same object is always present and undergoes a repetitive (but non periodic) nonrigid motion. By repetitive, we mean that the same deformation occurs repeatedly in the time, which is the case for the brain when the skull is opened. We do not make the stronger assumption of the motion periodicity. This allows accounting for temporal changes in the deformation caused for example by cardiac or respiratory rhythm changes. As the brain surface motion is similar all along the video timecourse, it should be possible to express the motion with a low degree of freedom model. This led us to consider subspace learning to estimate a low dimensional space to describe this motion. Subspace learning has already been used in several works [START_REF] Black | Learning parameterized models of image motion[END_REF][START_REF] Fleet | Design and use of linear models for image motion analysis[END_REF][START_REF] Irani | Multi-frame correspondence estimation using subspace constraints[END_REF][START_REF] Roberts | Learning general optical flow subspaces for egomotion estimation and detection of motion anomalies[END_REF][START_REF] Ricco | Dense lagrangian motion estimation with occlusions[END_REF][START_REF] Wulff | Efficient sparse-to-dense optical flow estimation using a learned basis and layers[END_REF][START_REF] Sdika | Robust real time motion compensation for intraoperative video processing during neurosurgery[END_REF], to find a good representation of the deformable motion. It is assumed that the motion in each frame belong to the same affine space. The basis of this space is found using a principal component analysis (PCA) on the motion estimated on similar video. The dimension of the motion is reduced from twice the number of pixels to the number of principal components kept (up to 500 in [START_REF] Wulff | Efficient sparse-to-dense optical flow estimation using a learned basis and layers[END_REF]). This enables the reconstruction of the full motion from a reduced number of sparse keypoints. To be robust to occlusion or tracking failures, M-estimation [START_REF] Huber | Robust statistics[END_REF][START_REF] Ricco | Dense lagrangian motion estimation with occlusions[END_REF], expectation maximization [START_REF] Roberts | Learning general optical flow subspaces for egomotion estimation and detection of motion anomalies[END_REF] or iterated reweighted least square (IRLS) [START_REF] Wulff | Efficient sparse-to-dense optical flow estimation using a learned basis and layers[END_REF] can be used.

In a previous work on the same application [START_REF] Sdika | Robust real time motion compensation for intraoperative video processing during neurosurgery[END_REF], the motion is modeled using the sum of an affine transform (for the camera motion) and a deformable transformation (for the object deformation) which lies in an affine subspace estimated with a PCA. The camera image formation process implies however that the camera motion is a composition and not an addition to the deformable brain motion. The affine camera model used is also very restrictive compared to the more realistic perspective model. In this work, solutions are proposed to these two problems.

Contributions

There are several contributions in this work. 1) We propose a motion model dedicated to the compensation of the repetitive brain motion during neurosurgery. The model explicitly distinguishes between the camera motion and the nonrigid brain motion. We assume and experimentally check that the nonrigid brain motion lies in a low dimensional affine space that can be learned from a few frames. 2) The main contribution is the resolution procedure. While the model is nonlinear, we will see that, with adequate variable changes, the equations involved to find our model parameters become considerably easier to solve: an original extension of the direct linear transform (DLT) [START_REF] Abel-Aziz | Direct linear transformation from comparator coordinates into object space coordinates[END_REF][START_REF] Hartley | In defense of the eight-point algorithm[END_REF] is proposed, involving only one singular value decomposition. These variable changes allow avoiding local minima considerations and to estimate the motion parameters easily. 3) From the robustness point of view, we propose a simple preprocessing of the keypoints using temporal consistency to detect outliers before parameters estimation. This enables us to robustly estimate the parameters with only one fit. We also propose a procedure to correctly handle keypoints tracking under large camera viewpoint change.

Evaluation on real in vivo data has been done to investigate the influence of the parameters of our model and to compare our method to standard optical flow routines. The proposed procedure has also been evaluated into the framework of an in vivo protocol for intraoperative identification of somatosensory brain areas during neurosurgery.

Theory

Transformation Model

To the best of our knowledge, when a linear model is used for the optical flow as in [START_REF] Black | Learning parameterized models of image motion[END_REF][START_REF] Fleet | Design and use of linear models for image motion analysis[END_REF][START_REF] Ricco | Dense lagrangian motion estimation with occlusions[END_REF][START_REF] Wulff | Efficient sparse-to-dense optical flow estimation using a learned basis and layers[END_REF], no global motion component accounts for camera or global subject displacements. In this work, the camera motion is explicitly integrated into the model. A Lagrangian representation is adopted and the geometric transformation between the initial and the t th frame is modeled by

T (x, t) = U(t)T d (x, t) (1)
where x ∈ R 2 is the spatial position of a given pixel in the initial frame, t ∈ R is the time and T (x, t) gives the position of x in the t th frame, T d (x, t) is the deformable part of the transform while the U(t) transform accounts for the camera and global subject motion at time t. In this work, the pinhole camera model is used for the projection of the 3D scene on the camera and U(t) is a homography (objects are approximated as planar surfaces). Using homogeneous coordinates [START_REF] Szeliski | Computer vision: algorithms and applications[END_REF] T (x, t) and T d (x, t) are in R 3 , the homography U(t) is a 3 × 3 matrix with 8 degrees of freedom and the homography applied to T d (x, t) is a simple matrix vector product.

It is assumed that the object deformable motion lies in a low dimensional affine space. This assumption will allow using mostly linear expressions in our derivation and to have only a limited few parameters to estimate. We consequently express the deformation T d as a weighted sum of basis transformations:

T d (x, t) = x + T µ (x) + K k=1 λ k (t)p k (x). ( 2
)
T µ is the average local deformation, p k are the stationary basis vectors of the deformation and λ k ∈ R are the time dependent deformation coefficients. We want to stress again that despite the linear dependency of T d on its parameters λ k , it represents a nonlinear deformation of the image. The low dimensional affine subspace assumption has been made in several works on optical flow estimation. It is especially well adapted for the application we are interested in, because the object undergoes a repetitive deformation over time. It also allows accounting for the camera motion separately from the brain surface motion and to learn a specific basis for each subject.

Once the basis vector p k and T µ are known, only 8 + K degrees of freedom are left for the estimation of the transformation at a given timepoint: 8 for the homography and K for the λ k coefficients. For our application, K is very low, around 5 in practice. The number of parameters to estimate is consequently reduced from twice the number of pixels for a standard optical flow routine to 8 + K, less than 15.

Compared to other works using affine subspaces, the camera motion and the object deformable motion are explicit in our model. As the full transformation is the composition of these two terms, it depends nonlinearly on the parameters U and λ. At a first sight, this may seem problematic as it implies solving nonlinear optimization problems to estimate these parameters. We propose to use a first variable change to eliminate this problem. If V(t) = U -1 (t), equation 1 becomes

V(t)T (x, t) = T d (x, t). (3)
We will see in the following sections that, consequently, all the steps of the registration process can be solved using linear algebra routines only.

Estimation of the Transformation

Assuming the deformation basis (T m u, p k ) is known, (details will be given in section 2.3), the transformation between the first frame and a new frame is estimated by fitting the parameters U and λ k (or equivalently V and λ k ). In this section, the fitting procedure is described and as we will see, it is both efficient (only one SVD) and robust to occlusions.

Sparse Parameter Estimation with Extended DLT

As the degree of freedom for a transformation is now very low, it is not necessary to use all the pixels to estimate the parameters V and λ k . In a manner similar to [START_REF] Ricco | Dense lagrangian motion estimation with occlusions[END_REF][START_REF] Wulff | Efficient sparse-to-dense optical flow estimation using a learned basis and layers[END_REF], Harris keypoints [START_REF] Harris | A combined corner and edge detector[END_REF] are detected on the first frame and tracked along the frames with a sparse version of the Lucas and Kanade method [START_REF] Bouguet | Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm[END_REF]. A more uniform distribution of the keypoint is obtained by setting the Harris threshold to 0 and by imposing a minimal distance between keypoints. A zero Harris threshold also makes the detection invariant to a global illumination change (see [START_REF] Faille | Adapting interest point detection to illumination conditions[END_REF]). One can consider now that for each keypoint x l detected on the first frame, its position at time t, T (x l , t), is known. In the figure 1, a frame is presented with the keypoint overlaid.

Using equation 3 on the keypoints at given time t, we obtain:

VT (x l ) - K k=1 λ k p k (x l ) = x + T µ (x l ), (4)
where the dependency on t has been dropped for readability.

From this set of equations (two for each keypoints), the transformation parameters V and λ k can be estimated using the original extension of the DLT method proposed below.

Extended DLT estimation. The DLT [START_REF] Abel-Aziz | Direct linear transformation from comparator coordinates into object space coordinates[END_REF][START_REF] Hartley | In defense of the eight-point algorithm[END_REF] is a standard method to find a homography given a set of corresponding points in two images. To solve the equation 4, although the correspondences are known, the DLT cannot be used out of the box to estimate V due to the unknown λ k variables. The following extended DLT enables the estimation of V and (λ k ) k all at once. For each point x, let's define T µ (x) = x + T µ (x). Let c ∈ {1, 2} indexes the two scalar components of the 2D equation 4 and let's use superscript to denote the component of vector variables. (For space reason, see fig. 3 for the derivation of the linear equation of the generalized DLT). Considering all the keypoints x and c ∈ {1, 2}, we end up with a linear system Rv = 0 where the unknown vector

v ∈ R 9+3K is v = (v, λ ) = v i j , λ l k and λ l k = λ k v 3,l .
Like v, v is defined up to a scale factor. As with the standard DLT, for a given weight matrix W, v is chosen as the solution of

min v =1 Rv W ,
which is given by the eigenvectors corresponding to the smallest eigenvalue of R T W 2 R. If v = (v, λ ) is one such eigenvector, the initial unknown parameters can be retrieved by solving the overdetermined system:

v i j = vi j λ k v 3l = λ l k
An analytical solution is available, if the first set of equation is solved exactly for v and the second set is solved in the least square sense for λ:

       v = v λ c k = l v3,l λ cl k l v2 3,l .

Tracking Keypoints Under Large Camera Motion

The transformation model (equation 1) proposed in this work explicitly includes perspective transform and thus, it already allows large camera motions.

However, in a Lagrangian setting, the Lucas and Kanade algorithm can be problematic when the camera undergoes a large rotation: the windows used to solve the optical flow equations can be significantly rotated. While it is reasonable to assume a small change between successive frames, this assumption is false between the initial and the current frame. If a known homography globally realign the two images, the following simple tracking procedure denoted as LRLK for Large Rotation Lucas and Kanade algorithm is applied: the second image is resampled and the starting value adjusted with the homography, the standard Lucas and Kanade algorithm is run with these new data and the resulting position is sent back with the homography. Thanks to its Lagrangian formulation, this procedure is stable: the errors do not accumulate from one frame to another. Similarly to the Eulerian approach, it allows large rotations and large camera motions in general. If for two images I 1 and I 2

q 2 = LK(I 1 , q 1 , I 2 , q 0
2 ) denotes the position in the image I 2 of the points q 1 ∈ I 1 as given by the sparse Lucas and Kanade algorithm run with q 0 2 as starting value, the procedure is formally described in algorithm 1 and illustrated in fig. 4.

Algorithm 1 Large Rotation Lucas and Kanade's algorithm

1: procedure LRLK(I 1 , q 1 , I 2 , q 0 2 , U) 2: Ī2 (x) = I 2 (U x) 3: q0 2 = U -1 (q 0 2 ) 4: q2 = LK(I 1 , q 1 , Ī2 , q0 2 ) 5:
q 2 = U q2 6:
return q 2

As the global perspective transform between the initial and the current frame appears explicitly in our model (variable U in the equation 1), this procedure fits well within our framework: the homography estimated at the previous frame is used for the LRLK routine. The keypoints positions at time t, q(t) are computed by LRLK (I(., 0), q(0), I(., t), q(t -1), U(t -1)) .

Outliers Detection

If equation 4 is solved using a standard unweighted least square, the L 2 norm will make this estimation sensitive to outliers. Keypoints are outliers if their tracking fails for some reason, for example, due to the occlusion of the scene by an external object. In related works, the outliers problem is accounted Each scalar component of equation 4 can be written as:

v c1 T 1 (x) + v c2 T 2 (x) + v c3 v 31 T 1 (x) + v 32 T 2 (x) + v 33 = T c µ (x) + K k=1 λ k p c k (x),
or equivalently:

v c1 T 1 (x) + v c2 T 2 (x) + v c3 = v 31 T 1 (x) + v 32 T 2 (x) + v 33        T c µ (x) + K k=1 λ k p c k (x)        v c1 T 1 (x) + v c2 T 2 (x) + v c3 = v 31 T 1 (x)T c µ (x) + v 32 T 2 (x)T c µ (x) + v 33 T c µ (x) + K k=1 λ k v 31 T 1 (x)p c k (x) + λ k v 32 T 2 (x)p c k (x) + λ k v 33 p c k (x) .
For brevity, let's define T ic (x) = T i (x)T c µ (x) and p ic k (x) = T i (x)p c k (x). To eliminate the nonlinear interaction between λ k and v 3l another variable change is done by defining: λ l k = λ k v 3l . The equations then become linear: using robust fit such as M-estimator [START_REF] Ricco | Dense lagrangian motion estimation with occlusions[END_REF][START_REF] Wulff | Efficient sparse-to-dense optical flow estimation using a learned basis and layers[END_REF], or IRLS [START_REF] Roberts | Learning general optical flow subspaces for egomotion estimation and detection of motion anomalies[END_REF][START_REF] Sdika | Robust real time motion compensation for intraoperative video processing during neurosurgery[END_REF].

v c1 T 1 (x) + v c2 T 2 (x) + v c3 = v 31 T 1c µ (x) + v 32 T 2c µ (x) + v 33 T c µ (x) + K k=1 λ 1 k p 1c k (x) + λ 2 k p 2c k (x) + λ 3 k p c k (x)
In this work, we propose to use the temporal smoothness of the transformation to detect the outliers before the fit. Concretely, for a given keypoint x l , equation 4 is weighted to remove the influence of points whose position changes abnormally fast. V and λ k are estimated with a weighted least square fit where the x l point equation for the fit at time t is weighted by:

w l (t) = exp       - T (x l , t) -T (x l , t -1) 2 2 2σ 2 O       (5)
where σ O is used to set the tolerance of what is an acceptable displacement between two frames. To summarize, when traking fails for some keypoints, they are given a lower weight in the fit. Motion parameters are estimated without these bad points while the global PCA model enables the recovery of the whole image deformation. Just a single IRLS iteration is necessary to robustly estimate the transformation parameters even with the presence of outliers. We will see however in section 3.4, that a few reweighting iterations have a low computational impact and can produces results that are more visually pleasing. In figure 2, a frame with occlusion is presented with the keypoints colorcoded by their weight. It can be clearly seen that the simple weighting scheme proposed allows the detection of tracking failure due to occlusions.

Full pipeline summary

To summarise, the brain motion compensation method proposed in this paper can be decomposed in two steps. 4: LRLK, algorithm 1: Frame at t is warped using the homography U = U(t -1), the standard Lucas and Kanade is used on the globally realigned images, keypoint are projected back in I(x, t). LRLK is effective because, with U = U(t -1), neighborhoods of the keypoints are approximatelly oriented after warping. The first step is to learn the affine subspace to which the brain motion belongs. To do so, the motion is estimated using a standard routine on a few initial frames of the video. The raw brain motion is then extracted from each dense motion vector (section 2.3) and the principal modes of the brain motion are estimated with a PCA.

I(t) I(0) x(t -1) U(t -1) LRLK Keypoint Tracking algo. 1 x(t) Extended DLT section 2.2.1 U(t), λ(t) p, T µ (learnt) Motion model eq. 1 T (t)
Once the brain motion basis is estimated, our fast method can be applied (see the pipeline on figure 5). Keypoints detected on the first frame are tracked with the LRLK algorithm (algorithm 1). The extended DLT (section 2.2.1) is then used to recover the motion parameters. These parameters enables the computation of the full dense motion (given by equation 1) in the given frame.

Learning the Nonrigid Transformation Basis

The first N tr frames of the video are used to learn the affine subspace in which T d lies, i.e. to estimate T µ and p k . First, (T (., t i )) i∈[1,N tr ] are estimated with a standard motion estimation routine from the literature. Then, to extract the brain motion T d (., t i ) from T (., t i ), the camera motion is estimated by solving min

V(t i ) 1 2 x∈P V(t i )T (x, t i ) -x 2 2 , ( 6
)
where P is a set of pixels and small deformations are assumed.

Here again, Harris keypoints are used to reduce the computation time.

Equation 6 is a least square problem solved with standard geometric transform estimation routine for each timepoint.

Once the V(t i ) are estimated, the deformable training transforms are given by

T d (x, t i ) = V(t i )T (x, t i ) for i ∈ [1, N tr ].
These deformable transformations are assumed to lie in a low dimensional affine space. A basis of this subspace is estimated using a PCA to retain the space that best captures the variability of these vectors: T µ is the mean of the (T d (., t i )) i vectors and p k are the K first eigenvectors of their covariance matrix. An example of mean deformation and basis images is presented in figure 6.

Experiments and Results

In this section, the proposed method in this paper has been evaluated on real neurosurgical videos taken in the operating suite. Experiments were done in the neurosurgery service of the Hospices Civils de Lyon. clinical surgical microscope (Zeiss OPMI Pentero) was used to record the video of opened brain areas. The microscope objective was positioned around 25 cm above the brain. The image plane was positionned to correspond to the plane tangent to the center of the brain area exposed. The brain was illuminated by a xenon lamp with a direction making a small angle with the optical axis to optimize the light collection and specular reflexion attenuation compromise. Each video lasts between 30 and 90 seconds, has a frame rate of 25 fps and a frame size of either 511x388 or 720x576. The Farnebäck method [START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF], denoted as GF, and the total variation Perez method [START_REF] Pérez | Tv-l1 optical flow estimation[END_REF], denoted as TV, as implemented in OpenCV 3.0 2 , were used for comparison. The GF method was also used as the motion estimation routine on the training frames for the subspace estimation. The GF method has been chosen for its availability in CPU version in OpenCV and its relative efficiency. Note that other methods could have been chosen for comparison as long as are also used for the training. For all but the training size investigation experiment, the 25 initial frames of each video were used as training set.

In section 3.1, the low dimensional affine model has been validated and the influence of its most important parameters has been investigated. In section 3.2, the robustness to occlusions of our method is evaluated. The robustness to large camera motions is finally evaluated in section 3.3. CPU considerations are presented in section 3.4. Finally, the use of our method into the framework of an in vivo protocol for intraoperative identification of somato-sensory brain area is illustrated in section 3.5.

Parameter Analysis

As no ground truth is available, a really satisfying and absolute error metric cannot be used. As it is often the case for this kind of work, we have to rely on a derivative error measure. Here, the error is measured by where n is the number of frames in the video and B is a brain mask manually segmented on the initial frame and T GF is the transformation obtained using the GF method. In the absence of ground truth, this measure is an indication of the adequacy of the model to the problem and that transformations obtained with high degrees of freedom methods can be represented by our low dimensional model.

Err(T ) = 1 n|B| n t=1 x∈B T (x, t) -T GF (x, t) ,
In figure 7, the average error is plotted against the number of principal components for several values of the minimal distance (hd) allowed between the Harris keypoints, with and without weighted fit (eqn. 5). As expected, the method gives better results when hd is the lowest, or equivalently, when the number of keypoints is the highest. One can also see that, regarding K, the number of principal components, a plateau is quickly attained: the deformable component of the transformation indeed lies in a low dimensional affine space. However, when hd is too high, the method seems less stable and the error tends to increase again with K. Although there were no occlusions on the video we used, the weighting (eqn. 5) of the keypoints does improve the estimation considerably. The error globally decreases and stability is preserved with less keypoints.

The mean error as a function of the number N tr of training frames is presented in figure 8 with K = 5 and hd = 6. The error quickly decreases for low N tr , the decrease is less important when N tr is greater than 40. Setting this parameter is a compromise between the desired accuracy and the training time: while a higher N tr improves the accuracy, it also implies heavier use of the slower standard optical flow routine and that the high dimensional PCA will be done on a larger set.

For reference, the average difference between GF and TV is 0.27 ± 0.10. There is a priori no reason why GF or TV would be better but these results show that the difference between the proposed method and GF seems within the range of differences between established optical flow methods.

Video with Occlusion

In this section3 , a black rectangle has been placed in the center of the brain after 3 seconds until the end of the video to mimic the apparition of an object in the field of view of the camera. The experiment has been reproduced with a rectangle size varying from 10% to 80% of the brain. The error between the GF transform computed with no occlusion and the result of the motion estimation algorithms with occlusions is measured and presented in figures 10 and 11. Snapshots of a motion compensated video are displayed in figure 9. Our method with the weighted fit produces videos with no artifacts while the unweighted estimation implies strong artifacts all over the image. While maybe less visible on the snapshots, the video processed with the GF method presents smooth distortion artifacts in the vicinity of the rectangle. The video compensated using TV also presents artifacts in a smaller region and more sharply defined.

In the previous section, we saw that weighting the fit with equation 5 enables to lower the error and to improve stability with larger inter keypoints distance. On figure 10, one can see that this weighting is necessary when foreign objects enter the field of view. The fit fails even when the size of these objects is small with respect to the brain size due to the high instability of the L2 norm to outliers. GF also fails in the presence of occlusions but to a lower extent. While not as robust as our method, TV is clearly more robust than GF. As expected, the results presented in figure 11 confirm that stability is better with low hd. It is however reassuring to see that even for quite large hd (30 pixels), the motion compensation still resists to about 60% or 70% of occlusions. In these cases, even if the motion compensation fails, in our experience, the fit get backs to normal when the object disappears.

Large Camera Motion

To assess both the validity of our model (eqn. 1) and the LRLK tracking (algorithm 1), we simulated camera motions by adding rotation, scaling or viewpoint changes to the input video after a few seconds. We stress that the initial nonrigid motion is still present in the video with additional motion. The mean intensity difference (MID) between the current and the initial frame in the brain mask

MID(t) = 1 |B| x∈B I(x, t) -I(x, 0) RGB ,
where . RGB is the Euclidean norm in the RGB space, is used to compare GF, TV and variants of our method. Pure rotation experiment results are presented in figure 12. A time dependent rotation is added to the initial video, the MID after compensation is plotted for GF, TV and our method with variants of Lucas and Kanade keypoint tracking: LK Lag is the tracking with a Lagrangian setting q(t) = LK(I(0), q(0), I(t), q(t -1)), LK Euler for an Eulerian specification q(t) = LK(I(t -1), q(t -1), I(t), q(t -1)) and LRLK is the algorithm 1 keypoint tracking q(t) = LRLK (I(0), q(0), I(t), q(t -1), U(t -1)). To assess the benefit of using the new physically meaningful composition model, we also included our previous additive model [START_REF] Sdika | Robust real time motion compensation for intraoperative video processing during neurosurgery[END_REF] in the comparison: the camera motion is an affine transform that is added to the deformable brain motion. This model, denoted as LRLKadd is used with the LRLK tracking.

One can see on figure 12 that both GF and LK Lag rapidly explode. While TV is better than GF, it is not resistant to even the slightest rotations. One can however notice that TV goes back to normal at every full rotation. In "Lucas and Kanade like" methods, even if the transformation is correctly initialized from the previous frame, the local window around each pixel does not rotate and ends up being really different. The known drawback of the Eulerian estimation is visible: the LK Euler error does not explode but is slowly but surely increasing. With LRLK, the error is stable and stays at the level before the rotation begins. This performance is the result of the combina- tion of two ingredients: the Lagrangian setting which provides the stability and the camera motion estimation and resampling which allows the large global perspective changes. One can also see that using the physically meaningful composition model is essential: despite the tracking is done with LRLK, the additive model LRLK-add fails for very small affine camera motion.

In figure 14 are displayed snapshots of motion compensation results when not only rotation but also scale changes (from 0.5 to 2) have been added 4 . Results of the previous paragraph are visually confirmed: neither GF, TV nor LK Lag are able to estimate the rotational motion of the camera; LK Euler is far better 4 The video and the results are given in additional materials. at first sight but on long time course, errors accumulate;when the tracking is done with LRLK, the motion estimation stays correct and the video is not artifacted even for a full rotation and scale changes.

For the next experiment, we simulated a viewpoint change in which the camera is moved around the pulsating brain. The MID is plotted in figure 13 as a function of the viewpoint angle. Our model with an orthographic camera (Aff.) and with a pinhole model (Persp.) is compared to TV and GF. For Aff, the only difference in our framework is to replace the homography U by an affine transform. Note that in this case, the extended DLT is not necessary: the equation 4 is linear and can be solved with linear least square routine. Both TV and GF perform reasonably well up to ±45 • . TV performs better than GF for high viewpoint changes but still, the results are not acceptable. One can see that the use of a perspective camera in our model is very important here: while the error is very high from the first viewpoint change degree with the orthographic camera, the pinhole camera model is very robust up to very large viewpoint changes.

One can also remark that real life camera motions will include pure viewpoint change mixed with rotation and scaling. Errors from the different camera motion experiments will be added and the discrepancy between our method with a perspective model and the other methods should be even clearer.

CPU Time

CPU measurements have been done on a Dell desktop with Intel Xeon E5-2640 2.40GHz processors. Measurements have been done using a single or 4 cores. The number of frames processed per second (fps) has been reported in table 1 for several variants of our method and the GF and TV method. Frame rate measurements includes reading from the input file, motion compensation and writing results to disk. Measurements have been done on 720 × 576 images. Frame rate of the input video was 25 fps.

Depending on the parameters chosen for our method, it is around 3 to 4 time faster than the GF method and these parameters can be chosen to have a real time motion compensation. The TV method is really expensive. The CPU criterion clearly discards this method for use in real clinical condition, even as the routine used for the training frames.

Among the two parameters hd and N IRLS of our method, only the minimal distance between keypoints hd has a real influence on the computation time. This parameter is directly linked to the number of keypoints used in the processing. Although the processing involved with N IRLS also depends on the number of keypoints, its influence on computation time is low because it does not depend on the LK tracking which represents a large part of processing time. The PCA to compute the T d basis takes about 2 GF iterations.

Although the efficiency of our multi core implementation is only around 40%, it is sufficient to be clearly above the real time frame rate of 25 fps with only 4 processors and to be able to add further processing for the analysis of the video for clinical parameter extraction.

Application to intraoperative brain areas identification

As an illustration of the relevance of the proposed motion compensation method in a real clinical application, its use for the intraoperative identification of somato-sensory brain areas during tumor resection in neurosurgery has been investigated. These areas are so far often identified using the potentially risky (epilepsy) electrostimulation method: brain surface areas are stimulated, if the stimulation of a given area triggers a finger motion, then the 'finger motion area' is identified. This identification method is used as gold standard and compared to a purely optical method: finger motions are applied to the anesthetized patient with a 30s rest; 30s stimulation; 30s rest paradigm. Variations in the hemodynamic parameters were then assessed based on the model described in [START_REF] Jakovels | Rgb imaging device for mapping and monitoring of hemoglobin distributionin skin[END_REF], for blood variation assessment. This allows to compute a hemodynamic map to identify the somato-sensory cortex area corresponding to the finger motion.

Figure 15 shows the variations in total hemoglogin concentration between the first rest period and the stimulation period for each pixel. The activated area has been identified by the gold standard in clinical practice, that is electrical stimulation. Fig 15 illustrates that our motion compensation method is effective to dramatically decrease artifacts appearing all over the image and showing artificial hemoglobin variations. In particular the motion compensated image shows mainly green/yellow (no/slight variations) or red (hemoglobin increase due to brain activation) events, as opposed to the non-compensated image which exhibits very noisy red and blue events. Therefore, while a red area corresponding to gold standard (electrical simulation) appears when the motion compensation is used, no clear activated brain areas can be identified without the motion compensation.

Conclusion

In this paper, we proposed a real-time motion estimation method to compensate repetitive brain motion on videos acquired during neurosurgery. Our method uses an original low degree of freedom transformation model defined as the composition of a perspective transformation and a nonrigid deformation lying on a low dimensional affine space. Despite the nonlinear nature of the transformation, two variable changes transform the nonlinear parameter estimation into a much simpler problem solved with only a single SVD computation. The motion is estimated with the tracking of only a sparse set of keypoints for efficiency and using a Lagrangian specification for stability. Using the perspective part of our transformation model, we also proposed an adaptation of the Lucas and Kanade procedure able to reliably track the keypoints under large rotation of the camera around its axes and more generally, under large viewpoint change. Outliers are handled by weighting the least square: a single fit is sufficient to handle occlusions caused by foreign objects entering the field of view. Few reweighting iteration of the least square fit has however very little impact on the computation time. Although the experiments in the paper have been done without these iterations, we found that in some extreme cases, adding these few iterations can help to produce visually better results at negligible cost.

As a clinical application of the proposed method we show its relevance as a preprocessing step in the intraoperative identification of brain areas during tumor resection. The usefulness of the proposed method is very clear to discriminate between real physiological hemodynamic events from noisy camera motion driven events. This step is then mandatory to realize a robust identification of brain areas in this demanding clinical context. The model used for areas identification was taken from [START_REF] Jakovels | Rgb imaging device for mapping and monitoring of hemoglobin distributionin skin[END_REF]. Further works will consist in the improvement of this conversion model to get hemodynamic parameters from colorimetric variations.

In this article, the videos considered for illustration were acquired with an incoherent white light to produce RGB reflectance images of the human brain. Other optical setup using coherent light from lasers have also been reported to be useful during neurosurgery. and Laser speckle imaging which can produce perfusion maps [START_REF] Raabe | Laser doppler imaging for intraoperative human brain mapping[END_REF][START_REF] Richards | Intraoperative laser speckle contrast imaging for monitoring cerebral blood flow: results from a 10-patient pilot study[END_REF][START_REF] Parthasarathy | Laser speckle contrast imaging of cerebral blood flow in humans during neurosurgery: a pilot clinical study[END_REF]. With such optical imaging modalities the blood vessels are also visible and the brain surface is highly textured. Therefore, the keypoints detection and tracking is possible and the real time motion compensation algorithm presented here could a priori also be applied with success in this extended field of brain imaging for neurosurgery.

The method has been evaluated on video recorded in neurosurgical unit. The motion estimation routine proposed in this paper can be done in real time, robustly and accounting for large viewpoint changes. This opens the way to further temporal analysis of these videos to help neurosurgeon in their action.

Figure 1 :Figure 2 :

 12 Figure 1: A frame with the Harris keypoints used for the tracking (distance between keypoints is hd = 6).

Figure 3 :

 3 Figure 3: Derivation of the generalized DLT.

Figure 5 :

 5 Figure 5: Pipeline of real time motion estimation method after training. I: image frame, x: set of keypoints positions, U:camera motion, T µ , p: motion basis learned on initial frames, λ: deformable brain motion parameters, T :dense motion.

2 Figure 6 :

 26 Figure 6: Images learned to model the brain deformable motion T d (eqn. 2): mean displacement T µ and three first eigenimages p k . First row is for displacement along x, second row is for displacement along y.

  2 http://opencv.org

Figure 7 :

 7 Figure 7: Mean error versus number of PCA components for several minimal distance between the keypoints (hd). Results are given when outliers weighting (eqn. 5) is used (Weighted) or not (Unweighted).

Figure 8 :

 8 Figure 8: Mean error as a function of the number of training frames (using K = 5, hd = 6).

Figure 9 :Figure 10 :Figure 11 :

 91011 Figure 9: Snapshot of the motion compensated video for several percentage of occlusion. Results are presented for the opencv GF and TV methods and our method when outliers weighting (eqn. 5) is used (W) or not (NW).

Figure 12 :

 12 Figure12: MID when the original video is rotated around the camera axis. Results are presented for the opencv Farnebäck routine (GF) and for our method when the tracking is done from one frame to the next (Euler LK), from the initial to the current (Lag LK), using our previous additive model (LRLK-add) and the proposed LRLK routine.

Figure 13 :

 13 Figure13: Average difference between the current and the initial frame when a time dependent viewpoint change is added to the video (in abscissa: the viewpoint angle). Results are presented for the opencv GF and TV routine and for our method with an orthographic (Aff.) or perspective (Persp.) camera model.

Figure 14 :

 14 Figure14: Snapshot of the motion compensation results when additional rotation/scale are added to the original video. Results are presented for the opencv Farnebaäck routine (GF) and for our method when the tracking is done from one frame to the next (LK Euler), from the initial to the current (LK Lag) and using the proposed LRLK routine.

Figure 15 :

 15 Figure15: The screenshot of a video (left) and the corresponding hemodynamic maps computed with (middle) and without (right) our motion compensation method as pre processing. The map without motion compensation presents motion artefact near the blood vessels. Colorbar show variations in total hemoglobin concentration in arbitrary units. The black arrow reveals the somato-sensory cortex identified by cortical stimulation during surgery; it corresponds to area of high value in our maps.

Table 1 :

 1 This includes Laser Doppler imaging Single Core Multi Core hd GF TV N IRLS = 1 N IRLS = 4 N IRLS = 1 N IRLS = 4 66.99 0.12 Number of frame per seconds for the OpenCV GF and TV methods, and our method with different parameters: N IRLS is the number of least square re-weighting (all other experiments have been done with N IRLS = 1), hd is the minimal distance between the pixels.

		23.81	21.74	56.13	49.28
	18	26.67	25.97	59.15	57.40
	30	28.17	27.40	63.27	61.34

A video showing results under occlusion is given in additional materials.

Acknowledgment

This work was supported by the LABEX PRIMES (ANR-11-LABX-0063) of Université de Lyon, within the program "Investissements d'Avenir" (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR).