Gilles Adda
email: adda@immi-labs.org

Laurent Besacier
email: laurent.besacier@imag.fr

Alain Couillault
email: alain.couillault@univ-lr.fr

Joseph J Mariani
email: joseph.mariani@limsi.fr

Hugues De Mazancourt
email: hugues.de-mazancourt@eptica.com

"Where the data are coming from?" Ethics, crowdsourcing and traceability for Big Data in Human Language Technology

Keywords: Ethics, crowdsourcing, traceability

Based on the experience gained on the observation of the corpora developement in HLT, the authors want to warn the Big Data community about some recent usage of human computation. For instance, the growing use in the HLT community of crowdsourcing methods, and especially of microworking retributed crowsourcing platforms, lead to many ethical, economical and juridical concerns. The authors want also to foster some behaviours, especially concerning traceability, implemented in the form of a charter, the Ethics and Big Data Charter .

INTRODUCTION

At the turn of the 90s, Human Language Technology (HLT) changed drastically, with the developpement of probabilistic machine learning methods and the establishment of the evaluation paradigm. One crucial result implied by these changes is the development of corpora of growing sizes and of varying natures. From speech corpora of few hours and text corpora of few millions of words, we developed our models from Multimedia, Multimodal, Multilingual (3M) data containing for instance thousands of hours of video and terawords text corpora coming from social media, newspapers, subtitles, transcriptions, translations, etc. * Alphabetic order

MASTODONS Paris, France

To be able to manage the constitution and the distribution of these corpora, specific agencies have been developed within the HLT community, such as LDC1 or ELDA. 2 But at a certain point, the growth of the needed corpora along the famous Big Data 3Vs3 axes, goes beyond the classical well-controlled developement of corpora, relying on a meticulous and safe selection of data, and retributed employees to annotate them. We did see more and more corpora design involving the use of crowdsourcing, and especially the use of microworking, monetary compensated work platform such as Amazon Mechanical Turk. The authors, as academic and industrial HLT actors, wrote numerous positions papers to warn the community on ethical, economical and juridical concerns about mturk [START_REF] Adda | Language resources and amazon mechanical turk: legal, ethical and other issues[END_REF][START_REF] Fort | Amazon mechanical turk: Gold mine or coal mine?[END_REF][START_REF] Adda | Crowdsourcing for Speech Processing, chapter Economic and Ethical Background of Crowdsourcing for Speech[END_REF][START_REF] Fort | HLT Challenges for Computer Science and Linguistics, chapter Crowdsourcing for Language Resource Development: Criticisms about Amazon Mechanical Turk Overpowering Use[END_REF]. From these statements, we found that retributed crowdsourcing was only one (important) symptom of way of doing in the HLT community for the constitution and annotation of corpora. Some of the authors have designed the Ethics and Big Data Charter [START_REF] Couillault | Charte Éthique et Big Data : parce que mon corpus le vaut bien ![END_REF] in collaboration with representatives from interest groups, private companies and academic organizations, including ELDA, ATALA,4 AFCP5 and APROGED. 6 The purpose of this Charter is to provide resources developers, not only in the HLT community, but more broadly in the Big Data community, with a precise framework to document their resources and ensure their traceability and transparency.

TOWARDS TRACEABILITY: THE ETHICS AND BIG DATA CHARTER

To adopt an ethical behavior in developing, funding, using or promoting language resources is first and above all a matter of choice: for the provider, deciding which approach to adopt -crowdsourcing or not -, or which platform to request on, or the level of remuneration of the workers; for the funding agency, choosing which project to fund; for users, choosing which resource to use or acquire.

Contents of the Charter

The Ethics and Big Data Charter is provided as a form to be filled in by the dataset provider. It is split into three major sections: traceability, intellectual property and specific legislation, preceded by a short identification section containing the names of the resource, the contact and responsible persons and a short description of the data set. We describe below some of the important points listed in the charter.

Traceability

Traceability is key to our purpose of putting forward ethical issues. The traceability part of the charter allows to precise the relationship between the resource provider and the workers involved in developing the resource, including legal bounding, workers skills, selection criteria. Specific focus is put on personal data, i.e. data such as voice or video recording, which can provide a means to identify a person directly or indirectly. The Charter requires to precise if and how the data is de-identified, and if and how the individuals were informed of the purpose of the data collection. Quality assurance is another major aspect of traceability addressed by the charter, as it requires to document the quality assurance strategy, so that the user of the data set is fully informed on the level of quality s/he can expect.

License and copyright

Thanks to a great deal of effort accomplished in the definition of -mainly open source -license schemes, it has become common practice to attach a license to a data set. The License and Copyright section of the Charter goes beyond this and puts the focus on questions which may be disregarded, like ensuring that the legal or moral copyrights of the persons who worked on compiling, enriching or transforming the data are respected.

Specific legal requirements

A third section of the Ethics and Big Data Charter deals with legal requirements that may arise from certain properties of the data set. For example, a country may have issued specific laws regarding the storing, use and/or dissemination of personal data. The Charter serves as a reminder for checking if such requirements exist.

Availability

The Ethics and Big Data Charter is available on-line. 7 Examples of charters are also provided.

CONCLUSION AND PERSPECTIVES

In [START_REF] Couillault | Evaluating Corpora Documentation with regards to the Ethics and Big Data Charter[END_REF], we showed that even the most widely referred to language resources present documentation lacks, in particular 7 http://wiki.ethique-big-data.org regarding the persons who produced the work and which external resources have been used. We assume that this also applies for many corpora used in the Big Data community.

The reality we are facing now in the call for projects of all the national and international funding research agencies is a growing interest for Big Data, and a foreseeable burst in the number of Big Data related projects, with the use, for instance, of personal data from Twitter or Facebook, the development of crowdsourcing and so on. But we also know that for these data, uncertainties about privacy, the way they have been acquired, who has really worked, is likely to be of an order of magnitude more important than for the present resources.

Other initiatives and articles have tackled this subject of Big data and Ethics (see for instance [START_REF] Davis | Ethics of Big Data: Balancing Risk and Innovation[END_REF][START_REF] Boyd | Critical questions for big data[END_REF][START_REF] Richards | Big Data Ethics[END_REF]), most of them dealing with the problem of privacy and surveillance.

For these reasons, we think it is crucial to gather all the initiatives such as the Ethics and Big Data Charter which aims at promoting ethics and traceability in resources, in order to propose, at the international level, a way to limit the risks for all the actors (funding agencies, research laboratories, private companies) of the data added value chain, regarding the use of Big Data and bring the benefits of the Ethics and Big Data Charter to current projects.

Linguistic Data Consortium, http://www.ldc.upenn.edu/

Evaluations and Language resources Distribution Agency, http://www.elda.org/

Volume, Velocity, Variety

Association pour le Traitement Automatique des Langues http://www.atala.org

Association Française de Communication Parlée http:// www.afcp-parole.org

Association de la Maîtrise et de la Valorisation des contenus http://www.aproged.org


Fabio Solari

Manuela Chessa

Kartheek Medathati

Pierre Kornprobst

What can we expect from a classical V1-MT feedforward architecture for optical flow estimation?

Keywords: Optical flow, spatio-temporal filters, motion energy, cortical areas V1 and MT, benchmarking, Middlebury dataset Flot optique, filtres spatio-temporels, énergie de mouvement, aires corticales V1 et MT, évaluation, données de Middlebury

Motion estimation has been studied extensively in neurosciences in the last two decades. The general consensus that has evolved from the studies in the primate vision is that it is done in a two stage process involving cortical areas V1 and MT in the brain. Spatio temporal filters are leading contenders in terms of models that capture the characteristics exhibited in these areas. Even though there are many models in the biological vision literature covering the optical flow estimation problem based on the spatio-temporal filters little is known in terms of their performance on the modern day computer vision datasets such as Middlebury. In this paper, we start from a mostly classical feedforward V1-MT model introducing a additional decoding step to obtain an optical flow estimation. Two extensions are also discussed using nonlinear filtering of the MT response for a better handling of discontinuities. One essential contribution of this paper is to show how a neural model can be adapted to deal with real sequences and it is here for the first time that such a neural model is benchmarked on the modern computer vision dataset Middlebury. Results are promising and suggest several possible improvements.

Que peut-on attendre d'une architecture feedforward classique de V1-MT pour estimer le flot optique?

1 Introduction

Interpretation of visual motion information is a key competency for biological vision systems to survive in a dynamic world and also for computer vision as it could enable several applications. Owing to application potential, optical flow estimation which is a fundamental problem in visual motion analysis has been studied extensively by both computer vision and biological vision communities. Estimating optical flow refers to assignment of 2-D vectors at sample locations of the visual image that describe their displacements within the sensor's frame of reference using the change of structured light in the retinal or camera image. This displacement vector field constitutes the image flow representing apparent 2-D motions resultant of their 3-D velocities being projected onto the sensor. Such 2-D motions are observable only at intensity variations (and are thus contrast dependent) as a consequence of the relative change between an observer (eye or camera) and the surfaces or objects in a visual scene.

In the past two decades efforts by computer vision researchers have lead to development of a large number of algorithms for computation of optical flow with majority of them extracting flow field using two consecutive frames in the video sequence (see [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF] for a review). Despite a large amount of work in this area, the problem remains still hard to solve as many of the algorithms either lack consistent accuracy across video sequences or suffer from computational cost. One of the prominent achievements of the research efforts that went in computer vision could be development of publicly available benchmarking datasets that provided an exciting opportunity to evaluate models and learn their short comings of the predictions in case of natural image scenarios.

On the other hand the neural mechanisms underlying motion analysis in the visual cortex have been extensively studied almost with little interaction with computer vision community [2,3,4,[START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF][START_REF] Bayerl | A fast biologically inspired algorithm for recurrent motion estimation, Pattern Analysis and Machine Intelligence[END_REF][START_REF] Tlapale | Modelling the dynamics of motion integration with a new luminance-gated diffusion mechanism[END_REF]8,[START_REF] Bouecke | Neural mechanisms of motion detection, integration, and segregation: From biology to artificial image processing systems[END_REF] resulting in few mathematical models. Even though there was some early interaction among the two communities [START_REF] Heeger | Optical flow using spatiotemporal filters[END_REF][START_REF] Nowlan | Filter selection model for motion segmentation and velocity integration[END_REF][START_REF] Simoncelli | A model of neuronal responses in visual area MT[END_REF] comparatively little work has been done in terms of examining or extending the mathematical models proposed in biology in terms of their engineering efficacy on modern optical flow estimation datasets.

In this work we take a step towards filling the critical gap between biological and computer vision communities working on visual motion estimation leveraging and testing ideas proposed in biology in terms of building scalable algorithms. This is a challenging task as many of the mathematical models proposed in biology are confined to highly primed stimuli or often only examine a local decision making process such as a receptive field (RF) property which demands non-trivial extensions to be made before the ideas could be tested on the complex real world datasets.

The paper is organized as follows: In Sec 2 we begin by presenting a brief summary of the models describing biological visual motion estimation with emphasis on spatio-temporal filter based ones, and present our baseline model which is a three-step feedforward model. The two first steps correspond to V1 and MT pattern cells responses and follow classical ideas from the literature. The third step is a decoding stage to obtain the optical flow from MT population response. Then we propose extensions of the baseline model by introducing a filtering stage at the level of the MT pattern cell layer in order to better preserve motion discontinuities. In Sec. 3 we present the algorithmic details which are an essential contribution here since it is the first time that this kind of V1-MT architecture is evaluated on modern computer vision datasets. We present a multi scale processing strategy to deal with a large range of displacements and the solutions to deal with boundary conditions and unreliable regions. In Sec. 4, we evaluate our approach, and parameters setting is described. First we use some test sequences to show the intrinsic properties of our approach and the influence of some parameters. Then we benchmark our approach using the modern computer vision datasets Middlebury [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF]. Finally, we present the conclusion of our work in Sec. 5.

Model description

Context: Biological visual motion estimation

Two primary cortical areas involved in motion estimation are V1 [START_REF] Sincich | The circuitry of v1 and v2: Integration of color, form, and motion[END_REF][START_REF] Rasch | Quantitative inference of population response properties across eccentricity from motion-induced maps in macaque v1[END_REF] and MT [START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF] (for reviews see, e.g., [START_REF] Perrone | Spatial integration by mt pattern neurons: a closer look at patternto-component effects and the role of speed tuning[END_REF][START_REF] Bradley | Velocity computation in the primate visual system[END_REF][START_REF] Pack | 2.11 -cortical mechanisms for the integration of visual motion[END_REF]).

Several models have been devised in order to account for the neural mechanisms of motion analysis. A model for the extraction of the image flow, inspired by the stages of the visual motion pathway of the primate brain, is proposed in [START_REF] Heeger | Model for the extraction of image flow[END_REF]. Such a model is based on spatio-temporal RFs of simple cells, modeled by Gabor functions [START_REF] Daugman | Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters[END_REF], and on the computation of the motion energy [START_REF] Adelson | Spatiotemporal energy models for the perception of motion[END_REF] by a layer of complex cells. The problem of detecting local image velocity is also addressed in [START_REF] Grzywacz | A model for the estimate of local image velocity by cells in the visual cortex[END_REF], by a two-layer model that first computes motion energies and eventually estimates velocity by combining these energies. The combination of the responses of the different cells, necessary to solve the aperture problem and to compute velocity, is the main difference between [START_REF] Heeger | Model for the extraction of image flow[END_REF] and [START_REF] Grzywacz | A model for the estimate of local image velocity by cells in the visual cortex[END_REF]. More recently, computational models to compute local image velocity through a two-layer architecture corresponding to V1 and MT cortical areas have been proposed [START_REF] Simoncelli | A model of neuronal responses in visual area MT[END_REF][START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF].

In general, the pattern selectivity of MT cells can be explained by following two different approaches [START_REF] Bradley | Velocity computation in the primate visual system[END_REF][START_REF] Pack | 2.11 -cortical mechanisms for the integration of visual motion[END_REF]. In [START_REF] Bradley | Velocity computation in the primate visual system[END_REF], the authors consider the motion computation related to some kind of 2-D feature extraction mechanism. The consequence is that the aperture problem does not affect the motion processing, though few evidence for a feature-tracking mechanism are reported [START_REF] Stoner | Transparency and coherence in human motion perception[END_REF][START_REF] Noest | The role of early mechanisms in motion transparency and coherence[END_REF][START_REF] Skottun | Neuronal responses to plaids[END_REF]. In [START_REF] Pack | 2.11 -cortical mechanisms for the integration of visual motion[END_REF] which our neural model is based on, the authors use a non-linear integration of the V1 afferents to obtain the MT pattern cells: in particular, the intersection of constraints (IOC) mechanism is indirectly considered through localized activations of V1 cells [START_REF] Deangelis | Spatiotemporal organization of simple-cell receptive fields in the cat's striate cortex. II. Linearity of temporal and spatial summation[END_REF][START_REF] Simoncelli | A model of neuronal responses in visual area MT[END_REF][START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF].

However, this class of models, although conceptually valuable, has always been characterized by a high computational cost, as the price to pay for mimicking the hierarchical processing of the visual signal. Indeed, these kinds of neural solutions were developed as models of how the visual cortex works, and their functionality were demonstrated on simple synthetic sequences, but were never designed to form a systematic alternative to computer vision algorithms, working on real video sequences. Here, it is the first time that a neural model is chosen as a starting point and adapted so that it could be tested against modern computer vision benchmarks.

Model overview

A global overview of the model studied in this paper is given in Fig. 1. It is a three-step feedforward model: Step 1 corresponds to the V1 simple and complex cells, Step 2 corresponds to the MT pattern cells and Step 3 corresponds to a decoding stage to obtain the optical flow from the MT population response. In term of modeling, Steps 1 and 2 follow a classical view, while Step 3 has been introduced to solve the task of optical flow.

Let us consider a grayscale image sequence I(x, y, t), for all positions p = (x, y) inside a domain Ω and for all time t > 0. Our goal is to find the optical flow v(x, y, t) = (v x , v y )(x, y, t) defined as the apparent motion at each position p and time t. Details for each steps are given in the following sections.

Step 1 : V1 (Motion energy estimation and normalization)

In the V1-layer two sub-populations of neurons are involved in the information processing, namely V1-direction selective simple cells and complex cells. Simple cells are characterized by the preferred direction θ of their contrast sensitivity in the spatial domain and their preferred velocity v c in the direction orthogonal to their contrast orientation often referred to as component speed. The RFs of the V1 simple cells are classically modeled using band-pass filters in the spatio-temporal domain. In order to achieve low computational complexity, the spatio-temporal filters are decomposed into separable filters in space and time. Spatial component of the filter is described by Gabor filters h and temporal component by an exponential decay function p. Given a spatial size of the receptive field σ and the peak spatial and temporal frequencies f s and f t , we define the following complex filters by: h(x, y; θ, f s ) =Be

-(x 2 +y 2 )
2σ 2 e j2π(fscos(θ)x+fssin(θ)y) ,

p(t; f t ) =e (-t τ ) e j2π(ftt) , (1)
where σ and τ are define the spatial and temporal scales respectively. Denoting the real and imaginary components of the complex filters h and p as h e , p e and h o , p o respectively, and a preferred velocity v c related to the frequencies by the relation

v c = f t f s , (3)
we introduce the odd and even spatio-temporal filters defined as follows, These odd and even symmetric and tilted (in space-time domain) filters characterize V1 simple cells. Using these expressions, we define the response of simple cells, either odd or even, with a preferred direction of contrast sensitivity θ in the spatial domain, with a preferred velocity v c and with a spatial scale σ by

g o (x,
R o/e (x, y, t; θ, v c ) = g o/e (x, y, t; θ, v c ) (x,y,t) * I(x, y, t) (4)
The complex cells are described as a combination of the quadrature pair of simple cells (4) by using the motion energy formulation (Fig. 1, inset a),

E(x, y, t; θ, v c ) = R o (x, y, t; θ, v c ) 2 + R e (x, y, t; θ, v c ) 2 ,
followed by a normalization. Assuming that we consider a finite set of orientations θ = θ 1 . . . θ N , to obtain the final V1 response

E V 1 (x, y, t; θ, v c ) = E(x, y, t; θ, v c ) N i=1 E(x, y, t; θ i , v c ) + ε , (5)
where 0 < ε ≪ 1 is a small constant to avoid divisions by zero in regions with no energies which happen when no spatio-temporal texture is present. The main property of V1 is its tuning to the spatial orientation of the visual stimulus (Fig. 1, inset b), since the preferred velocity of each cell is related to the direction orthogonal to its spatial orientation.

Step 2: MT pattern cells response

MT neurons exhibit velocity tuning irrespective of the contrast orientation. This is believed to be achieved by pooling afferent responses in both spatial and orientation domains followed by a non-linearity [START_REF] Simoncelli | A model of neuronal responses in visual area MT[END_REF]. The responses of an MT pattern cell tuned to the speed v c and to direction of speed d can be expressed as follows:

E M T (x, y, t; d, v c ) = F N i=1 w d (θ i )G σ pool x,y * E V 1 (x, y, t; θ i , v c ) , (6)
where G σ pool denotes a Gaussian kernel of standart deviation σ pool for the spatial pooling on the neighboring locations, F (s) = exp(s) is a static nonlinearity chosen as an exponential function [START_REF] Paninski | Maximum likelihood estimation of cascade point-process neural encoding models[END_REF][START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF] (Fig. 1, inset c), and w d represents the MT linear weights that give origin to the MT tuning (Fig. 1, inset d). The physiological evidence suggests that w d is a smooth function with central excitation and lateral inhibition. Cosine function shifted over various orientations is a potential function that could satisfy this requirement to produce the responses for a population of MT neurons [START_REF] Maunsell | Functional properties of neurons in middle temporal visual area of the macaque monkey. I. selectivity for stimulus direction, speed, and orientation[END_REF]. Considering the MT linear weights shown in [START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF], w d (θ) is defined by θ:

w d (θ) = cos(d -θ) d ∈ [0, 2π[. (7)
This choice allows us to obtain direction tuning curves of pattern cells that behave as in [START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF] (see illustration in Fig. 1, inset e).

Step 3: Velocity estimation

Given the target of this approach to be an estimation of dense optical flow, the velocity vector has to be obtained by decoding the population responses of the MT neurons. Indeed, a unique velocity vector cannot be recovered by activity of a single velocity tuned MT neuron as multiple scenarios could evoke the same activity, but unique vector can be recovered based on the activity of a population. In this section, we present a decoding step which was not present in [START_REF] Simoncelli | A model of neuronal responses in visual area MT[END_REF][START_REF] Rust | How MT cells analyze the motion of visual patterns[END_REF] to decode the MT population.

Inria

The velocity space could be sampled by considering MT neurons that span over the 2-D velocity space with a preferred set of tuning speed directions in [0, 2π[ and also a multiplicity of tuning speeds. Sampling the whole velocity space is not required, as a careful sampling along the cardinal axes could be sufficient to recover the full velocity vector.

In this paper, we sample the velocity space using two MT populations tuned to the directions d = 0 and d = π/2 with varying tuning speeds. Here, we adopt a simple weighted sum approach to decode the MT population response [START_REF] Rad | Information rates and optimal decoding in large neural populations[END_REF].

v x (x, y, t) = M i=1 v c i E M T (x, y, t; 0, v c i ), v y (x, y, t) = M i=1 v c i E M T (x, y, t; π/2, v c i ). (8)
Note that other decoding methods exist such as, e.g. the maximum likelihood estimator [START_REF] Pouget | Statistically efficient estimation using population coding[END_REF][START_REF] Pouget | Information processing with population codes[END_REF], however we have adopted the linear weighted sum approach, as a balancing choice between simplicity, computational cost and reliability of the estimates.

Extended approaches

As it will be shown in Sec. 4, the results obtained with the baseline model suffer from two main problems which are a direct consequence of the model choice. Resulting optical flow can be noisy in areas with smooth speeds, and discontinuities are lost since there is no mechanism in the baseline model to tackle these issues.

To reach a better performance, we propose to introduce a nonlinear filtering stage at the level of MT responses. Two kinds of informations can be taken into account to smooth the resulting optical flow while preserving its discontinuities. The first is directly based on the MT response and the idea is to prevent diffusion where MT response appears to be discontinuous in space. The second takes into account luminance similarity to gate diffusion, since in most real world sequences, motions discontinuities are observed at objects boundaries (and thus luminance discontinuities). Here, we propose solutions based on classical non-linear filters used in computer vision: bilateral or trilateral filtering. Note that these solutions do not have any direct biological interpretation. Although some analogies could be thought in term of functional principles with motion contrast cells in MT and some V2/V3 neurons, it is beyond the scope of the paper to give a truly and rigorous bio-inspired modeling of these mechanisms.

Bilateral filter solution (BF)

The first solution is based on the bilateral filter (BF) which can be interpreted as an extension of the linear Gaussian filtering. This method has been extensively used in the context of image smoothing leading to many applications (see [START_REF] Paris | Bilateral filtering: Theory and applications[END_REF] for a review). Given an image to be smoothed, the idea of BF is to consider that two pixels are close to each other not only if they occupy nearby spatial locations but also if they have some similarity in the values range. As a consequence, the interest of BF is that it allows to smooth an image while keeping its discontinuities.

In our context, we propose to apply BF on E M T (x, y, t; d, v c ) responses (at each time t and for each d, v c independently). Denoting E M T (x,y,t;d,v c ) by E M T (p) where p = (x, y) for sake of simplicity, the bilateral filter is defined by

BF α,β E M T (p) = 1 N (p) p ′ ∈Ω f α ( p -p ′ )f β (E M T (p ′ ) -E M T (p))E M T (p ′ )dp ′ , (9)
where α and β are two parameters determining the smoothing property and

f µ (s) = exp(s 2 /µ 2 ) s ∈ R, (10)
and N (p) is the normalizing term

N (p) = p ′ ∈Ω f α ( p -p ′ )f β (E M T (p ′ ) -E M T (p))dp ′ ,
Equation ( 9) can be interpreted as a nonlinear weighted sum of the values of E M T taken at positions p ′ close to p and such that values E M T (p ′ ) are also close to E M T (p). By doing so, the resulting filtered energy BF α,β E M T is smoothed while main discontinuities are preserved and enhanced. Several iterations of this filter can be made depending on the degree of smoothing desired.

Trilateral filter solution (TF)

A direct extension of ( 9) is to add an extra weighting term which depends on luminance similarity f γ (I(p ′ ) -I(p)). The proposed trilateral filter is then defined by

T F α,β,γ E M T (p) = 1 N (p) p ′ ∈Ω f α ( p -p ′ )f β (E M T (p ′ ) -E M T (p))f γ (I(p ′ ) -I(p))E M T (p ′ )dp ′ , (11)
where α, β, γ are parameters and N (p) is the normalizing term

N (p) = p ′ ∈Ω f α ( p -p ′ )f β (E M T (p ′ ) -E M T (p))f γ (I(p ′ ) -I(p))dp ′ .
The result of Eq. ( 11) is that the difference of luminance between I(p) and I(p ′ ) will be also taken into account when considering E M T (p ′ ). This luminance gated mechanism have some similarities with the former bio-inspired models [START_REF] Tlapale | Modelling the dynamics of motion integration with a new luminance-gated diffusion mechanism[END_REF][START_REF] Ringbauer | Mechanisms of adaptative spatial integration in a neural model of cortical motion processing[END_REF] although modeled differently here.

3 Algorithmic details: How to make the approach applicable to real sequences

The classic V1-MT model presented in Sect. 2 is primarily aimed at explaining the recorded neural and perceptual data on largely homogeneous synthetic images such as moving gratings and plaids. As our objective is to benchmark it on real sequences, we need to introduce algorithmic solutions to address several problems coming from the nature of real sequences but also from our model.

Multiscale approach

As detailed in the Section 2, the V1-like RFs are modeled through spatio-temporal filters. In order to keep as low as possible the computational load of the model, only one spatial radial peak frequency f s has been considered. This is in contrast with the physiological findings, and since information in natural images is spread over a wide range of frequencies, it is necessary to use a mechanism that allows us to get information from the whole range.

In this paper, a multi-scale processing (i.e., considering units tuned to different spatial peak frequencies) is implemented (Fig. 2). This is a classical approach used in computer vision. It consists in (i) a pyramidal decomposition [START_REF] Bergen | Pyramid methods in image processing[END_REF] with L levels and (ii) a coarse-to-fine refinement [START_REF] Simoncelli | Course-to-fine estimation of visual motion[END_REF], which is a computationally efficient way to take into account the presence of different spatial frequency channels in the visual cortex and their interactions. The optical flow at a finer scale Inria (S l ) is given by v l = δv l + v l+1 . Using this approach, the spatial distance between corresponding points is reduced, thus yielding to a more precise estimate, since the residual values of the velocities lie in the filters' range. This also allows large displacements to be estimated which is a crucial aspect when dealing with real sequences. Interestingly, at a functional level, there is an experimental evidence that MT neurons seems to follow a coarse-to-fine strategy [3] suggesting that motion signals become more refined over time.

Boundary conditions

The problem of boundary conditions arises as soon as we need to consider values outside the domain of definition Ω. Even with a simple Gaussian smoothing, when estimating results close to the boundaries, one needs to access values outside Ω. This is solved generally by choosing some boundary conditions like Neumann or Dirichlet. However, in our case, using such assumptions might introduce some strong errors at the boundaries. For this reason, we proposed instead to work on an inner region in which only available values are taken into account so that no approximation or assumption has to be made, and then to interpolate values in the remaining outer region. Note that this is an important issue to consider, especially because we use a multiscale approach since errors done at the boundaries at low scales can spread a lot as scales is getting finer.

The elements allowing to define the inner region Ω in are illustrated in Fig. 3(a). Region B 1 is first excluded so that V1 cell responses use only given values. Region B 2 is then excluded because of the pooling operation from V1 to MT. Given this definition of inner and outer regions (Fig. 3(b)) the idea is to make all the estimations in Ω in and to interpolate values in the outer region Ω out (Fig. 3(c)). So, given E M T estimated in Ω in , we propose that

E M T (p) = 1 N (p) p ′ ∈A f α ( p -p ′ )f γ (I(p) -I(p ′ ))E M T (p ′ )dp ′ ∀p ∈ Ω out , (12)
where A contains pixels at the boundary and inside Ω in (green region), function f µ is defined as in [START_REF] Heeger | Optical flow using spatiotemporal filters[END_REF], α and γ are parameters and N (p) is a normalising term

RR n°8618 (a) (b) (c) (d)
N (p) = p ′ ∈A f α ( p -p ′ )f γ (I(p) -I(p ′ ))dp ′ .
This method is based on luminance similarities, using the same idea as developed in Sec. 2.3. Note that other methods could be proposed such as more elaborated interpolation methods (like PDE-based inpainting methods) to further improve the quality of the results.

Unreliable regions

A problem is found with regions having a null spatio-temporal content, which is happening in the blank wall problem for example. In that case, locally, it is then not possible to find a velocity. Given a threshold T , a pixel (x, y) will be categorised as unreliable if and only if E M T (x, y, t; d, v c ) < T for all d and v c . For these pixels, the same interpolation as ( 12) is proposed (Fig. 3(d)).

Results

Parameters settings

Table 1 provides parameters used in our simulations.

Let us discuss some of the main parameters. The choice of the size of the spatial support of the RF is related to the necessity of processing fine details, i.e. at high image resolution, in realworld sequences. The temporal support of the filter is due to two different needs: to obtain good results on the benchmarking sequences, and to meet the experimental evidence. In particular, the V1 and MT RFs process the visual signal within a restricted epoch of time, we can consider an average period of 200 ms [START_REF] Deangelis | Receptive-field dynamics in the central visual pathways[END_REF]3]. Such a time period corresponds to about 5 frames for a standard video acquisition device. Such choices allows us to obtain good performances in terms of the execution time, though they do not allow us to have tuning to higher velocities than 1 Inria pixel/frame (i.e. the sampling constraints). The issues is addressed by considering a multi-scale approach, as explained in Sec. 3.1, the number of the considered scales depends on the size of the input images (and also on the speed range, though it is not a priori known), for the Middlebury sequences it has been fixed to 6 spatial scales. It is worth noting that to avoid the introduction of a loss of balance between the convolutions with the even and odd Gabor filters, the contribution of the DC component is removed [START_REF] Clausi | Designing Gabor filters for optimal texture separability[END_REF]. Finally, the support of the spatial pooling G σ pool is chosen in order to take into account the larger RFs of MT layer with respect to V1 layer, in particular the factor is 5, which is in accordance with findings reported in literature [START_REF] Albright | Local precision of visuotopic organization in the middle temporal area (MT) of the macaque[END_REF][START_REF] Bayerl | Disambiguating visual motion through contextual feedback modulation[END_REF].

All results presented here could be reproduced using our code available publicly.

Analysis of proposed approaches

In this section, we evaluate the baseline model and it extensions referred by BF (Sec. 2.3.1) and TF (Sec. 2.3.1), by using some test sequences to show the intrinsic properties of our approach and to show the influence of some parameters. When ground truth optic flow is available, average angular error (AAE) and endpoint error (EPE) will be estimated (with associated standard deviations) [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF].

L = 1 L = 3 L = 5
Baseline Sample input The influence of the number of spatial scales is shown in Fig. 4. In this sequence a dashed bar moves rightward with velocity (2,0) pixels. Focusing first on the baseline method, results show that increasing the number of scales improves the results. It is worth noting that the aperture problem is correctly solved by considering 3 spatial scales in the small segments, whereas 5 spatial scales are needed to handle longer segments, though a residual optic flow at the finest scale is not correctly recovered in the middle of the longest segment, since the spatial support of the RFs is too small with respect to the visual feature. This issue is partially solved by the trilateral filter, which has the effect of diffusing further the estimates along the bar, which slightly improves the results.

The effects of the bilateral and of the trilateral filter applied at the MT level, with respect to the baseline approach, are further shown in Fig. 5. To do so, we considered a synthetic sequence that represents a textured translating shape moving with velocity (-3,-3) pixels on top of a translating background with velocity (4,0) pixels. Two situations are tested: In input 1, there is no brightness difference between background and foreground (only motion can reveal the object); In input 2 there a brightness difference. Ground truth for both are the same. Optical flows and error measurements confirm the improvement in optic flow estimation by considering the extended approach (in particular TF). The improvement is higher in the case of input 2 where the boundaries of the foreground shape are better preserved. Nevertheless, it is worth noting that more reliable optic flow estimates in the inner part of the background are obtained when no brightness differences are present. This is due to the fact that the multi-scale approach might create artefacts at low-resolution scales, in particular on sharp boundaries. Such artefacts produce wrong optic flow estimates at coarser spatial scales, which propagate at finer spatial scales through the warping stage.

In order to better analyze the roles of the different stages of the model, Fig. 6 shows the V1 and MT activities, by considering the synthetic sequence shown in Fig. 5. For V1, two maps are shown:

E V 1 θ (x, y, v c ) = N i=1 E V 1 (x, y; θ i , v c ) 2 1/2 , E V 1 v c (x, y, θ) = M i=1 E V 1 (x, y; θ, v c i ) 2 1/2
, and for MT, the two populations E M T (x, y, v c , 0) and E M T (x, y, v c , π/2) are shown for different v c . These results confirm show that the V1 layer has a tuning on the spatial orientation (thus the cells are elicited by the spatial orientation of the shape), whereas at MT layer, a speed tuning ) and E M T (x, y, v c , π/2) maps, respectively. At MT layer, a speed tuning emerges: on the left, the energies are higher for the region related to the shape, this means that there is a negative speed for the horizontal and vertical velocities related to the shape. On the right, the energies are higher for the background (for the third row, only)), since the background moves rightwards.

v c 1 v c 2 v c 3 v c 4 v c 5 v c 6 v c 7 E V 1 θ (•, v c ) θ 1 θ 2 θ 3 θ 4 θ 5 θ 6 θ 7 θ 8 E V 1 v c (•, θ) v c 1 v c 2 v c 3 v c 4 v c 5 v c 6 v c 7 E M T (•, 0) E M T (•, 0)
no more related to spatial orientation emerges (i.e., the aperture problem is solved). Another representation consists in showing the distribution of E M T at different positions to understand its relation to velocities (Fig. 7). By observing the behaviour of MT energies in four different positions on the frame (highlighted by (a), (b), (c) and (d) in the figure), it is possible to note how the MT layer encodes the velocities. In particular: the behaviours in (a) and (c) are affected by the values of the neighboring borders, thus there are no prominent activities; in (b), which corresponds to a point on the foreground shape, cells tuned to negative speeds (v c 1 ) on both horizontal and vertical direction (E M T with d = 0 and d = π/2, respectively) have the maximum response; in (d), which corresponds to a point on the background, only the response of the horizontal direction has a maximum for positive horizontal speed (v c 7 ). Finally we give a comparison between the baseline, BF and TF approaches on the classical realistic Yosemite sequence, without and with clouds (see Fig. 8). Both the maps representing computed optic flows and error measurements show a consistent improvement using TF: Optical flow is smoother with less artefacts. The average angular error on the sequence without clouds, by considering the extended approach, is comparable with many of the approaches present in Middlebury evaluation table 2 . In the sequence with clouds, one can also notice the better preservation of discontinuties at some locations between the mountain and the sky.

Inria Sample input (a) (b) (c) (d)
Sequence

Performance evaluation on Middlebury dataset

In this Section, we benchmark our approach by using the modern computer vision datasets Middlebury [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF] 3 . In Fig. 9 and Table 2, we show qualitative and quantitative results obtained on the training dataset, which has public available ground truth. The sequences of this dataset have several challenges, such as sharp edges, high velocities and occlusions. Our model could fail in these situations essentially due to the support of the temporal filter: A small support of the temporal filters constrains the maximum detectable component velocity at each scale, thus the multi-scale approach is necessary in order to estimate higher velocities. On the other hand, the multi-scale approach creates artefacts at sharp edges, though the trilateral filter mitigates this issue. A large support of the temporal filters creates problems in occlusion regions.

Since Urban2 sequence is quite challenging, due to the presence of both occlusions and high velocities, we decided to consider it, in order further analyze the proposed approach by varying some parameters of the model. In particular, we tested the effects of the size of the temporal support T S, the temporal decay τ and the number of tuning speeds. Figure 10 shows how the choice of these parameters affects the optical flow computation. The last column corresponds to the parameters chosen for the final evaluation on the Middlebury test set.

The optic flows computed on the evaluation Datasets of Middlebury have been submitted for evaluation 4 . Figure 11 shows the computed optic flow and the error maps, from which it is possible to note that higher errors are in correspondence of occlusions (wee, e.g., Urban sequence) and sharp edges (see, e.g., Urban and Wooden sequences). A screenshot of the Middlebury table is presented in Fig 12, from which we can see how our approach performs compared to other state-of-the-art algorithms. It is worth noting that our V1-MT-FF model is the only neural model for motion estimation present in the table.

Conclusion

In this paper, we have presented a method that is based on mathematical model primarily developed to account for various physiological findings related to motion processing in primates. Starting from the classical hierarchical feedforward processing model involving V1 and MT cortical areas which is usually limited to a single spatial scale, we have extended it to consider the whole visual field by adapting a multi scale approach and analyzed the efficacy of the approach in estimating the dense optical flow in real world scenarios by considering an efficient velocity decoding step.

We have tested the performance of our model using various synthetic stimuli as well as the standard Middlebury database. A qualitative evaluation indicates that model could recover since the V1-MT feedforward model does not take into account the details of lateral interactions and scale space issues that need to be tackled in order to solve the blank wall problem. In order to address this issue, we have incorporated as an extension to the baseline approach bilateral and trilateral filtering, which could potentially simulate lateral interactions at the MT level in order to obtain better dense estimations. Incorporation of trilateral filtering has significantly improved the estimation of dense optic flow maps, which reinforces the need to consider lateral interactions and possibly feedback into the models to better account for natural image processing scenarios, going beyond the homogeneous synthetic stimuli such as grating and plaids that are traditionally used in experimental biology. We think that this work could act as a good starting point for building scalable computer vision algorithms for motion processing that are rooted in biology.

This study has shown that a V1-MT feedforward model, together with the improvements described in this paper, can be successfully used to compute optic flow also in modern datasets. Moreover, it has opened up several interesting sub problems, which could be of relevance to biologists as well, for example what could be afferent pooling strategy of MT when there are multiple surfaces or occlusion boundaries within the MT receptive field? Can a better dense optical flow map be recovered by considering different multi-scale strategies? These questions are currently under consideration.

Inria

Figure 1 :

 1 Figure 1: Baseline model overview: It is a three-step feedforward model where the first step corresponds to the V1 layer (motion energy estimation and normalization), the second step corresponds to an MT layer (motion energy pooling and non-linearity) and the third step is velocity estimation (decoding). Insets show different properties of each step.

Figure 2 :

 2 Figure 2: How the multi-scale processing works: Starting from the pyramidal decomposition S l (l = 0 . . . L -1) (l = 0 is the finer scale), the velocities obtained at a coarser level of the pyramid ( δv l ), are expanded (v l ) and used to warp the sequence of the spatially filtered images, then the residual optic flow is computed ( δv l-1 ).

Figure 3 :

 3 Figure 3: Illustration of the filling-in approach used to deal with boundary conditions and the unreliable regions. (a) How inner domain is defined taking into account V1 filter spatial size and V1 to MT pooling; (b) Image domain showing the inner region Ω in in grey where exact calculations (i.e., without any approximation) can be done, the outer region Ω out where an interpolation scheme is chosen, and an example of unreliable region explained in (d); (c) Illustration of the interpolation scheme for a pixel p ∈ Ω out , showing the spatial neighbourhood associated with the spatial support of the integration and in green the region A which is used to estimate the interpolated values; (d) Same as (c) but in the case of an unreliable region.

Figure 4 :

 4 Figure 4: Influence of the number of scales. Baseline and TF approaches are tested with L=1, 3 and 5 scales.

Figure 5 :

 5 Figure 5: Comparison between the baseline, BF and TF approaches on a synthetic scene (a translating shape (-3,-3) pixels moving on top of a translating background (4,0) pixels).

Figure 6 :

 6 Figure 6: Illustrations of V1 and MT activities on the synthetic case shown in Fig.5. The first row shows E V 1 θ (x, y, v c ): the activities do not identify specific tuning speeds, since all the spatial orientations are pooled in the norm and the tuning speeds are component speeds, i.e. they are orthogonal to the spatial orientation of the cell. The second row shows E V 1 v c (x, y, θ): the cells are elicited by the spatial orientation of the shape, the V1 layer shows a tuning on the spatial orientation. The third and fourth rows show E M T (x, y, v c , 0) and E M T (x, y, v c , π/2) maps, respectively. At MT layer, a speed tuning emerges: on the left, the energies are higher for the region related to the shape, this means that there is a negative speed for the horizontal and vertical velocities related to the shape. On the right, the energies are higher for the background (for the third row, only)), since the background moves rightwards.

Figure 7 :

 7 Figure 7: Distribution of MT energy at positions indicated in the sample input image

Figure 8 :

 8 Figure 8: Comparison between the baseline, BF and TF approaches on the classical realistic Yosemite sequence, without and with clouds.

Figure 9 :Figure 10 :

 910 Figure 9: Sample results on Middlebury training set

Figure 11 :

 11 Figure 11: Sample results on Middlebury test set

Table 1 :

 1 Parameter values used in our simulations. Equation number refers to the equation where it has been first introduced.

	1

Table 2 :

 2 Error measurements on Middlebury training set

	AAE ± STD EPE ± STD
	grove2 4.72 ± 10.02	0.33 ± 0.62
	grove3 10.65 ± 19.09 1.40 ± 1.92
	Hydrangea 6.23 ± 11.25	0.80 ± 0.99
	RubberWhale 8.18 ± 14.02	0.28 ± 0.44
	urban2 13.37 ± 19.48 1.42 ± 1.45
	urban3 18.82 ± 40.21 2.01 ± 3.10

RR n°8618

The link will be given, upon acceptance of the paper.RR n°8618

http://vision.middlebury.edu/flow/eval/results/results-a1.php RR n°8618

http://vision.middlebury.edu/flow/data/

http://vision.middlebury.edu/flow/eval/results-manuela-chessa/results-a1.php Inria

Inria

route des Lucioles -BP 93 06902 Sophia Antipolis Cedex Publisher Inria Domaine de Voluceau -Rocquencourt BP 105 -78153 Le Chesnay Cedex inria.fr ISSN 0249-6399

Acknowledgments

KM and PK acknowledge funding from the EC IP project FP7-ICT-2011-8 no. 318723 (Math-eMACS)