Mylène Tahar

Antoine Tran

Fabien Remond

Nicolas Renard

Vincent Rapp

Simon Fauchard

Vladyslav Gorbatiuk

Corentin Lacroix

Sergio Rodriguez

Aleksandr Liadov

Qiang Zhang

Denis Giri

Martin De Gourcuff

Gauthier Rousseau

Mathieu Babel

Romain Lozachmeur

Marie-Amélie Cousin

Nicolas Texier Le Roi Des Connemaras

Eric Si- Mon

Martin Liné

Pierre-Henri Oréfice

Adrien Ma- Tricon

Clément Masson

Céline Craye

Pauline Chevalier

Antonin Raffin

Gennaro Raiola

Yvon Ker- Doncuff

René Traoré

Ashley Hill

Hugo Caselles

David Iacob

Moad Kissai

Roxana Agrigoraie

Natalia Diaz

Timothée Lesort

Julien Alexandre

Dit Sandretto

Olivier Mullier

Thibault Toralba

Florence Carton

Alexan- Dre Chapoutot

Frehse Goran

Ce document clôt trois ans (et demi!) de thèse pendant lesquels j'ai pu travailler à la fois au sein du laboratoire U2IS de l'ENSTA et de l'équipe Computer Vision de Parrot. Cette thèse, dans le cadre d'une convention CIFRE a pour moi été très fructueuse, et je suis très fier d'avoir participé à ce projet.

Je remercie ainsi premièrement les gens sans qui le projet de thèse n'aurait pas pu se faire, à savoir l'entreprise Parrot, à travers Benoît Pochon et Laure Chevalley, l'ENSTA, à travers David Filliat et Antoine Manzanera, et enfin l'ANRT grâce à qui j'ai pu profiter de 4 mois de vacances forcées en attendant leur accord pour le début du contrat CIFRE. Je remercie d'autant plus Antoine, David et Laure qui m'ont accompagné et dirigé durant cette thèse, en réalisant le tour de force de s'intéresser à mes travaux, s'y investir tout en me laissant particulèrement libre dans mes choix de recherche. Merci à eux aussi pour leurs nombreuses relectures d'articles et de ce manuscrit. Cette thèse, c'est un peu la votre aussi! Dans la suite logique, après qui ont permis à cette thèse de commencer, merci à ceux qui ont permis son dénouement, en constituant mon jury de thèse. Merci donc à Patrick Perez et Nicolas Thome d'avoir accepté d'être mes rapporteurs, et merci à Pascal Monasse et Yann Gousseau d'avoir accepté d'être examinateurs. Vos différentes remarques et objections ont permis à ce manuscrit de grandement gagner en qualité, en lisibilité et en rigueur scientifique.

Je remercie également les gens avec qui j'ai eu l'occasion de travailler de près ou de loin durant ces trois ans, à Parrot et à l'ENSTA, grâce à qui j'ai pu garder un pied dans la recherche et un pied dans l'industrie, et que je peux compter non seulement dans mes collegues mais aussi dans mes amis.

Je remercie d'ailleurs tout particulièrement à Parrot:

• Léa Vauchier et Alexandre Briot, avec qui la meilleure équipe spécialisée en machine learning au monde a été créée en 2016 • Louis-Joseph Fournier et Edouard Leurent, qui ont dévelopé la premiere version de l'algorithme d'évitement d'obstacle à partir d'une profondeur et Aurélien Barre, qui a développé la librairie de streaming Vidéo à Parrot. C'est grâce à eux que la preuve de concept finale a pu être mise en place, et que la boucle a pu être bouclée.

List of Figures

2 [-1, 1, 1]: the camera goes down left and forward, field of view is 90 • . First row: presentation of θ t and theoretical results on this perfectly smooth depth map. Notice the foreground is also occluded because it goes out of field of view. Second row: presentation of on ν, a noisy equivalent of θ t , with a Gaussian noise with std of 0.1 ν t = θ t + N (0, 0.1). Result of direct warping (center) and filtered depth map with index back method. Last row: warp back of warped depth ν t→t+1 , and filtered depth map using the warp back method, followed by an erosion of 2 pixels. . . . . ix 4.11 Our two toy problems "Flower" (top) and "Horizon" (bottom). For each scene, first row are the three frames of the scene, and second row is the ground truth depth of the center frame. Images are 200 × 200 pixels. Scenes have no rotations. Flower scene has a displacement of 1 10 [1, 1, 0] (camera goes right and down, so the image goes left and up), Horizon scene has a displacement of 1 10 [0, 0, 2] (camera goes forward). Camera is a standard centered pinhole with 90 • of field of view. For optimization depth starting points are equal to background, i.e. 10m . . . . . . . . . . . 4.12 First row: Illustration of optical flow construction, and how a perfect depth produces imperfect warping. A "ghost" of the foreground can be seen on occluded areas, which the reprojection based optimization will wrongly try to solve. Second row, our two imperfectly warped frames from I t-1 and I t+1 to I t . Last row, the differences between warped images and I t are the occlusion areas. We can see that the non-zero areas don't cover the same pixels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 optimization results when only photometric loss is applied for depth optimization. Mean logarithmic error is worse at the end than at the beginning for both cases. For each scene, from left to right and top to bottom: depth result, difference with ground truth, final inverse warp of x 2 +y 2 +C used for l and c in SSIM (here, C is 0.01 2 ). This function is only convex around its maximum, which make its gradient based optimization difficult . . . . . . . . . . . . . . . . . . . . . and our method on a small UAV dataset. . . . . . . . . . . . . . . . . . . . . 5.4 Some failure cases of our method on KITTI. First column is a detail of a larger image. The foreground car is moving forward and it's detected as far away, while the background car is moving toward us and is detected as close. Second column is a poorly textured road. . . . . . . . . . . . . . . . 5.5 Comparison between our method and SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] on a normal example from KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] and its upside-down variation. . . . . . . . . vices. Batch sizes are 1 and 8 (when applicable). A batch size of 8 means 8 depth maps are computed at the same time. DeMoN was tested only on a 980Ti. On the second table, a quick comparison between tested GPUs . 3.7 Quantitative comparison between the three methods. FlowNetS → depth row is deducing depth from optical flow and displacement (perfectly known)

4.1 Best hyper parameters found and resulting quality indicators, when occlusion detection module (ODM) is active and when it is not. Red measures are to maximize, blue ones are to minimize . . . . . . . . . . . . . . . . . .

Quantitative tests on KITTI[Gei+13]

Eigen split [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF]. Measures are the same as in Eigen et al. [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF]. For blue measures, lower is better, for red measures, higher is better. For training, K is the KITTI dataset [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF], S is the Still Box dataset. For scale factor, GT is ground truth, P is pose.

When scale was determined with pose, we discarded frames where GPS uncertainty was greater than 1m.

Mathematical conventions

Throughout this thesis, unless specified otherwise, mostly for adequation with litterature notation, we take the following notation convention:

• Scalar will be written as simple lowercase.

• Vectors will be written bold and lowercase. For example p = (x, y). An exception can occur, for ease of notation when considering a 3D point P and its projection in 2D p.

• Scalar fields will be written as simple uppercase. Exception for depth and inverse depth map, called θ and ξ to keep consistency with literature. For example depth map:

θ(p) = z, p ∈ R 2 z ∈ R
• Vector fields will be written bold and upper case. For example optical flow F, or multi channel image

I(p) = c, p ∈ R 2 c ∈ R 3
• Matrices will be written bold, uppercase and with a straight font. For example M ∈ M 2,2 (R)

• Normalized values will be added a circumflex. It's only applied on vector or scalar fields for which we can compute spatial mean µ and variance σ. For example I is the normalized image

I-µ I σ I .
• Estimations will be added a tilde. Generally, it will be compared against a groundtruth. For example the depth map estimation θ compared with ground truth θ, or the warped image I 1 obtained from I 2 which aims at being the same as the original image I 1 .

• Spatial gradients of scalar fields will be noted with ∇. For example the gradient of depth θ is ∇θ(p = (x, y)) = ∂θ ∂x (p) ∂θ ∂y (p) .

• For sake of simplicity, we can mention the spatial gradient of a vector field (while we should talk about a Jacobian matrix). For example when talking about image spatial gradient ∇I, what is actually referred is the spatial gradient of each I channel.

• When the scalar field f is a result of a function taking a vector W for argument (for example a neural network with a set of weights W), we note ∇ W f (p) the gradients of f output at point p with respect to parameters W, with p fixed.

xiv

• Similarly, the Jacobian J W of a neural network with weights W outputting a scalar field f will be the function

J W : R n → R p p → ∇ W f (p) xv symbols
Here are presented typical symbols and acronyms for our notations. Even if they are specified when brought, this page can serve as a reminder for clarity. Although already well developed, the consumer UAV (Unmanned Aerial Vehicle) market is still said to show extremely high potential. Thanks to their simplicity of use, flying cameras democratized aerial footage. Today anyone can make a quality aerial footage without deep knowledge in aeronautics.

Acronyms

However, growing concerns on security of drone flights threatens the future growth of this market. Indeed, a consumer drone crashing can be deadly if falling from 100 meters high or more. This is why consumer drone regulations has strengthened by restricting the usage of flying cameras to certified pilots in isolated areas. Thus the enforced regulations diminish the number of potential customers. If we want the democratization process to be complete, we need to gain popular trust by making flights as safe as possible.

Motivations of this thesis

This work is motivated by improving security for consumer flying cameras by studying the obstacle avoidance possibilities for a UAV.

Anatomy of a consumer Drone

The application context of consumer drones implies particular binding conditions:

1. Design is cost-oriented. Unlike with professional UAVs, our strategy for an obstacle avoidance system should also be cost-oriented to keep the product attractive for consumers.

2. The aircraft needs to be compact. The consumer UAV is encouraged by the law to stay under a particular weight limit (e.g. 800grams in France [Fra16]) so that the consumer does not have to register it.

3. Battery life should be as long as possible, with a typical limit around ∼ 30minutes.

An obstacle avoidance system in this context should not rely on heavy or expensive hardware, and should not require too much energy.

Besides, there are several elements that we can already consider reliable because they are vital for a UAV designed to get quality aerial footage. Improving them is not in the scope of this thesis and will be considered solved, at least for our needs.

1. High level commands. For the sake of accessibility, the commands are made as intuitive as possible. Consumer UAVs can usually be controlled with the 6 degrees of freedom, without the need to explicitly control the rotation speed of the motors. More specifically, a UAV can be assimilated to its reference frame for which we can control its rotation and speed.

2. High quality image stabilization. The need for high quality footage makes the frame stabilization an essential feature for flying cameras. As such, the IMU system embedded guarantees an error of orientation such that the frame stabilization, active or passive (see figure 1.1) typically has at worst ≈ 1 pixel, for sensors up to 4K.

If we choose work on downscaled images, we can consider the stabilization to be perfect.

3. Accurately calibrated intrinsic parameters. Most likely, frames are perfectly rectified from geometric distortion, with perfectly known optical center and focal.

However, drone speed estimation is often noisy. Indeed, it mostly relies on inertial sensors that actually measure acceleration and GPS signal, which measures position with a precision of several meters. This can be mitigated by other sensors such as a vertical camera, which will measure apparent displacement using odometry and deduce speed from the altitude. Consequently we will not be able to easily link speed vector with its projection on the frame plane, the so-called flight path vector in more advanced aircraft heads up displays (HUD). 1. Return To Home (RTH): this use-case is an automatic flight mode with the simple objective of joining a particular 3D point based on its coordinate, generally the point where the UAV took off initially. This is particularly useful for out-of-range scenarios where the user can no longer pilot the drone. In this scenario, the user does not necessarily want to get good footage quality. The priority is to safely retrieve the aircraft.

2. Automatic flight for footage, like e.g. Parrot's Flight Plan (figure 1.2 and Follow me or DJI's ActiveTrack (figure 1.3): these flight modes require the video to be aesthetically pleasing. Flight Plan is similar to RTH, but the drone has to join several way-points in a particular sequence. Follow Me and ActiveTrack need the UAV to follow the user via GPS or Visual tracking.

3. Manual flight: the user has the entire control of the flight. Obstacle avoidance in this case would be only a flying helper, to avoid too dangerous piloting choices while still following the desired direction as closely as possible. This mode is for example featured with DJI's APAS (Advanced Pilot Assistance Systems)2 .

For Manual flight and Flight Plan or Follow me, the good video quality requirements not only refer to image quality, but also cinematography, with a smooth video. As such, Figure 1.4: A "jerky" trajectory (red) and a smoother one (green) we want to avoid visible maneuvers as much as possible. This implies both the lowest acceleration and acceleration variation possible (as known as "jerk"). Figure 1.4 shows two trajectories regarding the avoidance of an obstacle when the initial desired trajectory was a straight line. The red one has a very high jerk and acceleration and is not very smooth, which would only be acceptable in the context of RTH, while the green one is much smoother. This is the type of trajectory we want for our obstacle avoidance system.

We can see, however, that the green trajectory is altered well before the red one. The trajectory modification decision thus needs to be planned early, and thus the obstacle to be sensed with a longer range than with the RTH context with the sole purpose of not crashing into the obstacle, regardless of the jerk.

Depth perception for navigation in robotics

Now that we have established our main navigation requirements for obstacle avoidance for consumer drones, we can study the available techniques and related works to achieve this goal.

The most classical approach for obstacle avoidance relies on two distinct parts, sense and avoid. The first one is focused on perception, and the second one on action. [START_REF] Yu | Sense and avoid technologies with applications to unmanned aircraft systems: Review and prospects[END_REF] The perception can rely on several dedicated sensors to get information on the scene structure. The goal is to get the distance of other objects relative to the aircraft. More specifically, given a particular direction, we want to compute a depth map. Just like camera compute a RGB (Red Green Blue) pixel map of what can be seen, the depth map is a pixel map with a metric for each pixel, corresponding to the parallel distance relative to the direction where the sensor is pointed at (hence the name depth, and not distance). Figure 1.5 illustrates how we can link image pixels with their depth values. The RGB picture here only serves illustration purpose and for navigation, knowing depth perfectly is sufficient for obstacle avoidance.

It can be noted that the depth map is not the information we want per se, but rather the position of possible obstacles with respect to the camera, for which a simple conversion can be done based on the angle of view of a particular point with its depth. However, the depth map format illustrates that for a given orientation, only one depth value is known (the depth of the closest object). We don't want to be able to know the distance of occluded objects, since this would require prior knowledge of the scene.

Considered sensors for depth perception

For a UAV, we can list several solutions for active depth sensing. Contrary to e.g. cameras, these sensors require the emission of a signal (light or sound), whose reflexion will be measured to deduce depth:

• LiDar (Light detection and ranging) is a rotating light emitter (usually a laser beam) for a receptor that will compute distance from the time for the emitted beam to to be received back. If the direction of the beam is perfectly known, the depth can be deduced. This system has been used for example to get the depth of the KITTI dataset [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF].

• structured light, democratized with the first Kinect, and already equipped in products such as DJI phantom 4. The principle is to project a light pattern from a light source and identify it from a camera with a shifted position. The depth is deduced with triangulation. [START_REF] Paul | Active, optical range imaging sensors[END_REF] • ToF Cameras [START_REF] Hansard | Time-of-flight cameras: principles, methods and applications[END_REF] (Time of Flight), similar to Lidar but with no moving parts.

• Ultra sound sensors, which uses the same technology as Lidar but with sounds of high frequency. This sensor is very cheap and is already available on almost all consumer drones.

For passive depth sensing, which only requires measuring incoming signal such as light, we can list two camera based solutions:

• Stereo Vision • Monocular Vision

Active sensors advantages and limitations

Active sensors typically require less complex algorithm to deduce depth from perceived signal: the complexity resides in its hardware conception which is can be done by a specialized provider. Indeed, the emitted signal can be made in a particular pattern so that its detection and identification from receiver is easy.

However, for all active sensors, having a suitable range for smooth obstacle avoidance will require a powerful signal and thus a potentially heavy and power expensive dedicated device, which would greatly lower autonomy, along with increasing weight and manufacturing cost.

Besides, devices based on light emission and reception will suffer greatly for outdoor contexts. Indeed, most of them rely on Infra-Red emission, which is a major light component of sun rays. The receiver will then suffer a lot of noise and will have its typical range greatly reduced. The solution might be to emit more powerful light, but (in addition to power consumption) this is potentially hazardous for the human eye 3Finally, once the sensor is integrated with fixed signal emission specifications, range is fixed and thus not flexible to UAV speed. As a consequence, there will be a capped speed above which the drone cannot sense obstacle ahead enough. This is even more problematic to keep smooth trajectories since it needs long-time ahead planning.

Vision based depth perception

An interesting alternative is to use a passive sensor, like the camera. The main problem of the image coming from the camera is its extremely low information per pixels, which only gives a RGB color estimate of a particular point. Unlike active sensors, depth deduction from image signal is much more complex because it's less closely linked with image color.

Computer vision is a science field aimed at extracting high level information from these rudimentary "color maps" that are pictures taken from a camera, exploiting not only colors but also spatial and temporal contexts, by working on neighboring pixels in frame sequences or with multiple cameras.

In our context, a depth algorithm from computer vision would be an ideal solution because the camera sensor is already available. Besides, thanks to the UAV's first functionality, the image quality already benefits from heavy engineering to make it visually pleasant, and thus with a high functioning range of luminosity. Calibration, auto-exposition and white balance are thus out-of-scope problems already solved for video quality purpose.

One of the main characteristics of computer vision is the analogy that we can make with human vision, which is known to be one of the most important human sense, taking up to half of our brain's capacity [START_REF] Bhavin | Orientation Maps of Subjective Contours in Visual Cortex[END_REF]. Indeed, one could argue that the camera system is already derived from the eye, by projecting light beams on a plane sensor. The task given for computer vision could then be interpreted as the one given to the human brain to extract critical information from retina receptors intensity of activation, similar to color in image pixels.

Vision based depth sensing algorithms can then rely either on multiple cameras or one camera.

Stereo vision

The first case is a well known setup with two cameras on a stereo rig and already studied [START_REF] Nalpantidis | Stereovision-based algorithm for obstacle avoidance[END_REF] and used on several existing consumer products for obstacle avoidance like DJI's APAS already mentionned, or Parrot SlamDunk (see figure 1.6). It is also bio inspired since it reproduces the binocular human view, exploiting parallax to deduce depth. Provided computer vision techniques are able to match corresponding points on both frames (which is already not an easy task, since images only provide color information), a simple triangulation can be applied to deduce the depth. However, just as with structured light, the range of such algorithm heavily depends on image resolution and the distance between the two cameras (called the baseline), parallax being proportional to the ratio baseline/depth. Cameras too close to each other will have trouble sensing long range depth, and stereo camera integration complexity increases with stereo baseline. Typical stereo based obstacle avoidance systems on consumer drones can't guarantee to work at speeds higher than 10m.s -1 , and at this speed they can only perform urgent braking.

Monocular vision

The second case tries to get depth information from a single moving camera, either from appearance or using structure from-motion-algorithms. An in-depth comparison of these techniques is done at chapter 2. This is the use-case we will try to cover, because even if it now requires solving motion in addition to depth, the hardware integration is much easier. It is also flexible to already existing integrated hardware as long as a camera is available. Namely, if a stereo rig is already integrated in a high-end consumer drone, depth from single frame can also be used. This use-case can also be considered bioinspired, especially in the context of stabilized video because rotation compensation is Figure 1.6: Example of a stereo camera placed on a UAV in order to compute a depth map (here, a Parrot Slam Dunk on top of a Bebop2) somewhat related to human vision and vestibulo-ocular reflex (VOR) [START_REF] De Nó | Vestibulo-ocular reflex arc[END_REF]. Our eyes orientation is not induced by head rotation, our inner ear among other biological sensors allows us to compensate parasite rotation when looking toward a particular direction.

Monocular Vision based navigation is actually already a well studied field, especially within the context of SLAM algorithms[Cad+16; MAMT15; KM07; Fu+14]. Prior knowledge w.r.t. scene is used to infer a sparse depth map with its density usually growing over time. These techniques are typically used with unstructured movement, produce very sparse point-cloud based 3D maps and require heavy calculation to keep track of the scene structure and align newly detected 3D points to the existing ones. Although tools like Octomap [START_REF] Hornung | OctoMap: An Efficient Probabilistic 3D Mapping Framework Based on Octrees[END_REF] can be used, SLAM is not widely used for obstacle avoidance, but rather for 3D scan, where the cinematography and smooth video requirements are clearly ignored. It could also be argued that when trying to avoid obstacle while following a flight plan, mapping will mostly contain information about past elements which are no longer useful.

Deep Learning Approach

For this work we chose to focus on deep learning methods for vision. The deep learning approach is thus briefly presented here along with motivations to use it.

A brief historic

Just like camera and computer vision, initial motivations for deep learning were bioinspired. The core idea is to develop a learning algorithm that would capitalize on experience to get better over time.

The first iteration, called the perceptron [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF] was just the simulation of a neural networks, with neurons firing an activation or not based on a simple arithmetic operation over its inputs, as presented figure 1.7. We can already see how this structure could be parameterized and perform very different operations based on it. 4 The learning dimension was further improved by making every operation differentiable. That way it was possible to compute the gradient of a network output regarding every one of its parameters thanks to the back-propagation algorithm. The learning strategy was then to define a loss function to minimize and then perform a gradient descent to match the desired output. [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF] If we define a neural network as a function f taking trainable parameters vector W (called weights), and an input vector I (e.g. a colored image), we can define the result of an inference to be f (W, I). The optimization of this neural network then consists in updating the vector W to get a desired output. More specifically, if we define a loss function L to evaluate the output with respect to a known target, and an evaluation set V of images I and target outputs t, we want to reach the global miminum:

1.3. Deep Learning Approach output(x 0 • • • x n ) = 1 s>α , s = ∑ i w i × x n
W f inal = arg min W E (I,t)∼V L( f (W, I), t)
V is the set of valid samples on which we want the network to perform, e.g. the set of all possible video frames from a micro UAV. The most standard way of finding this global minimum is to apply a stochastic gradient descent (SGD) [START_REF] Bottou | Large-scale machine learning with stochastic gradient descent[END_REF] on a set of training samples (I, t), separated from the validation set.

W ← W -λ∇ W L( f (W, I), t)
Where λ is called the step size and the gradient vector is computed with backpropagation algorithm. Throughout a series of samples I with a way to evaluate the network output, the weights W are updated multiple times to try to achieve minimum loss on the validation set.

It is thus important to note that the learning workflow is essentially based on loss gradient (rather than loss value). Designing a loss is not the same as designing an evaluation function. It should not be forgotten and will be an important topic in chapter 4.

In this very brief presentation, we don't cover all the variation of this very basic gradient descent designed to solve optimization problems such as local minimum or over-fitting [START_REF] Tetko | Neural network studies. 1. Comparison of overfitting and overtraining[END_REF], and throughout this document, we will mention and use common good practices to make a particular network converge to the desired estimator without extensive justifications.

Deep learning with convolutional neural networks for computer cision

Convolutional Neural Network (or CNN), a particular set of neural network designed by Fukushima et al. [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[END_REF] are widely used for computer vision problems with a lot of success [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. This architecture tries to extract information from pixels using operations specially designed to link neighboring pixels together instead of having a dense network. This method has proven extremely useful and scalable for many computer vision benchmarks where deep learning solutions using CNNs are now very competitive. This includes in addition to classification, optical flow [But+12], depth from stereo [START_REF] Scharstein | A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[END_REF], odometry [START_REF] Geiger | Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite[END_REF], or depth completion [START_REF] Uhrig | Sparsity Invariant CNNs[END_REF]. 5It's also interesting to note that, as shown by Krizhevsky when working on AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], the first operations done by the convolutional kernels are very similar to intuitive hand crafted features such as gradient extraction. Semantically, it shows that a CNN can be divided into two parts: the first one being representation learning, where a map of pixels is projected into a map of interesting features for a specified learned tasks, and the second one being the task solving per se.

Deep learning for decision making

Recently popularized by DeepMind work on Atari [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] or Go [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF], deep learning has been used with some success on decision making, showing that highly semantic tasks could be performed as well.

In 2006, LeCun et al. [START_REF] Muller | Off-road obstacle avoidance through end-to-end learning[END_REF] had already shown that a CNN could be trained to mimic a human behavior on a simple case of off-road driving. This idea was also applied on UAV in order to follow a specified road or pedestrian path [START_REF] Giusti | A machine learning approach to visual perception of forest trails for mobile robots[END_REF][START_REF] Loquercio | Dronet: Learning to fly by driving[END_REF]. While obviously not enough for true autonomous driving or flying, these works enlighten the control learning capacity with an easily optimized loss function (here, the difference between human controls and predicted ones).

On the other hand, the field of reinforcement learning [START_REF] John | Learning from delayed rewards[END_REF] showed the potential of a truly Neural network powered autonomous system only supervised by a simple sparse reward function (e.g. positive reward for not crashing, negative reward for crashing). Counter intuitive but efficient strategies can then emerge and beat human level, that would have been impossible with imitation learning.

However, these systems are known to be extremely hard to train, requiring extensive trial and error before being able to shape a meaning reward where no degenerate strategy can be found [START_REF] Henderson | Deep reinforcement learning that matters[END_REF].

As suggested with DaGGer by Ross et al. [Ros+13;[START_REF] Ross | A reduction of imitation learning and structured prediction to no-regret online learning[END_REF], a mid point could be human supervision, only when needed, as an emergency intervention. That way more exploration could be done compared to imitation while still having supervision when needed. Of course this needs the constant attention of a human supervisor which is very expensive.

Interest of deep learning for consumer UAVs

One could argue that similarly to dedicated hardware sensors, deep learning vision solutions may not be a good option because most of them are fairly heavy and require a powerful GPU to be run real-time.

It is however reasonable to expect work to be done on very light neural networks, and the development of specialized hardware that can be easily embedded in consumer products like micro UAVs.

Approach and goal of the thesis

Most importantly, for a product like a flying camera, multiple computer vision tasks are at stake at the same time, e.g. tracking a moving target for a Follow Me/ActiveTrack use-case, odometry or depth sensing. A recent work [START_REF] Amir | Taskonomy: Disentangling Task Transfer Learning[END_REF] showed that several tasks could be combined in the same network in order to perform only one feature extraction that would be useful for both purposes. It then seems reasonable to think about such an architecture as a supplementary motivation to use embedded deep learning.

Approach and goal of the thesis 1.4.1 Hardware setup

From last sections, we define a minimal hardware context on which any solution proposed by this thesis is supposed to be used.

1. High quality stabilized camera, roughly pointed at the drone displacement direction.

2. No active sensor available, no stereo cameras.

3. Possibility to run embedded CNNs for inference. No fixed computing power is specified, but compact networks will be preferred when possible.

Throughout this thesis, this hardware choice will be taken in count in the different design motivations.

Discarding the naive solution

When considering deep learning for the goal of smooth obstacle avoidance, we could be tempted to test an end-to-end solution that would take monocular images as input, and provide maneuver commands for a safe and smooth flight. This would challenge the traditional workflow of sense-and-avoid by combining the two tasks within the same network, and could potentially result in a very light and responsive solution, the same way an human would avoid obstacle, without the need to explicitly measure objects distance.

Unfortunately, it would be naive to attempt at training a network from scratch for such a task, because of the reward sparsity and the obvious risks for a potential mistake. It goes without saying that the crash and retry strategy which would be inevitable at least at the beginning of the training is not easily scalable, although it has been tested in cluttered spaces at low speed [START_REF] Gandhi | Learning to fly by crashing[END_REF].

One could also use a first training pass with imitation learning, as shown in the already mentioned works [START_REF] Giusti | A machine learning approach to visual perception of forest trails for mobile robots[END_REF][START_REF] Loquercio | Dronet: Learning to fly by driving[END_REF]. The idea was to construct a dataset for imitation learning in a way that would not involve any UAV flying, but rather humans following normal paths. Constructing an imitation dataset only required humans to walk or bike normally, making the dataset very easy to construct, and the drone learned to imitate the motion of a human head. Unfortunately, these works were only successful in the context of a simple task: following a clear path on the ground, and many use-cases where no ground can be seen will fail, rendering the pretraining useless.

In contrast, comprehensive datasets for imitation learning of obstacle avoidance are very hard and expensive to construct because it involves flying a drone near obstacles numerous times risking human error each time.

In this section we thus discuss a potential path to reach obstacle avoidance by setting intermediate pre-objectives, with easier supervision, in the hope of using reinforcement learning with an already highly functional system.

Training with increasing complexity

This problem can be analyzed along two complexity axes: the domain complexity, and the task complexity. Figure 1.8 presents several tasks. 1. For a given task, the domain can be simplified by using a simulator, where data annotated either with ground-truth or reward is very easy to construct. However, a simulator will necessarily differ from the more complicated reality, and an algorithm with good performances on simulator has chances of performing poorly in reality.

2. Task complexity represents in a sense the sparsity of the useful information for a neural network to learn. Namely, regression problem with ground-truth are deemed much easier to train because each network inference will trigger a loss and thus a weight correction. On the other hand, reinforcement learning systems will only have meaningful rewards once in a while and might take a lot of experience replay to be trained.

From this analysis we decided to consider four different tasks and domains:

• Depth Sensing in simulator. The "easiest" of them, a classic supervised algorithm can be used, this will be the subject of chapter 3.

• Depth Sensing in real scenes. Simple task but with expensive ground truth. A solution to overcome this is to use already known computer vision techniques to get a self-supervised training. This will be the subject of chapter 4.

• Obstacle avoidance in simulator using perfect Depth maps as input. Several avoid algorithms can be used for supervision if the depth is perfect, such as Model Predictive Control [START_REF] Richalet | Model predictive heuristic control[END_REF][START_REF] Brett | Aggressive 3-d collision avoidance for high-speed navigation[END_REF]. Crashing the camera multiple times is not a problem since everything is simulated.

• Obstacle avoidance in real life. This features all levels of complexity, where crashes need to be extremely rare, and constitutes our final goal. A first working solution can 1.5. Contributions use the concatenation of a depth sensing in real scenes and obstacle avoidance in simulator.

The General strategy can then be summarized by solving the simplest task and adding complexity later on. These tasks still heavily rely on the sense and avoid paradigm since the last network will be the concatenation of a depth sensing and obstacle avoiding from depth. However, it will be possible to train a more compact network in a supervised way to get the same results. This technique is called knowledge distillation. [START_REF] Romero | FitNets: Hints for Thin Deep Nets[END_REF][START_REF] Hinton | Distilling the Knowledge in a Neural Network[END_REF]. These milestones are particularly interesting because each of these tasks if solved can be easily used in a classic sense-and-avoid system for a working solution even if the last step is not reached yet. This will be discussed in chapter 6.

Resulting scope of this thesis

The scope of this thesis does not cover all the possible milestones for an end-to-end obstacle avoidance neural network. Instead, we heavily focused on two tasks, by trying to conduct the most exhaustive study possible, while keeping the end goal of obstacle avoidance.

These two tasks are the depth sensing algorithms, inside a simulator and on real video footage, with the possible inclusion in a sense and avoid pipeline.

As such, we won't heavily study control technique for obstacle avoidance. Paradoxically, even if the long term end goal is to replace the sense-and-avoid paradigm to an end-to-end one, the main goal of the thesis will be to develop a reliable depth sensing solution. The knowledge distillation strategy in the end is what makes this strategy still relevant.

Contributions

This work led to two publications in international conferences and one publication to an international workshop, the two conferences are related to robotic navigation, and the workshop is related to computer vision and geometry.

Outline of this thesis

In addition to this introduction, this thesis is divided in five parts:

• chapter 2 will try to define an evaluation method for depth sensing, with obstacle avoidance in mind. The goal is to discuss current literature on this matter, with its applications and limitations. More specifically, we will see that current benchmarks for depth evaluation might not be suited for the flying camera context, and resulting best strategies for them, using depth from context rather than structure from motion are not ideal for us. In the last section, we will be interested in the amount of information a particular evaluation of an estimator gives on future estimations in the wild.

• chapter 3 will discuss the possibility for a CNN to deduce depth from a pair of stabilized frame. This is the first step in our general long term strategy. After having defined our geometric context, we will further subdivide this problem into an optical flow problem coupled with geometric triangulation, and show that an end-to-end solution (in the context of depth perception) that covers the whole problem at once outperforms the optical flow strategy in the context of a dedicated synthetic dataset created for this thesis. This solution uses a CNN called DepthNet.

• In chapter 4, we aim a finetuning our network trained on synthetic images with real data. After establishing that the training described in previous chapter was only theoretical and cannot be performed to finetune on real world videos where the ground truth is not easily available, we will try to cover the possible strategies for unsupervised learning, using the techniques of photometric reprojection error based optimization. This training technique being relatively new and based on heuristics, we try to describe currently used methods as thoroughly as possible, while linking them with already existing analytical structure from motion methods. More specifically, we discuss the core idea, the possible regularization terms in the optimization, with depth smoothing and occlusion detection, and finally the methods to measure the difference between two frames in a differentiable way.

• chapter 5 will apply the conclusions from previous chapter to perform an unsupervised training of our proof of concept network DepthNet, presented chapter 3. We then show that the resulting network, when used in the evaluation context presented in chapter 2, outperforms other unsupervised solution, mostly thanks to its structure from motion angle of attack in opposition to all other depth appearance related works.

• chapter 6 will present a strategy to use the resulting network with real videos. Namely, we will focus on how to construct an optimal input for DepthNet, and even how to exploit parallel inference to perform a high dynamic range of depth inference. Finally, we will present our proof of concept for obstacle avoidance using DepthNet with an off-the-shelf obstacle-avoidance-from-depth algorithm.

• Finally, chapter 7 will conclude this thesis regarding the long-term end-goal and the strategy that we initially designed, mentioning what issues should be solved.

We will try to propose potential future work for this, but also for other perspectives related to this work. This chapter will first focus on the different strategies that can be used to estimate the depth of an image. More specifically, depth from context and perspective will be compared to depth from motion.

On the second part, depth quality evaluation will be discussed in the light of obstacle avoidance context. This particular context will define our strategy on which quality measure is more meaningful, in terms of both safety and path planning.

Linking depth with reality: the scale factor problem

Measuring depth from the images of a monocular camera is inherently up to a scale factor. For example figure 2.1, when looking at a real castle or a much smaller model replica, the camera frames will look the same, and only other context elements can help estimate depth other than relatively with other pixels. This ambiguity is also appearing with movement estimation, consistently with depth: a video will look exactly the same when filming the real Chambord castle and translating by 10m, as filming the replica and translating in the same relative direction by 33cm (since the replica is at 1/30-scale). It does not mean we can't make estimation, but that all the depth values will only be relative to each other.

As a consequence, in order to get actual depth estimation, an absolute value is needed to serve as an anchor point, a measurement from an other dedicated sensor. The depth can then be deduced from the ratio between the estimated anchor value and the measured one. Possible solutions are: • Measuring depth, at least in one point, with additional sensor such as LiDAR, Timeof-Flight or stereo cameras. This is not a trivial solution and it needs embedded integration, as well as precise calibration to link this measurement with the right pixel in the picture plane.

• Assuming depth consistency across training and testing dataset. This can be particularly useful in datasets like KITTI, [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF], where the camera is always at the same height and looking at the floor from the same angle, but it is irrelevant on a dataset with high pose variability, e.g. UAV videos, and such assumptions will fail.

• Measure movement magnitude, with dedicated sensors, e.g. IMU or GPS in a UAV, or speed from wheels in a car, and compare it to estimated apparent movement, provided both depth and apparent movement are consistently estimated. The resulting ratio will be the scale factor.

The last solution is the preferred one, since movement estimation is a vital measurement in the context of navigation, and thus is almost always available on a UAV.

Besides, as shown on appendix A, a potential error in displacement estimation will also change the depth estimation, partially compensating the speed estimation error. For example if the vehicle underestimates speed magnitude, the depth will also be underestimated and the navigation strategy will try to avoid a potential obstacle more aggressively.

The drawback of this solution is that we need to estimate apparent movement, requiring multiple frames and preferably static scenes, without moving objects.

Depth from motion vs depth from context

Depth from context advantages and limitations

Depth inference networks have been shown to be able to estimate depth solely from perspective and context. Indeed, with only one image, they were able to get reasonable scale invariant quality measures [Fu+18; SCN08; EPF14; Zho+17]. It has been considered popular and convincing enough to develop its dedicated large scale challenge [START_REF] Uhrig | Sparsity Invariant CNNs[END_REF]. As said above, in a navigation context, the scale invariant quality is not really interesting without a way to link the estimation to the real world. As such, because it also estimates movement during training, Zhou et al. [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] work can be achieved within a consumer drone context. It will be discussed and tested in chapter 5. However, it is already fair to say that these estimations are reliable on a test set similar to what it has been trained with, here KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF].

Taking human perception as a reference, it has been shown [START_REF] Mcmanus | Is light in pictures presumed to come from the left side?[END_REF] that forcing perspective to the human was well known and studied, especially in art for dramatic effect. One of these techniques is Ames room. Figure 2.2 shows that the apparent perspective is contradicted by the person's size, the depth is thus different on the left and on the right while it looks the same. This illustrates the fact that the projection of a point P is not dependent to its distance to the camera, and highlights the lack of robustness of depth from context that is done by the human eye when looking at a single frame. We can extrapolate that a single frame depth network will suffer the same limitations.

One could argue that these counter examples of forced perspective are not really realistic: if they were to be found unintentionally, the human would have been robust to it. Figure 2.3 features a Dakar 2019 contestant, and an illusory cliff can be seen at first glance before realizing the picture is in fact a high-angle shot (that the human eye is not familiar with). This shows that confusing perspective can sometimes happen even with realistic imagery.

The conclusion of these examples, is that depth from context is not robust to new environments, and especially, depth error (even scale invariant) is not continuous with respect appearance. Because of multi-stable perception [START_REF] David M Eagleman | Visual illusions and neurobiology[END_REF], large errors can even be found in the very training set. As such, it means that a training set must be extremely thorough, because a slight perturbation of appearance, like a change of lighting or orientation might completely change the outcome. This would make a UAV depth from context dataset very challenging to construct.

Besides, a depth estimator trained on this hypothetic dataset might have to remember among all the possible perspective layouts in a particular image the most probable one with experience, which means heavy memory tasks might be at stake. We can suppose that a neural network dedicated to this task will probably have to be heavy and thus hard to embed on a mobile system.

Depth from motion advantages and limitations

Instead of depth from context, the depth from motion benefits from a possible training but mostly from geometrical relations between structure and movement. As it will be discussed on chapter 3, depth from motion only relies on how features are moved in the frame. In the absence of rotation, we know that infinitely distant objects won't move on the image and that features with large displacement will represent close objects.

As such, depth error is more continuous to light and structure conditions, and we predict that depth estimation will be more robust to unseen scenes. This suits both our needs in term of lightweight estimator, not needing to remember every possible realistic depth, and robustness, thus making dataset completeness requirements less vital.

Besides, this technique does not need to explicitly measure apparent displacement magnitude like it was the case for depth from context to solve the scale factor. If every depth from motion estimation is made with the assumption of a constant nominal speed, the apparent speed is fixed and the scale factor will only be a function of actual speed estimation from other sensors.

However, depth from motion is not robust to moving objects. The simplest example to illustrate this is an object that would be moving along with the camera. Object's relative movement to the camera will be null and its distance will appear infinite.

In the context of obstacle avoidance, this could be mitigated by the fact that this is less an obstacle since the relative movement is null. The time to collision, which is the time needed for the camera to reach the obstacle, should both the camera and the obstacle keep their relative motion constant, is infinite. However, knowing the actual depth value instead of the time to collision is important as maneuvers take typically more distance at high speed, making the best strategy dependent of the scene structure. This is a well known effect in road safety where the security distance must be a constant proportion of speed, regardless of actual relative motion [START_REF] Filzek | Distance behaviour on motorways with regard to active safety-a comparison between adaptive cruise control (acc) and driver[END_REF]. Regarding a stopping distance increasing non linearly with speed, like in the example of appendix A, this should even be an increasing proportion of speed.

Considering other movement such as perpendicular motion relative to actual camera path, optical flow anomalies could help detect the object as moving and then discard it from the depth triangulation. However, depth of the moving object will be unknown and the same scale ambiguity problem than with the whole rigid will be encountered: we either need to know our relative motion to it, or have any idea of its typical depth based on its appearance, if e.g. the moving object is an adult pedestrian, which gets back to all the already discussed depth from context problems.

Being robust to moving objects is thus potentially challenging for depth from motion, while it's not the case for depth from context. Although we discuss possible solutions for this particular problem at the end of this document, we chose to only consider rigid scenes throughout the next chapters. The rationale is that solving the rigid scene subproblem would be already a leap forward, and that in the context of UAV navigation where the camera is often at a high altitude when moving fast, this can be enough for most cases.

Our strategy

From this quick analysis of our problem, we chose to focus on depth from motion. The main reasons can be summarized to

• The context of heterogeneous stabilized aerial videos makes a harder use-case for depth from context and appearance, and an easier case for depth from motion.

• Priority is given to robustness, which, as discussed above is problematic for depth from context, given how easily even a system as evolved as the human can be fooled with unusual but realistic images.

• The estimator network needs to be lightweight for a chance to be embedded in the future, which seems more feasible with the lower level problem of feature movement recognition required by depth from motion

Obstacle avoidance oriented depth quality

Scale invariant estimation

Before trying to estimate the depth of a particular scene, we must be able to measure quality in the context of obstacle avoidance. More specifically, we must be able to take good trajectory correction based on the information given by a particular depth map and reasonable sensors.

For single frame networks, current depth quality measurements, originally introduced by Eigen et al. [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF], expect a relative depth map up to a scale factor. This is a scale invariant error measure, designed to evaluate the structure of the estimated depth rather than its actual values, particularly suited for depth completion tasks for example. This is however not suitable for navigation context, as they completely ignore the scale factor uncertainty, and thus rely on estimating the scale factor as the ratio of the medians of network's output and ground-truth. This is then representative to an ideal use case where the median of an unknown depth map has to be available, which is clearly unrealistic.

Following our decision to estimate both depth and apparent movement from section 2.1, we propose a new quality measurement, which depends not only on depth but also on speed magnitude estimation by slightly modifying the already prevalent ones. This new measurement is not relative anymore, it computes actual depth errors, and is more representative of a real application case where velocity is available.

Obstacle avoidance requirements

The depth θ from a particular estimator should give a reliable information in order to make a trajectory decision. As such, there is a particular range of depth values whose estimation needs to be precise.

prior work [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF][START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] Our proposition

Predictions Depth θ Depth θ, Velocity v Ground Truth Depth θ Depth θ, Velocity v Measure m = δ θ, θ × Me(θ) Me( θ) m = δ θ, θ × |v| | v|
Table 2.1: Me() is the median operator, and δ is a validation measure (e.g. L1 distance)

The obvious faulty case, is when an obstacle is thought to be farther than in reality, which makes the UAV chose a trajectory that will lead to a collision. As a result, too optimistic results in the validation set should be penalized in the range of maneuver of the drone. Thus we don't necessarily want depth to be of a constant precision until infinity, the range of maneuver will depend on the obstacle avoidance strategy and specifications. For our case of a relatively slow UAV (compared to e.g. a much faster military UAV) with a typical max speed of 20m.s -1 it might not be important to get precise depth above a maximum value, set here arbitrarily at 100m, and the closer depth will be to this maximum value, the less precision will be required.

Also, in our particular use-case we must keep a trajectory close to the one initially targeted, too pessimistic results should also be penalized because it will lead to unnecessary modifications of trajectory, resulting in a safe but inefficient flight.

The relative importance of depth overestimation (too dangerous) compared to underestimation (too conservative) is not clearly defined, but a depth estimator should at least not be encouraged to be biased in one way or another if it was to be optimized for a particular measure.

Anatomy of an error function

An error function f takes a depth map estimation θ and its ground truth θ, and outputs a positive scalar value. f ( θ, θ) > 0. Obviously, when estimation is perfect, θ = θ, the error function is supposed to be 0.

An error measure E f of this error function on a particular validation set V is then the mean of all the comparisons between estimation and ground truth.

E f = 1 |V| ∑ ( θ,θ)∈V f ( θ, θ)
If the validation set is thorough enough, we can write this equation as a mathematical expectation of the random variable f ( θ, θ).

E f = E( f ( θ, θ)) E f = ( θ,θ)∈R +2 f ( θ, θ) × P(θ) × P( θ|θ)d θdθ (2.1)
Since the mean is computed for all ( θ, θ) pairs, regardless of θ values, this measure is necessarily biased by the validation set distribution. If in a particular set the depth values in a certain range are under represented, the resulting measure won't reflect the actual depth at this range. ME , mean error gives more information. Above, standard error is the most informative.

Standard error SE

Mean error ME

E f 2 = E( f ( θ, θ) 2 ) E f = E| f ( θ, θ| P( f ( θ, θ) ≥ α) ≤ SE 2 α 2 P( f ( θ, θ) ≥ α) ≤ ME α Table 2

.2: Comparison of standard and mean errors and corresponding maximum for the tail distribution of f

It is thus very important to design a validation set V that fits our needs and the typical depth values that we need to be precise on. For example, when considering a high speed obstacle avoidance strategy, very low depth values might not be very important, since no good strategy can be used in that context, even if the measurement is accurate.

Standard error vs mean error

For a particular error function f , there are generally two ways of computing statistics on it. We can compute its mean, which we will call the mean error (ME) but we can also compute the mean of its square value (MSE). Both of these techniques measure how far from 0 the error f usually is, but the mean square error gives more priority to outliers. However, the Bayesian error distribution is more restricted: with Markov's inequality, we can see we have more concentrated repartition of possible error values than with mean error. (see table 2.2). The root of mean square error (sometimes called RMSE), which we will call here standard error (SE) because of its proximity with standard deviation will always be higher than mean error and thus give less information on lower possible error values, but gives more guarantee on the possible outliers.

It can be seen figure 2.5 how these two ways of measuring an error are complementary, but when it comes to comparison, the preferred measure will depend on the obstacle avoidance strategy. A more conservative system will prefer to minimize SE (standard error) over ME (mean error). In most cases, the mean error is given, because the distribution tail problems that are solved by standard error are often negligible in finite validation sets, without infinite depth values. Unless specified otherwise, mostly to compare the proposed algorithms with literature, we will use the mean error ME when measuring an error function.

Usual error functions

This section will try to cover usual error functions that are typically measured when evaluating a depth map estimation, not necessarily in the context of obstacle avoidance. Our goal is to determine how informative an estimator is with a particular error evaluation function. More specifically, we want to get an idea of θ Bayesian distribution given an estimation θ and an evaluation on a validation set V.

Absolute difference

The absolute difference is the simplest error function.

f ( θ, θ) = | θ -θ| ∀ θ ∈ R + , f ( θ, θ) = α ⇒ θ = θ ± α
It can already be noted that these error functions don't rely on the scale when computing its mean, contradicting our desire of being more tolerant on high depth values.

Another simple problem raised by this is the dependence to the validation set's actual distribution. This is not necessarily problematic if the set's distribution is well known, but it should not be forgotten.

Relative difference

With the goal of penalizing errors in accordance to depth scale, the relative errors are normalized by dividing by the ground truth values.

f ( θ, θ) = | θ -θ| θ = 1 - θ θ ∀ θ ∈ R + , f ( θ, θ) = α ⇒ θ = θ 1 ± α
The major issue is how penalizing this error function is for over-estimations. In fact, an error of 1 means that either the ground-truth is half the estimation or infinite. As such, an estimator chosen on this measure will be encouraged to under-estimate depth values.

Proposition 1. Given an estimator e responsible for estimations { θ}, in a corresponding validation dataset V = {( θ, θ)}, if the estimator is optimized for that validation set for a mean relative difference error, it will have more under-estimations than over-estimations.

P( θ < θ|( θ, θ) ∈ V) > 0.5
Corollarily, if we have an evenly balanced estimator { θ} such that P( θ < θ|( θ, θ) ∈ V) = 0.5, then we can find α < 1 such that {α θ} is a better estimator according to mean relative difference MRE.

See appendix B for a proof.

Log difference

The two aforementioned error functions have an additional drawback because the difference (normalized or not) implies that compared values are spread on the whole real spectrum, as if the distribution P(θ| θ) was a particular Gaussian with its mean on θ. The log error on the other hand makes this assumption on the log of depth values.

f ( θ, θ) = log( θ) -log(θ) = log θ θ ∀ θ ∈ R + , f ( θ, θ) = α ⇒ θ = θ × e ±α
This error has the advantage of being symmetrical, and penalizes 0 estimations as heavily as +∞, which better reflects our ambition to avoid under-estimation and unnecessary security maneuvers. Besides, since it is only a function of the ratio between estimation and ground truth, this measure is not tied to the validation set distribution the way absolute difference was.

Precision error

The precision error, contrary to other loss we discussed is a statistic loss that gives a proportion of outliers. Specifically, it cannot go outside of [0, 1]. Besides, it only tells statistic for a specific error, here noted δ, and does not tell any information regarding the distribution of the outliers, which could be all very close to the threshold δ or very far without influencing this evaluation.

f ( θ, θ, δ) = 1 -1 θ∈[ 1 δ θ,δθ] = 0 if θ ∈ 1 δ θ, δθ 1 otherwise 2.5. Conclusion on error measures Error Name Acronym Equation f ( θ, θ) Mean Absolute Error MAE E| θ -θ| Mean Relative Error MRE E | θ-θ| θ Mean Log Error MLE E| log( θ) -log(θ)| Standard Absolute Error SAE E( θ -θ) 2 Standard Log Error SLE E(log( θ) -log(θ)) 2
Precisions δ

P δ P log θ θ ≤ log(δ)

Conclusion on error measures

These different error functions and their respective means on a particular validation set give a complementary view of different strengths and weaknesses. More specifically, the absolute difference is a good performance index on the most represented depth values of a validation set, relative difference (when performance is below 1) is a good indication on depth over-estimation. Throughout this thesis, we will use the different error measures summarized table 2.3 to characterize a depth estimator, but the logarithm error will be used when tuning our algorithms, and especially the hyper-parameter search presented in chapter 4.

Chapter 3

Creating a depth map with a convolutional neural network

This chapter is dedicated to investigate the capacity of Fully Convolutional Neural

Networks to perform regression tasks, or more specifically to compute the depth of a stabilized image. As discussed before, depth for stabilized image is a too specific problem to have a good dedicated literature. Our strategy is to put this problem in relation with more general ones, and try to adapt their solutions to fit this problem's specificity.

In a first part, we will define the framework of this task, with geometric definitions, and especially the link between optical flow and depth map.

Then, we will look at some Deep Learning solutions for a widely studied problem, the optical flow problem. This would lead us to a first working solution, but with sensible limitations, especially when considering navigation and obstacle avoidance.

We proposed a solution to overcome these, by asking the Network to infer depth directly instead of deducing it from optical flow. This solution implies the creation of a synthetic dedicated dataset.

Finally, we propose a more in depth review of a proposed network and its training process, along with results both in synthetic and real images.

This chapter has been the subject to the following publication [START_REF] Pinard | End-to-end depth from motion with stabilized monocular videos[END_REF]: Clément Pinard, Laure Chevalley, Antoine Manzanera, and David Filliat. "End-to-end depth from motion with stabilized monocular videos". In: ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences IV-2/W3 (2017), pp. 67-74. DOI: 10.5194/isprs-annals-IV-2-W3-67-2017. URL: https://www.isprsann-photogramm-remote-sens-spatial-inf-sci.net/IV-2-W3/67/2017/.

Geometric definitions

Camera plane

The camera plane describes the image pixels as a grid of 3D colors (Red, Green and Blue). When considering our image to be of dimensions h × w, we can model the image I as a vector field defined on a discrete plane

Ω Ω = ([0, w -1] × [0, h -1]) ∩ Z 2 card(Ω) = |Ω| = h × w (3.1) ∀p = (u, v) ∈ Ω, I(p) = c = (c 1 , c 2 , c 3 )
The problem of depth generation is then to assign for each point of Ω a particular depth value.

Pinhole camera model

The Pinhole camera model is a way of describing the mathematical relationship between 3D objects and their projection in the camera plane. This model is very appreciated for its absence of distortion and in fact is readily usable on a wide number of cameras, be it from a dedicated lens [START_REF] Kingslake | A History of the Photographic Lens[END_REF] of from a software correction [START_REF] Ae Conrady | Lens-systems, decentered[END_REF]. Thanks to this, systems with a good image quality can be assumed to either provide directly a rectilinear image as if it came from a pinhole, or at least a good calibration to make the virtual equivalent image available at the cost of a rectification algorithm.

If we consider a camera with its coordinate system centered at the optical center of its lens, a 3D point P = (x, y, z) will have the projected point p = (u, v) with the following relation:

u = u 0 + f u x z v = v 0 + f v y z (3.2)
With u 0 and v 0 the coordinates of the optical center in the projection plane, and f u and f v the focal lengths. Note that with this thesis' convention, we then get θ(p) = z. Besides, p here is not necessarily on Ω: it can be out of bounds or not of integer coordinates.

Most of the time, f u and f v are very close and thus this model can be physically approximated to a pinhole camera where every light ray only pass through one point (discarding any diffraction phenomenon), illustrated figure 3.1. To get a more concise notation, we can introduce the function Π, its pseudo-inverse Π -1 , and the matrix K

Π : R 3 → R 2 P = (x, y, z) → ( x z , y z ) Π -1 : R 2 → R 3 p = (u, v) → (u, v, 1) (3.3) K =    f u 0 u 0 0 f v v 0 0 0 1   
Note that for an unoccluded 3D point P = (x, y, z), We have θ(Π(KP)) = z. We can then rewrite equation 3.2 as p = Π(KP)

If we want to get the 3D point from its image on the projection plane, we get:

P = θ(p) × K -1 Π -1 (p)
We can see how the depth is needed in order to figure the position of the 3D point. Besides, the Π operation is scale-invariant.

∀α ∈ R + , Π(αP) = Π(P)
This illustrates the fact that the projection of a point P does not depend on its distance (not to confuse with depth) to the camera, as said section 2.1.

Moving camera

If we suppose a moving camera, while our 3D point P stays motionless (the scene is supposed rigid), we can now compute the new projection p 2 of P.

The motion of our camera can be characterized by a rotation matrix R and a translation t from the second coordinate system to the first. It means that we consider the inverse of the actual displacement of the camera. That way, we can express a direct transformation for 3D points in the first frame coordinate. For example if the camera is going forward, the points will be closer and thus go backward, relatively to the frame coordinate. The new relative position of P is then

P 2 = t + RP = (x 2 , y 2 , z 2 )
We can then deduce the position of p 2 in our projection frame

p 2 = Π(KP 2 ) (3.4) p 2 = Π (K(t + RP)) (3.5) p 2 = Π KRK -1 Π -1 (p) + Kt θ(p) (3.6)
Again, we can see how the position z = θ(p) is needed in order to know the position of p 2 from p. However, when both projected points p and p 2 are known, we can try to to figure out the depth value θ(p) thanks to this equation. Note that if the 3D point P is chosen so that its projection p has integer coordinates (and more specifically p ∈ Ω), this is not necessarily true for p 2 .

It is interesting to note how translation, given by t, and rotation, given by R are separated in this equation. Besides, when considering a stabilized motion, i.e. with R equal to the identity matrix.

p 2 = Π Π -1 (p) + Kt θ(t) (3.7)

Optical flow and disparity

Definition 1. The optical flow is the displacement of a point in the image plane between two frames. It's a vector field defined on Ω, associating a displacement vector for each point in the image. For a particular set of point correspondences

C = {(p, p 2 )} ⊂ Ω × R 2
where each point p of Ω may only appear once in C, we have:

F : Ω → R 2 p → (F u , F v ) = p 2 -p
Definition 2. Disparity is the norm of optical flow.

D : Ω → R + p → F(p)
Disparity is a term mostly coined for stereo vision. It is a special case of stabilized pair where displacement is only lateral, with t = (t x , t y , 0). In that case, the 3D point θ(p)Π -1 (p) + Kt has the same depth value as p and the equation 3.7 becomes

p 2 = p + 1 z ( f x t x , f y t y )
As such, assuming

f x = f y = f , optical flow is then F(p) = f z (t x , t y
), and disparity is

D(p) = f θ(p) t
The value t is called the baseline in this case. When it is known, for example with two cameras on a fixed rig, it is easy to deduce the depth of a particular point. Because of this simple relation for this particular case, the term disparity is abusively used to refer to the inverse depth.

As such, it might be useful to remind that in other circumstances, disparity is not the inverse depth up to a scale factor. This is important as both terms will be used in this work.

Optical flow divergence

If we suppose optical flow F = (F u , F v ) to be spatially differentiable, we can compute the value

div(F) = ∇.F = ∂F u ∂u + ∂F v ∂v since F is
only defined on Ω which is not continuous, we can approximate its divergence with finite difference convolution for example.

div(F) = S T * F u + S * F v where S =    1 0 -1    and * is the convolution operation.
It can be geometrically interpreted as the magnification of the image objects. Positive optical flow divergence stands for object appearing bigger in the second frame than in the first, even if they did not move, see figure 3.2. As such, measuring the divergence of optical flow is the same as measuring the zooming factor.

Focus of expansion

When considering a stabilized camera displacement t with t z = 0, we introduce the Focus of Expansion (FOE) point, noted Φ t , which is the projection of a virtual point with t as coordinates.

Definition 3. Given a stabilized motion with translation t = (t x , t y , t z ) a pinhole camera with calibration matrix K, the focus of expansion is defined by

Φ t = Π(Kt)
This point is particularly interesting for stabilized motion, the resulting optical flow can be expressed as simply a function of p, Φ t and θ(p) and t z .

Deep learning for optical flow

F(p) = p 2 -p (3.8) = Π (Kt + θ(p)Π -1 (p)) -p (3.9) = 1 θ(p) + t z (t z Φ t + θ(p)p) -p (3.10) = t z θ(p) + t z (Φ t -p) (3.11)
This equation proves how closely optical flow is linked to FOE, and why its named like that. We can see that the vector is strictly proportional to the vector that goes from p to Φ t . When going forward (which means t z < 0), every optical flow vector will be headed away from FOE.

Finding depth from optical flow

In light of these different links between depth, we now have a first angle of attack to solve our depth problem. By trying to computing optical flow, and assuming we know t with external sensors, e.g. accelerometers (IMU) and GPS, we can deduce the depth map of our first frame.

Deep learning for optical flow

Although very popular, optical flow is a very difficult computer vision problem and many issues still remain unsolved, especially for some artifacts like motion blur or change of exposition. Considering a deep learning solution, if older datasets can only serve for evaluation purpose due to their size like the Middlebury dataset [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF], some bigger datasets can be used for training, be it from synthetic images with MPI Sintel [But+12] (with complete ground truth) or from real images with KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] with sparse groundtruth.

These datasets, although offering a training framework, were not sufficient to train a network to fully encompass the optical flow operation, but rather the projection function that could lead to feature maps to know if two pixels are referring to the same part in the 3D scene or not [ZK15; ZL16; Han+15]. That way, the optical flow is then found matching the most similar pixels.

∀p ∈ Ω, F(p) = argmin p ∈Ω (d( f (I 1 , W )(p), f (I 2 , W )(p )))
where f is the neural network outputting a feature map f (I, W ) from an image I and a set of trained weights W, and d is a distance function, possibly computed by another neural network, performing a classification task to infer how similar are two feature map pixels.

This sub-task might not need too much data to make a network converge, but lacks flexibility compared to end-to-end training algorithms powered by even bigger dataset, with synthetic purposely unrealistic images.

These datasets were proposed in order to get pretraining with a massive amount of data like Flying Chairs [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF], from which Figure 3.3 presents two examples, or Fly-ingThings3D[N.M+16]. The main purpose is to propose a "pretraining" framework, and then perform a fine-tuning on a smaller realistic dataset. That way, the network can understand the global notion of optical flow with a pretraining and not only the feature matching, and then specialize to a certain type of images as a fine-tuning. This is somewhat related to Curriculum Learning [START_REF] Bengio | Curriculum learning[END_REF] and gets the same conclusion about the network generalization.

FlowNet

One of the main network that particularly benefited from these very large datasets is FlowNet [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF], whose architecture is inspired from semantic segmentation networks [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] and presented figure 3.4. This architecture has been named with very different names such as tiramisu [START_REF] Jégou | The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation[END_REF], feature pyramid [START_REF] Lin | Feature Pyramid Networks for Object Detection[END_REF], U-Net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF], and it consists in a hourglass architecture, with skip connections with feature maps of the same size. As such, we get a "projection" part, that computes feature maps with high semantic level, and "refinement" part which concatenates up-sampled feature maps with corresponding earlier convolution outputs, with information more closely linked to pixels because it went through less strided convolutions.

This network is one of the first work that made possible a completely end-to-end train-3.3. Optical flow for depth generation ing of optical flow, and only simple convolution and transposed convolution operations are done, with no geometry, smoothing or matching function involved anywhere. For its simplicity despite its good results, we chose to study closely its usage and architecture for our problem. One should note that the following study does not necessarily depend on the architecture, and could be used with more sophisticated networks [Sun+18; Ilg+18], although the simplicity of this architecture also guarantees a good flexibility when contemplating the possibility of modifying its initial purpose or architecture.

Optical flow for depth generation

In this section we discuss the relation between depth and optical flow F coupled with translation estimation t. Most importantly, we discussed how uncertainties for optical flow and translation can lead to big errors in depth, even in the context of a stabilized camera with a perfectly known displacement. This navigation context can help us prioritize errors. To get the depth values of the last image I t we received from the stabilized camera, we need to get the optical flow of image I t compared to anterior frames I t-∆t . As such, our convention of taking displacement t matches the actual camera displacement: forward movement means positive forward translation t z > 0. Our sensible use-case is forward movement: FOE is inside the frame, and t z > 0.

From disparity to depth

Let us consider the norm of t, denoted t m (m stands for magnitude). We have t 2 m = t 2

x + t 2 y + t 2 z . We also consider the optical center p 0 , given by p 0 = (u 0 , v 0 ). Finally, we assume θ >> |t z |, since the opposite would mean objects potentially going behind the camera.

From definition 3 of Φ t , we can deduce that

t 2 m = t 2 z 1 + Φ t -p 0 2 f 2 |t z | = f t m f 2 + Φ t -p 0 2
Besides, from Eq.3.8, we have

(θ(p) + t z )D(p) = |t z | Φ t -p θ(p) = |t z | Φ t -p D(p) -sign(t z ) (3.12)
In the end, we can deduce depth θ from disparity D, focal length f , FOE Φ t and translation magnitude t m

θ(p) = f t m f 2 + Φ t -p 0 2 Φ t -p D(p) -sign(t z ) (3.13)
This result is in a useful form for limit values. Lateral movement corresponds to Φ t → +∞ and then On the other hand when Φ t is inside the frame boundaries, when p → Φ t , knowing depth is a bounded positive value, we can deduce:

lim Φ t →+∞ θ(p) = f t m D(p)
D(p) ∝ p→Φ t p -Φ t
Limit of disparity in this case is 0 and we use its inverse. As a consequence, small errors on disparity estimation will result in diverging values of depth near FOE while it corresponds to the direction the camera is moving to, which is clearly problematic for depth-based obstacle avoidance. Figure 3.5 illustrates how FlowNet errors behave with respect to target optical flow magnitude, and especially the fact that null ground truth optical flow can have great end point error, up to 10 pixels.

An other aspect of this equation shows that for a forward movement, sign(t z ) = 1, and thus disparity values can trigger a negative depth. This is expected as the dezooming from apparent backward movement (since we compute a reversed optical flow to get depth of last image) normally constraints the points to only get closer to the FOE while not going across it: an infinite depth makes the point p motionless, while a null depth moves the point to the FOE. As such, careless dense optical flow is almost guaranteed to output negative near the FOE in our sensible use-case.

An other issue can be the error from the estimation of the FOE from translation. Indeed, with a shifted Φ t , depth in its true position will be infinite since there the disparity D will be 0. This problem can however be partly solved when finding Φ t analytically, since every optical flow vector is supposed to be aligned with Φ t (see equation 3.8).

∀p ∈ Ω, F(p) × (Φ t -p) = 0
Where × is the 2D determinant.

From flow divergence to depth

If we consider inverse depth 1 θ to be spatially evolving slowly compared to optical flow, we can deduce it from divergence of F:

div(F) = t z div(Φ t -p) θ(p) + t z = -2t z θ(p) + t z
Here we are in a complementary situation: for lateral movement, t z is null, and so is divergence of F, but for forward movement, this technique can be used around the FOE [START_REF] Stabinger | Monocular obstacle avoidance for blind people using probabilistic focus of expansion estimation[END_REF][START_REF] Souhila | Optical flow based robot obstacle avoidance[END_REF]. Obviously this method will fail with a non-constant depth (and thus a non-zero divergence), its use is more adapted for very close incoming obstacle and urgent stop maneuver, when t z is of the same order of magnitude as z, and thus the obstacle likely to get reached within a few frames.

Conclusions on limitations

Even though optical flow is considered well predictable from networks like FlowNet [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF], its relation to depth induces inevitable errors when the movement is random, and computing depth from disparity is much harder than for a classic stereo rig. Some earlier works already enlightened the difficulty in estimating depth solely with flow [SK07; Zin+10], especially when the camera is pointed toward movement. One can note that rotation compensation was already used with fish-eye camera in order to have a more direct link between flow and depth.

Methods based on optical flow divergence can help around the FOE area, but are far from perfect, especially when a fine long range depth estimation is needed.

End-to-end learning of depth generation

From last section findings, we decide to study the possibilities in learning depth solely from a neural network.

We then focus on RGB-D datasets that would allow supervised learning of depth. RGB pairs (preferably with the corresponding displacement) being the input, and D the desired output. Our choice today to learn depth from motion in existing RGB-D datasets is either unrestricted w.r.t. ego-motion [START_REF] Firman | RGBD Datasets: Past, Present and Future[END_REF][START_REF] Sturm | A Benchmark for the Evaluation of RGB-D SLAM Systems[END_REF], or a simple stereo vision, equivalent to lateral movement [GLU12; SS02]. To our knowledge, no dataset proposes only translational movement.

An interesting work on a structure from motion network has been published and considers several IMU-enabled datasets [START_REF] Ummenhofer | DeMoN: Depth and Motion Network for Learning Monocular Stereo[END_REF]. The network, called DeMoN tries to get structure from a pair of images, but not necessarily stabilized. As such, several of these datasets could be used for our network, performing an online stabilization from IMU so that we would be back in our original use-case [START_REF] Xiao | Sun3d: A database of big spaces reconstructed using sfm and object labels[END_REF][START_REF] Sturm | A Benchmark for the Evaluation of RGB-D SLAM Systems[END_REF][START_REF] Aanaes | Large-Scale Data for Multiple-View Stereopsis[END_REF]. This is definitely an interesting solution, but a careful read of their article however enlightens the necessity of large synthetic datasets for their training in addition to these realistic datasets, named BlendSwap and Scenes11 (which don't have a dedicated publication). For each part of the network, it follows the curriculum learning workflow, training on a large unrealistic dataset before fine-tuning on a smaller harder real one: the joint training using all datasets at the same time was probably motivated by their validation set that includes samples from different datasets, and thus needs to stay robust, at the cost of not being specialized for a particular situation.

Another problem when considering leveraging from this work is that these datasets are all very suited for the "multi view synthesis" problem, where the general movement of the camera is to rotate around a subject in order to scan it. This is then not very representative of an obstacle avoidance context, with camera going forward and not only sideways.

Introducing StillBox, a depth synthetic dataset

In light of available literature for stabilized RGB-D datasets, we propose a new dataset which aims at proposing a suited environment for our navigation context: stabilized videos or perfectly known orientations, random movements, and random scenes.

This dataset has been shared with the scientific community and is available to download at http://stillbox.ensta.fr.

We called this dataset StillBox.

For this dataset, we used the rendering software Blender2 to generate an arbitrary number of random rigid scenes, composed of basic 3d primitives (cubes, spheres, cones and tores) randomly textured from an image set scrapped from Flickr3 (see figure 3.6).

These objects are randomly placed and sized in the scene, so that they are mostly in front of the camera, with possible variations including objects behind camera, or even camera inside an object. Scenes in which camera goes through objects are discarded. To add difficulty we also applied uniform textures on a proportion of the primitives. Each object thus has a uniform probability (corresponding to texture ratio) of being textured from a color-ramp and not from a photograph, along with a random size and position uniformly sampled from specified ranges.

Walls are added at large distances as if the camera was inside a box (hence the name). The camera is moving at a fixed speed value, but to a random direction (uniform distribution), which is constant for each scene. It can be anything from forward/backward movement to lateral movement (which is then equivalent to stereo vision). Tables 3.1 and 3.2 show a summary of our scenes parameters. They can be changed at will, and are Figure 3.6: Some examples of our renderings with associated depth maps, from 0 to 100m. Visible reticle is not comprised in raw images stored in a metadata JSON file to keep track of it. Our dataset is then composed of 4 sub-datasets with different resolutions, 64px dataset being the largest in term of number of samples, 512px being the heaviest in data.

The parameters described here present the version of this dataset we open sourced. However, thanks to the fact that we constructed it ourselves, we are able to generate very quickly a new dataset with custom camera properties such as focal length, image size or movement, potentially with rotation if we want to simulate a bad stabilization.

Dataset set augmentation

The way we store data in 10 images long videos, with each frame paired with its ground truth depth allows us to set a posteriori distances distribution with a variable temporal shift between two frames. A shift of 1, which means two consecutive frames, will have a lower temporal distance than a shift of 10 (the maximum shift in our dataset). If we use a baseline shift of 3 frames, we can e.g. assume a depth three times as great as for two consecutive frames.

In addition, we can also consider negative shift, which will only change displacement direction without changing speed value compared to opposite shift. This allows us, given a fixed dataset size, to get more evenly distributed depth values to learn, and also to decorrelate images from depth, preventing any over-fitting during training, that would result in a scene recognition algorithm and would perform poorly on a validation set.

DepthNet, a depth fully convolutional neural network

Now that we have a dataset that mimics UAV navigation, we can train a network to get depth from motion. This network is very inspired from FlowNetS presented in previous section 3.2.1.

The constant speed assumption

The network will accept two images I 1 and I 2 as input, and will output a depth map θ relative to I 2 . We want a network that assumes a constant displacement between frames. This will have two advantages:

• The way we augment the dataset, depth will be linked to apparent motion. For example if I 1 = I 2 the depth output by the network should be infinite.

• We make the assumption that since this constant speed assumptions makes depth very closely linked to disparity, a network performing well on optical flow will be a good candidate for depth.

On the optimal network output

As mentioned by authors of DeMoN [START_REF] Ummenhofer | DeMoN: Depth and Motion Network for Learning Monocular Stereo[END_REF], inverse depth might be a good value for our network to output, because its value is more linked to disparity. Besides, it gives less importance to far objects, which might be a good strategy to have good relative error such as MRE and MLE (see section 2.4.

2)

The major problem with this approach is that it does not provide an "optimal" depth on which the network should be the most reliable: the closer an object is, the more contrasted output values the network should have, and thus the more weights in the network should be dedicated to this depth distribution. In our case, betting on close 3.5. DepthNet, a depth fully convolutional neural network

Outputting method

Regular depth Inverse depth Log Depth

Loss function L |Out(p) -θ(p)| Out(p) -1 θ(p) |Out(p) -log(1 + θ(p))| Inverse Post processing IPP(θ(p)) = θ(p) IPP(θ(p)) = 1 θ(p) IPP(θ(p)) = log(1 + θ(p)) Post processing θ(p) = Out(p) θ(p) = 1 Out(p) θ(p) = exp(Out(p)) -1
Table 3.3: Comparison of three possible depth representations for the network output objects is probably not optimal in order to be able to avoid obstacles long before we reach them. We need to choose a more suiting hierarchy: our preferred depth quality measurement being the logarithmic error, we test three separate outputting methods, presented table 3.3: regular depth, inverse depth, logarithmic depth. Each method gets as a training loss the L1 difference between the outputting method and its corresponding ground truth. We thus need to design a post processing function to get the actual depth from the network output, and an inverse post processing function to get the value with which the network output will be compared for each loss function.

General structure

Our network is described figure 3.8. As mentioned above, this network architecture has been proven very efficient for flow and disparity computing while keeping a very simple supervised learning process. The main point of this experimentation is to show that end-to-end depth estimation can be beneficial regarding unknown translation. In the general effort of having a simple network that motivated our choice of inspiration from FlowNetS in the first place, all convolution are now 3 × 3 kernels, even the first ones that are 7 × 7 and 5 × 5 in FlowNetS, and activation functions are simple ReLU [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] instead of LeakyReLU [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF]. Most importantly, our architecture differs from FlowNetS by its width, where each layer has half the feature maps. These simplifications makes DepthNet a relatively light network (at least 4 times lighter than FlowNetS) with only standard computation blocks for a facilitated potential mobile deployment. The motivation for a thinner network also relies in the fact that a stabilized footage offers a much simpler optical flow pattern, as shown in equation 3.8. It can be noted that we added Spatial Batch Normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] in the convolution blocks to ease training (see figure 3.9), knowing that it can easily be replaced by simple addition and multiplication during the inference. Note that the "OutN" convolutions before the "1+ELU" activation functions neither have BatchNorm nor ReLU, and the "Up OutN" don't have BatchNorm.

The ELU trick

For final output, we use the activation function "1 + ELU" [START_REF] Clevert | Fast and accurate deep network learning by exponential linear units (elus)[END_REF]. It outputs only values above 0, which is to be expected, but unlike the ReLU, we avoid the null gradient when raw network output (before the activation function) is below zero. Besides, 1+ELU is still very close to a ReLU: it only differs for output values below 1, and could be replaced by a simple shifted ReLU (x → ReLU(x + 1)) for inference if needed and applicable, ie with regular depth. See figure 3.7 for a graphic comparison. Functions are exactly the same above y = 1, meaning that in our dataset, the difference will almost always be negligible.

1 + ELU : R → R + x → 1 + x if x > 0 e x else

Training details

Like for FlowNetS with optical flow, we use a multi-scale criterion, with a L1 reconstruction error for each scale.

Loss = ∑ s∈scales γ s 1 |Ω| ∑ p∈Ω |Out s (p) -IPP(θ(p))|

where

• Out s is the output map at scale s of the network, upscaled bilinearly to match dimension of θ. Note that depending on the post processing function, this is not yet a depth map per se.

• IPP is the inverse post processing function. (See 3.5.2)

• γ s is the weight of the scale, we arbitrarily chose higher weights for higher scales: (γ s ) 2≤s≤6 = [0.32, 0.08, 0.02, 0.01, 0.005] for s varying from 2 (highest scale) to 6 (lowest scale)

• θ is the depth ground-truth.

As said earlier, we apply data augmentation to the dataset using different shifts, along with classic methods such as flips and rotations. We also clamp depth to a maximum of 100m, and provide sample pairs without shift, assuming its depth is 100m everywhere. We use a stochastic gradient descent algorithm with a momentum of 0.9. The learning rate varies from 10 -3 to 10 -2 , depending on the post processing function.

We used the framework PyTorch [START_REF] Paszke | Automatic differentiation in PyTorch[END_REF], the full source code is available on Github4 , along with the StillBox dataset.

Influence of resolution on training

The special case of 64x64 images Figure 3.10 shows results from 64px dataset. Like FlowNetS, results are down-sampled by a factor of 4 and up-sampled to 64x64 with bilinear interpolation, just like during training.

One can notice that although the network is still fully convolutional, feature map sizes go down to 1x1 and then behave exactly like a Fully Connected Layer, which can serve to figure out implicitly motion direction and spread this information across the outputs. This can be interpreted as 3D shapes identification, along with their magnification: pixels belonging to the same shape are deemed to have close and continuous depth values, resulting in a FOE-independent depth inference. Figure 3.11 shows the depthwise relative error. While quality is not optimal at very low depth (and thus high displacement), the network is very precise even at very high depth. A relative error of 0.2 at depth of 90 means that the equivalent error on pixel displacement is less than 0.2× f ×0.3 90 = 0.02pixels. The error peak at 12m can be interpreted by the fact that most objects (and thus the scene's main source of diversity and estimation noise) are between 10 and 20 meters. It can be also reinforced by the fact that bilinear interpolation of our network's output (which is downscaled by a factor of 4) typically presents a constant slope at depth discontinuities, resulting in high error at object borders. Table 3. 4 shows quantitative results on a validation set with our three different post processing functions. It can appear surprising that in the end, inverse depth is clearly not the most efficient, especially when knowing that literature almost always tries to learn disparity (in the case of stereo vision) or inverse depth. Its poor results here can be explained by the mean disparity which is very low on our dataset and thus the L1 Error is too noisy. This result is thus particularly interesting for low disparity cases, where learning the inverse of disparity might be more efficient than learning it directly. As an example, we can cite the dataset CHairSDHorn, published with FlowNet2 [START_REF] Ilg | Flownet 2.0: Evolution of optical flow estimation with deep networks[END_REF] which features small displacements.

32x 1 /2Hx 1 /2W 64x 1 /4Hx 1 /4W 128x 1 /8Hx 1 /8W 256x 1 /16Hx 1 /16W 256x 1 /32Hx 1 /32W 512x 1 /64Hx 1 /64W 256x 1 /32Hx 1 /32W 128x 1 /16Hx 1 /16W 64x 1 /8Hx 1 /8W 32x 1 /4Hx 1 /4W 1x 1 /32Hx 1 /64W 1x 1 /16Hx 1 /32W 1x 1 /16Hx 1 /16W 1x 1 /8Hx 1 /8W 1x 1 /4Hx 1 /
For our two other losses, results are somewhat comparable. For sake of simplicity, we keep the identity post processing.

From 64px to 512px Depth inference

One could think that a fully convolutional network such as ours can not solve depth extraction for pictures greater than 64x64. The main concern is that for a fully convolutional network, each pixel is applied the same operation. For disparity, this makes sense because the problem is essentially similarity from different picture shifts. However, for depth inference when FOE is not diverging (forward movement is non negligible), result from equation 3.13 apparently shows that once the FOE is known, we then get different operations to do depending on distance from it and from the optical center p 0 . The only possible strategy for a fully convolutional network would be to compute the position in the frame as well and to apply the compensating scaling to the output. This problem then seems very difficult, if not impossible for a network as simple as ours, and if we run the training directly on 512x512 images, the network fails to converge to better results than with 128x128 images while better resolution would help getting more precision. However, if we take the converged network and apply a fine-tuning on it with 512x512 images, we get much better results. Figure 3.12 shows training results and shows that our deemed-impossible problem seems to be easily solved with multi-scale fine-tuning. As Table 3.5 shows, best results are obtained with multiple fine-tuning, with intermediate scales 64, 128, 256, and finally 512 pixels. Subscript values indicate finetuning processes. FlowNetS is performing comparably to DepthNet while being 5 times heavier and most of the time much slower, as shown Table 3.6.

Comparison with DeMoN and depth from optical flow

To show that a network dedicated to output depth map on stabilized videos can outperform more traditional methods, we compare our network to two methods: trained for depth used section 3.5) . FOE is supposed perfectly known. FlowNetS is finetuned for optical flow on still box.

• DeMoN network, which is supposed to work on any movement. But as discussed above, DeMoN might be more specialized on lateral movement, and not forward movement. Because of its heaviness which makes it unsuitable for light embedded integration, we did not take the time to fine-tune DeMoN on still box. The main point of comparing with DeMoN is to show that even though its inner architecture is based on optical flow, depth does not diverge near FOE. This is most probably thanks to the refinement network that regularizes the optical flow map and outputs a plausible depth map instead of deducing it geometrically from optical flow.

For this comparison, and because DeMoN which we don't retrain is supposed to work on a specific set of intrinsic parameters, we constructed a new still box dataset for fine tuning and validation that matches DeMoN's expected intrinsics. In order to help DeMoN, which might confuse translation and rotation while we know there is none, we feed the iterative network the right rotation (null) instead of the estimated value.

Figure 3.13 compares the three methods on a scene with lateral movement, and a scene with forward movement. We can see how near FOE, depth diverges for depth from optical flow.

Table 3.7 displays quantitative quality of these three methods. As both depth from FlowNet and DeMoN can output negative value that would result in an infinite log error, we clamped the depth to be between 0.1 and 100.

Finally, figure 3.14 shows the error with respect to distance to FOE. As shown, the depth from optical flow method, although with a better average score than DeMoN has poor values near FOE. This is an important result as it shows that even for optical flow based network like DeMoN, a refinement step makes it possible to output a better quality depth map near FOE. This shows that a depth from optical flow is salvageable with enough refinement.

UAV navigation use-case

Figure 3.15 shows qualitative results from our validation set, and from real conditions drone footage, on which we were careful to avoid camera rotation. These results did not benefit from any fine-tuning from real footage, indicating that our Still Box Dataset, although not realistic in its scenes structures and rendering, appears to be already sufficient for learning to produce decent depth maps in real conditions, the same way FlowNetS trained Flying Chairs [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF] dataset did get decent results on more realistic optical flow datasets, like MPI Sintel [But+12] and KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF].

We trained depth estimation from a moving camera, assuming its velocity is always the same. When running during flight, such a system can easily deduce the real depth map from the drone displacement magnitude t, knowing that the training speed was V 0 (here 9m.s -1 )

depth(t) = V t V 0 DepthNet( f rame t , f rame t-1 ) (3.14)
One of the drawbacks of this learning method is that the f value (which is focal length divided by sensor size per pixel) of our camera must be the same as the one used in training. Our dataset creation framework however allows us to change this value very easily for training. One must also be sure to have pinhole equivalent frames like during training.

However, since the network is fully convolutional, the inference will work as long as the focal length is the same. More specifically, this is supposed to be robust to optical center errors.

Figure 3.16 shows a typical failure case, where the sky is thought to be a very close object. This shows that although already decent on real videos, out network is probably over-fitted on typical StillBox images, even if it was designed with diversity in mind. More specifically in this case, we suspect a small defect on frame stabilization, which the network never had to be robust with when training on StillBox.

Conclusion of this chapter

We proposed a novel way of computing dense depth maps from monocular image sequences, along with a very comprehensive dataset for stabilized footage analysis. This algorithm can then be used for depth-based sense and avoid algorithm in a very flexible way, in order to cover all kinds of path planning, from collision avoidance to long range obstacle bypassing.

The proposed network outperforms more general depth from motion algorithms, and is yet very simple with only convolutions, transposed convolutions and ReLU activations.

We now have a very powerful pretraining for depth, the same way Flying chairs [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF] was for Optical Flow. The next chapter will cover the ways of fine-tuning this network in order to perform well on real videos, and especially how to leverage videos without ground-truth depth thanks to photometric reprojection based training.

Motivations

This chapter aims at studying the possibilities of learning depth without ground truth. This is one of the logical further steps after having shown how a neural network can learn robust depth inference from a supervised training.

Indeed, our network could probably be enhanced by training on real images instead of our unrealistic still box dataset. To do so, one could use supervised learning to train a system for depth from vision on an offline dataset featuring explicit depth measurement, such as KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF]. However, even setting up such recording devices can be costly and time demanding, which can limit the amount of data the system can be trained on.

One could also try to set up a synthetic but realistic dataset using state of the art game engines and assets [START_REF] Shah | AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles[END_REF]. Unfortunately, this solution could potentially lack diversity, as assets such as 3D scenes are not easy to setup and require human assumptions on what makes a rendering realistic.

The ground-truth-less learning solution, although more difficult in terms of designing a loss function, would have the huge advantage of not needing any sort of preprocessing. As a consequence, extending a dataset of training images would only require to capture them and directly add them to training database.

Training depth networks without ground-truth

Most recent works on autosupervised training networks for computing depth maps use differentiable bilinear warping techniques, which we will talk about in first section, first introduced in [START_REF] Jaderberg | Spatial transformer networks[END_REF]. The main idea is trying to match two frames using a depth map and a displacement. The new loss function to be minimized is the photometric error between the target frame and the projected one. Depth is then indirectly optimized. Although sensitive to errors coming from occlusions, non Lambertian surfaces and moving objects, this optimization shows great potential, especially when considering how little calibrated data is needed.

For instance [GMB17; Gar+16; XGF16] use stereo views and try to reconstruct one frame from the other. This particular use case for depth training allows to always consider the same displacement and rigid scenes since both images are captured at the same time and their relative poses are always the same. However, in addition to having a fixed displacement, with its already discussed depth range limitations, it constrains the training set to perfectly rectified stereo rigs, which are not as easy to set up as a monocular camera.

When trying to estimate both depth and movement, [Zho+17; YS18; MWA18; Ran+18] also achieved decent results on completely unconstrained ego-motion video. One can note that some methods [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] are assuming rigid scenes although the training set does not always conform to this assumption. The other ones try to do without this assumption by computing a residual optical flow to resolve the uncertainty from moving objects.

[Vij+17] explicitly considered non rigid scenes by trying to estimate multiple objects movements in the scene, to begin with the motion of the camera itself, which allowed them to deduce a flow map, along with the depth map.

Outline of this chapter

This chapter is dedicated to present as thoroughly as possible the main ideas between frame reprojection, and the existing related techniques in literature. This study will help us evaluate several extensions to the initial reprojection optimization before taking it to a full size deep learning workflow.

1. Core concept will be presented section 4.2, with inverse warp, pixel wise differentiation and optimization.

2. We will then discuss section 4.3 the depth optimization problem as a set of frame reprojection photometric errors to minimize in a particular scene, along with the SFMLearner[Zho+17] use-case.

3. On section 4.4, Smooth constraints, inspired from optical flow regularization will be here studied on inverse depth. A general smooth loss expression will be considered, along with several popular smooth losses which are in fact variations of the general form.

4. The occlusion problem will be addressed section 4.5. We call into question existing solutions in literature and propose an analytical alternative.

5. We will construct section 4.6 a toy problem designed to be challenging for these presented techniques, despite its size of only two scenes.

6. Finally, we will present section 4.7 several extensions to the initial photometric loss used for reprojection optimization Moreover, a collection of notebooks to recreate this chapter figures are available online1 .

Core concept for reprojection based optimization

The key feature of unsupervised depth training is the integration of images in the gradient propagation workflow. As a consequence, for any point p ∈ R 2 , we must be able to compute the gradients of I(p) with respect to p coordinates.

The image as a continuous 2D function

As in chapter 3, we consider an image I as a function that maps from integer values in Ω, defined eq 3.1 to channel luminance [0, 255] 3 .

Ω = ([0, W -1] × [0, H -1]) ∩ Z 2 cardinal(Ω) = |Ω| = H × W 4.2. Core concept for reprojection based optimization ∀p = (u, v) ∈ Ω, I(p) = c =    c 1 c 2 c 3   
Our current problem is to extend our image in the whole 2d plan R 2 . More specifically, we want to be able to compute for any point p = (u, v) ∈ R 2 the following values:

I(p) ∇I(p) = ∇ p I(p) = ∂I ∂u ∂I ∂v
That way, we will be able to tell the variation of the function I according to p and use it in an gradient retropropagation workflow. Here, we present the bilinear interpolation, which is the one used in Spatial Transformer Networks (STN) [START_REF] Jaderberg | Spatial transformer networks[END_REF].

First, if we consider the image size, the image function I can be extended to Z 2 , by setting its value arbitrarily outside of size limitations. Depending on conventions, it can be:

1. I(p) = 0, this is used when sampling an isolated point outside of Ω, in order for gradient to be null.

2. I(p) = I(q), q = arg min q∈Ω ( pq ), this is used when computing image gradient at image boundaries.

Interpolation can then extend the function to R 2 by finding possible values between integer positions. Several interpolations techniques can be used, such as nearest or bicubic.

The bilinear interpolation on which we focus on tries to find the quadratic function f i,j : [0, 1] 2 → R 3 between 4 adjacent points as shown in figure 4.1, that solves the equations related to these points, where value is known.

f p=(i,j)∈Ω : [0, 1] 2 → R 3 (u, v) → I(p) + u∂ x I + v∂ y I + uv∂ xy I
Where

∂ x I = I(i + 1, j) -I(p) ∂ y I = I(i, j + 1) -I(p) ∂ xy I = I(p) + I(i + 1, j + 1) -I(i + 1, j) -I(i, j + 1)
The extended image function is then the piecewise function composed of all different quadratic solutions to the systems stated above.

I : R 2 → R 3 (u, v) → f u , v ( f rac(u), f rac(v))
where f rac is the fractional part defined as xx for x ∈ R. One should note that this function is continuous everywhere and almost differentiable everywhere, but it's not a C 1 function, gradient generally has discontinuity when approaching p lim ∈ (R × Z) ∪ (Z × R). Besides, it can be shown that wherever defined, Laplacian is 0 everywhere. As discussed later, for these values to be computed on Ω, and especially when trying to optimize them, we will use the scale space representation [START_REF] Lindeberg | Scale-space: A framework for handling image structures at multiple scales[END_REF], which makes the picture C +∞ everywhere, but has a blurred representation of I.

From now on, for the sake of compact notation, we will consider any 2D function on Ω to be a differentiable function, and for any image I , the notation I(p) with p = (u, v) ∈ R 2 will implicitly refer to the bilinear interpolation extension of this image.

Optical flow and reprojection based optimization

Thanks to its differentiability, an optimization by gradient descent can be applied to the bilinear interpolation of an image. For a particular color value c in R 3 , one can try to find the optimal p coordinates where p = arg min p { I(p)c }.

An extension to this simple optimization is to consider an array of points to retrieve from one image I t to another I t+1 . Indeed, each point to retrieve is defined as a color that is assumed to be the same in the other image. This optimization then tries to compute optical flow F.

∀p ∈ Ω, I t+1 (p + F(p)) = I t (p) ∀p ∈ Ω, F(p) = arg min q∈R 2 I t (p) -I t+1 (p + q) (4.1)
Note that optical flow F, similarly to images is a 2D function defined on Ω, thus all aforementioned notations can be used, including interpolation. In practice, the optimization tries to minimize the following photometric loss, defined by L p .

L p = ∑ p∈Ω d (I t (p), I t+1 (p + F(p)))
where d is a distance function such as L1 distance.

By using gradient descent, we then get for each coordinate

F = F 1 F 2 of our optical flow ∀p = (i, j) ∈ Ω, ∂ t F(p) =   ∂ t F 1 (p) ∂ t F 2 (p)   = -   ∂L p ∂F 1 (p) ∂L p ∂F 2 (p)  
This equation can be more generally written as:

4.3. Retrieving a depth map using reprojection based optimization ∂ t F = -∇ F L p where ∇ T is the gradient operation with respect to the parameter vector T. In this notation, F is considered a 1D vector of size |Ω| × 2 By using Euler's method we then update F as follow:

F ← F -γ∇ F L p
where γ is the optimization step value.

The optical flow problem is actually the root of reprojection based optimization. Our depth problem, is in fact a composition of a function that outputs an optical flow and a photometric error.

Note on inverse warp

Throughout this subsection, we implicitly introduced the operation of inverse warp. Indeed, when optimizing optical flow from I t to I t+1 , we constructed an image similar to I t while sampling colors from I t+1 . This operation is called inverse warp because even if the optimized optical flow map indicates the displacement of I t pixels, the constructed image moves I t+1 pixels and not I t pixels, and the applied optical flow F from I t+1 to I t is the inverse of F.

∀p ∈ Ω, p = p + F(p) p = p + F (p )
The difference with direct warp, where I t pixels would have been moved so that constructed image would resemble I t+1 will be discussed on section4.5.

Retrieving a depth map using reprojection based optimization

Retrieving a depth map can be considered as a subproblem of our initial optical flow problem. Indeed, from an estimated depth θ t and pose estimation T t→t+1 = (R, t) (possibly known from other means, e.g. dedicated sensors), we can deduce an optical flow map. Using equation 3.4 from chapter 3, and assuming a rigid scene, we get for any point coordinate p in I t :

F(p) = p -Π KRK -1 Π -1 (p) + Kt θ(p) (4.2)
Practically, this subproblem consists in modeling the reprojection with less parameters: instead of optimizing the full optical flow, which is |Ω| × 2 independent elements, we simply optimize a depth map of |Ω| elements and a 6 degrees-of-freedom vector of displacement.

An important aspect of this problem to consider is the fact that estimated depth θ t is only a function of I t while pose T t→t+1 is function of both I t and I t+1 . This is particularly interesting when considering another frame pair still involving I t . For instance, when considering I t+2 with its associated pose relative to I t , T t→t+2 , the new reprojection optimization is still modifying θ t along with T t→t+1 . Only 6 new parameters are added to the problem.

More generally, when considering one key frame I t and a set of N reference frames I k , one can retrieve depth with joint reprojection optimization, as it only requires W × H + N × 6

T t→i = (R i , t i )

Training a neural network to output depth with reprojection based optimization

The next logical step from this depth retrieval is then to train a neural network to output depth with this technique, also called auto-supervision. The supervision is not made on the target depth map anymore, since the network trains on a lot of different samples. Indeed instead of computing a loss relative to the difference of our network's output and an estimated depth map from reprojection based optimization, the photometric loss function will serve as a direct value to optimize for the network, which will eventually result in an indirect optimization of depth values from its output.

SFMLearner from Zhou et al., a quick presentation

To illustrate this training workflow, we partly introduce a very influential and yet simple work on auto supervision from Zhou et al. [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF]. All other works of this field (including this one) are actually built on this fundamental basis.

As mentioned in the introduction of this chapter, this work relies on a neural network to produce a depth map (more specifically, the network called DispNet outputs an inverse depth map), but also on another network to produce a pose (called PoseNet). Both networks are simple CNNs. PoseNet is inspired from VGG[SZ14], while DispNet is inpired from FlowNet [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF], just like our DepthNet. This work may not be the most recent, but will serve as a basis for comparison, as further improvement built on this work [Cas+19; Ran+18; YS18] mostly try to address the non-rigid scene problem which we chose not to focus on. The general workflow is presented figure 4.2. The depth is estimated at a target frame called I t , located at the middle of a sequence. Other frames are called reference frames I i with i ∈ 0, N -1 . This work is very interesting for us, because the supervision need is very low, we don't even need to know the displacement which is trained to be estimated. Using this technique might help us to quickly get a first prototype for a real world DepthNet finetuning workflow with very few changes.

Conclusions of reprojection based optimization

Our review showed that it was possible to train a neural network to output depth only with a bilinear sampling operation which is differentiable, information about camera movement and geometric calibration. No depth groundtruth is involved in the training process.

We also have a way to construct a training algorithm that does not require camera movement to get significant results for both depth and pose estimation. Indeed, the work of Zhou et al. is very similar to what we want to achieve, and require a few changes for it to fit our needs regarding DepthNet :

• Replace DispNet with DepthNet

• feed two images to DepthNet, with rotation compensation, this is possible with PoseNet.

• normalize PoseNet translation estimation magnitude to get DepthNet nominal displacement These changes will be discussed during the next chapter. Instead, we are going to study this work as thoroughly as possible with the hope that fixing potential problems in this context will improve the basis for a better unsupervised DepthNet training.

The next sections will discuss the context of training a network to output depth within a sequence of frames, with a known pose, and especially the two main independent limitations of the reprojection based optimization used alone, and how we might be able to overcome them:

• Regularizing a depth map to be physically plausible (section 4.4)

• Filtering invalid reprojection errors due to occlusions (section 4.5)

Regularization with smooth loss and diffusion

In addition to photometric loss L p , SFMLearner's algorithm also introduces a smooth loss. This loss tries to regularize the depth network's output by averaging Laplacian absolute values of its output.

L s = 1 |Ω| ∑ p∈Ω |∆DispNet(I t )(p)|
This regularization problem is actually very common in classic image processing operations such as denoising or optical flow estimation.

After studying smooth regularization in context of the more well known optical flow estimation problem, this section will focus on presenting a review on possible smooth losses to apply on a network output to improve the resulting depth and making it as physically plausible as possible.

Optical flow smoothing

We discussed the color consistency optimization in its simplest form that assumes that finding the global minimum of eq 4.1 is the same as finding optical flow. Obviously this assumptions does not hold in general because the mapping from position to color is highly non-injective, and multiple points can have similar colors. This is the case for example in a texture-less area, where the color is similar on a large area of points.

To solve this problem, several assumptions are made on typical realistic optical flow. For example, Horn and Schunck [START_REF] Berthold | Determining optical flow[END_REF] extended the reprojection optimization with a smooth constraint, so that points close to each other have similar optical flow values. This assumption is related to the fact that when looking at moving objects, optical flow is continuous with respect to surface.

It can be noted that continuity on a discrete 2D map is not physically meaningful. In practice, reducing the discrete Laplacian ensures value proximity between nearby points. This is known as diffusion, inspired from the heat equation.

∂ t F = ∆F =   ∆F 1 ∆F 2   =   ∂ 2 ∂u 2 F 1 + ∂ 2 ∂v 2 F 1 ∂ 2 ∂u 2 F 2 + ∂ 2 ∂v 2 F 2  
where ∆ is the Laplacian operator, which is then approximated by updating its value using Euler's method and (u, v) are real values inside [0, H] × [0, W]. In practice, the derivative is constructed from difference values within integer positions inside Ω, using convolution kernels, such as Sobel for first order derivate and discrete Laplacian for second order, in accordance with the scale space method [START_REF] Lindeberg | Scale-space: A framework for handling image structures at multiple scales[END_REF].

∆F = 0 1 0 1 -4 1 0 1 0 * F F ← F + λ ∆F
where λ is the step value.

The new optimization workflow is then

∂ t F = -∇ F L p + λ∆F (4.3) F ← F -γ∇ F L p + λγ∆F
where λ is the smoothing rate and γ the optimization step.

Applying smoothing on a depth map

Similarly to optical flow, we can try to regularize a depth map. The first idea might be to simply smooth the resulting optical flow from depth and motion. However, we can take this regularization further by making several physical assumptions directly on the depth map and try to apply them as a diffusion operation. It is important to formulate them precisely to determine how a depth map can be regularized, and what constitutes a realistic depth. Instead of deciding how we want our depth maps to be like, we must first think about it the other way round: "Given a perfect depth map, what regularization would make it stable?"

• For naturally occurring scenes, depth discontinuity is possible only when image colors are also discontinuous. This is not necessarily true for the opposite, when for example the camera is looking at a textured plane. This is the case when looking at a road with markings for example.

• We suppose 3D points to describe a continuous differentiable surface. As a consequence, every surface point when zoomed enough from any point of view would look like a plane. A plane surface should be possible to converge to, even if it is not normal to the camera and goes to an infinite value, this case can be illustrated when looking at the horizon for example. In other words, the smoothing regularization on a depth map of a plane surface should be ineffective. • We can also mention the problem with a completely uniform background. Since there is no texture, the photometric loss does not create any incentive to change its value there since there is no error. One of the goal of regularization will be to keep this background at the right value. In other word, any regularization happening on a textured surface cannot spread to a texture-less area if there is a clear boundary between the two surfaces.

It is worth mentioning that the first assumption for image discontinuity is not always true, as the human mind itself assumes surface (and thus depth) discontinuities in well known optical illusions called subjective contours [START_REF] Coren | Subjective contours and apparent depth[END_REF]. This not only indicates that our learning workflow is not the same as the natural one experienced by humans, but also that under very specific light conditions, this could happen, making our whole optimization fail. While being probably worth studying in both neurological and computer vision contexts in further works, we believe these examples to be marginal in nature and choose to ignore them here.

In this section, we are interested in regularizing our optimization problem by smoothing a 2D map, the same way optical flow needed to be smoothed in Horn and Schunck [START_REF] Berthold | Determining optical flow[END_REF]. The first thing we need to do is to transpose the physical assumption of our surfaces into the depth map.

Regardless of the method used for smoothing, we consider a 2D map to be smooth at some point when its Laplacian is zero, because it's locally planar. Consequently, this is also a map stable to diffusion.

Translating physical assumptions into equations

For our optimization to be physically accurate, We must find a map where we would actually want its Laplacian to be zero. For example, as shown figure 4.4, when looking at the horizon, depth gradient ∂θ ∂y is not of constant gradient. However, inverse depth ξ is piecewise affine.

Proposition 2. The inverse depth map ξ = 1 θ of a perfect plane has a null Laplacian and is then stable with respect to smoothing. See appendix C for proof. From now on, our study will be focused on optimizing depth map, with a smooth constraint on inverse depth, called ξ.

Smooth loss presentation

In an attempt to generalize this "smooth constraint", this section will study loss based smoothing, as it is widely used e.g. in image denoising. It is an optimization problem where we design a smooth loss L s and add it to the other loss we try to minimize.

For a particular 2D map ξ defined on a surface Ω, we will study two smooth loss forms, trying to minimize gradients:

L s∇ = 1 2 Ω c ∇ξ 2 dS (4.4)
or trying to minimize Laplacians:

L s∆ = 1 2 Ω c (∆ξ) 2 dS (4.5)
These equations can be used to retrieve well known denoising loss functions. Total Variation, also called TV Loss [START_REF] Leonid I Rudin | Nonlinear total variation based noise removal algorithms[END_REF] is a special case of equation 4.4 where

c = x → 2 √ x. L TV = Ω ∇ξ dS
Besides, we will link the general expression of these losses with diffusion by studying their gradient. Definition 4. For a particular loss function L, if there is a scalar or vector field T such that ∇ T L = -∆T the loss wi ll be called diffusion loss with respect to T, because if T was the training parameter for the gradient descent optimization, we would get the diffusion operation

∂ t T = -∇ T L = ∆T
This is the case when considering L s∇ and c = x → x, as shown by Perona and Malik [PM90], because we have at the interior of Ω ∇ ξ L s∇ = -∆ξ (see appendix D.1) As such, we now identify the loss L s∇ with c = x → x as the diffusion loss.

L di f f = 1 2 Ω ∇ξ 2 dS (4.6)

Diffusing the output of a neural network

One must be careful when talking about diffusion in the context of a function output, here a neural network. In the case of a regular diffusion such as with Horn and Shunck [START_REF] Berthold | Determining optical flow[END_REF], the trainable parameters are the map we want to smooth itself.

As such, if we apply a smooth loss L s on ξ and ignore the other losses, the optimization step is as simple as:

∀p ∈ Ω, ∂ t ξ(p) = -∇ ξ(p) L s ξ(p) ← ξ(p) -γ∇ ξ(p) L s (4.7)
finding the value ∇ ξ(p) L s will completely characterize the smoothing process on ξ.

In the case of an output of a function (here, a neural network), the general optimization step becomes for a vector of trainable parameters W (for example, the weights of a neural network) and a function f such that f (W ) = ξ:

∂ t W = -∇ W L s = -J f (W ) ∇ ξ L s
Where J f (W ) is the Jacobian of f evaluated in W and M → M the transposition operation. It should be noted that for J f (W ) to be a matrix, we must consider ξ as a vector of values, ξ ∈ R |Ω| (the vector dimension is the number of pixels).

This equation can be also written like this:

∀i, ∂ t W i = ∑ p∈Ω ∂ f ∂W i (p) × ∂L s (ξ) ∂ξ(p)
As a consequence, we can have the evolution of ξ

∂ t ξ = J f (W )∂ t W = -J f (W )J f (W ) ∇ ξ L s (4.8)
It appears that when f is identity (and thus W = ξ), J f is also identity and when L s is the diffusion loss with respect to ξ, can retrieve the behaviour presented in definition 4. In practice, the Jacobian of a neural network is very computationally expensive to get, so there is no guarantee about ξ = f (W ) evolution. The only thing we know is given a step little enough, L s will be lowered, but we don't even know the maximum size of that step.

Of course, this study generalizes to any kind of loss whose gradient with respect to a function output can be computed. Although widely studied on the regular context (when Jacobian is identity matrix), loss functions may have robustness problems for an unknown Jacobian. Appendix D.2 shows in a simple example how a too large optimization step size might make a 2D map diverge, and in appendix D.3 how a simple function such as inverse will make the regularization step size unknown. This is particularly interesting for our use-case, since DepthNet is supposed to output depth directly, so we will end up trying to smooth its inverse. Note that the ELU function [START_REF] Clevert | Fast and accurate deep network learning by exponential linear units (elus)[END_REF] we used (see 3.5.3) might help mitigating gradients magnitude of ∇ξ when estimated depth θ is close to 0 (and thus ξ is high).

Adding robustness to smooth loss

From now on, for a particular smooth loss function L s with respect to a scalar or vector map T, its gradient ∇ T L s will be discussed but in the general case, additional steps might be added. Namely, we can decompose the optimization process into three separate steps:

1. consider the T = f (W ) map we would like to smooth that is the output of a neural network, like e.g. ξ or ∇ξ, as a set of trainable parameters.

2. perform a smoothing operation (such as diffusion) directly on it.

3. design the loss function as a distance between the smoothed map T smooth and T we actually want to smooth with the parameters W. L s = d(T = f (W ), T smooth ).

This method might use any kind of smoothing method, as long as it is applied directly on T.

Two step joint optimization

An interesting idea to investigate in the future would be to further apply this idea of decoupling smoothness and photometric optimizations. This idea first introduced by Steinbrucker [START_REF] Steinbrücker | Large displacement optical flow computation withoutwarping[END_REF] is also applied within DTAM algorithm [START_REF] Richard A Newcombe | DTAM: Dense tracking and mapping in real-time[END_REF], and aims at finding the solution of equation 4.9, for a fixed estimated inverse depth ξ = 1 θ . d is a distance function, e.g. L2 in the case of DTAM, and γ is an arbitrary positive hyper-parameter. Following e.g. DTAM method using Legendre-Fenchel transform and primal-dual optimization [START_REF] Chambolle | A first-order primal-dual algorithm for convex problems with applications to imaging[END_REF], a global minimum could be found in real time (with a 2011 high end customer hardware), and it can potentially be reused here, alternatively with optimization of the same equation 4.9 but with fixed α.

L( θ, α) = L p ( θ) + λL s (α) + γd α, ξ = 1 θ (4.9)
The global optimization can then become:

1. estimate θ = f (W ) from trainable weights W.

2. optimize L( θ, α) with θ fixed, find the global solution α = arg min α L( θ, α) with an off-the-shelf algorithm.

3. compute ∇ W L( θ, α) and update W.

update the γ value if needed, ideally it would lower throughout the training, enforcing

a stricter similarity between α and ξ.

Minimizing gradient with L s∇

As it is used by most published works, we can briefly analyze variations of the smooth loss function mentioned eq 4.4

L s∇ = 1 2 Ω c ∇ξ 2 dS
As said above, but in a more general way, we know that simple gradient descent applied on ξ of this cost function results in the following inside the boundaries.

∂ t ξ = -∇ ξ L s∇ = div c ( ∇ξ 2 )∇ξ (4.10)
For example, the total variation loss (called TV loss) [START_REF] Leonid I Rudin | Nonlinear total variation based noise removal algorithms[END_REF] mentioned above is a widely used constraint in partial derivative equations (PDE) based inpainting algorithms [START_REF] Shen | Mathematical models for local nontexture inpaintings[END_REF], and uses the same idea, but with gradient norm.

L TV = Ω ∇ξ dS From identifying c = x → 2 √ x, we can deduce ∇ ξ L TV = -2div ∇ξ ∇ξ
If instead, we have c = x → x, we end up with this already discussed equation:

∇ ξ L di f f = -∆ξ (4.11)
We then retrieve the gradient that would result in a diffusion if ξ was the trainable parameter.

Global solution of diffusion loss L di f f

Although not the global solution, from equation 4.10 it seems that a constant slope (and thus a null Laplacian) would be a stable position for this optimization, regardless of c function. This is not the case because the integration is made on a finite map, and the exact term on the boundary of Ω is:

∂ t ξ = -∇ ξ L di f f = div c ( ∇ξ 2 )∇ξ -c ( ξ 2 )(∇ξ.n) = ∆ξ -∇ξ.n
where n is the unit vector normal to the boundary. If we consider the 1D example and a map where ∆ξ = d 2 ξ dx 2 = 0 everywhere, we get:

∂ t ξ(0) = - dξ dx (0)
Following the convention of computing gradient in a picture boundary with values inside the interior of that set ( dξ dx (0) = lim x→0+ dξ dx (x)), we can see that increasing ξ(0) while keeping ∂ t ξ = 0 inside of Ω (since ∆(ξ) is null everywhere) will then decrease the gradient dξ dx (0) and vice-versa. ∂ t dξ dx (0) is then of the same sign asdξ dx . In other words, dξ dx will be lowered, and gradient will converge to 0. The same applies to ∂Ω, and can be extended to the 2D case:

∀p ∈ ∂Ω, ξ(p).n --→ t+∞

With propagation of this gradient, we can see how this will make the whole map converge to a constant plane.

Although not developed here, an interesting solution to this problem would be to not apply the optimization on δΩ. That way, the points on boundaries would be unaffected by the optimization making the rest of the image stable with a non-null gradient. This is potentially hazardous as points in boundaries often are noisy because it lacks neighbouring pixels informations for the main photometric loss to be reliable, making smooth regularization essential for them. In our 1D use-case, if we use this method, the only solution for the smooth loss would be a linear interpolation between the two boundary values, regardless of starting values in the interior of Ω

Minimizing Laplacian with L s∆

In the case we want a null Laplacian but not necessarily a constant map, some recent works proposed to minimize the sum of all absolute values of Laplacians of a 2D map [Zho+17; WB18]. For ease of notation, we will call this loss TVV for total variation of variation, in reference of total variation Loss [START_REF] Leonid I Rudin | Nonlinear total variation based noise removal algorithms[END_REF]. This equation can be retrieved from With TVV loss, the only points in the inverse depth map ξ where an optimization is actually done are the array bounds and the inflexion points, where Laplacian changes of sign.

This optimization, even though it has empirically proved to converge to blurry version of the initial image after several optimization steps is far from regular.

Let's instead look at the general smooth loss L s∆ that tries to lower Laplacians.

L s∆ = 1 2 Ω c (∆ξ) 2 dS
We can show that the resulting ξ-wise gradient loss is

∇ ξ L s∆ = -∆(c ((∆ξ) 2 )∆ξ) (4.12)
(see appendix D.4.1 for a demonstration) we can retrieve our first example of TVV Loss when c = x → 2 √ x. We then get

∇ ξ L TVV = 2∆ ∆ξ |∆ξ|
since ∆ξ |∆ξ| is either 1 or -1, its differentiation is always 0 or infinite. In practice, since discrete differentiation is approximated with finite differences, we actually get a finite value when ∆ξ changes of sign.

∀p ∈ Ω, ∇ ξ L TVV =      1
if ∆ξ < 0 and changes of sign -1 if ∆ξ > 0 and changes of sign 0 otherwise When c = x → x, we have:

∇ ξ L s = ∆ 2 ξ (4.13)
This equation is similar to the equation that qualifies a diffusion loss (definition 4) but is not of the form ∇ T L = -∆T However, knowing that ξ is in fact a function and not the trainable parameters, rather than studying Laplacian minimization of it, we can see this loss as minimizing the divergence of its gradient ∇ξ and get insight from the gradient of loss with respect to ∇ξ, ∇ A L s∆ with A = ∇ξ.

Diffusion of a vector map with divergence minimization

Let us consider the 2D vector map A. We want to minimize its divergence div(A). The motivation for this is that when A = ∇ξ, we get the equivalence div(A) = ∆ξ.

We can then modify equation 4.10 to work with divergence of a vector instead of gradient of a scalar:

Let us consider this modified loss function, from 4.5, by replacing ∆ξ with divA:

L s∆ = Ω c div(A) 2 dS
From this equation, assuming the trainable parameters are components of A, we get in Ω (the interior of Ω):

∇ A L s∆ = -∇ c div(A) 2 div(A) (4.14)
(see appendixD.4.2 for a demonstration) we can retrieve our first example of TVV Loss when c = x → 2 √ x. We then get

∇ A L s∆ = 2∇ div(A) |div(A)| When c = x → x, we have ∇ A L s∆ = -∇ (divA) (4.15)
And especially, when at first A is of the form ∇ξ, we get the following equation, no matter how A is derived from, even when it is diffused independently of ξ, for example in the fashion of two steps optimization (section 4.4.5). See appendix D.4.3 for a demonstration.

∇ A L s∆ = -∆A (4.16)
The minimization of square values of Laplacian, as described by equation 4.5 with c = x → x can then be called gradient diffusion loss L gdi f f . Because we have

L gdi f f = Ω (∆ξ) 2 dS ∇ ∇ξ L gdi f f = -∆(∇ξ) Figure 4
.5 presents a comparison for the four mentioned techniques in a 1D example: TV, diffusion, TVV and gradient diffusion. We test a smooth loss optimization with a sinusoid and and abs function with added noise. Stochastic gradient descent (SGD) is used, with indicated learning rates and momentums in each graph title. As shown and discussed section 4.4.6, TV loss and Diffusion Loss seem to converge to a constant slope, but also a constant value. On the other hand, the first TVV appears to smooth the function enough to denoise it, but fails to converge to a simple affine function. Finally, gradient diffusion loss converges to an affine function, but needs ten times more iterations to achieve a reasonable result, with a higher momentum of 9.99. This function which makes intervene the fourth derivative of ξ (see equation 4.13) is one of the reasons the convergence is so slow.

Edge-aware smooth loss

When trying to smooth a 2D map relative to a particular image, for example optical flow F, depth θ or inverse depth map ξ of image I, although useful for texture-less area, smoothing everything equally might be not ideal when considering discontinuities (and then the Laplacian is theoretically infinite). For example occlusion areas in a depth map. The map is then "over smoothed" [START_REF] Garg | Unsupervised cnn for single view depth estimation: Geometry to the rescue[END_REF].

As tested by many previous works [NE86; WB18; MWA18; Ran+18; YS18], a simple way to overcome the smearing of edges in our map, is to temper the smoothing when the image has edges, i.e. when it has gradients with high norm.

To establish an efficient strategy, we can look at the image denoising techniques, and where we can use it differently for inverse depth map texture-aware smoothing.

Taking back our smooth losses equations 4.4 and 4.5, we can replace c function with more complicated ones, especially function with decreasing derivate. As such, smoothing will be less enforced for high gradients. For example Perona and Malik [START_REF] Perona | Scale-space and edge detection using anisotropic diffusion[END_REF] proposed the following derivate function g = c (among others) for the image I:

g : ∇I 2 → exp - ∇I κ 2 (4.17)
This technique is called anisotropic diffusion, contrary to the regular diffusion (also called isotropic), which has the same diffusion strengh everywhere.

Besides, in their work, they presented a mathematical analysis of the function g.

• They show that with any positive function g, the smoothed image never diverges outside original image extrema (given a small enough optimization step), this is the maximum principle.

• They also show that if g is differentiable, given the sign of Φ where Φ = x → xg(x), the edges will be smoothed (if Φ > 0), or enhanced (if Φ < 0). The image diffused that way converges to a piecewise constant map where gradient is either null or infinite.

From equation 4.17 we can compute the function Φ and its differentiation Φ . In this particular function g, edges with a gradient of norm above κ √ 2 get enhanced, and mitigated otherwise. Figure 4.6 shows the picture of a flower (with values in [0, 1]) that got diffused with a κ of 0.3. We can see that the picture tends to converges to a 3 colors image.

∀x ∈ R + Φ (x) = g(x) + xg (x) = g(x) 1 -2 x κ
Our case is different here, since we don't have only the map ξ to work with: we also have the image I from which ξ is extracted. We don't want to preserve the edges of ξ (or ∇ξ), but we want ξ edges (or ∇ξ edges) to match image I edges. As discussed section 4.4.2, we want depth discontinuities to coincide with image discontinuities.

Nevertheless, we can be inspired by Perona and Malik proposition to adapt our loss functions so that c = x → h(∇I)x, where h is a decreasing function. For example:

∇ ξ L di f f (ξ) = div g ∇I 2 ∇ξ = div exp - ∇I 2 κ 2 ∇ξ
Similarly, with diffusion of ∇ξ, we would get

∇ ∇ξ L gdi f f (∇ξ) = ∇ g ∇I 2 ∇.A = ∇ exp - ∇I 2 κ 2 ∆ξ
Here, we mentioned ∇I which is in fact three different vectors (since image is composed of three channels). To compute the norm, we implicitly take the mean of the norms of the three gradient vectors.

One should note that if the maximum principle can still be respected for ξ diffusion, when diffusing gradient, this is only the case with respect to ∇ξ: the slope is bounded from the initial depth map, but not the actual values. It can have some problems, for example when the diffusion makes ξ go under 0 which is not physically possible. So it may be necessary to add regularization to prevent that. The simplest way is to manually bound the output of the network so that ξ = f (W ) > with a fixed strictly positive value.

The equivalent losses L di f f and L gdi f f to these equations can then be computed by simply integrating these g functions. It is actually fairly easy since ∇I is not a function of ξ, so g( ∇I 2 ) is constant with respect to ξ.

Final edge-aware forms of considered smooth losses

For loss functions L s∇ (equation 4.4) and L s∆ (equation 4.5, we can study a particular form of c which includes influence of ∇I. TV and TVV losses are modified to match literature, as it is used with DTAM [NLD11] and Godard et al. [START_REF] Godard | Unsupervised Monocular Depth Estimation with Left-Right Consistency[END_REF].

L di f f (ξ) = Ω exp - ∇I κ 2 ∇ξ 2 dS (4.18) L TV (ξ) = Ω exp - ∇I κ ∇ξ dS (4.19) L gdi f f (∇ξ) = Ω exp - ∇I κ 2 (∆ξ) 2 dS (4.20) L TVV (∇ξ) = Ω exp - ∇I κ |∆ξ|dS (4.21)
It's easy to see how a high κ value makes it equivalent to our first simple loss functions definitions.

Figure 4.7 shows a comparison of anisotropic diffusion and anisotropic gradient diffusion on a set of 1D examples with a simple toy image (dashed blue curve) whose gradient is 0 everywhere except in pixel coordinates 20 and 80 where it is above the κ value. These results indicate that as we foresaw, diffusing ∇ξ is more stable relative to image edges: last example shows a typical texture plane surface, and regular diffusion creates an edge from nowhere to make the depth piecewise constant. We can also see that our first example shows a typical transgression of the maximum principle: in order to be piecewise linear, ξ is rised above the sinusoid maximum. However, in the same time, it appears that the portion on which this happens seems to be a good linear regression of our function between the two discontinuities. Lastly, it can be noted that regular diffusion converges much faster than gradient diffusion.

As a conclusion for this section, we have a set of different losses that we can apply to a specific 2D map that might be the output of a network. In addition to the actual function c and the weight we can apply to this loss, we can also change a κ parameters to change behavior on images areas with texture (i.e. with a high gradient ∇I). We also have a robust counterpart of these losses designed to avoid divergence, which might be indicated for particularly non-linear neural network function. The inverse input of a network being one of them.

Dealing with occlusions

Occlusions occur when an object moves behind an other. From one frame to another, the object is then not visible anymore. This section studies how this phenomenon compromises the reprojection principle which tries to retrieve pixels from one frame to another, and what filtering can be applied to be perform a robust training.

Inverse warp based optimization vs occlusions

Taking back the first inverse warp photometric optimization problem, another aspect to take into account is the occluded areas. As seen figure 4.8, some pixels can be occluded from one frame to another. In this example, the torus should not be retrieved in the first image but can be found in the last image. Similarly, the cube should not be retrieved in the last frame, but can be found in the first frame. As a consequence, the task of retrieving them in both frames is impossible and the optimization will eventually converge to a false solution. Let's consider our problem with I t , I t 2 and T t→t 2 , we try to reconstruct θ t indirectly by reconstructing I t with mentioned parameters and camera intrinsics K. If we consider the function IW (for inverse warp) that performs this image reprojection, we can define it as:

IW : R |Ω| × R 3×|Ω| × R 6 × R 3×3 → R 3×|Ω| θ t , I t 2 , T t→t 2 , K → I t 2 →t
The photometric loss is then:

L p = ∑ p∈Ω d(I t , IW( θ t , I t 2 , T t→t 2 , K))(p) = ∑ p∈Ω d(I t , I t 2 →t )(p)
For example, if I t+1 is the image I t 2 in figure 4.8, the optimization process will try to reconstruct the cube in image I t with pixels from I t 2 although it does not appear in the image.

Another way to view this issue is to see that a perfect depth map would produce an imperfect reconstruction, resulting in a non-zero loss. The optimization will then modify the depth map to better satisfy the photometric loss. Figure 4.12 shows an example of imperfect inverse warp caused by a perfect depth map, with a "ghost" of foreground on the occluded background.

Filtering occlusion areas from optimization

One solution for solving this problem is to try to detect the occlusion and then filter them from the photometric error to get L p f (for photometric filtered).

L p f (t 2 ) = ∑ p∈Ω d(I t , I t 2 →t )(p) if p not occluded in I t 2 0 otherwise (4.22)
This technique, although not guaranteed to converge to the ground truth, at least makes the actual solution stable with the optimization. The smooth loss regularization is supposed to help in these areas where photometric loss is undefined.

As said section 4.3, the depth retrieval problem can benefit from multiple pairs of frames at the same time (one of them being invariably the frame of which we want to estimate the depth). As such, the occluded areas, relative to a particular frame pair won't be the same if the relative movement is different. Consequently, a simple solution to the occlusion problem in two frames stated before is here to retrieve the pixel in multiple frames, before and after, as illustrated figure 4.8. Although not guaranteed, the pixel has greater chances to be found in another frame, especially when the movement is consistent throughout the frame sequence: relative to I t , occlusion areas for the pairs I t → I t+n and I t → I t-n should be completely different since the associated relative displacement is deemed almost opposite. Assuming occluded areas could be identified between two frames, the depth map θ t can be estimated using redundancy in multiple optical flow maps.

L p f = ∑ t k L p f (t k ) (4.23)
The next sections will discuss the different methods for detecting the occlusion areas.

Self filtered depth map inference

Several works [Jan+18; Zho+17] proposed a dedicated network to figure out the occluded scenes, or more generally, the areas on which the photometric error won't be low even if the optical flow map is perfect, it can be interpreted as a confidence network for photometric loss. The main idea is to multiply the photometric loss by a confidence value E(I t , I t 2 ) outputted by a network. During the training, the network is then encouraged to have low values where photometric error is high. To avoid the degenerate solution with 0 confidence everywhere that would result in a 0 photometric loss, the network is also encouraged to output high confidence values with e.g. a binary cross entropy H 1 with ground-truth set to 1.

L = EL p + αH 1 ( E)
The confidence network tries to find a compromise between these two antagonistic losses, and is deemed to find it where photometric error are not easily lowered, be it from occlusion, moving object or graphical artifacts such as non Lambertian surfaces or change of exposition. This is a promising technique, somewhat related to Bayesian inference, where the network also predicts uncertainty, but it adds lots of new hyperparameters to the training workflow, especially the architecture of this new network and the weight to put to α.

Backward/forward consistency

Instead of asking a neural network to decide what constitutes an occluded area, we can solve this problem analytically. In this case, we extract the occlusion areas not from the image input like with self filtering, but rather from the pose and estimated depth. The occlusion areas will then be tied to the depth estimation, possibly erroneous but it can be geometrically deduced and not estimated. To better understand how we can analytically find these occlusion areas, we can look at the more general case of optical flow.

Optical flow consistency

When considering optical flow optimization, one solution for this occluded area problem is the forward/backward consistency check. This technique not only tries to compute forward flow F + from I t to I t 2 , but also the backward flow F -from I t 2 to I t . The general idea behind this redundancy is that these optical flow maps are expected to be consistent up to the occluded areas. Areas where optical flow is not consistent within forward/backward is excluded from the optimization.

∀p ∈ Ω, F -(p + F + (p)) + F + (p) = 0 (4.24)
In other words, a point warped from I t to I t+1 with F + should be warped back to its original place with F -.

Since the photometric optimization problem is based on optical flow, we can apply this filtering method to optical flow maps that results from depth and pose inference. However, it can be seen that two different optical flow maps will be needed, F + and F -. That means that we would need two different depth maps to deduce these optical flow maps, and we also would require them to be perfectly consistent.

This technique has been used for example in the case of GeoNet [START_REF] Yin | GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose[END_REF] and by Ranjan et al. [START_REF] Ranjan | Adversarial Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation[END_REF].

Depth map consistency

Occlusion zones can also be determined analytically directly from depth and displacement. We can adapt the backward/forward consistency method used for optical flow explained above to have a depth consistency method.

The idea is to adapt the inverse warp function to reconstruct depth maps instead of images. Instead of inverse warping I t 2 into I t 2 →t to reconstruct I t , we can try to inverse warp θ t 2 into θ t 2 →t and only optimize the difference between I t and I t 2 →t where θ t and θ t 2 →t are close. This solution however also requires θ t and θ t 2 to be both estimated, and these estimations to be consistent.

When training depth maps from scratch, and thus having them random and independent at first, how can we distinguish inconsistencies related to a bad depth estimation from those related to occlusion? We are not certain the neural network won't fall into a degenerate solution One solution could be to carefully engineer the training workflow. Since this problem typically appears at the end of the convergence, this regularization can be applied with a very low weight at first, and then when the network converges, we can assume sufficient reliability in the depth estimation so that inconsistency is mainly caused by the occlusions.

Unsurprisingly, these techniques of optical flow and depth forward-backward consistency with inverse warp encourage trial and error, and are prone to over-fitting with respect to the validation set. As our main goal is to be robust to drone videos heterogeneity, we need a more general regularization, that does not need to make assumptions on the dataset.

Occlusions detections with direct warp instead of inverse warp

Our main problem with inverse warp consistency check is that it needs two different depth maps. What we want is a way to estimate θ t 2 only with θ t and T t→t 2 . By looking at reprojection equation ( 4.2), we can see that for each point p in Ω a corresponding point in R 2 is found. The problem of inverse warp was initially to get the color of a point in R 2 using interpolation from points in Ω.

This is now the opposite problem: by a set of point of R 2 with corresponding colors, how can we know the values of points in Ω? This problem can be called depth based image rendering (DIBR) [START_REF] Sun | An overview of free view-point depth-image-based rendering (DIBR)[END_REF]. Mathematically, this can be understood as finding the inverse of equation 3.4, from chapter 3: for a point p ∈ Ω, find p 2 ∈ R 2 such that

p = Π KT t→t 2 ( θ t (p 2 )K -1 Π -1 (p 2 ) (4.25)
Unfortunately, this equation may have several solutions in case of occlusions, because the function Π is not injective. In that case, thanks to the known depth of these points, we can just take the closest point. For ease of notation, let's define the function C (for point cloud) as:

C : R 2 → R 3 p 2 → T t→t 2 ( θ t (p 2 )K -1 Π -1 (p 2 )
Let's also define for each point p ∈ Ω the set of possible projected points P (p), and the set of depth values D(p):

P (p) = {p 2 s.t. p = Π (KC(p 2 ))} D(p) = {C(p 2 ).u z , p 2 ∈ P (p)}
where u z = [0, 0, 1] is the forward unit vector. We can then define the direct projection of θ t as:

∀p ∈ Ω, θ t→t 2 (p) = +∞ if D(p) = ∅ min D(p) otherwise u z,t 2 ×P 1 ×P 2 × P 3 point cloud C(p 2 ), p 2 ∈ R 2
Set of possible values projected to frame point p: {P s.t. Π(KP) = p} Assuming we can perform direct warping, the idea is then to warp θ t into θ t→t 2 and then either warp θ t→t 2 back to θ t→t 2 →t or identify pixel that were painted with a 2D map of integer indices Id.

P (p) = {P 1 , P 2 , P 3 } θ t→t 2 (p) = P 1 .u z,t 2 T t→t 2
The index map Id is a way to retrieve what pixel was used in the point cloud constructed from θ t and the transformation T t→t 2 to be projected in each pixel of

θ t→t 2 . ∀p ∈ Ω, Id(p) = NaN if P (p) = ∅ arg min p 2 ∈P (p) (C(p 2 ).u z ) otherwise
We can then summarize these two methods:

1. Warp back:

θ t T t→t 2 --→ θ t→t 2 T t 2 →t --→ θ t→t 2 →t
If the warp operation is accurate enough, we can safely assume that differences between θ t and θ t→t 2 →t are occlusion areas.

∀p ∈ Ω, occ(p) = False if θ t→t 2 →t (p) = θ t (p) True otherwise
2. Index back:

θ t T t→t 2 --→ θ t→t 2 Id
We can identify which point was painted and which one was not from the index map Id that we created along

θ t 2 . ∀p ∈ Ω, occ(u, v) = False if p ∈ Id(Ω) True otherwise (4.26)
This pipeline is very simple as it always assumes the squares to be perfectly normal to the camera axis. As a consequence, this algorithm is not perfectly robust to rotations. In the case of rotations in our scene large enough for side effects not to be negligible, we could easily extend this algorithm to include square inclinations, but because our scenes usually have very low displacements relative to each other, we choose to keep it simple. The algorithm is not developed in this report, but we extended it to also warp the corresponding image pixel for visualization purpose, using the index map Id. That way we can see which pixel is mapped where. As mentioned above, unlike inverse warping, this operation is not spatially differentiable. Some works have been done to to make rasterization and even ray tracing differentiable [Li+18; LB14], but we don't need that functionality here, as we just want to identify the occlusion areas.

Occlusion detection module

Once we got direct warping developed, we know the occlusion and valid areas for θ t→t 2 with respect to T t 2 →t . To get the equivalent in θ t we can first look at the method based on index map (see equation 4.26).

This method we call "index back" will have a sparsity problem: overlapped pixels will not be indexed and thus considered occluded while a continuous rasterization would have made them appear. As a consequence, the filtered gradient map from photometric error is very sparse.

However, our main goal is to get rid of false negative for our occlusion filter: pixel that are falsely considered non-occluded should be avoided, at the cost of potential false positive if they are evenly distributed.

Because of its simplicity, our pipeline might not be robust to inclined plane. Indeed, unwanted disocclusion can appear, some background parts normally occluded will be visible, indexed and optimized while they should not be. As mentioned above, this problem is partly solved by the assumption of little displacement, but can appear when considering very thin local maxima.

The indexed map could probably be improved by filtering with e.g. morphological filters or median filters, but we can also chose to do the same direct warp operation with θ t→t+1 to avoid foreground sparsity. This is particularly interesting when considering false negative regarding occluded background. With a foreground continuous enough, false negative will be sparse, while true positive won't be. It means we can filter out these values with a simple dilation operation applied on the occlusion mask. The valid map is simply the coordinates where θ t→t 2 →t is within a range of θ t .

α > 1, ∀(u, v) ∈ Ω, occ(u, v) = False if θ t→t 2 →t ∈ [ 1 α θ t , α θ t ] True otherwise Figure 4
.10 shows a comparison with "index back" and "warp back + erosion of valid pixels by 2 pixels". This example features a depth map with a foreground at his center. The considered displacement is The depth map has a noise of σ = 0.1 and false negative appear beneath the foreground during warp (first frame).

Conclusions on occlusion considerations

This section presented the fact that in addition to not being injective which is partly solved with smooth regularization, the inverse warp function is not perfectly representative with respect to depth and movement. True minimum of the resulting photometric loss function is not indicating a true depth as soon as the scene features occlusions. Filtering invalid

√ 2 2 [-1, 1, 1]
: the camera goes down left and forward, field of view is 90 • . First row: presentation of θ t and theoretical results on this perfectly smooth depth map. Notice the foreground is also occluded because it goes out of field of view. Second row: presentation of on ν, a noisy equivalent of θ t , with a Gaussian noise with std of 0.1 ν t = θ t + N (0, 0.1). Result of direct warping (center) and filtered depth map with index back method. Last row: warp back of warped depth ν t→t+1 , and filtered depth map using the warp back method, followed by an erosion of 2 pixels. inverse warp areas is an attempt at making the photometric loss indeed minimum at the perfect value.

We now have along photometric error two studied tools to train a network to retrieve the depth of a picture within a sequence of frames. The next step in elaborating an unsupervised training workflow is to determine optimal values.

Determination of optimal hyper-parameters with toy problems

We now want to make a comprehensive study of our potential options for an unsupervised training before deploying it on a big drone dataset with our DepthNet network.

In order to study the different works on unsupervised depth learning, we decided to evaluate it on a very simple version of the problem. The key here is to evaluate the training quality and how it compares with the ideal supervised case, and not the network quality regarding over-fitting. As such, we reduce the problem to over-fit a small number of different depth maps: depth values are the very trainable parameters we apply the gradient descent on, which is very similar to DTAM [START_REF] Richard A Newcombe | DTAM: Dense tracking and mapping in real-time[END_REF]. In this toy problem, we make abstraction of the neural network: we simply want a training algorithm that can converge to a reasonable and stable depth map in the hope that it will be a good candidate for the full workflow with a neural network that tries to infer depth from images.

Indeed, one of the main drawbacks of this photometric based training is the lack of stability. It has been shown thanks to a test set with ground truth that photometric loss was not a good quality measure, and that depth quality did worsen when the training was too long [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF]. This indicates that training on a totally ground truth-less algorithm with only photometric loss is impossible since it would require a validation set for early stopping (as presented in [START_REF] Prechelt | Early Stopping -But When?[END_REF] among other methods to prevent overfitting) in order for the network to stop training further and compromising its quality.

It must be noted that this is different from over-fitting: the quality also worsens for training frames. It will appear that even in our toy problems that is designed to be overfitted, a lack of good regularization can make our depth maps diverge, while still lowering the photometric loss.

Scenes presentation

From physical assumptions made section 4.4.2, we want to design scenes that will test the robustness of an algorithm to them.

Namely, we identified problems that an auto-supervised training algorithm should be able to solve.

• A scene with two clear grounds, where the foreground has some texture discontinuity while being perfectly smooth.

• A scene with depth discontinuity that triggers occluded area from one view point to another.

• a scene with an infinite plane, not normal to camera plane

• a scene with a texture-less background, on which the regularization should not be applied.

To feature these problems, we focus on two very simple scenes, shown Figure 4.11:

1. a scene composed of one static foreground object and a background. This scene will be further referred to as "Flower", since the foreground is textured with a daisy.

2. a scene of an infinite horizontal plane. This scene will be referred to as "Horizon".

The scenes have three frames and a known displacement between the frames. The goal of this experiment is to find the depth map of the center frame only using reprojection optimization. Thanks to the uniformity of displacement, for the flower scene, the occluded areas shown figure 4.12 are not the same, which makes it theoretically possible for every pixel to be retrieved in at least one of the 2 frames.

A workflow not able to converge in our examples might not be robust to train on a larger dataset such as KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF], Still Box or our unlabeled drone sequence without the necessity of a validation dataset with depth ground truth. This statement is not necessarily true as stochasticity and sufficiently heterogeneous examples during training can lead to implicit regularization. As a consequence, the optimal learning schedule can be different from one dataset to another.

A first result on optimization can be seen figure 4.13 where only the photometric loss is optimized. This failed convergence serves as a baseline for our next regularization losses.

Metrics

To evaluate optimization tricks on our toy problems, we have to design a set of meaningful metrics for a potential extension to a wider problem. More specifically, we must optimize the error for both our scenes.

As a consequence, we must look at a joint metric that heavily penalizes performance discrepancy. A good way to do so is to find a normalized metric that is 1 when the estimated depth map θ is perfect, 0 if it's worse than our starting point (with everything set to background depth). That way, by the geometric mean of the scene metrics, we get a performance index that gives priority to consistency.

If we look at a standard performance evaluation such as logarithmic error LE (see chapter 3), for a particular scene S with associated depth target θ S and initial depth value θ S 0 , the normalized metric is then

LE(θ, θ) = log(θ) -log( θ) M S ( θ S ) = 0 if LE(θ S , θ S ) > LE 0 1 -LE(θ S , θ S )
LE 0 otherwise with LE 0 the initial error: LE 0 = LE( θ S 0 , θ S ). This value depends on the starting depth map.

The general quality metric is then for a set of

k scenes S = {S 0 , S 1 , • • • , S k } M = |S| ∏ S∈S M S
In our case k is only 2, but should we find a new suited scene representative of a characteristic problem not found in the first two, the formula would still hold.

From now on, unless specified otherwise, every quality measure will be based on M 4.12: First row: Illustration of optical flow construction, and how a perfect depth produces imperfect warping. A "ghost" of the foreground can be seen on occluded areas, which the reprojection based optimization will wrongly try to solve. Second row, our two imperfectly warped frames from I t-1 and I t+1 to I t . Last row, the differences between warped images and I t are the occlusion areas. We can see that the non-zero areas don't cover the same pixels. Figure 4.13: optimization results when only photometric loss is applied for depth optimization. Mean logarithmic error is worse at the end than at the beginning for both cases. For each scene, from left to right and top to bottom: depth result, difference with ground truth, final inverse warp of I t-1 and final inverse warp of I t+1

Tests on smooth loss regularization

From section 4.4.3, we know that the value we want to apply diffusion on is not our trainable parameters θ, but the inverse depth map ξ = 1 θ . Smooth loss functions presented section 4.4.10 will be tested. For each smooth loss (TV, TVV, diffusion and gradient diffusion), we will try to find the best loss hyper parameters for the measure M, and we will be able to compare them.

Robust smooth losses

As we advised section 4.4.5, we will also try "robust" versions of the considered loss function. In our tests (sec 4.6), they will be denoted by the prefix "robust".

The robust smoothing will consist in a simple gradient descent applied on a clone of ξ. It needs two parameters to compute the diffusion of our output, the optimization step, just as a regular optimization, and the number of iterations. In the following tests, to avoid unnecessary overhead of loss computation, we chose a step value of γ = 0.1 and 10 iterations. For distance function, mean square error will be used.

Besides, the training parameters are depth values, while the smooth loss functions are applied on inverse depth ξ. This makes a trivial use-case where applying a "robust" version of a smooth losses, might be useful. The derivative of the inverse function is highly non linear (see appendix D.3). As such, all our tests with smooth losses will be done using a regular potentially unstable version, and their robust counterpart.

Hyper-parameter search

The main goal of this experiment is to find a loss that is robust to our two supposedly complementary scenes. The two main hyper-parameters that can be searched are the influence of textureness over smooth loss dampening κ and the relative weight λ with which the smooth loss is applied, compared to the photometric loss.

Namely, given a smooth loss L s (ξ, κ), we apply the loss 4.17, present optimization results on our toy problems after selecting λ and κ maximizing measure M for each loss.

L(θ) = L p + λL s (ξ, κ)
Surprisingly, the diffusion methods (robust or regular) tend to the best results, even for our infinite plane in Horizon.

This can be mitigated by three observations:

• As shown figure 4.15, performance on Horizon Scene is very erratic for diffusion loss, for an unknown similar scene, there is a chance that the chosen hyperparameters (λ, κ) will perform very poorly. This is mostly due to not respecting the uniform background requirement, because the sky gets diffused with the foreground depth.

• It can be observed figure 4.17 that for Horizon scene, diffusion losses actually worsen over time after reaching a very good depth map, while gradient diffusion losses seem to keep going down until the end of the optimization. This shows that gradient diffusion losses may be more robust to extended training workflows which may happen when no ground-truth is available at validation time.

• When looking at parameter change robustness figures 4.14 and 4.15, gradient diffusion optimum appears to be more evenly distributed, without a strong peak at optimum. As a consequence, considering a different set of scenes would probably mean a set of optimal hyper-parameters somewhat different, gradient diffusion will probably outperform regular diffusion

Using Occlusion Module

Decent results using only smooth loss regularization have to be mitigated with the fact that foreground has a totally different color distribution than background. Which then makes the assumption of high color gradient for depth discontinuities plausible. But as seen figure 4.16, this has to be done with a compromise on textured planes (with no depth discontinuity), and the foreground is not perfectly determined. Keeping a κ parameter high enough to get good results on foreground worsens the background and especially the occlusion zones.

To check the relevance of our proof of concept occlusion mapper, we test this regularization with our toy problem, using our different losses, after a new hyper-parameter search. A quantitative comparison can be seen Table 4.1.

In every smooth loss, the quality index M is higher, even making TVV Loss eventually a good candidate. Both optimal κ and λ values are generally lower. This is expected for λ, as now several image areas are filtered from the photometric loss. The smooth loss then has no concurrency on them and thus don't need to be high. However, this is unexpected for κ, as we would have though that low κ values would be problematic for textured planes such as the Flower scene. It can be noted that the robustness index ∆M is worsened for diffusion and robust diffusion, while it was already their weak point, making gradient diffusion and robust gradient diffusion an even more interesting candidate regarding training on unknown datasets.

A final significant improvement of this occlusion regularization is the "ghosting" that is not longer appearing on our Flower scene. Figure 4.18 shows a comparison for the resulting depth map with regular diffusion.

This module then seems to help convergence without largely penalizing scenes without occlusions.

A more sophisticated warping algorithm, seems to be an interesting improvement axis to efficiently regularize occlusions.

Photometric losses

This section will discuss several variations on photometric loss. For our toy examples, we only used the simple photometric loss, which is obtained with the absolute difference between target images and reference frames, but some techniques have been studied to cope with large displacement, or non constant luminosity.

Multi-scale photometric loss

Multi scale (or pyramidal) loss consists in trying to minimize the photometric reprojection loss with multiple resolutions. This assumption is generally done to allow the photometric loss to have a greater receptive field, this is the main idea behind optical flow method and its variations such as Anandan [START_REF] Anandan | A computational framework and an algorithm for the measurement of visual motion[END_REF] or Farnebäck [START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF]. Indeed, when dealing with large optical flow, due to a high displacement magnitude or a small depth, gradient from photometric loss is not meaningful. Depth regularization can help, as it has been shown with our tests, but it requires an overlap of areas with the same modality. In the example of our occluded scene, if the foregrounds did not overlap, it would have been impossible to make the depth converge. As a consequence, the maximum optical flow over which foreground objects don't overlap and thus make depth impossible to converge directly depends on the size of foreground objects.

As a consequence, if we have a small foreground zone, without multi-scale loss, we might be unable to learn from it. However, for multi-scale loss to be usable, we would ODM TV Loss need to downscale the picture enough for the foreground zones to be less than one pixel across. This is required for gradients to be useful. In practice, this could require to downscale the picture a lot, with the loss of texture and precision that comes with it Another important element is that as shown in DepthNet and almost every variation of FlowNet [START_REF] Dosovitskiy | FlowNet: Learning Optical Flow with Convolutional Networks[END_REF], multi-scale training of a wanted map (be it optical flow, stereo disparity or depth) is very beneficial to the network, even in a supervised manner. It has been shown by Godart [START_REF] Godard | Digging Into Self-Supervised Monocular Depth Estimation[END_REF] that multi-scale learning for auto-supervision was not beneficial for the photometric loss, but rather for the network. In this work, he outputted downscaled depth maps, but did not use it to make a downscaled photometric error. Instead, he upscaled it to get a full scale photometric error, and could still easily make the network converge, better than with the highest scale depth output alone.

κ 1 0.1 λ 1 1 M 0.
Besides, multi-scale photometric error can trigger specific error, because the downscaling operation is similar to a low pass filter. As a consequence, high frequency pattern will lose their textureness on smaller pyramidal levels. This is one of the greatest difficulties of in-car environment datasets such as KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] or Cityscapes [START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF], where the road which is normally very textured appears homogeneous at low resolution. The basic idea behind this problem is that multi-scale photometric error is not robust where pyramidal refinement is not.

It thus appears that multi-scale photometric error is useful only at the beginning of the training, the same way pyramidal refinement is only useful when optical flow error is above one pixel.

The recommended training schedule can introduce multi-scale photometric error, but should be dismiss it as soon as the network can converge without it. When considering small enough displacements, it might never be useful, this is the case for KITTI where convergence is already possible at the beginning. Multi-scale photometric loss might also be dispensable when considering a pretrained network if the network performance is already decent enough.

Minimal loss sampling

Minimal loss sampling has been introduced by Godard et al. [START_REF] Godard | Digging Into Self-Supervised Monocular Depth Estimation[END_REF] as a way to automatically eliminate ambiguity. The assumption is the same as the one that made smooth losses work well in the first place: each ground (e.g. background or foreground) in the image has a specific color distribution. As a corollary, color differences of pixels from the same ground will be significantly lower than for colors of pixels from different planes.

When considering several inverse warps at the same time, we get multiple photometric loss maps. Minimizing the pixel wise minimal difference is deemed to be equivalent to applying optimization only on colors from the same ground.

L p = ∑ p∈Ω min k d(I t -I t k →t )(p)
The consequence of this is that suppressing "ghosting" is possible for occluded areas, provided there exists another frame where that same area is not occluded. This is for example the case in the Flower scene of our toy problem.

The main advantage of this technique is its simplicity, with no need to compute any custom module, and no computational overhead.

When testing with our toy problem, we can see figure 4.19 that indeed, no false positive can be seen on the background. However, foreground depth does not succeed in converging to right values, even with a very high smooth loss. The reason why the convergence is so poor in our case while it has shown great results and adhesion elsewhere [START_REF] Casser | Unsupervised Learning of Depth and Ego-Motion: A Structured Approach[END_REF] needs further investigations.

It might be due to the lack of stochasticity. Indeed, it appears that convergence is stopped very quickly, being stuck in a local minimum very specific to our example. Concretely, when adding noise to our examples at each optimization step, minimal sampling was improved.

Photometric distance

Since the beginning we worked with the simplest form of photometric distance, which is the absolute difference between color values. However, this assumes that images rightly projected have the exact same color. This is problematic as in reality some extrinsic scenes parameters can change. For example, light conditions can change and reflectance for non Lambertian surfaces makes the texture appear totally differently. Even camera parameters can change gradually such as white balance and auto-exposition. Some effects such as light changes can be assumed to change slowly and be neglected, some other effects such as auto-exposition when the camera goes out of a tunnel, or reflection patterns cannot be neglected. See an example figure 4.20. The following loss tries to address that problem by computing variations of correlation rather than absolute difference. This is particularly useful for changes of exposition.

Structural Similarity (SSIM)

As suggested subsequently by Zhao and Godart [Zha+15; GMB17], raw pixel difference can be coupled with structural similarity (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] maximization, in order to have a more natural quality measure. This has later been confirmed to yield good results

SSIM(X,

Y) = l(X, Y) α c(X, Y) β s(X, Y) γ (4.27) l(X, Y) = 2µ X µ Y + C 1 µ 2 X + µ 2 Y + C 1 (4.28) c(X, Y) = 2σ X σ Y + C 2 σ 2 X + σ 2 Y + C 2 (4.29) s(X, Y) = σ XY + C 3 σ X σ Y + C 3 (4.30) µ X = G * X (4.31) σ 2 X = G * X 2 -µ 2 X (4.32) σ XY = G * XY -µ X µ Y = G * (X -µ X )(Y -µ Y ) (4.33)
The SSIM function is a quality measure, which maximum value is 1, and minimum is -1. l and c have values within [0, 1] and s has values within [-1, 1]. In order to avoid too much computation, C 3 is usually set so that C 3 = C 2 2 which simplifies SSIM equation into Eq. 4.34.

SSIM(X

, Y) = (2µ X µ Y + C 1 )(2σ XY + C 2 ) (µ 2 X + µ 2 Y + C 1 )(σ 2 X + σ 2 Y + C 2 ) (4.34)
Nevertheless, general equation 4.27 can help grasp what values are actually measured and thus what is optimized when trying to maximize this measure by gradient ascent. An important point to note is that these functions are rather independent of one another. Namely, the function l is only a function of means µ X and µ Y , while being invariant to variance and correlation, the function c is a function of variance and s (roughly) a function of correlation when

C 2 is negligible compared to σ X σ Y . SSIM(X, Y) = l(µ Y , µ X )c(σ X , σ Y )s( X, Y) X = X -µ X σ X If we try to differentiate SSIM relative to X, we get ∇SSIM = cs∇l + ls∇c + lc∇s
The way to understand this is when trying to maximize SSIM, gradients of a function are dampened by the others. For example, when mean values don't match, l will be very low, and thus functions c and s won't be maximized. It can be problematic if at the beginning of our optimization the images patches are too dissimilar, for example when the optical flow is too high. As a consequence, it is a good idea to mix it with the more traditional L 1 distance, at least at the beginning. However, this loss adds some robustness: for a problematic sample with changing exposition (e.g, when exiting a tunnel), the difference of luminance and contrast lowers the SSIM gradient, and thus optimization is less effective on this sample. This loss is robust to changes of luminance in the sense that it filters them out of the optimization. In most works already mentioned [START_REF] Godard | Unsupervised Monocular Depth Estimation with Left-Right Consistency[END_REF] [WB18] [YS18], [START_REF] Mahjourian | Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints[END_REF], [START_REF] Casser | Unsupervised Learning of Depth and Ego-Motion: A Structured Approach[END_REF], the photometric is then a weighted sum of the L1 loss and DSSIM (the opposite version of SSIM, which needs to be lowered)

L p ( I, I) = (1 -α) ∑ i I -I 1 |Ω| + α 1 -SSIM( I, I) 2
It must also be noted that correlation is in fact the exact opposite of mean square error of normalized vectors (up to a constant value), as quickly shown Eq. 4.35. Likewise, the functions l and c can be interpreted as mean square errors, dampened by the denominator (because its value does not change much). This makes sense when considering the initial justification for SSIM, inspired by the human visual system, which supposedly has a relative scale. However, for us, this may not be the best assumption, because the higher the values X and Y we will have to match, the lower the gradient will be, inducing a vanishing gradient problem.

s( X, Y) = σ XY σ X σ Y = G * X Y = 1 2 G * ( X 2 + Y 2 -( X -Y) 2 ) = 1 - 1 2 G( X -Y) 2 (4.35)
Additionally, figure 4.21 shows the profile of a curve x → 2xy+C x 2 +y 2 +C . It appears obvious that optimization is impossible if x and y don't have the same sign. It must not be forgotten especially for the function l since in most deep learning training workflow, images are normalized before being fed to the network. Our advice if one wants to use SSIM is then to normalize images for the network, but not for the photometric loss.

For all of these reasons, SSIM seems suited to reach a better optimum, and benefit from being a widespread image quality metric, but can probably be improved in terms of gradient based optimization. In fact, in all the works that use it for image reconstruction problems, we have not seen other justification than the fact that it yields better results. x 2 +y 2 +C used for l and c in SSIM (here, C is 0.01 2 ). This function is only convex around its maximum, which make its gradient based optimization difficult

Conclusions on literature review for unsupervised depth training

In our toy problem, we have been able to identify and inspect different optimization methods for image reconstruction. Unfortunately, it turns out most of them are only motivated by heuristics, and thus can be improved by physical and theoretical considerations. It appears that the end goals are well defined: a smooth inverse depth map, with possible discontinuities. However, the loss gradients are not heaviliy engineered, as it can be seen e.g. for TVV Loss and SSIM distance. This tells us that this problematic is still in its "proof of concept" phase. The same way a simple network like VGG [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] was improved into Inception [START_REF] Szegedy | Going deeper with convolutions[END_REF], ResNet [START_REF] He | Deep residual learning for image recognition[END_REF], and finally DenseNet [START_REF] Huang | Densely connected convolutional networks[END_REF], we hope photometric distances and smooth losses will be refined to more robust and stable equivalent that could potentially lead to a true unsupervised learning, without having an annotated validation set for early stopping. This chapter presents our proposition for an unsupervised training of DepthNet, with robustness in mind. From the study made chapter 4 on unsupervised depth learning, we propose to discuss and adapt the work SFMLearner from Zhou et al. [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] and its variations in order to perform fine-tuning on real videos for our DepthNet network without ground-truth.

This chapter has been the subject to the following publication [START_REF] Pinard | Learning structure-from-motion from motion[END_REF]: Clément Pinard, Laure Chevalley, Antoine Manzanera, and David Filliat. "Learning structure-from-motion from motion". In: ECCV GMDL Workshop. Munich, Germany, Sept. 2018. URL: https://hal.archives-ouvertes.fr/hal-01995833.

Shortcomings of SFMLearner

As a reminder, the SFMLearner, presented section 4.3.1 trains a neural network (Disp-Net) to output inverse depth of an image, along with another network (PoseNet) to output frame poses of a particular sequence.

Whereas PoseNet takes all the images as input, DispNet only takes one picture I t . As such, this network is doing inverse depth from context and appearance. It should also be noted that KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] was the main dataset for this particular unsupervised learning task. Unfortunately, this dataset, which is only representative of car movements, 5.2. Workflow presentation is not really a true 6 degrees of freedom dataset, and lacks appearance diversity. As a consequence, depth from context is a very good strategy here. Indeed, it is mentioned by Zhou et al. [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] that feeding multiple frames to a network did not yield better results in their training workflow on KITTI dataset.

As discussed chapter 2, in the context of drone videos, the dataset can present camera movements with only 3 degrees of freedom since we don't have rotation, but flying cameras will present a huge variety of contexts and appearances, with different kinds of altitude and orientation, in addition to different ground profiles on which a car could not drive.

Another important point to note is the fact the scaling factor uncertainty is never explicitly solved, while DepthNet solves it by assuming constant displacement magnitude (as explained section 3.5.1), which is then compared against actual displacement magnitude. Here, no assumption is done on PoseNet translation magnitude, and the only supervision comes from the resulting optical flow and its corresponding warping, which is proportionate to both PoseNet displacement magnitude and disparity. As such, even if the two networks converge consistently so that the optical flow is well estimated, the depth and poses alone are only known up to a scale factor compared to ground truth.

Following discussion from section 2.3.1, we will display typical scale invariant quality measures for information purposes only. The actual measure we will use to evaluate and compare our solution asks for both translation estimation and depth estimation, to make the scale factor solvable with translation magnitude measured from dedicated sensors. The two different quality evaluations will be denoted in tables 5.1, 5.2 and 5.3 by the scale factor column. When using standard relative depth measurement, the indication GT (for Ground Truth) is used, when using translation magnitude, the letter P (for Pose) is used.

Workflow presentation

Inspired from [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF], we propose a framework for training a network similar to Depth-Net, but from unlabeled video sequences. The approach is slightly different from Sfm-Learner since our network takes two stabilized images for input instead of only one. The approach can be decomposed into three tasks, as illustrated by Fig. 5.1:

• From a certain sequence of frames (I i ) 0≤i<N , randomly choose one target frame I t and one reference frame I r , forming a pair to feed to DepthNet.

• For each i ∈ 0, N , estimate pose T t→i = (R i , t i ) of each frame I i relative to the target frame I t . Detailed section 5.2.1.

• Compensate rotation of reference frame I r to I STAB • Normalize the translation to constrain it so that the displacement magnitude t r with respect to I r , is always the same throughout the training. This point is very important, in order to guarantee the equivariance between depth and motion, imposed by the original DepthNet training procedure that makes network rely mostly on motion and not on appearance. Detailed section 5.2.3.

• As the problem is now made equivalent to the one used in [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF], perform a photometric reprojection of I t to every other frame I i to I i→t , thanks to depth θ t of I t and poses T t→i computed before. The resulting frame I t is then compared against I t to compute loss from photometric dissimilarity. Detailed section 5.2.4. Our algorithm, although relying on very little calibration, needs to get consistent focal length. This is due to the frame difference being dependent on focal length. However, this problem is easily avoided when training on sequences coming from the same camera. Also, as shown by Eq. 4.2, camera intrinsic matrix K needs to be known to compute warping and subsequent photometric reprojection loss properly. In practice, assuming optical center in the center of the focal plane worked for all our tests; this is corroborated by tests done by [START_REF] Mahjourian | Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints[END_REF] where they used uncalibrated camera, only knowing approximate focal length.

Finally, this algorithm can be simplified for stabilized videos, for which no rotation needs to be estimated, and no stabilization needs to be done before feeding I r to the network.

Pose estimation

PoseNet, as initially introduced by Zhou et al [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] is a classic fully convolutional neural network that outputs 6 degrees of freedom (or 3 in the case of stabilized videos) transformation pose descriptor for each frame. Output poses are initially relative to the last frame, and then compensated to be relative to the pose of the target frame. This way, PoseNet output is not dependent on the index of the target frame. Besides, computing by default with respect to the last frame makes the inference much more straightforward, as in real condition, target frame on which depth is computed should be the last of the sequence, to reduce latency.

Frame stabilization

For the general case with 6 degrees of freedom, in order to cancel rotation between target and reference frame, we can apply a warping using rotation estimation from PoseNet. When considering a transformation without translation, Eq. 4.2 no longer depends on the depth of each pixel, and becomes Eq. 5.1. As such, we can warp the frame to stabilize it using orientation estimation from PoseNet, before computing any depth.

p r t = KR r K -1 p t (5.1)
As mentioned earlier, UAV footages are either stabilized or with a reliable estimated orientation from inertial sensors. This information can be easily leveraged in our training workflow to supervise pose rotation, giving in the end only translation to estimate 5.3. Training datasets to PoseNet. In addition, when running in inference, no pose estimation is needed as DepthNet does not depend on translation estimation, and PoseNet is not used.

Depth computing and pose normalization

For depth to be provided assuming a constant displacement magnitude, the pose of the reference frame must be normalized to correspond to that magnitude.

As such, to get consistent poses throughout the whole sequence, we apply the same normalization ratio, as shown in Fig. 5.1. The resulting magnitude is thus only known to be scaled to d 0 between I t and I r . Every other translation might be of different magnitude.

∀i, t

NORM i = t i d 0 + t r (5.2)
The main drawback of normalizing translations is the lack of guarantee about absolute output values. Since we only consider translations relatively to the reference, translations are estimated up to a scale factor that could be -when they are very large -leading to potential errors for rotation estimation, or -when they are very close to 0 -leading to float overflow problems. To overcome these possible issues, along with classic L 2 regularization to avoid high values, we add a constant value to the denominator. The normalization is then valid only when t r .

Loss functions

Following what was discussed on chapter 4, our training loss will be composed of a photometric loss, and a smooth loss: we chose the gradient diffusion loss with optimal parameters κ and λ defined with the hyper-parameter search on the toy problem presented section 4.6.2. Our photometric loss L p is a mixture of mean L1 distance and DSSIM, α being an empirical weight set to 0.15.

L p = ∑ i (1 -α) I i→t -I t 1 |Ω| + α 1 -SSIM( I i→t , I t ) 2 (5.3) L s = λL gdi f f (ξ, I t , κ) = λ |Ω| ∑ p∈Ω exp - ∇I t 2 κ 2 × (∆ξ) 2 (5.4)
For L s , I t is downscaled to fit ξ resolution. Finally, we apply this loss to multiple scales s of DepthNet outputs, multiplied by a factor giving more importance to high resolution. In accordance with what was discussed section 4.7.1, the loss L s p is obtained with full scale target image and warped image from upscaled output of DepthNet.

Our final loss becomes

L = ∑ s 1 2 s L s p + L s s
(5.5)

Training datasets

Our experiments were made on three different datasets. KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] is one of the most well known datasets for training and evaluating algorithms on multiple computer vision tasks such as odometry, optical flow or disparity. It features stereo vision, LiDAR depth measures and GPS / RTK coupled with IMU for camera poses. During training we only used monocular frames and IMU values when supervising with orientation. We used LiDAR for evaluation. We applied the same training/validation/test split as [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF]: about 40k frames for training and 4k for evaluation. We also discarded the whole set scenes containing the 697 test frames from the Eigen [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF] split. We also constructed a filtered test set with the same frames as Eigen, but discarding 69 frames whose GPS position uncertainty was above 1 m. This set was used when displacement data was needed.

We also conducted experiments on StillBox, the dataset used in supervised DepthNet section 3.4.1, in which we added random rotations (i.e. we draw an initial rotation speed that remains the same through the sequence). As this dataset was initially developed to have depth independent on context, we can already predict that performance of single frame depth network like SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] will not be good. The dataset contains 1500 training scenes and 100 test scenes, ie 30k training frames and 2k validation frames.

Finally, we trained our network on a very small dataset of UAV videos, taken from the same camera the same day. We used a Bebop2 drone, with 30 frame per seconds videos, and flew over a small area of about one hectare for 15 minutes. Some of the flights have low displacement sequences and even hovering states where the drone is completely motionless. The training set contains around 14k frames while the test set is a sequence of 400 frames. This dataset is not annotated, and only subjective evaluation can be done.

Experiments

Qualitative results

Implementation details

As for supervised training chapter 4, we used PyTorch [START_REF] Paszke | Automatic differentiation in PyTorch[END_REF] for all tests and trainings. We used Adam optimizer [START_REF] Diederik | Adam: A Method for Stochastic Optimization[END_REF] with learning rate of 2 × 10 -4 and β 1 = 0.9, β 2 = 0.999. The full source code is available on Github1

Testing methodology for DepthNet

To stabilize the input of the network, we take a sequence of the same length as PoseNet input (here, 5 frames), with the goal of computing depth of the last frame. When training was done with orientation supervision, we stabilized the frames before feeding them to DepthNet, without the help of PoseNet.

Besides, as DepthNet output is a function of apparent movement, we have several options for the frame pairs we can use to compute depth. Our method relies on taking a pair for which the mean output of DepthNet is the closest of an arbitrary chosen value set to 50m, regardless of the dataset. The output is then rescaled with the ratio of nominal displacement. A more sophisticated method for depth inference with stabilized frames will be presented on chapter 6.

Quantitative results

In addition to using standard measurements from [START_REF] Eigen | Depth map prediction from a single image using a multi-scale deep network[END_REF], our goal is to measure how well a network would perform in real conditions. As stated in Section 2.3.1, depth map scale factor must be determined from reasonable external data and not from explicit depth ground truth. We thus compare our solution to SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] where the output is multiplied by the ratio between estimated displacement from PoseNet and actual values. For KITTI, displacement is determined by GPS RTK, but as we only need magnitude, speed from wheels would have been sufficient.

Input

SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] Ours [START_REF] Fu | Deep Ordinal Regression Network for Monocular Depth Estimation[END_REF] shows the single frame depth network trained with supervision, DORN being the current state of the art. It serves at giving an idea of theoretical limit. The second one with SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] and DDVO from Wang et al. [START_REF] Wang | Learning Depth from Monocular Videos using Direct Methods[END_REF] shows unsupervised methods where moving scenes are not explicitly considered. Last category with Casser et al [START_REF] Casser | Unsupervised Learning of Depth and Ego-Motion: A Structured Approach[END_REF] shows the state of the art on unsupervised training of depth with monocular camera on KITTI, which takes care of moving scenes with the help of an object detection network. These different works show that KITTI is indeed a dataset with non negligible moving scenes, which will be problematic for our solution, and that neural networks are extremely good at inferring depth from a single image on a dataset like KITTI.

We tried 5 different versions of our network. The first one is the exact same as supervised DepthNet, only trained on StillBox. It serves as a baseline purpose, without fine-tuning. The other four configurations are training from scratch or fine-tuning from StillBox, and training with orientation supervision or not. For our 5 versions we used occlusion module detection, our tests determined that this module had no influence on KITTI. This is probably due to the fact that contrary to our Stillbox dataset and typical drone videos, objects are never extremely close to the camera, and thus few occlusions happen and when they do, they induce an error that is neglectible compared to moving objects.

As we might expect, on KITTI our method fails to converge as well as single image methods using classic relative depth quality measurement. However, when scale factor is determined from poses, we match the performance of the adapted method from [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF]. It can also be noted that fine-tuning provides a better starting point for our network, and that when available on a training set, orientation supervision is very advantageous.

When trying to train a depth network with stabilized videos, it is then strongly recommended to do a first supervised training on a synthetic dataset such as StillBox.

Some failed test cases can be seen on Fig. 5.4. The main sources of error are moving objects and poorly textured areas (especially concrete roads), even though we applied depth smooth geometric loss. Our attempt at explaining this failure is the large optical flow value on a low textured area, meaning matching spatial structures is difficult. However, as KITTI acquisition rate is only 10 fps, we believe this problem would be less common on regular cameras, with typical rates of 30 fps or higher.

Table 5.2 presents results on the updated Still Box dataset. SFMLearner [Zho+17] performs surprisingly well given the theoretical lack of visual context in this dataset. However, our method performs better, whether from orientation supervision or completely unsupervised. We also compare it to supervised DepthNet. This can be considered a theoretical limit for training DepthNet on Still Box since it is completely supervised, and our unsupervised training method is very close to it, indicating the good convergence of our model and thus the validity of our training algorithm and loss design.

Note on Hyperparameter Search

As already discussed in chapter 4, the key idea behind our choice of regularization hyper parameters, and especially the smooth loss we chose, is domain-independence. That's why we chose gradient diffusion loss and not diffusion loss in the first place although it did perform better on the toy problem presented at section 4.6.2, because the optimum was more evenly distributed than diffusion.

For reference, in the context of unsupervised training of depth for KITTI, there has been a consensus on TV Loss for the best results. This loss is then particularly suited for the in-car environment, but not necessarily for drone videos.

If we wanted to test the domain independance hypothesis, we would have needed to test unsupervised training with every possible loss, with every possible hyperparameter. This was possible on a simple toy problem as a single training only required one minute, but is unfortunately much harder to do on a true unsupervised training task, where a single training takes around 10 hours. It might be a future work to consider with efficient sampling techniques such as Bayesian optimization [START_REF] Wilson | Maximizing acquisition functions for Bayesian optimization[END_REF].

Domain adaptation results

To test our method on our small UAV dataset, we first did a training on Still Box, then an unsupervised fine tuning. Likewise, when using SFMLearner method [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF], we pretrained on KITTI before fine tuning on our video. Fig. 5.3 compares SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] and our method on some test frames of our small UAV dataset. Our method shows much better domain adaptation when fine tuning in a few-shot learning fashion. Especially for foreground objects, as SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF] blends it with the trees near the horizon, which is very problematic for navigation. A more thorough qualitative comparison can be viewed in video1 .

Table 5.3 and figure 5.5 compare domain robustness without any training: we tried inference on an upside-down KITTI test set, with ground up and sky down, and our method performs much better than SFMLearner [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF], which is completely lost and performs worse than inferring a constant plane. However, contrary to Supervised DepthNet which performs as well on the reverse version of the test set, our method is not as performing as with regular KITTI (not reversed). This shows that our network may have learned to infer depth from both motion and context, which can be considered as a compromise between SFMLearner relying only on context and Supervised DepthNet relying only on motion.

Input

Ground truth SFMLearner [Zho+17] DepthNet Figure 5.5: Comparison between our method and SFMLearner [Zho+17] on a normal example from KITTI [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] and its upside-down variation.

Conclusion on unsupervised DepthNet

We have presented a novel method for unsupervised depth learning, using not only depth from context but also from motion. This method leverages the context of stabilized videos, which is a midway between common use cases of stereo rigs and unconstrained egomotion. As such, our algorithm provides a solution with embedded deployment in mind, especially for UAVs navigation, and requires only video and inertial data to be used. Our method is also much more robust to domain changes, which is an important issue when dealing with deployment in large scale consumer electronics on which it is impossible to predict all possible contexts and situations, and our method outperforms single frame systems on unusual scenes. This chapter aims at studying the possibilities of using our network DepthNet to get real-time depth maps. More precisely, we consider a strategy to feed our network the optimal frame pairs for real-time inference.

Conclusion on unsupervised

This chapter has been subject to the following publication [START_REF] Pinard | Multi range Real-time depth inference from a monocular stabilized footage using a Fully Convolutional Neural Network[END_REF]: Clément Pinard, Laure Chevalley, Antoine Manzanera, and David Filliat. "Multi range Real-time depth inference from a monocular stabilized footage using a Fully Convolutional Neural Network". In: European Conference on Mobile Robotics. ENSTA ParisTech. Paris, France, Sept. 2017. URL: https://hal.archives-ouvertes.fr/hal-01587658

Simplifying the scale factor determination by normalizing DepthNet

We learned depth inference from a moving camera, assuming its velocity is always the same. As already discussed chapter 2, when running during flight, such a system can deduce the real depth map estimation θ from the network output and the drone displacement, knowing that the nominal displacement was d 0 (here 0.3m), and the temporal difference between the frames δt.

θ(t) = DepthNet(I t , I t-δt ) d(t,δt) d 0 d(t, δt) = t t-δt V (τ)

dτ

In the context of a unconstrained displacement magnitude, the actual correct interpretation of the output of DepthNet is a depth relative to the camera displacement and not an absolute one, so its value should be a dimensionless quantity. To reflect the necessity 6.2. Optimal frame shift determination to estimate actual depth using the displacement magnitude d as the only distance related factor. θ(t) = αβ(I t , I t-δt )d(t, δt) (6.1)

Optimal frame shift determination

Depending on the depth distribution of the ground-truth depth map, it may be useful to adjust frame shift δt. For example, when flying high above the ground with low speed, big structure detection and avoidance requires estimating distance values that are outside the typical range of any dedicated depth sensor. The logical strategy would then be to increase apparent movement by increasing the temporal shift between the frame pairs fed to DepthNet as inputs. More generally, one must provide inputs to DepthNet in order to ensure a well distributed depth output within its typical range. This technique assumes that increasing temporal shift indeed increases apparent movement (and vice-versa for lowering temporal shift). This assumptions needs two conditions to be fulfilled between two frames:

1. The acceleration is small enough so that speed can be considered constant: this is the case for most smooth movements.

2. The depth is changing slowly enough to be considered constant: This is not always true, especially when distance traveled between two frames is comparable to measured depth. However, we assume this particular case to be rare.

The typical range is supposed to be a range of estimations where the error is minimal. Figure 6.1 shows the estimation-wise error for DepthNet512 on StillBox with a minimum between 40m and 90m (i.e. for β ∈ [0.4, 0.9]). This should not be confused with depthwise error presented at section 3.10: a neural network giving always the same value θ c would have a depth-wise error of 0 at this particular distance. The inverse representation of estimation-wise error gives us more information on what to expect for θ compared to the estimated θ.

We then need to choose the optimal spatial displacement d opt and corresponding temporal shift δt to minimize error on the next inference, by avoiding too low or too high values of β. We chose the optimal displacement magnitude as:

d opt = d 0 E(β) β mean = E( θ) αβ mean
With E( θ) the mean of depth values and β mean the optimal mean output of β, e.g. 0.5. δ is then computed numerically from displacement estimation to get the frame shift with the closest corresponding displacement magnitude possible. The mean E( θ) is computed by discarding the extremal values, and especially maximal values (for example the depth values of the sky) since they are not relevant for the actual depth values we want to measure. It is worth noting that the median value of β could be used instead of mean. Using mean here actually paves the way for a more general algorithm presented section 6.3

In order to smooth potential noise in speed estimation that would result in a bad displacement magnitude estimation, we smooth out the target displacement over time. More specifically, we perform an exponential moving average (EMA) with an arbitrary factor λ

d opt ← λd opt + (1 -λ) E( θ) αβ mean (6.2)
The main limitation of this optimal frame shift determination is the multi-modality of the depth distribution of a particular scene. In this case, computing mean of DepthNet output might not be representative of any output value. A representative case would be a scene with bimodal depth values at 10m and 100m (see an example figure 6.2): the resulting displacement will be optimal for a depth of 50m, but neither for 10m nor 100m where the objects of the scenes actually are.

Multiple shifts inference

As neural networks are traditionally computed within massively parallel architectures such as GPUs, multiple depth maps can be computed efficiently at the same time in a batch, especially for low resolution. Influence of batch size on inference speed has already been discussed in section 3.6.1 and has been shown to induce a less than linear slowdown on some architectures.

Batch inference of n frame pairs can then be used to compute depth with multiple shifts δt i . These multiple depth maps can then be combined to construct a higher quality depth map, with high precision for both long and short range. We propose a dynamic range algorithm, described figure 6.4 to compute and combine different depth maps. Instead of only one optimal displacement d opt from E(β), we use K-mean clustering algorithm [START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF] on the depth map to find n clusters on which each next input frame pair will be constructed to focus on. The clustering outputs a list of n centroids c i ( θ) and corresponding d opt,i (t) and δt i . n has to be decided before inference and usually ranges from 1 to 4.

Final depth map is then computed from fusing these outputs using a weighted mean for each pixel. Each weight is actually a linear interpolation from 0 to 1 according to distance of β from a target value β mean . See figure 6.3 for a graph of the weight function. That way, fusion will favor values that are closer to this optimal value. An value is added to solve fusion when every depth map is off its wanted range. This fusion process is very similar to exposure fusion [START_REF] Mertens | Exposure fusion: A simple and practical alternative to high dynamic range photography[END_REF], used in photography. Here, exposition is replaced by depth value, and the weight are based on distance to β mean instead of Wellexposedness.

One can see how when n = 1, this technique is similar to the one described equation 6.2.

w i (p) = f (β(I t , I t-δt i , p)) = f (β i (p)) f [0, 1] → [0, 1] x →            if x < β min + x-β min β mean -β min (1 -) if β min ≤ x < β mean β max -x β max -β mean if β mean ≤ x < β max 0 if x ≥ β max (6.3) ∀p ∈ Ω, θ f (p) = α ∑ i d * i w i (p)β i (p) ∑ i w i (p)
Or more simply

θ f = α ∑ i d i β i where d i = d * i w i ∑ j w j
In the rare case of ∑ i w i (p) = 0 ,meaning every β j (p) is above β max , we take the convention of w i (p) = +∞ and exclude this point from the K-mean clustering algorithm.

For our use-case, we set β min = 0.1 , β mean = 0.5, β max = 0.9 and = 10 -3 . i is the index of frame shift, j, k are the spatial indices. Figure 6.5 shows a result of the proposed algorithm for n = 2. Notice how the high shift detects buildings while low shift detects trees.

A more thorough presentation of the qualitative results can be viewed in video1 .

K-means 1 β mean X α (I t ,I t-δt 0 )• • • (I t ,I t-δt n-1 ) c 0 , • • • , c n-1 d opt,0 , • • • , d opt,n-1 δt 0 , • • • , δt n-1 d * 0 , • • • , d * n-1 β 0 , • • • , β n-1
β 0 , • • • , β n-1 to make a weighted mean of β 0 d * 0 , • • • , β n-1 d * n-1

Training a specific depth range with a clamped DepthNet

As discussed at the end of chapter 3, on very far objects (e.g. the sky), any minor optical flow caused by a default in stabilization will result in a massive error in depth. Moreover, our network being very good at recognizing shapes and giving it a continuous depth, this can result in the whole sky being computed as relatively close.

We thus propose an experimentation with a network designed for a simpler problem: during training on StillBox dataset, we clamp depth from 10m to 60m, with a shift of 5 images (instead of 3 for DepthNet). These new parameters allow the network to only focus on mid range objects, dismissing close and far objects. This training workflow is very well suited for multiple shift depth inference. Every image pair will have a dedicated depth to analyze, preventing the fusion to be bothered with redundant data, because of the high initial range of DepthNet.

Figure 6.6 shows results for multiple synthetic 256x256 scenes with ground truth, along with inference speed and a small noise added to camera initial orientation at each frame. R(t) = R 0 + Euler(N 0 µ(t)), with µ(t) being a 3-dimensional random unit vector and N 0 a constant fixed to 0.001. We also report performance of a thin version of our clamped network, that shows better results than DepthNet with 1 plane only in this noisy setup. The thin network has the same depth, but every layer has half as many feature maps as the original DepthNet.

It is worth noting that DepthNet Clamped and Tiny DepthNet Clamped are just experimentations and no unsupervised fine-tuning workflow has been designed to be robust on clamped depth values with multiple fused inferences.

Our proof of concept

In order to test our Network, we developed a small use-case using a Bebop drone from Parrot. We were able to have a relatively good obstacle avoidance using DepthNet as a depth provider for an depth-powered obstacle avoidance tool in the UAV.

Model Predictive Control

Model Predictive Control (MPC) [START_REF] Richalet | Model predictive heuristic control[END_REF] is a control strategy that can be used for obstacle avoidance with a depth map [START_REF] Brett | Aggressive 3-d collision avoidance for high-speed navigation[END_REF]. This control algorithm has been internally developed at Parrot as a tool for experimentation, and has been used off-the-shelf without any attempt at improving it for the depth-map delivered by DepthNet. This tool expects a depth map from any available source, be it from our network or a disparity map from a stereo camera, or a point cloud from a Lidar. This tool will correct a desired trajectory by either stopping or drifting around the obstacle. We are particularly interested by the second use-case, as our depth method needs motion.

Implementation details

The whole Obstacle avoidance proof-of-concept relies on Video streaming via Wifi into a GPU powered computer (Nvidia Quadro M1000M). Along with the video, additional information about the flight are embedded in the video stream, and extracted with Parrot internal demuxer, called PDraw2 . That way, We can use speed and timestamps of the received frames to apply the algorithm presented figure 6.4. The depth is then discretized This proof of concept has been used in cluttered area used by Parrot for drones prototyping, presented figure 6.8. It has been tested on two obstacles (here the plant and the desk chair), but also on the white wall which was not a problem for our depth sensing protocol and the drone could drift along the wall for multiple flights. For displacement estimation, we used the embedded odometry algorithm, using Bebop's vertical camera. A video is available for further results in our proof of concept.3 .

Conclusions for this chapter

We proposed in this chapter a way of using depth from our network to get a first working solution. We were also able to apply a multiple shift architecture to be robust to scenes with large depth heterogeneity, especially with long range obstacles.

Conclusions for this chapter

This proof of concept is also very promising because every operation used is actually differentiable, even the MPC algorithm, thanks to a recent work by Amos et al. [START_REF] Amos | Differentiable MPC for End-to-end Planning and Control[END_REF]. That way it becomes possible to imagine a training workflow where DepthNet output would not be trained to get a good depth quality measure anymore, but rather so that the resulting MPC based on it will get a good trajectory.

As discussed section 1.3.3, ensuring a good trajectory will still require expensive human supervision with the possible usage of DaGGer [START_REF] Ross | A reduction of imitation learning and structured prediction to no-regret online learning[END_REF]. Nonetheless, this chapter provides a good non-random starting point for training. We showed in this work a working strategy to train a neural network to sense depth based on motion from a stabilized camera, for any kind of domain. We even showed that the next step in the general obstacle avoidance strategy, which was to train a neural network to avoid obstacles from a perfect depth map provided by a simulator before concatenating it with a real depth sensing solution might be replaced by a differentiable MPC [START_REF] Amos | Differentiable MPC for End-to-end Planning and Control[END_REF]. The strategy presented in the introduction (section 1.4.3) and in figure 1.8 is still valid and might be the subject for future work.

Chapter 7

Perspectives and Conclusions

However, thanks to a thorough study on possible drawbacks of depth from vision algorithms, we could identify several drawbacks of DepthNet that might need to be solved before being used as a reliable source of information for obstacle avoidance.

Limitations and future perspectives

UAV depth validation dataset

As heavily discussed on chapter 5, the KITTI dataset [START_REF] Geiger | Vision meets robotics: The KITTI dataset[END_REF] is not well suited to demonstrate the advantages of our solution. On the other hand, due to the lack of other validation sets for depth sensing in the context of outdoor flight, we could only have a subjective validation for the UAV use-case.

It thus seems necessary to construct a validation set with reliable ground-truth depth and navigation data the same way KITTI was constructed. Several solutions can be considered:

• Construct a UAV with a calibrated Lidar, similarly to KITTI. It is highly probable that such a device would not be easy to construct and would be particularly heavy and dangerous. However, it would be robust to moving objects.

• Use a heavy structure-from-motion algorithm to get high quality depth map from 3D reconstruction. Algorithms such as Colmap [START_REF] Lutz Schönberger | Pixelwise View Selection for Unstructured Multi-View Stereo[END_REF] can be used, or even proprietary solutions like pix4D1 . The latter has the advantage of automatizing the flight for an optimized mapping. We can then imagine a two step recording, the first one being dedicated to mapping and the second one to realistic flight conditions in the same scene, getting the ground truth depth map from its 3D reconstruction. The obvious drawback of this solution is the necessity of a rigid scene.

• Using structure from motion with a stereo camera could be an interesting mid-point.

Moving objects can then be detected within a particular range.

Robust smooth and photometric loss

As heavily discussed in chapter 4, today's state of the art regarding self supervision for depth sensing mostly relies on heuristics, themselves based on validation results on the KITTI dataset. A clear understanding of theoretical aspect of self supervision is needed to perform a truly robust training workflow. A recent interesting work [START_REF] Barron | A More General Robust Loss Function[END_REF] proposed to replace the L1 photometric loss with a more general equation, where a differentiable parameter could modify the photometric loss profile, and make it more or less sensitive to outliers, depending on the sample. In this work, among other tests, Barron showed that by simply replacing L1 photometric loss with its general form without changing anything else from original SFMLearner code [START_REF] Zhou | Unsupervised Learning of Depth and Ego-Motion from Video[END_REF], he could improve the results initially obtained. Also, as already discussed section 4.4.5, smoothing could benefit from already developed solutions for large displacement optical flow. A workflow using the same two step joint optimization as DTAM [START_REF] Richard A Newcombe | DTAM: Dense tracking and mapping in real-time[END_REF] for smoothing seems very promising.

Finally, as hinted by the SSIM, a photometric loss that would also take neighboring pixels into account could be beneficial in order to get more information than only color to decide whether two points match or not. This was already studied by Brox and Malik [START_REF] Brox | Large displacement optical flow: descriptor matching in variational motion estimation[END_REF]. In this work, they considered several features and descriptors to be matched and deduced that including descriptor matching inside the energy minimization process was beneficial, especially for large displacement. The problem at the time was the lack of differentiability and sub pixel accuracy of descriptors. This problem could be easily adapted for photometric loss with descriptors composed of convolution kernels, resulting in a feature matching loss.

More generally, self supervision should rely more on techniques developed for optical flow using the variational approach, because occlusion problems along with efficient smoothing and illumination robustness are definitely not new.

Moving scenes

The most important limitation of our work is of course the problem of moving scenes which is explicitly not taken into account in the structure-from-motion paradigm.

As shown by Ranjan et al. [START_REF] Ranjan | Adversarial Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation[END_REF], it is possible to estimate moving objects in a scene, based on the fact that optical flow does not match a specific pattern from estimated camera movement. That way, depth completion techniques can be used to deduce their depth from context. Unfortunately, in addition to having to run another potentially heavy network, that takes us back to the robustness problems we raised in section2.2.1 for depth from a single frame.

A possible solution could also be to use a stereo rig for ambiguous cases. This solution has obviously a limited range, but might be enough for safety regarding humans: if the drone is moving too fast for emergency braking to be possible (typically at more than 10m.s -1 ), the human will be moving slowly compared to the camera, making the resulting structure-from-motion less erroneous.

If we want a truly moving scene robust depth inference solution, we might have to look at time-sequence based models such as recurrent neural networks, that explicitly exploit the time continuity of information: ambiguous perspectives and scales can be solved with anterior depth inferences [START_REF] Mancini | Toward domain independence for learning-based monocular depth estimation[END_REF].

Finally, it should be noted that depth from moving scene problem might only be important when the end goal is a quality depth map. It might be interesting to consider the possibility of a neural network dedicated to obstacle avoidance pretrained using depthfrom-motion and thus not being robust to moving scenes at first, but then being trained to avoid moving obstacle as well.

Bayesian depth inference

For obstacle avoidance, it is interesting to know how confident the network is about a particular inference. As heavily stressed by e.g. Kendall et al. [KG17;[START_REF] Mcallister | Concrete problems for autonomous vehicle safety: advantages of Bayesian deep learning[END_REF], Bayesian inference, not only giving an estimate of a particular value (here the depth) but also the uncertainty, and more generally the distribution of possible values can be very beneficial on decision making. That way, particular system can take extra precautions when in an ambiguous situation. Our main motivation being robustness, it's easy to see how beneficial it could be to safety when operating with moving objects or in an unknown domain.

Alternative Perspectives

This work has been very directed toward the end goal of obstacle avoidance, which might lead to believe that depth sensing is only interesting for pretraining purpose in a fully neural avoidance system.

However, some other applications derived from this work can be used in other contexts.

Depth sensing for multi-task learning

As discussed in section 1.3.4 and shown by Kendall et al. [START_REF] Kendall | Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[END_REF], multi task is not only a way to combine several tasks efficiently in term of computational cost, but also in term of quality when tasks are correlated. It is particularly interesting in the context of object proposal tasks. Indeed, products like Parrot or DJI usually propose a followme/ActiveTrack functionality that proposes targets to track in the streamed footage on the user's phone. There is then already a dedicated training for object proposal networks, grabbing a pretrained general purpose object detection such as SSD [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] or YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF] and finetuning it in a dedicated follow-me training set. The addition of depth sensing for this training set has the potential to significantly improve the feature extraction and thus the finetuning, because as illustrated figure 7.1, typical proposed objects highly correlate with depth discontinuities.

Robust and evolutive passive depth sensing

An interesting perspective that has been developed throughout this thesis it the idea of having a robust and evolutive depth sensing system based on vision. In other words, such a system does not require any kind of active sensors, and can improve during its real-time usage. This has been discussed by Casser et al. [START_REF] Casser | Unsupervised Learning of Depth and Ego-Motion: A Structured Approach[END_REF], where it was used to overfit to each example of the test set (with a reset of the network between examples).

This technique can be very beneficial in the context of stealth off-road autonomous vehicles. Due to their stealthiness, no active depth sensors can be used and depth has to be deduced from cameras. Besides, off-road environment are typically more heterogeneous than a clean road, and our robustness oriented solution can be beneficial compared to a classical structure from motion analytical algorithm or an evolutive single frame depth solution.

D.2. Diffusion stability

By identifying φ = h and v = c ( ∇ξ 2 )∇ξ in theorem 1, we can apply our theorem and deduce d dt t=0

L s (ξ + th) = -Ω div(c ( ξ 2 )∇ξ)hdV + ∂Ω c ( ξ 2 )∇ξh.ndS this equation is valid for every function h. As a consequence, we can deduce the aforementioned result.

D.2 Diffusion stability

First referenced in section 4.4.5.

We can consider the infinite 1D function ∀i ∈ Z, u 0 (i) = u(i, 0) = (-1) i (D.5)

The diffusion equation with a step size of η states ∀(i, n) ∈ Z × N + , u(i, nη) = u(i, (n -1)η) + η∆(u(i, (n -1)η))

Also, the discrete Laplacian of u 0 can be computed to get ∆u(i, 0) = -2u(i, 0) + u(i -1, 0) + u(i + 1, 0) = -4u(i, 0)

As a consequence, u(i, η) = u(i, 0) -4ηu(i, 0) = (1 -4η)u(i, 0)

By recurrence, it's easy to see ∀(i, n) ∈ Z × N, u(i, nη) = (-1) i × (1 -4η) n (D.6) If η is above 0.5, the solution diverges. The middle value oscillates around the null-Laplacian solution with increasing magnitude.

D.3 Diffusion of Inverse

Also first referenced in section 4.4.5. A simple and pratical example of instability is when the trainable parameters are the depth map θ and diffusion is applied on ξ. We then get:

f : R |Ω| → R |Ω| θ → ξ = 1 θ
The jacobian J f (of size |Ω| 2 ) can be written as

J f (i, j) = -δ ij × 1 θ 2 i
Where δ ij is 1 if i = j otherwise 0.

∂ t ξ = ∆ξ θ 4 = ∆ξ × ξ 4 (D.7)
The closer depth θ is to 0, the larger the effective optimization step, and possibly above the divergence threshold. (see section D.2) show our first method and the deemed more "robust" one when trying to smooth the inverse of 2 + sin(x) with noise, with both regular diffusion loss (4.4) and diffusion of gradient (4.5). While not being exactly the same as a regular diffusion, it appears that the robust method helps to get a more regularized smoothed map, especially where the gradient from trainable map to the map we want to diffuse is the highest: naive application of diffusion loss makes the parameters diverge. Proof. We proceed the same way as for theorem 2. The considered loss function is

D.4 Diffusion of Gradient

L s = 1 2 Ω c((∆ξ) 2 )dV (D.8)
We consider h a C 2 function on Ω and t ∈ R and compute the expression For ease of notation, we will now identify c ( ∆ξ 2 )∆ξ as K. From integration by part formula (theorem 1), we identify φ = ∇h and v = K. For second term (on ∂Ω), we identify φ = h and v = Kn . It can be noted that since ∂Ω is a closed set, its boundaries are an empty set ∂∂Ω = {∅}. We can also that n is piecewise constant in our case where ∂Ω is a rectangle. Thus divn = 0. Proof. If we retake the result from equation 4.15, we get:

∂ t A = ∇(div(A)) = ∆A + ρ(A) = ∆A +   ∂ 2 A y ∂y∂x -∂ 2 A x ∂y 2 ∂ 2 A x ∂x∂y - ∂ 2 A y ∂x 2   (D.13)
The second term of this equation ρA can be interpreted as the double rotational.

Indeed, when considering the 3D extension A 3 of our vector A, A 3 = we can extrapolate for our 2D case. Besides, by noting that for any 3D potential field V we have the equality rot(∇V)) = 0 we can retrieve the vectorwise diffusion equation. It can also be shown that the second term of our equation is constant throughout the whole diffusion We thus have to prove that for any 2D vector field A such that ρ(A) = 0, the minimization of its divergence is equivalent to vectorwise diffusion. This is already verified at the beginning of the diffusion process, when t=0. Our goal is then to show that the vector

ρ(A) =   ∂ 2 A y ∂y∂x -∂ 2 A x ∂y 2 ∂ 2 A x ∂x∂y - ∂ 2 A y ∂x 2   is constant
. By looking at the first coordinate of this vector, we get

∂ t ρ(A) x = ∂ t ∂ 2 A y ∂y∂x - ∂ 2 A x ∂y 2 = ∂ ∂y ∂ ∂x ∂ t A y - ∂ ∂y ∂ t A x
By substituing ∂ t A x and ∂ t A y with the first form of equation 4.13:

∂ t ρ(A) x = ∂ ∂y ∂ ∂x ∂ 2 A y ∂y 2 + ∂ 2 A x ∂x∂y - ∂ ∂y ∂ 2 A x ∂x 2 + ∂ 2 A y ∂x∂y = 0
We proceed with the same technique for ∂ t ρ(A) y to finally show that the vector ρ(A) is constant no matter its initial value. For their robustness in computer vision and their capacity to solve complex tasks, we chose to use convolutional neural networks (CNN). Our strategy is based on incrementally learning tasks with increasing complexity which first steps are to construct a depth map from the stabilized camera. This thesis is focused on studying ability of CNNs to be trained for this task.

In the case of stabilized footage, the depth map is closely linked to optical flow. We thus adapt FlowNet, a CNN known for optical flow, to output directly depth from two stabilized frames. This network is called DepthNet. This experiment succeeded with synthetic footage, but is not robust enough to be used directly on real videos. Consequently, we consider self supervised training with real videos, based on differentiably reproject images. This training method for CNNs being rather novel in literature, a thorough study is needed in order not to depend too much on heuristics. Finally, we developed a depth fusion algorithm to use DepthNet efficiently on real videos. Multiple frame pairs are fed to DepthNet to get a great depth sensing range.

Université Paris-Saclay

Espace Technologique / Immeuble Discovery Route de l'Orme aux Merisiers RD 128 / 91190 Saint-Aubin, France

1. 1

 1 Consumer products featuring camera stabilization. From left to right: DJI Phantom (mechanic gimbal), Parrot Bebop (dynamic cropping from fisheye), DJI Osmo mobile (mechanic handheld guimbal) 1 . . . . . . . . . . . . 1.2 Parrot's current Flight Plan interface . . . . . . . . . . . . . . . . . . . . . . 1.3 DJI's ActiveTrack interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 A "jerky" trajectory (red) and a smoother one (green) . . . . . . . . . . . . . 1.5 Example of RGB-D data, taken from[Lai+11]. For depth (right), bright colors indicate lower depth (and thus closer object to the camera plane) . . . . 1.6 Example of a stereo camera placed on a UAV in order to compute a depth map (here, a Parrot Slam Dunk on top of a Bebop2) . . . . . . . . . . . . . 1.7 First simulation of a neuron, x 0 • • • x n are the inputs, while w 0

4. 1

 1 Illustration of bilinear interpolation on the unit square. The four corners have fixed z values as indicated in the figure, the values in between are interpolated, and the interpolated values are represented by the color. 6 . . 4.2 General workflow architecture of SFMLearner[Zho+17]. (simplified, without regularization) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 In this example called Kanizsa Triangle [Kan55], humans instinctively see a triangle above everything else and thus depth discontinuity in areas with no luminance gradient. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Example of a scene with an horizontal plane. Right is the depth of the vertical line depicted on the rendered image. . . . . . . . . . . . . . . . . . 4.5 comparison of four smooth losses TV, TVV, diffusion, and gradient diffusion. Gaussian noise with σ = 0.2 has been added to input: a sinusoid and an abs function. SGD is used, with indicated learning rates and momentums in each graph title. Color indicates the number of iterations . . . . . . 4.6 Perona-Malik anisotropic diffusion of a picture of a flower, κ = 0.3 . . . . . . 4.7 1D Anisotropic diffusion and gradient diffusion of several functions. Dashed line indicates the image function with null gradient, except on x = 20 and x = 80. Plain line colors indicate number of iterations . . . . . . . . . . . . . 4.8 Example of occluded elements in a synthetic scene when trying to estimate depth of I t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9 Illustration of the direct projection workflow. The pixel p characterizes an occluded view with multiple possible depth values . . . . . . . . . . . . . . 4.10 Illustration of occlusion detection module. NaN values are white colored. Displacement is √ 2

  I t-1 and final inverse warp of I t+1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.14 Hyper-parameter search for λ and κ for our two scenes. An ideal (λ, κ) value should maximize the metric M . . . . . . . . . . . . . . . . . . . . . . 4.15 Logarithmic error (LE) for our two scenes, using different values of λ and κ. 4.16 Result of a photometric optimization with optimal weights λ and κ for our 6 different smooth losses on the Flower scene after 10 4 iterations. Learning rate is 0.15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.17 Result of a photometric optimization with optimal weights λ and κ for our 6 different smooth losses on the Horizon scene after 10 4 iterations. Learning rate is 0.15. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.18 Comparison of a simple optimization (first two rows) and an optimization using occlusion module (last two rows). Optimal parameters for regular diffusion loss are used. Blacked parts of warped images are ignored thanks to our occlusion module. Thanks to this, the usual ghost effect of depth on background is avoided. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.19 Result on Flower scene, with regular diffusion smooth loss. No ghost effect can be seen around the foreground depth, but it performs poorly on foreground depth. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.20 Example of change of luminosity inside a video, when entering the tunnel, the camera rises the exposition, and corresponding point have now different colors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.21 graph of the function f : (x, y) → 2xy+C

5. 1

 1 General workflow architecture. target and reference indices (t and r) are chosen randomly for each sequence (even within the same batch); output transformations of PoseNet are compensated so that T t→t is identity. d 0 is a fixed nominal displacement. . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Comparison between our method and SFMLearner. [Zho+17] on StillBox and KITTI [Gei+13]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 x 5.3 Subjective comparison of disparity maps between SFMLearner [Zho+17]

6. 1

 1 Estimation-wise log error of supervised DepthNet 64→128→256→512 on StillBox. 6.2 Example of a multi-modal depth distribution scene with corresponding depth: foreground is much closer than mean depth and background further. . . . . 6.3 graph of function f , defined in equation 6.3 . . . . . . . . . . . . . . . . . . 6.4 Multiple shifts architecture with n parallel depth inferences . Numeric integration, given a desired displacement d opt gives the closest possible displacement between frames, denoted d * , along with corresponding shift δt. The fusion block computes pixel-wise weights from β 0 , • • • , β n-1 to make a weighted mean of β 0 d * 0 , • • • , β n-1 d * n-1 . . . . . . . . . . . . . . . . . . . . . 6.5 real condition application of the multi-shift algorithm with DepthNet. Top left is input. Last row are raw outputs of the network (from 0 to 100, corresponding to a β from 0 to 1). Top right is fused output, capped up to 100m. Numbers in the upper left corner of raw inputs indicate respectively temporal frame shift and measured displacement magnitude in meters. The UAV is flying forward at a speed of 1m.s -1 and an altitude of ≈ 12m. . . . . . . . 6.6 results for synthetic 256x256 scenes with noisy orientation. DepthNet has been tested with n = 1 and n = 2, DepthNet Clamped with n = 1 • • • 3 Tiny DepthNet Clamped with n = 1 • • • 4. Y axis is Mean Relative Error (MRE), X axis is inference speed, in ms on a Quadro M1000M GPU. . . . . . . . . 6.7 Experimental setup for our proof of concept . . . . . . . . . . . . . . . . . . 6.8 Experimental Scene where the drone is expected to travel around obstacles when desired trajectory collides with them . . . . . . . . . . . . . . . . 7.1 Multiple different but correlated semantic levels of a particular scene. From left to right: input, semantic segmentation masks, instance masks, depth. Figure taken from Kendall et al. [KGC18] . . . . . . . . . . . . . . . . . . . D.1 Left: diffusion of the noisy function x → 1 2+sin(x) (bottom) and its corresponding inverse (top). Center: diffusion of the inverse of the noisy function x → 2 + sin(x) (bottom) and the resulting function (top). Right: Evolution obtained by interleaving diffusion (10 iterations, learning rate of 0.4) and minimization of difference with diffused. . . . . . . . . . . . . . . . . . . . . D.2 Gradient diffusion equivalent of figure D.1. . . . . . . . . . . . . . . . . . . . xi List of Tables 2.1 Me() is the median operator, and δ is a validation measure (e.g. L1 distance) 2.2 Comparison of standard and mean errors and corresponding maximum for the tail distribution of f . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Considered Losses Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Datasets sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 StillBox dataset parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Comparison of three possible depth representations for the network output 3.4 Quantitative results for 64x64 images. . . . . . . . . . . . . . . . . . . . . . 3.5 quantitative results for depth inference networks. FlowNetS is modified with 1 channel outputs (instead of 2 for flow), trained from scratch for depth with Still Box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Size (millions of parameters) and Inference speeds (fps) on different de-

  For supervision, D is depth and O is orientation. → denotes fine tuning. . . . . . . . . . . . . . . . . . . . . . . . 102 5.2 Quantitative tests on StillBox, no pretraining has been done. The supervised DepthNet is here to give an hint on a theoretical limit since it uses the same network, but with depth supervision . . . . . . . . . . . . . . . . . 103 5.3 Quantitative tests on upside-down KITTI [Gei+13]: sky is down and ground is up. No training has been done. Constant Plane outputs the same depth for every pixel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 xii xiii

  . . . . . . . . . . . . . . . . 1.3.4 Interest of deep learning for consumer UAVs . . . . . . . . . . . 1.4 Approach and goal of the thesis . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.1 Hardware setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.4.2 Discarding the naive solution . . . . . . . . . . . . . . . . . . . . 10 1.4.3 Training with increasing complexity . . . . . . . . . . . . . . . . . 11 1.4.4 Resulting scope of this thesis . . . . . . . . . . . . . . . . . . . . 12 1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.6 Outline of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Figure 1 . 1 :

 11 Figure 1.1: Consumer products featuring camera stabilization. From left to right: DJI Phantom (mechanic gimbal), Parrot Bebop (dynamic cropping from fish-eye), DJI Osmo mobile (mechanic handheld guimbal) 1

Figure 1 . 5 :

 15 Figure 1.5: Example of RGB-D data, taken from[Lai+11]. For depth (right), bright colors indicate lower depth (and thus closer object to the camera plane)

Figure 1 . 7 :

 17 Figure 1.7: First simulation of a neuron, x 0 • • • x n are the inputs, while w 0 • • • w n and α are changeable parameters.4

Figure 2 . 1 :

 21 Figure 2.1: Illustration of scale ambiguity. Aerial view of Chambord Castle and 1/30-scale model replica at France Miniature 1

Figure 2 . 2 :

 22 Figure 2.2: Ames room, a famous example of forced perspective, taken at Cité des Sciences, Paris 2

Figure 2 . 3 :

 23 Figure 2.3: Dakar 2019 photograph by Frank Fife / AFP

Figure 2 . 4 :

 24 Figure 2.4: Example of consecutive frames with a moving object in KITTI dataset [Gei+13]

Figure 2 . 5 :

 25 Figure 2.5: Maximum error probability guaranteed for standard and mean error. For f ∈ ME, SE 2 ME , mean error gives more information. Above, standard error is the most informative.

Contents 3 . 1

 31 Geometric definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Camera plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Pinhole camera model . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Moving camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.4 Optical flow and disparity . . . . . . . . . . . . . . . . . . . . . . 3.1.5 Focus of expansion . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.6 Finding depth from optical flow . . . . . . . . . . . . . . . . . . . 3.2 Deep learning for optical flow . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 FlowNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Optical flow for depth generation . . . . . . . . . . . . . . . . . . . . . . 3.3.1 From disparity to depth . . . . . . . . . . . . . . . . . . . . . . . 3.3.2 From flow divergence to depth . . . . . . . . . . . . . . . . . . . 3.3.3 Conclusions on limitations . . . . . . . . . . . . . . . . . . . . . . 3.4 End-to-end learning of depth generation . . . . . . . . . . . . . . . . . . 3.4.1 Introducing StillBox, a depth synthetic dataset . . . . . . . . . . . 3.4.2 Dataset set augmentation . . . . . . . . . . . . . . . . . . . . . . 3.5 DepthNet, a depth fully convolutional neural network . . . . . . . . . . . 3.5.1 The constant speed assumption . . . . . . . . . . . . . . . . . . 3.5.2 On the optimal network output . . . . . . . . . . . . . . . . . . . 3.5.3 General structure . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Training details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6.1 Influence of resolution on training . . . . . . . . . . . . . . . . . . 3.6.2 Comparison with DeMoN and depth from optical flow . . . . . . . 3.7 UAV navigation use-case . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Conclusion of this chapter . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 1 :

 31 Figure 3.1: Illustration of a pinhole camera. the focal length is the depth of the box. 1

Figure 3

 3 Figure 3.2: A flow field with constant positive divergence, even if the value at the center is a null vector. Figure taken from [HCC17]

Figure 3

 3 Figure 3.3: Samples of Flying Chairs for Optical Flow training, with more than 22K frame pairs with perfect ground-truth. Same color code as Middlebury dataset [Bak+11]

Figure 3 . 5 :

 35 Figure 3.5: Percentiles of optical flow magnitude wise end point error (EPE) of our implementation of flownet, tested on 100 images of Flying Chairs dataset. Mean EPE on Flying Chairs is below 2 pixels.

Figure 3 . 7 :

 37 Figure 3.7: Comparison between "1+ELU" and "ReLU(1+x)", functions and derived functions

Figure 3 .

 3 Figure 3.11: Median depthwise metric, relative and logarithmic errors of DepthNet on 64x64 images with Identity post processing. Shaded areas are delimited by first and third quartiles.

  Figure 3.13: Qualitative comparison between DeMoN [Umm+17], FlowNet [Dos+15] and DepthNet.

Figure 3 .

 3 Figure 3.14: Depth errors with respect to distance to FOE on our test set. Contrary to FlowNet, DepthNet and DeMoN do not depend on it

Figure 4 . 1 :

 41 Figure 4.1: Illustration of bilinear interpolation on the unit square. The four corners have fixed z values as indicated in the figure, the values in between are interpolated, and the interpolated values are represented by the color. 2

  Photo loss

Figure 4 . 2 :

 42 Figure 4.2: General workflow architecture of SFMLearner[Zho+17]. (simplified, without regularization)

Figure 4 . 3 :

 43 Figure 4.3: In this example called Kanizsa Triangle [Kan55], humans instinctively see a triangle above everything else and thus depth discontinuity in areas with no luminance gradient.

Figure 4 . 4 :

 44 Figure 4.4: Example of a scene with an horizontal plane. Right is the depth of the vertical line depicted on the rendered image.

  equation 4.5, with c = x → 2 √ x L TVV = Ω |∆ξ|dS However, every Laplacian value is dependent on the adjacent values. More generally, from Ostrogradski 2D theorem, we can state that Ω div(A)dS = ∂Ω A.ndl where A is any 2D function, Ω is a closed surface, and ∂Ω its boundaries, and n is the normal vector to the boundary tangent. By identifying A = ∇ξ, we get ∀Ω ⊆ Ω, Ω ∆ξdS = ∂Ω ∇ξ.ndl Here, since we would use the absolute values, the surfaces Ω to consider are the ones where Laplacian values keep the same sign.

Figure 4 .

 4 Figure 4.5: comparison of four smooth losses TV, TVV, diffusion, and gradient diffusion. Gaussian noise with σ = 0.2 has been added to input: a sinusoid and an abs function. SGD is used, with indicated learning rates and momentums in each graph title. Color indicates the number of iterations

Figure 4 .

 4 Figure 4.6: Perona-Malik anisotropic diffusion of a picture of a flower, κ = 0.3

Figure 4 .Figure 4 . 8 :

 448 Figure 4.7: 1D Anisotropic diffusion and gradient diffusion of several functions. Dashed line indicates the image function with null gradient, except on x = 20 and x = 80. Plain line colors indicate number of iterations

Figure 4 .

 4 Figure 4.9: Illustration of the direct projection workflow. The pixel p characterizes an occluded view with multiple possible depth values

Figure 4 .

 4 Figure 4.10: Illustration of occlusion detection module. NaN values are white colored. Displacement is

Figure 4 .

 4 Figure 4.11: Our two toy problems "Flower" (top) and "Horizon" (bottom). For each scene, first row are the three frames of the scene, and second row is the ground truth depth of the center frame. Images are 200 × 200 pixels. Scenes have no rotations. Flower scene has a displacement of 1 10 [1, 1, 0] (camera goes right and down, so the image goes left and up), Horizon scene has a displacement of 1 10 [0, 0, 2] (camera goes forward). Camera is a standard centered pinhole with 90 • of field of view. For optimization depth starting points are equal to background, i.e. 10m

Figure

  Figure 4.12: First row: Illustration of optical flow construction, and how a perfect depth produces imperfect warping. A "ghost" of the foreground can be seen on occluded areas, which the reprojection based optimization will wrongly try to solve. Second row, our two imperfectly warped frames from I t-1 and I t+1 to I t . Last row, the differences between warped images and I t are the occlusion areas. We can see that the non-zero areas don't cover the same pixels.

Figure 4 .

 4 Figure 4.14: Hyper-parameter search for λ and κ for our two scenes. An ideal (λ, κ) value should maximize the metric M

  and

Figure 4 .

 4 Figure 4.15: Logarithmic error (LE) for our two scenes, using different values of λ and κ.

Figure 4 .

 4 Figure 4.16: Result of a photometric optimization with optimal weights λ and κ for our 6 different smooth losses on the Flower scene after 10 4 iterations. Learning rate is 0.15

Figure 4 .

 4 Figure 4.19: Result on Flower scene, with regular diffusion smooth loss. No ghost effect can be seen around the foreground depth, but it performs poorly on foreground depth.

Figure 4 .

 4 Figure 4.20: Example of change of luminosity inside a video, when entering the tunnel, the camera rises the exposition, and corresponding point have now different colors.

Figure 4 .

 4 Figure 4.21: graph of the function f : (x, y) → 2xy+Cx 2 +y 2 +C used for l and c in SSIM (here, C is 0.01 2 ). This function is only convex around its maximum, which make its gradient based optimization difficult

  of SFMLearner . . . . . . . . . . . . . . . . . . . . . . . . 95 5.2 Workflow presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.2.1 Pose estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.2.2 Frame stabilization . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.2.3 Depth computing and pose normalization . . . . . . . . . . . . . 98 5.2.4 Loss functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.3 Training datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.4.1 Qualitative results . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.4.2 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . 99 5.4.3 Testing methodology for DepthNet . . . . . . . . . . . . . . . . . 99 5.4.4 Quantitative results . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.4.5 Domain adaptation results . . . . . . . . . . . . . . . . . . . . . . 105 5.5 Conclusion on unsupervised DepthNet . . . . . . . . . . . . . . . . . . . 105

r

  before feeding it to DepthNet, leading to the same situation as original DepthNet supervised training. Detailed section 5.2.2. • Compute the unnormalized depth map θ t . DepthNet(I STAB r , I t ) = θ(I STAB r , I t )

T

  Photo loss

Figure 5 .

 5 Figure 5.2 presents a qualitative comparison between SFMLearner and Unsupervised DepthNet, on KITTI and StillBox. If results are comparable on KITTI, DepthNet is much sharper on StillBox.

Figure 5 . 2 :

 52 Figure 5.2: Comparison between our method and SFMLearner. [Zho+17] on StillBox and KITTI [Gei+13].

Figure 5 . 3 :

 53 Figure 5.3: Subjective comparison of disparity maps between SFMLearner [Zho+17] and our method on a small UAV dataset.

Figure 5 . 4 :

 54 Figure5.4: Some failure cases of our method on KITTI. First column is a detail of a larger image. The foreground car is moving forward and it's detected as far away, while the background car is moving toward us and is detected as close. Second column is a poorly textured road.

Figure 6

 6 Figure 6.1: Estimation-wise log error of supervised DepthNet 64→128→256→512 on StillBox.

Figure 6

 6 Figure 6.2: Example of a multi-modal depth distribution scene with corresponding depth: foreground is much closer than mean depth and background further.

Figure 6

 6 Figure 6.3: graph of function f , defined in equation 6.3

Figure 6 . 4 :

 64 Figure 6.4: Multiple shifts architecture with n parallel depth inferences . Numeric integration, given a desired displacement d opt gives the closest possible displacement between frames, denoted d * , along with corresponding shift δt. The fusion block computes pixel-wise weights from β 0 , • • • , β n-1 to make a weighted mean of β 0 d * 0 , • • • , β n-1 d *

Figure 6

 6 Figure 6.5: real condition application of the multi-shift algorithm with DepthNet. Top left is input. Last row are raw outputs of the network (from 0 to 100, corresponding to a β from 0 to 1). Top right is fused output, capped up to 100m. Numbers in the upper left corner of raw inputs indicate respectively temporal frame shift and measured displacement magnitude in meters. The UAV is flying forward at a speed of 1m.s -1 and an altitude of ≈ 12m.

Figure 6

 6 Figure 6.6: results for synthetic 256x256 scenes with noisy orientation. DepthNet has been tested with n = 1 and n = 2, DepthNet Clamped with n = 1 • • • 3 Tiny DepthNet Clamped with n = 1 • • • 4. Y axis is Mean Relative Error (MRE), X axis is inference speed, in ms on a Quadro M1000M GPU.

Figure 6 . 7 :

 67 Figure 6.7: Experimental setup for our proof of concept

Contents 7 . 1

 71 Limitations and future perspectives . . . . . . . . . . . . . . . . . . . . . 117 7.1.1 UAV depth validation dataset . . . . . . . . . . . . . . . . . . . . 117 7.1.2 Robust smooth and photometric loss . . . . . . . . . . . . . . . . 118 7.1.3 Moving scenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 7.1.4 Bayesian depth inference . . . . . . . . . . . . . . . . . . . . . . 119 7.2 Alternative Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 7.2.1 Depth sensing for multi-task learning . . . . . . . . . . . . . . . . 119 7.2.2 Robust and evolutive passive depth sensing . . . . . . . . . . . . 119

Figure 7 . 1 :

 71 Figure 7.1: Multiple different but correlated semantic levels of a particular scene. From left to right: input, semantic segmentation masks, instance masks, depth. Figure taken from Kendall et al. [KGC18]

Figure D. 1 :

 1 Figure D.1: Left: diffusion of the noisy function x →

Figure D. 1

 1 Figure D.1 and figure D.2 show our first method and the deemed more "robust" one when trying to smooth the inverse of 2 + sin(x) with noise, with both regular diffusion loss (4.4) and diffusion of gradient (4.5). While not being exactly the same as a regular diffusion, it appears that the robust method helps to get a more regularized smoothed map, especially where the gradient from trainable map to the map we want to diffuse is the highest: naive application of diffusion loss makes the parameters diverge.

  Figure D.1 and figure D.2 show our first method and the deemed more "robust" one when trying to smooth the inverse of 2 + sin(x) with noise, with both regular diffusion loss (4.4) and diffusion of gradient (4.5). While not being exactly the same as a regular diffusion, it appears that the robust method helps to get a more regularized smoothed map, especially where the gradient from trainable map to the map we want to diffuse is the highest: naive application of diffusion loss makes the parameters diverge.

D. 4 . 1

 41 Laplacian minimizationFirst referenced in section 4.4.7. The goal is to prove the equation 4.12 ∇ ξ L s∆ = -∆(c ((∆ξ) 2 )∆ξ)

Figure D. 2 :

 2 Figure D.2: Gradient diffusion equivalent of figure D.1.

  d dt t=0 L s (ξ + th) = -Ω ∇K.∇hdV + ∂Ω (Kn).∇hdSThis time, we apply the formula twice. For first term (on Ω), we identify φ = h and v = ∇K.

∂Ω(D. 4 . 2 D. 4 . 3

 4243 Kn).∇hdS = -∂Ω hdiv(Kn)dS = -∂Ω h∇K.ndS (D.10)We can combine equations D.9 and D.10 to get:d dt t=0 L s (ξ + th) = Ω h∆KdV -2 ∂Ω h∇K.ndSWe can deduce the following expression for ∇ ξ L s :∀p ∈ Ω, ∇ ξ L s (p) = -∆(c ((∆ξ) 2 )∆ξ) + ∇(c ((∆ξ) 2 )∆ξ).n if p ∈ ∂Ω 0 otherwise (D.11) Divergence minimizationFirst referenced in section 4.4.8. The goal is to prove the equation 4.14∇ A L s∆ = -∇ c div(A) 2 div(A)withL s∆ = Ω c div(A) 2 dSProof. We want to compute ∇ A L s We now consider h a C 1 multidimensional function on Ω and t ∈ R and compute the expressiond dt t=0 L s (A + th) divA + tdivh) 2 )dV = Ω c ((divA) 2 )divAdivhdVWe now identify K = c ((divA) 2 )divA. We also apply formula from theorem 1 by identifying v = h and φ = K.d dt t=0 L s (A + th) = -Ω h.∇KdV + ∂Ω Kh.ndSWe can then deduce:∀p ∈ Ω, ∇ A L s (p) = -∇(c ((divA) 2 )divA) + (c ((divA) 2 )divA)n if p ∈ ∂Ω 0 otherwise (D.12) Isotropic gradient diffusion First referenced in section 4.4.8. The goal is to prove the equation 4.16 ∇ A L gdi f f = -∆A where L gdi f f = Ω (divA) 2 dS and ∇(divA 0 ) = ∆A 0 at the begining of the optimization.

  (A 3 )) = ∇ ∧ (∇ ∧ A 3 ) = relation ∆A 3 = ∇(div(A 3 )) -rot(rot(A 3 ))

Titre:

  Apprentissage robuste d'une carte de profondeur pour l'évitement d'obstacle dans le cas des caméras volantes, monoculaires et stabilisées Mots clés: carte de profondeur, robuste, caméra stabilisée, caméra monoculaire, reseau de neurones, apprentissage profond Résumé: Les drones orientés grand public sont principalement des caméras volantes, stabilisées et de bonne qualité. Ceux-ci ont démocratisé la prise de vue aérienne, mais avec leur succès grandissant, la notion de sécurité est devenue prépondérante. Ce travail s'intéresse à l'évitement d'obstacle, tout en conservant un vol fluide pour l'utilisateur. Dans ce contexte technologique, nous utilisons seulement une caméra stabilisée, par contrainte de poids et de coût. Pour leur efficacité connue en vision par ordinateur et leur performance avérée dans la résolution de tâches complexes, nous utilisons des réseaux de neurones convolutionnels (CNN). Notre stratégie repose sur un système de plusieurs niveaux de complexité dont les premières étapes sont de mesurer une carte de profondeur depuis la caméra. Cette thèse étudie les capacités d'un CNN à effectuer cette tâche. La carte de profondeur, étant particulièrement liée au flot optique dans le cas d'images stabilisées, nous adaptons un réseau connu pour cette tâche, FlowNet, afin qu'il calcule directement la carte de profondeur à partir de deux images stabilisées. Ce réseau est appelé DepthNet. Cette méthode fonctionne en simulateur avec un entraînement supervisé, mais n'est pas assez robuste pour des vidéos réelles. Nous étudions alors les possibilités d'auto-apprentissage basées sur la reprojection différentiable d'images. Cette technique est particulièrement nouvelle sur les CNNs et nécessite une étude détaillée afin de ne pas dépendre de paramètres heuristiques. Finalement, nous développons un algorithme de fusion de cartes de profondeurs pour utiliser DepthNet sur des vidéos réelles. Plusieurs paires différentes sont données à DepthNet afin d'avoir une grande plage de profondeurs mesurées. Title: Robust Learning of a depth map for obstacle avoidance with a monocular stabilized flying camera Keywords: depth map, robust, stabilized camera, monocular camera, neural networks, deep learning Abstract: Customer unmanned aerial vehicles (UAVs) are mainly flying cameras. They democratized aerial footage, but with their success came security concerns. This works aims at improving UAVs safety with obstacle avoidance, while keeping a smooth flight. In this context, we use only one stabilized camera, because of weight and cost incentives.

  • • • w n and α are changeable parameters. Dakar 2019 photograph by Frank Fife / AFP . . . . . . . . . . . . . . . . . . 2.4 Example of consecutive frames with a moving object in KITTI dataset [Gei+13] 19 2.5 Maximum error probability guaranteed for standard and mean error. For f ∈ ME, SE 2 ME , mean error gives more information. Above, standard error is the most informative. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Illustration of a pinhole camera. the focal length is the depth of the box. 5 . 3.2 A flow field with constant positive divergence, even if the value at the center is a null vector. Figure taken from [HCC17] . . . . . . . . . . . . . . . . . . 3.3 Samples of Flying Chairs for Optical Flow training, with more than 22K frame pairs with perfect ground-truth. Same color code as Middlebury dataset [Bak+11] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 FlowNet general architecture for end-to-end optical flow generation, illustrations taken from [Dos+15] . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Percentiles of optical flow magnitude wise end point error (EPE) of our implementation of flownet, tested on 100 images of Flying Chairs dataset. DepthNet structure parameters . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 General Structure of convolution and deconvolution blocks of DepthNet. . . 3.10 Qualitative results of DepthNet for 64x64 images. FOE is indicated by a red cross on Image1 and Image 2 . . . . . . . . . . . . . . . . . . . . . . . 3.11 Median depthwise metric, relative and logarithmic errors of DepthNet on 64x64 images with Identity post processing. Shaded areas are delimited by first and third quartiles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Qualitative results of finetuned DepthNet 64→128→256→512 for 512x512 images. FOE is indicated by a red cross on Image 1 and Image 2 . . . . . . . 3.13 Qualitative comparison between DeMoN [Umm+17], FlowNet [Dos+15] and DepthNet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.14 Depth errors with respect to distance to FOE on our test set. Contrary to FlowNet, DepthNet and DeMoN do not depend on it . . . . . . . . . . . . . 3.15 some results on real images input. Top is from a Bebop drone footage, bottom is from a gimbal stabilized smartphone video. . . . . . . . . . . . . . 3.16 Example of depth flickering for the sky. The two frames are ≈ 83ms apart. .

2 . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Graph of milestones complexity regarding domain and task complexity. Arrows indicate a possible reutilization of networks with a fine-tuning fashion. Red points indicate considered milestones in this thesis. . . . . . . . . . . . 2.1 Illustration of scale ambiguity. Aerial view of Chambord Castle and 1/30scale model replica at France Miniature 3 . . . . . . . . . . . . . . . . . . . . 2.2 Ames room, a famous example of forced perspective, taken at Cité des Sciences, Paris 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Mean EPE on Flying Chairs is below 2 pixels. . . . . . . . . . . . . . . . . . 3.6 Some examples of our renderings with associated depth maps, from 0 to 100m. Visible reticle is not comprised in raw images . . . . . . . . . . . . . 3.7 Comparison between "1+ELU" and "ReLU(1+x)", functions and derived functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii 3.8

  Motivations of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Anatomy of a consumer Drone . . . . . . . . . . . . . . . . . . . 1.1.2 The problem of obstacles avoidance . . . . . . . . . . . . . . . . 1.2 Depth perception for navigation in robotics . . . . . . . . . . . . . . . . .

	Chapter 1
	Introduction
	UAV Contents	Unmanned Aerial Vehicle
	IMU	Inertial Measurement Unit 1.1
	HUD		Head Up Display
	RTH		Return To Home
	RGB		Red Green Blue
	SLAM Simultaneous Localisation and Mapping
	CNN		Convolutional Neural Network
	FOE		Focus of Expansion (see below)
	P δ		Precision P θ ∈ 1 δ θ, δθ
			Vectors
	p		2D point in the image
	P		3D point in the world
	Φ t		Focus of Expansion (also referred to as FOE) with respect to camera translation t.
			Scalar Fields
	θ		depth map of a particular image.
	F		Optical Flow map

Scalars

L p

Photometric loss, denotes the difference between two images to be minimized L s Smooth loss, denotes the regularization term that tries to constrain a particular 2D map to be smooth.

Error measures (see 2.4.2)

MAE Mean Absolute Error E|θ -θ| MRE Mean Relative Error E |θ-θ| θ MLE Mean Logarithmic Error E|log(θ)log( θ)| SAE Standard Absolute Error E(|θ -θ| 2 ) SLE Standard Logarithmic Error E(|log(θ)log( θ)| 2 ) ξ inverse depth map. ξ(p) = 1 θ(p)

Vector Fields I Multi channel Image. Usually with 3 channels. 1.2.1 Considered sensors for depth perception . . . . . . . . . . . . . 1.2.2 Active sensors advantages and limitations . . . . . . . . . . . . . 1.2.3 Vision based depth perception . . . . . . . . . . . . . . . . . . . 1.3 Deep Learning Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 A brief historic . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Deep learning with convolutional neural networks for computer cision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 Deep learning for decision making .

Chapter 2 Depth estimation from a camera

	Contents
	2.1 Linking depth with reality: the scale factor problem . . . . . . . . . . . . 15
	2.2 Depth from motion vs depth from context . . . . . . . . . . . . . . . . . 16
	2.2.1 Depth from context advantages and limitations . . . . . . . . . . 16
	2.2.2 Depth from motion advantages and limitations . . . . . . . . . . 18
	2.2.3 Our strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
	2.3 Obstacle avoidance oriented depth quality . . . . . . . . . . . . . . . . . 19
	2.3.1 Scale invariant estimation . . . . . . . . . . . . . . . . . . . . . . 19
	2.3.2 Obstacle avoidance requirements . . . . . . . . . . . . . . . . . 19
	2.4 Anatomy of an error function . . . . . . . . . . . . . . . . . . . . . . . . 20
	2.4.1 Standard error vs mean error . . . . . . . . . . . . . . . . . . . . 21
	2.4.2 Usual error functions . . . . . . . . . . . . . . . . . . . . . . . . . 22
	2.5 Conclusion on error measures . . . . . . . . . . . . . . . . . . . . . . . 24

Table 2

 2

.3: Considered Losses Summary

Table 3

 3

	4W

Output

Figure 3.9: General Structure of convolution and deconvolution blocks of DepthNet. Image 1 Image 2 Ground Truth θ Estimation θ Figure 3.10: Qualitative results of DepthNet for 64x64 images. FOE is indicated by a red cross on Image1 and Image 2 .5: quantitative results for depth inference networks. FlowNetS is modified with 1 channel outputs (instead of 2 for flow), trained from scratch for depth with Still Box

The second noticeable fact is that near FOE, the network has no problem inferring depth. It means that it uses both optical flow divergence and interpolates with neighbor depth values, where disparity is easier to measure.

  Table 3.6: Size (millions of parameters) and Inference speeds (fps) on different devices. Batch sizes are 1 and 8 (when applicable). A batch size of 8 means 8 depth maps are computed at the same time. DeMoN was tested only on a 980Ti. On the second table, a quick comparison between tested GPUs

	Network	size	980Ti	Quadro M1000M	TX1
			1	8	1	8	1	8
	DeMoN	48.0 11	4.8 N/A N/A N/A N/A
	FlowNetS 64	39.4 225 153 86	41	27	14
	DepthNet 64	7.33 364 245 213	88	70	40
	FlowNetS 512 39.4 69	8.8	15	N/A	2.8 N/A
	DepthNet 128 7.33 294 118 205	74	51	15
	DepthNet 256 7.33 178 36 144	29	39	3.2
	DepthNet 512 7.33 68	8.8	58	8.2	9.2 N/A
	Card model		GFLOPS (FP32) Memory (GB)
	GTX 980Ti			5, 632			6
	Quadro M1000M		1, 390			2
	Tegra X1			512			8

  • A mixture of FlowNetS and depth from optical flow (not to be confused with FlowNetS Table3.7: Quantitative comparison between the three methods. FlowNetS → depth row is deducing depth from optical flow and displacement (perfectly known)

	Method	MAE MRE MLE
	DepthNet	2.67	0.162 0.132
	FlowNetS → depth 6.462 0.203 0.172
	DeMoN	18.85	1.57 0.726

Table 5 .

 5 2: Quantitative tests on StillBox, no pretraining has been done. The supervised DepthNet is here to give an hint on a theoretical limit since it uses the same network, but with depth supervision Table 5.1 presents quantitative results compared to prior works on KITTI dataset. Prior works are divided into three categories. The first one with Eigen et al. [EPF14] and DORN from Fu et al.

.536 0.424 0.673 0.807

  Table 5.3: Quantitative tests on upside-down KITTI [Gei+13]: sky is down and ground is up. No training has been done. Constant Plane outputs the same depth for every pixel Simplifying the scale factor determination by normalizing DepthNet . . . 107 6.2 Optimal frame shift determination . . . . . . . . . . . . . . . . . . . . . . 108 6.3 Multiple shifts inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.4 Training a specific depth range with a clamped DepthNet . . . . . . . . . 113 6.5 Our proof of concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 6.5.1 Model Predictive Control . . . . . . . . . . . . . . . . . . . . . . . 113 6.5.2 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . 113 6.6 Conclusions for this chapter . . . . . . . . . . . . . . . . . . . . . . . . . 115

	Method Supervised DepthNet Constant Plane SFMLearner [Zho+17] * SFMLearner [Zho+17] * Unsupervised DepthNet Unsupervised DepthNet DepthNet in the wild training set scale factor S P -GT K GT K P S → K P S → K P Contents 0.604 0.414 11.152 0Chapter 6 supervision MAE MRE MLE SAE SLE P 1.25 P 1.25 2 P 1.25 3 D + O 7.716 0.637 0.443 10.499 0.548 0.359 0.614 0.774 -7.547 0.458 0.476 12.124 0.600 0.296 0.547 0.752 -8.433 0.551 0.551 12.469 0.680 0.255 0.482 0.668 -16.085 1.511 0.790 19.677 0.927 0.176 0.342 0.494 -7.759 0.596 0.442 11.250 0.555 0.370 0.628 0.784 O 7.731 6.1
	DepthNet

Introduced with their product Mavic Air https://www.dji.com/fr/mavic-air, retrieved 07/24/2019

As reported e.g. in a 2010 ANSES report https://www.anses.fr/fr/system/files/AP2008sa0408. pdf, retrieved 07/24/2019

Figure source: http://neuralnetworksanddeeplearning.com, retrieved 07/24/2019

A compelling robust challenge was made during CVPR 2018. Unfortunately, no technical report has been made. Leaderboard can be found here: http://www.robustvision.net/leaderboard.php, retrieved 07/24/2019

1.6. Outline of this thesis

Sources: https://commons.wikimedia.org/wiki/File:ChateauChambordArialView01.jpg and

Source: https://commons.wikimedia.org/wiki/File:Ames_room_forced_perspective.jpg, retrieved

3/28/2019

https://www.blender.org/, retrieved 07/24/2019

https://www.flickr.com/, retrieved 07/24/2019

https://github.com/ClementPinard/DepthNet, retrieved 07/24/2019

https://gitlab.ensta.fr/pinard/thesis-notebooks and https://gitlab.ensta.fr/pinard/ direct-warper, retrieved 07/24/2019

https://github.com/ClementPinard/unsupervised-depthnet, retrieved 07/24/2019

https://stillbox.ensta.fr/demo_depthnet.mp4

http://stillbox.ensta.fr/DepthNetResults.mp4(retrieved 07/24/2019)

Available at https://github.com/Parrot-Developers/pdraw, retrieved 07/24/2019

http://stillbox.ensta.fr/poc_depthnet_mpc.mp4(retrieved 07/24/2019)

https://www.pix4d.com/, retrieved 07/24/2019

By contraposition, we can say that if we have |V -| = |V + |, we can find a scale factor α < 1 such that {α θ} will have a better mean relative error.

Chapter 4 Training Depth estimation with auto-supervision Direct warp Module Presentation

As a proof of concept, we propose a consistency check using direct warping instead of inverse warping. While the inverse warp will be used for photometric error minimization, occlusion areas will be detected thanks to this consistency check using direct warping. We will then be able to dismiss these areas from being optimized with photometric error. Besides, no differentiation needs to be done with direct warping, allowing us to use a very simple pipeline.

Here, instead of finding the exact set of possible depth for each point, which would imply using complicated geometry techniques such as ray-tracing [START_REF] Whitted | An improved illumination model for shaded display[END_REF], we chose to do the opposite: from a set of points in R 2 , we discretely paint pixels in Ω on the output that are close to points actual projections. This technique is known as rasterization: it is much simpler but also less precise and not easily differentiable, although the differentiability aspect is not our priority since we only want to know occlusion maps.

To model direct warping, we assumed the initial depth map θ t to be not continuous as we did for I t 2 for inverse warping, but a "square cloud". The difference with a point cloud is that each point has a "width". The width is computed so that the projection of the 3D square into the initial frame is exactly 1 pixel large. This algorithm is a simplified version of McMillan's image-space Gaussian reconstruction [START_REF] Mcmillan | AN IMAGE-BASED APPROACH TO THREE-DIMENSIONAL COMPUTER GRAPHICS[END_REF].

For frame coordinates we take the convention that for an integer coordinate, the point is at the center of the square pixel. As a consequence, when we have a float frame coordinate point (u, v), the corresponding pixel to paint is at (round(u), round(v))

The rationale for square cloud is that when we have a forward movement, and thus a zoom in our depth map, disocclusions don't appear, because along with the distance between the squares, their width is also increased.

The algorithm for our projection module is presented algorithm 1.

Algorithm 1 Compute θ t→t 2 set θ t→t 2 to infinity everywhere set Id to NaN everywhere for all p ∈ Ω do Width = θ t (p)

Appendix A

Illustration of compensating effect of linking depth scaled factor to movement estimation

This appendix is first referenced in section 2.1. Let's consider the simplest obstacle avoidance maneuver which consists in braking with a constant force opposed to velocity vector in front of an obstacle at a distance θ knowing the vehicle velocity magnitude v.

The vehicle speed needs to get to 0 before having traveled the distance θ. A simple application of work to kinetic energy relation can tell us that the force magnitude needed will be

where m is the mass of the vehicle. Now if we consider an error on measured speed so that the estimation is v = αv with α = 1, α > 0, in case of a depth estimated independently of v (say from a Lidar, or a stereo camera), the estimated force needed to brake on time will be F = α 2 F 0 , and the actual distance to stop will be θ α 2 . On the other hand, in the case described section 2.1, where the scale factor is given by the ratio between apparent speed and measured speed, the estimated depth θ will be θ = αθ, the resulting force will be F = αF 0 and the distance to brake will be θ α . The error in navigation strategy, while still present, is less impacted with depth estimated relative to motion than absolute depth.

Appendix B

Proof on under-estimated depth over-representation when optimizing relative error for a validation set

This appendix is first referenced in section 2.4.2. The goal is to prove the proposition 1 Proposition. Given an estimator e responsible for estimations { θ}, in a corresponding validation dataset V = {( θ, θ)}, if the estimator is optimized for that validation set for a mean relative difference error, it will have more under-estimations than over-estimations.

Corollarily, if we have an evenly balanced estimator { θ} such that P( θ < θ|( θ, θ) ∈ V) = 0.5, then we can find α < 1 such that {α θ} is a better estimator according to mean relative difference MRE.

Proof. Let's consider the mean relative error of a set of estimations θ with respect to groundtruth values θ. The error is computed on the set of the estimation/ground truth pairs. V = ( θ, θ) .

MRE({ θ}) = E ( θ,θ)∈V

| θ -θ| θ We consider an estimator { θ} to be optimized if for every possible real function f (which is not dependent to the input of the estimator), the estimator { f ( θ)} has a worse error.

In this proof we assume that the estimator is not perfect so that no value θ is exactly the same as the corresponding θ. ∀( θ, θ) ∈ V, θ = θ, which is reasonable considering continuous values.

Let's then consider a value such that | | < min

For ease of notation we write V -= ( θ, θ), θ < θ and V + = ( θ, θ), θ > θ . These are the sets where the target values are respectively under-estimated and over-estimated. We can already see that

If we then consider the estimator { θ(1 + )}, we should get a worse mean relative error.

Appendix C

Proof on smoothing an inverse depth map

This appendix is first referenced in section 4.4.3. The goal is to prove the proposition 2 Proposition. The inverse depth map ξ = 1 θ of a perfect plane has a null Laplacian and is then stable with respect to smoothing.

Proof. The surface the camera is looking at can be modelized as an affine function of x and y in the world coordinates (0, u x , u y , u z ), u z standing for the optical axis of the camera. z = θ 0 + αx + βy θ 0 , α and β being constant values. Besides, from the pinhole model, for a point (x, y, z), we get the coordinates

As a consequence, we can have the depth θ as a function of image coordinates (u, v)

As a consequence, there exist two constant values γ 1 , γ 2 such that:

This implies that the Laplacian of inverse depth is 0 everywhere, and thus the diffusion operation equivalent to identity.

Then, making the Laplacian of inverse depth 1 θ = ξ converge to 0 is stable with respect to any perfect plane.

Appendix D

A Collection of Proofs on diffusion D.1 Preamble

This appendix is first referenced in section 4.4.4. This is a collection of proofs regarding propositions on diffusion and gradient diffusion. They all use to some extent a form of the multi-dimensional integration by parts theorem: Theorem 1. let Ω ⊂ R n be a regular region, and let n be the unit outward normal to ∂Ω (boundaries of Ω) suppose φ : Ω → R and v : Ω → R n are C 1 functions. Then As a basis for our future demonstrations, we can reformulate the proof from Perona and Malik [PM90]: Theorem 2. let Ω ⊂ R n be a regular region, and let n be the unit outward normal to ∂Ω (boundaries of Ω). Finally, let L s be the function defined on scalar fields defined on Ω as