Matthieu Hog
email: matthieu.hog@technicolor.com

Neus Sabater
email: neus.sabater@technicolor.com

Christine Guillemot
email: christine.guillemot@inria.fr

Dynamic Super-Rays for Efficient Light Field Video Processing

One challenging problem with light field video editing is the dreadful volume of data to process. Image and video processing frequently rely on over-segmentation methods to reduce the computational burden of subsequent editing tasks. In this paper, we present the first approach for video light field over-segmentation called dynamic super-rays, which can be seen as temporally and angularly consistent superpixels. Our algorithm is memory efficient and fast. The results show timing close to editing time by leveraging GPU computational power.

Introduction

A light field is a description of the flow of light from and towards every direction in space. In most cases, it is captured by imaging a region of interest from varying viewpoints, yielding in a highly-redundant multiview scene representation. Because of this, a light field enables advanced processing such as depth estimation, refocusing, parallax shift, or super-resolution. The acquisition of such content is typically done with 3 classes of devices. Plenoptic cameras [START_REF] Lumsdaine | The focused plenoptic camera[END_REF][START_REF] Ng | Light field photography with a hand-held plenoptic camera[END_REF] use a microlens array in front of a camera sensor to re-arrange by direction the light rays coming from the camera main lens. Camera arrays [START_REF] Wilburn | Highspeed videography using a dense camera array[END_REF] are composed of a rig of cameras, often organized on a regular grid. Finally, camera gantries (e.g. Stanford gantry [START_REF] Adams | The (new) Stanford fight field archive website[END_REF]) have a mechanical system to move a single camera along a plane, taking photos at regular intervals. Despite being aimed at different applications and having quite diverse spatial and angular resolutions, the content captured by these devices is often described using a representation called lumigraph [START_REF] Steven J Gortler | The lumigraph[END_REF], which describes all light rays as a set of views taken with a regular baseline on a single plane.

Currently, most of light field content available is static. This is perhaps due to the difficulties of capturing a dynamic light field for either of the aforementioned devices. Specifically, camera gantries are by design unable to record dynamic light fields, and the available plenoptic cameras are either limited to static acquisition (e.g. [START_REF] Lytro | Lytro website[END_REF]) or cannot easily produce a lumigraph (e.g. type 2 plenoptic cameras [START_REF] Raytrix | Raytrix website[END_REF]). Camera arrays require a synchronized camera acquisition system, which is a real technical challenge. However, latest advances in light field video acquisition systems [START_REF] Łukasz D Ąbała | Efficient Multiimage Correspondences for On-line Light Field Video Processing[END_REF][START_REF] Sabater | Dataset and pipeline for multi-view light-field video[END_REF] show that it is possible to capture quite voluminous light fields in real time. The problem of efficiently processing the captured 5D light field content is then twofold. First, the amount of data to handle, already a problem for static light fields, reaches a critical point for video light fields. As a consequence, computational efficiency is a core aspect in many light fields processing tasks, such as editing. Secondly, to enable editing via user interaction on few key frames, algorithms must consistently propagate the edits angularly and temporally to the rest of the dynamic light field.

In this paper we present an over-segmentation scheme that exploits the light field redundancy in both the temporal and angular dimensions and leverages GPU computational power to enable easy and fast light field editing from a single reference view and frame. The proposed over-segmentation approach generalizes the concept of super-rays, introduced in [START_REF] Hog | Super-rays for efficient light field processing[END_REF] for static light fields, to dynamic super-rays for video light fields. Super-rays, seen as the light field analog of superpixels [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF], are a grouping of rays captured in different views of the light field that are similar in appearance and coming from the same scene area.

Several constraints are taken into consideration in the design of the proposed method. First, the approach is parallelizable to take advantage of a GPU implementation. Second, it processes different frames sequentially and on-the-fly, so the memory footprint is reasonable, in contrast to methods operating on sliding windows. This is mandatory as the amount of data per frame is much greater than a conventional 2D video. Finally, super-rays are consistent across views and across frames in the temporal dimension.

Specifically, our contributions are: 1. An end to end approach to create an angularly and temporally consistent light field over-segmentation that we call dynamic super-rays 2. A new update term for creating and deleting superpixels and super-rays specially tailored for dynamic content 3. A new strategy for creating a temporal over-segmentation that is consistent with the scene movement.

Related Work

We focus on the two areas of the literature that are related to the proposed method, namely image and video over-segmentation and light field editing.

Image and Video Over-Segmentation: Since the introduction of the term superpixel [START_REF] Ren | Learning a classification model for segmentation[END_REF], many approaches have been proposed for still image over-segmentation. Because superpixels are often used as a pre-processing step to speed up other algorithms, it is unsurprising to see that superpixel methods yielding low run-time are more popular than historical approaches [START_REF] Levinshtein | Turbopixels: Fast superpixels using geometric flows[END_REF][START_REF] Meyer | Multiscale morphological segmentations based on watershed, flooding, and eikonal pde[END_REF][START_REF] Vedaldi | Quick shift and kernel methods for mode seeking[END_REF]. Among them, we can mention SEEDS [START_REF] Van Den Bergh | Seeds: Superpixels extracted via energy-driven sampling[END_REF]. Starting from a regular lattice, it iteratively reassigns pixel blocks to superpixels based on a color consistency term, in a coarse to fine multi-scale fashion. Simple Linear Iterative Clustering (SLIC) [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] adapts the Loyd's algorithm for k-means clustering of image pixels in terms of spatial and color distance in the Lab space. In order to speed up the algorithm, the clusters are contained in a small local window. Arguably, the strongest advantage of SLIC is its ability to leverage massively parallel computation. Because each assignment and update step can be carried out efficiently on the GPU, to the best of our knowledge, SLIC is the only approach providing real-time performances [START_REF] Yuheng Ren | gSLICr: SLIC superpixels at over 250Hz[END_REF]. Many variations of this algorithm have been proposed [START_REF] Achanta | Superpixels and Polygons using Simple Non-Iterative Clustering[END_REF][START_REF] Li | Superpixel segmentation using linear spectral clustering[END_REF][START_REF] Wang | Structuresensitive superpixels via geodesic distance[END_REF], but all loosing somehow this computational advantage.

The problem of over-segmentation for multiview is a rather unexplored topic. Current approaches focus on generating superpixels on each view and group them afterwards to compute a consistent segmentation, rather than an over-segmentation [START_REF] Neill Df Campbell | Automatic object segmentation from calibrated images[END_REF][START_REF] Mičušík | Multi-view superpixel stereo in urban environments[END_REF]. On the contrary, video over-segmentation is a more researched topic. In [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF], volumetric superpixels are created by treating the temporal and spatial dimensions in the same way. Whereas in [START_REF] Galasso | Video segmentation with superpixels[END_REF], superpixels are computed separately on each frame and later merged to extract object segments. In [START_REF] Chang | A video representation using temporal superpixels[END_REF] a pre-computed dense optical flow is used along with a Gaussian process to update the labeling from a frame to another. The superpixels deletion or creation is done by inference using the same Gaussian process. In [START_REF] Van Den Bergh | Online video seeds for temporal window objectness[END_REF], the framework proposed in [START_REF] Van Den Bergh | Seeds: Superpixels extracted via energy-driven sampling[END_REF] is extended by performing the re-assignments as a new frame arrives. In [START_REF] Reso | Temporally consistent superpixels[END_REF], the most related work to ours, dynamic SLIC superpixels are computed in a sliding window of 20 frames. A dense flow is used to propagate the assignment from a frame to another and several SLIC iterations are run. The centroid color is shared between corresponding superpixels on several frames of the sliding window. The superpixel update criteria is solely based on the superpixel size evolution. Unfortunately, none of the aforementioned approaches are readily applicable for video light field over-segmentation for different reasons. Loading a large amount of the video frames as in [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF][START_REF] Reso | Temporally consistent superpixels[END_REF] is prohibitive in the case of a light field. Performing a late merge of frames superpixels as in [START_REF] Galasso | Video segmentation with superpixels[END_REF] does not provide any temporal consistency. Methods taking the assumption that the temporal dimension is densely sampled, as in [START_REF] Chang | A video representation using temporal superpixels[END_REF], will fail when large objects motion are involved, as in our case. Finally, all the approaches relying on a stack of granular operations as in [START_REF] Van Den Bergh | Online video seeds for temporal window objectness[END_REF], are ill-suited for a GPU implementation.

Light field editing: Because segmentation is the first step of many editing algorithms, it is natural to see most light field editing papers being centered around this topic. A level set method is proposed in [START_REF] Berent | Unsupervised extraction of coherent regions for image based rendering[END_REF] to extract objects layers in the scene. In [START_REF] Wanner | Globally consistent multi-label assignment on the ray space of 4d light fields[END_REF] and [START_REF] Mihara | 4d light field segmentation with spatial and angular consistencies[END_REF], the authors use user scribbles on a reference view to learn a joint color and depth object model that is used to infer a label for each ray of the light field. These label assignments are further regularized inside and between views. In these two approaches, because the regularization does not scale well with the size of the input light field, the running times are rather high. To solve this issue, in [START_REF] Hog | Light Field Segmentation Using a Ray-Based Graph Structure[END_REF] rays coming from the same scene point are represented in a single node on a ray-based graph structure. This reduces significantly the size of the graph and scales well with the number of input images. However, the quality and the run-time depends on the quality of the dense depth estimation on all views, which can be quite expensive to compute. Furthermore, as for conventional 2D images, the approach does not scale with the spatial resolution of the light field.

To overcome this problem, recent work focused on providing a light field over-segmentation. In [START_REF] Hao Zhu | 4d light field superpixel and segmentation[END_REF], a depth estimation is used to propagate and refine an initial SLIC over-segmentation on a reference view to all the views of the light field. In [START_REF] Hog | Super-rays for efficient light field processing[END_REF], a method to compute angularly consistent superpixels, named super-rays is proposed. The approach does not require a dense depth estimation and focuses on speed and parallelism but still provides satisfactory segmentations. This last algorithm has shown to be a good trade-off between speed and accuracy for still light fields, so we consider it as a starting point for our over-segmentation method for light field videos. We summarize it in Sec. 3.

Static super-rays summary

In this section we briefly describe and give the main notations of the super-rays algorithm for static light fields. Further details can be found in [START_REF] Hog | Super-rays for efficient light field processing[END_REF].

Notations: Let r be a ray of the light field LF, and (s, x) its coordinates using the two plane parametrization [START_REF] Steven J Gortler | The lumigraph[END_REF], where s = (s,t) and x = (x, y) are the angular and spatial coordinates respectively. Besides, each ray has an associated CIELab color value Lab r . Let

x := P d s (x) be the spatial pixel position in view s imaging the same scene point, at a distance d, as x in view s. This is, r = (s, x) and r = (s , x ) are corresponding rays imaging the same scene point in different views. Given a light field, super-rays are all the perceptually similar rays corresponding to the same scene area. That is, the mapping A : LF ⊂ Z 4 → Z, such that each ray r is assigned a super-ray label c is computed. Let SR c be the set of rays r such that A(r) = c. The super-ray computation is inspired by SLIC [START_REF] Achanta | Slic superpixels compared to state-of-the-art superpixel methods[END_REF] and has the same main steps. One major difference is that each super-ray SR c is characterized by a centroid ray r c with angular coordinates corresponding to the reference view s c and a depth d c associated to it.

Initialization: The spatial positions x c of the centroid rays are initialized on a regular grid of step S in the reference view. The corresponding CIELab color values on such positions are the initial color values of the centroid rays Lab r c , and the depth d c of each centroid ray r c is estimated via block-matching.

Assignment step: At each iteration, each ray r = (s, x) of the light field is assigned a super-ray label A(r). First, the depth estimated in the previous step is used to compute r c = (s , P d c s (x c )), the corresponding rays of r c . Then, each ray in a neighborhood N S (r c ) of size S around r c is assigned to the super-ray SR c if it minimizes the color and spatial distances:

A(r) = arg min c ∆ Lab (r, r c ) + λ ∆ xy (r, r c ) , (1)
where 2 and λ is the parameter weighting the color and spatial distances.

∆ Lab (r, r c ) = ||Lab r -Lab r c || 2 , ∆ xy (r, r c ) = ||x -P d c s (x c )||
Update step: Following the assignment step, the spatial coordinates of the ray centroid and its corresponding Lab values are updated. The new color value of r c is the average of the color values of all rays in SR c and the new spatial coordinates are the average coordinates of all light rays r = (s, x) in SR c projected on the reference view using the depth d c :

x c = 1 |SR c | ∑ r∈SR c P d c s c (x). (2)
Note that the angular coordinates s c of the centroid rays do not change along the iterations, while the spatial coordinates are updated. Iterations: As in SLIC, the two previous steps are repeated until the centroids are stable. This happens within 10 to 15 iterations. A cleanup step is optionally run to reassign labels to disconnected rays.

Dynamic super-rays

The proposed approach is inspired from techniques proposed to generate temporally consistent superpixels [START_REF] Chang | A video representation using temporal superpixels[END_REF][START_REF] Reso | Temporally consistent superpixels[END_REF], that can be decomposed into three main steps: 1. initialize the current frame segmentation by temporally propagating the segmentation of previous frames 2. adapt the segmentation to changes in geometry 3. create and delete segments to take into account occlusions and objects entering or leaving the scene. Our algorithm is summarized in Alg. 1 in the supplementary material and illustrated in Fig. 1.

Sparse temporal propagation

Computing a dense and accurate optical flow for light fields can be a quite tedious task, especially when memory and time requirements are taken into account. Moreover, because super-rays embed a depth information per segment, the problem we aim to resolve is a scene flow estimation problem. That is, we aim to find the displacements of 3D points in the scene rather than pixels shifts in the image plane. Fortunately, in the case of super-rays, the scene flow estimation needs to be estimated only for centroids and not for all rays of the light field. One way of computing the scene flow (δ x c , δ d c ) is to compute exhaustively a cost function for each possible motion vector (as in Eq. 1 in the supplementary material). However, this 3dimensional cost function being quite expensive to minimize, we have split the problem into optical flow and depth estimation, like other methods for light field scene flow estimation in the literature [START_REF] Basha | Structure and motion from scene registration[END_REF][START_REF] Pratul P Srinivasan | Oriented lightfield windows for scene flow[END_REF]. Now, in state of the art optical flow estimation methods Deep Flow [START_REF] Weinzaepfel | Deep-Flow: Large displacement optical flow with deep matching[END_REF] stands out for its performance in terms of quality and run-time [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF]. Deep flow first searches for sparse matches using Deep Match [START_REF] Revaud | Deepmatching: Hierarchical deformable dense matching[END_REF] between downsampled versions of the input frames, and then the matches are densified by regularizing a selected set of sparse matches. Deep Match has many properties which are interesting for our problem. It is robust and efficient since it can be implemented on GPU [START_REF] Harchaoui | Deep-Matching: Deep Convolutional Matching[END_REF] and the matches are searched in a limited local window. Thus, we solve the sparse flow estimation using deep matches. In contrast to Deep Flow, we do not seek to obtain a dense and precise optical flow, but rather a robust and fast sparse flow for each centroid.

We compute the set of deep matches [START_REF] Revaud | Deepmatching: Hierarchical deformable dense matching[END_REF] from two downsampled frames f and f + 1. Then, the estimated flow δ x m = x f +1 mx f m , using the deep matches in the full resolution coordinate system, is used to compute the flow of each centroid δ x c using a simple and fast bilinear interpolation. Precisely, δ x c is the distance-weighted average flow δ x m of its 4 nearest matches. Using the notation above, the depth is updated using the same strategy as in [START_REF] Hog | Super-rays for efficient light field processing[END_REF]:

δ d c = arg min δ ∈D min o∈Ω ∑ s o(s ) ∆ B RGB (r f +1 c , r f +1 c ) , (3)
where D is the small range of potential depth movements and Ω is a family of spatio-angular patches.

Centroid creation and deletion

Because of object movements in the scene, the super-ray topology can change in time. For instance, parts of the super-rays can be occluded or disoccluded, or completely appear or disappear due to objects entering or leaving the scene. For this reason, creating and deleting super-rays might be necessary. While the superpixel size or color consistency has been used to determine the creation or deletion in other research works, we propose to leverage the depth information associated to the super-ray to detect occlusions and disocclusions.

In particular, a new super-ray is candidate to be created at the midpoint of two superrays when their centroid distance exceeds a given threshold S * τ. Conversely, a super-ray will be a candidate to be deleted if two super-rays are too close from each other, i.e. their centroid distance is lower than a threshold S/τ. In particular, the occluded super-ray (with the smallest disparity or biggest depth) is the candidate for deletion. For the sake of efficiency, and to avoid duplicates, we search the candidate centroids to be deleted or created in a 4nearest neighborhood, computed as illustrated in Fig. 1(b). Specifically, the approximate neighborhood of a centroid c is defined as N (c) = {c le f t , c right , c up , c down } where

c le f t = arg min ĉ |y ĉ -y c | s.t. x ĉ < x c , |y ĉ -y c | < S , (4)
and similarly for the other neighbor centroids. Now, in order to maintain the number of super-rays constant, we create the same number of super-rays we delete. If the number of candidates for deletion is smaller (resp. bigger) than the number of candidates for creation, only the centroids with the biggest (resp. smallest) centroid distance are created (resp. deleted). Finally, because objects can move inside or outside of the reference view, the super-rays near the image borders are treated as follows. New super-rays are created in the reference view between the image borders and the closest centroids. For instance, if a centroid c does not have a neighbor c right , a new centroid will be ( x c +M 2 , y c ), M being the reference view width. Super-rays that leave the reference view image plane are automatically deleted.

Note that the centroid neighborhood can be used for further processing, as it is a convenient way of representing the super-rays structure.

New frame over-segmentation

In the new frame, after defining the set of centroids in the reference view, all the rays of the light field are assigned to a centroid. Similarly to super-rays, the assignment is done using Eq. 1 in an iterative process with color and position centroid updates. While the centroid color is updated with the same color average strategy as super-rays, the centroid position update changes for dynamic super-rays. So Eq. 2 becomes

x f +1 c = p |SR f +1 c | ∑ r∈SR f +1 c P d c s c (x f r ) + (1 -p)(x f c + δ x c ), (5)
where p is a parameter controlling how much the super-rays are allowed to move from their theoretical position. When p = 1, this step corresponds to the same SLIC iteration as in Eq. 2, and when p = 0, the super-ray centroids are not updated at all, providing the best temporal consistency. Newly created centroids (as described in Sec. 4.2) are updated using p = 1, allowing them to adapt to scene changes.

In [START_REF] Reso | Temporally consistent superpixels[END_REF], 5 SLIC iterations are run, where the centroids are allowed to move freely. As a consequence, superpixels of static objects tend to move since they are affected by the creation, deletion and movements of nearby superpixels. On the contrary, our dynamic superrays movement is congruous with the objects movement in the scene, providing a more consistent temporal over-segmentation.

Experiments

As an effort to quantitatively assess the efficiency of the proposed approach, we use the Monka dataset proposed in [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF], composed of ground truth images, disparity, optical flow and object labels for pairs of cameras. This can arguably be considered as a small light field, but to the best of our knowledge such dataset does not exists for more than 2 views. We use the standard superpixel quality metrics [START_REF] Neubert | Superpixel benchmark and comparison[END_REF] (ASA ,BR, UE) along with a Temporal Consistency (TC) measure, which is the analog of the angular consistency in [START_REF] Hog | Super-rays for efficient light field processing[END_REF] in the temporal dimension.

We compare our approach with statics super-rays [START_REF] Hog | Super-rays for efficient light field processing[END_REF] computed on each frame and put into correspondence using the ground truth optical flow such that the TC is maximized. In average, we find the dynamic super-ray to be slightly better for classical over-segmentation metrics but not significantly so. This can be explained by the fact that at each new frame, the previous over-segmentation is re-used, meaning that each frame iteration is refining the previous segmentation. However, we see that super-rays computed separately give worst temporal constancy, despite being advantaged by the fact that super-pixel matching is perfect. Detailed results for all the sequences can be found in the supplementary material.

Currently, two datasets for video light fields captured with camera arrays are available: 1. The Fraunhofer dataset [START_REF] Łukasz D Ąbała | Efficient Multiimage Correspondences for On-line Light Field Video Processing[END_REF], with full-HD sequences of 9 to 16 views and between 150 and 400 frames; 2. the Technicolor dataset [START_REF] Sabater | Dataset and pipeline for multi-view light-field video[END_REF] with 2K sequences of 16 pseudo-rectified views and between 300 and 390 frames. The cameras baseline is quite important, especially for the second dataset.

We use a fixed set of hyper parameters, that we did not fine tune for each sequence. The values are described in the supplementary material.

Fig. 2 shows the output of our algorithm for a small area of the dataset Birthday [START_REF] Sabater | Dataset and pipeline for multi-view light-field video[END_REF]. For the sake of visualization, results are only shown for two views, the reference view s c = (1, 1) and another view s = (1, 0), and three non-consecutive frames f = 95, 100, 105. For each view, we show the input image with the optical flow only on the reference view (top left), the color-coded assignment (top right), the super-ray average color (bottom left) and finally the super-ray contours (bottom right). [START_REF] Reso | Temporally consistent superpixels[END_REF] on the reference view over 5 frames. Our dynamic super-rays (second row) are consistent with the scene movement while the superpixels in [START_REF] Reso | Temporally consistent superpixels[END_REF] (third row) move in static regions.

Please note that it is hard to evaluate our over-segmentation results on paper due to the reduced number of frames or views we can illustrate compared with the full light-field videos. We strongly encourage the reader to visualize all our results on our web-page 1 .

The videos show concatenated views with usual visualization methods, namely, the super-ray average color, the color-coded super-ray labels, and the super-ray contours (as in Fig. 2). We also visualize the value of the flow for each centroid, as well as the coarse depth of each super-ray (each ray has assigned its super-ray depth) and finally, the centroids neighborhood structure (as in Fig. 3).

We compare our method with the algorithm in [START_REF] Reso | Temporally consistent superpixels[END_REF] which is the state of the art for temporal consistent superpixels on videos. Note that in this experiment we focus on the temporal aspect since it has already been shown [START_REF] Hog | Super-rays for efficient light field processing[END_REF] that computing superpixels on each of the views separately does not guarantee angular consistency. Fig. 3 shows this comparison on five frames, f = 260, 262, 263, 264, 265, of the central view. In particular, on the top row, we show the neighborhood structure described in Sec. 4.2. Each centroid appears as a blue dot, and horizontal and vertical neighborhoods are illustrated with cyan and magenta edges respectively. Centroids of deleted super-rays are represented in red, while new super-rays are represented in yellow. The second and third rows correspond to our results and the results of [START_REF] Reso | Temporally consistent superpixels[END_REF] respectively. The input view and the color segments are blended on theses images. We observe that the update step in [START_REF] Reso | Temporally consistent superpixels[END_REF] allows the superpixels on the static background to move freely. On the contrary, our super-rays are not moving so the scene movement is consistent with the super-rays movement. We believe this is a major benefit if dynamic super-rays are to be used in further editing tasks. We invite the reader to view the videos in the supplementary material or on our website, where temporal consistency is more visible.

Besides the qualitative comparison with [START_REF] Reso | Temporally consistent superpixels[END_REF] we have also observed considerable differences in terms of computational complexity. Depending on the datasets, the algorithm in [START_REF] Reso | Temporally consistent superpixels[END_REF] takes several hours and up to one day (using the original implementation) to run for all the frames of a single view video. In our case, the biggest advantage is the GPU friendliness. Indeed, the SLIC-based iterations, the deep flow computation and the super-ray creation and deletion, are highly parallelizable. On the same machine (equipped with a Nvidia GTX 1080 GPU hosted by an Intel Xeon E5-2630 CPU) our current Python/PyOpencl implementation gives an average running time for each iteration of 0.157s and 0.059 to 0.083s (depending on the input size), respectively on [START_REF] Sabater | Dataset and pipeline for multi-view light-field video[END_REF] and [START_REF] Łukasz D Ąbała | Efficient Multiimage Correspondences for On-line Light Field Video Processing[END_REF]. Further improvements are to be expected by a more optimized implementation.

Dynamic super-rays with the neighborhood structure presented in Sec. 4.3 offer a useful representation of the scene captured by the light field videos. Temporal super-rays can be seen as a powerful tool for efficient light-field video editing in which the edits in one reference view of the light-field can be easily propagated to other frames and views. For example, in [START_REF] Hog | Super-rays for efficient light field processing[END_REF] it is presented how to use super-rays to generate intermediate views to correct the angular aliasing caused by the poor angular sampling of sparse light fields. Similarly, dynamic super-rays can be used for temporal image interpolation without flickering caused by an inconsistent interpolation. Other examples of temporal super-rays applications include light-field video compression (e.g. adapting the approach in [START_REF] Fracastoro | Superpixeldriven graph transform for image compression[END_REF]), or light field color transfer (e.g. using the algorithm in [START_REF] Giraud | Superpixel-based color transfer[END_REF]).

Limitations: We have observed that our approach has some limitations, in particular, when the depth or the flow estimation becomes erroneous, the super-ray consistency is not guaranteed from one view to another. Such failure case is visible in the dataset Newspeaker [START_REF] Łukasz D Ąbała | Efficient Multiimage Correspondences for On-line Light Field Video Processing[END_REF], where a very uniform green background challenges both the depth and flow estimations. When the depth is inconsistent, the centroids are wrongly projected, leading to large areas with no nearby centroid for the rays to be assigned to. The other failure case involves small moving objects, because of our sparse flow computation strategy, the optical flow for small object can be wrongly evaluated to the flow value of its surrounding. This is visible on the dataset Train [START_REF] Sabater | Dataset and pipeline for multi-view light-field video[END_REF], where centroids struggle to follow the train wagons.

In conclusion, even if depth and flow estimation are mature research topics we have observed that challenging datasets may still produce inaccurate estimates. In particular, the images in the two datasets suffer heavily from motion blur, noise and over and under expo-sition. Furthermore, the dataset in [START_REF] Łukasz D Ąbała | Efficient Multiimage Correspondences for On-line Light Field Video Processing[END_REF] has some large texture-less areas. However, loosing consistency in flat areas is not critical. Indeed, if the zone to edit is totally uniform, the editing become trivial (e.g. using a simple color threshold), dismissing the needs for super-rays in the first place.

Conclusion

We presented an approach to generate angularly and temporally consistent light field video over-segmentation. Our algorithm design enabless a GPU implementation, allowing computational performances that are required to cope with the high volume of data. To the best of our knowledge, this is the first approach to deal with the problem of video light field editing.

Figure 1 :

 1 Figure1: Proposed dynamic super-ray scheme. Each super-ray is represented by a solid color and its centroid by a black dot. (a): Illustration of our algorithm in a simple case. The red foreground super-ray is tracked over the consecutive frames of a 2 × 1 light field. Other super-rays do not move since the background is static. The depth d is used to enforce angular consistency from a view to another, while the scene flow (δ x , δ d ) guarantees temporal consistency. On the third frame, the moving red super-ray becomes too close of the pink super-ray and too far from the green one, triggering the creation of the orange super-ray and the deletion of the pink one on the next frame. (b): Super-ray neighborhood. The search area for the left neighbor c le f t of the red super-ray c is represented by the blue dotting, and the final neighborhood connections of c by the black lines.

FramesViewsFigure 2 :

 2 Figure 2: Video super-rays for 3 frames and 2 views of the dataset Birthday [31].

Frames

Figure 3 :

 3 Figure 3: Over-segmentation comparison with[START_REF] Reso | Temporally consistent superpixels[END_REF] on the reference view over 5 frames. Our dynamic super-rays (second row) are consistent with the scene movement while the superpixels in[START_REF] Reso | Temporally consistent superpixels[END_REF] (third row) move in static regions.

Project website : https://www.irisa.fr/temics/demos/DynamicSuperrays/ index.html. Alternatively the videos can be viewed on YouTube : https://www.youtube. com/channel/UCHFkXPUSiV3UFxlABRmQkNA/videos


Frédéric Charillon

Matthias Dembinski

Hans-Georg Ehrhart

Cathérine Hoeffler

Stephanie C Hofmann

Markus Kaim

Ronja Kempin

Barbara Kunz

Manuel Lafont Rapnouil

Aline Leboeuf

Nina Leonhard

How can France and Germany contribute to reaching the goal of European strategic autonomy? This key question has been guiding the work with the present report. In the light of a more demanding security environment, but also a rare momentum for further European integration, Berlin and Paris have to take their security and defence cooperation to the next level, bilaterally as well as in the EU.

Progress in Franco-German security and defence cooperation is a key step towards strategic autonomy with its three dimensions: political, operational and industrial autonomy. Political autonomy has so far been neglected in Franco-German security and defence cooperation. Past cooperation was almost exclusively linked to the operational, sometimes industrial levels. The new impetus needs to be embedded in a farther reaching political dialogue. This can best be achieved by elaborating a Franco-German White Paper on Security and Defence. At the operational level, the primary objective for Paris and Berlin is to enable Europeans to autonomously carry out the Petersburg tasks, as defined in article 43(1) of the Treaty of Lisbon. Nevertheless, the type of capabilities needed must be derived from a political analysis of the strategic environment and the threats posed. The same accounts for industrial autonomy where France and Germany should work towards closer cooperation and shared rules embedded in a political dialogue.

With seven concrete recommendations, the paper aims to contribute to establishing strategic autonomy, starting at the bilateral level to provide leadership for an ambitious European project.

Introduction

The year 2017 -as turbulent as it was -saw the Franco-German couple rise anew. Although the two countries held national elections, Berlin and Paris left no doubt about their wish to re-boost their bilateral relationship and their determination to play a leading role in defining the future of the European project "under threat". 1 Security and defence, an area of politics where France and Germany certainly differ, became one of the main axes of their agenda.

Three reasons convinced Paris and Berlin to revitalise their bilateral relationship in the area of security and defence politics and to lead the European Union's (EU) member states' action to implement the "Global Strategy for the European Union's Foreign and Security Policy", in which the Union's strategic autonomy is the leitmotiv:

First, a mixture of traditional and non-traditional threats cumulated against a bleak global background. Security challenges rose outside the European Union, e.g. the Russian Federation violating the European security order or instability in North Africa, the Sahel and the Middle East. Numerous countries within Europe were shaken by terrorist attacks. The influx of over one million refugees challenged the present state of border control in the EU as well as the solidarity between member states all while contributing to the rise of populism throughout Europe.

Second, the United Kingdom will leave the European Union by 2019, a decision made irreversible when Prime Minister Theresa May triggered Article 50 of the Treaty of Lisbon on 29 March 2017. As a result, member states realised that there was room for quid-pro-quos on security and defence. This allows moving past the traditional and unproductive opposition between a southern European vision on defence policies versus an eastern European one, with largely different priorities when it comes to threat perception and consequently different priorities regarding the institutions that form the European security architecture. Such differences will remain -but in times when EU-NATO cooperation is making considerable progress, a dichotomous view on "EU versus NATO" makes less sense than ever. At their first meeting after the Brexit referendum, the EU's Heads of State and Government declared that "[w]e need the EU not only to guarantee peace and democracy but also the security of our people." 2 The European public's expectations and needs and the willingness of EU member states to follow that path created a momentum for European integration in the field of security and defence.

Third, the election of Donald Trump as the 45 th President of the United States of America has accelerated this process, as it has put the transatlantic security community under strain. President Trump's inconsistent -at times strongly sceptical -takes on NATO and Europe led to strategic uncertainty about the future and the credibility of U.S. engagement in Europe. With the U.S. more and more unpredictable, the EU and its member states need to be more autonomous in international relations, including security and defence.

France and Germany reacted jointly to these substantial political and strategic challenges. On 13 July 2017, the two countries presented an ambitious bilateral cooperation agenda. 3 Paris and Berlin are to work together on a series of defence projects. These include the replacement of their current fleets of fighter planes, the development of a joint indirect fire artillery system and cooperation around a new major ground combat system. Both countries also pledged to push ahead with the Eurodrone programme, which is expected to produce Europe's first fleet of military drones by 2020, and agreed on cooperation in the field of cyber security. Timetables and working groups shall ascertain this cooperation's success, which is perceived as a profound revolution on both sides of the Rhine. In the words of French President Emmanuel Macron, the aim of the joint fighter jet project is "to do research and development together […], to use it together […] and to coordinate on exports." Moreover, Berlin and Paris plan to advance their operational cooperation in the Sahel and within the context of NATO's Enhanced Forward Presence deployment in the Baltic States, as well as the creation of a common operational culture, based on deepened cooperation between the two countries' air forces. Last, but not least, France and Germany plan for better information-sharing in the fight against terrorism and the strengthening of EU directives in the area of internal security. Once these bilateral projects are sufficiently developed, they will be 2. The Bratislava Declaration, Bratislava, 16 September 2016, p. 1, available at: www.consilium.europa.eu. 3. Presse-und Informationsamt der Bundesregierung, Gemeinsame Erklärung zum Deutsch-Französischen Ministerrat, 13 July 2017. The conclusions of the German-French Defense and Security Council can be found at the end of the document available at: www.bundesregierung.de. opened up to other EU countries willing and able to further integrate in this policy field and to contribute to an ever more autonomous EU.

At the European level and since the publication of the EU Global Strategy in June 2016, France and Germany have been able to inject unforeseen dynamism into the EU's security and defence policy -a field hitherto largely unaffected by moves towards European integration and communitisation. By November 2016, Berlin and Paris had convinced their EU partners of the necessity for a comprehensive reform agenda for the Union's Common Security and Defence Policy (CSDP), including extending common funding for EU Battlegroups beyond the end of the year and intensifying cooperation between the EU and NATO. In June 2017, EU member states also agreed to establish a Military Planning and Conduct Capability (MPCC) for strategic command of (so far) non-executive CSDP missions, such as EU training missions. Otherwise, the focus of the Franco-German EU agenda concentrates on defence, in particular through three major projects aimed at strengthening member states' ability to intervene and to defend themselves. First of all, Permanent Structured Co-operation (PESCO) is to be activated. 4 Introduced under Article 42 (6) of the Treaty of Lisbon, PESCO allows for member states "whose military capabilities fulfil higher criteria and which have made more binding commitments to one another in this area with a view to the most demanding missions" to cooperate more closely than the EU 27 context permits. 5 Under the terms of Article 46, PESCO is open to all member states. Secondly, a Coordinated Annual Review on Defence (CARD) is meant to institutionalise a systematic exchange between member states under the auspices of the European Defence Agency (EDA). This should contribute to identifying and closing gaps in member states' military and civilian resources. Thirdly, the European Commission and the member states decided to establish a European Defence Fund (EDF) to incentivise cooperation on key defence capabilities acquisitions. The EDF will co-finance initiatives where at least three EU states join forces to develop and procure defence products and 4. The possibility for certain EU countries to strengthen their cooperation in military matters by creating permanent structured cooperation (PESCO) has been introduced by the Lisbon Treaty (Articles 42(6) and 46 of the Treaty on European Union (TEU)). In order to do this, interested countries must fulfil two main conditions provided in Protocol No 10 annexed to the treaty: They must intensively develop defence capacities through the development of national contributions and their participation in multinational forces, in the main European equipment programmes and in the activities of the European Defence Agency in the field of defence capabilities development, research, acquisition and armaments. Secondly, they must have the capacity by 2010 to supply combat units and support logistics for the tasks referred to in Article 43 TEU within a period of 5 to 30 days and, depending on needs, for a period of 30 to 120 days. 5. Denmark negotiated an "opt-out" from CSDP. Post-Brexit defence cooperation in the EU will consequently take place at 26, not 27.

technologies. This should allow states and companies to operate more costeffectively as benefits and funding will be strongest with initiatives falling under cooperation within the PESCO framework.

We stand behind an ambitions implementation of all these initiatives. We are convinced that the Franco-German relationship has to be the nucleus of any European defence integration. What was true in the past holds true in the post-Brexit EU 26: a truly joint Franco-German approach in the area of security and defence policy can be the starting point of a European project. Inversely, only a strong, committed and courageous bilateral relationship will enable the EU and Europe to confront the present threats and challenges. With the end of the election cycle, our two countries have four years to lead the process of defining guidelines for Europe's strategic autonomy.

However, past "revivals" of defence cooperation have often stayed with declarations or proved purely symbolic. For instance, the idea of a Coordinated Annual Review on Defence has been around for longer than PESCO. CARD's precursors are the Capability Development Mechanism introduced in 1999, and its expanded successor of 2001, the European Capability Action Plan. Those two initiatives fell by the way-side, largely because member states balked at revealing gaps in their national defence capacities. Member states therefore need to be offered incentives for transparency in defence planning, to acknowledge a key role for institutions like the EDA, and to follow-up on the many commitments made in the context of PESCO. If France and Germany want to unfold a cooperation dynamic that is up to the needs of our time and will lead to a deeper form of integration, the two will have to take their cooperation to the next leveland to make their respective security and defence policy paradigms evolve.

At the same time, it is crucial to bear in mind that reaching congruity of interests and approaches is an endeavour that has failed in the past. Seeking "sameness" will not work this time either. It is when looking for complementarities and compatibilities that France and Germany can join forces in the most beneficial way, combining their respective strengths. For instance, even though their current roles in the Sahel differ significantly, both French and German forces are central in addressing the many security challenges in the region.

Listed below are seven recommendations that -to our mind -are particularly suitable to move beyond the present state of affairs and to reach a new level of Franco-German entente.

Political Autonomy: A Key Aspect of Strategic Autonomy

The pre-condition for any successful Franco-German cooperationbilateral, European or within the framework of the EU's CSDP -is a common strategic understanding. French President Macron was right calling for "a common strategic culture" in his Sorbonne speech. 6 To our mind, political autonomy is one of three indispensable building blocks of the notion currently at the heart of security and defence policy debates in Brussels and in EU capitals: strategic autonomy. Any actor's strategic autonomy is based on political, operational and industrial autonomy. Operational autonomy is defined as the capacity, based on the necessary institutional framework and the required capabilities, to independently plan for and conduct civilian and/or military operations. Industrial autonomy means the ability to develop and build the capabilities required to attain operational autonomy. These two necessary elements of strategic autonomy have to be complemented by the capacity to define foreign and security policy goals and to decide over the tools to be used in their pursuit -political autonomy.

6. "Initiative pour l'Europe", Discours d'Emmanuel Macron pour une Europe souveraine, unie, démocratique, Paris, Université de la Sorbonne, 26 Septembre 2017, available at: www.elysee.fr.

The three dimensions of strategic autonomy

Political autonomy

The capacity to take security policy decisions and act upon them.

Operational autonomy

The capacity, based on the necessary institutional framework and the required capabilities, to independently plan for and conduct civilian and/or military operations.

Industrial autonomy

The capacity to develop and build the capabilities required to attain operational autonomy.

These different types of autonomy are mutually dependent. Proposals aimed at establishing institutions, acquiring capabilities or funding research and development cannot, therefore, fill the lacuna existing at a political level. Attaining to strategic autonomy requires a thorough analysis of the strategic environment as well as a definition of urgent matters and prioritisation. If several actors want to attain strategic autonomy together, their analyses and prioritisations must be compatible.

A permanent strategic dialogue is needed

When looking back at the track record of Franco-German cooperation in the field of security and defence, one thing is obvious: cooperation and joint proposals are almost exclusively linked to the capabilities, sometimes industrial levels. This holds also true for the latest Franco-German proposals. All the projects and ideas that have been presented since September 2016 -provided that they are really implemented -are important steps forward. However, they need to be embedded in a farther reaching political dialogue on the ways in which they may serve the overall objective of strategic autonomy. Berlin and Paris will only be able to serve that goal if they bring their national security and defence reflexions and analysis together.

Plea for a Franco-German White Paper on Security and Defence

This goal can best be reached through a "Franco-German White Paper on Security and Defence". The 55 th anniversary of the Élysée Treaty, in January 2018, would be an ideal opportunity for the Heads of State and Government of our two countries to launch such work. The document, which should be produced throughout the year, should contain a shared analysis of the current security environment and the risks and threats it presents. From this analysis, regional and functional priorities should be derived. Based on these priorities, a joint level of ambition has to be defined.

A strategic, political dialogue between Paris and Berlin in the true meaning of the term is no easy task. Although postulated by the 1963 Élysée Treaty, it still is neglected in Franco-German security and defence cooperation. 7 Foreign, security and more importantly defence policy 7. Interestingly, the French and German versions of the text differ slightly, already hinting at differences in strategic culture: while the French version uses the term "doctrine" ("rapprocher traditions are largely different in the two countries. Paris' and Berlin's views on their own positions on the international stage are seldom on par. Ideas on the role of the military and the use of force differ, as well as societies' perceptions of their armed forces. Security and defence policy priorities do not necessarily go together either, be it in terms of threat perception, regional priorities or readiness to resort to military action.

These differences in French and German strategic cultures in fact imply different approaches to the notion of strategic autonomy, its content and its implications. In France, strategic autonomy is understood and officially defined as the French state's ability to decide and to act freely in an interdependent world. This allows preserving French independence and sovereignty, while strengthening the partnerships that contribute to this independence. For France, this freedom of action makes the country a more significant contributor to collective action. The framework for this can be international organisations, like the UN, NATO or the EU, or bilateral partnerships, e.g. with the United Kingdom and other countries. The partnership with Germany in particular is seen as crucial, especially since President Macron's election. The French understanding of strategic autonomy does, of course, not only have political implications. It also has ramifications at the levels of operational and industrial autonomy which translate into national defence and military policies. What remains key to French autonomy is the country's capacity to lead operations on its own and to retain key capabilities allowing it to preserve a major influence on operations led with allies. The German government, in turn, seems to carefully avoid the term "strategic autonomy" in its strategic documents so far. All these differences notwithstanding, both Paris and Berlin committed to the goal of European strategic autonomy with the adoption of the European Union Global Strategy. 8 leurs doctrines en vue d'aboutir à des conceptions communes"), the German text uses the much weaker "Auffassungen", i.e. "views" or "perceptions" ("ihre Auffassungen einander anzunähern, um zu gemeinsamen Konzeptionen zu gelangen"). See the French and German versions at www.franceallemagne.fr. 8. In the EUGS, the idea of strategic autonomy is defined by a rather comprehensive, wide-ranging understanding of the term: "An appropriate level of ambition and strategic autonomy is important for Europe's ability to promote peace and security within and beyond its borders." The Global Strategy does, however, not offer any concise definition of the term, nor does it offer provisions on how to operationalise it. See Shared Vision, Common Action: A Global Strategy for the European Union's Foreign and Security Policy, June 2016, p. 9, available at: eeas.europa.eu.

Revitalising the Franco-German Defence and Security Council

In this context, transposing the various national approaches regarding strategic autonomy to the European level will be no easy task. For France and Germany to be able to take the lead in working towards this objective, reaching a common understanding at the bilateral level first is indispensable: Paris and Berlin need to agree on both the content and the implications of political autonomy as the sine-qua-non dimension of strategic autonomy. Therefore, the Franco-German dialogue on security and defence must be made permanent, fulfilling the provisions contained in the 1963 Élysée Treaty. The Franco-German Defence and Security Council should thus meet twice a year. Once a year, security affairs should be discussed at a summit with the French President, the German Chancellor, as well as both ministers of defence and foreign affairs. Furthermore, the Franco-German Defence and Security Council Secretariat should serve as a hub and facilitate coordination among parliamentarians (regular meeting of defence commissions), high-ranking military and diplomatic representatives as well as meetings between officials and scholars in track 1.5 settings.

In any case, Franco-German cooperation has to be seen as a starting point for an inclusive European project. The classic mechanism of European integration could function as always: States delegate parts of their sovereignty in order to regain their autonomy. Fundamental issues will need to be tackled, touching upon most basic convictions on the reasons behind and the objectives for European security cooperation. But as different as these national approaches may be, we are convinced that they are not incompatible because European security is indivisible. It is for this reason we make the following recommendations, taking Franco-German cooperation from a tactical to a truly strategic level:

Recommendation 1

Creation of a joint Franco German White Paper by 2018 in order to find a common definition for strategic autonomy without dodging political autonomy.

To be effective, the White Paper needs to contain a shared analysis of the security environment and the risks and threats it presents, including regional and functional priorities derived thereof. In the light of the above, Berlin and Paris must define a level of ambition and what exactly they want to do together, avoiding merely symbolic measures.

Recommendation 2

Make this exchange permanent beyond the above-mentioned

White paper, based on fulfilling the provisions contained in the 1963 Élysée Treaty.

The Franco-German Defence and Security Council should meet twice a year. Once a year, security affairs should be discussed at a summit with the French President, the German Chancellor, as well as both ministers of defence and foreign affairs. Furthermore, the Franco-German Defence and Security Council Secretariat should serve as a hub and facilitator among parliamentarians, military as well as diplomatic representatives and as a place for meetings between scholars and officials under a track 1,5 format.

Operational Autonomy

Operational autonomy refers to an actor's ability to use the institutional framework to assess and plan for, and more importantly, to conduct a variety of possible civilian-/military operations without major contributions of others. It also refers to the actor's having the necessary capabilities at its disposal. The latter is the dimension of European strategic autonomy that has received the most attention in the past, e.g. through the formulation of so-called headline goals. At the same time, however, operational autonomy for the European Union has not been achieved.

Increasing and directly contributing to European operational autonomy should be the primary objective for Franco-German cooperation in the field of capabilities. The building blocks of such operational autonomy are command and control assets as well as modern and deployable military and civilian capabilities. These enable Europeans to counter threats and to act in conflict prevention, crisis response, conflict management, conflict resolution, and peace building. The type of capabilities needed must be derived from a political analysis of the strategic environment and the threats posed. In other words, operational autonomy builds upon political autonomy.

In order to obtain such capabilities, several prerequisites need to be fulfilled. First, multilateral defence policies -regardless of whether they take place within the UN, the EU, NATO or other contexts -invariably rely on national capabilities under national control. Individual states must therefore invest in their own capabilities, based on appropriate budgets for security and defence politics. Second, national defence planning must be compatible with the multilateral framework's defence objectives. This is the purpose of mechanisms such as the NATO Defence Planning Process (NDPP) and the EU Coordinated Annual Review on Defence. Ideally, in the light of the single set of forces, these various "tracks" would be compatible in terms of scenarios and ambitions and accordingly of capabilities required -especially at a time when the pendulum has swung back to collective defence in large parts of Europe. Third, as the supreme discipline of defence cooperation, states can decide to cooperate in supplying such capabilities, e.g. through multinational units or create multinational assets -an approach labelled Pooling & Sharing in the context of the EU, Smart Defence in the context of NATO. The launching of PESCO offers new opportunities in this context. Needless to say, obstacles are manifold in that context, not least pertaining to decision-making on the assets' use.

Defence spending as the prerequisite for capabilities

Operational autonomy is directly contingent on defence spending. Europe, although collectively the world's second largest military spender, is far from being the world's second largest military power. This is due to both collective and individual inefficiency in defence spending, contributing in particular to a lack of interoperability: Twenty-seven, soon twenty-six, national armies do simply not add up to one European army. Europeans are not good at taking advantage of economies of scales and other benefits of cooperation. Concerning procurement for instance, 84% of all equipment procurement took place at national levels in 2013. 9 Furthermore, defence equipment procurement and Research and Development (R&D) investment has dropped from above the 20% benchmark in 2006 to 17,9% of total defence expenditure in 2014; defence R&D expenditure has even shrunken to 1,0% of total defence expenditure in 2014, whereas member states had originally convened on a benchmark of 2%. There's also a huge gap between the benchmark of 20% and the 8,6% actual spending on collaborative defence R&D reported for 2014. Last, but not least, the share of R&D in total defence spending shrunk as well: from peaking at 1,3% in 2006, it fell to its lowest of 1,0% in 2014. 10 In addition to these deficits, Europe suffers from nondeployable assets, a lack of training and problems related to hardware maintenance. This is all the more worrisome at a time when European armed forces are deployed in a multitude of theatres covering a wider spectrum than ever.

Since 2014, Europeans have realised that more sustainable defence policies are necessary. In both France and Germany, defence budgets are on the rise. In the light of growing security threats and challenges, this is a welcome development. However, the short-sighted debate on the 2% objective must be overcome as it is misleading. How an increased budget is allocated is more important than its sheer volume. France and Germany should concretely define the 2% standard and base it on qualitative criteria. Therefore, France and Germany should initiate a bilateral discussion, to be opened to the other EU and NATO member states in the short-term, about the apportionment of defence budgets.

In addition to national spending, the creation of the European Defence Fund, and in particular its acquisitions window, has the potential to help foster Europe's operational autonomy. As the two largest member states in a post-Brexit EU, France and Germany should therefore contribute the lion's share to the Fund, thus inciting other member states to take part in research and development activities leading to joint military capabilities. In the medium term, the EDF's budget has to be sustained. From 2020 onwards, the European Commission foresees a total annual investment in defence research and capability development of 5,5 billion Euros. Especially with Brexit in view, which will lead to the loss of a major contributor to the EU budget, this goal is ambitious. France and Germany should support the Commission's effort. The two countries should use the negotiations of the next Multiannual Financial Framework (MFF) to firmly establish the EDF in the EU's budget. In this context, the EU Commission should ensure that the EDF prioritises funding for capabilities required to achieve the goals of the EU Global Strategy, in a manner consistent with the gaps identified by the EDA. The definition of binding criteria for the use of EDF co-funding is thus key, and should be supported by Berlin and Paris. Very importantly for the EDF to represent true added value, it must not be filled at the expense of national defence budgets.

Military assets for collective defence and crisis management

Beyond strengthening their respective national military assets, France and Germany can contribute to European operational autonomy both through pushing for capability development and procurement at EU level and through engaging in "bottom-up" bilateral cooperation, open to others. Success stories such as the TIGER School, where French and German, but also Spanish helicopter crews are trained, or the European Air Transport Command fit into that pattern.

At the EU level, the idea of an EU Operational Headquarter (OHQ) has played a key role in European debates. In 2003 already, France and Germany, together with Belgium, the Netherlands and later on Poland, supported the creation of such an OHQ -without being able to convince the United Kingdom. As a consequence, the EU had to rely on ad-hoc and temporary headquarters to conduct its operations, recurring to national assets in the larger member states. During the European Council of 6 March 2017, the Heads of State and Government of the 27 CSDP EU member states finally agreed on the development of a permanent Military Planning and Conduct Capability for non-executive missions. The MPCC should be seen as a first meaningful step towards a future EU permanent OHQ. In order to demonstrate their willingness to provide the EU with a full-fledged permanent OHQ, France and Germany should delegate several high-ranking officers each to show their good will in this project and make concrete proposals on how to organise the MPCC, make it effective and design a roadmap to -in the medium term -transform the MPCC into the desired OHQ.

At the bilateral level, France and Germany have sought to deepen their military cooperation since the 1980s. With the set-up of the Franco-German Brigade (FGB) or the creation of the Franco-German TIGER School, the two countries did, at the time, pioneering work in the integration of armed forces. However, the enthusiasm Franco-German military co-operation may elicit notwithstanding, the long-lasting results of these endeavours remain ambivalent at best.

In order to further deepen their cooperation, France and Germany will have to overcome the shortcomings of existing bilateral projects. These shortcomings are essentially due to differences in strategic cultures and outlooks, which are unlikely to disappear. Another reason for the difficulties encountered in practice is that Franco-German military co-operation so far has notably been based on the idea of two equal partners. On the one hand, this idea of equality has provided the very basis for the willingness to cooperate and to look for common ground despite the differences reflected not only in different national rules and legal regulations concerning military action, but also in varying concepts and practices of leadership within the armed forces. The idea of equality cleared the way for testing new forms of military co-operation, challenging the principle of national responsibilities. On the other hand, co-operation focussing strictly on equality comes along with little tolerance for differences. It tends to result in fears of discrimination that are easily triggered and generate mistrust and discontent.

While co-operation between French and German military personnel is rather strained in Franco-German military units organised according to the principle of equality in numbers, it runs quite smoothly when French and German military personnel are collaborating within a multinational framework. Thus, integrated military co-operation in practice works best when collaboration is not based on the principle of strict equality but is realised in various forms allowing to develop "collective drills & skills", which generate professional trust and lead to task cohesion. This observation should be borne in mind in planning future cooperation. All the differences notwithstanding, pragmatic cooperation still seems be possible at various levels and in perhaps hitherto underexplored fields:

First, France and Germany should create a nucleus for more interoperable and potentially integrated military units. Germany is increasingly becoming a security hub in Europe, e.g. by developing integrated structures with the Netherlands, the Czech Republic and Romania and announcing a security partnership with the UK. Recent French proposals -such as the European Intervention Initiative (EII)go into a similar direction. In order for these units to be successful, the Franco-German leitmotiv of cooperation -strict equality -has to be abandoned.

These new trends could also be the basis for renewed cooperation between Paris and Berlin reviving and possibly intensifying existing Franco-German military structures. Given the structural difficulties encountered in the field of foreign interventions, it seems promising to not lose sight of the past decades' neglected dimension of European defence, namely collective defence. As both countries participate in reassurance measures on the eastern flank, this partaking could be used as an opportunity to plan yearly brigade-level military exercises in order to improve military coordination. This would have simultaneous advantages. For one, brigade-level exercises are now rare, and would thus be important for Germany's new military ambitions. Second, they would be vital to demonstrate the new Franco-German willingness to concretely cooperate in defence matters. These bilateral exercises should be open to other countries. Berlin and Paris should actively invite them to participate in the joint endeavour. Last, but not least, yearly brigadelevel military exercises would be a political signal towards allies on the eastern flank as well as to the United States, as they would be a contribution to NATO's deterrence posture. This would also allow for reviving the Franco-German Brigade around a realistic project. Until now, differences in force employment between Paris and Berlin have prevented real operational deployments of the FGB, with the exception of its deployment in Mali.

Moreover, Berlin and Paris should actively make use of the possibilities offered by PESCO and show the example by moving ahead bilaterally. France and Germany have proposed in September 2016 to develop a logistical hub for strategic transport in Europe with several aspects in mind: developing European strategic transport on land, in the air and sea and decreases the dependence on Antonov aircraft and helps to push forward the A400M to effectively make it the European first military transport aircraft. France and Germany also want to develop a European medical command, enhancing interoperability between both states as a model for the EU. Berlin and Paris perceive these projects to be major contributions under Permanent Structured Cooperation (PESCO). They should move ahead proactively.

Finally, French and German contributions to European operational autonomy need not automatically be strictly Franco-German in nature. Europe's autonomy in the area of security and defence depends on interoperable and jointly deployable military capabilities. France and Germany could greatly contribute to the creation of permanent multinational formations by serving as core nations and leading projects with dedicated multinational headquarters, which smaller nations can plug into. While encouraging and leading the creation of permanent multinational formations between EU member states, France and Germany should make sure these become the central tools for operations in all frameworks, be it CSDP, NATO or UN.

Civilian assets for crisis management

As important as interoperable and jointly deployable military capabilities are, civilian assets matter equally in crisis management. Although the civilian dimension's considerable added value to conflict prevention and post-conflict stabilisation and management is universally accepted, it has tended to be a source of disagreement in the Franco-German relationship. While Berlin, for a number of reasons, prefers recurring to civilian approaches over the deployment of military forces and tends to shift the attention on them, France seems to belatedly realise that they are not of subordinate importance. Recently, the civilian dimension has received increased attention, and there is a clear willingness on the EU's part to bolster civilian missions' reactivity. In order to meet this demand, Germany and France should intensify their bilateral cooperation in this hitherto rather neglected field.

A first step could be a structured exchange of experiences of and approaches to personnel management and recruitment. Another step could be regular joint seminars of experts from the French Gendarmerie and the German Federal Police in order to exchange experiences with challenges encountered during civilian CSDP missions. Third, to improve rapid response, France and Germany should become prime drivers for establishing a standing civilian CSDP capacity composed of seconded staff or contracted personnel. This capacity's core functions would be to act as a start-up capability and to provide temporary reinforcement with core expertise. France and Germany should form a standing bilateral start-up capacity with regard to monitoring activities, maintaining public order and providing security to mission personnel. Moreover, they should supply a permanent pool of experts for civilian CSDP missions, for example first-aid workers, firemen or civil security specialists.

In addition to this effort on personnel, France and Germany could further work together on key aspects in stabilisation and peace-building missions: security sector reform, disarmament, demobilisation, and reinsertion, development assistance in difficult security environments.

Recommendation 3

Since appropriate spending on security and defence is the key prerequisite for operational autonomy, France and Germany have to make sure that there is such appropriate funding.

Berlin and Paris have to overcome the debate on "2% defence spending" by initiating a discussion about the apportionment of defence budgets based on qualitative criteria.

In addition to national spending, the creation of the EDF has the potential to help foster Europe's operational autonomy. As the two largest member states in a post-Brexit EU, France and Germany should therefore contribute the lion's share to the fund.

Recommendation 4

Beyond strengthening their respective national military assets, France and Germany can contribute to European operational autonomy by pushing for capabilities at EU level and through engaging in "bottom-up" bilateral cooperation.

At the EU level, Berlin and Paris should support the MPCC by delegating several high-ranking officers to this newly established body and by making concrete proposals to its effective organisation. In the medium term, they should continue to issue plans on how to transform the MPCC into the EU's desired, fully-fledged permanent OHQ.

At the bilateral level, France and Germany have to overcome the existing shortcomings of their bilateral projects.

France and Germany should create a nucleus for more integrated military units.

 They should renew their bilateral military cooperation by reviving and possibly intensifying existing Franco-German military structures, especially through yearly brigade-level military exercises in order to improve military coordination.

 Berlin and Paris should actively make use of the possibilities offered by PESCO and show the example by moving ahead bilaterally.

 France and Germany should contribute to the creation of permanent multinational formations, serving as core nations and leading projects with dedicated multinational headquarters, which smaller nations can plug into.

Recommendation 5

In crisis management, civilian assets matter. In the EU, the civilian dimension has lately received increased attention.

Germany and France need to contribute to bolstering civilian missions' reactivity by intensifying their bilateral cooperation in this hitherto rather neglected field.

 A first step could be a structured exchange of experiences of and approaches to personnel management and recruitment.

 Another step could be regular joint seminars of experts from the French Gendarmerie and the German Federal Police exchanging experiences of challenges of civilian CSDP.

Third, to improve rapid response, France and Germany should become prime drivers of establishing a standing civilian CSDP capacity composed of seconded staff or contracted personnel.

Industrial Autonomy

Strategic autonomy also builds on access to defence technology and materiel -in other words, industrial autonomy. It is for this reason the defence industry plays a key role when it comes to achieving strategic autonomy, especially at the operational level: it provides its basis.

The defence industrial sector is yet special in many ways. States are the only clients, and even independent private companies are heavily dependent on governments and their procurement decisions. Research and development cycles are long due to products' complexity, and companies require state commitment in order to engage in them. At the European level, the sector is highly fragmented.

Moreover, no national defence industry is today sustainable without exporting its products beyond its domestic market. European defence industries consequently compete with each other within the EU and on global markets and are in a more disadvantageous position than their consolidated U.S. rivals. These rivals furthermore have higher R&D budgets at their disposal. EU member states as the industry's clients suffer from lacking economies of scale and lower degrees of interoperability due to the diversity of equipment in use across the Union.

In addition, economic considerations are not necessarily the determining factor in defence procurement. In a wider European context, countries have reasons other than financial or technological to buy American products, given that securing good relations with the United States are a top foreign policy objective for many governments.

These weaknesses of the European defence market and especially the problems that arise from its fragmentation and the absence of a true European Defence Technological and Industrial Base (EDTIB) have been known for quite some time.11 However, the industrial dimension has long been another neglected child of European defence cooperation.

In the past year, important steps have nevertheless been taken. The European Defence Action Plan (EDAP) presented by the Commission in late 2016 and since then translated into detailed measures, including the European Defence Fund, essentially aims at providing funding for research and development as well as funding facilities for joint acquisitions by states.

As the European Union, and the European Commission in particular, is more and more coming to the fore as an actor in the industrial field, Franco-German industrial cooperation is increasingly taking place against a European background rather than in a purely bilateral context. Given that France and Germany are key industrial players in an EU at 27, closer cooperation between those two partners is the obvious path to choose -and likely the only chance for European defence industries to survive in the long term. As a space for armament industries to organise themselves according to market logics emerges, deeper forms of integration appear feasible, as the KMW-Nexter merger seems to suggest.

Closer Franco-German cooperation is yet faced with a number of obstacles, arising both from features shared by France and Germany and differences between the two countries. What both countries have in common is that the defence industry matters as an important employer with sometimes significant relevance for regional economies. This non-security related aspect at times clearly trumps security and defence considerations. Likewise, fears of unwanted technology transfer and a certain degree of distrust between companies hamper cooperation on both sides.

Perhaps even more importantly, structural differences between France and Germany are also numerous in this field, be it with respect to the industries and their structures or with respect to the role they play in national (defence) policies. In a French understanding, the defence industry is a key aspect of the nation's strategic autonomy, although the 2017 Strategic Review insists on the need to move towards a more European approach (including mutual dependency) and explicitly suggests a European preference. Germany largely views the defence industry as a technological asset and industry-related issues are primarily considered as economic issues. In France, the industry is seen as both an economic and strategic asset, including in its nuclear dimension. The defence industry is (partly) state-owned in France, private and often family-run in Germany. French companies tend to be much larger than German SMEs. As a consequence, the state's role is essentially different, as well as perceptions on what the state's role should be.

In sum, given the fact that these differences are deeply embedded in the respective country's strategic culture, they are unlikely to disappear any time soon. In the past, more effective and cost-efficient solutions have been impeded by political reservations and national interests, not by armament industries. Major issues indeed need to be solved at the political level rather than in companies' research departments or at the level of military procurement agencies.

Structural differences between defence industries in both countries notwithstanding, French and German interests converge on a number of points. The devil naturally is in the detail, as past -successful as well as failed -attempts of Franco-German industrial cooperation have demonstrated. However, strengthening the European Defence Technological and Industrial Base is a declared objective in both countries.

Harmonising export policies

For closer Franco-German cooperation to contribute to a strengthened EDITB, and given European defence industries' dependency on export markets, selling its products is key. Creating new capabilities necessitates to first answer the question of the future use of these capabilities: next to their use for security reasons, is there an export market? With shrinking domestic markets, export potentials are actually likely to be a key incentive for countries to join PESCO or cooperate actively in armament programmes. Yet, export policies have traditionally been a bone of contention, with Germany being stricter than France. Industry representatives consequently declared on a number of occasions that the German approach represents a true obstacle for Franco-German cooperation. If France and Germany are truly to work together in the industrial field, a common outlook on the matter is consequently indispensable. An update of the 1972 Schmidt-Debré agreement is therefore rather frequently identified as a necessity.

Looking ahead, the issue of differing export control standards and their implications will inevitably rise again following the KMW-Nexter merger. Existing models will not provide mutually satisfactory solutions. Defining a joint approach to arms exports is thus a condition sine qua non for successful cooperation. In so doing, both sides' concerns must be taken seriously. One possibility would be a governmental agreement on export control criteria or a country list of admissible customers and the setting up of a common authority that would decide on export applications.

A more strategic approach to defence cooperation

Moreover, France, Germany and the EU would benefit from a more strategic approach to defence cooperation. This approach should not be driven by what is (politically) feasible, but derived from strategic necessities defined at the political level. In line with the measures taken to improve European political autonomy, Paris and Berlin should thus jointly identify key technologies and where to build them: what critical systems need to be available "at home", i.e. at the national level, what can and perhaps even should be imported from allies, and what can be purchased on the world market? A bold political statement would consist of defining France and Germany as a single "at home". This may have different meanings: it could imply a division of labour between German and French manufacturers (e.g. like -although not necessarily a case of Franco-German cooperation in the true meaning of the term -Heckler & Koch providing the FAMAS replacement "HK 416" for the French armed forces or Germany contributing to funding the French surveillance satellite Composante Spatiale Optique), but it can also mean cooperation on genuinely Franco-German products such as in the case of Airbus or KNDS (formerly known as KANT). Such a joint approach would also require Paris and Berlin to agree on common criteria for defining certain technologies as key technologies.

Moreover, for defence industrial cooperation to truly lead to economies of scale and increased interoperability, a major mistake of the past must not be repeated: France and Germany as cooperating countries must agree on only one set of specifications, resulting in one type of final product.

It would of course be unrealistic to assume that no other factors but strategic considerations are part of the equation. For capitals' decisionmaking, economic factors, including regional development and employment, certainly matter as well. Moreover, the structural differences between both countries' defence industries bear on strongly in this context. Yet, without political initiatives to work towards a Franco-German approach, things are unlikely to evolve. A systematic and joint approach to mapping key technologies with a view to cooperation potentials would be a start in attaining industrial autonomy.

Recommendation 6

Work towards shared rules on arms exports.

Defining a joint approach to arms exports is a condition sine qua non. One possibility would be a governmental agreement on export control criteria or a country list of admissible customers and the setting up of a common authority that would decide on export applications.

  9. European Commission, European Political Strategy Centre, In Defence of Europe. Defence Integration as a Response to Europe's Strategic Moment, EPSC Strategic Notes 4/2015, p. 3, available at: https://ec.europa.eu. 10. See European Defence Agency, Defence Data 2014, Brussels, 2016, available at : www.eda.europa.eu.

See e.g. B. Ballester, The Cost of Non-Europe in Common Security and Defence Policy, CoNE 4/2913, European Parliamentary Research Service, 2013, www.europarl.europa.eu.

Recommendation 7

Engage in a joint mapping of key technologies as a basis for industrial autonomy.

Like in all other fields, industrial cooperation should be guided by a strategic approach. In jointly and systematically defining industries that form the backbone of European industrial autonomy, France and Germany can create a conceptual basis for further cooperation.

France, Germany and European Strategic Autonomy: Complementarity, Not Congruence

If at least some of the ambitious proposals outlined here were to be implemented, this would constitute a leap forward towards European strategic autonomy. There is currently a real window of opportunity for Paris and Berlin taking European defence cooperation to the next level.

In so doing, two aspects should always be borne in mind. First, France and Germany have different strategic cultures. None of these two countries will simply adopt the other's strategic culture. Therefore, complementarity is the key word. Rather than deploring national differences and putting the blame on the respective other for being different, making use of both sides' strength should be the key objective.

Second, if Franco-German security and defence cooperation is to show the way for cooperation at the European level, Paris and Berlin must make sure that their European partners are on board. For European security and defence cooperation to make the necessary leap forward, it has to be in everybody's interest. It is this spirit that should guide Franco-German leadership, as in previous historical steps towards deeper European integration.