Pierre-Henri Conze
email: pierre-henri.conze@imt-atlantique.fr

Florian Tilquin

Mathieu Lamard

Fabrice Heitz

Gwenolé Quellec

Unsupervised learning-based long-term superpixel tracking

Keywords: superpixel matching, unsupervised learning, superpixel tracking, multi-step integration, random forests, foward-backward consistency

Finding correspondences between structural entities decomposing images is of high interest for computer vision applications. In particular, we analyze how to accurately track superpixels -visual primitives generated by aggregating adjacent pixels sharing similar characteristics -over extended time periods relying on unsupervised learning and temporal integration. A two-step video processing pipeline dedicated to long-term superpixel tracking is proposed. First, unsupervised learning-based superpixel matching provides correspondences between consecutive and distant frames using new context-rich features extended from greyscale to multi-channel and forward-backward consistency contraints. Resulting elementary matches are then combined along multi-step paths running through the whole sequence with various inter-frame distances. This produces a large set of candidate long-term superpixel pairings upon which majority voting is performed. Video object tracking experiments demonstrate the accuracy of our elementary estimator against state-of-the-art methods and proves the ability of multi-step integration to provide accurate long-term superpixel matches compared to usual direct and sequential integration.

Introduction

Finding correspondences between multiple images is a fundamental problem in computer vision tasks including scene segmentation [START_REF] Lezama | Track to the future: Spatiotemporal video segmentation with long-range motion cues[END_REF], 3D reconstruction [START_REF] Seitz | A comparison and evaluation of multi-view stereo reconstruction algorithms[END_REF], visual tracking [START_REF] Yang | Robust superpixel tracking[END_REF], trajectory analysis [START_REF] Wang | Dense trajectories and motion boundary descriptors for action recognition[END_REF] or video editing like 2D-to-3D video conversion [START_REF] Cao | Semi-automatic 2D-to-3D conversion using disparity propagation[END_REF] and graphic elements propagation [START_REF] Conze | Multi-reference combinatorial strategy towards longer long-term dense motion estimation[END_REF]. Established via local or global search, correspondences are usually either sparse through key feature points [START_REF] Shi | Good features to track[END_REF] or for the dense pixel grid as with optical flow [START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF]. Patch-based approximate nearest neighbor (ANN) search methods such as PatchMatch (PM) [START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF] and its extension to multi-resolution [START_REF] Barnes | The generalized patchmatch correspondence algorithm[END_REF] are mainly followed to find correspondences between patches. Alternatively, finding associations between structural entities decomposing images by grouping pixels enables a semi-dense coverage of whole images while drastically reducing the cost of correspondence. Contrary to a regular decomposition of the image grid into patches which would not offer enough consistent support regions for matching, superpixels -visual primitives generated by aggregating adjacent pixels sharing similar characteristics into visually meaningful entities [START_REF] Stutz | Superpixels: An evaluation of the stateof-the-art[END_REF] -provide more reliable areas preserving both image geometry and object contours. Moreover, the hypothesis that motion discontinuities are a subset of photometric contours is usually used to preserve boundaries between objects exhibiting different motion. In particular, these findings have motivated recent optical flow algorithms using image data and smoothness terms adapted to the superpixel level [START_REF] Chang | Superpixel-based large displacement optical flow[END_REF][START_REF] Donné | Fast and robust variational optical flow for high-resolution images using SLIC superpixels[END_REF]. Conversely, we claim that image matching relying on superpixels could benefit from these advantages to offer more consistent associations than pixel or patch matches while providing a better management of motion discontinuities. More precisely, this paper focuses on how to accurately find correspondences between superpixels over extended time periods.

In this context, superpixel correspondences have been already employed for visual tracking through superpixel-based discriminative appearance models [START_REF] Yang | Robust superpixel tracking[END_REF] or object-background confidence maps [START_REF] Fan | Visual tracking by local superpixel matching with markov random field[END_REF]. However, these works perform superpixel matching based on comparisons of intrinsic superpixel features only, without taking full advantage of neighborhood information. Conversely, [START_REF] Giraud | Superpatchmatch: An algorithm for robust correspondences using superpixel patches[END_REF] exploits a structure of superpixel neighborhood called SuperPatch involved in a superpixel PM framework. Superpixel neighborhood information greatly improves correspondences since it alleviates some matching failures due to irregular decomposition of the same image content, not directly comparable between images. However, even with incorporated neighborhood information, directly computing a matching distance between irregular structures can be tedious, especially when images are divided into a large collection of superpixels.

A prior pixel-to-superpixel mapping can drive the matching at the superpixel level to provide more precise correspondences. In this direction, [START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF][START_REF] Conze | Hierarchical multi-scale supervoxel matching using random forests for automatic semi-dense abdominal image registration[END_REF] uses random forests (RF) [START_REF] Breiman | Random Forests[END_REF] to establish supervoxel correspondences between two 3D images in an unsupervised fashion. RF is trained on one image by using supervoxel indexes as voxel-wise class labels and robust context-rich features to describe the extended neighborhood. Applying RF on the other image yields a voxel labelling which is then regularised using majority voting within supervoxel boundaries. First validated on medical image registration, we explore the use of such learning-based superpixel matching for accurate superpixel matching across long video sequences.

Despite recent advances related to optical flow integration [START_REF] Crivelli | Robust optical flow integration[END_REF][START_REF] Conze | Multi-reference combinatorial strategy towards longer long-term dense motion estimation[END_REF], the temporal tracking of superpixels over long-term video sequences has received little attention in the literature. [START_REF] Wang | Constrained superpixel tracking[END_REF] uses a constrained graph where nodes denote superpixels and edges encode spatial, temporal, and appearance constraints.

However, temporal constraints only model short-term smoothness between consecutive frames. Generative probabilistic model [START_REF] Chang | A video representation using temporal superpixels[END_REF] and proximity-weighted patch matching [START_REF] Lee | Temporal superpixels based on proximity-weighted patch matching[END_REF] employed to generate temporal superpixels only exploit short-term correspondences from one frame to the subsequent one. The same finding arises in [START_REF] Yang | Robust superpixel tracking[END_REF] whose tracker is conducted sequentially and therefore prone to motion drift. Establishing long-term superpixel correspondences requires to perform superpixel matching between consecutive and distant frames and therefore to handle simultaneously small and large displacements. To address this challenge, we exploit the concept of multi-step integration introduced for long-term motion estimation using optical flow [START_REF] Conze | Multi-reference combinatorial strategy towards longer long-term dense motion estimation[END_REF]. The idea is to generate a large set of elementary displacement estimations performed between consecutive frames or with larger inter-frame distances. Once combined, elementary multi-step estimations result in a large set of long-term correspondences which are significative enough to be fused through statistical processing. We are not aware of any studies that have recovered this concept for long-term superpixel tracking while it could bring many benefits in this context. Indeed, it can alleviate matching errors during superpixel trajectory estimation since new steps can give a chance to match with a correct location again compared to sequential processing whose tracks may be lost. Moreover, statistical processing upon large representative long-term superpixel candidates can solve the uncertainty component present for matching tasks.

It must be reported that deep learning has become popular for object tracking relying on convolutional networks to learn discriminative features to encode the target appearance [START_REF] Ma | Hierarchical convolutional features for visual tracking[END_REF][START_REF] Li | Deeptrack: Learning discriminative feature representations online for robust visual tracking[END_REF] or recurrent networks trained with reinforcement learning to learn how to predict object locations across videos [START_REF] Zhang | Deep reinforcement learning for visual object tracking in videos[END_REF]. Despite their high performance, these methods only provide very sparse bounding box tracking and do not describe how the boundaries of an irregular shaped object evolves in time as expected through long-term superpixel tracking.

In summary, two main contributions are proposed towards accurate longterm superpixel tracking. First, unsupervised learning-based superpixel matching is generalized and adapted from medical image processing [START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF][START_REF] Conze | Hierarchical multi-scale supervoxel matching using random forests for automatic semi-dense abdominal image registration[END_REF] to computer vision in order to find associations along video sequences between consecutive and distant images decomposed into superpixels (Sect.2). The approach is carried out using classifiers such as k-nearest neighbors (kNN) or RF [START_REF] Breiman | Random Forests[END_REF], incorporates new forward-backward consistency contraints and fully exploits dedicated context-rich features we extended from greyscale [START_REF] Glocker | Robust registration of longitudinal spine CT[END_REF][START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF][START_REF] Conze | Hierarchical multi-scale supervoxel matching using random forests for automatic semi-dense abdominal image registration[END_REF] to multi-channel to incorporate neighborhood information on RGB frames. Second, based on this learning-based matching approach used as an elementary displacement estimator, we propose a multi-step integration strategy for longterm superpixel tracking (Sect.3). It combines multiple elementary superpixel matches obtained for some intermediate images following randomly selected multi-step paths. This produces a large set of candidate long-term superpixel pairings upon which a majority voting selection is performed. Based on object tracking experiments extending a preliminary study presented in [START_REF] Conze | Long-term superpixel tracking using unsupervised learning and multi-step integration[END_REF], Sect.4 assesses the accuracy of the proposed elementary estimator against state-ofthe-art methods and proves the ability of multi-step integration to provide an efficient long-term superpixel tracking compared to both standard direct and sequential integration. We end with conclusions and perspectives in Sect.5.

Superpixel matching with unsupervised learning

Let V be a video sequence of RGB images. In this section, unsupervised learning-based superpixel matching is addressed between two consecutive or distant images I f and I s of V. Each image I q : Ω q ⊂ N 2 → N 3 associates a RGB color vector I q (x q ) to each pixel p q located at x q ∈ Ω q with q ∈ {f, s} 1 .

Problem formulation

Let F = {f i } i∈{1,...,|F |} and S = {s j } j∈{1,...,|S|} be respectively the set of |F| and |S| connected superpixels partitioning I f and I s . The superpixel decomposition can be performed using any superpixel algorithm [START_REF] Stutz | Superpixels: An evaluation of the stateof-the-art[END_REF] such as Simple Linear Iterative Clustering (SLIC) [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] which aggregates neighboring pixels p q based on spatial and intensity proximity criteria. Forward superpixel matching from I f to I s (f < s) consists in automatically learning a matching function h f,s that maps each superpixel f i ∈ F of I f to a given superpixel s j ∈ S of I s [START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF] such that:

∀i ∈ {1, . . . , |F|}, ∃j ∈ {1, . . . , |S|} | h f,s (f i ) = s j (1)
Backward matching from I s to I f can be similarly considered by estimating h s,f mapping each superpixel s j ∈ S to a given superpixel f i ∈ F. In what follows, learning-based superpixel matching is described in forward from I f to I s .

1 stands for first and second source I f SLIC [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] pixel-wise prediction majority voting source Is SLIC [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] pixel-wise training kNN or RF [START_REF] Breiman | Random Forests[END_REF] Figure 1: Superpixel matching between I f and Is using unsupervised learning applied with SLIC [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] superpixel indexes as label entities followed by majority voting following [START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF]. This example is produced for image pair {I 50 , I 70 } of the lapa sequence [START_REF] Sznitman | Datadriven visual tracking in retinal microsurgery[END_REF] using RF [START_REF] Breiman | Random Forests[END_REF].

Overall strategy

Instead of relying on nearest neighbor search at the superpixel level through superpixel feature comparisons [START_REF] Fan | Visual tracking by local superpixel matching with markov random field[END_REF][START_REF] Giraud | Superpatchmatch: An algorithm for robust correspondences using superpixel patches[END_REF], which is prone to ambiguity due to possible severe overlaps in feature space, we explore the use of pixel-wise k-nearest neighbors (kNN) or random forests (RF) [START_REF] Breiman | Random Forests[END_REF] to establish correspondences between superpixels over-segmenting I f and I s , as formulated in Eq. [START_REF] Glocker | Robust registration of longitudinal spine CT[END_REF][START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF][START_REF] Conze | Hierarchical multi-scale supervoxel matching using random forests for automatic semi-dense abdominal image registration[END_REF] to multi-channel RGB and described in Sect.2.3.

The key idea is to perform training on the target image (I s ) by using superpixel indexes as pixel-wise class labels and testing on the first image (I f ) to get a pixel-to-superpixel mapping, as introduced in [START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF]. In particular, the classifier aims at assigning a superpixel s j ∈ S to each pixel p f ∈ Ω f . A training set is thus built by considering all pixels p s of Ω s with their associated superpixel index, i.e. the index of the superpixel s j they belong to. Once trained, the classifier is applied to I f to predict for each p f the index of a superpixel of S.

This pixel-to-superpixel mapping is addressed in detail in Sect.2.4.

Mapping results are further regularized following superpixel boundaries to reach robust superpixel matches. Within each superpixel f i of I f , the most represented superpixel index among all pixel-wise predictions indicates the best superpixel match h f,s (f i ). This final superpixel matching step is detailed in Sect.2.5 with new foward-backward (FW-BW) consistency constraints.

Context-rich multi-channel features

Extracting contextual information from near size-variable patches is key to discriminate pixels towards robust pixel-to-superpixel mapping. Let Īw q (p q , c) be the average intensity on a local box of size w centered on p q for channel c ∈ {r, g, b}. Pixel-wise context-rich features θ(p q ) = {θ m (p q )} m∈{0,...,Ka-1} assigned to pixels p q are extended from greyscale [START_REF] Glocker | Robust registration of longitudinal spine CT[END_REF][START_REF] Kanavati | Supervoxel classification forests for estimating pairwise image correspondences[END_REF][START_REF] Conze | Hierarchical multi-scale supervoxel matching using random forests for automatic semi-dense abdominal image registration[END_REF] to multi-channel as follows:

θ m (p q ) = Īw p (p q + ∆ r , c) -β × Īw p (p q + ∆ r , c) (2)
where displacements ∆ {r,r } are randomly defined starting from p q in a disc of maximal radius Φ (Fig. 2). β ∈ {0, 1} is a binary parameter which focuses whether on intensity differences between two boxes randomly located in the extended neighborhood (β = 1) or on the value obtained from one single box only (β = 0). Color intensities around p q are included in the feature vector θ(p q ) by forcing ∆ r = β = 0 for all possible pre-defined box sizes w and channels c.

By randomly generating many different box sizes w and offsets ∆ r , we obtain a large set of K a features describing the extended spatial context for all color channels c. Parameters {w, w , ∆ r , ∆ r , β, c} are randomly generated once and remain similar whatever the image I f or I s under consideration.

Pixel-to-superpixel mapping

Pixel-to-superpixel mapping relies on machine learning to compute pixelto-superpixel mapping probabilities denoted as p(h f,s (p f )=s n ) for each pixel p f ∈ Ω f with respect to all superpixels s n ∈ S with n ∈ {1, . . . , |S|}.

The procedure with random forests (RF) [START_REF] Breiman | Random Forests[END_REF] is conducted as follows. The forest is formed by T uncorrelated trees made of both internal nodes splitting data according to binary tests Ψ and leaf nodes which reach all together a final data partition. At each internal node, the split sends pixels p q to left and right child nodes during training (q =f ) and prediction (q =s). The associated binary test Ψ focuses on a random subset θ(p q ) of context-rich multi-channel visual features θ(p q ) assigned to p q (Sect.2.3) and divides the input pixel set based on the following split rule:

Ψ(p q , θ(p q )) =    true, if θ(p q ) > τ false, otherwise (3)
where θ(p q ) is compared to a threshold τ and q ∈ {f, s}.

Internal node parameters ({τ, θ(p q )}) are optimized via information gain maximization with respect to the training dataset L = {p s , c(s j )} combining pixels p s belonging to s j with their associated superpixel index c(s j ) = j with j ∈ {1, . . . , |S|}. Any additional training data is required additionally to L which makes this strategy unsupervised. After optimization, each leaf node l t of the t th tree receives a partition L lt of L and produces the class probability distribution p lt (c(s j )|L) for all superpixels s j .

To predict the corresponding superpixel index c(s n ) of a given pixel p f ∈ Ω f with associated visual features θ(p f ) during testing, p f is injected into each optimized tree and finally reaches a leaf node l t per tree following the successive split rules (Eq.3). The pixel-to-superpixel mapping probability

p(h f,s (p f ) = s n )
denoting the probability that s n is assigned to p f is obtained for each s n by: Nearest neighbors are estimated using Euclidean distance on context-rich features assigned to pixels p q with q ∈ {f, s}.

p(h f,s (p f ) = s n ) = 1 T T t=1 p lt (c(s n )|L) = 1 T T t=1 |{p s , c(s l )} ∈ L lt | l = n | |L lt | (4)

Superpixel-to-superpixel matching

Once pixel-to-superpixel mapping probabilities p(h f,s (p f ) = s n ) are computed for each superpixel s n ∈ S using context-rich features (Sect.2.3) involved in RF or kNN (Sect.2.4), two steps are required to get final superpixel pairings.

First, the final pixel-to-superpixel mapping for each p f of I f can be found using:

h f,s (p f ) = s j = arg max sn∈S p(h f,s (p f ) = s n ) (5)
Second, majority voting among all pixels of a given superpixel f i ∈ F can be performed by selecting the most represented superpixel index. The final matching h f,s (f i ) = s j is defined such that c(s j ) satisfies:

c(s j ) = arg max hist({c(h f,s (p f )) | p f ∈ f i }) (6)
An alternative consists in averaging the pixel-to-superpixel mapping probabilities at the superpixel level instead of making hard decision for each p f as performed in Eq.5:

p(h f,s (f i ) = s n ) = 1 |f i | p f ∈ f i p(h f,s (p f ) = s n ) (7)
We keep at this point all possible outcomes between candidate matches. Decisions are postponed to the superpixel level by finding the superpixel s j ∈ S

which maximizes p(h f,s (f i ) = s n ): h f,s (f i ) = s j = arg max sn∈S p(h f,s (f i ) = s n ) (8)
Forward-backward consistency can be enforced in the context where two mapping functions are learned: h f,s (resp. h s,f ) that maps each supervoxel

f i ∈ F (s j ∈ S) to a given s j (f i ) belonging to S (F) in forward (backward).
Thus, we extend Eq.8 with a new consistency constraint that guides the mutual matching between f i and s j :

h f,s (f i ) = arg max sn∈S p(h f,s (f i ) = s n ) × p(h s,f (s n ) = f i ) (9)
The whole unsupervised learning-based strategy described above can be performed all along the video V to match superpixels decomposing consecutive or distant images, both in forward and backward directions.

Long-term superpixel tracking using multi-step integration

We address at this stage long-term superpixel tracking for sequence V com- Both superpixel trajectory estimation between the reference frame and all the images of the sequence and superpixel matching to match each image to the reference frame can be considered, as in [START_REF] Crivelli | Robust optical flow integration[END_REF][START_REF] Conze | Multi-reference combinatorial strategy towards longer long-term dense motion estimation[END_REF]. From-the-reference estimation is useful for information pushing from superpixels of Thus, for each step α q ∈ A n , we have a superpixel match in I n+αq for each superpixel of I n through the mapping function h n,n+αq , and this for each frame.

posed of N + 1 RGB frames I n : Ω n ⊂ N 2 → N 3 using
I
The starting point of multi-step integration consists in initially generating all possible step sequences (Fig. 3), i.e. combinations of steps, to join I n from I ref .

Then, each generated step sequence defines a multi-step path (Fig. 4) linking each superpixel f i of I ref to a superpixel s j in I n passing through superpixels of some intermediate frames.

Let Γ ref,n = {γ 0 , γ 1 , . . . , γ K-1 } be the set of the K possible step sequences γ k between I ref and

I n . A step sequence γ k = {α k 1 , α k 2 , . . . , α k Kγ k } is defined by a set of K γ k steps which once cascaded join I n from I ref . Γ ref,n is computed
by building a tree structure (Fig. 3) where each node corresponds to a field of superpixel matches assigned to a given frame for a given step value (node value).

Going from the root node to leaf nodes of this tree structure gives the possible step sequences which are stacked into Γ ref,n . For instance, the tree displayed in Fig. 3 indicates the 4 possible step sequences from I 0 to I 3 with steps 1, 2, and 3:

Γ 0,3 = {{1, 1, 1}, {1, 2}, {2, 1}, {3}}.
Once all the K possible step sequences γ k between I ref and I n are generated, the corresponding multi-step paths are constructed (Fig. 4). For step sequence

γ k = {α k 1 , α k 2 , . . . , α k Kγ k } ∈ Γ ref,n composed of K γ k steps, superpixel matching
between I ref and I n is performed via:

h ref,n (f i )| γ k = h ref + Kγ k -1 p=1 α k p ,n • . . . • h ref +α k 1 ,ref +α k 1 +α k 2 • h ref,ref +α k 1 (f i ) (10)
with ref

+ Kγ k p=1 α k p = n. Once all the steps α k j ∈ γ k have been run through, one gets h ref,n (f i )| γ k , the superpixel in I n corresponding to f i ∈ I ref obtained
with step sequence γ k . For γ k = {1, 2} ∈ Γ 0,3 for instance (Fig. 4), we have:

h 0,3 (f i )| {1,2} = h 1,3 h 0,1 (f i ) = h 1,3 • h 0,1 (f i ) (11)
A large set of candidate superpixels in I n is finally reached by considering all the step sequences of Γ ref,n and this for each superpixel f i defined in I ref .

Thus, to each f i is associated a large set of K candidate superpixels in I n defined as

T ref,n (f i ) = { h ref,n (f i )| γ k } k∈{0,1,...,K} .
Multi-step integration has been previously presented as an exhaustive candidate generation process. In practice, selecting only a subset of all possible step sequences and therefore associated multi-step paths is required to be able to build and keep in memory the multi-step integration stage outputs growing exponentially [START_REF] Conze | Multi-reference combinatorial strategy towards longer long-term dense motion estimation[END_REF]. For instance, 5877241 multi-step paths can be generated for a distance of 30 frames using steps 1, 2, 5 and 10. Up to a few thousands can be actually considered to avoid computational and memory issues. The selection of L step sequences Γ * ref,n = {γ 0 , γ 1 , . . . , γ L-1 } among the K possible step sequences Γ ref,n is therefore necessary, with L << K.

Two complexity reduction rules are taken from [START_REF] Conze | Multi-reference combinatorial strategy towards longer long-term dense motion estimation[END_REF]. We start by removing the largest step sequences in terms of number of constituting steps. A threshold of K max number of steps is thus set and only step sequences

γ k = {α k 1 , α k 2 , . . . , α k Kγ k } for which K γ k ≤ K max are kept.
Indeed, too many steps may induce an important drift due to multiple intermediates. Then, random selection of L step sequences among remaining ones is performed.

Long-term match selection

Once step selection is performed, we obtain for each superpixel

f i of I ref a set of L candidate superpixels T ref,n (f i ) = { h ref,n (f i )| γ l ∈Γ * ref,n
} defined in I n with l ∈ {0, ..., L -1}. The final candidate selection is performed via majority voting among T ref,n (f i ), i.e. the final matching h ref,n (f i ) = s j is defined such that c(s j ) satisfies:

c(s j ) = arg max hist({c( h ref,n (f i )| γ l ) | γ l ∈ Γ * ref,n }) (12)
Thank to the random step sequence selection (Sect.3.1), the set of generated superpixel candidates is both significative and uncorrelated enough to assume that the most represented superpixel provides an accurate superpixel match.

Forward-backward consistency can be also considered in this context by providing to-the-reference multi-step paths additionally to from-the-reference ones.

We thus incorporate in

T ref,n (f i ) superpixels s j such that h n,ref (s j )| γ l ∈Γ * n,ref = f i where Γ * n,ref
is the set of L selected step sequences in the to-the-reference direction. The resulting additional superpixel candidates are referred as reverse candidates in opposition to direct ones, i.e. those which were formerly stacked into T ref,n (f i ). To further guide mutual matching between f i and s j , one can apply majority voting (Eq.12) only on superpixel candidates generated in both from/to-the-reference directions.

Application to video object tracking

Different aspects of the proposed methodology are evaluated through video object tracking experiments. First, the ability of unsupervised learning-based superpixel matching to provide a reliable accurate elementary estimator between consecutive and distant frames is proven with comparisons to state-of-the-art methods (Sect.4.1). Second, the capacity of the proposed multi-step integration stage to perform robust long-term superpixel tracking is shown using both kNN and RF-based multi-step elementary superpixel matches (Sect.4.2). Moreover, multi-step integration results are assessed with respect to straightforward direct and sequential integration outputs. Third, multi-step integration is further analyzed by studying the impact of different candidate generation strategies in terms of tracking accuracy (Sect.4.3).

To provide a generic evaluation while ensuring content diversity and representativity, video object tracking is performed over 10 sequences (Tab.1) extracted from 4 databases: bag, fish3 (denoted fsh3) and octopus (octo) from the Visual Object Tracking (VOT) database [START_REF] Kristan | The visual object tracking VOT2016 challenge results[END_REF], sleep1 (sle1) with albedo from MPI Sintel [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF], lapa from the laparoscopy dataset [START_REF] Sznitman | Datadriven visual tracking in retinal microsurgery[END_REF] VIdeo Segmentation (DAVIS) database [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF]. As detailed in Tab.1, these sequences cover altogether many challenging situations such as complex non-rigid . Then, contour-based precision P c and recall R c between estimated and GT masks can be estimated relying on bipartite graph matching to be robust to small inac-curacies [START_REF] Martin | Learning to detect natural image boundaries using local brightness, color, and texture cues[END_REF]. In practice, we focus on the F-measure combining precision and recall using F = 2PcRc

Pc+Rc . Bi-partite matching is approximated using morphology operators, as in [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF]. Finally, consistency-based assessment is performed relying on the percentage of pixels of I ref located inside the ROI and whose belonging superpixel f ref is consistent in terms of forward-backward binary consistency:

h n,ref (h ref,n (f ref )) = f ref (13)
In terms of computation time, performing RF-based matching followed by multi-step integration using steps {1, 2, 5, 10, 20} on a sequence of 640×360 frames such as octo with 500 SLIC [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] superpixels takes approximately 6 minutes per frame using a 3.1GHz Intel Xeon CPU processor and Python implementation, without extensive code optimization. Processing time is reduced about 17% when relying on kNN for unsupervised learning. images decomposed into 500 SLIC superpixels. PM [START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF] is looking for the best patch matches using 9×9 windows with 6 iterations including both propagation and random refinement steps. Farnebäck [START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF] and SIFT Flow [START_REF] Liu | Sift flow: Dense correspondence across scenes and its applications[END_REF] superpixel-to-superpixel matching using superpixel-wise average color (RGBm) and color histogram (RGBh) features, PatchMatch (PM) [START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF], optical flow through Farnebäck [START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF] and SIFT Flow [START_REF] Liu | Sift flow: Dense correspondence across scenes and its applications[END_REF] as well as the proposed unsupervised learning-based superpixel matching using kNN/RF classifiers. Bold and underline results indicate first and second best scores. is more deeply demonstrated in the next section.

Long-term superpixel tracking

Long-term ROI tracking resulting from direct (DIR), sequential (SEQ) and multi-step (MSI) integrations are compared based on unsupervised learningbased SLIC [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] superpixel matching whose accuracy against state-of-the-art methods has been demonstrated in Sect.4.1 with both kNN and RF classifiers.

MSI is applied with L=200 maximal step sequences per image pairs. Only step sequences whose length is less than or equal to K max = 7 are kept to prevent from motion drift (Sect. series of medical images can be also processed with the proposed methodology.

Reliable ROI tracking through MSI is also shown on synthetic images (Fig. 8) despite strong scale variations. Propagation of matching errors with SEQ is clearly illustrated Fig. 7 for lapa (kNN) and Fig. 9 for bag. Tracking failures with DIR are temporally uncorrelated but strong enough to damage the propagation task for the fish (Fig. 9) due to color variations of its right part. ROI tracking for octo (Fig. 9) gives correct results both with kNN and RF despite significant color similarities with the dynamic background. Experiments for swan (Fig. 5) and sle1 (Fig. 8) videos demonstrate that RF-based MSI outperforms kNN-based MSI as well as RF-based DIR and SEQ. More globally, results suggest that the SLIC superpixel algorithm [START_REF] Achanta | SLIC superpixels compared to state-of-the-art superpixel methods[END_REF] is enough accurate to provide visually meaningful entities to be accurately tracked along video sequences.

Comparisons with [START_REF] Yang | Robust superpixel tracking[END_REF] are provided for bag, fsh3, and octo sequences (Fig. 9).

The first finding is that [START_REF] Yang | Robust superpixel tracking[END_REF] only provides sparse bounding box tracking and do not describe how the boundaries evolves in time, as performed by the proposed superpixel tracker. Despite an accurate tracking of the foreground fish (fsh3 sequence), [START_REF] Yang | Robust superpixel tracking[END_REF] results in long-term tracking failures for bag (especially in I 71 ).

Furthermore, it does not show robustness to strong scale variations and color similarities with the dynamic background (I 110 and I 120 for octo) contrary to RF-based MSI which offers an efficient pixel-wise long-term object delineation.

Another visualization through prediction masks given Fig. 5

Figure 2 :

 2 Figure 2: Pixel-wise context-rich multi-channel features provide a description of the extended spatial context (see Sect.2.3 for further details).

  Contrary to RF, the kNN classifier simply stores instances of training data instead of building a general internal model. Pixel-to-superpixel mapping probabilities p(h f,s (p f )=s n ) are computed by looking at the class (superpixel index) distribution among the k nearest neighbors p s of each pixel p f in feature space.

  the learning-based superpixel matching strategy, described Sect.2 for a given pair of consecutive or distant frames, as elementary estimator. Each frame I n is decomposed into a set of superpixels with the same amount of visual primitives for each. One particular frame (usually the first one) of V is defined as the reference frame and denoted I ref . In this context, we aim at finding correspondences between superpixels over-segmenting I ref and superpixels defined in frames I n with n ∈ {0, . . . , N } = ref . Let F = {f i } i∈{1,...,|F |} and S = {s j } j∈{1,...,|S|} be the set of |F| and |S| connected superpixels partitioning I ref and I n .

Figure 3 :

 3 Figure 3: Generation of step sequences from I 0 to I 3 with steps 1, 2, and 3 by creating a tree structure: Γ 0,3 = {{1, 1, 1}, {1, 2}, {2, 1}, {3}}.

Figure 4 :

 4 Figure 4: Generation of multi-step paths corresponding to step sequences {{1, 2}, {2, 1}} ⊂ Γ 0,3 from I 0 to I 3 .

  Superpixel correspondences with respect to I ref are provided for the whole sequence relying on multi-step integration applied independently for each pair {I ref , I n } ∀n = ref and based on unsupervised learning-based superpixel matching as an elementary estimator.

  motion (NR), large displacement (LD), background clutter (BC), i.e. color similarities with background or between objects, dynamic background (DB) including moving background objects and camera viewpoint changes, scale variations (SV), partial occlusions (PO), thin structures (TS), illuminations changes and shadows (IC). Image sizes vary from 364 × 270 to 640 × 360 (Tab.1). Except for lapa whose ground-truth (GT) masks have been created from our own, all sequences were provided with associated GT masks indicating exact object delineations. Video object tracking, also called semi-supervised video object segmentation task, consists in estimating for the whole sequence the exact location of a semantically meaningful free-shape region of interest (ROI) manually defined in one single image referred as reference frame. Once produced, tracking results are assessed for each pair {I ref , I n } with n = ref based on three complementary measures. First, Dice scores [35] measure the region-based segmentation similarity between estimated X and GT Y masks by computing 2|X∩Y | |X|+|Y |

4. 1 .

 1 Elementary superpixel matchingOur first experiments consist in evaluating the proposed unsupervised learningbased superpixel matching (Sect.2) between consecutive and distant frames against state-of-the-art methods. In this direction, ROI tracking is performed through direct integration (DIR) in the to-the-reference direction, i.e. relying on direct processing of image pairs {I n , I ref } without any sequential or multistep combinations of pre-estimated superpixel matches. Unsupervised learningbased matching using both kNN and RF classifiers is compared to three other methodologies: superpixel-to-superpixel matching using superpixel-wise average color (RGBm) and color histogram (RGBh) features, PatchMatch (PM)[START_REF] Barnes | Patchmatch: A randomized correspondence algorithm for structural image editing[END_REF], as well as optical flow through Farnebäck[START_REF] Farnebäck | Two-frame motion estimation based on polynomial expansion[END_REF] and SIFT Flow[START_REF] Liu | Sift flow: Dense correspondence across scenes and its applications[END_REF]. Unsupervised learning-based matching works with |F| = |S| = 500 SLIC superpixels per frame with the same compactness parameter (i.e. same weighting between spatial and intensity proximity for each frame) and employs K a = 80 context-rich multichannel features computed with Φ = 40 as maximal radius and w ∈ {3, 5, 7} as possible box sizes (Sect.2.4). RF is made of T = 100 trees whereas kNN relies on 5 neighbors for queries. RGBm and RGBh use respectively average RGB colors and RGB histograms (using 10 bins) as superpixel-wise features to give correspondences in I ref for each superpixel of I n in a nearest neighbor manner. As for unsupervised learning-based matching, RGBm and RGBh exploit

  kNN -SEQ kNN -MSI RF -SEQ RF -MSI training RF -DIR pred. RF -SEQ pred. RF -MSI pred.

Figure 5 :

 5 Figure 5: ROI propagation for swan ({I 1 , I 34 }) [34] with DIR, SEQ and MSI (steps {1, 2, 5, 10, 20, 30}) integrations using kNN and RF. Results are compared with: superpixelto-superpixel matching with average color (RGBm) and color histogram (RGBh) features, PatchMatch (PM) [9], optical flow through Farnebäck (Far) [8] and SIFT Flow (SF) [37]. Blue boundaries in I 1 indicate superpixel labelling resulting from GT assignment. Green and red boundaries correspond to groundtruth (GT) and estimated tracking results. The last raw displays training (I 1 ) and prediction (I 34 ) masks resulting from DIR, SEQ and MSI integrations of RF-based elementary pairings.

  remains an interesting and faster alternative, depending on expected tracking quality requirements. Since the balance between processing time and tracking quality is guided by the application, we can imagine that k-NN could be chosen against RF in some contexts.Temporal evolutions of Dice and F-measure scores are displayed Fig.6along lapa, sle1.2 (sle1 for object 2), octo and swan sequences with both classifiers.As already confirmed, best tracking results are reached with MSI compared to DIR and SEQ, especially for distant pairs. Contrary to SEQ whose performance decreases across sequences due to error accumulations (lapa and swan especially), multi-step estimations involved in MSI allow to fix uncorrect superpixel tracks as we can notice for sle1.2 from frame I 20 . Moreover, DIR is not prone to motion drift as SEQ but direct matching becomes tedious when inter-frame distances increase as shown for octo starting from I 109 . Finally, it can be noticed that the temporal behavior remains almost the same regardless of the classifier.Finally, quantitative results are illustrated by series of ROI selection and visual tracking examples for several pairs of lapa (Fig.7), sle1 (Fig.8), bag, fsh3, and octo (Fig.9) sequences. Fig.7shows that kNN-based MSI provides a very good delineation of the surgical tool for all image pairs, which suggests that

Figure 6 :Figure 7 :

 67 Figure 6: Temporal evolution of Dice (left) and F-measure (right) scores during ROI tracking across lapa [29], sle1.2[START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF], octo[START_REF] Kristan | The visual object tracking VOT2016 challenge results[END_REF] and swan[START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] sequences. We compare direct (DIR), sequential (SEQ) and multi-step (MSI) integration based on kNN and RF-based elementary pairings.

I 1 I 1 -Figure 8 :Figure 9 :

 1189 Figure 8: ROI selections and tracking across sle1 sequence [33] from I 1 to I 50 . We compare direct (DIR), sequential (SEQ) and multi-step (MSI, steps {1, 2, 5, 10, 20}) integration based on unsupervised learning-based superpixel matching with kNN and RF.

  generated using direct candidates only (MSId), both direct and reverse candidates (MSIr) or only candidates generated in both direct and reverse directions (MSIm). Note that the previously given MSI results corresponded to MSIm where only superpixel duplicates are taken into account for majority voting (Eq.12).MSId, MSIr and MSIm are comparatively evaluated in terms of tracking accuracy based on RF-based superpixel elementary matches. Dice, F-measure and consistency scores are reported across the 10 sequences used for ROI tracking.Results from Tab.4 bring two main findings. First, we observe that tracking accuracy is improved when reverse candidates are used additionally to direct ones. Consistency ratios are clearly improved (from 63.3 to 71.5% when comparing MSId/MSIr) as expected but Dice and F-measure improvements can be also observed with gains of 0.7 and 1.3 between MSId/MSIr. Second, relying on superpixel duplicates only (MSIm) brings a slighltly better ROI tracking compared to MSIr. Average results slightly increase from 90.4 to 90.6, 85.2 to 85.5 and 71.5 to 71.7 for Dice, F-measure and consistency which shows that extensive mutual matching guidance as performed with MSIm is the best way to perform long-term superpixel tracking from multi-step elementary correspondences.5. ConclusionIn this work, we proposed a two-step pipeline dedicated to long-term superpixel tracking. Unsupervised learning-based superpixel matching is firstly considered as an elementary displacement estimator to provide correspondences between consecutive and distant images using either nearest neighbors or random forests with robust context-rich features we extended from greyscale to multi-channel and forward-backward consistency contraints. Resulting elementary matches are then combined along multi-step paths running through the sequence with various inter-frame distances to produce a large set of candidate long-term superpixel pairings upon which majority voting selection is performed. Compared to state-of-the-art methods including pixel or patch-based strategies which may suffer from regular support regions, video object tracking experiments demonstrate that unsupervised learning can produce reliable correspondences between visually meaningful entities. Moreover, the ability of multi-step integration to combine these pairings towards accurate long-term superpixel tracking has been shown against usual direct and sequential integrations. Extending this work from single to hierarchical multi-scale superpixel decomposition would deserve further investigation for future research since dealing with multiple spatial extends can drive the matching process in a coarse-to-fine fashion. Other superpixel algorithms as well as other features such as spectral features could be employed to further improve unsupervised learning-based matching while reducing processing time. In addition, very long-term superpixel tracking could be reached by combining superpixel pairings estimated with respect to multiple reference frames. Our contributions also give new insights for optical flow and registration initialization, in particular to provide a better management of large displacements, appearance and illumination changes. More generally, the proposed framework could be easily extended to other imaging modalities including series of medical images for anatomical structure tracking.

  ref whereas to-the-reference estimation allows information propagation over superpixels of each frame I n by pulling it from I ref . The description below focuses on a given pair {I ref , I

	Issues related to both sequential and direct superpixel tracking could be
	partially compensated by complexifying the superpixel matching models and
	criteria, but an uncertainty component remains. This argues in favor of a sta-
	tistical processing (Sect.3.2) which takes into account a large set of candidate
	long-term superpixel matches obtained using multi-step combination (Sect.3.1)
	of elementary superpixel pairings previously established through unsupervised
	learning following Sect.2.

n } where I n is located far away from I ref . Correspondences for the whole sequence are obtained by processing each pair {I ref , I n } independently ∀n = ref .

Starting from learning-based superpixel matching (Sect.2) as elementary motion estimator, two temporal integration schemes can be considered at first glance to find h ref,n mapping each superpixel f i ∈ F to a given superpixel s j of S such that h ref,n (f i ) = s j . First, sequential integration can be employed passing through all intermediate frames, similarly to dense point tracking algorithms

[START_REF] Brox | Large displacement optical flow: descriptor matching in variational motion estimation[END_REF]

. This step-by-step strategy can gradually apprehend appearance changes and large displacements but may lead to large error accumulation, resulting in a substantial drift over extended time periods. This drawback is further enhanced when using superpixels since superpixel decompositions across the sequence may result in an irregular partitioning of the image content. Second, to avoid error accumulations, direct matching

[START_REF] Roth | Discrete-continuous optimization for optical flow estimation[END_REF]

can be applied between superpixels of I ref and I n , exactly as in Sect.2. However, this ignores that V consists of interrelated images with redundant and smoothly evolving content, which makes large displacement and aspect changes challenging to handle.

Table 1 :

 1 as well as swan, bear, camel (caml), cows and flamingo (flam) from the Densely Annotated Overview of sequences extracted from[START_REF] Kristan | The visual object tracking VOT2016 challenge results[END_REF][START_REF] Sznitman | Datadriven visual tracking in retinal microsurgery[END_REF][START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF][START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] and used for object tracking

		img	size	obj	NR	LD	BC	DB	SV	PO	TS	IC
	bag [32]	101	480×360	1	x	x		x				x
	fsh3 [32]	101	480×270	1		x		x				x
	octo [32]	51	640×360	1	x		x	x	x
	lapa [29]	81	364×270	1			x	x				x
	sle1 [33]	50	512×218	3			x		x		x
	swan [34]	50	427×240	1		x		x				x
	bear [34]	82	427×240	1	x	x	x	x		x		x
	caml [34]	90	427×240	1	x	x	x	x		x	x	x
	cows [34]	104	427×240	1	x	x		x	x	x		x
	flam [34]	80	427×240	1	x	x	x	x			x	x

experiments with associated sequence length, tracked object number and video attributes including complex non-rigid motion (NR), large displacement (LD), background clutter (BC), dynamic background (DB), scale variations (SV), partial occlusions (PO), thin structures (TS), illuminations changes and shadows (IC).

  estimators are used using by-default parameters. Learning-based and superpixel-to-superpixel matching are performed once a groundtruth label is assigned to each superpixel of I ref to indicate its belonging to the ROI to be tracked. In practice, 50% of the constituting pixels must be included into the ROI to label a superpixel as part of the object in I ref . Label propagation can be then easily done at the super-These findings are illustrated visually Fig.5 for the pair {I 1 , I 34 } of swan.

					Dice						F-measure
		spx matching RGBm RGBh	PM[9]	optical flow Far[8] SF[37]	proposed DIR kNN RF	spx matching RGBm RGBh	PM[9]	optical flow Far[8] SF[37]	proposed DIR kNN RF
	bag	67.1	87.5	11.6	11.8	11.0	96.9	92.8	59.3	82.6	10.7	10.1	10.6	97.8	89.0
	fsh3	89.8	89.7	23.8	16.7	15.2	64.8	89.4	91.9	91.4	27.4	21.4	18.7	67.3	81.5
	octo	25.6	47.6	98.6	96.5	96.4	84.8	85.5	17.2	32.1	95.2	92.9	92.9	75.6	74.9
	lapa	65.8	67.7	70.4	61.1	48.5	89.2	87.9	57.1	57.7	53.0	51.2	51.5	86.1	85.3
	sl1.1	84.5	84.2	41.7	35.2	32.5	82.9	94.9	81.2	83.3	53.1	42.2	43.1	81.8	94.4
	sl1.2	24.5	51.2	24.3	13.4	11.8	82.0	89.4	21.6	44.6	50.4	34.1	24.4	82.4	90.3
	sl1.3	38.5	51.2	13.5	7.95	7.75	66.7	88.3	39.5	49.1	24.7	12.1	11.4	72.9	77.5
	swan	91.0	89.2	78.5	84.9	83.9	90.5	91.3	87.4	83.9	62.6	70.7	69.2	85.7	88.0
	bear	81.5	73.8	79.7	85.6	84.0	84.3	89.1	57.7	46.5	56.3	64.5	65.9	65.2	76.3
	camel	58.5	49.3	85.2	85.2	80.5	67.5	70.3	45.6	38.2	72.4	64.3	67.2	56.4	59.3
	cows	64.2	67.5	89.6	89.6	84.4	16.8	87.1	32.2	36.9	77.3	68.3	60.4	15.5	69.8
	flam	48.2	52.4	77.5	77.2	66.3	70.0	76.2	45.6	46.7	55.6	48.8	45.9	56.3	67.2
	avg	61.6	67.6	57.9	54.2	51.9	74.7	86.9	53.0	57.8	53.2	48.4	46.8	70.3	80.3
			optical flow methods do not globally outperform unsupervised learning-based
			and superpixel-to-superpixel matching with averaged Dice (F-measure) of 57.9
			(53.2), 54.2 (48.4) and 51.9 (46.8) for PM, Farnebäck and SIFT Flow.
			Red and green boundaries denote propagated and GT ROI location. We can
			notice that PM, Farnebäck and SIFT Flow under-estimate the area covered
			by the swan, especially for the neck and near the water. RGBm, RGBh as

pixel level once to-the-reference superpixel pairings are obtained. Conversely, PM and optical flow estimators use dense to-the-reference fields to propagate labels at the pixel level from I ref to the whole sequence. Dice and F-measure scores temporally averaged across each of the previously described sequences are given Tab.2 for each method. Bold and underline results indicate first and second best scores. Results indicate a good matching accuracy reached using the proposed unsupervised learning-based strategy for both consecutive and distant frames. On average, RF-based superpixel pairings provide the best direct tracking results with Dice and F-measure of 86.9 and 80.3, followed by k-NN-based results which reach 74.7 and 70.3. Both methods are significantly superior to the other existing approaches. Averaged Dice (F-measure) goes down to 67.6 (57.8) and 61.6 (53.0) for RGBh and RGBm respectively. Despite fairly good scores for octo, caml, cows and flam, PM and

Table 2 :

 2 Dice and F-measure scores for ROI tracking across 10 sequences using direct integra-

tion (DIR), i.e. direct processing of image pairs {I ref , In}. Four methodologies are compared:

  3.1). kNN and RF-based elementary multi-step superpixel matches are obtained with steps {1, 2, 5, 10, 20} for octo, swan and sle1 and {1, 2, 5, 10, 20, 30, 50} for longer sequences (fsh3, bag, lapa, bear, caml, cows

				Dice					F-measure					consistency
			kNN			RF			kNN			RF			kNN			RF
		DIR	SEQ	MSI	DIR	SEQ	MSI	DIR	SEQ	MSI	DIR	SEQ	MSI	DIR	SEQ	MSI	DIR	SEQ	MSI
	bag	96.9	97.7	97.7	92.8	74.9	92.9	97.8	99.7	99.4	89.0	65.9	88.3	39.2	12.1	32.1	30.2	12.2	27.8
	fsh3	64.8	90.5	85.7	89.4	91.1	92.0	67.3	92.6	81.5	91.8	93.9	95.4	56,5	37,6	74,3	69.5	38.8	58.7
	octo	84.8	91.3	93.0	85.5	91.3	92.8	75.6	83.6	86.3	74.9	83.6	86.0	84,6	67,5	76,6	82.9	67,1	75,7
	lapa	89.2	86.5	92.8	87.9	88.3	92.8	86.1	86.2	95.0	85.3	86.8	94.8	96.1	68.9	96.8	90.2	64.6	91.5
	sl1.1	82.9	95.3	95.6	94.9	95.5	95.9	81.8	93.2	94.8	94.4	94.0	95.5	89,2	63,4	96,8	94,5	76,1	93,4
	sl1.2	82.0	80.9	90.3	89.4	80.5	90.9	82.4	77.9	93.0	90.3	73.1	93.9	76.0	62,4	91,6	84,9	61,6	91,2
	sl1.3	66.7	92.4	92.2	88.3	92.4	94.2	72.9	76.7	78.1	77.5	76.7	81.6	86.7	78.1	95.0	87.7	75.6	100
	swan	90.5	85.0	93.5	91.3	86.9	93.4	85.7	73.9	93.6	88.0	78.4	93.8	86.1	57.7	83.0	82.3	59.3	77.7
	bear	84.3	87.8	87.7	89.1	87.0	92.5	65.2	68.2	72.2	76.3	68.9	82.1	65.6	61.2	68.5	62.7	62.1	67.2
	camel	67.5	76.6	76.9	70.3	77.7	79.6	56.4	64.7	67.2	59.3	63.3	69.1	58.7	41.5	51.8	63.8	42.3	54.7
	cows	16.8	80.1	87.3	87.1	79.1	89.1	15.5	59.6	66.8	69.8	61.4	74.1	15.5	39.6	68.5	75.8	39.3	67.6
	flam	70.0	63.5	76.7	76.2	66.5	80.8	56.3	62.5	67.1	67.2	62.8	71.9	55.4	34.9	63.4	43.0	39.7	55.0
	avg	74.7	85.6	89.1	86.9	84.3	90.6	70.3	78.2	82.9	80.3	75.7	85.5	67.5	52.1	74.9	72.3	53.2	71.7
				and flam). Context-rich features are estimated using the same parameters as
				in Sect.4.1. Majority voting (Eq.12) focuses only on superpixel candidates gen-
				erated in both to/from-the-reference directions to improve forward-backward
				consistency (see Sect.4.3 for further details).

Table 3 :

 3 Dice, F-measure and consistency scores for ROI tracking across 10 sequences. We compare direct (DIR), sequential (SEQ) and multi-step (MSI) integration based on unsupervised learning-based superpixel matching using kNN and RF. Bold results indicate the best performance between DIR, SEQ and MSI. Underline scores highlight best results between kNN and RF-based methods. Tab.3 presents temporally averaged metrics (Dice, F-measure and consistency scores) obtained by DIR, SEQ and MSI across all sequences using kNN and RF. Except for consistency scores when relying on RF, Tab.3 confirms that MSI is the best integration strategy towards long-term superpixel tracking compared to DIR and SEQ. For instance, RF and kNN-based MSI reach the highest Dice scores with 90.6 and 89.1 in comparison to 84.3 (86.9) and 85.6 (74.7) obtained with RF and kNN-based SEQ (DIR). Second and third positions in terms of Dice and F-measure vary depending on the classifier. SEQ outperforms DIR for kNN whereas RF exhibits the opposite behavior. Except for MSI in terms of consistency and SEQ for F-measure, another main finding is that RF-based elementary matches usually make better long-term tracking than kNN-based pairings. This suggests that RF is more able to deal with a large amount of classes in this context. Despite a slightly poorer management of context-rich features, kNN

Table 4 :

 4 for the {I 1 , I 34 } swan pair confirms the ability of RF-based MSI to reach accurate long-term correspondences (see for instance the swan beak). In such maps, a given object part must keep the same color between training and prediction in case of correct matching. Improvements from RF-based DIR to MSI can be noticed for both low-textured areas (as in the water) and highly-textured ones. Dice, F-measure and consistency scores for ROI tracking across 10 sequences. Based on RF-based superpixel elementary matches, we compare three different superpixel candidate generation strategies for multi-step (MSI) integration using: only direct candidates (MSId), direct and reverse candidates (MSIr), only candidates generated in both direct and reverse directions (MSIm). Bold results indicate the best performance.

	4.3. Long-term candidate generation
	We propose to perform a more in-depth study of multi-step integration by
	comparing different long-term candidate generation strategies in terms of track-
	ing accuracy. As described Sect.3.2, long-term superpixel candidates can be


Claire Filloux

Hélène Géniaux

Jacques Buxeraud
email: jacques.buxeraud@unilim.fr

Praticien Pharmacien

Hélène Attaché

Géniaux

Dochead pratique

Keywords: effet indésirable, fluoroquinolone, iatrogénie, pharmacovigilance

Les quinolones sont aujourd'hui principalement représentées par les fluoroquinolones : la norfloxacine, l'ofloxacine, la lévofloxacine, la ciprofloxacine, la moxifloxacine et la loméfloxacine. Les quinolones sont des inhibiteurs de l'ADN gyrase (topoisomérase de type II) et de l'ADN topoisomérase de type IV. La cible principale est la gyrase chez les bactéries à Gram négatif alors que l'action sur les germes à Gram positif est plus particulièrement liée à la topoisomérase IV. Dans les deux cas, les quinolones induisent l'inhibition de la réplication et de la transcription de l'ADN bactérien.

T1 Les indications

TEG1 Les fluoroquinolones peuvent être administrées par voie systémique ou par voie inhalée dans de nombreuses indications [START_REF] Antibioclic | Antibiothérapie rationnelle en soins primaires[END_REF][START_REF] Popi | Maladies infectieuses et tropicales. Guide de traitement[END_REF][START_REF]Base de données publique des médicaments[END_REF]. Dans certaines infections, elles sont également utilisées par voie oculaire et auriculaire.

TEG1 Les infections urinaires : infections urinaires compliquées ou pyélonéphrites et infections urinaires basses masculines (anciennement prostatites).

TEG1 Les infections sexuellement transmissibles et infections gynécologiques (pas nécessairement en première intention) : infections gonococciques non compliquées localisées, chancre mou, les infections génitales non gonococciques.

TEG1 Les infections gastro-intestinales : diarrhées infectieuses aiguës, fièvre typhoïde, salmonelloses et autres infections intra-abdominales TEG1 Les infections de la sphère ORL : otites externes, sinusites bactériennes.

TEG1 Les infections ostéo-articulaires, dans lesquelles les fluoroquinolones sont parfois utilisées du fait de leur bonne diffusion osseuse. Infections ostéo-articulaires (ostéomyélite, ostéite chronique, infection sur prothèses, arthrite, spondylodiscite) à staphylocoque sont, dans certaines circonstances (allergie) et en association, traitées par ofloxacine. Lorsque l'espèce en cause est Pseudomonas aeruginosa, la ciprofloxacine peut être utilisée en association avec une céphalosporine ou un carbapénème. Les durées de traitement sont souvent de plusieurs mois.

TEG1 Les infections bronchopulmonaires : surinfections des bronchopathies chroniques obstructives (BPCO), pneumopathies communautaires, pneumopathies nosocomiales, infections associées à la mucoviscidose.

T1 Des précautions à respecter

TEG1 Le recours aux fluoroquinolones doit être limité au maximum en raison des dommages écologiques collatéraux observés. Les indications validées sont nombreuses, mais l'usage de ces médicaments tend à être restreint en raison de l'augmentation des résistances acquises, de la pression de sélection exercée et de l'usage potentiel de certaines fluoroquinolones en troisième ligne dans la tuberculose [START_REF] Popi | Maladies infectieuses et tropicales. Guide de traitement[END_REF].

TEG1 Des contre-indications doivent être respectées [START_REF]Base de données publique des médicaments[END_REF]. Ainsi, de manière générale, le prescripteur veillera à ne pas utiliser ces médicaments dans certains cas :

• hypersensibilité aux fluoroquinolones ;

• épilepsie, en raison du risque d'abaissement du seuil épileptogène ;

• antécédents de tendinopathie liée à leur administration ;

• enfants ou adolescents en période de croissance ;

• femmes enceintes ou allaitantes, bien que dans certaines situations très spécifiques, l'utilisation des fluoroquinolones soit possible, en seconde intention, dans la pyélonéphrite ou la cystite aiguë gravidique (la ciprofloxacine, voire la norfloxacine seront préférées en raison d'un plus recul clinique) [START_REF] Jonville-Béra | Médicaments et grossesse : prescrire et évaluer le risque[END_REF][5] ;

• déficit en glucose-6-phosphate déshydrogénase (G6PD) pour certaines fluoroquinolones.

La moxifloxacine, du fait de sa toxicité cardiaque, est contre-indiquée chez les patients présentant un allongement de l'espace QT, des troubles hydro-électrolytiques, une bradycardie, une insuffisance cardiaque avec fraction d'éjection ventriculaire gauche réduite, et des antécédents de troubles du rythme.

Compte tenu de données cliniques limitées, la moxifloxacine est également contre-indiquée chez les patients insuffisants hépatiques sévères (Child-Pugh C) et chez ceux ayant des valeurs de transaminases cinq fois supérieures à la normale.

T1 Les effets indésirables

Leurs effets indésirables nombreux parfois graves doivent être bien connus et mis en balance avec les bénéfices attendus (encadré 1) [START_REF]sante[END_REF].

Encadré 1. Une mise en garde de l'ANSM

Dans une lettre adressée aux professionnels de santé datant du 10 avril 2019, l'Agence nationale de sécurité du médicament et des produits de santé (ANSM) attire leur attention sur les fluoroquinolones : « Antibiotiques de la famille des quinolones et fluoroquinolones administrés par voie systémique ou inhalée : risque d'effets indésirables invalidants, durables et potentiellement irréversibles et restrictions d'utilisation » [START_REF]sante[END_REF].

Sur 2 colonnes près de son appel (une seule colonne de texte à l'intérieur)

TEG1 Les atteintes tendineuses, qui peuvent aller jusqu'à une rupture du tendon d'Achille (encadré 2), constituent l'un des effets indésirables majeurs des fluoroquinolones [START_REF] Neumayer | Diagnostic et traitement de la rupture du tendon d'Achille[END_REF]. Elles sont liées à leur effet négatif sur la synthèse du collagène et de l'élastine par majoration de la synthèse de métalloprotéinases extracellulaires.

Encadré 2. La rupture du tendon d'Achille

TEG1 Une tendinopathie peut survenir dès les premières 48 heures de traitement par une fluoroquinolone. Touchant principalement le tendon d'Achille, elle est parfois bilatérale et atteint préférentiellement les personnes à risque (sujets âgés, corticothérapie prolongée, insuffisance rénale chronique, etc.).

TEG1 Ces tendinites peuvent conduire à une rupture tendineuse, le risque de rupture aiguë étant le plus souvent lié à un effort d'accélération brusque (activité sportive) ou au moment de la reprise de la marche d'un patient alité.

TEG1 La rupture se traduit par une douleur vive du talon et du mollet, un gonflement de la face postérieure de la cheville, une impossibilité ou grande difficulté à marcher. Une dépression marquée peut de former au-dessus du talon. Un hématome peut parfois s'installer progressivement.

TEG1 En cas d'apparition de cet effet indésirable, il est indispensable d'arrêter le traitement et de mettre au repos le ou les tendons atteints.

Sur 2 colonnes près de son appel (une seule colonne de texte à l'intérieur)

TEG1 Un risque vasculaire et rétinien a été mis en exergue (encadré 3). Plusieurs études récentes ont montré que l'altération du collagène par les fluoroquinolones pourrait avoir, en effet, d'autres cibles vasculaires à composante conjonctive et élastique (aorte, artères cérébrales, artères des membres). Après leur utilisation per os, il a été observé un risque accru de décollement rétinien, de même que des anévrismes et dissections de l'aorte. L'incidence cumulée des anévrismes et dissections de l'aorte rapportée à une semaine de traitement par une fluoroquinolone a été de 2,3 pour 100 000 patients traités versus 1,3 pour 100 000 patients sous amoxicilline avec un risque relatif estimé de 1,7 (intervalle de confiance [IC] à 95 % : 1,1 à 2,5). Sur la totalité des cas rapportés, 40 % sont survenus dans les dix premiers jours de traitement [START_REF]Fluoroquinolones : anévrismes et dissections de l'aorte[END_REF].

Encadré 3. Qu'est-ce qu'un décollement de rétine ?

Un décollement de rétine fait suite, dans la grande majorité des cas, à une déchirure qui permet au liquide intraoculaire de s'infiltrer et de soulever la rétine de proche en proche. Le décollement provoque des troubles de la vision de l'oeil atteint. Il existe un risque non négligeable d'atteinte de l'autre oeil. Dès les premiers signes, il est important de consulter un ophtalmologiste. Un traitement précoce permet d'éviter l'aggravation. L'oeil atteint n'est ni rouge, ni douloureux. La survenue de certains symptômes inexistants jusqu'alors doit alerter :

• apparition, dans le champ visuel, de points lumineux, diffus, plus ou moins fixes, ou d'un éclair de lumière, généralement bleuté, unique, fixe, présent toujours au même endroit, qui persiste yeux fermés ou dans le noir ;

• vision de mouches volantes ou de pluie de suie (pluie de très fines particules) ;

• présence d'une ombre prononcée ou d'un voile noir en bordure du champ de vision ;

• vision floue ou embuée puis une baisse de la vision qui peut être rapide ;

• perte de la vision d'un seul oeil concernant le centre ou la périphérie du champ de vision.

Sur 2 colonnes près de son appel (une seule colonne de texte à l'intérieur)

TEG1 Une petite série d'observations d'insuffisances rénales aiguës imputées à l'association de ciprofloxacine (ou autre quinolone) avec un inhibiteur de l'enzyme de conversion (IEC), a été rapportée, au début de l'année 2018, par le Centre de pharmacovigilance de l'Organisation mondiale de la santé à Uppsala, en Suède. Les patients traités au long cours par un IEC avaient des facteurs de risque d'insuffisance rénale, mais celle-ci n'est apparue qu'après le début du traitement par fluoroquinolone. Une étude cas/témoins effectuée chez des hommes âgés de 40 à 85 ans, a montré un risque d'insuffisance rénale deux fois plus grand chez les patients exposés à une fluoroquinolone par rapport aux témoins, et 4,5 fois plus grand quand elle est associée à un IEC (IC à 95 % : 2,8 à 7,0). La prise d'une fluoroquinolone potentiellement néphrotoxique nécessite une surveillance accrue de la fonction rénale quand elle est associée à des médicaments exposant à des insuffisances rénales fonctionnelles comme les diurétiques, les anti-inflammatoires non stéroïdiens (AINS), les IEC et les sartans [START_REF]Fluoroquinolone + IEC : insuffisances rénales aiguës[END_REF]. TEG1 D'autres effets indésirables peuvent parfois survenir, notamment en cas de prédisposition. Il s'agit de convulsions, manifestations neuropsychiatriques (perturbations de l'attention, désorientation, agitation, nervosité, pertes de mémoire, délire) pouvant, dans de rares cas, conduire à des suicides, parfois même après une seule prise, des troubles de la glycémie, etc. [START_REF]Profil de sécurité des quinolones administrées par voie générale[END_REF][START_REF]FDA updates warnings for fluoroquinolone antibiotics on risks of mental health and low blood sugar adverse reactions[END_REF][START_REF]Quinolones et fluoroquinolones : retour d'information sur le PRAC de juin[END_REF].

Encadré

Retrouver les coordonnées du CRPV

Pour retrouver les coordonnées du centre régional de pharmacovigilance (CRPV) dont l'officine dépend, il convient de consulter la carte interactive (www.rfcrpv.fr/contacter-votre-crpv/) ou le site de l'Agence nationale de sécurité du médicament et des produits de santé (ANSM) (http://ansm.sante.fr/Declarer-un-effet-indesirable/Pharmacovigilance/Centres-regionaux-depharmacovigilance/(offset)/4).

En dernière page sur 2 colonnes en haut à gauche T1 Conclusion

Toutes les manifestations attribuées à l'administration des fluoroquinolones ont conduit la Food and Drug Administration (FDA) à renforcer les recommandations de leur utilisation dès mai 2016 [START_REF]FDA updates warnings for fluoroquinolone antibiotics on risks of mental health and low blood sugar adverse reactions[END_REF]. Il a été préconisé de restreindre leur indication et de ne les utiliser qu'en l'absence d'alternatives thérapeutiques pour traiter des infections peu sévères telles que les sinusites et aiguës et certaines infections urinaires [START_REF] Antibioclic | Antibiothérapie rationnelle en soins primaires[END_REF][START_REF] Popi | Maladies infectieuses et tropicales. Guide de traitement[END_REF]. En juillet 2018, cette même FDA a demandé que figurent sur les notices des antibiotiques à base de fluoroquinolones, des avertissements sur les effets indésirables neuropsychiatriques et les hypoglycémies.

Dans le cadre d'un arbitrage européen, le Comité pour l'évaluation des risques en matière de pharmacovigilance de l'Agence européenne (PRAC), au cours de la séance d'octobre 2018 a clôturé les discussions par l'existence d'un risque d'effets indésirables durables, handicapants et potentiellement irréversibles touchant principalement les muscles, les articulations et le système nerveux. Le PRAC, comme l'avait fait la FDA a recommandé une restriction des indications des fluoroquinolones [START_REF]Quinolones et fluoroquinolones : retour d'information sur le PRAC de juin[END_REF].

Points à retenir

• Les atteintes tendineuses, pouvant aller jusqu'à une rupture du tendon d'Achille, représentent l'un des effets indésirables les plus connus avec les fluoroquinolones.

• Après utilisation de fluoroquinolones per os, il a été observé un risque accru de décollement rétinien, de même que des anévrismes et des dissections de l'aorte.

• La prise d'une fluoroquinolone potentiellement néphrotoxique nécessite une surveillance accrue de la fonction rénale quand elle est associée à d'autres médicaments néphrotoxiques.

• Au niveau européen, il a été préconisé de restreindre l'indication des fluoroquinolones et de ne les utiliser qu'en l'absence d'alternatives thérapeutiques pour traiter des infections peu sévères.

• La Food and Drug Administration (FDA) a demandé que figurent sur les notices des antibiotiques à base de fluoroquinolones des avertissements sur les effets indésirables neuropsychiatriques et les hypoglycémies.

Sur 2 colonnes à droite après la puce de fin

Références