Clément Farabet

Camille Couprie

Laurent Najman

Yann Lecun

Cl Ément Farabet

Learning Hierarchical Features for Scene Labeling

Keywords: Convolutional networks, deep learning, image segmentation, image classification, scene parsing. ✦

published or not. The documents may come   L'archive ouverte pluridisciplinaire

INTRODUCTION

I MAGE UNDERSTANDING is a task of primary impor- tance for a wide range of practical applications. One important step towards understanding an image is to perform a full-scene labeling also known as a scene parsing, which consists in labeling every pixel in the image with the category of the object it belongs to. After a perfect scene parsing, every region and every object is delineated and tagged. One challenge of scene parsing is that it combines the traditional problems of detection, segmentation, and multi-label recognition in a single process.

There are two questions of primary importance in the context of scene parsing: how to produce good internal representations of the visual information, and how to use contextual information to ensure the self-consistency of the interpretation.

This paper presents a scene parsing system that relies on deep learning methods to approach both questions. The main idea is to use a convolutional network [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] operating on a large input window to produce label hypotheses for each pixel location. The convolutional net is fed with raw image pixels (after band-pass filtering and contrast normalization), and trained in supervised mode from fully-labeled images to produce a category for each pixel location. Convolutional networks are composed of multiple stages each of which contains a filter bank module, a non-linearity, and a spatial pooling module. With end-to-end training, convolutional networks can automatically learn hierarchical feature representations.

Unfortunately, labeling each pixel by looking at a small region around it is difficult. The category of a pixel may depend on relatively short-range information (e.g. ). E-mails: cfarabet@cs.nyu.edu, ccouprie@cs.nyu.edu, l.najman@esiee.fr, yann@cs.nyu.edu the presence of a human face generally indicates the presence of a human body nearby), but may also depend on long-range information. For example, identifying a grey pixel as belonging to a road, a sidewalk, a gray car, a concrete building, or a cloudy sky requires a wide contextual window that shows enough of the surroundings to make an informed decision. To address this problem, we propose to use a multi-scale convolutional network, which can take into account large input windows, while keeping the number of free parameters to a minimum.

Common approaches to scene parsing first produce segmentation hypotheses using graph-based methods. Candidate segments are then encoded using engineered features. Finally, a conditional random field (or some other type of graphical model), is trained to produce labels for each candidate segment, and to ensure that the labelings are globally consistent.

A striking characteristic of the system proposed here is that the use of a large contextual window to label pixels reduces the requirement for sophisticated postprocessing methods that ensure the consistency of the labeling.

More precisely, the proposed scene parsing architecture is depicted on Figure 1. It relies on two main components:

1) Multi-scale, convolutional representation: our multi-scale, dense feature extractor produces a series of feature vectors for regions of multiple sizes centered around every pixel in the image, covering a large context. The multi-scale convolutional net contains multiple copies of a single network (all sharing the same weights) that are applied to different scales of a Laplacian pyramid version of the input image. For each pixel, the networks collectively encode the information present in a large contextual window around the given pixel (184 × 184 pixels in the system described here). The convolutional network is fed with raw pixels and trained end to end, thereby alleviating the need for hand-engineered features. When properly trained, these features produce a representation that captures texture, shape and contextual information. While using a multiscale representation seems natural for FSL, it has rarely been used in the context of feature learning systems. The multiscale representation that is learned is sufficiently complete to allow the detection and recognition of all the objects and regions in the scene. However, it does not accurately pinpoint the boundaries of the regions, and requires some post-processing to yield cleanly delineated predictions.

2) Graph-based classification:

An over-segmentation is constructed from the image, and is used to group the feature descriptors. Several over-segmentations are considered, and three techniques are proposed to produce the final image labeling.

2.a. Superpixels:

The image is segmented into disjoint components, widely over-segmenting the scene. In this scenario, a pixelwise classifier is trained on the convolutional feature vectors, and a simple vote is done for each component, to assign a single class per component. This method is simple and effective, but imposes a fixed level of segmentation, which can be suboptimal.

2.b. Conditional random field over superpixels: a conditional random field is defined over a set of superpixels. Compared to the previous, simpler method, this post-processing models joint probabilities at the level of the scene, and is useful to avoid local aberrations (e.g. a person in the sky). That kind of approach is widely used in the computer vision community, and we show that our learned multiscale feature representation essentially makes the use of a global random field much less useful: most scene-level relationships seem to be already captured by it.

2.c. Multilevel cut with class purity criterion:

A family of segmentations is constructed over the image to analyze the scene at multiple levels. In the simplest case, this family might be a segmentation tree; in the most general case it can be any set of segmentations, for example a collection of superpixels either produced using the same algorithm with different parameter tunings or produced by different algorithms. Each segmentation component is represented by the set of feature vectors that fall into it: the component is encoded by a spatial grid of aggregated feature vectors. The aggregated feature vector of each grid cell is computed by a component-wise max pooling of the feature vectors centered on all the pixels that fall into the grid cell. This produces a scale-invariant representation of the segment and its surrounding. A classifier is then applied to the aggregated feature grid of each node. This classifier is trained to estimate the histogram of all object categories present in the component. A subset of the components is then selected such that they cover the entire image. These components are selected so as to minimize the average "impurity" of the class distribution in a procedure that we name "optimal cover". The class "impurity" is defined as the entropy of the class distribution. The choice of the cover thus attempts to find a consistent overall segmentation in which each segment contains pixels belonging to only one of the learned categories. This simple method allows us to consider full families of segmentation components, rather than a unique, predetermined segmentation (e.g. a single set of superpixels).

All the steps in the process have a complexity linear (or almost linear) in the number of pixels. The bulk of the computation resides in the convolutional network feature extractor. The resulting system is very fast, producing a full parse of a 320 × 240 image in less than a second on a conventional CPU, and in less than 100ms using dedicated hardware, opening the door to real-time applications. Once trained, the system is parameter free, and requires no adjustment of thresholds or other knobs.

An early version of this work was first published in [START_REF] Farabet | Scene parsing with multiscale feature learning, purity trees, and optimal covers[END_REF]. This journal version reports more complete experiments, comparisons and higher results.

RELATED WORK

The scene parsing problem has been approached with a wide variety of methods in recent years. Many methods rely on MRFs, CRFs, or other types of graphical models to ensure the consistency of the labeling and to account for context [START_REF] He | Learning hybrid models for image annotation with partially labeled data[END_REF], [39], [START_REF] Gould | Decomposing a scene into geometric and semantically consistent regions[END_REF], [START_REF] Kumar | Efficiently selecting regions for scene understanding[END_REF], [START_REF] Munoz | Stacked hierarchical labeling[END_REF], [START_REF] Tighe | Superparsing: scalable nonparametric image parsing with superpixels[END_REF], [START_REF] Lempitsky | A pylon model for semantic segmentation[END_REF]. Most methods rely on a pre-segmentation into superpixels or other segment candidates, and extract features and categories from individual segments and from various combinations of neighboring segments. The graphical model inference pulls out the most consistent set of segments which covers the image. [START_REF] Socher | Parsing Natural Scenes and Natural Language with Recursive Neural Networks[END_REF] proposed a method to aggregate segments in a greedy fashion using a trained scoring function. The originality of the approach is that the feature vector of the combination of two segments is computed from the feature vectors of the individual segments through a trainable function. Like us, they use "deep learning" methods to train their feature extractor. But unlike us, their feature extractor operates on hand-engineered features.

One of the main question in scene parsing is how to take a wide context into account to make a local decision. [START_REF] Munoz | Stacked hierarchical labeling[END_REF] proposed to use the histogram of labels extracted from a coarse scale as input to the labeler that looks at finer scales. Our approach is somewhat simpler: our feature extractor is applied densely to an image pyramid. The coarse feature maps thereby generated are upsampled to match that of the finest scale. Hence with three scales, each feature vector has multiple fields which encode multiple regions of increasing sizes and decreasing resolutions, centered on the same pixel location.

Like us, a number of authors have used families of segmentations or trees to generate candidate segments by aggregating elementary segments. The approaches of [39], [START_REF] Lempitsky | A pylon model for semantic segmentation[END_REF] rely on inference algorithms based on Graph Cuts to label images using trees of segmentation. Other strategies using families of segmentations appeared in [36], [5]. None of the previous strategies for scene labeling used a purity criterion on the class distributions. Combined to the optimal cover strategy, this purity criterion is general, efficient and could be applied to solve different problems.

Contrary to the previously cited approaches using engineered features, our system extracts features densely from a multiscale pyramid of images using a convolutional network (ConvNet) [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. These networks can be fed with raw pixels and can automatically learn lowlevel and mid-level features, alleviating the need for Each scale is fed to a 3-stage convolutional network, which produces a set of feature maps. The feature maps of all scales are concatenated, the coarser-scale maps being upsampled to match the size of the finest-scale map. Each feature vector thus represents a large contextual window around each pixel. In parallel, a single segmentation (i.e. superpixels), or a family of segmentations (e.g. a segmentation tree) are computed to exploit the natural contours of the image. The final labeling is produced from the feature vectors and the segmentation(s) using different methods, as presented in section 4.

!! !" !# !# "#$%$&' !!$ " !(!!%!& !' !("!%"& "' !(#!%#& #' )*+,-./ 0!%' 1! 1" 1# 1( 1) 1* 1+ 1, 1- 2&0-&$','.#$ 3!%'
hand-engineered features. One of their advantage is the ability to compute dense features efficiently over large images. They are best known for their applications to detection and recognition [START_REF] Vaillant | Original approach for the localisation of objects in images[END_REF], [START_REF] Garcia | Convolutional face finder: A neural architecture for fast and robust face detection[END_REF], [START_REF] Osadchy | Synergistic face detection and pose estimation with energy-based models[END_REF], [21], but they have also been used for image segmentation, particularly for biological image segmentation [START_REF] Ning | Toward automatic phenotyping of developing embryos from videos[END_REF], [START_REF] Jain | Supervised learning of image restoration with convolutional networks[END_REF], [START_REF] Turaga | Maximin affinity learning of image segmentation[END_REF]. The only previously published work on using convolutional networks for scene parsing is that of [START_REF] Grangier | Deep Convolutional Networks for Scene Parsing[END_REF]. While somewhat preliminary, their work showed that convolutional networks fed with raw pixels could be trained to perform scene parsing with decent accuracy. Unlike [START_REF] Grangier | Deep Convolutional Networks for Scene Parsing[END_REF] however, our system uses a boundary-based hierarchy of segmentations to align the labels produced by the network to the boundaries in the image and thus produces representations that are independent of the size of the segments through feature pooling. Slightly after [START_REF] Farabet | Scene parsing with multiscale feature learning, purity trees, and optimal covers[END_REF], Schulz and Behnke proposed a similar architecture of a multiscale convolutional network for scene parsing [START_REF] Schulz | Learning object-class segmentation with convolutional neural networks[END_REF]. Unlike us, they use pairwise class location filters to predict the final segmentation, instead of using the image gradient that we found to be more accurate.

MULTISCALE FEATURE EXTRACTION FOR SCENE PARSING

The model proposed in this paper, depicted on Figure 1, relies on two complementary image representations. In the first representation, an image patch is seen as a point in R P , and we seek to find a transform f : R P → R Q that maps each patch into R Q , a space where it can be classified linearly. This first representation typically suffers from two main problems when using a classical convolutional network, where the image is divided following a grid pattern: (1) the window considered rarely contains an object that is properly centered and scaled, and therefore offers a poor observation basis to predict the class of the underlying object, (2) integrating a large context involves increasing the grid size, and therefore the dimensionality P of the input; given a finite amount of training data, it is then necessary to enforce some invariance in the function f itself. This is usually achieved by using pooling/subsampling layers, which in turn degrades the ability of the model to precisely locate and delineate objects. In this paper, f is implemented by a multiscale convolutional network, which allows integrating large contexts (as large as the complete scene) into local decisions, yet still remaining manageable in terms of parameters/dimensionality. This multiscale model, in which weights are shared across scales, allows the model to capture long-range interactions, without the penalty of extra parameters to train. This model is described in Section 3.1.

In the second representation, the image is seen as an edge-weighted graph, on which one or several oversegmentations can be constructed. The components are spatially accurate, and naturally delineate the underlying objects, as this representation conserves pixel-level precision. Section 4 describes multiple strategies to combine both representations. In particular, we describe in Section 4.3 a method for analyzing a family of segmentations (at multiple levels). It can be used as a solution to the first problem exposed above: assuming the capability of assessing the quality of all the components in this family of segmentations, a system can automatically choose its components so as to produce the best set of predictions.

Scale-invariant, scene-level feature extraction

Good internal representations are hierarchical. In vision, pixels are assembled into edglets, edglets into motifs, motifs into parts, parts into objects, and objects into scenes. This suggests that recognition architectures for vision (and for other modalities such as audio and natural language) should have multiple trainable stages stacked on top of each other, one for each level in the feature hierarchy. Convolutional Networks (ConvNets) provide a simple framework to learn such hierarchies of features.

Convolutional Networks [START_REF] Lecun | Handwritten digit recognition with a back-propagation network[END_REF], [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] are trainable architectures composed of multiple stages. The input and output of each stage are sets of arrays called feature maps. For example, if the input is a color image, each feature map would be a 2D array containing a color channel of the input image (for an audio input each feature map would be a 1D array, and for a video or volumetric image, it would be a 3D array). At the output, each feature map represents a particular feature extracted at all locations on the input. Each stage is composed of three layers: a filter bank layer, a non-linearity layer, and a feature pooling layer. A typical ConvNet is composed of one, two or three such 3-layer stages, followed by a classification module. Because they are trainable, arbitrary input modalities can be modeled, beyond natural images.

Our feature extractor is a three-stage convolutional network. The first two stages contain a bank of filters producing multiple feature maps, a point-wise nonlinear mapping and a spatial pooling followed by subsampling of each feature map. The last layer only contains a bank of filters. The filters (convolution kernels) are subject to training. Each filter is applied to the input feature maps through a 2D convolution operation, which detects local features at all locations on the input. Each filter bank of a convolutional network produces features that are equivariant under shifts, i.e. if the input is shifted, the output is also shifted but otherwise unchanged.

While convolutional networks have been used successfully for a number of image labeling problems, imagelevel tasks such as full-scene understanding (pixel-wise labeling, or any dense feature estimation) require the system to model complex interactions at the scale of complete images, not simply within a patch. To view a large contextual window at full resolution, a convolutional network would have to be unmanageably large.

The solution is to use a multiscale approach. Our multiscale convolutional network overcomes these limitations by extending the concept of spatial weight replication to the scale space. Given an input image I, a multiscale pyramid of images X s , ∀s ∈ {1, . . . , N } is constructed, where X 1 has the size of I. The multiscale pyramid can be a Laplacian pyramid, and is typically pre-processed, so that local neighborhoods have zero mean and unit standard deviation. Given a classical convolutional network f s with parameters θ s , the multiscale network is obtained by instantiating one network per scale s, and sharing all parameters across scales:

θ s = θ 0 , ∀s ∈ {1, . . . , N }.
We introduce the following convention: banks of images will be seen as three dimensional arrays in which the first dimension is the number of independent feature maps, or images, the second is the height of the maps and the third is the width. The output state of the L-th stage is denoted H L .

The maps in the pyramid are computed using a scaling/normalizing function g s as X s = g s (I), for all s ∈ {1, . . . , N }.

For each scale s, the convolutional network f s can be described as a sequence of linear transforms, interspersed with non-linear symmetric squashing units (typically the tanh function [START_REF] Lecun | Efficient backprop[END_REF]), and pooling/subsampling operators. For a network f s with L layers, we have:

f s (X s ; θ s ) = W L H L-1 , (1)
where the vector of hidden units at layer l is

H l = pool(tanh(W l H l-1 + b l )) (2)
for all l ∈ {1, . . . , L -1}, with b l a vector of bias parameters, and H 0 = X s . The matrices W l are Toeplitz matrices, therefore each hidden unit vector H l can be expressed as a regular convolution between kernels from W l and the previous hidden unit vector H l-1 , squashed through a tanh, and pooled spatially. More specifically,

H lp = pool(tanh(b lp + q∈parents(p) w lpq * H l-1,q )). (3)
The filters W l and the biases b l constitute the trainable parameters of our model, and are collectively denoted θ s . The function tanh is a point-wise non-linearity, while pool is a function that considers a neighborhood of activations, and produces one activation per neighborhood. In all our experiments, we use a max-pooling operator, which takes the maximum activation within the neighborhood. Pooling over a small neighborhood provides built-in invariance to small translations.

Finally, the outputs of the N networks are upsampled and concatenated so as to produce F, a map of feature vectors of size N times the size of f 1 , which can be seen as local patch descriptors and scene-level descriptors

F = [f 1 , u(f 2 ), . . . , u(f N )], (4)
where u is an upsampling function.

As mentioned above, weights are shared between networks f s . Intuitively, imposing complete weight sharing across scales is a natural way of forcing the network to learn scale invariant features, and at the same time reduce the chances of over-fitting. The more scales used to jointly train the models f s (θ s ) the better the representation becomes for all scales. Because image content is, in principle, scale invariant, using the same function to extract features at each scale is justified.

Learning discriminative scale-invariant features

As described in Section 3.1, feature vectors in F are obtained by concatenating the outputs of multiple networks f s , each taking as input a different image in a multiscale pyramid.

Ideally a linear classifier should produce the correct categorization for all pixel locations i, from the feature vectors F i . We train the parameters θ s to achieve this goal, using the multiclass cross entropy loss function. Let ĉi be the normalized prediction vector from the linear classifier for pixel i. We compute normalized predicted probability distributions over classes ĉi,a using the softmax function, i.e.

where w is a temporary weight matrix only used to learn the features. The cross entropy between the predicted class distribution ĉ and the target class distribution c penalizes their deviation and is measured by

!"#$$%&'()*!+*,-$ !"#$$*.*(' /%"#0('%1%2"& #3('#4(% #!',$$ $5&('1 &*6("$ $5&('&*6("$ #'42#6
Fig. 2. First labeling strategy from the features: using superpixels as described in Section 4.1.

L cat = - i∈pixels a∈classes c i,a ln(ĉ i,a ). ( 6
)
The true target probability c i,a of class a to be present at location i can either be a distribution of classes at location i, in a given neighborhood or a hard target vector: c i,a = 1 if pixel i is labeled a, and 0 otherwise. For training maximally discriminative features, we use hard target vectors in this first stage.

Once the parameters θ s are trained, the classifier in Eq 5 is discarded, and the feature vectors F i are used using different strategies, as described in Section 4.

SCENE LABELING STRATEGIES

The simplest strategy for labeling the scene is to use the linear classifier described in Section 3.2, and assign each pixel with the argmax of the prediction at its location. More specifically, for each pixel i

l i = arg max a∈classes ĉi,a (7)
The resulting labeling l, although fairly accurate, is not satisfying visually, as it lacks spatial consistency, and precise delineation of objects. In this section, we explore three strategies to produce spatially more appealing labelings.

Superpixels

Predicting the class of each pixel independently from its neighbors yields noisy predictions. A simple cleanup can be obtained by forcing local regions of same color intensities to be assigned a single label.

As in [START_REF] Fulkerson | Class segmentation and object localization with superpixel neighborhoods[END_REF], [START_REF] Gould | Multiclass segmentation with relative location prior[END_REF], we compute superpixels, following the method proposed by [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF], to produce an oversegmentation of the image. We then classify each location of the image densely, and aggregate these predictions in each superpixel, by computing the average class distribution within the superpixel.

For this method, the pixelwise distributions dk at superpixel k are predicted from the feature vectors F i using a two-layer neural network:

y i = W 2 tanh(W 1 F i + b 1 ), (8)
di,a = e yi,a b∈classes e y i,b ,

L cat = - i∈pixels a∈classes d i,a ln( di,a ), (9)
dk,a = 1 s(k) i∈k di,a , (10)
with d i the groundtruth distribution at location i, and s(k) the surface of component k. Matrices W 1 and W 2 are the trainable parameters of the classifier. Using a twolayer neural network, as opposed to the simple linear classifier used in Section 3.2, allows the system to capture non-linear relationships between the features at different scales. In this case, the final labeling for each component k is given by

l k = arg max a∈classes dk,a . ( 12
)
The pipeline is depicted in Figure 2.

Conditional Random Fields

The local assignment obtained using superpixels does not involve a global understanding of the scene. In this section, we implement a classical CRF model, constructed on the superpixels. This is a quite standard approach for image labeling. Our multi-scale convolutional network already has the capability of modeling global relationships within a scene, but might still be prone to errors, and can benefit from a CRF, to impose consistency and coherency between labels, at test time.

A common strategy for labeling a scene consists in associating the image to a graph and define an energy function whose optimal solution corresponds to the desired segmentation [START_REF] Shotton | TextonBoost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation[END_REF], [START_REF] Fulkerson | Class segmentation and object localization with superpixel neighborhoods[END_REF].

For this purpose, we define a graph G = (V, E) with vertices v ∈ V and edges e ∈ E ⊆ V × V . Each pixel in the image is associated to a vertex, and edges are added between every neighboring nodes. An edge, e, spanning two vertices, v i and v j , is denoted by e ij . The Conditional Random Field (CRF) energy function is typically composed of a unary term enforcing the variable l to take values close to the predictions d and a pairwise term enforcing regularity or local consistency of l. The CRF energy to minimize is given by

E(l) = i∈V Φ( di , l i ) + γ eij ∈E Ψ(l i , l j ) (13)
We considered as unary terms

Φ( di,a , l i ) = exp (-α di,a )1(l i = a), (14)
where di,a corresponds to the probability of class a to be present at a pixel i computed as in Section 4.1, and 1(•) is an indicator function that equals one if the input is true, and zero otherwise.

The pairwise term consists in

Ψ(l i , l j ) = exp (-β ∇I i )1(l i = l j ) (15)
where ∇I i is the ℓ 2 norm of the gradient of the image I at a pixel i. Details on the parameters used are given in the experimental section.

The CRF energy ( 13) is minimized using alphaexpansions [4], [START_REF] Boykov | An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision[END_REF]. An illustration of the procedure appears in Figure 3.

Parameter-free multilevel parsing

One problem subsists with the two methods presented above: the observation level problem. An object, or object part, can be easily classified once it is segmented at the right level. The two methods above are based on an arbitrary segmentation of the image, which typically decomposes it into segments that are too small, or, more rarely, too large.

In this section, we propose a method to analyze a family of segmentations and automatically discover the best observation level for each pixel in the image. One special case of such families is the segmentation tree, in which components are hierarchically organized. Our method is not restricted to such trees, and can be used for arbitrary sets of neighborhoods.

In Section 4.3.1 we formulate the search for the most adapted neighborhood of a pixel as an optimization problem. The construction of the cost function that is minimized is then described in Section 4.3.2.

Optimal purity cover

We define the neighborhood of a pixel as a connected component that contains this pixel. Let C k , ∀k ∈ {1, . . . , K} be the set of all possible connected components of the lattice defined on image I, and let S k be a cost associated to each of these components. For each pixel i, we wish to find the index k * (i) of the component that best explains this pixel, that is, the component with the minimal cost S k * (i) :

k * (i) = argmin k | i∈C k S k (16)
Note that components C k * (i) are non-disjoint sets that form a cover of the lattice. Note also that the overall cost

S * = i S k * (i) is minimal.
In practice, the set of components C k is too large, and only a subset of it can be considered. A classical technique to reduce the set of components is to consider a hierarchy of segmentations [START_REF] Najman | Geodesic saliency of watershed contours and hierarchical segmentation[END_REF], [START_REF] Arbeláez | Contour Detection and Hierarchical Image Segmentation[END_REF], that can be represented as a tree T . This was previously explored in [START_REF] Farabet | Scene parsing with multiscale feature learning, purity trees, and optimal covers[END_REF]. Solving Eq 16 on T consists in the following procedure: for each pixel (leaf) i, the optimal component C k * (i) is the one along the path between the leaf and the root with minimal cost S k * (i) . The optimal cover is the union of all these components. For efficiency purposes, it can be done simply by exploring the tree in a depth-first search manner, and finding the component with minimal weight along each branch. The complexity of the optimal cover procedure is then linear in the

!!"" #$%&'()!*#+,- # ./!0" 0$ 0% 0& 0' 0( 0) 0* 0+ 0, !.-!"-1"" 2 1$ 1% 1& 1' 1( 1) 1* 1+ 1,
!"./&0-1"./&0"

1$ 1% 1& 1' 1( 1) 1* 1+ 1, 2 '(3"&456$##)&45! (!/0"-0 *)(33&7&,-! *!/"1 *0 )(8,)&45
)!/-2-!0""0 Fig. 4. Third labeling strategy from the features: using a family of of segmentations, as described in Section 4.3. On this figure, the family of segmentations is a segmentation tree. The segment associated with each node in the tree is encoded by a spatial grid of feature vectors pooled in the segment's region. A classifier is then applied to all the aggregated feature grids to produce a histogram of categories, the entropy of which measures the "impurity" of the segment. Each pixel is then labeled by the minimally-impure node above it, which is the segment that best "explains" the pixel. Optimal cover:

{C 1 , C 3 , C 4 , C 5 }
Fig. 5. Finding the optimal cover on a tree. The numbers next to the components correspond to the entropy scores S i . For each pixel (leaf) i, the optimal component C k * (i) is the one along the path between the leaf and the root with minimal cost S k * (i) . The optimal cover is the union of all these components. In this example, the optimal cover {C 1 , C number of components in the tree. Figure 5 illustrates the procedure. Another technique to reduce the set of components considered is to compute a set of segmentations using different merging thresholds. In Section 5, we use such an approach, by computing multiple levels of the Felzenszwalb algorithm [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF]. The Felzenszwalb algorithm is not strictly monotonic, so the structure obtained cannot be cast into a tree: rather, it has a general graph form, in which each pixel belongs to as many superpixels as levels explored. Solving Eq 16 in this case consists in the following procedure: for each pixel i, the optimal component C k * (i) is the one among all the segmentations with minimal cost S k * (i) . Thus the complexity to produce a cover on the family of components is linear on the number of pixels, but with a constant that is proportional to the number of levels explored.

Producing the confidence costs

Given a set of components C k , we explain how to produce all the confidence costs S k . These costs represent the class purity of the associated components. Given the groundtruth segmentation, we can compute the cost as being the entropy of the distribution of classes present in the component. At test time, when no groundtruth is available, we need to define a function that can predict this cost by simply looking at the component. We now describe a way of achieving this, as illustrated in Figure 6.

Given the scale-invariant features F, we define a compact representation to describe objects as an elastic spatial arrangement of such features. In other terms, an object, or category in general, can be best described as a spatial arrangement of features, or parts. We define a simple attention function a used to mask the feature vector map with each component C k , producing a set of K masked feature vector patterns {F C k }, ∀k ∈ {1, . . . , K}. The function a is called an attention function because it suppresses the background around the component being analyzed. The patterns {F C k } are resampled to produce fixed-size representations. In our model the sampling is done using an elastic max-pooling function, which remaps input patterns of arbitrary size into a fixed G × G grid. This grid can be seen as a highly invariant representation that encodes spatial relations between an object's attributes/parts. This representation is denoted O k . Some nice properties of this encoding are:

(1) elongated, or in general ill-shaped objects, are nicely handled, (2) the dominant features are used to represent the object, combined with background subtraction, the features pooled represent solid basis functions to recognize the underlying object. Once we have the set of object descriptors O k , we define a function c : O k → [0, 1] Nc (where N c is the number of classes) as predicting the distribution of classes present in component C k . We associate a cost S k to this distribution. In this paper, c is implemented as a simple 2-layer neural network, and S k is the entropy of the predicted distribution. More formally, let O k be the feature vector associated with component C k , dk the predicted class distribution, and S k the cost associated to this distribution. We have

y k = W 2 tanh(W 1 O k + b 1 ), (17)
dk,a = e y k,a b∈classes e y k,b ,

S k = - a∈classes d k,a ln( dk,a ), (18)
with d k the groundtruth distribution for component k. Matrices W 1 and W 2 are noted θ c , and represent the trainable parameters of c. These parameters need to be learned over the complete set of segmentation families, computed on the entire training set available.

The training procedure is described in Section 4.3.3.

For each component C k chosen by the optimal purity cover (Section 4.3.1) the label is produced by:

l k = arg max a∈classes dk,a C k ∈ cut. (20)

Training procedure

Let F be the set of all feature maps in the training set, and T the set of all families of segmentations. We construct the segmentation collections (T ) T ∈T on the entire training set, and, for all T ∈ T train the classifier c to predict the distribution of classes in component C k ∈ T , as well as the costs S k . Given the trained parameters θ s , we build F and T , i.e. we compute all vector maps F and segmentation collections T on all the training data available, so as to produce a new training set of descriptors O k . This time, the parameters θ c of the classifier c are trained to minimize the KL-divergence between the true (known) distributions of labels d k in each component, and the prediction from the classifier dk (Eq 18):

l div = a∈classes dk,a ln( dk,a d k,a ). (21)
In this setting, the groundtruth distributions d k are not hard target vectors, but normalized histograms of the labels present in component C k . Once the parameters θ c are trained, dk accurately predicts the distribution of labels, and Eq 19 is used to assign a purity cost to the component.

EXPERIMENTS

We report our semantic scene understanding results on three different datasets: "Stanford Background" on which related state-of-the-art methods report classification errors, and two more challenging datasets with a larger number of classes: "SIFT Flow" and "Barcelona". The Stanford Background dataset [START_REF] Gould | Decomposing a scene into geometric and semantically consistent regions[END_REF] contains 715 images of outdoor scenes composed of 8 classes, chosen from other existing public datasets so that all the images are outdoor scenes, have approximately 320 × 240 pixels, where each image contains at least one foreground object. We use the evaluation procedure introduced in [START_REF] Gould | Decomposing a scene into geometric and semantically consistent regions[END_REF], 5-fold cross validation: 572 images used for training, and 143 for testing. The SIFT Flow dataset [START_REF] Liu | Nonparametric scene parsing: Label transfer via dense scene alignment[END_REF] is composed of 688 images, that have been thoroughly labeled by LabelMe users, and split in 2, 488 training images and 200 test images. The authors used synonym correction to obtain 33 semantic labels. The Barcelona dataset, as described in [START_REF] Tighe | Superparsing: scalable nonparametric image parsing with superpixels[END_REF], is derived from the LabelMe subset used in [START_REF] Russell | Object recognition by scene alignment[END_REF]. It has 14, 871 training and 279 test images. The test set consists of street scenes from Barcelona, while the training set ranges in scene types but has no street scenes from Barcelona. Synonyms were manually consolidated by [START_REF] Tighe | Superparsing: scalable nonparametric image parsing with superpixels[END_REF] to produce 170 unique labels.

To evaluate the representation from our multiscale convolutional network, we report results from several experiments on the Stanford Background dataset: (1) a system based on a plain convolutional network alone; (2) the multiscale convolutional network presented in Section 3.1, with raw pixelwise prediction; (3) superpixelbased predictions, as presented in Section 4.1; (4) CRFbased predictions, as presented in Section 4.2; (5) coverbased predictions, as presented in Section 4. 3. Results are reported in Table 1, and compared with related works. Our model achieves very good results in comparison with previous approaches. Methods of [START_REF] Kumar | Efficiently selecting regions for scene understanding[END_REF], [START_REF] Lempitsky | A pylon model for semantic segmentation[END_REF] achieve similar or better performances on this particular dataset but to the price of several minutes to parse one image.

Pixel Acc.

Class Acc.

CT (sec.) Gould et al. 2009 [15] 76.4% -10 to 600s Munoz et al. 2010 [START_REF] Munoz | Stacked hierarchical labeling[END_REF] 76.9% 66.2% 12s Tighe et al. 2010 [START_REF] Tighe | Superparsing: scalable nonparametric image parsing with superpixels[END_REF] 77.5% -10 to 300s Socher et al. 2011 [START_REF] Socher | Parsing Natural Scenes and Natural Language with Recursive Neural Networks[END_REF] 78.1% -? Kumar et al. 2010 [START_REF] Kumar | Efficiently selecting regions for scene understanding[END_REF] 79.4% -< 600s Lempitzky et al. 2011 [START_REF] Lempitsky | A pylon model for semantic segmentation[END_REF] 81 TABLE 2 Performance of our system on the SIFT Flow dataset [START_REF] Liu | Nonparametric scene parsing: Label transfer via dense scene alignment[END_REF]: per-pixel / average per-class accuracy. Our multiscale network is trained using two sampling methods: 1 balanced frequencies, 2 natural frequencies. We compare the results of our multiscale network with the raw (pixelwise) classifier, Felzenszwalb superpixels [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF] (one level), and our optimal cover applied to a stack of 10 levels of Felzenszwalb superpixels. Note: the threshold for the single level was picked to yield the best results; the cover automatically finds the best combination of superpixels.

We then demonstrate that our system scales nicely when augmenting the number of classes on two other datasets, in Tables 2 and3. Results on these datasets were obtained using our cover-based method, from Section 4.3. Example parses on the SIFT Flow dataset are shown on Figure 9.

For the SIFT Flow and Barcelona datasets, we experimented with two sampling methods when learning the multiscale features: respecting natural frequencies of classes, and balancing them so that an equal amount of each class is shown to the network. Balancing class occurrences is essential to model the conditional likelihood of each class (i.e. ignore their prior distribution). Both results are reported in Table 2. Training with balanced frequencies allows better discrimination of small objects, and although it decreases the overall pixelwise accuracy, it is more correct from a recognition point of view. Frequency balancing is used on the Stanford Background dataset, as it consistently gives better results. For the Barcelona dataset, both sampling methods are used as well, but frequency balancing worked rather poorly in that case. This can be explained by the fact that this dataset has a large amount of classes with very few training examples. These classes are therefore extremely , segments results of a threshold in the gPb hierarchy [START_REF] Arbeláez | Contour Detection and Hierarchical Image Segmentation[END_REF], and segments recovered by the maximum purity approach with an optimal cover (cf 4.3). The result (c) is obtained with a CRF on the superpixels shown in (d), as described in Section 4.2.

hard to model, and overfitting occurs much faster than for the SIFT Flow dataset. Results are shown on Table 3.

Results in Table 1 demonstrate the impressive computational advantage of convolutional networks over competing algorithms. Exploiting the parallel structure of this special network, by computing convolutions in parallel, allows us to parse an image of size 320 × 240 in less than one second on a 4-core Intel i7 laptop. Using GPUs or other types of dedicated hardware, our scene parsing model can be run in real-time (i.e. at more than 10fps).

Multiscale feature extraction

For all experiments, we use a 3-stage convolutional network. The first two layers of the network are composed of a bank of filters of size 7×7 followed by tanh units and 2 × 2 max-pooling operations. The last layer is a simple filter bank. The filters and pooling dimensions were chosen by a grid search. The input image is transformed into YUV space, and a Laplacian pyramid is constructed from it. The Y, U and V channels of each scale in the pyramid are then independently locally normalized, such that each local 15×15 patch has zero-mean and unit variance. For these experiments, the pyramid consists of 3 rescaled versions of the input (N = 3), in octaves: 320 × 240,160 × 120, 80 × 60.

The network is then applied to each 3-dimension input map X s . This input is transformed into a 16dimension feature map, using a bank of 16 filters, 10 connected to the Y channel, the 6 others connected to the U and V channels. The second layer transforms this 16-dimension feature map into a 64-dimension feature map, each map being produced by a combination of 8 randomly selected feature maps from the previous layer. Finally the 64-dimension feature map is transformed into Class Acc. Tighe et al. 2010 [START_REF] Tighe | Superparsing: scalable nonparametric image parsing with superpixels[END_REF] 66.9% 7.6% raw multiscale net 1 37.8% 12.1% multiscale net + superpixels 1 44.1% 12.4% multiscale net + cover 1 46.4% 12.5% multiscale net + cover 2 67.8% 9.5% TABLE 3 Performance of our system on the Barcelona dataset [START_REF] Tighe | Superparsing: scalable nonparametric image parsing with superpixels[END_REF]: per-pixel / average per-class accuracy. Our multiscale network is trained using two sampling methods: 1 balanced frequencies, 2 natural frequencies. We compare the results of our multiscale network with the raw (pixelwise) classifier, Felzenszwalb superpixels [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF] (one level), and our optimal cover applied to a stack of 10 levels of Felzenszwalb superpixels. Note: the threshold for the single level was picked to yield the best results; the cover automatically finds the best combination of superpixels.

a 256-dimension feature map, each map being produced by a combination of 32 randomly selected feature maps from the previous layer. The outputs of each of the 3 networks are then upsampled and concatenated, so as to produce a 256 × 3 = 768dimension feature vector map F. Given the filter sizes, the network has a field of view of 46 × 46, at each scale, which means that a feature vector in F is influenced by a 46×46 neighborhood at full resolution, a 92×92 neighborhood at half resolution, and a 184×184 neighborhood at quarter resolution. These neighborhoods are shown in Figure 1.

The network is trained on all 3 scales in parallel, using stochastic gradient descent with no second-order information, and mini-batches of size 1. Simple grid-search was performed to find the best learning rate (10 -3 ) and regularization parameters (L2 coefficient: 10 -5 ), using a holdout of 10% of the training data for validation. The holdout is also used to select the best network, i.e. the network that generalizes the most on the holdout.

Convergence, that is, maximum generalization performance, is typically attained after between 10 to 50 million patches have been seen during stochastic gradient descent. This typically represents between two to five days of training. No special hardware (GPUs) was used for training.

The convolutional network has roughly 0.5 million trainable parameters. To ensure that features do not overfit some irrelevant biases present in the data, jitterhorizontal flipping of all images, rotations between -8 and 8 degrees, and rescaling between 90 and 110%was used to artificially expand the size of the training data. These additional distortions are applied during training, before loading a new training point, and are sampled from uniform distributions. Jitter was shown to be crucial for low-level feature learning in the works of [START_REF] Simard | Best practices for convolutional neural networks applied to visual document analysis[END_REF] and [START_REF] Ciresan | A committee of neural networks for traffic sign classification[END_REF].

For our baseline, we trained a single-scale network and a three-scale network as raw site predictors, for each location i, using the classification loss L cat defined in Eq 10, with the two-layer neural network defined in Eq 9. Table 1 shows the clear advantage of the multi-scale representation, which captures scene-level dependencies, and can classify more pixels accurately. Without an explicit segmentation model, the visual aspect of the predictions still suffers from inaccurate object delineation.

Parsing with superpixels

The results obtained with the strategy presented in section 4.1 demonstrate the quality of our multiscale features, by reaching a very high classification accuracy on all three datasets. This simple strategy is also a real fit for real time applications, taking only an additional 0.2 second to label a 320 × 240 image on Intel i7 CPU. An example of result is given in Figure 7.

The 2-layer neural network used for this method (Eq 9) has 768 input units, 1024 hidden units; and as many output units as classes in each dataset. This neural network is trained with no regularization.

Multilevel parsing

Although the simple strategy of the previous section seems appealing, the results can be further improved using the multilevel approach of Section 4.3.

The family of segmentations used to find the optimal cover could be a simple segmentation tree constructed on the raw image gradient. For the Stanford Background dataset experiments, we used a more sophisticated tree based on a semantic image gradient. We used the gPb hierarchies of Arbelaez et al. , which are computed using spectral clustering to produce semantically consistent contours of objects. Their computation requires one minute per image.

For the SIFT Flow and Barcelona datasets, we used a cheaper technique, which does not rely on a tree: we ran the superpixel method proposed by Felzenszwalb in [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF] at 10 different levels. The Felzenszwalb algorithm is not strictly monotonic, so the structure obtained cannot be cast into a tree: rather, it has a general graph form, in which each pixel belongs to 10 different superpixels. Our optimal cover algorithm can be readily applied to arbitrary structures of this type. The 10 levels were chosen such that they are linearly distributed and span a large range.

Classically, segmentation methods find a partition of the segments rather than a cover. Partitioning the segments consists in finding an optimal cut in a tree (so that each terminal node in the pruned tree corresponds to a segment). We experimented with graph-cuts to do so [START_REF] Ford | A simple algorithm for finding maximal network flows and an application to the hitchcock problem[END_REF], [START_REF] Boykov | Interactive graph cuts for optimal boundary & region segmentation of objects in n-d images[END_REF], but the results were less accurate than with our optimal cover method (Stanford Background dataset only).

The 2-layer neural network c from Eq 17 has 3 × 3 × 768 = 6912 input units (using a 3 × 3 grid of feature vectors from F), 1024 hidden units; and as many output units as classes in each dataset. This rather large neural network is trained with L2 regularization (coefficient: 10 -2 ), to minimize overfitting.

Results are better than the superpixel method, in particular, better delineation is achieved (see Fig. 7).

Conditional random field

We demonstrate the state-of-the-art quality of our features by employing a CRF on the superpixels given by thresholding the gPb hierarchy, on the Stanford Background dataset. A similar test is performed in Lempitsky et al. [START_REF] Lempitsky | A pylon model for semantic segmentation[END_REF], where the authors also use a CRF on the same superpixels (at the threshold 20 in the gPb hierarchy), but employ different features. Histograms of densely sampled SIFT words, colors, locations, and contour shape descriptors. They report a ratio of correctly classified pixels of 81.1% on the Stanford Background dataset. We recall that this accuracy is the best one has achieved at the present day on this dataset with a flat CRF.

In our CRF energy, we performed a grid search to set the parameters of ( 13) (β = 20, α = 0.1 γ = 200), and used a grey level gradient. The accuracy of the resulting system is 81.4, as reported in Table 1. Our features are thus outperforming the best publicly available combination of handcrafted features.

Some comments on the learned features

With recent advances in unsupervised (deep) learning, learned features have become easier to analyze and understand. In this work, the entire stack of features is learned in a purely supervised manner, and yet we found that the features obtained are rather meaningful. We believe that the reason for this is the type of loss function we use, which enforces a large invariance: the system is forced to produce an invariant representation for all the locations of a given object. This type of invariance is very similar to what can be achieved using semisupervised techniques such as Dr-LIM [START_REF] Hadsell | Dimensionality reduction by learning an invariant mapping[END_REF], where the loss enforces pairs of similar patches to yield a same encoding. Figure 10 shows an example of the features learned on the SIFT Flow dataset.

Some comments on real-world generalization

Now that we have compared and discussed several strategies for scene parsing based on our multiscale features, we consider taking our system in the realworld, to evaluate its generalization properties. The work of [START_REF] Torralba | Unbiased look at dataset bias[END_REF], measuring dataset bias, raises the question of the generalization of a recognition system learned on specific, publicly available datasets. We used our multiscale features combined with classification using superpixels as described in Section 4.1, trained on the SiftFlow dataset (2,688 images, most of them taken in non-urban environments, see Table 2 and Figure 9). We collected a 360 degree movie in our workplace environment, including a street and a park. introducing difficulties such as lighting conditions and image distortions: see Figure 11.

The movie was built from four videos that were stitched to form a 360 degree video stream of 1280 × 256 images, thus creating artifacts not seen during training. We processed each frame independently, without using any temporal consistency or smoothing.

Despite all these constraints, and the rather small size of the training dataset, we observe rather convincing generalization of our models on these previously unseen scenes. The two video sequences are available at http://www.clement.farabet.net/. Two snapshots are included in Figure 11. Our scene parsing system constitutes at the best of our knowledge the first approach achieving real time performance, one frame being processed in less than a second on a 4-core Intel i7. Feature extraction, which represent around 500ms on the i7 can be reduced to 60ms using dedicated FPGA hardware [9], [START_REF] Farabet | Neuflow: A runtime reconfigurable dataflow processor for vision[END_REF].

DISCUSSION

The main lessons from the experiments presented in this paper are as follows: • Using a high-capacity feature-learning system fed with raw pixels yields excellent results, when compared with systems that use engineered features. The accuracy is similar or better than competing systems, even when the segmentation hypothesis generation and the post-processing module are absent or very simple. • Feeding the system with a wide contextual window is critical to the quality of the results. The numbers in table 1 show a dramatic improvement of the performance of the multi-scale convolutional network over the single scale version. • When a wide context is taken into account to produce each pixel label, the role of the post-processing is greatly reduced. In fact, a simple majority vote of the categories within a superpixel yields state-ofthe-art accuracy. This seems to suggest that contextual information can be taken into account by a feedforward trainable system with a wide contextual window, perhaps as well as an inference mechanism that propagates label constraints over a graphical model, but with a considerably lower computational cost.

• The use of highly sophisticated post-processing schemes, which seem so crucial to the success of other models, does not seem to improve the results significantly over simple schemes. This seems to suggest that the performance is limited by the quality of the labeling, or the quality of the segmentation hypotheses, rather than by the quality of the contextual consistency system or the inference algorithm.

• Relying heavily on a highly-accurate feed-forward pixel labeling system, while simplifying the postprocessing module to its bare minimum cuts down the inference times considerably. The resulting system is dramatically faster than those that rely heavily on graphical model inference. Moreover, the bulk of the computation takes place in the convolutional network. This computation is algorithmically simple, easily parallelizable. Implementations on multi-core machines, general-purpose GPUs, Digital Signal Processors, or specialized architectures implemented on FPGAs is straightforward. This is demonstrated by the FPGA implementation [9], [START_REF] Farabet | Neuflow: A runtime reconfigurable dataflow processor for vision[END_REF] of the feature extraction scheme presented in this paper that runs in 60ms for an image resolution of 320 × 240.

CONCLUSION AND FUTURE WORK

This paper demonstrates that a feed-forward convolutional network, trained end-to-end in a supervised manner, and fed with raw pixels from large patches over multiple scales, can produce state of the art performance on standard scene parsing datasets. The model does not rely on engineered features, and uses purely supervised training from fully-labeled images to learn appropriate low-level and mid-level features. Perhaps the most surprising results is that even in the absence of any post-processing, by simply labeling each pixel with the highest-scoring category produced by the convolutional net for that location, the system yields near state-of-the-art pixel-wise accuracy, and better per-class accuracy than all previously-published results. Feeding the features of the convolutional net to various sophisticated schemes that generate segmentation hypotheses, and that find consistent segmentations and labeling by taking local constraints into account improves the results slightly, but not considerably.

While the results on datasets with few categories are good, the accuracy of the best existing scene parsing systems, including ours, is still quite low when the number of categories is large. The problem of scene parsing is far from being solved. While the system presented here has a number of advantages and shortcomings, the framing of the scene parsing task itself is in need of refinement. First of all, the pixel-wise accuracy is a somewhat inaccurate measure of the visual and practical quality of the result. Spotting rare objects is often more important than accurately labeling every boundary pixel of the sky (which are often in greater number). The average per-class accuracy is a step in the right direction, but not the ultimate solution: one would prefer a system that correctly spots every object or region, while giving an approximate boundary to a system that produces accurate boundaries for large regions (sky, road, grass), but fail to spot small objects. A reflection is needed on the best ways to measure the accuracy of scene labeling systems.

Scene parsing datasets also need better labels. One could imagine using scene parsing datasets with hierarchical labels, so that a window within a building would be labeled as "building" and "window". Using this kind of labeling in conjunction with graph structures on sets of labels that contain is-part-of relationships would likely produce more consistent interpretations of the whole scene.

The framework presented in this paper trains the convolutional net as a pixel labeling system in isolation from the post-processing module that ensures the consistency of the labeling and its proper registration with the image regions. This requires that the convolutional net be trained with images that are fully labeled at the pixel level. One would hope that jointly fine-tuning the convolutional net and the post-processor produces better overall interpretations. Gradients can be backpropagated through the post-processor to the convolutional nets. This is reminiscent of the Graph Transformer Network model, a kind of non-linear CRF in which an un-normalized graphical model based post-processing module was trained jointly with a convolutional network for handwriting recognition [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. Unfortunately, preliminary experiments with such joint training yielded lower test-set accuracies due to overtraining.

A more importantly advantage of joint training would allow the use of weakly-labeled images in which only a list of objects present in the image would be given, perhaps tagged with approximate positions. This would be similar in spirit to sentence-level discriminative training methods used in speech recognition and handwriting recognition [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF].

Another possible direction for improvement includes the use of objective functions that directly operates of the edge costs of neighborhood graphs in such as way that graph-cut segmentation and similar methods produce the best answer. One such objective function is is Turaga's Maximin Learning [START_REF] Turaga | Maximin affinity learning of image segmentation[END_REF], which pushes up the lowest edge cost along the shortest path between two points in different segments, and pushes down the highest edge cost along a path between two points in the same segment.

Our system so far has been trained using purely supervised learning applied to a fairly classical convolutional network architecture. However, a number of recent works have shown the advantage of architectural elements such as rectifying non-linearities and local contrast normalization [21]. More importantly, several works have shown the advantage of using unsupervised pre-training to prime the convolutional net into a good starting point before supervised refinement [37], [START_REF] Kavukcuoglu | Learning invariant features through topographic filter maps[END_REF], [START_REF] Kavukcuoglu | Fast inference in sparse coding algorithms with applications to object recognition[END_REF], [START_REF] Lee | Convolu-tional deep belief networks for scalable unsupervised learning of hierarchical representations[END_REF], [START_REF] Kavukcuoglu | Learning convolutional feature hierachies for visual recognition[END_REF]. These methods improve the performance in the low training set size regime, and would probably improve the performance of the present system.

Finally, code and data are available online at http://www.clement.farabet.net/.

Fig. 1 .

 1 Fig. 1. Diagram of the scene parsing system. The raw input image is transformed through a Laplacian pyramid.Each scale is fed to a 3-stage convolutional network, which produces a set of feature maps. The feature maps of all scales are concatenated, the coarser-scale maps being upsampled to match the size of the finest-scale map. Each feature vector thus represents a large contextual window around each pixel. In parallel, a single segmentation (i.e. superpixels), or a family of segmentations (e.g. a segmentation tree) are computed to exploit the natural contours of the image. The final labeling is produced from the feature vectors and the segmentation(s) using different methods, as presented in section 4.

  ĉi,a = e w T a Fi b∈classes e w T b Fi ,

Fig. 3 .

 3 Fig. 3. Second labeling strategy from the features: using a CRF, described in Section 4.2.

6 C 1 C 2 C 3

 6123

Fig. 6 .

 6 Fig. 6. The shape-invariant attention function a. For each component C k in the family of segmentations T , the corresponding image segment is encoded by a spatial grid of feature vectors that fall into this segment. The aggregated feature vector of each grid cell is computed by a component-wise max pooling of the feature vectors centered on all the pixels that fall into the grid cell; this produces a scale-invariant representation of the segment and its surroundings. The result, O k , is a descriptor that encodes spatial relations between the underlying object's parts. The grid size was set to 3×3 for all our experiments.

Fig. 7 .

 7 Fig. 7. Example of results on the Stanford background dataset. (b),(d) and (f) show results with different labeling strategies, overlaid with superpixels (cf Section 4.1), segments results of a threshold in the gPb hierarchy[START_REF] Arbeláez | Contour Detection and Hierarchical Image Segmentation[END_REF], and segments recovered by the maximum purity approach with an optimal cover (cf 4.3). The result (c) is obtained with a CRF on the superpixels shown in (d), as described in Section 4.2.

Fig. 8 .

 8 Fig. 8. More results using our multiscale convolutional network and a flat CRF on the Stanford Background Dataset.

Fig. 10 .

 10 Fig. 10. Typical first layer features, learned on the SIFT Flow dataset. (a) to (c) show the 16 filters learned at each scale, when no weight sharing is used (networks at each scale are independent). (d) show the 16 filters obtained when sharing weights across all 3 scales. All the filters are 7 × 7. We observe typical oriented edges, and highfrequency filters. Filters at higher layers are more difficult to analyze.

Fig. 9 .

 9 Fig. 9. Typical results achieved on the SIFT Flow dataset.

Fig. 11 .

 11 Fig. 11. Real-time scene parsing in natural conditions. Training on SiftFlow dataset. We display one label per component in the final prediction.

  3 , C 4 , C 5 } will result in a segmentation in disjoint sets {C 1 , C 2 , C 3 , C 4 }, with the subtle difference that component C 2 will be labelled with the class of C 5 , as C 5 is the best observation level for C 2 . The generalization to a family of segmentations is straightforward (see text).

TABLE 1

 1 Performance of our system on the Stanford Background dataset[START_REF] Gould | Decomposing a scene into geometric and semantically consistent regions[END_REF]: per-pixel / average per-class accuracy. The third column reports compute times, as reported by the authors. Our algorithms were computed using a 4-core Intel i7.

		Pixel Acc.	Class Acc.
	Liu et al. 2009 [31]	74.75%	-
	Tighe et al. 2010 [44]	76.9%	29.4%
	raw multiscale net 1	67.9%	45.9%
	multiscale net + superpixels 1	71.9%	50.8%
	multiscale net + cover 1	72.3%	50.8%
	multiscale net + cover 2	78.5%	29.6%

ACKNOWLEDGMENT

We would like to thank Marco Scoffier for fruitful discussions and the 360 degree video collection. We are also grateful to Victor Lempitsky who kindly provided us with his results on the Stanford Database for comparison.

This work was funded in part by DARPA contract "Integrated deep learning for large scale multi-modal data representation", ONR MURI "Provably-stable visionbased control of high-speed flight", ONR grant "Learning Hierarchical Models for Information Integration".