Juhong Min

Jongmin Lee

Jean Ponce

Minsu Cho

Learning to Compose Hypercolumns for Visual Correspondence

Keywords: visual correspondence, multi-layer features, dynamic feature composition

Feature representation plays a crucial role in visual correspondence, and recent methods for image matching resort to deeply stacked convolutional layers. These models, however, are both monolithic and static in the sense that they typically use a specific level of features, e.g., the output of the last layer, and adhere to it regardless of the images to match. In this work, we introduce a novel approach to visual correspondence that dynamically composes effective features by leveraging relevant layers conditioned on the images to match. Inspired by both multi-layer feature composition in object detection and adaptive inference architectures in classification, the proposed method, dubbed Dynamic Hyperpixel Flow, learns to compose hypercolumn features on the fly by selecting a small number of relevant layers from a deep convolutional neural network. We demonstrate the effectiveness on the task of semantic correspondence, i.e., establishing correspondences between images depicting different instances of the same object or scene category. Experiments on standard benchmarks show that the proposed method greatly improves matching performance over the state of the art in an adaptive and efficient manner.

Introduction

Visual correspondence is at the heart of image understanding with numerous applications such as object recognition, image retrieval, and 3D reconstruction [START_REF] Forsyth | Computer Vision: A Modern Approach[END_REF]. With recent advances in neural networks [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Hu | Squeeze-and-excitation networks[END_REF][START_REF] Huang | Densely connected convolutional networks[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], there has been a significant progress in learning robust feature representation for establishing correspondences between images under illumination and viewpoint changes. Currently, the de facto standard is to use as feature representation the output of deeply stacked convolutional layers in a trainable architecture. Unlike in object classification and detection, however, such learned features have often achieved only modest performance gains over hand-crafted ones [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF][START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] in the task of visual correspondence [START_REF] Schonberger | Comparative evaluation of hand-crafted and learned local features[END_REF]. In particular, correspondence between images under large intra-class variations still remains an extremely challenging problem [START_REF] Choy | Universal correspondence network[END_REF][START_REF] Fathy | Hierarchical metric learning and matching for 2d and 3d geometric correspondences[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Jeon | Parn: Pyramidal affine regression networks for dense semantic correspondence[END_REF][START_REF] Kanazawa | Warpnet: Weakly supervised matching for single-view reconstruction[END_REF][START_REF] Kim | Recurrent transformer networks for semantic correspondence[END_REF][START_REF] Kim | Fcss: Fully convolutional self-similarity for dense semantic correspondence[END_REF][START_REF] Kim | Dctm: Discrete-continuous transformation matching for semantic flow[END_REF][START_REF] Lee | Sfnet: Learning object-aware semantic correspondence[END_REF][START_REF] Long | Do convnets learn correspondence?[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Novotny | Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF][START_REF] Ufer | Deep semantic feature matching[END_REF][START_REF] Zhou | Learning dense correspondence via 3d-guided cycle consistency[END_REF] while modern neural networks are known to excel at classification [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Huang | Densely connected convolutional networks[END_REF]. What do we miss in using deep neural features for correspondence?

Most current approaches for correspondence build on monolithic and static feature representations in the sense that they use a specific feature layer, e.g., the last convolutional layer, and adhere to it regardless of the images to match. Correspondence, however, is all about precise localization of corresponding positions, which requires visual features at different levels, from local patterns to semantics and context; in order to disambiguate a match on similar patterns, it is necessary to analyze finer details and larger context in the image. Furthermore, relevant feature levels may vary with the images to match; the more we already know about images, the better we can decide which levels to use. In this aspect, conventional feature representations have fundamental limitations.

In this work, we introduce a novel approach to visual correspondence that dynamically composes effective features by leveraging relevant layers conditioned on the images to match. Inspired by both multi-layer feature composition, i.e., hypercolumn, in object detection [START_REF] Hariharan | Hypercolumns for object segmentation and fine-grained localization[END_REF][START_REF] Kong | Hypernet: Towards accurate region proposal generation and joint object detection[END_REF][START_REF] Lin | Feature pyramid networks for object detection[END_REF][START_REF] Liu | Receptive field block net for accurate and fast object detection[END_REF] and adaptive inference architectures in classification [START_REF] Figurnov | Spatially adaptive computation time for residual networks[END_REF][START_REF] Srivastava | Highway networks[END_REF][START_REF] Veit | Convolutional networks with adaptive inference graphs[END_REF], we combine the best of both worlds for visual correspondence. The proposed method learns to compose hypercolumn features on the fly by selecting a small number of relevant layers in a deep convolutional neural network. At inference time, this dynamic architecture greatly improves matching performance in an adaptive and efficient manner. We demonstrate the effectiveness of the proposed method on several benchmarks for semantic correspondence, i.e., establishing visual correspondences between images depicting different instances of the same object or scene categories, where due to large variations it may be crucial to use features at different levels.

Related work

Feature representation for semantic correspondence. Early approaches [START_REF] Bristow | Dense semantic correspondence where every pixel is a classifier[END_REF][START_REF] Cho | Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals[END_REF][START_REF] Ham | Proposal flow[END_REF][START_REF] Kim | Deformable spatial pyramid matching for fast dense correspondences[END_REF][START_REF] Liu | Sift flow: Dense correspondence across scenes and its applications[END_REF][START_REF] Taniai | Joint recovery of dense correspondence and cosegmentation in two images[END_REF][START_REF] Yang | Object-aware dense semantic correspondence[END_REF] tackle the problem of visual correspondence using hand-crafted descriptors such as HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] and SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. Since these lack high-level image semantics, the corresponding methods have difficulties with significant changes in background, view point, deformations, and instance-specific patterns. The advent of convolutional neural networks (CNN) [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] has led to a paradigm shift from this hand-crafted representations to deep features and boosted performance in visual correspondence [START_REF] Fathy | Hierarchical metric learning and matching for 2d and 3d geometric correspondences[END_REF][START_REF] Novotny | Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching[END_REF][START_REF] Zhou | Learning dense correspondence via 3d-guided cycle consistency[END_REF]. Most approaches [START_REF] Choy | Universal correspondence network[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Kim | Fcss: Fully convolutional self-similarity for dense semantic correspondence[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF] learn to predict correlation scores between local regions in an input image pair, and some recent methods [START_REF] Jeon | Parn: Pyramidal affine regression networks for dense semantic correspondence[END_REF][START_REF] Kanazawa | Warpnet: Weakly supervised matching for single-view reconstruction[END_REF][START_REF] Kim | Recurrent transformer networks for semantic correspondence[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF] cast this task as an image alignment problem in which a model learns to regress global geometric transformation parameters. All typically adopt a CNN pretrained on image classification as their backbone, and make predictions based on features from its final convolutional layer. While some methods [START_REF] Long | Do convnets learn correspondence?[END_REF][START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF] have demonstrated the advantage of using different CNN layers in capturing low-level to high-level patterns, leveraging multiple layers of deeply stacked layers has remained largely unexplored in correspondence problems. Multi-layer neural features. To capture different levels of information distributed over all intermediate layers, Hariharan et al. propose the hypercolumn [START_REF] Hariharan | Hypercolumns for object segmentation and fine-grained localization[END_REF], a vector of multiple intermediate convolutional activations lying above a pixel for fine-grained localization. Attempts at integrating multi-level neural features have addressed object detection and segmentation [START_REF] Kong | Hypernet: Towards accurate region proposal generation and joint object detection[END_REF][START_REF] Lin | Feature pyramid networks for object detection[END_REF][START_REF] Liu | Receptive field block net for accurate and fast object detection[END_REF]. In the area of visual correspondence, only a few methods [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Novotny | Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching[END_REF][START_REF] Ufer | Deep semantic feature matching[END_REF] attempt to use multi-layer features. Unlike ours, however, these models use static features extracted from CNN layers that are chosen manually [START_REF] Novotny | Anchornet: A weakly supervised network to learn geometry-sensitive features for semantic matching[END_REF][START_REF] Ufer | Deep semantic feature matching[END_REF] or by greedy search [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]. While the use of hypercolumn features on the task of semantic visual correspondence has recently been explored by Min et al. [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF], the method predefines hypercolumn layers by a greedy selection procedure, i.e., beam search, using a validation dataset. In this work, we clearly demonstrate the benefit of a dynamic and learnable architecture both in strongly-supervised and weakly-supervised regimes and also outperform the work of [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF] with a significant margin. Dynamic neural architectures. Recently, dynamic neural architectures have been explored in different domains. In visual question answering, neural module networks [START_REF] Andreas | Learning to compose neural networks for question answering[END_REF][START_REF] Andreas | Neural module networks[END_REF] compose different answering networks conditioned on an input sentence. In image classification, adaptive inference networks [START_REF] Figurnov | Spatially adaptive computation time for residual networks[END_REF][START_REF] Srivastava | Highway networks[END_REF][START_REF] Veit | Convolutional networks with adaptive inference graphs[END_REF] learn to decide whether to execute or bypass intermediate layers given an input image. Dynamic channel pruning methods [START_REF] Gao | Dynamic channel pruning: Feature boosting and suppression[END_REF][START_REF] Hua | Channel gating neural networks[END_REF] skip unimportant channels at run-time to accelerate inference. All these methods reveal the benefit of dynamic neural architectures in terms of either accuracy or speed, or both. To the best of our knowledge, our work is the first that explores a dynamic neural architecture for visual correspondence.

Our main contribution is threefold: [START_REF] Andreas | Learning to compose neural networks for question answering[END_REF] We introduce a novel dynamic feature composition approach to visual correspondence that composes features on the fly by selecting relevant layers conditioned on images to match. [START_REF] Andreas | Neural module networks[END_REF] We propose a trainable layer selection architecture for hypercolumn composition using Gumbelsoftmax feature gating. [START_REF] Bristow | Dense semantic correspondence where every pixel is a classifier[END_REF] The proposed method outperforms recent state-ofthe-art methods on standard benchmarks of semantic correspondence in terms of both accuracy and speed.

Dynamic hyperpixel flow

Given two input images to match, a pretrained convolutional network extracts a series of intermediate feature blocks for each image. The architecture we propose in this section, dynamic hyperpixel flow, learns to select a small number of layers (feature blocks) on the fly and composes effective features for reliable matching of the images. Figure 1 illustrates the overall architecture. In this section, we describe the proposed method in four steps: (i) multi-layer feature extraction, (ii) dynamic layer gating, (iii) correlation computation and matching, and (iv) training objective.

Multi-layer feature extraction

We adopt as a feature extractor a convolutional neural network pretrained on a large-scale classification dataset, e.g., ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], which is commonly used in most related methods [START_REF] Choy | Universal correspondence network[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Kim | Recurrent transformer networks for semantic correspondence[END_REF][START_REF] Kim | Dctm: Discrete-continuous transformation matching for semantic flow[END_REF][START_REF] Lee | Sfnet: Learning object-aware semantic correspondence[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF][START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF]. Following the work on hypercolumns [START_REF] Hariharan | Hypercolumns for object segmentation and fine-grained localization[END_REF], however, we view the layers of the convolutional network as a non-linear counterpart of image pyramids and extract a series of multiple features along intermediate layers [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF].

Let us assume the backbone network contains L feature extracting layers. Given two images I and I , source and target, the network generates two sets of L intermediate feature blocks. We denote the two sets of feature blocks by B = {b l } L-1 l=0 and B = {b l } L-1 l=0 , respectively, and call the earliest blocks, b 0 and b 0 , base feature blocks. As in Fig. 1, each pair of source and target feature blocks at layer l is passed to the l-th layer gating module as explained next.

Dynamic layer gating

Given L feature block pairs {(b l , b l )} L-1 l=0 , L layer gating modules learn to select relevant feature block pairs and transform them for establishing robust correspondences. As shown in the top of Fig. 1, the module has two branches, one for layer gating and the other for feature transformation.

Gumbel layer gating. The first branch of the l-th layer gating module takes the l-th pair of feature blocks (b l , b l ) as an input and performs global average pooling on two feature blocks to capture their channel-wise statistics. Two average pooled features of size 1 × 1 × c l from b l and b l are then added together to form a vector of size c l . A multi-layer perceptron (MLP) composed of two fully-connected layers with ReLU non-linearity takes the vector and predicts a relevance vector r l of size 2 for gating, whose entries indicate the scores for selecting or skipping ('on' or 'off') the l-th layer, respectively. We can simply obtain a gating decision using argmax over the entries, but this naïve gating precludes backpropagation since argmax is not differentiable.

To make the layer gating trainable and effective, we adopt the Gumbelmax trick [START_REF] Gumbel | Statistical theory of extreme values and some practical applications: a series of lectures[END_REF] and its continuous relaxation [START_REF] Jang | Categorical reparameterization with gumbel-softmax[END_REF][START_REF] Maddison | The concrete distribution: A continuous relaxation of discrete random variables[END_REF]. Let z be a sequence of i.i.d. Gumbel random noise and let Y be a discrete random variable with Kclass categorical distribution u, i.e., p(Y = y) ∝ u y and y ∈ {0, ..., K -1}. Using the Gumbel-max trick [START_REF] Gumbel | Statistical theory of extreme values and some practical applications: a series of lectures[END_REF], we can reparamaterize sampling Y to y = arg max k∈{0,...,K-1} (log u k + z k ). To approximate the argmax in a differentiable manner, the continuous relaxation [START_REF] Jang | Categorical reparameterization with gumbel-softmax[END_REF][START_REF] Maddison | The concrete distribution: A continuous relaxation of discrete random variables[END_REF] of the Gumbel-max trick replaces the argmax operation with a softmax operation. By expressing a discrete random sample y as a one-hot vector y, a sample from the Gumbel-softmax can be represented by ŷ = softmax((log u + z)/τ ), where τ denotes the temperature of the softmax. In our context, the discrete random variable obeys a Bernoulli distribution, i.e., y ∈ {0, 1}, and the predicted relevance scores represent the log probability distribution for 'on' and 'off', i.e., log u = r l . Our Gumbel-softmax gate thus has a form of

ŷl = softmax(r l + z l ), (1)
where z l is a pair of i.i.d. Gumbel random samples and the softmax temperature τ is set to 1.

Convolutional feature transformation. The second branch of the l-th layer gating module takes the l-th pair of feature blocks (b l , b l ) as an input and transforms each feature vector over all spatial positions while reducing its dimension by 1 ρ ; we implement it using 1 × 1 convolutions, i.e., position-wise linear transformations, followed by ReLU non-linearity. This branch is designed to transform the original feature block of size h l × w l × c l into a more compact and effective representation of size h l × w l × c l ρ for our training objective. We denote the pair of transformed feature blocks by ( bl , b l ). Note that if l-th Gumbel gate chooses to skip the layer, then the feature transformation of the layer can be also ignored thus reducing the computational cost.

Forward and backward propagations. During training, we use the straightthrough version of the Gumbel-softmax estimator [START_REF] Jang | Categorical reparameterization with gumbel-softmax[END_REF]: forward passes proceed with discrete samples by argmax whereas backward passes compute gradients of the softmax relaxation of Eq.( 1). In the forward pass, the transformed feature pair ( bl , b l ) is simply multiplied by 1 ('on') or 0 ('off') according to the gate's discrete decision y. While the Gumbel gate always makes discrete decision y in the forward pass, the continuous relaxation in the backward pass allows gradients to propagate through softmax output ŷ, effectively updating both branches, the feature transformation and the relevance estimation, regardless of the gate's decision. Note that this stochastic gate with random noise increases the diversity of samples and is thus crucial in preventing mode collapse in training. At test time, we simply use deterministic gating by argmax without Gumbel noise [START_REF] Jang | Categorical reparameterization with gumbel-softmax[END_REF]. As discussed in Sec. 4.2, we found that the proposed hard gating trained with Gumbel softmax is superior to conventional soft gating with sigmoid in terms of both accuracy and speed.

Correlation computation and matching

The output of gating is a set of selected layer indices, S = {s 1 , s 2 , ..., s N }. We construct a hyperimage H for each image by concatenating transformed feature blocks of the selected layers along channels with upsampling:

H = ζ( bs1 ), ζ( bs2 ), ..., ζ( b s N )
, where ζ denotes a function that spatially upsamples the input feature block to the size of b 0 , the base block. Note that the number of selected layers N is fully determined by the gating modules. If all layers are off, then we use the base feature block by setting S = {0}. We associate with each spatial position p of the hyperimage the corresponding image coordinates and hyperpixel feature [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]. Let us denote by x p the image coordinate of position p, and by f p the corresponding feature, i.e., f p = H(x p ). The hyperpixel at position p in the hyperimage is defined as h p = (x p , f p ). Given source and target images, we obtain two sets of hyperpixels, H and H . In order to reflect geometric consistency in matching, we adapt probablistic Hough matching (PHM) [START_REF] Cho | Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF] to hyperpixels, similar to [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]. The key idea of PHM is to re-weight appearance similarity by Hough space voting to enforce geometric consistency. In our context, let D = (H, H ) be two sets of hyperpixels, and m = (h, h ) be a match where h and h are respectively elements of H and H . Given a Hough space X of possible offsets (image transformations) between the two hyperpixels, the confidence for match m, p(m|D), is computed as p(m|D) ∝ p(m a ) x∈X p(m g |x) m∈H×H p(m a )p(m g |x) where p(m a ) represents the confidence for appearance matching and p(m g |x) is the confidence for geometric matching with an offset x, measuring how close the offset induced by m is to x. By sharing the Hough space X for all matches, PHM efficiently computes match confidence with good empirical performance [START_REF] Cho | Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals[END_REF][START_REF] Ham | Proposal flow[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]. In this work, we compute appearance matching confidence using hyperpixel features by p(m a ) ∝ ReLU

fp•f p fp f p 2
, where the squaring has the effect of suppressing smaller matching confidences. On the output |H| × |H | correlation matrix of PHM, we perform soft mutual nearest neighbor filtering [START_REF] Rocco | Neighbourhood consensus networks[END_REF] to suppress noisy correlation values and denote the filtered matrix by C. Dense matching and keypoint transfer. From the correlation matrix C, we establish hyperpixel correspondences by assigning to each source hyperpixel h i the target hyperpixel ĥ j with the highest correlation. Since the spatial resolutions of the hyperimages are the same as those of base feature blocks, which are relatively high in most cases (e.g., 1/4 of input image with ResNet-101 as the backbone), such hyperpixel correspondences produce quasi-dense matches.

Furthermore, given a keypoint p m in the source image, we can easily predict its corresponding position p m in the target image by transferring the keypoint using its nearest hyperpixel correspondence. In our experiments, we collect all correspondences of neighbor hyperpixels of keypoint p m and use the geometric average of their individual transfers as the final prediction p m [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]. This consensus keypoint transfer method improves accuracy by refining mis-localized predictions of individual transfers.

Training objective

We propose two objectives to train our model using different degrees of supervision: strongly-supervised and weakly-supervised regimes. Learning with strong supervision. In this setup, we assume that keypoint match annotations are given for each training image pair, as in [START_REF] Choy | Universal correspondence network[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]; each image pair is annotated with a set of coordinate pairs M = {(p m , p m )} M m=1 , where M is the number of match annotations.

To compare the output of our network with ground-truth annotations, we convert the annotations into a form of discrete correlation matrix. First of all, for each coordinate pair (p m , p m ), we identify their nearest position indices (k m , k m ) in hyperimages. On the one hand, given the set of identified match index pairs {(k m , k m )} M m=1 , we construct a ground-truth matrix G ∈ {0, 1} M ×|H | by assigning one-hot vector representation of k m to the m-th row of G. On the other hand, we construct Ĉ ∈ R M ×|H | by assigning the k m -th row of C to the m-th row of Ĉ. We apply softmax to each row of the matrix Ĉ after normalizing it to have zero mean and unit variance. Figure 2a illustrates the construction of Ĉ and G. Corresponding rows between Ĉ and G can now be compared as categorical probability distributions. We thus define the strongly-supervised matching loss as the sum of cross-entropy values between them:

L match = - 1 M M m=1 ω m |H | j=1 G mj log Ĉmj , (2)
where ω m is an importance weight for the m-th keypoint. The keypoint weight ω m helps training by reducing the effect of the corresponding cross-entropy term if the Eucliean distance between predicted keypoint p m and target keypoint p m is smaller than some threshold distance δ thres :

ω m = ( p m -p m /δ thres ) 2 if p m -p m < δ thres , 1 otherwise. ( 3
)
The proposed objective for strongly-supervised learning can also be used for self-supervised learning with synthetic pairs [45, 49] * , which typically results in trading off the cost of supervision against the generalization performance.

Learning with weak supervision. In this setup, we assume that only imagelevel labels are given for each image pair as either positive (the same class) or negative (different class), as in [START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF]. Let us denote the correlation matrix of a positive pair by C + and that of a negative pair by C -. For C ∈ R |H|×|H | , we define its correlation entropy as

s(C) = -1 |H| |H| i=1 |H | j=1 φ(C) ij log φ(C) ij
where φ(•) denotes row-wise L1-normalization. Higher correlation entropy indicates less distinctive correspondences between the two images. As illustrated in Fig. 2b, assuming that the positive images are likely to contain more distinctive correspondences, we encourage low entropy for positive pairs and high entropy for negative pairs. The weakly-supervised matching loss is formulated as

L match = s(C + ) + s(C + ) s(C -) + s(C -) . (4)
Layer selection loss. Following the work of [START_REF] Veit | Convolutional networks with adaptive inference graphs[END_REF], we add a soft constraint in our training objective to encourage the network to select each layer at a certain rate:

L sel = L-1 l=0 (z l -µ) 2
where zl is a fraction of image pairs within a mini-batch for which the l-th layer is selected and µ is a hyperparameter for the selection rate. This improves training by increasing diversity in layer selection and, as will be seen in our experiments, allows us to trade off between accuracy and speed in testing.

Finally, the training objective of our model is defined as the combination of the matching loss (either strong or weak) and the layer selection loss: L = L match + L sel .

Experiments

In this section we compare our method to the state of the art and discuss the results. The code and the trained model are available online at our project page.

Feature extractor networks. As the backbone networks for feature extraction, we use ResNet-50 and ResNet-101 [START_REF] He | Deep residual learning for image recognition[END_REF], which contains 49 and 100 conv layers in total (excluding the last FC), respectively. Since features from adjacent layers are strongly correlated, we extract the base block from conv1 maxpool and intermediate blocks from layers with residual connections (before ReLU). They amounts to 17 and 34 feature blocks (layers) in total, respectively, for ResNet-50 and ResNet-101. Following related work [START_REF] Choy | Universal correspondence network[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Kim | Recurrent transformer networks for semantic correspondence[END_REF][START_REF] Lee | Sfnet: Learning object-aware semantic correspondence[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF][START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF], we freeze the backbone network parameters during training for fair comparison. * For example, we can obtain keypoint annotations for free by forming a synthetic pair by applying random geometric transformation (e.g., affine or TPS [START_REF] Donato | Approximate thin plate spline mappings[END_REF]) on an image and then sampling some corresponding points between the original image and the warped image using the transformation applied.

Table 1: Performance on standard benchmarks in accuracy and speed (avg. time per pair). The subscript of each method name denotes its feature extractor. Some results are from [START_REF] Jeon | Parn: Pyramidal affine regression networks for dense semantic correspondence[END_REF][START_REF] Kim | Recurrent transformer networks for semantic correspondence[END_REF][START_REF] Lee | Sfnet: Learning object-aware semantic correspondence[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF]. Numbers in bold indicate the best performance and underlined ones are the second best. The average inference time (the last column) is measured on test split of PF-PASCAL [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF] and includes all the pipelines of the models: from feature extraction to keypoint prediction. Datasets. Experiments are done on four benchmarks for semantic correspondence: PF-PASCAL [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF], PF-WILLOW [START_REF] Ham | Proposal flow[END_REF], Caltech-101 [START_REF] Li | One-shot learning of object categories[END_REF], and SPair-71k [START_REF] Min | SPair-71k: A large-scale benchmark for semantic correspondence[END_REF]. PF-PASCAL and PF-WILLOW consist of keypoint-annotated image pairs, 1,351 pairs from 20 categories, and 900 pairs from 4 categories, respectively. Caltech-101 [START_REF] Li | One-shot learning of object categories[END_REF] contains segmentation-annotated 1,515 pairs from 101 categories. SPair-71k [START_REF] Min | SPair-71k: A large-scale benchmark for semantic correspondence[END_REF] is a more challenging large-scale dataset recently introduced in [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF], consisting of keypoint-annotated 70,958 image pairs from 18 categories with diverse view-point and scale variations.

Evaluation metrics. As an evaluation metric for PF-PASCAL, PF-WILLOW, and SPair-71k, the probability of correct keypoints (PCK) is used. The PCK value given a set of predicted and ground-truth keypoint pairs

P = {(p m , p m )} M m=1 is measured by PCK(P) = 1 M M m=1 1[ p m -p m ≤ α τ max (w τ , h τ )].
As an evaluation metric for the Caltech-101 benchmark, the label transfer accuracy (LT-ACC) [START_REF] Liu | Nonparametric scene parsing: Label transfer via dense scene alignment[END_REF] and the intersection-over-union (IoU) [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] are used. Running time (average time per pair) for each method is measured using its authors' code on a machine with an Intel i7-7820X CPU and an NVIDIA Titan-XP GPU.

Hyperparameters. The layer selection rate µ and the channel reduction factor ρ are determined by grid search using the validation split of PF-PASCAL. As a result, we set µ = 0.5 and ρ = 8 in our experiments if not specified otherwise. The threshold δ thres in Eq.( 3) is set to be max(w τ , h τ )/10.

Results and comparisons

First, we train both of our strongly and weakly-supervised models on the PF-PASCAL [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF] dataset and test on three standard benchmarks of PF-PASCAL (test split), PF-WILLOW and Caltech-101. The evaluations on PF-WILLOW and Caltech-101 are to verify transferability. In training, we use the same splits of PF-PASCAL proposed in [START_REF] Han | Scnet: Learning semantic correspondence[END_REF] where training, validation, and test sets respectively contain 700, 300, and 300 image pairs. Following [START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF], we augment the training pairs by horizontal flipping and swapping. Table 1 summarizes our result and those of recent methods [START_REF] Ham | Proposal flow[END_REF][START_REF] Han | Scnet: Learning semantic correspondence[END_REF][START_REF] Kim | Recurrent transformer networks for semantic correspondence[END_REF][START_REF] Kim | Dctm: Discrete-continuous transformation matching for semantic flow[END_REF][START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF]. Second, we train our model on the SPair-71k dataset [START_REF] Min | SPair-71k: A large-scale benchmark for semantic correspondence[END_REF] and compare it to other recent methods [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Rocco | Convolutional neural network architecture for geometric matching[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF]. Table 2 summarizes the results.

Strongly-supervised regime. As shown in the bottom sections of Table 1 and 2, our strongly-supervised model clearly outperforms the previous state of the art by a significant margin. It achieves 5.9%, 3.2%, and 9.1% points of PCK (α img = 0.1) improvement over the current state of the art [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF] on PF-PASCAL, PF-WILLOW, and SPair-71k, respectively, and the improvement increases further with a more strict evaluation threshold, e.g., more than 15% points of PCK with α img = 0.05 on PF-PASCAL. Even with a smaller backbone network (ResNet-50) and smaller selection rate (µ = 0.4), our method achieves competitive performance with the smallest running time on the standard benchmarks of PF-PASCAL, PF-WILLOW, and Caltech-101.

Weakly-supervised regime. As shown in the middle sections of Table 1 and 2, our weakly-supervised model also achieves the state of the art in the weaklysupervised regime. In particular, our model shows more reliable transferablility compared to strongly-supervised models, outperforming both weakly [START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF] and strongly-supervised [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF] state of the arts by 6.4% and 5.8% points of PCK respectively on PF-WILLOW. On the Caltech-101 benchmark, our method is comparable to the best among the recent methods. Note that unlike other benchmarks, the evaluation metric of Caltech-101 is indirect (i.e., accuracy of mask transfer). On the SPair-71k dataset, where image pairs have large view point and scale differences, the methods of [START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF] as well as ours do not successfully learn in the weakly-supervised regime; they (FT) all underperform transferred models (TR) trained on PF-PASCAL. This result reveals current weakly-supervised objectives are all prone to large variations, which requires further research in the future.

Effect of layer selection rate µ [START_REF] Veit | Convolutional networks with adaptive inference graphs[END_REF]. The plot in Fig. 3a shows PCK and running time of our models trained with different layer selection rates µ. It shows that smaller selection rates in training lead to faster running time in testing, at the cost of some accuracy, by encouraging the model to select a smaller number of layers. The selection rate µ can thus be used for speed-accuracy trade-off.

Analysis of layer selection patterns. Category-wise layer selection patterns in Fig. 3b show that each group of animal, vehicle, and man-made object categories shares its own distinct selection patterns. The model with a small rate (µ = 0.3) tends to select the most relevant layers only while the model with larger rates [START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF], (e) A2Net [START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF], (f) NC-Net [START_REF] Rocco | Neighbourhood consensus networks[END_REF], and (g) HPF [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF].

(µ > 0.3) tends to select more complementary layers as seen in Fig. 3c. For each µ ∈ {0.3, 0.4, 0.5} in Fig. 3c, the network tends to select low-level features for vehicle and man-made object categories while it selects mostly high-level features for animal category. We conjecture that it is because low-level (geometric) features such as lines, corners and circles appear more often in the vehicle and man-made classes compared to the animal classes. Figure 4 plots the frequencies over the numbers of selected layers with different selection rate µ, where vehicles tend to require more layers than animals and man-made objects.

Qualitative results. Some challenging examples on SPair-71k [START_REF] Min | SPair-71k: A large-scale benchmark for semantic correspondence[END_REF] and PF-PASCAL [START_REF] Ham | Proposal flow: Semantic correspondences from object proposals[END_REF] are shown in Fig. 5 and 6 respectively: Using the keypoint correspondences, TPS transformation [START_REF] Donato | Approximate thin plate spline mappings[END_REF] is applied to source image to align target image. The object categories of the pairs in Fig. 6 are in order of table, potted plant, and tv. Alignment results of each pair demonstrate the robustness of our model against major challenges in semantic correspondences such as large changes in view-point and scale, occlusion, background clutters, and intra-class variation.

Ablation study. We also conduct an ablation study to see the impacts of major components: Gumbel layer gating (GLG), conv feature transformation (CFT), probabilistic Hough matching (PHM), keypoint importance weight ω m , and layer selection loss L sel . All the models are trained with strong supervision 3 and Table 1.

Computational complexity. The average feature dimensions of our model before correlation computation are 2089, 3080, and 3962 for each µ ∈ {0.3, 0.4, 0.5} while those of recent methods [START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF][START_REF] Lee | Sfnet: Learning object-aware semantic correspondence[END_REF][START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF] are respectively 6400, 3072, 1024, 1024. The dimension of hyperimage is relatively small as GLG efficiently prunes irrelevant features and CFT effectively maps features onto smaller subspace, thus being more practical in terms of speed and accuracy as demonstrated in Table 1 and 3. Although [START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF] use lighter feature maps compared to ours, a series of 4D convolutions heavily increases time and memory complexity of the network, making them expensive for practical use (31ms (ours) vs. 261ms [START_REF] Rocco | Neighbourhood consensus networks[END_REF][START_REF] Huang | Dynamic context correspondence network for semantic alignment[END_REF]).

Comparison to soft layer gating

The Gumbel gating function in our dynamic layer gating can be replaced with conventional soft gating using sigmoid. We have investigated different types of soft gating as follows: (1) 'sigmoid': The MLP of dynamic gating at each layer predicts a scalar input for sigmoid and the transformed feature block pairs are weighted by the sigmoid output. (2) 'sigmoid µ=0.5 ': In training the 'sigmoid' gating, the layer selection loss L sel with µ = 0.5 is used to encourage the model to increase diversity in layer selection. (3) 'sigmoid + 1': In training the 'sigmoid' gating, the 1 regularization on the sigmoid output is used to encourage the soft selection result to be sparse. Table 4 summarizes the results and Fig. 7 compares their layer selection frequencies.

While the soft gating modules provide decent results, all of them perform worse than the proposed Gumbel layer gating in both accuracy and speed. The slower per-pair inference time of 'sigmoid' and 'sigmoid µ=0.5 ' indicates that soft gating is not effective in skipping layers due to its non-zero gating values. We find that the sparse regularization of 'sigmoid + 1' recovers the speed but only at the cost of significant accuracy points. Performance drop of soft gating in accuracy may result from the deterministic behavior of the soft gating during training that prohibits exploring diverse combinations of features at different levels. In contrast, the Gumbel gating during training enables the network to perform more comprehensive trials of a large number of different combinations of multi-level features, which help to learn better gating. Our experiments also show that discrete layer selection along with stochastic learning in searching the best combination is highly effective for learning to establish robust correspondences in terms of both accuracy and speed.

Conclusion

We have presented a dynamic matching network that predicts dense correspondences by composing hypercolumn features using a small set of relevant layers from a CNN. The state-of-the-art performance of the proposed method indicates that the use of dynamic multi-layer features in a trainable architecture is crucial for robust visual correspondence. We believe that our approach may prove useful for other domains involving correspondence such as image retrieval, object tracking, and action recognition. We leave this to future work.

Fig. 1 :

 1 Fig. 1: The overall architecture of Dynamic Hyperpixel Flow (DHPF).

  Weakly-supervised loss.

Fig. 2 :

 2 Fig. 2: Matching loss computation using (a) keypoint annotations (strong supervision) and (b) image pairs only (weak supervision). Best viewed in electronic form.

Fig. 3 :

 3 Fig. 3: Analysis of layer selection on PF-PASCAL dataset (a) PCK vs. running time with varying selection rate µ (b) Category-wise layer selection frequencies (x-axis: candidate layer index, y-axis: category) of the strongly-supervised model with different backbones: ResNet-101 (left) and ResNet-50 (right) (c) ResNet-101 layer selection frequencies of strongly (left) and weakly (right) supervised models at different layer selection rates µ. Best viewed in electronic form.

Fig. 4 :Fig. 5 :

 45 Fig.4: Frequencies over the numbers of selected layers with different selection rates µ (x-axis: the number of selected layers, y-axis: frequency). Best viewed in electronics.

Fig. 6 :

 6 Fig. 6: Example results on PF-PASCAL [16]: (a) source image, (b) target image and (c) DHPF (ours), (d) WeakAlign[START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF], (e) A2Net[START_REF] Seo | Attentive semantic alignment with offset-aware correlation kernels[END_REF], (f) NC-Net[START_REF] Rocco | Neighbourhood consensus networks[END_REF], and (g) HPF[START_REF] Min | Hyperpixel flow: Semantic correspondence with multi-layer neural features[END_REF].

Fig. 7 :

 7 Fig. 7: ResNet-101 layer selection frequencies for 'sigmoid' (left), 'sigmoidµ=0.5' (middle), and 'sigmoid + 1' (right) gating.

Table 2 :

 2 Performance on SPair-71k dataset in accuracy (per-class PCK with α bbox = 0.1). TR represents transferred models trained on PF-PASCAL while FT denotes fine-tuned (trained) models on SPair-71k. ] 23.[START_REF] Cho | Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals[END_REF] 17.0 41.6 14.6 37.6 28.1 26.6 32.6 12.6 27.9 23.0 13.6 21.3 22.2 17.9 10.9 31.5 34.8 21.1 FT WeakAlignres101 [46] 22.2 17.6 41.9 15.1 38.1 27.4 27.2 31.8 12.8 26.8 22.6 14.2 20.0 22.2 17.9 10.4 32.2 35.1 20.9 TR NC-Netres101 [47] 24.0 16.0 45.0 13.7 35.7 25.9 19.0 50.4 14.3 32.6 27.4 19.2 21.7 20.3 20.4 13.6 33.6 40.4 26.4 FT NC-Netres101 [47] 17.9 12.2 32.1 11.7 29.0 19.9 16.1 39.2 9.9 23.9 18.8 15.7 17.4 15.9 14.8 9.6 24.2 31.1 20.1 TR DHPFres101 (ours) 21.5 21.8 57.2 13.9 34.3 23.1 17.3 50.4 17.4 34.8 36.2 19.7 24.3 32.5 22.2 17.6 30.9 36.5 28.5 FT DHPFres101 (ours) 17.5 19.0 52.5 15.4 35.0 19.4 15.7 51.9 17.3 37.3 35.7 19.7 25.5 31.6 20.9 18.5 24.2 41.1 27.7 strong FT HPFres101 [42] 25.2 18.9 52.1 15.7 38.0 22.8 19.1 52.9 17.9 33.0 32.8 20.6 24.4 27.9 21.1 15.9 31.5 35.6 28.2 TR DHPFres101 (ours) 22.6 23.0 57.7 15.1 34.1 20.5 14.7 48.6 19.5 31.9 34.5 19.6 23.0 30.0 22.9 15.5 28.2 30.2 27.4 FT DHPFres101 (ours) 38.4 23.8 68.3 18.9 42.6 27.9 20.1 61.6 22.0 46.9 46.1 33.5 27.6 40.1 27.6 28.1 49.5 46.5 37.3

	Sup.	Methods	aero bike bird boat bottle bus car cat chair cow dog horse mbike person plant sheep train tv all
		TR CNNGeores101 [45] 21.3 15.1 34.6 12.8 31.2 26.3 24.0 30.6 11.6 24.3 20.4 12.2 19.7 15.6 14.3 9.6 28.5 28.8 18.1
	self	FT CNNGeores101 [45] 23.4 16.7 40.2 14.3 36.4 27.7 26.0 32.7 12.7 27.4 22.8 13.7 20.9 21.0 17.5 10.2 30.8 34.1 20.6 TR A2Netres101 [49] 20.8 17.1 37.4 13.9 33.6 29.4 26.5 34.9 12.0 26.5 22.5 13.3 21.3 20.0 16.9 11.5 28.9 31.6 20.1
		FT A2Netres101 [49]	22.6 18.5 42.0 16.4 37.9 30.8 26.5 35.6 13.3 29.6 24.3 16.0 21.6 22.8 20.5 13.5 31.4 36.5 22.3
		TR WeakAlignres101 [46
	weak

Table 3 :

 3 Ablation study on PF-PASCAL. (GLG: Gumbel layer gating with selection rates µ, CFT: conv feature transformation)

	Module	PCK (αimg) time
	GLG CFT PHM 0.05 0.1 0.15 (ms)
	0.5	75.7 90.7 95.0 58
	0.4	73.6 90.4 95.3 51
	0.3	73.1 88.7 94.4 47
		70.4 88.1 94.1 64
	0.5	43.6 74.7 87.5 176
	0.5	68.3 86.9 91.6 57
		37.6 68.7 84.6 124
		68.1 85.5 91.6 61
	0.5	35.0 54.8 63.4 173
	w/o ωm	69.8 86.1 91.9 57
	w/o L sel	68.1 89.2 93.5 56

Table 4 :

 4 Comparison to soft layer gating on PF-PASCAL.

	Gating function	PCK (αimg) time 0.05 0.1 0.15 (ms)
	Gumbelµ=0.5 75.7 90.7 95.0 58
	sigmoid	71.1 88.2 92.8 74
	sigmoidµ=0.5 72.1 87.8 93.3 75
	sigmoid + 1 65.9 87.2 91.0 60

Acknowledgements. This work is supported by Samsung Advanced Institute of Technology (SAIT) and also by Basic Science Research Program (NRF-2017R1E1A1A01077999) and Next-Generation Information Computing Development Program (NRF-2017M3C4A7069369) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT, Korea. Jean Ponce was supported in part by the Louis Vuitton/ENS chair in artificial intelligence and the Inria/NYU collaboration and also by the French government under management of Agence Nationale de la Recherche as part of the "Investissements dâavenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).