Dan Xu

Jingkuan Song

Xavier Alameda-Pineda

Elisa Ricci

Nicu Sebe

Multi-Paced Dictionary Learning for Cross-Domain Retrieval and Recognition

Several applications benefit from learning coupled representations able to describe data from multiple sources. For instance, cross-domain dictionary learning methods demonstrated to be particularly effective. In this paper we introduce Multi-Paced Dictionary Learning (MPDL) and propose an instantiation of it under the framework of cross-domain dictionary learning. MPDL is inspired by previous works on self-paced learning, a framework able to enhance the accuracy of conventional learning models by presenting the training data in a meaningful order, i.e. easy samples are provided first. However, most of existing self-paced learning methods only consider a single modality, while MPDL is specifically designed to assess the learning pace when data from multiple sources are available. We present the model and propose an efficient algorithm to learn the dictionaries and codes. The approach is validated via experiments on two different tasks, namely cross-media retrieval and sketchto-photo face recognition, using publicly available datasets.

I. INTRODUCTION

Many problems in computer vision, natural language processing and multimedia analysis require solving cross-modal knowledge association tasks. Dictionary learning approaches [START_REF] Yang | Image super-resolution via sparse representation[END_REF], [START_REF] Wang | Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[END_REF], [START_REF] Huang | Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition[END_REF] have proven to be successful for learning coupled representations using data from multiple domains. One recurrent issue with latent variable models, and with cross-modal dictionary learning in particular, is that the learning algorithms usually find sub-optimal solutions corresponding to bad local minima of the objective function.

Recently, self-paced learning (SPL) [START_REF] Kumar | Self-paced learning for latent variable models[END_REF] has raised as the framework insignia within which many classical learning techniques have shown to improve the generalization performance. The idea of self-paced learning is inspired from the way a good teacher instills knowledge to students: prioritizing easy training samples over the more difficult ones. Since the learning priority is understood from the data themselves, we are left with the challenging task of designing a meaningful strategy to assess the difficulty of the samples. Previous works have addressed this issue in different ways. The most common strategy is to measure the easiness of a sample by its loss [START_REF] Kumar | Self-paced learning for latent variable models[END_REF]. Alternatively, Jiang et al. [START_REF] Jiang | Easy samples first: Self-paced reranking for zero-example multimedia search[END_REF] defined the sample order taking into account the dissimilarity with respect to what has already been learned. A hybrid strategy, where prior knowledge (i.e. a curriculum [START_REF] Bengio | Curriculum learning[END_REF]) is integrated into the computation of the easiness measure, was presented in [START_REF] Jiang | Self-paced curriculum learning[END_REF].

In this paper we address the problem of inferring the sample order using data coming from multiple sources. Specifically, we introduce Multi-Paced Dictionary Learning and propose an Fig. 1. Application of the multi-paced dictionary learning method to sketchto-photo face recognition. Faces and sketches are represented as sparse codes by means of the (learned) dictionaries D 1 and D 2 . The shape size represents the reconstruction error of the corresponding codes s 1 n , s 2 n . Intuitively, our method first selects the sample associated with the red circle, then with the yellow triangle and finally with the blue square. Details are given in main text.

instantiation of it under the framework of cross-domain dictionary learning. MPDL is related to both curriculum learning and SPL. As explained above, while in curriculum learning the ordering is set by the "teacher" in advance, in SPL the ordering is data-driven, i.e. inferred from the "student's" understanding. The rationale behind MPDL is that, within this teacher-student analogy, most successful students learn from multiple sources. In other words the learning pace is jointly determined by the knowledge gathered from the teacher, textbook, electronic resources, etc. Importantly, the learning order must be decided, not only upon one particular explanation of a concept, but also taking into account how much two interpretations of the same concept from different sources match. In the framework of dictionary learning, this corresponds to evaluating not only the representation power of each domain's dictionary, but also the coherence between the domain-specific codes. Thus, the learning rhythm is determined by multiple sources, giving rise to MPDL.

Figure 1 illustrates the idea behind the proposed method. MPDL first trains with those samples that have a good trade-off between reconstruction error and code matching. Intuitively, our method first selects the sample associated with the red circle, then with the yellow triangle and finally with the blue square. Even if the yellow codes show low reconstruction error, they do not match. Contrarily, the red codes match very well while keeping a fairly low reconstruction error. Finally, the samples associated with blue codes are selected at the end, as they correspond to higher reconstruction errors. To assess the appropriateness of our intuition, we derive a novel coupled dictionary learning framework and propose an efficient algorithm to learn the dictionaries and codes and simultaneously infer the optimal sample order. Finally, we evaluate the effectiveness of our framework by conducting an extensive experimental evaluation on publicly available benchmarks for two applications, namely cross-media retrieval and sketch-to-photo face recognition.

II. RELATED WORK

In this section we describe related work on (i) cross-domain dictionary learning and (ii) self-paced and curriculum learning, since multi-paced dictionary learning lies in the cross-road of these two research directions.

In the last few years, several dictionary learning algorithms have been proposed [START_REF] Aharon | K-svd: An algorithm for designing overcomplete dictionaries for sparse representation[END_REF], [START_REF] Mairal | Online dictionary learning for sparse coding[END_REF], [START_REF] Lee | Efficient sparse coding algorithms[END_REF], pushing the state-of-theart in many applications. When data from different domains are available, traditional dictionary learning approaches can be extended to benefit from the information encoded in multiple sources. In [START_REF] Yang | Image super-resolution via sparse representation[END_REF], corresponding samples from different domains were used to learn per-source dictionaries in a coupled fashion. Wang et al. [START_REF] Wang | Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[END_REF] introduced a semi-coupled dictionary learning scheme for cross-domain image sparse representation. Huang and Wang [START_REF] Huang | Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition[END_REF] proposed a framework to learn a pair of dictionaries along with the corresponding feature representations. Finally, a multi-task dictionary learning framework was proposed in [START_REF] Liu | Inferring painting style with multi-task dictionary learning[END_REF] in the context of the automatic analysis of paintings to learn multiple style-specific dictionary representations.

Coupled dictionary learning strategies have shown to be especially effective in transfer learning tasks, i.e. when knowledge obtained from a given dataset (source) is used to facilitate the learning on another dataset (target). For instance, Ding et al. [START_REF] Ding | Latent low-rank transfer subspace learning for missing modality recognition[END_REF] presented a framework to handle the missing modality problem (target domain data are not available). Similarly, Ni et al. [START_REF] Ni | Subspace interpolation via dictionary learning for unsupervised domain adaptation[END_REF] used dictionaries to learn a set of subspaces minimizing the distance between the data distributions of source and target domains.

Curriculum [START_REF] Bengio | Curriculum learning[END_REF] and self-paced [START_REF] Kumar | Self-paced learning for latent variable models[END_REF] learning develop from the idea that models must be learned in an incremental fashion, using the easy samples before the difficult ones. However, while curriculum learning requires the a priori identification of easy and hard samples in a given training dataset, in self-paced the learning order is automatically determined from the data themselves. Due to its generality, curriculum and self-paced learning have been considered in a broad spectrum of latent variable models, including matrix factorization [START_REF] Zhao | Self-paced learning for matrix factorization[END_REF], domain adaptation [START_REF] Tang | Shifting weights: Adapting object detectors from image to video[END_REF], dictionary learning [START_REF] Tang | Self-paced dictionary learning for image classification[END_REF] and clustering [START_REF] Xu | Multi-view self-paced learning for clustering[END_REF]. Among all previous works, the two studies most related to this paper are [START_REF] Tang | Self-paced dictionary learning for image classification[END_REF] and [START_REF] Xu | Multi-view self-paced learning for clustering[END_REF]. In [START_REF] Tang | Self-paced dictionary learning for image classification[END_REF] a SPL strategy is proposed in the context of dictionary learning. Opposite to [START_REF] Tang | Self-paced dictionary learning for image classification[END_REF], in this paper we analyze the challenging problem of combining data from multiple domains and develop a novel dictionary learning framework for coupling multiple modalities and selecting the sample order accordingly. In [START_REF] Xu | Multi-view self-paced learning for clustering[END_REF] the traditional SPL scheme is embedded in an algorithm which clusters multi-view data. As a consequence, the order of datapoints is determined considering each view in isolation, while in our MPDL the samples are sorted by jointly looking at multiple domains.

III. THE MODEL OF MPDL

In this section we present our MPDL approach. We first introduce the problem of dictionary learning when samples are gathered from multiple data sources. Then, we describe the proposed MPDL method along with the algorithm we devised to solve the associated optimization problem.

A. Cross-Domain Dictionary Learning

We assume the existence of N samples observed in I different modalities. We denote the feature vector of the n th sample in the i th modality as x i n ∈ R Mi , where M i is the dimension of the i th feature space. We choose to instantiate MPDL within the framework of sparse coding and dictionary learning [START_REF] Aharon | K-svd: An algorithm for designing overcomplete dictionaries for sparse representation[END_REF], [START_REF] Mairal | Online dictionary learning for sparse coding[END_REF], [START_REF] Lee | Efficient sparse coding algorithms[END_REF]. More precisely, we assume the existence of I dictionaries -one per modality -that represent the data as x i n ≈ D i s i n , where s i n ∈ R K is the code of the n th sample in the i th modality and D i ∈ R Mi×K is the dictionary of the i th modality containing K words. Compactly, we write:

X i ≈ D i S i , (1)
where

X i = [x i 1 , . . . , x i N ] and S i = [s i 1 , . . . , s i N ]
. Dictionary learning boils down to a loss minimization problem with normalization constraints and sparse regularization:

min {S i ,D i } I i=1 I i=1 X i -D i S i 2 2 + α S i 1 s.t. d i k ≤ 1, (2)
where d i k is the k th word of the i th dictionary, i.e. ,

D i = [d i 1 , . . . , d i K ].
The constraint allows to skip scale ambiguities due to the matrix product D i S i .

Prior relational knowledge among the samples is very useful when learning new representations [START_REF] Zheng | Graph regularized sparse coding for image representation[END_REF], [START_REF] Song | Multiple feature hashing for real-time large scale near-duplicate video retrieval[END_REF]. To embed this knowledge into dictionary learning, we propose to use the original set of features X i to create an undirected proximity graph. In practice, the weights of this graph w i nm can be computed, for instance, with the Gaussian kernel. The weights are then used to create the Laplacian matrix L i with

l i mn = N p=1 w i np -w i nm .
A regularization term is then added to the dictionary problem (2) leading to:

min {S i ,D i } I i=1 I i=1 X i -D i S i 2 2 +α S i 1 +βTr S i L i S i s.t. d i k ≤ 1. (3)
The problem (3) learns a set of domain-specific dictionaries along with the associated code independently for each domain.

Previous works [START_REF] Wang | Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[END_REF], [START_REF] Yang | Image super-resolution via sparse representation[END_REF], [START_REF] Huang | Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition[END_REF] have shown that a coupled learning framework can be beneficial in several applications. Our MPDL approach develops from this intuition. However, opposite to previous works, in MPDL modality-specific dictionaries and codes are learned jointly by presenting the training data in a meaningful order, i.e. easy samples are provided first. Our experiments demonstrate that the proposed strategy is superior to previous multi-domain dictionary learning methods [START_REF] Wang | Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[END_REF], [START_REF] Huang | Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition[END_REF].

B. The Formulation of MPDL

As in curriculum and self-paced learning, MPDL develops from the idea that models must be learned in an incremental fashion, using the easy samples before the difficult ones. To model the easiness of learning the n th sample in MPDL a binary variable v n ∈ {0, 1} is introduced that indicates whether or not the n th sample is currently selected for training. Consequently only the samples selected for training are taken into account and the i th dictionary loss becomes

(X i -D i S i )V 2 2
, where V is a N -by-N diagonal matrix with entries v 1 , . . . , v N . As it is traditionally done, the method iteratively increases the amount of samples selected for learning by adding a regularization term with a penalty evolving over time [START_REF] Kumar | Self-paced learning for latent variable models[END_REF], [START_REF] Jiang | Easy samples first: Self-paced reranking for zero-example multimedia search[END_REF], [START_REF] Jiang | Self-paced curriculum learning[END_REF]. In the present study, we assume that the data is split into C groups (either provided or learned from the data) and define a group-specific indicator vector p c ∈ R N , where p c,n = 1 if and only if sample n belongs to group c, and p c,n = 0 otherwise. We devise a penalty over V that is normalized over the groups' size, denoted by B c :

C c=1 1 B c Vp c 1 . (4)
Importantly, the penalty induced by this term has to evolve over time so to allow more and more samples to be part of the training set. At the same time, this term enforces learning from different groups and therefore it is closely related to SPL with diversity [START_REF] Jiang | Selfpaced learning with diversity[END_REF]. Similarly to [START_REF] Jiang | Selfpaced learning with diversity[END_REF], the idea is to learn not only from easy samples as in standard SPL [START_REF] Kumar | Self-paced learning for latent variable models[END_REF] but also from samples that are dissimilar from what has already been learned. However, with respect to [START_REF] Jiang | Selfpaced learning with diversity[END_REF], the proposed regularizer has two prominent advantages:(i) we avoid using group norms that significantly increase the complexity of the optimization solvers and (ii) we introduce the normalization factor B c that softens the bias induced by dissimilar group cardinalities. In all:

min V,{S i ,D i } I i=1 I i=1 (X i -D i S i )V 2 2 +α S i 1 +βTr S i L i S i -µ C c=1 1 B c Vp c 1 s.t. d i k ≤ 1 and v n ∈ {0, 1}. (5)
As delineated in the introduction, MPDL aims to determine the learning pace from multiple sources, that is, from the different I domains. Importantly, MPDL assesses the learning easiness of the n th sample, not only from the representation precision in each of the I domains (as is already done in [START_REF] Jiang | Easy samples first: Self-paced reranking for zero-example multimedia search[END_REF]), but also from the correspondence between cross-domain representations. This is the main conceptual contribution of the present research study. Moreover, within the framework of dictionary learning this is naturally implemented as a penalty over the codes' dissimilarity. Importantly, this dissimilarity is also weighted by the pacing variables V. The regularization term is defined as:

i<j S i -S j V 2 2 . (6)
Finally, the optimization problem for MPDL writes:

min V,{S i ,D i } i i (X i -D i S i )V 2 2 +α S i 1 + βTr S i L i S i -µ c Vp c 1 B c +γ i<j S i -S j V 2 2 s.t. d i k ≤ 1 and v n ∈ {0, 1}, (7)
where the regularization parameters α, β, µ, γ balance the effect of sparseness, the prior knowledge, the general pacing regime and the plurality of the learning pace, respectively.

C. The Optimization of MPDL

The optimization problem for MPDL is not jointly convex. We propose an alternating optimization approach to solve [START_REF] Jiang | Self-paced curriculum learning[END_REF] in three steps: dictionaries, codes and pacing variables. Solve for D i : With fixed codes S i and selection matrix V, the optimization problem for D i writes:

min D i (X i -D i S i )V 2 2 s.t. d i k ≤ 1. ( 8
)
This problem is a Quadratically Constrained Quadratic Program (QCQP) that can be solved using gradient descent with iterative projection [START_REF] Albers | Quadratic minimisation problems in statistics[END_REF] or Lagrangian duality, e.g. [START_REF] Lee | Efficient sparse coding algorithms[END_REF].

Solve for S i : With fixed dictionaries D i and selection matrix V, the optimization function for the codes writes:

f (S i ) = I i=1 X i -D i S i V 2 2 + α S i 1 +βTr S i L i S i + γ i<j S i -S j V 2 2 (9)
According to FISTA [START_REF] Beck | A fast iterative shrinkage-thresholding algorithm for linear inverse problems[END_REF], ( 9) can be viewed as a proximal regularization problem, iteratively solved using the following recursion (over r):

S i r = argmin S i S i -S i r-1 -t r ∇f S i r-1 2 2 2t r + α S i 1 , (10
) where 1 :

∇f (S i ) = 2D i (D i S i -X)V + 2βS i L i + 2γ j =i (S i -S j )V (11)
is the gradient of f (S i ) and t r > 0 is a suitable step size.

Since the 1 norm is separable, the computation of each s i n,r

reduces to solving a one-dimensional minimization problem for each of its components, which by simple calculus produces s i n,r = τ αtr s i n,r-1t r ∇f s i n,r-1 , where τ λ is the shrinkage operator defined by: τ λ (x) m = (|x m |λ) + sgn (x m ) and m is the dimensionality of x. Solve for V: By fixing the dictionaries D i and the codes S i , we can update V solving:

min V I i=1 X i -D i S i V 2 2 + γ i<j S i -S j V 2 2 -µ c Vp c 1 B c , s.t. v n ∈ {0, 1}. (12)
Because V is a diagonal matrix with binary entries, we can rewrite the previous optimization problem as N independent optimization problems of the following form:

min vn v n (e n + γf n -µg n ) s.t. v n ∈ {0, 1}, (13)
with e n = I i=1

x i n -D i s i n 2 2 , f n = i<j s i n -s j n || 2 2 , g n = 1 Bc n
, being c n the class of the n th sample. The solution of the previous optimization problem is trivial and given by:

v * n = 1 if e n + γf n < µg n , 0 otherwise. ( 14
)
Importantly, this solution should be understood with the following intuition. If the error of a training point is smaller than a threshold µg n , the training point will be selected for training.

To make sure that the training samples are equally selected independently of the clusters, we impose a higher threshold (determined by g n ) for large classes/clusters. The summary of the learning procedure is shown in Algorithm 1.

IV. EXPERIMENTS

In this section, we extensively evaluate the proposed approach when applied to two tasks, namely cross-media retrieval and sketch-to-photo face recognition.

A. Cross-media retrieval

Cross-media retrieval aims at robustly finding query results in various media by submitting queries in one of the media. We use the proposed method to retrieve text and images from the widely used "Wikipedia" dataset [START_REF] Rasiwasia | A new approach to cross-modal multimedia retrieval[END_REF].

Setup. The Wikipedia dataset consists of 2866 image-text pairs generated from featured articles of Wikipedia. These articles are continuously updated and selected by the Wikipedia editors since 2009. While the text is a semantic description of people, places or events, the images are a visual representation of them: both modalities are complementary. Each text-image pair sample is labeled with one out of 10 possible semantic classes. The dataset is randomly split in 2173 training samples and 693 test samples. Each image is then represented by a 128-dimensional codeword obtained using the bag-of-words framework over SIFT features. The text descriptor is computed by a latent Dirichlet allocation (LDA) model with 10 topics, as in [START_REF] Rasiwasia | A new approach to cross-modal multimedia retrieval[END_REF]. Following previous works on dictionary learning for retrieval tasks, after training our MPDL, we used the learned dictionaries to compute the codes associated to test samples. These codes are then employed for cross-media retrieval applying a nearest neighbor scheme.

Baselines. We compared the proposed approach with several state-of-the-art methods: Random retrieval, Cross-modal Factor Analysis (CFA) [START_REF] Li | Multimedia content processing through cross-modal association[END_REF], Canonical Correlation Analysis (CCA) [START_REF] Rasiwasia | A new approach to cross-modal multimedia retrieval[END_REF], Partial Least Squares (PLS) [START_REF] Rosipal | Overview and recent advances in partial least squares[END_REF], Linear Cross-Modal Hashing (LCMH) [START_REF] Zhu | Linear cross-modal hashing for efficient multimedia search[END_REF], Cross-modality Metric Learning with Similarity-Sensitive Hashing (CMLSSH) [START_REF] Bronstein | Data fusion through cross-modality metric learning using similarity-sensitive hashing[END_REF], Cross-View Hashing (CVH) [START_REF] Kumar | Learning hash functions for cross-view similarity search[END_REF] and Multi-modal Stacked Auto-Encoders (MSAE) [START_REF] Wang | Effective multi-modal retrieval based on stacked auto-encoders[END_REF]. Importantly, our method learns the 1 We used V 2 = V. dictionary in an unsupervised manner, where only pair-wise information is utilized. Semantic information is also important for cross media retrieval, but constructing high-level semantic description usually requires label information [START_REF] Rasiwasia | A new approach to cross-modal multimedia retrieval[END_REF]. Therefore, we only consider and compare with unsupervised methods. The chosen baselines can be grouped by the way they achieve cross-modal retrieval: CFA and CCA learn modality-specific subspaces; LCMH, CVH, CMLSSH and PLS find a projection matrix by means of spectral hashing schemes or correlation metrics; and MSAE uses two parallel multi-layer stacked autoencoder networks to learn cross-modal correlations. To guarantee a fair comparison all methods use the same visual features (except MSAE that uses the raw images). The dictionary size was set to

K 1 = K 2 = 7.
Results. We report mean average precision (MAP) performance of all benchmarked methods on the four possible singleand cross-media retrieval tasks in Table I 2 . The regularization parameters α, β and γ of our method were set by crossvalidation to 1, 0.1 and 1, respectively. In addition to the proposed MPDL method with the cross-validated parameters, we also show two special cases, namely γ = 0, i.e. no crossmodal coupling, and µ → ∞, i.e. disabling the self-pacing feature of the method.

We remark that MPDL systematically obtains the best performance among all methods. Specifically, for single-media retrieval, our method obtains MAP scores of 0.185 and 0.512 for the text-to-text query task (Q T →T ) and the image-to-image query task (Q I→I ), respectively. Q I→I shows a significant gap of 2.4 points over MSAE, the best state-of-the-art method. This is probably because MSAE fails to learn joint representations from low-resolution images. For cross-modal retrieval, our method achieves 0.274 for Q I→T , which is 2.5 points above the best (CCA), and 0.198 for Q T →I , outperforming all methods. The results show the beneficial effect of assessing the learning pace from cross-modal correspondences, that is when compared to MPDL (γ = 0). A similar conclusion can be drawn when the learning pace is disabled.

In addition to the MAP, we plot precision-recall (PR) curves for the two cross-modal tasks in Figure 3. The PR results are consistent with Table I, and our method outperforms previous approaches. Finally, we also show some qualitative results in Figure 2, where the images on the right correspond to the first five images selected from the text query on the left.

B. Sketch-to-photo face recognition

The sketch has become an important clue in image query systems, specially if a picture of the face can be retrieved from the sketch. We therefore applied the proposed method to the sketch-to-photo recognition task. Experiments were conducted on the CUHK Face Sketch Database (CUFS) [START_REF] Wang | Face photo-sketch synthesis and recognition[END_REF].

Setup. CUFS contains pair-wise samples of sketch and photo faces collected from 188 CUHK students, and a few pair examples from it are shown in Fig. 4. In our experiments, 88 sketch-photo pairs are randomly selected for training the model, and the remaining 100 pairs are used for testing. The recognition task is to identify the face photo corresponding to given a sketch. Similarly to the experiments on the Wikipedia dataset, the dictionaries learned during training are used to calculate the codes associated to test samples. Then, recognition is based on the computed codes.

Baselines. We compare the MPDL method to seven stateof-the-art methods, namely: CCA, Sketch Transform Method (STM) [START_REF] Tang | Face sketch recognition[END_REF], PLS [START_REF] Rosipal | Overview and recent advances in partial least squares[END_REF], [START_REF] Sharma | Bypassing synthesis: Pls for face recognition with pose, low-resolution and sketch[END_REF], Bilinear model (Bil) [START_REF] Tenenbaum | Separating style and content with bilinear models[END_REF], Semi-Coupled Dictionary Learning (SCDL) [START_REF] Wang | Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[END_REF], Joint Dictionary Learning (JDL) [START_REF] Yang | Image super-resolution via sparse representation[END_REF] and Coupled Dictionary Learning (CDL) [START_REF] Huang | Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition[END_REF]. Similarly to MPDL, the SCDL, JDL and CDL methods are based on dictionary learning, and the dictionary size is set to 50 in all cases. For the bilinear model, we used 70 PLS bases and 50 eigenvectors (see [START_REF] Sharma | Bypassing synthesis: Pls for face recognition with pose, low-resolution and sketch[END_REF]).

Results. Our model learns sparse representation from the raw sketch and face images. The regularization parameters α, β and γ were set by cross-validation to 1, 0.1 and 4.5 respectively. The dictionaries were initialized using joint dictionary learning [START_REF] Yang | Image super-resolution via sparse representation[END_REF]. The recognition is done using nearest neighbor classifiers, as in [START_REF] Sharma | Bypassing synthesis: Pls for face recognition with pose, low-resolution and sketch[END_REF], [START_REF] Huang | Coupled dictionary and feature space learning with applications to cross-domain image synthesis and recognition[END_REF], on the newly learned sparse representation.

Table II (b) Q I→T Fig. 5. MAP for the cross-modal retrieval tasks as a function of the dictionary size for MPDL and MPDL (µ → ∞) on the Wikipedia dataset. dictionary based approaches. Among the compared methods, SCDL, JDL and CDL are the strongest competitors, achieving 95.2%, 95.4% and 97.4% recognition rate respectively. This means that dictionary learning for cross-modal learning is a promising line of research further improved by the proposed MPDL. Specially, when the learning pace is driven by the data as shown by the 1.1% improvement over MPDL (µ → ∞).

C. Analysis of MPDL

In order to further analyze the proposed approach we first assess the performance as a function of the dictionary size. Figure 5 show MAP measures for the Q T →I and Q I→T tasks of MPDL and MPDL (µ → ∞) as a function of the dictionary size. Generally speaking, MPDL outperforms its non-paced version for most values of the dictionary size.

We also discuss and provide experimental evidence of the convergence of the learning algorithm. As illustrated in Section III-C, our MPDL relies on an alternating optimization approach and the proposed optimization problem ( 7) is solved separately with respect to dictionaries D i , codes S i and selfpaced variables V. According to [START_REF] Meng | What objective does self-paced learning indeed optimize?[END_REF] the alternative search implemented in Algorithm 1 converges. Moreover, as illustrated in Fig. 6 for the Wikipedia dataset, we experimentally observe that MPDL attains a stable solution within few iterations, proving the efficiency of the algorithm proposed to solve the MPDL optimization problem. Regarding computational complexity, it is worth noting that for MPDL, as well as for any algorithm incorporating a SPL scheme, the improvement in terms of generalization performance comes at the price of an increased computational cost. However, in case of the considered retrieval and recognition tasks, the training time is not particularly important, as training is performed only once in an offline phase. Oppositely, the test phase, where codes are computed by projecting test samples into the learned dictionaries, is very efficient.

To conclude this in-depth analysis of the proposed method, we remark that both applications clearly show the advantage of using the cross-modal coupling term. In other words MPDL outperforms MPDL (γ = 0) by at least 2 points and up to 4 points (see Tables I andII).

V. CONCLUSIONS

This study introduces the multi-paced learning framework and an instantiation of it under dictionary learning. The rationale between multi-paced learning is that, when data from multiple modalities are available, the learning pace of each sample should be determined, not only by the reconstruction error (loss) in each modality but also from the coherence of the representations across modalities. We formulated the multipaced dictionary learning problem and propose an efficient algorithm to solve it. An extensive evaluation campaign is performed on two cross-modal data analysis tasks, namely cross-modal retrieval and sketch-to-photo face recognition. Future works will focus on improving the proposed framework, e.g. exploiting an hybrid curriculum/multi-paced learning scheme [START_REF] Jiang | Self-paced curriculum learning[END_REF], and on extending our multi-domain approach in case of other latent variables models.

Algorithm 1 6 :

 16 Multi-Paced Dictionary Learning Input: Initialize D i , S i using single modality dictionary learning[START_REF] Yang | Image super-resolution via sparse representation[END_REF], µ to use 10% of the training set and the parameters α, β, γ;Update the v n 's following (14);

7 :

 7 Increase µ by λ to enlarge the training set; 8: until All the training data points are selected. Output: D i , S i , V;

Fig. 2 .

 2 Fig. 2. Two examples of image query text (the first row) and text query image (the second row). Top four results are retrieved by our method on the Wikipedia dataset.

  Q I→T Fig.3. Precision-recall curves of different methods for the cross-modal retrieval tasks.

Fig. 4 .

 4 Fig. 4. Example of face-sketch pairs in the CUFS dataset.TABLE II AVERAGE RECOGNITION RATE FOR ALL BENCHMARKED METHODS ON

Fig. 6 .

 6 Fig. 6. The objective function value at varying number of iterations on Wikipedia dataset.

TABLE I MEAN

 I AVERAGE PRECISION OF ALL BENCHMARKED METHODS ON DIFFERENT RETRIEVAL TASKS USING THE WIKIPEDIA DATASET.

	Algorithm		Retrieval Task
		Q I→I	Q T →T	Q I→T	Q T →I
	Random	0.118	0.118	0.118	0.118
	CFA	0.157	0.488	0.246	0.195
	CCA	0.161	0.495	0.249	0.196
	PLS	0.158	0.490	0.240	0.163
	LCMH	0.146	0.359	0.133	0.117
	CMLSSH	0.148	0.318	0.138	0.140
	CVH	0.147	0.153	0.126	0.122
	MSAE	0.162	0.462	0.182	0.179
	MPDL (γ = 0)	0.171	0.494	0.237	0.185
	MPDL (µ → ∞)	0.181	0.498	0.269	0.191
	MPDL	0.186	0.512	0.274	0.198

TABLE II AVERAGE

 II RECOGNITION RATE FOR ALL BENCHMARKED METHODS ON SKETCH-TO-PHOTO FACE RECOGNITION.

	Algorithm	Average Recognition Rate
	STM	81.0%
	PLS	93.6%
	Bil	94.2%
	CCA	94.6%
	SCDL	95.2%
	JDL	95.4%
	CDL	97.4%
	MPDL(γ = 0)	96.5%
	MPDL (µ → ∞)	97.3%
	MPDL	98.4%

  reports average recognition results over five trials. MPDL achieves the best average recognition rate: 98.4%. Remarkably, MPDL outperforms CDL, which is the best of the

		0.2 0.22				MPDL (7!1) MPDL	0.26 0.28				MPDL (7!1) MPDL
		0.18
	MAP	0.16				MAP	0.24
		0.14					0.22
		0.12					0.2
		0.1	3	5	7	The dictionary size 10 15 20 30 50 60 100 200	0.18	3	5	7	The dictionary size 10 15 20 30 50 60 100 200
						(a) Q T →I

The numbers corresponding to the performance of the other approaches are partially taken from[START_REF] Wang | Effective multi-modal retrieval based on stacked auto-encoders[END_REF].