Indrayush De

Jean-Luc Battaglia

Gérard L Vignoles

Thermal properties measurements of a silica/pyrocarbon composite at the microscale

Laminar pyrocarbons are used as interphases or matrices of carbon/carbon and ceramic-matrix composites in several high-temperature aerospace applications. Depending on their organization at the microscale, they can have a variety of mechanical and thermal properties. Hence, it is important to know, before thermal processing, the properties of these matrices at the micrometer scale in order to improve and control the composite behavior in a macroscopic scale. We use Scanning Thermal Microscopy (SThM) on a silica fiber / regenerative laminar pyrocarbon matrix composite to provide an insight into the effective thermal conductivity of pyrocarbon as well as the thermal contact resistance at the interface between fiber and matrix. The conductivity of pyrocarbon is discussed as a function of its nanostructural organization.

I Introduction

Carbon/carbon (C/C) composite materials are choice materials for use in extreme environments, such as space propulsion rocket nozzles, atmospheric re-entry thermal protection systems, aircraft brake discs, and Tokamak plasma-facing components [START_REF] Savage | Carbon-Carbon composites[END_REF] . In addition to carbon fibers, they contain interphases and matrices made of pyrolytic carbon, or pyrocarbon (PyC) [START_REF] Manocha | Carbon reinforcements and C/C composites[END_REF] . This special type of carbon can be though of as a heavily faulted graphite. It is prepared via a gas-phase route, called Chemical Vapor Deposition (CVD) or Infiltration (CVI). It is therefore quite unavailable in bulk form. It has, depending on its processing parameters, a very versatile nanostructure [START_REF] Oberlin | Pyrocarbons[END_REF][4][5] and consequently, broadly varying mechanical and thermal properties, usually anisotropic to a more or less large extent [START_REF] Vignoles | Carbones Pyrolytiques ou Pyrocarbonesdes Matériaux Multi-Echelles et Multi-Performances[END_REF] . Posterior heat treatments may further alter their structure and properties [START_REF] Oberlin | Carbonization and graphitization[END_REF] . Hence, it is important to know the properties of these matrices at the micrometer scale in order to improve and control the composite behavior in a macroscopic scale. In this frame, a large variety of PyC samples have been prepared 8 . That represented in Fig. 1 consists of an asdeposited regenerative laminar (ReL) PyC 9 deposit made on 5-µm radius glass fibers. The general orientation of the anisotropic texture is concentric around the fibers, as exhibited in Fig. 2, and results in orthotropic thermal properties of the matrix in the cylindrical coordinate frame following the fiber axis. This is due to the fact that the graphitic sheets exhibit strong thermal anisotropy. The thermal behavior of these non-homogeneous composites can be captured through characterization that will provide the thermal properties of the PyC. Previous thermoreflectance (TR) experiments [START_REF] Jumel | AIP Conf. Procs[END_REF][START_REF] Jumel | [END_REF][START_REF] Jumel | AIP Conf. Procs[END_REF] have been performed to assess the anisotropic thermal diffusivity of the Smooth Laminar (SL) PyC and of the Rough Laminar (RL) PyC, either pristine or after different heat treatments. It was obtained that the in-plane thermal diffusivity (in orthoradial direction) for the as-prepared SL PyC matrix was 0.14 cm².s -1 while the ratio of the in-plane and out-of-plane thermal diffusivities was 7; the as-prepared RL exhibits higher figures (0.42 cm 2 .s -1 and 20, respectively), denoting a more graphitic and anisotropic structure. ReL PyC, which is a highly anisotropic form of PyC, differs from RL by a larger amount of defects [START_REF] Farbos | [END_REF] and had not been investigated so far. The TR method has in the current case some possible drawbacks: first, its spatial resolution is of the same size as the deposit thickness, a fact that could result in inaccuracies; second, this method requires a rather strong temperature increase on the heating area in order to increase the signal-tonoise ratio, therefore yielding an effective diffusivity characteristic at temperature markedly higher than the ambient and nonlinear effects. On the other hand, the thermal boundary resistance (TBR) at the interface between the fiber and the matrix has not been investigated so far. Since the thermal conductivity for both the silica fiber and the PyC along the radial axis is low, the TBR was not expected to be a key parameter on heat transfer. However, its quantitative identification from measurements at the microscale could bring complementary information regarding the chemical bonding and/or structural arrangement at the interface 14 . In order (i) to overcome the drawbacks of the TR method, (ii) to provide thermal conductivity value for ReL PyC, and (iii) to measure as well the thermal boundary resistance at the interface between the PyC and the glass fiber, we have implemented the scanning thermal microscopy (SThM) experiment involving the 3ω mode 15 . The advantage of using SThM is that (i) the spatial resolution achieved is in the submicron scale and (ii) that high temperature differences are not involved, avoiding thus any risk of nonlinearity. In addition, SThM leads to absolute temperature measurements of the probe as well as to phase measurements when working under the 3 mode. Therefore, advanced inverse techniques can be implemented that can benefit from the frequency and spatial variations of both functions in order to investigate the thermal properties of the PyC and the TBR at the interface between the fiber and the matrix. In the present study the fiber is made of a single glass structure whose properties are available in literature 16 ( k =1.4 W.m -1 .K -1 , r = 2200kg.m -3 , C p = 787 J.kg -1 .K -1 ). The density and specific heat of ReL PyC have been also measured as [START_REF] Farbos | [END_REF]  = 2110 kg.m -3 and C p = 748 J.K -1 .kg -1 respectively.

II Scanning Thermal Microscopy in 3 mode -experiment

Scanning thermal microscopy is a well-established and almost ideal tool for investigating nanostructures like semiconductors and nano-electronic devices, due to its intrinsic sensitivity with respect to local material properties and to the thermal wave's ability to propagate through the material with sub micrometer lateral resolution. Since its inception in 1986, based on the principle of Scanning Thermal Profile (STP) [START_REF] Williams | [END_REF] , SThM has seen a lot of improvements and developments [18][19][20][21] including the Wollaston probe and thermocouple probes. The employed SThM (Nanosurf Easyscan AFM) uses a 400 nm-thick silicon nitride (Si 3 N 4 ) AFM probe provided by Kelvin Nanotechnology, on which is deposited a palladium strip (Pd, 1.2 µm thick and 10 µm long) that plays both the roles of the heater and of the a)

10 µm b) ReL PyC Silica fiber c) d) OA = 27°
thermometer. The SThM probe has a tip curvature radius of r s = 100 nm. The contact force between the probe and the sample was chosen between 5 and 10 nN during our experiments and was accurately controlled during the probe motion using a feedback-closed loop on a piezo element, which ensures the displacement in the z direction with precise steps of 1 nm. The contact area between the probe and the surface is assumed to be a disk with radius r 0 . A periodic current I = I 0 cos w t ( ) with angular frequency w = 2p f passes through the strip with electrical resistance R 0 at room temperature, generating Joule's effect and, thus, being a heat source, dissipating the power

P 2w ( ) = P 0 1+ cos 2w t ( ) ( ) 2
DT 2w = 2U 3w R 0 I 0 a R .
In our configuration, R 0 =155  and I 0 =750 µA. The quantity DT 2w must be viewed as an average temperature value of the Pd wire since the change is expected to occur very close to the probe tip when it enters into contact with the investigated material surface. The harmonic contribution is measured using a differential stage coupled with a lock-in amplifier. The thermal coefficient of the Pd strip was calibrated by measuring the change in sensor resistance as a function of temperature and a value of a

R = 1.3± 0.2 ( ) ´10 -3
K -1 was obtained. The contact between the probe and the surface involves a thermal boundary resistance that plays a very significant role on the measured temperature. This resistance involves at least three main contributions 22,23 : (i) the solid-solid contact resistance, (ii) the heat diffusion through the gas surrounding the probe and the water meniscus that forms inevitably at the probe tip and (iii) the change in the temperature gradient within the Pd strip between the out-of-contact mode (used to evaluate the probe thermal impedance) and when the probe comes into contact with the surface. In the present study, although working under argon flow (after a preliminary air removal from primary vacuum), the diffusion through the gas and the water meniscus may still be present even if its contribution is bounded below. Moreover, for silicon nitride probes, the increase in the contact area can also be explained by the flattening of the tip apex when in contact with the sample. As we showed in a previous study 18 , the thermal contact resistance integrates all the physical phenomena listed above and that occur to form the thermal contact resistance R c at the interface between the probe and the surface. On the other hand, we observed 18 that, in this experimental condition, the contact resistance as well as the radius r 0 of the heated area did not vary significantly when the thermal conductivity of the sample varied from 0.2 to 40 W.m -1 .K -1 . This observation was made considering the probe was motionless and the roughness of all samples was less than 10 nm. Finally, it was also observed 18 that the sensitivity to the thermal conductivity variation started to vanish above 25 W.m -1 .K -1 . This is obviously related to the very small contact area since the smaller this area, the lower is the sensitivity to thermal conductivity change for high conductive samples. Finally, it was observed that the measured phase did not vary significantly with respect to the measurement error whatever the thermal conductivity of the material. The heat transfer in the probe and the investigated material, in both the out-of-contact and contact modes, is described in Fig. 3, using the thermal impedances formalism [START_REF] Maillet | Thermal quadrupoles: solving the heat equation through integral transforms[END_REF] . This contact mode configuration assumed that the probe was located on either the fiber or the matrix and was only sensitive to the thermal properties of the contacting material. In other words, this configuration suggests that the probe was put in static contact on both materials far away enough from the interface. Denoting w 2 = 2w , the average temperature T p w ( ) dq of the amplitude versus the parameter  = {k r,PyC , k z,PyC , r 0 , R c }, have been calculated and reported on the Fig. 4. Sensitivity on k r,PyC and k z,PyC being exactly the same, they cannot be identify separately. Therefore, the measurements achieved when the probe is in contact with the PyC will only lead to identify its effective thermal conductivity k PyC,eff . On the other hand, the ratio of the sensitivity functions to r 0 and R c is constant; there is thus no chance to identify separately those two parameters simultaneously from frequency dependent measurements.

III Scanning Thermal Microscopy in 3 modeheat transfer model

Z m w 2 ( ) = 2 k z,m p r 0 J 1 (x) é ë ù û 2 x x 2 a r ,m a z,m + j w 2 r 0 2 a z,m dx 0 ¥ ò , j 2 = -1
In order to simulate the probe temperature when the probe sweeps the surface of the composite at a given frequency, we used the analytical model derived by Lepoutre et al. [START_REF] Lepoutre | [END_REF] assuming semi-infinite domains on both sides of the interface. This assumption is realistic since the probe is only sensitive to the material bulk thermal conductivity at distances that do not exceed 5 to 6 times the contact radius r 0 . In addition, in order to validate this assumption, we also performed calculations based on a finite elements model that is not presented here. They confirm the reliability of the solutions obtained using the analytical model that requires less computation times and that can thus be implemented in an inverse procedure to estimate the sought parameters. We assume here the motion of the probe is along the radial direction r, when the probe passes from the fiber to the matrix, through the interface. The reduced sensitivity functions

S

A q,r ( ) =q dA r ( ) dq of the amplitude to the parameters = {k r,Pyc , k z,PyC , r 0 , R c , TBR} at the frequency 1125 Hz, when r varied from -0.5 to 0.5 µm assuming the interface is at r=0, are represented in figure 5a.

The sensitivity functions with respect to r 0 and R c , are, as for the frequency behavior, linearly dependent. However, it appears, as revealed on Fig. 5b, that the parameters k r,Pyc , r 0 and TBR can be identified since the ratios of the associated sensitivity functions are not constant along r.

Whatever the experimental configuration for the probe, static or dynamic, the sought properties are identified by minimizing the quadratic gap:

J = A p r ,w 2 ( ) -A meas r ,w 2 ( ) ( ) 2

IV Experimental results

In a first time, we estimated the contact radius r 0 at the probe/surface interface using a calibrated "step" sample [START_REF] Puyoo | [END_REF] that consists in a 100 nm thick SiO 2 step deposited on a Si substrate. We found r 0 =100 ±10 nm with a constant force of 10 nN applied on the cantilever. It is assumed this value for the contact radius when the probe is in contact either with the glass fiber or the PyC considering the same force (10 nN) applied on the cantilever. We performed frequency dependent measurements when the probe was out-ofcontact and in static contact at the center of the glass fiber and on the PyC matrix far away from the fiber. Figure 6 shows the measured frequency dependent amplitude (Fig. 6a) and phase (Fig. 6b) for those three configurations. As said previously, we retrieve that the difference in the phase for each condition is very small, meaning that only the amplitude can be used. The out-of-contact measurements lead to the probe thermal impedance Z p ( 2 ). Then, since the silica thermal properties are known, we identified the thermal contact resistance at the interface between the SThM probe and the material starting from the amplitude measurements when the probe is in static contact at the center of the fiber. Using the minimization technique and the model for the probe in static contact with the material, we found that

R c = 7.83± 0.3 ( ) ´10 -8 K.m 2 .W -1 .
The fit between experimental data and the theoretical ones is very satisfying as presented in Fig. 6. Finally, using this value for R c , we identified the effective thermal conductivity of the PyC from the measured amplitude when the probe is in static contact with the PyC. We found that k PyC,eff = 20.8± 4.2 W.m -1

.K -1 , which leads again to a very satisfactory fit between the measurements ad the simulation as showed in Fig. 6. It must be emphasized that the standard deviation on the identified thermal conductivity is high (20% uncertainty) since a small variation in r 0 leads to a very large change in k PyC,eff . We noticed also that the minimum value of the quadratic J that is minimized is obtained for r 0 = 110 nm . This value is thus in the expected range obtained using the calibrated sample.

We have however to mention that the value we found for k PyC,eff is at the detection limit (25 W.m -1 .K -1 ) of the instrument, as said in section II. .K -1 and r 0 = 105± 7 nm . The calculated temperature along  using those identified parameters is reported in Fig. 8 with the simulations. Therefore we retrieve well the values for r 0 and k PyC,eff that have been determined using the step sample and the frequency identification procedure.

V Conclusion

The SThM method has been applied to a composite made of silica fibers embedded in a regenerative laminar pyrocarbon (RL PyC) matrix. It has allowed obtaining values of the effective conductivity of this type of pyrocarbon, therefore completing the existing database obtained by TR on other types of PyC. The method has proved efficient in yielding effective values of the thermal conductivity. Unfortunately, it cannot give the details of the conductivity tensor elements; the uncertainty margin is also rather large. On the other hand, it allows identifying the thermal boundary resistance between the carbon matrix and silica fibers. A main advance in the field of scanning thermal microscopy is that we implemented an inverse technique in order to identify simultaneously (i) the radius of the contact area between the probe and the sample, (ii) the thermal conductivity of the sample and (iii) the effective thermal conductivity of the PyC. Therefore, whereas it was shown that the frequency dependent temperature at a point located on the surface could not lead to this simultaneously identification, it has been demonstrated in this paper that such an identification can be achieved considering from the spatial temperature variation at a given frequency. However, it is obviously required working with a heterogeneous surface where at least some of the materials are known in terms of their thermal conductivity (in the present case, the SiO 2 fiber). As measured by TR experiments [START_REF] Jumel | AIP Conf. Procs[END_REF][START_REF] Jumel | [END_REF][START_REF] Jumel | AIP Conf. Procs[END_REF] , thermal conductivity of RL and SL PyC are respectively 66.7 and 20.4 W.K -1 .m -1 (using the same heat capacity, and densities of 2120 kg.m -3 for RL [START_REF] Jumel | AIP Conf. Procs[END_REF] and 1930 kg.m -3 for SL [START_REF] Jumel | [END_REF] ). Actually both RL and ReL have the same degree of textural anisotropy, as measured e.g. by polarized light optical microscopy or by selected area electron diffraction in a transmission electron microscope, and only differ by the amount of in-plane defects, as measured by X-ray diffraction, neutron diffraction and by Raman spectroscopy 29 , and confirmed by HRTEM image-based atomistic modeling 30 . On the other hand, SL has a lesser anisotropy but a comparable, though lesser, amount of defects as compared to ReL. We conclude here that the room temperature conductivity is more sensitive to structural perfection than to textural arrangement. Indeed, either phonons or electrons, which are responsible for heat transfer in carbons, are scattered by the defects present in the planes.

The value of TBR is unexpectedly rather low. A possible reason for this low value is that the PyC finds itself in a state of compression around the fiber: as a matter of fact, no decohesion has been found between the fibers and the matrices. Another effect is the fact that, on the carbon side, the conductivity is much larger parallel to the interface instead of perpendicularly, therefore providing easy "escape routes" to heat around defects present at the interface. Therefore, the hypothesis of 1D transfer across the interface could be questioned. Finally, we have also to mention that the surface of the sample is not fully flat at the interface between the fiber and the PyC. This comes from the different mechanical properties of both materials and their impact on the roughness at the end of the surface polishing. On the other hand, the fiber being an insulator already, the sensitivity of the measured temperature vs. the TBR remains low. Additional measurements of the TBR between a carbon fiber and the PyC matrix are in course.

Further investigations are desirable in at least two directions. First, the SThM method should be improved in order to reduce its large degree of uncertainty and to obtain direction-dependent data. Second, measurements should be carried out on other pyrocarbons and fibers, in order to confirm the tendencies obtained here; measurements at higher temperatures are possible and would be highly interesting, since virtually no actual experimental data is available on these materials at elevated temperatures.

"PyroMaN" project, funded by ANR. The authors thank the late Patrick Weisbecker (1973-2015) for the TEM images.

Figure 1 .

 1 Figure 1. Microscopic Image of the Composite Structure obtained using a Scanning Electron Microscope (SEM). Silica fibers (5 µm in radius) in grey color, surrounded by the PyC matrix in dark grey color, are perpendicular to the surface that was prepared by mechanical surfacing.

Figure 2 .

 2 Figure 2. a) Image showing the silica fiber and the cylindrical arrangement of the graphitic sheets; b) SEM image at a higher resolution confirming the presence of a concentric arrangement of the anisotropic texture, c) Dark field TEM image showing the anisotropic nature of the pyrocarbon, d) High-Resolution 002 Lattice Fringe TEM imaging of the pyrocarbon; the inset is a Selected Area Electron Diffraction Diagram illustrating the high anisotropy through the low value of the 002 diffraction arc opening angle.

2 .

 2 The resulting temperature increase ΔT in the strip is composed of a continuous component (DC) and of a periodic one at 2ω, as DT = DT DC + DT 2w cos 2w t +f ( ) . This leads to changes of the strip electrical resistance as R= R is the thermal coefficient. The voltage drop between the two pads of the probe is therefore expressed according to , 2  and 3 .The third harmonic U 3w is related to the transient contribution of the temperature change to resistance as

Figure 3 .

 3 Figure 3. Heat transfer model for the out-of-contact operation mode and the contact mode (considering the probe in contact either with the glass fiber or the PyC matrix). T p OFC is the out-of-contact probe temperature that is used to identify the probe thermal impedance Z p (). The current generator represents the heat source, localized within the Pd strip of the probe.

2 () 2 ( ) P w 2 ( 2 (p w 2 ( 2 (

 222222 of the Pd strip in the contact mode is related to the total heat flux ) . Expressions for Z p and Z m are required to calculate Tp . We have chosen to express the thermal impedance of the probe as Z ) are respectively the amplitude and the phase measured within the out-of-contact mode. Finally, the heat transfer model within the investigated material leads to express the thermal impedance Z m in an analytical way as25 :

Figure 4 .A w 2 ( ) = T p w 2 ( ) and d p w 2 ( 2 ( ) =q dA w 2

 422222 Figure 4. Reduced sensitivities of the amplitude A w 2( ) to: the axial and radial thermal conductivity (k r,PyC , k z,Pyc ) of the PyC, the contact radius r 0 and the thermal contact resistance R c at the interface between the probe and the investigated surface, as a function of frequency. Ratio between sensitivity functions are also presented.In this relation, k z,m is the thermal conductivity along the longitudinal axis (perpendicular to the investigated surface) and a r,m and a z,m are the thermal diffusivity of the material (either the fiber or the matrix, i.e., m=SiO 2 or PyC) along the longitudinal and radial axis respectively. J 1 is the Bessel function of the first kind of order 1. Finally, the theoretical expressions for the temperature and the phase are respectively:A w

Figure 5 .

 5 Figure 5. a/ Reduced sensitivity of the amplitude A r ( ) = T p r ,w 2( ) with respect to R c , r 0 , k r,PyC , k z,PyC and TBR at 1125 Hz, along a path crossing the fiber/matrix interface b/ reduced sensitivity ratios along the same path.

Figure 6 .(

 6 Figure 6. Measured amplitude (a) and phase (b) vs. 2. Plain lines are simulations using the identified R c when the probe is in contact with the silica fiber (green line) and the identified k eff of the PyC when the probe is in contact with the PyC (red line). We performed an SThM sweep of the specimen at 1125 Hz with a current of 750 µA under atmospheric condition. The topography, amplitude and phase images are recorded during the sweep. The experiments have been performed first over an image edge size of 50 micrometers with 256 points of measurement per line at a speed of 0.25 lines per second (fig. 7a). Then, a sweep over the  domain (see fig. 7a) has been performed and reported on Fig. 7b. The value for the amplitude along line  (see Fig. 7b), when the probe moves from the fiber to the PyC, is represented in Fig. 8. Using the minimization technique described previously, we found TBR= 5±1 ( ) ´10 -8 K.m 2 .W -1 , k PyC,eff = 20.18± 0.12 W.m -1

Figure 7 .

 7 Figure 7. a/ The 5050 µm 2 images obtained via experiments under atmospheric conditions using Scanning Thermal Microscopy at 1125 Hz showing the topography, amplitude and phase from left to right respectively. b/ sweep over the  domain.

Figure 8 .

 8 Figure 8. Green circles: measured probe temperature along the  line (see Fig. 7) and simulated probe temperature values considering the identified values for TBR, r 0 and k PyC,eff and two different values for TBR (in K.m 2 .W -1 ) in order to show the sensitivity of this parameter on the calculated temperature.

Acknowledgement

This work has been funded by Conseil Régional d'Aquitaine and the CNRS 102758 project with Epsilon Engineering Company. The observed sample was produced during the execution of the BLAN-10-0929


Aliaksandr Siarohin

Enver Sangineto

Stéphane Lathuilière
email: stephane.lathuiliere@inria.fr

Nicu Sebe
email: niculae.sebe@unitn.it

Deformable GANs for Pose-based Human Image Generation

In this paper we address the problem of generating person images conditioned on a given pose. Specifically, given an image of a person and a target pose, we synthesize a new image of that person in the novel pose. In order to deal with pixel-to-pixel misalignments caused by the pose differences, we introduce deformable skip connections in the generator of our Generative Adversarial Network. Moreover, a nearest-neighbour loss is proposed instead of the common L 1 and L 2 losses in order to match the details of the generated image with the target image. We test our approach using photos of persons in different poses and we compare our method with previous work in this area showing state-of-the-art results in two benchmarks. Our method can be applied to the wider field of deformable object generation, provided that the pose of the articulated object can be extracted using a keypoint detector.

Introduction

In this paper we deal with the problem of generating images where the foreground object changes because of a viewpoint variation or a deformable motion, such as the articulated human body. Specifically, inspired by Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF], our goal is to generate a human image conditioned on two different variables: [START_REF] Cao | Realtime multiperson 2D pose estimation using part affinity fields[END_REF] the appearance of a specific person in a given image and (2) the pose of the same person in another image. The task our networks need to solve is to preserve the appearance details (e.g., the texture) contained in the first variable while performing a deformation on the structure of the foreground object according to the second variable. We focus on the human body which is an articulated "object", important for many applications (e.g., computer-graphics based manipulations or re-identification dataset synthesis). However, our approach can be used with other deformable objects such as human faces or animal bodies, provided that a significant number of keypoints can be automatically extracted from the object of interest in order to represent its pose.

Pose-based human-being image generation is motivated by the interest in synthesizing videos [START_REF] Walker | The pose knows: Video forecasting by generating pose futures[END_REF] with non-trivial human movements or in generating rare poses for human pose estimation [START_REF] Cao | Realtime multiperson 2D pose estimation using part affinity fields[END_REF] or re-identification [START_REF] Zheng | Unlabeled samples generated by GAN improve the person re-identification baseline in vitro[END_REF] training datasets. However, most of the recently proposed, deepnetwork based generative approaches, such as Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF] or Variational Autoencoders (VAEs) [START_REF] Kingma | Auto-encoding variational bayes[END_REF] do not explicitly deal with the problem of articulated-object generation. Common conditional methods (e.g., conditional GANs or conditional VAEs) can synthesize images whose appearances depend on some conditioning variables (e.g., a label or another image). For instance, Isola et al. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] recently proposed an "image-toimage translation" framework, in which an input image x is transformed into a second image y represented in another "channel" (see Fig. 1a). However, most of these methods have problems when dealing with large spatial deformations between the conditioning and the target image. For instance, the U-Net architecture used by Isola et al. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] is based on skip connections which help preserving local information between x and y. Specifically, skip connections are used to copy and then concatenate the feature maps of the generator "encoder" (where information is downsam-pled using convolutional layers) to the generator "decoder" (containing the upconvolutional layers). However, the assumption used in [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] is that x and y are roughly aligned with each other and they represent the same underlying structure.

This assumption is violated when the foreground object in y undergoes to large spatial deformations with respect to x (see Fig. 1b). As shown in [START_REF] Ma | Pose guided person image generation[END_REF], skip connections cannot reliably cope with misalignments between the two poses. Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF] propose to alleviate this problem using a two-stage generation approach. In the first stage a U-Net generator is trained using a masked L 1 loss in order to produce an intermediate image conditioned on the target pose. In the second stage, a second U-Net based generator is trained using also an adversarial loss in order to generate an appearance difference map which brings the intermediate image closer to the appearance of the conditioning image. In contrast, the GAN-based method we propose in this paper is end-to-end trained by explicitly taking into account pose-related spatial deformations. More specifically, we propose deformable skip connections which "move" local information according to the structural deformations represented in the conditioning variables. These layers are used in our U-Net based generator. In order to move information according to a specific spatial deformation, we decompose the overall deformation by means of a set of local affine transformations involving subsets of joints, then we deform the convolutional feature maps of the encoder according to these transformations and we use common skip connections to transfer the transformed tensors to the decoder's fusion layers. Moreover, we also propose to use a nearest-neighbour loss as a replacement of common pixelto-pixel losses (such as, e.g., L 1 or L 2 losses) commonly used in conditional generative approaches. This loss proved to be helpful in generating local information (e.g., texture) similar to the target image which is not penalized because of small spatial misalignments.

We test our approach using the benchmarks and the evaluation protocols proposed in [START_REF] Ma | Pose guided person image generation[END_REF] obtaining higher qualitative and quantitative results in all the datasets. Although tested on the specific human-body problem, our approach makes few human-related assumptions and can be easily extended to other domains involving the generation of highly deformable objects. Our code and our trained models are publicly available 1 .

Related work

Most common deep-network-based approaches for visual content generation can be categorized as either Variational Autoencoders (VAEs) [START_REF] Kingma | Auto-encoding variational bayes[END_REF] or Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. VAEs are based on probabilistic graphical models and are trained by maximizing a lower 1 https://github.com/AliaksandrSiarohin/pose-gan bound of the corresponding data likelihood. GANs are based on two networks, a generator and a discriminator, which are trained simultaneously such that the generator tries to "fool" the discriminator and the discriminator learns how to distinguish between real and fake images.

Isola et al. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] propose a conditional GAN framework for image-to-image translation problems, where a given scene representation is "translated" into another representation. The main assumption behind this framework is that there exits a spatial correspondence between the low-level information of the conditioning and the output image. VAEs and GANs are combined in [START_REF] Zhao | Multi-view image generation from a single-view[END_REF] to generate realistic-looking multi-view clothes images from a single-view input image. The target view is filled to the model via a viewpoint label as front or left side and a two-stage approach is adopted: pose integration and image refinement. Adopting a similar pipeline, Lassner et al. [START_REF] Lassner | A generative model of people in clothing[END_REF] generate images of people with different clothes in a given pose. This approach is based on a costly annotation (fine-grained segmentation with 18 clothing labels) and a complex 3D pose representation.

Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF] propose a more general approach which allows to synthesize person images in any arbitrary pose. Similarly to our proposal, the input of their model is a conditioning image of the person and a target new pose defined by 18 joint locations. The target pose is described by means of binary maps where small circles represent the joint locations. Similarly to [START_REF] Lassner | A generative model of people in clothing[END_REF][START_REF] Zhao | Multi-view image generation from a single-view[END_REF], the generation process is split in two different stages: pose generation and texture refinement. In contrast, in this paper we show that a single-stage approach, trained end-to-end, can be used for the same task obtaining higher qualitative results.

Jaderberg et al. [START_REF] Jaderberg | Spatial transformer networks[END_REF] propose a spatial transformer layer, which learns how to transform a feature map in a "canonical" view, conditioned on the feature map itself. However only a global, parametric transformation can be learned (e.g., a global affine transformation), while in this paper we deal with non-parametric deformations of articulated objects which cannot be described by means of a unique global affine transformation.

Generally speaking, U-Net based architectures are frequently adopted for pose-based person-image generation tasks [START_REF] Lassner | A generative model of people in clothing[END_REF][START_REF] Ma | Pose guided person image generation[END_REF][START_REF] Walker | The pose knows: Video forecasting by generating pose futures[END_REF][START_REF] Zhao | Multi-view image generation from a single-view[END_REF]. However, common U-Net skip connections are not well-designed for large spatial deformations because local information in the input and in the output images is not aligned (Fig. 1). In contrast, we propose deformable skip connections to deal with this misalignment problem and "shuttle" local information from the encoder to the decoder driven by the specific pose difference. In this way, differently from previous work, we are able to simultaneously generate the overall pose and the texture-level refinement.

Finally, our nearest-neighbour loss is similar to the perceptual loss proposed in [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] and to the style-transfer spatial-analogy approach recently proposed in [START_REF] Liao | Visual attribute transfer through deep image analogy[END_REF]. However, the perceptual loss, based on an element-by-element difference computed in the feature map of an external classifier [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF], does not take into account spatial misalignments. On the other hand, the patch-based similarity, adopted in [START_REF] Liao | Visual attribute transfer through deep image analogy[END_REF] to compute a dense feature correspondence, is very computationally expensive and it is not used as a loss.

The network architectures

In this section we describe the architectures of our generator (G) and discriminator (D) and the proposed deformable skip connections. We first introduce some notation. At testing time our task, similarly to [START_REF] Ma | Pose guided person image generation[END_REF], consists in generating an image x showing a person whose appearance (e.g., clothes, etc.) is similar to an input, conditioning image x a but with a body pose similar to P (x b ), where x b is a different image of the same person and P (x) = (p 1 , ...p k ) is a sequence of k 2D points describing the locations of the human-body joints in x. In order to allow a fair comparison with [START_REF] Ma | Pose guided person image generation[END_REF], we use the same number of joints (k = 18) and we extract P () using the same Human Pose Estimator (HPE) [START_REF] Cao | Realtime multiperson 2D pose estimation using part affinity fields[END_REF] used in [START_REF] Ma | Pose guided person image generation[END_REF]. Note that this HPE is used both at testing and at training time, meaning that we do not use manually-annotated poses and the so extracted joint locations may have some localization errors or missing detections/false positives.

At training time we use a dataset X = {(x

(i) a , x (i)
b )} i=1,...,N containing pairs of conditioningtarget images of the same person in different poses. For each pair (x a , x b ), a conditioning and a target pose P (x a ) and P (x b ) is extracted from the corresponding image and represented using two tensors H a = H(P (x a )) and H b = H(P (x b )), each composed of k heat maps, where H j (1 ≤ j ≤ k) is a 2D matrix of the same dimension as the original image. If p j is the j-th joint location, then:

H j (p) = exp - p -p j σ 2 , (1)
with σ = 6 pixels (chosen with cross-validation). Using blurring instead of a binary map is useful to provide widespread information about the location p j . The generator G is fed with: (1) a noise vector z, drawn from a noise distribution Z and implicitly provided using dropout [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] and (2) the triplet (x a , H a , H b ). Note that, at testing time, the target pose is known, thus H(P (x b )) can be computed. Note also that the joint locations in x a and H a are spatially aligned (by construction), while in H b they are different. Hence, differently from [START_REF] Ma | Pose guided person image generation[END_REF][START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], H b is not concatenated with the other input tensors. Indeed the convolutional-layer units in the encoder part of G have a small receptive field which cannot capture large spatial displacements. For instance, a large movement of a body limb in x b with respect to x a , is represented in different locations in x a and H b which may be too far apart from each other to be captured by the receptive field of the convolutional units. This is emphasized in the first layers of the encoder, which represent low-level information. Therefore, the convolutional filters cannot simultaneously process texture-level information (from x a ) and the corresponding pose information (from H b ).

For this reason we independently process x a and H a from H b in the encoder. Specifically, x a and H a are concatenated and processed using a convolutional stream of the encoder while H b is processed by means of a second convolutional stream, without sharing the weights (Fig. 2). The feature maps of the first stream are then fused with the layerspecific feature maps of the second stream in the decoder after a pose-driven spatial deformation performed by our deformable skip connections (see Sec. 3.1).

Our discriminator network is based on the conditional, fully-convolutional discriminator proposed by Isola et al. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF]. In our case, D takes as input 4 tensors: (x a , H a , y, H b ), where either y = x b or y = x = G(z, x a , H a , H b ) (see Fig. 2). These four tensors are concatenated and then given as input to D. The discriminator's output is a scalar value indicating its confidence on the fact that y is a real image.

Deformable skip connections

As mentioned above and similarly to [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], the goal of the deformable skip connections is to "shuttle" local information from the encoder to the decoder part of G. The local information to be transferred is, generally speaking, contained in a tensor F , which represents the feature map activations of a given convolutional layer of the encoder. However, differently from [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], we need to "pick" the information to shuttle taking into account the object-shape deformation which is described by the difference between P (x a ) and P (x b ).

To do so, we decompose the global deformation in a set of local affine transformations, defined using subsets of joints in P (x a ) and P (x b ). Using these affine transformations and local masks constructed using the specific joints, we deform the content of F and then we use common skip connections to copy the transformed tensor and concatenate it with the corresponding tensor in the destination layer (see Fig. 2). Below we describe in more detail the whole pipeline.

Decomposing an articulated body in a set of rigid subparts. The human body is an articulated "object" which can be roughly decomposed into a set of rigid sub-parts. We chose 10 sub-parts: the head, the torso, the left/right upper/lower arm and the left/right upper/lower leg. Each of them corresponds to a subset of the 18 joints defined by the HPE [START_REF] Cao | Realtime multiperson 2D pose estimation using part affinity fields[END_REF] we use for extracting P (). Using these joint locations we can define rectangular regions which enclose the specific body part. In case of the head, the region is simply chosen to be the axis-aligned enclosing rectangle of all the corresponding joints. For the torso, which is Figure 2: A schematic representation of our network architectures. For the sake of clarity, in this figure we depict P (•) as a skeleton and each tensor H as the average of its component matrices H j (1 ≤ j ≤ k). The white rectangles in the decoder represent the feature maps directly obtained using up-convolutional filters applied to the previous-layer maps. The reddish rectangles represent the feature maps "shuttled" by the skip connections from the H b stream. Finally, blueish rectangles represent the deformed tensors d(F ) "shuttled" by the deformable skip connections from the (x a , H a ) stream.

the largest area, we use a region which includes the whole image, in such a way to shuttle texture information for the background pixels. Concerning the body limbs, each limb corresponds to only 2 joints. In this case we define a region to be a rotated rectangle whose major axis (r 1 ) corresponds to the line between these two joints, while the minor axis (r 2 ) is orthogonal to r 1 and with a length equal to one third of the mean of the torso's diagonals (this value is used for all the limbs). In Fig. 3 we show an example. Let R a h = {p 1 , ..., p 4 } be the set of the 4 rectangle corners in x a defining the h-th body region (1 ≤ h ≤ 10). Note that these 4 corner points are not joint locations. Using R a h we can compute a binary mask M h (p) which is zero everywhere except those points p lying inside R a h . Moreover, let R b h = {q 1 , ..., q 4 } be the corresponding rectangular region in x b . Matching the points in R a h with the corresponding points in R b h we can compute the parameters of a body-part specific affine transformation (see below). In either x a or x b , some of the body regions can be occluded, truncated by the image borders or simply miss-detected by the HPE. In this case we leave the corresponding region R h empty and the h-th affine transform is not computed (see below).

Note that our body-region definition is the only humanspecific part of the proposed approach. However, similar regions can be easily defined using the joints of other articulated objects such as those representing an animal body or a human face.

Computing a set of affine transformations. During the forward pass (i.e., both at training and at testing time) we decompose the global deformation of the conditioning pose with respect to the target pose by means of a set of local affine transformations, one per body region. Specifically, given R a h in x a and R b h in x b (see above), we compute the 6 parameters k h of an affine transformation f h (•; k h ) using Least Squares Error: Figure 3: For each specific body part, an affine transformation f h is computed. This transformation is used to "move" the feature-map content corresponding to that body part.

min k h pj ∈R a h ,qj ∈R b h ||q j -f h (p j ; k h )|| 2 2 (2)
The parameter vector k h is computed using the original image resolution of x a and x b and then adapted to the specific resolution of each involved feature map F . Similarly, we compute scaled versions of each M h . In case either R a h or R b h is empty (i.e., when any of the specific body-region joints has not been detected using the HPE, see above), then we simply set M h to be a matrix with all elements equal to 0 (f h is not computed).

Note that (f h (), M h ) and their lower-resolution variants need to be computed only once per each pair of real images (x a , x b ) ∈ X and, in case of the training phase, this is can be done before starting training the networks (but in our current implementation this is done on the fly).

Combining affine transformations to approximate the object deformation. Once (f h (), M h ), h = 1, ..., 10 are computed for the specific spatial resolution of a given tensor F , the latter can be transformed in order to approximate the global pose-dependent deformation. Specifically, we first compute for each h:

F h = f h (F M h ), (3)
where is a point-wise multiplication and f h (F (p)) is used to "move" all the channel values of F corresponding to point p. Finally, we merge the resulting tensors using:

d(F (p, c)) = max h=1,...,10 F h (p, c), ( 4
)
where c is a specific channel. The rationale behind Eq. 4 is that, when two body regions partially overlap each other, the final deformed tensor d(F ) is obtained by picking the maximum-activation values. Preliminary experiments performed using average pooling led to slightly worse results.

Training

D and G are trained using a combination of a standard conditional adversarial loss L cGAN with our proposed nearest-neighbour loss L N N . Specifically, in our case L cGAN is given by:

L cGAN (G, D) = E (xa,x b )∈X [log D(x a , H a , x b , H b )]+ E (xa,x b )∈X ,z∈Z [log(1 -D(x a , H a , x, H b ))], (5)
where x = G(z, x a , H a , H b ).

Previous works on conditional GANs combine the adversarial loss with either an L 2 [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] or an L 1 -based loss [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Ma | Pose guided person image generation[END_REF] which is used only for G. For instance, the L 1 distance computes a pixel-to-pixel difference between the generated and the real image, which, in our case, is:

L 1 (x, x b ) = ||x -x b || 1 . (6)
However, a well-known problem behind the use of L 1 and L 2 is the production of blurred images. We hypothesize that this is also due to the inability of these losses to tolerate small spatial misalignments between x and x b . For instance, suppose that x, produced by G, is visually plausible and semantically similar to x b , but the texture details on the clothes of the person in the two compared images are not pixel-to-pixel aligned. Both the L 1 and the L 2 loss will penalize this inexact pixel-level alignment, although not semantically important from the human point of view. Note that these misalignments do not depend on the global deformation between x a and x b , because x is supposed to have the same pose as x b . In order to alleviate this problem, we propose to use a nearest-neighbour loss L N N based on the following definition of image difference:

L N N (x, x b ) = p∈x min q∈N (p) ||g(x(p)) -g(x b (q))|| 1 , (7)
where N (p) is a n × n local neighbourhood of point p (we use 5 × 5 and 3 × 3 neighbourhoods for the DeepFashion and the Market-1501 dataset, respectively, see Sec. 6). g(x(p)) is a vectorial representation of a patch around point p in image x, obtained using convolutional filters (see below for more details). Note that L N N () is not a metrics because it is not symmetric. In order to efficiently compute Eq. 7, we compare patches in x and x b using their representation (g()) in a convolutional map of an externally trained network. In more detail, we use VGG-19 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], trained on ImageNet and, specifically, its second convolutional layer (called conv 1 2 ). The first two convolutional maps in VGG-19 (conv 1 1 and conv 1 2 ) are both obtained using a convolutional stride equal to 1. For this reason, the feature map (C x ) of an image x in conv 1 2 has the same resolution of the original image x. Exploiting this fact, we compute the nearest-neighbour field directly on conv 1 2 , without losing spatial precision. Hence, we define: g(x(p)) = C x (p), which corresponds to the vector of all the channel values of C x with respect to the spatial position p. C x (p) has a receptive field of 5 × 5 in x, thus effectively representing a patch of dimension 5 × 5 using a cascade of two convolutional filters. Using C x , Eq. 7 becomes:

L N N (x, x b ) = p∈x min q∈N (p) ||C x(p) -C x b (q)|| 1 , (8)
In Sec. A, we show how Eq. 8 can be efficiently implemented using GPU-based parallel computing. The final L N N -based loss is:

L N N (G) = E (xa,x b )∈X ,z∈Z L N N (x, x b ). (9)
Combining Eq. 5 and Eq. 9 we obtain our objective:

G * = arg min G max D L cGAN (G, D) + λL N N (G), (10)
with λ = 0.01 used in all our experiments. The value of λ is small because it acts as a normalization factor in Eq. 8 with respect to the number of channels in C x and the number of pixels in x (more details in Sec. A).

Implementation details

We train G and D for 90k iterations, with the Adam optimizer (learning rate: 2 * 10 -4 , β 1 = 0.5, β 2 = 0.999). Following [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] we use instance normalization [START_REF] Ulyanov | Instance normalization: The missing ingredient for fast stylization[END_REF]. In the following we denote with: (1) C s m a convolution-ReLU layer with m filters and stride s, (2) CN s m the same as C s m with instance normalization before ReLU and (3) CD s m the same as CN s m with the addition of dropout at rate 50%. Differently from [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], we use dropout only at training time. The encoder part of the generator is given by two streams (Fig. 2), each of which is composed of the following sequence of layers:

CN The discriminator architecture is:

C 2 64 -C 2 128 -C 2 256 -C 2 512 -C 2 1
, where the ReLU of the last layer is replaced with sigmoid.

The generator for the DeepFashion dataset has one additional convolution block (CN 2 512 ) both in the encoder and in the decoder, because images in this dataset have a higher resolution.

Experiments

Datasets The person re-identification Market-1501 dataset [START_REF] Zheng | Scalable person re-identification: A benchmark[END_REF] contains 32,668 images of 1,501 persons captured from 6 different surveillance cameras. This dataset is challenging because of the low-resolution images (128×64) and the high diversity in pose, illumination, background and viewpoint. To train our model, we need pairs of images of the same person in two different poses. As this dataset is relatively noisy, we first automatically remove those images in which no human body is detected using the HPE, leading to 263,631 training pairs. For testing, following [START_REF] Ma | Pose guided person image generation[END_REF], we randomly select 12,000 pairs. No person is in common between the training and the test split.

The DeepFashion dataset (In-shop Clothes Retrieval Benchmark) [START_REF] Liu | Deepfashion: Powering robust clothes recognition and retrieval with rich annotations[END_REF] is composed of 52,712 clothes images, matched each other in order to form 200,000 pairs of identical clothes with two different poses and/or scales of the persons wearing these clothes. The images have a resolution of 256×256 pixels. Following the training/test split adopted in [START_REF] Ma | Pose guided person image generation[END_REF], we create pairs of images, each pair depicting the same person with identical clothes but in different poses. After removing those images in which the HPE does not detect any human body, we finally collect 89,262 pairs for training and 12,000 pairs for testing.

Metrics Evaluation in the context of generation tasks is a problem in itself. In our experiments we adopt a redundancy of metrics and a user study based on human judgments. Following [START_REF] Ma | Pose guided person image generation[END_REF], we use Structural Similarity (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], Inception Score (IS) [START_REF] Salimans | Improved techniques for training gans[END_REF] and their corresponding masked versions mask-SSIM and mask-IS [START_REF] Ma | Pose guided person image generation[END_REF]. The latter are obtained by masking-out the image background and the rationale behind this is that, since no background information of the target image is input to G, the network cannot guess what the target background looks like. Note that the evaluation masks we use to compute both the mask-IS and the mask-SSIM values do not correspond to the masks ({M h }) we use for training. The evaluation masks have been built following the procedure proposed in [START_REF] Ma | Pose guided person image generation[END_REF] and adopted in that work for both training and evaluation. Consequently, the maskbased metrics may be biased in favor of their method. Moreover, we observe that the IS metrics [START_REF] Salimans | Improved techniques for training gans[END_REF], based on the entropy computed over the classification neurons of an external classifier [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], is not very suitable for domains with only one object class. For this reason we propose to use an additional metrics that we call Detection Score (DS). Similarly to the classification-based metrics (FCN-score) used in [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], DS is based on the detection outcome of the state-of-theart object detector SSD [START_REF] Liu | SSD: Single shot multibox detector[END_REF], trained on Pascal VOC 07 [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2007[END_REF] (and not fine-tuned on our datasets). At testing time, we use the person-class detection scores of SSD computed on each generated image x. DS(x) corresponds to the maximumscore box of SSD on x and the final DS value is computed by averaging the scores of all the generated images. In other words, DS measures the confidence of a person detector in the presence of a person in the image. Given the high accuracy of SSD in the challenging Pascal VOC 07 dataset [START_REF] Liu | SSD: Single shot multibox detector[END_REF], we believe that it can be used as a good measure of how much realistic (person-like) is a generated image.

Finally, in our tables we also include the value of each metrics computed using the real images of the test set. Since these values are computed on real data, they can be considered as a sort of an upper-bound to the results a generator can obtain. However, these values are not actual upper bounds in the strict sense: for instance the DS metrics on the real datasets is not 1 because of SSD failures.

Comparison with previous work

In Tab. 1 we compare our method with [START_REF] Ma | Pose guided person image generation[END_REF]. Note that there are no other works to compare with on this task yet. The mask-based metrics are not reported in [START_REF] Ma | Pose guided person image generation[END_REF] for the DeepFashion dataset. Concerning the DS metrics, we used the publicly available code and network weights released by the authors of [START_REF] Ma | Pose guided person image generation[END_REF] in order to generate new images according to the common testing protocol and ran the SSD detector to get the DS values.

On the Market-1501 dataset our method reports the highest performance with all but the IS metrics. Specifically, our DS values are much higher than those obtained by [START_REF] Ma | Pose guided person image generation[END_REF]. Conversely, on the DeepFashion dataset, our approach significantly improves the IS value but returns a slightly lower SSIM value.

User study

In order to further compare our method with the state-ofthe-art approach [START_REF] Ma | Pose guided person image generation[END_REF] we implement a user study following the protocol of Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF]. For each dataset, we show 55 real and 55 generated images in a random order to 30 users for one second. Differently from Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF], who used Amazon Mechanical Turk (AMT), we used "expert" (voluntary) users: PhD students and Post-docs working in Computer Vision and belonging to two different departments.

We believe that expert users, who are familiar with GANlike images, can more easily distinguish real from fake images, thus confusing our users is potentially a more difficult task for our GAN. The results 2 in Tab. 2 confirm the significant quality boost of our images with respect to the images produced in [START_REF] Ma | Pose guided person image generation[END_REF]. For instance, on the Market-1501 dataset, the G2R human "confusion" is one order of magnitude higher than in [START_REF] Ma | Pose guided person image generation[END_REF]. Finally, in Sec. D we show some example images, directly comparing with [START_REF] Ma | Pose guided person image generation[END_REF]. We also show the results obtained by training different person re-identification systems after augmenting the training set with images generated by our method. These experiments indirectly confirm that the degree of realism and diversity of our images is very significant.

Table 2: User study (%). ( * ) These results are reported in [START_REF] Ma | Pose guided person image generation[END_REF] and refer to a similar study with AMT users.

Market-1501

DeepFashion Model R2G G2R R2G G2R Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF] ( * ) 11.2 5.5 9.2 14.9 Ours 22.67 50.24 12.42 24.61

Ablation study and qualitative analysis

In this section we present an ablation study to clarify the impact of each part of our proposal on the final performance. We first describe the compared methods, obtained by "amputating" important parts of the full-pipeline presented in Sec. 3-4. The discriminator architecture is the same for all the methods.

• Baseline: We use the standard U-Net architecture [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] without deformable skip connections. The inputs of G and D and the way pose information is represented 2 R2G means #Real images rated as generated / #Real images; G2R means #Generated images rated as Real / #Generated images. (see the definition of tensor H in Sec. 3) is the same as in the full-pipeline. However, in G, x a , H a and H b are concatenated at the input layer. Hence, the encoder of G is composed of only one stream, whose architecture is the same as the two streams described in Sec.5.

• DSC: G is implemented as described in Sec. 3, introducing our Deformable Skip Connections (DSC). Both in DSC and in Baseline, training is performed using an L 1 loss together with the adversarial loss.

• PercLoss: This is DSC in which the L 1 loss is replaced with the Perceptual loss proposed in [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF]. This loss is computed using the layer conv 2 1 of [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], chosen to have a receptive field the closest possible to N (p) in Eq. 8, and computing the element-to-element difference in this layer without nearest neighbor search.

• Full: This is the full-pipeline whose results are reported in Tab. 1, and in which we use the proposed L N N loss (see Sec. 4).

In Tab. 3 we report a quantitative evaluation on the Market-1501 and on the DeepFashion dataset with respect to the four different versions of our approach. In most of the cases, there is a progressive improvement from Baseline to DSC to Full. Moreover, Full usually obtains better results than PercLoss. These improvements are particularly evident looking at the DS metrics, which we believe it is a strong evidence that the generated images are realistic. DS values on the DeepFashion dataset are omitted because they are all close to the value ∼ 0.96. In Fig. 4 and Fig. 5 we show some qualitative results. These figures show the progressive improvement through the four baselines which is quantitatively presented above. In fact, while pose information is usually well generated by all the methods, the texture generated by Baseline often does not correspond to the texture in x a or is blurred. In same cases, the improvement of Full with respect to Baseline is quite drastic, such as the drawing on the shirt of the girl in the second row of Fig. 5 or the stripes on the clothes of the persons in the third and in the fourth row of Fig. 4. Further examples are shown in the Appendix.

Conclusions

In this paper we presented a GAN-based approach for image generation of persons conditioned on the appearance and the pose. We introduced two novelties: deformable skip connections and nearest-neighbour loss. The first is used to solve common problems in U-Net based generators when dealing with deformable objects. The second novelty is used to alleviate a different type of misalignment between the generated image and the ground-truth image.

Our experiments, based on both automatic evaluation metrics and human judgments, show that the proposed method is able to outperform previous work on this task. Despite the proposed method was tested on the specific task of human-generation, only few assumptions are used which refer to the human body and we believe that our proposal can be easily extended to address other deformable-object generation tasks.

Appendix

In this Appendix we report some additional implementation details and we show other quantitative and qualitative results. Specifically, in Sec. A we explain how Eq. 8 can be efficiently implemented using GPU-based parallel computing, while in Sec. B we show how the human-body symmetry can be exploited in case of missed limb detections. In Sec. C we train state-of-the-art Person Re-IDentification (Re-ID) systems using a combination of real and generated data, which, on the one hand, shows how our images can be effectively used to boost the performance of discriminative methods and, on the other hand, indirectly shows that our generated images are realistic and diverse. In Sec. D we show a direct (qualitative) comparison of our method with the approach presented in [START_REF] Ma | Pose guided person image generation[END_REF] and in Sec. E we show other images generated by our method, including some failure cases. Note that some of the images in the DeepFashion dataset have been manually cropped (after the automatic generation) to improve the overall visualization quality.

A. Nearest-neighbour loss implementation

Our proposed nearest-neighbour loss is based on the definition of L N N (x, x b ) given in Eq. 8. In that equation, for each point p in x, the "most similar" (in the C x -based feature space) point q in x b needs to be searched for in a n × n neighborhood of p. This operation may be quite time consuming if implemented using sequential computing (i.e., using a "for-loop"). We show here how this computation can be sped-up by exploiting GPU-based parallel computing in which different tensors are processed simultaneously.

Given C x b , we compute n 2 shifted versions of C x b : {C (i,j)

x b }, where (i, j) is a translation offset ranging in a relative n × n neighborhood (i, j ∈ {-n-1 2 , ..., + n-1 2 }) and

C (i,j) x b
is filled with the value +∞ in the borders. Using this translated versions of C x b , we compute n 2 corresponding difference tensors {D (i,j) }, where:

D (i,j) = |C x -C (i,j) x b | (11)
and the difference is computed element-wise. D (i,j) (p) contains the channel-by-channel absolute difference between C x(p) and C x b (p + (i, j)). Then, for each D (i,j) , we sum all the channel-based differences obtaining:

S (i,j) = c D (i,j) (c), (12)
where c ranges over all the channels and the sum is performed pointwise. S (i,j) is a matrix of scalar values, each value representing the L 1 norm of the difference between a point p in C x and a corresponding point p + (i, j) in C x b :

S (i,j) (p) = ||C x(p) -C x b (p + (i, j))|| 1 . (13)
For each point p, we can now compute its best match in a local neighbourhood of C x b simply using:

M (p) = min (i,j) S (i,j) (p). (14)
Finally, Eq. 8 becomes:

L N N (x, x b ) = p M (p). (15)
Since we do not normalize Eq. 12 by the number of channels nor Eq. 15 by the number of pixels, the final value L N N (x, x b ) is usually very high. For this reason we use a small value λ = 0.01 in Eq. 10 when weighting L N N with respect to L cGAN .

B. Exploiting the human-body symmetry

As mentioned in Sec. 3.1, we decompose the human body in 10 rigid sub-parts: the head, the torso and 8 limbs (left/right upper/lower arm, etc.). When one of the joints corresponding to one of these body-parts has not been detected by the HPE, the corresponding region and affine transformation are not computed and the region-mask is filled with 0. This can happen because of either that region is not visible in the input image or because of falsedetections of the HPE.

However, when the missing region involves a limb (e.g., the right-upper arm) whose symmetric body part has been detected (e.g., the left-upper arm), we can "copy" information from the "twin" part. In more detail, suppose for instance that the region corresponding to the right-upper arm in the conditioning image is R a rua and this region is empty because of one of the above reasons. Moreover, suppose that R b rua is the corresponding (non-empty) region in x b and that R a lua is the (non-empty) left-upper arm region in x a . We simply set: R a rua := R a lua and we compute f rua as usual, using the (now, no more empty) region R a rua together with R b rua .

C. Improving person Re-ID via dataaugmentation

The goal of this section is to show that the synthetic images generated with our proposed approach can be used to train discriminative methods. Specifically, we use Re-ID approaches whose task is to recognize a human person in different poses and viewpoints. The typical application of a Re-ID system is a video-surveillance scenario in which images of the same person, grabbed by cameras mounted in different locations, need to be matched to each other. Due to the low-resolution of the cameras, person re-identification is usually based on the colours and the texture of the clothes [START_REF] Zheng | Person reidentification: Past, present and future[END_REF]. This makes our method particularly suited to automatically populate a Re-ID training dataset by generating images of a given person with identical clothes but in different viewpoints/poses.

In our experiments we use Re-ID methods taken from [START_REF] Zheng | Person reidentification: Past, present and future[END_REF][START_REF] Zheng | A discriminatively learned CNN embedding for person reidentification[END_REF] and we refer the reader to those papers for details about the involved approaches. We employ the Market-1501 dataset that is designed for Re-ID method benchmarking. For each image of the Market-1501 training dataset (T ), we randomly select 10 target poses, generating 10 corresponding images using our approach. Note that: (1) Each generated image is labeled with the identity of the conditioning image, (2) The target pose can be extracted from an individual different from the person depicted in the conditioning image (this is different from the other experiments shown here and in the main paper). Adding the generated images to T we obtain an augmented training set A. In Tab. 4 we report the results obtained using either T (standard procedure) or A for training different Re-ID systems. The strong performance boost, orthogonal to different Re-ID methods, shows that our generative approach can be effectively used for synthesizing training samples. It also indirectly shows that the generated images are sufficiently realistic and different from the real images contained in T .

D. Comparison with previous work

In this section we directly compare our method with the results generated by Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF]. The comparison is based on the pairs conditioning image-target pose used in [START_REF] Ma | Pose guided person image generation[END_REF], for which we show both the results obtained by Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF] and ours.

Figs. 67show the results on the Market-1501 dataset. Comparing the images generated by our full-pipeline with the corresponding images generated by the full-pipeline presented in [START_REF] Ma | Pose guided person image generation[END_REF], most of the times our results are more realistic, sharper and with local details (e.g., the clothes texture or the face characteristics) more similar to the details of the conditioning image. For instance, in the first and the last row of Fig. 6 and in the last row of Fig. 7, our results show human-like images, while the method proposed in [START_REF] Ma | Pose guided person image generation[END_REF] produced images which can hardly be recognized as humans.

Figs. 89show the results on the DeepFashion dataset. Also in this case, comparing our results with [START_REF] Ma | Pose guided person image generation[END_REF], most of the times ours look more realistic or closer to the details of the conditioning image. For instance, the second row of Fig. 8 shows a male face, while the approach proposed in [START_REF] Ma | Pose guided person image generation[END_REF] produced a female face (note that the DeepFashion dataset is strongly biased toward female subjects [START_REF] Ma | Pose guided person image generation[END_REF]). Most of the times, the clothes texture in our case is closer to that depicted in the conditioning image (e.g., see rows 1, 3, 4, 5 and 6 in Fig. 8 and rows 1 and 6 in Fig. 9). In row 5 of Fig. 9 the method proposed in [START_REF] Ma | Pose guided person image generation[END_REF] produced an image with a pose closer to the target; however it wrongly generated pants while our approach correctly generated the appearance of the legs according to the appearance contained in the conditioning image.

We believe that this qualitative comparison using the pairs selected in [START_REF] Ma | Pose guided person image generation[END_REF], shows that the combination of the proposed deformable skip-connections and the nearestneighbour loss produced the desired effect to "capture" and transfer the correct local details from the conditioning image to the generated image. Transferring local information while simultaneously taking into account the global pose deformation is a difficult task which can more hardly be implemented using "standard" U-Net based generators as those adopted in [START_REF] Ma | Pose guided person image generation[END_REF].

E. Other qualitative results

In this section we present other qualitative results. Fig. 10 and Fig. 11 show some images generated using the Market-1501 dataset and the DeepFashion dataset, respectively. The terminology is the same adopted in Sec. 6.2. Note that, for the sake of clarity, we used a skeleton-based visualization of P (•) but, as explained in the main paper, only the point-wise joint locations are used in our method to represent pose information (i.e., no joint-connectivity information is used).

Similarly to the results shown in Sec. 6.2, also these images show that, despite the pose-related general structure is sufficiently well generated by all the different versions of our method, most of the times there is a gradual quality improvement in the detail synthesis from Baseline to DSC to PercLoss to Full.

Finally, Fig. 12 and Fig. 13 show some failure cases (badly generated images) of our method on the Market-1501 dataset and the DeepFashion dataset, respectively. Some common failure causes are:

• Errors of the HPE [START_REF] Cao | Realtime multiperson 2D pose estimation using part affinity fields[END_REF]. For instance, see rows 2, 3 and 4 of Fig. 12 or the wrong right-arm localization in row 2 of Fig. 13.

• Ambiguity of the pose representation. For instance, in row 3 of Fig. 13, the left elbow has been detected in x b although it is actually hidden behind the body. Since P (x b ) contains only 2D information (no depth or occlusion-related information), there is no way for the system to understand whether the elbow is behind or in front of the body. In this case our model chose to generate an arm considering that the arm is in front of the body (which corresponds to the most frequent situation in the training dataset).

• Rare poses. For instance, row 1 of Fig. 13 shows a girl in an unusual rear view with a sharp 90 degree profile face (x b ). The generator by mistake synthesized a neck where it should have "drawn" a shoulder. Note that rare poses are a difficult issue also for the method proposed in [START_REF] Ma | Pose guided person image generation[END_REF]. [START_REF] Zheng | Person reidentification: Past, present and future[END_REF] 73.9 48.8 78.5 55.9 IDE + XQDA [START_REF] Zheng | Person reidentification: Past, present and future[END_REF] 73.2 50.9 77.8 57.9 IDE + KISSME [START_REF] Zheng | Person reidentification: Past, present and future[END_REF] 75.1 51.5 79.5 58.1 Discriminative Embedding [START_REF] Zheng | A discriminatively learned CNN embedding for person reidentification[END_REF] 78.3 55.5 80.6 61.3

• Rare object appearance. For instance, the backpack in row 1 of Fig. 12 is light green, while most of the backpacks contained in the training images of the Market-1501 dataset are dark. Comparing this image with the one generated in the last row of Fig. 10 (where the backpack is black), we see that in Fig. 10 the colour of the shirt of the generated image is not blended with the backpack colour, while in Fig. 12 it is. We presume that the generator "understands" that a dark backpack is an object whose texture should not be transferred to the clothes of the generated image, while it is not able to generalize this knowledge to other backpacks.

• Warping problems. This is an issue related to our specific approach (the deformable skip connections). The texture on the shirt of the conditioning image in row 2 of Fig. 13 is warped in the generated image. We presume this is due to the fact that in this case the affine transformations need to largely warp the texture details of the narrow surface of the profile shirt (conditioning image) in order to fit the much wider area of the target frontal pose.

x a

x b Full (ours) Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF] Figure 6: A qualitative comparison on the Market-1501 dataset between our approach and the results obtained by Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF]. Columns 1 and 2 show the conditioning and the target image, respectively, which are used as reference by both models. Columns 3 and 4 respectively show the images generated by our full-pipeline and by the full-pipeline presented in [START_REF] Ma | Pose guided person image generation[END_REF].

x a

x b Full (ours) Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF] Figure 7: More qualitative comparison on the Market-1501 dataset between our approach and the results obtained by Ma et al. [START_REF] Ma | Pose guided person image generation[END_REF].

Figure 1 :

 1 Figure 1: (a) A typical "rigid" scene generation task, where the conditioning and the output image local structure is well aligned. (b) In a deformable-object generation task, the input and output are not spatially aligned.

  x a P (x a ) P (x b ) x b Baseline DSC PercLoss Full

Figure 4 :

 4 Figure 4: Qualitative results on the Market-1501 dataset.Columns 1, 2 and 3 represent the input of our model. We plot P (•) as a skeleton for the sake of clarity, but actually no joint-connectivity relation is exploited in our approach. Column 4 corresponds to the ground truth. The last four columns show the output of our approach with respect to different baselines.

Figure 5 :

 5 Figure 5: Qualitative results on the DeepFashion dataset with respect to different baselines. Some images have been cropped for visualization purposes.

Figure 8 :

 8 Figure 8: A qualitative comparison on the DeepFashion dataset between our approach and the results obtained by Ma et al. [12].

Figure 9 :

 9 Figure 9: More qualitative comparison on the DeepFashion dataset between our approach and the results obtained by Ma et al. [12].

PFigure 10 :

 10 Figure 10: Other qualitative results on the Market-1501 dataset.

PFigure 11 :

 11 Figure 11: Other qualitative results on the DeepFashion dataset.

PFigure 12 :

 12 Figure 12: Examples of badly generated images on the Market-1501 dataset. See the text for more details.

PFigure 13 :

 13 Figure 13: Examples of badly generated images on the DeepFashion dataset.

Table 1 :

 1 Comparison with the state of the art. ( * ) These values have been computed using the code and the network weights released by Ma et al.[START_REF] Ma | Pose guided person image generation[END_REF] in order to generate new images.

				Market-1501			DeepFashion
	Model	SSIM	IS	mask-SSIM mask-IS	DS	SSIM	IS	DS
	Ma et al. [12] 0.253 3.460	0.792	3.435	0.39 * 0.762 3.090 0.95 *
	Ours	0.290 3.185	0.805	3.502	0.72 0.756 3.439 0.96
	Real-Data	1.00	3.86	1.00	3.36	0.74	1.000 3.898	0.98

Table 3 :

 3 Quantitative ablation study on the Market-1501 and the DeepFashion dataset.

				Market-1501			DeepFashion
	Model	SSIM	IS	mask-SSIM mask-IS	DS	SSIM	IS
	Baseline	0.256 3.188	0.784	3.580	0.595	0.754 3.351
	DSC	0.272 3.442	0.796	3.666	0.629	0.754 3.352
	PercLoss	0.276 3.342	0.788	3.519	0.603	0.744 3.271
	Full	0.290 3.185	0.805	3.502	0.720 0.756 3.439
	Real-Data	1.00	3.86	1.00	3.36	0.74	1.000 3.898
	x

a P (x a ) P (x b ) x b Baseline DSC PercLoss Full

Table 4 :

 4 Accuracy of Re-ID methods on the Market-1501 test set (%)

		Standard training set (T ) Augmented training set (A)
	Model	Rank 1	mAP	Rank 1	mAP
	IDE + Euclidean

Acknowledgements

We want to thank the NVIDIA Corporation for the donation of the GPUs used in this project.