Sébastien Briot

Ilian A Bonev
email: ilian.bonev@etsmtl.ca

ARE PARALLEL ROBOTS MORE ACCURATE THAN SERIAL ROBOTS?

It is widely claimed that parallel robots are intrinsically more accurate than serial robots because their errors are averaged instead of added cumulatively, an assertion which has not been properly addressed in the literature. This paper addresses this void by comparing the kinematic accuracy of two pairs of serialparallel 2-DOF planar robots. Only input errors are considered and all robots are optimized for accuracy, the only constraint being that they cover a given desired workspace. The results of this comparison seem to confirm that parallel robots are less sensitive to input errors than serial robots. However, this comparison is too limited to draw any general conclusions. Besides, it is virtually impossible to make a meaningful comparison between other pairs of serial and parallel robot. Therefore, there is no simple answer to this question of superiority.

Introduction

The development of parallel robots has always been driven by promises of (1) greater rigidity, (2) higher speed, and (3) higher accuracy than serial robots.

The fact that virtually all the hundreds, or even thousands, of motion simulators with load capacities of up to several tons are based on parallel robots (mostly hexapods), with serial robots able to carry at most five hundred kilograms or so, unquestionably demonstrates that the first promise has been fulfilled. The commercial success of the Delta parallel robot and the performance of the recently launched Quickplacer by Fatronik (200 cycles per minute) confirms fulfillment of the second promise, though serial robots are not far behind. But has the third promise been fulfilled yet?

The boom in the development of parallel kinematic machines (PKMs) in the 1990s, particularly those based on hexapods, was driven mainly by that third promise. But none of these hexapods is more accurate than a conventional serial machine tool. Some three-axis and five-axis PKMs are now gaining commercial success, but precision is still not their best feature. While a number of alignment stages are based on parallel robots, the fact remains that great precision is attained by the use of special technologies, such as flexures. Flexures rely on deformation of material to achieve a motion between two elastically joined parts. Flexures are mainly used as passive joints, thus mostly in parallel robots, so it could be said that parallel robots are more accurate for this reason alone. But is it true that parallel robots are kinematically more accurate than serial robots because errors are averaged instead of added cumulatively, as widely claimed:

-"The parallel actuator technology promises to offer […] advantages relative to conventional machine tools, such as […] higher accuracy…" [START_REF] Wavering | Parallel kinematic machine research at NIST: past, present and future[END_REF]; -"Parallel manipulators are preferred to serial manipulators for their […] high positioning accuracy." [START_REF] Rauf | A new measurement device for complete parameter identification of parallel manipulators with partial pose measurements[END_REF]; -"Comparing [sic] to the traditional serial-chain mechanism […], the parallel mechanism exhibits the following advantages: […] better accuracy due to non-cumulative joint error." [START_REF] Song | Error modeling and compensation for parallel kinematic machines[END_REF] ; -"The errors of parallel manipulators are averaged out in the serial chains and the errors of serial manipulator are accumulated [sic]." [START_REF] Guan | Kinematics of a Tricept-like parallel robot[END_REF]; -"Moreover the links [of a serial robot] magnify errors: a small measurement error in the internal sensors of the first one or two links will quickly lead to a large error in the position of the end effector.

[…] The errors of the internal sensors [of a parallel robot] only slightly affect errors on the platform position." [START_REF] Merlet | Parallel Robots[END_REF]. Obviously, the sources of positioning error are numerous (design errors, flexibility of the links, thermal expansion, etc.). But, according to Merlet [START_REF] Merlet | Computing the worst case accuracy of a PKM over a workspace or a trajectory[END_REF], joint sensor errors are the largest source of error in the positioning of a robot. Surprisingly, we have found no reference that explains this "accumulation/averaging of errors" or which compares the input error sensitivity of serial and parallel robots. The most relevant work was reported in [START_REF] Wenger | A comparative study of serial and parallel mechanism topologies for machine tools[END_REF], where several two-degrees-of-freedom (2-DOF) planar serial and parallel robots are compared on the basis of four performance criteria, none of which is purely input error sensitivity (one is called sensitivity, but this takes into account errors in the design parameters, in addition to input errors). Our paper addresses this void and provides some new results regarding the input error sensitivity of serial and parallel robots.

In this work, we perform a comparative study of the kinematic accuracy of two serial and two parallel 2-DOF planar robots, one of which was not considered in [START_REF] Wenger | A comparative study of serial and parallel mechanism topologies for machine tools[END_REF]. The only source of error that we consider is that caused by an uncertainty on the input joint sensor measurement (input errors). The robots are compared in pairs, the robots in each pair being subject to identical actuation (and the same input errors). To make the comparison meaningful, all robots are optimized to have the best accuracy, while covering the same desired square workspace area.

In the next section, we will define the four robots and specify the criteria for comparison. In the third section, we will study the maximal position error of each robot, and, in the fourth section, compare the dexterity index to the maximal position error. Conclusions will be presented in the last section.

Criteria for Comparison and Description of the Planar Robots Under Study

In most cases, the so-called dexterity index is used to study the kinematic accuracy of robots [START_REF] Gosselin | The optimum design of robotic manipulators using dexterity indices[END_REF]. Merlet [START_REF] Merlet | Jacobian, manipulability, condition number, and accuracy of parallel robots[END_REF] criticizes such an index, stressing that its major drawbacks are that it mixes both translational and rotational terms of the Jacobian matrix and that it is usually not invariant on the choice of units. As a consequence, the Jacobian matrix must be split into its translational and rotational parts to calculate the dexterity of each of them, but this is not satisfactory for estimating the amplification factor for motion involving both translational and rotational displacements. As we will see in this paper, the dexterity index does not even work properly for robots having only translational degrees of freedom. Thus, the most suitable method for computing the accuracy of robots (actually the input error sensitivity) is to calculate the maximal position error, or orientation error, due to input errors, at a given nominal configuration. This is very easy to do for 2-DOF planar parallel robots using a simple geometrical method. Now that we have decided how to measure the kinematic accuracy of robots, we have to define some criteria for a fair comparison. The first-and most obvious-criterion is that the robots must have the same actuators (only revolute or only prismatic) so that they can have the same input errors. Another criterion is that the robots must be able to accomplish the same task. We impose the constraint in this paper that the robots must be able to displace their end-effectors inside a 1 m by 1 m square (the desired workspace), and do so with the best accuracy possible, meaning that their designs should be optimized to have the smallest mean maximal position error over this desired workspace. This square should obviously be free of singularities. Note that the authors of [START_REF] Wenger | A comparative study of serial and parallel mechanism topologies for machine tools[END_REF] do not compare robots with optimized kinematic accuracy.

That said, we will compare the following two pairs of 2-DOF planar robots for positioning:

a RRRRR parallel robot (Fig. 1a) and a RR serial robot (Fig. 1b); -a PRRRP parallel robot, the directions of its base-mounted prismatic actuators being parallel (Fig. 1c), and a Cartesian serial robot (Fig. 1d).

While the choice of the first pair is fairly obvious, the choice of the second pair might look a bit arbitrary, but it is not. We choose a PP serial robot in which the directions of the prismatic joints are orthogonal, simply because any other non-Cartesian PP serial robot will have worse maximal position error. As for the PRRRP parallel robot, of course, we could choose another architecture with prismatic actuators, but this one is surely the most practical one. Finally, we choose to have the directions of its two prismatic actuators parallel, simply because this gives the most compact design having the desired singularity-free workspace.

Both serial robots are well known and trivial to design to obtain the best accuracy within the desired workspace.

The RR serial robot is designed by finding the optimal values of the parameters OA, AP and d (Fig. 2a). For this robot, it is obvious that the smaller the workspace, the higher the accuracy. Therefore, the design parameters have to define a compact workspace with respect to the desired workspace. The geometric conditions for compactness are: (1 2 ) ( .5 )

+ = + γ+ + +γ OA AP d O , ( 1
)
OA AP d - = . ( 2
)
where γ is a safety distance added to avoid that the desired workspace includes singularities along its boundary. In this study, γ = 0.1 m.

Solving equations ( 1) and (2), we obtain the possible values for OA and AP as functions of d: The mean value of the maximal position error within the desired workspace of the RR serial robots corresponding to any of the two solutions of eq. ( 3) is shown in Fig. 3a, as a function of d. The calculation of the maximal position error will be presented in Section 3. As expected, the optimal design occurs at d = 0 m, and from eq. ( 3), we have OA = 0.67 m and AP = 0.67 m.

( )/2 ( )/2 OA T d AP T d = + ⎧ ⎨ = - ⎩ or ( )/2 ( )/2 OA T d AP T d = - ⎧ ⎨ = + ⎩ (3)
The accuracy of the Cartesian serial robot is the same for any position and any actuator stroke. Therefore, there are no optimal design parameters to look for.

The two parallel robots are more difficult to optimize in terms of accuracy. These difficulties are due to the complexity of their direct kinematics and to the presence of singularities inside their workspaces. These two robots have recently been studied in detail [START_REF] Liu | Kinematics, singularity and workspace of planar 5R symmetrical parallel mechanisms[END_REF][START_REF] Liu | Performance atlases and optimum design of planar 5R symmetrical parallel mechanisms[END_REF][START_REF] Liu | On the optimal kinematic design of the PRRRP 2-DOF parallel mechanism[END_REF]. In these references, the authors analyze the robots using different performance indices depending on the link lengths. From [START_REF] Liu | Performance atlases and optimum design of planar 5R symmetrical parallel mechanisms[END_REF], we can roughly estimate that a nearly-optimal RRRRR design occurs when A 1 A 2 = 0.3 m, A 1 B 1 = 0.6 m, and B 1 P = 0.8 m. Although this is not the actual optimal design, for the purposes of our purely qualitative study, it will be good enough. What is important is that the RR serial robot has been given all the chances to win the competition-its design is optimal.

For the PRRRP parallel robot, the shorter the links A i P, the higher the accuracy. While this fact seems obvious, it was nevertheless verified numerically. Thus, it is possible to find a relationship between a and A i P which defines the minimal link length as (Fig. 2b):

( 1)/2 i A P a = + +γ . (4)
The mean value of the maximal position error within the desired workspace of the PRRRP parallel robots whose parameters obey eq. ( 4) is shown in Fig. 3b, as a function of the parameter a. The method for calculating the maximal position error will be presented in Section 3. Figure 3b also shows the required stroke of the actuators (in dashed line). One can see that higher accuracy calls for longer actuators. Thus, we chose a nearly-optimal design at A 1 P = A 2 P = 2.1 m and a = 3 m.

Figure 4 shows the optimized designs of three of the robots under study, their workspaces and the square within them that constitutes the desired workspace, all to the same scale. The workspace of the Cartesian serial robot, which is not shown here, is obviously a rectangular region. In the next section, we will analyze the maximal position errors of these four robots using a geometrical method.

Analysis of the Maximal Position Error of the Robots Under Study

Comparison of the RRRRR parallel robot and the RR serial robot

For these robots, we consider that the maximal input error is equal to ±2×10 -4 rad. The maximal position error for these robots is quite easy to determine. For each of them, this error occurs at one of the four sets of extreme input errors, i.e., at one of the corners of the so-called uncertainty zone, as shown in Fig. 5.

Thus, it is possible to calculate the maximal position error at each nominal position. These errors are presented in Fig. 6. In addition, Table I gives some statistics regarding the maximal position error for each robot over the desired workspace.

Comparison of the PRRRP parallel robot and the Cartesian serial robot

For these robots, we consider that the maximal input error is equal to ±100 µm. The maximal position error for these robots is also quite easy to determine. It is equal to approximately 141 µm for the Cartesian serial robot (Fig. 7b). For the PRRRP parallel robot, this error occurs at one of the four sets of extreme input errors, i.e., at one of the corners of the so-called uncertainty zone, as shown in Fig. 7a.

Thus, it is possible to obtain the maximal position error at each position for the PRRRP parallel robot. This maximal position error is virtually equal to the input error, i.e., ±100 µm, for any position. Therefore, no contour plot as in Fig. 6 is given for this robot. Table II gives statistics regarding the maximal position error for each robot over the desired workspace. In concluding this section, we warn readers that our study is too limited to draw any general conclusions. It is quite possible, for example, that if the desired workspace is different (e.g., an annular region or an elongated rectangular region) or if the input errors are much smaller, some of the results could be quite different.

Nevertheless, our study suggests that a RRRRR parallel robot is much less sensitive to input errors than an equivalent RR serial robot, while having nearly the same overall dimensions. (We again point out that by "equivalent" we mean that the robots have the same desired square workspace and the same input errors.) Similarly, a PRRRP parallel robot is much less sensitive to input errors than an equivalent Cartesian serial robot, but only when its overall dimensions are much greater than those of the serial robot. In fact, the mean maximal position error of a nearlyoptimal PRRRP parallel robot is equal to its maximal input error, which means that both the Cartesian robot and the parallel robot are dimension invariant; hence the comparison is fair. However, with greater dimensions there are more manufacturing errors, and a need for larger actuators stroke, which means higher manufacturing costs. For example, the actuators of the PRRRP parallel robot should be three times as long as those of the Cartesian robot. Thus, we find it hard to believe that, in practice, a PRRRP parallel robot would be more precise than a Cartesian robot.

Finally, we are tempted to comment on this widely claimed "accumulation of errors" in serial robots in contrast to the "averaging of errors" in parallel robots. A ±100 µm input error produces a maximal position error of approximately 141 µm in a Cartesian serial robot, and a maximal mean positioning error of about 100 µm in a (optimally designed) PRRRP parallel robot. So, in this example, one might indeed say that there is an averaging of errors in the parallel robot. However, when its end-effector is close to certain singularities, then the maximal position error could be several times larger than the input error (in the case of prismatic actuators). We therefore believe that this is, in general, too strong a statement and should be avoided.

Analysis of the Dexterity Index

In this section, we will compare the dexterity indices of the four robots under study to their maximal position errors over the desired workspace. The dexterity of a robot is calculated as defined in [START_REF] Merlet | Jacobian, manipulability, condition number, and accuracy of parallel robots[END_REF]:

1 1 - = J J ξ , (5)
where we use the Euclidean norm defined as

1 2 T tr ⎛ ⎞ = ⎜ ⎟ ⎝ ⎠ J J J , ( 6
)
and J is the Jacobian matrix of the robot. The Jacobian matrices for the four robots under study will not be derived here, since this is fairly simple to do and the calculation of the dexterity index is not the main subject of this paper.

The dexterity maps for the two parallel robots and for the RR serial robot are represented in Fig. 8. The dexterity index of the Cartesian serial robot is constant and equal to unity, and is therefore not shown in this figure .  Several observations can be made. Firstly, the shape of the dexterity map does not correspond to the shape of the maximal position error map. Areas of highest dexterity do not correspond to areas of lowest maximal position error. Secondly, the values of the dexterity indices of the RRRRR parallel robot and of the RR serial robots are similar, even though the first robot is more accurate than the second in terms of maximal position error. Likewise, the values of the dexterity index for the PRRRP parallel robot are not constant, even though its maximal position error is nearly constant.

It is therefore clear that the dexterity map cannot be used to evaluate the global nature of the accuracy of a robot, let alone to compare the accuracy of different robot designs, and this is true even for robots having only positioning capabilities (i.e., when there are no problems involving mixed units).

Conclusions

Obviously, our study is quite simplistic and too limited for us to draw any general conclusions as numerous other authors have done who claim that parallel robots are more accurate than serial robots. Yet, in spite of the limited nature of our study, it still suggests that parallel robots are indeed theoretically more accurate than serial robots, when input errors are assumed to be the only source of inaccuracy. This was shown using the natural concept of maximal position error instead of the dexterity index, which proved to be meaningless for comparing accuracy.

But, are parallel robots more accurate than serial robots in practice, where the larger dimensions required in parallel robots might induce greater mechanical errors? Moreover, are we not, after all, comparing apples and oranges when we state that parallel robots are more accurate than serial robots? How can we say, for example, that a hexapod (with six linear actuators) is theoretically more accurate than a serial machine tool (with three linear and three rotary actuators)? The truth is that only practice, and not theory, will show whether or not parallel robots can be manufactured to be more accurate than serial robots. In other words, we believe that the mechanical design of a robot, its manufacture and its calibration are much more important drivers of accuracy than the optimal kinematic design. So, the question is whether parallel robots can be mechanically designed, manufactured and calibrated so as to be more precise than the most accurate serial robots on the market. For the time being, we can only claim that some parallel robots seem to be less sensitive to input errors than their equivalent serial counterparts.

Fig. 1 .

 1 Fig. 1. The two pairs of planar robots under comparison (not to scale).

( a )Fig. 2 .

 a2 Fig. 2. Design constraints for obtaining the desired workspace.

Fig. 3 .

 3 Fig. 3. Variations of the mean value of the maximal position error over the desired workspace.

Fig. 4 .

 4 Fig. 4. Workspace of the planar robots under study (to scale).

Fig. 5 .

 5 Fig. 5. Uncertainty zones for the first two planar robots.

Fig. 6 .

 6 Fig. 6. Maximal position errors (in µm) in the desired workspace for the first two planar robots.

Fig. 7 .

 7 Fig. 7. Uncertainty zones for the second pair of planar robots.

Fig. 8 .

 8 Fig. 8. Contour plots of the dexterity index for three of the robots over the desired workspace.

TABLE I

 I

	Statistics for the Maximal Position Errors
	Max. position error	RRRRR parallel robot RR serial robot
	Max. value (µm)	197	367
	Mean value (µm)	175	240
	Standard deviation (µm) 24

TABLE II

 II

	Statistics for the Maximal Position Errors
	Max. position error	PRRRP parallel robot PP serial robot
	Max. value (µm)	100	141
	Mean value (µm)	100	141
	Standard deviation (µm) 0	0


l> Julian Thevenard

Dominique Lo

Hine Tong

Nicolas Corinne

Christian Person

Jean-Philippe Coupez

Romain Pascaud

Raphaël Gillard

Renaud Loison

A Louzir
email: ali.louzir@thomson.net

C Nicolas

J Ph

Coupez

Integration of a multi-sector antenna system on a wireless camera

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Wireless television cameras are already widely used for Electronics News Gathering (ENG) and Outside Broadcast (OB) and their usage is growing with the increasingly need of rapid deployment of mobile video production platform for the coverage of live sporting and news events. The use of wireless cameras gives also the camera crew much more freedom to capture the event without the hurdle of running cables. Unlike many wireless applications, the upstream link (from the mobile camera to the intermediate receiver) is the most constraining in terms of required bit rate and link reliability to achieve a professional picture quality, in particular in the case of live HDTV production. The power consumption of the wireless module that may limit the battery life of the camera is also an important design parameter. The use of a multi-sector antenna at the camera side may be very helpful to respond to the above constraints while at the same time it limits the interferences between several wireless production cameras. This paper addresses the integration issue of a multi-sector antenna on a wireless camera from simulation and application point of view. First the multi-sector antenna system is presented. Then simulation results of the full structure (antenna + camera) using both Ansoft's HFSS software and a Dual-Grid FDTD (DG FDTD) multi-resolution method are presented. The obtained results are finally compared with measurements.

The multi-sector antenna system description A 4-sector antenna system operating in the 5GHz band is used. The chosen concept of the multi-sector antenna system, shown in Figure 1, is based on the use of 4 Vivaldi antennas arranged vertically around a common central axis to cover the entire 360° of the azimuth plane [START_REF] Thevenard | 30 multi-sector Vivaldi antennas based on metallized plastic technology[END_REF][START_REF] Thevenard | Switched beam 3d-vivaldi antennas for WLAN applications[END_REF]. The Vivaldi antennas are excited through microstrip to slot line transitions and mounted onto a horizontal mother board that includes the feeding networks and the beam switching circuits. Such 3D antenna arrangement offers more freedom in the optimization of the number of sectors and the control of the radiation pattern and gain of each individual beam. Figure 1 also shows the radiation patterns (measured at 6 GHz) in the azimuth plane of the designed multi-sector antenna taking into account a mother board square of 90x90 mm 2 • Simulation and measurements of the integrated multi-sector antenna

1 Sector! Sector2 Sector3 Sector4 -00 .. l-+-l-f---+--+-l--'*+-+-+-+-1--+-1 60 -150 60
After a comprehensive study taking into account both radio and application constraints, the place of the multi-sector antenna on the camera was chosen at the rear of the camera body, i.e. near the radio module enclosure and in a relatively cleared/quiet zone (Figure 2). Besides, the orientation of the 4 beams which are 90° from each other are chosen at 45° from the camera axis in order to limit the interaction of the radiation patterns with the camera metallic handling. Despite these precautions the surrounding environment of the multi-sector antenna system could not be neglected since it might affect its performances.

Lens

Camera body radio part ), the simulation of the full structure is very time and power consuming, approaching the processing limit of the used machine. Figure 3 shows the simulated S parameters with sector 1 (less perturbed by the camera) activated compared to the case with sector 2 (close to the camera handling) activated. While the return loss remains better than 10 dB over the full [4.8-6]GHz band in both cases, the isolation between sectors rises from typically -25 dB (which is the value obtained for the multi-sector antenna system alone) to the value of -15 dB corresponding to the worst case when part of the power transmitted by sector 1 is reflected by the camera handle and captured by sector 3 antenna. Figure 4 presents the simulated radiation patterns for the 4 sectors in azimuth and elevation planes showing a significant disturbance due to the camera handle. The above conclusions have been confirmed by radiation pattern measurements of the antenna system attached to the camera. The results presented in Figure 5 at the frequency of 6 GHz, correspond to the total radiated field, obtained by adding E and H components and are representative of the measurements made at other frequencies. However, while the measured radiation patterns in the elevation plane are relatively similar to those obtained by simulations using HFSS software, a significant discrepancy between simulation and measurement could be noticed in the azimuth plane.

A second simulation of the full structure has then been done using the Dual-Grid Finite-Difference Time-Domain (DG-FDTD) method [START_REF] Pascaud | Dual-grid finite-difference time-domain scheme for the fast simulation of surrounded antennas[END_REF]. This method divides the classical FDTD simulation of the overall problem into two different FDTD simulations. First the antenna system is simulated without its environment using the required high resolution mesh, while its "primary radiation" is saved. Then the saved contribution is used as an excitation of the second FDTD simulation that uses a coarse mesh to represent the antenna with the large camera. Figure 7 shows the simulation results of the radiation patterns in the azimuth plane obtained by this method, compared with both HFSS simulations and measurements made at 6 GHz for both sector 1 and 3. It could be noticed that the obtained results with the DG-FDTD simulation seems closer to the measurements. In the elevation plane, the obtained results by the DG-FDTD method and HFSS are very close and fairly match the measured results.

Conclusion and fu ture work 90

The issue of the integration of a multi-sector antenna system onto a wireless camera has been addressed from simulation and application point of view. The obtained results show that the effect of the camera on the multi-sector antenna system could not be neglected and should be taken into account at the early phase of the antenna design. Simulations and measurements have shown the interest of using a multi-resolution method such as the DG-FDTD method in the simulation of antenna system including its close environment. One potential solution to limit the disturbance of the antenna performances by the camera handling may rely on the tilt of the concerned beams by using metallic directors properly positioned into the antenna radome [START_REF] Thevenard | Radome Enhancing Multi-Sector Antenna Radiation Performances[END_REF].

Figure 1 : 4 -

 14 Figure 1: 4-sector switched beam antenna in FR4 substrate and associated radiation pattern at 6GHz in the azimuth plane (0=90°).

Figure 2 :

 2 Figure 2: Camera+ antenna system global views.

Figure 3 :Figure 4 :

 34 Figure 3 : Simulated S-parameters with sector 1 (a) and sector 2 activated (b).

Figure 5 :

 5 Figure 5 : Radiation patterns at 6GHz in the azimuth plane (0=90°).


Xi Shen

Robin Champenois

Shiry Ginosar

Ilaria Pastrolin

Morgane Rousselot

Oumayma Bounou

Tom Monnier

Spyros Gidaris

François Bougard

Pierre-Guillaume Raverdy

Marie-Françoise Limon

Christine Bénévent

Marc Smith

Olivier Poncet

K Bender

Béatrice Joyeux-Prunel

Elizabeth Honig

Alexei A Efros

Mathieu Aubry

P.-G Raverdy

Spatially-consistent Feature Matching and Learning for Heritage Image Analysis

Keywords: Feature learning, Self-supervised learning, Artwork analysis, Watermark recognition

Progress in the digitization of cultural assets leads to online databases that become too large for a human to analyze. Moreover, some analyses might be challenging, even for experts. In this paper, we explore two applications of computer vision to analyze historical data: watermark recognition and one-shot repeated pattern detection in artwork collections. Both problems present computer vision challenges which we believe to be representative of the ones encountered in cultural heritage applications: limited supervision is available, the tasks are fine-grained recognition, and the data comes in several different modalities. Both applications are also highly practical, as recognizing watermarks makes it possible to date and locate documents, while detecting repeated patterns allows exploring visual links between artworks. We demonstrate on both tasks the benefits of relying on deep mid-level features. More precisely, we define an image similarity score based on geometric verification of mid-level fea-

Introduction

Learning global image features has been successful in many image recognition tasks, such as image classification [START_REF] He | Deep residual learning for image recognition[END_REF], landmark retrieval [START_REF] Gordo | End-toend learning of deep visual representations for image retrieval[END_REF][START_REF] Radenović | Fine-tuning cnn image retrieval with no human annotation[END_REF], metric learning [START_REF] Kim | Proxy anchor loss for deep metric learning[END_REF][START_REF] Elezi | The group loss for deep metric learning[END_REF][START_REF] Teh | Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis[END_REF] and few-shot classification [START_REF] Vinyals | Matching networks for one shot learning[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF]. In this paper, we highlight that matching and learning mid-level features and taking into account spatial information is better adapted to two cultural heritage applications required to recognize exactly repeated patterns across wide appearance changes, namely, historical watermark recognition and one-shot repeated pattern detection in artwork collections.

For watermark recognition (Figure 1(a)), we aim at retrieving the exact drawing (green) corresponding to the query photograph (blue) of a watermark from a very large database, which includes many similar watermarks each corresponding to a different category. In artwork collections, we tackle one-shot repeated pattern detection (Figure 1(b)), i.e. given a query detail (blue) we retrieve it in artworks with various styles (green). Both applications share similar Computer Vision challenges: (i) since annotations are very costly and require expert knowledge, there are only few category-level annotations available for training for watermark recog- nition and no training annotations for repeated detail detection in art collections; (ii) the artwork details and the watermarks are represented in different visual domains; (iii) both tasks require fine-grained separation between similar but distinct classes. We believe these challenges are common to many cultural heritage applications.

The two problems we tackle are also highly practical for historians and archivists. For watermark recognition, the drawings we recognize come from large catalogs [10,1] where they are associated with information such as the date and location of the paper fabrication, which are important clues to analyze historical documents. Similarly, repeated patterns in art collections are important to art historians and allow them to analyze influences, find provenance, and even establish authorship [START_REF] Castellano | Visual link retrieval and knowledge discovery in painting datasets[END_REF].

For each application, we introduced datasets for training and evaluation adapted to modern deep learning approaches [START_REF] Shen | Discovering visual patterns in art collections with spatially-consistent feature learning[END_REF][START_REF] Shen | Large-scale historical watermark recognition: dataset and a new consistency-based approach[END_REF] and extend them in this paper. For historical watermark recognition, we first collected more than 6k new photographs for hundreds of classes, then searched and photographed systematically the exact original watermarks corresponding to hundreds of line drawings printed in Briquet's classic catalog of more than 16k [10] watermarks. This allows us to tackle at scale for the first time the scenarios of practical interest for scholars: one-shot instance recognition and crossdomain one-shot instance recognition among more than 16k fine-grained classes. For one-shot detection of the artwork's details, we used a database of 1,587 artworks from Brueghel's workshop [START_REF] Brueghel Family | Jan brueghel the elder[END_REF][START_REF] Honig | Jan Brueghel and the Senses of Scale[END_REF], selected 10 of the most commonly repeated details, and annotated their 273 occurrences in the full dataset for evaluation. Note that these visual patterns are near-duplicated but with various styles as artworks are in different media (e.g. oil, ink, chalk, watercolor) and on different materials (e.g. paper, panel, copper). In this paper, we provide additional results on a collection of 25,681 depictions of Venus [START_REF] Bender | Distant viewing in art history. a case study of artistic productivity[END_REF]. It is more than an order of magnitude larger and more diverse than the Brueghel dataset, with works spreading over centuries and continents.

We demonstrate that both problems can be successfully addressed by leveraging the geometric consistency of mid-level feature matches both for training features and for scoring candidate correspondences. Our idea can be seen as revisiting the geometric verification of classical feature matches [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] for mid-level deep feature training and matching. More precisely, we propose a local matching score to compare pairs of images, which combines spatial consistency and local feature similarity. We also present a feature fine-tuning strategy that can be used without any annotations or with weak category-level annotations: we mine positive and negative pairs using geometric verification and use them to optimize a standard triplet loss.

We demonstrate experimentally that the proposed local matching score provides important performance gains for both tasks compared to global features, and that our feature fine-tuning -without any annotations on the Brueghel dataset and with weak category-level supervision on the watermark dataset -further improves the results. For example, on the challenging crossdomain one-shot recognition over more than 16k finegrained watermark categories, for which average pooling fails (0% accuracy), our local matching score using classification pretrained features achieves 45% accuracy, and our fine-tuning further improves it to 55%.

Our code and data are available at our project page http://imagine.enpc.fr/ ~shenx/HisImgAnalysi s/.

Related work

In this section, we first discuss the use of mid-level features for image recognition, then works related to our two applications, artwork analysis and watermark recognition.

Mid-level features for image recognition. Recent work analyzing the performance of CNNs [START_REF] Brendel | Approximating cnns with bagof-local-features models works surprisingly well on imagenet[END_REF][START_REF] Geirhos | Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness[END_REF] suggests that they might ignore a large part of the spatial information present in the image, and rather work in a way similar to classical order-less bags-of-features methods [START_REF] Wallraven | Recognition with local features: the kernel recipe[END_REF][START_REF] Grauman | Pyramid match kernels: Discriminative classification with sets of image features[END_REF][START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF]. This might not be suitable for recognizing repeated patterns such as watermarks and near-duplicated details in artworks, where the actual shape is key. To build on the local features learned by CNNs but consider spatial information, we follow an approach closely related to the classic spatial verification step introduced in Video Google [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] with SIFT features [START_REF] Lowe | Distinctive image features from scaleinvariant keypoints[END_REF]. Rather than using SIFTs, which we found were not adapted to non-photographic depictions, we use intermediary deep features, that can be thought of as mid-level image features. Mid-level features [START_REF] Singh | Unsupervised discovery of mid-level discriminative patches[END_REF][START_REF] Doersch | Mid-level visual element discovery as discriminative mode seeking[END_REF][START_REF] Doersch | Context as supervisory signal: Discovering objects with predictable context[END_REF] have been used in cross-domain matching [START_REF] Aubry | Painting-to-3d model alignment via discriminative visual elements[END_REF], video instance segmentation [START_REF] Wang | Learning correspondence from the cycle-consistency of time[END_REF] and coarse parametric transformation estimation [START_REF] Shen | Ransacflow: generic two-stage image alignment[END_REF]. Our feature fine-tuning also leverages the spatial structure of images: thus, it is related both to selfsupervised feature learning methods that use spatial information to learn deep visual features in images [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF] or videos [START_REF] Wang | Learning correspondence from the cycle-consistency of time[END_REF][START_REF] Jabri | Space-time correspondence as a contrastive random walk[END_REF], and to [START_REF] Rocco | Neighbourhood consensus networks[END_REF] that uses neighborhood consensus to learn correspondences from the correlation map. We adapted [START_REF] Rocco | Neighbourhood consensus networks[END_REF] to use as baseline.

Computer vision and art.

There is a long-standing and fruitful collaboration between computer vision and art. On the synthesis side, promising results have been obtained for transferring artistic style to a photograph [START_REF] Hertzmann | Image analogies[END_REF][START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF][START_REF] Zhu | Unpaired image-to-image translation using cycle-consistent adversarial networks[END_REF], or even trying to create art [START_REF] Elgammal | Can: Creative adversarial networks, generating" art" by learning about styles and deviating from style norms[END_REF][START_REF] Hertzmann | Can computers create art?[END_REF]. On the analysis side, there are several efforts on the collection and annotation of large-scale art datasets [START_REF] Karayev | Recognizing image style[END_REF][START_REF] Mensink | The rijksmuseum challenge: Museum-centered visual recognition[END_REF][START_REF] Picard | Challenges in content-based image indexing of cultural heritage collections[END_REF][START_REF] Wilber | Bam! the behance artistic media dataset for recognition beyond photography[END_REF][START_REF] Strezoski | Omniart: multi-task deep learning for artistic data analysis[END_REF][START_REF] Mao | Deepart: Learning joint representations of visual arts[END_REF][START_REF] Paumard | Jigsaw puzzle solving using local feature co-occurrences in deep neural networks[END_REF], and using them for genre and authorship classification [START_REF] Karayev | Recognizing image style[END_REF][START_REF] Tan | Ceci n'est pas une pipe: A deep convolutional network for fineart paintings classification[END_REF][START_REF] Strezoski | Omniart: multi-task deep learning for artistic data analysis[END_REF] or fragment alignment [START_REF] Paumard | Jigsaw puzzle solving using local feature co-occurrences in deep neural networks[END_REF]. Others focus on applying and generalizing visual correspondence and object detection methods to paintings using both classical [START_REF] Shrivastava | Data-driven visual similarity for cross-domain image matching[END_REF][START_REF] Ginosar | Detecting people in cubist art[END_REF][START_REF] Crowley | Of gods and goats: Weakly supervised learning of figurative art[END_REF][START_REF] Aubry | Painting-to-3d model alignment via discriminative visual elements[END_REF][START_REF] Seguin | Tracking transmission of details in paintings[END_REF], as well as deep [START_REF] Crowley | Face painting: querying art with photos[END_REF][START_REF] Crowley | The art of detection[END_REF][START_REF] Westlake | Detecting people in artwork with cnns[END_REF][START_REF] Gonthier | Weakly supervised object detection in artworks[END_REF] approaches. In particular, Yin et al. [START_REF] Yin | Object recognition in art drawings: Transfer of a neural network[END_REF] used the same Brueghel data [START_REF] Brueghel Family | Jan brueghel the elder[END_REF][START_REF] Honig | Jan Brueghel and the Senses of Scale[END_REF] as us and annotate it to train detectors for five object categories (carts, cows, windmills, rowboats, and sailboats). Beyond classification, several projects have built search engines for artworks [3,4]. An original and related approach was also proposed by [START_REF] Jenicek | Linking art through human poses[END_REF] to relate artwork by estimating human poses rather than simply comparing abstract features. Our work is also related to [START_REF] Úbeda | Pattern spotting in historical documents using convolutional models[END_REF] which performs pattern spotting in historical documents but for patterns with little appearance change. To the best of our knowledge, our work is the first to demonstrate a method capable of detecting small copied details across different depiction styles: this fine-grained analysis is the most relevant for art historians, as it may allow them to discover influences, find provenance and establish authorship.

Historical watermark imaging and recognition. Manual tracing and back-lit photography are two standard ways to reproduce watermarks. Manual tracing [START_REF] Piccard | Die Wasserzeichenkartei Piccard im Hauptstaatsarchiv Stuttgart: Wasserzeichen Buchstabe P[END_REF]10] consists in copying the watermark pattern on tracing paper and was tyically used to create the classical catalogs of watermarks, which are nowdays aggregated in online databases such as Briquet Online [1] which includes specifically the drawings from [10] we worked with. Back-lit photography is the most common technique to acquire an actual photograph of a watermark. While a watermark is often barely visible, placing it in front of a light source and looking at the transmitted light reveals it more or less clearly.

This cross-modality between the drawings and the photographs one would like to identify, is one of the main challenges of watermark recognition. Several studies have focused on localizing and extracting the pattern of a watermark from a photograph [START_REF] Hiary | A system for segmenting and extracting paper-based watermark designs[END_REF][START_REF] Hiary | Paper-based watermark extraction with image processing[END_REF][START_REF] Said | Watermark location via back-lighting modelling and verso registration[END_REF], which could potentially be helpful to match a photograph to a database of drawings [START_REF] Rauber | Retrieval of images from a library of watermarks for ancient paper identification[END_REF]. However, these approaches are complex, not end-to-end and have not yet been demonstrated on a large scale. Some works on watermark recognition focuse on drawings. [START_REF] Rauber | Retrieval of images from a library of watermarks for ancient paper identification[END_REF] uses histogrambased descriptors in a spirit similar to shape context [START_REF] Belongie | Shape context: A new descriptor for shape matching and object recognition[END_REF], while more recent works use machine learning approaches, such as dictionary learning [START_REF] Picard | Non-negative dictionary learning for paper watermark similarity[END_REF] or neural networks [START_REF] Pondenkandath | Identifying cross-depicted historical motifs[END_REF]. The study most similar to us is probably [START_REF] Pondenkandath | Identifying cross-depicted historical motifs[END_REF], that used a non-publicly available database [START_REF] Frauenknecht | Wzis -wasserzeicheninformationssystem: Verwaltung und präsentation von wasserzeichen und ihrer metadaten. Kodikologie und Paläographie im Digitalen Zeitalter 3: Codicology and Palaeography in the Digital Age[END_REF] to train a convolutional neural network classifying 106,000 watermark into 12 coarse categories.

Method

In this section, we start by introducing our similarity score named "local matching score" which combines spatial consistency and local feature similarity in Section 3.1. Then, in Section 3.2, we detail our feature learning strategies for both the weakly supervised and the unsupervised scenario, which correspond to the two applications focused in this work. Precisely, for watermark recognition, a small number of image-level annotations are accessible and watermarks are approximately centered. For artworks, no annotations are available. The overview of our feature learning strategies is shown in Figure 2. Finally, we provide some implementation details in Section 3.3. (b) In case of category-level supervisions (top), we verify whether matched pairs belong to the same category and their spatial difference is small. If no supervisions is available (bottom), the candidate correspondences are verified by matching the features of the verification region (in blue, note that, to make the figure visible, the blue regions are visualized as 5 × 5 squares while we use 10 × 10 squares in our algorithm.) of the query in the candidate images and checking for consistency. (c) Finally, we extract features from the positive (in green) and negative (in yellow) regions and use them to improve the features using a metric learning loss.

Local matching score

Let us assume we have access to a trained convolutional neural network that we want to use to compare two images I 1 and I 2 . The simplest solution is to use the similarity between aggregated features, typically the cosine similarity between average-pooled features for a ResNet network. However, such features will completely discard location information, which is likely not well adapted for fine-grained differentiation between similar classes. Another natural solution is then to consider convolutional features and use the similarity between the resulting descriptors. However, this gives the same importance to all local features, while some might be discriminative and others might not be, and this amounts to comparing only convolutional features at the same spatial location, i.e., assuming that the images are exactly aligned. This is a strong assumption and is not realistic in the cases we study: precisely centering and scaling the watermarks is difficult and the copied details in artworks might have undergone some amount of local deformation.

Thus, we propose a similarity metric to compare a pair of images I 1 and I 2 which relies on matching densely mid-level CNN features (e.g., conv4 in ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]) computed at different scales. To select only discriminative features, we check that they correspond to reciprocal matches and discard them otherwise. More formally, for every local feature f i 1 in I 1 , we first find its best match in I 2 , f i 2 . We then check that it is a reciprocal match, i.e. that the best match i.e. the best match in I 1 of f i 2 is f i 1 and we write M the set of all reciprocal matches. We also write x i 1 the position of f i 1 in I 1 , and

x i 2 the position f i 2 in I 2 .
Assuming that the two images are approximately aligned, which is true for example in the watermark dataset, we define the local matching score S by combining a Spatial Consistency score (SC) measuring the similarity between the positions x i 1 and x i 2 , and a Feature Similarity (FS) measuring the distance between f i 1 and f i 2 :

S(I 1 , I 2 ) = i∈M e - x i 1 -x i 2 2 2σ 2 SC s(f i 1 , f i 2 ) F S (1)
where s is a feature level similarity -for which we use cosine similarity in all of our experiments -, and σ is a tolerance parameter.

If the images are not coarsely aligned, which is the case for repeated details detection in artworks, we need to further model the alignment between the two images. We do so by estimating a parametric transformation T between the images, using RANSAC on the set of reciprocal matches M. We then compute the score:

S(I 1 , I 2 ) = i∈M e - T (x i ) 1 -x i 2 2 2σ 2 SC s(f i 1 , f i 2 ) F S (2)
T is an Affine transformation and therefore has 6 parameters. Classical RANSAC is employed to estimate T : for each iteration, we sample 3 matches to compute a candidate transformation ; a match is considered as an inlier if its difference with the candidate transformation is smaller than 2 in the feature space; the final transformation is the one with maximum number of inliers after 1k iterations.

Spatially-consistent feature learning

The overview of our feature learning strategies is shown in

Hypothesis and objective function

Our goal is to improve local features for matching. Starting with a standard pretrained deep feature extractor, we extract matching regions from the dataset that we can then circle back and use to improve the features. Our two key hypotheses are that: (i) our dataset includes large parts of images that contain repeated patterns, and (ii) the initial feature descriptor is good enough to extract some positive matches. We follow a metric learning approach. Assuming we have a set of positive pairs P and a set of negative pairs N , we learn our feature extractor f by minimizing a standard triplet loss:

L(f ) = (n1,n2)∈N max(1 -λ, s(f (n 1 ), f (n 2 ))) - (p1,p2)∈P min(λ, s(f (p 1 ), f (p 2 )) (3)
where the similarity measure s is the cosine similarity and λ is a hyper-parameter.

The main challenge for learning such features is defining the sets of positive and negative pairs. In the following sections, we show how to find these pairs in both weakly supervised and unsupervised cases.

Candidates

We randomly sample several query features from each image in the dataset and match them densely at every scale to all the images in the dataset using cosine similarity. This can be done efficiently and in parallel for many queries using the normalized query features as the weights of a convolution applied to the normalized features. For each query, we select its top K matches in the entire dataset as candidate correspondences (Figure 2(a)).

Verification

We now explain how we filter the positive and negative pairs from these candidate local feature pairs both in the case with and without image-level supervision.

Image-level supervisions In the case where image level category labels are available, they can be used as a first cue to select positive matches: they can only exist between images belonging to the same category. Furthermore, if we assume that the images are coarsely aligned, as is the case for our watermark recognition problem, we can additionally filter only matches that are spatially consistent up to a given spatial threshold τ . This threshold allows to take into account misalignment between the source and target images and we study its influence in our experiments. Once a positive pair has been identified, we look for hard negatives by matching one of the positive features with all the photographs of watermarks from other categories, and select the the most similar feature as negative (Figure 2 The choice of the verification region is important to the verification step. The key aspect is that the features in the verification region should be, as much as possible, independent of the features in the proposal region. On the other hand, having them too far apart would reduce the chances of the region being completely matched.

For our experiments, we used the 10x10 feature square centred around the query region.

Finally, given a set of verified correspondences, we have to decide which features to use as positive training pairs (Figure 2(c) bottom). One possibility would be to directly use features from the proposal region, since they have been verified. However, since the proposal region has already been "used" once (to verify the matches), it does not bring enough independent signal to make quality hard positives. Instead, we propose to sample positives from a different positive region. We evaluated different configurations for the positive region, as visualized in Figure 3 (in green). We choose to keep only 4 positive pairs per verified proposal, positioned at the corners of a square and denote the different setups as P2 to P14, the number corresponding to the size of the square. We will show in the experiments (Section 5.2) that P12 and P14 perform better than the alternatives. The features from the positive regions (Figure 2(b) bottom in green) are then used as hard positives for feature fine-tuning (Figure 2(c) bottom).

For a given positive pair, we extract top-N closest features from the same image as negatives. This selects hard negatives and avoids any difference in the distribution of the depiction styles in our positive and negative samples. We chose a relatively high number (N = 20) of negatives to account for the fact that some of them might actually correspond to matching regions, for example, in the case of repeated elements, or to locations near the optimal match.

Implementation details

In all of our experiments, we used conv4 features of the ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] architecture and train the models with the Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] optimizer with learning rate 1e-5 and β = [0.9, 0.99]. For both applications, we use K = 10 candidate matches for each query.

For fine-tuning on watermark datasets, the initial features are pretrained on our classification dataset (see Section 4.1). The hyper-parameter λ is set to 1. Since the watermarks might be flipped and rotated with respect to the references, we consider matches with four rotated (0 • , 90 • , 180 • , 270 • ) reference images and their flipped versions. Each source image was resized to 352×352 so that it was represented by 22×22 features. To be robust to scale discrepancies, we matched the source features to features extracted from the target image resized at five scales, corresponding to square features of widths 16, 19, 22, 25 and 28. Using a single Geforce GTX 1080 Ti GPU, training converged in approximately 2 hours

For learning features on art collections, the initial features are pretrained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. The hyperparameter λ is set to 0.8. Since repeated patterns might have large scale differences in different artworks, we used 7 different scales ranging from 20 to 80 features in the largest dimension, regularly sampled on two octaves with 3 scales per octave. From these candidates, the top 10% with the most votes from neighbours were considered verified. Note that these parameters might need to be adjusted depending on the diversity and size of the dataset, but we found that they performed well for the Brueghel data [START_REF] Brueghel Family | Jan brueghel the elder[END_REF]. Using a single GPU Geforce GTX 1080 Ti, the training converged in approximately 10 hours.

Experiments on watermark recognition

In this section, we provide experimental results on watermarks recognitions. We first introduce the datasets we used, then present detailed results and analysis of the proposed local matching score and the proposed feature learning strategies.

Datasets

In this section, we explain how we created datasets to evaluate (A) one-shot and (B) cross-domain watermark recognition in the case where the watermarks are coarsely aligned. In the supplementary material available on our project page, we explain in detail the definition of classes and preprocessing procedures. More samples can be found in the project page. Dataset B An important challenge is to use the drawings from existing watermark catalogs to perform recognition. Indeed, collecting watermarks and information for such catalogs is a very tedious and expensive work. We focused on the Briquet catalog [10,1]. The available images include additional information, such as IDs of watermarks, paper line positions, or complementary marks that can be found at another position on the paper sheet. Such information is valuable to experts, but cannot easily be used in an automatic recognition system based on a single photograph. We thus extracted the main part of the watermarks whenever it was clear and ended up with 16,753 drawings that could be used as references for photograph recognition. Examples of these drawings can be seen on the project page.

The challenge that we want to evaluate on this dataset is to recognize photographs in a drawing database. We thus collected dataset B by searching the original archives that provided the material for the Briquet catalog [10], in a specific city (Paris) and collected photographs for 240 classes (see examples in Figure 4(d)). We use 140 classes with a total of 463 photographs as training and keep the remaining 100 classes with 2 photographs in each class for testing. Because comparing photographs (Fig. 4(c)) directly with a line drawing (Fig. 4(d)) as a reference is very challenging, we also report results using a synthetic image generated from the drawing simply by using the average watermark color as background and making the drawing pattern lighter (Fig. 4(e)). In our experiments, we first compare the methods for 100class one-shot cross-domain classification and then give the results for the even more challenging 16,753-class classification. In the following sections, we refer the 100class one-shot cross-domain classification as the task on the dataset B. The 16,753-class classification is the task on Briquet.

One-shot cross-domain recognition

In this section, we present results on the one-shot recognition on the dataset A-one-shot and one-shot crossdomain recognition on the dataset B. In all experiments, we use a network pretrained on A-classification (100 classes, 50 training and 10 validation images for each class). The best performances were obtained with a strong dropout (0.7 ratio), which is not surprising given the relatively small size of our dataset. The validation accuracy is 98.8%, the mis-classified images being only very difficult or ambiguous cases, which shows that our 6k images dataset was large enough to train a good network for fine-grained watermark classification.

Comparison to one-shot recognition methods

On dataset A-one-shot, which does not include any domain shift, we compare our method with some one-shot recognition methods: -Baseline: directly using the features learned by training a network on the classification task and computing the dot-product as the distance between classification weights and features. -Cosine Classifier : [START_REF] Gidaris | Dynamic few-shot visual learning without forgetting[END_REF][START_REF] Qi | Low-shot learning with imprinted weights[END_REF] have shown that the performance of the baseline can be improved if the dotproduct operation (between classification weights and features) in the last linear layer of the network is replaced with the cosine similarity operation. -Matching Networks: we tried the metric-learning approach of Matching Networks [START_REF] Vinyals | Matching networks for one shot learning[END_REF], performing metatraining to solve one-shot recognition tasks using a differentiable nearest-neighbor-like classifier. -Weights Prediction: the one-shot recognition approach of Gidaris and Komodakis [START_REF] Gidaris | Dynamic few-shot visual learning without forgetting[END_REF]. The key insight is that the novel class weights have two components: i) average features of annotated samples; ii) linear combinations of the base class weights, which are meta-learned during the training. It uses a feature extractor learned with a cosine-similarity-based classifier which remains frozen during the meta-training procedure.

For each feature, we report three different similarities:

-AvgPool: cosine similarity using the average pooled features.

-Concat: cosine similarity on the descriptor formed by the concatenation of all the spatial features. -Local Sim.: computing the cosine similarity over each local feature individually, then averaging.

For all baselines, we report the best performance over conv4/conv5 features. The optimal parameters and training strategy for each baseline and a qualitative analysis of our results are reported on the project page.

The results are in Table 1, our local matching score leads to 85% accuracy for 256 × 256 images, which is close to the best one-shot approach, Weights Prediction [START_REF] Gidaris | Dynamic few-shot visual learning without forgetting[END_REF], but without any specific feature learning. This demonstrates the interest of our local matching score for one-shot fine-grained watermark recognition. The performance can further be boosted to 90% by resizing images to a larger resolution, 352×352 pixels, which was not possible for Weights Prediction for very large features due to computational cost. This is clearly above the performances of the other methods.

Comparison of feature similarities for one-shot crossdomain recognition In Table 2, we compare our local matching score (eq. 1) with alternative feature similarities on our two datasets. On dataset B, we use either the drawing (B-4(d)) or our synthetic image (B-4(e)) as reference. We always used the features trained for classification on dataset A-classification, and compare on each dataset to the similarities described in the previous paragraph (AvgPool, Concat, and LocalSim.). Our local matching score consistently outperforms all these baseline similarities. In our naive implementation, our approach is slower than direct feature comparison, but both can be mixed to obtain fast results on very large datasets.

Learning features for cross-domain recognition

We now focus on cross-domain recognition. We first compare our approach with different feature training strategies on our dataset B. We scale our watermark recognition method to the full Briquet catalog, showing we can perform classification with more than 16k classes.

Feature fine-tuning In Table 3, we compare the results from our fine-tuning strategy to different baselines:

-Unsupervised (translation): in a spirit similar to [START_REF] Sun | Return of frustratingly easy domain adaptation[END_REF], we translated the features in our target domain so that, on our training set, they have the same mean as the features from the target domain. We then use our score to perform nearest-neighbor classification.

Table 3: Accuracy (in %) on one-shot cross-domain recognition with different feature fine-tuning approaches. τ is the hyper-parameter to constraint the misalignment of matches (Sec. 3.2.3). τ = 3/22 indicates we tolerates a difference of 3 features while the entire feature map is 22×22. We experiment with different reference images ("B-4(e)" : synthetic image ; "B-4(d)" : drawing). Note that all the approaches except NC-Net [START_REF] Rocco | Neighbourhood consensus networks[END_REF] use local matching score -Supervised (affine): since we have aligned images from both domains, we can adapt features from the source and target domains in a supervised way, similarly to [START_REF] Massa | Deep exemplar 2d-3d detection by adapting from real to rendered views[END_REF][START_REF] Rad | Feature mapping for learning fast and accurate 3d pose inference from synthetic images[END_REF]. We found that a simple affine adaptation gave the best results, likely because of the small size of our dataset. -Randomization: we also report results using randomized generated synthetic images from the drawings. The randomized synthetic image S are generated by computing S = B + R × (G * E), where B is a background sampled from photographs of paper without watermarks, G is a Gaussian filter, R is a random image and E is the binary watermark pattern extracted from the drawing. Generated images can be found in the supplementary material.

We trained a standard classifier on the generated images. Such an approach has been shown to be very successful for example for 3D pose estimation [START_REF] Su | Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views[END_REF][START_REF] Loing | Virtual training for a real application: Accurate object-robot relative localization without calibration[END_REF]. -Triplet-loss: similarly to our method, we tried to improve the features using a triplet loss on local features, but using as positive all aligned features in the images from the same category. -NC-Net [START_REF] Rocco | Neighbourhood consensus networks[END_REF]: while it was not initially designed for domain adaptation, we trained NC-Net on our database because of the intuition that, similarly to our method, it is able to learn to leverage spatial information. We use our pre-trained ResNet for the feature extractor and freeze it during the training. The other parts are kept the same as the category level matching model proposed in [START_REF] Rocco | Neighbourhood consensus networks[END_REF]. The positive pairs are composed Table 4: Top-1 and top-1000 accuracy on our Briquet dataset with different models ("Briquet-4(e)" : model trained on classification on dataset A-classification ; "Briquet-4(e)+Fine-tuning" : fine-tuned model). Our approach first applies Local Similarity to obtain N top ranked references and then uses our score to re-rank the N references.

with one image from each domain, which results in 463 pairs in the training set. The training converges in 20 epochs. We then consider the sum of the scores over all correspondences as the score between a pair of images.

We detail these approaches and the training details in the supplementary material available in our project website. All results except for NC-Net are reported for matching done with our local matching score -using other metrics leads to worse performance. The effectiveness of this metric might be the reason why standard domain adaptation approaches only marginally improve performances over the baseline. Another possible reason is the small size of our training set (463 photographs, 140 references). On the contrary, our fine-tuning strategy boosts performances by a clear margin.

Large-scale recognition on Briquet We finally evaluate one-shot cross-domain recognition using the test photographs of our dataset B and our full curated version of the Briquet dataset as reference. This comparison with 16,753 fine-grained classes is very challenging, but also corresponds to a realistic scenario for watermark recognition. We use our synthetic images (Figure 4(e)) to represent the drawings. with our local matching score takes 37s on a single GPU Geforce GTX 1080 T. This is acceptable for practical use, and we thus believe the application of our algorithm will be a game-changer and widen considerably the potential use of watermark analysis, which until now has been limited to a small number of experts.

Web application

The ultimate goal of watermark recognition is to develop an application to simplify the search of watermarks. We developed a first version of such a web application [START_REF] Bounou | A web application for watermark recognition[END_REF] which can be accessed at https: //filigranes.inria.fr/. Although the task is challenging, as some queries are cluttered and some other categories are highly similar to the ground-truth, our approach allows to recognize the correct categories. Note that more visual results can be found in the project page.

Qualitative results

Experiments on one-shot art pattern detection

In this section, we present results on one-shot art pattern detection. We first introduce the datasets we used, then present detailed results on one-shot detection on Brueghel [START_REF] Brueghel Family | Jan brueghel the elder[END_REF] dataset. We finally show qualitative results on the Venus dataset.

Datasets

Brueghel The Brueghel [START_REF] Brueghel Family | Jan brueghel the elder[END_REF] dataset contains 1,587 artworks done in different media (e.g. oil, ink, chalk, watercolour) and on different materials (e.g. paper, panel, copper), describing a wide variety of scenes (e.g. landscape, religious, still life). This dataset is especially adapted for our task since it assembles paintings from artists related to the same workshop, who thus had many interactions with each other, and includes many copies, preparatory drawings, and borrowed details.

With the help of our art history collaborators, we selected 10 of the most commonly repeated details in the dataset and annotated the visual patterns in the full dataset using the VGG Image Annotator tool [START_REF] Dutta | VGG image annotator (VIA)[END_REF]. The 10 annotated patterns can be seen in Figure 6 as queries (blue boxes), and our full annotations can be found in the project page. We were careful to select diverse patterns, and for each of them to annotate only duplicates, and not full object classes. Note for example, that for the horses and lion classes, we annotated separately two variants of the details (front and back facing lion, front and right facing horse). This resulted in 273 annotated instances, with a minimum of 11 and a maximum of 57 annotations per pattern.

These annotations allow us to evaluate one-shot duplicate detection results. In our evaluation, we use an IoU threshold of 0.3 for positives, because precise annotations of the bounding boxes in different environment is difficult and approximate detection would be sufficient for most applications. In practice, our detected bounding boxes, visualised in Figure 6 (green boxes) often appear more consistent than the annotations. We compute the Average Precision for each query, average them per class and report class level mean Average Precision (mAP).

Venus To demonstrate the generality of our approach, we show results on a very different database, containing 25,681 depictions of Venus [START_REF] Bender | Distant viewing in art history. a case study of artistic productivity[END_REF]. This database is much larger than the Brueghel database and more diverse since it includes depictions from many different periods and very diverse styles. Because it is thematic, it however includes many cases of revisited artwork and artistic citations. It is thus much better suited to our purpose than standard artwork databases. The size and diversity of the database makes annotation unpractical, but we report qualitative results which demonstrate matches of interest beyond the case of copied details. Fig. 6: Detection example with our trained features on the Brueghel dataset. We show the top 5 matches (in transparency) as well as the annotations (green bounding box) for one example of query from each of our 10 annotated categories. Notice how the style of the matches can be different from the one of the query. Table 5: Experimental results on Brueghel, IoU > 0.3.

One-shot detection

We evaluated our feature learning strategy using oneshot detection mainly on the Brueghel dataset which was small enough to be manually annotated. We performed one-shot detection simply by computing dense features on the dataset and computing their cosine similarity with the features corresponding to the query. The query was resized so its largest dimension in the feature map would be 8. Note that unlike standard deep detection approaches [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], we do not use region proposals because we want to be able to match regions which do not correspond to objects.

Comparison and analysis. We compare one-shot detection performance with different features using both cosine similarity and the score described in equation 1.

In Figure 8 we show an example of the top-6 matches for the same query using different approaches. In the first line, the matches obtained with the cosine similarity between features trained on ImageNet are all from styles similar to the style of the query, and while three of them include horses, they are not in the same pose as the query. On the contrary, the matches with our trained features, shown on the second line, mainly associate horses in the exact same position, including in depictions with different styles. Finally, the results with the same trained features but using the local matching The corresponding quantitative results are presented in Table 5 and confirm these observations. Indeed, learning our features improves the score by approximately 30%. The discovery procedure and score provide an additional boost.

Positive region configuration. We now focus on evaluating the different positive region settings described in Section 3.2.3 and Figure 3. For each of them, we analyse the performance of the features on one-shot learning on the Brueghel dataset and its evolution during training. The results can be seen in Figure 7. Interestingly, the performance always initially improves over ImageNet features. However, when the positive region is close to the proposal region, the performance decreases after some iterations of our training procedure and ends up with worse performance than the initial features. However, if the positive region is far enough from the query (P12 and P14), the performance improves much more and does not subsequently deteriorate. We thus use P12 for all our other experiments.

Training visualization. To visualise the influence of the feature training for our discovery task, we selected a pair of matching images from the Brueghel dataset and visualised the the inlier set I (defined in equation 1) at different steps of training in Figure 9. During training, larger and larger parts of the images can be matched in a consistent way. This shows both the efficiency of our feature training and its relevance for our task.

Qualitative results. Qualitative results using our approach for each of the 10 details we annotated on the Brueghel dataset are shown in Figure 6. It gives a sense of the difficulty of the task we target and the quality of the results we obtain. Note for example how the matches are of different styles, and how the two types of lions (top row) and the two types of horses (bottom row) are differentiated. In the following, we compare these results with baselines and analyze the differences.

In Figure 10, we show qualitative results on the Venus dataset. In this case, the retrieved images might not be exact copies but sets of works that revisit at different periods the same model, sometime borrowing only part of the works and keeping details, such as Flora from the Primavera from Botticelli in the last row, sometime focusing on the composition such as for La Venus del espejo from Velázquez in the 7th row. Note how our method is able to retrieve relevant results in all of these cases, despite strong variations. These results showcase the interest of our method for Art Historians to explore such collections where one can hardly expect exhaustive annotations and meta-data.

Limitations Our method has several limitations. First, the approach need to compare image pairs, which is not possible for fast retrieval applications. Second, in both applications, our feature learning strategy requires a good feature initialization. For watermark recognition, we start from features learned from A-classification. For artwork detection, we start from ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] pretrained features.

Conclusion

In this paper, we have introduced two cultural heritage applications and associated datasets: watermark recognition and one-shot repeated art pattern detection. We have presented a unified mid-level feature training and matching method for both applications. We showed that the proposed local matching score clearly improves over standard feature similarities and that our mid-level feature fine-tuning approach can further boost performances on both tasks.

  (a) Historical watermark recognition (b) One-shot repeated pattern detection

Fig. 1 :

 1 Fig. 1: We present a unified approach to two problems: (a) historical watermark recognition, and (b) repeatedpattern detection in artwork collections.

Fig. 2 :

 2 Fig.2: Mid-level feature learning with weak supervisions or without supervisions. (a) Our approach relies on candidate correspondences obtained by matching the features of a proposal region (in red) to the full database. (b) In case of category-level supervisions (top), we verify whether matched pairs belong to the same category and their spatial difference is small. If no supervisions is available (bottom), the candidate correspondences are verified by matching the features of the verification region (in blue, note that, to make the figure visible, the blue regions are visualized as 5 × 5 squares while we use 10 × 10 squares in our algorithm.) of the query in the candidate images and checking for consistency. (c) Finally, we extract features from the positive (in green) and negative (in yellow) regions and use them to improve the features using a metric learning loss.

Figure 2 ,

 2 which can be separated into three parts: 1. searching candidates of training samples (Figure 2(a)); 2. verification of candidates with weak category-level supervision or no supervision (Figure 2(b)); 3. training on selected samples (Figure 2(c)). We first explain the objective function we use to improve our features in Section 3.2.1, then present how we sample candidate matches in Section 3.2.2. Finally, we detail how to filter training samples in the candidate matches in Section 3.2.3.

Fig. 3 :

 3 Fig. 3: Different region configurations. red: query regions, blue: verification regions, green: positive regions.

DatasetA

  The goal of this dataset is to train and evaluate methods for one-shot fine-grained watermark recognition from photographs. It thus only includes photographs and is split into two parts: a first part that can be used for feature training or meta-training, with many examples of each watermark, and a second part, with few examples, to evaluate one-shot recognition.We first created dataset A-classification by collecting 50 training and 10 validation images for 100 watermarks, which we found was large enough to perform pre-training / meta-training of CNNs. Examples of images from the same class are given in Figure4(a).

  (a) Samples from the same class in A-classification (b) Clean references (top) and associated test sample (bottom) in A-one-shot (c) Photograph (d) Drawing (e) Synthetic

Fig. 4 :

 4 Fig. 4: Images in watermark datasets: (a) Samples from the same class in A-classification; (b) three categories in A-one-shot. (c) -(e): Photograph, Drawing and Synthetic in B-cross-domain.

  Fig.5: Top-3 matches retrieved with our approach with synthetic reference (Figure4(e)) on Briquet dataset (16,753 categories). For all the four queries, our approach is able to recognize the correct categories.

  Typical examples of top-3 matches with synthetic references (Figure 4(e)) are shown in Figure 5.

Fig. 7 :

 7 Fig. 7: Evolution of the mean Average Precision for oneshot matching on the Brueghel dataset during training. Performance decreases after a few iterations for settings where we extract positive regions correlated with the proposal region.

Fig. 8 :

 8 Fig. 8: From a single query, shown on the left, we show the detection results obtained with cosine similarity with ImageNet feature (a) and our trained features (b) as well as the ones (c) obtained with our features and the local matching score presented in Section 3.1.

Fig. 9 :

 9 Fig. 9: Visualizing matched correspondences between a pair of images during training.

  (a) Query (b) Top six detection results

Fig. 10 :

 10 Fig. 10: Each row shows the top 6 matches on the Venus database given queries outlined in blue on the left images.

Table 2

 2

	Method	A	B-4(d) B-4(e) Time (s / Query)
		Exact features comparison
	AvgPool	78	4	12	1
	Concat	86	55	61	2
	Local Sim.	81	56	65	2
		Our local matching score
	Ours	90	66	72	15
	: Comparison of our local matching score (eq. 1)
	with alternative feature similarities. Accuracy in % for
	one-shot recognition on dataset A-one-shot (A column)
	and one-shot cross-domain recognition on dataset B us-
	ing either the drawing (B-4(d)) or our synthetic image
	(B-4(e)).

Table 4

 4

	shows the top-1 and

Acknowledgements This work was supported in part by ANR project EnHerit ANR-17-CE23-0008, PSL Filigrane pour tous project, project Rapid Tabasco, gifts from Adobe to Ecole des Ponts. We thank Minsu Cho, Pascal Monasse and Renaud Marlet for fruitful discussions, and Kenny Oh and Davienne Shields for thier help on defining the task and building the Brueghel dataset.

data are available at http://imagine.enpc.fr/