Théo Moins
email: theo.moins@inria.fr

Julyan Arbel
email: julyan.arbel@inria.fr

Anne Dutfoy
email: anne.dutfoy@edf.fr

Stéphane Girard
email: stephane.girard@inria.fr

A local version of R-hat for MCMC convergence diagnostic

Keywords: Chaînes de Markov, Statistiques bayésienne, Diagnostique de convergence Markov chains, Bayesian statistics, Convergence diagnostic

Diagnostiquer la convergence des chaînes de Markov est un enjeu crucial pour les méthodes de Monte Carlo par chaîne de Markov. Parmi les méthodes les plus populaires, le diagnostique de Gelman-Rubin, communément appelé R, est un indicateur qui permet de vérifier la convergence en se basant sur une comparaison des variances inter-et intra-chaîne. Nous proposons ici une version localisée R(x) qui se concentre sur un quantile x de la distribution, et analysons certaines propriétés de la valeur théorique associée R(x). Ceci conduit à proposer un nouvel indicateur R∞ , qui permet à la fois de localiser la convergence de la chaîne en différents quantiles de la distribution, et en même temps de traiter certains problèmes de convergence non détectés par d'autres versions de R.

Introduction

Markov chain Monte Carlo (MCMC) algorithms have strongly contributed to the popularity of Bayesian models to sample from posterior distributions, especially in highdimensional or high computational settings. The fundamental idea behind those algorithms is the convergence of the sampling distribution to the target (typically the posterior) when the number of samples goes to infinity. A major challenge is therefore to know if the behaviour for a finite number of draws is satisfactory or not. This allows for a handle on the number of iterations to be drawn, which is all the more crucial in complex models with costly sampling schemes. See [START_REF] Roy | Convergence diagnostics for Markov chain Monte Carlo[END_REF] for a recent literature review on convergence diagnostics.

A property that is frequently mentioned to verify chain convergence is mixing (see [START_REF] Vats | Invited discussion: Rank-normalization, folding, and localization: An improved R for assessing convergence of MCMC[END_REF] for a discussion). It refers in practice to the exploration of the support of F : slow mixing chains correspond to chains that only explore a subset of the parameter space, which can lead to strong bias in the distribution.

A common way to avoid mixing issues is to run several chains in parallel with different starting points, which also allows comparing the chains together. We place ourselves in that case: consider m chains of size n, with θ (i,j) denoting the ith draw from chain j. We focus here on the Gelman-Rubin diagnostic [START_REF] Gelman | Inference from iterative simulation using multiple sequences[END_REF], named potential reduction scale factor and commonly denoted by R. It is by far one of the most popular methods to assess MCMC convergence, used in particular in Stan, PyMC3, or NIMBLE. The original heuristic for R construction is the comparison between two estimators that converge to the target variance Var[θ], based on Ŵ and B, respectively the estimated within-and between-variances. This diagnostic has the advantage of being scalar even in the case of a huge number of chains and comes with a rule of thumb that makes it very easy to use: generally R ≥ 1, and if it is greater than a given threshold (for example 1.01), then a convergence issue is raised.

The use of the original version of [START_REF] Gelman | Inference from iterative simulation using multiple sequences[END_REF] has some limitations, which can be found for example in [START_REF] Vats | Revisiting the Gelman-Rubin Diagnostic[END_REF]; [START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF]: lack of interpretability, lack of robustness for certain types of non-convergence, arbitrary choice of a threshold, etc. We hope to take a step forward in addressing these limitations with a localized version of R introduced in [START_REF] Moins | Contributed discussion: Rank-Normalization, Folding, and Localization: An Improved R for Assessing Convergence of MCMC[END_REF] and developed here and in Moins et al. (2022): we analyze R(x), a local version of R associated with a given quantile x, and the corresponding population value R(x). This study leads us to suggest a new indicator R∞ , which in addition to being more interpretable, shows better results in terms of MCMC convergence diagnostic.

A local version of R

Population version. For all x ∈ R, introduce the Bernoulli random variable I x = I{θ ≤ x}, where I{•} denotes the indicator function. The idea of our local convergence estimate is decidedly simple: we use I x in place of θ in the original Gelman-Rubin construction.

If Z ∈ {1, . . . , m} denotes the corresponding index of the chain, the population withinchain and between-chain variances at point x are then defined respectively as

W (x) = E [Var[I x | Z]] and B(x) = Var [E[I x | Z]] .
Note that both quantities exist whatever the tail heaviness of θ distribution thanks to introduction of the indicator function, thus relaxing moment conditions of the original R. We define the associated population R(x)

as

R(x) = W (x) + B(x) W (x) .
It turns out that under the assumption of stationarity for each chain, R(x) can be expressed in closed form with respect to the chains' distribution.

Proposition 2.1 Suppose that, for any j ∈ {1, . . . , m}, P(Z = j) = 1/m and θ given Z = j has cumulative distribution function (cdf ) F j . Then, one has for any x ∈ R:

R(x) = 1 + m j=1 m k=j+1 (F j (x) -F k (x)) 2 m m j=1 F j (x)(1 -F j (x))
.

(1) Thus, using I x instead of θ defines a local convergence estimate at any point x which quantifies a distance between the F j 's. This allows for diagnostic convergence relatively to a quantile one wants to estimate (for a posterior credible interval for example).

In order to summarize this continuous index into a scalar one, we may also consider its supremum over R:

R ∞ = sup x∈R R(x). (2)
Note that R ∞ is finite simply as soon as the F j 's are continuous with overlapping supports. Considering R ∞ amounts to considering the local version R(x) corresponding to the quantile x with the poorest convergence when no information is given on the posterior interval used for inference.

Sample version. Population version R(x) can be estimated by replacing F j (x) in (1) by its empirical counterpart Fj (x) = 1 n n i=1 I{θ (i,j) ≤ x}. This is equivalent to computing the original version of R on indicator variables I (i,j) x = I{θ (i,j) ≤ x} instead of θ (i,j) . This connects with the Raftery-Lewis diagnostic [START_REF] Raftery | How Many Iterations in the Gibbs Sampler?[END_REF]) and more recently with [START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF] who suggest this transformation for effective sample size (ESS) to construct graphical diagnostics or "tail-versions" of this diagnostic. Moreover, a ranknormalization step is added in [START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF]'s versions to prevent from infinite moments, although using I (i,j) x ensures the index existence whatever the θ (i,j) distribution is. Skipping this step for R yields an explicit expression of what is estimated in the stationary case with (1). This makes the diagnostic more interpretable and allows us to obtain key theoretical results for the associated theoretical R and R ∞ .

Illustrations

In this section, we consider toy distributions for the chains, where the computation of R ∞ can be done explicitly. In particular, we first focus on two cases raised by [START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF] of deficient behaviour of the traditional R. Then, we exhibit a situation of discrepancy of rank-R, the updated version of [START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF]. All these theoretical behaviours are illustrated on a small simulation study.

Example 1: chains with same mean and different variances. To tackle the first situation of poor behaviour of the traditional R, we consider m chains following centered uniform distributions with different variances. More specifically, assume that the m -1 first chains have the cdf

F 1 = • • • = F m-1 of the uniform distribution U(-σ, σ) while the last chain has the cdf F m of the uniform distribution U(-σ m , σ m ) with 0 < σ ≤ σ m .
In such a case, the between-variance is zero and it is thus expected that R ≈ 1. In contrast, R ∞ can be written Example 2: chains with heavy-tails and different locations. As a second example of poor behaviour of R, we consider chains following Pareto(α, η) distributions, with cdf

R ∞ = 1 + m -1 m 1 - 2 1 + σ m /σ . It appears that R ∞ is an increasing function of σ m /σ starting from R ∞ = 1 when σ m /σ = 1,
F (x | α, η) = 1 -(x/η) -α , ∀x ∈ [η, +∞),
shape parameter α > 0 and lower bound η > 0. Let us recall that such a distribution is heavy-tailed (Embrechts et al., 2013, Table 3.4.2) and has a finite first moment when α > 1. We focus on the case where one chain is shifted from the other ones:

F 1 (x) = • • • = F m-1 (x) = F (x | α, η) and F m (x) = F (x | α, η m ) with 0 < η ≤ η m and 0 < α ≤ 1.
Here, the within-and between-variances do not exist and it is expected in practice that R ≈ 1. In contrast, R ∞ can be written as

R ∞ = 1 + 1 m η m η α -1 .
Clearly, R ∞ is an increasing function of η m /η starting from R ∞ = 1 when η m = η and such that R ∞ → ∞ as η m /η → ∞. Results are shown in the second column of Figure 1. This experiment corresponds to the second example of convergence issue raised by [START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF]. The same observations as for Example 1 can be made here: R∞ is more prone to indicating a convergence issue than rank-R.

Ex. 1: Example 3: chains with same mean and mean over the median. Finally, we look at an example where both R and rank-R fail to detect non-convergence. Consider m -1 exponential chains Exp(1) and one uniform U(1 -2 log 2, 1 + 2 log 2). This results in chains with same mean and mean over the median, which by construction should fool rank-R. Results are illustrated in the third row of Figure 1: the histograms of replications confirm that R∞ is able to spot the convergence issue that both R and rank-R do not.

Ex. 2: Ex. 3: U(-σ, σ)-U(-σ m , σ m ) Pareto(α, η)-Pareto(α, η m ) Exp(1)-U 1 -λ 2 ; 1 + λ 2 -1 -0.5 0 0.
In the communication, we shall also present applications to other models in a more practical case for Bayesian inference.

  and upper bounded by 2 -1/m when σ m /σ → ∞. Results are illustrated in the first column of Figure1. In the third row, the histograms of replications confirm that R∞ is able to spot the same convergence issue as the one[START_REF] Vehtari | Ranknormalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion)[END_REF] suggests.

Figure 1 :

 1 Figure 1: Illustrations with m = 4 chains, n = 200 independent iterations each. Top row: Empirical distributions F 1 = • • • = F m-1 in green distinct from F m in blue. For the uniform example (left), σ = 3/4 and σ m = 1, for the Pareto (middle) η = 1 and η m = 1.5, and for the uniform with Laplace (right) σ = 1/4. Second row: The corresponding population version R(x) and empirical version R(x) as function of x for one replication. Bottom row: Histograms of 500 replications of R, rank-R and R∞ . The dashed lines correspond to the threshold of 1.01 for R and rank-R and 1.02 for R∞ .


Matthieu Pizenberg

Abderrahim Elmoataz

Yvain Qu Éau
email: yvain.queau@ensicaen.fr

Image Processing for Cultural Heritage Accessibility: Digitizing the Bayeux Tapestry

This work showcases image processing and computer vision algorithms, in the light of accessibility to cultural heritage artifacts. We emphasize the potential of multimodal 2D image registration and fine-scale 3D-reconstruction techniques, with the aim to ease the work of historians and museum curators, as well as to make artifacts more accessible to the general public or to visually impaired people. This study focuses on the Bayeux Tapestry, a world-famous medieval wool embroidery included in UNESCO's Memory of the World register, and of fundamental importance for both the scientific community and the general public. This exceptional testimony on the society at the eleventh century is both a singular artwork and a historical source on a major event in the history of medieval Europe. Developing state-of-the-art image processing tools for its digitization will ease not only its access, but also its analysis, inspection and reproduction.

Introduction

The Bayeux Tapestry is a unique historical document and a unique artwork. Indeed, it is the only embroidery of this importance that has reached us almost intact, crossing nine and a half centuries while others are known only from historical textbooks. This document of the eleventh century is a 70m long and 50cm high wool embroidery on a linen canvas. Throughout 55 scenes, it tells the epic of William, Duke of Normandy, who left Normandy with his armada in 1066 to conquer the kingdom of England (Figure 1), leading to the famous battle of Hastings where William defeated Harold (Figure 12) who who had seized the throne in his place following the death of old King Edward (Figure 8). This unique artifact of the Middle Ages is a contemporary testimony of the way people used to live in the eleventh century. It is also a historical documentation of events that changed the course of history for two nations (France and England), and for Europe. As such, it is of interest to textual historians, to art historians, to specialists in ancient textiles and architecture, to archaeologists, not to mention the general public. It is the first furniture to have been classified as a historical monument in 1840, and it was included in the UNESCO "Memory of the World" register in 2007. This artifact, which is the property of the French state, is part of the "Musée de France" collection of the city of Bayeux, and it has been exhibited at the Centre Guillaume le Conquérant (Bayeux Tapestry Museum) since 1983.

The Tapestry has been an object of investigation, research and hypothesis for almost three centuries. It was and it is still studied by historians, academics, novelists and journalists. It has been the subject of many questions about its sponsor, its date and place of manufacture, etc. However, researchers and scientists interested in the study of this artifact are confronted with problems related to temporal or geographical constraints i.e., to accessibility:

• The Tapestry is accessible to researchers only one month a year, in January, when the museum is closed to the general public. The rest of the time, research is complicated by the number of visitors (more than 400 000 in 2019).

• Studies rarely consider the artifact in its overall structure. The exceptional size of the document (70m long and 50cm high, see Figure 1a), the profusion of scenes (55), the variety of characters and decorations, the texts in Latin associated with the images: all of this forces specialists to focus on a precise portion of the artwork.

• As a furniture, it raises questions about the stages of the artistic process of its assembly and realization. For example, the possibility of detecting obvious technical changes between different parts of the Tapestry, may indicate that they were embroidered in several workshops. Some of these questions can only be answered by a careful examination of the face and reverse of the whole Tapestry. Yet, since the last digitization in 1983, this reverse side is protected and no researcher or curator can access it.

• There is a lack of a consensual reference system to find one's way around the work. Some authors choose to propose reproductions of the Tapestry in their works and to refer to it by illustrated images. Others try to rely on geolocalization systems more or less known by the readers in order to direct them (the scene numbering present on the canvas being part of the Tapestry).

• The embroidery being extremely fragile, it is protected by a glass (see Figure 1c), preventing its tactile exploration. Therefore, it remains inaccessible to visually impaired persons.

As part of a restructuration project of the Bayeux Tapestry Museum by 2024, a digitization of the face of the tapestry was jointly carried out in 2017 by La Fabrique du Patrimoine en Normandie and by the GREYC laboratory (Normandy University, UNICAEN, ENSICAEN, CNRS). These scans have generated a mass of data under the form of 86 high resolution, multispectral images corresponding to different modalities, as well as 3D point clouds. The objective of this digitization is to circumvent the aforementioned accessibility constraints, by providing open-access 2D and 3D digitizations for different modalities. Along with image processing tools, such data will allow scientists, curators and the general public to visualize the whole embroidery and navigate through it with a very high precision, while discovering elements that were previously inaccessible. One could then observe the artifact on their personal tablet or computer, or connect to a server at the Bayeux museum in order to study it without being subject to geographical and temporal constraints. These tools also have the advantage of increasing the possibilities in terms of cultural mediation and offering interactive ways to involve the public in the discovery process, making them more active in their approach to knowledge. For instance, the construction of a 2D panorama in daylight 1, 2 may already serve as a background image for any indexing or information system, hence solving the geo-referencing problem. Lastly, the high-resolution 3D-reconstruction of the tapestry will allow to both virtually visualize the Tapestry in augmented reality applications, and to realize faithful 3D-printed mockups, which may be touched without damaging the original artwork.

In the present paper, we describe the image processing tools which were developed for facilitating the access to the Tapestry to scientists and to the general public. We first elaborate on 2D imaging technologies allowing to navigate through the artwork in an offline manner, through a large-scale, multimodal panorama and semi-automatic tools for registering historical layers. Then, we demonstrate the potential of 3D computer vision tools for virtually relighting it in surface inspection or augmented reality applications, and for realizing high-detail 3D-printed mockups. A conclusion on this digitization work lastly follows, along with a discussion on accessibility of the softwares developed during this project.

Registering Multimodal Digitizations

The Bayeux Tapestry is extraordinary in many regards, yet one of its specificities is its exceptional size (70m long and 50cm high). Providing a way to the general public to visualize it without direct access is therefore a nontrivial problem. To that aim, a digitization campaign was carried out by La Fabrique de patrimoines en Normandie in January 2017. This digitization had several goals: serve as reference for studying future evolution of the artifact; serve as basis for different research programs; feed a geo-referenced image database; and provide high-resolution images of the artifact for research, cultural, education and communication purposes. The campaign resulted in a series of 86 high resolution images (roughly 8000 × 5500 pixels), in multiple modalities such as daylight, infrared and ultraviolet (fluorescence and reflection). The daylight images enable a first high definition digital coverage of the Tapestry, by opening the way to the constitution of a readily available reference background [START_REF]Official digital representation of the Bayeux Tapestry -XIth century. Credits: City of Bayeux[END_REF] . Yet, other images obtained with different modalities bring complementary information which can help in the restoration or the study of the Tapestry, as previously done for instance in the context of wall paintings studies 3 . For instance,

2/12

the ultraviolet fluorescence images highlight some restoration materials. Other images such as those obtained by ultraviolet reflection, converted into false colors can highlight alterations of the support of the tapestry such as stains. In practice though, capturing the different modalities involves manual operations on the sensor, which induces spatial displacements of up to a few hundreds of pixels. For these reasons, creating a large-scale panorama of the artifact under different registered modalities, as illustrated in Figure 2, would be worthwile yet nontrivial. Two questions thus arise: 1) how to spatially register (i.e., stitch together) the 86 images of each modality, and 2) how to spectrally register (at pixel accuracy) the different modalities. Existing panorama creation softwares fall short in our use case for two reasons. First, an image of the full embroidery length would amount to approximately 680.000 × 5500 pixels, meaning around 3 gigapixels. Such sizes bring technical difficulties due to memory consumption of stitching algorithms and incompatibilities with many image formats (the JPG format, for example, cannot handle more than 65535 pixels width or height). The second reason is that existing panorama softwares do not handle registrations of multiple modalities. Consequently, we developed our own processing pipeline in order to create a multi-modal panorama from the N frames = 86 high resolution (8000 × 5500 pixels) images under N modalities = 6 different modalities. We considered the two following peculiarities of our problem:

• Every image in one modality has approximately the same framing and contents as the corresponding images in the other modalities. Moreover, two corresponding images in two modalities only exhibit rigid deformations that can be modelled by small camera displacements. Indeed, during capture, the embroidery stayed still between the changes of filters on the sensors.

• Two consecutive images in one modality share roughly 30% overlap (see Figure 3, left), and the overlapping region between two consecutive images presents non-rigid deformations. This is because the camera setup was still, while the embroidery was slid on a rail, leading to folds and other non-rigid image deformations.

These observations tend to indicate that we can proceed in a two-steps procedure: 1) achieve spectral registration frame-byframe, so as to register each scene independently in the same coordinates as the daylight modality, which serves as reference; 2) create a panorama in one modality (daylight), and apply the same panorama deformations to the other, already registered, modalities.

Spectral Registration For each frame, the problem consists in registering a series of N modalities = 6 images of different modalities together, where these images have roughly the same frame and the displacement between frames can be assumed rigid. In some sense, these images have been captured under unknown, varying lighting, as it is done for instance in the context of photometric stereo [START_REF] Woodham | Photometric stereo: A reflectance map technique for determining surface orientation from image intensity[END_REF] . Let us denote by N observations the number of pixels of the reference (daylight) frame, by T the sought transformation, and by M ∈ R N observations ×N modalities a matrix stacking all the data. The registration problem can be recast as a rank minimization problem, and then relaxed as a convex optimization problem [START_REF] Pizenberg | Low-rank registration of images captured under unknown, varying lighting[END_REF] :

rank(T (M)) → min, ⇒ Relaxation min A,T A + λ A -T (M) 1 ,
where the latter can be solved efficiently using augmented Lagrangian methods. After this spectral registration step, every coordinate (x,y) in each modality, should now correspond to the same location on the embroidery, as illustrated in the top-left of Figure 4. Panorama Creation Now that all images are aligned with the reference daylight modality, we may compute the full panorama in that reference modality. The algorithm presented here converts the input, overlapping sequence into a new sequence of images which may be smoothly concatenated into a panorama (see Figure 3, right). Each frame is split into roughly overlapping (30% of sides) and non-overlapping areas (central part), and the latter are copy-pasted in the output (cf. the "sand" texture in Figure 3, right). Then, the problem reduces to merging each right-side overlapping area with the left-side of the following scene as seamlessly as possible (cf. the "dots" and "wave" textures in Figure 3, right), which requires computing the deformation mapping from the left image to the right one. Since our scene is well textured, we follow a keypoint detection and matching approach (bottom row in Figure 4). On both images, we compute Akaze keypoints [START_REF] Alcantarilla | Fast explicit diffusion for accelerated features in nonlinear scale spaces[END_REF] , then compute pair matches between them, and filter the pairs using RANSAC [START_REF] Raguram | A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus[END_REF] . Then, we create a new virtual coordinates frame and transform keypoint pairs into keypoint triplets, where the third virtual keypoint is on the line segment between the first and second keypoints with the following expression:

p 3 = (β * p 1 + α * p 2 )/(α + β )
where α is the distance from the left on the first image, and β is the distance from the right on the second image. Finally, we warp both images into this new virtual target image using the moving least squares algorithm (MLS [START_REF] Schaefer | Image deformation using moving least squares[END_REF] ) with respectively the (p 1 , p 3 ) and (p 2 , p 3 ) pairs of keypoints. To avoid seams, the resulting image is a weighted mean of both images, such that it equals the left (resp., right) image on its left (resp., right) side, and blends both images in its central part. Once the panorama fusion has been computed on the reference daylight modality, we can re-apply exactly the same transformations on other modalities, since they have all been previously aligned with daylight images.

For this to work we simply need to save the keypoints and warping paramaters used for the MLS daylight warps. An example of concatenation obtained with this procedure is presented in the top right of Figure 4.

4/12

Visualizing the Panorama Concatenating the whole set of images would lead to a single image of 3 gigapixels. This is not pragmatic, both since very few image formats can hold such an image (one notable exception being TIFF), and since most image viewers choke while loading it. An alternative strategy is to use a zoomable format, and a dedicated viewer able to display such a format. Many options exist, and we chose the combination of the Deepzoom format [START_REF]Deep Zoom[END_REF] and the Openseadragon web visualization tool [START_REF] Openseadragon | an open-source, web-based viewer for zoomable images[END_REF] . Deepzoom is composed of a pyramid of image tiles (typically 512 × 512) where each level of the pyramid halves the image resolution, resulting in roughly four times less tiles than the previous level. All the tools we found to generate the Deepzoom tiles need as input a single image, but at the same time struggle with images as big as the one we can generate for the Bayeux Tapestry. We thus developed a custom made program called Zoomtiler 11 able to take as input our collection of consecutive images and output a hierarchy of image tiles following the Deepzoom format (see Figure 5). Finally, once a Deepzoom image has been created for each modality, we may view them with the Openseadragon Web viewer.

It enables panning and zooming with dynamic loading of the relevant tiles depending on the current zoom level and viewing area, while also switching between the different modalities and preserving the current location. Right: OpenSeadragon Web image viewer used on a small section of the Bayeux Tapestry. Users may navigate through the entire content of the embroidery with their mouse, and also easily switch between the different imaging modalities, as illustrated in Figure 2.

Registering the Back of the Tapestry In the previous paragraphs, we described how to create a large-scale, multimodal digital copy of the Tapestry as it can now be seen when actually visiting the museum. On the other hand, other images of the artwork are not available to the general public, yet of infinite value. One example is the back of the Tapestry, which has never been exposed to the general public, and remains unavailable even to scientists since its last study in 1983. During the previous digitization campaign, the opportunity arised to take pictures of this back. The same panorama generation algorithm could then be applied, allowing the researchers to also navigate through this usually unavailable piece of data. Navigating through this new panorama can be very useful for understanding the creation process of the artwork. Indeed, while the front of the Tapestry (see Figure 6, left) tells the History, its back informs us about the story of its manufacture. As illustrated in Figure6, right, one can for instance visualize the different restorations carried out through the ages.

5/12

Semi-automatic Registration of Historical Layers During the winter of 1982-1983, the Tapestry had been transferred from its former location (hôtel du Doyen) to the current one. On this occasion, a condition assessment of the artwork was carried out and a series of layers of the back side of the artifact was manually created by the curators. They first drew a grid pattern forming a tracking system scaled such that each square of the layer's grid corresponds to a 10 cm by 10 cm square on the Tapestry. Thanks to this tracking system, curators were able to precisely manually draw the back side of the Tapestry. Finally these layers were used to annotate a large number of elements such as restoration pieces, holes, tears, seams, samples, or stains. Those annotations constituted a reference of the state of the artifact, which was intended to serve as basis for the evaluation of future deteriorations. A registration of these layers on the digital reference background will allow a more detailed analysis. Therefore, a specific tool has been developed in order to proceed to its registration on the previously discussed reference panorama. For each layer:

1. The grid is automatically detected using image binarization followed by Hough transform line detection;

2. The image of the layer is projected onto the corresponding scene in the panorama. This operation is performed semiautomatically, the user choosing four noncolinear points on the layer image and their counterparts on the panorama image;

3. The nonrigid transformation between the images is calculated and the two images are aligned;

4. We apply the same transformation to the grid that we have detected.

By iterating this process, we can build a panorama of layers and grids for the whole tapestry, as illustrated in Figure 7. This example shows that, logically, the grids are no longer perfectly rectilinear: it is an illustration of the difficulty of these different image processing operations, due to the very nature of the work, which is moving, and for which there are no perfect and fixed illustrations. Tensions due to the embroidery and its mode of presentation (vertical suspension, posed on a horizontal support) as well as its important length, have always induced deformations that all imaging operations on the Tapestry are subject to. Indeed, no Bayeux Tapestry imaging campaign to date has ever been able to deviate from this rule; to do otherwise would require the Tapestry to be "stabilized" in a room at least 70 m long, with the security conditions that are necessary for the preservation of this major artwork.

Fine-scale 3D Digitization

So far, we have shown how to use image processing algorithms to enable visual inspection of the Tapestry on a multimedia device, so as to guarantee accessibility to the artwork for the general public and scientists. However, these tools only reveal what is "visible", while the geometry of the surface might also be of interest. Indeed, allowing one to touch the Tapestry could allow visually-impaired people to "feel" the wool string and linen canvas composing the artifact, without actually seeing it. Obviously, this is not allowed because it would cause serious damages, hence a glass is placed before the artwork to ensure its protection (see Figure 1c). A workaround consists in creating highly-detailed 3D-printed copies of some scenes of the Tapestry, which could be touched by anyone or used in relighting applications, as illustrated in Figure 8. This, on the other hand, raises a nontrivial challenge: how to reconstruct in three dimensions the very thin geometric details of the surface (wool strings), without damaging it and without a direct access to it (presence of the protective glass) ? From a set of photographies captured under the same viewing angle but varying lighting directions (left), the photometric stereo technique recovers a highly detailed representation of the surface geometry. This geometry can then either be printed in three dimensions to enable tactile explorations (center), or used in augmented/virtual reality applications (right).

Existing Techniques for Surface 3D-reconstruction Digitization of real-world assets in three dimensions is an open problem where choosing a reconstruction approach requires balancing the advantages and drawbacks of each method. The main criteria are the precision of the reconstruction, the subject scale and fragility, the setup cost and difficulty, as well as specific accessibility constraints of the subject. We are especially interested in digitizing the Bayeux Tapestry, which, as previously mentioned, is a thousand year old wool embroidery on a linen canvas. Preservation of this piece of European history is the main constraint, so coordinate measuring machines, with their probe-mounted moving arms, are obviously out of the equation. Other traditional 3D-scanning approaches for small and medium size objects include laser scanning [START_REF] Levoy | The digital Michelangelo project: 3D scanning of large statues[END_REF] , structured light projections [START_REF] Zhang | Microsoft kinect sensor and its effect[END_REF][START_REF] Zaman | Simultaneous capture of the color and topography of paintings using fringe encoded stereo vision[END_REF] , and photogrammetry [START_REF] Chen | Photogrammetry-based 3d printing reproduction method for oil paintings[END_REF] . However, for conservation purposes, we were not authorized to take the embroidery out of its protective glass and exhibition room for our 3D-digitization in January 2021. Therefore, laser and and structured light projections could not be used, as the glass prevents these techniques to perform correctly. On the other hand, photogrammetry had already been used for the 3D-reconstruction of small oil paintings [START_REF] Chen | Photogrammetry-based 3d printing reproduction method for oil paintings[END_REF] . The setup is quite straightforward since one just needs a uniform lighting, a textured subject, and to take many images with a significant overlap. For a bigger subject however, it is highly recommended to create custom hardware such as the dedicated rack designed by Pamart et al. for the digitization of wall paintings [START_REF] Pamart | Multispectral photogrammetric data acquisition and processing for wall paintings studies[END_REF] . The precision obtained by photogrammetry can reach subpixel accuracy for the control points matched in multiple images. For a sufficiently dense sensor, and with images close to the subject, this can result in high resolution 3D scans. In addition, many open-source photogrammetry consumer software exist, such as Alice Vision Meshroom [START_REF] Griwodz | AliceVision Meshroom: An open-source 3D reconstruction pipeline[END_REF] , making the technique easily available for non experts. We experimented with photogrammetric pipelines [START_REF] Lozes | PDE-based graph signal processing for 3-D color point clouds: Opportunities for cultural heritage[END_REF][START_REF] Elmoataz | Un cadre unifiant le traitement d'images et de nuages de points 3D : applications à la tapisserie de Bayeux[END_REF] , and found that photogrammetry was very efficient at recovering the overall structure of the scene's geometry (see Figure 9, yet it was not able to reveal the high-frequency details we are interested in for tactile experiments. Therefore, we turned our attention to another technique which has recently been shown to excel at revealing the finest details of cultural heritage artifacts [START_REF] Huang | Near light correction for image relighting and 3d shape recovery[END_REF][START_REF] Cheveau | 3D Scanning Solution for Textured Object using Photometric Stereo with Multiple Known Light Sources[END_REF][START_REF] Brenner | An investigation of optimal light source setups for photometric stereo reconstruction of historical coins[END_REF][START_REF] Valach | 3D Digitization of Selected Collection Items Using Photometric Stereo[END_REF][START_REF] Vanweddingen | A status quaestionis and future solutions for using multi-light reflectance imaging approaches for preserving cultural heritage artifacts[END_REF][START_REF] Salvant | Photometric stereo by UV-induced fluorescence to detect protrusions on Georgia O'Keeffe's paintings[END_REF][START_REF] Xu | 3d surface measurement and analysis of works of art[END_REF][START_REF] Abzal | Development of an automatic map drawing system for ancient bas-reliefs[END_REF][START_REF] Dulecha | SynthPS: a Benchmark for Evaluation of Photometric Stereo Algorithms for Cultural Heritage Application[END_REF] : photometric stereo.

7/12

Photometric Stereo for Fine-scale 3D-reconstruction of the Tapestry The photometric stereo technique was developed in the late 70s by Woodham [START_REF] Woodham | Photometric stereo: A reflectance map technique for determining surface orientation from image intensity[END_REF] . It is a photometric 3D-reconstruction technique [START_REF] Durou | A comprehensive introduction to photometric 3d-reconstruction[END_REF] i.e., a technique which aims at inverting a physical model describing the image formation process, rather than matching geometric features as in photogrammetry. In photometric stereo, 3D-reconstruction is achieved from a series of photos taken under the same viewing angle, while varying the lighting directions so as to create shading effects on the images. The camera being fixed, the precision of the 3D scan is mostly dependent on the sensor resolution: indeed, this approach provides geometric clues under the form of per-pixel surface normal estimates, which can then be integrated into a depth map [START_REF] Quéau | Normal integration: a survey[END_REF] . The resolution of recent cameras tending to dramatically increase, one may have good hopes to recover structures as thin as the wool strings of the Tapestry. Besides, the setup for photometric stereo is quite straightforward and relatively cheap, as it only requires a remotely triggered camera, a tripod, and small spherical objects used for calibrating the light directions. Moreover, cultural heritage digitization teams are already very familiar with Reflectance Transformation Imaging (RTI) setups [START_REF] Malzbender | Polynomial texture maps[END_REF] , which are similar in spirit and can thus be used for the purpose of photometric stereo [START_REF] Dulecha | SynthPS: a Benchmark for Evaluation of Photometric Stereo Algorithms for Cultural Heritage Application[END_REF] . Another advantage of photometric stereo over photogrammetry is that photometric stereo enables better relighting options for the 3D scan, since it not only estimates the scene's geometry, but also its reflectance i.e., it decouples the intrinsic colour of the subject from the the lighting phenomena, as illustrated in Figure 10. Digitization Pre-study and Protocol Photometric stereo infers geometry from the brightness variations arising in the images as the light direction changes. As such, getting sharp images without noise is key to the reconstruction quality. We thus decided to use a DSLR camera with high light sensibility, which can be remotely triggered and connected to a flash light (Figure 11d). An important point to mention is that a pre-study of the digitization protocol was performed and submitted to the "Direction régionale des affaires culturelles" (DRAC Normandie) to ensure there was no deterioration risk. In that report, we established that the energy brought by our flash light amounts to 6 seconds of the Bayeux Tapestry normal lighting conditions in its exhibition room. Apart from choosing an appropriate camera and light source, it is also extremely important to keep the exact same point of view between images. However, at the scale of a pixel, any sensor vibration due to manual trigger or floor vibrations due to walking around can create misalignments in the sequence of images. A strong and stable tripod, as well as remote triggers are thus very much recommended. In order to calibrate the location of the flash light for each image of a sequence, we placed small reflective spheres at the corner of the images. Since the artifact was kept behind its protective glass, we designed cheap, hand-made calibration spheres, by joining plastic door knobs and bathroom wall suction cups (Figure 11b), enabling easy displacements of the calibration spheres. Once the equipment is gathered, the digitization steps are the following: 1) place the camera on its tripod and adjust the frame; 2) place the reflective spheres on the image corners; 3) focus on the subject; 4) remotely launch a sequence of shots while changing the flash location between each shot. A minimum of 3 shots with non-co-planar light sources is required for photometric stereo but we went with 12 shots to enable robustness to outliers such as cast shadows and caustics induced by the spheres and glass panel (see Figure 12a, left). As illustrated in Figure 11c, eight of the sources were uniformly placed at en elevation angle of roughly 50 • (the optimal value for the digitization of quasi-planar artifacts [START_REF] Brenner | An investigation of optimal light source setups for photometric stereo reconstruction of historical coins[END_REF] ), while four additional shots were taken with a smaller elevation angle to increase the number of outlier-free images.

8/12

(a) Digitization of the scene 55 (see also Figure 10). The camera is fixed on a tripod, and the flash is held on the side, directed toward the captured scene. Algorithm for 3D-reconstruction Our goal is to turn a sequence of photometric stereo images into a highly detailed 3Dmodel, as depicted in Figure 12. To this end, we first estimate the incident lighting directions using the reflective spheres. The flash source being non-directional, each sphere provides a different estimate: we simply average them to have a reliable estimate at the center of the scene. Then, we simultaneously compensate the slight sensor's displacements and remove the outliers, by using a low-rank approximation procedure [START_REF] Pizenberg | Low-rank registration of images captured under unknown, varying lighting[END_REF] . After this preprocessing, we iteratively estimate the reflectance, the normals and each source intensity, by semi-calibrated photometric stereo [START_REF] Cho | Semi-calibrated photometric stereo[END_REF] . This provides us with a normal field which is integrated into a depth map using discrete cosine transform [START_REF] Quéau | Normal integration: a survey[END_REF] . Since we wrongly assumed directional lighting for simplicity, the depth map exhibits a well-known low-frequency bias ("potato cheap" effect [START_REF] Huang | Near light correction for image relighting and 3d shape recovery[END_REF] ). We found that this bias could be removed simply by fitting and substracting a low-dimension polynomial to the depth map, which is simpler to implement than, e.g., resorting to photogrammetry [START_REF] Karami | Investigating 3d Reconstruction Of Non-Collaborative Surfaces Through Photogrammetry And Photometric Stereo[END_REF] . Next, we recompute the surface normals using finite difference and integrate them again using discrete sine transform [START_REF] Quéau | Normal integration: a survey[END_REF] to force constant values on the boundaries in order to ease 3D-printing. Lastly, the depth map is meshed and a small support is added, resulting in a printable 3D-volume (see Figure 8b). This pipeline is summarized in Figure 13. . From a series of pictures taken under varying lighting (one of these images in shown in Figure 1b), a reflectance map and a normal map are computed using photometric stereo. The latter is then integrated into a depth map, which can further be meshed and 3D-printed, in view of tactile experiments for visually-impaired people.

Conclusion

In this article, we described a few image processing techniques which could be used to reinforce the accessibility to a unique cultural heritage artifact, namely the Bayeux Tapestry. We first introduced a 2D multimodal registration technique which enables the virtual visit of the artwork on a multimedia device, for the general public or for scientists interested in the study of the manufacturing process. Then, we elaborated on 3D-reconstruction techniques permitting to create 3D-models which may be used in relighting applications, or 3D-printed in view of tactile explorations by visually-impaired people. Future work will explore the possibility to bring such 3D-models onto force-feedback tablets [START_REF] Gay | A Novel Force-Feedback Haptic Architecture Delivering 2D Data to Visually Impaired People[END_REF] , so as to enable tactile experiments without the need for an actual 3D-printed mockup.

Beside the accessibility to the public and to scientists interested in this exceptional artifact, we are genuinely convinced that pushing for accessibility and reproducibility of research is important. Therefore, throughout our work, we have focused on the accessibility of the results and the reproducibility of the methods. For the multimodal panorama, we carefully described the algorithms, released code for some of the steps [START_REF] Pizenberg | a simple CLI program to generate zoomable tiled images[END_REF][START_REF] Pizenberg | Rust implementation of image deformation using moving least squares[END_REF] , and are planing to release the full pipeline with even more details provided in its documentation. Similarly, for the 3D digitization pipeline, we are releasing a suite of web applications as well as their code to the public. The source code will enable further collaboration, and the web applications will provide re-usability for the broader cultural heritage community, less familiar with programming. Currently we are making two web applications available [START_REF] Pizenberg | Light calibration from reflective spheres[END_REF][START_REF] Pizenberg | Calibrated quasi-planar photometric stereo in Rust and on the Web[END_REF] . The first aims at performing light calibration of a sequence of photometric stereo shots by identifying bright spots on the reflective spheres and computing the light direction for each shot. The second aims at performing the 3D reconstruction when provided with a photometric stereo image sequence and its light calibration parameters, obtained with the first application. We hope that in the future, such tools will help others digitizing other unique cultural heritage artifacts.

( a )

 a Overview of the first half of the Tapestry, as it is exhibited in the Bayeux Tapestry Museum. Credits: Bayeux Tapestry Museum, City of Bayeux. (b) William, Duke of Normandy, as depicted in scene 23. (c) Close-up on William's armada crossing the Channel to conquer the kingdom of England (scene 37). Credits: Bayeux Tapestry Museum, City of Bayeux.

Figure 1 .

 1 Figure 1. The Bayeux Tapestry in its exhibition hall.

Figure 2 .

 2 Figure 2. Multi-modal panorama of the beginning of the Tapestry. Five different imaging modalities are represented here with different colour codes. They are both spectrally registered at pixel accuracy, and spatially registered into a large-scale panorama which can be explored in a web interface.

3 / 12 Figure 3 .

 3123 Figure 3. Left: Diagram representing two consecutive overlapping scenes, imaged under two modalities picturing the same contents with a small rigid deformation. The modalities need first to be registered (spectral registration), then the overlapping frames may be stitched into a panorama (spatial registration). Right: sketch of the panorama procedure. The input overlapping sequence is turned into a new sequence of non-overlapping images which may be concatenated into a panorama.

Figure 4 .

 4 Figure 4. Top, left: four overlapping images, of size 8000 × 5500 pixels, of the beginning of the Tapestry under daylight, and their registered fluorescence UV counterparts (revealing the restored parts). Top, right: reconstructed panoramas for the two modalities. Bottom: detected keypoints (circles) in the first and second daylight images, and keypoint matching (lines).

Figure 5 .

 5 Figure 5. Left: Zoomtiler, a simple CLI program to generate zoomable tiled images from a horizontal succession of images.Right: OpenSeadragon Web image viewer used on a small section of the Bayeux Tapestry. Users may navigate through the entire content of the embroidery with their mouse, and also easily switch between the different imaging modalities, as illustrated in Figure2.

Figure 6 .

 6 Figure 6. Reconstructed panoramas of the front (left) and back (right) sides of the Tapestry under daylight, visualized at different scales. Navigating through the front of the artifact allows one to follow the epic of William; on the flip side, navigating through its back allows one to follow the different stages of the manufacturing process, for instance by exploring parts which have been restored.

Figure 7 .

 7 Figure 7. From left to right: one of the digitized layers from the 1982-1983 campaign; overview of the registration program, where the detected grid and matching points are highlighted; registration of the layer onto the panorama.

  Two images of King Edward (scene 1), captured under varying lighting. Reflective spheres are stick onto the protection glass to calibrate the lighting directions. (b) Printable 3D-model computed using photometric stereo. (c) Relighting of the 3D-model, applying a gold reflectance to the surface.

Figure 8 .

 8 Figure 8. From a set of photographies captured under the same viewing angle but varying lighting directions (left), the photometric stereo technique recovers a highly detailed representation of the surface geometry. This geometry can then either be printed in three dimensions to enable tactile explorations (center), or used in augmented/virtual reality applications (right).

( a )

 a Dense point cloud representing the whole photogrammetric 3D-reconstruction. (b) Close-up on a small part of the photogrammetric 3D-reconstruction.

Figure 9 .

 9 Figure 9. The photogrammetric 3D-reconstruction of the Tapestry allows to virtually navigate through the entire Tapestry as if visiting the Museum, yet this 3D-reconstruction still lacks the finest geometric details.

Figure 10 .

 10 Figure 10. Left: 3 (out of 12) images of William revealing his face (scene 55). Right: using photometric stereo in the area highlighted in red, the input image sequence (one image of the sequence is recalled on the left) can be decomposed between reflectance (top, right) and normals (shown in false colors on bottom, right). Reflectance encodes the scene's color, independently from shape and illumination: see how the folds on the linen canvas have been eliminated. Normals, on the other hand, encode the scene's micro-geometry through the surface's orientation, which reveals thin details such as wool strings.

( b )

 b Sphere stick on the glass panel.(c) Distribution of flash locations on a hemisphere around the subject.(d) Equipment for the 3D scan: a DSLR and a flash connected via a wireless radio command.

Figure 11 .

 11 Figure 11. Equipment and protocol for the 3D-digitization of the Bayeux Tapestry by photometric stereo.

( a )

 a Two images from the death of Harold sequence (scene 57). Note the shadows and caustics induced by the spheres and the glass.(b) 3D-reconstruction obtained using photometric stereo, superimposed on one of the input images.

Figure 12 .

 12 Figure 12. Highly detailed 3D-reconstruction using photometric stereo.

Figure 13 .

 13 Figure13. Summary of the photometric stereo pipeline for creating 3D-printed mockups of the Tapestry, illustrated on the Duke William sequence (scene 27). From a series of pictures taken under varying lighting (one of these images in shown in Figure1b), a reflectance map and a normal map are computed using photometric stereo. The latter is then integrated into a depth map, which can further be meshed and 3D-printed, in view of tactile experiments for visually-impaired people.

Acknowledgements

This work was funded by the RIN grant Guide Muséal, and the ANR grant Inclusive Museum Guide (ANR-20-CE38-0007). A.E is partially supported by ANR Grant SUMUM (ANR-17-CE38-0004). Multimodal images were acquired by La Fabrique de patrimoines en Normandie. Access to the Tapestry was made possible by the Bayeux Tapestry Museum and the City of Bayeux, under the supervision of Clémentine Berthelot.

Data availability

The source codes of the softwares which were developed to produce the results on multimodal panorama creation and visualization, as well as fine-scale 3D reconstruction, are available at https://github.com/mpizenberg. Datasets are available from the corresponding author on reasonable request.

Author contributions statement

All authors contributed to the preparation, writing and revisions of the manuscript.

Additional information

The authors declare no competing interests.

12/12


Patrice Guyot

Thierry Malon

Geoffrey Roman-Jimenez

Sylvie Chambon

Vincent Charvillat

Alain Crouzil

André Péninou

Julien Pinquier

Florence Sèdes

Christine Sénac

Audiovisual annotation procedure for multi-view field recordings

Keywords: Audiovisual, annotation, multi-view, multi-modal, field recording, multimedia, ground truth

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Production of audiovisual documents is a fast-growing phenomenon which is founded on an increasing number of recording devices, for instance smartphones. In comparison to the data conceived in a controlled domain (e.g. TV, radio, music studio, motion capture studio, etc.), many recordings are generally produced in an uncontrolled context. They will be further referred to as field recordings.

Moreover, different audiovisual documents may correspond to the same scene, for instance a public event that is filmed by different points of view. These multi-view scenes contain lots of information and provide new opportunities for high-level automatic queries.

In the context of automatic analysis, the aim of the different tasks (e.g. detection, classification) is to reduce the quantity of information embedded in audiovisual documents towards some particular semantic concept. For example, a video with a car in the foreground contains lots of information (type of car, ground, objects in the background, weather, localization, etc.) that could be reduced to the concepts car or nice weather.

In order to produce a model and to evaluate the performances of the algorithms on a set of data, researchers generally build a manual annotation that expresses this semantic information. The result of such manual annotation is generally called ground truth. As it usually refers to information provided by direct observation, it requires researchers to develop objective criteria. The ground truth depends on the definition of a space in which the data are projected in the most appropriate manner within a specific context. This task is not always straightforward: for instance, in the context of Music Information Retrieval, the evaluation of musical artist similarity requires the development of an objective measurement, meanwhile artist similarity relies on an elusive concept [START_REF] Ellis | The quest for ground truth in musical artist similarity[END_REF]. Thus, it appears that the term ground truth is sometimes misleading because it does not reflect an objective truth [START_REF] Aroyo | Truth is a lie: Crowd truth and the seven myths of human annotation[END_REF]. In that respect, we will use the term reference which seems more accurate to designate the manual annotations.

Audiovisual documents are in essence based on two modalities: audio and video. Yet in the context of audiovisual documents, the annotations are generally mono-modal (audio or video), while the perception of an audiovisual content is multi-modal and thus leads to a richer interpretation. Moreover, the different modalities influence each other, making the mono-modality annotation difficult in a multi-modality context.

This paper addresses the issue of producing multi-modal annotations in an audiovisual context. We propose a low-cost procedure to manually annotate multi-view field recordings. This paper is organized as follows. We first present the relative works about annotation and perception of audiovisual contents. Section 3 presents a specific procedure to solve the multi-modal issues of audiovisual annotations. This procedure is usable in mono or multi-view contexts. Finally, different applications of this procedure are described in Section 4.

Related works

Audio and video ground truth: from precise to weak annotations

The challenge of multimedia modeling, developed intensively during the 2000s, has produced multiple campaigns for information retrieval, for instance with video [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] or audio events [START_REF] Mesaros | Dcase 2017 challenge setup: Tasks, datasets and baseline system[END_REF]. In this framework, vast amount of data have been manually annotated. These annotations are usually precise and time consuming. For example, audio events, such as speaker turns in case of speaker diarisation, or music notes in the case of Music Information Retrieval, are usually annotated at a millisecond scale [START_REF] Bird | A formal framework for linguistic annotation[END_REF]. Moreover, as these annotations are hardly objective, an agreement between annotators is usually needed [START_REF] Turnbull | Semantic annotation and retrieval of music and sound effects[END_REF]. For these different tasks of annotation, different softwares have been proposed (see [START_REF] Rohlfing | Comparison of multimodal annotation tools-workshop report[END_REF] for a comparison).

Lately, Deep Learning based approaches [START_REF] Lecun | Deep learning[END_REF] outperform the state of the art in many domains. However, they require a large amount of data. Because a pre-cise annotation of these data is almost impossible, recent datasets include only weak annotations. These weak annotations really differ from a precisely annotated ground truth, as they may be incomplete, not relevant and heterogeneous. For example, the Audioset dataset [START_REF] Gemmeke | Audio set: An ontology and human-labeled dataset for audio events[END_REF] provides a large set of audio data extracted from videos, but the annotations were tagged by YouTube users on the audiovisual content. In the area of vision, the AVA dataset [START_REF] Gu | AVA: A video dataset of spatio-temporally localized atomic visual actions[END_REF] provides precise spatio-temporal annotations of persons conducting actions, but the sound of the video is not taken into account.

Finally, most of the research works are usually mono-modal based (only audio or video stream). The issue of merging audio and visual information to richer concepts is rarely addressed by the different scientific communities. In that scope, the softwares used for manual annotations seem to deal with multi-modality as a juxtaposition of mono-modal annotations.

Multi-modal and multi-view

The modeling of multi-modal (or cross-modal) inputs is very challenging, for example when studying discourses containing speech and non-linguistic signs [START_REF] Iedema | Multimodality, resemiotization: Extending the analysis of discourse as multi-semiotic practice[END_REF]. Some applications rely on a precise interaction between image and sound. For instance, the detection of talking heads has been addressed [START_REF] Li | Multimedia content processing through cross-modal association[END_REF]. In this context, various works deal with the fusion of audio and video modalities, for example with early, intermediate or late fusion [START_REF] Pinquier | Strategies for multiple feature fusion with hierarchical hmm: application to activity recognition from wearable audiovisual sensors[END_REF]. Besides, other applications deal with other modalities, for instance image and texts [START_REF] Pereira | On the role of correlation and abstraction in cross-modal multimedia retrieval[END_REF].

The issue of annotating a multi-modal dataset in the case of audiovisual content is clearly addressed in [START_REF] Lefter | Addressing multimodality in overt aggression detection[END_REF]. Whereas this study aims at automatically detecting overt aggression in public places, the authors state that "problems with automatically processing multi-modal data start already from the annotation level." The complexity of the interactions between modalities forced the authors to produce three different types of annotations: audio, video, and multimodal. The combination of these three annotations increase the performances of an automatic detector based on a machine learning approach. However, the processing of these annotations is time-consuming and sensitive. Firstly, this procedure necessitates at least three different kinds of playbacks (audio, video and audiovisual) to perform the annotation. Secondly, in order to process independent annotations with limited influence among modalities, three different annotators at least are required.

Furthermore, increasing amounts of scenes are filmed simultaneously from different points of view. In particular, in the context of video surveillance, different cameras are usually used [START_REF] Wang | Intelligent multi-camera video surveillance: A review[END_REF]. The framework of Motion Capture also provides interesting databases that include different views [START_REF] Ofli | Berkeley mhad: A comprehensive multimodal human action database[END_REF]. The reflective markers placed on a human body allow the recording of the absolute position of each part of the body, which can be directly used as a reference. However, these applications usually remain in the field of laboratory studies and are hard to deploy in a real-life context.

The context of field recordings is generally more challenging [START_REF] Auer | Automatic annotation of media field recordings[END_REF] due to the number of overlapping events and objects. Considering audio, many events over-lap and produce a mixture. Moreover, the movement of the audio sources (for instance a passing car) makes it difficult to position the starting and ending boundaries of the events. Same kind of difficulties arise considering images, with occlusion, superposition, illumination and size of objects.

Different works review datasets from the perspective of multi-modal and multi-view features [START_REF] Ofli | Berkeley mhad: A comprehensive multimodal human action database[END_REF][START_REF] Liu | Benchmarking a multimodal and multiview and interactive dataset for human action recognition[END_REF][START_REF] Malon | Toulouse Campus Surveillance Dataset: scenarios, soundtracks, synchronized videos with overlapping and disjoint views (regular paper)[END_REF]. However, as observed in [START_REF] Malon | Toulouse Campus Surveillance Dataset: scenarios, soundtracks, synchronized videos with overlapping and disjoint views (regular paper)[END_REF], these datasets are often limited by different criteria including presence of audio, realism for real life applications, and number of overlapping and disjoint views.

Audio-vision

The relationship between sound and image has been investigated for a long time, in particular in the context of cinema. A reference book [START_REF] Chion | Audio-vision: sound on screen[END_REF] details the different possibilities of using sounds in videos.

Focusing on the area where the action takes place, the first distinction has to be drawn between sounds of the scene that could be heard by the film's characters, and sounds that could not. The first category is called diegetic sounds. The second category consists of non-diegetic sounds that are added in a postproduction step, for example in the case of voice-over. More precisely, the diegetic sound source can be on-screen or off-screen. Furthermore, the source of the sound can be at times visualized in the image. Otherwise, if the sound source is not visible, the sound is called acousmatic. Figure 1 summarizes these different interactions.

off-screen non-diegetic on-screen In a multi-view sequence, the source of a sound may be either visualized or off-screen according to the different points of view. If the source of a sound is ambiguous or not visible in the current video, a different viewpoint may disclose it. We speak about causal identification when the source of a sound can be identified, whether it is visible or not in the current viewpoint.

V I S U A LI ZED Z O N E A C O U S M ATIC Z O N E S
As these different types of interaction are precisely depicted in a movie script, they are quite unusual in research papers. To our knowledge, research datasets do not provide information about on-screen and off-screen sounds. However, it seems that these different types of interaction have an influence on the perception and the understanding of the audiovisual content.

Finally, the audio and video parts interact in different ways to create an audiovisual perception. One of the clearest examples of the influence between audio and video lies in the McGurk effect [START_REF] Mcgurk | Hearing lips and seeing voices[END_REF], that demonstrates an interaction between hearing and vision in speech perception. Its most known implementation consists of a video of a human face saying a pseudo-word (ga-ga) with a voice over saying another one (ba-ba), leading to the perception of a third one (da-da). When annotating, this kind of phenomenon could occur in the same way and would lead to three different annotations (audio, video and audiovisual).

3 Audio/visual annotation

Problematic

Audio and video annotations are usually based on different paradigms, but are both based on predefined categories to annotate, such as car or speech. Audio annotations usually consist in determining the start and the end of audio events and tag each event with a category (engine noise, speech, horn sound, etc.). Considering video, a usual procedure of annotation is to set a bounding box on each object of interest in each frame of the video and tag each object with some categories (car, person, clothes, etc.).

Procedures of annotation usually consider the audio and video streams as if they were disconnected and each media is annotated separately. In that process, a valuable information may be lost. In this article, we argue that the whole information embedded in an audiovisual content is greater than the sum of its audio and video parts. For example, if we separately annotate speech events (audio only) and person objects (video only), we cannot deduce if a visible person is the speaker or not.

In that context, some issues are clearly observable with the Audioset dataset (see Section 2.1). Most of the tags seem to have been set according to the video part, which usually dominates the audiovisual content. As a consequence, a video of a cat annotated as cat will also be annotated cat in the audio annotation, even if the cat remains silent in the video. To address these issues, we intend to merge the audio and video modalities into audiovisual objects. Practically, we aim to create an audiovisual object based on a moving bounding box and a corresponding audio event. Surprisingly, this task proved to be very difficult and many issues appeared and are detailed below.

A first challenge is about matching and merging one visual object and one audio event. First of all, we have considered a systematic fusion of events from the two modalities considering that this fusion could match segments from audio and video streams in the case of temporal overlapping. Unfortunately, this matching may introduce some wrong annotations when the audio annotation corresponds to an off-screen source. For example, Figure 2 shows a car at the foreground. At the same moment the sound track is overpowered by an off-screen motorbike. A second challenge lies in the case of defining several annotations with temporal overlapping. The context of field recordings induces an audio mixture. Depending on his expertise, a human annotator may not be able to set precisely the starting and ending boundaries of the different audio events of this mixture. In this case, the matching between a specific audio event and the potential corresponding visual object can be impossible. In the same way, considering visual annotations, when annotating a group of objects, the annotator might be unable to draw bounding boxes around each element. Depending on the scale of the image or the mixing of objects in the image, the annotator may annotate each element separately, the entire group as a single object, or a mixture of single elements and rest of the group.

In this context, the issue consists of matching several audio and visual annotations with ensuring their relevancy. When many visual objects may have produced some sound events, the separation of the sound sources in the audio signal may be impossible. Let us consider the Figure 3 that represents audio segments (time boundaries) and video annotations. In that scene, the passing of two consecutive vehicles have been auditory annotated as a single audio event. The solution for creating an audiovisual object from these annotations relies on the segmentation of the audio event in two parts to create two audiovisual objects. We have tested many possibilities to obtain the boundaries of the audio events but none of them was satisfying whatever the situation.

In a more sophisticated way, we could directly build audio, video and audiovisual annotations from the audiovisual stream. However, the completion of this task is not straightforward. Indeed, the influence of the audiovisual content may influence the annotation of the mono-modal streams. For instance, an annotator would more likely create an audio event for a moving car than for a stopped one, even if they both produce a motor noise. ?

Engine noise Fig. 3. Audiovisual annotation of the passing of two cars. In the audio modality, the passage of cars is heard as a unique lengthy sound. On the contrary, the video annotation clearly exhibits two different vehicles. Consequently, the automatic fusion of these two modalities to create audiovisual object(s) is very difficult to define.

Procedure of annotation

We present here a procedure to obtain audio and visual annotations, as well as audiovisual information. It aims at satisfying the following goals:

-Audiovisual added value: the annotations must embed multi-modal information that allows a better understanding of the scene and an added value in comparison of the whole set of mono-modal annotations. -Mono-modal used: the audio and visual annotations must be usable in a mono-modal context. Therefore, additional information from other modality are not to be considered when creating mono-modal annotations. -Low additional cost: the audiovisual annotation must be objective and straightforward, and must not generate a heavy additional cost.

To address these different constraints, we propose the following two-steps protocol, which is designed to be processed manually.

Step 1: mono-modal annotations. In this step, the audio and video annotations are processed separately. Optimally, the annotations have to be processed by different persons, without access to the other modality. For example, the annotator of the audio stream works only with audio. These two annotations can be processed in parallel.

For each modality, a unique identifier is set for each object in the scenes. The objects visible at different moments of the video (or on different videos in a multi-view context) must bear the same identifier. Similarly, the same identifier is set for each annotation of the same audio event in the case of a clearly unique event, for example a big explosion recorded by various devices.

At the end, an audio annotation contains description of audio events that are made up of time boundaries, categories and identifier. Visual annotations allow the description of objects on the basis of time, spatial coordinates of bounding boxes, categories, and identifier.

Step 2: multi-modal links. In a second step, audio and visual modalities are linked with each other. Links between audio and video identifiers are created in case of causal identification (see Section 2.3). In a multi-view case, an audio event could be associated with an off-screen object that is visible on another view. This process is detailed in next section.

In this step, the mono-modal annotations (audio or video) cannot be modified regarding the other modality, even if they appear to be wrong in the multi-modal context (see the McGurk effect in Section 2.3). These annotations were valid from a mono-modal annotation point of view and remain as they stand.

Implementation of multi-modal links

Considering the mono-modal annotations, we focus on audio and video annotations that temporally overlap. These annotations may refer to the same audiovisual document, or to different documents in the multi-view case.

Each of the audio annotations is considered in terms of sound source. If a causal identification is possible (see Section 2.3), we link audio to video annotations. A link means that audio annotations are enriched with the list of the linked visual objects considered as the source of the audio event. When an audio annotation is linked to several visual objects, the sources of the audio event can be all of the objects or some of them indifferently.

Table 1 summarizes the different annotation links between audio and video. Note that we only link audio event to video object (not video object to audio event) because of the unbalanced relationship between audio and video.

We detail below some concrete examples of links between audio and video annotations. In these examples, we focus on vehicles. However, as our procedure is generic, it can be applied on different kinds of events and objects.

Table 1. Annotation link procedure depending on the presence of audio and video annotations and the possibility of causal identification. A corresponds to an annotation of a single audio event (e.g. engine noise, speech, etc.). V corresponds to an annotation of a single visual object (e.g. car, person, etc.). {Ai} corresponds to a set of audio annotations. {Vj} corresponds to a set of video annotations. A link between annotations is denoted by → .

(1) (2)

(3) (4)

link - --A → V -A → {Vj} -{Ai → V } -{Ai → {Vj}}
Passing vehicle: the audio and video events are linked if they undoubtedly originate from the same vehicle. If any doubt exists, for instance if the source of the audio event could be another vehicle that is not visible, the events are not linked (see Table 1 column 3 and Figure 2).

Slammed door: if a sound event occurs from the interaction of several visually annotated objects, we link the audio event to the each visual objects (see Table 1 column 4). For instance, in case of the closure of a car door with annotations for two objects (car and person), the audio event slammed door is linked to each of the two objects.

Passing vehicle and horn: in the case of multiple audio events that obviously originate from the same visual object, we link all audio events to the object (see Table 1 column 5). Thus, if an object car has been annotated visually and two audio events engine and horn are produced by the car, then the two audio events are linked to the visual object.

Passing of multiple vehicles: in the case of multiple vehicles passing with a different number of audio events (see Figure 3), we link the audio events to all visual objects (see Table 1 column 6). However, if the audio source is not obvious (for instance a car horn when different vehicles are present), we do not link the audio event to any visual object.

Applications

We present hereafter different applications that are made possible by our procedure of annotation. In the case of mono-modal request, the annotated corpus can be used for different purposes. The audio annotations can be used in audio detection tasks (see [START_REF] Mesaros | Dcase 2017 challenge setup: Tasks, datasets and baseline system[END_REF] for examples). Similarly, the video annotations provide a framework for objects detection (see [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] for example). Using the bounding boxes drawn on each object, object re-identification based on image appearance can be driven. In a context of surveillance with a network of recording devices (cameras recording video, microphones recording audio, smart-phones recording both video and audio...), our annotations allow users to perform different kinds of requests. Figure 4 illustrates this application in the context of the ToCaDa dataset [START_REF] Malon | Toulouse Campus Surveillance Dataset: scenarios, soundtracks, synchronized videos with overlapping and disjoint views (regular paper)[END_REF]. Several devices are set around a scene: devices 3 and 7 record both audio and video, whereas devices 5 and 14 only record video stream. Finally, microphone 28 only records audio. From an audiovisual document, we may perform queries that can be either video only (for example by clicking the bounding box containing the vehicle on the video from camera 14) or audio only (for example by clicking on the represented audio event from the same video) in order to retrieve the object ID. All the audio events and video objects associated to the same ID are returned as results. These results can either be audio, visual, or audiovisual.

In a more complex application, this framework also allows multi-modal queries that aim to retrieve audiovisual objects, for example a vehicle with distinct sound and appearance.

Conclusion

In this paper, we propose a simple procedure to produce audiovisual annotations in different contexts such as multi-view dataset. Our approach aims to produce audio, visual, and audiovisual information. It is based on separate annotations on the audio and video modalities, followed by an audiovisual matching. In this way, an audiovisual annotation is produced, as well as audio and video annotations that remain relevant in a mono-modal context.

This procedure is simple. With respect to mono-modal annotations, our method does not extend the time of processing significantly. It can be deployed at a large scale, but, unlike weak annotations, maximizes the relevance of the annotation. Moreover, in the context of multi-view annotations, the required uniqueness of annotation identifiers allows for creating possibly relevant annotations not only with on-screen objects but also with off-screen objects. Finally, the resulting annotations produce a valuable approximation of what should be a ground truth.

Fig. 1 .

 1 Fig.1. The audiovisual scene (adapted from[START_REF] Chion | Audio-vision: sound on screen[END_REF]).

Fig. 2 .

 2 Fig. 2. Image bounding boxes around the cars. If a passing car is clearly visible at the foreground, a motorbike behind the camera overpowers the corresponding soundtrack.

Fig. 4 .

 4 Fig.4. Within a network of recording devices, our multi-modal annotation procedure allows to retrieve either visual objects only (camera 5), multi-modal objects (camera 7), or audio events only (camera 3, microphone 28) from audio or video queries. Note that camera 3 records audio and video, but the audible object is off-screen.