Sz Borsanyi

, , , , J. N Z Fodor

C Guenther

S D Hoelbling

L Katz

T Lellouch

K Lippert

Miura

L Parato

K K Szabo

F Stokes

B C Toth

Cs Torok

L Varnhorst
email: varnhorst@uni-wuppertal.de

High precision scale setting on the lattice

Recently, the hadronic vacuum polarization contribution to the anomalous magnetic moment of the muon was determined by the BMW collaboration with sub-percent precision. Such a precision requires to control many sources of uncertainty. One of these is the uncertainty in the determination of the lattice spacing. In this talk, we present the scale setting entering this computation. It relies on the mass of the Ω baryon as input which is directly used to set the scale of our main calculation. It also allows us to calculate the value of the intermediate scale setting quantity 0 . Here, we present our calculation of this quantity with a relative precision of about 0.4%.

Introduction

This work is part of the lattice determination of the anomalous magnetic moment of the muon by the BMW collaboration that was published in [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF]. To achieve the precision necessary for a meaningful comparison with experiments, many sources of errors had to be controlled. One such source is the determination of the lattice scale which is described in this contribution. For an overview of the whole calculation, see also [START_REF] Toth | Muon g-2: BMW calculation of the hadronic vacuum polarization contribution[END_REF]. For our main analysis of , we set the scale directly with the mass of the omega baryon. To keep this contribution as self contained as possible, however, the determination of 0 [START_REF] Borsanyi | High-precision scale setting in lattice QCD[END_REF] using the omega mass is explained. Besides being a useful quantity by itself, it was also used in [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF] to determine the individual isospin splitting contributions to .

In the first section, the determination of the omega mass on individual ensembles of gauge configurations is discussed, both for the isospin symmetric, pure QCD part as well as for the strong isospin breaking and QED corrections. After that, in a second section, our global fits are briefly explained with a special emphasis on the determination of 0 .

Mass of the omega baryon

Any precise determination of the lattice scale requires a precisely known input quantity. In this work, we choose the mass of the omega baryon. It can be measured both in experiment [START_REF]Review of Particle Physics[END_REF] and on the lattice with sufficient precision.

Our calculation is based on a set of 22 ensembles of gauge configurations generated with a Symmanzik improved gauge action. The 2 + 1 + 1 flavors of the dynamical fermions are realized by a staggered action constructed from four times stouts smeared gauge fields. On top of these QCD configuration QED configuration where generated. For details of the action and the methods used to generated the configurations, see [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF]. The configurations where generated at six values of the gauge coupling and the lattice spacing ranges from 0.132 fm to 0.064 fm. The quark masses where chosen such that the ensembles scatter closely around the physical point to allow for a reliable interpolation to the physical point.

In the first part of this section, the accurate lattice determination of the omega mass in pure QCD is discussed. In the second part, the calculation of the QED corrections, which are important at the target accuracy, is discussed.

QCD contributions

The mass of the omega baryon can be calculated by studying the exponential falloff of the correlation function of two suitably chosen temporally separated interpolating operators. In the staggered formulation, translations by one lattice spacing are intertwined with transformations among the spinor degrees of freedoms of the quark fields. Hence, the construction of interpolating operators that couple to specific hadrons is more complicated than in the case of e.g. Wilson fermions. For mesons, constructions of such operators are discussed in [START_REF] Golterman | Staggered mesons[END_REF]. For the construction of baryon operators, see [START_REF] Golterman | Lattice Baryons With Staggered Fermions[END_REF]. Amongst these baryon operators, the ones labeled as VI and XI couple to the omega baryon as the lowest state. The operators can be constructed from a staggered quark

( ) = ì , = even ( 1 ) ( 12 ) ( 13 ) -( 2 ) ( 21 ) ( 23 ) +( 3 ) ( 31 ) ( 32 ) , ( 1a
) VI ( ) = ì , = even ( 1 ) ( 2 ) ( 3 ) (1b)
where

( ) = 1 2 -ˆ + + ˆ and = . (2)
The omega baryon, however, comes in multiple staggered tastes which become degenerate only in the continuum limit. The operators shown above couple to more then one of these tastes. In [START_REF] Bailey | Staggered baryon operators with flavor SU(3) quantum numbers[END_REF] an additional flavour degree of freedom is used to construct an operator which solely couples to one taste of the omega baryon. This operator reads

Ba = 2 1 2 3 -3 1 2 -2 3 1 + (• • • ↔ • • • ) × ( 1 ( )) ( 12 ( )) ( 13 ( )) -( 2 ( )) ( 21 ( )) ( 23 ( )) + ( 3 ( )) ( 31 ( )) ( 23 ( )
where the Greek indices refer to the additional flavour degree of freedom. We applied Wuppertal smearing [START_REF] Gusken | Nonsinglet Axial Vector Couplings of the Baryon Octet in Lattice QCD[END_REF], but with a kernel that includes only two-hop terms, to all three operators. For the gauge field appearing in this kernel we applied multiple steps of stout smearing [START_REF] Morningstar | Analytic smearing of SU(3) link variables in lattice QCD[END_REF] in three dimensions. For details, see [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF].

We have compared the performance of the three operators mentioned above on the coarsest ensemble for which we generated about 3000 extra configuration that where used only to compare the three operators with an increased precision. In figure 1 the extracted mass is shown for all three operators. As it can be seen, even on this ensemble the three values agree within errors. Furthermore, the difference between the largest and the smallest value is of the same size than the typical statistical and systematic error on the masses extracted on other ensembles. We conclude that it is therefore justified to use the VI operator to extract the mass of the omega baryon.

Due to the presence of excited states in the correlation functions of the interpolating operators, a simple one-state ansatz does not allow for a controlled extraction of the ground state mass.

Consequently, excited states must be included in the analysis. For this purpose, we employed two procedures: The first one was a fit with a ansatz containing four states. In the second one we used a generalized eigenvalue problem (GEVP) based on a matrix constructed out of time shifted correlation functions [START_REF] Aubin | A new approach for Delta factors[END_REF].

In the first approach, we considered the first four lowest energy states and fitted the resulting ansatz to the correlation function. Neglecting contributions with even higher energy, the correlation function has the form

( ; , ) = 1 ℎ + ( ; 0 ) + 2 ℎ -( ; 2 ) + 3 ℎ + ( ; 3 ) + 4 ℎ -( ; 4 ) + • • • (3a)
where

ℎ + ( , ) = -+ (-1) -1 -( -) , (3b)
ℎ -( , ) = -ℎ + ( -, ). (3c)
Each of the terms corresponds to one state and the sign factors included in the ℎ ± originate from the staggered structure. The function in equation (3a) has 8 free parameter. Four masses and four prefactors. To stabilize the fit, we imposed priors on all but the ground state mass. These priors are motivated by measured excited states of the omega baryon [START_REF]Observation of an Excited Ω -Baryon[END_REF] and quark model predictions [START_REF] Capstick | Baryons in a Relativized Quark Model with Chromodynamics[END_REF] further motivated by [START_REF] Bazavov | Additional Strange Hadrons from QCD Thermodynamics and Strangeness Freezeout in Heavy Ion Collisions[END_REF][START_REF] Alba | Constraining the hadronic spectrum through QCD thermodynamics on the lattice[END_REF]. The central values of the priors where 2021 MeV, 2250 MeV, and 2400 MeV. The width where chosen to be 10 %, 10 %, and 15 %. No priors where applied to the prefactors. Since at early times, more than four states contribute to the correlation functions, we varied the time range of the fit. In addition, we fitted all ensembles at a given gauge coupling together by assuming that the quark mass dependencies of the mass parameters can be described by a linear function of the strange quark mass. The results from these fits combining multiple ensembles did not enter our main analysis but only served as a crosscheck.

As a complementary procedure, we also applied a GEVP based approach [START_REF] Aubin | A new approach for Delta factors[END_REF]. For that purpose we constructed the matrix

H ( ) = +0 +1 +2 +3 +1 +2 +3 +4 +2 +3 +4 +5 +3 +4 +5 +6 . (4)
On such a matrix the well know staggered variational approach [START_REF] Detar | Variational method with staggered fermions[END_REF] can be applied. We solved the GEVP

H ( 0 )ì ( , ) = ( , )H ( 1 )ì ( , ) (5)
for two times 0 and 1 and then constructed a projected correlation function

( ) = ì † ( 1 , 2 )H ( ) ( 1 , 2 ) (6)
from the -th generalized eigenvector ì . This projected correlation function is a linear combination of the time shifted correlation functions entering the matrix in equation [START_REF]Review of Particle Physics[END_REF]. In contrast to the individual contributions in this matrix, the excited state contamination is however greatly reduced in ( ). The effect of this procedure on a mock correlation function can be seen in figure 2 We

Figure 3:

The projected correlation function for the groundstate and the first excited states on one ensemble. The dot show the effective masses of the projected correlation functions. The lower band indicate the mass extracted by a single exponential fit to the projected correlation function. The upper band is not obtained by the fit but indicates the width of the prior used in the alternative four state fit approach. As it can be seen, the first excited state mass as seen by the GEVP method is consistent with the prior. used a standard one-state ansazt to extract the ground state mass from 0 ( ). The performance of the GEVP based method is shown for one ensemble in figure 3. There, also the effective mass of 1 ( ) is shown and compared to the prior for the first excited state in the four state fit approach. No priors are necessary for the GEVP based approach.

For our final determination, we employed three fits per ensemble: Two four state fits with different fit ranges and one combination of times for the GEVP base method. The combinations of fit ranges can be found in [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF]. For each of the three procedures, a Kolmogorov-Smirnov test [START_REF] Borsanyi | Ab initio calculation of the neutron-proton mass difference[END_REF] was used to verify hat the resulting 2 values where distributed as expected. For one value of the gauge coupling, the values of the omega masses determined with the described methods, are compared in figure 4.

QED contributions

The correlation function of a baryon operator in QCD+QED can be written to quadratic order in the valence and sea electrical charge and linear order in the strong isospin breaking as For more details of the formalism, see [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF][START_REF] Toth | Muon g-2: BMW calculation of the hadronic vacuum polarization contribution[END_REF][START_REF] Parato | QED and strong isospin corrections in the hardonic vacuum polarization contribution to the anomalous magnetic moment of the muon[END_REF]. To determine the mass corrections, we apply the effective mass M [ ] to instead of the four state fits or GEVP based extractions because it features a closed form derivative with respect to the values of the correlation function at different times which will be denoted as [ ]/ . This allows to derive expression for the QED and strong isospin breaking correction to hadron masses. These corrections are

QCD+QED ≈ | 0 + • + 2 2 • 20 + • 11 + 2 2 • 02 . (7
0 = M [ 0 0 ] (8a) 02 = M [ ] 0 0 02 = M [ ] 0 0 ( 0 -0 0 ) dets 2 dets 0 (8b) 20 ≈ 1 2 (M [ 1 2 + + -0 ] -M [ 0 0 ]) (8c) 11 ≈
M [ ]

+ + -0 + -- 2 dets 1 dets 0 0 (8d) ≈ (M [ 0 ] -M [ 0 0 ]) (8e)
where • • • 0 is the QCD expectation value of • • • , 0 is a combination of interpolating operators that can be used to measure the Omega mass in QCD, ± are similar operators with positive and negative charges, is a combination of interpolating operators with finite strong isospin breaking and dets 0 , dets 1 , and dets 2 are the appropriate products of quartic roots of fermion determinant and their first and second derivative with respect to the sea electric charge.

Global fits and the determination of 0

For the determination of 0 at the physical point and in the continuum, we employed a global fit. The approach discussed here is described in [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF] under the name of Type 1 fit. For other global fits used in the determination of , especially to define the isospin splitting contributions, see that work. Let = 0 Ω . In general, such a dimensionless observable can be expanded around the physical point as

= + + + + 2 + + 2 (9)
where

= 2 0 / 2 Ω - 2 0 / 2 Ω * , = 2 / 2 Ω - 2 / 2 Ω * , = ( 0 - + ) 2 2 Ω (10)
L. Varnhorst

with 2 = 1 2 ( 2 0 + 2 + -2 + )
are suitable parametrizations of the light and strange quark mass deviations from the physical point and of the strong isospin splitting. The coefficients , . . . , are not assumed constant but to depend on the lattice spacing , , and according to

= 0 + 2 2 (1/ ) + 4 2 (1/ ) 2 + 6 2 (1/ ) 3 (11a) = 0 + 2 2 (11b) = 0 + 2 2 (11c) = 0 + 2 2 + 4 4 + + (11d) = 0 + 2 2 + 4 4 + + (11e) = 0 + 2 2 (11f) = 0 + 2 2 (11g)
where , . . . , are parameters to be determined by a fit and determines the power of the corrections to the continuum approach of the term. For more details see [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF].

Assuming 02 being the first and second order derivatives with respect to the strong isospin breaking and the valence-valence, valence-sea, and and sea-sea electric charges, equation ( 9) can be written as

[ ] 0 = [ + + ] 0 (12a) [ ] = [ ] (12b) [ ] 20 = [ + + + ] 20 + [ ] 0 (12c) [ ] 11 = [ + + + ] 11 + [ ] 0 (12d) [ ] 02 = [ + + + ] 02 + [ ] 0 (12e)
For the observable = 0 Ω , there is no strong isospin breaking component and hence equation (12b) is not needed. The other equations can be fitted to the lattice data in a simultaneous fit that takes into account the statistical correlations between the various quantities appearing in these equations. Not all of the parameters listed above contribute to the determination of every observable. For this specific observable, we always enabled the parameters 0 , 2 , 0 , 0 , 0 , 0 and set = 0. To estimate the systematic uncertainty of our determination, we varied our global fitting form and the mass fits: One set of variations relates to changes in the fitranges and methods of the mass fits for the omega mass and for the pseudoscalar meson masses. Another set of variation is introduced by imposing various cuts on the lattice spacing included in the fit. To estimate the systematic uncertainty introduced by our expansion we switched the fit parameters 4 , 0 , 2 , 2 , , , 2 , and 2 on and off. We also varied the input mass of the omega baryon within the experimental uncertainty. All these variation lead to a large number of fits that we combine with the histogram method described in [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF]. Some representative fits are shown in figure 5. We arrive at a result of The continuum approach of the components in equations ( 12) and some representative global fits.

The red points are the values of the components of measured on the various configurations. These points have already been extrapolated using the global fit with linear 2 dependence to the physical point. For some fits these extrapolations yield significant different points. In that case, the points are shown in grey.

Conclusion

We have determined the 0 scale in the continuum limit in full 2+1+1 flavour QCD+QED. This result can be used to precisely set the scale in subsequent lattice calculation. The method described here for the scale setting with the omega mass was also used in the high precision determination of the anomalous magnetic moment published in [START_REF] Borsanyi | Leading hadronic contribution to the muon magnetic moment from lattice QCD[END_REF].

Acknowledgments

The computations were performed on JUQUEEN, JURECA, JUWELS and QPACE at Forschungszentrum Jülich, on SuperMUC and SuperMUC-NG at Leibniz Supercomputing Centre in München, on Hazel Hen and HAWK at the High Performance Computing Center in Stuttgart, on Turing and Jean Zay at CNRS' IDRIS, on Joliot-Curie at CEA's TGCC, on Marconi in Roma and

Figure 1 :

 1 Figure 1: The effective mass of the three operators used for the omega mass determination on the coarsest ensemble. The green data points show the effective masses calculated directly from the correlation function of the respective operators. The red band indicates the ground state mass as extracted with a multistate fit (for details see main text). The red dotted line indicates the effective mass of that multistate fit.

Figure 2 :

 2 Figure 2: The effect of the GEVP based procedure as described in the main text on the effective mass. The black dots show the 2-lag effective mass of a mock correlation function and the colored dots show the same effective mass of the projected correlation function. The mock correlation function contains 6 states. For the the left panel, a 6 × 6 matrix was employed whereas for the right panel, a 4 × 4 matrix was employed. The overlap with the ground state is improved significantly in both cases

Figure 4 :

 4 Figure 4: Comparison between the different determination of the omega mass on four ensemble with gauge coupling = 3.84.

2 ]Figure 5 :

 25 Figure 5:The continuum approach of the components in equations (12) and some representative global fits.

  )

	m s : m s /m l		4state fit -combined
	0.039130 : 26.893	4state fit -range#2 4state fit -range#1
			GEVP fit
	0.040750 : 28.007
	0.043194 : 30.205
	0.043194 : 28.500
	β=3.8400	0.780 0.785 0.790 0.795 0.800 0.805 0.810 0.815
			aM Ω

  [. . .] 0 to be the pure isospin symmetric QCD part of a given observable and [. . .] , [. . .] 20 , [. . .] 11 , [. . .]

= 0.17236(29) (63) [70] fm.[START_REF] Bazavov | Additional Strange Hadrons from QCD Thermodynamics and Strangeness Freezeout in Heavy Ion Collisions[END_REF]

on GPU clusters in Wuppertal and Budapest. We thank the Gauss Centre for Supercomputing, PRACE and GENCI (grant 52275) for awarding us computer time on these machines. This project was partially funded by the DFG grant SFB/TR55, by the BMBF Grant No. 05P18PXFCA, by the Hungarian National Research, Development and Innovation Office grant KKP126769 and by the Excellence Initiative of Aix-Marseille University -A*MIDEX, a French "Investissements d'Avenir" program, through grants AMX-18-ACE-005, AMX-19-IET-008 -IPhU and ANR-11-LABX-0060.


Ina Reiche

Myriam Eveno

Clément De Mecquenem

Thomas Calligaro

Eric Laval

Ruven Pillay

Elisabeth Ravaud

Victor Etgens

Kristina Mösl

Matthias Alfeld

The original appearance and beauty of ancient paintings virtually reconstructed

Keywords: Smalt, discolouration, macro-XRF scanning, hyperspectral imaging, virtual reconstruction

As part of questions about methods of cultural heritage conservation, transdisciplinary research was performed in order to develop a set of analytical processes allowing us to reconstruct the appearances of historical paintings at different moments in their history. This reconstruction is based on a detailed knowledge of the materiality of the works, their preservation state and the conditions of conservation and storage. These analyses, necessarily non-or microinvasive for precious artworks, are accompanied by actual and virtual modelling of the evolution of the materials composition of the works. The example of a restored French 16 th c. painting illustrates our approach.

I. Introduction

Although generally well exhibited or correctly stored, easel paintings can change over time. As complex works of art, the degradation processes can be variable depending on multiple factors. Therefore, preserving paintings over a long time represents a challenge. When the painting is already weathered, conservation-restoration measures are intended to maintain as much as possible the original material, to restore an appearance as original as possible and to prevent the painting from further degradation. Restoration generally needs a practical intervention into the materiality of the painting. The choice to be made is not always simple and straightforward. Virtually proposing different restoration solutions based on physicochemical analyses and imaging support the conservation decision making. Models need to be developed for determining the original state of the studied painting on the basis of quantified physicochemical markers by comparing them to artificially aged mock-ups.

Additionally, these mock-ups allow establishing criteria for the assessment of the state of preservation and identifying factors responsible for the alteration phenomena, and explaining visible changes. The gained knowledge on the conservation state and the alteration mechanisms of the painting allows for creating a virtual reconstruction of the original state of the painting using mathematical tools.

The approach to achieve this goal is necessarily transdisciplinary and is based on analysis and imaging of the painting in its current state. Chemical, physical and mathematical tools are used to study the painting in situ or on cross-sections prepared from micro-samples. X-rays analyses and imaging as well as reflectance imaging spectroscopy or hyperspectral imaging (RIS) are carried out. The acquired data are combined and treated by statistical tools. The challenge is to be able to carry out these investigations and reconstructions in the less invasive way avoiding sampling. In a practical way and as the first step in this research, we want to set up an operational, analytical strategy for the study of paints when the question of the original appearance of the painted works is raised.

II. Recent technological development

In the last decade, much progress has been made in the instrumental development of analytical devices adapted to the non-invasive imaging of paintings. Above all, the setup of macro-X-ray fluorescence imaging (MA-XRF) has to be cited [START_REF] Alfeld | Revealing hidden paint layers in oil paintings by means of scanning macro-XRF: a mock-up study based on Rembrandt's "An old man in military costume[END_REF][START_REF] Ravaud | Development of a versatile XRF scanner for the elemental imaging of paintworks[END_REF]. Besides, micro-analyses at synchrotron sources (Cotte et al. 2010, Janssens et al. 2010) allowed important progress in combination with combined micro-analyses on cross-sections. To gain non-invasive insights into the paint stratigraphy, confocal micro-XRF was developed and applied to easel paintings [START_REF] Reiche | Depth profiling reveals multiple paint layers of Louvre Renaissance paintings using non-invasive compact confocal micro-X-ray fluorescence[END_REF][START_REF] Reiche | New insight into the painting stratigraphy of L'Homme blessé by Gustave Courbet combining scanning macro-XRF and confocal micro-XRF[END_REF]. This technique allows minimizing sampling for cross-section analysis. Last but not least, hyperspectral imaging (RIS) was developed as a precious complementary imaging tool for studying the palette of paintings [START_REF] Delaney | Mapping the pigment distribution of Vermeer's Girl with a Pearl Earring[END_REF][START_REF] Alfeld | Revealing hidden paint layers in oil paintings by means of scanning macro-XRF: a mock-up study based on Rembrandt's "An old man in military costume[END_REF][START_REF] Pillay | Hyperspectral imaging of art: Acquisition and calibration workflows[END_REF]. In summary, there are available methods ranging from point analysis to multiscale and multimodal imaging at large scale facilities, in the laboratory and using portable devices, allowing in situ studies. These methods need to be applied in the framework of an integrated interdisciplinary approach that fulfils the special requirements and challenges of studying precious easel paintings, and that is adapted to the research questions addressed in each special case.

III. Results of a case study

The steps of the process presented here are illustrated here through the following case study provided by a scientific examination at the C2RMF (E. Ravaud, pers. comm.) of the French painting Woman (Artemise?) doing a Libation from an anonymous painter from the Ecole de Fontainebleau, around 1570 (320 x 780 mm 2 , inv. no. RFML.PE.2019.40.1), Louvre Museum, Paris, France (Figure 1).

We were especially interested in the discolouration of the smalt pigment inducing a colour change of the smalt-based paint layer. Smalt is corresponding to a crushed Cobalt-colored potassium-based blue glass. It turns grey and brown during ageing leading to an irreversible discolouration of the pigment and the paint layer. In this case, a virtual reconstruction of the original appearance is especially appreciable because no restoration treatment allows to turn back to the original aspect.

Figure 2 shows the characteristic chemical maps obtained by MA-XRF for smalt containing paints and the corresponding RIS spectral angle maps obtained from the distribution of defined endmembers.

IV. Conclusions and work in progress

Our work shows that the analytical tools are available for in situ non-invasive combined and multi-modal imaging of artworks such as easel paintings. The challenge relies on data evaluation, fusion and storage. Although established for isolated methods,

Figure 1 :

 1 Figure 1 : A) Woman doing a Libation (Artemise?) from an anonymous painter from the Ecole de Fontainebleau, around 1570, 320 x 780 mm 2 , inv. no. RFML.PE.2019.40.1, Louvre museum, before restoration, B) Detail of reflected light image before restoration and C) Detail of reflected light image after restoration

  progress needs to be made in data fusion to allow for integrated data interpretation and propose dedicated answers in the field of heritage sciences. For the presented case study, work is in progress. The digital tools are available for a colour reconstruction based on combined data of MA-XRF, RIS and colour measurements. It is intended to apply machine learning methods in order to propose different hypotheses of the original appearance (de Mecquenem et al. forthcoming). Virtual reconstructions allow for the modelling of different states of the painting by providing new aesthetic projections. These reconstructions can help in a long perspective in the decision-

Figure 2 :

 2 Figure 2: A) MA-XRF maps of Co, Cu and Hg, B) Map showing the pigment distribution obtained from RIS and C) Reflectance spectra of the end members used for the RIS distribution map.

Acknowledgements:

We gratefully acknowledge the financial support of the Fondation des Sciences du Patrimoine in the framework of the PhD project entitled MARCS (2021-2024).


Florent Bartoccioni

Éloi Zablocki

Andrei Bursuc

Patrick Pérez

Matthieu Cord

Karteek Alahari

Valeo Ai Inria

Valeo Ai

LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation

Keywords:

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

To plan and drive safely, autonomous cars need accurate 360-degree perception and understanding of their surroundings from multiple and diverse sensors, e.g., cameras, RADARs, and LiDARs. Most of the established approaches tardily aggregate independent predictions from each sensor [START_REF] Liu | SMOKE: Single-stage monocular 3D object detection via keypoint estimation[END_REF][START_REF] Roddick | Orthographic feature transform for monocular 3D object detection[END_REF][START_REF] Wang | FCOS3D: Fully convolutional one-stage monocular 3D object detection[END_REF]. Such a late fusion strategy has limitations for reasoning globally at the scene level and does not take advantage of the available prior geometric knowledge that links sensors. Alternatively, the bird's-eye-view's (BEV) representational space, a.k.a. top-view occupancy grid, recently gained considerable interest within the community. BEV appears as a suitable and natural space to fuse multiple views [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF] or sensor modalities [START_REF] Hendy | FISHING net: Future inference of semantic heatmaps in grids[END_REF][START_REF] Bai | Transfusion: Robust LiDARcamera fusion for 3D object detection with transformers[END_REF] and to capture semantic, geometric, and dynamic information. Besides, it is a widely adopted choice for downstream driving tasks including motion forecasting [START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF][START_REF] Ettinger | Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset[END_REF][START_REF] Chang | Argoverse: 3D tracking and forecasting with rich maps[END_REF][START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF] and planning [START_REF] Zeng | End-to-end interpretable neural motion planner[END_REF][START_REF] Casas | MP3: A unified model to map, perceive, predict and plan[END_REF][START_REF] Chitta | NEAT: Neural attention fields for end-to-end autonomous driving[END_REF][START_REF] Caesar | NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles[END_REF]. In this paper, we focus on BEV perception from multiple cameras. The online estimation of BEV representations is usually done by: (i) imposing strong geometric priors such as a flat world [START_REF] Reiher | A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird's eye view[END_REF] or correspondence between pixel columns and BEV rays [START_REF] Roddick | Predicting semantic map representations from images using pyramid occupancy networks[END_REF], (ii) predicting depth probability distribution over pixels to lift from 2D to 3D and project back in BEV [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF], a system subject to compounding errors, or, (iii) learning a costly dense mapping between multi-camera features and the BEV grid pixels [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF].

Figure 1: LaRa overview. Semantic features (green) are extracted from the images with a shared CNN and are concatenated with ray embeddings (multi-colored) that inform about geometric information to spatially relate pixels within and across cameras. This representation is then fused into a compact latent representation through one cross-attention (CA) and L self-attention (SA) layers (yellow). The final BEV map is obtained by querying the latent representation with a cross-attention and then refined with BEV CNN (red). ⊕ denotes concatenation. The orange letters indicate tensor dimensions. K, Q, and V are the Key, Query, and Value of the cross-attentions.

Here, we depart from these dominant strategies and introduce 'LaRa', a novel transformer-based model for vehicle segmentation from multiple cameras. In contrast to prior works, we propose to use a latent 'internal representation' instantiated as a collection of vectors. Fusing multiple views into a compact latent space comes with several benefits. First, it provides an explicit control on the memory and computation footprint of the model, instead of the quadratic scaling of the full mapping between multi-camera features and the BEV grid pixels [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. By design, the number of latents that we use is much smaller compared to the spatial resolution of the BEV grid, enabling a highly-efficient aggregation of information at the latent-level while exploiting spatial cues within and across camera views. Moreover, we also hypothesize that discarding error-prone modules in the pipeline such as depth estimation [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF] can boost model accuracy and robustness. Finally, we can directly predict at the full-scale BEV resolution bypassing noisy upsampling operations. This is infeasible, within a reasonable computational budget, for prior works restricted to coarser BEV grids as they map densely between all the image and BEV grid pixels [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. Besides, as an orthogonal contribution, we augment input features with ray embeddings that encode geometric relationships within and across images. We show that such spatial embeddings, encoding prior geometric knowledge, help guide the cross-attention between input features and the latent vectors.

Our approach is extensively validated against prior works on the nuScenes [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF] dataset. We significantly improve the performance on the vehicle segmentation task, outperforming recent highperforming models [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. Moreover, we show interesting properties of our cross-attention, which naturally stitches multiple cameras together. We also perform several ablation and sensitivity studies of our architecture with respect to hyper-parameters changes. Overall, LaRa is a novel model that learns the mapping from camera views to bird's-eye-view for the task of vehicle semantic segmentation. In summary, our contributions are as follows:

• We encode multiple views into a compact latent space that enables precise control on the model's memory and computation footprint, decoupled from the input size and output resolution.

• We augment semantic features with spatial embeddings derived from cameras' calibration parameters and show that it strongly helps the model learn to stitch multiple views together.

• Our architectural contributions are validated on nuScenes where we reach new SOTA results.

2 Related work

BEV semantic segmentation

Models for BEV segmentation are typically structured in two parts. They first extract features of each camera and then project them into a common top-view grid, called the bird's-eye-view. There are different strategies for this projection, which can be grouped into the following categories.

IPM-based. Inverse perspective mapping (IPM) defines the correspondence between the camera and the ground planes as a homography matrix. IPM makes strong assumptions that the world is planar and the cameras' horizontal axes are parallel to the ground. Early works [START_REF] Bertozzi | Experience of the ARGO autonomous vehicle[END_REF][START_REF] Sengupta | Automatic dense visual semantic mapping from street-level imagery[END_REF] apply it directly to raw camera pixels or features. This approach suffers from blurring and stretching artifacts for distant objects (as they have fewer pixels in the camera view) and objects with a height (as they violate the planar world assumption). To alleviate these shortcomings, a generative adversarial network [START_REF] Zhu | Generative adversarial frontal view to bird view synthesis[END_REF] or training a BEV decoder with synthetic ground-truth [START_REF] Reiher | A sim2real deep learning approach for the transformation of images from multiple vehicle-mounted cameras to a semantically segmented image in bird's eye view[END_REF] has been used to refine the IPM projection.

'Lift-splat'-based: guiding with depth. Using depth information to lift features from 2D to 3D and then 'splatting' them in BEV space recently gained popularity for its effectiveness and sound geometric definition. Among the formulations of depth estimation for BEV projection [START_REF] Roddick | Orthographic feature transform for monocular 3D object detection[END_REF][START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF][START_REF] Srikanth | INFER: Intermediate representations for future prediction[END_REF][START_REF] Ng | BEV-Seg: Bird's eye view semantic segmentation using geometry and semantic point cloud[END_REF], estimating depth probabilities along camera rays appears to perform the best [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF]. However, such a strategy, depth being the most influential factor [START_REF] Simonelli | Are we missing confidence in pseudo-LiDAR methods for monocular 3D object detection?[END_REF], is subject to compounding errors. Inaccuracies in depth prediction will propagate into the BEV features, which themselves can be erroneous.

Implicitly learned with dense networks. An alternative to explicit geometric projection is to learn the mapping from data. For instance, VPN [START_REF] Pan | Cross-view semantic segmentation for sensing surroundings[END_REF] uses an MLP to make a dense correspondence between pixels in the camera views and BEV. These methods rely on such expensive operations and do not use readily available spatial information given by the calibrated camera rig capturing the images. The BEV projection must be entirely learned, and as it is determined by training data, it can hardly apply to new settings with slightly different camera calibrations. Alternatively, PON [START_REF] Roddick | Predicting semantic map representations from images using pyramid occupancy networks[END_REF] builds on the observation that a column in the camera image contains all the information of the corresponding ray in BEV: it first encodes each column into a feature vector, which is then decoded into a ray along the depth dimension. However this relies on two implicit assumptions: (i) the camera follows a pinhole projective model, and (ii) it is horizontally aligned with the ground plane.

Implicitly learned with transformer architectures. The attention system at the core of transformer architectures [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF][START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF] allows learning of long-range dependencies and correspondences explicitly. These architectures have recently been employed for the BEV semantic segmentation task, yielding among the best-performing methods [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF][START_REF] Li | BEVFormer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers[END_REF][START_REF] Gong | GitNet: Geometric prior-based transformation for birds-eye-view segmentation[END_REF]. Nonetheless, a direct cross-attention [START_REF] Vaswani | Attention is all you need[END_REF] between camera images and the BEV grid is computationally expensive. BEVFormer [START_REF] Li | BEVFormer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers[END_REF] alleviates this issue by only cross-attending BEV pixels with cameras in which the BEV pixel is visible and by replacing the heavier multi-head attention [START_REF] Vaswani | Attention is all you need[END_REF] with deformable attention [START_REF] Zhu | Deformable DETR: Deformable transformers for end-to-end object detection[END_REF]. CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] keeps the vanilla multi-head cross-attention [START_REF] Vaswani | Attention is all you need[END_REF] but applies it between low-resolution camera feature maps and a small BEV grid which is then upsampled to reach the final resolution. GitNet [START_REF] Gong | GitNet: Geometric prior-based transformation for birds-eye-view segmentation[END_REF] restrains the cross-attentions to column-ray pairs making the same original implicit assumptions as PON [START_REF] Roddick | Predicting semantic map representations from images using pyramid occupancy networks[END_REF]. Our proposed model LaRa belongs to this category as it learns the BEV representation with a transformer architecture. On the other hand, our attention scheme does not impose strong geometric assumptions while still being efficient enough to attend to a full-resolution BEV grid.

Incorporating geometric priors in Transformers

Since transformer architectures are permutation-invariant, spatial relationships between image regions are lost if no precautions are taken. A standard practice to retain this spatial knowledge is to add a positional embedding to the input of attention layers [START_REF] Vaswani | Attention is all you need[END_REF]. A popular approach is to encode the position of pixels with sine and cosine functions of varying frequencies [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF][START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF] applied over the horizontal and vertical axes. An alternative solution to induce spatial awareness in the model is to concatenate x, y positions to feature maps fed to convolutional layers [START_REF] Liu | An intriguing failing of convolutional neural networks and the coordconv solution[END_REF].

Related to our ray embedding proposition, recent works [START_REF] Yifan | Input-level inductive biases for 3D reconstruction[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] embed the parameters of the calibrated cameras in the image features, improving training efficiency and segmentation performance. Similar to LaRa, IIB [START_REF] Yifan | Input-level inductive biases for 3D reconstruction[END_REF] also encodes the camera center and ray direction in the input feature sequence, but it applies it to depth estimation on image pairs in an indoor environment. Furthermore, Yifan et al. [START_REF] Yifan | Input-level inductive biases for 3D reconstruction[END_REF] embed the origin and direction of rays into Fourier features, which can become memory intensive depending on the number of frequency bands and also introduces additional hyper-parameters to tune. CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] adds up a ray direction embedding to the input feature sequence, but, differently from ours, uses the camera center embedding in the BEV query. This requires a BEV query and 'cross-view attention' operation per camera, increasing the memory and computational footprint, thus limiting the maximum resolution of the BEV query.

3 LaRa: Our Latents and Rays Model

Given multiple cameras observing the scene, our goal is to estimate a binary occupancy grid [START_REF] Elfes | Occupancy grids: A stochastic spatial representation for active robot perception[END_REF] ŷ ∈ {0, 1} hbev×wbev of size h bev × w bev ∈ N 2 for vehicles in the surroundings of the ego car. We propose 'LaRa' a transformer-based architecture to efficiently aggregate information gathered from multiple cameras into a compact latent representation before expanding back into the BEV space.

Besides, as we believe that the geometric relationship between cameras should guide the fusion across each camera view, we propose to augment each pixel with the geometry of the ray that captured it. The LaRa architecture is illustrated in Figure 1.

Input modeling with geometric priors

We consider C cameras described by

(I k , K k , R k , t k ) C k=1 , with I k ∈ R H×W ×3 the image pro- duced by camera k, K k ∈ R 3×3 the intrinsics, R k ∈ R 3×3 and t k ∈ R 3 the
extrinsic rotation and translation respectively. From these inputs, two complementary types of information are extracted: semantic information from raw images and geometric cues from the camera calibration parameters.

Semantic information from raw images.

A shared image-encoder E extracts feature maps for each image F k = E(I k ) ∈ R h×w×c . Following [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF], we instantiate E with a pretrained Efficient-Net [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] backbone to produce the multi-camera features. These spatial feature maps in R C×h×w×c are then rearranged as a sequence of feature vectors, in R (C h w)×c .

Leveraging geometric priors. To enrich camera features with geometric priors, commonly used sine and cosine spatial embeddings [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF][START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF] are ambiguous in presence of multiple cameras. A straightforward solution would be to use camera-dependant learnable embeddings in addition to the Fourier embeddings to disambiguate between cameras. However, in our setting, we argue that the geometric relationship between cameras, which is defined by the structure of the camera rig, is crucial to guide the fusion of the views. This motivates our choice to leverage the cameras' extrinsics and intrinsics to encode the position and orientation of each pixel in the vehicle ego-frame.

More precisely, we encode the camera calibration parameters by constructing the viewing ray for each pixel of the cameras. Given a pixel coordinate x ∈ R 2 within a camera image I k , the direction d k (x) ∈ R 3 of the ray that captured x is computed with:

d k (x) = R -1 k K -1 k x, (1)
where x are the homogeneous coordinates of x, and d k (x) is expressed in ego-coordinates. The origin of the ray d k (x) is the camera center given by t k .

Then, to fully describe the position and the orientation of the ray that captured pixel x, we use the embedding ray k (x) ∈ R d computed as follows:

ray k (x) = MLP ray (t k ⊕ d k (x)), (2)
where ⊕ is a concatenation operation and MLP ray a 2-layer MLP with GELU activations [START_REF] Hendrycks | Gaussian error linear units (GELUs)[END_REF]. The computation is consistent within and across cameras and it exhibits an interesting property: overlapping regions for two cameras with the same optical center have the same ray embedding. Note that the intrinsics are scaled according to the difference in resolution between I k and F k .

As shown in Figure 1, the final input vector sequence, in R (C h w)×(d+c) , is produced by concatenating each of the C h w feature vectors F k (x) ∈ R c with its geometric embedding ray k (x) ∈ R d .

Building latent representations and deep fusion

To control the computational and memory footprint of the image-to-BEV block, we leverage findings from general-purpose architectures [START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF] and propose to use an intermediate fixed-sized latent space instead of learning the quadratic all-to-all correspondence between multi-camera features and BEV space [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. Formally, the visual representations F k from all cameras, along with their corresponding geometric embeddings ray k , are compressed by cross-attention [START_REF] Vaswani | Attention is all you need[END_REF] into a collection of N learnable latent vectors of dimension M ∈ N and processed by a series of L self-attention blocks [START_REF] Vaswani | Attention is all you need[END_REF] (see yellow elements in Figure 1). We stress that N ≪ C h w, which enables to fuse and process efficiently the visual information coming from all the cameras regardless of the input feature resolution or the number of cameras. Thanks to latent-based querying, this formulation decouples the network's deep multi-view processing from the input and output resolution. Our architecture can thus take advantage of the full resolution of the BEV grid.

Generating BEV output from latents

The final step is to decode the binary segmentation prediction ŷ ∈ {0, 1} hbev×wbev from the latent space. In practice, the latent vectors are cross-attended [START_REF] Vaswani | Attention is all you need[END_REF] with a BEV 'query' grid Q ∈ R hbev×wbev×dbev at the final prediction resolution, with d bev ∈ N a hyper-parameter (illustrated by the red blocks in Figure 1). Each element of the query grid is a feature vector encoding the spatial position in the bird's-eye-view which specifies what information the cross-attention would extract from the latent representations. This last cross-attention yields a feature map in BEV space, in dimension h bev × w bev × 256, that is further refined with a small convolutional encoder-decoder U-Net ('BEV CNN' in Figure 1) to finally predict the binary bird's-eye-view semantic map ŷ ∈ {0, 1} hbev×wbev×1 .

Specifically, we consider a combination of two types of queries: normalized coordinates in the BEV space and radial distance. Normalized coordinates encode ego-centered normalized coordinates of the BEV plane. They are obtained with:

Q coords [i, j] = 2i h bev -1 -1, 2j w bev -1 -1 , ∀i, j ∈ {0, . . . , h bev -1} × {0, . . . , w bev -1}. (3)
Normalized radial distances are simply Euclidean distances of pixels w.r.t. the origin:

Q radial [i, j] = Q coords [i, j] 2 i + Q coords [i, j] 2 j . (4)
While the network could produce a similar embedding from Q coords using MLP bev , we find that introducing these radial embeddings along Q coords empirically improves results. Moreover, this query decoding choice compares favorably against more classical Fourier embeddings [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF][START_REF] Yifan | Input-level inductive biases for 3D reconstruction[END_REF] and learned query embeddings [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF], as shown in Table 2.

Experiments

Dataset. We conduct experiments on the nuScenes dataset [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF], which contains 34k annotated sets of frames captured by C=6 synchronized cameras covering the 360°field of view around the ego vehicle. The extrinsics and intrinsics calibration parameters are given for all cameras in every scene. Raw annotations come in the form of 3D bounding boxes that are simply rendered in the discretized top-down view of the scenes to form the ground-truth for our binary semantic segmentation task.

Precise settings for training and validation. With no established benchmarks to precisely compare model's performances, there are almost as many settings as there are previous works. Differences are found at three different levels: The resolution of the output grid, the level of visibility used to select objects as part of the ground-truth, and the task considered. In this paper, we address the task of binary semantic segmentation of all vehicles (cars, bicycles, trucks, etc.) [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. This choice is made to have fair and consistent comparisons with our baselines [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF], however, it should be noted that our model is not constrained to this setting. To enable and ease future comparison, we have published our code2 . We also present additional settings in the supplementary material. In all the settings we considered, models are evaluated with the IoU metric.

Training and implementation details. We train our model by optimizing the Binary Cross Entropy with our predicted soft segmentation maps and the binary ground-truth. Images are processed at resolution 224 × 480. We use the AdamW [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] optimizer with a constant learning rate of 5e-4 and a weight decay of 1e-7. We train our model on 4 Tesla V100 16GB GPUs with a total batch size of 8 for 30 epochs. Training takes on average 11 hours. We use an EfficientNet-B4 [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] with an output stride of 8 as our CNN image encoder. For the BEV CNN we follow Philion and Fidler [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF]. MLP bev is a 2-layer MLP producing d bev = 128-dimensional features.

Comparison with previous works

In Table 1, we compare the IoU performances of LaRa against two baselines Lift-Splat [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF] and CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] on vehicle BEV segmentation in their respective training and evaluation setups. In all cases, we improve results by a significant margin. More precisely, we improve by 10% compared to Lift-Splat in their settings, by 10% and 8% compared to CVT respectively in Setting 1 and Setting 2. This suggests that our model can better extract the geometric and semantic information from all cameras with a very general architecture that does not necessitate any strong geometric assumptions. Besides, when compared with CVT, we observe that LaRa obtains better results in the setting with finer resolution (+10% in Setting 1 vs. +8% in Setting 2).

Since our attention mechanism does not rely on all-to-all attention between camera images and BEV map as CVT does, LaRa can directly decode to the final BEV resolution which helps for fine prediction at a high resolution.

Model ablation and sensitivity to hyper-parameters

Input and Output-level embeddings. To assess the contribution of the geometric embeddings that we use, we compare the different choices at both the input and output level in Table 2. As hypothesized, embedding the geometric relationship between cameras in the input is better suited for our task than the generic sine and cosine spatial embeddings. The additional camera index, while performing better than Fourier feature alone, is not enough to link pixels across cameras. For the output query embedding, the combination of normalized coordinates and radial distance gives the best results. This simple choice outperforms both the Fourier features [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF] and learned embeddings [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF] that also have the disadvantage of increasing the number of parameters.

Sensitivity to hyper-parameters. To delve into the influence of hyper-parameters, we conduct a sensitivity analysis in Figure 2 where we vary the number N of latent vectors, their dimension M and the number of self-attention blocks L. We clearly observe that the performance increases with the number of latent vectors used. This is expected as it is the main parameter controlling the attentional bottleneck between input and output. Such a parametrization allows for an easy tuning of the performance/memory trade-off. We observe no clear correlation between the dimension M of latent vectors, the number L of self-attention layers, and the obtained IoU performance. This indicates that our architecture is not too sensitive to these hyper-parameters and can work efficiently with a wide range of values for these parameters. Although we obtain better results with 512 latent vectors, we use a maximum of 256 to stay in the same computational regime as the baseline we compare against; training with 512 latent vectors requires 32GB GPUs.

Study of attention

As quantitatively studied in Section 4.2, embedding camera rays impacts significantly the performance of LaRa. By analyzing the input-to-latent attention map, we further investigate the geometric reasoning of LaRa in Figure 3. In this figure, we show two representations of the attention: a reprojection of the attention in the camera-space (left) and a top-view projection of the attention in polar coordinates by collapsing, i.e., averaging the vertical dimension (right). In the latter, the radial distance is proportional to the attention level and shows the directions the network attends the most.

The study is conducted at three different levels. First, for a couple of one latent vector and one attention head (n = 10, h = 5 and n = 50, h = 30), among N = 256 possible latents and H = 32 possible attention heads. Second, for one latent vector and the averaged attention from all attention heads (n = 10, h = avg and n = 50, h = avg). Third, for one attention head and the averaged attention over all latents (n = avg, h = 5 and n = avg, h = 30). From these three settings, we note the followings: First, the attention map between one latent vector and one attention head targets a specific direction (about a 90°field of view). Additionally, it can be clearly observed that the attention is continuous across cameras, proving the network is able to retrieve the pixel relationships between views. Second, while one attention head fires in a specific direction, the attention averaged over all the heads for one latent vector spans over half of the scene. This allows one latent vector to extract long-range context between views with the capacity to disambiguate them. Third, the attention for one head aggregated over all the latent vectors covers all directions, suggesting that the latent vectors contain all of the directional information and that the whole scene is attended across the latents. To summarize, by integrating early multi-view geometric cues instantiated by camera rays embedding (Section 3.1), we show that LaRa learns to reason across views. We also provide quantitative evidence in the supplementary material.

Qualitative Results

We show the segmentation results of two complex scenes in Figure 4. For a fair comparison, we use our model trained with visiblity > 40% against CVT and > 0% against Lift-Splat. Compared to LaRa, CVT missed two objects, one at a long distance and the other in the dark (red box). We also estimate the boundaries of the vehicles better than Lift-Splat (green box). Interestingly, models trained on all vehicles (visibility > 0%) tend to hallucinate cars in occluded or distant regions (highlighted with black circles in the figure).

Conclusion

We presented LaRa, which leverages transformer-based architectures and encoder-decoder models, with respectively efficient deep cross-and self-attentions as well as an explicit control on the computation and memory footprint thanks to decoupling the bulk of the processing from the input and output resolution. By incorporating ray embeddings into LaRa, we augment semantic features with geometric cues of the scene and show that this leads to multi-view stitching.

Limitations. Our model operates on camera inputs only. Thus, in adverse conditions, e.g., with glares and darkness, its performance remains limited. To better handle these challenging situations, one avenue of improvement would be the extension of LaRa to handle complementary modalities, e.g., coming from LiDARs or radars.

Broader impacts. LaRa demonstrates that the geometry and semantics of a complex scene can be compacted in a small collection of latent vectors. We believe that this formulation would allow for efficient temporal reasoning. Currently, the temporal modeling is done in the BEV space, which is high resolution and mostly represents empty space [START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF][START_REF] Li | BEVFormer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers[END_REF]. We show the six camera views surrounding the vehicle along with segmentation map ground-truth for reference. In the ground-truth (GT) map, vehicles are shown in blue (visibility > 40%) or purple (visisibility < 40%). The ego vehicle is located in the center and facing downwards. We show results for our two baselines [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. For a fair comparison, we always compare using their respective level of visibility. Setting 2 is used.

LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation -Supplementary Material -A Implementation details

Following common practice [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] we employ an EfficientNet [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] as our CNN image encoder E. In particular, we use an EfficientNet-B4 [START_REF] Tan | Efficientnet: Rethinking model scaling for convolutional neural networks[END_REF] with an output stride of 8. It extracts feature maps for each image

F k = E(I k ) ∈ R h×w×c .
In practice, h = 224/8 = 28, w = 480/8 = 60 and we define c = 128.

For the BEV CNN, we follow Philion and Fidler [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF] and use an encoder-decoder architecture with a ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] as backbone. It produces features at three levels of resolutions (1:1, 1:2 and 1:8), which are progressively upsampled back to the input resolution with bilinear interpolation (first ×4 for the 1:8th scale then ×2 for the 1:2th). Skip connections are used between encoder and decoder stages of the same resolution.

Both MLP ray and MLP bev are 2-layer MLPs producing 128-dimensional features. Each consists of two linear transformations with a GELU [START_REF] Hendrycks | Gaussian error linear units (GELUs)[END_REF] activation function:

MLP(x) = W 2 GELU(W 1 x + b 1 ) + b 2 . (5)
The exact specification of other modules will be available in our code upon publication.

A.1 Attention modules

Following the original formulation and notations [START_REF] Vaswani | Attention is all you need[END_REF], the attention operation is defined as:

Attn(Q, K, V ) = softmax( QK ⊤ √ d K )V (6)
with its multi-headed extension:

MultiheadAttn(Q, K, V ) = Concat(head 1 , . . . , head h )W O where head i = Attn(QW Q i , KW K i , V W V i ). (7)
with d q , d v , d k the dimensions of Q, K and V . In practice, we use d model , a hyperparameter, to define the dimension of the queries, keys and values for the inner attention (Equation 6) as well as h the number of attention heads. More precisely, we linearly project queries, keys and values h times with different projections, each with dimension d emb = d model /h. The learnable projection matrices of each head are defined as

W Q i ∈ R dq×demb , W K i ∈ R d k ×demb , W V i ∈ R dv×demb and W O i ∈ R h•demb×dv .
Our architecture integrates three attention modules [START_REF] Vaswani | Attention is all you need[END_REF]: (i) a cross-attention between latent vectors and input features; (ii) a sequence of self-attention on the latent vectors; (iii) a cross-attention between BEV query and latent vectors. More precisely, and with a slight abuse of notation: Latent-Input cross-attention latents := MultiheadAttn(LayerNorm(latents), LayerNorm(input), LayerNorm(input)) + latents latents := MLP(LayerNorm(latents)) + latents [START_REF] Ettinger | Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset[END_REF] Latent self-attention latents := MultiheadAttn(LayerNorm(latents), LayerNorm(latents), LayerNorm(latents)) + latents latents := MLP(LayerNorm(latents)) + latents

BEVquery-Latent cross-attention output := MultiheadAttn(LayerNorm(BEVquery), LayerNorm(latents), LayerNorm(latents)) output := MLP(LayerNorm(output)) + output [START_REF] Caesar | nuScenes: A multimodal dataset for autonomous driving[END_REF] In particular, the cross-attention between BEV query and latent vectors is not residual. Since the query is made of coordinates, imposing the network to predict segmentation as residual of coordinates does not make sense.

A.2 Output embedding

In the main paper, we considered Fourier features and learned query as alternative BEV query embeddings. Here we detail both of them.

Fourier features. The Fourier encoding has been proven to be well suited for encoding fine positional features [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF][START_REF] Yifan | Input-level inductive biases for 3D reconstruction[END_REF]. This is done by applying the following on an arbitrary input z ∈ R:

fourier(z) = (z, sin(f 1 πz), cos(f 1 πz), . . . , sin(f B πz), cos(f B πz)) , (11)
where B is the number of Fourier bands, and f b is spaced linearly from 1 to a maximum frequency f B and typically set to the input's Nyquist frequency [START_REF] Jaegle | Perceiver IO: A general architecture for structured inputs & outputs[END_REF]. The maximum frequency f B and number of bands B are hyper-parameters. This Fourier embedding is applied on the normalized coordinate grid such that:

Q fourier [i, j] = fourier(Q coords [i, j] i ) ⊕ fourier(Q coords [i, j] j ). (12)
Learned. Another alternative, following common transformer practice [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF] and most notably proposed by CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF], is to let the network learn its query of dimension d bev-query from data. However, this is memory intensive as it introduces h bev × w bev × d bev-query additional parameters to be optimized. In other words, the number of parameters grows quadratically to the resolution of the BEV map. For experiments using learned output query embedding, we use d bev-query = 32.

B Evaluation details

With no established benchmarks to precisely compare model's performances, there are almost as many settings as there are previous works. Differences are found at three different levels:

• The resolution of the output grid where two main settings have been used: a grid of 100m×50m at a 25cm resolution [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF][START_REF] Roddick | Predicting semantic map representations from images using pyramid occupancy networks[END_REF][START_REF] Pan | Cross-view semantic segmentation for sensing surroundings[END_REF][START_REF] Saha | Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation[END_REF] and a grid of 100m×100m at a 50cm resolution [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. These settings are respectively referred as 'Setting 1' (h bev × w bev = 400 × 200) and 'Setting 2' (h bev × w bev = 200 × 200).

• The considered classes. There are slight differences in the classes used to train and evaluate the model. For instance, some models are trained with a multi-class objective to simultaneously segment objects such as cars, pedestrian or cones [START_REF] Roddick | Predicting semantic map representations from images using pyramid occupancy networks[END_REF][START_REF] Pan | Cross-view semantic segmentation for sensing surroundings[END_REF][START_REF] Saha | Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation[END_REF]. Some others only train and evaluate in a binary semantic segmentation setting on a meta-class vehicles which includes cars, bicycles, trucks, etc. [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. Some works also use instance segmentation information to train their model where the centers of each distinct vehicle is known at train time [START_REF] Hu | FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras[END_REF]. In all of our experiments, we place ourselves in the binary semantic segmentation setting of the meta-class vehicles. This choice is made to have fair and consistent comparisons with our baselines [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF], however, it should be noted that our model is not constrained to this setting.

• The levels of visibility of objects. Objects selected as ground truth, both for training and evaluating the model, differ in terms of their levels of visibility. Three options have been considered: objects that are in line-of-sight with the ego car's LiDAR [START_REF] Roddick | Predicting semantic map representations from images using pyramid occupancy networks[END_REF], or objects with a nuScenes visibility above a defined threshold, either 0% [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF] or 40% [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. In this section, we also provide results for the driveable area segmentation task, also addressed by CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. Contrary to vehicle segmentation, this task requires the network to do "amodal completion" to a high degree, i.e., to correctly estimate regions of the road despite parts of it being severely occluded.

C Extension to driveable area segmentation task

We followed the protocol of CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] for this segmentation task; the ground truth is generated using HD-map's polygons from the dataset. We kept the same hyperparameters we used for the vehicle segmentation task, with a minor difference to the learning rate: we divide it by a factor 10 after 15 epochs (compared to a constant learning rate for vehicle segmentation).

Quantitative and qualitative results for this additional task are given respectively in Table 3 and Figure 5. When compared with CVT, we observe that LaRa achieves better performance (+0.9). Note that we do not do multi-tasking: following CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF], we train a model specifically for the task of driveable area segmentation. The qualitative examples in Figure 5 are produced by fusing predictions from two models. In this section, we propose a quantitative analysis to support our claim that "our network is able to retrieve the pixel relationships between views thanks to our ray embedding" (Sec. 4.3 in the main paper).

To this end, we introduce a metric that directly quantifies the consistency and alignment of attention values across camera by analyzing behavior in "overlapping" regions, i.e., regions seen by two different cameras. We provide a visual description of this metric and its computation in Figure 7.

In short, knowing the orientation of each camera, we compute the Mean Squared Error (MSE) of the directional attention intensity between cameras on their overlapping regions. This score is averaged for all the overlapping regions, latents and attention heads, and examples in the validation set. A score of zero indicates a perfect match of the attention levels on overlapping regions (i.e., across cameras). Results with this metric, reported in Table 4, show that our 'Cam. rays' embedding is 10 times more "consistent" across cameras than the baseline 'Fourier + Cam. idx'.

Additionally, we provide qualitative examples of the 'Fourier + Cam. idx' embedding to compare against our ray embedding in Figure 6. Contrary to the attention yield by our ray embedding, the one derived from the 'Fourier + Cam. idx' embedding is much more spread out and less consistent across cameras.

E Comparison to PETR encoding

In PETR [START_REF] Liu | Position embedding transformation for multi-view 3d object detection[END_REF], the embedding of each pixel is computed by sampling its ray given D predefined depths. The 3D coordinates of the D sampled points along the ray are normalized, concatenated, processed by an MLP and summed with the visual features. Conceptually, the embedding is a way to indicate to the network "this pixel can observe these 3D points in the camera frustum space".

The embedding in PETR differs in that it is limited by the sampling resolution (i.e., the D predefined depths), as computation and memory footprint increase linearly with respect to D. In contrast, we showed that our constant-complexity embedding is effective as a 3D positional embedding.

In addition, we include quantitative results to compare PETR embedding against our ray embedding in Table 5. We trained our model with PETR input embedding in place of ours. The results show that our ray embedding performs better (+72%).

F Additional attention qualitative analysis

We also provide additional analysis of attention maps for the multi-camera input shown in Figure 8 with a network using 256 latents and 32 attention heads. As in the main paper, the polar plots represent the directional attention intensity, showing the directions the network attends the most. The contribution of each camera is indicated by a color code coherent with Figure 8. Each polar plot is oriented in an upward direction (i.e., the front of the car points upward).

G Additional qualitative examples

We also provide videos of our segmentation results on complex scenes in various visual conditions (daylight, rain, night). In these videos, we compare against our two baselines CVT [START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF] and Lift-Splat [START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF]. For a fair comparison, we use our model trained with visibility > 40% against CVT and > 0% against Lift-Splat. This suggests that the latent vectors contain most of the directional information and that the whole scene is attended across the latent. More rarely, an attention head's polar plot will be directional but will maintain a level of generality by being symmetrical.

Figure 10: Input-to-latent attention study -average over heads. These polar plots represent the directional attention intensity averaged over all attention heads for the 32 attention heads. When averaging over attention heads, we observe that the average attention spans over half of the scene. This allows latent vectors to extract long-range context between views with the capacity to disambiguate them.

Figure 11: Input-to-latent attention study -all the attention heads of a latent vector. These polar plots represent the directional attention intensity of the 32 attention heads for a randomly chosen latent vector (latent vector #10). As shown in Figure 10, one latent vector approximately covers half of the scene over its attention heads.

Figure 12: Input-to-latent attention study -all the latent vectors for an attention head. These polar plots represent the directional attention intensity of the 256 latent vectors for a randomly chosen attention head (head #4). As shown in Figure 9, one attention head generally covers the full scene over the latent vectors.

Figure 2 :

 2 Figure 2: Sensitivity study of LaRa to hyper-parameters. We vary the number of latent vectors (N ), their dimension (M ), and the number of self-attention layers (L) and report IoU performances.

Figure 3 :

 3 Figure 3: Input-to-latent attention study. Attention maps analysis for a network using 256 latents and 32 attention heads. The attention for one attention head and one latent is shown on the left superimposed with RGB images. The polar plots represent the directional attention intensity for one (or the average) attention head with one (or the average) latent vector. The radial distance is proportional to the attention level and shows the directions the network attends the most.

Figure 4 :

 4 Figure4: Qualitative results on complex scenes. We show the six camera views surrounding the vehicle along with segmentation map ground-truth for reference. In the ground-truth (GT) map, vehicles are shown in blue (visibility > 40%) or purple (visisibility < 40%). The ego vehicle is located in the center and facing downwards. We show results for our two baselines[START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF][START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF]. For a fair comparison, we always compare using their respective level of visibility. Setting 2 is used.

Figure 5 :

 5 Figure5: Qualitative results on complex scenes. We show the six camera views surrounding the vehicle along with segmentation ground truth for reference. Vehicles are shown in blue and driveable area in gray. Vehicles and driveable area predictions are from two different models trained independently for their respective ground-truth, the predictions are then merged for vizualization purpose. The ego vehicle is located in the center and facing downwards. Predictions of both driveable area and vehicle segmentation are thresholded at 0.5 for visualization purpose.

Figure 6 :

 6 Figure 6: Input-to-latent attention study. Analysis of attention maps for two networks trained with different input embeddings. Top row is with 'Fourier + Cam. idx' and bottom row is with our proposed 'Cam. rays' embedding. The attention for one attention head and one latent is shown on the left superimposed with RGB images. The polar plots represent the directional attention intensity for one attention head with one latent vector. The radial distance is proportional to the attention level and shows the directions the network attends to the most.

Figure 7 :

 7 Figure 7: Measuring the attention consistency across cameras. The proposed metric computes the Mean-Squared-Error (MSE) of the attention intensity on overlapping regions between cameras (as illustrated for two cameras and one latent and one attention head), and averages it over all cameras, latents, heads and scenes.

Figure 8 :

 8 Figure8: Six input camera images coming from the 360-degree camera rig of nuScenes. Note small overlaps between views, e.g., the front of the white truck is both seen in the front-left and front cams.

Figure 9 :

 9 Figure9: Input-to-latent attention study -average over latents. These polar plots represent the directional attention intensity averaged over all the 256 latent vectors for each attention head. When averaging over latent vectors, we observe that each attention head generally covers all directions. This suggests that the latent vectors contain most of the directional information and that the whole scene is attended across the latent. More rarely, an attention head's polar plot will be directional but will maintain a level of generality by being symmetrical.

Table 1 :

 1 Intersection-over-Union (IoU) for vehicle segmentation on nuScenes. 'Setting 1' refers to a 100m×50m grid with a 25cm resolution and 'Setting 2' to a 100m×100m grid with a 50cm resolution. For training and validation, vehicles are considered only if their visibility level is above a predefined threshold (either 0% or 40%). To compare against other works, we refer the reader to Lift-splat[START_REF] Philion | Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[END_REF] and CVT[START_REF] Zhou | Cross-view transformers for real-time map-view semantic segmentation[END_REF].

			visibility > 0%	visibility > 40%
	Method	Conference	Setting 2	Setting 1 Setting 2
	Lift-splat [4]	ECCV'20	32.1	-	-
	CVT [17]	CVPR'22	-	37.5	36.0
	LaRa (ours)	-	35.4	41.4	38.9

Table 2 :

 2 Ablation study for the input and output query embedding. Training and evaluation are done in Setting 2 (100m×100m at 50cm resolution), with a visibility > 0%.

	Input geometry embedding			Output query embedding
	Cam. rays Cam. idx Fourier IoU	Radial dist. Norm. coords Fourier Learned IoU
	✔	✗	✗	35.4	✔	✔	✗	✗	35.4
	✔	✔	✔	34.4	✗	✔	✗	✗	35.1
	✗	✔	✔	32.3	✗	✗	✔	✗	30.6
	✗	✗	✔	30.5	✗	✗	✗	✔	21.8

Table 3 :

 3 Driveable area segmentation. Results (in IoU) on nuScenes.

	Method	IoU
	CVT	74.3
	LaRa (ours) 75.2

n:10 h:5

Table 4 :

 4 Impact

	Embedding	MSE on overlap
	2D Fourier + Cam. idx 0.0896
	Cam. rays (ours)	0.0068

of ray embedding on cross-camera attention consistency. Cross-camera attention consistency (measured with proposed MSE metric, see Fig.

6

) on nuScene.

Table 5 :

 5 Impact of ray embedding on performance. Vehicle segmentation performance (in IoU) for vehicle segmentation on nuScenes.

	Embedding	IoU
	PETR [40]	34.8
	Cam. rays (ours) 35.4

https://github.com/valeoai/LaRa

Acknowledgments

This work was supported in part by the ANR grants AVENUE (ANR-18-CE23-0011), VISA DEEP (ANR-20-CHIA-0022), and MultiTrans (ANR-21-CE23-0032). It was granted access to the HPC resources of IDRIS under the allocation 2021-101766 made by GENCI.