M Diaz Ortiz

J Gleason

H Grote

A Hallal

M T Hartman

H Hollis

K.-S Isleif

A James

K Karan

T Kozlowski

A Lindner

G Messineo

G Mueller

J H Põld

R C G Smith

A D Spector
email: aaron.spector@desy.de

D B Tanner

L.-W Wei

B Willke

Design of the ALPS II Optical System

Keywords: axion search, laser interferometry, optical cavities, light-shining-through-a-wall

The Any Light Particle Search II (ALPS II) is an experiment currently being built at DESY in Hamburg, Germany, that will use a light-shining-through-a-wall (LSW) approach to search for axionlike particles. ALPS II represents a significant step forward for these types of experiments as it will use 24 superconducting dipole magnets, along with dual, high-finesse, 122 m long optical cavities. This paper gives the first comprehensive recipe for the realization of the idea, proposed over 30 years ago, to use optical cavities before and after the wall to increase the power of the regenerated photon signal. The experiment is designed to achieve a sensitivity to the coupling between axion-like particles and photons down to g αγγ = 2 × 10 -11 GeV -1 for masses below 0.1 meV, more than three orders of magnitude beyond the sensitivity of previous laboratory experiments. The layout and main components that define ALPS II are discussed along with plans for reaching design sensitivity. An accompanying paper (Hallal, et al [1]) offers a more in-depth description of the heterodyne detection scheme, the first of two independent detection systems that will be implemented in ALPS II.

Introduction

While the standard model has been extremely successful, a variety of phenomena suggest that it is still an incomplete description of our universe. One example of this is known as the strong CP problem in quantum chromodynamics (QCD), in which a term in the QCD Lagrangian will break CP symmetry if one of it constituent variables has a non zero value [START_REF] Peccei | CP conservation in the presence of pseudoparticles[END_REF]. While, as an angle, it appears that this variable should be of order unity, no violations of CP symmetry in the strong force have ever been measured [START_REF] Abel | Measurement of the Permanent Electric Dipole Moment of the Neutron[END_REF].

The solution proposed by Peccei and Quinn was to promote this variable to a dynamic field associated with a new elementary particle called the axion. This pseudo-Goldstone boson is characterized by the energy scale of the Peccei-Quinn symmetry breaking: the axion mass as well as its coupling strengths to standard model particles is inversely proportional to this energy scale. Experimental results indicate that the energy scale is far beyond the reach of particle accelerators resulting in very light-weight and very weakly coupled axions. These axions would also couple to photons via the following term in the Lagrangian [START_REF] Peccei | CP conservation in the presence of pseudoparticles[END_REF][START_REF] Weinberg | A new light boson?[END_REF][START_REF] Wilczek | Problem of Strong P and T Invariance in the Presence of Instantons[END_REF]:

L a = g aγγ ϕ a ⃗ E • ⃗ B (1)
Here g aγγ is the axion-photon coupling strength, ϕ a is the axion field, the oscillating electric field is given by ⃗ E, and ⃗ B represents a static magnetic field.

As it happens, axions could be the cause of a number of other phenomena that the standard model also fails to explain. While the most prominent example of this is the existence of cold dark matter [START_REF] Tanabashi | Review of particle physics[END_REF], other anomalies such as stellar cooling rates that exceed predictions may also be explained by axions [START_REF] Giannotti | Cool WISPs for stellar cooling excesses[END_REF]. Furthermore, 'axion-like' particles which are not themselves the QCD axion, but share similar characteristics to it could explain other anomalous observations such as the transparency of the universe to TeV photons [START_REF] Meyer | First lower limits on the photon-axion-like particle coupling from very high energy gamma-ray observations[END_REF]. All of these hints are shown as the red regions in the axion-like particle parameter space in Figure 1.

One of the most promising strategies for axion searches involves the Sikivie effect, where in the presence of an external magnetic field, an axion will generate a photon [START_REF] Sikivie | Experimental Tests of the "Invisible" Axion[END_REF]. There are a number of experiments that try to use this effect to observe axion-like particles and their designs are largely dependent on their respective sources of axions. Haloscopes such as ADMX and MADMAX look for axions that reside in the dark matter halo using microwave resonators immersed in a magnetic field [START_REF] Braine | Extended Search for the Invisible Axion with the Axion Dark Matter Experiment[END_REF][START_REF] Brun | A new experimental approach to probe QCD axion dark matter in the mass range above 40 µeV[END_REF]. Helioscopes such as CAST and IAXO use the sun as a source of ultra-relativistic axions that then are converted to photons in a magnetic field and detected with X-ray telescopes placed at the end of the magnet [START_REF] Anastassopoulos | New CAST limit on the axion-photon interaction[END_REF][START_REF] Armengaud | Conceptual design of the international axion observatory (IAXO)[END_REF]. In contrast to these searches, light-shining-through-a-wall (LSW) experiments take place entirely in the laboratory using a high-power laser (HPL) propagating through a magnetic field. This generates a beam of axion-like particles that travel through a lighttight wall into a second magnetic field region where some of these axion-like particles convert back to photons [START_REF] Van Bibber | Proposed experiment to produce and detect light pseudoscalars[END_REF]. The current limits on the axion-photon coupling from these different types of searches are shown as the solid blue regions in Figure 1, while limits from astronomical observations are shown as the solid green regions. axion-like-particles given in blue for the helioscopes [START_REF] Anastassopoulos | New CAST limit on the axion-photon interaction[END_REF], dark blue for the LSW experiments [START_REF] Ehret | New ALPS results on hidden-sector lightweights[END_REF][START_REF] Ballou | New exclusion limits on scalar and pseudoscalar axionlike particles from light shining through a wall[END_REF][START_REF] Valle | First results from the new PVLAS apparatus: A new limit on vacuum magnetic birefringence[END_REF], light blue for dark matter haloscopes [START_REF] Asztalos | Large-scale microwave cavity search for darkmatter axions[END_REF][START_REF] Du | Search for invisible axion dark matter with the axion dark matter experiment[END_REF][START_REF] Braine | Extended Search for the Invisible Axion with the Axion Dark Matter Experiment[END_REF][START_REF] Boutan | Piezoelectrically Tuned Multimode Cavity Search for Axion Dark Matter[END_REF][START_REF] Depanfilis | Limits on the abundance and coupling of cosmic axions at 4.5 < m a < 5.0 µeV[END_REF][START_REF] Wuensch | Results of a laboratory search for cosmic axions and other weakly coupled light particles[END_REF][START_REF] Hagmann | Results from a search for cosmic axions[END_REF][START_REF] Brubaker | First Results from a Microwave Cavity Axion Search at 24 µeV[END_REF][START_REF] Zhong | Results from phase 1 of the HAYSTAC microwave cavity axion experiment[END_REF], and green for the astronomical observations [START_REF] Beznogov | Constraints on axion-like particles and nucleon pairing in dense matter from the hot neutron star in HESS J1731-347[END_REF][START_REF] Meyer | Fermi Large Area Telescope as a Galactic Supernovae Axionscope[END_REF][START_REF] Grin | Telescope search for decaying relic axions[END_REF]. Regions of the parameter space which offer hints of their existence are shown in red [START_REF] Kim | Weak-Interaction Singlet and Strong CP Invariance[END_REF][START_REF] Shifman | Can confinement ensure natural CP invariance of strong interactions?[END_REF][START_REF] Dine | A simple solution to the strong CP problem with a harmless axion[END_REF]25,[START_REF] Cheng | Axion-photon couplings in invisible axion models[END_REF][START_REF] Meyer | First lower limits on the photon-axion-like particle coupling from very high energy gamma-ray observations[END_REF][START_REF] Giannotti | Cool WISPs for stellar cooling excesses[END_REF][START_REF] Abbott | A cosmological bound on the invisible axion[END_REF][START_REF] Dine | The Not So Harmless Axion[END_REF][START_REF] Preskill | Cosmology of the invisible axion[END_REF][START_REF] Ipser | Can Galactic Halos Be Made of Axions?[END_REF].

Please note that several exclusion limits rely on mostly untested assumptions such as axions or axion-like particles making up 100% of the dark matter in the milky way, structures of galactic and intergalactic magnetic fields, or axion and axion-like particles production in dense stellar plasmas.

It is worth emphasizing that LSW experiments make no assumptions regarding the natural prevalence of any of these particles, but merely probe the interactions themselves without the need for an external source. These experiments can therefore determine the photon-coupling strength independent of any astrophysical models, while solar searches and haloscopes, not only depend on the coupling strength, but also rely on models of the axion-flux.

The Any Light Particle Search II (ALPS II) [START_REF] Bähre | Any light particle search II-technical design report[END_REF] is a LSW experiment currently under construction at DESY in Hamburg, Germany. With DESY as the host site, the experiment takes advantage of the tunnels, magnets, and cryogenic infrastructure formerly used by the HERA accelerator. It will use 122 m long optical cavities to amplify light at a wavelength of 1064 nm that is generated on one side of the wall by a high-power laser (HPL) and by the photon regeneration process on the the PC, shown as solid red, will be at least 150 kW, while the resonant enhancement of the RC β RC , must be greater than 10,000. The coupling efficiency between the PC and RC, given by η, will be at least 0.9, with the projected spatial mode of the PC shown as the dotted purple line. A wall between the cavities prevents any light from the PC from directly entering the RC while a half wave plate (HWP) after the high power laser enables the polarization of the laser to be controlled with respect to the magnetic field. The axion like particle beam is seen as the blue line with the regenerated photon signal represented by the dotted red line that is incident on the detector at the right side of the diagram.

other side. The experiment is designed to increase the photon regeneration rate by twelve orders of magnitude over earlier LSW experiments [START_REF] Sikivie | Resonantly enhanced axion-photon regeneration[END_REF][START_REF] Redondo | Light shining through walls[END_REF][START_REF] Hoogeveen | Production and detection of light bosons using optical resonators[END_REF][START_REF] Fukuda | Production and detection of axions by using optical resonators[END_REF]. Like all LSW experiments, ALPS II will not only probe the existence of pseudo-scalar fields whose coupling to electromagnetic fields is described by Equation 1, but also scalar fields whose coupling can be described by the Lagrangian:

L s = g aγγ ϕ s ( ⃗ E 2 -⃗ B 2 ). (2)
Experimentally, while the pseudo-scalar fields from Equation 1 require the polarization of the E-field and the B-field to be parallel, scalar fields require that the polarization of the E-field is orthogonal to the B-field. A signal running in both polarization modes with no observed polarization dependence on the production rate could be detected as well. This may indicate the existence of other types of Weakly Interacting Sub-eV Particles (WISPs) that are produced by kinetic mixing such as millicharged particles or hidden sector photons which do not require a static magnetic field to interact with photons [START_REF] Gies | Polarized light propagating in a magnetic field as a probe for millicharged fermions[END_REF]. In the following, we refer to particles whose interaction strength depends on ⃗ B as axion-like particles, however the observation of any such particle would represent a profound discovery as it would be the first detection of an interaction beyond the standard model.

With a target sensitivity of 2 × 10 -11 GeV -1 , ALPS II aims to go beyond the current CAST limits by a factor of ∼3 for masses below 0.1 meV, investigating a new region of the axion-like particle parameter space that is occupied by hints from both stellar cooling excesses and the TeV transparency. In addition to this, ALPS II will also be the first model independent search in a significant region of the parameter space explored by CAST.

To achieve this, the entire optical setup including the two cavities must be tightly controlled to maintain and accurately calibrate the coupling of the generated axion field to the cavities. The following text will focus primarily on the core components that define the optical system for the experiment and discuss how we plan to reach the targeted sensitivity, while also providing a set of top level requirements for the subsystems. A complementary paper [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF], builds on the work presented here and describes the heterodyne detection scheme (HET) in more detail.

ALPS II

As mentioned earlier, the optical system for ALPS II will consist of two 122 m long, high-finesse optical cavities whose circulating fields will propagate through strings of 12 superconducting HERA dipole magnets [START_REF] Albrecht | Straightening of superconducting HERA dipoles for the any-light-particlesearch experiment ALPS II[END_REF], as shown in Figure 2. A current of 5.7 kA will flow through the 8.8 m long dipoles and produce a magnetic field of 5.3 T giving a product of the magnetic field and length of B 0 L = 560 T•m on each side of the wall. On the left side of the wall, light with a wavelength of 1064 nm from the HPL will be resonant with the production cavity (PC), building up the stored energy due to the high cavity finesse. This light will generate axion-like particles, shown as the transparent blue line in Figure 2, with an identical energy and spatial mode, outlined by the purple dotted line.

These particles pass through the light-tight barrier on the central optical bench (COB) before they enter the regeneration cavity (RC) where they convert back to photons, shown as the dotted red line. The regenerated photon rate,

n reg = η 16 (g aγγ F (qL B )B 0 L B ) 4 P PC hν β RC , (3)
scales with (g aγγ B 0 L) 4 and is proportional to the power inside the PC in terms of photons per second, P PC /hν, and the resonant enhancement β RC of the RC [START_REF] Arias | Optimizing light-shining-through-a-wall experiments for axion and other weakly interacting slim particle searches[END_REF]. The form factor can be approximated by the following equation with L B representing the 106 m length of each magnet string.

|F (qL B )| ≈ 2 qL B sin qL B 2 q = m 2 a 2hν (4)
This is a typical phase matching condition which accounts for the possible mass m a , of the relativisitic axion-like particles. For masses m a < 0.1 meV, the form factor is essentially unity in ALPS II. For a more detailed discussion on this please see [START_REF] Arias | Optimizing light-shining-through-a-wall experiments for axion and other weakly interacting slim particle searches[END_REF].

The coupling efficiency η between the relativistic axion field and the eigenmode of the RC takes into account all transversal and spectral mismatches between the axion mode, which is identical to the PC eigenmode, and the eigenmode of the RC. Here η is given in terms of axion to photon coupling and therefore the axion field to electromagnetic field coupling would be given by √ η. It will be possible to verify η before and after measurement runs by opening a shutter in the light tight barrier and allowing the PC transmitted field to couple to RC. provides a sensitivity gain of ∼ 25 in g aγγ when compared to ALPS I, TLR 1, requiring a PC internal power of 150 kW, and TLR 4, requiring an RC resonant enhancement β RC > 10 000, together increase the sensitivity of the experiment by a factor of ∼ 40, demonstrating the importance of the optical system. Achieving both of these requirements depends on the coatings and surface roughnesses of the cavity mirrors as well as clipping losses in the magnet strings. This will be discussed further in section 2. It should be noted TLR 4 is not far from the limits of what is possible for mirrors of these dimensions with state of the art polishing techniques. TLR 3 refers to the coupling of the axion field to the RC and is discussed in section 3. TLR 5 not only establishes requirements on the sensitivity of the detection systems, but can also be used to set the maximum permissible background signal due to stray light coupling from the PC to the detectors.

The first science run will be a search based on the above listed parameters that could set an upper limit of g αγγ = 2.8 × 10 -11 /GeV corresponding to an upper limit on the regenerated photon rate of 2.8 × 10 -5 /s or roughly 2.4 photons per 24 h of valid data. This search will be followed by a scalar particle search at the same sensitivity by changing the polarization (TLR 2). We will then attempt to improve the sensitivity by increasing the PC circulating power, the RC resonant enhancement, and the duty cycle to aim for g αγγ = 2 × 10 -11 /GeV or better [START_REF] Põld | ALPS II design requirements document[END_REF] for pseudo-scalar and scalar particles.

Detection Systems

ALPS II will have the benefit of using two independent detection systems, each with very different systematic uncertainties, to measure the reconverted photons. While two different detector types are not explicitly needed, this approach will help increase confidence that signals observed with the same strength in both detectors, while also above the measured backgrounds, are indeed the result of photon-axion conversion-reconversion process. The detectors themselves require different optical systems in order to be operated and therefore cannot be used in parallel. Working at a wavelength of 1064 nm implies that free-space, off-the-shelf, silicon photodetectors have plenty of sensitivity to be used as detectors in the control system and also for the ALPS II heterodyne readout. Only the single-photon counting approach requires more sensitive detectors, which are under development.

The first detection scheme to be implemented will be the HET, described in an accompanying paper in this journal [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF]. The HET measures the interference beatnote between a laser, referred to as the local oscillator (LO), and the regenerated photon field on a photodetector. Demodulating the electronic signal from this photodetector at the known difference frequency will create a signal proportional to the regenerated field strength that can be integrated over the measurement time τ .

The regenerated photon signal will thus accumulate proportional to τ while the background signal from laser shot noise will sum incoherently and thus be proportional to √ τ [START_REF] Bush | Coherent detection of ultraweak electromagnetic fields[END_REF].

A transition edge sensor (TES) will be used in the second detection system [START_REF] Dreyling-Eschweiler | Characterization, 1064 nm photon signals and background events of a tungsten TES detector for the ALPS experiment[END_REF]. In contrast to the HET system based on interference effects, the TES system will count individual photons.

The TES consists of an absorptive tungsten film which is held at a temperature at the threshold of superconductivity. When a photon is absorbed by the tungsten it will lead to a slight increase in its temperature. This will suddenly raise the resistance of the chip causing a drop in the bias current that is flowing through it. This current drop can be measured over an inductive coil with a superconducting quantum interference device. Therefore, the reconverted photons can be individually counted as these pulses occur, with an energy resolution of ∼ 7%. A laboratory prototype TES currently triggers at about 0.8 eV, but this is not necessarily the final configuration. Saturation occurs at energies well above 2.3 eV (wavelengths below 532 nm), and can be tuned with the TES working point. A forthcoming paper will offer more details about the TES detection system, including details of the involved optical fiber system as well as the interplay between the 1.17 eV energy of the reconverted photons and the TES system.

ALPS II Cavities

Both the PC and RC will be plano-concave cavities with a stability parameter g = 0.43. The curved mirrors will be located at the end stations of the experiment while the flat mirrors will be at the center as shown in Figure 2. The radius of curvature of the mirrors at the end stations were chosen such that the Rayleigh lengths of the cavity eigenmodes, z R , are equal to the length of their corresponding magnet strings. This geometry will help minimize aperture losses in the cavities while also avoiding higher order mode degeneracies that would occur if the cavities were exactly half-confocal. The configuration also ensures that the eigenmodes of both cavities can have a high spatial overlap as the nominally identical Gaussian beam waists are located on the flat mirrors. The distance between the flat cavity mirrors (< 1 m) is much smaller than the Rayleigh length (∼ 106 m) of the modes rendering the resulting loss in coupling negligible compared to other contributions.

The cavity eigenmodes will need to be centered within the beam tube of the magnet string to reduce clipping losses. The diameter of the magnet aperture is nominally 55 mm, however since the magnets were originally used to steer protons around the arcs of the HERA accelerator, their central axis followed a curvature of 600 m and therefore required straightening. This process was very successful, and expanded the free apertures of the magnets from ∼37 mm to between 46 and 51 mm [START_REF] Albrecht | Straightening of superconducting HERA dipoles for the any-light-particlesearch experiment ALPS II[END_REF]. The largest free aperture magnets will be used near the end stations where the beam size and risk of clipping losses is the highest.

The magnets along with the rest of the vacuum system that will house the cavities are now aligned within ± 200 µm and ± 1 µrad of a line defining the theoretical optical axis of the experiment. The cavity mirrors will then be placed within ± 1 mm and ± 8 µrad of the resulting central line of the combined magnet string, reducing clipping losses inside the two cavities to below 1 ppm.

Regeneration cavity

The resonant enhancement provided by the regeneration cavity:

β RC ≈ 4T out (T out + T b + ρ) 2 , (5)
is very similar to the power build-up ratio between the input power and the maximum circulating power for a generic optical cavity. In this case, it expresses the gain in signal power at the detector due to the amplification of the cavity. As Equation 5shows, it depends on the losses and transmissivities of each mirror. Here T out is the transmissivity of the mirror located nearest to the main regenerated photon detector while T b is the transmissivity of the other cavity mirror. Ideally, the maximum resonant enhancement occurs when the RC has minimal round-trip losses ρ, and the mirror transmissivities are as low as possible and the cavity is in the so-called 'impedance matched' configuration where

T out = T b + ρ.
Losses in our cavities are expected to be dominated by the surface roughness of the mirrors and the associated scattering of light. Simulations using measured surface maps of the actual mirror substrates predict that the scatter losses inside the cavity will likely be between 40 and 60 ppm per round-trip. Operating the RC in an 'over-coupled configuration' (T out > T b + ρ) will simplify the HET detection scheme as it will allow for a more stable power in the LO at the science photodetector [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF]. Therefore, to be conservative, we decided to use 100 ppm as the design value for T out for the initial science run. T b will then be 5 ppm since the HET also requires some nonzero transmission for this mirror to realize the sensing and control scheme. This is discussed in more detail in Section 3.

The dielectric cavity mirror coatings consist of alternating λ/4 layers of silica and tantala to minimize the absorptive losses; note that the same coatings will also be used in the PC where laser beam absorption will lead to thermal distortions of the cavity eigenmode. These mirrors were received and measured to have transmissivities of ∼ 110 ppm and ∼ 6.7 ppm at normal incidence which results in an expected resonant enhancement of

β RC ≈ 16 000 ± 2 000 ( 6
)
given the expected scattering losses and their corresponding uncertainty.

Production cavity

The PC will be seeded with the HPL, a linear polarized laser beam from a single-frequency, low-noise laser system operating at 1064 nm capable of injecting up to 50 W to the cavity. The HPL is based on a master-oscillator power-amplifier configuration (MOPA) with an ultra stable Nd:YAG seed laser in non-planar ring-oscillator configuration [START_REF] Kane | Monolithic, unidirectional single-mode Nd:YAG ring laser[END_REF] and a low-noise Nd:YVO 4 amplifier [START_REF] Frede | Fundamental mode, single-frequency laser amplifier for gravitational wave detectors[END_REF] with a high spatial purity. The challenge for the laser design is to generate a high power beam with small fluctuations and drifts that allows for a highly efficient coupling of the light into the PC. In this regard the chosen MOPA design is advantageous as it preserves the high frequency stability of the seed laser and has, compared to an oscillator, relatively low intensity levels in the gain material such that thermally induced wave front distortions and depolarization effects can be minimized.

Light from the HPL will then be incident on the PC. The PC will use mirrors with identical dielectric coatings as the RC mirrors. Therefore, the PC power build-up factor is also expected to be 16 000 ± 2 000, the same as the resonant enhancement of the RC. A frequency stabilization system with a unity gain frequency of ∼ 300 kHz will maintain the resonance of the laser with respect to the length of the PC using the standard Pound-Drever-Hall (PDH) technique [START_REF] Drever | Laser phase and frequency stabilization using an optical resonator[END_REF][START_REF] Black | An introduction to Pound-Drever-Hall laser frequency stabilization[END_REF].

The sensing scheme and the loop gain are expected to keep the laser frequency within a hundredth of the half-width-half-maximum (HWHM) of the cavity resonance corresponding to a relative power noise inside the cavity to a RMS value <100 ppm.

The input optics between the high-power laser (HPL) and the PC will also be equipped with an automatic alignment system based on a differential wavefront sensing (DWS) scheme. This system uses a pair of quadrant photodetectors (QPDs) which measure the lateral shift and angular offset between the laser mode and the cavity eigenmode [START_REF] Morrison | Automatic alignment of optical interferometers[END_REF]. These signals are then fed back to a pair of actuators to maintain the alignment of the laser into the cavity. The goal is to also limit the RMS relative power noise inside the cavity due to alignment fluctuations to <100 ppm for each degree of freedom. The entire system will guarantee that the total relative power noise stays below 0.1% RMS, which should be sufficient to reduce the impact of dynamic thermal effects on the HET [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF]. The input optics for the PC will also employ a half-waveplate to rotate the polarization of the circulating field with respect to the polarity of the magnet string. This will satisfy TLR2 and allow the experiment to search for either scalar or pseudo-scalar particles.

This combination of the HPL and cavity finesse may allow powers as high as 1 MW inside the PC, however, the final power level will likely be limited by the absorption in the highly-reflective coating layers of the two cavity mirrors. There are a number of ways this absorbed light could lead to thermal effects that cause higher intracavity losses. For example, point absorbers heating up on the surface of the mirror could cause the formation of low spatial frequency features which, in turn, leads to an increase in the scattering losses [START_REF] Buikema | Sensitivity and performance of the Advanced LIGO detectors in the third observing run[END_REF][START_REF] Glover | Optical scattering measurements and implications on thermal noise in gravitational wave detectors test-mass coatings[END_REF]. Absorption in the mirror coatings could also cause the size of the mode circulating in the PC to change and lead to additional clipping loses from the beam tube [START_REF] Winkler | Heating by optical absorption and the performance of interferometric gravitational-wave detectors[END_REF]. The loss in sensitivity due to the mode mismatch between the cavity eigenmodes as the PC mirrors heat up is expected to be insignificant in comparison. Nevertheless, 150 kW with the beam size on the PC mirror is roughly equivalent to the peak circulating intensity in the aLIGO pre-mode cleaner and is therefore expected to be attainable [START_REF] Kwee | Stabilized high-power laser system for the gravitational wave detector advanced LIGO[END_REF].

Maintaining the Axion Field Coupling to the RC

The primary obstacles to optimizing the coupling of the axion field to the RC are related to maintaining its coherence and spatial mode matching with respect to the RC eigenmode. These parameters will depend on the residual changes of the frequency and spatial mode of the PC circulating field with respect to the RC. As Table 3 shows, we allow each to contribute a 5% loss of the signal to meet the 90% coupling efficiency listed under TLR 3. Here we can also see that each of these requirements can be further subdivided into requirements on specific parameters of the optical system. The following sections discuss how each of these requirements are derived and how the optical system will work to satisfy them.

Coherence of the PC field with the RC

Maintaining the coherence between the electromagnetic field regenerated from the axion beam and the RC eigenmode is critical to ALPS II achieving its target sensitivity. Therefore, the optical system is designed such that the regenerated field should experience no more than 5% average reduction from its optimal resonant enhancement over the duration of the measurement due to its frequency noise with respect to the RC resonance frequency. This requirement is further divided into one on static frequency offset, denoted by ∆f , and one on the dynamic phase noise, denoted by ∆ϕ SD . As the regenerated field is a replica of the field circulating in the PC, the first challenge is to accurately tune the frequency of the PC transmitted field to be resonant with the RC. The second challenge is to precisely control the phase of the PC transmitted field around this nominal value.

PC Tuning

The signal loss in regenerated photons due to a small offset of ∆f in the frequency of the PC transmitted light relative to a resonance frequency of the RC is quadratic in the offset and can be approximated by the following expression.

1 -η ∆f ≈ ∆f HWHM 2 (7)
Here HWHM is the half-width half-max linewidth of the RC. To limit the loss of regenerated photons to 1%, we require that the detuning is less than 10% of the HWHM or less than 1.5 Hz during the science run.

As Figures 3 shows, the optical system when using the HET will control the frequency of the PC transmitted field, ν PC , via a series of offset phase lock loops (PLLs). This will allow ν PC to track the changes in the resonance frequency of the RC due to the presence of environmental noise. To do this the LO is stabilized to the length of the RC with a high bandwidth control loop (>300 kHz) ensuring that the frequency of the transmitted LO field is equivalent to the RC resonance ν RC . The phase of the transmitted LO field, ϕ RC , will therefore be encoded with the RC length noise. An auxiliary laser, referred to as the reference laser (RL), will then be phase locked to ν RC with some offset f 0 giving it a frequency ν RL = ν RC + f 0 . By setting f 0 ̸ = F SR RC , the RL will not be resonant in the RC. Finally, since the HPL is frequency stabilized to the length of the PC, an offset phase lock loop between RL and the PC transmitted field will be established to suppress the dynamic phase noise δϕ PC -δϕ RC (as discussed in Section 3.1.2). This gives ν PC the following value with any unintended offsets expressed as ∆f :

ν PC = ν RL -f 0 + N • F SR RC + ∆f = ν RC + N • FSR RC + ∆f. ( 8
)
With ∆f sufficiently small, the PC circulating field should be offset in frequency from ν RC by some integer multiple of FSR RC , N , thus making it resonant in the RC. ∆f will be minimized tuning the shutter open. During the science runs these values must be maintained when the shutter is closed [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF]. This approach requires that the source for the offset frequency and the FSR of the RC are both stable.

ϕ RC = RC t + ϕ RC (t) ϕ = ϕ PC -ϕ RC PC = RC + N•FSR RC ϕ PC = PC t + ϕ PC (t) RL = RC + f 0 RC N•FSR RC -f g
The RL-LO beatnote frequency, f 0 , will be derived from a clock that is synchronized to a 10 MHz rubidium frequency standard with a yearly frequency drift on the order of mHz, well below our requirement. However, macroscopic changes of the length of the RC will change the optimum offset frequency by:

∆f ∆FSR = N • ∆FSR RC = N • FSR RC ∆L RC L RC (9)
The length changes of the RC then have to be

∆L RC < ∆f ∆FSR FSR RC L RC N = 150 µm N ( 10
)
between retuning measurements to ensure that ∆f < 1.5 Hz. A FSR sensing system that uses a modified PDH sensing technique which uses phase modulated sidebands at some multiple (̸ = N ) of the FSR will also be implemented [START_REF] Thorpe | Laser frequency stabilization and control through offset sideband locking to optical cavities[END_REF]. With this ∆L will be measured continuously during the science run and if it becomes larger than 150 µm/N the run will be paused and the length of the RC will be adjusted back to its initial value before it is started again. Options to actively control the length of the RC during measurement runs are also being evaluated.

Phase lock of the PC

The series of PLLs mentioned in the previous section will also be used to reduce the frequency or phase fluctuations δϕ(t), of the PC transmitted field relative to the resonant frequency of the RC. This system must provide the precision necessary to meet the requirements on the coherence.

Phase noise will spread the energy of the ideally monochromatic PC transmitted field over a finite frequency band and only the frequency components which are resonant in the RC will contribute to the signal. The energy in all frequency components outside the line-width of the RC will be attenuated.

We require that the power integrated over all frequency components outside this bandwidth is less than 4% of the total power. This requirement roughly translates into an upper limit for the standard deviation (SD) of the phase noise evaluated over the storage time T of the cavity of [START_REF] Põld | Demonstration of a length control system for ALPS II with a high finesse 9.2 m cavity[END_REF]:

∆ϕ SD (t) ≈ ⟨δϕ 2 (t)⟩ T < 0.2 rad (11)
which the phase lock loop between the PC transmitted field and the reference field must achieve.

Due to the high gain and fast bandwidth (∼ 300 kHz) of the control loop that stabilizes the HPL to the PC, the phase of the PC transmitted light should be almost entirely determined by the PC

High Power Laser Production Cavity

Reference Laser length. The PLL must then act on the length of the PC such that it follows all length changes of the RC which are impressed on the phase of the reference laser as shown in Figure 3.

PD P COB Regeneration Cavity PD R ϕ RC = RC t + ϕ RC (t) ϕ = ϕ PC -ϕ RC / 2 PC = RC / 2 -f g + N•FSR RC ϕ PC = PC t + ϕ PC (t) RL = RC / 2 RC N•FSR RC -f 0 ' ' ' '
For this purpose, we developed a piezo-electric actuated mirror mount for the 2" diameter PC input mirror that supports a control bandwidth of 4 kHz. Based on seismic measurements in the HERA tunnel, this bandwidth, paired with an aggressive gain function is expected to be sufficient to suppress the environmental noise [START_REF] Põld | Demonstration of a length control system for ALPS II with a high finesse 9.2 m cavity[END_REF].

Maintaining coherence with the TES optical system

The TES optical system is also designed to maintain the resonance of the PC circulating field with respect to the RC, but has slightly different constraints due to the nature of the detector. While the HET is capable of distinguishing fields with sub-Hz frequency differences [START_REF] Bush | Coherent detection of ultraweak electromagnetic fields[END_REF], the presence of these signals at the detector would saturate the TES, in addition to creating backgrounds indistinguishable from the regenerated field. Therefore, the frequency of any field that may be incident on the TES must be different enough from ν PC such that it can be sufficiently suppressed by optical filters and can also be distinguished from the signal field during data processing. For this reason the optical system for the TES will appear as seen in Figure 4. Here the RL is frequency doubled from 1064 nm to a wavelength of 532 nm and the laser is then frequency stabilized to the RC length with the green light coupled to the cavity. With this ν RL = ν ′ RC /2, where ν ′ RC is the resonance frequency of the RC for 532 nm. This allows the green light to be filtered out of the path to the detector (not shown).

Furthermore, the TES is capable of differentiating between photons with wavelengths of 532 nm and 1064 nm.

As is the case in the HET optical system, the PC transmitted field is then offset phase locked to the 1064 nm light from RL by actuating on the length of the PC. The frequency of the PC transmitted field is therefore:

ν PC = ν RL -f g + N • F SR RC = ν ′ RC 2 -f g + N • F SR RC (12)
In this equation f g accounts for the relative frequency offset between the green and IR resonances due to the difference in the reflected phase between the two wavelengths. To ensure that the PC circulating field is resonant in the RC, the beatnote frequency, ν PC -ν RL , must be tuned for either an even or odd number integer multiple of green wavelengths in the RC because of f g . If the RC length for green is an odd number wavelengths of the 532 nm frequency double light from RL and the beatnote frequency is set for an even number of wavelengths, then ν PC will not be resonant with the RC length. Like the optical system for HET, this tuning will be optimized with the shutter open, by maximizing the power of the PC transmitted light that couples to the RC.

In Figure 4 we can see that, like the optical system for the HET, the PLL between RL and the light transmitted by the PC is used to control its length by feeding back to a piezo-electric actuator at the curved mirror. Other than the differences discussed in this section, the optical system for the TES adheres to the same requirements and uses the same methods as those that are used for the HET optical system and discussed in Sections 3.1.1 and 3.1.2.

Maintaining Spatial Coupling

Another effect which could lead to a loss in sensitivity is related to the alignment of the axion field into the RC. Any difference in the angular alignment or transversal position between the incoming axion field and the optical axis of the cavity eigenmode will reduce the coupling of the axion field to the cavity and lead to a decrease in the regenerated photon rate. To be clear, this will not lead to any misalignment between the detection system and the resonant spatial part of the reconverted field that is amplified in the RC, as its spatial mode will be in the eigenmode of the RC, which is itself aligned to the detectors.

The spatial mode of the axion field entering the RC is an extension of the spatial mode of the field inside the PC and the loss in coupling due to small alignment errors can be calculated from the following equation:

1 -η TM ≈ ∆x w 0 2 + ∆θ θ Div 2 ( 13
)
where ∆x is the transversal shift and ∆θ is the angular misalignment between the two modes measured at the waist of the RC [START_REF] Mueller | Detailed design of a resonantly enhanced axion-photon regeneration experiment[END_REF][START_REF] Mueller | Resonantly-enhanced axion-photon regeneration[END_REF]. The power loss is quadratic in both terms and required to be less than 5% in total. With a waist size of w 0 = 6 mm, our requirements of ∆x < 1.2 mm correspond to a loss in the mode matching of 4% due to transversal shifts. Since the divergence angle of the cavities is θ Div = 57 µrad, the requirements of ∆θ < 5.7 µrad are equivalent to a 1% loss in the mode matching. The systems that will control the alignment of the cavity mirrors are discussed in the following sections.

Central Mirror Alignment

As the left side of Figure 5 shows, due to the geometry of the cavities, the angular misalignment between their eigenmodes will be equal to the relative misalignment of the flat mirrors at the center of the experiment. Therefore the alignment of these mirrors must be maintained to better than 5.7 µrad to meet our requirements. To accomplish this, both mirrors are mounted to the central optical bench (COB), which is effectively a large stable surface designed for this purpose. It uses no active alignment system and the system instead relies on its passive stability. A system for monitoring the relative alignment drift between the mirrors on the COB for the HET optical system is detailed in [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF], but not shown in Figure 5.

The COB is constructed from a single aluminum plate on which all mirrors, beam splitters, and waveplates, as well as position sensors are either mounted directly or through additional ULE base plates [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF] using ultra-stable optical mounts. Tests with an autocollimator have shown that a prototype COB was capable of maintaining a long term alignment stability of 2 µrad over one week in air with measured thermal alignment coefficients of ≈ 4 µrad/K in pitch and ≈ 1 µrad/K in yaw [START_REF] Kulkarni | Ultrastable optical components using adjustable commercial mirror mounts anchored in a ULE spacer[END_REF][START_REF] Wei | Optics mounting and alignment for the central optical bench of the dual cavity enhanced light-shining-through-a-wall experiment ALPS II[END_REF]. The air conditioning system of the cleanroom has been designed to maintain an 0.1 K absolute temperature stability which would, in principle, eliminate any relevant misalignment.

However, the impact of the heating of mirror PC 2 by the cavity internal field on the alignment still needs to be evaluated during the commissioning of the experiment.

End Mirror Alignment

Transversal shifts in the relative positions of the eigenmodes will be determined by the relative alignment of the curved mirrors with respect to the flat mirrors on the COB, (∆θ M ). In this case, the coupling is determined by the ROC of the curved mirrors, R.

∆x = R∆θ M (14)
We should note here that ∆θ M is not the same as the misalignment given by the cavity eigenmodes ∆θ. With a radius of curvature in the curved mirrors of 214 m, the relative alignment between the curved and flat mirrors for each cavity must be stabilized to better than 5.6 µrad to meet our requirements of ∆x < 1.2 mm. As we can see, this is quite similar to the requirements on the alignment of the flat mirrors on the COB.

As Figure 5 shows, this is done using two in-vacuum QPDs housed on the COB to monitor the position of the cavity eigenmodes by performing DC differential measurements of the light incident on their quandrants. These QPDs are optimized to sense the position of the 6 mm radius beam with sub 100 µm precision [START_REF] Wei | Optics mounting and alignment for the central optical bench of the dual cavity enhanced light-shining-through-a-wall experiment ALPS II[END_REF]. Assuming that the components on the COB remain stationary, the positions of the eigenmodes on the flat cavity mirrors PC 2 and RC 1 , and in extension on the two QPDs, only depend on the orientations of the curved cavity mirrors PC 1 and RC 2 , respectively. The differential signals from the QPDs will be fed back to active alignment stages that are capable of controlling the pitch and yaw of the curved cavity end mirrors.

As we will see in the next section, due to the presence of wedge angles in the optics in between the cavities mirrors on the COB, the eigenmodes will have some static transversal shift between them. Nevertheless, this is expected to be below 200 µm. When including the < 3 µrad offsets in the COB cavity mirrors the cumulative mode matching losses due to the static misalignments should be:

1 -η TM ≈ ∆x w 0 2 + ∆θ θ Div 2 < 0.5%. ( 15
)
This is on the same order as the matching between the axion field and the PC transmitted field after traversing the COB optics, leaving significant margin for systematic errors and drifts.

Quantifying the transversal matching

Like the tuning of PLL offset frequency, the alignment of the cavity eigenmodes will be quantified using the PC transmitted field when the shutter is open. This quantification has a systematic error due to the refraction of the beams that pass through the optical components located between the cavity internal fields. The substrates of the cavity end mirrors have known wedge angles θ W between 3 and 4 µrad which will refract the PC transmitted field, but not the axion field. By clocking the two substrates correctly to minimize their differential wedge angle ∆θ W , the refraction angles will compensate each other such that the final angular refraction is:

θ refr = (n -1)∆θ W < 1 µrad (16)
The refraction in PC 2 will laterally shift the PC transmitted beam by ∆x < 4 µm, well below the level of the other sources of alignment noise.

The optics in between the cavity mirrors are also made from substrates with known wedge angles between 2 and 5 µrad and will also be clocked to reduce the overall deflection to below 2 µrad. The total deflection angle between the beams will therefore be below 3 µrad.

Each substrate in between the cavity mirrors will laterally shift the beam off of the optical axis of the system by:

y = d sin(θ 1 -θ 2 ) cos θ 2 sin θ 1 = n sin θ 2 ( 17
)
where d is the thickness of the substrate, θ 1 the angle of incidence, θ 2 the angle inside the material, and n the index of refraction. To cancel the cumulative effect, the number of substrates which shift the beam to the left has to be equal to the number of substrates which shift the beam to the right, assuming that the substrates are equal in thickness and in material, and that the angle of incidence is the same. The COB design [START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF] uses two substrates that shift the beam left and two that shift it right; all at 35 • angle of incidence and all made from fused silica. According to the vendor, the substrates are 9.5 mm thick with a tolerance of [+0, -0.5 mm]. This results in a worst case lateral shift of ∼ 200 µm.

Based on these numbers, the uncertainty in the resulting mode mismatch between the PC transmitted field and the axion field for both degrees of freedom will be below 0.4% which will allow us to use the PC transmitted field to verify that the overall alignment of the axion mode into the RC is better than 95%.

Summary and Conclusion

ALPS II is designed to be the most sensitive LSW experiment to date with a regenerated photon rate more than 12 orders of magnitude larger than previous experiments. The innovations in the optical system alone account for more than 6 orders of magnitude of the increase in the regenerated signal while the additional boost will come from the long magnetic field length and improvements in detector technologies [START_REF] Albrecht | Straightening of superconducting HERA dipoles for the any-light-particlesearch experiment ALPS II[END_REF][START_REF] Dreyling-Eschweiler | Characterization, 1064 nm photon signals and background events of a tungsten TES detector for the ALPS experiment[END_REF][START_REF] Hallal | The heterodyne sensing system for the ALPS II search for sub-ev weakly interacting particles[END_REF]. Two 122 m, high-finesse optical cavities are largely responsible for the gains from optical system. The use of these cavities, however, presents a unique set of challenges related to maintaining their coherence and spatial overlap. In this paper we have described the core components of ALPS II, and how the optical system is designed to address these challenges.

We will take great care that the spatial mode of the transmitted light from the PC that is incident on the RC is an accurate representation of the axion mode. This will allow us to quantify the coherence and mode matching of the axion field with respect to the RC. The length and alignment sensing system for the lasers and the cavities is based on PDH and DWS, well established phase sensing schemes with sufficient sensitivity to monitor all relevant degrees of freedom. Additionally, we developed and tested different actuators which should have enough range and bandwidth to operate ALPS II in the HERA tunnels.

We also discussed our plans to employ two different schemes to detect the regenerated photon signal. The first one is the HET and is described in detail in an accompanying paper, while the second scheme uses a TES and will be implemented following the HET science runs. The experiment itself is presently under construction and aiming for a first science run in 2021. Once fully operational, the optical system should allow ALPS II to be able to detect axions with a 95% confidence level for coupling constants as low as g aγγ = 2 × 10 -11 /GeV using 20 days of valid science data.

Figure 1 :

 1 Figure 1: ALPS II target sensitivity, shown in the transparent blue region, along with current exclusion limits for

Figure 2 :

 2 Figure 2: Experimental layout of ALPS II. The circulating fields in each cavity propagate through 560 T • m of the magnetic field times length product, with its length shown as L B and the field strength given by B 0 . The power inside

Figure 3 :

 3 Figure 3: Layout of the cavities and control architecture for maintaining the phase lock of the PC in the HET optical system.

Figure 4 :

 4 Figure 4: Layout of the cavities and control architecture for maintaining the phase lock of the PC in the TES optical system.

Figure 5 :

 5 Figure 5: Control architecture for maintaining spatial overlap. The QPDs on the COB monitor the position of the cavity eigenmodes on the cavity mirrors PC 2 and RC 1 . These signals are used to align the cavity mirrors at the end stations PC 1 and RC 2 . The aligned configuration is shown as the solid image in the foreground, while examples of the potential alignment errors are shown as transparent images in the background.

Table 1 :

 1 Top level requirements for the first ALPS II science run targeting a sensitivity of gαγγ = 2.8 × 10 -11 GeV -1

	(adapted from [44]).
		Requirement
	TLR1	> 150 kW power circulating in PC (fundamental mode, linearly polarized, 1064 nm)
	TLR2	Parallel and perpendicular polarization adjustment possibility with respect to the
		magnetic field
	TLR3	Coupling between the axion mode and the RC fundamental mode: η > 90% (power
		ratio)
	TLR4	RC resonant enhancement β RC > 10 000
	TLR5	Detector sensitive enough to exclude a reconverted photon rate > 2.8 × 10 -5 /s with
		a 95% confidence level within 20 days
	TLR6	Magnetic field × length product of 560 T•m for PC and RC magnet string

Table 1

 1

lists the top level requirements (TLR) of ALPS II. While the long magnet string

(TLR 6)

Table 2 :

 2 Parameters of the ALPS II cavities

	Parameter (symbol/acronym)	Value
	Length (L PC/RC )	122 m
	Free spectral range (FSR)	1.2 MHz
	Half-width-half-maximum (HWHM)	15 ± 2.5 Hz
	End mirror radius of curvature (R)	214 ± 6 m
	1/e 2 waist radius (w 0 )	6.0 mm
	1/e 2 end mirror beam radius (w m )	9.2 mm
	Divergence half angle (θ Div )	57 µrad
	RC resonant enhancement (β)	16 000 ± 2 000

Table 3 :

 3 Requirements for fulfilling TLR3

	Parameter (symbol)	Requirement
	Axion coupling to RC (η)	>90%
	Coherence (η ∆f )	>95%
	dynamic phase noise (∆ϕ SD )	<0.2 rad
	static frequency offset (∆f )	<1.5 Hz
	Spatial overlap (η T )	>95%
	Angular alignment (∆θ)	<5.7 µrad
	Transversal shift (∆x)	<1.2 mm

Acknowledgments

The work is supported by the Deutsche Forschungsgemeinschaft [grant number WI 1643/2-1], by the German Volkswagen Stiftung, the National Science Foundation [grant numbers PHY-2110705 and PHY-1802006], the Heising Simons Foundation [grant numbers 2015-154 and 2020-1841], and the UK Science and Technologies Facilities Council [grant number ST/T006331/1].


Rokiatou Diarra Satie
email: rokiatou.diarra@u-psud.fr

Alain Merigot
email: alain.merigot@u-psud.fr

Bastien Vincke
email: bastien.vincke@u-psud.fr

Exploring performance improvement opportunities in directive-based GPU programming

Keywords: GPU, OpenMP, OpenACC, CUDA

GPUs offer an impressive computing power, but their effective exploitation remains an open problem. Kernel based programming models such as CUDA or OpenCL, allow a direct programming of the GPU architecture and can drive to excellent performance. However, these programming models require significant code changes, often tedious and errorprone, before getting an optimized program. Directive-based programming models (such as OpenMP and OpenACC) are now available for GPU and can offer good trade-off between performance, portability and development cost. In this paper, we do a comparative performance study between OpenACC, OpenMP 4.5 and CUDA, which is essential for facilitating parallel programming for GPUs. In order to find most significant performance issues, we port a suite of representative benchmarks and three computer vision applications to OpenACC, OpenMP and CUDA. Beyond runtime, we explore factors that influence performance, such as register counts, workload, grid and block sizes. The results of this work show that either OpenACC or OpenMP are good alternatives to kernel based programming models, provided some careful manual optimization is performed. Through the analysis of generated PTXs files, we discover that there is in general a systematic overhead in the kernel launch for OpenMP, but, for most applications, it is not a big issue provided the kernel has a sufficient workload.

I. INTRODUCTION

Heterogeneous programming has become a reality with the omnipresence of accelerators, such as the graphics processing unit (GPU) and Xeon Phi, in current architectures. GPUS can achieve significant performance for certain categories of application (e.g.: computer vision algorithms, dense linear algebra). Nevertheless, achieving this performance is not only dependent on an important effort of programming and code tuning, but also a good knowledge of GPUS architecture. Thereby, porting sequential applications on these systems requires large efforts of rewriting. Kernel based languages such as CUDA or OpenCL are well suited to GPUS. They offer a number of features for performance optimization as the architecture is directly accessible to the user that makes possible to obtain excellent performance but adds complexities for application developers.

Directive-based programming models may therefore become an interesting solution. GPUS programming using directives is an alternative to kernel based language (CUDA and OpenCL). Because of their ease of use, directive-based programming models can offer a good trade-off between productivity, portability and performance. However, such programming strategies impose technical challenges on compiler optimizations, which could result in lower performance than with CUDA or OpenCL. With standards such as OpenACC and OpenMP 4.0/4.5, programmers can easily implement an accelerated code by adding compiler directives in their sequential code and architecture management being left to the compiler. Each API has its own execution model, which is intentionally abstract to avoid coupling it to the specificities of any device.

Although many works [START_REF] Hoshino | Cuda vs openacc: Performance case studies with kernel benchmarks and a memory-bound cfd application[END_REF]- [START_REF] Ikeda | An openacc optimizer for accelerating histogram computation on a gpu[END_REF] have been done on GPUS programming with OpenACC and OpenMP since their release, we shall demonstrate that, despite all the attention this topic has received, current compilers still need improvement in order to generate codes that can be equal to the performance of a hand-optimized CUDA code. We make the following contributions:

• The configuration of OpenMP and OpenACC directives affects the overall performance of the application. We test different combinations of directives to show their impact on performance. We show also that a good comprehension of OpenACC and OpenMP offloading model is necessary on behalf of the programmer in order to help compilers efficiently parallelize. Section III and Section V presents an extensive performance evaluation and analysis. Section VI summarize concluding remarks.

II. BACKGROUND

A. CUDA

CUDA is a parallel computing programming model that fully utilizes hardware architecture and software algorithms to accelerate various types of computation [START_REF] Li | Comparing programmer productivity in openacc and cuda: An empirical investigation[END_REF], [START_REF] Han | hicuda: High-level gpgpu programming[END_REF]. In CUDA, the programmer writes device code in functions called kernels. A kernel will be executed by many GPU threads. Before launching the kernel on the GPU the user must specify the number of threads, by setting grid and block sizes, that will execute the kernel. A grid consists of multiple thread blocks and each block contains several threads. During kernel execution, threads have access to different types of memory on the GPU. Each type of memory has its properties such as access latency, address space, scope, and lifetime. Before launching kernels, data must be transferred to GPU memories. To obtain optimized code, the programmer must understand well not only GPU architecture, but also CUDA optimization strategies like memory-coalescing access, efficient usage of shared memory or tiling technology. Additionally, grid and block configurations, computing behaviors of each thread, and synchronization problems need to be carefully tuned [START_REF] Li | Comparing programmer productivity in openacc and cuda: An empirical investigation[END_REF], [START_REF] Ryoo | Optimization principles and application performance evaluation of a multithreaded gpu using cuda[END_REF].

B. OpenMP

OpenMP is undoubtedly the most used standards for several years in the parallel programming for shared memory CPUs [10]. OpenMP 4.0 and 4.5 extended the OpenMP shared memory programming model with the introduction of device constructs to support accelerators. In order to offload a region of code into the device, OpenMP 4.0 and 4.5 uses the target construct to create a data environment on the device and then execute the code region on that device. Various directives are provided to express the levels of parallelism. The teams construct, creates a league of thread teams where the master thread of each team executes the region. The number of teams created and of threads participating in the contention group that each team initiates are implementation defined, but can be specified respectively by the num teams and thread limit clauses. The distribute construct specifies loops which are executed by the thread teams. OpenMP 4.0 provides the target data construct to handle data transfers and update data on both of the host and accelerators within the target data regions. The use device ptr clause allows to indicate that a list item is a device pointer already. The target enter data and target exit data constructs, newly added by OpenMP 4.5, allow programmers to specify that variables are respectively mapped/unmapped to/from a device data environment. Finally, OpenMP provides the declarative declare target construct that specifies that variables and functions are mapped to a device.

C. OpenACC

OpenACC is another specification focused on directivebased ways to program accelerators [START_REF]The openacc application programming interface[END_REF]. Unlike OpenMP, OpenACC standard is relatively new, the first version was released in 2011. OpenACC has fewer constructs, but most of them are analogous to those of OpenMP 4.0/4.5. The OpenACC parallel construct starts parallel execution on the current accelerator device by creating one or more gangs of workers. The number of gangs, workers and vector lanes are set by the compiler if no num gangs, num workers and vector length are specified. The kernels construct defines a region of the program that is to be compiled into a sequence of kernels for execution on the current accelerator device. The loop construct specifies the type of parallelism for the associated loop. In addition to the collapse directive, the loop construct provides the tile clause which specifies that the implementation should partition a loop's iteration space into smaller chunks or blocks. Like OpenMP 4.0/4.5, OpenACC provides the data construct for data transfer management between host and accelerator. The enter/exit data or declare constructs and deviceptr clause work in the same way as their correspondent in OpenMP. OpenACC offers also the cache construct which allows to specify that this data should be fetched into the highest level of the cache. Listing 1 shows matrix addition kernel example in CUDA, OpenMP and OpenACC. # pragma omp p a r a l l e l f o r s c h e d u l e ( s t a t i c , 1 )

13 f o r ( i n t j = 0; j < W; j ++) 14 C[ i * W + j ] = A[ i * W + j ] + B[ i * W + j ] ; 15 } 16 v o i d addOpenACC ( f l o a t * A, f l o a t * B , f l o a t * C) {

III. PREVIOUS WORK

In recent years, much work has been done on directivebased programing models for accelerators. Par4All [START_REF] Amini | Source-to-source automatic program transformations for gpu-like hardware accelerators[END_REF] is an open source directive-based programing that allows to target parallel architectures including GPUS. PPCG [START_REF] Verdoolaege | Polyhedral parallel code generation for cuda[END_REF] is a source-to-source compiler based on polyhedral compilation techniques. HMPP [START_REF] Sa-Garriga | Omp2hmpp: Hmpp source code generation from programs with pragma extensions[END_REF] is another high-level directive-based language and source-to-source compiler that can generate CUDA and OpenCL code. Since the release of the OpenACC and OpenMP 4.0 standards, many works have been done to compare their performance with that of CUDA. Hoshino et al. [START_REF] Hoshino | Cuda vs openacc: Performance case studies with kernel benchmarks and a memory-bound cfd application[END_REF] have studied the performance aspects of OpenACC and found that in general OpenACC is approximately 50% slower than CUDA, but for some applications, it can reach up to 98% with careful manual optimizations. Winke et al. [START_REF] Wienke | A pattern-based comparison of openacc and openmp for accelerator computing[END_REF] compared OpenMP 4.0 and OpenACC, predicting that OpenMP 4.0 would likely achieve best adoption in the longterm because it is such a prominent standard. Martineau et al. evaluated in [START_REF] Martineau | Evaluating openmp 4.0s effectiveness as a heterogeneous parallel programming model[END_REF] OpenMP 4.0 effectiveness as a heterogeneous parallel programming model and found that OpenMP 4.0 can achieve good performance while decreasing development cost. They also analyzed, in [START_REF] Martineau | Performance analysis and optimization of clangs openmp 4.5 gpu support[END_REF], the Clang OpenMP 4.5 current implementation on an NVIDIA Kepler GPU and suggested some potential solutions that can improve Clang code generation performance. Graham et al. [START_REF] Lopez | Towards achieving performance portability using directives for accelerators[END_REF] explored the performance portability of directives provided by OpenMP 4 and OpenACC to program various types of node architectures. They concluded that due to the slightly different interpretations of the OpenMP 4 specification, it is crucial to understand how the specific compiler being used implements a particular feature on different platforms. Hayashi et al. [START_REF] Hayashi | Exploring compiler optimization opportunities for the openmp 4.x accelerator model on a power8+gpu platform[END_REF] evaluated and analyzed OpenMP 4.x on an IBM POWER8 + NVIDIA Tesla K80 platform. They found that the OpenMP generated codes are in some cases faster, in some cases slower than straightforward CUDA implementations written without complicated hand-tuning. The work presented in this paper brings, to the state of art, a clarification about the impacts of directives configuration on performance, variations in generated PTX and the additional costs related to the launch of kernel and thread management.

IV. EXPERIMENTAL METHODOLOGY

In this section, we present the experiments carried out to perform our performance comparison between CUDA, Ope-nACC and OpenMP.

A. Applications

As part of this research it was necessary to port a number of applications that would be used in the performance analysis. For what purpose, applications: Back Propagation (BP), Breadth First Search (BFS), Heartwall Tracking (HW), HotSpot (HS), LavaMD (LMD), LU Decomposition (LUD), Needleman -Wunsch (NW) and Speckle Reducing Anisotropic Diffusion (SRAD) come from Rodinia 1 benchmark suite. We chose the Rodinia benchmark because it contains several classes of applications, and this variety allowed us to show the impacts of our different combinations of directives on performance. Detailed description of Rodinia benchmark is provided in [START_REF] Che | Rodinia: A benchmark suite for heterogeneous computing[END_REF]. We also used three applications typically used in preprocessing steps of complex computer vision algorithms: Canny filter (Canny), Harris Corner Detector (HCD) and Horn & Schunck (H&S) method of estimating optical flow.

1 https://www.cs.virginia.edu/ skadron/wiki/rodinia/index.php/Downloads

B. Hardware platforms & Compilers

For our experiments, performance data were collected on an NVIDIA GPU Quadro M2000M hosted in an Intel I7 CPU and an NVIDIA Tegra X2 GPU. LLVM/Clang2 (4.0) being the only free compiler with OpenMP 4.5 support for NVIDIA GPUs, it is the latter that was used to compile our OpenMP versions. For OpenACC, we used the PGI 17.10.0 compiler and CUDA Driver/Runtime version is 9.1. On all platforms, we used Ubuntu 16.04.

C. Approach

Since our goal is to explore performance improvement opportunities in directive-based GPU programming, we have studied several aspects that can impact on OpenACC and OpenMP achievable performance. For that purpose, our study will take place in two steps:

1) Although the use of directive-based methods does not necessarily require a knowledge of GPU architecture, it is nevertheless useful to understand the offloading mechanism used by OpenACC and OpenMP in order to better exploit them. Indeed, the quality of the PTX code generated by the OpenACC and OpenMP compilers is dependent on how the programmer configures the directives. So, as a first step, we will analyze in detail the impact of directives configurations on performance. These configurations include: using directives separately on multiple lines or combining them into one, loops collapsing and grid and block sizes setting. Indeed, it is well known that using the optimal grid and block sizes helps to improve GPU code performance. Since default grid and block sizes chosen by OpenACC and OpenMP compilers are not always the right ones, the developer must often set theses sizes using the clauses dedicated for this purpose. 2) For the second step, we will study the performance evolution according to the workload. This will allow us to see how the additional cost related to the preparation and launch of the kernel as well as thread management varies depending on the body of the kernel. All performance data were collected with NVIDIA profiler nvprof. In our experiment, we used Rodinia CUDA implementations for BP, BFS, HW, HS, HS3D, LMD, LUD, NW and SRAD and ours for Canny, HCD and H&S. In the following, we will take the performance of CUDA versions as a baseline.

V. EXPERIMENTS RESULTS AND ANALYSIS

In this section, we will discuss the results obtained from our experiences.

A. Evaluation of directives configuration impacts on performance

Although accelerator directive based programming model does not exhibit the same level of complexity as CUDA, it is necessary to understand how OpenACC and OpenMP 4.5 map the data as well as the distribution of the computation on the device in order to achieve good performance. With directive-based programming methods, the first opportunity to improve performance is how to combine directives. Figure 1 shows examples of directives configuration in OpenMP and OpenACC. In order for comparisons to be as fair as possible, all memory allocations are made with CUDA APIs in all versions. Thus, data transfers between the CPU and the GPU are managed in the same way as in CUDA versions. In all that follows, the CUDA versions of our applications are optimized using texture memory, constant memory or even shared memory. All optimizations made in OpenMP and OpenACC rely mainly on the use of directives and clauses. It should be noted that unlike OpenACC, whose cache directive allows the developer to tell the compiler that this data needs to be put in shared memory, OpenMP does not offer any directive for this service. In all following graphs, nonComb corresponds to the uncombined configuration of directives as we can see an example, in the red box in figure 1, Comb means combined configuration of directives (green box in figure 1) and Setting-Grid&Block corresponds to the case of configuration of grid and block sizes in addition to directives combination. It should be noted that, directives configuration only has an impact on application performance (runtime, executed instruction count, register used per thread, etc.) and not on the accuracy of the result.

Figure 2 shows speedup factors relative to the baseline CUDA versions, overall execution time (including API calls and data transfer), on an NVIDIA Quadro M2000M and Tegra X2 GPUs. Overall, for both OpenACC and OpenMP, SettingGrid&Block versions are faster than nonComb and Comb versions. With OpenACC, as we cans see on sub figure 2a, directives combination (plus loops collapsing or tiling when they are ready for it) improves applications compute time by a reduction of 1.269⇥, on average, with respect to uncombined versions. Typically, an OpenACC optimized version (here SettingGrid&Block versions) requires 1.556⇥ hand-optimized CUDA runtime. With OpenMP, as we can see in sub figures 2a and 2b, directives combination allows a reduction in application execution time by a factor of 1.456⇥ on average, with respect to uncombined version, on both Quadro M2000M and Tegra X2 GPUs. Typically, an OpenMP optimized version requires 2.114⇥ hand-optimized CUDA runtime. However, these factors are slightly higher for HW and LMD due to the fact that their CUDA versions extensively use the shared memory while in PGI compiler, current version, the cache directive does not work well when there are many arrays to put in shared memory and OpenMP has nos directives to put data in shared memory.

Directives combination helps, OpenACC and OpenMP compilers, to generate efficient code by influencing on the numbers of registers used per thread, the grid and block sizes, the number of generated instructions in PTX file, as well as the number of executed instructions by the threads. Inspecting generated PTXs, we have found that, for both OpenACC and OpenMP, the difference between uncombined and combined versions lies in how the parallelization is made. Table I shows the grid and block sizes and the number of registers used per thread automatically chosen by OpenACC/OpenMP compilers. We can see that directives combination reduces the number of registers used per thread in general. This reduction is more important in the case of OpenMP due to the fact that unlike OpenACC and CUDA, OpenMP uses more registers. For grid and block sizes, OpenACC and OpenMP compilers tend to use the same configurations in general.

By inspecting all PTXs generated by LLVM/Clang for all applications used in our experiment, we found what a third of instructions are used to compute parameters of functions such as kmpc spmd kernel init, kmpc for static fini, etc. These functions, systematically called in each kernel, are used by OpenMP in order to prepare the launch of the kernel on GPU, team creation, threads management and synchronization. So, it seems that the number of these functions called is higher in the case of non combination of directives, which contributes to degrade performance by increasing the number of registers used per thread for example. Thus, it is important to combine directives, in OpenMP, as much as possible in order to minimize the additional cost associated with these functions.

In summary directives must be combined and loops fused as much as possible. Indeed, we found that the collapse clause improves performance. With OpenACC, a significant improvement of performance can be achieved with the tile clause. Additionally, setting block and grid sizes contribute to improve application performance.

B. Performance evolution with workload

Code generation quality and cost, global memory access pattern and workload are factors which can affect OpenACC and OpenMP performance. In this section, we gradually increase in the loops the number of executed instructions. In order to prevent the compiler from using temporal variables, we use different memory address. In OpenMP and OpenACC Figure 3 shows matrix and vector additions kernels compute time and executed instructions count evolution with the workload. On sub figures 3a and 3b, we found that global memory access pattern does not significantly affect performance for OpenACC.

The variation in computing time, according to the number of instructions in kernel (in source code), is relatively stable with an average slope of 878µs for both OpenACC, OpenMP and Cuda. However, the most important thing to see in figure 3 is that there is in general a systematic overhead in the kernel launch for OpenMP. As said in V-A and as can be seen in the figure 3b, OpenMP versions execute 2 to 8⇥ instructions more than CUDA versions and 2 to 6⇥ more than OpenACC versions. This shows that the current implementation of OpenMP in LLVM/Clang still needs improvement in order to generate better PTX code.

VI. CONCLUSION

This paper did a comparative study between OpenACC, OpenMP and CUDA. For that, we made investigations on applications taken from the Rodinia benchmark suite, three computer vision applications and synthetic programs in order to find most significant performance issues. We tested different configurations of OpenMP and OpenACC directives and evaluate their impact on overall application performance. We show that directives must be combined and loops must be collapsed as much as possible and, additionally, setting block and grid sizes contribute to improved application performance. It happens from our experiments that, provided some proper tuning of compilation directives is performed, either OpenACC or openMP can be good alternatives to direct CUDA programming. Kernel code generation is similar to the CUDA compiler. However, there is in general a systematic overhead in the kernel launch for OpenMP, but, for most applications, it's not a big issue provided the kernel has a sufficient workload.

We show that it is important to understand well most features of OpenACC and OpenMP in order to get competitive optimized codes with hand-optimized CUDA. We found that OpenACC optimized code typically requires 1.269⇥ hand- 3. Performance evolution with workload of matrix and vector addition kernels on Quadro M2000M GPU optimized CUDA runtime which is consistent with factors found by Hoshino et al. in [START_REF] Hoshino | Cuda vs openacc: Performance case studies with kernel benchmarks and a memory-bound cfd application[END_REF]. On the other hand, OpenMP requires in general 2.114⇥ hand-optimized CUDA runtime while Martineau et al. found in [START_REF] Martineau | Evaluating openmp 4.0s effectiveness as a heterogeneous parallel programming model[END_REF] a factor of 2.2⇥ with CCE implementation of OpenMP 4.0. However, it should be noted that current compilers (especially true for LLVM/Clang) still need some improvements to increase the competitiveness of OpenMP and OpenACC against CUDA. OpenACC and OpenMP standards also have to evolve to give more possibility (e.g.: access to constant and texture memories) to developers.

1 g l o b a l 2 v 6 C 7 } 8 v 9 #

 126789 o i d addCuda ( f l o a t * A, f l o a t * B , f l o a t * C) { 3 i n t x= blockDim . x * b l o c k I d x . x + t h r e a d I d x . x ; 4 i n t y= blockDim . y * b l o c k I d x . y + t h r e a d I d x . y ; 5 i f ( x >= 0 && x < W && y >= 0 && y < H) [ x + y * W] = A[ x + y * W] + B [ x + y * W] ; o i d addOpenMP ( f l o a t * A, f l o a t * B , f l o a t * C) { pragma omp t a r g e t teams i s d e v i c e p t r (A, B , C) 10 # pragma omp d i s t r i b u t e 11 f o r ( i n t i = 0; i < H; i ++) 12

17 #}Listing 1 .

 171 pragma a c c k e r n e l s d e v i c e p t r (A, B , C) 18 f o r ( i n t i =0; i<H; i ++) 19 # pragma a c c l o o p i n d e p e n d e n t 20 f o r ( i n t j =0; j<W; j ++) 21 C[ i * W + j ] = A[ i * W + j ] + B[ i * W + j ] ; 22 Matrix addition kernel example

Fig. 1 .

 1 Fig. 1. Examples of directives configuration

Fig. 2 .

 2 Fig. 2. Directives configuration impact on performance

  Fig.3. Performance evolution with workload of matrix and vector addition kernels on Quadro M2000M GPU

  While kernel compute time is not sufficient to understand what is really happening at the architectural level when kernel running, we carry out a detailed analysis of the results with the help of performance counters and PTXs codes to identify execution models differences between OpenACC, OpenMP 4.5 and CUDA. • Many embedded systems integrate GPUs. Embedded systems applications developers can use OpenMP or Ope-nACC to accelerate their codes. Currently, LLVM/Clang is the only open source compiler that has a support for NVIDIA GPU and ARM CPU. To the best of our knowledge, this is the first work evaluating performance of LLVM/CLang implementation of OpenMP 4.5 on NVIDIA Tegra embedded platforms. The paper is organized as follows. Section II provides background information on CUDA, OpenMP and OpenACC.

• Using various applications from the Rodinia benchmark suite and the field of computer vision, we do a comparative performance study between OpenACC, OpenMP 4.5 and CUDA, which is essential for facilitating parallel programming for accelerators.

•

TABLE I REGISTERS

 I , GRID & BLOCK CONFIGURATION GENERATED BY OPENMP/OPENACC COMPILERS ON QUADRO GPU: KN = KERNEL N; RG = REGISTERS/THREAD; THREADS/BLOCK = BLK; BLOCKS/GRID = GD

	CUDA	OpenMP nonComb	OpenMP Comb	OpenACC nonComb	OpenACC Comb
	versions, directives are combined, but we did not set grid and
	block sizes. Thus, compilers have the freedom to choose grid
	and block sizes. CUDA versions are not optimized. In this
	experiment, arrays sizes are 4096 ⇥ 4096 and data type is float.

https://github.com/clang-ykt