Kévin

Meriem

Philomène, Abdelkader, Chloé, Victor, Claudia, Olivier, Fairouz, Modibo, Sophie Charles

Musaab Arthur Flora

Matthieu Maria Belen

Keywords: Transport sédimentaire, couplage bidirectionnel, méthodes particulaires, double diffusivité, HPC, GPU, HySoP Sediment transport, dual-way coupling, particle methods, double diffusivity, HPC, GPU, HySoP Sediment transport, dual-way coupling, particle methods, double diffusivity, HPC, GPU, HySoP

Je souhaite remercier en premier lieu mon directeur de thèse, Georges-Henri Cottet, pour m'avoir permis de réaliser cette thèse dans son équipe au sein du laboratoire Jean Kuntzmann. Je lui suis également reconnaissant pour ses qualités scientifiques, sa patience et sa franchise. J'aimerais ensuite remercier mon codirecteur de thèse, Iraj Mortazavi, pour les nombreux séjours au CNAM de Paris, sa sympathie et pour tout son enthousiasme dans ce travail.

Je souhaite également remercier mon encadrant de thèse, Christophe Picard, sans qui ce travail ne serait peut-être pas arrivé à son terme. Je le remercie pour sa grande disponibilité, sa bonne humeur et tout le soutien qu'il m'a apporté durant ces quatre longues années de thèse. J'espère qu'une fois son habilitation à diriger les recherches en poche il pourra continuer à transmettre cette ambiance de travail à la future génération de thésards de l'équipe.

Je remercie mes deux rapporteurs de thèse, à savoir Emmanuel Jeannot et Florian De Vuyst, pour avoir pris le temps de relire ce document conséquent et pour leurs retours très positifs et pertinents. Je tiens également à remercier tous les autres membres du jury : la présidente du jury Valérie Perrier ainsi que tous les autres examinateurs Delphine Depeyras et Eckart Meiburg pour avoir accepté de parcourir ce travail et pour avoir tenté de rejoindre Grenoble malgré une situation quelque peu compliquée.

Je remercie chaleureusement Eckart Meiburg d'avoir fait le déplacement depuis la Californie pour la soutenance et pour les nombreux échanges autour de son modèle de sédimentation. Je remercie également son ancien étudiant de thèse Peter Burns pour les précisions qu'il a pu m'apporter quant à l'aspect numérique du problème. J'espère que ce travail n'est que la première étape d'une longue collaboration entre Grenoble et Santa Barbara.

Je remercie Delphine Depeyras, Matthieu Puyo, Quentin Desbonnets et Florent Braure de l'accueil qu'ils m'ont réservé au sein du groupe Ingéliance sur le site de Mérignac, et de m'avoir permis de découvrir le monde de la simulation numérique en entreprise par le biais du dispositif de doctorat-conseil. Cette collaboration s'est avérée très enrichissante, avec en prime l'obtention conjointe du prix de la simulation numérique de l'usine nouvelle. J'adresse aussi mes remerciements aux personnes qui ont gravité autour de ce projet de thèse, Chloé Mimeau, Jean-Matthieu Etancelin et Franck Pérignon, pour ces nombreux échanges qui ont grandement contribué à la complétion de travail. Mention spéciale à Chloé pour son soutien et sa positivité sans failles, à Jean-Matthieu pour ses retours et contributions dans la librairie et à Franck pour m'avoir mis le pied à l'étrier (et le café, faute de quoi ce travail ne serait bien évidemment pas arrivé à terme).

Je n'oublie pas bien sur tout le personnel du laboratoire pour m'avoir permis de mener ce travail sans accrocs. Tout d'abord je remercie le personnel informatique, notamment Fred, Patrick et Bruno, pour la résolution rapide de tous ces petits tracas informatiques du quotidien. Je remercie aussi tout le personnel administratif et en particulier Laurence, Gaëlle, Aurore et Cathy de nous simplifier la vie au jour le jour.

iii iv Je pense également aux autres membres de l'équipe, notamment à Christophe, amateur de vins et autres spiritueux, au passé texan, qui tente de maintenir tant bien que mal l'équipe à flot; à Brigitte, qui malgré les plaisanteries dont elle est souvent la cible, propose tout de même quelques douceurs pour accompagner le café, voire même, dans son immense bonté, un graphe des relations de publication sur HAL; à Emmanuel, toujours vêtu d'une belle chemise (et qui oublie parfois son polycopié dans l'ascensseur), qui m'a permis de mettre en place la visioconférence en ce vendredi 13 sans transports; à Clément qui a dû s'incliner sur l'épreuve des clous lors de la journée de rentrée du laboratoire et à Ksenia qui a su appliquer chaque jour avec vigueur la méthode scientifique pour finalement devenir développeuse CFD sur Poitiers. Toutes mes pensées vont bien évidemment aux développeurs Fortran 77 et à leurs familles. Je pense enfin au dernier venu, Simon, un grand artiste qui m'a tout de suite pris sous son aile et qui a accepté de devenir mon coach de vie. Sa fameuse méthode TPLM a sans doute été la clé de la rédaction de ce manuscrit, même si Olivier Minne, les cartes à trous, le PEP-8, le reblochon, la chartreuse M.O.F. et le télex y sont aussi sûrement pour quelque chose.

Je remercie ensuite mes collègues de bureau, fussent-ils contemporains de la tour irma ou du batimag. Je pense notamment à Quentin, ce bon vieux Matthias des familles (toujours célibataire), Thomas le web développeur aguerri (qui fera un très bon ingénieur), la maman du LJK (avec ses plats succulents et ses petites gâteries), bouclette l'ours du LJK fan d'échecs et bien évidemment el famoso Dr. Meyron qui peut vous faire miroiter tout et n'importe quoi (et qui apprécie les fruits au sirop). Grâce à vous tous, cette période a été riche d'échanges : échanges de quantité de mouvement tout d'abord grâce aux 54 balles habilement éparpillées dans tous les recoins du bureau, échanges musicaux avec la découverte de nombreuses superstars (le choeur universitaire, Damien Jean, Khaled Freak, Salut c'est cool, Philippe Katerine, Vald, Kaaris, Liza M., Al K. et autres grands artistes en 3 lettres), échanges culturels (Kaamelott et le cul de chouette, Coincoin et les z'inhumains, la scène underground), échanges sportifs (le chamboule-tout et le championnat du monde de fléchettes), échanges d'opportunités business avec la création de nombreuses jeunes pousses (dont la licorne frigofit) et échanges scientiques enfin avec la fameuse étude sur l'angle optimal de rebond balistique et l'étude approfondie de la tribologie. Je pense également aux trois générations de stagiaires qui se sont succédé et qui n'ont eu d'autre choix que de baigner dans cette ambiance : les louveteaux de la meute (Nathan et Gregouze), les deux supernanas (Mélanie et Anne) et mes étudiants de compétition (Nicolas et Joël). Tous auront été placés astucieusement sur les trajectoires balistiques habituelles du bureau et, chose surprenante, aucun n'y est resté. Sophie la girafe est, de tou•te•s mes collègues de bureau, la seule qui a survécu à mon passage au LJK. Sa datation au carbone 14 indique qu'elle proviendrait au moins de l'ère irma, où les abeilles butinaient encore en plein air afin d'y produire le fameux miel mural amianté bio tant prisé des connaisseurs.

Je dois également beaucoup à tous mes collègues de proximité, présent ou passé : Émilie (la tigresse), Arnaud (un grand homme), Mr. Bȃlȃşoiuz (en trajectoire ballistique), Lionel (petit ange parti trop tôt), David (le hipster), Alexis (les maths marchent), Zhu (Dédé), Jean-Baptiste (qui se tape des barres), Nicolas (le béret noir) et François (le kiné) qui ont su ambiancer avec brio le couloir central du premier étage du batimag. Je remercie également les startuppers pour leur curiosité quant à la présence de peluches poulpiques dans le bureau.

Introduction

Continental erosion is controlled by climate and tectonics and can be enhanced by human activities such as large-scale agriculture, deforestation, and sand extraction. Erosion produces sediments that are mostly evacuated by rivers. Downstream, the supply of sediment carries organic matter and nutrients to the ocean, which are fundamental for marine ecosystems and for oil and gas reservoirs [START_REF] Mouyen | Assessing modern river sediment discharge to the ocean using satellite gravimetry[END_REF]]. Estimation of the global sediment discharge in the ocean remains unknown and its measurement still represents a difficult challenge in earth sciences. While it has been estimated that the largest river sediment discharges probably reach one billion of metric tons per year [START_REF] Milliman | River discharge to the coastal ocean: a global synthesis[END_REF], the processes by which this sediment settles out from buoyant river plumes are still poorly understood. A better understanding of the dominant sediment settling mechanisms will help to determine the location of sediment deposits on the sea floor. This represents important information for predicting the location of certain hydrocarbon reservoirs [START_REF] Meiburg | Turbidity currents and their deposits[END_REF]].

During the seventies, a simple laboratory experiment showed that when a layer of fluid, made denser by the presence of suspended material, was put above colder, distilled water, a fingering phenomenon similar to thermohaline convection occurred, result of double-diffusive instabilities [START_REF] Houk | Descent rates of suspension fingers[END_REF]]. In the nature, this event has been observed when a muddy river enters a stratified lake or ocean [START_REF] Schmitt | An estimate of the vertical mixing due to salt fingers based on observations in the North Atlantic Central Water[END_REF], as shown on figure 1. The density stratification of lakes and oceans can be due to thermal or compositional density gradients such as salinity gradients. The density of the seawater is greater than the density of fresh water. When a river plume loaded with fine sediments enters the sea, the river often continues to flow above the seawater (hypopycnal flow) and sediment fingering can occur [START_REF] Mulder | Marine hyperpycnal flows: initiation, behavior and related deposits. A review[END_REF]]. Hypopycnal flows supposes that the sediment concentration remains dilute (less than 40kg/m 3 ), else the density of the river is more than the density of sea, in which case the river continues to flow near the bottom of the sea floor (hyperpycnal flow). For dilute suspensions of particles, Stokes law predicts the settling velocity of small particles in fluid [START_REF] Ferguson | A simple universal equation for grain settling velocity[END_REF]].

In the passed years, most oceanographic sedimentation models relied on Stokes settling as the main mechanism of sediment transfer out of river plumes. Aware that double-diffusive instabilities could greatly increase the effective settling velocity of fine sediments, [START_REF] Burns | Sediment-laden fresh water above salt water: linear stability analysis[END_REF]] performed a numerical investigation to the explore the effect of double-diffusive mechanisms on sediment settling. Through extensive three-dimensional simulations, it was determined that when sediment-laden water flows over clearer and denser water, both doublediffusive and Rayleigh-Taylor instabilities may arise [Burns et al. 2015]. In their analysis both of those convective instabilities led to downward sediment fluxes that could be orders of magnitude above the ones that would normally be obtained with gravitational settling alone. They however faced a major problem when trying to apply their model to fine sediments such as clay and fine silts. The very low diffusivity of small particles [START_REF] Segre | An effective gravitational temperature for sedimentation[END_REF]] with respect to the momentum diffusivity (the kinematic viscosity of the carrier fluid), prevented them to perform the corresponding numerical simulations. This fluid-to-particle diffusivity ratio is often reduced to a dimensionless Schmidt number, and high Schmidt number flows are known to be very computationally demanding [START_REF] Lagaert | Particle method: an efficient tool for direct numerical simulations of a high Schmidt number passive scalar in turbulent flow[END_REF]]. This work aims to achieve high Schmidt number sediment-laden flow simulations by using adapted numerical methods in a high performance computing context.

In the field of fluid mechanics, the evolution of numerical simulations greatly contributes to improve the understanding of such complex physical phenomena. A numerical simulation makes it possible to obtain a solution to a mathematical model that is generally not possible to solve analytically. These models consist of equations modeling a phenomenon to study. One of the objective of numerical simulations is to allow the validation of the physical and mathematical models used for the study of a phenomenon, compared to the theory and to possible experimental data. Since the thirties, the use of calculators has continued to grow with the rapid development of computing machines. In recent years, advances in computing enabled a large enhancement of numerical simulations related to computational fluid dynamics (CFD). These simulations allow to predict the physical behavior of complex flow configurations, possibly coupled with other physical models. The large increase in computing power over the last decades allows the computation of numerical simulations of increasing accuracy and complexity while maintaining comparable computation times. Numerical simulations are based on one or more numerical methods and a computer code that implements these methods on computing machines. The choice of the numerical methods depends essentially the problem to be solved. The numerical efficiency of a code is measured, among other things, by the calculation time required to obtain the solution according to the desired precision. Thus, the implementation of a method requires special attention in order to make the best use of available computing resources. The development of an efficient code in a context of high performance computing is a real challenge on nowadays massively parallel machines. Therefore, numerical methods must not only be adapted to the characteristics of the problems to be solved, but also to the architectures of the machines on which they will be implemented.

In particular, the eventual presence of multiple compute nodes and/or accelerators such as GPUs or other coprocessors must be taken into account from the resolution algorithm to the development of the numerical method.

In this work we limit ourselves to the study of non-cohesive particle-laden hypopycnal flows above salt water. The physics of this type of problem is characterized by the presence of several phenomena at different physical scales that is due to the large difference in the diffusivities of the transported components [START_REF] Batchelor | Small-scale variation of convected quantities like temperature in turbulent fluid Part 1. General discussion and the case of small conductivity[END_REF]]. In this thesis, we consider the problem of active scalar transport at high Schmidt number as an application framework. The transported scalars, representing particles concentration and salinity, will be coupled to the carrier fluid bidirectionally. The approach considered here is that of a resolution by an hybrid particle-grid method [START_REF] Lagaert | Hybrid spectral-particle method for the turbulent transport of a passive scalar[END_REF]. Remeshed particle methods makes it possible to naturally solve conservation equations without imposing a Courant-Friedrichs-Lewy stability constraint, but a less restrictive stability condition allowing the use of larger time steps [START_REF] Cottet | A new approach for the analysis of vortex methods in two and three dimensions[END_REF]]. The use of this method leads, through the presence of an underlying grid, to algorithms and regular data structures particularly adapted to heavily vectorized hardware such as GPUs [START_REF] Rossinelli | Vortex methods for incompressible flow simulations on the GPU[END_REF][START_REF] Rossinelli | GPU accelerated simulations of bluff body flows using vortex particle methods[END_REF]][Etancelin et al. 2014].

Numerically speaking, the starting point of this work is the HySoP library, a Python-based solver based on hybrid MPI-OpenCL programming that targets heterogeneous compute platforms, originally developed in the context of passive scalar transport in high Schmidt number flows [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]]. In particular, we strive to follow the original library mantra, that is the development of non-architecture-specific, performance-portable and easily reusable numerical code. This helps to support the rapid evolution of hardware, favours code reusability and problem expressiveness. A particular attention will be paid to code generation techniques and runtime optimization for GPU-enabled numerical routines. By pursuing the OpenCL porting efforts of the original developer, we will be able to solve custom user-specified numerical problems such as incompressible Navier-Stokes fully on accelerator. The resulting solver will be generic and modular enough to be able to run high performance GPU accelerated numerical simulation of particle-laden sediment flows.

The sequence of the four chapters of this manuscript is the following. In a first chapter, the mathematical models required for the modeling of sediment flows are introduced. The second chapter is dedicated to the different numerical methods required to simulate high Schmidt number flows. The existing numerical methods are adapted to fit GPU-computing by using operator splitting techniques. Chapter three is dedicated to the numerical code and the high performance implementation of the method, giving technical details on the development of performance-portable multi-architecture numerical routines. In particular, we will see how code generation techniques simplify the use of accelerators from a user point of view. The fourth chapter is dedicated to the simulation of particle-laden clear water above salt-water with the resulting solver. Finally, we will conclude this manuscript giving rise to a critique of the work carried out as well as a statement of the perspectives revealed by this study.

Introduction

The context of this work is in applied mathematics, at the intersection of a physical problem, numerical methods and a high performance numerical code. From observations of natural or artificial sedimentation phenomena, empirical physical models are elaborated and put into equations in the form of mathematical models. For numerical simulations, mathematical models are discretized according to one or more numerical methods leading to the expression of algorithms that are implemented in numerical codes. A given model generally depends on several parameters, and can be mapped to a given physical situation by specifying these parameters as well as the initial and boundary conditions of the model. The execution of these codes on computers makes it possible to validate both the numerical methods and the models with respect to the physical phenomena.

Once a mathematical model and associated numerical method are considered valid for a certain range of simulation parameters, the numerical code can be used to explore parameter configurations that are not easily observable or measurable. As it will be the case for our sediment-related problem, the required computational power can be largely dependent on those parameters. When a given computational problem requires more than trillion operations per second (10 12 OP/s) in order to finish in acceptable time, it becomes a necessity to distribute the computations on multiple compute nodes containing multiple CPU ressources. When those nodes also contain accelerators, such as GPUs, it quickly becomes a necessity to adapt the numerical codes to harness their additional compute power. In order to support a wide diversity of hardware efficiently, associated technologies must be taken into account from the design stage of the numerical methods, especially in the perspective of the exascale.

This chapter is dedicated to the presentation of the mathematical background of fluid mechanics in the context of sediment flows and high performance computing. The first section is dedicated to the construction of the model that describes the motion of an incompressible viscous fluid. Some classical resolution methods will be given and a particular attention will be paid to high Schmidt number flows. The second part present some physical models associated to sedimentary processes and lead to the physical model of interest in this work. The last part is focused on high performance computing and introduces distributed computing and coprocessors.

Computational fluid dynamics

The goal of computational fluid mechanics is to analyze and understand behaviors of fluids in motion, possibly in the presence of obstacles or structures with which they interact. The major part of fluid mechanics problems arise from engineering problems encountered in the aviation industry and more generally in the fields of transport, energy, civil engineering and environmental sciences. As an example, CFD1 can be used in sedimentology to study sediment dynamics in an estuary system.

Mathematical modeling of these problems leads to systems of equations too complex to be solved formally. For most problems, the existence of solutions to these systems is still an open problem. Numerical modeling of these problems makes it possible to approach a solution by the use of numerical methods. The Navier-Stokes-Fourier equations provide a general description of the behavior of a viscous fluid based on physical principles of conservation of mass, momentum, and energy.

Derivation of the Navier-Stokes-Fourier equations

The Navier-Stokes equations, named after Claude-Louis Navier and George Gabriel Stokes, are a set of nonlinear partial differential equations that describe the motion of a viscous fluid. This set of equations is a generalization of the equation devised by Leonhard Euler in the 18th century to describe the flow of an incompressible and frictionless fluid [START_REF] Euler | Principes généraux de l'état d'équilibre des fluides[END_REF]]. In 1821, the French engineer Claude-Louis Navier introduced friction for the more complex problem of viscous fluids [START_REF] Navier | Sur les Lois des Mouvement des Fluides, en Ayant Egard a L'adhesion des Molecules[END_REF]]. Throughout the middle of the 19th century, British physicist and mathematician Sir George Gabriel Stokes improved on this work, though complete solutions were obtained only for the case of simple two-dimensional flows [START_REF] Stokes | On the effect of the internal friction of fluids on the motion of pendulums[END_REF]][Stokes 1880].

Way later, in 1934, Jean Leray proved the existence of weak solutions to the Navier-Stokes equations, satisfying the equations in mean value, not pointwise [START_REF] Leray | Sur le mouvement d'un liquide visqueux emplissant l'espace[END_REF]]. One had to wait until the 1960s to obtain results about existence of regular solutions to the Navier-Stokes equations in two dimensions thanks to the work of [START_REF] Ladyzhenskaya | The mathematical theory of viscous incompressible flow[END_REF]]. To this day, the complex vortices and turbulence that occur in three-dimensional fluid flows as velocities increase have proven intractable to any but approximate numerical analysis methods. The existence and uniqueness of classical solutions of the three-dimensional Navier-Stokes equations is still an open mathematical problem and thus constitues an active research area in mathematics.

The balance of equations arise from applying Isaac Newton's second law to fluid motion, together with the assumption that the stress in the fluid is the sum of a diffusing viscous term proportional to the gradient of velocity and a pressure term. Supplementing the Navier-Stokes equations with the conservation of energy leads to the Navier-Stokes-Fourier equations. This section is dedicated to give the idea behind those famous centuries old equations.

8

Chapter 1. High performance computing for sediment flows

Integral form of the Navier-Stokes equations

Mass conservation principle: There is no creation, nor annihilation of mass. Put in other words it means that the rate of mass accumulation within a control volume Ω is equal to the rate of mass flow that goes into the control volume minus the rate of mass flow that leaves the control volume:

d dt Ω ρ dv + ∂Ω ρu • n ds = 0 (1.1)
where ρ is the density of the fluid, u the velocity of the fluid, Ω the fluid control volume, ∂Ω the boundaries of the control volume and n the normal to the boundary at integration point pointing outwards Ω.

Momentum conservation principle:

This equation is given by Isaac Newton's second law applied on the control volume. As Ω is an open system, we need to take into account the momentum flux due to the particles entering and leaving the control volume:

d dt Ω ρu dv + ∂Ω (ρu ⊗ u)n ds = Ω F dv (1.2)
where F represents the forces acting on the fluid in Ω and u ⊗ u = uu T is the outer product of the velocity vector with itself. This equation means that the accumulation of momentum within the control volume is due to the rate of momentum flow going into the control volume minus the rate of momentum flow leaving the control volume supplemented by forces acting the fluid.

The forces applied on the fluid can be expressed as the sum of two forces:

F = ∇ • σ + F ext
where the divergence of the tensor is defined as the vector of the divergence of all its lines.

Here F ext represents all the external forces acting on the fluid and σ is the Cauchy stress tensor that completely define the state of stress at any point inside the control volume in a deformed state. The Cauchy stress tensor can further be decomposed into σ = τ -P I d where τ is the viscous stress tensor, I d is the identity tensor in dimension d and P = (1/d) T r(τ)

the hydrostatic pressure field. The viscous stress tensor accounts for the stress that can be attributed to the strain rate at a given point (the rate at which it is deforming) whereas the pressure field accounts for the local mean compression due to normal stresses.

The decomposition of F as volumetric forces and surface forces becomes straightforward when using the divergence theorem (A.2c):

Ω F dv = Ω (∇ • σ + F ext ) dv = ∂Ω σn ds + Ω F ext dv (1.3)
The constitutive equation of a given fluid gives the expression of the viscous stress tensor τ with respect to other physical quantities. Most commonly encountered fluids, such as water, are Newtonian. In this case, the viscous stress is linear with respect to the strain rate. More complex fluids are known as non-Newtonian and can display a wide range of behaviors like paint or honey. Once τ is specified, the deduction of the total stress tensor σ follows.

Computational fluid dynamics

Energy conservation principle: The first law of thermodynamics states the change in the internal energy of a closed system is equal to the amount of heat Q supplied to the system, minus the amount of work W done by the system on its surroundings. Let q be the conductive heat flux vector accounting for thermal conduction on the boundaries of the control volume.

We also consider an additional volumetric heat source Q that may arrise from radiation or chemical reactions. Because Ω is an open system, we need to take into account the energy flux due to the particles entering and leaving the control volume:

d dt Ω ρe dv + ∂Ω ρe (u • n) ds = ∂Ω σn • u ds + Ω F ext • u dv - ∂Ω q • n ds + Ω Q dv (1.4)
where k is the thermal conductivity of the fluid, T the absolute temperature of the fluid, e = U + 1 2 u 2 the total energy defined as the sum of the internal energy U and the kinetic energy.

For compressible flows the relation between density, pressure and temperature is given by a special equation called equation of state. The most commonly used one is the ideal gas relation P = ρRT , R being the gas constant. This additional equation closes the set of equations.

Differential form of the Navier-Stokes equations

As equations (1.1), (1.2) and (1.4) remain valid for a control volume Ω as small as we want, we can apply the divergence theorem (A.2) to get equivalent local equations. This relies on the continuum hypothesis, stating that a fluid can be regarded as a continuum rather than a collection of individual molecules. Flows where molecular effects are of significance are known as rarefied flows. Most liquids can be considered as a continua, as can gases under ordinary circumstances.

Conservation of mass:

∂ρ ∂t + ∇ • (ρu) = 0 (1.5)

Conservation of momentum:

∂ρu ∂t

+ ∇ • (ρu ⊗ u) = ∇ • σ + F ext (1.6)

Conservation of energy:

∂ρe ∂t

+ ∇ • (ρu) = ∇ • σu + F ext • u -∇ • q + Q (1.7)
The principle of conservation of angular momentum implies that the total stress tensor is symmetric, which means that σ defines a self-adjoint linear operator. The local equation for Chapter 1. High performance computing for sediment flows energy (1.7) is obtained by using this additional equality:

∂Ω σn • u ds = ∂Ω n • σu ds = Ω ∇ • σu dv (1.8)
Equations (1.6) and (1.7) can be simplified using equation (1.5), yielding the following nonconservative local forms of momentum and energy equations:

ρ ∂u ∂t + (u • ∇) u = ∇ • τ -∇P + F ext (1.9a) ρ ∂e ∂t + (u • ∇) e = ∇ • τu -∇ • (P u) + F ext • u -∇ • q + Q (1.9b)
In addition, taking the dot product of equation (1.9a) with the velocity u and subtracting it to equation (1.9b) together with equations (A.1a) and (A.1c) gives the equation relating to the internal energy U = e -1 2 (u • u):

ρ ∂U ∂t + (u • ∇) U = τ : ∇u -P (∇ • u) -∇ • q + Q (1.10)
In this equation, the : operator denotes the double dot product between two tensors. This new equation allows us to define a new equation based on the enthalpy h which is related to the internal energy through h = U + P/ρ:

ρ ∂h ∂t + (u • ∇) h = τ : ∇u + ∂P ∂t + (u • ∇) P -∇ • q + Q (1.11)
This equation is easier to obtain by going back to the conservative formulation of equation (1.10) using equation (1.5) and by using usual divergence formulas (A.1).

Finally we can relate enthalpy h to the temperature of the fluid T using the following differential equation:

dh = C p dT + 1 ρ (1 -αT )dP (1.12)
where C p denotes the heat capacity at constant pressure and α the bulk expansion coefficient also known as the cubic thermal expansion coefficient. We also consider that the heat transfer by conduction is governed by Fourier's law q = -k∇T , giving the following equation for the temperature:

ρC p ∂T ∂t + (u • ∇) T = τ : ∇u + αT ∂P ∂t + (u • ∇) P + ∇ • (k∇T ) + Q (1.13)
For an ideal gas the variation of enthalpy dh is independent of the variation of pressure dP and thus αT = 1. Liquids generally have higher expansivities than solids because their bulk expansion coefficient greatly vary with temperature. For example water has an anomalous thermal expansion coefficient as it is densest at 3.983 • C where it reaches ρ = 999.973 kg/m 3 .

Computational fluid dynamics

Summary of unknowns in a d-dimensional space: Full Navier-Stokes-Fourier system of equations:

The Navier-Stokes-Fourier system describes the motion of a compressible, viscous and heat conducting fluid, and can be expressed as the following:

∂ρ ∂t + ∇ • (ρu) = 0 ρ ∂u ∂t + (u • ∇) u = ∇ • τ -∇P + F ext ρC p ∂T ∂t + (u • ∇) T = τ : ∇u + αT ∂P ∂t + (u • ∇) P + ∇ • (k∇T ) + Q (1.14a) (1.14b) (1.14c)
The Navier-Stokes equations are a simplification of this model for a Newtonian isothermal fluid. When temperature is not involved, the energy equation (1.14c) is not required anymore.

Chapter 1. High performance computing for sediment flows

Vorticity formulation

The vorticity is a field that describes the local spinning motion of a continuum near some point, as would be seen by an observer located at that point and traveling along with the flow. Mathematically, the vorticity ω is a pseudovector field defined as the curl of the velocity:

ω = ∇ × u (1.15)
Some phenomena are more readily explained in terms of vorticity, rather than the basic concepts of pressure and velocity. It is possible to rewrite the momentum equation (1.14b) in terms of vorticity with the help of equation (1.15):

∂ω ∂t + (u • ∇) ω = (ω • ∇)u -ω(∇ • u) + 1 ρ 2 (∇ρ × ∇P ) + ∇ × ∇ • τ + F ext ρ (1.16)
Note that if the density ρ is constant, the baroclinic term that contains the pressure vanishes.

Newtonian fluids

A fluid is considered to be Newtonian only if the tensors that describe the viscous stress and the strain rate are related by a constant fourth order viscosity tensor µ that does not depend on the stress state and velocity of the flow:

τ(u, T ) = µ(T ) : ∇u

(1.17)

If the fluid is also isotropic, this viscosity tensor reduces to two scalar coefficients, the molecular viscosity coefficient µ(T ) and the bulk viscosity coefficient λ(T ). Those constants describe the fluid's resistance to shear deformation and compression, respectively:

τ(u, T ) = µ ∇u + ∇u T + λ(∇ • u)I d (1.18)
Stokes made the hypothesis that λ = -(2/3)µ which is frequently used but which has still not been definitely confirmed to the present day [Stokes 1880[START_REF] Hak | Questions in fluid mechanics[END_REF]. This is not a problem because, as we will see later, the bulk viscosity λ can be neglected when the fluid can be regarded as incompressible.

For an isotropic Newtonian fluid, the divergence of the viscous stress tensor becomes:

∇ • τ = ∇ • µ ∇u + ∇u T + λ(∇ • u)I d (1.19)
The divergence of the strain rate can be rearranged as:

2(∇ • γ) = ∇ • ∇u + ∇u T = ∆u + ∇(∇ • u) (1.20)
Expanding equation (1.19) using (1.20) and divergence formulas gives:

∇ • τ = µ∆u + (µ + λ)∇(∇ • u) + ∇u + ∇u T ∇µ + (∇ • u)∇λ
(1.21)

1.1. Computational fluid dynamics

Compressible Navier-Stokes equations

Under the Stoke's hypothesis and considering an isotropic Newtonian fluid at constant temperature T 0 and constant molecular viscosity µ we finally obtain the compressible Navier-Stokes equations from equations (1.5), (1.6) and (1.21):

∂ρ ∂t + ∇ • (ρu) = 0 ρ ∂u ∂t + (u • ∇) u = µ ∆u + 1 3 ∇(∇ • u) -∇P + F ext (1.22a) (1.22b)
As stated before this set of equation is valid for non-rarefied flows where the representative physical length scale of the system is much larger than the mean free path of the molecules.

As a side note we will only consider isotropic Newtonian fluids throughout the manuscript.

By dividing by the density, equations (1.22a) and (1.22b) can be rewritten as the following:

1 ρ ∂ρ ∂t + (u • ∇) ρ + ∇ • u = 0 ∂u ∂t + (u • ∇) u = ν ∆u + 1 3 ∇(∇ • u) - ∇P ρ + f ext (1.23a) (1.23b)
where ν = µ/ρ is the cinematic viscosity and f ext = F ext /ρ represents the external mass forces acting on the fluid.

Incompressible flows

An incompressible flow refers to a flow in which the material density is constant within a infinitesimal control volume that moves with the flow velocity. From the continuity equation (1.23a), an equivalent statement that implies incompressibility is that the divergence of the flow velocity is zero:

∂ρ ∂t + (u • ∇) ρ = -ρ(∇ • u) = 0 (1.24)
Under the incompressibility hypothesis the viscous stress tensor reduces to τ = µ ∇u + ∇u T for an isotropic Newtonian fluid, removing the need to provide the bulk viscosity λ. This allows us to define the incompressible Navier-Stokes equations with variable density and viscosity:

∇ • u = 0 ρ ∂u ∂t + (u • ∇) u = µ∆u + ∇u + ∇u T ∇µ -∇P + F ext (1.25a) (1.25b)
Note that this hypothesis also simplifies the energy conservation equation (1.14c):

ρC p ∂T ∂t + (u • ∇) T = 2µ( γ : ∇u) + αT ∂P ∂t + (u • ∇) P + ∇ • (k∇T ) + Q (1.26) 14
Chapter 1. High performance computing for sediment flows

In addition, if the density and the viscosity are assumed constant, is follows that the flow is incompressible and we can obtain the following set of equations:

∇ • u = 0 ∂u ∂t + (u • ∇) u = ν∆u - ∇P ρ + f ext (1.27a) (1.27b)
that can also be expressed in terms of vorticity:

ω = ∇ × u ∂ω ∂t + (u • ∇) ω = (ω • ∇)u + ν∆ω + ∇ × f ext (1.28a) (1.28b)
It is important to note that an incompressible flow does not imply that the fluid itself is incompressible. Homogeneous fluids with constant density always undergo flow that is incompressible, but the reverse is not true. As an example it is possible to have an incompressible flow of a compressible fluid in the case of low Mach numbers or in stratified flows.

The Boussinesq approximation

The Boussinesq approximation is a way to solve the incompressible Navier-Stokes equations with a slightly varying density, without having to solve for the full compressible formulation of the Navier-Stokes-Fourier equations (1.14) applied to an isotropic Newtonian fluid (1.21). This approximation, introduced by Joseph Boussinesq in 1877, assumes that variations in density have no effect on the flow field, except that they give rise to buoyancy forces [Boussinesq 1877]. The Boussinesq approximation is mostly used to simplify the equations for non-isothermal flows, such as natural convection problems, but can also be used for problems where the density variations are not due solely to temperature variations inside the fluid.

General framework

Suppose that tiny density variations δρ ρ 0 are created by some physical phenomenon such that the density can be expressed as ρ = ρ 0 + δρ where ρ 0 is a constant. Injecting ρ into the compressible continuity equation (1.23a) gives:

1 ρ ∂δρ ∂t + (u • ∇) δρ 0 +∇ • u = 0 (1.29)
Under the Boussinesq approximation, the continuity equation reduces to the incompressible form ∇ • u = 0 because the first term is assumed to be small compared to the velocity gradients.

Rewriting the momentum equation (1.22b) for an incompressible flow and using the buoyancy as only external force leads to:

ρ ∂u ∂t + (u • ∇) u = µ∆u -∇P + ρg (1.30)
Taking into account that the density variations have only an effect on buoyancy forces yields:

ρ 0 ∂u ∂t + (u • ∇) u = µ∆u -∇P + ρg (1.31)
Finally equation (1.31) can be rewritten as:

ρ 0 ∂u ∂t + (u • ∇) u = µ∆u -∇P + δρg (1.32)
where g = -ge z and P = P + ρ 0 gz is referred to as a pressure shift. A pressure shift can be done whenever the external force vector field is conservative, meaning that there exists some scalar field φ such that F ext = -∇φ, by defining the pressure term as P = P + φ.

Special case for temperature dependent density variations

If density variations are only due to temperature variations δT around some base temperature T 0 , we can rewrite the buoyancy term δρg as -αρ 0 (T -T 0 )g where α is the coefficient of thermal expansion. In this special case, the momentum equation becomes:

∂u ∂t + (u • ∇) u = ν∆u - ∇P ρ 0 -α(T -T 0 )g (1.33)
Keep in mind that this equation is only valid if the viscosity µ does not depend on temperature, an hypothesis that is generally assumed, along with constant thermal conductivity k. Under those additional assumptions and considering the energy conservation equation (1.14c) we get the following set of equations: (1.34b)

T = T 0 + δT ρ = ρ 0 (1 -αδT ) P = P -ρ 0 gz ∇ • u = 0 ∂u ∂t + (u • ∇) u = ν∆u - ∇P ρ 0 -
(1.34c)

(1.34d)

(1.34e)

(1.34f)

Chapter 1. High performance computing for sediment flows

Most of the time, the work of pressure forces and the heat generated by viscous friction are neglected. Those terms however constitute intrinsic components in many physical problems like buoyancy-induced natural convection [START_REF] Pons | Modeling natural convection with the work of pressure-forces: a thermodynamic necessity[END_REF]]. When those terms and heat Q can be neglected we can rewrite equation (1.14c) with the thermal conductivity κ t as the following:

∂δT ∂t + (u • ∇) δT = κ t ∆δT (1.35)

Dimensionless numbers

In this subsection we quickly present some computational fluid dynamics related dimensionless numbers that will be used throughout the manuscript.

Reynolds number

The Reynolds number is used to predict the transition from laminar to turbulent flow, and may be used in the scaling of similar but different-sized flow situations. The predictions of onset of turbulence and the ability to calculate scaling effects can be used to help predict fluid behaviour on a larger scale.

• Laminar flow occurs at low Reynolds numbers, where viscous forces are dominant, and is characterized by smooth, constant fluid motion.

• Turbulent flow occurs at high Reynolds numbers and is dominated by inertial forces, which tend to produce chaotic eddies, vortices and other flow instabilities.

The Reynolds number is defined as:

Re = inertial forces viscous forces = ρuL µ = uL ν
where ρ is the density of the fluid, u is the velocity of the fluid with respect to the object, L is a characteristic length, µ the dynamic viscosity of the fluid and ν the kinematic viscosity of the fluid.

This concept was introduced by George Stokes in 1851 [START_REF] Stokes | On the effect of the internal friction of fluids on the motion of pendulums[END_REF]], but this number was named by Arnold Sommerfeld in 1908 [START_REF] Sommerfeld | Ein beitrag zur hydrodynamischen erklaerung der turbulenten fluessigkeitsbewegungen[END_REF]] after Osborne Reynolds who popularized its use in 1883 [START_REF] Reynolds | XXIX. An experimental investigation of the circumstances which determine whether the motion of water shall be direct or sinuous, and of the law of resistance in parallel channels[END_REF]].

Schmidt number

The Schmidt number Sc is a dimensionless number defined as the ratio of kinematic viscosity (momentum diffusivity) to the mass diffusivity of some transported component:

Sc = viscous diffusion rate molecular diffusion rate = ν κ = µ ρκ
where κ is the mass diffusivity, ρ is the density of the fluid, µ the dynamic viscosity of the fluid and ν the kinematic viscosity of the fluid.

A Schmidt number of unity indicates that momentum and mass transfer by diffusion are comparable, and velocity and concentration boundary layers coincide with each other. Thus it is used to describe whether momentum or diffusion will dominate mass transfer.

Prandtl and Lewis numbers

The Prandtl number P r, named after the German physicist Ludwig Prandtl, is the heat transfer analog of the Schmidt number. It is defined as the ratio of momentum diffusivity to thermal diffusivity:

P r = viscous diffusion rate thermal diffusion rate = ν κ t = µ ρκ t
The ratio between the thermal diffusivity and the mass diffusivity is defined as the Lewis number Le = κ t κ = Sc P r and was named after Warren K. Lewis [START_REF] Lewis | The evaporation of a liquid into a gas[END_REF]].

Rayleigh number

The Rayleigh number Ra for a fluid is a dimensionless number associated with buoyancydriven flow (natural convection) named after Lord Rayleigh [Rayleigh 1916]. This number can be used to describe flows when the mass density of the fluid is non-uniform. When below a critical value, there is no flow and heat transfer is purely achieved by conduction, otherwise heat is transferred by natural convection [C ¸engel et al. 2001].

When the density difference is caused by temperature difference, it is defined as the following:

Ra = time scale for thermal transport by diffusion time scale for thermal transport by convection = gL 3 νκ t ∆ρ ρgL 3 νκ t α∆T where α is the thermal expansion coefficient of the fluid, ρ is the density of the fluid, g is the acceleration due to gravity, L is a characteristic length, ν the kinematic viscosity of the fluid, κ t is the thermal diffusivity of the fluid and ∆ρ and ∆T represent respectively density and temperature differences.

Chapter 1. High performance computing for sediment flows

Kolmogorov microscales

Kolmogorov microscales are the smallest scales in turbulent flow. At the Kolmogorov scale, viscosity dominates and the turbulent kinetic energy is dissipated into heat:

• Kolmogorov length scale: η K = ν 3 1 4

• Kolmogorov time scale: τ K = ν 1 2

• Kolmogorov velocity scale: u K = (ν )

1 4
where ν is the kinematic viscosity and is the average rate of dissipation of turbulence kinetic energy per unit mass. The Kolmogorov length scale ν can be obtained as the scale at which the Reynolds number is equal to 1,

Re = uL ν = 1, L = ν U = µ ρU .

Numerical resolution

The simulation of turbulent flows by numerically solving the Navier-Stokes-Fourier equations (1.14) requires resolving a very wide range of time and length scales, all of which affect the flow field. Such a time and space resolution can be achieved with Direct Numerical Simulation (DNS), but is computationally expensive. Its computational cost can prohibit simulation of practical engineering systems with complex geometry or flow configurations. To circumvent those limitations, various class of mathematical models have been developed to numerically solve complex fluid problems.

Direct Numerical Simulation (DNS)

A direct numerical simulation is a simulation in computational fluid dynamics in which the Navier-Stokes equations are numerically solved without any turbulence model. This means that the whole range of spatial and temporal scales of the turbulence must be resolved. All the spatial scales of the turbulence must be resolved on the computational mesh, from the smallest dissipative scales (Kolmogorov microscales), up to the integral scale L associated with the motions containing most of the kinetic energy. In 3D we have that the number of mesh points satisfying this condition is proportional to Re 9 4 = Re 2.25 and the number of time steps is proportional to Re 3 4 implying that the number of floating point operations grows as Re 3 . Therefore, the computational cost of a fully resolved DNS is very high, even at low Reynolds numbers.

Large Eddy Simulation (LES)

The principal idea behind LES is to reduce the computational cost by ignoring the smallest length scales, which are the most computationally expensive to resolve, via low-pass filtering of the Navier-Stokes equations. Such a low-pass filtering, which can be viewed as a time-and spatial-averaging, effectively removes small-scale information from the numerical solution. This information is not irrelevant, however, and its effect on the flow field must be modeled, a task which is an active area of research for problems in which small-scales can play an important role, such as near-wall flows, reacting flows, and multiphase flows.

Reynolds-averaged Navier-Stokes equations (RANS)

The Reynolds-averaged Navier-Stokes equations are time-averaged equations of motion of fluid flow. The idea behind the equations is Reynolds decomposition, whereby an instantaneous quantity is decomposed into its time-averaged and fluctuating quantities, an idea first proposed by Osborne Reynolds. The RANS equations are primarily used to describe turbulent flows. These equations can be used with approximations based on knowledge of the properties of flow turbulence to give approximate time-averaged solutions to the Navier-Stokes equations.

Vortex methods

Since their first use in the 1930s [START_REF] Prager | Die druckverteilung an körpern in ebener potentialströmung[END_REF]] [START_REF] Rosenhead | The formation of vortices from a surface of discontinuity[END_REF]], vortex methods have gained attraction in the numerical fluid mechanics community by their ability to model accurately, robustly and naturally flows where the convective phenomenon is dominant. However, their widespread use was hampered by their difficulty to take into account adhesion conditions on the boundaries of immersed solids and to treat effects due to the viscosity of the flow [START_REF] Cottet | Vortex methods: theory and practice[END_REF]]. This last point is explained by the phenomenon of grid distortion which is characteristic of lagrangian methods. This distorsion is due to the accumulation or rarefaction of the particles in areas of strong velocity gradients.

Despite multiple studies until the 1990s in order to remedy these weaknesses, the vortex method did not achieve to impose themselves in their original context [START_REF] Chorin | Numerical study of slightly viscous flow[END_REF]] [START_REF] Leonard | Numerical simulation of interacting three-dimensional vortex filaments[END_REF]][Chorin 1978[START_REF] Leonard | Numerical simulation of interacting three-dimensional vortex filaments[END_REF]]. The emergence of particle remeshing schemes and their use in hybrid eulerian/lagrangian solvers enabled the vortex methods to circumvent their intrinsic difficulties while maintaining their original strengths [START_REF] Anderson | On vortex methods[END_REF][START_REF] Cottet | A new approach for the analysis of vortex methods in two and three dimensions[END_REF]]. The remeshing procedure allowed to redistribute the particles on an underlying grid, ensuring a distortionless procedure with uniform particle distribution, and the use of eulerian methods on Cartesian grid facilitated the treatment of viscous effects. The gain of competitiveness obtained with this new semi-lagrangian framework then aroused many efforts, particularly in terms of algorithmic development and high performance computing, resulting in multicore distributed parallel solvers [START_REF] Sbalzarini | PPM-A highly efficient parallel particle-mesh library for the simulation of continuum systems[END_REF]] [START_REF] Lagaert | Hybrid spectral-particle method for the turbulent transport of a passive scalar[END_REF]] and efficient implementations of high order remeshing kernels on coprocessors and accelerators [START_REF] Rossinelli | Vortex methods for incompressible flow simulations on the GPU[END_REF]] [START_REF] Etancelin | Multi-CPU and multi-GPU hybrid computations of multi-scale scalar transport[END_REF]]. The remeshed particle method is described in section 2.3.

Chapter 1. High performance computing

for sediment flows

Scalar transport in high Schmidt number flows

The prediction of the dynamics of a scalar transported in a turbulent flow constitute an important challenge in many applications. A scalar field can describe a temperature field, the concentration of chemical species or even a levelset function that captures the interface in multiphase flows [START_REF] Shraiman | Scalar turbulence[END_REF]]. For a passive scalar θ, the transport equation is an advection-diffusion equation that is coupled with the velocity of the carrier-fluid:

∂θ ∂t + (u • ∇) θ = κ∆θ (1.36)
where κ is the molecular scalar diffusivity and u the turbulent velocity of the carrier fluid.

Similarly to the Kolmogorov length scale η K defined in section 1.1.7, the Batchelor scale η B describes the smallest physical length scales that the scalar will develop before being dominated by molecular diffusion [START_REF] Batchelor | Small-scale variation of convected quantities like temperature in turbulent fluid Part 1. General discussion and the case of small conductivity[END_REF]]. The Schmidt number is a dimensionless parameter defined as the viscosity-to-diffusivity ratio:

S c = ν κ (1.37)
The Batchelor and Kolmogorov scales are related by η B = η K / √ S c . For a Schmidt number larger than one, the Batchelor scale is thus smaller than the Kolmogorov scale and scalar dynamics can occur at scales smaller than the smallest velocity eddies. Direct numerical simulations of turbulent transport with a pseudo-spectral methods have been conducted by [START_REF] Donzis | Resolution effects and scaling in numerical simulations of passive scalar mixing in turbulence[END_REF] to study universal scaling laws of passive scalars. For Schmidt numbers higher than one, they investigated three-dimensional grid discretizations up to N = 2048 in each direction with a Fourier-based spectral solver to determine the effects of the grid resolution on small-scale scalar statistics. Their main conclusion is that the grid resolution should effectively be of the order of the Batchelor scale ∆x η K .

Figure 1.1 -Three-dimensional turbulent scalar transport at high Schmidt number: Slice of the norm of the vorticity (left, coarse grid 256 3 ) and passive scalar (right, fine grid 3064 3 ) obtained by [START_REF] Lagaert | Hybrid spectral-particle method for the turbulent transport of a passive scalar[END_REF] for a Schmidt number S c = 128.

Sediment flows 21

For numerical simulations, the dependency of the Batchelor scale on the Schmidt number suggests that the prediction of scalar dynamics for high Schmidt numbers is more demanding in terms of spatial resolution than the prediction of momentum. The two-scale nature of turbulent scalar transport makes it natural to use different grids [START_REF] Gotoh | Spectral compact difference hybrid computation of passive scalar in isotropic turbulence[END_REF], however, classical advection schemes for the scalar equation impose a CFL condition which requires adaptation of the time step to the finest scales, thus increasing the computational cost. Particle methods, that are not constrained by any CFL condition, can overcome this limitation. This technique has been introduced by [Cottet et al. 2009a] in the context of LES simulations of homogeneous isotropic turbulence where both the momentum and scalar equations were solved by particle methods at different resolutions. This method leads to a significative speed up over more conventional grid-based methods and allows to address challenging Schmidt numbers [START_REF] Lagaert | Particle method: an efficient tool for direct numerical simulations of a high Schmidt number passive scalar in turbulent flow[END_REF]. A distributed hybrid spectral-particle method has also been proposed in [START_REF] Lagaert | Hybrid spectral-particle method for the turbulent transport of a passive scalar[END_REF] where three-dimensional simulations up to S c = 128 were performed (see figure 1.1). In this case, the grid resolution was 256 3 for the flow variables (u, ω) and 3064 3 for the scalar θ. The speed-up provided by the spectral-particle method over a purely spectral method was estimated to be around O 10 2 .

Sediment flows

Dispersion of solid particles in turbulent shear flows is of importance in many natural and engineering processes. The transport of sediments in oceans and slurries in pipes are typical examples. At sufficiently high mass loading, the particles have an effect on the turbulence structure of the carrier fluid and thus modify the transport of physical quantities. To predict these transport phenomena, the knowledge of the coupling between the particles and the fluid is required. This section introduces sediment transport processes and aims to give a state of the art of existing fluid-particles coupling models.

Sediment transport mechanisms

A sediment is a naturally occurring material that originates from the alteration of continental geological formations. Sedimentary erosion, transport and deposit are driven by many physical processes, their transport being mainly due to wind, water and gravity. As an example, sand can be carried in suspension by rivers, reaching the sea bed where they will be deposited by sedimentation once the flow becomes sufficiently quiescent. Their geological origin determining their chemical composition, sediments exhibit a huge variety of physical and mechanical properties including varying sizes, shapes and densities.

In the literature, many sediment size classifications have been proposed [START_REF] Schmid | Descriptive nomenclature and classification of pyroclastic deposits and fragments[END_REF][START_REF] Moncrieff | Classification of poorly-sorted sedimentary rocks[END_REF]]. Those classifications are important because the granulometry plays a great role into the transport mechanisms: the same sediment calibers will not be found near or far from the erosion zone as it can be seen on figure 1.2. As we may expect, the finest sediments will be transported as suspensions and the coarsest sediments that are heavier will travel the least distance from the erosion zone. The granulometry is not the only factor that should be taken for sediment flows Figure 1.2 -Illustration of the different sediment transport mechanisms: the upper layer is composed of ions that are transported as dissolved load and light sediments like clay, silt and sand that are transported as suspended loads. The bed load is composed of intermediate size sediments like gravel alternating between transport and sedimentation phases, jumping on the bed (saltation) and pebbles and boulders rolling and sliding on the sea floor. A typical mean vertical flow velocity profile u(z) is also provided, showing the effect of the high concentration of sediments in the bed load, acting as a porous media, on the flow velocity.

in account, the amount of clay and organic matter is also of importance because it determines whether the sediments exhibit a cohesive behaviour or not. Gravels and coarse to medium sands are non-cohesive sediments that are constituted of particles that are independent from each other. In cohesive sediments however, particles are attracted between each other and form higher density aggregates that will settle [START_REF] Teisson | Cohesive suspended sediment transport: feasibility and limitations of numerical modeling[END_REF]] having significant effects on the flow [START_REF] Edmonds | Significant effect of sediment cohesion on delta morphology[END_REF]. Polydisperse flows are sediment-laden flows that contains sediment of different sizes and shapes. Polydispersity of particles or droplets introduces a wide range of length and time scales that introduce further modeling challenges.

The ability to transport sediments is constrained by the energy available in the flow. When water flows over the sediment bed, it generates a shear stress τ that may trigger the sediment transport if above a critical value. The comparison of the destabilizing force due to shear F s ∝ τ d 2 to the stabilizing gravity force acting on a single particle F g ∝ (ρ s -ρ f )gd 3 gives a nondimensional number, the Shields number, that can be used to calculate the initiation of motion of sediments [START_REF] Shields | Application of similarity principles and turbulence research to bedload movement[END_REF]]. It is defined as the following:

τ * = τ (ρ s -ρ f )gd
where ρ s and ρ f are the particle and fluid density. As the transport is facilitated when particles lay on an inclined bed, the formula can be adapted to depend on the slope angle [START_REF] Fredsøe | Mechanics of coastal sediment transport[END_REF]].

Depending on the value of the Shields number, sediments will move differently. We usually distinguish four layers of sediment loads as represented on figure 1.2:

• Dissolved load: Dissolved matter is invisible, and is transported in the form of chemical ions, the simplest example being salt (NaCl). Water can disrupt the attractive forces that hold the sodium and chloride in the salt compound together and dissolve it. All flows carry some type of dissolved load as water is a universal solvent. The total dissolved solid load transported in major world rivers has been estimated to 3843 megatons per year [START_REF] Zhang | Water chemistry of the Zhujiang (Pearl River): natural processes and anthropogenic influences[END_REF]].

• Suspended load: The suspended load is composed of fine particles that are transported in suspension in the flow. Those particles are too large to be dissolved, but too light to lie on the sea bed. However, suspended loads can settle if the flow is quiescent or in the presence of cohesive particles that will agglomerate and thus settle on the bed.

• Bed load: The bed load is constituted of sediments that are in continuous or intermittent contact with the bed. One can distinguish pebbles and boulders that can only slide and roll on the bed and coarse sands and gravels that moves by successively jumping on the bed in the flow direction. This last process is called saltation. The fluid velocity rapidly decreasing near the sea bed, stratification occurs and the near bed sediment concentration rises, resulting in non-Newtonian rheologies [START_REF] Faas | Time and density-dependent properties of fluid mud suspensions, NE Brazilian continental shelf[END_REF]].

• Bed: When the sediment concentration exceeds a critical value, the structural density, a porous network of sediments is formed and a weak saturated soil is formed. Due to the continuous deposit of sediments, this structure will slowly collapse and compress on its own weight, increasingly consolidating the soil. The soil then consolidates even further due to time dependent processes such as thixotropy [START_REF] Freundlich | Sedimentation Volume, Dilatancy, Thixotropic and Plastic Properties of Concentrated Suspensions[END_REF].

In reality, the transition between these four zone is rather gradual. This highlights the fact that sediment transport is a vertically continuous varying phenomenon. The Hjulström curve (fig. 1.3) can be used to determine whether a river flowing at a certain velocity will erode, transport, or deposit sediments of a certain size.

This simple model, proposed in the early 20th century, determines the state of a particle at a certain depth by the knowledge of its size and the velocity of the carrier fluid. On this curve we can already see that for cohesive sediment (clay and silt) the erosion velocity increases with decreasing grain size, highlighting the fact that the cohesive forces become more and more important as particles sizes get smaller. On the other hand, the critical velocity for deposition depends only on the settling velocity which decreases with decreasing particle size.

Sediments-laden flows classification

Physical models can be split into three classes depending on the volume fraction of particles in the carrier fluid and the ratio between particle response time and the Kolmogorov time scale [START_REF] Elghobashi | Particle-laden turbulent flows: direct simulation and closure models[END_REF]] as it can be seen on figure 1.4. The particle response time τ p characterizes how quickly a particle responds to turbulent fluctuations in the carrier fluid. In this subsection, φ p is the volume fraction of particles in the carrier fluid, S represents the average distance between the centers of two neighboring particles, d is the diameter of a particle, τ k is the Kolmogorov time scale and τ e is the turnover time of large eddy.

When the volume fraction of particles is very small φ p ≤ 10 -6 , the particles effect on turbulence can be neglected. In this first case, the particles behaves as tracers: the particle dispersion only depends on the carrier fluid, and there is not feedback from particles to the fluid. This situation is often referred as "one way coupling". The second regime characterizes dilute suspensions where the volume fraction φ p is comprised between 10 -6 and 10 -3 In this case, the mass loading is high enough to modify the turbulence of the carrier fluid and "two way coupling" becomes necessary as particles modify the fluid characteristics. Depending on their size, particles can enhance either the production or the dissipation of turbulence energy:

1. Lowering the particles diameter d diminishes their response time τ p and increases the total surface area of the particles. As a consequence, particles have a tendency to increase the dissipation rate of the turbulence energy.

2. Increasing the particles diameter d increases their response time τ p and diminishes the total surface area of the particles. This increases the particles Reynolds number Re p and vortex shedding builds up. As a result, particles increase the production of turbulence energy in the carrier fluid.

The third regime characterizes dense suspensions where the particle loading is greater than 0.1%. Because of the increased particle loading, we also have to take into account interactions between particles, hence the term "four way coupling". Increasing further the particle loading leads to collision-dominated flows (fluidized beds) followed by contact-dominated flows (nearly settled bed). When φ p reaches 100%, there is no carrier fluid anymore and the flow becomes fully granular.

Chapter 1. High performance computing for sediment flows

Numerical models

In this work, we are only interested in suspensions that are dilute enough to be accurately modelled by either one-way or two-way coupling (φ p ≤ 10 -3 ), evolving as dissolved or suspended loads. One-way coupling just consists into adding an additional scalar advection-diffusion equation to the Navier-Stokes equations (see section 1.1.10). Some modelling difficulties arrise when considering suspended loads of sediments where two-way coupling becomes mandatory. Such flow configurations are referred to as sediment-laden flows and are included into the more general framework of particle-laden flows. In a particle-laden flow, one of the phases is continuously connected (the carrier phase) and the other phase is made up of small immiscible particles (the particle phase). This section presents the two main approches commonly used to mathematically represent particle-laden flows.

Eulerian-eulerian approach

In the EE 2 approach the carrier flow and the particles are modelled as a continuum, allowing the use of the same discretization techniques for both the fluid and the particles. This class of methods can further be split into two subclasses depending on how the particulate phase is modelled (deterministically or statistically):

1. Deterministic separated-fluid formulation: When the particles can be considered as a continuum behaving like the carrier fluid, is is possible to perform a two-way fluidfluid coupling through two sets of Navier-Stokes equations modelling each continuum:

φ f + φ p = 1 F f p + F p f = 0 ∂φ k ρ k ∂t + ∇ • (φ k ρ k u k ) = 0 ∂φ k ρ k u k ∂t + ∇ • (φ k ρ k u k ⊗ u k ) = ∇ • (φ k τ k ) + φ k ρ k g + F k k (1.38a) (1.38b) (1.38c) (1.38d)
where φ k , ρ k , u k and τ k represent the volume fraction, density, velocity and total stress tensor of phase k ∈ {f, p}. This two-fluid model is based on the particle-phase continuum assumption of [START_REF] Drew | Theory of multicomponent fluids[END_REF]]. Note that for the particulate phase, those quantities are to be determined for a huge number of particles in a small control volume which imposes d dx. The fluid-particles interactions are taken into account through the interphase hydrodynamic force F k k resulting in a two-way coupling of the model. The total stress tensor of the particles can take into account the pressure gradient due to the carrier flow and for dense suspensions where φ p ≥ 0.1%, complex particle-particle interactions like collisions and particle viscous stress due to altered flow characteristics around each particle. If the particles are polydisperse, it is also possible to consider independently each particle group as an individual phase [START_REF] Schwarzkopf | Multiphase flows with droplets and particles[END_REF]].

2 Eulerian-Eulerian 2. Deterministic mixed-fluid formulation: This formulation is a simplification of the separated-fluid formulation where the relative velocities between the fluid and particles u f -u p are small compared to the predicted flow velocity field u = u f . This simplification results in a unique set of Navier-Stokes equations for the two phases [START_REF] Faeth | Mixing, transport and combustion in sprays[END_REF]]:

φ f + φ p = 1 ρ = φ f ρ f + φ p ρ p ∂ρ ∂t + ∇ • (ρu) = 0 ∂ρu ∂t + ∇ • (ρu ⊗ u) = ∇ • τ + ρg (1.39a) (1.39b) (1.39c) (1.39d)
3. Statistical kinetic method: Statistical methods also make the assumption that particles can be seen as a continuum. They can be used to track the evolution of ensembleaveraged particle properties. In this first statistical method, the particles are described by a probability density function W (u p , x p , t) which represents the probability that a particle has a certain velocity and position at given time. This framework being more general, the resulting transport equations are harder to derive but it becomes easier to handle near-wall boundary conditions. It has been first introduced by [START_REF] Buyevich | Statistical hydromechanics of disperse systems Part 1. Physical background and general equations[END_REF]], and since has been continuously improved by many authors including notably [START_REF] Reeks | Eulerian direct interaction applied to the statistical motion of particles in a turbulent fluid[END_REF]], [START_REF] Derevich | Particle deposition from a turbulent flow[END_REF]], [START_REF] Swailes | A generalized Fokker-Planck equation for particle transport in random media[END_REF]] and [START_REF] Hyland | Derivation of a pdf kinetic equation for the transport of particles in turbulent flows[END_REF]]. Moreover, this framework tends to invalidate the traditional assumption that the particles behaves as a Newtonian fluid [START_REF] Elghobashi | Particle-laden turbulent flows: direct simulation and closure models[END_REF]].

Statistical generalized Langevin model:

This model is a generalization of the statistical kinetic model where the probability density function now also depends on the velocity of the carrier fluid W (u f , u p , x p , t). It was first introduced by [START_REF] Simonin | Eulerian prediction of the fluid/particle correlated motion in turbulent two-phase flows[END_REF]] and is built upon the work of [START_REF] Haworth | A generalized Langevin model for turbulent flows[END_REF]] on the generalized Langevin model. However statistical approches are most suited for gases and are not relevant for sediment-laden flows were all the particles are solid.

Lagrangian-eulerian approach:

The LE3 approach denotes a group of models where the carrier fluid is modelled as a continuum and the particles are described in a lagrangian frame. Within this framework all particles are tracked individually within the carrier fluid, allowing a better handling of particle-particle collisions [START_REF] Subramaniam | Lagrangian-Eulerian methods for multiphase flows[END_REF]]. We usually split those methods into the following categories:

1. Fully-resolved DNS: The exact Navier-Stokes equations are solved by imposing noslip boundary conditions at each particle surface. In this case, particle collisions are usually modelled by using either a soft-sphere or hard-sphere collision model [START_REF] Hoomans | Discrete particle simulation of bubble and slug formation in a two-dimensional gas-fluidised bed: a hard-sphere approach[END_REF][START_REF] Matuttis | Discrete element simulations of dense packings and heaps made of spherical and non-spherical particles[END_REF]]. This kind of simulation is very demanding in terms of compute resources but no additional closure models are required to compute the solution.

Chapter 1. High performance computing for sediment flows

This method has been successfully applied for a limited amount of particles by [START_REF] Zhang | A second-order method for three-dimensional particle simulation[END_REF]] and [Uhlmann 2005]. As the domain space step dx should be much smaller than the particle diameter to capture the surface of each particle d, increasing the number of particles quickly becomes a problem when considering thousands particles. However, boundary handling can be simplified by introducing an immersed boundary method to easily enforce the particle boundary conditions. By using the penalization method of [START_REF] Kempe | An improved immersed boundary method with direct forcing for the simulation of particle laden flows[END_REF], particle-resolved DNS has been used to perform fluid simulations of a polydisperse sediment bed made up of thousands of particles, sheared by a viscous flow [START_REF] Biegert | A collision model for grain-resolving simulations of flows over dense, mobile, polydisperse granular sediment beds[END_REF]]. More recently the same technique was used to study the settling of cohesive sediments [START_REF] Vowinckel | Settling of cohesive sediment: particle-resolved simulations[END_REF]. Simulations using levelsets and vortex penalization were performed in [START_REF] Coquerelle | A vortex level set method for the two-way coupling of an incompressible fluid with colliding rigid bodies[END_REF]] and [START_REF] Jedouaa | An efficient interface capturing method for a large collection of interacting bodies immersed in a fluid[END_REF]].

Point-particle DNS:

If the particle size is smaller than the Kolmogorov scale, the particle can be considered as a point particles, relieving the dx d condition. The main difference with the fully-resolved DNS is that this model requires an additional closure model for interphase momentum transfer. As each particle can be evolved independently, it is possible to handle millions of particles [START_REF] Kuerten | Point-Particle DNS and LES of Particle-Laden Turbulent flow-a state-of-the-art review[END_REF]]. This category englobes two subcategories depending on the type of computational particles that is considered:

• PP-DNS with physical particles: each particle represents a physical particle [START_REF] Squires | Measurements of particle dispersion obtained from direct numerical simulations of isotropic turbulence[END_REF][START_REF] Sundaram | Collision statistics in an isotropic particle-laden turbulent suspension. Part 1. Direct numerical simulations[END_REF]].

• PP-DNS with stochastic particles: each particle represents some particle density that interacts with the carrier fluid [START_REF] Boivin | Direct numerical simulation of turbulence modulation by particles in isotropic turbulence[END_REF]].

In point-particle based models statistical treatment of collisions is usually employed [START_REF] O'rourke | Collective drop effects on vaporizing liquid sprays[END_REF][START_REF] Schmidt | A new droplet collision algorithm[END_REF]].

Point-particle LES:

It is also possible to take into account the carrier fluid using Large Eddy Simulation instead of DNS, leading to two new subcategories similar to the previous ones: point-particle LES with physical particles [START_REF] Apte | Large-eddy simulation of swirling particle-laden flows in a coaxial-jet combustor[END_REF]] and point-particle LES with stochastic particles [START_REF] Sommerfeld | Validation of a stochastic Lagrangian modelling approach for inter-particle collisions in homogeneous isotropic turbulence[END_REF][START_REF] Okong'o | Consistent large-eddy simulation of a temporal mixing layer laden with evaporating drops. Part 1. Direct numerical simulation, formulation and a priori analysis[END_REF].

From a numerical point of view, the lagrangian-eulerian approach minimizes numerical diffusion in the particulate-phase fields such as volume fraction or mean velocity when compared to eulerian-eulerian approaches. This comes at the cost of higher computational requirements compared to the EE averaged equations due to the particle-based representation of the multiphase flow, even when considering stochastic point particle based models.

Density-driven convection 29

Density-driven convection

For the rest of this chapter, we place ourselves into the eulerian-eulerian framework. This section describes some sedimentation problems that are buoyancy-driven such as gravity currents and double-diffusive fingering of salt and particle-laden flows.

Gravity currents

Gravity currents or density currents are a primarily horizontal flows in a gravitational field that is driven by a density difference [START_REF] Benjamin | Gravity currents and related phenomena[END_REF]]. Gravity currents can be finite volume like in the event of a dam break, or continuously supplied like for lava flows. Depending on the parameters, mostly the density difference and the concentration, different approaches that can account for this kind of two-way coupling are to be considered. An overview of those approaches can be found in [START_REF] Sinclair | Tectonostratigraphic model for underfilled peripheral foreland basins: An Alpine perspective[END_REF]] as well as in [START_REF] Schwarzkopf | Multiphase flows with droplets and particles[END_REF]]. In his book, [START_REF] Simpson | Gravity currents: In the environment and the laboratory[END_REF]] describes and illustrates a whole range of naturally occurring densitydriven flows such as turbidity currents in the ocean, sea breeze fronts propagating inland from the coast, salt water wedges in river estuaries, and snow avalanches. The density difference that drives this kind of currents may be cause by the presence of additional chemicals in some regions of the flow, like in the case of fresh water flowing into salty sea water [START_REF] Huppert | The propagation of two-dimensional and axisymmetric viscous gravity currents over a rigid horizontal surface[END_REF]]. Thermal differences can also be the cause of density gradients creating gravity currents [START_REF] Bercovici | A theoretical model of cooling viscous gravity currents with temperature-dependent viscosity[END_REF]]. Turbidity currents in the ocean are driven by suspended particles increasing the bulk density of the surrounding fluid [START_REF] Bonnecaze | Particle-driven gravity currents[END_REF]. In this specific case, the density can change through sedimentation and the entrainment of the lighter carrier fluid. [START_REF] Britter | Experiments on the dynamics of a gravity current head[END_REF] provides experimental and analytical data for the case of a heavy fluid released next to a lighter fluid which show that gravity currents mix trough Kelvin-Helmholtz billows generated at the moving head of the mixing layer.

The density difference between the two fluids can range from very small to very large. In many geophysical situations such as sea water and fresh water the density difference is very small (within 5%). In cases of small density difference, density variations can be neglected in the inertia term, but retained in buoyancy term. Hence, for dilute suspensions of sediments that are small enough such that their inertia can be neglected, the most convenient model uses a single-phase Boussinesq fluid whose density ρ depends only on carrier fluid base density ρ 0 augmented by the local particle concentration C.

The particulate phase is driven by a convection-diffusion equation for its concentration:

∇ • u = 0 P = P -ρ 0 gz ρ = ρ 0 (1 + C) ∂u ∂t + (u • ∇) u = ν∆u - ∇P ρ 0 + Cg ∂C ∂t + (u • ∇) C = κ c ∆C (1.40a) (1.40b) (1.40c) (1.40d) (1.40e)

Chapter 1. High performance computing for sediment flows

This simple model can directly be obtained from the incompressible eulerian-eulerian mixedfluid formulation (1.39) in its Boussinesq approximation (1.34) where we used the so called pressure shift (1.32). This model is particularly adapted for the velocity-vorticity formulation as the gradient of shifted pressure disappears when taking the curl of the momentum equations leading to two scalar convection-diffusion equations in 2D. For three-dimensional problems we get three convection-diffusion-stretch equations for the vorticity and one convection-diffusion equation for the scalar concentration C:

∇ • u = 0 ρ = ρ 0 (1 + αC) with α 1 and C ∈ [0, 1] ∂ω ∂t + (u • ∇) ω = ν∆ω + ∇ × (Cg) ∂C ∂t + (u • ∇) C = κ c ∆C (1.41a) (1.41b) (1.41c) (1.41d)
This model has extensively been used by [START_REF] Necker | Direct numerical simulation of particle-driven gravity currents[END_REF] to investigate particle-driven currents and to compare the results to a eulerian-lagrangian numerical model as described in subsection (1.2.3). Their results indicate that this model is a good candidate in the presence of low particle Stokes number. [START_REF] Härtel | Analysis and direct numerical simulation of the flow at a gravity-current head. Part 1. Flow topology and front speed for slip and no-slip boundaries[END_REF]] than further used direct numerical simulation of this model to perform a detailed analysis of the flow structure exhibited at the front of this kind of flows, where Kelvin-Helmholtz instabilities appear.

To illustrate the fact that simpler models than the full Navier-Stokes equations for the carrier fluid can be used we can take as an example [START_REF] Bonnecaze | Particle-driven gravity currents down planar slopes[END_REF]. Using only the shallowwater equations, they were able to solve for a turbulent current flowing down a uniform planar slope from a constant-flux point source of particle-laden fluid. This work has later been extended by [START_REF] Ross | Gravity currents on slopes[END_REF]]. Another example of this kind would be [START_REF] Monaghan | Gravity currents descending a ramp in a stratified tank[END_REF] who used SPH to model gravity currents descending a ramp in a stratified tank.

There can be practical situations of interest where the density difference between the two fluids forming the gravity current is larger than a few percent. In order to study such flows, we cannot use the above Boussinesq approximation. We instead need to solve the mixed formulation involving the full incompressible Navier-Strokes equations with variable density (1.39) along with convection-diffusion equation for the particles concentration: By using equations 1.42, detailed numerical simulations of gravity currents released from a lock and propagating at the bottom have been first studied by [START_REF] Ungarish | Towards accelerating smoothed particle hydrodynamics simulations for freesurface flows on multi-GPU clusters[END_REF] and extended later by [START_REF] Birman | High-resolution simulations of gravity currents[END_REF]] and [START_REF] Birman | On gravity currents in stratified ambients[END_REF]] in the case of linearly stratified ambients, using this more complete model as illustrated on figure 1.5. Hoffmann and its collaborations than extended the numerical investigation to 2D buoyancy-driven flows in the presence of slopes [START_REF] Hoffmann | Sediment wave formation caused by erosional and depositional turbidity currents: A numerical investigation[END_REF]. This model has also been used by Meiburg and collaborators to take into account complex 3D sea floor topologies like bumps [START_REF] Nasr-Azadani | Turbidity currents interacting with threedimensional seafloor topography[END_REF]][Meiburg et al. 2015].

∇ • u = 0 P = P -ρ 0 gz ρ = ρ 0 (1 + αC) with α = O (1) and C ∈ [0, 1] ρ ∂u ∂t + (u • ∇) u = µ∆u -∇P + Cρg ∂C ∂t + (u • ∇) C = κ c ∆C (1.42a) (1.42b) (1.42c) (1.42d) (1.42e)
More recently, DNS and LES simulations of three-dimensional non-Boussinesq gravity currents using a discontinuous Galerkin finite elements method have been conducted by [START_REF] Bassi | Direct and Large Eddy Simulation of three-dimensional non-Boussinesq gravity currents with a high order DG method[END_REF]]. Multiple levels of stratifications have also been investigated by [START_REF] Khodkar | Gravity currents propagating into two-layer stratified fluids: vorticity-based models[END_REF]] using this model.

Double diffusive convection

Double diffusive gravity currents constitute a specific set of problems where two fluids, initially in a lock-exchange configuration, exhibit different densities that depends on some of their physical properties (concentration, temperature, ...) that diffuses at different rates creating two different density gradients. In this kind of configuration, the horizontal movement for sediment flows is initiated by initial density differences between each side of the lock as in classical lockexchange configurations that have been presented in subsection (1.3.1). After the lock-gate is withdrawn, the light fluid flows over the heavy fluid transporting dissolved chemicals, physical particles or heat by convection down through the interface between the two fluids. This forms vertical density gradients, generating double diffusive convection oriented perpendicularly to the flow direction. In practice, double-diffusive processes are interesting enough that they are often studied independently without the initial horizontal lock-exchange configuration.

In double-diffusive problems, two basic type of convective instabilities may arrise: diffusive and fingering configurations (see figure 1.6 for the latter case). Such transport processes have long been ignored because their effects have been thought to be always overwhelmed by turbulent transport, but in both cases, the double diffusive fluxes can be much larger than the vertical transport in a single-component fluid because of the coupling between diffusive and convective processes [START_REF] Turner | Multicomponent convection[END_REF]. As an example, double diffusion convection plays an important role in upwelling of nutrients and vertical transport of heat and salt in oceans. Thus, finger convection helps to support flora and fauna. It also plays an important role in controlling the climate.

The form of the resulting vertical motions depends on whether the driving energy comes from the component having the high or lower diffusivity. When one layer of fluid is placed above another denser layer having different diffusive properties, two types of instabilities may arise:

• If the stable stratification is provided by the component with the lower molecular diffusivity, the stratification will be of "diffusive" type. This can happen when small pockets of saline water are trapped near the ocean bottom in the trenches where there is a supply of geothermal heat from hot vents [START_REF] Kelley | The diffusive regime of double-diffusive convection[END_REF]].

• On the other hand if the stable stratification is provided by the most diffusive component, the stratification will be called to be of "finger" type. This configuration is occurring frequently in oceanographic studies as salt fingers [START_REF] Stern | The "salt-fountain" and thermohaline convection[END_REF]].

Double diffusive convection has been actively studied during the last six decades as its effects are not limited to oceanography. It is also of importance in the understanding in a number of other phenomena that have multiple causes for density variations [START_REF] Huppert | Double-diffusive convection[END_REF][START_REF] Chen | Precise equation of state of seawater for oceanic ranges of salinity, temperature and pressure[END_REF]]. This includes convection in magma chambers [START_REF] Huppert | Double-diffusive convection due to crystallization in magmas[END_REF]] and formation of basalt fingers [START_REF] Singh | A study of basalt fingers using experiments and numerical simulations in double-diffusive systems[END_REF] in geology, solidification of metallic alloys [START_REF] Beckermann | Double-diffusive convection during dendritic solidification of a binary mixture[END_REF]] in metallurgy, convection of solar winds due to temperature gradients and magnetic buoyancy [START_REF] Hughes | Magnetic fields in the solar convection zone: magnetoconvection and magnetic buoyancy[END_REF]] and more recently convection in Jupiter due to hydrogen-helium phase separation [START_REF] Nettelmann | An exploration of double diffusive convection in Jupiter as a result of hydrogen-helium phase separation[END_REF] in astrology. What is interesting in these systems is the wide variety of length scales and generated structures that are controlled by large variations in the governing parameters. Salt fingering in oceans operates on a length scale of centimeters over days while for the basalt columns formation the convective structures scale up to meters over time scales of decades [START_REF] Singh | Effect of Rayleigh numbers on the evolution of doublediffusive salt fingers[END_REF].

Thermohaline convection and salt fingers

Historically double diffusive convection has been studied first in oceanography where heat and salt concentrations exist with different gradients and diffuse at differing rates. The term thermohaline convection has already been introduced in 1945 to describe this specific heat-salt system by Deacon and his collaborators when they studied water circulation and surface boundaries in the oceans [START_REF] Deacon | Water circulation and surface boundaries in the oceans[END_REF]]. A decade later, Stommel and its collaborators tried to describe an "oceanographic curiosity" relative to thermohaline convection that they named the salt fountain [START_REF] Stommel | An oceanographic curiosity: the perpetual salt fountain[END_REF]]. In 1960 Stern and its collaborators studied those salt fountains and observed that gravitationally stable stratification of salinity and temperature can be explained by the fact that the molecular diffusivity of heat κ t is much greater than the diffusivity of salt κ s [START_REF] Stern | The "salt-fountain" and thermohaline convection[END_REF]]. Some years later, experimental data obtained by [START_REF] Ingham | The salinity extrema of the world ocean[END_REF] suggested that the relationship between temperature and salinity was much better described by a curve of constant density ratio R ρ rather than by a straight line as showed on figure (1.7). Those early experimental observations were later taken as an evidence of double diffusive mixing by [START_REF] Schmid | Descriptive nomenclature and classification of pyroclastic deposits and fragments[END_REF]] using the empirical model proposed by [START_REF] Chen | Precise equation of state of seawater for oceanic ranges of salinity, temperature and pressure[END_REF].

Salt fingers appear when a hot and salty water lies over cold and fresh water of higher density (see figure 1.8). Salt fingers are formed because the temperature T , which is the fastest diffusing component, contributes to the negative density gradient ( ∂ρt ∂z < 0, stable stratification) and slow diffusing salinity S contributes to the positive density gradient ( ∂ρs ∂z > 0, unstable stratification) with overall density stratification remaining gravitationally stable ( ∂ρ ∂z < 0). for sediment flows Figure 1.7 -Potential temperature versus salinity plot of four hydrographic stations in the North and the South Atlantic and the North and the South Pacific obtained by [START_REF] Schmid | Descriptive nomenclature and classification of pyroclastic deposits and fragments[END_REF]] based on the experimental work of [START_REF] Ingham | The salinity extrema of the world ocean[END_REF]] and the numerical model proposed by [START_REF] Chen | Precise equation of state of seawater for oceanic ranges of salinity, temperature and pressure[END_REF] to estimate density expansion parameters α and β. Dotted lines represents linear fit of data whereas solid lines represents optimal least squares fit of constant density ratio R ρ = α∆T β∆S . See equations (1.43) for the modelization of thermohaline convection.

Hot saline water

Cold fresh water

x z T = T 0 + ∆T S = S 0 + ∆S ρ bottom = ρ 0 = ρ(T 0 ; S 0 ) ρ top = ρ 0 (1 -α∆T + β∆S) < ρ 0 T = T 0 S = S 0
Figure 1.8 -Initial configuration for a thermohaline convection: hot and salty water lies over cold and fresh water of higher density ρ 0 . Initial temperature and salinity differences, denoted ∆T and ∆S modify the bulk density of water ρ 0 by a factor 1 -α∆T + β∆S < 1 such that the system is initially gravitationally stable. The density stability ratio R ρ > 1 is defined as the ratio between the thermal and salt density contributions. The temperature, diffusing 100 times faster than salt, will reduce density bellow the initial interface making the system gravitationally unstable before salinity can compensate for the sudden density change.

On the contrary, diffusive stratification occur when a deep layer of cold fresh water is placed above a warm salty (and heavier) layer. For example, such situation can occur in the case of a melting iceberg that inputs cold freshwater from above [START_REF] Stern | The "salt-fountain" and thermohaline convection[END_REF]]. See [START_REF] Kelley | The diffusive regime of double-diffusive convection[END_REF]] for a complete discussion about the diffusive configuration.

The usual model for salt fingering can be found in [START_REF] Schmitt | The growth rate of super-critical salt fingers[END_REF] and uses a single-phase Boussinesq fluid whose density ρ depends only on carrier fluid base density ρ 0 augmented by the local salinity S and temperature T analog to the one presented in section (1.1.6) and equations (1.40):

T = T 0 + δT S = S 0 + δS ρ = ρ 0 (1 -αδT + βδS) P = P -ρ 0 gz ∇ • u = 0 ∂u ∂t + (u • ∇) u = ν∇ 2 u - ∇P ρ 0 + (βδS -αδT )g ∂δT ∂t + (u • ∇) δT = κ t ∇ 2 δT ∂δS ∂t + (u • ∇) δS = κ s ∇ 2 δS (1.43a) (1.43b) (1.43c) (1.43d) (1.43e) (1.43f) (1.43g) (1.43h)
In this simple model, the work of pressure forces and the heat generated by viscous friction are neglected, the notation ∇ 2 has been used instead of ∆ for the Laplacian operator, α corresponds to usual thermal expansion coefficient and β to the density expansion parameter taking into account the presence of additional salt in the flow. As stated before, κ t represents the thermal diffusivity, κ s is the diffusion coefficient for salt in water and we applied a pressure shift to get P from constant bulk density ρ 0 = ρ(T 0 , S 0 ).

Within this specific setup we can estimate that around T 0 = 20°C, the kinematic viscosity of water is approximately ν = 1.003 × 10 -6 m 2 /s, the thermal diffusivity of water is κ t = 1.430 × 10 -7 m 2 /s and from [START_REF] Caldwell | Thermal and Fickian diffusion of sodium chloride in a solution of oceanic concentration[END_REF]] the diffusivity of salt in sea water is κ s = 1.286×10 -9 m 2 /s (which seems to agree with [START_REF] Poisson | Diffusion coefficients of major ions in seawater[END_REF]). This configuration yields the following dimensionless numbers comparing the different diffusivities that are involved in the double diffusive thermohaline convection process:

• Schmidt number: Sc = ν κ s = 7.799 × 10 2 = O 10 3 • Prandtl number : P r = ν κ t = 7.014 × 10 0 = O (10) • Lewis number : Le = κ t κ s = Sc P r = 1.112 × 10 2 = O 10 2 • Diffusivity ratio : τ = Le -1 = κ s κ t = 8.993 × 10 -3 = O 10 -2 for sediment flows
With those numbers, it is easy to see that in heat-salt systems, heat is the faster diffusing component (about 100 times) and that both heat and salt diffuse slower than velocity through viscous friction (by a factor 10 and 1000 respectively). The evolution of salt fingers is known to be influenced by many other non-dimensional parameters such as the thermal Rayleigh number Ra t , the salinity Rayleigh number Ra s and the initial density stability ratio R ρ :

• Initial stability ratio: R ρ = α∆T β∆S ∈ ]1, 10]

• Salinity Rayleigh number: Ra s = ρgL 3 νκ t β∆S

• Thermal Rayleigh number:

Ra t = R ρ Ra s = ρgL 3 νκ t α∆T ∈ [10 8 , 10 12 ]
It can be noted that when R ρ < 1, the density gradient becomes gravitationally unstable. The initial density stability ratio R ρ , the Schmidt number Sc and diffusivity ratio τ have been observed to be the most important non-dimensional parameter that controls the fingering behavior, see [START_REF] Taylor | Laboratory experiments on the structure of salt fingers[END_REF][START_REF] Piacsek | Nonlinear evolution and structure of salt fingers[END_REF][START_REF] Traxler | Dynamics of fingering convection. Part 1 Small-scale fluxes and large-scale instabilities[END_REF]]. However, first decades of research on salt fingers has been done in systems where Rayleigh numbers are high (10 8 or higher, see [START_REF] Kelley | Fluxes through diffusive staircases: A new formulation[END_REF]]) but it has been recently observed that variation of flux ratios (the density fluxes anomaly due to the presence of heat and salt) also greatly depend on the Rayleigh numbers [Sreenivas et al. 2009b]. Smaller Rayleigh numbers than the ones naturally occurring for heat-salt systems were experimentally investigated by [START_REF] Krishnamurti | Salt fingers at low Rayleigh numbers[END_REF] and numerically by [START_REF] Singh | Effect of Rayleigh numbers on the evolution of doublediffusive salt fingers[END_REF]. Finally it has been shown by [START_REF] Huppert | Limiting conditions for salt-fingering at an interface[END_REF]] that the stability criterion for finger formation in a two-layer system is 1 R ρ Le 3/2 . Note that P e can be much smaller for other configurations like salt-sugar systems where Le 3.

While forty years ago it was unclear that salt-fingering even existed outside of the laboratory, it is now well understood that the vertical flux of sea salt is dominated by double diffusive convection [START_REF] Schmitt | An estimate of the vertical mixing due to salt fingers based on observations in the North Atlantic Central Water[END_REF]. It was also observed that a small amount of horizontal water movement or turbulence could destroy the fingers [START_REF] Linden | Salt fingers in the presence of grid-generated turbulence[END_REF][START_REF] Konopliv | The influence of shear on double-diffusive and settling-driven instabilities[END_REF]]. However, the fingers usually reformed within a few minutes when the flow became laminar again.

Double diffusive sediment-laden flows

During the seventies, a simple laboratory experiment showed that a similar fingering phenomenon occurred when a warm fluid, made denser by the presence of suspended material, was put above colder, distilled water [START_REF] Houk | Descent rates of suspension fingers[END_REF]. As for thermohaline convection, the presence of an originally sharp horizontal interface between two such fluid layers, the system is double-diffusive and vertical sediment fingering motion occurs [START_REF] Green | The importance of double diffusion to the settling of suspended material[END_REF]]. This sediment fingering can also occur when vertical sediment concentration gradients are associated with vertical temperature gradients instead of salinity gradients [START_REF] Maxworthy | The dynamics of sedimenting surface gravity currents[END_REF]]. Temperature can also be complementary to salinity as observed in hypersaline lakes [START_REF] Ouillon | Halite precipitation from double-diffusive salt fingers in the Dead Sea: Numerical simulations[END_REF]]. The vertical motion of suspended particles being fundamental to sedimentation, it was then anticipated that the resulting double-diffusive instabilities could greatly contribute to the distribution of nearshore sediments when compared to gravitational settling alone in laminar flows.

In the nature, this event has been observed when a muddy river enters a stratified lake or ocean [START_REF] Schmitt | An estimate of the vertical mixing due to salt fingers based on observations in the North Atlantic Central Water[END_REF]. Such sediment-laden riverine outflows can be classified as either hypopycnal or hyperpycnal currents. An hyperpycnal current is a flow in which the density of the river is more than that of the stratified ambiant, so that the river continues to flow near the bottom of the sea floor [START_REF] Mulder | Marine hyperpycnal flows: initiation, behavior and related deposits. A review[END_REF]]. Such flows have been numerically investigated in [START_REF] Schulte | Clear salt water above sediment-laden fresh water: Interfacial instabilities[END_REF]]. The flows of interest here are hypopycnal currents, where the combined density fresh water and suspended sediments is less than the density of the ocean. In this case the river continues to flow along the top of the stratified ambiant and sediment fingering can occur. As predicted, the resulting downward sediment flux can be orders of magnitude above the one that would normally be obtained with only gravitational settling [START_REF] Hoyal | The influence of diffusive convection on sedimentation from buoyant plumes[END_REF][START_REF] Parsons | P3DFFT: A framework for parallel computations of Fourier transforms in three dimensions[END_REF]]. It has been later observed that double-diffusive fingering was not the only type of instabilities that could arrise. When sediment-laden water flows over clearer and denser water, the dominant instability mode may instead become Rayleigh-Taylor like, in which case the instabilities are located at the lower boundary of the particle-laden flow region [START_REF] Burns | Sediment-laden fresh water above salt water: linear stability analysis[END_REF].

The important parameter that drives the instabilities has been identified as being the ratio of the particle settling velocity to the diffusive spreading velocity of salinity or temperature layer that initially provides the stable stratification [Burns et al. 2015]. While the particle settling velocity can be estimated from physical particle properties [START_REF] Gibbs | Estuarine flocs: their size, settling velocity and density[END_REF]], the diffusivity of salinity κ s or temperature κ t is fully determined by the knowledge of associated Schmidt number between the considered field and water. The upward layer spreading velocity depends on this Schmidt number but also the other dimensionless parameters of the system, such as the stability ratio R ρ and the diffusivity ratio τ between the salinity and sediments (or between temperature and sediments). In order for sediment fingers to form in hypopycnal flows, the particles have to diffuse slower than the considered agent (τ > 1).

The numerical models used to explore the nonlinear behavior of convective sedimentation in hypopycnal flows are similar to the one of the heat-salt system (1.43). Those models consider dilute monodisperse distributions of fine particles with negligible inertia that can be considered as a continuum. The particle concentration C contributes positively to the density and is transported into the carrier fluid with respect to some relative particle settling velocity V st that is usually the same everywhere in the flow [START_REF] Yu | Convective instability in sedimentation: Linear stability analysis[END_REF]]. In the case of the particle-salt system, equations (1.43c) and (1.43g) are thus replaced by the following expressions:

ρ = ρ 0 (1 + αδS + βδC) ∂δC ∂t + [(u -V st e z ) • ∇] δC = κ c ∇ 2 δC (1.44a) (1.44b)
The full set of equations relating to this particle-salt system is introduced in chapter 4.

Chapter 1. High performance computing for sediment flows

The particle settling velocity can be determined directly from the grain diameter d for given values of carrier fluid viscosity ν 0 and density ρ 0 . For dilute suspensions, Stokes law predicts the settling velocity of small spheres in fluid:

V st = Rgd 2 C 1 ν 0 R = ρ c -ρ 0 ρ 0 (1.45a) (1.45b)
where ρ c is the density of particles, R represent the submerged specific gravity and C 1 is a constant that has a value of 12 for smooth spheres, and that is larger for non-spherical or rough particles [START_REF] Raudkivi | LIX. On convection currents in a horizontal layer of fluid, when the higher temperature is on the under side[END_REF]]. Stokes law holds for particle Reynolds numbers bellow unity, else the settling of large particles is slowed down by the turbulent drag of the wake behind each particle:

V st = 4Rgd 3C 2 (1.46a)
with C 2 = 0.4 for smooth spheres and C 2 = 1 for natural grains. The expressions for Stokes flow and turbulent drag law can be combined into a single expression that works for all sizes of sediment [START_REF] Ferguson | A simple universal equation for grain settling velocity[END_REF]]. The fall velocity of various sizes of particle estimated with this law is shown on figure 1.9. The only considered particles being the ones with negligible inertia (clay and fine silts), the settling velocity of a single particle will always be determined by Stokes law (1.45). The small diffusion coefficient κ c associated to the particles accounts for Brownian motion or the mixing that would occur in real polydisperse suspensions. In practice, for grain sizes between 1 and 20µm, the particle diffusivity is mainly induced by the velocity fluctuation due to the presence of neighboring particles [START_REF] Yu | Convective instability in sedimentation: Linear stability analysis[END_REF]. Depending on the volume fraction of particles φ, the neighboring particles also have an influence on effective viscosity and settling velocity. A semi-empirical formulation based on the particle diameter d and particles Stokes velocity V st has been proposed by [START_REF] Segre | An effective gravitational temperature for sedimentation[END_REF]:

ν c (φ) = ν 0 (1 -φ/0.71) -2 V c (φ) = V st (1 -φ) 5 κ c (φ) = K(φ)V st r (φ) = 11.4 (1 -φ) 2 (1 -φ/0.71) 2 (1 + 4φ + 4φ 2 -4φ 3 + φ 4 ) (1.47a) (1.47b) (1.47c) (1.47d)
where r = d/2 is the particle radius and ν c and V c are respectively the effective viscosity of the sediment-laden mixture and the effective settling velocity of the particles.

As shown on figure 1.10, for dilute suspensions under φ = 2%, the resulting values stay really close to the particle properties obtained at infinite dilution. The average yearly sediment concentration of dirty rivers ranges from 10 to 40kg/m 3 [START_REF] Mulder | Turbidity currents generated at river mouths during exceptional discharges to the world oceans[END_REF]. For quartz (ρ c 2650kg/m 3 ) this corresponds roughly to volume fractions between 0.4% and 1.5%. When a river outflow is loaded with a large amount of suspended sediment (more than 40kg/m 3 ) the density of the mixture becomes greater than the one of seawater and the flow becomes hyperpycnal. The diffusivity of salt in sea water has already been estimated in the last subsection, leading to κ s = 1.286 × 10 -9 m 2 /s. If we estimate κ c for sediment flows from equations (1.47) for a given volume fraction and particle size, we can also compute the diffusivity ratio τ = κ s /κ c . This is illustrated on figure 1.11.

Figure 1.11 -Estimation of the diffusivity ratio between salinity and particles.

From figure 1.11 we can estimate that τ 25 for a particle diameter of 2µm. This is effectively the usual values that we can find in the literature [START_REF] Yu | Convective instability in sedimentation: Linear stability analysis[END_REF]][Burns et al. 2015]. The dimensionless parameters corresponding to grain sizes of 2µm and volume fractions of roughly 1% are usually the following:

• Dimensionless settling velocity: V p 0.04

• Schmidt number: S c = ν 0 κ s 700

• Diffusivity ratio: τ = κ s κ c 25

• Initial stability ratio: R ρ = α∆S β∆C 2

The value R ρ = 2 corresponds to a mass loading of roughly 20kg/m 3 of sediments and a salinity of 3%. The Schmidt number between sediments and water can be computed as ν 0 /κ c = τS c 17500. Because of the high Schmidt numbers involved and associated Batchelor scales for salinity and particle concentration, numerical simulations with excessively large values of S c or τ cannot be performed (see section 1.1.10). With τ = 25, simulations up to S c = 70.0 in 2D and S c = 7.0 in 3D have been performed in [Burns et al. 2015]. Accessing higher Schmidt numbers is the current challenge to be able to fully understand sediment fingering naturally occurring near sediment-laden riverine outflows. The Batchelor scale of fine particles being around η B = η K / √ S c η K /132, a multilevel approach with a grid ratio of 132 in each direction could be considered. Even for a two-dimensional flow, this will require important computational ressources and distributed high performance computing is here mandatory. This work aims to achieve high Schmidt number sediment-laden flow simulations by using adapted numerical methods in a high performance computing context.

High Performance Computing

High performance computing (HPC) is a field of applied mathematics that consists in using supercomputers for applications that generally target scientific computing. The goal is to take full advantage of all the available compute resources to solve a given numerical problem. When high performance computing is used to solve multiple independent problems on a set of resources of a computing machine, the problem is said to be embarrassingly parallel. In this case little to no effort is needed to separate the problem into a number of parallel tasks. When a set of compute resources is used to solve a single problem, the numerical method employed has to be distributed on the computational elements in a collaborative manner. This usually implies communication of intermediate results between the different compute ressources. The implementation is here less straightforward because the problem has to be decomposed into parallel tasks. This often requires to rethink the underlying algorithms and to find a compromise between calculations and communications. As communication between distant compute ressources is usually a costly operation, it is not uncommon to perform redundant calculations to be able to reduce the volume of communications.

Hierarchy of a computing machine

A computing machine is made up of different hierarchical level of resources. It is composed of interconnected compute nodes that generally comprises a hierarchy of memory (multiple levels of caches, RAM, local disks), processors and possibly accelerators (coprocessors or graphics cards). Including multiple cores on a single chip has become the dominant mechanism for scaling processor performance. Modern supercomputers are thus hierarchical and the hierarchy depth tends to grow [START_REF] Smith | Future general purpose supercomputer architectures[END_REF][START_REF] Fu | The Sunway TaihuLight supercomputer: system and applications[END_REF]. Structural hierarchy (core, socket, node, rack, system) implies significant differences in communication time [START_REF] Blaauw | Computer architecture: Concepts and evolution[END_REF]]. Memory hierarchy also induces differences in access time: the larger is the size of the level, the slower is the access and the data movement overhead become the most significant factor of inefficiency.

A given compute node containing multiple processors usually exhibit one of the following memory architecture [START_REF] Lameter | NUMA (Non-Uniform Memory Access): An Overview[END_REF]]:

• Uniform memory access: UMA is a shared memory architecture used in parallel computers where all the processors share the physical memory uniformly. In an UMA architecture, access time to a memory location is independent of which processor makes the request or which memory chip contains the transferred data.

• Non-uniform memory access: On NUMA architectures, memory access time depends on the memory location relative to the processor. A processor can access its own local memory faster than non-local memory.

A compute node usually contain one or more sockets (NUMA nodes), that each can contain one or more physical CPUs (corresponding to the physical sockets on the motherboard). A for sediment flows compute cluster contain one or more compute nodes and effectively constitute a distributed memory architecture. Most usual cluster configurations ranges from one to eight sockets per compute node, each containing a single physical multi-core CPU.

Evaluation of performance

Performance of supercomputers is usually evaluated in terms of floating point operations per second (FLOPS or FLOP/s) because most scientific computations require floating-point calculations. Although the notion of single-and double-precision is architecture specific, we will here always refer to 32 bits floating point arithmetic (FP32) as single-precision and 64 bits floating point arithmetic (FP64) as double precision. The TOP500 project was started in 1993 and ranks the 500 most powerful supercomputer systems in the world [START_REF] Meuer | Die TOP 500 Supercomputer in der Welt[END_REF]]. Since 1993, performance of the number one ranked position has grown in accordance with the revised Moore's law, doubling roughly every 18 months [START_REF] Moore | Progress in digital integrated electronics[END_REF]]. As of June 2019, Summit is the fastest supercomputer with theoretical peak computing performance of 200.8 PFLOPS of double-precision floating point arithmetic. As a comparison, this is over 1.53 × 10 6 times faster than the Connection Machine CM-5/1024, which was the fastest system in November 1993 with 131.0 GFLOPS theoretical peak computing performance. This represents a performance doubling every 15 months. Even tough, for the first time, all top 500 systems deliver a petaflop or more on the High Performance Linpack (HPL) benchmark [START_REF] Dongarra | Toward a new metric for ranking high performance computing systems[END_REF]], Moore's law is slowing down as we get closer and closer to physical limitations of current lithography processes [START_REF] Waldrop | The chips are down for Moore's law[END_REF]].

Figure 1.12 -Exponential growth of supercomputing power -TOP500 list HPL is a simple benchmark that factors and solves a large dense system of linear equations using Gaussian elimination with partial pivoting. Figure 1.12 shows the evolution of su-percomputing power as achieved on the HPL benchmark (which is less then the theoretical peak compute). Currently the combined power of all top 500 supercomputers exceeds the exaflop, or one quintillion floating point operations per second. Exascale computing refers to computing systems capable of at least 1 EFLOPS, and the first systems to break the exascale compute wall are expected in the upcoming years [START_REF] Geller | Supercomputing's exaflop target[END_REF]]. When HPL gained prominence as a performance metric in the early nineties, vendors logically pursued architectural designs that would increase HPL performance. As there was a strong correlation between benched performance and general application performance, this in turn improved real application performance. While microprocessors have increased in speed at a rate of 60%/year in the last decades, access times to memory have only been improving at a rate of less than 10%/year [START_REF] Carvalho | The gap between processor and memory speeds[END_REF]]. Because of the exponential growing disparity between compute and memory performance the HPL metric now gives a skewed picture relative to application performance where the computation-to-data-access ratios (FLOP/B) are low. The HPL benchmark is only relevant for high computation-to-data-access ratios as obtained for dense matrix-matrix multiplications [START_REF] Dongarra | Toward a new metric for ranking high performance computing systems[END_REF]]. Since November 2017, a new benchmark has been introduced in the TOP500 project to evaluate computational and data access patterns that more closely match a broad set of scientific applications, leading to an alternative ranking of the TOP500 supercomputers.

This new benchmark, named High Performance Conjugate Gradients (HPCG), has been designed to put more stress on memory and interconnect by performing sparse matrix operations. This benchmark is especially designed for HPC applications that require higher memory bandwidth and lower memory latencies. It aims to deter hardware vendors from making design choices that would be detrimental for a wide range of applications, but beneficial for the HPL benchmark. Such applications are usually memory-bound on current architectures, and do not benefit at all from additional floating point compute units. On the contrary, when a given algorithm is limited by computing power instead of memory bandwidth, it is said to be compute-bound. Because HPCG is memory-bound, this benchmark generally achieves only a tiny fraction of the peak FLOPS of a given supercomputer. As an example, the Summit supercomputer only achieves 2.93 PFLOPS on HPCG versus 148.6 PFLOPS on the HPL benchmark. For a given algorithm, the computation-to-data-access ratio in FLOP/B is defined as the arithmetic intensity. Particle methods and dense linear algebra (HPL) perform the order of 10 floating point operations per byte loaded or stored in memory. Those algorithms are typically compute-bound on current architectures. Sparse linear algebra (HPCG) and stencil computations perform less than 1 FLOP/B and are usually memory-bound. Spectral methods and Fast Fourier Transforms (FFT) have intermediate arithmetic intensities and may be either memory-or compute-bound for a given architecture [START_REF] Williams | Roofline: An insightful visual performance model for floating-point programs and multicore architectures[END_REF]]. An associated visual performance model is described in section 1.4.7.

Last but not least, the performance can also be evaluated in terms of power efficiency. As power consumption of supercomputers increases, energy efficiency will move from desirable to mandatory in order to push beyond the exascale limit [START_REF] Hemmert | Green hpc: From nice to necessity[END_REF]]. The GREEN500 is a third list that classifies supercomputers in terms of achieved floating point operations per second per watt (FLOPS/W). Here Summit is ranked second with 14.72 GFLOPS/W, for a total consumption of roughly 10 MW.

Accelerators and coprocessors

Since 2010, high performance computing system architectures have shifted from the traditional clusters of homogeneous nodes to clusters of heterogeneous nodes with accelerators [START_REF] Kindratenko | Trends in high-performance computing[END_REF]. Hardware acceleration consists in the use of computing hardware specially designed to perform some operations more efficiently than in software running on general-purpose CPUs. Accelerators include many integrated core architectures (MICs), graphics processing units (GPUs), digital signal processors (DSPs), application-specific instruction-set processors (ASIPs), field-programmable gate arrays (FPGAs) and other processors or hardware accelerators. In the realm of high performance computing, the hardware accelerators have to be reconfigurable and the current tendency is clearly in favour of dedicated GPUs, followed to a lesser extent by MICs and a slow but increasing adoption of FPGAs [START_REF] Weber | Comparing hardware accelerators in scientific applications: A case study[END_REF]]. The TOP500 of June 2019 lists 134 supercomputers containing accelerators. Those supercomputers represent roughly 40% of the total computing power of the 500 machines (in terms of HPL performance) as exposed on figure 1.13. Accelerator architectures tend to be either heavily specialized or very generic and both of these models present real challenges when it comes to programming. General-purpose processing on graphics processing units (GPGPU) is the use of a GPU as a coprocessor for general-purpose computing [START_REF] Luebke | GPGPU: general-purpose computation on graphics hardware[END_REF]]. The GPU accelerates applications running on the CPU by offloading some of the compute-intensive portions of the code while the rest of the application still runs on the CPU. The share of coprocessors on the 134 supercomputers containing accelerators is further described in figure 1.14. Here the advantage is clearly to Nvidia GPUs, as most of the accelerators present in the TOP500 are last generation Volta GPUs (Volta GV100, Tesla V100, Tesla P100 and variants). Out of those 134 machines, 62 contain Nvidia Volta GPU architectures (46.6%), 50 contain NVidia Pascal GPU architectures (37.6%), 12 contain NVidia Kepler GPU architectures (9%) and 5 contain Intel Xeon Phi MICs (3.8%). The remaining machines contain either custom accelerators or NVidia Fermi GPUs. Due to thermal and energy constraints, the trend in processor and accelerator development has been towards an ever-increasing number of cores with reduced frequency. The exponential growth in the number of cores is expected to lead to consumer grade CPUs containing hundreds of cores in the near future [START_REF] Mccool | Scalable programming models for massively multicore processors[END_REF]]. Another way to get extend the computing power of a supercomputer is to simply add more compute nodes. As the number of processors and compute nodes increase, efficient interprocessor communication and synchronization on a supercomputer becomes a challenge. In the nineties, two parallel programming open standards grew to dominate the parallel computing landscape:

• Message Passing Interface: MPI is a specification for a standard library for message passing that was defined by the MPI Forum, a group of parallel computer vendors, library writers, and applications specialists. MPI includes point-to-point and collective internode communications routines through communicator abstractions [START_REF] Barker | Message passing interface (mpi)[END_REF]].

The third version of the standard (MPI-3) introduced remote memory accesses [START_REF] Hoefler | Remote memory access programming in MPI-3[END_REF]]. Because it is a standard, multiple open-source implementations of MPI have been developed such as MPICH [START_REF] Walker | MPI: a standard message passing interface[END_REF] and OpenMPI [START_REF] Gabriel | Open MPI: Goals, concept, and design of a next generation MPI implementation[END_REF]].

Implementations optimized for vendor-specific hardware are also available directly from hardware vendors.

• Open Multiprocessing: OpenMP is an implementation of multithreading. Once spawned, threads run concurrently, with the runtime environment allocating threads to different physical CPU cores. It is an industry-standard API for shared-memory programming [START_REF] Dagum | OpenMP: An industry-standard API for shared-memory programming[END_REF]] targeting C, C++ and Fortran and consists in a set of compiler directives, library routines, and environment variables that influence runtime behavior. The standard require specific compiler support [START_REF] Novillo | OpenMP and automatic parallelization in GCC[END_REF]] and as for MPI, both open-source and vendor-specific implementations exist. As for OpenMP 4.1, the standard began to introduce accelerator support [START_REF] Antao | Offloading support for OpenMP in Clang and LLVM[END_REF].

Those two standards still require an in-depth understanding of computing ressource hierarchies. Multicore architectures with OpenMP requires an explicit management of the memory hierarchy while distributed computing with MPI requires explicit communications between the different processes that are mapped to the physical compute nodes. As MPI process placement can play a significant role concerning communication performance [START_REF] Rashti | Multi-core and network aware MPI topology functions[END_REF]], tools such as TopoMatch, have been developed in order optimize the placement of processes on NUMA compute nodes with respect to interprocess data exchange [START_REF] Jeannot | Near-optimal placement of MPI processes on hierarchical NUMA architectures[END_REF]]. Concerning OpenMP, thread and memory placement can be tweaked by using the numactl utility [START_REF] Kleen | A numa api for linux[END_REF]]. While those two standards still plays an important role in high performance computing, the learning curve can be steep for some programmers. A survey of programming languages that try to hide the complexity of multicore architectures is available in [START_REF] Kim | Multicore software technologies[END_REF] An application built with the hybrid model of parallel programming can run on a cluster using both OpenMP and MPI. In this case OpenMP is used for parallelism within a compute for sediment flows node (or within a NUMA node) while MPI is used for internode communication. OpenMP can be replaced by other multithreading implementations such as POSIX threads [START_REF] Barney | POSIX threads programming[END_REF]] and OpenCL. Since 2006, the two industry standards for GPU programming are CUDA [START_REF] Luebke | CUDA: Scalable parallel programming for high-performance scientific computing[END_REF] and OpenCL [START_REF] Stone | OpenCL: A parallel programming standard for heterogeneous computing systems[END_REF]]. While the former has been designed only for Nvidia GPUs, the latter can drive a wide range of CPUs and accelerators. In this work we will focus on hybrid MPI-OpenCL programming.

The OpenCL standard

OpenCL is a framework for writing programs that execute across heterogeneous platforms consisting of central processing units (CPUs), graphics processing units (GPUs

Target OpenCL devices

In this work we will focus on performance obtained on Intel and Nvidia OpenCL platforms on the four following compute devices:

• Server grade dual socket Intel Xeon E5-2695 v4 CPUs (36 physical cores, 128GB RAM)

• Server grade GPU: Nvidia Tesla V100 (5120 CUDA cores, 32GB VRAM)

• Consumer grade CPU: Intel Core i7-9700K CPU (8 physical cores, 16GB RAM)

• Consumer grade GPU: Nvidia GeForce RTX 2080 Ti (4352 CUDA cores, 11GB VRAM)

Table 1.1 and 1.2 contain theoretical and achieved performance characteristics of the four considered benchmark configurations.

The speed of the bus interconnect usually constitute the bottleneck for host-device interactions but host-to-host and device-to-device memory bandwidth are also of importance in the design of OpenCL based numerical codes. Actual generation of PCIe buses (PCIe3.0 16x) provides 16 GB/s of theoretical bidirectional memory bandwidth (host-to-device and device-to-host) for dedicated accelerators. This may be slow compared to the device-to-device memory bandwidth, also called global memory bandwidth, provided by the two considered GPUs which is of the order of 1TB/s. When the data can entirely fit in device memory, a given kernel is either compute or memory bound depending on the device global memory bandwidth and compute capabilities for the required type of operations (single or double precision floating point operations, integer operations, ..

.). for sediment flows

This can be summed up into the designed arithmetic intensity of a given device which is given in operations per byte of memory traffic (OP/B). It can be seen in table 1.3 that the average device will require fifty operations per element loaded from memory in order not to be memory bound. Arithmetic intensity represent the minimal number of operations to execute per byte loaded in order to be compute bound. Data corresponds to the worst case scenario where we consider only one memory transaction of the given type. The numbers between parenthesis reflect more realistic values for the considered OpenCL platforms (using actual bandwidth and clock).

Double precision operations on consumer grade GPUs constitute an exception to this rule because of they target graphical tasks that mostly require single-precision operations. In practice because of the small number of operations per byte required for most of the numerical kernels we will develop in this work, the situation is likely to be memory bound. As an example, explicit finite differences based solvers use stencils operations while implicit finite differences solvers use sparse algebra. Both of those methods have very low arithmetic intensity. The performance of memory-bound kernels can quickly drop when the full potential of the memory controller is not used. How to correctly access memory is architecture dependent but most of accelerators use vector operations in a SIMD fashion [START_REF] Duncan | A survey of parallel computer architectures[END_REF]]. It usually entails to access contiguous memory with the good granularity and alignment. Grouping of work-items into work-groups is not only relevant to computation, but also to global memory accesses.

The device coalesces global memory loads and stores issued by neighboring work-items of a work-group into as few transactions as possible to minimize required memory bandwidth. For strided global memory access, the effective bandwidth is poor regardless of architecture and declines with increasing stride (distance between two accessed elements).

Figure 1.17 -Achieved memory bandwidth vs. memory transaction size Each result (black cross) corresponds to the median bandwidth obtained for 1024 device-todevice copies performed with the builtin OpenCL copy kernel on the Tesla V100-SXM2 GPU.

Global memory accesses on cache miss have a latency of hundreds device cycles (of the order of 100ns). In order to achieve the full device bandwidth, this latency needs to be hidden using the following strategies:

• Perform contiguous memory accesses: adapt the data structures and reformulate associated algorithm.

• Perform independent loads and stores from the same work-item: increase problem size or increase work-item workload by playing on the global work-space size (global size).

• Perform memory transactions from increased number of work-items (local size)

• Use larger word sizes by using vector transactions: use float4 instead of float when the device is designed to handle 16B per work-item per coalesced memory transaction. for sediment flows

All those strategies are so that there is enough memory transactions in flight to saturate the memory bus (the price of latency for small memory transaction is shown on figure 1.17).

Instead of relying only on some compiler automatic vectorization process, the OpenCL standard provide vector types and vector intrinsics to manually vectorize kernels. Those vector intrinsics have a great impact on achieved memory bandwidth as shown on figure 1.18 for single precision floating point vectors.

Figure 1.19 clearly shows that for the Tesla V100 GPU, 16B per work-item per memory transaction is optimal. Considering that all types perform equally well at given type size (which is surprisingly not the case here for 64B transactions) one can deduce from figure 1.18 that the two CPUs prefer 64B transactions and that the consumer grade GPU is designed for 32B transactions. Those values do not match the preferred and native vector sizes provided by the OpenCL platforms and encourages microbenchmarking for the design of any kernel. A given hard-coded kernel will thus be portable across our four platforms but performance will not. The solution to this problem is to implement one kernel per vector type, and this can be done with a simple code generation technique associated to some kernel template. The best kernel can then be determined at runtime by self-tuning [START_REF] Dolbeau | One OpenCL to rule them all?[END_REF]]. In fact, kernel auto-tuning can be extended to determine other runtime parameters such as the local and global grid sizes that have an effect on device occupancy and work-item workload. Median bandwidth achieved for 1 GiB buffers at 1 op/E over 1024 runs per kernel. The memory transaction ratio is the amount of memory loads compared to the total number of memory transactions (loads + stores). A different kernel is generated for every data point.

The roofline model

When a kernel may be compute bound we can switch to performance model that also take into account the compute capabilities of the device. The roofline model is an intuitive visual performance model used to provide performance estimates of a given compute kernel for sediment flows running on a given architectures by showing inherent hardware limitations [START_REF] Williams | Roofline: An insightful visual performance model for floating-point programs and multicore architectures[END_REF]. The most basic roofline model can be visualized by plotting performance as a function of device peak performance Π and device peak bandwidth β (see table 1.2). For a given hardware, the best attainable performance is expressed by P = min(Π, β I) where I is the arithmetic intensity in operations per byte. The resulting plot usually exhibits fractions of maximum attainable performance P with both axes in logarithmic scale. Within this model, the performance obtained for a given kernel can be plotted as a single point which coordinates represents the arithmetic intensity I of the considered algorithm (this value is fixed for a given algorithm and do not depend on kernel optimizations) and the obtained performance obtained from microbenchmarking (most of the time in operations per second). A kernel is said to be memory-bound when I ≤ Π/β and compute bound when I ≥ Π/β (see table 1.3).

A reference roofline for single and double precision floating point operations can be obtained with the mixbench benchmark [Konstantinidis et al. 2017]. It generates kernel of increasing arithmetic intensity (in FLOP/B) and run them on arrays with fixed number of elements (an element has a size 4B for float and 8B for double). The joint knowledge of the total number of elements, the arithmetic intensity and the result of the runtime is required to deduce the obtained performance in FLOP/s. The following figure shows the performance obtained with this benchmark on the two GPUs for single and double precision operations: All fluid solvers are based on a numerical approximation of the variables and the equations both in terms of time and space. The common principle is to spatially discretize the field of computation in various elements: grid, mesh or particles. In any case, the solution is given on these discrete elements and an increase in the number of elements leads to a better spatial description of the solution. Timesteps are usually chosen to be the largest possible while ensuring the stability of the underlying numerical schemes. Most methods allow an estimation of the approximation errors in terms of spatial and temporal resolutions of the discretization.

In practice, obtaining results on finer resolutions generally allows greater accuracy in the resolution of the considered problem, which contributes to a better understanding of the physical mechanisms involved. The consequence is that the computing resources necessary for a refinement of numerical solutions become important. Hence, flow simulation relies heavily on high performance computing to accurately simulate complex fluid problems [START_REF] Tezduyar | Flow simulation and high performance computing[END_REF]].

In the passed years, many high-performance numerical fluid-related solvers have been released and many other research-oriented closed-source solvers are still in active development:

• Finite element based solvers:

-FEniCS is a C++/Python library to translate finite element variational formulations into efficient distributed finite element code [START_REF] Logg | Automated solution of differential equations by the finite element method: The FEniCS book[END_REF]. It is based on DOLPHIN, an automatic finite element code generator [START_REF] Logg | DOLFIN: Automated finite element computing[END_REF]. This solver has been used to simulate turbulent flows on massively parallel architectures and scales up to 5000 cores [START_REF] Hoffmann | Sediment wave formation caused by erosional and depositional turbidity currents: A numerical investigation[END_REF]].

-Fluidity is a general purpose, multiphase computational fluid dynamics code capable of numerically solving the Navier-Stokes equations on arbitrary unstructured finite element meshes. It is parallelised using MPI and is capable of scaling to many thousands of cores [START_REF] Davies | Fluidity: A fully unstructured anisotropic adaptive mesh computational modeling framework for geodynamics[END_REF]].

-Feel++ is another C++ MPI-based finite element (and spectral element) solver based on domain decomposition [START_REF] Prud'homme | Feel++: A computational framework for galerkin methods and advanced numerical methods[END_REF]]. It implements one-, two-and three-dimensional Galerkin methods and can be applied to fluid mechanics. It scales up to thousands of cores [START_REF] Abdelhakim | FEEL++ applications to engineering problems[END_REF]].

• Finite-volume based solvers:

-OpenFOAM is a C++ toolbox for the development of custom numerical solvers targeting continuum mechanics problems and most prominently computational fluid dynamics [START_REF] Chen | OpenFOAM for computational fluid dynamics[END_REF]. Although the solver is MPI-based, multiple parts of the library have been accelerated by introducing GPU support [START_REF] Malecha | GPU-based simulation of 3D blood flow in abdominal aorta using Open-FOAM[END_REF][START_REF] He | Acceleration of the OpenFOAMbased MHD solver using graphics processing units[END_REF]. This solver has been used to solve the three-dimensional two-phase eulerian model for sediment transport [START_REF] Chauchat | SedFoam-2.0: a 3-D two-phase flow numerical model for sediment transport[END_REF]].

-MRAG is an hybrid CPU-GPU wavelet adaptive solver based on high-order finite volume scheme and OpenCL. It has been implemented to solve multiphase compressible flows [START_REF] Rossinelli | Multicore/multi-gpu accelerated simulations of multiphase compressible flows using wavelet adapted grids[END_REF].

Chapter 1. High performance computing for sediment flows -Using an hybrid OpenMP-MPI NUMA-aware solver, [START_REF] Hejazialhosseini | High throughput software for direct numerical simulations of compressible two-phase flows[END_REF]] performed a three-dimensional simulation of shock-bubble interaction discretized on 200 billion computational elements (9728 × 6656 × 3072). Achieved performance was 30% of the total peak computing power of a cluster containing 47.000 cores.

Using one MPI process per NUMA node and replacing OpenMP by Intel TBB improved the runtime [START_REF] Willhalm | Putting intel® threading building blocks to work[END_REF]].

-COOLFluid is an open computational platform for multi-physics simulation and research. It features a collection of parallel numerical solvers for unstructured meshes such as finite-differences and finite-volume based solvers for computational fluid dynamics problems [START_REF] Lani | COOLFluiD: an open computational platform for multi-physics simulation and research[END_REF]. Some parts of the solver are GPUenabled [START_REF] Lani | A GPU-enabled Finite Volume solver for global magnetospheric simulations on unstructured grids[END_REF]] and the implicit multi-fluid finite volume solver has been successfully run on up to 60.000 cores.

• Lagrangian particle methods:

-Many GPU enabled SPH implementation have been proposed [START_REF] Amada | Particle-based fluid simulation on GPU[END_REF][START_REF] Hérault | Sph on gpu with cuda[END_REF] -GADGET-2 is a code for collisionless cosmological simulations using SPH. A parallel version has been designed to run with MPI and reached total particle numbers of more than 250 million [START_REF] Springel | TTC: A tensor transposition compiler for multiple architectures[END_REF]. A third version of the solver is being written to prepare for the exascale era [START_REF] Goz | Cosmological Simulations in Exascale Era[END_REF]].

-A new family of eulerian solvers based on lagrangian particles and finite-volume approximation (Lagrange-flux) is being developed as an alternative to Lagrangeremap schemes with better scalability in mind [De Vuyst 2016a][De Vuyst 2016b].

-Partial GPU acceleration of eulerian-lagrangian particle-laden turbulent flow simulations has been performed by [START_REF] Sweet | GPU acceleration of Eulerian-Lagrangian particle-laden turbulent flow simulations[END_REF]].

• Remeshed particle methods:

-PPM has been as the first highly efficient MPI parallel solver based on remeshedparticles [START_REF] Sbalzarini | PPM-A highly efficient parallel particle-mesh library for the simulation of continuum systems[END_REF]].

It provides a general-purpose, physicsindependent infrastructure for simulating systems using particle methods. Using PPM along with the vortex method, simulations of vortex rings with up to 6 × 10 9 particles were performed on 16.000 cores [Chatelain et al. 2008b]. The same method was then applied to numerical simulations of aircraft wakes [Chatelain et al. 2008a] -In the mean time, two-dimensional remeshed particle methods were ported to GPU by using CUDA and extended with vortex penalization techniques [START_REF] Rossinelli | Vortex methods for incompressible flow simulations on the GPU[END_REF][START_REF] Rossinelli | GPU accelerated simulations of bluff body flows using vortex particle methods[END_REF]]. Corresponding spectral fluid solver was implemented by using the CUDA FFT library (cuFFT).

High Performance Computing

57

-SCALES: This library proposes a MPI-based Fortran implementation of remeshedparticles methods. It has been successfully run up to 8192 cores with 89% scalability for grids of size 2048 3 [START_REF] Lagaert | Hybrid spectral-particle method for the turbulent transport of a passive scalar[END_REF]].

-HySoP: In parallel an hybrid MPI-OpenCL code has been developed for the transport of a scalar in a turbulent flow [START_REF] Etancelin | Multi-CPU and multi-GPU hybrid computations of multi-scale scalar transport[END_REF]]. The fluid solver was also FFT-based (FFTW) and solved on CPUs while the scalar, defined on a finer grid, was transported on GPUs.

-MRAG-I2D: This is an open source software framework for multiresolution of twodimensional incompressible viscous flows on multicore architectures (OpenMP). The solver uses remeshed particles and high order interpolating wavelets [START_REF] Rossinelli | MRAG-I2D: Multi-resolution adapted grids for remeshed vortex methods on multicore architectures[END_REF]].

• Spectral and pseudo-spectral methods:

-Tarang is a general-purpose pseudo-spectral parallel code developed for turbulence and instability studies. It has been validated up to grid of size 4096 3 and offers good weak and strong scaling up to several thousand processors. It features periodic as well as homogeneous boundary conditions [START_REF] Verma | Benchmarking and scaling studies of pseudospectral code Tarang for turbulence simulations[END_REF]].

-SpectralDNS is a periodic FFT solver for turbulent flows that has been implemented by using only Python modules: pyFFTW and MPI4py. This solver scales well up to 3000 cores by using the hybrid MPI-OpenMP capabilities offered by FFTW [START_REF] Mortensen | High performance Python for direct numerical simulations of turbulent flows[END_REF]].

-Dedalus translates symbolic user-supplied partial differential equations into an efficient MPI distributed spectral solver. It supports Fourier, sine and cosine bases as well as polynomial bases (Chebyshev, Hermite, Laguerre) and scales up to thousand of cores [START_REF] Burns | Dedalus: Flexible framework for spectrally solving differential equations[END_REF]].

-CaNS is a FFTW based code for massively parallel numerical simulations of turbulent flows. It aims at solving any fluid flow of an incompressible, Newtonian fluid that can benefit from a FFT-based solver [START_REF] Costa | A FFT-based finite-difference solver for massively-parallel direct numerical simulations of turbulent flows[END_REF]]. It proposes the same boundary conditions as Tarang supplemented with general boundary conditions on the last axis and offers good strong scaling performances up to 10.000 cores.

-FluidSim is a framework for studying fluid dynamics on periodic domains by using Python [START_REF] Mohanan | FluidSim: Modular, object-oriented python package for high-performance CFD simulations[END_REF]. It is based on FluidFFT is a common API for FFT HPC libraries. This Python module allows to perform two-and three-dimensional FFTs with FFTW and cuFFT.

-Shenfun is a high performance computing platform for solving PDEs by the spectral Galerkin method on multi-dimensional tensor product grids [START_REF] Mortensen | High performance Python for direct numerical simulations of turbulent flows[END_REF]].

It proposes a Python user interface similar to FEniCS and can solve periodic problems (Fourier) and problems where the last direction is inhomogeneous by using a mix of Chebyshev and Legendre bases. It is parallelised by using mpi4py-fft package and scales up to thousand of cores for periodic boundary conditions [START_REF] Mortensen | mpi4py-fft: Parallel Fast Fourier Transforms with MPI for Python[END_REF]. for sediment flows

Most of the libraries presented here are available as open-source projects and propose a large set of numerical methods using many different spatial and temporal discretization techniques. Those numerical routines are often distributed up to thousands of CPU cores by using hybrid MPI-OpenMP or MPI-OpenCL programming models. More and more solvers are starting to integrate at least partial GPU support. This partial GPU support often comes from the drop-in replacement of underlying linear solvers (CUDA libraries cuSparse, cuBLAS and OpenCL equivalents clSparse, clBLAS) or other accelerated libraries (cuRAND, cuFFT or clRNG, clFFT).

Distributed GPU computing is becoming more and more popular especially for particle based methods. For many applications GPU adoption is penalized by additional CPU-GPU memory transfers. As an example multi-GPU distributed implementations of spectral Fourier based solvers do not scale well because of the additional cost implied by host-device transfers [START_REF] Wu | High performance FFT based poisson solver on a CPU-GPU heterogeneous platform[END_REF]].

The tendency is clearly to simplify the user interfaces from compiled Fortran or C++ configuration files to accessible scripting languages such as Python while relying on compiled code under the hood for performance. An ever increasing number of high performance libraries are made available through simple Python modules (pyFFTW, pyBLAS, pyMUMPS, ...). Because those wrappers provide a pythonic access to equivalent performances, it is now possible to implement performant distributed numerical solvers using directly Python as the core language (FluidSim, PySPH, HySoP), enabling faster development cycles [START_REF] Wagner | Performance Analysis of Parallel Python Applications[END_REF]].

Writing multi-threaded compiled extensions to Python has never been easier than today with the advent of tools like Cython [START_REF] Behnel | Cython: The best of both worlds[END_REF], Numba [START_REF] Lam | Numba: A llvm-based python jit compiler[END_REF] and Pythran [START_REF] Guelton | Pythran: Enabling static optimization of scientific python programs[END_REF]]. The same goes for GPU-programming with the PyOpenCL and PyCUDA Python modules provided by [START_REF] Klöckner | PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation[END_REF]]. Finally the mpi4py module allows fast prototyping and development of MPI-based distributed solvers [START_REF] Smith | Performance of MPI Codes Written in Python with NumPy and mpi4py[END_REF]].

Because the problems to be solved are not known in advance, some libraries use code generation techniques from user specified inputs such as symbolic equations (FEniCS, Dedalus).

Code generation is also a common technique used for GPU-programming [START_REF] Klöckner | PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation[END_REF].

In this work we will be particularly interested in the development of numerical routines using finite-differences, remeshed-particles and spectral methods in an hybrid MPI-OpenCL context. Similar code generation techniques will be used to generate finite-difference based numerical routines from user-supplied symbolic expressions. A particular attention will be paid to OpenCL performance portability between different compute devices. The corresponding numerical methods are introduced in section 2 and are implemented within the HySoP library in section 3.

Conclusion

We first described the derivation of the Navier-Stokes equations along with their simplification to the incompressible case. The velocity-vorticity formulation has been derived from the associated velocity-pressure formulation. Remeshed particle methods are CFL free numerical schemes used to discretize the transport that fit well with the velocity-vorticity formulation of the incompressible Navier-Stokes equations. Those methods are particularly adapted for the transport of fields that diffuse slower than the vorticity due to viscous momentum diffusion.

Those flows are more commonly known as high Schmidt number flows. In such a flow, the slowly diffusive agent will develop smaller physical scales, up to a ratio depending on the square root of the Schmidt number, when compared to the smallest velocity eddies.

Sediment dynamics is still poorly understood and multiple physical models with increasing complexity exist. Those models mostly depend on the mass loading and the physical properties of the said sediments in the carrier fluid. Monodisperse distribution of particles with a mass loading of less than 1% can be reduced to a dual-way coupling between the particle concentration and the fluid, in an eulerian-eulerian framework. When the fluid density is only slightly affected by the presence of these additional particles, the equations can be simplified by using the Boussinesq approximation. As small particles may diffuse slower than other fields such as temperature or salinity, a configuration where two such species are present can lead to double-diffusive instabilities. In the nature, this situation can happen near sediment-laden riverine outflows flowing into the ocean. Although double-diffusivity is known to enhance vertical fluxes, effective sedimentation velocity cannot be numerically investigated because of the high Schmidt numbers in play for fine particles.

From a single mathematical problem, different numerical methods can be considered. The choice of the numerical method strongly depend on the target problem and available computing architectures. Here the choice of a remeshed particle method is natural but not sufficient. The use of high performance computing is required to be able to achieve sufficient resolution and solve all the physical scales of the sediment concentration up to the Batchelor scale. In this work, we will extend the HySoP library, a Python-based solver based on hybrid MPI-OpenCL programming that targets heterogeneous compute platforms. We will be able to solve custom user specified numerical problems such as incompressible Navier-Stokes fully on accelerator. The resulting solver will be modular enough to be able to run high performance GPU accelerated numerical simulation of particle-laden sediment flows.

Introduction

In this chapter we derive a computational framework that will allow the resolution of the transport of a passive scalar in a turbulent flow, possibly at high Schmidt number. As seen in section 1.1.10, the modeling of the transport of a passive scalar in such a flow can be carried out using a system of continuous equations consisting of Navier-Stokes equations coupled with a scalar advection-diffusion equation (1.36). Both the momentum equation and the scalar equation can be viewed, at least partially, as advection-diffusion equations, one for the vorticity and the other for the scalar. Those two equations can be split into transport and diffusion terms, by relying on so-called operator splitting methods. The idea behind the proposed numerical method is to split the equations such that each subproblem can be solved by using a dedicated solver based on the most appropriate numerical scheme and by employing a space discretization that is regular enough to be handled by accelerators. Semi-lagrangian (remeshed) particle methods makes it possible to solve convection problems without imposing a Courant-Friedrichs-Lewy stability constraint, but a less restrictive stability condition allowing the use of larger time steps [START_REF] Cottet | Vortex methods: theory and practice[END_REF][START_REF] Van Rees | A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers[END_REF][START_REF] Mimeau | Direct numerical simulations of three-dimensional flows past obstacles with a vortex penalization method[END_REF]]. On the one hand this is method particularly adapted for the transport of scalars in high Schmidt number flows, as introduced in chapter 1. On the other hand, through the presence of an underlying grid, this method allows to the use of eulerian solvers. In particular we will use Cartesian grids that are compatible with a wide variety of numerical methods such as finite difference methods and spectral methods.

The resulting solver should be able to run on multiple compute nodes, each exhibiting one or more accelerators. Thus, the numerical scheme is designed by employing numerical methods that are known to perform reasonably well in this context. In particular, each numerical scheme should exhibit adequate theoretical strong scalability with respect to the number of compute nodes used for the simulation. Choosing a numerical scheme that is not known to scale well would constitute a real performance bottleneck [START_REF] Mathew | The Performance of Parallel Algorithms by Amdahl's Law, Gustafson's Trend[END_REF]]. Maximizing the memory bandwidth is crucial in this context because most of the aforementioned numerical methods are generally memory bound. The use of regular grids, enabled by the remeshed particle method, is particularly adapted to heavily vectorized hardware such as GPUs. Indeed, Cartesian grids will allow us to saturate memory controller of the target devices in memorybound situations.

The chapter is organized as the following: we first give an overview of the method by splitting the problem into several subproblems, each of them accounting for a specific part of the underlying physics. Each of the resulting subproblems can be solved by using a grid compatible numerical method. Those additional numerical methods are introduced in the following subsections. We then show that some of those operators can further be split directionally reducing the overall cost and complexity of the method. The two last subsections focus on spectral methods from periodic problems to problems with any general set of boundary conditions. Spectral methods are mainly used to compute the velocity from the vorticity but also provide an alternative method to solve timestep limiting diffusion problems. The resulting hybrid particle-spectral-finite differences method will be particularly adapted for the problem of interest in this work.

Outline of the method

In this section we describe how we can split the problem into simpler subproblems to solve the incompressible Navier-Stokes equations on a n-dimensional spatial domain Ω coupled with a passive scalar. We will then decompose the problem by applying operator splitting strategies, consisting in solving successively different operators within the same time iteration.

Let n be the dimension of the spatial domain Ω ⊂ R n , T = [t start , t end ] be the time domain, u(x, t) : Ω × T → R n be the velocity field, ω(x, t) : Ω × T → R p be the vorticity field, and θ(x, t) : Ω×T → R be the scalar field passively transported in the flow. The numerical method is obtained by starting with the incompressible Navier-Stokes equations in their velocityvorticity formulation coupled with a passive scalar, recalled bellow.

∇ • u = 0 ω = ∇ × u ∂ω ∂t + (u • ∇) ω = (ω • ∇)u + ν∆ω + ∇ × f ext ∂θ ∂t + (u • ∇) θ = κ∆θ (2.1a) (2.1b) (2.1c) (2.1d)
In those equations, f ext represents an external forcing term that depends on the problem being solved, ν is the constant kinematic viscosity of the fluid, and κ is the constant diffusivity of the scalar. Additionally, equation (2.1b) imposes that p = n(n-1)

2

, such that the vorticity field has three components in 3D and only one in 2D. In two dimensions, the vorticity is orthogonal to the flow plane, so that stretching term (ω • ∇)u vanishes and the vorticity equation (2.1c) reduces to a convection-diffusion equation that may include an additional external forcing term. Operator splitting will be applied for both the momentum equation (2.1c) and the scalar equation (2.1d).

Operator splitting

The approach followed here is summarized here for one time iteration. First, the scalar θ and the vorticity ω = ∇ × u are evolved at fixed velocity u using an operator splitting on their respective equations. More precisely, we first split the scalar equation (2.1d) into independent convection and diffusion equations as the following:

1. Scalar convection:

∂θ ∂t + (u • ∇) θ = 0 (2.2) 2. Scalar diffusion: ∂θ ∂t = κ∆θ (2.3)
We also split the vorticity transport equation (2.1c) in which the successive terms respectively model the convection, the stretching, the diffusion and the external forcing applied to the vorticity field:

1. Convection:

∂ω ∂t + (u • ∇) ω = 0 (2.4) 2. Diffusion: ∂ω ∂t = ν∆ω (2.5) 3. Stretching: ∂ω ∂t = (ω • ∇)u (2.6) (only in 3D) 4. External forces: ∂ω ∂t = ∇ × f ext (2.

7) (depends on the problem being solved)

As we will see later, the order of resolution of those equations is not important, but this is the one we will use throughout the manuscript. Viscous splitting within the framework of vortex methods was originally proposed in the early seventies by [START_REF] Chorin | Numerical study of slightly viscous flow[END_REF]]. Its convergence has been proven by [START_REF] Beale | Vortex methods. II. Higher order accuracy in two and three dimensions[END_REF]] and reformulated by [START_REF] Cottet | Vortex methods: theory and practice[END_REF]] to suit to the present vortex formulation.

The vorticity transport equation (VTE) has to be coupled to the system giving the velocity in terms of the vorticity. Hence, once the scalar and the vorticity have been evolved, the new divergence free velocity is computed from the vorticity by using a Poisson solver. The velocity is directly linked to the vorticity through the following relation obtained by taking the curl of (2.1b) and the incompressibility condition (2.1a):

∆u = -∇ × ω (2.8)
In some situations it is more convenient to use the Helmholtz decomposition of the velocity:

u = ∇ × ψ + ∇φ (2.9)
The stream function ψ and potential function φ then satisfy the following independent systems that have to be complemented with appropriate boundary conditions:

∆ψ = -ω ∇ • ψ = 0 (2.10a) (2.10b) and ∆φ = 0 (2.11)
The pressure can be easily recovered from the velocity field and the forcing term in the case of constant density ρ and dynamic viscosity µ by taking the divergence of equation (1.27b) together with the incompressibility condition (2.1a) to obtain the following Poisson equation:

∆P = ρ∇ • [f ext -(u • ∇)u] = ∇ • [F ext -ρ(u • ∇)u]
(2.12)

Temporal resolution

The time is discretized on time domain T from t start to t end using a variable timestep dt k > 0 that will be determined later for each iteration k such that t 0 = t start and t k+1 = t k + dt k ≤ t end . In addition we define T = t end -t start > 0 as the total duration of the simulation. For a given field F ∈ {u, ω, θ} and a given k with associated time t k ∈ T , we define F k as F (t k ).

Given an initial state (u k , ω k ) at a given time t k , we can compute (u k+1 , ω k+1 ) at time t k+1 = t k + dt k using the following algorithm:

1. (a) Update the scalar and obtain θ k+1 from θ k and u k using equation (2.1d) by solving successively each split operator (2.2) and ( 2.3).

(b) Update the vorticity and obtain an intermediate vorticity field ω k, * from ω k and u k using equation ( 2.1c) at fixed velocity u k by solving successively each split operator (2.4), (2.5), (2.6) and (2.7).

2. Correct the newly obtained vorticity ω k, * to obtain ω k+1 such that the associated velocity u k+1 becomes divergence free by using the Helmholtz decomposition of the velocity (2.9). Compute the new velocity u k+1 from ω k+1 by solving the Poisson problem (2.8) or by solving independently (2.10) and (2.11).

3. Compute the new timestep dt k+1 from the stability criteria of the numerical schemes depending on u k+1 , ω k+1 , θ k+1 and the spatial discretization parameters. Advance in time by setting

t k+1 = t k + dt k and k = k + 1.
Steps (1a) and (1b) are independent and can be computed concurrently. Moreover the pressure term P k+1 can be recovered after step 2 by solving the Poisson problem (2.12).

The timestep only depends on the numerical methods used for each of the subproblems obtained in the splitting steps. We will denote ∆t adv , ∆t dif f , ∆t stretch and ∆t f ext the timesteps computed from the stability criteria of each of the subproblems for the vorticity ω. The scalar transport also constraints the timestep to be less than ∆t adv,θ and ∆t dif f,θ . A constant maximum timestep ∆t max is also introduced to cover the case of dual-way couplings where the stability constraints would be too loose to ensure the convergence of the method. As we will never be interested in the pressure term, this results in algorithm 1.

The transported fields may exhibit a multiscale property. This means that the scalar θ may be discretized on a finer mesh compared to the velocity or vorticity mesh. We will see that for performance reasons we will only use Cartesian grid discretizations which are parametrized by a constant space step dx = (dx 1 , • • • , dx n ). Specific non-uniform grid discretizations may Chapter 2. Numerical method be required for Chebyshev spectral solvers introduced in the last section.

k ← 0 t 0 ← t start u 0 ← u(t = t start ) ω 0 ← ω(t = t start ) θ 0 ← θ(t = t start ) dt 0 ← min(∆t max , ∆t 0 adv , ∆t 0 dif f , ∆t 0 stretch , ∆t 0 f ext , ∆t 0 adv,θ , ∆t 0 dif f,θ , t end -t 0 ) while t k < t end do θ k,1 ← convect(θ k , dt k , u k ) Transport θ k at velocity u k during period dt k θ k+1 ← diffuse(θ k,1 , dt k , κ) Diffuse θ k,1 with coefficient κ during period dt k ω k,1 ← convect(ω k , dt k , u k ) Transport ω k at velocity u k during period dt k ω k,2 ← diffuse(ω k,1 , dt k , ν) Diffuse ω k,1 with coefficient ν during period dt k ω k,3 ← stretch(ω k,2 , dt k , u k ) Stretch ω k,2 at velocity u k during period dt k ω k,4 ← forcing(ω k,3 , dt k ) Apply external forces to ω k,3 during period dt k ω k+1 ← correct(ω k,4 ) Correct the vorticity ω k,4 to ensure ∇ • u k+1 = 0 u k+1 ← poisson(ω k+1 )
Compute u k+1 from ω k+1 using a Poisson solver

t k+1 ← t k + dt k dt k+1 ← min(∆t max , ∆t k+1 adv , ∆t k+1 dif f , ∆t k+1 stretch , ∆t k+1 f ext , ∆t k+1 adv,θ , ∆t k+1 dif f,θ , t end -t k+1 ) k ← k + 1 end
Algorithm 1: Algorithm to solve the one-way coupling of a scalar with the incompressible Navier-Stokes equations (2.1) by using operator splitting and variable timestep dt.

Classical eulerian solvers on such grids are submitted to a CFL condition [START_REF] Courant | On the partial difference equations of mathematical physics[END_REF]] that caps the timestep ∆t adv depending on velocity u and space step dx:

C = ∆t adv n i=1 u i ∞ dx i < C max (2.13)
When ω and θ live on the same grid, it entails that dx ω = dx θ and thus ∆t adv = ∆t θ,adv . Under the assumption of high Schmidt number, Sc = ν κ 1, it is of interest to choose a finer grid to solve for all the physical scales of the scalar. In this case we may choose dx = √ Sc dx θ , which implies ∆t adv,θ = ∆t adv / √ Sc. Within this framework, passively coupling a single scalar with a high Schmidt number can potentially slow down the whole simulation by a factor √ Sc.

Using a lagrangian scheme instead of an eulerian scheme for advection allows to get rid of this factor at the cost of loosing the underlying uniform grid discretization of eulerian methods. The present method use semi-lagrangian schemes, that are able to combine the best of both worlds. With particle methods, the stability of the method just depends on the gradient of the velocity:

LCF L = ∆t ∇u ∞ ≤ C max (2.14)

Outline of the method 67

The scalar diffusion term may also restrict the timestep because of stability considerations. The use of an explicit finite difference scheme on a diffusion equation with diffusion coefficient α is stable under:

C = ∆t dif f max i∈ 1,n α dx 2 i < C max (2.15)
When ω and θ live on the same grid, we have dx ω = dx θ such that ∆t dif f = ∆t dif f,θ /Sc and thus ∆t dif f ∆t dif f,θ , the scalar has no impact on the diffusion time step for high Schmidt numbers. It is however possible to use an unconditionally stable solver such as spectral diffusion to get rid of ∆t dif f . If we choose dx = √ Sc dx θ , we obtain ∆t dif f,θ = ∆t dif f . Hence the additional diffusion of scalar at high Schmidt number should never impact the number of iterations of the solver given the same explicit diffusion schemes are used. When the diffusion limits the timestep is is possible to use an implicit spectral formulation instead.

Directional splitting

The operator splitting introduced in section 2.1.1 and illustrated in algorithm 1 is further decomposed by using a directional Strang splitting on advection, diffusion, stretching and external forcing operators. Such a Strang splitting can be used to speed up calculation for problems involving multidimensional partial differential equations by reducing them to a sum of one-dimensional problems [START_REF] Strang | On the construction and comparison of difference schemes[END_REF]]. This method makes it possible to decouple the spatial directions by solving one-dimensional advection problems by alternating the directions.

Let L be a linear differential operator on y such that

∂ ỹ ∂t = L(ỹ) t ∈ [t k , t k + dt[ ỹk = ỹ(t k ) = y k (2.16a) (2.16b)
We want to solve problem 2.16 to obtain y k+1 = ỹ(t k + dt).

A Strang splitting allows to solve problem (2.16) when L can be decomposed as a sum of p differential operators by solving successively partial differential equations arrising from the splitting. It is particularly interesting when

L = L 1 + L 2 + • • • + L p is
relatively difficult or costly to compute directly while there are readily available methods to compute each of the splitted differential operators L i separately. In such a case, the method can then be used to speed up calculations at first and second order in time.

First order in time Strang splitting is achieved by solving successfully the p operators during a full timestep dt as the following:

→ y k =y k 0 L 1 (dt) → y k 1 L 2 (dt) → y k 2 • • • → y k p-1 L p (dt) → y k p =y k+1 68 Chapter 2. Numerical method input : dt, y k = y(t k ) output: y k+1 = y(t k + dt), solution of problem 2.16 y k 0 ← y k for i ∈ 1, p do Solve ∂ ỹ ∂t = L i (ỹ) t ∈ [t k , t k + dt[ ỹk = ỹ(t k ) = y k i-1 y k i ← ỹ(t k + dt) end y k+1 ← y k p Algorithm 2: First order Strang splitting applied to operator L = L 1 + • • • + L p
Second order (in time) Strang splitting can be achieved by solving successfully p operators back and forth with half a timestep:

→ y k =y k 0 L 1 dt 2 → y k 1 L 2 dt 2 → y k 2 • • • → y k p-1 L p dt 2 → y k p =y k+ 1 2 0 L p dt 2 or equivalently L p (dt) → y k+ 1 2 1 • • • → y k+ 1 2 p-2 L 2 dt 2 → y k+ 1 2 p-1 L 1 dt 2 → y k+ 1 2 p =y k+1 input : y k = y(t k ) output: y k+1 = y(t k + dt), solution of problem 2.16 y k 0 ← y k for i ∈ 1, p do Solve ∂ ỹ ∂t = L i (ỹ) t ∈ [t k , t k + 1 2 dt[ ỹk = ỹ(t k ) = y k i-1 y k i ← ỹ(t k + 1 2 dt) end y k+ 1 2 0 ← y k p for i ∈ 1, p do Solve ∂ ỹ ∂t = L p-i+1 (ỹ) t ∈ [t k + 1 2 dt, t k + dt[ ỹk+ 1 2 = ỹ(t k + 1 2 dt) = y k+ 1 2 i-1 y k+ 1 2 i ← ỹ(t k + dt) end y k+1 ← y k+ 1 2 p Algorithm 3: Second order Strang splitting applied to operator L = L 1 + • • • + L p

Outline of the method 69

In order to obtain only one-dimensional problems, the idea is to split directionally all partial differential equations relating to advection (2.2, 2.4), diffusion (2.3, 2.5), stretching (2.6) and external forcing (2.7) by breaking them down into n directional terms of the form:

∂f ∂t = L x, t, f , ∂f ∂x 1 , • • • , ∂f ∂x n , ∂f 2 ∂x 2 1 , • • • , ∂f 2 ∂x 2 n = L 1 x, t, f , ∂f ∂x 1 , ∂f 2 ∂x 2 1 + • • • + L n x, t, f , ∂f ∂x n , ∂f 2 ∂x 2 n (2.17) where f = [ω 1 , • • • , ω p , u 1 , ..., u n , θ] T .
This kind of directional splitting offers many advantages as only one-dimensional problems have to be solved for directionally split operators leading to a dimension agnostic implementation. It also reduces the arithmetic intensity while being well suited for vectorized hardware architectures such as CPUs and GPUs [START_REF] Magni | Accurate, non-oscillatory, remeshing schemes for particle methods[END_REF][START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]]. Once the decomposition L 1 , • • • , L n is known for all operators we can apply first or second order Strang splitting by grouping each direction to solve the respective operators as exposed in algorithm 4.

(k, t 0 ) ← (0, t start ) (u 0 , ω 0 , θ 0 ) ← u(t = t start ), ω(t = t start ), θ(t = t start ) dt 0 ← min(∆t max , ∆t 0 adv , ∆t 0 dif f , ∆t 0 stretch , ∆t 0 f ext , ∆t 0 adv,θ , ∆t 0 dif f,θ , t end -t 0 ) while t k < t end do // Advection and diffusion of the scalar θ at constant velocity u θ k,0 ← θ k for i ← 0 to (n -1) do θ k,2i+1 ← directional convect i+1 (θ k,2i+0 , dt k , u k i+1 ) θ k,2i+2 ← directional diffuse i+1 (θ k,2i+1 , dt k , ν) end // Advection,

diffusion, stretching and ext. forcing of vorticity ω at constant

u ω k,0 ← ω k for i ← 0 to (n -1) do ω k,4i+1 ← directional convect i+1 (ω k,4i+0 , dt k , u k i+1 ) ω k,4i+2 ← directional diffuse i+1 (ω k,4i+1 , dt k , ν) ω k,4i+3 ← directional stretch i+1 (ω k,4i+2 , dt k , u k ) ω k,4i+4 ← directional forcing i+1 (ω k,4i+3 , dt k ) end //

Correction of the vorticity and computation of the new divergence-free velocity

ω k+1 ← correct(ω k,4n ) u k+1 ← poisson(ω k+1 ) // Compute the new timestep and advance in time t k+1 ← t k + dt k dt k+1 ← min(∆t max , ∆t k+1 adv , ∆t k+1 dif f , ∆t k+1 stretch , ∆t k+1 f ext , ∆t k+1 adv,θ , ∆t k+1 dif f,θ , t end -t k+1 ) k ← k + 1 end
Algorithm 4: Extension of algorithm 1 by using first order directional Strang splitting on both the scalar θ and the vorticity ω.

Spatial discretization

The physical domain Ω is first embedded into a n-dimensional bounding box B that has origin

x min = (x min 1 , • • • , x min n ) and length L = (L 1 , • • • , L n ) such that x max = (x min 1 + L 1 , • • • , x min n + L n ) and Ω ⊂ [x min 1 , x max 1 ] × • • • × [x min n , x max n ].
If Ω is not initially box-shaped, it is possible to inject additional penalization terms into the momentum equation (2.1c) and split it accordingly to impose the right boundary conditions on Ω. This includes homogeneous Dirichlet boundary conditions [START_REF] Angot | A penalization method to take into account obstacles in incompressible viscous flows[END_REF], homogeneous Neumann boundary conditions [START_REF] Kadoch | A volume penalization method for incompressible flows and scalar advection-diffusion with moving obstacles[END_REF]] and its non homogeneous counterpart [START_REF] Sakurai | Volume penalization for inhomogeneous Neumann boundary conditions modeling scalar flux in complicated geometry[END_REF]] and can also be used to model the boundaries of complex obstacles moving inside the fluid domain. However, in this manuscript, the physical domain Ω will systematically represent a rectangular cuboid of dimension n and we will not make any distinctions between Ω and B.

Cartesian grids

Such a box is then discretized by using rectilinear grids defined by a tessellation of hyperrectangle cells that are all congruent to each other. We will denote dx = (dx 1 , • • • , dx n ) the size of each of those elements and

N c = (N c 1 , • • • , N c n ) their number in each direction such that L = N c dx where denotes elementwise multiplication (L i = N c i dx i ∀i ∈ 1, n ). When dx 1 = • • • = dx n ,
all the elements tesselating the domain are hypercubes and we obtain a Cartesian grid. All the components of velocity u and vorticity ω are discretized on the same collocated grid, formed by the N v cell vertices defined by The number of vertices on axe i depends on whether the domain is L i -periodic or not on the said axis. If we define P = (P 1 , • • • , P n ) ∈ {0, 1} n such that P i = 1 if the domain is periodic on axe i and P i = 0 if not, the number of vertices N v is defined by

x i = x min + i dx for all i = (i 1 , • • • , i n ) ∈ 0, N v 1 × • • • × 0, N v n .
N v i = N c i + 1 -P i ∀i ∈ 1, n .
The vertices are globally indexed and stored in memory by using row-major ordering, where consecutive elements of a row reside next to each other. This ordering is defined by the following relation:

I i = ((((• • • + i n-3 )N v n-2 + i n-2 )N v n-1 + i n-1 )N v n + i n ) = i • S with stride S = (S 1 , • • • , S n ) where S i = Π n j=i+1 N v j as illustrated on figure 3.29.
The convention in 2D is that the first axe corresponds to the y-axis and the second one corresponds to the x-axis such that a domain of size

L = (L 1 , L 2 ) = (L y , L x ) is discretized using N c = (N c y , N c x ) rectangle cells of size dx = (dy, dx) forming N v = (N v y , N v x ) vertices globally indexed by I i = I (i 1 ,i 2 ) = i 1 N v
x + i 2 by taking into account the periodicity of the domain P = (P y , P x ). The 3D case follows the same idea, the first axis being the z-axis. In this case, the axes are ordered as (z, y, x) such that vertex

v i = v (i 1 ,i 2 ,i 3 ) has a global index I i = ((i 1 N v y + i 2 )N v x + i 3 = i • S with S = (N v x N v y , N v
x , 1) and position x i = (z min + i 1 dz, y min + i 2 dy, x min + i 3 dx). This convention is made such that the x-axis always represent the last axis where the vertex indices (and memory offsets) are contiguous.

The rationale behind this choice of discretization is that parallel computing is easy on rectilinear grids. First, when the grid is big enough with respect to the number of compute ressources, it is easy to perform domain decomposition on multiple processes as shown on figure 2.2. Secondly, Cartesian grids enable a wide variety of numerical methods such as finite differences [START_REF] Courant | On the solution of nonlinear hyperbolic differential equations by finite differences[END_REF], spectral methods [START_REF] Canuto | Spectral methods in fluid dynamics[END_REF]] and pseudo-spectral methods [START_REF] Fornberg | Generation of finite difference formulas on arbitrarily spaced grids[END_REF]]. It can be used in conjonction with remeshed particle methods [START_REF] Cottet | Vortex methods: theory and practice[END_REF]]. Last but not least, such a regular data structure offers uniform mapping from indexes to vertex coordinates, and can be allocated as a single large contiguous chunk of memory. Hence, this data structure is adapted for use with parallel computing platforms such as CUDA [START_REF] Luebke | CUDA: Scalable parallel programming for high-performance scientific computing[END_REF]] and OpenCL [START_REF] Stone | OpenCL: A parallel programming standard for heterogeneous computing systems[END_REF]]. Indeed, hardware peculiarities such as memory hierarchy or vector execution units inherent to the processors or coprocessors this kind of application programming interfaces can drive make them attractive [START_REF] Owens | GPU computing[END_REF][START_REF] Cottet | Is GPU the future of Scientific Computing ?[END_REF].

Figure 2.2 -Example of domain decomposition in 24 subdomains

Field discretization, interpolation and restriction

Each scalar field ω i , u i and θ are then discretized on such regular grids. In all this work the velocity and the vorticity components will be discretized on grids with the same size N v . The scalar θ may be discretized on a finer grid such that this new grid contains N c θ = m N c cells where m = N c θ N c ∈ N n is the grid ratio. Its associated number of vertices can be obtained as before by computing

N v θ = N c θ + 1 -P. From now on we denote N = n i=1 N v i and N θ = n i=1 N v θ,i .
In practice those variables may interact due to the couplings and thus we need to be able to compute a restriction of θ on the coarse grid and interpolate the velocity and vorticity fields on the fine grid. There exist many different methods to compute an interpolation:

• Local polynomial interpolation (n-linear or n-cubic splines, see [START_REF] Keys | Cubic convolution interpolation for digital image processing[END_REF]])

• Local interpolation by using interpolating kernels defined in section 2.3.

• Global Fourier spectral interpolation for periodic and homogeneous boundary conditions. Can be computed in O (N log N + N θ log N θ ) by using a n-dimensional forward and backward Fast Fourier transform with zero-padding (see section 2.6).

• Global Chebyshev spectral interpolation for general boundary conditions. Can be computed in O (N log N + N θ log N θ ) at the Chebyshev-Gauss-Lobatto points by using Fast Chebyshev transforms and in O N 2 + N 2 θ on a regular grid (see section 2.7).

For the local interpolation methods, the associated interpolation kernel has a compact support s = s dx with s ∈ N n . Local interpolation algorithms usually treats one direction after another to reduce the computational complexity of the method, going from O ([ Each of those methods have a corresponding restriction method such as low-pass filtering for global spectral methods and local restrictions for the local methods.

n i=1 s i ] N θ ) to O ([ n i=1 s i ] N θ ).

Hybrid particle-grid method 73

Hybrid particle-grid method

In this section we propose to introduce an hybrid method for the transport of scalar quantities using the particle method. This method will allow us to solve equations (2.2) and(2.4) and constitute the very first step of our numerical method as shown bellow:

A particle method is able to solve a transport equation of the form:

∂θ ∂t + (u • ∇) θ = 0 (2.18)
where the scalar field θ is transported at velocity u.

With particle methods, the variables are discretized on a set of particles which correspond to an elementary volume of the domain. The general principle is that particles carry quantities corresponding to the variables of the problem. The particles are then transported in the velocity field and are free to evolve over the whole domain in a grid-less fashion.

In such a lagrangian framework, the scalar field θ(x) can be approximated by the discrete sum of P particles {p i | i ∈ 1, P } each carrying a quantity θ i at position x i :

θ(x) P i=1 θ i W ε (x -x i ) (2.19)
The carried quantity θ i is obtained by computing the mean value of θ inside the elementary discrete volume V i that the particle represent:

θ i = 1 |V i | V i θ(x)dx (2.20)
The function W ε that appears in the equation (2.19) is a regularization function whose choice and properties are specific to each particle method. In all cases, it is constructed so that its limit when ε tends to 0 is the Dirac measure δ.

During the resolution, the particles are displaced relative to the velocity field and then interact to give a new distribution of the transported quantities. If the transported quantity is the vorticity ω, as in equation (2.4), this leads to a new velocity field u. Particles are transported along their characteristics and evolve on a grid free domain. The motion of the particles is dictated by the following set of equations:

dx i dt = u(x i , t) ∀i ∈ 1, P (2.21)
As a grid-free method, it has the advantage of being free from any stability condition that would restrict the timestep depending on some grid discretization parameter or the distance between two particles. Indeed, with traditional eulerian methods, the timestep is constrained by a Courant-Friedrichs-Lewy (CFL) condition where C is often less than one.

CF L = ∆t u ∞ ∆x ≤ C (2.22)
With particle methods, the stability of the method, and thus the timestep, just depends on the gradient of the velocity:

LCF L = ∆t ∇u ∞ ≤ C (2.23)
In practice, the constant C, called the LCFL (for lagrangian CFL), leads to a less restrictive condition than the CFL condition (2.22). This generally makes it possible to use larger time steps than in the case of an eulerian method. This feature is particularly interesting for high Schmidt number flows [START_REF] Etancelin | Multi-CPU and multi-GPU hybrid computations of multi-scale scalar transport[END_REF]].

Semi-lagrangian methods

A well-known disadvantage of lagrangian methods is that the particles have trajectories that follow the velocity field and thus particle distribution may undergo distortion. This manifests itself by the clustering or spreading of the flow elements in high strain regions and leads to the deterioration of the representation of the fields by lack or excess of particles in some areas of the flow [START_REF] Cottet | Vortex methods: theory and practice[END_REF]]. This particle overlapping condition has been recognized as a major difficulty to perform accurate simulations, in particular for non-linear problems.

Semi-lagrangian methods were developed for the resolution of flows dominated by advection, mainly in the field of meteorology [START_REF] Staniforth | Semi-Lagrangian integration schemes for atmospheric models-A review[END_REF]] and are able to combine the advantages of both eulerian (grid-based) and lagrangian (grid-free) methods. To maintain the regularity of the particle distribution, particles are remeshed on an underlying eulerian grid every few timesteps [START_REF] Koumoutsakos | High-resolution simulations of the flow around an impulsively started cylinder using vortex methods[END_REF]. When the particles are remeshed at every timestep, we obtain a systematic representation of the transported quantities on the grid, hence the name semi-lagrangian method. Within this framework particle methods can be combined with grid-based techniques such as spectral methods and finite difference methods.

Because the particles are systematically remeshed it is possible to create the particles only at the beginning of each advection step and to destroy them just after remeshing.

There exist two classes of semi-lagrangian methods:

• Backward semi-lagrangian method: The resolution consists to start from a grid point and integrate backward in time along the characteristics (advection step). The quantity interpolated at this new point is then transported to the starting point (interpolation step).

• Forward semi-lagrangian method: In this case we also start from a grid point, but the trajectory is integrated forward in time (advection step). The quantity transported by the particle is then distributed on the surrounding points (remeshing step).

Figure 2.4 -Illustration of the two classes of semi-lagrangian methods

In this work we will use the high-order forward semi-lagrangian particle method proposed by [START_REF] Cottet | High order semi-Lagrangian particle methods for transport equations: numerical analysis and implementation issues[END_REF]]. As a forward method, it features an advection and a remeshing step.

Advection step

At each time iteration of the method, particles are created on the underlying eulerian grid at positions

{x i = x min + i dx | i ∈ 0, N v 1 × • • • × 0, N v n }
and are transported along their characteristics while the associated transported quantities θ i remain constant. The particle positions are driven by transport equations (2.21) and the new position of the particles are given by the following set ordinary differential equations: (2.24) In this work we will numerically integrate particle positions in time by using explicit Runge-Kutta methods, which are described in section 2.3.2. The only difficuly that may arrise during this step is the interpolation of the velocity field, at all the intermediate positions of the particles (see figure 2.5). For a first order scheme, namely the Euler scheme, no interpolation is needed because the particles initially coincide with the eulerian grid where the velocity is known from the last timestep. In this case, the new position of the particles can be computed as the following:

∂x i ∂t = u(x i ) t ∈ [t k , t k+1 [ x k i = x i (t k ) = x min + i dx
x k+1 i = x k i + dt u k i (2.25)
Using a Runge-Kutta scheme of the second order leads to this second expression:

x k i,1 = x k i + dt 2 u k i x k+1 i = x k i + dt u x k i,1 (2.26)
Here it is clear that the intermediate position x k i,1 will not always be aligned with the grid and the computation of the velocity at this point requires an interpolation. Figure 2.5 shows the difference between the two methods, by using bilinear interpolation.

Figure 2.5 -Advection with first and second order explicit Runge-Kutta schemes

During the advection step, n-linear interpolation is used to evaluate velocity at positions that are not grid aligned. A value inside a cuboid cell is computed by the weighted average of the values present at the 2 n vertices forming the cell. As the scheme is not CFL constrained, a given particle can potentially travel over multiple cells at once and the cell in which the particle may arrive intersects the sphere centered at initial particle position x i = x min +i dx with radius r = dt u ∞ . Let x be the position at which the velocity has to be interpolated and y = (xx min ) dx be the grid-normalized coordinate of x. The multi-index j of the cell that contains the particle is obtained by taking the integer part of the grid-normalized coordinate: j = y . Similarly the weights α ∈ [0, 1[ n for the n-linear interpolation are obtained by extracting the fractional part: α = yy = yj. The value of the velocity at the new position x and time t k is then obtained by the following formula:

u k (x) = d∈{0,1} n n i=1 α d i i (1 -α i ) 1-d i u k j+d (2.27)
Figure 2.5 illustrates the two-dimensional interpolation of the velocity required for the second order Runge-Kutta method and corresponds to the case j = (0, 0) and α = (α y , α x ) = (1/3, 1/3). The velocity at position x = j + α is bilinearly interpolated using formula (2.27) that can be expanded to the following expression:

u k (x) = (1 -α y )(1 -α x )u k 0,0 + (1 -α y )α x u k 0,1 + α y (1 -α x )u k 1,0 + α y α x u k 1,1
In a multiscale approach, the velocity is known only on a coarser grid than that of the scalar and we have to use interpolation even in with the first order method. In this case we have

N c θ = m N c
u where m ∈ N n denotes the grid ratio and we proceed in two steps:

1. First we interpolate the velocity from the coarse grid to the fine grid by using a given interpolation method. When the grid ratio between the velocity and the transported quantity is high, higher order interpolation schemes may be used (n-cubic splines or spectral interpolation, see section 2.2.2).

2. Now that the transported quantity θ and the velocity u are known on the same grid, particles are advected on the fine grid by using n-linear interpolation to get the velocity at intermediate particles positions.

Figure 2.6 -Example of multiscale advection by using bilinear interpolation

Remeshing step

Once the particles have been advected, they have to be remeshed onto the grid by using a remeshing kernel W ε , also called remeshing formula. The multidimensional case is generally treated by tensor product of one-dimensional kernels

W ε (x) = n i=1 W ε (x i ).
Those formulas usually have a high arithmetic cost, especially for high order remeshing formula that exhibit large supports. Figure 2.7 illustrates the advection and remeshing procedure applied to a single particle marked with a blue cross. Initially particles are grid-aligned and their size represent the value they are carrying. After the displacement of the particles according to the velocity field represented by the thick red arrows, the particles are remeshed onto neighboring grid points:

Figure 2.7 -Example of two-dimensional remeshing by tensor product

In this case, a single particle is remeshed onto S = 16 surroundings grid location (represented by the fine blue arrows) and the remeshing formula has a support s of 2dx (2 dx i in each direction i), or simply 2 in grid-normalized coordinates of the particle y p = (x p -x min ) dx.

If we simplify equation (2.19) by using the grid normalized coordinates and grid-independent kernels W (y) = W ε (y dx) we obtain the following expression:

θ k+1 i = P p=1 θ k p W (x k+1 p -x i ) dx = P p=1 θ k p W   y k+1 p -y i =i   (2.28)
This expression can be simplified by taking into account the support s of the remeshing kernel. Let us denote P k i the set of index of the particles p that where advected inside the rectangular cuboid centered around i with size 2s. Equation (2.28) can be rewritten as:

θ k+1 i = p∈P k i θ k p W y k+1 p -i P k i = p ∈ 1, P | y k+1 p -i ∈ ] -s 1 , s 1 [× • • • ×] -s n , s n [ (2.29a) (2.29b)
A given particle of global index p, associated to multi-index j such that p = I(j), has an initial grid-aligned grid-normalized coordinate y k p = y k j = j. In this configuration we can express the ball that contains the position of the particle after advection:

y k+1 j ∈ y | y -j 2 ≤ dt max x∈Ω ( u(x) dx 2 ) ⊂ y | |y i -j i | ≤ dt u i ∞ dx i ∀i ∈ 1, n Let d k j = y k+1 j
y k j be the grid-normalized displacement of a given particle p j and c = 2.29) can be rewritten by using directly the neighbor particle coordinates and displacements:

(c 1 , • • • , c n ) with c i = s i + dt u i ∞ /dx i . Equation (
θ k+1 i = c 1 j 1 =-c 1 • • • cn jn=-cn θ k i+j W y k+1 i+j -i = c 1 j 1 =-c 1 • • • cn jn=-cn θ k i+j W d k i+j + j (2.30)
In practice we only need to store the particle displacements d k i and the remeshing procedure goes the other way arround: each advected particle is remeshed on the S = n i=1 s i surrounding grid locations as illustrated on figures 2.7, 2.8 and in algorithm 5.

Input : Input scalar field θ k+1 ∈ R N v 1 ו••×N v n Input : Grid-normalized particle displacement field d ∈ R N v 1 ו••×N v n n Output: Output scalar field θ k ∈ R (N v 1 +2c 1 )ו••×(N v n +2cn) for i ∈ -c 1 , N v 1 + c 1 × • • • × -c n , N v n + c n do θ k+1 i ← 0 end for i ∈ 0, N v 1 × • • • × 0, N v n do j ← i + d i α ← d i -d i for p ∈ -s 1 + 1, s 1 × • • • × -s n + 1, s n do θ k+1 j+p ← θ k+1 j+p + θ k i W (p -α) = n l=1 W (p l -α l )
end end Algorithm 5: Remeshing procedure using a remeshing kernel W with support s

The inner loop of algorithm 5 computes the contribution of particle p i carrying quantity θ k i to the S grid locations present in the neighborhood of its new position y k+1 i = i + d k i = j + α as illustrated on figure 2.8 for the one-dimensional case. The values that are remeshed in the boundary cells, outside of the domain, are handled depending on the boundary conditions as a post-processing step. Multi-dimensional remeshing is generally achieved by tensor product of one-dimensional kernels, resulting in S calls to the one-dimensional remeshing formula per particle. We will see that the evaluation of W implies to evaluate high order polynomials, and thus the remeshing procedure through tensor product has high arithmetic intensity. It is possible to reduce the number of remeshing kernel calls by performing a directional splitting trough the use of a Strang splitting as proposed by [START_REF] Magni | Accurate, non-oscillatory, remeshing schemes for particle methods[END_REF]. This is presented in section 2.5.1.

Explicit Runge-Kutta methods

The accuracy in time can be improved in the advection step by using more ellaborate time stepping schemes such as the explicit Runge-Kutta methods [START_REF] Kutta | Beitrag zur naherungsweisen Integration totaler Differentialgleichungen[END_REF]]. The most widely known scheme of Runge-Kutta family is generally referred to as RK4 or the Runge-Kutta method. Here for simplicity and without any loss in generality and consider that f is scalar valued. We consider the following initial value problem

∂f ∂t = F (t, f ) with f (t k ) = f k and we seek to find f k+1 = f (t k + dt).
The fourth-order (in time) Runge-Kutta scheme is obtained as the following:

f k+1 = f k + dt 6 F k 1 + 2F k 2 + 2F k 3 + F k 4 F k 1 = F (t k , f k ) F k 2 = F t k + dt 2 , f k + dt 2 F k 1 F k 3 = F t k + dt 2 , f k + dt 2 F k 2 F k 4 = F t k + dt, f k + dt F k 3 (2.31a) (2.31b) (2.31c) (2.31d) (2.31e)
Here f k+1 is the RK4 approximation of f (t k+1 ) at order p = 4. It is determined by the current value plus the weighted average of four estimated slopes F i multiplied by the timestep dt.

The four slopes are estimated by evaluating

F at different intermediate estimations (t i , f i ) such that t i ∈ {t k , t k + dt 2 , t k + dt}: • F k 1 is the initial slope F (t k , f k )
as seen before with the Euler method.

• F k 2 is the slope at the midpoint t k + 1 2 dt based on initial slope F k 1 .

• F k 3 is the slope at the midpoint t k + 1 2 dt based on the second slope F k 2 .

• F k 4 is the slope at the endpoint t k + dt based on the third slope F k 3 .

The final slope is obtained by giving greater weights to the slopes obtained at the midpoints. The family of explicit Runge-Kutta methods is simply a generalization of the RK4 method 2.3. Hybrid particle-grid method 81 presented above. An explicit scheme of s stages is given by the following expressions:

f k+1 = f k + dt s i=1 b i F k i F k 1 = F t k , f k F k 2 = F t k + c 1 dt, f k + a 1,1 F k 1 F k 3 = F t k + c 2 dt, f k + a 2,1 F k 2 + a 2,2 F k 1 . . . F k s = F t k + c s-1 dt, f k + s-1 i=1 a s-1,i F k i (2.32a) (2.32b) (2.32c) (2.32d) (2.32e) (2.32f)
All the required coefficients can be summed up in a Butcher tableau [START_REF] Butcher | Coefficients for the study of Runge-Kutta integration processes[END_REF]]:

0 1 2 1 2 1 2 0 1 2 1 0 0 1 1 6 1 3 1 3 1 6 Butcher tableau for RK4 0 c 1 a 11 c 2 a 21 a 22 . . . . . . . . . c s-1 a s-1,1 a s-1,2 • • • a s-1,s-1 b 1 b 2 • • • b s-1 b s General Butcher tableau 0 1 3 1 3 2 3 -1 3 1 2 1 1 -1 1 1 8 3 8 3 8 1 8
Butcher tableau for RK4 38

The method is consistent under i j=1 a i,j = c i for all i ∈ 1, s -1 and a method of order p has at least s stages. The only consistent explicit Runge-Kutta method with one stage is the Euler method. In practice we will use methods with up to four stages, denoted RK1 (Euler), RK2, RK3 and RK4 which are the classical Runge-Kutta methods complemented with RK4 38 which is a variation of the fourth-order scheme called the 3/8-rule [START_REF] Butcher | Coefficients for the study of Runge-Kutta integration processes[END_REF]]. All those methods offer the same accuracy in time as their respective number of stages (p = s). This is not possible with methods using more than four stages. The primary advantage the RK4 38 method is that almost all of the error coefficients are smaller for the price of some extra FLOPs and memory.

As those methods are explicit, there is no need to solve any linear system and the total cost of the method is due to the s evaluations of F along with 2s + k -1 additions and 2s + k multiplications where s -1 ≤ k ≤ s(s + 1)/2 represent the number of non-zero coefficients a ij . Those same Runge-Kutta schemes will also be used for diffusion and stretching in a finite differences framework. In this case f represent a space dependent field, and the cost of the method is hidden in the s evaluations of F where partial space derivatives of f can be obtained by applying centered finite difference stencils.

Construction of the remeshing formulas

This subsection is dedicated to the construction of one-dimensional remeshing kernels W (x). Due to their nature, particle methods are suitable for solving conservation equations. It is therefore important that the remeshing step, performed in equation (2.19), enforce those conservation properties. In particular, remeshing formula should be designed to preserve the moments of the quantities transported. In dimension one, the discrete moment of order m ∈ N of the transported field θ is defined by

N v x -1 i=0 x m i θ i .
Therefore, a remeshing kernel must satisfy the conservation equalities of the following discrete moments up to the p-th moment:

N v x -1 i=0 x m i θ k i = N v x -1 i=0 x m i θ k+1 i ∀m ∈ 0, p (2.33)
where p is directly linked to the spatial order of the method (and the support s of the kernel).

By using equation (2.30) with n = 1, equation (2.33) is satisfied under:

j∈Z j m W (x -j) = x m ∀x ∈ R ∀m ∈ 0, p (2.34)
It also has to be symmetric W (x) = W (-x) so that there is no preferred remeshing direction.

(this implies that the odd moments are all zero). As W approximates a Dirac, it is preferable that it satisfies the interpolation condition W (j) = δ j,0 ∀j ∈ Z which is compatible with the 0-th moment equation that states that the global scalar quantity is preserved during the transport j∈Z W (x -j) = 1. The interpolation property allows an exact conservation of remeshed quantities carried by particles that happen to be grid-aligned (particles whose position coincides with a grid point after advection). Thus, at zero-velocity, the scalar field θ remains the same.

A first family of kernels can be obtained by looking for functions with support p + 1 built with p + 1 piecewise polynomials of order p (with support 1). The coefficients of the polynomials are obtained enforcing the conservation of the p first moments with equations (2.34). Kernels arising from this family are not regular enough and some of them are not even continuous. This entails a loss of numerical precision [Cottet et al. 2009b]. A second family of kernels that are more regular can be obtained up by the use of regular B-splines [START_REF] Schoenberg | Contributions to the problem of approximation of equidistant data by analytic functions[END_REF]]. This family does not enforce the interpolation condition and offers up to second order moment conservation for a given regularity r. The lack of high order moment preservation properties of this family have been later compensated by using Richardson extrapolation while preserving their regularity [START_REF] Monaghan | Extrapolating B splines for interpolation[END_REF]]. This family has been used since the end of the 90's [START_REF] Koumoutsakos | Inviscid axisymmetrization of an elliptical vortex[END_REF]][Salihi 1998] in the context of semi-lagrangian methods. However, high order kernels obtained by this method also lack the interpolation property.

The present method will use the family high order kernels proposed by [START_REF] Cottet | High order semi-Lagrangian particle methods for transport equations: numerical analysis and implementation issues[END_REF]] that allows to obtain a remeshing kernel of any order p and any regularity r while preserving the interpolation property. The kernels are found as with the first family, by looking for piecewise polynomials with compact support. Those kernels satisfy the following properties:

• P1: Parity W (x) = W (-x) • P2: Compact support [-s, s] with s ∈ N
• P3: Piecewise polynomial of degree q on the integer subsets.

• P4: Overall regularity r.

• P5: Preservation of the p first moments.

• P6:

Interpolation property W (j) = δ j0 ∀j ∈ Z
The three first properties restrict the problem to find s polynomials of degree q yielding s(q + 1) unknowns. Because the odd moment are automatically preserved by the parity of the kernel, we only take into account an even number moments p ∈ 2N. Choosing support s = 1 + p/2 and polynomials of degree q = 2r + 1 with r ≥ p/2 give enough equations to solve the linear system satisfying all the properties. The kernels obtained by this method are named Λ p,r and some of them are listed in the following table:

Kernel Max. preserved moment Regularity Support Degree Λ p,r p ∈ 2N C r , r ≥ p/2 [-s, s], s = 1 + p/2 q = 2r + 1 Λ 2,1 2 C 1 [-2, 2] 3 Λ 2,2 2 C 2 [-2, 2] 5 Λ 4,2 4 C 2 [-3, 3] 5 Λ 4,4 4 C 4 [-3, 3] 9 Λ 6,4 6 C 4 [-4, 4] 9 Λ 6,6 6 C 6 [-4, 4] 13 Λ 8,4 8 C 4 [-5, 5] 9 Λ 8,6 8 C 6 [-5, 5] 13
Table 2.1 -Comparative of some high order semi-lagrangian remeshing kernels

The default remeshing kernel that we will use in this work is Λ 4,2 : This specific formula is illustrated on figure 2.8. Other remeshing formulas can be found in appendix A of [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]] and some of them are illustrated on figure 2.9.

Λ 4,2 (x) =                          1 - 5

Finite difference methods

Finite-difference methods (FDM) are numerical methods for solving differential equations by approximating them with difference equations, in which finite differences approximate the derivatives [START_REF] Smith | Numerical solution of partial differential equations: finite difference methods[END_REF]. Let f : R → R be the function whose derivatives are to be approximated, such that f is p times continuously differentiable, f ∈ C p (R, R). Then by Taylor's theorem, we can write a Taylor series expansion around point x ∈ R:

f (x + h) = p q=0 f (q) (x) q! h q + R p (h) (2.36)
where R p (x) = O h p+1 is a remainder term, resulting of the difference between the Taylor polynomial of degree p and the original function f . It is then possible to extract the target derivative f (d) (x) with d ≤ p in terms of elements in f (x + Zh) by building a linear system consisting of s = i max -i min + 1 Taylor series expanded around x + ih at order p for i ∈ i min , i max ⊂ Z. If we denote f x = (f (x + ih)) i∈ i min ,imax then the d-th derivative of f evaluated at x, can be approximated by h -d (S • f x ) where S ∈ Q s is the associated finite difference stencil. The resulting order of the stencil depends on the order of the Taylor series p, the approximated derivative d and how errors terms cancel each other out. When i min = -i max the finite difference scheme is said to be centered, i max = 0 yields a backward finite difference scheme and i min = 0 a forward finite differences scheme.

The centered finite difference stencils approximating f (d) with accuracy m contains s = 2q + 1 coefficients S with q = d+1 2 + m+1

2

. The s coefficients of the stencil (S -q , S -q+1 , • • • , S 0 , • • • , S q-1 , S q ) can be computed by solving the following linear system:

          (-q) 0 (-q + 1) 0 • • • (q -1) 0 q 0 (-q) 1 (-q + 1) 1 • • • (q -1) 1 q 1 • • • • • • • • • • • • • • • (-q) d (-q + 1) d • • • (q -1) d q d • • • • • • • • • • • • • • • (-q) 2q (-q + 1) 2q • • • (q -1) 2q q 2q                     S -q S -q+1 • • • S -q+d • • • S q           =           0 0 • • • d! • • • 0           (2.37)
The following table contains the coefficients of the central differences obtained for the first and second derivative with several orders of accuracy for a uniform grid spacing h:

derivative accuracy S -3 S -2 S -1 S 0 S 1 S 2 S 3 1 2 -1/2 0 1/2 1 4 1/12 -2/3 0 2/3 -1/12 1 6 -1/60 3/20 -3/4 0 3/4 -3/20 1/60 2 2 1 -2 1 2 4 -1/12 4/3 -5/2 4/3 -1/12 2 6 1/90 -3/20 3/2 -49/18 3/2 -3/20 1/90
f (1) (x) = 1 h 2 + 1 12 f (x -2h) - 2 3 f (x -h) + 2 3 f (x + h) - 1 12 f (x + 2h) + O h 4 f (2) (x) = 1 h 2 - 1 12 f (x -2h) + 4 3 f (x -h) - 5 2 f (x) + 4 3 f (x + h) - 1 12 f (x + 2h) + O h 4 (2.38a) (2.38b)
For a fourth order accuracy stencil, the local truncation error is proportional to h 4 .

Spatial derivatives

Finite difference coefficients can be extended to any function f (R n , R) ∈ C p (R n , R) and can be computed on arbitrarily spaced grids [START_REF] Fornberg | Generation of finite difference formulas on arbitrarily spaced grids[END_REF]]. As part of this work we will remain in the uniform grid spacing case for discretization in space as presented in subsection 2.2. We will also approximate derivatives with at least fourth order accuracy to prevent odd-even decoupling which is a discretization error that can occur on collocated grids and which leads to checkerboard patterns in the solution [START_REF] Harlow | Numerical calculation of time-dependent viscous incompressible flow of fluid with free surface[END_REF]].

Let F be the discretization of a scalar field f :

B → R on the cuboid domain B ∈ R n with global grid size N v containing a total of N = n i=1 N v i elements and let S ∈ Q s 1 × • • • Q sn be a centered stencil of size s = 2q + 1 containing k ≤ n
i=1 s i non-zero coefficients that approximate some mixed spatial derivative of f on the grid at order m. The approximation of the derivative D over the whole domain is obtained by performing a convolution between the stencil S and the grid data

F = {F i = f (x min + i dx) | i ∈ 1, N v 1 × • • • 0, N v n }
containing additional boundary values that we will later call ghost nodes (see algorithm 6). This operation costs (k -1)N additions and kN multiplications and can thus be computed in O (kN ) operations.

Input : n-dimensional centered stencil S ∈ R s 1 ו••×sn with s = 2q + 1 Input : n-dimensional array F ∈ R (N v 1 +2q 1 )ו••×(N v n +2qn) Output: n-dimensional array D = F * S ∈ R N v 1 ו••×N v n for i ∈ 0, N v 1 × • • • × 0, N v n do D i ← 0 for j ∈ -q 1 , +q 1 × • • • × -q n , +q n do if S j = 0 then D i ← D i + S j F q+i+j end end end
Algorithm 6: Convolution of a discretized field F with a centered stencil S of size s = 2q + 1. F is discretized on a grid of size N v + 2q, q being the size of additional required boundary layers for input. The n-dimensional stencil S is indexed as in equation ( 2.37) whereas input and output arrays have indexes starting at zero. The additional inner loop conditional is here to highlight the fact that S may be sparse. Stencils can in general represent more than simple derivatives, it can for example approximate sums of derivatives as shown on figure 2.10. The mask of non-zero coefficients of a given stencil, represented in red, gives the location of values that have to be read to compute one value of the output, located at the center. The number of non-zero coefficients k of a finite difference stencil is typically small. Although dense centered stencils such as the second order nine-point Laplacian stencil have (2q + 1) n non-zero coefficients, they are seldomly used for fourth-order accuracy because of performance reasons. In 3D, the fourth order accuracy dense Laplacian stencil requires (2 * 2 + 1) 3 = 125 input values, 125 multiplications and 124 additions to compute one single output value. Assuming the values are represented with double precision floating point numbers of 8 bytes each, and that the multiplications and additions are done by using fused multiply-add operations, this gives an arithmetic intensity of 125/8 15.6 FLOP/B. Nowadays, most common architectures like CPUs and GPUs are optimized to target 5 FLOP/B for double precision computations [START_REF] Rupp | CPU, GPU and MIC Hardware Characteristics over Time[END_REF]]. A higher FLOP per byte ratio indicates that the given algorithms fully saturate the hardware compute units [START_REF] Williams | Roofline: An insightful visual performance model for floating-point programs and multicore architectures[END_REF]]. In this case, this means that this stencil would, in average, be compute-bound by a factor of 3 or put in other words that it would only use at most one third of the available memory bandwidth for an given device. Sparse stencils, such as the one obtained by sum or tensor product of 1D stencils, have the order of 1 + 2qn non-zero coefficients and thus are memorybound. The same calculation for the sparse fourth-order accuracy 3D Laplacian stencil yields an arithmetic intensity of 8/8 = 1 FLOP/B. Hence sparse stencils are likely to to be memory bound.

Finite difference schemes

A finite difference scheme can be used to approximate the solution of a partial differential equation such as (2.5), (2.6) and (2.7). It is obtained by approximating both space and time using finite differences. In this section we will focus on the two-dimensional diffusion equation as encountered in equation (2.5). The simplest temporal integration scheme is the explicit Euler scheme. It can be directly obtained by taking the Taylor expansion of ω(x, t) around t k + dt at order p = 1 with gives:

∂ω ∂t (t = t k ) = ω k+1 -ω k dt k + O (dt) (2.39)
By using equations (2.38b) and (2.39) applied to ω(t = t k ) along with equation (2.5) we obtain:

ω k+1 i 1 ,i 2 -ω k i 1 ,i 2 dt + O (dt) = + ν dx 2 - 1 12 ω i 1 ,i 2 -2 + 4 3 ω i 1 ,i 2 -1 - 5 2 ω i 1 ,i 2 + 4 3 ω i 1 ,i 2 +1 - 1 12 ω i 1 ,i 2 +2 + O dx 4 + ν dy 2 - 1 12 ω i 1 -2,i 2 + 4 3 ω i 1 -1,i 2 - 5 2 ω i 1 ,i 2 + 4 3 ω i 1 +1,i 2 - 1 12 ω i 1 +2,i 2 + O dy 4
(2.40) from which we can easily express the value of the variable at the next timestep ω k+1 i 1 ,i 2 :

ω k+1 i 1 ,i 2 = ω k i 1 ,i 2 + νdt dx 2 - 1 12 ω i 1 ,i 2 -2 + 4 3 ω i 1 ,i 2 -1 - 5 2 ω i 1 ,i 2 + 4 3 ω i 1 ,i 2 +1 - 1 12 ω i 1 ,i 2 +2 + νdt dy 2 - 1 12 ω i 1 -2,i 2 + 4 3 ω i 1 -1,i 2 - 5 2 ω i 1 ,i 2 + 4 3 ω i 1 +1,i 2 - 1 12 ω i 1 +2,i 2 +O dx 4 + O dy 4 + O (dt) (2.41)
This constitutes a fourth order scheme in space and first order in time finite differences scheme.

It is illustrated on figure 2.11. Higher order in time can be achieved trough the use of an higher order Runge-Kutta scheme.

Figure 2.11 -Explicit 2D finite difference scheme used to solve the heat equation with first order accuracy in time and fourth order accuracy in space. This scheme requires one read and one write and a total of 9 multiplications and 8 additions, giving an operational intensity of about 0.56FLOP/B in double precision by using fused multiply-add operations.

The major drawback of this explicit method is that the scheme in not unconditionally stable and for problems with large diffusivity the timestep restriction can be too severe. The von Neumann stability analysis [START_REF] Crank | A practical method for numerical evaluation of solutions of partial differential equations of the heat-conduction type[END_REF] states that this scheme is stable under the following condition that restricts the timestep:

max νdt dx 2 , νdt dy 2 ≤ 1 2 (2.42)
This explicit numerical scheme can be generalized in any dimension for all partial differential equations of the form:

∂f ∂t = L x, t, f , ∂f k 1 ∂x k 1 , • • • , ∂f km ∂x km with m ∈ N and k i ∈ N n where ∂f k ∂x k = ∂f k 1 +•••+kn ∂x k 1 1 • • • ∂x kn n Equations (2.
3), (2.5), (2.6) and (2.7) however do not contain any mixed spatial partial derivatives and fall into the following family of partial differential equations:

∂f ∂t = L x, t, f , ∂f ∂x 1 , • • • , ∂f ∂x n , ∂f 2 ∂x 2 1 , • • • , ∂f 2 ∂x 2 n (2.43) with f = [ω 1 , • • • , ω p , u 1 , ..., u n , θ] T .

Performance considerations

Achieving theoretical peak bandwidth or peak performance when dealing width the evaluation of three-dimensional (and higher dimensions) stencils is not an easy task [START_REF] Datta | Stencil computation optimization and auto-tuning on state-of-the-art multicore architectures[END_REF]]. This comes from the fact that when we estimate the number of operations per bytes we assume that the device has infinite cache such that each input value can be remembered after its first read. It is also assumed that data can be read and written at theoretical peak bandwidth, which may not be the case here because of the non-contiguous memory accesses.

For dense stencils that require O (100) input values per output (fourth order Laplacian stencil in 3D), the data can simply not fit in cache requiring the implementation of stencil-dependent caching strategies. For sparse stencils the problem is by definition memory bound and memory accesses have to be optimized.

As stencil computation is a key part of many other high-performance computing applications, such as image processing and convolutional neural networks, numerous techniques have been developed to achieve optimal performance for multi-dimensional stencils:

• Rearrange the n-dimensional data to preserve the locality of the data points by using space-filling curves like Z-ordering instead of the linear row-major ordering [START_REF] Nocentino | Optimizing memory access on GPUs using morton order indexing[END_REF]. Note that for GPUs, textures already provide space filling curves builtin for 2D and often 3D arrays [START_REF] Li | GPU-based flow simulation with complex boundaries[END_REF][START_REF] Sugimoto | Improving cache locality for GPU-based volume rendering[END_REF]]. This optimization technique is easiest to implement but do not provide enormous gains for general stencils.

• Manually tune the vectorization, memory accesses and how the data is cached with respect to the target architecture and the given stencil [START_REF] Brandvik | SBLOCK: A framework for efficient stencilbased PDE solvers on multi-core platforms[END_REF]]. This has been the main approach for early CUDA and OpenCL implementations [START_REF] Micikevicius | 3D finite difference computation on GPUs using CUDA[END_REF][START_REF] Su | On the GPU performance of 3D stencil computations implemented in OpenCL[END_REF]].

• As manual tuning requires a lot of work, the next idea was logically to develop frameworks, that for a given stencil, automatically generate optimized code for a target language and autotune its runtime parameters [START_REF] Zhang | Auto-generation and auto-tuning of 3D stencil codes on GPU clusters[END_REF][START_REF] Holewinski | Highperformance code generation for stencil computations on GPU architectures[END_REF][START_REF] Cummins | Autotuning OpenCL workgroup size for stencil patterns[END_REF]].

• Although the first solutions are all software based, there also exist hardware based solutions. It is for example possible to redesign the memory hardware to be optimized for stencil computations on FPGAs by using FPGA-compatible OpenCL platforms [START_REF] Verma | Accelerating workloads on fpgas via opencl: A case study with opendwarfs[END_REF][START_REF] Wang | A comprehensive framework for synthesizing stencil algorithms on FPGAs using OpenCL model[END_REF][START_REF] Zohouri | Combined spatial and temporal blocking for high-performance stencil computation on FPGAs using OpenCL[END_REF]].

The approach we will choose in this work is somehow orthogonal to what is usually done for finite differences based fluid solvers: we will not use any high dimensional stencils at all. In fact the absence of mixed spatial partial derivatives in equations (2.1c) and (2.1d) allows us to split all n-dimensional stencils that arrise from the approximation of the spatial derivatives to be split as efficient 1D stencils. This offers numerous advantages:

• One-dimensional stencil optimizations like vectorization are easy to implement. As they only require contiguous memory accesses they maximize the cache hits.

• By implementing only 1D computations, the algorithm becomes dimension-agnostic.

The support of n-dimensional problems is added by simply looping over the dimensions.

• This idea fits nicely into the directional splitting framework already introduced in section 2.1.3.

All the optimization aforementioned only work if the data is contiguous for all axes. This suppose that between the treatment of each direction, the data is rearranged in memory such that the next axis to be computed becomes contiguous in memory. This just shifts the performance bottleneck and the implementation effort on the permutations, that by definition do not compute anything, and are thus intrinsically memory-bound. However transpositions and permutations are also required for many other applications such as to compute fast Fourier transforms [START_REF] Nukada | Bandwidth intensive 3-D FFT kernel for GPUs using CUDA[END_REF]] (that will be required for the spectral method) and many cache-aware and cache-oblivious algorithms have been developed for many different architectures [START_REF] Ruetsch | Optimizing matrix transpose in CUDA[END_REF][START_REF] Jodra | Efficient 3D transpositions in graphics processing units[END_REF][START_REF] Lyakh | An efficient tensor transpose algorithm for multicore CPU, Intel Xeon Phi, and NVidia Tesla GPU[END_REF][START_REF] Springel | TTC: A tensor transposition compiler for multiple architectures[END_REF]].

Directional splitting

In order to obtain only one-dimensional stencils and one dimensional remeshing kernels, the idea is to split directionally all partial differential equations by breaking them down into n directional terms:

∂f ∂t = L x, t, f , ∂f ∂x 1 , • • • , ∂f ∂x n , ∂f 2 ∂x 2 1 , • • • , ∂f 2 ∂x 2 n = L 1 x, t, f , ∂f ∂x 1 , ∂f 2 ∂x 2 1 + • • • + L n x, t, f , ∂f ∂x n , ∂f 2 ∂x 2 n (2.44)
Once F 1 , • • • , F n are known for diffusion, stretching and external forces, we can apply a first or second order Strang splitting to solve the respective operators (2.1.3).

This section is dedicated to express the directional splitting of all those operators. The analysis is done in the three-dimensional case where the velocity and the vorticity have three scalar components each but all the results can easily be extended to any dimension when applicable.

Advection

The three-dimensional transport equation ( 2.18) can be split into three distinct differential operators, each taking into account the effect of advection in a given direction:

∂θ ∂t = -(u • ∇)θ L(u,∂xθ) = -u x ∂θ ∂x Lx(ux,∂xθ)
+ u y ∂θ ∂y

Ly(uy,∂yθ)

+ u z ∂θ ∂z Lz(uz,∂zθ) (2.45)
As the advection is treated with a forward remeshed particle method the method is decomposed into an advection and a remeshing step. In the current direction i, the directionally split method just consists into a one-dimensional advection with velocity component u i followed by a one-dimensional remeshing procedure as described in [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]]. This is illustrated on the following figure:

Diffusion

This operator appears in equation ( 2.1c) as a momentum diffusion due to shear stress and in equation ( 2.1d) as a scalar diffusion term due to molecular diffusivity:

∂ω ∂t + (u • ∇) ω = (ω • ∇)u + ν∆ω + ∇ × f ext ∂θ ∂t + (u • ∇) θ = κ∆θ
To illustrate the directional operator splitting (2.17), let us define f = [ω x , ω y , ω z , θ] T and D = diag(ν, ν, ν, κ). Merging equations ( 2.3) and (2.5) leads to

∂ t f = L (∂ xx f ) = D∆f and
we want to express L as a sum of directional operators L x , L y and L z where L i contains only spatial derivatives with respect to axe i. The expression of L i for each axis can be obtained by expanding the equation to four independent scalar partial differential equations:

∂f ∂t = D∆f ⇔                ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t ∂θ ∂t                =                ν∆ω x ν∆ω y ν∆ω z κ∆θ                =                  ν ∂ω 2 x ∂x 2 + ∂ω 2 x ∂y 2 + ∂ω 2 x ∂z 2 ν ∂ω 2 y ∂x 2 + ∂ω 2 y ∂y 2 + ∂ω 2 y ∂z 2 ν ∂ω 2 z ∂x 2 + ∂ω 2 z ∂y 2 + ∂ω 2 z ∂z 2 κ ∂θ 2 ∂x 2 + ∂θ 2 ∂y 2 + ∂θ 2 ∂z 2                 
We are looking to express

L ∂f 2 ∂x 2 , ∂f 2 ∂y 2 , ∂f 2 ∂z 2 as L x ∂f 2 ∂x 2 + L y ∂f 2 ∂y 2 + L z ∂f 2 ∂z 2 .
As the right hand side of the equations does not contain any mixed derivative we can split it into three parts, each of them containing the partial derivatives specific to each axis. Here the red, green and blue terms represent the splittings for axe x, y and z.

1. x-axis splitting:

∂f ∂t = L x ∂f 2 ∂x 2 = D ∂f 2 ∂x 2 (2.47)
2. y-axis splitting:

∂f ∂t = L y ∂f 2 ∂y 2 = D ∂f 2 ∂y 2 (2.48)
3. z-axis splitting:

∂f ∂t = L z ∂f 2 ∂z 2 = D ∂f 2 ∂z 2 (2.49)
The same procedure can be repeated for any dimension n and for any operator that does not contain mixed spatial derivatives.

Stretching

Vortex stretching is the lengthening of vortices in three-dimensional fluid flow, associated with a corresponding increase of the component of vorticity in the stretching direction, due to the conservation of angular momentum. Here we want to evaluate explicitly the contribution due to the stretching term appearing in the conservation of momentum equation (2.1c).

∂ω ∂t + (u • ∇) ω = (ω • ∇)u + ν∆ω + ∇ × f ext
In the following, we will rather use the equivalent notation (∇u)ω = (ω • ∇)u where ∇u is the (spatial) velocity gradient of u in dimension n:

∂ω ∂t = (∇u)ω (2.50)
Equation ( 2.50) can be rewritten equivalently in either its conservative form or nonconservative form:

1. Conservative formulation: The stretching term can be rewritten in a conservative fashion by using the incompressibility condition (2.1a) and the definition of the vorticity (2.1b):

∂ω ∂t = ∇ • (u ⊗ ω) (2.51)
This expression can be obtained by expanding the right hand side as the following ∇ • (u ⊗ ω) = u (∇ • ω) + (∇u)ω and by taking into account the fact that the vorticity is also divergence free:

∇ • ω = ∇ • (∇ × u) = ∇ × (∇ • u) = 0.
From a numerical point of view, the discretized version of the solver will not lead to the same property. Put in other words, this means that numerically we may have ∇ • ω = 0 during the different stages of the solver.

Non-conservative formulations:

We can split the Jacobian matrix of the velocity field into its symmetric and antisymmetric parts:

∇u = ∇u + ∇u T 2 + ∇u -∇u T 2 = 1 2 γ + Ω (2.52)
and then express the coefficients of Ω using vorticity components ω = ∇ × u:

Ω = ∇u -∇u T 2 = 1 2            0 ∂u x ∂y - ∂u y ∂x ∂u x ∂z - ∂u z ∂x ∂u y ∂x - ∂u x ∂y 0 ∂u y ∂z - ∂u z ∂y ∂u z ∂x - ∂u x ∂z ∂u z ∂y - ∂u y ∂z 0            = 1 2            0 -ω z +ω y +ω z 0 -ω x -ω y +ω x 0            (2.53)
From the above relation we can deduce that Ωω = 0 from which we obtain directly that (∇u)ω = (∇u T )ω which gives a new non-conservative stretching formulation:

∂ω ∂t = (∇u) T ω (2.54)
More generally we can choose a linear combination of those two terms:

∂ω ∂t = α∇u + (1 -α)(∇u) T ω ∀α ∈ R (2.55)
With α = 1 2 , we obtain the following expression for the stretching term:

∂ω ∂t = ∇u + ∇u T 2 ω = γω (2.56)
Equations ( 2.51) and ( 2.55) with α ∈ {0, 0.5, 1} give the classical stretching formulations used in the literature [START_REF] Mimeau | Conception and implementation of a hybrid vortex penalization method for solid-fluid-porous media : application to the passive control of incompressible flows[END_REF]]. Those formulations are directionally split as the following:

1. Conservative formulation:

∂ω ∂t = ∇ • (u ⊗ ω) ⇔           ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           ∂u x ω x ∂x + ∂u x ω y ∂y + ∂u x ω z ∂z ∂u y ω x ∂x + ∂u y ω y ∂y + ∂u y ω z ∂z ∂u z ω x ∂x + ∂u z ω y ∂y + ∂u z ω z ∂z          
The splitting of the conservative formulation is straightforward once the scalar equations have been expressed:

(a) x-axis splitting:

∂ω ∂t = ∂ ∂x [ω x u] (2.
∂ω ∂t = α∇u + (1 -α)∇u T ω ⇔           ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           = α           ∂u x ∂x ∂u x ∂y ∂u x ∂z ∂u y ∂x ∂u y ∂y ∂u y ∂z ∂u z ∂x ∂u z ∂y ∂u z ∂z                     ω x ω y ω z           + (1 -α)           ∂u x ∂x ∂u y ∂x ∂u z ∂x ∂u x ∂y ∂u y ∂y ∂u z ∂y ∂u x ∂z ∂u y ∂z ∂u z ∂z                     ω x ω y ω z          
The directional splitting of the non-conservative formulations is less elegant but is easily obtained by extracting the directional space derivatives:

(a) x-axis splitting:

          ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           ∂u x ∂x ω x + (1 -α) ∂u y ∂x ω y + ∂u z ∂x ω z α ∂u y ∂x ω x α ∂u z ∂x ω x          
(2.60) (b) y-axis splitting:

          ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           α ∂u x ∂y ω y ∂u y ∂y ω y + (1 -α) ∂u x ∂y ω x + ∂u z ∂y ω z α ∂u z ∂y ω y           (2.61) (c) z-axis splitting:           ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           α ∂u x ∂z ω z α ∂u y ∂z ω z ∂u z ∂z ω z + (1 -α) ∂u x ∂z ω x + ∂u y ∂z ω y          
(2.62) 2.5. Directional splitting 97

External forces and immersed boundaries

External forces (2.7) are handled by an operator of the form:

∂ω ∂t = ∇ × A ⇔           ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           ∂A z ∂y - ∂A y ∂z ∂A x ∂z - ∂A z ∂x ∂A y ∂x - ∂A x ∂y          
They can be split as the following:

1. x-axis splitting:

          ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           0 - ∂A z ∂x ∂A y ∂x           (2.63) 2. y-axis splitting:           ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           ∂A z ∂y 0 - ∂A x ∂y          
(2.64)

3. z-axis splitting:

          ∂ω x ∂t ∂ω y ∂t ∂ω z ∂t           =           - ∂A y ∂z ∂A x ∂y 0           (2.65)

Fourier spectral methods

Spectral methods are a class of techniques to numerically solve certain differential equations in which the solution of the differential equation is decomposed in a function space whose basis are non-zero over the whole domain [START_REF] Orszag | Numerical methods for the simulation of turbulence[END_REF]]. Unlike finite differences for which only local information are used to find a solution, spectral methods use basis functions that spans over the whole domain. Because spectral methods constitute global approaches, they offer excellent convergence properties given that the solution is smooth [START_REF] Gottlieb | Numerical analysis of spectral methods: theory and applications[END_REF].

In this section we will first focus on spectral methods arising the decomposition of periodic functions in Fourier series to numerically solve diffusion and Poisson problems on Cartesian grids. Spectral diffusion is unconditionally stable. It is an interesting alternative to explicit finite-differences based solvers when the diffusivity is high. Poisson solvers allow to correct the vorticity and compute the velocity from the corrected vorticity. This results into fast numerical solvers by using the fast Fourier transform [START_REF] Cooley | An algorithm for the machine calculation of complex Fourier series[END_REF]]. The method is then extended to handle Dirichlet and Neumann homogeneous boundary conditions by changing the basis of functions while remaining compatible with the use of the fast Fourier transform (FFT). The difficulties concerning domain decomposition within this spectral framework are discussed in the next chapter.

Discrete Fourier transform

In spectral methods for differential equations, considering one dimension here for simplicity, one has a periodic function y : [0, L[→ C with period L that expands as a Fourier series:

y(x) = ∞ p=-∞ y p e +2iπ px L y p = 1 L L 0 y(x) e -2iπ px L dx (2.66a) (2.66b)
where i denotes the imaginary unit.

One then wishes to apply a differential operator like d q dx q for some q ∈ N * . Differentiation is performed term-by-term in the Fourier domain:

dy q dx q (x) = ∞ p=-∞ 2iπ px L q y p e +2iπ px L (2.67)
which is just a pointwise multiplication of each y p by a complex factor depending on p.

To implement this on a computer, one approximates the Fourier series by a discrete Fourier transform (DFT) and we replace the function y(x) by N discrete samples y = (y j ) j∈ 0,N such that y j = y(jdx) with dx = L N . Within this discrete framework y = ( y p ) p∈ 0,N is approximated by the discrete Fourier transform (DFT):

y p = 1 N N -1 j=0 y j e -2iπ jp N (2.68)
and the discrete samples y j are recovered from the discrete Fourier coefficients y p by the associated inverse transform (IDFT):

y j = N -1 p=0 y p e +2iπ jp N (2.69)
An alternative and more compact notation is y = F (y) and y = F -1 ( y). With these notations, it follows that F -1 (F (y)) = y.

Fast Fourier Transform

The nice thing about the discrete Fourier transform expressions is that y can be computed from y in only O (N logN ) operations by a fast Fourier transform (FFT) algorithm instead of O N 2 operations obtained for a naïve implementation based on its definition. By far the most commonly used FFT is the Cooley-Tukey algorithm that is a divide and conquer algorithm that recursively breaks down a DFT of any composite size N = N 1 N 2 into many smaller DFTs of sizes N 1 and N 2 [START_REF] Cooley | An algorithm for the machine calculation of complex Fourier series[END_REF]]. The best known use of this algorithm is to divide the transform into two pieces of size N/2 at each step (radix-2 decimation), and is therefore limited to power-of-two sizes N = 2 p , but any factorization can be used in general. This algorithm has led to multiple variants called split-radix FFT algorithm [START_REF] Duhamel | Split radix'FFT algorithm[END_REF]]. The Cooley-Tukey has the disadvantage that it also requires extra multiplications by roots of unity called twiddle factors, in addition to the smaller transforms. The prime-factor algorithm (PFA) or Good-Thomas algorithm is a variant of Cooley-Tukey when N 1 and N 2 are coprimes that do not require those extra multiplications [START_REF] Good | The interaction algorithm and practical Fourier analysis[END_REF]].

Transforms of arbitrary sizes, including prime sizes can be computed by the use of Rader [START_REF] Rader | Discrete Fourier transforms when the number of data samples is prime[END_REF]] or Bluestein [START_REF] Bluestein | A linear filtering approach to the computation of discrete Fourier transform[END_REF]] algorithms that work by rewriting the DFT as a convolution. In the presence of round-off error, many FFT algorithms are much more accurate than evaluating the DFT definition directly. Considering that ε is the machine floating-point relative precision, the upper bound on the relative error for the Cooley-Tukey algorithm is O (ε log N ) versus O εN √ N for the naïve DFT formula [START_REF] Gentleman | Fast Fourier Transforms: for fun and profit[END_REF]]. The root mean square (RMS) errors are much better than these upper bounds, being only O (ε log N ) for Cooley-Tukey and O ε √ N for the naïve DFT [START_REF] Schatzman | Accuracy of the discrete Fourier transform and the fast Fourier transform[END_REF]]. Those algorithms makes working with Fourier series practical: we can quickly and accurately transform back and forth between space domain (where multiplying by functions is easy) and Fourier domain (where operations like derivatives are easy) by using single or double precision floating point numbers.

Trigonometric interpolation and derivatives

FFT implementations like FFTW, FFTPACK, MKL-FFT or clFFT compute DFTs and IDFTs in forms similar to equations (2.68) and (2.69), with the coefficients arranged in order (from p = 0 to N -1 for the forward transform and from j = 0 to N -1 for the backward transform). This ordering turns out to make the correct implementation of FFT-based differentiation more obscure. Let y denote the sampling of the q-th derivative of y(x) and y its corresponding discrete Fourier transform. From equation ( 2.67) one should expect that y can be expressed from y by the following relation: y p = 2iπ px L q y p . However in practice, this is not the case.

In order to compute derivatives, we need to use the IDFT expression to define a continuous interpolation between the samples y j and then differentiate this trigonometric interpolation. At first glance, interpolating seems very straightforward: one simply evaluates the IDFT expression (2.69) at non-integer x ∈ [0, L[:

y N (x) = N -1 p=0 y p e +2iπ px L (2.70)
This indeed defines an interpolation but this is not the only one. The reason there is more than one interpolation is due to aliasing: Any term of the form y p e +2iπ pj N in the IDFT can be replaced by y p e +2iπ j(p+mN ) N for any integer m and still give the same samples y because

y p e +2iπ pj N = y p e +2iπ pj N e +2iπjm =1 ∀m∈Z = y p e +2iπ j(p+mN ) N .
Essentially, adding mN to p means that the interpolated function y(x) oscillates m extra times in between the sample points y j and y j+1 . This has no effect on y but has a huge effect on its derivatives such as y . The unique minimal-oscillation trigonometric interpolation of order N is obtained by the following formula [START_REF] Johnson | Notes on FFT-based differentiation[END_REF]]:

y N (x) =            y 0 + (N/2-1) p=1 y p e +2iπ px L + y N -p e -2iπ px L + y N/2 cos π N x L if N is even y 0 + (N -1)/2 p=1 y p e +2iπ px L + y N -p e -2iπ px L if N is odd (2.71a) (2.71b)
In this formula, the N/2 (Nyquist) term is absent for odd N .

The treatment of the maximum frequency (Nyquist component) is especially tricky when computing derivatives on even sized grid that are typically fast to compute FFTs: an odd mode is left without a conjugate partner as seen in the interpolant (2.71a).

We define the

N wavenumbers k = (k p ) p∈ 0,N with k p =        2iπ L p if p ≤ N -1 2 2iπ L (p -N ) if p > N -1 2 .
Computing ∂y q ∂x q (x) from spectral interpolation (2.71b) and evaluating it x j ∈ x for an odd 2.6. Fourier spectral methods

101

N gives directly y p = (k p ) q y p . For even N and even derivatives, the result stays the same because the Nyquist component remains a cosine that evaluates to +1 or -1 at all discrete samples x j . For even N and odd derivatives however the derivation of (2.71a) contains a Nyquist component with a sine that evaluates to 0 at all x j such that y N/2 vanishes. In this case we have to modify the formula to y p = (1 -δ p,N/2 )(k p ) q y p to impose y N/2 = 0. Hence performing a second derivative is not equivalent to performing the spectral first derivative twice unless N is odd (the discretized spectral differential operators do not inherit all of the properties of the continuum operators).

To express any derivative, we define vector k = (k p ) p∈ 0,N with

k p = (1 -δ p,N/2 )k p . g N (x j ) = d 2q f dx 2q N (x j ) = N -1 p=0 g p e +2iπ jp N with g p = (k p ) 2q ∈R f p h N (x j ) = d 2q+1 f dx 2q+1 N (x j ) = N -1 p=0 h p e +2iπ jp N with h p = k p (k p ) 2q ∈C f p (2.72a) (2.72b)

Smoothness and spectral accuracy

In order to apply the discrete Fourier transform to interpolate a signal, compute derivatives or solve partial differential equations, the spectral approximation f N (x) from equation ( 2.71) should converge rapidly to f (x) as N → ∞. Let f ∈ L 2 (R) be a L-periodic function.

The convergence rate of the spectral approximation f N is dictated by the regularity of the periodized function f :

1. If f (x) is analytic, the Fourier series converges exponentially fast:

f ∈ L 2 (R) ∩ C ω (R) ⇒ ∃α > 0 f N (x) -f (x) ∞ = O N →∞ e -αN (2.73)
2. If f (x) has p square-integrable continuous derivatives and a (p + 1)-th derivative of bounded variation, the Fourier series converges at order p:

         f ∈ L 2 (R) ∩ C p (R) f (q) ∈ L 2 (R) ∀q ∈ 1, p f (p+1) ∈ BV (R) ⇒ f N (x) -f (x) ∞ = O N →∞ 1 N p+1
(2.74)

3. If f is piecewise continuous (discontinuous at some points or simply non-periodic), with a first derivative of bounded variation, f N is not a good approximation anymore:

   f ∈ L 2 (R) ∩ C 0 I (R) \ C 0 (R) f (1) ∈ BV (R) ⇒      f N (x) -f (x) ∞ = O N →∞ (1) f N (x) -f (x) 2 = O N →∞ 1 √ N (2.75)
The three different cases are illustrated on figure 2.13 and 2.14. The inability to recover point values of a non-periodic (or discontinuous), but otherwise perfectly smooth function from its Fourier coefficients is known as the Gibbs phenomenon. Away from the discontinuity the convergence is rather slow (|f (x) -f n (x)| = O (1/N )) and close to the discontinuous locations x d there is an overshoot that does not diminish with increasing

N (|f (x d ) -f N (x d )| = O (1)).

Figure 2.13 -Spectral interpolation of 1-periodic functions of varying regularity.

The three functions considered here are all one-periodic square-integrable functions with at least a first derivative with bounded variations. For all x ∈ [0, 1[ we define f , g and h as f

(x) = cos(2πx)sin(4πx) ∈ C ω (R), g(x) = 1 -2|0.5 -x| ∈ C 0 (R) and h(x) = |x| ∈ C 0 I (R) \ C 0 (R).
Those functions have decreasing regularity and match the three convergence cases (2.73), (2.74) and (2.75). Each leftmost plot represent the function f (x) to be interpolated (blue) and its spectral DFT interpolant f 7 (x) obtained by evaluating (2.71) using O (N ) operations per point (orange) with N = 7 samples. The samples x j = j/8 ∈ [0, 1[ used to compute the DFT are shown as red points and discrete interpolation of size 3N is represented by magenta crosses. This interpolation is obtained by an IDFT of the N spectral coefficients f padded with 2N zeros, computed in O (log 3N ) operations per point. Rightmost plots represent the pointwise errors between each function and their interpolant f (x) -f 7 (x).

DFT of a real signal

When the DFT takes a purely real signal y : [0, L[→ R as input, it is possible to use a smaller index set due to the Hermitian symmetry of the output: y p = y N -p * where c * denotes the complex conjugate of c. It is possible to take advantage of these circumstances in order to achieve roughly a factor of two improvement in both speed and memory usage. This real-tocomplex forward discrete Fourier transform that takes N reals as input and outputs N 2 + 1 complex numbers will be denoted H as opposed to F that performs a complex-to-complex transform The corresponding forward H and backward H -1 transforms are defined as:

y p = 1 N N -1 j=0 y j e -2iπ jp N ∀p ∈ 0, N/2 + 1 y j = y 0 + N -1 2 p=1 y p e +2iπ jp N + y * p e -2iπ jp N + 1 2Z (N ) y N/2 cos(jπ) =(-1) j ∀j ∈ 0, N (2.76a) (2.76b)
where 1 2Z (N ) is one when N is even and zero when N is odd.

Using the Hermitian symmetry of the output and equation (2.71) leads to:

y N (x) = y 0 + 2 N -1 2 p=1 Re ( y p )cos 2π px L -Im ( y p )sin 2π px L +1 2Z (N ) y N/2 cos π N x L (2.77)
In this formula ŷ0 ∈ R represent the mean of the signal and when N is even, ŷN/2 ∈ R is also a real coefficient associated to the highest frequency (Nyquist frequency). As a direct consequence, real-valued samples y j = y(x j ) will result in a purely real-valued interpolation y(x) for all x. Equations (2.72) remain valid to compute even and odd derivatives with modified versions of k and k , mainly k p = 2iπ L p and k p = (1 -δ p,N/2 )k p for p ∈ 0, N/2 .

Multi-dimensional discrete Fourier transform

The n-dimensional DFT is achieved by tensor product of one-dimensional DFTs. Let θ(x, t) : Ω×T → R be a n-dimensional L-periodic (in space) and time-dependent scalar field with

L = (L 1 , • • • , L n ) and Ω = [0, L 1 [× • • •×[0, L n [. The field is discretized on a Cartesian grid of N = N v = N c cells and vertices such that θ k = θ k j = θ(j dx, t k ) | j ∈ 0, N 1 × • • • × 0, N n with dx = L N and N = n i=1 N i .
The forward n-dimensional discrete Fourier transform applied to θ k is defined as:

θ k p = 1 N N 1 -1 j 1 =0   e -2iπ p 1 j 1 N 1 N 2 -1 j 2 =0   e -2iπ p 2 j 2 N 2 • • • Nn-1 jn=0 e -2iπ pnjn Nn θ k j     = 1 N N 1 -1 j 1 =0 • • • Nn-1 jn=0 θ k j e -2iπ n q=1 pq jq Nq ∀p ∈ 0, N 1 × • • • × 0, N n (2.78)
which can be simplified by using the following notation:

θ k = F θ k = F 1 F 2 • • • F n θ k
where F i denotes the forward partial discrete Fourier transform along axis i. Similarly we define the backward n-dimensional DFT as

θ k = F -1 θ k = F -1 n F -1 n-1 • • • F -1 1 θ k .
Those two transforms can be computed in O (N log N ) operations by using successive calls to any compatible one-dimensional FFT algorithm to compute each transforms F i or F -1 i .

The q-th spatial derivative of θ is obtained as in the one-dimensional case and for all j ∈ 0, N 1 × • • • × 0, N n we have:

ζ N (x j ) = ∂ q θ ∂x q N (x j ) = ∂ q 1 +•••+qn θ ∂x q 1 1 • • • ∂x qn n (j dx) = N 1 -1 p 1 =0 • • • Nn-1 pn=0 n i=1 exp 2iπ p i j i N i ζ p 1 ,••• ,pn with ζ p 1 ,••• ,pn = n i=1 1 -1 2Z+1 (q i ) δ p i , N i /2 (k p i ) q i θ p 1 ,••• ,pn (2.79)
If the partial derivative is not performed on every axis, it is possible to compute only a partial m-dimensional transform with m < n.

As we will only deal with real scalar fields as inputs, the n-dimensional real-to-complex forward discrete Fourier transform is defined as the following:

H : R N 1 ו••×N n-1 ×Nn → C N 1 ו••×N n-1 × Nn 2 +1 θ k → θ k = F 1 F 2 • • • F n-1 H n θ k (2.80)
Its corresponding complex-to-real backward transform is defined as

H -1 = H -1 n • F -1 n-1 • • • • • F -1
1 . In this case we have H -1 H θ k = θ k where where H i and H -1 i use the Hermitian symmetry of the DFT on the last axis (n is the first axis being forward transformed).

Spectral diffusion

Diffusion problems were already encountered in equation ( 2.3) and were already solved by using the finite differences approach in section 2.4.2. Let Ω be a n-dimensional periodic box of size L discretized on a grid of

N c = N v = N point with N = n i=1 N i . Let θ k be the discretization of θ(x, t) : Ω × T → R with θ k j = θ(j dx, t k ) ∀j ∈ J = 0, N 1 × • • • × 0, N n with dx = L N .
Here we suppose that θ(t = t k ) ∈ L 2 (R) and θ(t = t k+1 ) ∈ L 2 (R) are at least C 2 with first and second order derivatives that are square integrable and with third order derivatives of bounded variations.

We want to solve the following diffusion equation implicitely in time so that there will be no restriction on the timestep:

∂θ ∂t = κ ∆θ t ∈ [t k , t k + dt[ θ k = θ(t k ) (2.81a) (2.81b)
where κ is a constant. Implicit discretization in time using an Euler scheme yields:

θ(x, t k+1 ) = θ(x, t k ) + dt κ ∆θ(x, t k+1 ) ∀x ∈ Ω (2.82)
In order to solve this equation, we impose equality (2.82) for the variable discretized on the grid θ k j = θ(t k , x j ):

θ k+1 j = θ k j + dt κ ∆θ k+1 j ∀j ∈ J (2.83)
The derivatives can then be approximated at order p by a weighted sum of the discretized variable values:

θ k+1 j = θ k j + dt κ i∈J l j,i θ k+1 i + O N -p ∀j ∈ J ⇔ θ k+1 = θ k + dt κ L θ k+1 + O N -p for some L ∈ M N (R) (2.84) (2.85)
This equation can be rewritten

A θ k+1 = θ k + O N -p with A = I N -dt κ B ∈ M N (R) (2.86)
Using centered finite differences to approximate the derivatives leads to a linear system that is not easy to solve efficiently for a general n > 1. In this case A is a sparse Toeplitz matrix [START_REF] Recktenwald | Finite-difference approximations to the heat equation[END_REF]] and it is known that general Toeplitz systems can be solved by the Levinson algorithm in O N 2 operations [START_REF] Trench | An algorithm for the inversion of finite Toeplitz matrices[END_REF]] and can be reduced to O N 1.5 by using iterative algorithms such as successive over-relaxation [START_REF] Liu | Numerical solution of forward and backward problem for 2-D heat conduction equation[END_REF]]. The order in space of this method in space will depend on the order of the stencil choosed to approximate the second derivatives and the sparsity of A decreases with increasing order. When n = 1, the complexity drops to O (N ) for slightly modified tridiagonal Toeplitz systems by using the Scherman-Morison formula to recover a trigiagonal Toeplitz system along with the Thomas algorithm [START_REF] Stone | An efficient parallel algorithm for the solution of a tridiagonal linear system of equations[END_REF]].

We can also rewrite equation ( 2.82) by using the spectral interpolation of θ(x), denoted θ N (x) at time t k and t k+1 = t k + dt in order to obtain a solution in only O (N log N ) operations by using two FFTs along with O (N ) posprocessing:

θ N (x, t k+1 ) = θ N (x, t k ) + dt κ ∆θ k+1 N (x) ∀x ∈ Ω (2.87)
As before, in order to solve this equation, we impose equation (2.87) on the N spatial discretization points x j = i dx:

θ k+1 N (x j ) = θ k N (x j ) + dt κ ∆θ k+1 N (x j ) ∀j ∈ J (2.88)
Let θ k represent the n-dimensional complex-to-complex DFT of θ k and θ k+1 be the one of θ k+1 . We can rewrite equation 2.88 by identifying the coefficients in the spectral basis:

θ k+1 p = θ k p + dt κ ∆ θ k+1 p ∀p ∈ J = θ k p + dt κ n i=1 k 2 p i θ k+1 p ∀p ∈ J (2.89) (2.90)
which can be rewritten

1 -dt κ n i=1 k 2 p i θ k+1 p = θ k p ∀p ∈ J ⇔ A θ k+1 = θ k with A = I N -dt κ diag (k p • k p ) p∈J and k p = [k p 1 , • • • , k pn ] T (2.91) (2.92)
Hence in the frequency space the matrix A ∈ M N (R) becomes diagonal due to the properties of the Fourier transform with respect to the derivatives (2.72). This system can be solved in O (N ) operations. Here θ k is obtained by DFT in O (N log N ) operations and θ k+1 can be recovered by IDFT in O (N log N ) operations. The order of convergence of the method is O (N -p ) with p ≥ 2 depending n the smoothness of θ k+1 (x).

Spectral convergence is achieved only for smooth θ:

• Polynomial convergence is achieved for θ ∈ C ∞ given its derivatives are all squareintegrable. In this case the error is of order O (N -p ) for all p ∈ N.

• Exponential convergence is achieved for θ ∈ C ω , the additional condition being that there exists some constant C ∈ R such that the all its spatial derivatives θ (k) are bounded by [START_REF] Komatsu | A characterization of real analytic functions[END_REF]).

C k 1 +k 2 +•••+kn k 1 !k 2 ! • • • k n ! (see
In practice those two cases are indistinguishable at the discrete level. Although the diffusive process may help to enhance the local regularity away from the boundaries, this method does not work well for non-periodic boundary conditions because there is no hope of uniform convergence for discountinuous θ. The specific handling of general non-periodic boundaries is described in the last section of this chapter.

The algorithm to compute θ k+1 from θ k is straightforward:

input : dt, θ k = θ(t k ) output: θ k+1 = θ(t k + dt), solution of problem (2.81) θ k ← F θ k for p ∈ J do θ k+1 p ← θ k p 1 -dt κ (k p • k p ) end θ k+1 ←= F -1 θ k+1
Algorithm 7: Spectral resolution of a periodic Poisson problem

Vorticity correction and computation of velocity

Let ω * be the state of the discretized vorticity after convection, diffusion, stretching and external forcing corresponding to ω k,4 in algorithm 1 or ω k+ 1 2 ,4n in algorithm 4. The correction step consists into correcting ω * to ω k+1 such that ω k+1 = ∇ × u k+1 where u k+1 is divergence free. Once the vorticity has been projected, the divergence-free velocity can be recomputed from vorticity by solving Poisson equation (2.8) spectrally as in the last subsection. To derive required equations, it is convenient to work with the Helmholtz decomposition of the velocity u k+1 and the vorticity ω * :

u k+1 = ∇ × ψ + ∇φ ω k+1 = ω * + ∇e (2.93) (2.94)
In addition we are free to impose ∇ • ψ = 0 and we want the velocity to stay divergence free at time t k+1 : ∇ • u k+1 = 0

ω k+1 = ∇ × u k+1 (2.
∇ • ω k+1 = ∇ • (∇ × u k+1 ) = ∇ × (∇ • u k+1 =0 ) = 0 ∇ • u k+1 = ∇ • (∇ × ψ + ∇φ) = 0 ⇒ ∆φ = 0 ∇ • ω k+1 = ∇ • (ω * + ∇e) = 0 ⇒ ∆e = -∇ • ω * (2.97) (2.98) (2.99)
The Poisson problem relating to the correction of the vorticity is obtained by applying the Laplace operator to equation (2.94) along with equation (2.99):

∆ω k+1 = ∆ (ω * + ∇e) = ∆ω * + ∇∆e = ∆ω * -∇(∇ • ω * ) (2.100)
Finally, the equation relating to ψ is obtained by taking the curl of equation ( 2.93) along with equation (2.96):

ω k+1 = ∇ × u k+1 = ∇ × ∇ × ψ + ∇φ = ∇ × ∇ × ψ = -∆ψ + ∇ (∇ • ψ =0 ) = -∆ψ (2.101)
The general algorithm to compute ω k+1 and u k+1 from ω * is thus the following:

1. Correct ω * such that ω k+1 is divergence-free using a Poisson solver: 4. Compute u k+1 using the two computed velocities:

∆ω k+1 = ∆ω * -∇(∇ • ω * ) ( 2
u k+1 = u k+1 0 + u k+1 1 (2.108)
On a fully periodic domain Ω, the only solution to u 1 is 0 so step 3 and 4 can be skipped, and step 2 is simplified because we do not have to impose ∇ • ψ = 0 on the domain boundaries while solving for ψ. Other boundary conditions are taken in account in next subsection. In 2D, ∇ • ω * is 0 because ω * = [0, 0, ω * z (x, y)] T , so there is no correction to do, and step 1 can also be skipped. Basically this leads to the same algorithm as for the diffusion in the previous subsection, followed by the computation of a curl as exposed in algorithm 8. This algorithm requires one DFT and two IDFTs such that the total cost of the method is proportional to 3N log N with N = N 1 N 2 . The 3D case shown in algorithm 9 is a bit more computationally intensive because the vorticity has to be corrected, leading to 3 DFTs and 6 IDFTS for a total cost proportional to 9N log N with N = N 1 N 2 N 3 .

Fourier spectral methods

109

input : ω * , (u k+1

x , u k+1 y ), vorticity and spatial mean of the velocity components output: ω k+1 , (u k+1

x , u k+1 y ), divergence free vorticity and velocity ), divergence free vorticity and velocity

ω k+1 = ω * ω k+1 ← F ω k+1 for p = (p x , p y ) ∈ 0, N x × 0, N y do if p = (0, 0) then u k+1 x,0 , u k+1 y,0 ← u k+1 x , u k+1 y else ψ p ← ω k+1 p k 2 px + k 2 py u k+1 x,p , u k+1 y,p ← -k py ψ p , +k px ψ p end end u k+1 x , u k+1 y ← F -1 u k+1 x , F -1 u k+1 y
( ω * x , ω * y , ω * z ) ← F (ω * x ) , F ω * y , F (ω * z ) for p = (p x , p y , p z ) ∈ 0, N x × 0, N y × 0, N z do if p = (0, 0, 0) then ω k+1 x,0 , ω k+1 y,0 , ω k+1 z,0 ← ω * x,0 , ω * y,0 , ω * z,0 u k+1 x,0 , u k+1 y,0 , u k+1 z,0 ← u k+1 x , u k+1 y , u k+1 z else ω k+1 x,p , ω k+1 y,p , ω k+1 y,p ← ω * x,p , ω * y,p , ω * z,p -k p k px ω * x,p + k py ω * y,p + k pz ω * z,p k 2 px + k 2 py + k 2 pz ψ x,p , ψ y,p , ψ z,p ←   ω k+1 x,p k 2 px + k 2 py + k 2 pz , ω k+1 y,p k 2 px + k 2 py + k 2 pz , ω k+1 z,p k 2 px + k 2 py + k 2 pz   u k+1 x,p , u k+1 y,p , u k+1 z,p ← k pz ψ y,p -k py ψ z,p , k px ψ z,p -k pz ψ x,p , k py ψ x,p -k px ψ y,p end end u k+1 x , u k+1 y , u k+1 z ← F -1 u k+1 x , F -1 u k+1 y , F -1 u k+1 z ω k+1 x , ω k+1 y , ω k+1 z ← F -1 ω k+1 x , F -1 ω k+1 y , F -1 ω k+1 z
Algorithm 9: Algorithm to compute divergence-free vorticity and velocity on a 3D fully periodic domain discretized on N = (N z , N y , N x ) points with N * ∈ 2N + 1.

Real to real transforms and homogeneous boundary conditions

It is possible to extend the validity of algorithms 7, 8 and 9 to homogeneous Dirichlet and homogeneous Neumann boundary conditions by periodizing the signal by introducing left and right even or odd extensions [START_REF] Martucci | Symmetric convolution and the discrete sine and cosine transforms[END_REF]]. In order for this to work, the discrete Fourier transform (DFT) has to be replaced by either a discrete sine transform (DST) or a discrete cosine transform (DCT) corresponding to the required boundary condition. The distinction between a DST or a DCT and a DFT is that the former use only sine or cosine functions, while the latter uses both cosines and sines with the frequencies halved. Given N discrete equispaced samples f j = f (x j ) of a non-periodic function f : [0, L] → R, the signal can be periodized in various ways, each of them imposing a particular boundary condition. Each boundary can be either even or odd (2 choices per boundary) and can be symmetric about a data point or the point halfway between two data points (also 2 choices per boundary). This gives a total of 16 transforms, half of these possibilities where the left extension is even corresponding to the 8 types of DCTs and the other half where the left extension is odd to the 8 types of DSTs. In practice, boundary conditions are imposed on collocated or staggered grids so only 4 types of DCTs and DSTs are of interest in order to solve PDEs.

The four transforms for which the homogeneous boundary conditions can be prescribed at the grid points (collocated grid) are given bellow:

f p = N -1 j=0 (2 -δ j,N -1 )f j sin π N (p + 1 2 )(j + 1) Neumann-Dirichlet even-odd DCT-III f p = N -1 j=0 (2 -δ j,0 )f j cos π N (p + 1 2 )(j) Neumann-Neumann even-even DCT-I f p = N -1 j=0 (2 -δ j,0 -δ j,N -1 )f j cos π N -1 (p)(j)
The four other transforms correspond to boundaries prescribed in between two grid points:

Left-right B.C. Extensions Transform Unnormalized discrete transform Dirichlet-Dirichlet odd-odd DST-II f p = 2 N -1 j=0 f j sin π N (p + 1)(j + 1 2 ) Dirichlet-Neumann odd-even DST-IV f p = 2 N -1 j=0 f j sin π N (p + 1 2 )(j + 1 2 ) Neumann-Dirichlet even-odd DCT-IV f p = 2 N -1 j=0 f j cos π N (p + 1 2 )(j + 1 2 ) Neumann-Neumann even-even DCT-II f p = 2 N -1 j=0 f j cos π N (p)(j + 1 2 )
Choosing one of those transforms impose the left and right boundary conditions of the spectral interpolant f N for a given function f . In other words, f is projected into a basis that verify the chosen boundary conditions. The eight different cases are shown on the following figure: The effective size of the forward and backward transforms is not the same in all cases (3,4,4,5) top-to-bottom for the collocated cases and (5,5,5,5) for the staggered cases.

Chapter 2. Numerical method

Upper part of figure 2.15 shows the even and odd extensions for the transforms of interest where the boundary conditions are imposed on the grid. Other transforms are however required to compute the corresponding inverse transforms. Let (φ p ) p∈ 0,N -1 be the basis of sines or cosines of the considered transform, the spectral interpolant of f (x) is given by:

f (x) f N (x) = N -1 p=0 f p φ p (x) (2.109)
The reconstruction basis are summarized in the following table:

Transform Inverse Odd inv. s k p Basis functions φ p (x) DST-I DST-I DCT-I 0 π(p + 1)/L 2 sin (k p x) DST-II DST-III DCT-III 0 π(p + 1)/L (2 -δ p,N -1 ) sin (k p x) DST-III DST-II DCT-II 0 π (p + 1 2 )/L 2 sin (k p x) DST-IV DST-IV DCT-IV 0 π (p + 1 2 )/L 2 sin (k p x) DCT-I DCT-I DST-I 1 π p/L (2 -δ p,0 -δ p,N -1 ) cos (k p x) DCT-II DCT-III DST-III 1 π p/L (2 -δ p,0 ) cos (k p x) DCT-III DCT-II DST-II 1 π (p + 1 2 )/L 2 cos (k p x) DCT-IV DCT-IV DST-IV 1 π (p + 1 2 )/L 2 cos (k p x)
This brings to six the total number of required transforms to handle all homogeneous boundary conditions on a collocated grid (DCT-I, DCT-II, DCT-III, DST-I, DST-II and DST-III). All those transforms will be referred to as real-to-real transforms as opposed to the DFT and the IDFT which are complex-to-complex, real-to-complex or complex-to-real transforms.

The computation of even derivatives is done similarly to the DFT, by multiplying the Fourier coefficients f by some constant wavenumbers k that depends on p and by applying the corresponding inverse transform. For odd derivatives, things get a little more complicated because of the change of basis: sines become cosines, and cosines become sines. Thus the inverse transform to compute odd derivatives is not the same as the inverse transform used to compute even derivatives (the standard backward transform). This specific odd derivative transform will be referred to as the odd inverse transform. All inverse and odd inverse transforms are summarized in the previous table. As for the DFT, all the inverse transforms have to be scaled according to some implementation dependent normalization factor that depends on the transform size N . The size of the transform may be less than the grid size N because of implicit zeros enforced by the homogeneous Dirichlet boundary conditions as it can be seen on figure 2.15.

f (q) (x) f (q) N (x) = N -1 p=0 f p φ (q) p (x) = N -1 p=0 f p ϕ p (x)
(2.110)

Fourier spectral methods 113

For even derivatives φ p = ϕ p and N = N but for odd derivatives φ p and N may be different than ϕ p and N . The odd collocated case can be especially tricky to handle depending on the considered transforms, because of the change of the basis and the number of coefficients. Here because there is no periodicity, P = 0 and the grid size corresponds to the number of grid vertices N = N v = N c + 1, but the presence of homogeneous Dirichlet boundary conditions may alter the size N of the corresponding transform in the collocated case:

L

1. Collocated grid: boundaries are prescribed on the grid vertices. In this case we have dx = L/(N -1) and x j = j dx with j ∈ J ⊂ 0, N -1 :

Staggered grid:

Here the boundary conditions are imposed in between two grid points so the transforms are all of size N = N c = N v -1 = N -1. For all the cases we have dx = L/N = L/(N -1) and x j = (j + 1 2 ) dx with j ∈ 0, N -2 .

Computation of derivatives

Let q ∈ 2N, the computation of the even q-th derivative in the frequency space is done as in the Fourier space by elementwise multiplication of f by the real wavenumbers k = (k p ) p∈ 0,N -1 corresponding to the transform: k = (-1) (s+q)/2 k q . Here k q = k k • • • k is defined as the elementwise q-th power of k and s and k are forward transform dependent quantities that are summarized in the table under equation (2.109). The values of the derivatives are obtained at the same points than for the forward transform.

= N -(1 - d l ) -(1 -d r ) = N -d l -d r .
The location x j of the N resulting samples f (q) (x j ) correspond to the ones associated to the odd derivative forward transform: dx = L/(N -1)

x j = j dx ∀j ∈ d l , N -d r x j = j dx ∀j ∈ d l , N -d r k p = (-1) (s+q)/2 k q p ∀p ∈ 0, N f p =        0 if p = d l d r -1 0 if p = N -d l d r k p+d l d r f p+d l d r else ∀p ∈ 0, N (2.111a) (2.111b) (2.111c) (2.111d) (2.111e)

Convergence of the method

The real-to-real transforms are subject to the same convergence properties as the DFT when considering the discrete signal obtained by applying left and right extensions. For the DFT we saw that implicit periodicity of the transform meant that discontinuities usually occur at the boundaries because any random segment of a signal is unlikely to have the same value at both the left and right boundaries. A similar problem arises for all real-to-real transforms that happens to impose en odd symmetry (an homogeneous Dirichlet boundary condition).

For those six transforms there is a discontinuity for any function that does not happen to be zero at Dirichlet boundaries. When there is such a discontinuity, there is no expected uniform convergence of the interpolant ( f -f N ∞ = O (1)) so we cannot use this method for interpolation. Because of the integrations we may however expect to uniformly converge with second order to the solution of a Poisson equation (algorithm 7) and with first order to the solution of a Poisson-curl problem (algorithm 8 and 9). In practice we will only use those last two algorithms when the vorticity is zero at the Dirichlet boundaries such that the order only depend on the smoothness of the vorticity. In contrast, the Neumann-Neumann cases where both extensions are even always yields a continuous extension at the boundaries and we recover exponential convergence with any smooth input. This is why DCT-I and DCT-II generally perform better for interpolation and signal compression than other real-toreal transforms.

Multi-dimensional transforms

Let F = F 1 • F 2 • • • • • F n denote a n-
dimensional real-to-real or real-to-complex forward transform with associated real-to-real or complex-to-real backward transform

F -1 = F -1 n • • • • • F -1 1 .
Let S X and C X with X ∈ {I, II, III} denote the six required sine and cosine transforms and S -1 X and C -1 X their corresponding inverse transforms. We recall that the complex-to-complex DFT is denoted F and the real-to-complex DFT with Hermitian symmetry is denoted H. With these notations we have

F i ∈ {F, H, S I , S II , S III , C I , C II , C III , } and F -1 i ∈ F -1 , H -1 , S -1 I , S -1 II , S -1 III , C -1 I , C -1 II , C -1
III , providing reconstruction basis (φ i,p ) p∈ 0,M i where M represent the size of the transform. In practice the axes of the problem are reordered for performance reasons and the permutation is such that all the real-to-real transforms come first (non-periodic axes), followed an Hermitian real-to-complex DFT for the first periodic axis, and complex-to-complex DFTs for all the subsequent periodic axes. The first transform being done on axis x (the last one, contiguous in memory), such a configuration yields a periodicity mask of the form

P = (1, • • • , 1, 0, • • • , 0).
We recall that the number of grid vertices (the grid size) is defined by N v = N c + 1 -P where N c represent the number of grid cells and that dx = L N c . This defines nodes

x j = x start + j dx ∀j ∈ J = 0, N v 1 × • • • × 0, N v n .
Let

d l = (d i,l ) i∈ 1,
f = (f (x j )) j∈K with K = d 1,l , N 1 -d 1,r × • • • × d n,l , N n -d n,r be the M = n i=1 M i samples such that f = F (f ) = ( f p ) p∈K .
The spectral interpolant f M is defined similarly to the multi-dimensional DFT as:

f (x) f M (x -x start ) = M 1 -1 p 1 =0 • • • Mn-1 pn=0 n i=1 φ i,p i (x) f p 1 ,••• ,pn (2.112)
Each forward directional transform F i has associated diagonal derivative matrices of any given order q ∈ N such that in general D (q) i is not square and has shape (M i , M i ), D (q) i = D

(1) i q defined by (2.72) and (2.111). For all the transforms, those matrices depend on the parity Chapter 2. Numerical method of q, the simplest case being that for an even DFT derivative

D (q) i = diag(k 0 , • • • , k N i -1 ) q ∈ M N i (C)
where the k i are the associated complex wavenumbers. Let (F ) -1 denote the inverse transform required to compute the q-th derivative. (F ) -1 may be different than F -1 due to the change of basis entailed by odd derivatives on non-periodic axes. This also modifies the points x j where the derivative samples are recovered and we define

d l = 1 -d l , d r = 1 -d r , M = N -d l -d r , M = n i=1 M i and K = d 1,l , N 1 -d 1,r × • • • × d n,l , N n -d n,r .
With those notations we have:

f = f (x j ) j ∈ K f ∂ q 1 +•••+qn f ∂x q 1 1 • • • ∂x qn n (x j ) j ∈ K f = F (f ) f = K f f = (F ) -1 ( f ) K = D (q 1 ) 1 ⊗ D (q 2 ) 2 ⊗ • • • ⊗ D (qn) n ∈ M M ,M (K) with K ∈ {R, C} (2.113a) (2.113b) (2.113c) (2.113d) (2.113e) (2.113f)
In practice the matrices D

(q i ) i
and K are never built and we proceed by iterating on the spectral coefficients as in the pure periodic case. From now on for simplicity we consider only the case M = M where D

(q) i = diag(d (q) i,0 , • • • , d (q) i,M i -1
) is always a square diagonal matrix.

Modification of the algorithms to handle homogeneous boundary conditions

Spectral diffusion algorithm 7 only use second derivatives and requires minimal modification to handle all real-to-real transforms:

input : dt, θ k = θ(t k ) output: θ k+1 = θ(t k + dt), solution of problem (2.81) θ k ← F θ k for p ∈ K do θ k+1 p ← θ k p 1 -dt κ n i=1 d (2) i,p i end θ k+1 ←= F -1 θ k+1
Algorithm 10: Spectral diffusion with mixed periodic and homogeneous Dirichlet/Neumann boundaries Algorithm 8 and 9 that solve the Poisson-curl problem have to be modified such that the mean velocity can only be imposed for fully periodic problems, fully homogeneous Neumann boundaries problems or a mix of the two (which is coherent with the fact those are the only transforms for which d (q) 0 = 0). Moreover, the presence of odd derivatives may introduce zero-padding and offsets as seen in ( 2.111) on every axes.

Fourier spectral methods 117

For clarity those extra offsets have not been included in the following modified algorithm:

input : ω * , (u k+1
x , u k+1 y ), vorticity and weighted mean of the velocity components output: ω k+1 , (u k+1

x , u k+1 y ), divergence free vorticity and velocity

ω k+1 = ω * ω k+1 ← F ω k+1 for p ∈ K do if p 1 = p 2 = 0and d (2) 1,0 + d (2)
2,0 = 0 then u k+1

x,0 , u k+1 y,0

← u k+1 x , u k+1 y //Only happens for periodic/Neumann BCs else

ψ p ← ω k+1 p d (2) 1,p 1 + d (2) 2,p 2 u k+1 x,p , u k+1 y,p ← -d (1) 1,p 1 ψ p , +d (1)
2,p 2 ψ p end end

u k+1 x , u k+1 y ← (F ∂y ) -1 u k+1 x , (F ∂x ) -1 u k+1 y
Algorithm 11: Algorithm to compute divergence-free vorticity and velocity on a 2D domain with a mix of periodic and homogeneous boundary conditions.

Extension to general non-periodic boundary conditions

In order to solve spectrally Poisson problems with general boundary conditions, the spectral basis has once again to be changed. We already saw that because of the Gibbs phenomenon, non-periodic signals cannot be reconstructed accurately by IDFT, the error being of the order of the jump discontinuity (2.75). The same goes for real-to-real transforms enforcing Dirichlet boundary conditions where the signal does not happen to be zero on the said boundaries. In those problematic cases, increasing the number of samples N does not improve accuracy because f -f N ∞ = O (1) as shown on figure 2.14 for L ∞ convergence results.

In this subsection we propose to solve the following one-dimensional problem where

L (φ, ∂ x φ, ∂ x 2 φ, • • •
) is a linear operator with left and right boundary conditions specified:

L φ = f on [0, L] α l φ(0) + β l ∂φ ∂x (0) = γ l α r φ(L) + β r ∂φ ∂x (L) = γ r (α l , α r ) = (0, 0) (2.114a) (2.114b) (2.114c) (2.114d) Boundary condition α β γ Homogeneous Dirichlet 1 0 0 Homogeneous Neumann 0 1 0 Dirichlet 1 0 γ Neumann 0 1 γ Robin α β γ

General spectral methods

As already seen for the periodic case, the idea behind a spectral method is to approximate a solution f (x) by a finite sum:

f N (x) = N p=0 f p φ p (x) f p = b a h(x)f (x)φ p (x)dx (2.115a) (2.115b)
where {φ p } p∈ 0,N is a set of orthogonal basis functions with respect to some weighting function h(x) and where f represent a square integrable function with respect to h, noted as f ∈ L 2 h . Successful expansion basis meets the following requirements:

1. Convergence: The approximation f N (x) should converge uniformly to f (x) as N → ∞.

Differentiation:

Given coefficients f , it should be easy to determine the set of coefficients f representing some derivative:

f (q) N (x) = N p=0 f p φ (q) p (x) N p=0 f p φ p (x) (2.116)
3. Transformation: The computation of expansion coefficients f from function values u j = u(x j ) for j ∈ 0, N should be algorithmically efficient.

Extension to general non-periodic boundary conditions 119

Spectral accuracy for non-periodic functions is generally achieved trough the use of Fourier series in orthogonal polynomials on some interval (a, b). All of the classical orthogonal polynomials [START_REF] Gautschi | Orthogonal polynomials[END_REF]] can be reduced to one of the following four types:

Polynomial Symbol Weight h(x) Interval [a,b] Domain Trigonometric P p (x) 1 [0, 2π[ periodic Jacobi P α,β p (x) (1 -x) α (1 + x) β [-1, +1] bounded Laguerre L α p (x) x α e -x [0, +∞[ semi-bounded Hermite H p (x) e -x 2 ]-∞, +∞[ unbounded
As we only need to solve PDEs on bounded domains [0,L] the natural choice is to use Jacobi polynomials and their derivatives when the domain is not periodic.

Jacobi polynomials

Jacobi polynomials [Jacobi 1859] are useful to handle non-periodic functions on bounded domains Ω = [0, L] with a simple change of variable to [-1, 1]. The class of orthonormal Jacobi polynomials are defined for α > -1 and β > -1 by the following formula:

P α,β p (x) = ω α,β p (-1) p p!2 p (1 -x) -α (1 + x) -β d p dx p (1 -x) α (1 + x) β (1 -x 2 ) p ω α,β p = p! (α + β + 2p + 1) Γ (α + β + p + 1) 2 α+β+1 Γ (α + p + 1) Γ (β + p + 1) (2.117a) (2.117b)
Special cases of the Jacobi polynomials are the ultraspherical polynomials C λ p (x) (or Gegenbauer polynomials) obtained with α = β = λ -1 2 . Special cases of ultraspherical polynomials include Legendre polynomials L p (x) with α = β = 0, Chebyshev polynomials of the first kind T p (x) with α = β = -1 2 and Chebyshev polynomials of the second kind U p (x) with α = β = + 1 2 . Like for the periodic case, the Fourier-Jacobi series of a function f is uniformly convergent on [-1, 1] under regularity constraints and the convergence rate that depends on the smoothness of f but also the distribution of samples used to approximates the series coefficients.

As trigonometric interpolation in equispaced points suffers from the Gibbs phenomenon where f -f N ∞ = O (1), polynomial interpolation in equispaced points suffers from the Runge phenomenon where f -f N ∞ = O 2 N , which is much worse [START_REF] Grandclement | Stellar fluid dynamics and numerical simulations: from the sun to neutron stars[END_REF]]. A particularly interesting non-uniform distribution of points x j to limit the Runge phenomenon are the zeros of some Chebyshev and Legendre polynomials, also called Gauss-Lobatto-Chebyshev (GLC) and Gauss-Lobatto-Legendre points (GLL), respectively. Those nodes gives a polynomial interpolant that is not to far from the optimal polynomial of order N , denoted f opt N :

f -f opt N ∞ ≤ (1 + Λ N ) f -f N ∞ (2.118)
where Λ N is the Lebesgue constant [START_REF] Ibrahimoglu | Lebesgue functions and Lebesgue constants in polynomial interpolation[END_REF]] for the considered interpolation nodes:

Λ GLC N = O N →∞ (log N ) Λ GLL N = O N →∞ √ N Λ U nif orm N = O N →∞ 2 N N log N 120 Chapter 2.

Numerical method

The uniform convergence rate f -f N ∞ also depends on the interpolation technique:

1. Expansions in Legendre polynomials are optimal approximants in the L 2 -norm:

   f ∈ L 2 ([-1, 1]) ∩ C p ([-1, 1]) f (q) ∈ L 2 ([-1, 1]) ∀q ∈ 1, p ⇒ f -f N 2 = O N →∞ N -p-1-1 2 (2.119) 2.
Expansions in Chebyshev polynomials are optimal approximants in the L ∞ -norm:

   f ∈ L 2 h ([-1, 1]) ∩ C p ([-1, 1]) f (q) ∈ L 2 h ([-1, 1]) ∀q ∈ 1, p ⇒        f -f N L 2 h = O N →∞ N -p-1-1 2 f -f N ∞ = O N →∞ N -p-1
(2.120)

with weights h(x) = h Chebyshev (x) = 1 √ 1 -x 2 .

Chebyshev collocation method

The Chebyshev polynomials of the first kind can alternatively be defined by following the recurrence relation:

T 0 (x) = 1 T 1 (x) = x T p+1 (x) = 2xT p (x) -T p-1 (x) (2.121a) (2.121b) (2.121c)
T p is the only polynomial satisfying T p (cos(θ)) = cos(pθ).

The Chebyshev-Gauss-Lobatto points have a better asymptotic behaviour for the Lebesgue constant and have the advantage of being known analytically. The N +1 points have analytical formula x j = -cos (jπ/N ) ∈ [-1, 1] for j ∈ 0, N . Moreover, with the knowledge of the N +1 samples f = (f (x j )) j∈ 0,N it is possible to compute the expansion coefficients f in the basis of orthogonal Chebyshev polynomials of order N efficiently by applying a type one cosine transform (DCT-I) with O (N log N ) complexity by using a specialized variant of FFT [START_REF] Neagoe | Chebyshev nonuniform sampling cascaded with the discrete cosine transform for optimum interpolation[END_REF]]. In practice, it is better to reverse the indices of the Chebyshev-Gauss-Lobatto points, yielding x j = cos (jπ/N ) to have a direct correspondance with the DCT-I. In this case if we define weights w = (1 + δ p0 + δ pN ) p∈ 0,N we have f = C I (f ) w, the first and the last coefficients being scaled by a factor one-half. As the inverse of the DCT-I is the DCT-I itself, f can also be computed from f with the same complexity with f = C I ( f w). The result of these procedures has eventually to be scaled by some factor depending of N according to the chosen implementation. This fast algorithm only works when f has no sharp discontinuity on [-1, 1] because the DCT-I coefficients are subject to the same convergence results as the DFT (without the periodicity constraint) and thus are subject to the Gibbs phenomenon.

The same error result in O ε √ N holds when using a suitable algorithm (FFTW has a correct implementation unlike FFTPACK, see [Frigo et al. 2005] part B). Another way to compute the coefficients is to compute the N + 1 scalar products f, φ j h directly by using the discrete variant of formula (2.115b).

The convergence of the interpolation at the Chebyshev-Gauss-Lobatto points is illustrated on figure 2.17 for three different functions defined on [-1, 1] with varying regularity. As for the Fourier transform in the periodic case, a limit case can be provided by a discontinuous function where convergence can not be guaranteed as formulated in (2.120).

Figure 2.17 -Spectral Chebyshev interpolation of functions of varying regularity.

The three functions considered here are all square-integrable functions with respect to weight function h(x) = 1/ √ 1 -x 2 . For all x ∈ [-1, 1] we define f , g and h similarly to the DFT interpolation presented on figure 2.13 as f (x) = cos(2πy)sin(4πy

) ∈ C ω (R) with y = (x+1)/2, g(x) = 1 -|x| ∈ C 0 (R) and h(x) = [ [ [x > 0] ] ] ∈ C 0 I (R).
Those functions have decreasing regularity and theoretical convergence rate is given by (2.120). Each leftmost plot represent the function f (x) to be interpolated (blue) and its spectral Chebyshev interpolant f 16 (x) (orange) obtained by evaluating (2.71) with N + 1 = 16 samples. The 16 samples f j = f (x j ) with x j = cos(jπ/15)) ∈ [-1, 1] used to compute the interpolation are shown as red points. See figure 2.17 for the definition of functions f , g and h. Here the coefficients f and g are computed efficiently by using the DCT-I while coefficients h are computed up to N = 2 12 by scalar products because of the discontinuity of the function h(x). The black dotted line represents the machine error for double precision floating point numbers y = ε 2.22×10 -16 .

Given N+1 values of at the Gauss-Lobatto-Chebyshev points one can also approximate f (x) by using the N +1 interpolating polynomials L N,j (x) associated to the Chebyshev polynomial of degree N :

L N,j (x) = (-1) j+1 (1 -x 2 )T N (x) (1 + δ j0 + δ jN )N 2 (x -x j ) ∀ j ∈ 0, N f (x) N j=0 L N,j (x) f (x j ) = N j=0 L N,j (x) f j L N,i (x j ) = δ ij ∀(i, j) ∈ 0, N 2 (2.122a) (2.122b) (2.122c)
This leads to a first way to compute derivatives at the Chebyshev extreme points:

f (q) (x) N j=0 L (q) N,j (x) f j f = D q N f (2.123a) (2.123b)
Multiplication by the first order Chebyshev differentiation matrix

D N = ∂L N,j ∂x (x i ) ij trans-
forms a vector data at the Gauss-Lobatto-Chebyshev points into approximative derivative on those same points. D N is a matrix of size (N + 1) × (N + 1) and is explicitly defined by:

(D N ) 00 = -(D N ) N N = 2N 2 + 1 6 (D N ) jj = -x j 2(1 -x 2 j ) ∀ j ∈ 1, N -1 (D N ) ij = 1 + δ i0 + δ iN 1 + δ j0 + δ jN (-1) i+j x i -x j ∀ i = j

Extension to general non-periodic boundary conditions 123

The Chebyshev differentiation matrix is nilpotent of degree N + 1 and has the interesting property of being skew-centrosymmetric implying that D 2q N is centrosymmetric and D 2q+1 N is skew-centrosymmetric for all positive integers q. In practice we loose the symmetry properties when taking into account certain boundary conditions because D N is modified to take into account the boundary conditions at x = x 0 = -1 and x = x N = 1. Suppose we want to solve the following second order boundary value partial differential equation:

N i=0 a i (x) ∂ i φ ∂x i (x) = f (x) on ] -1, +1[ α l φ(-1) + β l ∂φ ∂x (-1) = γ l α r φ(+1) + β r ∂φ ∂x (+1) = γ r (α l , α r ) = (0, 0) (2.124a) (2.124b) (2.124c) (2.124d) Let x 0 , • • • , x N be the N+1 Chebyshev-Gauss-Lobatto collocation points in [-1,+1]. We can define A i = diag(a i (x 0 ), • • • , a i (x N )
) for all i ∈ 0, N and:

φ =            φ(x 0 ) φ(x 1 ) φ(x 2 ) . . . φ(x N -1 ) φ(x N )            f =            f (x 0 ) f (x 1 ) f (x 2 ) . . . f (x N -1 ) f (x N )            f =            γ r f (x 1 ) f (x 2 )
. . .

f (x N -1 ) γ l           
(2.125)

L N = N i=0 A i (D N ) i and L N =         α r (I N +1 ) 0j + β r (D N ) 0j (L N ) ij α l (I N +1 ) N j + β l (D N ) N j         (2.126)
The solution to the linear system L N φ = f satisfies the two boundary conditions (2.124b) at x = x 0 = -1 and (2.124c) at x = x N = +1 while equation (2.124a) is satisfied at all the N -1 inner collocation points x 1 , • • • , x N -1 .

Chebyshev-Tau method

An other way to solve this system is to use the representation of φ(x) in the basis of Chebyshev polynomials. The derivatives can then be obtained in the spectral basis similarly to the DFT 124 Chapter 2. Numerical method via the multiplication of the coefficient by an upper triangular first derivative matrix D N .

f N (x) = N p=0 f p T p (x) f (q) N (x) = N p=0 f p T (q) p (x) = N p=0 f p T p (x) (2.127a) (2.127b)
where f can be obtained from f with the knowledge of the of the matrix D N = ∂ x T i , T j h ij that transforms the spectral coefficients of the interpolant to the spectral coefficients of the derivative in the orthogonal Chebyshev basis:

f = ( D N ) q f ( D N ) ij = 1 2Z+1 (i + j) j 2 -δ i0 ∀(i, j) ∈ 0, N 2 (2.128a) (2.128b)
The procedure to solve the problem is then same then for the Fourier transform:

• All spatial derivatives are computed in the spectral space with equation (2.128).

• All linear operations (additions, subtractions, multiplication by a scalar) can be equally made in the spectral or in the physical space:

(c 0 φ + c 1 ψ) N (x) = c 0 φ N (x) + c 1 ψ N (x) F (c 0 φ + c 1 ψ) = c 0 φ + c 1 ψ
• All non-linear products are made in the real space, and the result is transfered back to the spectral space:

(φ ψ) N (x) = φ N (x) ψ N (x) F (φ ψ) = F F -1 φ F -1 ψ
where special care may be taken to dealias the product [START_REF] Uhlmann | The need for de-aliasing in a Chebyshev pseudo-spectral method[END_REF] If we suppose that all the a i do not depend on x with a i (x) = c i in equation ( 2.124a) then we

obtain the system L N φ = f with L N = N i=0 c i D N i
. The left (2.124b) and right (2.124c) boundary conditions are then included in the two last rows of L N . The value of the Chebyshev polynomials and their first derivative at left and right boundaries can be computed with the following formulas:

T (0) p (-1) = (-1) p T (0) p (1) = 1 T (1) p (-1) = (-1) p+1 p 2 T (1) p (1) = p 2 (2.129a) (2.129b)

Extension to general non-periodic boundary conditions 125

Thus by defining T l = (T p (-1)) p , T l = T

L N =            ( L N ) ij α l T l + β l T l α r T r + β r T r            and f =            f (x 0 ) f (x 1 ) . . . f (x N -2 ) γ l γ r            (2.130)
The solution to the linear system L N φ = f satisfies the system including the two boundary conditions. Once the system has been solved, it is possible to recover the solution φ at the Chebyshev extreme points by performing an inverse Chebyshev transform on φ, which is nothing more than a scaled inverse type one cosine transform. Similarly f is obtained from f by using a forward Chebyshev transform.

Chebyshev-Galerkin method

An alternative method to solve second order boundary value problems with Fourier-Chebyshev series is to take into account the boundary conditions directly in the basis, similarly to the real-to-real transforms of previous section. In the context of spectral methods this is known as the Galerkin method.

For the Chebyshev polynomials it is known that for homogeneous boundary conditions of the form (2.124b)-(2.124c) with γ l = γ r = 0, there exist a unique set of constants (a p , b p ) ∈ R N +1 × R N +1 such that:

G p (x) = T p (x) + a p T p+1 (x) + b p T p+2 (x) (2.131)
satisties the boundary conditions for given (α l , α r , β l , β r ). The resulting basis is not orthogonal but spans the solution space [START_REF] Shen | Spectral methods: algorithms, analysis and applications[END_REF]].

Here we propose to extend this result to non-homogeneous boundary conditions. The boundary values being given by equations ( 2.129), we can derive a system to solve for the coefficients that satify any boundary condition (with γ l = 0 and γ r = 0):

G (0) p (-1) = (-1) p + a p (-1) p+1 + b p (-1) p+2 G (0) p (+1) = 1 + a p + b p G (1) p (-1) = (-1) p p 2 + a p (-1) p+1 (p + 1) 2 + b p (-1) p+2 (p + 2) 2 G (1) p (+1) = p 2 + a p (p + 1) 2 + b p (p + 2) 2 (2.132a) (2.132b) (2.132c) (2.132d)
the system being given by:

α l G (0) p (-1) + β l G (1) p (-1) = γ l α r G (0) p (+1) + β r G (1) p (+1) = γ r (2.133a) (2.133b)

Chapter 2. Numerical method

The general solution to this system can be found by any symbolic computation software and we obtain the following:

a p = A p B p and b p = C p D p A p = (-1) 1-p α r γ l + 4 β r γ l (p + 1) + β r γ l p 2 + α l γ r + 4(α l β r -α r β l + β l γ r )(p + 1) + β l γ r p 2 B p = (2 α l α r + 5 α l β r + 5 α r β l + 8 β l β r ) + (6 α l β r + 6 α r β l + 24 β l β r )p + (2 α l β r + 2 α r β l + 26 β l β r )p 2 + (12 β l β r )p 3 + (2 β l β r )p 4 C p = -α l -β l p 2 + (-1) -p γ l α r + β r p 2 + 2β r p + β r + -α r -β r p 2 + γ r α l + β l p 2 + 2β l p + β l D p = α l + β l p 2 + 2β l p + β l α r + β r p 2 + 4β r p + 4β r + α l + β l p 2 + 4β l p + 4β l α r + β r p 2 + 2β r p + β r
The expression of a p and b p is greatly simplified for the common cases of Dirichlet-Dirichlet (α l = α r = 1), Dirichlet-Neumann (α l = β r = 1), Neumann-Dirichlet (β l = α r = 1) and Neumann-Neumann (β l = β r = 1) boundary conditions:

(α l , β l , α r , β r ) a p b p (1, 0, 1, 0) γ r 2 - (-1) -p γ l 2 γ r 2 + (-1) -p γ l 2 -1
(1, 0, 0, 1) (γ r + 4(p + 1)) + (-1) 1-p γ l p 2 + 4γ l p + 4γ l 2p 2 + 6p + 5

γ r -p 2 + (-1) -p γ l -1 p 2 + 2p + 1 2p 2 + 6p + 5 (0, 1, 1, 0) γ r p 2 + 4(γ r -1)(p + 1) + (-1) 1+p γ l 2p 2 + 6p + 5 (-1) p γ l -p 2 + (γ r -1) p 2 + 2p + 1 2p 2 + 6p + 5 (0, 1, 0, 1) γ r + (-1) 1+p γ l 2 (p 2 + 2p + 1) γ r -2p 2 + (-1) p γ l 2 (p 2 + 4p + 4)

Table 2.3 -Chebyshev-Galerkin coefficients (2.131) for usual boundary conditions

Once the coeffients a p and b p are known for given left and right boundary conditions, it is possible to express any function f in terms of Chebyshev and Chebyshev-Galerkin Fourier series:

f N (x) = N p=0 f p T p (x) = N -2 p=0 f p G p (x) with G p (x) = T p (x) + a p T p+1 (x) + b p T p+2 (x)
(2.135)

Extension to general non-periodic boundary conditions 127

By identifying terms in the Chebyshev orthogonal basis, it is possible to express the N + 1 Chebyshev coefficients f in terms of the N -1 Chebyshev-Galerkin coefficients f :

f 0 = f 0 f 1 = f 1 + a 0 f 0 f p = f p + a p-1 f p-1 + b p-2 f p-2 ∀p ∈ 2, N -2 f N -1 = a N -2 f N -2 + b N -3 f N -3 f N = b N -2 f N -2 (2.136a) (2.136b) (2.136c) (2.136d) (2.136e)
This gives an O (N ) procedure to go from the Chebyshev-Galerkin coefficients to the Chebyshev coefficients and we denote R c the associated matrix of size (N + 1, N -1) such that f = R c f . Because R c is not a square matrix, we can not compute its inverse, fortunately there exist two different iterative approaches with the same complexity to compute f from f :

Forward algorithm:

f 0 = f 0 f 1 = f 1 -a 0 f 0 f p = f p -a p-1 f p-1 -b p-2 f p-2
Backward algorithm (defined for b p = 0):

f N -2 = f N /b N -2 f N -3 = ( f N -1 -a N -2 f N -2 )/b N -3 f N -2-p = ( f N -p -f N -p -a N -p-1 f N -p-1 )/b N -p
We denote R g the matrix of size (N -1, N + 1) associated to one of those algorithms. With this definition we have f = R g f under the assumption that the N + 1 coefficients f represent a function with the correct boundary conditions (2.124b) and (2.124c). With these matrices with have

R g R c = I N -2 but R c R g = I N .
The Chebyshev-Galerkin derivative matrix D N of shape (N -1, N -1) is obtained from the Chebyshev derivation matrix D N of shape (N + 1, N + 1) by computing

D N = R g D N R c .
The procedure to find the solution to problem (2.124) with constant coefficients at the Chebyshev-Gauss-Lobatto points is then the following:

1. Preprocessing:

(a) Compute the N -1 coefficients a p and b p corresponding to the left and right boundary condition by using equation (2.134).

(b) Compute D N and L N = N i=0 c i D N i , L N is a square matrix of size N + 1. (c) Compute R c , R g and L N = R g L N R c , L N is a square matrix of size N -1.
2. Compute the coefficients of f from f in O (N log N ) by using a scaled DCT-I transform.

Compute

f = R g f in O (N ) operations.
4. Solve L N φ = f to find the coefficients of φ in the Chebyshev-Galerkin basis.

5. Compute φ = R c φ in O (N ) operations.

6. Compute the φ from φ in O (N log N ) by using a scaled DCT-I transform.

Comparison of the different methods

In this section we check the convergence of the different methods with respect to the following one-dimensional Poisson problem:

∂φ 2 ∂x 2 (x) = f (x) = e 4x on Ω = ] -1, 1[ φ(-1) = φ(1) = 0 (2.139a) (2.139b)
The analytical solution is φ(x) = (f (x) -xsinh( 4) -cosh( 4))/16. As the problem features homogeneous Dirichlet boundary conditions we can use the DST-I and DST-II transforms with associated collocated or staggered grid. With those transforms we cannot expect better than second order convergence because of the Gibbs phenomenon: f (1) 54.6 0. We compare the results obtained by implicit centered finite differences and various spectral methods with the analytical solution. The associated linear systems are shown on figure 2.19 for a discretization of N = 32 nodes. Each dot corresponds to a non-zero entry in the matrix. Coefficients are color graded from lowest (blue) to highest absolute value (red). FDCx corresponds to implicit centered finite differences of given spatial order (decentered on the boundaries when required).

For the discrete sine transforms, there is no linear system to solve because the forward transform performs the diagonalization of the problem. Implicit centered finite-differences schemes lead to banded Toeplitz matrices of increasing bandwidth. The Chebyshev collocation method requires the resolution of a dense matrix while the Chebyshev-Galerkin method yields a checkerboard matrix. Finally, the Chebyshev-Tau method gives an upper triangular checkerboard matrix. All those linear systems are solved without using any preconditioner.

As it can be seen on figure 2.20, we recover the expected convergence rates for all methods.

Conclusion

In this chapter we built an hardware agnostic numerical method that allows us to solve the incompressible Navier-Stokes equations passively coupled with the transport of a scalar, possibly at high Schmidt number. The main idea behind the proposed numerical method is the use of a remeshed particle method. While this method is particularly adapted for the transport of scalars in high Schmidt number flows, the presence of an underlying grid allows to the use of efficient eulerian solvers. The resulting method consists in an hybrid particle, spectral and finite differences based numerical method especially designed to reduce the timestep constraints associated to high Schmidt number flows. Those numerical methods are further adapted in order to become accelerator-friendly.

Within this framework, the governing equations are discretized on a rectangular computational box and the domain is meshed by using regular grids. Boundary conditions are prescribed on the walls. The numerical method is not limited to periodic boundary conditions and can take into account other kind of boundary conditions such as homogeneous Dirichlet, homogeneous Neumann or even more general boundary conditions. The numerical method is based on two different kinds of operator splitting, namely viscous and directional splitting.

Viscous splitting decomposes the problem into subproblems that each can be solved with a specific set of numerical methods. All of those subproblems but the recovery of velocity from vorticity are further split directionally by using a Strang splitting of first or second order, yielding one-dimensional subproblems. This makes the corresponding numerical scheme dimension-agnostic and thus simpler to implement. It is predicted that the choice of a Cartesian discretization along with a simple data permutation strategy in between the treatment of different directions allows efficient implementations on vectorized hardware such as CPUs and GPUs.

Transport is solved at fixed velocity, one direction at a time, by using a high order remeshed particle method that does not suffer from a timestep restriction depending on the grid size (CFL) but rather on velocity gradients (LCFL). Stretching and external forcing are handled with directionally split explicit finite differences. Diffusion can be treated either by using explicit finite differences or with a spectral solver. When the overall timestep of the simulation is limited by some timestep associated to a diffusion subproblem, it may be better to choose an implicit spectral method that does not enforce any timestep restriction. Finally, the recovery of velocity from the vorticity is always done through a spectral solver. All of those operators are solved one after another within a simulation iteration with variable timestepping. Explicit finite differences and transport each enforce a minimum timestep for a given iteration. A wide variety of spectral methods has been presented to handle any second order boundary value problems. Periodicity is handled with the Fourier transform while homogeneous boundary conditions can be handled by sine and cosine transforms under certain assumptions. All the other cases are handled by Chebyshev polynomials, at the cost of increased computational complexity. The recovery of velocity from vorticity and diffusion can be solved by using a tensor product of spectral bases statisfying the right boundary conditions. With this spectral method, there is no linear system to solve unless Chebyshev polynomials are used.

In the next chapter we will see how to implement efficiently those numerical methods depending on the target hardware. Results obtained with this numerical framework as well as performance statistics will also be given. The split operators presented in this chapter constitute the basic bricks that will be used to solve every other incompressible Navier-Stokes problems.

Chapter 3

Implementation and High

Introduction

One of the main objective of this thesis consists in the efficient implementation of the numerical schemes presented in the previous chapter. The resulting implementation should be able to operate on accelerators, possibly in a distributed environment. The numerical code should be easy to adapt to new architectures while retaining the possibility of operating on conventional machines for the sake of sustainability.

In a first part, we give an overall presentation of the HySoP library, the general highperformance computing framework in which we will combine multiple levels of parallelism. Within this framework, the description of a numerical algorithm is handled by building a graph of operators. This first global view on the library is the opportunity to focus on task parallelism.

In a second part, we focus on the specific care required for the implementation on hardware accelerators. Numerical methods are implemented once for multiple different architectures by relying on the OpenCL standard. We show that the regularity of the data structures combined to data permutations leads to coalesced memory accesses that are well suited for vectorized hardware. Performance-portability can achieved by the way of code generation techniques associated to automatic runtime kernel performance tuning. Benchmark results are given and analyzed for four reference configurations ranging from simple consumer grade desktop CPUs to professional grade GPUs. In particular we focus on the individual OpenCL implementation of each algorithm as well as runtimes statistics obtained for full two-and three-dimensional incompressible Navier-Stokes simulations. The numerical scheme is then validated on the Taylor-Green Vortex problem.

The last part is dedicated to the distributed implementation of the algorithms, domain decomposition and the specific challenges raised by distributed computing in the context of spectral transforms.

The HySoP library 133

The HySoP library

HySoP is a library dedicated to high performance direct numerical simulation of fluid related problems based on semi-lagrangian particle methods, for hybrid architectures providing multiple compute devices including CPUs, GPUs or MICs. The high level functionalities and the user interface are mainly written in Python using the object oriented programming model. This choice was made because of the large software integration benefits it can provide [START_REF] Sanner | Python: a programming language for software integration and development[END_REF]]. Moreover, the object oriented programming model offers a flexible framework to implement scientific libraries when compared to the imperative programming model [START_REF] Arge | Modern software tools for scientific computing[END_REF]] [START_REF] Cary | Comparison of C++ and Fortran 90 for object-oriented scientific programming[END_REF]. It is also a good choice for the users as the Python language is easy to use for beginners while experienced programmers can pick it up very quickly [Oliphant 2007a]. The numerical solvers are mostly implemented using compiled languages such as Fortran, C/C++, or OpenCL for obvious performance reasons. It is also possible to implement numerical algorithms using directly Python, which is an interpreted language, hence slower for critical code paths under heavy arithmetic or memory load. The Python language support is however the key for rapid development cycles of experimental features. It also allows to easily implement routines that compute simulation statistics during runtime, relieving most of the user post-processing efforts and enabling live simulation monitoring.

The compiled Fortran and C++ compute backends allow us to integrate a variety of external dependencies by connecting them to the main HySoP python module with interface wrappers such as F2PY or SWIG. Note that many scientific libraries already provide Python interfaces so that they can be directly used in python without needing the user to implement his own wrapper. In addition to the compiled languages, the library offers the possibility to compile generated code just-in-time during execution. This is the case for OpenCL, the language used to drive OpenCL-compatible accelerators, like GPUs, but also to translate python methods to fast machine code by using the Numba just-in-time compiler. Most of the dependencies that the HySoP library uses are described in appendix B.

This subsection provides implementation details of the HySoP library from its origin to the most recent developments (2012)(2013)(2014)(2015)(2016)(2017)(2018)(2019). It aims to cover computational-backend agnostic design choices that allows us to discretize the variables and equations of a given fluid-related problem into subproblems, each being solved individually by some operator embedded into a directed acyclic graph. A first level of parallelism is explored here, namely task parallelism.

Origin and current state of the library

The name of the library, Hybrid Simulation with Particles (HySoP), comes from its original purpose to propose a high performance framework for flow simulations based on semilagrangian particle methods. This library got created under the impulsion of [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]].

To this day the scope of the library is a bit larger and the library has more or less become a framework to develop high performance computational fluid-related routines operating on fields discretized on regular meshes. The library still uses the original Fortran implementa-tion [START_REF] Lagaert | Particle method: an efficient tool for direct numerical simulations of a high Schmidt number passive scalar in turbulent flow[END_REF] of the remeshed particle proposed by [Cottet et al. 2009b] and further developed by [START_REF] Magni | Accurate, non-oscillatory, remeshing schemes for particle methods[END_REF]]. The original Fortran code, named SCALES, was capable of running on massively parallel CPU architectures by using the Message Passing Interface (MPI). At this time the advection remeshing routines were coupled with a fully pseudo-spectral Navier-Stokes solver.

Shortly after, HySoP was born, by wrapping a subset of the SCALES library, mainly the particle remeshing functions, using F2PY and by implementing a custom solver for the fluid simulation, arising from successive developments of the underlying numerical schemes [Balarac et al. 2014] [Lagaert et al. 2014]. Current developments of the library builds on the original implementation of [START_REF] Etancelin | Multi-CPU and multi-GPU hybrid computations of multi-scale scalar transport[END_REF]] featuring an high order remeshing method and the porting of CPU intensive particle remeshing routines onto GPU accelerators by using the OpenCL standard [START_REF] Cottet | High order semi-Lagrangian particle methods for transport equations: numerical analysis and implementation issues[END_REF]]. This was the genesis of the OpenCL computing backend in the HySoP library. During this phase, the directional splitting approach was introduced in order to optimize the memory accesses, and thus improve achieved the memory bandwidth, for the OpenCL-based remeshing kernels. In the mean time, an hybrid vortex penalization method was developed to be able to handle complex geometries on Cartesian grids [START_REF] Mimeau | Passive flow control around a semi-circular cylinder using porous coatings[END_REF]] and to solve more challenging problems [START_REF] Mimeau | Vortex penalization method for bluff body flows[END_REF][START_REF] Mimeau | Passive control of the flow around a hemisphere using porous media[END_REF]].

Apart from developing the HySoP library, current efforts focus on the integration of high order remeshed particles within the YALES2 library [START_REF] Moureau | Design of a massively parallel CFD code for complex geometries[END_REF]]. One of the main objective of this thesis was to continue the porting efforts towards accelerators for other sequential CPU operators that quickly became the new simulation bottleneck on GPUenabled compute nodes. In addition to this performance objective, there was also the need to support homogeneous boundary conditions as the library historically only handled periodic boundary conditions which are not suitable for most sedimentation problems. Due to the huge number of features to add to the existing library and the existing partition of code bases, it was decided to fork the project to rethink the entire codebase. The main vision concerning the new implementation was to keep most of the parts that worked in the original library while using the experience gained during those first development years to improve what did not work as expected. From now on the original code prior to the fork will be referred as hysop-origin and the current state of the library as hysop-current.

Using lines of code (LOC) to measure software progress is like using kilograms for measuring progress in aircraft manufacturing. It is however a good metric to grasp where the developing efforts went during the transition period. Figure 3.2 shows that the entire code base has grown by a factor 2.6 in terms of total source code lines. Most of the efforts went into adding new features directly into the core of the library by using the Python programming language (x3.5). The Fortran backend, implementing legacy operators like the remeshing kernels and some spectral operators has remained roughly the same as the original implementation. The C++ backend is completely new and mainly addresses non-periodic boundary conditions.

At first glance, the lack of OpenCL sources in hysop-current might be surprising for a version of the library that pretends to support more OpenCL operators compared to its predecessor. This is due to the change of philosophy in the handling of OpenCL source code, passing from parsed static kernels to fully code generated ones for more versatility (see section 3.2 for more information). Finally an increase can be seen in terms of tests and continuous integration, represented by the Other section that contains mainly scripts, build utilities and Docker files used to automate the testing process. During this period Git records 1034 changed files, 135k insertions and 62k deletions. The main differences between the two versions are summarized in table 3.1. Details about the original design of the library can be found mostly in [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]] and [START_REF] Mimeau | Conception and implementation of a hybrid vortex penalization method for solid-fluid-porous media : application to the passive control of incompressible flows[END_REF]].

The HySoP library 137

Design of the library

As stated in the introduction, the library is designed around a Python core interface sitting on the top of different computing backends such as Fortran, C++ and OpenCL. The end user needs to specify his problem using a Python script to interact with the library. Figure 3.3 show the backends interact with the base Python layer.

The code is organized around various mathematical and logical concepts.

Physical domain

The first thing the user has to do is to specify the domain of definition of the variables involved in the problem. A physical domain Ω ⊂ R n represents the physical domain where the simulation will take place. At this time the only available type of domain in the library are n-dimensional rectangular cuboids, obstacles being handled by using a vortex penalization method [START_REF] Mimeau | Direct numerical simulations of three-dimensional flows past obstacles with a vortex penalization method[END_REF]. Rectangular boxes can be described by an origin A Box is a Domain that owns all those properties (L, P, • • • ), while BoxView is a read-only access to those properties with axes permutated. In practice a Box is a BoxView with default axes (1, • • • , n). Views will prove themselves useful in the presence of local data permutations.

x start = (x 0 , • • • , x n 1 )

Continuous fields

Once a domain is defined, it becomes possible to create named continuous scalar, vector or tensor fields. In HySoP, a continuous field is an abstract object which represents the usual vector field, in a mathematical sense, i.e some function which associates integer, real or complex valued scalar, vector or tensor to each point x of the space at a given time t:

f : Ω × T → K m = K m 1 ×m 2 ו••×m M (x, t) → f (x, t) with Ω ⊂ R n , T = [t start , t end ], K ⊂ C, n ∈ N * , M ∈ N * and m i ∈ N * ∀i ∈ 0, M .
In practice M = 1 for scalar fields, M = 2 for vector fields and M = 3 for second order tensor fields such as the gradient of velocity. Such objects are used as input and/or output variables of operators and must be defined with at least:

• a name: compulsory (required for logs, i/o and code generation).

• a domain: the n-dimensional physical domain Ω of definition of the field.

The underlying scalar data type that will be set upon discretization can also be specified. By default fields are set to data type HYSOP REAL which will be either float (single precision floating point numbers) or double (double precision floating point numbers) depending on HySoP build configuration. Integer and complex data types are also supported. Default boundary conditions can also be specified. As an example if the field will only be discretized on boxed domains, one can specify left and right boundary conditions for each of the n axes. Scalar, vector and tensor fields are seen as a scalar field containers and each inherit the common FieldContainerI interface such that independent scalar field operations are implemented only once. Each scalar field is associated to a symbolic representation with the help of Sympy. Tensor fields are associated to NumPy arrays of scalar field symbols. Those symbolic representations can be used later within a code generation framework.

Continuous fields are later discretized on one or more topologies. Topologies are domain discretizations associated to a specific domain decomposition. Before this discretization step, no memory is dynamically allocated for fields.

Parameters

They constitute another source of time-dependent inputs and outputs for operators.

p : T → K m = K m 1 ×m 2 ו••×m M t → p(t)
Together with continuous fields, they compose what we will call input and output variables. They are designed like continuous fields that have no space dependencies and follow exactly the same implementation design: ScalarParameter and TensorParameter can be of any data type and all inherit from a common ScalarParameterContainerI interface. They are associated to symbolic variables and the memory required by a given parameter is allocated at its creation (there is no spatial discretization step for parameters). The dependence in time can be removed by declaring the parameter constant and by providing an initial value.

Discretization parameters

A DiscretizationParameter instance contains information about the way to discretize a given type of domain (type of the mesh, distribution and number of mesh nodes). A DiscretizationParameter does not assume anything about the boundary conditions but the boundary conditions may alter the way fields are discretized because they impose a specific choice of spectral methods. A CartesianDiscretizationParameter just contains the number of cells N c a Cartesian grid (uniform space step dx) should contain in each direction and is compatible with Box domains. Other Box-compatible variants include ChebyshevDiscretizationParameter that also contains the number Chebyshev-Gauss-Lobatto nodes (variable space step dx) the resulting regular grid should contain in each direction (here N c corresponds the order of the Chebyshev polynomials). Finally RegularGridDiscretizationParameter allows the user to enforce the node distribution (uniform or CGL) on a per-axis basis. It can also be configured to deduce the node distributions from the knowledge of the scalar field left and right boundary conditions.

Compatible left and right boundary pairs include:

• Periodic boundary conditions : PERIODIC-PERIODIC

• Homogeneous boundary conditions: ODD-ODD, ODD-EVEN, EVEN-ODD, EVEN-EVEN

• General boundary conditions:

BC(α l , β l , γ l )-BC(α r , β r , γ r )
The two first kind of boundary pairs are mapped to uniform sample spacings (compatible with the usual discrete Fourier transform and collocated real-to-real transforms). Axes for which general boundary conditions are imposed are mapped to Chebyshev-Gauss-Lobatto nodes (compatible with the fast Chebyshev transform). For performance reasons, the user has to order the axes in such a way that the general boundary condition pairs come first, followed by the periodic boundary pairs and finally homogeneous boundary pairs. This is so that spectral transforms are performed by chaining real-to-real (sine and cosine transforms), real-to-complex (FFT with Hermitian symmetry), complex-to-complex (complex FFT) and finally fast Chebyshev transforms (scaled type one cosine transforms).

Discretization

A Discretization is obtained when bringing together a Domain and DiscretizationParameter.

The most common type of discretization we will use is the CartesianDiscretization that is the result of applying a CartesianDiscretizationParameter to a Box domain.

The boxed domain origin x min , size L and periodicity P allows to compute the physical node coordinates x j . The resulting grid has size N v = N c + 1 -P and the node coordinates are deduced from the specified node distribution. For regular grids we store only the per direction offsets (x i,j ) j∈ 0,N v i for every axe i ∈ 0, n . The position of a grid node on a regular grid is then given by x j = (x 1,j 1 , x 2,j 2 , • • • , x n,jn ) and the different node distributions are recalled here:

• Uniform distribution: x i,j = x min,i + j dx i with dx i = L i /N c i .
• Gauss-Chebyshev-Lobatto distribution:

x i,j = x min,i + L i (1 -cos(jπ/N c i ))/2

Topologies and meshes

A Topology brings together a Discretization, a MPI communicator, a compute backend and a specified number of ghosts G ∈ N n . It is responsible to generate a Mesh object and handle domain decomposition among all processes of the communicator. For a regular discretization of boxed domains this mainly consists into splitting the global regular grid into smaller local regular grids. Ghosts are extra boundary nodes required by some operators to compute values close to the local subdomain boundaries. A Mesh also maps the localto-process mesh properties to global mesh properties. For the usual regular grid case, one obtain a CartesianTopology operating on a MPI Cart communicator (with splitting mask S ∈ {0, 1} n and periodicity mask P ∈ {0, 1} n ) on the top of a CartesianMesh. The splitting mask S controls the directions in which the domain can be split while the periodicity mask enables or disables the communications between processes handling global domain left and right boundaries. It is required that S • S ≥ 1 when the number of processes P is more than one so that domain decomposition can happen. The CartesianTopology provides facilities to perform local ghost exchanges and accumulation with direct and diagonal neighbor processes. The compute backend dictates the memory allocation policy (where the memory will be allocated) and drives eventual host-to-device and device-to-host memory exchanges required by any OpenCL context.

Compute backends

Currently there exist three kinds of compute backends that dictates where memory will be physically allocated and what memory exchanges are required to perform local data redistributions and inter-process communications:

• HOST BACKEND: Memory is allocated in the main memory. This first kind of backend is used to implement Python operators (PythonOperator) or wrap C++ and Fortran operators (CppOperator and FortranOperator). With this backend, MPI exchanges can be done directly from and to host buffers.

• OPENCL BACKEND: Memory is allocated in the memory of a specific OpenCL device on a specific OpenCL platform. Dedicated GPUs and MICs have their own embedded memory and CPUs can use OpenCL buffers stored the main memory. Here two variants are possible depending on the location of the OpenCL buffers. CPU devices can map and unmap their memory directly to the main memory such that all HOST BACKEND compatible operators and MPI exchanges works seamlessly. Devices providing their own memory require data exchanges between host and device memories. In this case MPI exchanges are done less efficiently trough temporary hosts buffers (unlike CUDA, MPI cannot work directly with OpenCL buffers [START_REF] Wang | GPU-aware MPI on RDMA-enabled clusters: Design, implementation and evaluation[END_REF]).

• HYBRID BACKEND: Memory is allocated in the main memory and is partially allocated on every supplied backend that cannot map memory to host (mostly OpenCL devices with dedicated memory banks). This is the only backend that allows hybrid CPU-GPU computations or multi-device computations by using only one MPI process. CPU cores can be used by any OpenCL platform compatible with the target CPU (such as AMD, Intel and POCL OpenCL platforms) or in a multi-threaded context (Numba, MKL-FFT, FFTW).

Operators

An Operator represents a function that takes zero or more continuous fields and parameters as inputs and zero or more continuous fields and parameters as outputs. As one may expect, operators do not work with every type of topologies, discretizations and boundary conditions. Each operator proposes a default implementation (on a specific compute backend) and enforce requirements on the discretization of the fields and the layout of the topology on a per variable basis (FieldRequirements) and on a global variable basis (OperatorRequirements). Operators are usually bound to a compute backend trough the common interfaces (PythonOperator, CppOperator, FortranOperator, OpenClOperator and HybridOperator) that enforce default requirements such as the memory ordering and memory location.

Simulation

A Simulation instance is responsible to handle the time parameter t as well as the timestep parameter dt of a given discretized problem. It calls the underlying problem operators implementation in order at every timestep. The timestep can be either fixed or decided during the current timestep by any operator. Many timestep criterias are implemented in the library such as CFL and LCFL constraints.

Problems

A Problem is a sequence of operators that are organized in a directed acyclic graph (DAG).

Operators are created by the user and inserted one by one into the graph. The nodes represent operators while edges represent read and write dependencies of scalar fields and parameters.

The order of application of operations is deduced by performing a topological sort of this graph. Apart from giving the order of operations this graph also allows us to extract task parallelism. Subproblems can be represented with subgraphs or operator generators and can be inserted directly into a Problem.

Discrete fields

Discrete fields are continuous fields that have been discretized on a given topology. A single continuous field can be discretized on a given domain on multiple different topologies (different mesh, number of ghosts, compute backend and domain decomposition). The DiscreteField class hierarchy is a bit more complicated because it has to handle views. In general DiscreteScalarField is a ScalarField discretized on a Mesh handled by some Topology. A CartesianDiscreteScalarField is a ScalarField discretized on a CartesianMesh handled a CartesianTopology backed up by a MPI Cart communicator. All processes handling a given DiscreteScalarField allocate their own local data (typically one value per local mesh node). The size and location of the memory depends on the chosen discretization, topology and compute backend. For Cartesian meshes, the resulting local subgrids are also regular grids that can be allocated as one big local chunk of contiguous memory. A DiscreteScalarFieldView is a read-only access to a given DiscreteScalarField property (but not necessarily to the discrete field's data). As for continuous fields, discrete fields come with their tensor counterparts (which are M -dimensional containers of DiscreteScalarFieldViews). As usual all those types inherit common container interfaces to group common functionalities. An operator can declare to require some temporary discrete fields. In this case the discrete fields data is automatically provided by a temporary buffer memory pool that is common to all operators. The validity of the content of a temporary field (or any other kind of temporary buffer) is tied to the lifetime of the current applied operator.

CartesianDiscreteScalarField

Topology state

A TopologyState is a type of object that can alter the functionality of a View (such as DomainView, TopologyView, MeshView, DiscreteFieldView and their child class variants). One property is independent of the type of topology and record whether the underlying data should be considered to be read-only or read-write (only relevant to DiscreteFields which are objects that own data). A view that is set up with a read-only state will only return host and device buffers marked as read-only (whenever the associated compute backend offers this possibility). The topology state of a CartesianTopology is extended to handle a current local transposition state and C or Fortran ordering. This comes from the fact that once the boxed domain has been decomposed, each process allocates a unique contiguous chunk of local memory for all its required discrete fields. Those two additional states record the actual logical representation of this memory.

Operators are implemented by accessing to domain, topology, mesh, discrete field information through views that are set up during the construction of the problem (the graph of operators) where each topology states are tracked and updated depending on operator requirements. For example accessing the global grid size

N v = (N v z N v y , N v x ) of a three-dimensional discrete field trough a view will return (N v y , N v x , N v z )
if the actual local transposition state is set to (2,3,1). An operator that enforces the last axis (x-axis) to be continuous in memory for a given field has to ask for either (1, 2, 3) or (2, 1, 3) transposition states trough a FieldRequirement for the considered input or output field. The graph generator is then responsible to modify the last known state of the field by inserting automatically operators such as local transpositions to satisfy the operator requirements. The graph builder also automatically handles memory transfers between topologies with different compute backends (host-host, host-device and device-device transfers), and data redistribution for discrete fields that are defined on different topologies.

HySoP from a user point of view

In practice, the user only defines its variables (fields and parameters) and builds a problem by inserting operators. Operators can be obtained through simplified operator interfaces that gather all the available implementations. The user specifies associated input and output variables along with their discretizations (most of the time the knowledge of N c is enough for uniform Cartesian grids). The library then handles the rest of the operations by generating a directed graph of operators, finding common topologies between operators depending on their requirements and discretizing the fields. Prior to simulation, the initialization of the fields can be done with simple Python methods. A typical user script looks like this: #> Compute initial vorticity formula symbolically and translate symbolic # expressions to equivalent numerical functions by using numpy and sympy import numpy as np, sympy as sm x, y = sm.symbols('x y', real=True) u, v = sm.tanh(30*(0.25-sm.Abs(y-0.5))), 0.5*sm.sin(2*sm.pi*x) w = v.diff(x) -u.diff(y) U,V,W = (sm.lambdify((x,y), f) for f in (u,v,w)) def init_velocity (data, coords, component) [START_REF] Brown | Performance of under-resolved two-dimensional incompressible flow simulations[END_REF]] (p.17) by using the same ideas as algorithm 4: advection and diffusion of vorticity ω at fixed velocity u by using the directional splitting approach (introduced in section 2.5), followed by the recovery of the velocity from the vorticity using a spectral Poisson-curl operator (section 2.6). The full example including variable timestep is available as one of the many examples provided by the library, see hysop/examples/shear layer/shear layer.py.

Because nothing has been specified by the user, the library assumes that all numerical methods use their default parameters. The library defaults to:

• Time integrator: explicit second order Runge-Kutta scheme (RK2).

• Finite differences: explicit fourth order centered finite differences (FDC4)

• Interpolation: linear interpolation (LINEAR)

• Remeshing kernel: C 2 with four preserved moments Λ 4,2 (L42)

The DirectionalDiffusion operator defaults to explicit finite differences with default time integrator (RK2+FDC4) and DirectionalAdvection operator to default time integrator, linear interpolation of velocity and default remeshing kernel (RK2+L42+LINEAR). Default boundary conditions and discretization are all set to PERIODIC and CartesianDiscretization so that the PoissonCurl operator uses a Fourier spectral solver on a regular Cartesian grid. The library then deduces the number of required ghosts depending on the order of the numericals methods, build the topologies and the graph of operators. The resulting DAG can be obtained by calling problem.display() right after problem.build(). Operators, represented by nodes have default names that can be overridden by the user (here 'adv', 'diff', 'poisson' corresponds to DirectionalAdvection, DirectionalDiffusion and PoissonCurl operators). For performance reasons, the library automatically inserts LocalTranpose operators that are responsible to make the current axe contiguous in memory in directional splitting contexts (default name 'T'). The edges show the variable dependencies between operators. Edge dependencies are shown as discrete fields (continuous field + topology index) or parameters:

Graph of operators and task parallelism

The HySoP library is responsible to generate a Directed Acyclic Graph of operators based on user input. In this graph, the vertices represent tasks to be performed (operators), and the edges represent constraints that one task must be performed before another. This DAG of tasks is then analyzed to extract independent execution paths that can be scheduled on available hardware [START_REF] Kwok | Static scheduling algorithms for allocating directed task graphs to multiprocessors[END_REF][START_REF] Simon | Scheduling task graphs on modern computing platforms[END_REF][START_REF] Canon | Online scheduling of task graphs on hybrid platforms[END_REF]. Once the user has described and inserted all operators, the graph builder proceeds as the following:

1. Generate initial directed graph: Iterate over all operators as ordered by the user and compute the following on a per operator basis:

(a) Iterate over operator input fields and check their current topology state. If the current topology state does not match one of the operator required topology states, do not insert this operator directly into the graph but insert as many as required local permutations and topology redistributions as required to fulfill current operator needs. Data redistribution include host-device memory transfers as well as inter-process communications.

(b) Handle input dependencies: For each input scalar field, insert an edge from the operator lastly wrote the field to the current one. Add this operator to the reading record of all input fields.

(c) Handle output dependencies: For each output scalar field, insert an edge from all operators that are reading the field to the current one. Clear the reading and writing records of all output fields and add this operator to the writing record.

From the knowledge of all the input field topology states, compute resulting output field topology states and update the topology state record of all output fields.

An example of initial DAG obtained after this step is given on figure 3.7.

2. Insert additional operators to match input topology states: Additional local permutations and redistributions of data may be required in order to loop over a problem (output states have to match input states). The graph is not closed so that is does not contain any directed cycles.

3. Perform topological sort: Apart from drastically reducing the number of edges contained in the graph, the result of the topological ordering gives a valid execution sequence of the tasks.

Extract independent execution queues:

Iterate on operators in topological order to find out which one can be executed concurrently. Generate a new work queue until all independent operators are affected (see figure 3.8).

Independent tasks can be distributed on different compute nodes (different processes) or locally on the same compute nodes by using multi-threading, OpenCL device fission or asynchronous programming techniques (or a mix of all those methods). Support of compute-node level task parallelism has already been implemented in [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]].

Integration of numerical methods

When required, the HySoP library dynamically generates the exact numerical coefficients of the numerical schemes. This includes finite-differences coefficients and remeshing formulas (polynomial coefficients), which values can be obtained by solving a corresponding linear system (equation 2.37). Those linear systems are solved by using infinite precision rational numbers based linear solvers provided either by Sympy or Python-FLINT Python modules.

Implementing a new explicit Runge-Kutta scheme only requires the user to provide a Butcher tableau (equation 2.31). Those numerical schemes are used later for the implementation of host-side operators but also device-side operators such as in the OpenCL code generation framework. The code generation framework allows kernels to be specialized and optimized exactly for user chosen numerical methods. Once generated, coefficients are cached to disk.

For efficiency reasons, explicit centered finite differences associated to multi-stage time integrators are treated in one go without any ghost exchanges happening in the middle of the process. In a given direction, this implies to multiply the number of required ghosts g by the number of stages s, resulting in a ghost layer of size s g. The number of required ghosts g is at least the spatial order of approximation of the derivatives divided by two. This means that for fourth order centered finite differences FDC4 (requires 2 ghosts vertices, see section 2.4.1) and a second order Runge-Kutta time integrator RK2 (two integration steps), the ghost layer will be of size 4. With this configuration, one-dimensional diffusion would be solved as the following:

u = u(x, t), solve ∂u ∂t = ∂u 2 ∂x 2 on Ω from t = t 0 to t 0 + dt u k,0 i = u n i ∀i ∈ -4, N + 3 F k,0 i = -u k,0 i-2 + 16u k,0 i-1 -30u k,0 i + 16u k,0 i+1 -u k,0 i+2 12 dx 2 ∀i ∈ -2, N + 1 u k,1 i = u k,0 i + dt 2 F k,0 i ∀i ∈ -2, N + 1 F k,1 i = -u k,1 i-2 + 16u k,1 i-1 -30u k,1 i + 16u k,1 i+1 -u k,1 i+2 12 dx 2 ∀i ∈ 0, N -1 u k+1 i = u k,0 i + dt F k,1 i ∀i ∈ 0, N -1 u k -4 u k -3 u k -2 u k -1 u k 0 u k 1 u k 2 u k 3 u k 4 u k 5 u k 6 u k 7 u k 8 u k 9 u k N u k N +1 u k N +2 u k N +3 u k,0 -4 u k,0 -3 u k,0 -2 u k,0 -1 u k,0 0 u k,0 1 u k,0 2 u k,0 3 u k,0 4 u k,0 5 u k,0 6 u k,0 7 u k,0 8 u k,0 9 u k,0 N u k,0 N +1 u k,0 N +2 u k,0 N +3 F k,0 -2 F k,0 -1 F k,0 0 F k,0 1 F k,0 2 F k,0 3 F k,0 4 F k,0 5 F k,0 6 F k,0 7 F k,0 8 F k,0 9 F k,0 N F k,0 N +1 u k,1 -2 u k,1 -1 u k,1 0 u k,1 1 u k,1 2 u k,1 3 u k,1 4 u k,1 5 u k,1 6 u k,1 7 u k,1 8 u k,1 9 u k,1 N u k,1 N +1 F k,1 0 F k,1 1 F k,1 2 F k,1 3 F k,1 4 F k,1 5 F k,1 6 F k,1 7 F k,1 8 F k,1 9 u k,2 0 u k,2 1 u k,2 2 u k,2 3 u k,2 4 u k,2 5 u k,2 6 u k,2 7 u k,2 8 u k,2

Implementation on hardware accelerators

One of the main objective of this thesis was to pursue the OpenCL development efforts towards hardware-accelerated numerical simulation in HySoP, initiated by [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]]. Graphics Processing Units (GPUs) and other accelerators promise tremendous advantages in throughput over conventional processor architectures. Those advantages ideally result in a large reduction of execution time for suitable compute or bandwidth-bound algorithms. However, execution time is not the only time scale to consider when comparing architectures. The development time for a scientific code will, in many cases, constitute a significant fraction of its useful lifespan. CUDA and OpenCL are the two competing standards capable of driving GPUs [Su et al. 2012b]. They both provide a language that is a subset of C or C++ and a runtime library dedicated to compile and execute code on compatible target devices. A short overview and comparison of both those standards is proposed in [START_REF] Tompson | An introduction to the opencl programming model[END_REF].

Within the HySoP library, the choice of the OpenCL standard over CUDA has been made so that more devices can be supported within a single implementation (such as CPUs, MICs and GPUs from any vendor). Under fair comparison, OpenCL offers roughly the sames performances guaranties than CUDA [START_REF] Fang | A comprehensive performance comparison of CUDA and OpenCL[END_REF]]. The HySoP library requires the OpenCL platforms to be compliant with the OpenCL 1.2 standard:

• OpenCL 1.1 offers memory operations on subregions of a buffer including read, write and copy of 1D, 2D, or 3D rectangular subregions [Khronos2010]. This is required to perform efficient boundary layer exchanges (ghost exchanges).

• OpenCL 1.2 introduces device partitioning and IEEE 754 compliance for single-precision floating point math [Khronos2011] required for the correctness of single-precision numerical routines.

While the OpenCL standard provides a unified abstraction to widely divergent hardware architectures, performance portability remains a difficult problem, even for contemporary competing architectures available today [START_REF] Klöckner | PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation[END_REF]. As an example, writing efficient software for GPUs using low-level programming environments such as OpenCL or CUDA requires the developer to map computation directly onto GPU architectural design (number of compute units, on-chip memory type and size, designed arithmetic intensity, optimal memory access patterns, ...). Architectural design often being unavailable to the programmer, GPU programming relies extensively on experimentation and microbenchmarking.

This section describes the new strategies that have been incorporated into the HySoP library to reduce the time passed in OpenCL development cycles and to guaranty some performance portability across nowadays, and hopefully future generations of architectures. The resulting implementation is able to run fully accelerated Navier-Stokes simulations on CPUs and GPUs.

It can be extended by users without prior OpenCL experience by using a code generation framework based on symbolic expressions. Benchmark results are given for four reference compute platforms ranging from consumer grade CPUs to server grade GPUs (section 1.4.5).

The resulting implementation is validated on a three-dimensional reference simulation.

Arrays, buffers and memory handling

All the memory is allocated prior to simulation during the building of the problem. Problem building is constituted of the following three steps:

1. Initialization: Each operator contained in the problem is initialized, operator topology requirements are set up (min and max number of ghosts G, splitting mask S, local transposition state T , C or Fortran data ordering, compute backend, ...).

2. Discretization: the topology requirements are collected at a scalar field level and compatible requirements are merged together in order to create the minimal number of different topologies for a given scalar field. Topologies are created and continuous scalar fields are discretized on each of their associated topologies. The graph of operators is created and simplified with topological sort, giving the order of application as well as independent execution of operators.

Setup:

Each operator asks for extra temporary buffers to work with. Those extra work buffers can be of any shape and data type and can even require special memory alignment. Temporary fields memory requirements are automatically registered. All the memory requests are collected, reduced to one-dimensional contiguous chunks of memory and allocated in common memory pools. Each memory sharing group (execution queue and compute backend pair) gets its own memory pool. Those raw contiguous buffers are then sliced, reshaped and viewed as any other data type to suit each of the memory requests. Temporary buffers obtained by this mechanism are only considered valid while the operator is being applied because of this memory sharing policy (the temporary buffer content may be overwritten by any subsequent operator). This of importance since GPU memory is usually a precious ressource [START_REF] Rupp | CPU, GPU and MIC Hardware Characteristics over Time[END_REF]].

Dynamic allocation of memory is a slow operation and is not suited for time critical sections of the code such as the execution of an operator. This is mainly why all the memory is preallocated in the discretization and setup steps. It is not released until the very end of the simulation through a call to problem.finalize(). The memory is either provided by Python or OpenCL buffers which are respectively wrapped by numpy.ndarray or pyopencl.Array ndimensional array types. The NumPy library is responsible to allocate host side memory while the PyOpenCL library is responsible to allocate device memory. A pyopencl.Array is a numpy.ndarray work-alike that stores its data and performs its computations on a given OpenCL compute device. PyOpenCL provides some functionalities that are equivalent to numpy through a runtime code generation framework provided by the library itself [START_REF] Klöckner | PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation[END_REF]. This framework includes elementwise algebraic operations, reductions and scans with automatic type promotion. Basically, each different array operation triggers the code generator, the compilation of the generated code and the call to the resulting OpenCL kernel (a kernel refers to a function that can be executed on a OpenCL device). As an example, the expression a+b * sin(c) where a, b and c are three n-dimensional arrays will generate, compile and execute three different kernels (sin(•), • * • and • + •) for every different shape and data type of the operands (the library also supports broadcasting of scalars). It also requires to dynamically allocate three temporary arrays of the same shape and datatype as the intermediate result for every kernel. This leads to the generation of kernels with smaller arithmetic intensities and dynamic allocation of temporaries. Evaluating array expressions by using overloaded operators is thus less efficient. It is possible to use the lower level kernel generation interfaces of the PyOpenCL library (ElementwiseKernel, ReductionKernel and GenericScanKernel) to generate kernels that evaluate multi-stage expressions on several operands at once. Moreover, generated kernel objects can be kept so that they can be used again later.

We take advantage of this kernel generation framework to extend the support of ndimensional arrays to a broader range of NumPy functionalities. The HySoP array implementation include support for the following subset of NumPy functionalities: binary operations (elementwise bit operations), logic functions (truth value testing, array contents, logical operations and comparisons), mathematical functions (trigonometric functions, hyperbolic functions, rounding, sums and products, exponents and logarithms, other special functions, arithmetic operations, complex numbers functions and miscellaneous), random sampling (uniform distributions), statistics (order statistics, averages and variances). See https://docs.scipy.org/doc/numpy/reference/routines.html for corresponding numerical routines. The support of standard host-to-host sliced array memory transfer is extended to host-to-device, device-to-host and device-to-device memory transfers by performing multiple calls to clEnqueueCopyBufferRect so that the user never bothers to call any low-level OpenCL function. It is powerful enough to support and encourage the creation of custom application by its users. All of the implemented NumPy-like routines are tested against the reference NumPy ones for various shapes and data type associations such that the HySoP array interface closely mimic the NumPy one. This test is one of the many tests present in the HySoP test suite that is driven by the continuous integration software development practice [START_REF] Duvall | Continuous integration: improving software quality and reducing risk[END_REF][START_REF] Meyer | Continuous integration and its tools[END_REF]].

The need for such feature-complete implementations of numpy compatible array interfaces targeting accelerators has recently led to the release of two new Python modules, namely CuPy for CUDA-compatible arrays [START_REF] Nishino | CuPy: A NumPy-Compatible Library for NVIDIA GPU Calculations[END_REF] and ClPy for OpenCL-compatible arrays [START_REF] Higuchi | ClPy: A NumPy-Compatible Library Accelerated with OpenCL[END_REF]. Some other interesting C++ libraries such as Thrust, VexCL and ArrayFire also offers n-dimensional array support for various compute backends (although they do not support the NumPy interface). Thrust is a library that offers parallel algorithms on arrays based on the Standard Template Library (STL) on C++, CUDA, OpenMP and TBB compute backends [START_REF] Bell | Thrust: A productivity-oriented library for CUDA[END_REF]]. Multiple expressions involving arrays are automatically merged by using C++ expression templates [START_REF] Veldhuizen | Expression templates[END_REF]] and iterators. VexCL provide the same ideas for OpenCL [START_REF] Demidov | VexCL: Vector expression template library for OpenCL[END_REF]]. Unfortunately those two libraries do not offer any Python bindings. ArrayFire has array support for CPU, CUDA and OpenCL compute backends [START_REF] Malcolm | ArrayFire: a GPU acceleration platform[END_REF]. It recently got Python support through the arrayfire-python module [START_REF] Chrzeszczyk | Matrix Computations on the GPU with ArrayFire-Python and ArrayFire-C/C++[END_REF]]. Multiple expressions involving arrays are automatically merged by its own just-in-time engine. Those libraries provide productive high-performance computational software ecosystems for creators of scientific libraries. All the Python solutions being posterior to the beginning to this work, the HySoP module uses its own implementation of n-dimensional array functionalities backed up by PyOpenCL code generation capabilities.

OpenCL code generation framework

OpenCL 1.2 is a subset of the C language, and because target architectures are often not known is advance, OpenCL sources are usually compiled at runtime. Just-in-time (JIT) compilation is thus the natural way to obtain executable binaries (kernels) in OpenCL. For simple application, the source code can be loaded from external files or directly embedded into the application as a literal string constants. When the source code has to be modified to suit some of the user inputs, the JIT capability makes it possible to generate code at runtime instead. This is known as Run-Time Code Generation (RTCG) and is used in all of the array libraries presented in the previous section. Code can be generated by three different methods:

• Simple string formatting: Using raw Python builtin string formatting capabilities (PEP 3101) is already sufficient for a large range of applications. It can be used to inject specific values for given keywords such as the data type and loop ranges. This simple technique can compensate for the lack of generic programming capabilities such as C++ templates that are available in CUDA and OpenCL 2.x kernels. Note that it is also possible to use the C-preprocessor [START_REF] Stallman | The C preprocessor[END_REF] within OpenCL source code and to provide custom defines at compilation.

• Textual templating: When the generated code depends on some control flow while the code variants are textually related, it may be easier to use a templating engine that can do more than simple keyword substitution. As an example, the templating engine used by PyOpenCL to provide code generation capabilities such as ElementwiseKernel is Jinja2 [START_REF] Ronacher | Jinja2 (the Python template engine[END_REF]].

• Code generation from abstract syntax trees (AST): When the generated code variants are not textually related, it may become appropriate to introduce a full representation of the target code in the host language (here Python). Abstract syntax trees allow code to be generated by using all the facilities provided by the host language.

In Python, C-like code generation can be done with a variety modules such as CodePy, cgen and cfile but the most mature AST based code generator seems to be provided by Sympy (even if it does not offer direct OpenCL support). A list of scientific libraries using AST based code generation techniques can be found in [START_REF] Terrel | From equations to code: Automated scientific computing[END_REF]].

Runtime code generation has been used in the HySoP library since the first OpenCL implementation [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]]. The library was capable of dynamically generating efficient advection, remeshing, diffusion and transposition OpenCL kernels by using file-based static sources, the C-preprocessor to select the numerical methods and a custom source parser to enable the vectorization of expressions. This method would be classified as a simple string formatting technique. The main disadvantage with this approach is that each new numerical method requires sources and C macros to be modified, even if it is just a simple variant of an existing one. The source code is also obfuscated in the presence of many preprocessor macros.

In the present work, this code generation framework has been replaced by two complementary AST-based approaches:

1. An OpenCL code generation framework based on Python context managers (PEP 343) capable of generating any OpenCL kernel directly in Python. Context managers are used to handle code sections and code blocks such that the code generator does reflect the structure of the generated code [Filiba 2012]. Inside a block, each individual line of code is then generated by using simple Python string formatting capabilities. This process is assisted by many helper classes representing OpenCL variables, arrays, and functions to be called.

2. An OpenCL code generator based on the Sympy AST representation. This is particularly useful to automatically generate and vectorize symbolic expressions supplied by the user.

The code generating task is facilitated by the PyOpenCL library that already provides mappings between OpenCL vector types and NumPy types and a way to convert NumPy structured types to OpenCL ones (structured data has no reason to be identically laid out in memory between the host and device because of alignment concerns). The code generator offers helper routine to handle struct, union, enum and various type of variables such as PyOpenCL arrays that each can have given different offset, strides and data type. Functions are implemented as separate code generators such that they can be reused across different kernels (and other functions). The code generator uses by default eleven ordered code sections to generate valid OpenCL: the first block consists in C-preprocessor macros and is followed by enum definitions (enums cannot be forward declared) and union, struct, function and kernel prototypes. Those six header sections are followed by union and struct declarations, global scope constants and finally function and kernel implementations. The direct implication of runtime code generation is that most variables are now known at compile time. Within the previous code sample, this include the array shape N but also local and global work sizes (L and G which specify respectively the number of workitems that make up a work-group and the total number of global work-items that have to be spawned). The knowledge of those variables allows the compiler to optimize the generated code appropriately. To give an example, the same code is regenerated with work dim set to 2 and by using the known vars capabilities of the generator and providing the value of N, G and L: The generated kernel signature now exposes one less argument (that has become a global scope constant) and enforces an explicit work group size that is set to (512, 1, 1). This simple kernel already raise the problem of choosing appropriate values for the work-group size and the global work size for given device and array shape. In general, the values of L and G are obtained by a simple heuristic. For a given device, better values may be obtained by performing kernel microbenchmarking.

Sympy offers a way to generate compilable code directly from Sympy expressions. (see http://docs.sympy.org/latest/modules/utilities/codegen.html). In particular its C code generator can be extended by implementing code printers to handle symbolic representations of additional OpenCL constructs. Each Sympy symbolic expression is represented by an underlying AST where each node is either a subexpression or a concrete symbol. With such a representation it is possible to determine the OpenCL type (int, float8, custom complex type, ...) of each node by taking into account the implicit casting rules of the language. It also makes it possible to inject additional explicit cast and conversion expressions to handle the non-implicit cases of promotion or demotion of builtin vector types (see [Khronos2011], section 6.2). The idea behind this approach is to propose to the users (and implementers) a way to generate efficient OpenCL based operators without having to use the tedious class-based code generator to generate custom elementwise kernels.

Code generation from symbolic expressions use the class-based code generator under the hood to generate the final code. It is mainly capable of generating elementwise operations, the iteration shape being determined by field of array symbols present in the expressions. The generated code targets contiguous cached memory accesses and the optimal kernel configuration and runtime parameters are automatically determined by kernel microbenchmarking (see subsection 3.2.3). The resulting symbolic DSL also incorporate features aimed at generating kernels that are able to integrate PDEs by using explicit finite differences with explicit Runge-Kutta time integrators. The basic symbolic codegen interface allows to mimic the capabilities provided by PyOpenCL elementwise kernels but with additional automatic tuning of kernel runtime parameters. Symbolic operator interfaces are also provided such that once a kernel has been generated, it is automatically wrapped into an operator and its topology requirements are automatically set up depending on determined numerical methods. Finally the process of splitting expressions according to directional derivatives (as presented in section 2.5) is automated to generate directional operators from general symbolic expressions.

To sum up, given four arrays (a, b, c, d) ...

The last method offers more flexibility but requires the user to master the OpenCL standard.

The HySoP symbolic code generator offers good performances when compared to state of the art array libraries. The performance is first compared for OpenCL CPU platforms where HySoP outperforms all considered array libraries. The Xeon E5-2695v4 achieves 57.99 GB/s (42.5% of peak memory bandwidth) while the Core i7-9700K achieves 14.16 GB/s (66.5% of peak memory bandwidth) for double-precision arrays of 512 3 = 2 27 elements (1GiB / array). All considered OpenCL and CUDA GPU platforms perform roughly the same under enough workload. CUDA based solutions (PyCuda and cupy) exhibit 45% faster runtimes than their OpenCL counterparts (PyOpenCl and HySoP) for the consumer grade GPU (GeForce RTX 2080Ti). The generated code being roughly the same between PyOpenCl and PyCuda, this could be explained by differing kernel compilation flags between the two platforms [START_REF] Fang | A comprehensive performance comparison of CUDA and OpenCL[END_REF]]. The Tesla V100-SXM2 achieves 774.4 GB/s (86.0% of peak bandwidth) while the GeForce RTX 2080Ti achieves 263.4 GB/s (42.8% of peak bandwidth vs. 62.1% for CUDA).

Permutations and automatic tuning of kernel runtime parameters

Performance auto-tuning is a well established technique which has been applied in a variety of high performance libraries targeting CPUs with multi-level caches, such as ATLAS for dense linear algebra [START_REF] Whaley | Automatically tuned linear algebra software[END_REF]], OSKI for sparse linear algebra [START_REF] Vuduc | OSKI: A library of automatically tuned sparse matrix kernels[END_REF] Datta2009 for stencil based computations [START_REF] Datta | Auto-tuning stencil codes for cache-based multicore platforms[END_REF], and FFTW for fast Fourier transforms [Frigo et al. 2012]. It has been naturally suggested as a solution to the OpenCL performance portability problem [START_REF] Fang | A comprehensive performance comparison of CUDA and OpenCL[END_REF]]. Determining the best set of optimizations that have to be applied to a kernel to be executed on a given OpenCL device is a challenging problem [START_REF] Dolbeau | One OpenCL to rule them all?[END_REF]]. For instance, some kernel optimizations may result in excellent performance on GPUs, but reduce performance on CPUs. In the case the target device is not known in advance, the choice of the best kernel execution parameters has to rely on some runtime kernel microbenchmarking strategy. Such OpenCL kernel autotuning strategy has been successfully applied to the same application specific problems as on the CPU: stencil based computations [START_REF] Datta | Stencil computation optimization and auto-tuning on state-of-the-art multicore architectures[END_REF], fast Fourier transforms (MPFFT [START_REF] Li | MPFFT: An auto-tuning FFT library for OpenCL GPUs[END_REF]), sparse linear algebra (clSpMV [Su et al. 2012a]), and dense linear algebra (clMagma [START_REF] Cao | clMAGMA: High performance dense linear algebra with OpenCL[END_REF]).

Numerical simulations performed with HySoP with the OpenCL backend generate tens to hundreds of different OpenCL kernels, each of them requiring a specific configuration to achieve the best performance. Moving an application to a different device often requires a new optimization configuration for each kernel [START_REF] Grauer-Gray | Auto-tuning a high-level language targeted to GPU codes[END_REF]]. The aim of automatic kernel parameter tuning is to guaranty some performance portability by relying solely on empirical data, instead of any performance model or a priori knowledge of the target compute device [START_REF] Du | From CUDA to OpenCL: Towards a performance-portable solution for multi-platform GPU programming[END_REF]]. In section 1.4.5 we already identified at least three parameters to optimize in order to get closer to device peak memory bandwidth: local and global grid size and vectorization. The global and local grid sizes can easily be tuned at runtime without performing any modification to the code and constitute parameters of choice for quick kernel autotuning [START_REF] Spafford | Maestro: data orchestration and tuning for OpenCL devices[END_REF]]. In practice, other kernel optimization parameters are mostly application dependent [START_REF] Zhang | Improving performance portability in OpenCL programs[END_REF]].

In the present work, all kernels are code generated and autotuned during problem setup and the same kernels are then called a certain number of times at every timestep. A unique kernel may be called thousands of time during a single simulation. It is thus computationally efficient to spend some time in the exploration of the kernel parameter space. Many different methods exist to explore the parameter space, from simple brute-force approaches to autotuners based on efficient discrete search algorithms [Werkhoven 2019] or even on machine learning [START_REF] Falch | Machine learning based auto-tuning for enhanced opencl performance portability[END_REF]]. Those two last approaches are more efficient since they only explore a subset of the entire tuning parameter configuration space. Here we choose simplicity and opt for a classical brute forcing algorithm as provided by [START_REF] Nugteren | CLTune: A generic auto-tuner for OpenCL kernels[END_REF] (see figure 3.11). The autotuner is implemented directly into the HySoP library and provides a planning interface similar to FFTW to reduce the time passed during the autotuning step.

Kernel code generator

It features four levels of parameter sets (enabled with the planning rigor flags ESTIMATE, MEASURE, PATIENT and EXHAUSTIVE) depending on how much time the user is willing to pay during the planning phase.

One kernel is code generated per kernel parameters sets (including hard-coded runtime parameters such as local and global grid size) and is benchmarked by averaging statistics over a given amount of successive runs. Kernels that underperform the actual best candidate over some threshold value (usually 120%) are automatically pruned out from the optimization process. The procedure then consists into keeping half of the best candidates and doubling the number of successive runs to refine bench statistics until there is only one kernel left.

Once the autotuner has found an optimal set of parameters, the results are cached on disk for the actual device configuration, allowing faster subsequent runs. The case of the local permutation kernels is interesting in this context because of the large space of kernel generation parameters.

Transposition or permutations kernels are generated by using the following set of classical optimizations:

• Only coalesced memory transactions from and to device global memory.

• Use of a square window of local memory (called a tile) to handle transposition between the axis contiguous in memory (the last one) and its corresponding permutation. The array is permutated tile by tile out of place (inplace permutation can also be done easily for 2D arrays but requires two tiles in local memory).

• Minimize bank conflicts on local memory transposition by offsetting each line by one element. Shared memory usually is divided into equally-sized memory modules, called banks. They are organized such that successive words are assigned to successive banks. These banks can be accessed simultaneously, and to achieve maximum bandwidth workitems should access local memory associated with different banks

• Use of diagonal work-group reordering to minimize partition camping. As with local memory, global memory is often divided into multiple partitions that have all to be accessed simultaneously by work-items of a given work-group to achieve global maximal bandwidth. Given a work-group index, the position of tile is found by using diagonal array coordinates.

Those optimizations, described in [START_REF] Ruetsch | Optimizing matrix transpose in CUDA[END_REF]] and [START_REF] Baqais | Bank Conflict-Free Access for CUDA-Based Matrix Transpose Algorithm on GPUs[END_REF], can be generalized to permutations in any dimensions. The kernel autotuner tries to find optimal parameters such as the tile size, global and local sizes and has to enable or disable the use of diagonal coordinates and tile offsetting. The maximum tile size is determined by the device available local memory per work-group: 32kiB for the two CPUs and 48kiB for the two GPUs.

To transpose 4B elements (float or int), this leads to maximum tiles size of 90 and 110 (90 × 90 and 110 × 110 elements). For 2D inplace transpositions, two tiles are required and the maximal tile size drops to 64 and 78 respectively. In practice the autotuner only explore tile sizes that are powers of 2 (and of the form 2 i 3 j in EXHAUSTIVE mode).

The HySoP library provides options to configure the kernel autotuner and to dump resulting kernel execution statistics. As an example, the following statistics represent all the different configurations generated for the transposition of a single-precision floating point array of shape (16384, 16384): The kernel autotuner generate hundreds of different kernels which mean runtimes are measured initially over 8 runs. The resulting distribution of relative runtimes (with respect to the mean mean-runtime across all benched kernels) is represented in an histogram with log scale. Sky blue overlay correspond to kernels that have been pruned out during the procedure. Green, orange and red vertical lines represent, respectively, best, median and worst runtime.

Table 3.2 -Kernel autotuning results for 2D and 3D OpenCL permutation kernels:

Best configuration determined by the kernel autotuner in PATIENT mode for two and threedimensional single-precision floating point arrays of given shape. Mean runtime value is obtained by at least 128 runs of the best resulting kernel. Axes corresponds to the state of axes after permutation starting from unpermutated configuration (1,2) and (1,2,3) where the first axis (resp. the last axis) has the greatest (resp. smallest) stride in memory. DC corresponds to diagonal coordinates, TO to tile offsetting, TS to tile size, BDW to achieved global memory bandwidth and G and L to global and local grid sizes. Achieved global memory bandwidth is also given in percentage of theoretical peak memory bandwidth of the target device.

Apart from the dual CPU configuration, every possible permutations for two and threedimensional arrays achieve more than 50% of the device theoretical peak global memory bandwidth. On the two GPUs , some permutations even achieve more than 85% of the maximal theoretical value. Diagonal coordinates is an optimization that could be disabled for every benched platforms whereas tile offsetting seems to help in half the cases. Those performances are in agreement with dedicated multidimensional array permutation libraries targeting CUDA such as TTC (Tensor Transposition Compiler [START_REF] Springel | TTC: A tensor transposition compiler for multiple architectures[END_REF], cuTT (CUDATensor Transpose library [START_REF] Hynninen | cutt: A high-performance tensor transpose library for cuda compatible gpus[END_REF]] and TTLG (Tensor Transposition Library for GPUs [START_REF] Vedurada | TTLG-an efficient tensor transposition library for gpus[END_REF]). To the author's knowledge there exist no such dedicated dense tensor permutation library for OpenCL. Performance on CPUs is also comparable to HPTT (High Performance Tensor Transposition, [START_REF] Springer | HPTT: a high-performance tensor transposition C++ library[END_REF]).

The Intel OpenCL platform seems unable to surpass half the bandwidth in a multi-socket configuration (dual Xeon E5-2695v4 with four memory channels per CPU). This behaviour was already seen on clpeak and elementwise benchmarks, and could be the result of the memory policy that is in place for the allocation of the CPU OpenCL buffer on the underlying NUMA architecture [START_REF] Lameter | NUMA (Non-Uniform Memory Access): An Overview[END_REF]]. We recall that a NUMA node is defined as an area where all memory has the same speed as seen from a particular CPU (a node can contain multiple CPUs). We study the effect of the memory policy on a quad socket Intel Xeon E7-8860 v4 platform with a theoretical peak global memory bandwidth of 51.2 GB/s per node (one node containing one CPU). The kernel considered here is the 2D transposition of single-precision floating point arrays of size 16384 2 . To enforce memory policies we use the NUMA library (through its Python binding of the same name) and the numactl utility [START_REF] Kleen | A numa api for linux[END_REF]].

A first batch of statistics is collected with the numactl utility:

• Only one CPU with local allocation (numactl --cpunodebind=0 --membind=0) yields 80.7ms mean execution time, or 26.6GB/s (52.0% of the peak memory bandwidth of one memory node).

• All CPUs with allocation on NUMA node 0 (numactl --cpunodebind=0,1,2,3 --membind=0) yields 83.5ms mean execution time, or 25.7GB/s (50.0% of the peak memory bandwidth of one memory node).

• Only one CPU with local allocation interleaved on all nodes (numactl --cpunodebind=0 --interleave=0,1,2,3) yields 62.3ms mean execution time, or 34.5GB/s (16.8% of the peak memory bandwidth of all memory nodes).

• All CPUs with local allocation (numactl --cpunodebind=0,1,2,3 --localalloc) yields 28.8ms mean execution time, or 74.5B/s (36% of the peak memory bandwidth of all memory nodes).

A second batch of statistics is gathered directly in OpenCL by using device fission capabilities offered by the OpenCL 1.2 standard [START_REF] Sych | OpenCL Device Fission for CPU Performance[END_REF]] and Intel specific extension cl intel device partition by names [Yariv et al. 2013]. We create four non-overlapping sub-devices containing specific compute units based on libnuma.node to cpu, splitting the OpenCL device into the four subdevices corresponding to the four physical CPUs. The best kernel obtained with the command numactl --cpunodebind=0 --membind=0 is recovered and compiled for every of those subdevices. One array buffer is allocated on each NUMA node and each device is benched on all those four buffers by using a subdevice-specific queue. The joint use of the four subdevices results in mean runtimes of 27.4ms which is similar with the 28.8ms result obtained with numactl --cpunodebind=0,1,2,3 --localalloc. Enforcing memory policies with numactl or by manual device fission does not seem to help much in this context, the maximal obtained mean relative bandwidth being of the order of 37% (and 50% for a single socket). As a point of comparison, the STREAM memory bandwidth benchmark [START_REF] Bergstrom | Measuring NUMA effects with the STREAM benchmark[END_REF]] achieves up to 54% of peak memory bandwidth for its best run.

Directional advection operator

In this subsection we study the performance of the directional advection operator. As seen in chapter 2, the transport of a scalar θ in a velocity field u is decomposed into n (or 2n) substeps with directional splitting, where n is the dimension of the problem. Each substep is further decomposed as transport, remeshing and local permutation of data such that the next advected axis becomes contiguous in memory. At a first sight, this results into enqueuing 3n kernels for a first order or 6n kernels for second order Strang splitting, but we also have to take into account ghost exchanges and ghost accumulation (even for serial processes).

Given a timestep dt that fulfills a LCFL condition (2.23), a user chosen CFL condition (2.22) and a direction i, directional advection consists into the following steps:

1. Interpolate velocity to the scalar grid by using an interpolation scheme (only if the two grids have different resolutions).

2. Create particles aligned with the underlying scalar field grid and perform particle advection at fixed velocity u i (x) by using an explicit Runge-Kutta time integrator 2.3.2.

Particles may be transported up to C ≥ dt u i ∞ /dx i cells outside of the domain (where C is obtained from the arbitrary CFL condition). The required ghosts for the velocity and input scalar is thus 1 + C .

3. Remesh all the particles with algorithm 5 by using a remeshing kernel Λ p,r that has support [-s, s] with s = 1 + p/2. This requires 1 + C + s ghosts to remesh the output scalar field.

4. Accumulate ghost remeshed quantities to the left and to the right processes (with respect to the current direction). For a serial process with periodic boundary conditions in the current direction, this will accumulate left ghost values to rightmost compute grid values.

5. Update ghosts by performing a full ghost exchange.

6. Perform a local permutation such that the next axis becomes contiguous in memory. In 2D, this corresponds to a transposition but in 3D multiple different permutations are available.

Within a multi-grid context where the scalar is discretized on a finer grid, the number of scalar ghosts is multiplied by the grid ratio in the considered direction. The user chosen CFL number should be such that ghost exchange has a reasonable cost (see section 3.3.2).

The OpenCL implementation in mainly based on the one of [START_REF] Etancelin | Model coupling and hybrid computing for multi-scale CFD[END_REF]] and has been fit into the code generation and auto-tuning framework. Each line that is contiguous in memory represent and independent one-dimensional transport problem that is treated iteratively by one work-group (by iterating on sublines which length is a multiple of the work-group size).

During the advection step, the velocity is only read once and is cached in local memory to allow for efficient linear interpolations. During the remeshing step, the particles are remeshed to local memory before being written back to global memory. In the case local memory do not improve performance, this caching behaviour can be disabled by the autotuner.

Advection kernel

The arithmetic intensity of the advection kernel only depends on the Runge-Kutta scheme (the Butcher tableau), and the chosen interpolation scheme. In practice we only use linear interpolation during the advection step (a more elaborate interpolation method may be used to interpolate velocity to the fine grid as a pre-processing step). For fields discretized on N v points and a Runge-Kutta scheme of s steps, the advection step performs s evaluations of the velocity per particle along with 2s + k -1 additions and 2s + k multiplications per particles where s -1 ≤ k ≤ s(s + 1)/2 represent the number of non-zero coefficients a ij contained in the Butcher tableau (see section 2.3.2). Evaluation of velocity has no additional cost on the first stage (particles are grid aligned) and for all subsequent stages it cost approximately 5 FLOP / particle, see equation ( 2.27)). The computation of the initial particle position requires one multiplication per particle. The total number of particles is

N = n i=1
N v i and if we consider all operations we obtain a total of (4s + 2k + 5s)N operations. The total number of memory transaction in terms of elements is N reads of velocity and N writes of resulting particle position (2N memory transactions). If we denote S the element size in bytes this gives a total arithmetic intensity of (9s + 2k)/(2S) FLOP/B. The following table shows the usual cases of single-and double-precision Euler, RK2, RK3, RK4 and RK4 38 Runge-Kutta methods:

s k FLOP/E FLOP/B (SP) FLOP/B (DP) Euler 1 0 4.5 1.1 0.6 RK2 2

Table 3.4 -Approximative arithmetic intensity of the advection kernels

In practice, the arithmetic intensity also depends on vectorization and the size of input at constant CFL number. The total memory that has to be read include velocity ghosts and is thus larger than N . The vectorization also plays a role because some particle independent scalar quantities can be computed once for multiple particles at a time.

Because computing arithmetic intensity by hand is a tricky task, even for simple algorithms, we rely on an automatic tool to collect kernel execution statistics from code generated kernels.

This tool is based on Oclgrind, a virtual OpenCL device simulator which goal is to provide a platform for creating tools to help OpenCL development [START_REF] Price | Oclgrind: An extensible OpenCL device simulator[END_REF]]. The simulator is built on an interpreter for the LLVM IR (LLVM intermediate representation [START_REF] Lattner | LLVM: A compilation framework for lifelong program analysis & transformation[END_REF]) and provide a lightweight method to run kernels in isolation from any OpenCL host code. This interface is provided via the oclgrind-kernel command, which takes as input a simulator file (*.sim) describing how to run the kernel (location of the OpenCL source code, configuration of kernel arguments, and global and local sizes). Perhaps the most interesting feature of Oclgrind is that it provides a simple plugin interface that allows third-party developers to extend its functionality. This interface allows a plugin to be notified when various events occur within the simulated kernel, such as an instruction being executed or memory being accessed. Plugins are implemented in C++ by extending the virtual oclgrind::Plugin class and have to be compiled as external dynamically loadable libraries.

As most of HySoP kernels are code generated such kernel simulation files can easily be generated by the autotuner that has the full knowledge of the source code and associated kernel arguments. By adding a simple callback support, the autotuner can run arbitrary user scripts with the knowledge of the source location, simulation file and runtime statistics. Such a script can run oclgrind-kernel with a custom plugin and analyze its outputs. The present work use this approach with a custom plugin that computes instructions statistics occurring during the simulation of some target kernel. The plugin collects data about integer and floating point operations as well as memory transactions (private/local/constant/global) and outputs them to disk for further analysis. An example of statistics obtained with this plugin is given appendix (C). The user script can then proceed to compute the effective floating point operations per byte FLOP/B and integer operations per byte IOP/B by using global memory data (all target kernels only load and store a given piece of global data once). The mean runtime being provided by the autotuner, it is then possible to compute mean achieved bandwidth and mean achieved compute statistics.

Implementation on hardware accelerators 167

We use this method to collect statistics of directional advection on the Tesla V100-SXM2 GPU for various Runge-Kutta schemes and grid resolutions. Full results, including execution times are available in tables C.1 and C.2 and corresponding rooflines are plotted here: Figure 3.14 -Roofline of 2D and 3D, single-and double-precision advection From figure 3.14 it is clear that advection is a memory-bound problem on the Tesla GPU and that arithmetic intensity increases with integration order in time. Achieved global memory bandwidth increases with the size of the array. This is expected because one work-group is scheduled per line (particles are all independent) and device occupancy increases when more work-groups are scheduled at a time. The achieved bandwidth also depends on the size of the work-groups (number of work items) and associated workload. Although the smallest 3D problem spawns twice the number of work-groups when compared to the smallest 2D problem (4096 vs. 2048), the 2D problem achieves much higher bandwidth simply because the smallest 2D problem is 16 times bigger than the smallest 3D problem (2048 2 vs. 64 3 ). Under enough workload the directional advection kernels achieve at least 70% of global peak theoretical memory bandwidth for single-precision and 80% for double-precision, the maximal achieved bandwidth being of 86.9%. Thanks to the massive memory bandwidth provided by the second generation of High Bandwidth Memory (HBM2) stacked onto the device, it takes only 1.62 milliseconds to transport 512 3 ( 134.2 millions) particles by using single-precision. Put the other way around, this represent one particle every 12 picoseconds or 83 giga-particles per second. Once all particles have been spawned and advected, the remeshing kernel is responsible to remesh those free particles onto the grid.

Remeshing kernel

The main problem with remeshing is that consecutive particles can be accumulated at the same time to the same local memory locations by neighboring work-items. The historical OpenCL implementation of the library is based on the assumption that because of the regularity of the velocity (imposed by the LCFL), the maximal number of particles per cell is kept small after the advection step. Under this hypothesis, local memory data races can be avoided by enforcing a given work-item to treat a certain amount of consecutive neighboring particles (usually 2, 4, 8 or 16 to fit OpenCL vector builtins). In practice this implementation has shown good results on many different test cases but it tends to reduce device occupancy (reduced work-group sizes). A new implementation of the remeshing kernel has been introduced by using atomic additions in local memory. Unfortunately most devices do not provide 64bit atomic intrinsics so this second implementation may only be available for single-precision floating point numbers on certain devices such as NVidia GPUs. Atomic operations on floating point values are usually not readily available and are thus implemented with the usual union + atomic cmpxchg workaround [START_REF] Hamuraru | Atomic operations for floats in OpenCL[END_REF]]. Apart from these parameters, the autotuner has to find the optimal vectorization and workload per work-item. As before, results are given for two-and three-dimensional, single-and double-precision arrays in tables C.3 and C.4 and the corresponding rooflines are depicted here:

Implementation on hardware accelerators 169

The remeshing formula Λ p,r is a piecewise polynomial of a certain order that is evaluated with Horner's method. High order remeshing formulas that preserve multiple moments with decent overall regularity tend to be compute-bound but multiple scalars can be advected at a time to save on expensive calls to the remeshing formula. Because additional scalars decrease arithmetic intensity, we only consider the best case scenario (in terms memory bandwidth) of a single advected scalar. As it can be seen on figure 3.15, for a single scalar the Tesla V100 GPU and the Λ 8,4 remeshing formula, we obtain an algorithm that is at the same time compute and memory bound (the sweet spot of designed arithmetic intensity). Out of the 32 single-precision problem configurations, the kernel autotuner has chosen the implementation based on atomic operations 30 times. For all those cases the preferred vectorization parameter alternates between 1 and 2 and under high load the achieved relative global memory bandwidth lies between 83.1% and 90.0%. As there is no double-precision atomic support on this device, all the double-precision results arrise from the version without atomic intrinsics (where the preferred vectorization becomes 2). For those cases, the achieved relative bandwidth is between 77.0% and 86.6%. The Λ 8,4 remeshing kernel exceeds 10 TFLOPS of compute for the single-precision (10779 GFLOPS or 72.3% of the device capabilities) and about half of it for the double-precision version. For all cases the achieved performance is roughly 16 picoseconds per remeshed particles or 64 giga-particles per second (single-precision).

Overall n-dimensional transport performance

The n-dimensional transport of a scalar is obtained by chaining n advection and remeshing operators interleaved with n permutations of the scalar. The graph builder automatically determines that each of the velocity components have to be in permutated state so that velocity data is permutated only once at problem initialization (in fact the velocity components are just initialized in permutated state). This gives a total of 3n operators for first order Strang splitting and this number doubles for second order Strang splitting. As only two-dimensional square arrays support inplace transpositions, inplace two-dimensional rectangle and threedimensional permutations are performed out-of-place to a temporary buffer, followed by a device-to-device copy to the source array. This decreases the permutation performance by a factor of two (transposition is memory bound). Those 3n operators map to 3n OpenCL kernels plus n device-to-device copies (except for 2D square discretizations), plus additional copy and accumulation of hyperplane layers required for ghost exchange and ghost accumulation. As some OpenCL platforms do not correctly support device-to-device rectangular copies from and to non-overlapping subregions of the same buffer, ghost exchange and ghost accumulation are done inefficiently by using temporary preallocated device buffers and by performing twice the required number of copies. A ghost exchange costs 4 copies per direction and a ghost accumulation 6 copies and two accumulation kernels. Each directional advection operation require a directional ghost accumulation and a full ghost exchange. For advection, this brings the total number of ghost related copy kernels to d(4d + 6). Besides paying extra kernel launch latency, the device-to-device copies of discontinuous chunks of data perform poorly. The solution to this problem would be to implement custom ghost exchange and ghost accumulation kernels. This would however not improve the distributed implementation of the library because device-to-device copies of ghosts layers have to be replaced with device-to-host and host-to-device copies from and to host buffers (to perform MPI exchanges). Figure 3.16 show single-precision kernel execution statistics gathered on 100 iterations of ndimensional advection on the NVidia GeForce RTX 2080Ti GPU for various configurations of time integrators and remeshing kernels:

Figure 3.16 -Distribution of runtime for n-dimensional single-precision transport

Mean runtime and runtime ratio is given for different kind of kernels. Each bar corresponds to a specific configuration (Runge-Kutta scheme and remeshing formula). D2D copy corresponds to copies required for permutations and ghosts to ghost exchange and accumulation. The number between parenthesis, when given, corresponds to the number of calls per iteration.

From figure 3.16 it is clear that the whole advection problem is memory bound on the consumer-grade GPU, as it was the case for the server-grade GPU in the previous roofline analyses. In 2D, every numerical method results into the same runtime. In 3D the situation is the same but the runtime is increased by roughly 5% due to increased number of ghosts required by higher order remeshing formula Λ p,r (the support of the remeshing kernel increases with the number of even preserved moments p).

Due to the directional splitting approach and differences in the number of copies between the 2D and 3D cases the overall achieved performance for transport depends on the number of dimensions. On this device, a particle can be transported every 160 picoseconds on a 2D domain (6.3 giga-particles per second) and every 410 picoseconds in 3D (2.44 giga-particles per second) regardless of the order of the methods. Permutations constitute most of the runtime (45 to 48%) and further optimization of the transposition kernels could lead up to 30% improvement in total runtime. We saw that different permutations perform differently and at this time the operator graph builder does not perform microbenchmarking to determine the best permutation axes. Another improvement track would be to use one of the CUDA transposition library and to convert the generated CUDA code to OpenCL (or more elegantly to implement OpenCL code generation in one of those frameworks). Proper CUDA support also constitute a long-term goal of the HySoP library.

The same benchmark run on the dual CPU device reveals that transport is compute-bound mainly by the remeshing step (figure 3.17). On this device the advection step takes longer with increasing time integration order (Runge-Kutta schemes). Remeshing also have bigger runtimes for remeshing formulas with increasing number of preserved moments and take up to 59% of the runtime for the Λ 8,4 formula. Achieved performance range from 1099 to 2127 picoseconds per particle (0.5 to 0.9 giga-particles per second). Two-dimensional single-precision performance is roughly ten times worse than the consumer GPU for approximately a factor four in price tag. For double-precision computations this performance gap reduces (transport is compute-bound in double-precision on consumer GPUs due to architecture design).

Figure 3.17 -Distribution of runtime for single-precision transport on the CPU Mean runtime and runtime ratio is given for different kind of kernels. Each bar corresponds to a configuration consisting in a specific Runge-Kutta scheme that affects advection and a specific remeshing formula Λ p,r that is used during the remeshing step. D2D copy corresponds to copies required for transpositions and ghosts to ghost exchange and accumulation. The number between parenthesis, when given, corresponds to the number of calls per iteration.

Finite differences operators

Finite difference operators such as diffusion and stretching are all implemented by using the symbolic code generation framework. Given a symbolic expression containing n-dimensional scalar fields and spatial derivatives of scalar fields (but no mixed spatial derivatives) the expression is automatically split into n expressions corresponding to what we did manually in section 2.5. This is done by simple depth-first Sympy expression AST traversal. Once the directional symbolic expressions are found, spatial symbolic variables are permutated so that all memory accesses become coalesced. As for advection and remeshing, it will be the graph builder's responsibility to inject permutations operators prior to each directional operator. The symbolic code generator then generates kernels corresponding to those expressions by mapping the symbolic representation to OpenCL. Each expression requires multiple analysis and transformation steps until the final code is generated.

Because expressions are directional, we always fall in the case of independent 1D problems. As before, a work-group is responsible to handle one contiguous line by iterating by sublines. All global memory loads and stores happen only once, and data is cached in local memory in order not to be memory-bound. Symbolic expressions are vectorized at code generation depending on a vectorization parameter. As for all other code generated kernels, the autotuner tries different combinations of parameters to find the best kernel candidate. The whole process is depicted on figure 3.18. The customization of the numerical method is done by passing method arguments such as the type of time integrator or the order of centered finite differences that have to be used to approximate the derivatives. The number of required ghosts per scalar field and per direction is extracted from the symbolic expressions and the knowledge of the methods. Because multiple symbolic expressions can be supplied to integrate multiple scalar fields at once with right hand sides that depend on all those fields and their spatial derivatives (like in equation 2.44) this analysis is not straightforward.

Directional symbolic operator generator

user supplied symbolic expression(s)

Figure 3.18 -Overview of the directional symbolic code generation process

At this time, symbolic code generation only work for input and output scalar fields discretized on the same Cartesian grid (but with varying number of ghosts). This means that the symbolic code generator do not handle on the fly interpolation of fields defined on coarser grids. Some frequently used operators such as finite differences based diffusion and stretching are directly provided by the library. Withing this framework, a user can easily generate custom OpenCL finite difference operators. As an example diffusion and conservative stretching can be generated by building the following symbolic expressions: This will print the following symbolic expressions: and To save on memory bandwidth, stretching and diffusion kernels can be merged together by just summing the two right hand sides together. Note that here all directional expressions will be equivalent and generate exactly the same kernels. Autotuning will only happen once.

import
Kernels statistics are collected for 3D single-and double-precision directional operators on 512 3 fields on the Nvidia Tesla V100-SXM2 GPU. Diffusion and stretching results are available in table C.5 and C.6. The corresponding rooflines for the stretching operator are depicted here:

Figure 3.19 -Rooflines of single-and double-precision directional stretching

Those kernels are memory-bound and achieved relative global memory bandwidth lies between 26 and 84% depending on the number of stages of the Runge-Kutta time integrator and finite difference order (FDCx). The theoretical local memory bandwidth can be computed from [START_REF] Jia | Dissecting the nvidia volta gpu architecture via microbenchmarking[END_REF]] for the Volta GPU architecture which leads to 13.2TB/s at device base clock (1290MHz). Achieved local memory bandwidth goes as high as 9.0TB/s for the RK4-FDC8 configuration (68% of device capabilities). As all memory accesses are coalesced and without bank conflicts, it is unclear why higher local memory bandwidth cannot be achieved, and the lack of proper OpenCL profiling tools for NVidia GPUs does not help. Without further analysis it is assumed that those performance results are linked to increasing pressure on local memory coupled to local memory barriers (one barrier per time integrator stage) and possible memory alignment issues. A local memory barrier is a work-group level construct that will flush any variables to local memory. It is required between integrator stages so that local memory transactions are visible to all work-items in given work-group. The alternative would be to call the kernel s times where s is the number of stages. This would decrease the overall performance of the operator because of additional global memory transactions.

Spectral transforms

The last kind of operators that we need to be able to solve an incompressible Navier-Stokes problem are spectral operators. This include the recovery of velocity from vorticity (Poisson problem with computation of curl, see algorithm 9) and spectral diffusion (algorithm 7). As we saw in section 2.6.4, the boundary conditions directly determine what kind of solver to use, the simplest configuration being a fully periodic domain, in which case only forward and backward discrete Fourier transforms are required. All kind of homogeneous boundary conditions can be solved by using real-to-real transforms (DCT and DST variants). General boundary conditions are solved spectrally by using the most suitable representation in Chebyshev polynomials (with Tau or Galerkin methods). In this subsection we do not consider the distributed case.

The general idea is to build a n-dimensional transform by tensor product of one-dimensional transforms, interleaved with data permutations for exactly the same memory coalescing reasons that justified directional operator splitting. Additionally we impose a forward transform ordering for efficiency: first real-to-real transforms (homogeneous BCs), followed by a real-tocomplex transform (first periodic BC), followed by complex-to-complex transforms (additional periodic BCs) and finally fast Chebyshev transforms (general BCs). This transform ordering imposes the ordering of corresponding boundary conditions and thus the physical ordering of the problem to be solved. Backward transforms just follow the reversed order with corresponding inverse transforms. Spectral operators always perform the following steps: first forward transform input scalar fields from physical to spectral space, compute output scalar fields in spectral space and finally transform them back to physical space.

Let N v be the size of the grid discretization and N = n i=1 N v i be the total number of grid points. We saw that all forward and backward transforms can be implemented in

O (N log N ) = O     n i=0     n j=1 j =i N v j     N v i log N v i    
complexity with some variant of the Fast Fourier Transform. This is usually the most costly step in spectral operators because the post-processing step can be done in only O (N ) complexity. The presence of general boundary conditions can drastically increase the post-processing step up to O N 3 due to the necessity of solving a general linear system of size N × N . However in practice there is rarely more than one axis that has general boundary conditions and in this case the complexity drops to

O N (N v n ) 2 = O     n-1 j=1 N v j   (N v n ) 3  
because there would be N/N v n independent linear systems to solve.

In practice many FFT libraries exist targeting classical CPUs such as FFTW, FFTPACK and MKL-FFT but also accelerators such as the OpenCL library clFFT and the CUDA li-brary cuFFT. Those libraries are accessible in Python through pyFFTW (FFTW wrapper), NumPy (C FFTPACK wrapper), SciPy (Fortran FFTPACK wrapper), mkl fft (MKL-FFT wrapper), gpyfft (clFFT wrapper) and scikit-cuda (cuFFT wrapper). All those libraries and wrappers but the CUDA based ones are usable in the HySoP library and are described in appendix B. The performance varies greatly from one FFT backend to another and we use gearshifft, a benchmark suite for heterogeneous implementations of FFTs [START_REF] Staniforth | Semi-Lagrangian integration schemes for atmospheric models-A review[END_REF], to gather runtimes for reference 2D DFTs of increasing size (see figure 3.20). Without any surprises, vendor specific libraries are always faster and open-source alternatives such as FFTW and clFFT lie behind with a two-to ten-fold performance hit.

Those libraries come with various levels of support for FFT related transforms. This includes available type of transforms (real-to-real, real-to-complex, complex-to-real, complexto-complex), their sizes (usually a product of powers of 2, 3, 5, 7 and so on), their type (single or double-precision transforms) and whether or not they offer multithreading or multiprocessing implementations. The main problem for GPU compatible libraries (clFFT and cuFFT) is that none of them provide the real-to-real sine and cosine transforms required to implement solvers with homogeneous boundary conditions. Fortunately it is possible to implement DCT-I, II and III as well as DST-I, II and III by using classical DFTs with O (N ) pre-or post-processing steps. The general idea is to transform input or output signals by reordering and multiplication by some twiddle factors. The method may also depend on the signal size [START_REF] Wang | On computing the discrete Fourier and cosine transforms[END_REF][START_REF] Chan | Direct methods for computing discrete sinusoidal transforms[END_REF][START_REF] Chan | Fast algorithms for computing the discrete cosine transform[END_REF]. With those methods DCT-II, DCT-III, DST-II and DST-III can be computed efficiently in O (N log N ) complexity and O (ε log N ) root mean square error where N is the size of the transform and ε the machine floating-point relative precision. For the DCT-I and the DST-I transforms there exist O (N log N ) algorithms with O ε √ N RMS error. We use the same approach as FFTW and we use O (2N log 2N ) implementations with O (ε log N ) error to get comparable accuracy between all implementations [Frigo et al. 2005]. We use the pre-and post-callback facilities offered by the clFFT library to perform those steps without requiring additional global memory transactions.

The n-dimensional transforms are then computed by iteratively performing batched onedimensional transforms followed by local data permutations with the local permutation kernels introduced earlier. In order to do this all FFT backends are wrapped to fit a common FFT planning and queuing interface that has to handle all types of transforms, data movements, data permutations and zero-filling. The planner then proceeds to generate all required compute plans by using two additional temporary buffers to perform the required transforms. The best case scenario is when all of the data can fit inside the device's memory, in which case the transform is entirely handled on the GPU. In all others cases, the data has to pass from the device to the host, leading to a drastic performance drop due to memory transfers. As an example the NVidia Tesla V100-SXM2 offers 900GB/s of global memory bandwidth but host-device memory transfers are capped at 16GB/s (more then fifty times less). With such a reduced effective memory bandwidth, the FFT computation is likely to be memory bound.

Navier-Stokes runtime distribution

Now that all operators have been written to work with OpenCL, we can run 2D and 3D incompressible Navier-Stokes simulations in their velocity-vorticity formulation (1.28). We recall that stretching only appears in the 3D case. We run 100 iterations of the problem on a fully periodic domain to analyse the average cost of one iteration without computing any timestep criteria. Spectral operators include curl computation, the poisson solver and diffusion of vorticity. Advection and stretching are split on the grid with a first order Strang splitting. Figure 3.21 shows statistics obtained on the dual CPU device while figure 3.22 exposes 2D and 3D statistics obtained on the two GPU devices. The transpose category now takes into account local permutations due to the directional splitting but also the permutations required to perform FFTs. The D2D category represent device-to-device copies requires for the permutations and for out-of-place FFTs. diffusion, curl and poisson categories correspond to the post-transform elemenwise kernels that are required to compute output fields in the Fourier space.

For 2D simulations the only relevant parameter seems to be the order of the remeshing kernel that goes from 12 to 17% of total runtime. Fourier transforms and permutations represent 50% of the total runtime, followed by ghosts (18%), remeshing and advection (< 10%). Apart from the small increase in runtime due to high order remeshing (+6%) all the mean iteration times fluctuate around 50ms. In the case of 3D simulations, data movements represent about 60% of the runtime (permutations, ghost exchanges and device-to-device copies). Stretching is sensible to both the order of the finite differences and the number of integration stages and represents 10 to 25% of total runtime. The highest order in space and time configuration only cost about 25% more runtime than the lowest order configuration. The CPU platform seems to be really inefficient in terms of ghost exchanges and is already compute bound at the advection step. Mean runtime and runtime ratio is given for different kind of kernels. Each bar corresponds to a specific configuration (time integrator, remeshing formula and finite differences order). The number between parenthesis, when given, corresponds to the number of calls per iteration.

Overall this corresponds in the worst case scenario to 1.3 × 10 9 vertices per second (0.74 ns/vertex) on the NVidia GeForce RTX 2080Ti for the 2D case and 240 × 10 6 vertices per second (4.2 ns/vertex) on the NVidia Tesla V100 SXM2 for the 3D case. The dual CPU setup only provides 43 × 10 6 vertices per second (22 ns/vertex) in 2D and 7.1 × 10 6 vertices per second (142 ns/vertex) in 3D. From those results it is clear that the best way to achieve even better performances is to optimize and possibly eliminate useless ghost exchanges. The solver should also benefit from improved bandwidth in permutation kernels.

Validation of the solver

In this section we are interested in the validation of the OpenCL solver and associated numerical methods with the Taylor-Green vortex benchmark, which constitutes a relevant threedimensional test case before applying those methods to more complex models. The simulation is handled in a fully periodic cubic domain of size [0, 2π] 3 and the evolution of viscous incompressible flow is described by the Navier-Stokes equations in their velocity-vorticity formulation:

ω = ∇ × u ∇ • u = 0 ∂ω ∂t + (u • ∇) ω = (ω • ∇)u + ν∆ω (3.1a) (3.1b) (3.1c) Velocity is recovered with ∆ψ = -ω u = ∇ × ψ (3.2a) (3.2b)
The velocity is initialized with the following smooth conditions:

u x (x, t = 0) = sin(x)cos(y)cos(z) u y (x, t = 0) = -cos(x)sin(y)cos(z) u z (x, t = 0) = 0 (3.3a) (3.3b) (3.3c)
All fields are discretized on a cubic grid with n points in each directions (N = n 3 ) and the viscosity is given by ν = 1/Re = 1/1600. The Poisson solver and diffusion of vorticity are computed spectrally whereas transport and stretching are solved respectively with remeshed particles and finite differences. The whole simulation is run from t = 0 to t = 20 on a single OpenCL device. For all runs, the problem always fits in the device's embedded memory.

We first consider a fixed timestep dt = 10 -3 (dimensionless time) and two configurations:

1. A first configuration using single-precision floating point numbers, a second order time integrator RK2, a remeshing formula that preserves the four first moments Λ 4,2 , fourth order centered finite differences FDC4 and first order Strang splitting.

2. A second configuration featuring double-precision floating point numbers, a fourth order time integrator RK4 38, a remeshing formula that preserves the eight first moments Λ 8,4 , eighth order centered finite differences FDC8 and second order Strang splitting.

Those two configurations are representative of default and highest order numerical methods available in the HySoP library. They should thus be representative of average and worst runtime per iteration. On the NVidia Tesla V100-SXM2 the first configuration can be run up to N = 896 3 while the second configuration up to N = 512 3 . The expected compute complexity of one timestep is O (N log N ) because of the FFTs.

This is effectively what we get in practice as seen on figure 3.23.

Comparison with reference data

The results will be compared to two reference flow solutions obtained by dealiased pseudospectral solvers. The first reference solution is obtained with a dimensionless timestep dt = 10 -3 and a low storage 3-steps Runge-Kutta scheme [START_REF] Hillewaert | Direct numerical simulation of the taylor-green vortex at Re= 1600[END_REF]]. The second reference solution uses a fourth order Runge-Kutta time integrator with a CFL set to 0.75 [START_REF] Van Rees | A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers[END_REF]]. These two reference pseudo-spectral solvers have been grid-converged to a 512 3 grid.

The first parameter of interest is the temporal evolution of the enstrophy integrated on the domain:

ε(t) = 1 |Ω| Ω ω(t) • ω(t) dΩ (3.4)
The second set of reference data consists in the isocontours of the dimensionless norm of the vorticity at t = 8 and t = 9 in the plane x = 0. Reference enstrophy and isocontour data are also available from [START_REF] Van Rees | A comparison of vortex and pseudo-spectral methods for the simulation of periodic vortical flows at high Reynolds numbers[END_REF]] and [START_REF] Hillewaert | Direct numerical simulation of the taylor-green vortex at Re= 1600[END_REF]].

It can be clearly seen on figure 3.24 that both cases spatially converge to some enstrophy curve that is not far away from the reference enstrophy as spatial discretization increases. Figure 3.25 shows the distance of the enstrophy obtained with the present method to the reference data. Without any surprises the higher order method with N = 512 3 is closer to the reference enstrophy curve than the lower order method (even at higher discretizations N = 896 3 ). Isocontours of the norm of the vorticity are compared on figure 3.26. The discrepancies at t = 9 suggest to reduce the timestep around the enstrophy peak, and this can be done by introducing a variable timestep that is driven by a lagrangian CFL. Hence we introduce a third case that corresponds to Case-II with variable timestepping. In order to fully converge in time, the third case introduces variable timestepping with drastic timestep constraints: a CFL condition of 1/10 along with a lagrangian CFL of 1/32. The resulting timestep lies between 6.7 × 10 -5 and 1.3 × 10 -3 , the mean timestep being around 2.3 × 10 -4 (4.3 times more iterations than Case-I and Case-II). The simulation is run with the same parameters as case two (high order methods with double-precision up to 512 3 discretization). At 512 3 , the resulting simulation runs on a single GPU and spans over 85982 iterations at 1.09s per iteration (49% performance hit due to the computation timestep criterias) which represent approximatively one day and two hours. As seen on figure 3.27, the resulting enstrophy curve at N = 512 3 is really close to the reference curve, even after t = 10 when compared to the same simulation at fixed timestep dt = 10 -3 . Without variable time stepping, the total number of iterations would be around 3 × 10 6 (two days and 18 hours).

Distributed computing

This section focus on the last kind of parallelism provided by the HySoP library, namely dataparallelism. The regular grid is split into multiple subgrids to perform domain decomposition.

Each process is then responsible to handle one subdomain by using one or more compute backends. The second part focuses on the different strategies that can be adopted concerning ghost layers data exchanges with neighboring processes such as cross and diagonal ghost exchanges. In the last part we will see that the domain decomposition process is constrained by the Fourier based spectral solver.

Domain decomposition

The large cuboid computational domain B is divided into smaller cuboid subdomains. Its discretization, consisting in a rectilinear global grid of N v vertices, can be split in K local subgrids each labelled by their local cuboid subdomain position p = (p

1 , • • • , p n ) ∈ 0, P 1 × • • • 0, P n .
Every subdomain is affected to a specific compute process denoted P p = P (p 1 , ..., pn) . The K = Π n j=1 P j subdomains and processes are defined by P = (P 1 , • • • , P n ) splittings of the box, one for each direction of the space, and are indexed in a row-major ordering similar to the vertices. The local vertex grid belonging to process P (p 1 , ..., pn) has size

N v,p = (N v,p 1 1 , • • • , N v,pn n
) and is allocated locally to the process it belongs in the main memory or in some dedicated accelerator memory by using the OpenCL standard. An example of uniform domain decomposition is shown on figure 3.29. However, the domain decomposition is not required to be uniform, the only condition for a splitting to be valid being, at the discrete level, the following:

P i -1 p i =0 N v,p i i = N v i ∀i ∈ 1, n (3.5)
For the rest of the notations, we adopt the following convention: calligraphic letters such as N v represent global parameters while their uppercase counterpart N v represent local parameters.

All local parameters are associated to a given subdomain indexed by p but the multi-index can be dropped when there is no ambiguity. Similarly to the global grid, the vertices belonging to the subdomain discretization are indexed using row-major ordering such that vertex v j has index

I j = ((((• • • + j n-3 )N v n-2 + j n-2 )N v n-1 + j n-1 )N v n + i n ) = j • S
p = (O p 1 1 , • • • , O pn n ) vertices such that O p k i = p k p=1 N v,p i ∀i ∈ 1, n ∀p k ∈ 0, P i .

Distributed computing 187

This leads to the following relations that links the global index I i and local index I j for a given subdomain of coordinate p, global vertex multi-index i and local vertex multi-index j:

I i = i • S I j = j • S i k = O p k k + j k (3.6a) (3.6b) (3.6c) with i k =              I i S k if k = 1 I i mod S k-1 S k else (3.7)
and

j k =              I j S k if k = 1 I j mod S k-1 S k else (3.8)
In practice each subdomain is extended on the boundaries such that the discretization contains additional ghosts vertices, useful for stencil based numerical methods such as finite differences [START_REF] Micikevicius | 3D finite difference computation on GPUs using CUDA[END_REF]]. The number of required ghost nodes G = (G 1 , • • • , G n ) depends on the spatial order of the numerical method and the frequency of interprocess boundary data exchanges. Those ghost layers are required to compute values close to the local subdomain boundaries. Once a scalar field has been written to by an operator, its ghosts layers are invalidated and will become valid again after a ghost exchange step where all processes exchange their inner data with their neighbors as illustrated by the blue arrows on figure 3.30. In 2D, each process may exchange data with up to 4 direct neighbors and 4 diagonal neighbors. Subdomains that are on domain boundary B may exchange less data if the boundary conditions are not periodic, in which case the ghost node values can be computed locally, directly from the knowledge of the boundary conditions and inner node values.

Interprocess communications

Each subdomain is of coordinate p is affected to a specific process P p . All processes can communicate with each other via MPI by using a MPI Cart Cartesian topology. With such a configuration, calculations, such as a global sum, require very small amounts of data to be globally communicated between the K processes (O (K) communications). When possible, it is of interest to overlap communications with regular computations, so that the interprocess data exchanges happen while something is being computed, taking no extra communication time when compared with sequential execution [START_REF] Khajeh-Saeed | Direct numerical simulation of turbulence using GPU accelerated supercomputers[END_REF]. This is an important optimization because MPI interconnect hardware has a bandwidth O(10-200GB/s) that is generally up to an order of magnitude less than local to process hardware memory bandwidth O(50-1000GB/s) [START_REF] Bode | Cluster interconnect overview[END_REF]. , N v,px x ) = (3, 4), is extended with G = (G y , G x ) = (1, 2) ghost nodes that overlaps the neighbor processes subdomains. Process P 3 = P (1,1) sends and receives data from and to its neighboring processes in the topology presented on figure 3.29. In this case the periodicity condition on the x-axis is such that process P 2 is at the same time the left and right neighbor. Upper ghost nodes represented in grey accounts for a specific non-periodic boundary condition on the y-axis. The local extended subgrid has size N v,p + 2G = (5, 8) and is represented with a dashed line.

All stencil based operators that require G ghosts to operates need to perform a ghost layer exchange for each of the scalar discrete fields they are writing to. Although the boundary data is typically small (less than 10%) compared to the internal data of the partitions, the MPI communications are also typically very slow (more than 10 times slower) than the internal operations. In the case of nD stencils that are the result of a tensor product of 1D stencils, it is possible for each process to exchange data only with its 2n direct neighbors, connected by an hyperrectangle of dimension n -1 that we will call a (n-1)-rectangle or simply a (n-1)-face. However when the stencil is dense, a process has to communicate with all its 3 n -1 neighbors, including all the diagonals as represented on figure 3.30 with dashed blue arrows. This splits the ghost exchange methods into two categories:

• Cross exchange: Each process exchange data with its direct left and right neighbors in each direction. A given process sends, for a given direction i, a n-rectangle ghost layer of size

(N 1 , • • • , N i-1 , G i , N i+1 , • • • , N n )
to the left and to the right, such that the size of the communication is 2( n i=1 N v,p i )( n i=1 G i /N v,p i ) elements per process. If the grid is uniformly distributed between processes, such that N v i = P i N v i the total data sent by all the K = Π n i=1 P i processes becomes:

E send = 2K n i=1 N v i n i=1 G i N v i = n i=1 N v i n i=1 2G i N v i (3.9)
We can compare this quantity with the total grid data in terms of elements E tot = ( n i=1 N v i ) constituting the global grid and compute the ratio:

α = E send E tot = n i=1 2G i N v i = n i=1 2P i G i N v i (3.10)
Equation (3.10) tells us that to minimize the communications we should choose a number of processes P i and a number of ghosts G i that are small enough compared to the global grid size N v i :

2P i G i N v i ∀i ∈ 1, n 2G i N v,p i i ∀i ∈ 1, n ∀p ∈ 0, P 1 × • • • × 0, P n (3.11a) (3.11b)
• Full exchange: In this case each process exchange data with all its 3 n neighbors.

There exists exactly 2 n-k n k k-rectangles on the boundary of a n-rectangle. The cross exchange corresponds to the case where each neighbors pair share a common k-face where k = n -1. For a given process and a given k, the size of the sent data belongs to

{G 1 , N 1 } × • • • × {G n , N n } where we select k components of N and n -k components of G depending on the chosen direction d ∈ {-1, 0, 1} n \ {0} where k = n -d • d ∈ 1, n .
If the grid is uniformly distributed between processes, such that N v i = P i N v i the total data sent by all the K = Π n i=1 P i processes becomes:

E send = K d∈{-1,0,1} n d =0 n i=1 (G i ) |d i | (N v i ) 1-|d i | = K n i=1 (N v i + 2G i ) - n i=1 N v i (3.12)
As for the cross exchange case we can compare this quantity with the total data constituting the global grid and compute the ratio of elements to send:

α = n i=1 (N v i + 2G i ) - n i=1 N v i n i=1 N v i = n i=1 1 + 2G i N v i -1 = n i=1 1 + 2P i G i N v i -1 (3.13)
As one could expect, equation (3.13) gives exactly the same condition on the number of processes and ghosts as (3.11a) and (3.11b) in order to minimize the communications.

• Full exchange with only 2n neighbors: It is also possible to exchange diagonal data by exchanging a bit more elements but with only 2n messages instead of 3 n . This can be achieved by performing successfully for each direction i, to the left and to the right, a cross exchange of size (N

1 +2G 1 , • • • , N i-1 +2G i-1 , G i , N i+1 +2G i+1 , • • • , N n +2G n ) per process.
If the grid is uniformly distributed between processes, such that N v i = P i N v i , this gives the following numbers of elements to send for all processes:

E send = 2K n i=1 (N v i + 2G i ) n i=1 G i N v i + 2G i (3.14)
which gives a send ratio of

α = n i=1 1 + 2G i N v i n i=1 2G i N v i + 2G i (3.15)
In case of a uniform Cartesian grid this can be rewritten α = nx(1+x) n-1 with x = 2 G N .

If we want to keep the communication ratio under α for a Cartesian grid decomposed in hypercubes

(N = N 1 = • • • = N n ) with the same number of ghosts in any direction (G = G 1 = • • • = G n ), this implies that G < N 2 α n for the cross exchange and G < N 2 (1 + α) 1/n -1 = N 2 α n + O α 2
when α → 0 for the full exchange. The full exchange with limited number of neighbors does not give a nice expression to describe the number of ghosts G in function of the ratio α. For all the three cases, the expression is without any surprises the same for one-dimensional problems (n = 1).

As expected, the number of ghosts at fixed α decreases by decreasing local grid size N . For example if each subdomain is discretized by using local grids of size 512 n and we want to keep the communications bellow α = 10% this gives a maximum of 12 ghosts in 2D and 8 ghosts in 3D independently of the type of exchange. In 4D, the cross and full exchange can be made with up to 6 ghosts, but the full exchange with limited number of neighbors only allows a maximum of 5 ghosts. Although the number of ghosts may not be constrained by the exchange method for reasonable values of N and n, we must not forget that with cross exchange a process only performs communication with up to 2n neighbors whereas with the full exchange a process communicates with up to 3 n -1 other processes (6 vs 26 in communications 3D). The higher number of communications is likely to saturate the MPI interconnect system and is not free if we take into account communication latencies. This is why we will use the full exchange with only 2n neighbors when full ghost exchange is required, at the cost of more data to communicate. Some operations require full interprocess global communications (α ≥ 100%). This is especially the case for the Fast Fourier Transform (FFT), extensively used by spectral operators. For each axis i, the FFT algorithm requires the knowledge of all the lines in the current axis, imposing P i = 1 at the i th step such that N v i = N v i . A solution is to set P n to 1, yielding slab or pencil like MPI topologies, and to perform global data permutations in between each step [Pekurovsky 2012]. This is described in the next subsection. 4) forming what we will call the local compute domain, represented by the green vertices. The discretization is extended on the boundaries by G = (G y , G x ) = (1, 2) additional ghosts vertices represented in grey, forming the local domain. The local grid is allocated as as single block of memory, consisting into the local domain grid of size N v + 2G which rows are eventually padded by some virtual vertices, represented in turquoise, to enforce eventual memory alignement requirements. The local to process grid data is stored contiguously using a row-major ordering as shown in red. Each local compute node v is indexed by its global index I i , represented in black, and its local index I j , represented in blue. Local and global vertex indices are linked by equations (3.6). In this case, if the elements stored in the grid would be double precision floating point numbers of 8 bytes each, the padding would ensure that each line is aligned on a 16x8=128B boundary.

v = (N v y , N v x ) = (3,

Field discretization

Each operator, as listed in algorithm (1), generally takes multiple scalar fields R i ∈ R as inputs and writes to multiple scalar fields W i ∈ W as outputs, some scalar fields being at the same time inputs and outputs in which case they belong to R ∩ W. For example the 3D stretching operator defined by equation (2.6) takes R = {ω x , ω y , ω z , u x , u y , u z } as inputs and outputs the following scalar fields W = {ω x , ω y , ω z }. Each scalar field is discretized on one or more topology. A topology T (B, N v , O, G) is defined by a domain B, a global grid resolution N v along with the local domain decomposition defined by the knowledge of the number of splittings defined by the subgrid offsets O ∈ M K,n (N) for all subdomains and the number of ghosts G that are the same for all the K local grids. A process is responsible to compute the outputs from the local domain discretization of the inputs of size N v + 2G, comprising the local compute domain of size N v and the additional ghost layers (see figure 3.31).

Spectral solvers

In section 2.6 we saw that the Fast Fourier Transform of multidimensional data could be performed as a sequence of one-dimensional transforms along each dimension. An threedimensional array of size (N z , N y , N x ) can be Fourier transformed by first performing N y N z independent one-dimensional transforms of size N x , followed by N x N z transforms of size N y , and finally N x N y transforms of size N z . However in a domain-decomposition context, the signal to be transformed is distributed in the memory of multiple different compute nodes. In such a case, there exists at least one direction (a splitting direction) where the required one-dimensional data is not available locally to the node. There exist two main approaches when the signal does not fit entirely on one node, the first approach consisting in global redistributions of data to ensure that the required one-dimensional signal data is locally available when needed. The second approach consists into implementing directly a distributed FFT algorithm. This is known as the binary exchange method (as opposed to the transpose or permutation method). Those two methods are reviewed in [START_REF] Foster | Parallel algorithms for the spectral transform method[END_REF]] and the implementation based on global permutations of data has often be proven to be superior for large problems and in the presence of many compute nodes [START_REF] Gupta | The scalability of FFT on parallel computers[END_REF]]. As the HySoP library provides its own abstraction for FFT-based transforms, it also has to provide a way to compute those transforms in a distributed context. The implementation of distributed spectral transforms is still a work in progress but several execution policies and data redistribution methods are to be explored.

Slab decomposition is the simplest way to decompose a domain (only one direction is distributed) and a full forward transform can be computed in performing batched 2D transforms in the slabs followed by a global permutation of data and batched 1D transforms on the last axis. After this forward transform, local data is contiguous on the first axis (the z-axis). The backward transform just follows the reversed operation order to recover a contiguous x-axis. Slab decompositions are very efficient but they are limited to a small number of processes (the maximum number of processes scales with min(N x , N y , N z )). This decomposition method is the one provided by FFTW [Frigo et al. 2012]. Apart from the differences in the way to distribute data, some methods exhibit different techniques to handle local data transpositions. Most of the methods perform local transposes prior to global data redistributions (as in the serial case, see [Frigo et al. 2005]) but permutation of data can also happen during data exchange through modern MPI features such as MPI ALLTOALLW [START_REF] Dalcin | Fast parallel multidimensional FFT using advanced MPI[END_REF]]. The best method will depends on many factors such as the latencies and bandwidth provided by the inter-node communication network and the ones involved in eventual host-device memory exchanges.

Conclusion

In this chapter we introduced HySoP, a high-performance library dedicated to fluid-related numerical simulations. The library is developed since 2012 and consists in a Python interface that sits on top of compiled languages such as Fortran and C++. It also relies on just-in-time compilation techniques through OpenCL and Numba. Within this framework, problems are described by building a directed acyclic graph of operators which execution order is deter-mined by topological sort. Three different kind of parallelism are exposed, task parallelism that is directly extracted from the knowledge of the graph, accelerator parallelism that allows to use efficiently various kind of coprocessors by using the OpenCL-1.2 standard and finally data parallelism through domain decomposition. The main contributions of this work to the HySoP library include the dynamic generation of certain numerical methods depending on user needs, the simplification of the user interface with the introduction of DAGs and an extensive command line interface, the generalization of directional splitting, the possibility to generate efficient accelerated elementwise and finite-differences based operators through a symbolic backend and code generation techniques, the introduction of kernel parameter autotuning, the support of non-periodic boundary conditions and finally the implementation of accelerated spectral operators.

A numpy like n-dimensional array interface targeting the OpenCL backend has been implemented. This interface extends pyopencl.Array and allows quick prototyping of OpenCL based operators. Advection and remeshing have been adapted to fit the new OpenCL code generation framework. The generated code is such that full lines of cache are not required anymore yielding some interesting performance improvements on GPUs and the possibility to run the solver on larger discretizations. Local data permutations kernels have been generalized to any dimension to satisfy graph builder and FFT planner requirements. An OpenCL code generator based on symbolic expressions has been implemented. All code generated kernels pass through a user-configurable autotuning process that strives to minimize kernel runtime by tweaking vectorization, global and local sizes and other kernel specific parameters. When not readily available, real-to-real transforms are computed with a classical FFT algorithm with Hermitian symmetry associated to some pre-or post-processing steps. Those transforms are required to handle homogeneous boundary conditions in spectral operators and are implemented on the top of many different external FFT libraries. An Oclgrind plugin has been implemented to collect OpenCL kernel instruction statistics and, apart from remeshing, all kernels are generally memory bound on the considered devices. For all kernels, achieved global memory bandwidth lies between 25 and 90% of maximal theoretical device bandwidth.

Each individual operator is validated within a continuous-integration development practice. The numerical method is validated on a three-dimensional Taylor-Green vortex benchmark that runs exclusively on accelerator with either fixed or variable timestep. It is shown that even if adaptive timestepping increases the mean runtime per iteration due to the computation of additional flow characteristics, the overall simulation duration decreases when compared to equivalent fixed timestep simulation. A three-dimensional incompressible Navier-Stokes problem discretized on a 512 3 grid with double-precision runs roughly at the rate of one iteration per second on a single server-grade GPU. There is still a lot of room for optimizations, ranging from the elimination of useless ghost exchanges to the improvement of achieved memory bandwidth in some permutation kernels. Current implementation efforts focus on MPI spectral transforms to allow fully distributed simulations. In the next chapter we use the HySoP library to perform high-performance simulations of sediment flows. In this chapter we apply our numerical method to sediment flows that can occur in sedimentladen riverine outflows as depicted on figure 4.1. The processes that drive sediment settling are complex and still poorly known. Understanding the settling mechanisms is the key to be able to predict where sediments will settle. We consider a sedimentary process where small particles with negligible inertia, seen as a continuum, settle with a constant Stokes velocity above salt water. When such a layer of particle-laden fresh water flows above clear saline water, both Rayleigh-Taylor and double diffusive fingering instabilities may arise. Both of those instabilities increase the vertical transport of sediments. This simple model already raises numerical difficulties because of the high Schmidt numbers involved.

Introduction

We first verify that our numerical method match the results obtained in the literature. The comparison is drawn in 2D and 3D for a large set of dimensionless parameters specific to this physical problem. We then show that within our framework, the performance delivered by GPUs allows us to compute large ensemble averages of parameters characterizing the flow. The ultimate goal being to understand vertical sediment transport with Schmidt numbers encountered in the nature, we finally develop strategies to increase the simulated Schmidt number.

Statement of the problem

This section follows the description of the problem originally introduced in [START_REF] Burns | Sediment-laden fresh water above salt water: linear stability analysis[END_REF]]. We consider an initial stationary, unbounded single-phase fluid with a vertically stratified density profile. Within this carrier fluid, a layer of particle-laden fresh water is initially placed above denser clear, saline water. Sediment concentration C and salinity S are taken as a physical continuum and are coupled with incompressible Navier-Stokes equations. The fluid has a velocity u(x, t) and an associated vorticity ω(x, t). Its density ρ(x, t) depends on the base density of clear water ρ 0 , augmented by local sediment concentration 0 ≤ C(x, t) ≤ 1 and salinity 0 ≤ S(x, t) ≤ 1. The evolution of the species in the carrier fluid is governed by transport-diffusion equations, and the sediment concentration is transported relatively to some constant Stokes settling velocity V st that is determined by particle properties. The presence of sediment and salt induces local gravity current which modifies the behavior of the fluid through the change of local density profiles (see figure 4.2).

Solid particles in suspension

Salt water ρ = ρ 0 (1 + αS + γC) We do not take into account any particle-particle interactions such that the numerical model only exhibits dual-way coupling. As seen in section 1.2 this assumes that particles have negligible inertia, which is the case for silt and clay. The Stokes settling velocity of a given particle depends on its size and shape [START_REF] Gibbs | Estuarine flocs: their size, settling velocity and density[END_REF]]. Imposing the same constant settling velocity for all particles characterizes a dilute monodisperse suspension. The particles diffusion coefficient can be obtained from the Einstein-Stokes equation [START_REF] Miller | The Stokes-Einstein law for diffusion in solution[END_REF]], and is much smaller than the typical salinity diffusion coefficient. The small diffusion coefficient of the particles accounts for Brownian motion or the mixing that occurs in real polydisperse suspensions of particles, as a result of a distribution of particle shapes and sizes. The resulting dynamic of the problem should not vary significantly under κ c = 0.

u z = 0 ∂u || ∂z = 0 V p C - ν 0 κ c ∂C ∂z = 0 ∂S ∂z = 0 u z = 0 ∂u || ∂z = 0 ∂C ∂z = 0 ∂S ∂z = 0 ∂S ∂t + (u • ∇)S = κ s ∆S ∂C ∂t + (u • ∇)C -V p ∂C ∂z = κ c ∆C x z V p periodic boundary conditions
The density ρ is assumed to be a linear function in S and C:

ρ(x, t) = ρ 0 [1 + αS(x, t) + γC(x, t)] (4.1)
where (α, γ) are the density expansion coefficients of salinity and sediments. In practice, typical density expansion coefficients encountered for sediments and salt are of the order of 2 to 4%.

Governing equations

The resulting numerical model is very similar to the heat-salt system (1.43) used to model thermohaline convection introduced in section (1.3.2). It is extended by introducing an additional relative settling velocity in the transport term relating to the particles: The nondimensionalization procedure is fully described in [START_REF] Burns | Sediment-laden fresh water above salt water: linear stability analysis[END_REF]] and yields the following set of dimensionless parameters: R s = αSmax /γCmax Stability ratio S c = ν 0/κ s Schmidt number between salt and water τ = κs /κc Diffusivity ratio between salt and sediments V p = Vst /(ν 0 g )

ρ = ρ 0 (1 + αS + γC) ∂ρ ∂t + ∇ • ρu = 0 ρ ∂u ∂t + (u • ∇) u = µ 0 ∆u -∇P + ρg ∂S ∂t + (u • ∇) S = κ s ∆S ∂C ∂t + (u • ∇) C -V p ∂C ∂z = κ c ∆C (4.2a) (4.2b) (4.2c)
1 3
Dimensionless settling velocity of the particles g = γC max g Reduced gravity

where αS max and γC max refer to the maximum added density due to salinity and particles.

The average yearly sediment concentration of dirty rivers ranges from 10 to 40kg/m 3 [START_REF] Mulder | Turbidity currents generated at river mouths during exceptional discharges to the world oceans[END_REF]. The value R s = 2 corresponds to a mass loading of roughly 20kg/m 3 of sediments and a salinity of 3%. V p = 0.04 corresponds to the dimensionless settling velocity of spherical particles of radius 10µm such as clays and silts. The Schmidt number between salt and water is approximately S c = 700. Assuming a diffusivity ratio τ = 25, this gives a Schmidt number between sediments and water of τS c = 17500.

The physics tells us that the scalars may develop physical scales up to 27 and 133 times smaller than the finest velocity scales, for salt and sediment respectively (see section 1.1.10).

Once the problem is grid converged, multiplying the grid resolution by two makes it possible to multiply the Schmidt number by a factor of four. It is thus estimated that passing from S c = 7 to S c = 700 on a 3D simulation will require to pass from a grid of size 1537 × 512 × 512 as obtained in [Burns et al. 2015] to a grid of size 15370 × 5120 × 5120 which is a problem that is 10 3 times larger then the current best simulation. The increased spatial resolution naturally comes with increased number of iterations because of timestep restrictions. While numerical considerations make it hard to achieve S c = 700, the physical mechanisms to be explored are also relevant for smaller Schmidt numbers such as a warm river outflow into a cold lake (S c = 7) or the settling of small water droplets in a temperature gradient in the air (Sc = 0.7).

Boussinesq approximation

As we consider only small local density perturbations, we place ourselves under the Boussinesq approximation where local variations of density are only taken into account in external forces (represented in blue in equations 4.2). This approximation has been introduced in section 1.1.6, and we perform the usual pressure shift to include the base density of water into the pressure term. In the end, when passing to the velocity-vorticity formulation the shifted pressure disappears and the resulting single-phase Boussinesq fluid is evolved with the following dimensionless set of equations:

ω = ∇ × u ∇ • u = 0 ∂ω ∂t + (u • ∇) ω -(ω • ∇)u = ∆ω -∇ × (R s S + C) e z ∂S ∂t + (u • ∇) S = 1 S c ∆S ∂C ∂t + (u • ∇) C -V p ∂C ∂z = 1 τS c ∆C (4.3a) (4.3b) (4.3c) (4.3d) (4.3e)
The velocity is recovered from the vorticity by solving the following Poisson problem:

∆ψ = -ω ∇ • ψ = 0 u = ∇ × ψ (4.4a) (4.4b) (4.4c)
Note that from now on u, ω, S and C refer to dimensionless fields.

Boundary conditions

As depicted on figure 4.2 for the 2D case, the boundary conditions in the horizontal directions are periodic. The 3D case adds an additional horizontal axis (x, y), the vertical direction being always considered as z. Top and bottom walls, denoted Γ t and Γ b respectively, are slip walls with no penetration. On those walls, we impose a no-flux boundary condition for the salinity field. For the sediment field we use a no-flux boundary condition formulated with respect to the Stokes settling velocity of the particles on Γ t and the no-flux boundary condition on Γ b .

∂u || ∂z = 0 and u z = 0 on Γ t ∪ Γ b V p C - 1 τS c ∂C ∂z = 0 on Γ t et ∂C ∂z = 0 on Γ b ∂S ∂z = 0 on Γ b ∪ Γ t (4.5a) (4.5b) (4.5c)
The • || notation will always denote horizontal components • x in 2D and (• x , • y ) in 3D.

Initial conditions

As stated earlier the flow is initially quiescent and the initial sediment concentration field is given by a smoothed step profile. The initial salinity field is then obtained by computing S = 1 -C. -200, 200] and l 0 = 1.5.

Unless otherwise specified we define δ(x, y) as p 0 l 0 [U(0, 1) -0.5] with p 0 = 0.1 so that the initial perturbation stays within ±5% of initial profile thickness l 0 . Here U(0, 1) represent random samples obtained from a uniform distribution over [0, 1].

Interfacial instabilities

Similarly to the heat-salt system, the initial vertical density profile ρ 0 (z) = ρ(z, t = 0) is such that the system is initially gravitationally stable ( ∂ρ 0 ∂z < 0). Here both the settling velocity of the particles and the difference in the scalar diffusivities may destabilize the initial interface.

If we first consider V p = 0, the problem is purely double-diffusive and we know that the form of the resulting vertical motions depends on whether the driving energy comes from the component having the high or low diffusivity (section 1.3.2). We recall that in this case two basic type of convective instabilities may arrise: diffusive and fingering configurations. In our problem, a layer of particle-laden fresh water initially lies on top of a denser layer of saline water. Because the salinity S, which diffuses τ = 25 times faster than sediments, provides the stable stratification, the resulting instabilities will be of finger type.

The presence of a non-zero settling velocity V p can significantly increase the instability growth rate because particles settle downwards into the upper region of the saline layer, where they form an unstable layer of excess density which is gravitationally unstable. An important parameter is the ratio of the particle settling velocity to the diffusive spreading velocity of the salinity interface. When this ratio is kept below unity, instabilities are still mostly determined by double-diffusive effects. For ratios above unity, large settling velocities or equivalently large Schmidt numbers, the salinity does not diffuse fast enough to keep up with the particles that are settling. In this case, it is expected that Rayleigh-Taylor instabilities dominate the flow [START_REF] Lord | Investigation of the character of the equilibrium of an incompressible heavy fluid of variable density[END_REF]].

Hence, when a layer of particle-laden fresh water flows above clear saline water, both Rayleigh-Taylor and double diffusive fingering instabilities may arise. Those two kind of instabilities are represented on figure 4.4. In both cases, the vertical fluxes can be much larger than the vertical transport in a single-component fluid because of the coupling between diffusive and convective processes [START_REF] Turner | Multicomponent convection[END_REF]].

Solid particles in suspension

Setup and validation of the solver

In this section we describe how the algorithm to solve incompressible Navier-Stokes equations coupled with a passive scalar is extended to handle dual-way coupling with sediment and salinity fields. After defining all the timestep criterias required for this particular problem, the problem is described within the HySoP framework. The resulting solver is compared to state of the art numerical simulations for reference two-and three-dimensional flow configurations.

Outline of the method

In order to solve the dimensionless problem 4.3 we use the same approach as in algorithms 1 and 4. In those algorithm we replace the transport and diffusion of the passive scalar θ at fixed velocity u, by the independent transport and diffusion of scalar field S and sediment concentration field C. The stretching and buoyancy term ∇ × (R s S + C) are then computed and integrated into the vorticity ω by using the directional splitting approach introduced in section 2.5.

Given an initial state (u k , ω k , C k , S k ) at a given time t k , we can compute (u k+1 , ω k+1 , C k+1 , S k+1 ) at time t k+1 = t k + dt k by using the following algorithm:

1. Independently perform each transport-diffusion steps for the vorticity, salinity and sediment concentration fields at fixed velocity. Diffusion can be either spectral or finitedifferences based depending on the timestep criterias and actual grid discretization. (c) Obtain C k+1 from C k , u k -V p e z and equation (4.3e).

2. Compute conservative stretching (1) and the Buoyancy term ∇ × (R s S + C) directionally by using explicit finite differences. Integrate those terms into ω k,1 with an explicit Runge-Kutta time integrator and a directional Strang splitting to obtain ω k,2 .

3. Correct the newly obtained vorticity ω k,2 to obtain the divergence-free vorticity ω k+1 and compute u k+1 spectrally with equations 4.3a, 4.3b, 4.4 and algorithm 11.

= k + 1.
Within a multiscale approach where the sediments and salinity fields are discretized on a finer grid than the one of the velocity and the vorticity, this algorithm is modified such that velocity is interpolated to the fine grid prior to step 1 and scalar fields are restricted to the coarse grid in-between step 1 and 2. By default all fields are defined on the same grid and the fastest diffusing component being the vorticity (for S c above unity) the vorticity is diffused spectrally while the scalars are diffused by using explicit finite differences split on the grid.

Setup and validation of the solver 203

Timestep criterias

Let dx = (dz, dy, dx) denote the space step associated to the coarse grid where the flow is discretized (u, ω) and dx * = (dz * , dy * , dx * ) the one associated to the grid where the scalars are discretized (S, C). In a multiscale context we have dx * i ≤ dx i for all i ∈ 1, n else dx * = dx. As for the Taylor-Green problem with variable timestep introduced in section 3.2.8 the timestep is determined by:

1. An arbitrary CFL to determine the maximum number of advection ghosts on the coarse grid. Here we have also to take into account directional splitting and additional settling velocity V p :

CF L = max dt u x ∞ dx , dt u y ∞ dy , dt u z ∞ dz , dt u z -V p ∞ dz , ≤ C CFL (4.8) 2.
A lagrangian CFL condition bellow unity that does not depend on any space step so that particles do not cross during the advection step. Because the settling velocity is constant, we recover the classical LCFL timestep criteria, here formulated with respect to vorticity instead of the gradient of velocity:

LCF L = max dt ω x ∞ , dt ω y ∞ , dt ω z ∞ ≤ C LCFL ≤ 1 (4.9)
3. A stability condition for the stretching term:

dt max i∈ 1,3   j∈ 1,3 ∂u i ∂x j ∞   ≤ C STRETCH (4.10)
where C STRETCH is a constant depending on the time integration scheme used to discretize the stretching. For explicit Runge-Kutta schemes we have C STRETCH ≥ 2 (see [START_REF] Mimeau | Conception and implementation of a hybrid vortex penalization method for solid-fluid-porous media : application to the passive control of incompressible flows[END_REF]], appendix B).

4. A stability condition for the explicit finite-differences based diffusion terms:

max dt S -1 c (dx * ) 2 , dt S -1 c (dy * ) 2 , dt S -1 c (dz * ) 2 , dt (τS c ) -1 (dx * ) 2 , dt (τS c ) -1 (dy * ) 2 , dt (τS c ) -1 (dz * ) 2 ≤ C DIFF (4.11)
where C DIFF ≥ 0.5 is a constant depending on the chosen explicit Runge-Kutta scheme.

An additional constant timestep criteria dt MAX ensures the convergence of the coupling between the carrier fluid (u, ω) and the transported scalars (S, C). Note that because the flow is initially quiescent, this constant timestep drives the simulation during the first timesteps.

In practice because the flow is laminar (R e = 1), this timestep criteria may constraint the timestep during the whole simulation under reasonable discretization and particle settling velocity V p . Unless otherwise specified, we use C CFL = 1.0, C LCFL = 0.95, C STRETCH = 2, C DIFF = 0.5 and dt MAX = 0.1.

Numerical setup

Most of the required numerical routines have already been introduced in chapter 3 and the problem is described within the HySoP framework by using existing operators. We use the Λ 4,2 remeshing kernel along with fourth-order Runge-Kutta scheme RK4, fourth-order centered finite-differences FDC4 and second order Strang splitting. The fields are discretized by using single-precision floating point numbers. The resolution of this problem on the OpenCL computing backend requires minimal changes to the existing codebase: the buoyancy term is computed from the directional symbolic code-generation interface introduced in section 3.2.5 and advection routines are adapted to take into account an optional relative velocity. The library generates a graph containing 85 operators with up to nine independent execution queues. This graph includes operators that compute the timestep criterias as well as monitors that compute flow characteristics during the simulation.

The boundary conditions on Γ b and Γ t are homogeneous boundary conditions that are compatible with real-to-real spectral transforms introduced in section 2.6.4. In all the other directions the domain is periodic and we use classical Fourier transforms. The slip walls with no penetration boundary condition ∂u || ∂z = 0 and u z = 0 also yields real-to-real transform compatible vorticity boundary conditions. All top and bottom no-flux boundary conditions on the scalar fields are handled with homogeneous Neumann boundary conditions. The height of the domain is set to be sufficiently large such that the results are not affected by the lower or upper boundary planes.

Flow characteristics

In order to verify that our solver performs well when compared to state of the art numerical results we have to compute many different horizontally averaged flow characteristics. We use the same notations as in [Burns et al. 2015]. Horizontally averaged quantities are computed as:

• (z, t) = xmax x min ymax y min •(x, y, z, t)dxdy (4.
12)

The quality of horizontally averaged statistics can be improved by using a large horizontal domains or by computing ensemble averages over multiple runs with the same parameters.

Scalar interfaces tracking:

The error function is solution of the laminar diffusion equation with constant diffusion coefficient starting from a discontinuous initial condition [START_REF] List | Mixing in inland and coastal waters[END_REF]] and serves to provide a good physical approximation of actual sediment and salt concentration profiles. We can thus track both the thickness of a given interfacial region as well as its position by computing the least-squares fit of the horizontally averaged scalar profiles to an error function: z • is the location at which • f it = 0.5, location of the horizontally averaged interface, whereas l • represents the thickness of the horizontally averaged profile. The optimization process is done every few timesteps and takes as input the last known interface location z * • and thickness l * • , initial values being set to z c = z s = 0 and l c = l s = l 0 . The fitting procedure is weighed more heavily around the last known position of the interface z *

• than near the end points of the profile. The weights are chosen to have analytical formula w

(z) = 1 1 + 10 -2 (z -z * • ) 2 .
Height, thickness ratio and nose region ratio: After the fitting procedure we can compute the sediment to salinity thickness ratio as well as the offset between the interface locations. The thickness ratio is defined as ξ(t) = l c (t)/l s (t) and the height is defined as the difference between the upward salinity interface and downward sediment interface locations:

H(t) = z s (t) -z c (t) ≤ 0.
Within this nose region both salinity and sediment are present in high concentrations.

The nose region height to salinity profile thickness ratio is defined as R t (t) = H(t)/l s (t). Note that both H(t) and l s (t) diffuse with time but the nose region remains embedded within the larger region containing the salinity gradients: H(t) < l s (t). It has been observed in [START_REF] Burns | Sediment-laden fresh water above salt water: linear stability analysis[END_REF]] that this ratio will dictate the dominant type of instability:

• H/l s ≤ O (0.1): double-diffusive fingering dominates.

• H/l s ≥ O (0.1): sediment and salinity interfaces become increasingly separated in space, the dominant instability mode becomes Rayleigh-Taylor like.

They show that the ratio R t initially grows and then plateaus, at a value that is determined by the balance between the flux of sediment into the nose region coming from above, the sediment flux out of the nose region below, and the rate of sediment accumulation within the region.

Vertical interface velocities: Upwards salinity interface velocity v zs is obtained by a linear fit of z s (t -t 0 ) = v zs t where t 0 represents the delay before the salinity interface begins to move upwards. Downwards sediment interface velocity v cs is obtained by a linear fit of z c (t -t 0 ) = v cs t. Before the initial delay t 0 it is expected that z c (t) = -V p t and z s (t) 0.

Turbulent interface diffusivities: Fitting the interfacial thicknesses l • to diffusive √ t profiles allows us to compute the turbulent diffusivities of the sediment and salt interfaces. If we assume that the averaged profiles evolve as

∂ S ∂t = 1 S c ∂ 2 S ∂z 2 and ∂ C ∂t = 1 τS c ∂ 2 C ∂z 2 we can fit • l s (t) to l s,f it (t) = 1 S c t 1 2 to obtain S c . • l c (t) to l c,f it (t) = 1 τ S c t 1 2 to obtain τ.

206

Chapter 4. Sediment-laden fresh water above salt water

Comparison with two-dimensional reference solution

The first reference simulation is obtained from [Burns et al. 2015] with the following low Schmidt number configuration, discretized over a grid of size N v = (3n + 1, n) with n = 1024:

• t ∈ [t start , t end ] = [0, 500] • Ω = [z min , z max ] × [x min , x max [= [-600, +600] × [0, 750[ • S c = 0.7, τ = 25, V p = 0.04, R s = 2
Sediment and salt interfaces are fitted with error functions during the simulation every ∆ t = 1 dimensionless time. This yields 500 estimations of interface locations (z c , z s ) and interface thicknesses (l c , l s ). Snapshots of horizontally averaged profiles of sediment concentration and salinity as well as their error function fit are plotted on figure 4.5. Mean sediment interface and error function fit Mean sediment interface and error function fit

C C 1 1 1 1 1 1 1 1 1 1 -200 0 200 z t=0 t=2
C C 1 1 1 1 1 1 1 1 1 1 -200 0 200 z t = 1 0 0 t = 2 0 0 t = 3 0 0 t = 4

Quantitative comparison

Figure 4.5 shows that the horizontally averaged salinity profile shares many features with the horizontally averaged sediment profile. The salinity profile is much smoother due to the faster diffusion of salinity and the results of the error function fitting procedure are much closer to the original curves than the ones of obtained from averaged sediment concentrations. The evolution of fitted average interface locations (z c (t), z s (t)) and interface thicknesses (l c (t), l s (t)) is shown on figure 4.6 and compared to reference results. The fitting procedure used to estimate those parameters is not exactly the same in the reference data because the weighting procedure used in [Burns et al. 2015] is not described. The parameter that is the most sensible to the optimization procedure for a given scalar is the interface location and the method employed to obtain reference interface locations seems to be much more sensible than ours (data has been smoothed out in the data extraction process, see figure 8 of [Burns et al. 2015] for original curves). The two runs also differ in the random displacement of the initial interface determined by δ(x) in equation 4.6. As stated earlier, we use a uniform distribution of the form δ(x) = 10 -1 l 0 [U(0, 1) -0.5] which may be different in amplitude than the one used to obtain reference data. -300, 300] so that our results can be compared to reference data available from [Burns et al. 2015], figure 4. As it can be seen on the top frames, the interface is initially displaced by a random perturbation δ(x) but develops a definite preferred wavelength during the early development of the instability. Overall our solver seems to capture the same physical scales and interface displacements as the reference one. Here the instability is clearly double-diffusive with the emergence of fingers at the interface. As for the reference solution, we observe both positively and negatively buoyant plumes that detach from the interfaces at t = 200 and eventually collide around t = 300. When the simulation starts, it can be clearly seen that the sediment interface begins to move downward with velocity V p as a result of initial particle settling (orange and red curves on figure 4.6). Initial sediment interface location prediction -V p t is shown as a dashed line and after t 0 = 40 double-diffusive processes take over. Before t 0 we also observe that the initial salinity interface does not move much (it only diffuses upwards at rate 1/S c ). After t 0 , both the sediment and salinity interface thicknesses can be fit to √ t diffusive profiles (represented with dashed lines) and we obtain roughly the same profiles as in the reference data. In this configuration, the turbulent diffusion resulting from the double-diffusivity instability along with the settling velocity of sediments has modified the sediments evolution from diffusing τ = 25 times more slowly than salinity, to diffusing close to twice as fast.

Finally we can compare the ratio of the nose region height H(t) = z s (t) -z c (t) to the salinity interface thickness l s (t). It is expected from [START_REF] Burns | Sediment-laden fresh water above salt water: linear stability analysis[END_REF]] that once is fully developed (after t 0 ), the ratio H/l s remains roughly constant. This is effectively what we get in practice, with a ratio R t (t) that is in agreement to the one obtained in the reference data. The fluctuations are mainly due to the differences in the estimation of interface locations z c (t) and z s (t). Those fluctuations can be smoothed out by computing an ensemble average over multiple runs with different initial random perturbations (see section 4.3.1).

Comparison with three-dimensional reference solution

The second reference simulation is also obtained from [Burns et al. 2015] and features the following configuration:

• t ∈ [t start , t end ] = [0, 150] • Ω = [z min , z max ] × [y min , y max [×[x min , x max [= [-110, +65] × [0, 100[×[0, 100[ • S c = 7, τ = 25, V p = 0.04, R s = 2
It is discretized over a grid of size N v = (3n + 1, n, n) with n = 512.

Setup and validation of the solver 211

This moderate Schmidt number configuration will highlight different flow features. Note that the domain is now much smaller so that the space step allows to simulate this ten-fold increase in the Schmidt number. Our simulation is carried out on a slightly bigger domain then the one of the reference simulation.

High performance computing for sediment flows

In this section we explore the performance and limitations of our solver with respect to the sediment-laden fresh water above salt water problem.

Ensemble averaged two-dimensional simulations

The two reference simulations with S c = 0.7 and Sc = 7 have already shown the influence of the Schmidt number on instabilities. For small Schmidt numbers, molecular diffusion is high and the settling velocity is too low to have a real effect on the evolution of the interfaces in which case double diffusion dominates. Larger Schmidt number values result in a sharper salinity interface and the double-diffusive mode is replaced by Rayleigh-Taylor instabilities. We cannot however draw any conclusions from a single numerical simulation. The idea of [Burns et al. 2015] is to perform a parametric study over 170 simulations with various configurations of the dimensionless parameters of this problem (V p , R s , τ, S c ). Those simulations each yield horizontally averaged flows quantities that are ensemble-averaged over 10 runs to see the effect of the parameters on the H/l s ratio that determines the type of instabilities. This represents an embarrassingly parallel problem that is a first application to our high performance solver.

To illustrate this, we run four different two-dimensional configurations from Sc = 0.07 to Sc = 70.0 and compute ensemble-averaged parameters every ∆ t = 5 over 100 runs. It takes 227s to compute a case from t = 0 to 500 with dt MAX = 1, discretized on (n + 1, n) points with n = 4096 on a single Nvidia Tesla V100 GPU. Those four different ensemble-averaged cases thus run in approximately 6 hours and 20 minutes on a single compute node containing four of such GPUs. Samples of obtained statistics are shown on figure 4.12 where the thick lines represent ensemble averaged statistics. Estimated interface velocities and diffusivities as well as median thickness ratios and nose region height to salinity thickness ratios are shown in table 4.1 for n = 2048 and n = 4096. It can be seen that the cases Sc = 7 and Sc = 70 do no converge to the expected solution because of insufficient spatial discretization.

n = 2048 S c

V zs V zc S c τ ξ R t 0.07 9.27 × 10 -3 -2.45 × 10 -2 1.07 × 10 1 1.15 × 10 0 1.16 × 10 0 6.71 × 10 -2 0.7 1.57 × 10 -2 -3.24 × 10 -3 3.39 × 10 0 1.65 × 10 0 1.67 × 10 0 1.63 × 10 -1 7.0 1.71 × 10 -2 9.40 × 10 -3 8.64 × 10 -1 1.45 × 10 0 1.43 × 10 0 3.24 × 10 -1 70.0 1.22 × 10 -2 9.09 × 10 -3 2.17 × 10 -1 1.21 × 10 0 1.21 × 10 0 5.68 × 10 -1 S c n = 4096 0.07 8.84 × 10 -3 -2.57 × 10 -2 1.05 × 10 1 1.18 × 10 0 1.19 × 10 0 6.66 × 10 -2 0.7 1.52 × 10 -2 -3.52 × 10 -3 3.40 × 10 0 1.66 × 10 0 1.66 × 10 0 1.62 × 10 -1 7.0 1.65 × 10 -2 9.08 × 10 -3 8.70 × 10 -1 1.45 × 10 0 1.44 × 10 0 3.22 × 10 -1 70.0 1.24 × 10 -2 9.48 × 10 -3 2.10 × 10 -1 1.19 × 10 0 1.19 × 10 0 5.88 × 10 -1

Three-dimensional simulations

On a GPU with 32GB of memory, our implementation is able to solve single-precision twodimensional problems with discretizations up to (32768, 16384) and three-dimensional configurations up to (1537,512,512). On an Nvidia Tesla V100, the three-dimensional problem is memory-bound and can be solved at a rate of 10.9s per iteration, including the computation of the statistics and timestep criterias. A compute-node with 256GB of memory and a sufficient number of CPUs can handle a simulation of size up to (3073,1024,1024). In this case the problem is compute-bound and it takes around 24 minutes per iteration on a quad socket Intel Xeon E7-8860 v4 platform when using the Intel MKL-FFT library to perform the spectral transforms. The total number of iterations required to solve a typical three The resulting isocontours of sediment concentration obtained with the GPU setup and S c = 7 are shown on figure 4.13. This specific configuration is at the interface between the two kind of instabilities we may encounter: double-diffusion and Rayleigh-Taylor instabilities. Before t = 100 we can clearly see sediment fingers moving both upwards and downwards, result of a double-diffusive process. After t = 100, the instabilities become more and more Rayleigh-Taylor like. As introduced in section 1.3.2 the dominant type of instability can be deduced from the knowledge of the particle settling velocity and the diffusive spreading velocity of the salinity layer. More precisely, the key parameter that determines whether the settling process is dominated by double-diffusive or Rayleigh-Taylor instabilities is the ratio of the nose region height to the salinity thickness R t = H/l s . By performing a parametric study over 15 different two-dimensional configurations, [Burns et al. 2015] showed that the value of the nose thickness ratio R t could be predicted a priori based on the knowledge of the dimensionless grouping R s V p √ S c . Because there currently exist no three-dimensional reference data for Schmidt numbers higher than 7, we propose to validate our implementation to higher Schmidt numbers by checking the fit between the predicted values of H/l s and obtained results. Table 4.2 (A-B-C-D) shows the two-dimensional configurations used by Burns and its collaborators to perform their parametric study. We introduce a new dimensionless group (E) that characterizes our three-dimensional test cases. We explore Schmidt numbers ranging from S c = 3.5 to S c = 28.0 while other dimensionless parameters are kept constant. For small values of R s V p √ S c (small settling velocity or big salinity diffusivity) the sediment inflow into the nose region from above is small and double-diffusive fingering compensate with an equal sediment outflow from within the nose region (H/l s remains small). For large values of R s V p √ S c (large settling velocity or small salinity diffusivity), the sediment inflow into the nose region from above is high and the nose region height H grows faster than the salinity layer thickness l s . As sediments accumulate in the nose region, H/l s grows until achieving a critical value where Rayleigh-Taylor instabilities kicks in, releasing downward sediments plumes. The transition between small and large values has been identified as R s V p √ S c = O (0.1).

Our four three-dimensional test cases (E) are run simultaneously on a single compute node containing four GPUs and give the following results: We compare those results with the a priori prediction of the nose thickness ratio from [Burns et al. 2015]. As seen on figure 4.14, all the four predictions are in agreement with our results.

τ R s V p S c H(t = 150) l s (t = 150) R s V p √ S c R t = H/

Chapter 4. Sediment-laden fresh water above salt water

The sediment concentration isocontours corresponding to averaged quantities from table 4.3 are shown on figure 4.16. As expected, with increasing Schmidt number, we see the transition from a double-diffusive fingering behaviour (R s V p √ S c = 0.15) to a Rayleigh-Taylor dominated configuration (R s V p √ S c = 0.42). The two lowest Schmidt number configurations feature salt fingers initially present on both sides of the interface that are eventually suppressed by Rayleigh-Taylor instabilities after t = 100. For the two highest Schmidt number configurations no more sediment fingers can be seen. Instead, we can see downward moving sediment plumes that have a mushroom like structure characteristic of Rayleigh-Taylor instabilities.

Achieving higher Schmidt numbers

Achieving a Schmidt number of 100, an order of magnitude above the grid-converged simulation at Sc = 24, will require a discretization of at least (3073,1024,1024). This would be too costly for a single compute-node and too big too fit into a single GPU memory. Some memory could be saved by using the multiscale approach where only the scalars are discretized on a fine grid of size (3073, 1024, 1024) while velocity and vorticity are discretized on a coarser grid of size (1537,512,512). It is however not possible to perform this simulation on a GPU with only 32GB of RAM because a single field discretized on the fine grid with single-precision would take up to 13GB of memory.

We recall that when using a multiscale approach, the ratio between the scalar and velocity grids can be initially computed as the square root of the respective Schmidt numbers (see section 1.1.10). For τ = 25 and Sc = 100 we have √ τS c 50. This indicates that we could use grid ratios up to 50 in each direction between the velocity and sediment concentration grid (and 10 for the salinity grid). To illustrate this point, the different physical scales obtained for a problem with Sc = 7 and τ = 25 are shown on figure 4.15 In practice to simplify the interpolation and restriction steps, we restrict ourselves to power of two grid ratios (2, 4, 8, 16, • • • ). A two-and three-dimensional convergence study with fixed initial displacement δ has shown that, when the flow is sufficiently resolved, such a multiscale approach worked well with linear interpolation up to a grid ratio of 2. Higher grid ratios required at least cubic polynomial interpolation. Currently the only limitation of our implementation relies in the fact that we perform in-core computations on single GPUs with rather limited memory (32GB per GPU). This means that the problem has to fit in the embedded GPU memory (VRAM). On the contrary CPU devices can directly use the main memory (RAM) which is usually larger (currently, 128GB to 512GB per compute node is rather common) but offer far lower floating-point compute capabilities (see table 1.2).

Hence, a solution to achieve higher Schmidt numbers on accelerators having limited memory capabilities is to perform out-of-core simulations by storing the whole problem in the host memory instead of device memory. The idea is then to overlap compute kernels, device-tohost and host-to-device memory transfers by using multiple OpenCL command queues [START_REF] Van Werkhoven | Performance models for CPU-GPU data transfers[END_REF]. The second solution is to distribute the computation across multiple GPUs by using domain decomposition. Ongoing developments efforts are thus focused on distributed spectral operators and support for out-of-core computations.

Conclusion

This chapter was dedicated to the simulation of particle-laden clear water above salt-water.

The full physical model has first been introduced, followed by the specification of initial and boundary conditions. The computational domain consists in a box, compatible with Cartesian grid discretization. The use of the remeshed particle method make it possible to discretize the sediment concentration and salinity on a finer grid than that of the flow variables, without impacting the timestep. An algorithm to solve this problem within our framework has been proposed along with associated timestep criterias. Homogeneous boundary conditions are handled by using the spectral solver introduced in section 3.2.6. The horizontally averaged flow characteristics of interest in this kind of problem have been detailed. In the end, the whole simulation, including the computation of averaged quantities, can run exclusively on the OpenCL compute backend.

The proposed implementation was run for two-and three-dimensional configurations and compared to reference solutions available from state of the art numerical simulations. Obtained numerical results were in agreement both qualitatively and quantitatively. We then showed that within our framework, the performance delivered by GPUs allowed us to compute large ensemble averages of parameters characterizing the flow. Subsequent three-dimensional runs for for Schmidt numbers comprised between S c = 3.5 and S c = 24 showed the transition from a double-diffusive fingering behaviour to a Rayleigh-Taylor dominated configuration. Because all other dimensionless parameters were kept constant between the different cases, this was the expected behaviour and our results all fit the prediction of the nose thickness ratio. We thus demonstrated that it was possible to simulate flows at Schmidt numbers while keeping reasonable compute times on a single GPU. The ultimate goal being to understand vertical sediment transport with Schmidt numbers encountered in the nature (Sc = 1750), we finally developed strategies to increase the simulated Schmidt number. While the results obtained on a single GPU looks very promising, going even higher in Schmidt number will require to distribute the computations on multiple GPUs.

Conclusion and perspectives

General conclusion

The main objective of this work was to achieve high Schmidt number sediment-laden flow simulations by using adapted numerical methods in a high performance computing context.

The physics of this kind of problem is characterized by the presence of several phenomena at different physical scales that is due to the large difference in the diffusivities of the transported components. Those type of flows are more commonly referred to as high Schmidt number flows. The need for high performance computing and adapted numerical methods was motivated by the fact that such flows are very demanding in terms of computing ressources.

Achieving high Schmidt numbers is the key to be able to numerically investigate the settling of fine particles arising from sediment-laden riverine outflows. The underlying numerical simulations are designed to identify the dominant sediment settling mechanisms which in turn can help to determine the location of sediment deposits on the sea floor. Based on the state of the art, this work proposes an efficient numerical method that is able to capture the dynamics of high Schmidt number flows in which there exist a two-way coupling between the transported quantities and the fluid.

A first chapter has been dedicated to a review of the underlying physical model and its derivation. The mathematical models required for the modeling of sediment flows have first been introduced, leading to the modelization of dilute monodisperse distribution of small particles as a continuum, that is, a particle concentration. The corresponding model was reduced to a coupling between the particles and the fluid, in an eulerian-eulerian framework. This model served as a base model for the physics of sediment-laden fresh water above salt water and was then further simplified by using a Boussinesq approximation. The final model consisted in a coupling between two scalars, representing salinity and sediment concentration, with the carrier fluid. The small diffusivity of fine sediments such as clay or fine silts was estimated by a semi-empirical model and led to Schmidt numbers between sediment and water as high as 17500. We saw that in such a flow, the slowly diffusive particles could develop smaller physical scales, up to a ratio depending on the square root of the Schmidt number, when compared to the smallest velocity eddies.

The second chapter was focused on the development the numerical method. This numerical method has been designed by starting from a state of the art numerical method adapted to the resolution of incompressible Navier-Stokes equations passively coupled with the transport of a scalar, at high Schmidt number. The key idea behind the proposed numerical method relies in the use of a semi-lagrangian (remeshed) particle method which benefits are twofold. Firstly, this method do not suffer from a timestep restriction depending on the grid size (CFL) but rather on velocity gradients (LCFL), reducing the timestep constraints associated to high Schmidt number flows. Secondly, the presence of an underlying grid allows to the use of

Conclusion and perspectives

efficient eulerian solvers. The final proposed method consisted in an hybrid method based on semi-lagrangian particles, spectral methods and finite differences based methods. By relying heavily on operator splitting techniques, the method has been further adapted in order to become accelerator-friendly.

In the third chapter, the high performance implementation of the method has been described. Details about the development of associated numerical routines were given. Required numerical methods were implemented once for multiple different architectures by relying on the OpenCL standard. Performance-portability has been achieved by the way of code generation techniques associated to automatic runtime kernel performance tuning. Within the solver, the description of a numerical algorithm has been handled by building a graph of operators.

The translation of finite differences based methods to GPU accelerated operators has been simplified by an OpenCL code generator based on symbolic expressions. An implementation of efficient OpenCL real-to-real transforms, required to handle homogeneous boundary conditions in spectral operators, has been proposed. A tool dedicated to the collection of OpenCL kernel instruction statistics has been implemented to evaluate the performance of the proposed numerical routines. For all kernels, achieved global memory bandwidth lied between 25 and 90% of peak device memory bandwidth. Two other levels of parallelism were also described: task parallelism that is automatically extracted from a directed acyclic graph of operators and domain decomposition. This implementation has been validated on a three-dimensional Taylor-Green vortex benchmark running exclusively on single GPU accelerator.

Chapter four has been devoted to the simulation of particle-laden clear water above saltwater. The problem to be solved as well as target averaged flow statistics were introduced. Reference solutions were obtained from state of the art two-and three-dimensional numerical simulations. Similar configurations were run with the proposed implementation by using a single GPU. Obtained numerical results were in agreement with the reference data both qualitatively and quantitatively. Even though no reference data exists for Schmidt numbers higher than 7, subsequent run at higher Schmidt numbers showed that the resulting flow was dominated by the expected type of instabilities, slowly passing from a double-diffusive fingering behaviour to Rayleigh-Taylor dominated settling as the Schmidt number increased.

We thus demonstrated that it was possible to simulate Schmidt numbers four times higher than current state of the art simulations by using a single GPU accelerator. Distributing the computations on multiple GPUs will allow us to reach even higher Schmidt numbers while keeping reasonable compute times.

Perspectives

Many perspectives emerge from the numerical and applicative aspects of this work. First of all, since the spectral solvers have been rewritten from scratch to implement the support of homogeneous boundary conditions on accelerators, the distributed MPI implementation introduced in section 3.3.4 has not been implemented yet. This is the current blocking point for distributed multi-GPU simulations and constitute an essential step to develop the full potential of the proposed numerical method. Moreover, two performance bottlenecks have been identified in chapter 3.2: ghost exchanges and permutations kernels.

The roadmap to achieve higher Schmidt numbers within the HySoP library is the following:

1. Short term perspectives to achieve Schmidt numbers up to Sc 100:

(a) Implement CPU-GPU data streaming to perform numerical simulations up to 8 GPUs on a single compute node. The problem will have to fit in the compute node memory while being sent back and forth through the PCI bus. While this will reduce the performance of memory-bound kernels, it will allow to compute bigger problems in terms of spatial resolution without having to wait for the next generation of GPU cards. Here the basic idea is to overlap kernels, device-to-host and host-todevice copies by using multiple command queues [START_REF] Van Werkhoven | Performance models for CPU-GPU data transfers[END_REF]].

(b) This will also be the opportunity to implement hybrid OpenCL-OpenCL computing by using CPU and GPU OpenCL platforms at the same time. A simple microbenchmark could be used to determine the optimal compute granularity and the workload between CPU and GPU [START_REF] Henry | Toward OpenCL automatic multi-device support[END_REF][START_REF] Aji | Automatic command queue scheduling for task-parallel workloads in opencl[END_REF]].

2. Mid-term perspectives to achieve Schmidt numbers Sc 400:

(a) Implement distributed spectral solvers and perform numerical simulations up to 64 GPUs. Here an approach similar to [START_REF] Aji | MPI-ACC: An integrated and extensible approach to data movement in accelerator-based systems[END_REF]] and [START_REF] Dalcin | Fast parallel multidimensional FFT using advanced MPI[END_REF]] should be preferred. This would yield three levels of parallelism by using hybrid MPI-OpenCL-OpenCL programming.

(b) If possible, introduce better permutations kernels, based on open-source CUDA tensor-transpose libraries [START_REF] Hynninen | cutt: A high-performance tensor transpose library for cuda compatible gpus[END_REF][START_REF] Vedurada | TTLG-an efficient tensor transposition library for gpus[END_REF]. Determine preferred permutation axes by microbenchmarking prior to graph building.

(c) Reduce the number of required ghost exchanges by integrating ghosts exchange to the graph analysis step presented in section 3.1.4.

3. Long-term perspectives to achieve Schmidt numbers Sc 1600 (512 GPUs):

(a) Implement time sub-stepping for finite-difference based diffusion operators. Here the numerical method of choice would be RKC, a family of Runge-Kutta-Chebyshev formulas with a stability bound that is quadratic in the number of stages [START_REF] Sommeijer | RKC: An explicit solver for parabolic PDEs[END_REF]].

(b) Depending on the performance of the distributed spectral solver, implement an alternative solver based on an Aitken-Schwarz algorithm [START_REF] Garbey | On some Aitken-like acceleration of the Schwarz method[END_REF][START_REF] Baranger | On Aitken like acceleration of Schwarz domain decomposition method using generalized Fourier[END_REF]]. This algorithm trades the global permutations required by the spectral transforms with simple ghost exchanges, at the price of increased complexity.

(c) Use the task parallelism provided by the graph analysis by using asynchronous capabilities of the Python language [START_REF] Hunt | Concurrency with AsyncIO[END_REF]]. At this point, this would yield four levels of parallelism (Task-MPI-OpenCL-OpenCL). Simulations containing blocking MPI communications and disk I/O should largely benefit from this feature.

The HySoP library is planed to be open-sourced by the end of the year (2019). The short term perspectives listed here are currently a work in progress within the library.

Conclusion and perspectives

Numerically speaking many interesting tracks could be explored, ranging from the comparison of the different interpolation and restriction methods in multiscale approaches to the use of higher order directional splitting schemes. When considering the applicative aspects of this work, many things come to mind because of the great flexibility of the numerical code developed in this work. Indeed, the HySoP library can be used as a high performance numerical toolbox for any fluid-related problem which resolution algorithm can be expressed in terms of operators. The library is modular and can be extended by implementing additional Python, C++, Fortran, or OpenCL operators.

If we stick to the original aim of this work, the next logical step is to extend the work of [Burns et al. 2015] by performing ensemble averaged three-dimensional simulations of increasing Schmidt numbers (Sc > 28). Those numerical results could eventually be used to identify the dominant settling mechanism of fine particles for Schmidt numbers encountered in the nature (Sc = 1750). While this application was focused on vertical convection, the study of horizontal convection at high Schmidt numbers is also of importance [START_REF] Cohen | Experiments on turbulent horizontal convection at high Schmidt numbers[END_REF][START_REF] Konopliv | The influence of shear on double-diffusive and settling-driven instabilities[END_REF]]. The developed method could also be used to explore many other physical problems where transported quantities, diffusing at different rates, would be coupled. This is especially the case for chemical reactions happening in turbulent flows [START_REF] Schwertfirm | A numerical approach for simulation of turbulent mixing and chemical reaction at high Schmidt numbers[END_REF][START_REF] Watanabe | Mixing and chemical reaction at high Schmidt number near turbulent/nonturbulent interface in planar liquid jet[END_REF]].

• Space of analytic functions: The space of analytic function C ω (Ω, K) is the space of inifinitely differentiable functions that are locally given by a convergent power series:

f ∈ C ω (Ω, K) ⇔ f ∈ C ∞ (Ω, K) and ∀x 0 ∈ Ω lim x→x 0 +∞ n=0 f (n) (x 0 ) n! (x -x 0 ) n = f (x 0 )
• Relation between spaces C k , C k I , D k and C ω :

C 0 I ⊃ C 0 ⊃ D 1 ⊃ C 1 I ⊃ C 1 ⊃ • • • ⊃ D k ⊃ C k I ⊃ C k ⊃ • • • ⊃ D ∞ = C ∞ I = C ∞ ⊃ C ω
• Lebesgue spaces: Let 1 ≤ p < ∞ and K ∈ {R, C}, we define the space L p (Ω) of functions for which the p-th power of the absolute value is Lebesgue integrable:

f : Ω → K ∈ L p (Ω) ⇔ f p =   Ω |f (x)| p dx   1/p

B.3 Build tools

• Numba: Numba translates Python functions to optimized machine code at runtime using the industry-standard LLVM compiler library [START_REF] Lam | Numba: A llvm-based python jit compiler[END_REF]. Numba compiled numerical algorithms in Python can approach the speeds of C or Fortran. Numba is designed to be used with NumPy arrays and functions, the source code remains pure Python while Numba handles the compilation at runtime just in time (JIT). Numerical modelling and High Performance Computing for sediment flows Abstract -The dynamic of sediment flows is a subject that covers many applications in geophysics, ranging from estuary silting issues to the comprehension of sedimentary basins. This PhD thesis deals with high resolution numerical modeling of sediment flows and implementation of the corresponding algorithms on hybrid calculators. Sedimentary flows involve multiple interacting phases, giving rise to several types of instabilities such as Rayleigh-Taylor instabilities and double diffusivity. The difficulties for the numerical simulation of these flows arise from the complex fluid/sediment interactions involving different physical scales. Indeed, these interactions are difficult to treat because of the great variability of the diffusion parameters in the two phases. When the ratio of the diffusivities, given by the Schmidt number, is too high, conventional methods show some limitations. This thesis extends the recent results obtained on the direct resolution of the transport of a passive scalar at high Schmidt number on hybrid CPU-GPU architectures and validates this approach on instabilities that occur in sediment flows. This work first reviews the numerical methods which are adapted to high Schmidt flows in order to apply effective accelerator implementation strategies and proposes an open source reference implementation named HySoP. The proposed implementation makes it possible, among other things, to simulate flows governed by the incompressible Navier-Stokes equations entirely on accelerator or coprocessor thanks to the OpenCL standard and tends towards optimal performances independently of the hardware. The numerical method and its implementation are first validated on several classical test cases and then applied to the dynamics of sediment flows which involve a two-way coupling between the transported scalars and the Navier-Stokes equations.

We show that the joint use of adapted numerical methods and their implementation on accelerator makes it possible to describe accurately, at a very reasonable cost, sediment transport for Schmidt numbers difficult to reach with other methods.

Figure 1 -

 1 Figure 1 -Le Havre, Seine estuary, France -February 23rd, 2019. Picture adapted from Sentinel Hub. It is licensed under CC BY 2.0 and contains modified Sentinel data.

  αδT g ρC p ∂δT ∂t + (u • ∇) δT = 2µ ( γ : ∇u) viscous heating + αT ∂P ∂t + (u • ∇) P work of pressure forces +k∆δT + Q (1.34a)

Figure 1 . 3 -

 13 Figure1.3 -Plot of the Hjulström curve in logarithmic scale. At high flow velocity, gravel, pebbles and boulders are transported on the sea bed whereas clay, silt and sand are already transported as a suspension at smaller velocities. The erosion velocity curve corresponds to the critical velocity where sediments begins to erode from the bed. Eroded sediments may then be transported in the carrier fluid at lower velocities until the critical settling velocity is reached, where they will deposit. Figure adapted from the original[START_REF] Hjulstrom | Studies of the morphological activity of rivers as illustrated by the River Fyris, Bulletin[END_REF]] curve.

Figure 1 . 4 -

 14 Figure 1.4 -Map of sediment laden flow regimes as a function of the volume fraction of the particles φ p ∝ dS and the ratio between particle response time τ p and the Kolmogorov time scale τ k . Here d represents the particle diameter, S the distance between two neighboring particles and τ e is the eddy turnover time. Figure adapted from[START_REF] Elghobashi | Particle-laden turbulent flows: direct simulation and closure models[END_REF]].

Figure 1 . 5 -

 15 Figure 1.5 -Example of 2D gravity current obtained by [Birman et al. 2007]. A dense fluid is released in a linearly stratified ambient of lesser density. The density contours show the formation of an advancing front, behind Kelvin-Helmholtz instabilities giving rise to strong vortices.

Figure 1 . 6 -

 16 Figure 1.6 -Effect of Rayleigh numbers on the evolution of double-diffusive salt fingers: Numerical simulations results obtained by [Sreenivas et al. 2009a] that show concentration fields of a thermohaline system at different thermal Rayleigh numbers for fixed value of R ρ = 6. Finger characteristics such as width, evolution pattern are a function of Rayleigh numbers. Image adapted from Salt fingers by Faria Rehman and licensed under CC BY-SA 4.0.

Figure 1 . 9 -

 19 Figure 1.9 -Settling velocity with respect to grain grain size. The blue zone corresponds to the dilute suspensions considered here. Figure adapted from [Sylvester 2013].

Figure 1 .

 1 Figure 1.10 -Semi-empirical model [Segre et al. 2001] for φ between 0.1 and 10%.

Figure 1 .

 1 Figure 1.13 -Treemap representing TOP500 coprocessor share. Each rectangle represent a specific accelerator, and None means that no accelerator is present (leftmost part of the plot in blue). Each subrectangle represent supercomputer of the TOP500 list. The area of the rectangles is proportional to the benchmarked HPL performance on each supercomputer.

Figure 1 .

 1 Figure 1.14 -Coprocessor share in the supercomputers containing accelerators as of June 2019. Data represents the ratio of obtained HPL performance with respect to the 134 machines having accelerator support, not the ratio of accelerator computing capabilities.

Figure 1 .

 1 Figure 1.18 -Achieved memory bandwidth versus float vector type for all devices

Figure

  Figure 1.20 -Roofline of mixbench achieved single-precision performance

Figure 2 . 1 -

 21 Figure 2.1 -Illustration of the discretization of rectangle-shaped domain B spanning from x min = (0, 0) to x max = (L y , L x ). Here P = (0, 1), the domain is L xperiodic on the x-axis. It is discretized by using N c = (5, 8) rectangle cells of size dx = (dy, dx) = (L y /N c y , L x /N c x ), yielding N v = (6, 8) vertices where the velocity u and vorticity ω fields are defined. The blue vertices are not included in the discretization because they represent redundant vertices arising from the periodization of the x-axis, as ω * ,8 = ω * ,0 .

Figure 2 . 3 -

 23 Figure 2.3 -Local interpolation by using an interpolating kernel(2.35)

Figure 2 .

 2 Figure 2.8 -One-dimensional remeshing kernel with support s = [-3, 3]

  Figure 2.9 -Illustration of some semi-lagrangian remeshing kernels Λ p,r

Figure 2 .

 2 Figure 2.10 -Stencils approximating the 2D Laplacian: Examples of five-point and nine-point centered stencils that approximate the Laplacian operator ∆ = ∂ ∂x 2 +∂ ∂y 2 on a uniform grid with second order accuracy (leading error coefficients are however not the same).

Figure 2 .

 2 Figure 2.12 -Directional splitting of the advection-remeshing procedure

Figure 2 .

 2 Figure 2.14 -Spectral convergence of 1-periodic functions of varying regularity.Here the black dotted line represents the machine error y = ε 2.22 × 10 -16 , see figure2.13.

Algorithm 8 :

 8 Algorithm to compute divergence-free vorticity and velocity on a 2D fully periodic domain discretized on N = (N y , N x ) points where N x and N y are odd.

Figure 2 .

 2 Figure 2.15 -Spectral interpolation by using the eight real-to-real transformsInterpolation of f (x) = 1 -2|x -0.5| on interval [0, 1] by using eight different sine and cosine transforms each corresponding to specific homogeneous boundary conditions. Red dots represent the five samples f j = f (x j ) used to compute the forward transform and blue dots represent implicit zeros due to the homogeneous Dirichlet boundary conditions (those points are thus not used in the forward and backward transforms). All leftmost plots represent the four collocated boundary configurations while the rightmost plots represent the four staggered boundary configurations. The function f is represented in blue and its interpolant f N obtained by evaluating (2.109) (orange). The magenta crosses are obtained efficiently with the corresponding backward transforms of roughly three times the size of the forward transform. The effective size of the forward and backward transforms is not the same in all cases(3,4,4,5) top-to-bottom for the collocated cases and (5,5,5,5) for the staggered cases.

Figure 2 .

 2 Figure 2.16 -Zoom on the symmetries exhibited by the different transforms.In each case, we look at a DCT or DST of N v = 10 input samples for the collocated case and N c = 9 for the staggered case (shown as red and purple dots). Purple dots corresponds to implicit zeros enforced by the sine or cosine basis φ p . The size of the transform N is the number of samples minus the number of collocated Dirichlet boundary conditions. The data extension, shown as gray dots, is the result of applying left and right even or odd extensions enforcing the left and right homogeneous boundary conditions. Image adapted from DCTsymmetries and DST-symmetries from Steven G. Johnson, licensed under CC BY-SA 3.0.

( a )

 a Dirichlet-Dirichlet: j ∈ 1, N -2 , the forward DST-I transform is of size N = N -2 with dx = L/(N + 1) because of implicit x 0 = x N -1 = 0. (b) Dirichlet-Neumann: j ∈ 1, N -1 , the forward DST-III transform is of size N = N -1 with dx = L/N because of implicit x 0 = 0. (c) Neumann-Dirichlet: j ∈ 0, N -2 , the forward DCT-III transform is of size N = N -1 with dx = L/N because of implicit x N -1 = 0. (d) Neumann-Neumann: j ∈ 0, N -1 , the forward DCT-I transform is of size N = N with dx = L/(N -1).

For

  odd derivatives, q ∈ 2N + 1, the left and right boundaries are swapped due to the change of basis and this may change the size of the transform in the collocated Dirichlet-Dirichlet and Neumann-Neumann cases. Here we describe the algorithm to compute odd derivatives for all collocated cases: Let d l and d r represent the setup of left and right boundaries (d l = 1 when there is a left homogeneous boundary and 0 otherwise, and similarly for the right boundary condition). For a collocated boundary setup, the size of the forward transform is N = N -d l -d r and the size of the odd derivative backward transform is N

  n and d r = (d i,r ) i∈ 1,n such that d i,l (respectively d i,r ) denote whether problem imposes a left (respectively a right) homogeneous boundary condition on the i-th axis. The size of the forward transform is M = N -d l -d r because of the implicit hyperplanes containing only zero-valued nodes due the homogeneous Dirichlet boundary conditions. Let

Figure 2 .

 2 Figure 2.18 -Convergence of the GLC Chebyshev interpolation method.See figure2.17 for the definition of functions f , g and h. Here the coefficients f and g are computed efficiently by using the DCT-I while coefficients h are computed up to N = 2 12 by scalar products because of the discontinuity of the function h(x). The black dotted line represents the machine error for double precision floating point numbers y = ε 2.22×10 -16 .

,

  T r = (T p (1)) p and T r = T N we can include the boundary conditions in L and f :

Figure 2 .

 2 Figure 2.19 -Matrix layouts associated to the different methods (N = 32).Each dot corresponds to a non-zero entry in the matrix. Coefficients are color graded from lowest (blue) to highest absolute value (red). FDCx corresponds to implicit centered finite differences of given spatial order (decentered on the boundaries when required).

Figure 2 .

 2 Figure 2.20 -Convergence of the different methods for a one-dimensional homogeneous Dirichlet Poisson problem.FDCx corresponds to implicit centered finite differences of given order, ChebC to the collocated Chebyshev method, ChebT to the Chebyshev-Tau method, ChebG to the Chebyshev-Galerkin method and DST-x to the discrete sine transforms.

Figure 3 . 1 -

 31 Figure 3.1 -HySoP logo representing an hyssop (Hyssopus officinali) plant whose seeds are transported by the wind, referring to the lagrangian nature of particle methods.

Figure 3 . 2 -

 32 Figure 3.2 -Evolution of the programming languages used in the HySoP library.

Figure 3 . 3 -

 33 Figure 3.3 -Different computing backends present in the HySoP library.

Figure 3 . 4 -

 34 Figure 3.4 -Simplified UML diagram representing the Domain class hierarchy.

  Figure 3.5 -Simplified UML diagram of the ContinuousField class hierarchy.

  Figure 3.6 -Simplified UML diagram of the DiscreteField class hierarchy.

Figure 3 . 7 -

 37 Figure 3.7 -Example of directed graph of operators obtained after step one

  /* kernel_prototypes */ __kernel __attribute__((reqd_work_group_size(512,1,1))) void copy_kernel(__global const float *restrict src, __global float *restrict dst); /* global_scope_constants */ __constant const int2 N = (int2)((+1024),(+1024)); /* kernel_declarations */ __kernel void copy_kernel(__global const float *const restrict src, __global float *const restrict dst) { const int2 lid = (int2)(get_local_id(0),get_local_id(1)); const int2 L = (int2)((+512),(+1)); const int2 G = (int2)((+512),(+512)); int2 k; for (k.y=0; k.y<N.y; k.y+=G.y) { for (k.x=lid.x; k.x<N.x; k.x+=L.x) { dst[k.x*1+k.y*N.x] = src[k.x*1+k.y*N.x]; } } }

  of given shape and data type it is possible to compute d = a + b * sin(d) on the OpenCL backend by using one of the following code generation frameworks provided by the HySoP library: • Simple elementwise operations on arrays based on the PyOpenCL stack: kernel = array_backend.nary_op((A,B,C), (D,), operation='y0[i] = x0[i]+x1[i]*sin(x2[i])', build_kernel_launcher=True) • Elementwise operations from symbolic expressions with automatic parameter tuning: As, Bs, Cs, Ds = OpenClElementwiseKernelGenerator.arrays_to_symbols(A,B,C,D) expr = Assignment(Ds, As+Bs*sympy.sin(Cs)) kernel = kernel_generator.elementwise('custom_kernel', expr) • A more general code generator requiring the implementation of a custom kernel class. class CustomKernel(KernelCodeGenerator):

Figure 3 . 9 -

 39 Figure 3.9 -Three dimensional array benchmark on CPU: Mean runtime required to compute d=a+b*sin(c) over 1024 runs where a, b, c, and d are double-precision arrays of shape (N z , N y , N x ). This computation is run on the following array backends: Numpy [Van Der Walt et al. 2011] (sin, multiply, add), Numexpr [Cooke et al. 2017], Pythran [Guelton et al. 2015] (-DUSE XSIMD -fopenmp -march=native -O2), Cython[START_REF] Behnel | Cython: The best of both worlds[END_REF]] (-fopenmp -march=native -O2), PyOpenCl[START_REF] Klöckner | PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation[END_REF]] (ElementwiseKernel), and autotuned HySoP symbolic code generator (MEASURE). The NumPy implementation is not multithreaded.

Figure 3 .

 3 Figure 3.10 -Three dimensional array benchmark on GPU: Mean runtime required to compute d=a+b*sin(c) over 1024 runs where a, b, c, and d are double-precision arrays of shape (N z , N y , N x ). This computation is run on the following array backends: PyOpenCl and PyCuda [Klöckner et al. 2012] (ElementwiseKernel), CuPy [Nishino et al. 2017] and autotuned HySoP symbolic OpenCL code generator (MEASURE). PyCuda and CuPy are CUDA based libraries whereas PyOpenCL and HySoP are based on OpenCL (using Nvidia OpenCL platforms). The kernel is compute-bound on the consumer-grade GPU.

  Figure 3.11 -Interactions between the kernel code generator and kernel autotuner

Figure 3 .

 3 Figure 3.12 -Kernel statistics obtained for 2D transposition of arrays of size 16384 2The kernel autotuner generate hundreds of different kernels which mean runtimes are measured initially over 8 runs. The resulting distribution of relative runtimes (with respect to the mean mean-runtime across all benched kernels) is represented in an histogram with log scale. Sky blue overlay correspond to kernels that have been pruned out during the procedure. Green, orange and red vertical lines represent, respectively, best, median and worst runtime.

Figure 3

 3 Figure 3.13 -Overview of the process to gather kernel execution statistics

Figure 3 .

 3 Figure 3.15 -Roofline of 2D and 3D, single-and double-precision remeshing

#

  Get associated symbolic variables xs, ts = box.s, t.s Us, Ws, nus = velocity.s, vorticity.s, nu.s # Vorticity diffusion (three scalar expressions) lhs = Ss.diff(ts) rhs = nus*laplacian(Ws, xs) print Assignment.assign(lhs, rhs) # Conservative stretching (three scalar expressions) lhs = Ws.diff(ts) rhs = div(np.outer(Us, Ws).freeze(), xs) print Assignment.assign(lhs, rhs)

Figure 3 . 20 -

 320 Figure3.20 -FFT benchmark, 2D complex-to-complex discrete Fourier transforms Comparison of different CPU and GPU compatible FFT backends. Each point represent the mean over 64 runs of complex-to-complex DFT of size N = n 2 . FFTW is compiled with ICC 19.0.5 and run with either the OpenMP or the Pthreads threading layer with planning rigor flag set to FFTW MEASURE. The POCL OpenCL platform provides support for CPUs but also Nvidia GPUs by generating directly PTX for the target device.

Figure 3 .

 3 Figure 3.21 -Distribution of runtime for single-precision Navier-Stokes on CPU

Figure 3 .

 3 Figure 3.23 -Mean runtime per timestep for the resolution of the Taylor-Green vortex problem. Results averaged over at least 20000 iterations on grids of increasing size.

( a )

 a Figure 3.24 -Evolution of enstrophy with respect to time for various configurations and comparison with reference [Hillewaert 2012]. Simulation at fixed timestep dt = 10 -3 .

( a )Figure 3

 a3 Figure 3.26 -Slice of vorticity at x=0 and comparison of isocontoursfor ω 2 (x = 0) ∈ {1, 5, 10, 20, 30} in the top-left part of the slice (represented by a green rectangle). Solution is obtained with fixed timestep dt = 10 -3 (red) are compared with the 768 3 results of [Van Rees et al. 2011] (black). This corresponds to iterations 8000 and 9000 of the numerical simulation.

( a )

 a Figure 3.27 -Evolution of enstrophy with respect to time -variable timestep

  where j represent the local index and the local strideS = (S 1 , • • • , S n ) is defined by S i = Π n j=i+1 N v j .Passing from the global index i to the local index j is made possible by the knowledge for each processes of their coordinates and the exclusive prefix sum of the local grid sizes as illustrated on figure 3.29. More specifically, the process of coordinates p = (p 1 , • • • , p n ) has its local grid offset by O

Figure 3 .

 3 Figure3.29 -Example of uniform grid distributionbetween 4 processes P (0,0) , P (0,1) , P (1,0) and P (1,1) aliased by P 0 , P 1 , P 2 and P 3 corresponding to the example presented in figure2.1. The global grid of size N v = (6, 8) is split by P = (2, 2) processes in each direction such that the local grid size handled by process of coordinates p, namely P (py,px) = P 2py+px , contains N v,(py,px) = (N v,py y , N v,px x ) = (3,4) vertices. The index present above each node correspond to the global node indexI i = i 1 N v x + i 2 = i • S with global stride S = (N vx , 1).

Figure 3 .

 3 Figure 3.30 -Extension of the subdomain with ghosts nodes: The subdomain of coordinates p = (p y , p x ) = (1, 1), discretized on a local grid of size N v,p = (N v,py y

Figure 3 .

 3 Figure 3.31 -Local grid data layout owned by process P 3 = P (1,1) as shown on figure 3.30. The full domain split by using a uniform domain decomposition with underlying local discrete grid size Nv = (N v y , N v x ) = (3,4) forming what we will call the local compute domain, represented by the green vertices. The discretization is extended on the boundaries by G = (G y , G x ) = (1, 2) additional ghosts vertices represented in grey, forming the local domain. The local grid is allocated as as single block of memory, consisting into the local domain grid of size N v + 2G which rows are eventually padded by some virtual vertices, represented in turquoise, to enforce eventual memory alignement requirements. The local to process grid data is stored contiguously using a row-major ordering as shown in red. Each local compute node v is indexed by its global index I i , represented in black, and its local index I j , represented in blue. Local and global vertex indices are linked by equations (3.6). In this case, if the elements stored in the grid would be double precision floating point numbers of 8 bytes each, the padding would ensure that each line is aligned on a 16x8=128B boundary.

Figure 3 .

 3 Figure 3.32 -Example of slab decomposition of a three-dimensional domain

Figure 3 .

 3 Figure 3.33 -Example of pencil decomposition of a three-dimensional domain

Figure 4 . 1 -

 41 Figure 4.1 -River delta of Irrawaddy, Myanmar (European Space Agency)

Figure 4 . 2 -

 42 Figure 4.2 -Two-dimensional problem configuration. Here V p corresponds to the dimensionless Stokes settling velocity of the particles while κ * correspond to scalar diffusivities.

  may exhibit double-diffusive behaviours depending on the settling velocity of the particles but also on the other constants of the problem.The physical constants of the problem are:ρ 0Fresh water density ν 0 Kinematic viscosity of fresh water α, γ Density expansion coefficients of salinity and sediments κ s , κ c Diffusion coefficients of salinity and sediments V stStokes settling velocity of the particles g = -ge z Acceleration due to gravity

u 0

 0 (x, y, z) = 0 and ω 0 (x, y, z) = 0 C 0 (x, y, z) x, y, z) = 1 -C 0 (x, y, z) ) is a small initial random perturbation that is uniformly distributed. The initial thickness of the profile l 0 represent less than 1% of the total domain height and for all this chapter we fix l 0 to 1.5. Initial sediment concentration profile C 0 (z) and salinity profile S 0 (z) are given for δ = 0 on figure 4.3.

Figure 4 . 3 -

 43 Figure 4.3 -Initial vertical sediment profile for z ∈ [-200, 200] and l 0 = 1.5.

Figure 4 . 4 -

 44 Figure 4.4 -Sketch of the different kind of instabilities the flow can develop. Figure adapted from [Burns et al. 2015]. The dotted line represent sediment interface location.

  (a) Obtain ω k,1 from ω k , u k and equation (4.3c). (b) Obtain S k+1 from S k , u k and equation (4.3d).

Figure 4 . 5 -

 45 Figure 4.5 -Snapshots of horizontally averaged sediment and salt profiles

Figure 4 . 6 -

 46 Figure 4.6 -Evolution of horizontally averaged interface locations and thicknesses

Figures 4 .

 4 Figures 4.7 and 4.8 show the sediment and salinity fields at four different times t ∈ {40, 100, 200, 300}, top-to-bottom. The domain has been truncated in the vertical direction z ∈ [-300, 300] so that our results can be compared to reference data available from[Burns et al. 2015], figure4. As it can be seen on the top frames, the interface is initially displaced by a random perturbation δ(x) but develops a definite preferred wavelength during the early development of the instability. Overall our solver seems to capture the same physical scales and interface displacements as the reference one. Here the instability is clearly double-diffusive with the emergence of fingers at the interface. As for the reference solution, we observe both positively and negatively buoyant plumes that detach from the interfaces at t = 200 and eventually collide around t = 300.

Figure 4 .

 4 Figure 4.7 -Comparison of 2D sediment slices with reference data: C(x, z, t) for x ∈ [0, 750], z ∈ [-300, 300] and t ∈ {40, 100, 200, 300}. Left slices corresponds to [Burns et al. 2015] and right slices to our results. Parameters: V p = 0.04, Sc = 0.7, R s = 2.0, τ = 25.

4. 2 .Figure 4

 24 Figure 4.8 -Comparison of 2D salt slices with reference data: S(x, z, t) for x ∈ [0, 750], z ∈ [-300, 300] and t ∈ {40, 100, 200, 300}. Left slices corresponds to [Burns et al. 2015] and right slices to our results. Parameters: V p = 0.04, Sc = 0.7, R s = 2.0, τ = 25.

  zc)/lsRatio of the nose region height to the salinity interface ls (current method) ls(Meiburg et al.)

Figure 4 . 9 -

 49 Figure 4.9 -Evolution of the ratio of the nose region height to the salinity interface thickness

  We use Ω = [z min , z max ]×[y min , y max [×[x min , x max [= [-128, +64] × [0, 128[×[0, 128[ so that the space step is dx = (dz, dy, dx) = (0.125, 0.25, 0.25).

Figure 4 .

 4 Figure 4.10 -Comparison of 3D sediment slices with reference data: C(x, y, z, t) for (x, y, z) ∈ [0, L x [×[0, L y [×{-10, +10}, and t ∈ {75, 100, 125, 150}. Left slices corresponds to reference and right slices to our results. Parameters: V p = 0.04, Sc = 7, R s = 2.0, τ = 25.

Figure 4 .

 4 Figure 4.10 shows four different snapshots of horizontal slices of the sediment concentration, above an below the initial interface position at z = ±10. Left reference slices represent a dimensionless physical surface of size [0, 100[ 2 while right slices corresponds to our results obtained on slightly larger surfaces of size [0, 128[ 2 .Above the interface at z = +10, the plots show circular spots representing the upward moving salt fingers. The only change in this structure is that more and more fingers appear with time. The first frame at z = -10 and t = 75 show downward moving sediment fingers that are similar in size, although larger in count, when compared to the upward moving salt fingers at the same period. After t = 100, sediment fingers are no longer circular but stretched by local convection zones. This is suggests that Rayleigh-Taylor instabilities dominate in the lower region of the flow. It can be better seen on the lower isocontours of the sediment concentration C = -1.5 at t = 100 that are shown on figure4.11. Exactly as for[Burns et al. 2015] we observe that the convection zones become larger with time, yielding polygon-like shapes. Overall we observe the same structures as in the reference data, but our results seem to be lagging behind with a delay of ∆ t = 15 (dimensionless time). It is believed that the amplitude of the initial perturbation δ(x, y) is the cause of such a difference. Here we use the same initialization as for the 2D case, δ(x, y) as p 0 l 0 [U(0, 1) -0.5] with p 0 = 0.1 whereas for the reference data δ has not been specified.

Figure 4 .

 4 Figure 4.11 -Comparison of 3D sediment isocontours with reference data: contours C(x, y, z) = 0.5 at t = 100 showing upward (top) and downward (down) moving fingers.

Figure 4 .

 4 Figure 4.12 -Ensemble averaged quantities at varying Schmidt number of 2D flows, averaged over 100 runs with N v = (4097, 4096) from Sc = 0.07 (top) to Sc = 70 (bottom) on domain Ω = [-1024, 1024] 2 . With this discretization, the two highest Schmidt configurations are under-resolved, dx = (0.5, 0.5), and result in positive sediment interface velocities. Parameters: V p = 0.04, Sc = {0.07, 0.7, 7, 70} , R s = 2.0, τ = 25.

  dimensional problem from t = 0 to 150 ranges from 300 to 2000 depending on the setting of dt MAX and imposed CFL constant. With dt MAX = 0.1 and C CFL = 1, the 1537 × 512 × 512 simulation takes under 6 hours on the aforementioned GPU device and can be used for Schmidt numbers up to Sc = 28 on the three-dimensional domain Ω = [-128, 64] × [0, 128] × [0, 128].

Figure 4 .

 4 Figure 4.14 -Prediction of the nose thickness ratio H/l s versus obtained results. Dots, squares, triangles and crosses correspond to the two-dimensional results obtained by [Burns et al. 2015] with parameters groups (A,B,C,D) defined in table 4.2. The superimposed plus signs correspond to our three-dimensional results obtained for parameter group (E), from smallest Schmidt number (left, Sc = 3.5) to highest Schmidt number (right, Sc = 28.0).

Figure 4 .

 4 Figure 4.15 -Example of the physical scales obtained with Sc = 7 and τ = 25.

Figure 4 .Chapter 4 .

 44 Figure 4.16 -3D sediment isocontours for varying Schmidt number: Plot of the contours C(x, y, z) = 0.5 for t ∈ {100, 125, 150} showing upward (top) and downward (bottom) moving fingers at n = 512. Parameters: V p = 0.04, Sc ∈ {3.5, 7, 14, 28} , R s = 2.0, τ = 25.

UNIVERSIT É DE GRENOBLE ALPES MSTII Mathématiques, Sciences et Technologies de l'Information, Informatique

	T H È S E
	pour obtenir le titre de
	docteur en sciences
	de l'Université Grenoble Alpes
	Spécialité : Mathématiques appliquées
	Présentée et soutenue par
	Jean-Baptiste KECK

Numerical modelling and High Performance Computing for sediment flows

			お
			箸 は 汁 物
			を 摑 み ま せ ん
			ミ シ
			ラ ン 博 士
		Thèse dirigée par Georges-Henri COTTET
		codirigée par Iraj MORTAZAVI
	préparée au sein du Laboratoire Jean Kuntzmann (LJK)
		soutenue le 13 décembre 2019
	Composition du Jury :
	Président :	Valérie PERRIER	-CNRS, Université Grenoble Alpes
	Rapporteurs :	Emmanuel JEANNOT	-INRIA, Bordeaux Sud-Ouest
		Florian DE VUYST	-Université de Technologie de Compiègne
	Examinateurs : Christophe PICARD	-CNRS, Université Grenoble Alpes
		Delphine DEPEYRAS	-Ingéliance Technologies, Mérignac
		Eckart MEIBURG	-University of California, Santa Barbara
	Directeurs :	Georges-Henri COTTET -Université Grenoble Alpes
		Iraj MORTAZAVI	-CNAM Paris
	Invités :	Chloé MIMEAU	-CNAM Paris
		Matthieu PUYO	-Ingéliance Technologies, Mérignac

« C'est pas faux ! » Perceval i Modélisation

numérique et calcul haute performance de transport de sédiments Résumé -

  La dynamique des écoulements sédimentaires est un sujet qui concerne de nombreuses applications en géophysiques, qui vont des questions d'ensablement des estuaires à la compréhension des bassins sédimentaires. Le sujet de cette thèse porte sur la modélisation numérique à haute résolution de ces écoulements et l'implémentation des algorithmes associés sur accélérateurs. Les écoulements sédimentaires font intervenir plusieurs phases qui interagissent, donnant lieu à plusieurs types d'instabilités comme les instabilités de Rayleigh-Taylor et de double diffusivité. Les difficultés pour la simulation numérique de ces écoulements tiennent à la complexité des interactions fluides/sédiments qui font intervenir des échelles physiques différentes. En effet, ces interactions sont difficiles à traiter du fait de la grande variabilité des paramètres de diffusion dans les deux phases et les méthodes classiques présentent certaines limites pour traiter les cas où le rapport des diffusivités, donné par le nombre de Schmidt, est trop élevé. Cette thèse étend les

récents résultats obtenus sur la résolution directe de la dynamique du transport d'un scalaire passif à haut Schmidt sur architecture hybride CPU-GPU et valide cette approche sur les instabilités qui interviennent dans des écoulements sédimentaires. Ce travail revisite tout d'abord les méthodes numériques adaptées aux écoulements à haut Schmidt afin de pouvoir appliquer des stratégies d'implémentations efficaces sur accélérateurs et propose une implémentation de référence open source nommée HySoP. L'implémentation proposée permet, entre autres, de simuler des écoulements régis par les équations de Navier-Stokes incompressibles entièrement sur accélérateur ou coprocesseur grâce au standard OpenCL et tend vers des performances optimales indépendamment du matériel utilisé. La méthode numérique et son implémentation sont tout d'abord validées sur plusieurs cas tests classiques avant d'être appliquées à la dynamique des écoulements sédimentaires qui font intervenir un couplage bidirectionnel entre les scalaires transportés et les équations de Navier-Stokes.

Nous montrons que l'utilisation conjointe de méthodes numériques adaptées et de leur implémentation sur accélérateur permet de décrire précisément, à coût très raisonnable, le transport sédimentaire pour des nombres de Schmidt difficilement accessibles par d'autres méthodes.

Table 1 .1 -Characteristics and compute capabilities of the considered devices:

 1 BDW corresponds to the maximal theoretical device global memory bandwidth (for CPUs , this corresponds to maximum memory frequency and all memory channels in use). For CPU devices, the second value between parenthesis corresponds to the maximal theoretical value enabled by actual memory configuration. FP32 and FP64 correspond to theoretical single and double precision peak floating point operations per second at device base clock. For GPU devices, the second value between parenthesis correspond to same values at maximal boost clock (this provides a performance boost of +38.9% for the 2080Ti and +18.6% for the V100 under sufficient cooling). CPU characteristics are estimated from Intel compliance metrics (CTP).

	Device	# H2D (GB/s) D2H (GB/s) BDW (GB/s) FP32 (GFLOPS) FP64 (GFLOPS)
	i7-9700K	1	19.4	8.9	18.1 (85.0%)	514.8 (87.6%)	260.1 (88.5%)
	E5-2695 v4 2	13.5	6.1	67.4 (49.4%) 1434.1 (96.1%)	713.1 (95.8%)
	RTX 2080Ti 1	8.98	6.1	496.13 (80.5%) 14508.6 (88.9%)	451.4 (88.5%)
	V100-SXM2	1	7.08	4.9	807.6 (89.7%) 15676.0 (88.7%) 7854.6 (88.9%)

Table 1 .2 -Peak performance metrics obtained for the four device configurations: H2D

 1

	corresponds to host-to-device memory bandwidth, D2H to device-to-host memory band-
	width, BDW to device global memory bandwidth, FP32 and FP64 to floating point opera-
	tions per second. FLOPS (floating point operations per seconds) are taken as the best result
	achieved by clpeak [Bhat 2017] for all considered vector types. The results are also given in
	percentage of theoretical peak values at actual memory bandwidth and max. boost frequency.
		Operations per byte (OP/B)	Operations per element (OP/E)
	Device	FP32 (FLOP/B) FP64 (FLOP/B) FP32 (FLOP/E) FP64 (FLOP/E)
	i7-9700K	14.1 (27.6)	7.1 (13.8)	56.5 (110.4)	56.5 (110.4)
	E5-2695 v4	9.7 (10.9)	4.9 (5.5)	38.8 (43.7)	38.8 (43.7)
	RTX 2080Ti	19.1 (26.5)	0.6 (0.8)	76.3 (106.0)	4.8 (6.6)
	V100-SXM2	16.6 (19.6)	8.3 (9.8)	66.2 (78.5)	66.2 (78.5)

Table 1 .3 -Designed arithmetic intensity for usual floating point operations:

 1

Table 2 . 2 -

 22 Finite differences coefficients for first and second derivatives For example, table 2.2 states that the first and second derivatives of f at order 4 at point x can be approximated by:

Performance Computing Contents 3.1 The HySoP library

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 3.1.1 Origin and current state of the library . . . . . . . . . . . . . . . . . . . 133 3.1.2 Design of the library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 3.1.3 HySoP from a user point of view . . . . . . . . . . . . . . . . . . . . . . . 143 3.1.4 Graph of operators and task parallelism . . . . . . . . . . . . . . . . . . 146 3.1.5 Integration of numerical methods . . . . . . . . . . . . . . . . . . . . . . 148 3.2 Implementation on hardware accelerators . . . . . . . . . . . . . . . . . 149 3.2.1 Arrays, buffers and memory handling . . . . . . . . . . . . . . . . . . . 150 3.2.2 OpenCL code generation framework . . . . . . . . . . . . . . . . . . . . . 152 3.2.3 Permutations and automatic tuning of kernel runtime parameters . . . . 159 3.2.4 Directional advection operator . . . . . . . . . . . . . . . . . . . . . . . 164 3.2.5 Finite differences operators . . . . . . . . . . . . . . . . . . . . . . . . . 172 3.2.6 Spectral transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 3.2.7 Navier-Stokes runtime distribution . . . . . . . . . . . . . . . . . . . . . 178 3.2.8 Validation of the solver . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 3.3 Distributed computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 3.3.1 Domain decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 3.3.2 Interprocess communications . . . . . . . . . . . . . . . . . . . . . . . . 187 3.3.3 Field discretization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 3.3.4 Spectral solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192

Table 3 .

 3

			HySoP
			Python User Interface
	Fortran Backend	C++ Backend	Python Backend		OpenCL Backend	CUDA Backend
	f2hysop	cpp2hysop	Numpy	Numba	pyOpenCL	pyCUDA
					POCL
	HySoP Fortran Library (f2hysop.so)	HySoP C++ Library (cpp2hysop.so)	Numpy C API	C++ (JIT)	OpenCL (JIT)	CUDA (JIT)
	F2PY	SWIG
	HySoP	HySoP		HySoP	HySoP	HySoP
	Fortran	C++		Python	OpenCL	OpenCL
	sources	sources		bytecode	sources	Code Generator

1 -Comparison of some features between hysop-origin and hysop-current.

  Those fifty lines of Python solve a fully periodic two-dimensional Navier-Stokes equation with fixed timestep, discretized on a Cartesian grid of size N c = (128, 128) entirely on the OpenCL compute backend (default OpenCL platform and device are assumed here). It solves the double shear layer case presented in

	:
	data[...] = U(*coords) if (component==0) else V(*coords)
	def init_vorticity(data, coords, component):
	data[...] = np.nan_to_num(W(*coords))
	from hysop import Box, Simulation, Problem, MPIParams
	from hysop.constants import Implementation
	from hysop.defaults import VelocityField, VorticityField, TimeParameters
	from hysop.operators import (DirectionalAdvection, DirectionalDiffusion,
	StrangSplitting, PoissonCurl)
	ndim, npts = 2, (128,128)
	#> Define domain, default MPI communicator and common operator parameters
	box = Box(dim=ndim)
	mpi_params = MPIParams(comm=box.task_comm)
	extra_params = {'mpi_params': mpi_params, 'implementation': Implementation.OPENCL}
	#> Variables (time, timestep, velocity, vorticity)
	t, dt = TimeParameters()
	velo = VelocityField(box)
	vorti = VorticityField(velo)
	#> Directionally split advection and diffusion of vorticity (second order Strang)
	advection_dir = DirectionalAdvection(velocity=velo, advected_fields=vorti, dt=dt,
	velocity_cfl=2, variables={velo: npts, vorti: npts}, name='adv', **extra_params)
	diffusion_dir = DirectionalDiffusion(fields=vorti, coeffs=1e-4, dt=dt,
	variables={vorti: npts}, name='diff', **extra_params)
	splitting = StrangSplitting(ndim, order=2)
	splitting.push_operators(advection_dir, diffusion_dir)
	#> Poisson operator to recover the velocity from the vorticity
	poisson = PoissonCurl(velocity=velo, vorticity=vorti, name='poisson',
	variables={velo:npts, vorti: npts}, **extra_params)
	#> Create and setup the problem (graph of operators)
	problem = Problem()
	problem.insert(splitting, poisson)
	problem.build()
	#> Create a simulation, initialize fields and solve the problem
	simu = Simulation(start=0.0, end=1.25, dt0=1e-2, t=t, dt=dt)
	problem.initialize_field(velo, formula=init_velocity)
	problem.initialize_field(vorti, formula=init_vorticity)
	problem.solve(simu)

Table 3 .3 -Performance of transposition on a quad socket NUMA architecture 164 Chapter 3. Implementation and High Performance Computing

 3

Table 4 .1 -Ensemble averaged statistics for different Schmidt numbers

 4

. Parameters are averaged over 100 runs for V p = 0.04, R s = 2, τ = 25 and S c ∈ {0.07, 0.07, 7, 70}.

Table 4 .2 -Dimensionless group values and grid properties for the parametric study

 4

. The first four classes of parameters (A,B,C,D) corresponds to the 15 two-dimensional setups of

[Burns et al. 2015

] that have non-vanishing settling velocities. The last class (E) corresponds to our three-dimensional cases with increasing Schmidt numbers up to Sc = 28.

Table 4 .3 -Obtained nose region ratio for three-dimensional test cases (E).

 4

	l s

  pyFFTW is a pythonic wrapper around FFTW. The ultimate aim is to present a unified interface for all the possible transforms that FFTW can perform. Both the complex DFT and the real DFT are supported. Real to real transforms are not supported in the master branch yet (but are available as a merge request). This wrapper does not support the MPI capabilities of FFTW. Python wrapper for the OpenCL FFT library clFFT. It is designed to tightly integrate with PyOpenCL and consists of a low-level Cython based wrapper with an interface similar to the underlying C library. On top of that it offers a high-level interface designed to work on data contained in instances of pyopencl.array.Array. The high-level interface takes some inspiration from pyFFTW. The mkl fft Python module wraps the MKL-FFT library and provides numpy.fft and scipy.fftpack compatible FFT interfaces to NumPy and SciPy. Python module for manipulation and statistical analysis of graphs. The core data structures and algorithms are implemented in C++, making extensive use of template metaprogramming, based heavily on the Boost Graph Library. An extensive array of features is included, such as support for arbitrary vertex, edge or graph properties and topological aglorithms.

	• pyFFTW:
	< +∞
	225

Link: https://github.com/pyFFTW/pyFFTW • gpyfft: gpyfft is a Link: https://github.com/geggo/gpyfft • mkl fft: Link: https://github.com/IntelPython/mkl_fft • graph-tool: graph-tool is is an efficient Link: https://graph-tool.skewed.de

Table C .1 -2D and 3D single-precision directional advection kernel statistics.

 C Numba generates specialized code for different array data types and layouts to optimize performance. It supports vector instructions and adapts to the CPU capabilities (SSE, AVX, AVX-512) and enable GPU acceleration through CUDA and ROCm. The purpose of F2PY, the Fortran to Python interface generator, is to provide a connection between Python and Fortran languages[START_REF] Peterson | F2PY: a tool for connecting Fortran and Python programs[END_REF]]. F2PY is part of NumPy and is also available as a standalone command line tool. It makes it possible to call Fortran 77/90/95 external subroutines and Fortran 90/95 module subroutines as well as C functions and allows access to Fortran 77 COMMON blocks and Fortran 90/95 module data, including allocatable arrays.Benchmark of advection kernels on theNVidia Tesla V100-SXM2 GPU. Kernel parameter P represent the number of simultaneously advected particles (vectorization) and S determines whether the shared memory version of the kernel was chosen by the autotuner or not.

	Kernel Configuration global size local size P S	global size local size P S (1024, 2048) (1024, 2048) (1024, 1) 1 (1024, 1) 1 (1024, 4096) (512, 4096) (512, 1) 1 (1024, 1) 1 (1024, 8192) (512, 8192) (512, 1) 1 (1024, 1) 1 (1024, 8192) (512, 16384) (512, 1) 1 (1024, 1) 1 (512, 2048) (512, 2048) (512, 1) 1 (512, 1) 1 (1024, 4096) (256, 4096) (256, 1) 1 (1024, 1) 1 (1024, 8192) (512, 8192) (512, 1) 1 (1024, 1) 1 (1024, 8192) (512, 16384) (512, 1) 1 (1024, 1) 1 (512, 2048) (1024, 2048) (1024, 1) 1 (512, 1) 1 (512, 4096) (256, 4096) (256, 1) 1 (512, 1) 1 (512, 8192) (512, 8192) (512, 1) 1 (512, 1) 1 (1024, 8192) (512, 16384) (512, 1) 1 (1024, 1) 1	(1024, 2048) (1024, 1) 1 (512, 2048) (512, 1) 1	(256, 4096) (256, 1) 1 (512, 4096) (512, 1) 1	(256, 8192) (256, 1) 1 (512, 8192) (512, 1) 1	(256, 16384) (256, 1) 1 (512, 4096) (512, 1) 1	(1024, 2048) (1024, 1) 1 (512, 2048) (512, 1) 1	(256, 4096) (256, 1) 1 (512, 4096) (512, 1) 1	(256, 8192) (256, 1) 1 (512, 8192) (512, 1) 1	(256, 16384) (256, 1) 1 (512, 4096) (512, 1) 1	global size local size P S global size local size P S	(64, 16, 64) (64, 1, 1) 1 (64, 16, 64) (64, 1, 1) 1	(128, 64, 128) (128, 1, 1) 1 (64, 128, 128) (64, 1, 1) 1	(128, 256, 256) (128, 1, 1) 1 (128, 256, 256) (128, 1, 1) 1	(256, 128, 512) (256, 1, 1) 1 (512, 256, 512) (512, 1, 1) 1	(64, 16, 64) (64, 1, 1) 1 (64, 32, 64) (64, 1, 1) 1	(128, 64, 128) (128, 1, 1) 1 (64, 64, 128) (64, 1, 1) 1	(128, 256, 256) (128, 1, 1) 1 (128, 128, 256) (128, 1, 1) 1	(512, 128, 512) (512, 1, 1) 1 (512, 256, 512) (512, 1, 1) 1	(64, 32, 64) (64, 1, 1) 2 (64, 16, 64) (128, 32, 128) (128, 1, 1) 1 (64, 1, 1) 1 (128, 128, 128) (128, 1, 1) 1 (128, 256, 256) (128, 1, 1) 1 (256, 256, 256) (256, 1, 1) 1 (256, 128, 512) (256, 1, 1) 1 (512, 128, 512) (512, 1, 1) 1 (64, 16, 64) (64, 1, 1) 1 (64, 32, 64) (128, 32, 128) (128, 1, 1) 1 (64, 1, 1) 1 (128, 128, 128) (128, 1, 1) 1 (128, 128, 256) (128, 1, 1) 1 (256, 256, 256) (256, 1, 1) 1 (256, 128, 512) (256, 1, 1) 1 (512, 512, 512) (512, 1, 1) 1 (32, 32, 64) (32, 1, 1) 1 (64, 32, 64) (128, 32, 128) (128, 1, 1) 1 (64, 1, 1) 1 (128, 128, 128) (128, 1, 1) 1 (128, 128, 256) (128, 1, 1) 1 (256, 256, 256) (256, 1, 1) 1 (256, 128, 512) (256, 1, 1) 1 (512, 512, 512) (512, 1, 1) 1 global size local size P S	global size local size P S
	Mean Achieved Performance BDW (%) GFLOPS (FP64) FLOP/B	BDW (%) GFLOPS (FP32) FLOP/B 77.0% 66.9% 1202.50 (8.1%) 2.0 691.22 (9.3%) 1.0 84.1% 72.3% 1175.35 (7.9%) 1.8 708.18 (9.5%) 0.9 85.2% 77.2% 1233.16 (8.3%) 1.8 693.77 (9.3%) 0.9 84.1% 77.3% 1222.91 (8.2%) 1.8 672.92 (9.0%) 0.9 75.8% 64.1% 1724.63 (11.6%) 3.0 1019.68 (13.7%) 1.5 81.4% 70.4% 1828.36 (12.3%) 2.9 1096.84 (14.7%) 1.5 84.0% 74.7% 1945.37 (13.1%) 2.9 1108.26 (14.9%) 1.5 83.1% 75.8% 1964.90 (13.2%) 2.9 1084.14 (14.6%) 1.5 72.3% 59.7% 2683.20 (18.0%) 5.0 1580.16 (21.2%) 2.4 79.4% 67.3% 2875.13 (19.3%) 4.7 1711.91 (23.0%) 2.4 82.0% 72.3% 3098.34 (20.8%) 4.8 1758.39 (23.6%) 2.4 80.8% 73.5% 3140.59 (21.1%) 4.7 1736.06 (23.3%) 2.4	56.0% 3078.87 (20.7%) 6.1 71.4% 1921.06 (25.8%) 3.0	64.3% 3392.58 (22.8%) 5.9 76.9% 2047.33 (27.5%) 3.0	70.1% 3687.21 (24.7%) 5.8 80.8% 2140.00 (28.7%) 2.9	71.5% 3756.32 (25.2%) 5.8 79.4% 2097.52 (28.2%) 2.9	56.2% 3467.51 (23.3%) 6.9 71.0% 2149.37 (28.9%) 3.4	63.1% 3750.95 (25.2%) 6.6 78.6% 2355.51 (31.6%) 3.3	70.2% 4161.02 (27.9%) 6.6 80.6% 2403.80 (32.3%) 3.3	70.9% 4200.76 (28.2%) 6.6 79.2% 2357.46 (31.6%) 3.3	BDW (%) GFLOPS (FP32) FLOP/B BDW (%) GFLOPS (FP64) FLOP/B	26.0% 538.57 (3.6%) 2.3 49.1% 508.65 (6.8%) 1.2	61.2% 1356.85 (9.1%) 2.5 75.0% 735.91 (9.9%) 1.1	77.4% 1544.04 (10.4%) 2.2 85.4% 851.73 (11.4%) 1.1	79.5% 1463.93 (9.8%) 2.0 86.9% 974.00 (13.1%) 1.2	23.1% 705.77 (4.7%) 3.4 47.5% 751.80 (10.1%) 1.8	58.9% 1892.97 (12.7%) 3.6 65.5% 929.08 (12.5%) 1.6	75.0% 2242.32 (15.0%) 3.3 82.9% 1193.75 (16.0%) 1.6	78.0% 2447.29 (16.4%) 3.5 86.1% 1399.34 (18.8%) 1.8	22.7% 1214.26 (8.1%) 5.9 45.1% 56.7% 2701.87 (18.1%) 5.3 1057.83 (14.2%) 2.6 70.3% 70.4% 3274.36 (22.0%) 5.2 1791.24 (24.0%) 2.8 80.6% 76.2% 3445.49 (23.1%) 5.0 2068.91 (27.8%) 2.9 84.3% 23.8% 1352.33 (9.1%) 6.3 2030.66 (27.3%) 2.7 45.1% 54.0% 3112.21 (20.9%) 6.4 1303.95 (17.5%) 3.2 71.1% 68.1% 3771.86 (25.3%) 6.2 2164.81 (29.1%) 3.4 82.4% 74.5% 4117.36 (27.6%) 6.1 2528.57 (33.9%) 3.4 83.3% 21.7% 1285.02 (8.6%) 6.6 2566.18 (34.4%) 3.4 44.7% 54.2% 3484.04 (23.4%) 7.1 1438.48 (19.3%) 3.6 71.6% 70.6% 4376.88 (29.4%) 6.9 2419.91 (32.5%) 3.8 82.8% 74.2% 4595.66 (30.8%) 6.9 2818.26 (37.8%) 3.8 83.9% BDW (%) GFLOPS (FP32) FLOP/B 2867.50 (38.5%) 3.8	BDW (%) GFLOPS (FP64) FLOP/B
	Runtime (us) Mean Memory Bandwidth mean min max private local global	mean min max private local global 97.09 55.81 54.27 64.51 52.4TiB/s 841.0GiB/s 561.0GiB/s 95.23 104.45 37.7TiB/s 966.9GiB/s 645.0GiB/s 355.36 206.98 204.80 210.94 44.3TiB/s 908.3GiB/s 606.3GiB/s 353.28 359.42 35.7TiB/s 1.0TiB/s 704.9GiB/s 1402.59 1400.83 1409.02 775.49 773.12 780.29 45.1TiB/s 969.7GiB/s 647.3GiB/s 33.4TiB/s 1.0TiB/s 714.4GiB/s 5684.51 5675.01 5701.63 3100.51 3095.55 3110.91 44.0TiB/s 970.1GiB/s 647.6GiB/s 31.1TiB/s 1.0TiB/s 705.0GiB/s 98.72 58.37 56.32 68.61 55.4TiB/s 1.3TiB/s 537.5GiB/s 97.28 101.38 43.4TiB/s 1.5TiB/s 635.6GiB/s 367.10 213.34 211.97 218.11 54.7TiB/s 1.4TiB/s 590.5GiB/s 365.57 372.74 46.7TiB/s 1.7TiB/s 682.3GiB/s 1423.01 1419.26 1426.43 802.05 795.65 807.94 58.2TiB/s 1.5TiB/s 625.8GiB/s 45.5TiB/s 1.7TiB/s 704.1GiB/s 5756.74 5743.62 5769.22 3159.23 3154.94 3166.21 58.0TiB/s 1.5TiB/s 635.5GiB/s 43.1TiB/s 1.7TiB/s 696.2GiB/s 103.52 62.53 61.44 64.51 71.0TiB/s 1.7TiB/s 500.8GiB/s 102.40 106.50 53.1TiB/s 2.1TiB/s 606.1GiB/s 377.31 223.20 221.18 225.28 66.2TiB/s 1.9TiB/s 564.4GiB/s 373.76 380.93 55.7TiB/s 2.3TiB/s 665.2GiB/s 1459.81 1457.15 1464.32 828.48 818.18 838.66 71.4TiB/s 2.1TiB/s 605.9GiB/s 56.2TiB/s 2.3TiB/s 687.7GiB/s 5914.34 5904.38 5925.89 3258.66 3254.27 3269.63 71.5TiB/s 2.1TiB/s 616.1GiB/s 55.0TiB/s 2.3TiB/s 677.6GiB/s	66.75 64.51 77.82 77.4TiB/s 2.1TiB/s 469.1GiB/s 104.80 102.40 107.52 63.0TiB/s 2.6TiB/s 598.7GiB/s	233.66 231.42 236.54 75.8TiB/s 2.4TiB/s 539.1GiB/s 389.25 385.02 392.19 65.4TiB/s 2.8TiB/s 644.8GiB/s	857.70 845.82 868.35 81.6TiB/s 2.6TiB/s 587.5GiB/s 1481.73 1477.63 1491.97 67.4TiB/s 3.0TiB/s 677.5GiB/s	3363.20 3355.65 3369.98 82.8TiB/s 2.6TiB/s 599.3GiB/s 6030.94 6017.02 6047.74 65.3TiB/s 2.9TiB/s 665.8GiB/s	66.53 65.54 69.63 78.4TiB/s 2.1TiB/s 470.6GiB/s 105.38 102.40 107.52 63.6TiB/s 2.6TiB/s 595.4GiB/s	238.18 234.50 241.66 75.1TiB/s 2.3TiB/s 528.9GiB/s 381.06 378.88 388.10 67.8TiB/s 2.9TiB/s 658.6GiB/s	856.80 852.99 864.26 82.6TiB/s 2.6TiB/s 588.1GiB/s 1486.62 1480.70 1497.09 68.2TiB/s 3.0TiB/s 675.3GiB/s	3390.78 3383.30 3400.70 82.9TiB/s 2.6TiB/s 594.4GiB/s 6049.15 6037.50 6064.13 66.0TiB/s 2.9TiB/s 663.8GiB/s	mean min max private local global mean min max private local global	9.25 8.19 12.29 26.5TiB/s 323.4GiB/s 217.8GiB/s 9.79 9.22 12.29 30.1TiB/s 610.9GiB/s 411.4GiB/s	30.91 29.70 33.79 73.6TiB/s 766.1GiB/s 513.4GiB/s 51.30 50.18 55.30 50.1TiB/s 932.9GiB/s 628.3GiB/s	195.58 193.54 197.63 87.0TiB/s 968.7GiB/s 649.1GiB/s 354.56 353.28 360.45 58.0TiB/s 1.0TiB/s 716.1GiB/s	1512.77 1507.33 1520.64 69.9TiB/s 996.7GiB/s 666.2GiB/s 2756.00 2753.54 2764.80 63.3TiB/s 1.1TiB/s 728.5GiB/s	10.40 9.22 16.38 27.9TiB/s 475.4GiB/s 193.7GiB/s 10.11 9.22 13.31 40.6TiB/s 977.8GiB/s 398.4GiB/s	32.13 30.72 37.89 82.2TiB/s 1.2TiB/s 493.9GiB/s 58.69 56.32 70.66 47.0TiB/s 1.3TiB/s 549.1GiB/s	202.02 198.66 206.85 98.7TiB/s 1.5TiB/s 628.4GiB/s 365.41 362.50 370.69 60.4TiB/s 1.7TiB/s 694.9GiB/s	1535.62 1531.90 1543.17 96.9TiB/s 1.6TiB/s 653.7GiB/s 2781.54 2778.11 2786.30 75.5TiB/s 1.8TiB/s 721.8GiB/s	10.59 10.24 15.36 36.7TiB/s 668.4GiB/s 190.2GiB/s 10.66 33.38 32.77 36.86 81.3TiB/s 1.6TiB/s 475.5GiB/s 9.22 14.34 41.3TiB/s 1.3TiB/s 53.86 215.20 212.99 219.14 107.1TiB/s 2.0TiB/s 589.9GiB/s 52.22 57.34 85.7TiB/s 2.0TiB/s 373.02 1577.66 1571.84 1583.10 97.6TiB/s 2.2TiB/s 638.8GiB/s 370.69 378.88 99.0TiB/s 2.3TiB/s 2842.11 2836.48 2851.84 10.08 9.22 14.34 38.2TiB/s 878.0GiB/s 199.8GiB/s 79.2TiB/s 2.4TiB/s 10.65 35.04 33.79 39.94 87.9TiB/s 2.0TiB/s 452.9GiB/s 10.24 13.31 52.1TiB/s 1.6TiB/s 53.28 222.40 219.14 230.40 105.5TiB/s 2.5TiB/s 570.8GiB/s 52.22 58.37 97.1TiB/s 2.6TiB/s 364.93 1613.60 1606.66 1620.99 110.0TiB/s 2.7TiB/s 624.6GiB/s 363.52 367.62 113.4TiB/s 3.0TiB/s 2876.64 2857.98 2889.73 115.1TiB/s 11.42 10.24 15.36 32.3TiB/s 780.0GiB/s 181.7GiB/s 3.1TiB/s 10.75 34.91 33.79 46.08 88.9TiB/s 2.0TiB/s 454.5GiB/s 10.24 14.34 52.2TiB/s 1.6TiB/s 52.86 214.66 211.97 218.11 110.1TiB/s 2.6TiB/s 591.4GiB/s 51.20 55.30 98.8TiB/s 2.6TiB/s 363.14 1620.90 1615.87 1627.14 110.4TiB/s 2.7TiB/s 621.8GiB/s 360.45 371.71 115.0TiB/s 3.0TiB/s 2855.20 2844.67 2864.13 117.0TiB/s mean min max private local global 3.1TiB/s 378.0GiB/s 589.3GiB/s 675.4GiB/s 706.5GiB/s 378.1GiB/s 595.7GiB/s 690.4GiB/s 698.0GiB/s 374.7GiB/s 600.4GiB/s 693.8GiB/s 703.2GiB/s	mean min max private local global
	Configuration integrator discretization	integrator discretization Euler Euler (2048, 2048) (2048, 2048) Euler Euler (4096, 4096) (4096, 4096) Euler Euler (8192, 8192) (8192, 8192) Euler Euler (16384, 16384) (16384, 16384) RK2 RK2 (2048, 2048) (2048, 2048) RK2 RK2 (4096, 4096) (4096, 4096) RK2 RK2 (8192, 8192) (8192, 8192) RK2 RK2 (16384, 16384) (16384, 16384) RK3 RK3 (2048, 2048) (2048, 2048) RK3 RK3 (4096, 4096) (4096, 4096) RK3 RK3 (8192, 8192) (8192, 8192) RK3 RK3 (16384, 16384) (16384, 16384)	RK4 (2048, 2048) RK4 (2048, 2048)	RK4 (4096, 4096) RK4 (4096, 4096)	RK4 (8192, 8192) RK4 (8192, 8192)	RK4 (16384, 16384) RK4 (16384, 16384)	RK4 38 (2048, 2048) RK4 38 (2048, 2048)	RK4 38 (4096, 4096) RK4 38 (4096, 4096)	RK4 38 (8192, 8192) RK4 38 (8192, 8192)	RK4 38 (16384, 16384) RK4 38 (16384, 16384)	integrator discretization integrator discretization	Euler (64, 64, 64) Euler (64, 64, 64)	Euler (128, 128, 128) Euler (128, 128, 128)	Euler (256, 256, 256) Euler (256, 256, 256)	Euler (512, 512, 512) Euler (512, 512, 512)	RK2 (64, 64, 64) RK2 (64, 64, 64)	RK2 (128, 128, 128) RK2 (128, 128, 128)	RK2 (256, 256, 256) RK2 (256, 256, 256)	RK2 (512, 512, 512) RK2 (512, 512, 512)	RK3 (64, 64, 64) RK3 RK3 (128, 128, 128) (64, 64, 64) RK3 RK3 (256, 256, 256) (128, 128, 128) RK3 RK3 (512, 512, 512) (256, 256, 256) RK3 RK4 (64, 64, 64) (512, 512, 512) RK4 RK4 (128, 128, 128) (64, 64, 64) RK4 RK4 (256, 256, 256) (128, 128, 128) RK4 RK4 (512, 512, 512) (256, 256, 256) RK4 RK4 38 (64, 64, 64) (512, 512, 512) RK4 38 RK4 38 (128, 128, 128) (64, 64, 64) RK4 38 RK4 38 (256, 256, 256) (128, 128, 128) RK4 38 RK4 38 (512, 512, 512) (256, 256, 256) RK4 38 integrator discretization (512, 512, 512)	integrator discretization

Link: https://numba.pydata.org • F2PY: Link: https://docs.scipy.org/doc/numpy/f2py

Table C .2 -2D and 3D double-precision directional advection kernel statistics.

 C Benchmark of advection kernels on theNVidia Tesla V100-SXM2 GPU. Kernel parameter P represent the number of simultaneously advected particles (vectorization) and S determines whether the shared memory version of the kernel was chosen by the autotuner or not.

	Kernel Configuration	global size local size P A	(256, 2048) (256, 1) 2	(512, 4096) (512, 1) 2	(256, 8192) (256, 1) 1	(1024, 16384) (1024, 1) 2	(512, 2048) (512, 1) 2	(512, 4096) (512, 1) 2	(256, 8192) (256, 1) 1	(1024, 16384) (1024, 1) 2	(512, 2048) (512, 1) 1	(512, 4096) (512, 1) 2	(256, 8192) (256, 1) 1	(512, 16384) (512, 1) 1	(512, 2048) (512, 1) 1	(256, 4096) (256, 1) 1	(256, 8192) (256, 1) 1	(512, 16384) (512, 1) 1	global size local size P A	(64, 32, 32) (64, 1, 1) 1	(64, 128, 128) (64, 1, 1) 2	(64, 256, 256) (64, 1, 1) 1	(64, 512, 512) (64, 1, 1) 2	(64, 64, 32) (64, 1, 1) 1	(64, 128, 128) (64, 1, 1) 2	(64, 256, 256) (64, 1, 1) 1	(64, 512, 512) (64, 1, 1) 2	(64, 32, 32) (64, 1, 1) 1	(128, 128, 128) (128, 1, 1) 1	(64, 256, 256) (64, 1, 1) 1	(128, 512, 512) (128, 1, 1) 1	(64, 16, 64) (64, 1, 1) 1	(128, 128, 128) (128, 1, 1) 1	(64, 256, 256) (64, 1, 1) 1	(128, 512, 512) (128, 1, 1) 1	global size local size P A
	Mean Achieved Performance	BDW (%) GFLOPS (FP32) FLOP/B	77.1% 1270.86 (8.5%) 1.83	83.4% 1374.61 (9.2%) 1.83	86.6% 1427.78 (9.6%) 1.83	86.6% 1428.48 (9.6%) 1.83	76.2% 3139.79 (21.1%) 4.58	83.6% 3446.79 (23.1%) 4.58	86.2% 3553.73 (23.9%) 4.58	85.5% 3527.62 (23.7%) 4.58	72.8% 7299.94 (49.0%) 11.15	81.0% 8136.03 (54.6%) 11.16	85.1% 8550.43 (57.4%) 11.16	84.3% 8467.79 (56.8%) 11.16	71.5% 8987.79 (60.3%) 13.97	78.0% 9822.73 (65.9%) 13.99	84.1% 10595.18 (71.1%) 13.99	83.1% 10465.59 (70.2%) 14.00	BDW (%) GFLOPS (FP32) FLOP/B	35.0% 559.70 (3.8%) 1.78	70.0% 1137.96 (7.6%) 1.81	87.9% 1439.09 (9.7%) 1.82	90.0% 1478.86 (9.9%) 1.83	34.7% 1373.66 (9.2%) 4.40	69.8% 2820.41 (18.9%) 4.49	87.5% 3573.66 (24.0%) 4.54	89.3% 3665.94 (24.6%) 4.56	33.8% 3228.32 (21.7%) 10.61	68.3% 6688.37 (44.9%) 10.88	86.1% 8545.75 (57.4%) 11.02	88.3% 8815.85 (59.2%) 11.09	30.5% 3612.22 (24.2%) 13.18	66.3% 8095.62 (54.3%) 13.58	85.0% 10546.02 (70.8%) 13.78	86.2% 10779.47 (72.3%) 13.89	BDW (%) GFLOPS (FP32) FLOP/B
	Runtime (us) Mean Memory Bandwidth	mean min max private local global	72.61 70.66 80.90 23.5TiB/s 435.4GiB/s 646.2GiB/s	268.51 266.24 279.55 25.4TiB/s 468.3GiB/s 698.6GiB/s	1034.05 1030.14 1043.46 30.9TiB/s 494.9GiB/s 725.5GiB/s	4134.18 4128.77 4142.08 24.9TiB/s 485.2GiB/s 725.7GiB/s	73.47 71.68 83.97 32.5TiB/s 428.7GiB/s 638.8GiB/s	267.71 265.22 281.60 32.8TiB/s 470.6GiB/s 700.8GiB/s	1038.62 1035.26 1052.67 38.0TiB/s 496.5GiB/s 722.3GiB/s	4185.25 4177.92 4196.35 32.1TiB/s 479.7GiB/s 716.9GiB/s	76.99 75.78 86.02 47.0TiB/s 413.8GiB/s 609.8GiB/s	276.32 273.41 286.72 44.2TiB/s 456.8GiB/s 679.1GiB/s	1051.71 1048.58 1059.84 50.2TiB/s 494.0GiB/s 713.4GiB/s	4247.90 4240.38 4255.74 49.6TiB/s 480.0GiB/s 706.4GiB/s	78.40 76.80 88.06 52.1TiB/s 407.9GiB/s 599.1GiB/s	286.94 281.60 301.06 53.1TiB/s 456.0GiB/s 654.1GiB/s	1064.10 1059.84 1074.18 56.6TiB/s 491.9GiB/s 705.2GiB/s	4309.09 4298.75 4318.21 55.9TiB/s 475.0GiB/s 696.4GiB/s	mean min max private local global	10.30 9.22 21.50 21.6TiB/s 207.3GiB/s 293.2GiB/s	40.54 38.91 51.20 38.4TiB/s 403.4GiB/s 587.1GiB/s	256.48 254.98 267.26 47.7TiB/s 533.1GiB/s 736.8GiB/s	1996.67 1993.73 2007.04 33.7TiB/s 524.3GiB/s 754.2GiB/s	10.50 9.22 20.48 27.9TiB/s 209.3GiB/s 290.8GiB/s	40.90 38.91 51.20 44.1TiB/s 405.9GiB/s 585.0GiB/s	258.21 257.02 268.29 54.7TiB/s 544.6GiB/s 733.7GiB/s	2013.66 2012.16 2022.40 40.9TiB/s 527.6GiB/s 748.8GiB/s	10.88 10.24 19.46 28.0TiB/s 207.5GiB/s 283.3GiB/s	42.02 40.96 51.20 75.8TiB/s 400.9GiB/s 572.3GiB/s	263.07 262.14 272.38 66.4TiB/s 549.4GiB/s 722.0GiB/s	2040.10 2036.74 2048.00 68.3TiB/s 528.5GiB/s 740.0GiB/s	12.19 11.26 25.60 26.9TiB/s 190.2GiB/s 255.3GiB/s	43.52 41.98 52.22 78.6TiB/s 392.7GiB/s 555.4GiB/s	267.26 266.24 279.55 72.5TiB/s 555.4GiB/s 712.5GiB/s	2091.81 2087.94 2104.32 73.8TiB/s 522.9GiB/s 722.7GiB/s	mean min max private local global
	Configuration	remesh discretization	Λ 2,1 (2048, 2048)	Λ 2,1 (4096, 4096)	Λ 2,1 (8192, 8192)	Λ 2,1 (16384, 16384)	Λ 4,2 (2048, 2048)	Λ 4,2 (4096, 4096)	Λ 4,2 (8192, 8192)	Λ 4,2 (16384, 16384)	Λ 6,4 (2048, 2048)	Λ 6,4 (4096, 4096)	Λ 6,4 (8192, 8192)	Λ 6,4 (16384, 16384)	Λ 8,4 (2048, 2048)	Λ 8,4 (4096, 4096)	Λ 8,4 (8192, 8192)	Λ 8,4 (16384, 16384)	remesh discretization	Λ 2,1 (64, 64, 64)	Λ 2,1 (128, 128, 128)	Λ 2,1 (256, 256, 256)	Λ 2,1 (512, 512, 512)	Λ 4,2 (64, 64, 64)	Λ 4,2 (128, 128, 128)	Λ 4,2 (256, 256, 256)	Λ 4,2 (512, 512, 512)	Λ 6,4 (64, 64, 64)	Λ 6,4 (128, 128, 128)	Λ 6,4 (256, 256, 256)	Λ 6,4 (512, 512, 512)	Λ 8,4 (64, 64, 64)	Λ 8,4 (128, 128, 128)	Λ 8,4 (256, 256, 256)	Λ 8,4 (512, 512, 512)	remesh discretization

Table C .3 -2D and 3D single-precision directional remesh kernel statistics.

 C NVidia Tesla V100-SXM2 GPU. Kernel parameter P represent the number of remeshed particles and A that atomic intrinsics are used.

	BDW (%) GFLOPS (FP64) FLOP/B global size local size P A	76.9% 633.89 (8.5%) 0.9 (256, 256) (256, 1) 2	82.7% 681.94 (9.2%) 0.9 (128, 512) (128, 1) 2	77.6% 640.35 (8.6%) 0.9 (1024, 4096) (1024, 1) 2	78.6% 648.37 (8.7%) 0.9 (1024, 8192) (1024, 1) 2	75.4% 1553.32 (20.8%) 2.3 (32, 2048) (32, 1) 2	81.4% 1677.47 (22.5%) 2.3 (128, 512) (128, 1) 2	80.5% 1659.59 (22.3%) 2.3 (64, 1024) (64, 1) 2	78.7% 1622.06 (21.8%) 2.3 (32, 2048) (32, 1) 2	72.7% 3646.94 (49.0%) 5.6 (512, 2048) (512, 1) 2	79.5% 3990.15 (53.6%) 5.6 (128, 512) (128, 1) 2	79.7% 4004.31 (53.7%) 5.6 (64, 1024) (64, 1) 2	77.6% 3899.96 (52.3%) 5.6 (32, 2048) (32, 1) 2	72.2% 4539.29 (60.9%) 7.0 (512, 2048) (512, 1) 2	77.2% 4856.66 (65.2%) 7.0 (512, 4096) (512, 1) 2	78.7% 4954.95 (66.5%) 7.0 (64, 1024) (64, 1) 2	77.0% 4847.17 (65.1%) 7.0 (32, 2048) (32, 1) 2	BDW (%) GFLOPS (FP64) FLOP/B global size local size P A	54.9% 439.57 (5.9%) 0.9 (32, 32, 64) (32, 1, 1) 2	76.5% 621.46 (8.3%) 0.9 (64, 128, 128) (64, 1, 1) 2	86.1% 705.12 (9.5%) 0.9 (128, 256, 256) (128, 1, 1) 2	86.6% 711.38 (9.5%) 0.9 (256, 256, 512) (256, 1, 1) 2	54.7% 1083.08 (14.5%) 2.2 (32, 32, 64) (32, 1, 1) 2	76.6% 1547.65 (20.8%) 2.2 (32, 128, 128) (32, 1, 1) 2	84.2% 1718.67 (23.1%) 2.3 (128, 256, 256) (128, 1, 1) 2	85.0% 1744.13 (23.4%) 2.3 (256, 256, 512) (256, 1, 1) 2	50.3% 2402.03 (32.2%) 5.3 (32, 32, 64) (32, 1, 1) 2	74.8% 3662.15 (49.2%) 5.4 (64, 128, 128) (64, 1, 1) 2	83.4% 4138.71 (55.6%) 5.5 (128, 256, 256) (128, 1, 1) 2	84.4% 4215.66 (56.6%) 5.5 (256, 256, 512) (256, 1, 1) 2	47.6% 2820.20 (37.9%) 6.6 (32, 64, 64) (32, 1, 1) 2	74.1% 4525.30 (60.7%) 6.8 (64, 128, 128) (64, 1, 1) 2	82.3% 5104.63 (68.5%) 6.9 (128, 256, 256) (128, 1, 1) 2	83.9% 5247.33 (70.4%) 6.9 (256, 256, 512) (256, 1, 1) 2	BDW (%) GFLOPS (FP64) FLOP/B global size local size P A	Benchmark of remeshing kernels Λ p,r on the
	mean min max private local global	145.57 142.34 157.70 15.9TiB/s 434.4GiB/s 644.7GiB/s	541.25 535.55 549.89 16.8TiB/s 472.7GiB/s 693.2GiB/s	2305.60 2297.86 2316.29 16.5TiB/s 435.0GiB/s 650.7GiB/s	9108.29 8866.82 9209.86 16.3TiB/s 440.4GiB/s 658.8GiB/s	148.51 145.41 156.67 21.3TiB/s 473.4GiB/s 632.1GiB/s	550.08 543.74 559.10 22.8TiB/s 468.7GiB/s 682.2GiB/s	2224.03 2206.72 2245.63 22.5TiB/s 477.7GiB/s 674.7GiB/s	9101.98 9018.37 9196.54 22.0TiB/s 494.4GiB/s 659.3GiB/s	154.11 152.58 165.89 33.3TiB/s 409.5GiB/s 609.3GiB/s	563.42 557.06 587.78 33.6TiB/s 461.0GiB/s 666.1GiB/s	2245.73 2229.25 2263.04 33.7TiB/s 480.1GiB/s 668.2GiB/s	9223.26 9133.06 9280.51 32.8TiB/s 501.4GiB/s 650.7GiB/s	155.23 152.58 163.84 38.6TiB/s 407.3GiB/s 605.1GiB/s	580.35 577.54 599.04 39.9TiB/s 435.8GiB/s 646.8GiB/s	2275.36 2262.02 2300.93 39.3TiB/s 480.7GiB/s 659.6GiB/s	9303.81 9230.34 9376.77 38.5TiB/s 510.5GiB/s 645.1GiB/s	mean min max private local global	13.12 12.29 26.62 15.9TiB/s 325.6GiB/s 460.6GiB/s	74.24 72.70 86.02 26.4TiB/s 440.7GiB/s 641.3GiB/s	523.46 521.22 532.48 29.9TiB/s 488.8GiB/s 722.0GiB/s	4150.82 4145.15 4159.49 25.6TiB/s 487.5GiB/s 725.6GiB/s	13.31 12.29 24.58 19.8TiB/s 330.1GiB/s 458.5GiB/s	74.53 72.70 88.06 27.4TiB/s 471.7GiB/s 642.1GiB/s	536.90 532.48 547.84 35.6TiB/s 480.2GiB/s 705.7GiB/s	4232.48 4221.95 4248.58 31.6TiB/s 479.9GiB/s 712.5GiB/s	14.62 13.31 26.62 24.9TiB/s 308.8GiB/s 421.5GiB/s	76.74 75.78 88.06 41.6TiB/s 439.1GiB/s 626.8GiB/s	543.20 539.65 555.01 47.0TiB/s 478.2GiB/s 699.3GiB/s	4266.27 4254.72 4278.27 43.3TiB/s 477.9GiB/s 707.8GiB/s	15.62 14.34 30.72 29.2TiB/s 297.0GiB/s 398.7GiB/s	77.86 76.80 89.09 46.6TiB/s 439.0GiB/s 620.9GiB/s	552.16 548.86 562.18 52.5TiB/s 474.0GiB/s 689.8GiB/s	4297.15 4286.46 4310.02 49.5TiB/s 476.3GiB/s 703.6GiB/s	mean min max private local global
	remesh discretization	Λ 2,1 (2048, 2048)	Λ 2,1 (4096, 4096)	Λ 2,1 (8192, 8192)	Λ 2,1 (16384, 16384)	Λ 4,2 (2048, 2048)	Λ 4,2 (4096, 4096)	Λ 4,2 (8192, 8192)	Λ 4,2 (16384, 16384)	Λ 6,4 (2048, 2048)	Λ 6,4 (4096, 4096)	Λ 6,4 (8192, 8192)	Λ 6,4 (16384, 16384)	Λ 8,4 (2048, 2048)	Λ 8,4 (4096, 4096)	Λ 8,4 (8192, 8192)	Λ 8,4 (16384, 16384)	remesh discretization	Λ 2,1 (64, 64, 64)	Λ 2,1 (128, 128, 128)	Λ 2,1 (256, 256, 256)	Λ 2,1 (512, 512, 512)	Λ 4,2 (64, 64, 64)	Λ 4,2 (128, 128, 128)	Λ 4,2 (256, 256, 256)	Λ 4,2 (512, 512, 512)	Λ 6,4 (64, 64, 64)	Λ 6,4 (128, 128, 128)	Λ 6,4 (256, 256, 256)	Λ 6,4 (512, 512, 512)	Λ 8,4 (64, 64, 64)	Λ 8,4 (128, 128, 128)	Λ 8,4 (256, 256, 256)	Λ 8,4 (512, 512, 512)	remesh discretization

Table C .4 -2D and 3D double-precision directional remesh kernel statistics.

 C NVidia Tesla V100-SXM2 GPU. Kernel parameter P represent the number of remeshed particles and A that atomic intrinsics are used.

	Mean Achieved Performance Kernel Configuration	BDW (%) GFLOPS (FP32) FLOP/B global size local size V S	76.9% 1294.59 (8.7%) 1.87 (256, 512, 32) (256, 1, 1) 2	73.4% 1439.93 (9.7%) 2.18 (64, 512, 512) (64, 1, 1) 2	64.5% 1658.28 (11.1%) 2.86 (256, 512, 32) (256, 1, 1) 2	60.9% 1734.53 (11.6%) 3.16 (64, 512, 256) (64, 1, 1) 2	67.3% 2301.70 (15.4%) 3.80 (64, 512, 128) (64, 1, 1) 2	51.9% 2344.54 (15.7%) 5.02 (64, 512, 64) (64, 1, 1) 2	62.0% 3293.99 (22.1%) 5.90 (64, 512, 512) (64, 1, 1) 1	60.5% 4258.28 (28.6%) 7.82 (128, 512, 256) (128, 1, 1) 1	55.4% 3930.50 (26.4%) 7.88 (64, 512, 256) (64, 1, 1) 1	52.8% 5321.90 (35.7%) 11.20 (128, 512, 64) (128, 1, 1) 1	44.1% 5382.66 (36.1%) 13.56 (128, 512, 512) (128, 1, 1) 1	35.5% 5072.08 (34.0%) 15.88 (128, 512, 256) (128, 1, 1) 1	54.6% 4285.57 (28.8%) 8.71 (64, 512, 64) (64, 1, 1) 1	52.1% 5680.45 (38.1%) 12.12 (128, 512, 32) (128, 1, 1) 1	41.3% 5383.14 (36.1%) 14.47 (128, 512, 256) (128, 1, 1) 1	34.2% 5167.33 (34.7%) 16.79 (128, 512, 64) (128, 1, 1) 1	BDW (%) GFLOPS (FP64) FLOP/B global size local size V S	83.7% 704.77 (9.5%) 0.94 (256, 512, 512) (256, 1, 1) 2	80.9% 861.22 (11.6%) 1.18 (256, 512, 512) (256, 1, 1) 2	79.7% 1025.52 (13.8%) 1.43 (256, 512, 512) (256, 1, 1) 2	78.2% 1177.92 (15.8%) 1.67 (256, 512, 512) (256, 1, 1) 2	56.0% 855.72 (11.5%) 1.70 (32, 512, 512) (32, 1, 1) 2	46.5% 941.26 (12.6%) 2.25 (32, 512, 512) (32, 1, 1) 2	34.7% 968.81 (13.0%) 3.10 (32, 512, 256) (32, 1, 1) 2	33.1% 1103.62 (14.8%) 3.71 (32, 512, 512) (32, 1, 1) 2	35.1% 1278.07 (17.2%) 4.05 (32, 512, 512) (32, 1, 1) 2	28.4% 1342.64 (18.0%) 5.25 (32, 512, 512) (32, 1, 1) 2	25.9% 1499.15 (20.1%) 6.44 (64, 512, 128) (64, 1, 1) 2	28.4% 2029.71 (27.2%) 7.94 (128, 512, 256) (128, 1, 1) 1	35.0% 1421.85 (19.1%) 4.51 (32, 512, 512) (32, 1, 1) 2	28.3% 1452.13 (19.5%) 5.71 (32, 512, 512) (32, 1, 1) 2	26.9% 1668.46 (22.4%) 6.90 (64, 512, 512) (64, 1, 1) 2	28.6% 2160.17 (29.0%) 8.39 (128, 512, 512) (128, 1, 1) 1	Benchmark of remeshing kernels Λ p,r on the
	Configuration Runtime (us) Mean Memory Bandwidth	integrator order mean min max private local global	Euler FDC2 1555.14 1549.31 1569.79 49.0TiB/s 2.2TiB/s 644.3GiB/s	Euler FDC4 1631.20 1627.14 1643.52 44.4TiB/s 2.7TiB/s 615.4GiB/s	Euler FDC6 1861.57 1855.49 1872.90 49.4TiB/s 2.9TiB/s 540.3GiB/s	Euler FDC8 1973.18 1970.18 1988.61 42.5TiB/s 3.3TiB/s 510.8GiB/s	RK2 FDC2 1778.53 1775.62 1789.95 54.5TiB/s 3.7TiB/s 564.5GiB/s	RK2 FDC4 2318.50 2311.17 2326.53 49.8TiB/s 3.9TiB/s 434.7GiB/s	RK2 FDC6 1945.63 1943.55 1954.82 108.1TiB/s 5.5TiB/s 520.0GiB/s	RK2 FDC8 2001.47 1996.80 2015.23 134.3TiB/s 7.0TiB/s 507.4GiB/s	RK4 FDC2 2168.38 2165.76 2176.00 104.8TiB/s 4.9TiB/s 464.8GiB/s	RK4 FDC4 2295.01 2290.69 2304.00 138.8TiB/s 7.2TiB/s 442.5GiB/s	RK4 FDC6 2767.81 2765.82 2774.02 143.3TiB/s 7.8TiB/s 369.8GiB/s	RK4 FDC8 3466.53 3463.17 3476.48 130.6TiB/s 7.7TiB/s 297.5GiB/s	RK4 38 FDC2 2200.13 2196.48 2209.79 102.6TiB/s 4.8TiB/s 458.1GiB/s	RK4 38 FDC4 2327.36 2320.38 2335.74 137.2TiB/s 7.1TiB/s 436.4GiB/s	RK4 38 FDC6 2954.56 2952.19 2966.53 132.2TiB/s 7.3TiB/s 346.4GiB/s	RK4 38 FDC8 3597.44 3585.02 3608.58 124.7TiB/s 7.4TiB/s 286.7GiB/s	integrator order mean min max private local global	Euler FDC2 2856.64 2852.86 2865.15 41.7TiB/s 2.4TiB/s 701.5GiB/s	Euler FDC4 2961.09 2953.22 2970.62 43.5TiB/s 3.0TiB/s 678.1GiB/s	Euler FDC6 3010.18 3004.42 3023.87 46.0TiB/s 3.6TiB/s 668.3GiB/s	Euler FDC8 3076.51 3069.95 3084.29 48.2TiB/s 4.1TiB/s 655.2GiB/s	RK2 FDC2 4274.08 4261.89 4283.39 25.0TiB/s 2.8TiB/s 469.8GiB/s	RK2 FDC4 5169.02 5156.86 5183.49 25.0TiB/s 3.2TiB/s 389.9GiB/s	RK2 FDC6 6961.57 6940.67 6983.68 23.8TiB/s 3.4TiB/s 290.7GiB/s	RK2 FDC8 7327.36 7310.34 7348.22 26.3TiB/s 3.9TiB/s 277.2GiB/s	RK4 FDC2 6852.29 6837.25 6870.02 28.3TiB/s 3.4TiB/s 294.2GiB/s	RK4 FDC4 8522.08 8504.32 8546.30 28.7TiB/s 3.9TiB/s 238.4GiB/s	RK4 FDC6 9445.31 9426.94 9459.71 31.0TiB/s 4.6TiB/s 216.7GiB/s	RK4 FDC8 8662.56 8657.92 8670.21 61.3TiB/s 6.1TiB/s 238.1GiB/s	RK4 38 FDC2 6867.33 6852.61 6888.45 28.8TiB/s 3.4TiB/s 293.5GiB/s	RK4 38 FDC4 8572.74 8541.18 8595.46 28.9TiB/s 3.9TiB/s 236.9GiB/s	RK4 38 FDC6 9090.21 9078.78 9105.41 33.4TiB/s 4.7TiB/s 225.2GiB/s	RK4 38 FDC8 8605.41 8599.55 8616.96 63.0TiB/s 6.2TiB/s 239.7GiB/s

Table C .5 -3D single-and double-precision directional diffusion kernel statistics.

 C Benchmark of diffusion on arrays of 512 3 elements on the NVidia Tesla V100-SXM2 GPU. Kernel parameter V represent the vectorization and S that shared memory is used.

	global size local size V S	(64, 512, 512) (64, 1, 1) 1	(128, 512, 256) (128, 1, 1) 1	(64, 512, 512) (64, 1, 1) 1	(64, 512, 512) (64, 1, 1) 1	(64, 512, 256) (64, 1, 1) 2	(64, 512, 256) (64, 1, 1) 1	(64, 512, 512) (64, 1, 1) 1	(128, 512, 512) (128, 1, 1) 1	(64, 512, 256) (64, 1, 1) 1	(128, 512, 128) (128, 1, 1) 1	(128, 512, 64) (128, 1, 1) 1	(128, 512, 64) (128, 1, 1) 1	(64, 512, 128) (64, 1, 1) 1	(128, 512, 128) (128, 1, 1) 1	(128, 512, 64) (128, 1, 1) 1	(128, 512, 64) (128, 1, 1) 1	global size local size V S	(512, 512, 512) (512, 1, 1) 1	(64, 512, 256) (64, 1, 1) 1	(512, 512, 512) (512, 1, 1) 1	(64, 512, 512) (64, 1, 1) 1	(64, 512, 512) (64, 1, 1) 1	(64, 512, 128) (64, 1, 1) 1	(64, 512, 512) (64, 1, 1) 1	(128, 512, 64) (128, 1, 1) 1	(64, 512, 256) (64, 1, 1) 1	(128, 512, 64) (128, 1, 1) 1	(128, 512, 32) (128, 1, 1) 1	(128, 512, 256) (128, 1, 1) 1	(64, 512, 256) (64, 1, 1) 1	(128, 512, 128) (128, 1, 1) 1	(128, 512, 256) (128, 1, 1) 1	(128, 512, 256) (128, 1, 1) 1
		80.9% 716.96 (4.8%) 0.98	79.4% 1068.14 (7.2%) 1.49	78.3% 1393.15 (9.4%) 1.98	77.7% 1725.50 (11.6%) 2.47	42.2% 819.54 (5.5%) 2.16	46.5% 1357.54 (9.1%) 3.24	42.8% 1671.74 (11.2%) 4.34	44.9% 2447.16 (16.4%) 6.05	45.7% 1942.52 (13.0%) 4.72	41.3% 2852.40 (19.1%) 7.67	34.1% 3080.95 (20.7%) 10.04	28.3% 3148.82 (21.1%) 12.38	46.5% 2210.84 (14.8%) 5.28	41.0% 3058.76 (20.5%) 8.29	34.0% 3257.88 (21.9%) 10.65	28.0% 3274.09 (22.0%) 12.98	BDW (%) GFLOPS (FP64) FLOP/B	84.3% 410.26 (5.5%) 0.54	80.3% 534.95 (7.2%) 0.74	80.5% 751.13 (10.1%) 1.04	81.8% 908.56 (12.2%) 1.23	67.0% 642.28 (8.6%) 1.07	58.7% 856.50 (11.5%) 1.62	52.5% 1026.35 (13.8%) 2.17	49.6% 1350.57 (18.1%) 3.03	56.0% 1190.30 (16.0%) 2.36	47.7% 1647.96 (22.1%) 3.84	40.3% 1820.80 (24.4%) 5.02	34.4% 1917.35 (25.7%) 6.19	55.1% 1310.49 (17.6%) 2.64	47.5% 1771.70 (23.8%) 4.14	40.0% 1915.91 (25.7%) 5.33	33.9% 1980.19 (26.6%) 6.49
		Euler FDC2 6645.73 6642.69 6656.00 49.5TiB/s 1.4TiB/s 678.3GiB/s	Euler FDC4 6785.41 6774.78 6795.26 65.6TiB/s 2.2TiB/s 665.5GiB/s	Euler FDC6 6888.38 6882.30 6895.62 78.5TiB/s 3.1TiB/s 656.7GiB/s	Euler FDC8 6961.76 6956.03 6980.61 92.9TiB/s 3.9TiB/s 650.9GiB/s	RK2 FDC2 12774.18 12744.70 12832.77 28.8TiB/s 1.9TiB/s 353.5GiB/s	RK2 FDC4 11617.06 11575.30 11650.05 70.0TiB/s 3.1TiB/s 390.1GiB/s	RK2 FDC6 12685.22 12656.64 12726.27 83.6TiB/s 3.9TiB/s 358.4GiB/s	RK2 FDC8 12121.06 12105.73 12140.54 120.7TiB/s 5.7TiB/s 376.4GiB/s	RK4 FDC2 11849.73 11812.86 11880.45 79.4TiB/s 3.4TiB/s 383.1GiB/s	RK4 FDC4 13222.24 13204.48 13255.68 119.5TiB/s 5.5TiB/s 346.2GiB/s	RK4 FDC6 16162.18 16131.07 16195.58 130.2TiB/s 6.4TiB/s 285.7GiB/s	RK4 FDC8 19650.02 19612.67 19695.62 134.0TiB/s 6.7TiB/s 237.0GiB/s	RK4 38 FDC2 11640.93 11588.61 11680.77 80.9TiB/s 3.4TiB/s 389.9GiB/s	RK4 38 FDC4 13317.50 13297.66 13337.60 119.0TiB/s 5.5TiB/s 343.8GiB/s	RK4 38 FDC6 16211.36 16180.22 16241.66 130.2TiB/s 6.3TiB/s 284.8GiB/s	RK4 38 FDC8 19820.54 19780.61 19885.06 133.1TiB/s 6.7TiB/s 234.9GiB/s	integrator order mean min max private local global	Euler FDC2 12758.98 12730.37 12782.59 43.6TiB/s 1.5TiB/s 706.6GiB/s	Euler FDC4 13422.98 13412.35 13436.93 35.2TiB/s 2.3TiB/s 672.8GiB/s	Euler FDC6 13401.54 13390.85 13419.52 59.0TiB/s 3.1TiB/s 675.1GiB/s	Euler FDC8 13221.44 13209.60 13236.22 54.2TiB/s 4.1TiB/s 685.4GiB/s	RK2 FDC2 16090.85 16072.70 16110.59 40.7TiB/s 2.8TiB/s 561.3GiB/s	RK2 FDC4 18412.93 18384.90 18440.19 49.2TiB/s 3.9TiB/s 492.2GiB/s	RK2 FDC6 20662.05 20622.34 20709.38 57.3TiB/s 4.8TiB/s 440.1GiB/s	RK2 FDC8 21962.69 21924.86 21996.54 72.9TiB/s 6.3TiB/s 415.5GiB/s	RK4 FDC2 19338.21 19295.23 19363.84 56.0TiB/s 4.1TiB/s 469.4GiB/s	RK4 FDC4 22885.92 22849.54 22929.41 78.3TiB/s 6.4TiB/s 400.1GiB/s	RK4 FDC6 27347.74 27272.19 27399.17 87.0TiB/s 7.5TiB/s 337.7GiB/s	RK4 FDC8 32270.72 32228.35 32302.08 92.3TiB/s 8.2TiB/s 288.6GiB/s	RK4 38 FDC2 19638.59 19610.62 19680.26 55.7TiB/s 4.1TiB/s 462.3GiB/s	RK4 38 FDC4 22992.03 22963.20 23011.33 78.6TiB/s 6.4TiB/s 398.2GiB/s	RK4 38 FDC6 27566.40 27531.26 27604.99 87.2TiB/s 7.5TiB/s 335.0GiB/s	RK4 38 FDC8 32771.71 32734.21 32804.86 91.3TiB/s 8.1TiB/s 284.2GiB/s

Table C .6 -3D single-and double-precision directional stretching kernel statistics.

 C elements on the NVidia Tesla V100-SXM2 GPU. Kernel parameter V represent the vectorization and S that shared memory is used.

	Modélisation	Benchmark of stretching on arrays
		of 512 3

numérique et calcul haute performance de transport de sédiments Résumé -

  La dynamique des écoulements sédimentaires est un sujet qui concerne de nombreuses applications en géophysiques, qui vont des questions d'ensablement des estuaires à la compréhension des bassins sédimentaires. Le sujet de cette thèse porte sur la modélisation numérique à haute résolution de ces écoulements et l'implémentation des algorithmes associés sur accélérateurs. Les écoulements sédimentaires font intervenir plusieurs phases qui interagissent, donnant lieu à plusieurs types d'instabilités comme les instabilités de Rayleigh-Taylor et de double diffusivité. Les difficultés pour la simulation numérique de ces écoulements tiennent à la complexité des interactions fluides/sédiments qui font intervenir des échelles physiques différentes. En effet, ces interactions sont difficiles à traiter du fait de la grande variabilité des paramètres de diffusion dans les deux phases et les méthodes classiques présentent certaines limites pour traiter les cas où le rapport des diffusivités, donné par le nombre de Schmidt, est trop élevé. Cette thèse étend les récents résultats obtenus sur la résolution directe de la dynamique du transport d'un scalaire passif à haut Schmidt sur architecture hybride CPU-GPU et valide cette approche sur les instabilités qui interviennent dans des écoulements sédimentaires. Ce travail revisite tout d'abord les méthodes numériques adaptées aux écoulements à haut Schmidt afin de pouvoir appliquer des stratégies d'implémentations efficaces sur accélérateurs et propose une implémentation de référence open source nommée HySoP. L'implémentation proposée permet, entre autres, de simuler des écoulements régis par les équations de Navier-Stokes incompressibles entièrement sur accélérateur ou coprocesseur grâce au standard OpenCL et tend vers des performances optimales indépendamment du matériel utilisé. La méthode numérique et son implémentation sont tout d'abord validées sur plusieurs cas tests classiques avant d'être appliquées à la dynamique des écoulements sédimentaires qui font intervenir un couplage bidirectionnel entre les scalaires transportés et les équations de Navier-Stokes. Nous montrons que l'utilisation conjointe de méthodes numériques adaptées et de leur implémentation sur accélérateur permet de décrire précisément, à coût très raisonnable, le transport sédimentaire pour des nombres de Schmidt difficilement accessibles par d'autres méthodes.

	Mots clés : Transport sédimentaire, couplage bidirectionnel, méthodes particulaires,
	double diffusivité, HPC, GPU, HySoP
	Laboratoire Jean Kuntzmann
	Bâtiment IMAG -Université Grenoble Alpes
	700 Avenue Centrale
	Campus de Saint Martin d'Hères
	38401 Domaine Universitaire de Saint-Martin-d'Hères
	France

Computational Fluid Dynamics

Lagrangian-Eulerian

(a) Norm of vorticity -t=8.0 (b) Norm of vorticity -t=9.0 (c) Top view -t=8.0 (d) Top view -t=9.0 (e) Side view of the upper part of the compute domain -t=8.0 (f) Side view of the upper part of the compute domain -t=9.0 Figure 3.28 -Volume rendering of the norm of the vorticity ω ≥ 10

We fit C (z, t) against C f it (z, t) = 0.5 1 + erf z -z c (t) l c (t)to obtain z c (t) and l c (t).

We fit S (z, t) against S f it (z, t) = 0.5 1 -erf z -z s (t)l s (t) to obtain z s (t) and l s (t).

Figure 4.13 -3D sediment isocontours at moderate Schmidt number S c = 7 : Plot of the contours C(x, y, z) = 0.5 for t ∈ {75, 100, 125, 150} (top-to-bottom) showing upward (left) and downward (right) moving sediment fingers. Color represent vertical velocity. Parameters: V p = 0.04, Sc = 7, R s = 2.0, τ = 25.

Remerciements Acknowledgments

The author thanks Professor Eckart Meiburg and Peter Burns for their valuable inputs about the numerical aspects of their model. The author would also like to thank Jean-Matthieu Etancelin, Chloé Mimeau and Franck Pérignon for numerous helpful discussions around the HySoP library and its development. The author also thanks Delphine Depeyras, Matthieu Puyo, Quentin Desbonnets, Florent Braure and more generally the Ingeliance company (https://www.ingeliance.com) for their interest in this work and for providing additional financial support through the establishment of consulting duties (dispositif de doctoratconseil).

This work has been supported by the ANR MPARME project funded by grant ANR-17-CE23-0024-01 of the programme Investissements d'Avenir supervised by the Agence Nationale pour la Recherche. Some of the computations presented in this work have been performed using the GRICAD infrastructure (https://gricad.univ-grenoble-alpes.fr), which is partly supported by the Equip@Meso project funded by grant ANR-10-EQPX-29-01.

Chapter 3. Implementation and High Performance Computing

The kernel autotuner gives the following optimal parameters on the different devices for two and three-dimensional arrays of size 16384 2 Sediment-laden fresh water above salt water

Contents

Appendix A

Mathematical notations

A.1 Function spaces

The class of infinitely differentiable functions D ∞ (Ω, K) s defined the intersection of the sets D k (Ω, K) as k varies over the non-negative integers.

• Spaces of piecewise continously differentiable functions: Let k ∈ N * , we define the space consisting of functions f ∈ D k (Ω, K) whose k-th derivative is piecewise continuous as C k I (Ω, K). The class of infinitely piecewise continuously differentiable functions C ∞ I (Ω, K) s defined the intersection of the sets C k I (Ω, K) as k varies over the non-negative integers.

• Spaces of continuously differentiable functions: Let k ∈ N * , we define the space consisting of functions f ∈ C k I (Ω, K) whose k-th derivative is continuous as C k (Ω, K). The class of infinitely continuously differentiable functions C ∞ (Ω, K) is defined the intersection of the sets C k (Ω, K) as k varies over the non-negative integers.

Appendix A. Mathematical notations

A.2 Functions and symbols

• Set of integers: Let (a, b) ∈ Z 2 such that a < b, we define the following integer sets:

• Indicator function: The indicator function of a set K is defined as:

A.3 Vector operations

Let n ∈ N, a ∈ R n and b ∈ R n we use the following notations:

A.4. Vector calculus identities

227

A.4 Vector calculus identities

The following vector calculus identities are required to derive the Navier-Stokes equations. In those identities ϕ represent a scalar field, u and v are vector fields and A is a tensor field.

Divergence formulas

Divergence theorem (Green-Ostrogradski)

The divergence theorem states that the outward flux of a tensor field through a closed surface is equal to the volume integral of the divergence over the region inside the surface. It relates the flow of a vector field through a surface to the behavior of the tensor field inside the surface:

HySoP dependencies and tools

This appendix provides some information and references about the dependencies the HySoP library uses. Il also covers basic tools that are usefull to build the library and analyze code metrics. It is better used with the electronic version of this work through hyperlinks.

B.1 FFT libraries

• FFTPACK: FFTPACK is a package of Fortran and C subroutines for the fast Fourier transform (FFT). It includes complex, real, sine, cosine, and quarter-wave single and double precision one-dimensional transforms [START_REF] Swarztrauber | Vectorizing the ffts[END_REF]]. Link: https://www.netlib.org/fftpack

• FFTW: The Fastest Fourier Transform in the West is a C subroutine library for computing the discrete Fourier transform (DFT) in one or more dimensions, of arbitrary input size, and of both real and complex data [START_REF] Frigo | FFTW: An adaptive software architecture for the FFT[END_REF]]. It can also computes discrete cosine and sine transforms (DCT/DST). The library supports single and double precision floating point numbers as well as extended precision long double and non standard float128 quadruple-precision floating point types. It provides two different multithreaded implementations using OpenMP and POSIX threads (pthread) in addition to distributed-memory transforms through MPI. Benchmarks comparing the performance of FFTW to other FFT implementations can be found in [Frigo et al. 2012]. Link: http://www.fftw.org

• MKL-FFT: Intel specific FFT implementation for multidimensional complex-tocomplex, real-to-complex, and real-to-real transforms of arbitrary length. • LLVM: The LLVM Project is a collection of modular and reusable compiler and toolchain technologies [START_REF] Lattner | LLVM: A compilation framework for lifelong program analysis & transformation[END_REF]]. The LLVM Core libraries provide a modern source and target-independent optimizer, along with code generation support for many popular CPUs. These libraries are built around a well specified code representation known as the LLVM intermediate representation (LLVM IR). Link: https://llvm.org

B.4 Benchmarking and debugging tools

• clpeak: clpeak is a synthetic benchmarking tool to measure peak capabilities of OpenCL devices [START_REF] Bhat | clpeak: A tool which profiles OpenCL devices to find their peak capacities[END_REF]]. It measures peak metrics that can be achieved using vector operations such as global memory bandwidth, host-to-device and device-to-host memory bandwidth, peak compute for single and double precision floating point numbers and kernel launch latencies. Link: https://github.com/krrishnarraj/clpeak

• mixbench: The purpose of mixbench is to evaluate performance bounds of GPUs on mixed operational intensity kernels [START_REF] Konstantinidis | A practical performance model for compute and memory bound GPU kernels[END_REF]. The executed kernels are customized on a range of different operational intensity values. Modern GPUs are able to hide memory latency by switching execution to threads able to perform compute operations. Using this tool one can assess the practical optimum balance in both types of operations for GPUs with CUDA, OpenCL or HIP. Link: https://github.com/ekondis/mixbench

• gearshifft: This is a simple and easy extensible benchmark system to answer the question, which FFT library performs best under which conditions [START_REF] Staniforth | Semi-Lagrangian integration schemes for atmospheric models-A review[END_REF]. Conditions are given by compute architecture, inplace or outplace as well as real or complex transforms, data precision, and so on. This project supports FFTW, MKL-FFT, clFFT, rocFFT and cuFFT libraries. Link: https://github.com/mpicbg-scicomp/gearshifft

• Oclgrind: Oclgrind implements a virtual OpenCL device simulator [START_REF] Price | Oclgrind: An extensible OpenCL device simulator[END_REF]. and aims to provide a platform for creating tools to aid OpenCL development. This project implements utilities for debugging memory access errors, detecting data-races and barrier divergence and for interactive OpenCL kernel debugging. The simulator is built on an interpreter for LLVM IR. Oclgrind provides a simple plugin interface that allows third-party developers to extend its functionality. This interface allows a plugin to be notified when various events occur within the simulator, such as an instruction being executed, memory being accessed, or work-group synchronisation constructs occurring. Link: https://github.com/jrprice/Oclgrind Appendix C

Oclgrind plugin and kernel statistics

This plugin aims to output execution statistics about OpenCL kernels run in isolation in Oclgrind. It is heavily inspired from the InstructionCounter plugin already provided by the device simulator and is thus subject to the same 3-clause BSD license (https://github. com/jrprice/Oclgrind/blob/master/LICENSE). This plugin has been tested with LLVM 6.0.1 and Oclgrind 18.3.

C.1 Requirements, build and usage

The plugin consists into a single source file with associated header (InstructionDumper.h and InstructionDumper.cpp) which are compiled to a shared library (libOclgrindInstructionDumper.so).

To build the plugin, it is required to copy oclgrind core directory (core/*.h and core/*.cpp) in the current directory. It also requires LLVM to be present at LLVM DIR, a c++17 compatible compiler defined as CXX and boost progress and property tree headers to be present in include path. Please refer to the online documentation to obtain more information about the plugin build process (https://github.com/jrprice/Oclgrind/wiki/Creating-Plugins).

The simplest way to get the source code and a working plugin is to clone the repository: #!/usr/bin/env bash export CXX=... export LLVM_DIR=... git clone https://gitlab.com/keckj/oclgrind-instruction-dumper make Oclgrind can be built from the cloned submodule or directly installed from the system package manager (sudo apt-get install oclgrind). A basic kernel simulation file is present in the repository in the test subdirectory. After build the test can be run with make test (see https://gitlab.com/keckj/oclgrind-instruction-dumper).

The general command to run the plugin on an arbitrary kernel is the following: oclgrind-kernel --plugins $(pwd)/libOclgrindInstructionDumper.so kernel.sim Although the plugin supports multithreaded execution, kernel isolation remains a slow process for large kernels (its runs exclusively on CPU). When execution is too long, instruction statistics can be extrapolated from the first work-group by defining the OCLGRIND ONLY FIRST WORKGROUP environment variable (its value is not checked).

C.2 Output sample and obtained kernel statistics

Here is a sample of statistics that are output by the plugin for the three dimensional directional remeshing kernel with Λ 8,4 on a 512 3 single-precision array. The values are printed to the standard output stream correspond only to the first work-group (out of 512 2 work-groups). The output has been formatted to fit the page: { "op_counts": { "Add": "9604", "Sub": "3072", "Mul": "3072", "FSub": "512", "FMul": "8192", "Alloca": "18944", "GetElementPtr": "10988", "Load": "101400", "Store": "39288", "Trunc": "10928", "BitCast": "3072", "ZExt": "4608", "SExt": "3692", "ICmp": "3900", "FCmp": "5120", "PHI": "512", "Call": "48256", "ExtractElement": "11068", "InsertElement": "384", "Ret": "5760", "Br": "28820" }, "function_calls": { "lambda_8_4__0_86b2": "512", "lambda_8_4__1_86b2": "512", "lambda_8_4__2_86b2": "512", "lambda_8_4__3_86b2": "512", "lambda_8_4__4_86b2": "512", "lambda_8_4__5_86b2": "512", "lambda_8_4__6_86b2": "512", "lambda_8_4__7_86b2": "512", "lambda_8_4__8_86b2": "512", "lambda_8_4__9_86b2": "512", "barrier(unsigned int)": "2560", "convert_int_rtn(float)": "512" }, "memory_op_bytes": { "load": { "private": "483344", "global": "4096", "constant": "0", "local": "2240" }, "store": { "private": "164080", "global": "2096", "constant": "0", "local": "2240" }, "total": { "private": "647424", "global": "6192", "constant": "0", "local": "4480" } }, "integer_ops": { "32": "15748" }, "floating_point_ops": { "32": "86016" }, "work_group": { "executed": "1", "total": "262144" } } If we consider that all work group will perform an equivalent number of instructions and memory transactions, this kernel has an arithmetic intensity of 86016/6192=13.89 FLOP/B. By knowing the runtime of this kernel on a considered target device, one can deduce achieved bandwidth (GB/s) and compute (GFLOP/s).