I am deeply grateful to all those who supported me from my high-school years, when my interest in fluid mechanics and computer sciences began, to my Ph.D. completion, where I developed expertise in turbulence and deep learning.

First and foremost, I extend my heartfelt gratitude to my parents, Shri. Vijay Patil and Sau. Aruna Patil, for their unwavering support and encouragement. Their tireless efforts and sacrifices for my education and well-being are beyond measure. As I reflect upon their struggles, my own journey from bachelors to doctorate seems comparatively effortless. I cannot thank them enough for everything they have done for me.

I would like to express my heartfelt gratitude to my advisor, Elie Hachem, for providing me with everything I needed to succeed in my research. Right from the day I was hired, Elie ensured that I had state-of-the-art hardware, computing infrastructure, and a generous conference and travel budget. Under his guidance, I learned valuable skills such as leadership, confidence, independence, and the freedom to conduct research on my own terms. He instilled in me the importance of the socio-economic impact of research, which helped me ask the right questions. Throughout the challenging times of the COVID-19 pandemic, he checked on me like an elder brother and encouraged me to take days off to rest and recharge. I am grateful for his constant support, encouragement, and insistence on work-life balance.

Jonathan, my colleague and unofficial advisor, played a crucial role in my Ph.D. journey. I am grateful for his guidance in writing research papers and his validation of every small step. Without his support, this Ph.D. would not have been possible. He also helped to advocate for my ideas and at times even pushed my advisor, Elie, to support them. Special thanks to Florence, Aurelien, Philippe, and Franck for interesting discussions.

Thanks to Mrunmayee, the love of my life, for supporting my unconventional atrangi ideas of changing the world and for ensuring that my Ph.D. meets my expectations. She has been my best friend since my teenage years, and now, as my wife, her constant company and support have been invaluable to me. Also, my younger sister Mrunalini has been a constant support throughout my journey, always there for me like an older sister. She played a key role in keeping me sane during the challenging times and accompanied me throughout the writing of my thesis, validating my flow of ideas. I am grateful for her unwavering support and encouragement.

iii I would like to thank my dear kutumb best friends, Nikhil, Shivraj, Chaitanya, Manish, Srijith, Sakshi, and Parag for their life advice and constant support throughout my journey. I am also grateful to my master's and Ph.D. friends, Mohan, Akshay, Diana, Leo, Diego, Bipin, Bastien, Pierre-Alexandre, and Puneeth, for showing me the importance of work-life balance and enjoying life. I would like to express my heartfelt gratitude to my family in Antibes -Ashish sirji, Megha vahini, Nisarg, and Shrikant, for their care, engaging discussions, and for the best time we had in French Riviera. They were my constant source of encouragement and support, and I cherish the memories of our time together.

I am grateful to my colleagues at Mines ParisTech -CEMEF for making the Ph.D. journey enjoyable and memorable. I had the pleasure of working alongside Jonathan, Robin, Sofia, Gulia, Luca, and later Diego and Adrien. I am also thankful to Junfeng, Hassan, George, Victor, Franco, Joe, Wassim, Ramy, Ghanniya, Sasha, Juhi, and Prasanth for their company and good times. I would always remember our dicussions on a spectrum of topics from computing to economics to finance, and ofcourse food and culture.

I am grateful to Salunkhe sir for being my long-term mentor, who introduced me to research projects, grant writing, and guided me towards a career in research from a young age. During my master's in turbulence at Lille, I would like to thank Jean Phillpe for his mentorship and encouragement, Jean Marc for honing my experimental skills, and Thomas for giving me multiple opportunities and pushing me to explore Machine Learning for studying turbulence. Frank in Poitiers provided me with experimental access and helped me develop my skills in studying turbulence. I am also thankful to Corentin in Toulouse, who gave me my first exposure to French supercomputers and deep learning infrastructure.

I am grateful for the invaluable opportunities provided by the Lille Turbulence Program, CNRS-GDR Turbulence meetings, and the overall turbulence community in France. These opportunities allowed me to share my perspectives and defend my views on important topics such as reproducibility, replicability, the future of applied artificial intelligence, and the importance of giving it a test of time.

Finally, I am grateful to the gods and goddesses of all religions for providing opportunities and taking care of my family and friends during the challenging time of the Covid-19 pandemic. Pursuing a Ph.D. during the pandemic was not easy, especially while living far away from loved ones, but I feel fortunate to have received support from somewhere or something. Or, as some of my friends would say, perhaps I was just lucky ! I am grateful for the opportunities and blessings that have come my way, and I will always be humbled by the experience.

Table of Contents

Chapter 3

Robust Learning for Turbulent Flows

Des modèles les plus simples aux réseaux neuronaux profonds complexes, la modélisation de la turbulence à l'aide de techniques d'apprentissage automatique présente encore de nombreux défis. Dans ce contexte, la présente contribution propose une stratégie robuste utilisant l'apprentissage par patch pour apprendre la viscosité turbulente à partir des vitesses d'écoulement et démontre son utilisation efficace sur le modèle de turbulence Spalart-Allmaras. Des ensembles de données d'entraînement sont générés pour l'écoulement à travers des obstacles bidimensionnels à des nombres de Reynolds élevés et utilisés pour entraîner un réseau neuronal convolutionnel de type auto-encodeur avec des entrées de patchs locaux. Par rapport à une technique d'apprentissage standard, l'apprentissage par patchs permet non seulement d'améliorer la précision, mais aussi de réduire les coûts de calcul nécessaires à l'apprentissage.

Introduction

Computational fluid dynamics (CFD) is an essential asset for research and industrial applications. Despite advances in computational power over the years, industrial CFD tools still largely rely on the Reynolds Averaged Navier-Stokes (RANS) turbulence models due to cost-savings and lesser time-to-solution offered by RANS when compared with intensive Large Eddy Simulations (LES) and Direct Numerical Simulations (DNS), especially for flows at high-Reynolds numbers. Among the variety of one-equation to many equations RANS models, the Spalart Allmaras (SA) turbulence model [START_REF] Spalart | A one-equation turbulence model for aerodynamic flows[END_REF], that solves for the kinematic eddy turbulent viscosity, and has not been derived from the existing turbulent kinetic energy-based RANS models. In this sense, it can be described as a proper one-equation model which does not require knowledge of a specific problem and additional advantages include numerical stability as well as reliability for convergence of results. Due to these reasons, the SA model has been widely used, documented, and serves as a benchmark turbulence model for many CFD applications Ferziger et al. [2002]; Spalart [2000].

The Spalart-Allmaras turbulence model has also been subject to machine learning-based investigations in several works. [START_REF] Matthew O Williams | A datadriven approximation of the koopman operator: Extending dynamic mode decomposition[END_REF], Tracey et al. Tracey et al. [2015] demonstrated one of the first works on the SA model using neural networks to predict a part of the RANS closure model (namely the source terms of the eddy viscosity transport). Their study features hand-picking of features from the SA eddy viscosity transport equation in order to predict its source term. Later on, Singh et al. Singh et al. [2017] followed a similar procedure by exploiting hand-picking of input features deploying neural networks for the prediction of source terms of SA eddy viscosity transport, and later presented both a priori and a posteriori analysis. More recently, Liang et al. [2019]; Maulik et al. [2020] proposed to predict the eddy viscosity from input features consisting of data from Navier-Stokes and transport equations, while Pal [2020] used neural networks for predicting subgrid-scale viscosity in the geophysical applications.

Although convolutional neural networks are traditionally trained using fullscale inputs, patch-wise deep learning models have been successfully applied in the past in the computer-vision community. In particular, Long et al. proposed a work on object detection Long et al. [2015] where data was spatially divided into patches of information, which were then fed to the model with a primary intention of reducing memory consumption during training. This study proved efficient not only in terms of error reduction, but also in memory consumption during the training. Also, it has been demonstrated in the image classification tasks in Farabet et al. [2012]; Pinheiro and Collobert [2014] that the patch-wise training can correct class imbalance as well as assist in the spatial correlation of dense patches. More than saving training memory, patch-wise training offers a great opportunity for deep learning research in CFD, primarily because the spatial data can be divided into small patches of neighboring nodes to achieve global as well as local learning, independent of the size of the domain. It also CHAPTER 3. ROBUST LEARNING FOR TURBULENT FLOWS offers a way for CFD data augmentation by increasing the quantity of the same data with flipping and rotation of patches.

In the present contribution, we aim at learning the SA turbulent viscosity from the velocity field using a convolutional neural network trained following a patch-based approach. The proposed novelties are (i) the blind-learning of SA eddy viscosity from input velocities, without any hand-picking, manual feature selection, non-dimensionalization, or tailored losses, and (ii) a novel patch-based learning strategy with an auto-encoder type convolutional neural network, provided as a way towards generalized deep learning in turbulence modeling. The remaining of the paper is organized as follows: first, the problem setup and its governing equations are described, and the methods used for the dataset generation are covered; then, the selected network architecture is presented, and the patch-based learning procedure is described thoroughly; finally, the interests of the proposed approach are assessed, and results are discussed and compared to baseline solutions.

Problem setup & data generation

Governing equations

The evolution of the velocity u and pressure p in an incompressible fluid flow with given positive constant density ρ and dynamic viscosity µ is governed by the Navier-Stokes equations:

   ρ (∂ t u + u • ∇u) -∇ • σ = f , ∇ • u = 0, (3.1)
where σ = 2µ ε(u) -p I d is the Cauchy stress tensor for a Newtonian fluid, ε(u) the strain-rate tensor, and I d the d-dimensional identity tensor. Equations (5.6) are supplemented with adequate boundary and initial conditions, to be specified. Reynolds-Averaged Navier-Stokes (RANS) equations are then obtained by applying the Reynolds decomposition to the system (5.6), such that velocity and pressure are expressed as the sum of a mean-field and a fluctuation. Applying a time averaging operator to the resulting expressions yields a forcing term under the form of the divergence of the so-called Reynolds stress tensor. The latter consists of correlations of velocity fluctuations and accounts for the effect of the turbulent fluctuations on the averaged flow. In the Boussinesq approximation, first-order closure of the system of averaged equations amounts to a mean gradient hypothesis: turbulence is therefore modelled as an additional diffusivity called eddy viscosity µ t . The eddy viscosity µ t itself proceeds from a model involving one or more turbulent scales, each of which is the solution of a nonlinear convectiondiffusion-reaction equation. For additional details, the reader is referred to the works of Pope [2001] on turbulent flows.

The turbulence model chosen to compute the eddy viscosity is the one-equation CHAPTER 3. ROBUST LEARNING FOR TURBULENT FLOWS Spalart-Allmaras (SA) model [START_REF] Spalart | A one-equation turbulence model for aerodynamic flows[END_REF], which describes the evolution of the kinematic eddy viscosity by solving a convection-diffusion-reaction problem and serves as baseline for future testing of other models. Applying this model, the eddy viscosity µ t in the Navier-Stokes equations is obtained by µ t = ρ ν t f v1 , where f v1 is a given damping function to enforce linear profile in the viscous sublayer. The turbulent scale ν t is itself governed by the following nonlinear convection-diffusion-reaction equation:

∂ν t ∂t +u•∇ν t -c b1 (1-f t2 ) Sν t + c w1 f w - c b1 κ 2 f t2 ν t d 2 - c b2 σ ∇ν t •∇ν t - 1 σ ∇•[(ν + ν t )∇ν t ] = 0 (3.
2) where d is the distance to the nearest wall boundary, σ = 2/3, and S is the modified vorticity magnitude given as,

S = S + ν t κ 2 d 2 f v2 , S = 2W (u) : W (u),
Here κ = 0.4 is the von Kármán constant, W is the rotation-rate tensor, f v2 is a damping function to enforce the logarithmic profile, with other damping functions given as:

f v1 = χ 3 χ 3 + c 3 v1 , χ = ν t ν , f v2 = 1 - χ 1 + χf v1 f t2 = c t3 e -c t4 χ 2 f w = g 1 + c 6 w3 g 6 + c 6 w3 1 6 , g = r + c w2 (r 6 -r), r = ν t Sκ 2 d 2 ,
and model coefficients are specified as:

c b1 = 0.1355
, c b2 = 0.622 , c v1 = 7.1 , c v2 = 0.7 , c v3 = 0.9

c w1 = c b1 κ + 1 + c b2 σ , c w2 = 0.3 , c w3 = 2 , c t3 = 1.2 , c t = 0.5.
From dimensional considerations, ν t is proportional to the product of characteristic length and velocity, and as a result proportional to the Reynolds number:

ν t ∝ uL ∼ f (Re) (3.3)
More details on the implementation of this model can be found in [START_REF] Guiza | Anisotropic boundary layer mesh generation for reliable 3d unsteady rans simulations[END_REF], and more details on the turbulent viscosity models can be found in Pope [2001]. Variants of the SA model exist in the literature, most of which are collected in NASA's turbulence modeling resource webpage Rumsey et al. [2010]. In this present work, the negative Spalart-Allmaras Model was selected due to its capability to avoid the generation of negative turbulent viscosity without the use of clipping Allmaras and Johnson [2012]. These equations were cast into a stabilized finite element formulation and solved using an in-house variational multi-scale solver CimLib CFD [START_REF] Hachem | Immersed stress method for fluid-structure interaction using anisotropic mesh adaptation[END_REF]. For additional details, the reader is referred to [START_REF] Hachem | Immersed stress method for fluid-structure interaction using anisotropic mesh adaptation[END_REF] and [START_REF] Guiza | Anisotropic boundary layer mesh generation for reliable 3d unsteady rans simulations[END_REF] CHAPTER 3. ROBUST LEARNING FOR TURBULENT FLOWS

∂ y u x = u y = 0, ∂ y ν = 0 free pressure u x = u y = 0 u = u in 6.5H H 6.5H 4.5H H 20H 14H (

Datasets of turbulent flow around obstacle

We consider the widely benchmarked turbulent flow past a two-dimensional (2D) square cylinder Rodi et al. [1997]; [START_REF] Guiza | Anisotropic boundary layer mesh generation for reliable 3d unsteady rans simulations[END_REF]. A sketch of the problem, including its dimensions, is presented in figure 3.1, along with the associated mesh.

The baseline Reynolds number is set to 22 × 10 3 , based on the inlet velocity and the cylinder diameter. The inflow boundary conditions are u = (V in , 0), together with ν = 3ν, which corresponds to a ratio of eddy to kinematic viscosity of approximately 0.2. For the lateral boundaries, we use symmetry conditions ∂ y u x = u y = 0 and ∂ y ν = 0. For the outflow, ∂ x u x = ∂ x u y = 0, ∂ x ν = 0 together with p = 0 are prescribed. Finally, no-slip conditions u = 0 and ν = 0 are imposed at the cylinder surface.

Following the problem setup and methods, a baseline dataset (hereafter re- lished). Each snapshot is sampled on a rectilinear grid having spatial dimensions of (N x × N y ) = (360 × 300). The sampling on a rectilinear grid was performed to facilitate the use of CFD data coming from unstructured meshes. The same rectilinear grid was used to perform sampling on the square and circular obstacles.

For points inside the obstacle, the velocities and turbulent viscosities were zeroed out, following the no-slip boundary conditions on the obstacle. In practice, it would be possible to skip the unstructured-to-structured sampling by making use of the graph neural networks, as presented in recent works Chen et al. [2021]. The dataset is deliberately not normalized to achieve robust and generalizable training. For testing purposes, additional datasets are also generated by changing the obstacle to a 2D circular cylinder, and by modifying the Reynolds number. As is summarised in table 3.1, six different datasets are obtained. Sample snapshots of velocity and turbulent viscosity from SqRe22k are shown in figure 3.2. In the following, the training subset is composed of 75% of the SqRe22k samples and 25% of the CyRe44k samples, while the remaining samples are reserved for the validation and testing subsets (each of the latter is therefore composed of 12.5% of the SqRe22k and 37.5% of the CyRe44k).

Network architecture and training procedure

Deep learning model

Given the input dataset x (here, the velocity snapshots from the RANS simulations) and the desired output dataset y (here, the turbulent viscosity snapshots from the RANS simulations), we desire to find the optimal set of weights and biases θ = (w, b) in a deep-learned model f such that f (x; θ) = y. The set of free parameters θ is optimized using Adam Kingma and Ba [2014], in order to iteratively minimize the mean squared error (MSE) loss defined as:

L = 1 n s ns i=1 (y i -f (x; θ) i ) 2 , (3.4)
where n s is the number of samples. The full training dataset is shown repeatedly to the network after a shuffling step during the training, and each pass is referred to as an epoch. An early stopping criterion is used along with a reduction of learning rate if learning doesn't improve after every 100 epochs. The neural network was implemented using TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF], and trained on an Nvidia Tesla V100 GPU.

The network architecture proposed for the present work is an auto-encoder structure Hinton and Salakhutdinov [2006] . Auto-encoders contain two parts: (i) a converging part that decreases the spatial dimension of the input (the encoder) and compresses the input using successive convolutions, and (ii) a diverging part that rebuilds a predicted output of the same size as input (the decoder). The encoder and decoder handle the spatial-dimensionality reduction by compressing the high-dimensional spatial data, using convolutional layers, to a low-dimensional representation called latent space. For example, a N y × N z feature map can be reduced to N y /2 × N z /2 using a convolutional layer with a stride of 2. An essential aspect of this operation is that it preserves the most important features of the map. To increase robustness and generalization of the trained model, data standardization was not performed. Instead, batch normalization layers were used, which apply a transformation that maintains the mean and standard deviation of output close to 0 and 1, respectively. The proposed network architecture is shown in figure 3.3. In the literature, similar architectures (trained with full-scale inputs) were successfully exploited for studies focusing on turbulent flows [START_REF] Fukami | Super-resolution reconstruction of turbulent flows with machine learning[END_REF]; Mohan et al. [2019].

The convolutional filters used in the proposed architecture incorporate a symmetric boundary condition into the padding operation. Classically, padding is Figure 3.3: Proposed auto-encoder network architecture. The encoder branch is based on a convolution-convolution-batch-normalization pattern: the first convolution has a stride of s = 1, while the second has a stride of s = 2. The batch-normalization layer is followed by a rectified linear unit (ReLU) layer. At each occurrence of the pattern, the spatial dimensions are divided by two, while the number of filters, noted m, is doubled. In the decoder branch, a transposed convolution step is first applied to the input from the previous layer, while the number of filters is halved and two convolution layers are applied. At the end of the last layer, a 1 × 1 convolution is applied to obtain the final output. used to preserve the spatial dimensions of the field being convoluted, but the standard zero-padding approach doesn't usually represent the expected physical behavior. Indeed, padding with zeros everywhere would violate the representation of existing boundary conditions, for example, the notion of wall-boundaries would have lesser significance if a region is padded with zeros on all the sides in a channel flow Patil and Lapyere [2019] . To preserve the boundary conditions after multiple successive convolutions, a boundary condition formulation was implemented such that the walls could be padded with zeros if required, while the periodic sides could be padded with adequate values from the periodic cells. The ReLU function was used as an activation function, which is known to be an effective tool for stabilizing the weight update in the machine learning process [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF].

Patch-based training procedure

We remind the goal of the present work, which is to train a deep learning model to infer the turbulent viscosity ν at every grid point from the velocities (u, v) at the same position. As underlined earlier, no data-preprocessing tasks such as normalization or standardization were used, and the input-output fields were used "as is" from the RANS simulation output. Similar to splitting between training and validation dataset as described in section 3.2.2, we use a mixture of the SqRe22k and CyRe44k datasets. The first stage of patch-based learning consists of dividing each snapshot of the dataset into smaller n × n overlapping patches with stride s, as is shown in figure 3.4. In this case, the number of patches obtained can be doubled by considering an up-down flipping transformation on the same snapshot. For a baseline comparison, the proposed network is also trained conventionally over the full-spatial field dimensions, without using patch-based learning (this training method is hereafter referred to as M1). In this context, the batch size is 32, and the learning rate is 0.001. When a sufficient accuracy level is reached and no more improvement is observed, the training is terminated using the early-stopping criterion. A decent accuracy after convergence is obtained for both training and validation subsets, with a mean-squared error of 1 × 10 -6 , as presented on the learning curve in figure 3.5. Total training time is 0.85 hours on a Tesla V100 GPU card, for 28 million degrees of freedom.

For patch-based training, patches from the different samples are randomly shuffled together and presented to the network in batches of size 32, with a learning rate equal to 0.001 (this training method is hereafter referred to as M2). Baseline values for the patch size n and the stride s are chosen to be 50 and 75, respectively, but their respective impact on the training performance is evaluated in section 3.4. Similarly, the impact of batch size is assessed in the following section. The model is trained for 850 epochs, after which the accuracy stops improving, resulting in a final MSE error of the order of 1 × 10 -7 , i.e.one order of magnitude lower than that of method M1. Total training time is 2.38 hours for 1.7 million degrees of freedom. Although this represents about 3 times the training time of method M1, it must be noticed that the final M2 accuracy is significantly lower than that of M1, as is visible in figure 3.5. More, the final generalization level is also superior, evidenced by the negligible gap between validation and training curves. As the patch-based approach grounds the learning in a local velocity-to-turbulent-viscosity inference, it is argued that the trained network is able to re-use local mappings from one snapshot to another, leading to improved generalization capabilities compared to a monolithic snapshot-tosnapshot inference.

Results and discussion

In this section, the benefits induced by the patch-based training procedure are compared with that of the regular M1 training method on predictive tasks. To this end, predictions of both models are evaluated against reference solutions obtained from the CFD solver. In the remaining of this section, training data consists of 75% of samples from the SqRe22k dataset and 25% of samples from the CyRe44k dataset. Such a mixing of datasets is used to assess the generalization capabilities of the two methods, as both datasets present similar flow features, but with different obstacles. First, comparisons are made on out-of-training samples from the SqRe22k dataset using baseline training parameters. Then, predictions obtained with snapshots from different datasets (SqRe44k, SqRe88k, CyRe22k, and CyRe88k) are evaluated against their references. Finally, a parametric study considering the impact of batch size b, the patch size n, and the stride size s on the final performance is proposed. Overall, comparisons are made on the basis of (i) contour plots of predicted and expected ν, (ii) 1D plots of ν along streamwise and spanwise lines at different locations in the domain, as shown in figure 3.6, and (iii) scatter and density plots of the predicted ν against reference values.

Comparison on out-of-training snapshot

In this section and the following, baseline training parameters are used, i.e.batch size is equal to 32, patch size n is equal to 50, and stride size s is equal to 75. As stated above, the training data consists of 75% of samples from the SqRe22k dataset and 25% of samples from the CyRe44k dataset. M1 and M2 models' predictive capabilities are compared on an out-of-training snapshot from the SqRe22k dataset, as shown in figure 3.7. As can be observed on the scatter plot (figure 3.7a), both M1 and M2 methods are in good accordance with the reference regarding the predicted ν. Still, the M2 prediction presents an average relative deviation of 2.25% on the entire sample, against 5.04% for M1. More, its maximum relative deviation is also lower, with 36.44% for M2, against 76.23% for M1. To illustrate, the error fields obtained with M1 and M2 predictions are shown on the same snapshot in figure 3.8.

Comparison on out-of training datasets

In this section, models M1 and M2 (trained on a mixed dataset composed of samples from SqRe22k and CyRe44k) are used to make predictions on snapshots from datasets SqRe44k, SqRe88k, CyRe22k, and CyRe88k, which were not used for training. M1 and M2 predictions for one snapshot of each dataset are compared against CFD reference on stream-wise and span-wise 1D plots of ν, at the locations presented in figure 3.6. Results are shown in figure 3.9. As can be observed, the patch-based trained model consistently outperforms the M1 model, while presenting an excellent agreement with reference data. On the x = 10.02 line, which represents full developed wake region, performances of M1 and M2 models are close on SqRe44k and SqRe88k datasets, but M1 significantly overestimates the ν values on the CyRe22k and CyRe88k datasets, indicating that model M1 is unable to fully leverage the diversity of the training dataset, and only learns full-scale velocity-turbulent viscosity patterns. Conversely, the M2 model here proves its ability to learn local feature mapping from velocity field to turbulent viscosity field and accurately reconstructs it, independently of the obstacle-type and Reynolds number. Similarly, on the y = 0.02 line, which passes through the obstacle boundaries as well as the wake regions, M1 and M2 models show similar performances on datasets with a square obstacle, while M1 largely deviates from the reference data on snapshots coming from datasets with a cylindrical obstacle.

Contrarily, the M2 model again provides accurate predictions. The latter results are further emphasized on the contour plots of figure 3.10, where M1 predictions on cylindrical obstacles present inaccurate features and saturated fields in the turbulent area downstream of the obstacle. This again indicates the inability of training procedures on full-scale samples to infer proper mapping from velocity fields to turbulent viscosity fields at the local scale, which is not the case of patchbased training. The Reynolds numbers are of similar orders in magnitudes which explains the capabilities of M1 and M2 to extrapolate on Re values outside of their training datasets. Hence, the extrapolation capabilities of the M2 model could be assessed even at higher Reynolds number.

Parametric study

A parametric study is performed to explore the impact of the batch size b, the patch size n and the stride size s on the MSE error L M SE (as defined in equation (5.5)) computed on validation data. To this end, the performances of various (n, s) pairs with relation s = 1.5 × n are first compared in terms of final validation performance and training time. To select the best performance of each pair, early stopping is used during training, and the average validation error over the last 50 epochs, noted L M SE , is retained. As shown in table 3.2a, the pairs (100, 150) and (50, 75) yield close performances in terms of final MSE error. Although the (100, 150) is slightly better in accuracy and training time, the (50, 75) pair is preferred for its larger amount of patches per snapshot. The larger errors of the (20, 30), (10, 15), (6, 9), and (2, 3) pairs can be attributed to the low number of points per patch making it difficult to train the model with the same hyper-parameters, while the (200, 300) pair prevents the efficient learning of local features, and is likely to present the same flaws as method M1.

In a second time, we consider the impact of varying stride size s for the previously-select n value, equal to 50. Results are presented in table 3.2b. As can be seen, no significant difference is observed for stride values ranging from 30 to 300, indicating that in this context, the amount of patches per snapshot (and thereby total samples) is not a limitation. Finally, the effect of varying batch size is assessed for (n, s) = (50, 75). As shown in table 3.2c, small batch sizes 8, 16, and 32 yield close error levels, while larger batch sizes are associated with errors larger by roughly one order of magnitude. Although b = 8 is slightly lower than the other values, b = 32 is retained as the best accuracy/training time ratio.

Conclusions

In

Introduction

Turbulence is multiscale in nature, with a very wide range of scales coexisting and interacting. Turbulence is governed by the Navier-Stokes equation in most practical problems, and is still very difficult to predict and model. In order to understand its detailed physics, spatio-temporal resolved information is necessary.

Unfortunately, none of the current facilities, even within academic research, can provide this information over a sufficiently wide spatial range and different flow conditions. Despite advances in computational resources, direct numerical simu- Fukami et al. [2019aFukami et al. [ , 2020]]. Some investigations have considered machine learning methods and superresolution data reconstruction techniques for reproducing turbulent flows [START_REF] Nakamura | Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow[END_REF]; Jiang et al. [2020]. Once trained on high-quality data, these methods have been shown capable of reproducing the underlying flow field using remarkably coarse measurements. Previous studies have also considered purely statistical data fusion methods for flow reconstruction. In Van Nguyen et al. [2015], a "model-free" maximum a posteriori (MAP) algorithm was proposed for fusing low-temporal-high-spatial resolution data with high-temporal-low-spatial resolution data for turbulent flow reconstruction. On the experimental side, Krishna et al. [2019,2020] showed that the gaps between PIV snapshots can be filled with simple linear assumptions on the Navier Stokes equations. In that work, simple linear regression models were used to propagate PIV snapshots forward and backward in time, with a weighted average of forward and backward predictions used to estimate the turbulent flow between snapshots.

The current chapter addresses the problem of estimating subgrid-scale information from grid-resolved information to provide a complete view of turbulent flows at both large and small scales. Available turbulence databases, both computational and experimental, are used to reconstruct subgrid-scale turbulence. First, the DNS database used for training the deep learning model is described, followed by the rationale of the deep learning model's architecture and the as-sociated transfer learning. Later, the subgrid-scale reconstruction statistics are compared with reference to the DNS data. The trained model is also tested against a PIV dataset to demonstrate its robustness and applicability to the experimental dataset.

Turbulent channel flow datasets and preparation

One of the important concepts in the subgrid-scale reconstruction of the turbulent flow between consecutive snapshots is the model used to approximate the flow evolution. We consider the three-dimensional incompressible Navier-Stokes equations (NSE) linearized around the mean flow profile to satisfy the momentum equation and continuity constraint. Considering the dynamics of turbulent velocity fluctuations u = (u 1 , u 2 , u 3 ) around a mean flow U = (U (x 2 ), 0, 0), we have:

∂u ∂t + U • ∇u + u • ∇U = -∇p + 1 Re τ ∇ 2 u + (NL), (4.1)
∇ • u = 0, (4.2)
where U = [U (x 2 ), 0, 0] denotes the mean flow profile in a three-dimensional coordinate system, as shown in Fig. 4.1. We use (x 1 , x 2 , x 3 ) to represent the streamwise, wall-normal, and spanwise directions, respectively. u = [u 1 , u 2 , u 3 ] denotes the turbulent velocity fluctuations, and p represents the pressure fluctuations. Re τ denotes the friction Reynolds number.

We assume linearity as proposed by Hunt and Carruthers [1990] in such a way that the non-linear interactions can be neglected on scales that are much shorter than typical scales of interaction, which in the case of time evolution means that the interactions shorter than typical eddy turnover time can be neglected. These assumptions allow for linearized equations of motions for the subgrid-scale reconstruction of flow over shorter spatial or temporal scales. This means the non-linear terms can be neglected when the grid resolution (∆) or time step (∆t) is less than a typical eddy length or eddy turnover time. The simplified NSE would thus read as:

∂u 1 ∂t + U ∂u 1 ∂x 1 advection = 1 Re τ ∂ 2 u 1 ∂x 2 1 + ∂ 2 u 1 ∂x 2 2 diffusion -u 2 ∂U ∂x 2 coupling , ∂u 2 ∂t + U ∂u 2 ∂x 1 advection = 1 Re τ ∂ 2 u 2 ∂x 2 1 + ∂ 2 u 2 ∂x 2 2 diffusion .
(4.3)

A direct numerical simulation database of turbulent channel flow is used to validate the subgrid-scale reconstruction method described in this chapter. To

u = u t i -u t i ′ , (4.4)
where (.) denotes the mean quantity along symmetric or homogeneous directions, and (.) ′ denotes the fluctuating quantity. In this case, we can take the mean along t and x, making the resulting velocities a function in y -z planes. Hereafter, the fully resolved fluctuating velocities in a plane normal to flow direction are denoted by u ′ . This data contains a total P = 512 × 1700 snapshots representing samples in x and t. Each snapshot has a spatial resolution of 384 × 512 denoting the samples in the y -z plane. This is our true data, and the target of higher resolution against which we would compare our errors of subgrid-scale reconstruction.

The lower-resolution dataset is synthetically produced from the high-resolution data with spatial coarse-graining. We perform spatial coarse-graining with meanpooling, max-pooling, and sparse sampling of the higher-resolution data. The lower resolution dataset coefficients are 4∆, 16∆, and 64∆ for four-times, sixteentimes, and sixty-four times coarse-grained snapshots respectively. For example, when 4∆ is spatially coarse-grained along the y -z plane, a 384×512 higher resolution snapshot becomes a 96 × 128 lower resolution snapshot. These datasets are used to estimate high-resolution turbulent flow fields from low-resolution ones,

Deep Learning Architecture and Training

A deep learning model to estimate mapping between subgrid scales and large scales is required. This can be considered as whether a non-linear model M exists such that, given a lower-resolution field, a corresponding higher-resolution field would be predicted by:

u t i ′ HR = M(u t i ′ LR ), (4.5)
where u t i ′ HR is a higher-resolution snapshot at instant t containing the complete subgrid-scale information, and u t i ′ LR is a lower-resolution snapshot at the same instant containing the large scales.

A deep learning architecture based on enhanced super-resolution generative adversarial networks (ESRGAN) [START_REF] Guo | Long text generation via adversarial training with leaked information[END_REF] is used to reconstruct highresolution flow fields from low-resolution fields. Figures 4.2 show the architectures of a generator and discriminator in the ESRGAN. A generator consists of a deep convolution neural network with residual in residual dense blocks (RRDBs) [START_REF] Han | A novel spatialtemporal prediction method for unsteady wake flows based on hybrid deep neural network[END_REF]. A low-resolution input snapshot is first fed to the generator which is passed through a convolution layer followed by a series of RRDBs, and passed through a final convolutional layer to generate a high-resolution snapshot. The original ESRGAN uses an additional discriminator to which the generated and true snapshots are fed by passing through a series of convolutional, batch normalization, and rectified linear unit (ReLU) layers. After successful training, the generator is expected to produce a snapshot with statistics similar to the true snapshot that a discriminator cannot distinguish from the true snapshot. Readers are referred to the work of Goodfellow et al. [2020] for details on generative-adversarial networks (GANs) where two adversarial neural networks, the generator, and the discriminator, compete with each other. For the present study, we deploy transfer learning by making use of a previously trained ESR-GAN model on super-resolution of images from various sources in Wang et al. [2018]. Transfer-learning is used to re-train a model for a new dataset which helps in saving training time.

This deep learning architecture aims at proposing a nonlinear model to estimate subgrid-scale information given the large-scale information. The use of deep learning models is in the context where data at all scales are available from the DNS dataset. The proposed model learns an empirical relation between large and subgrid scales, which is presented in the form of a function M(.) to predict subgrid scale information on new datasets and situations. The deep learning architecture aims at finding nonlinear mapping functions with a more adaptive kernel space and allows to model nonlinear phenomena.

The resulting deep learning model is trained with the Adam Kingma and Ba [2014] optimizer, to iteratively minimize the total Root Mean Square Error (RMSE) of the reconstruction loss defined by:

ǫ(t) = ( h x 1 =0 h x 2 =0 (u 1 -û1 ) 2 + (u 2 -û2 ) 2 dx 1 dx 2 ) 1/2 ( h x 1 =0 h x 2 =0 ((u 1 ) 2 + (u 2 ) 2 ) dx 1 dx 2 ) 1/2 , (4.6)
where û1 and û2 are the reconstructed velocity fluctuations while u 1 and u 2 are the velocity fluctuation from the DNS ground truth. Following the same metrics, we can also evaluate the RMSE with respect to the streamwise and wall-normal velocity components separately with an aim to evaluate the reconstruction errors for each component of the 2D velocity profiles:

ǫ x 1 (t) = ( h x 1 =0 h x 2 =0 (u 1 -û1 ) 2 dx 1 dx 2 ) 1/2 ( h x 1 =0 h x 2 =0 u 2 1 dx 1 dx 2 ) 1/2 , ǫ x 2 (t) = ( h x 1 =0 h x 2 =0 (u 2 -û2 ) 2 dx 1 dx 2 ) 1/2 ( h x 1 =0 h x 2 =0 u 2 2 dx 1 dx 2 ) 1/2 . (4.7)

Results and Discussions

The deep learning model is trained on 70% of the available DNS snapshots and the remaining 30% snapshots are kept for evaluation of results presented in this section. First, the results from various coarse-graining methods for generating synthetic low-resolution datasets are compared, followed by the effects of refinement direction on reconstructing the subgrid-scales. Later, successive refinement through various scaling factors is compared. Lastly, the trained deep learning model is tested on a PIV experimental dataset of wall-bounded turbulence.

To compare the results of subgrid-scale reconstructions with DNS, several turbulence statistics are compared. Turbulence intensity, dissipation, and enstrophy are compared along with turbulent kinetic energy spectra in streamwise and spanwise directions. The turbulence intensity, also often referred to as turbulence level, is defined as:

I = u ′ U , (4.8)
where u ′ is the root-mean-square of the turbulent velocity fluctuations and U is the mean velocity. If the turbulent kinetic energy k is known, u ′ can be computed as:

u ′ = 1 3 (u ′2 x + u ′2 y + u ′2 z ) = 2 3 k. (4.9)
U can be computed from the three mean velocity components U x , U y and U z as:

U = U 2 x + U 2 y + U 2 z . (4.10)
Turbulence dissipation, ǫ is the rate at which turbulence kinetic energy is dissipated, which is written as:

ǫ ≡ ν ∂u ′ i ∂x k ∂u ′ i ∂x k . (4.11)
Finally, the enstrophy Ω is a type of potential density which is directly related to the kinetic energy in the flow that corresponds to dissipation effects in turbulent flows. For incompressible flows with ∇ • u = 0, the enstrophy can be described as the integral of the square of the vorticity ω given by:

Ω ≡ Ω |ω| 2 dx.
(4.12)

Effects of coarse-graining

Since it is not known which pooling method for spatial coarse-graining is better than other pooling methods, the effects of these methods are examined. As described in the previous section, the lower-resolution dataset is synthetically produced from the high-resolution data with spatial coarse-graining. Spatial coarsegraining is performed with mean-pooling, max-pooling, and sparse sampling of the higher-resolution data. As shown in figure 4.3 max-pooling refers to taking the maximum value at each filter, average or mean-pooling refers to taking the average value at each filter, whereas sparse-pooling refers to sparse-sampling of the available feature space at each filter. When max, mean, and sparse pooling methods are compared, it is found that the choice of these coarse-graining methods does not have a significant impact on the subgrid-scale reconstruction. Figures 4.4b and 4.4c show the comparison of turbulence dissipation and enstrophy for various coarse-graining methods when measured along streamwise directions. Figure 4.4 shows a comparison of these methods by measuring turbulence intensity in a streamwise direction, where it can be noted that subgrid-scale reconstruction from all three methods have similar accuracy and are comparable to DNS. Additionally, when the spectral contributions are compared as shown in figure 4.5, it is observed that these three methods have similar accuracy for streamwise energy spectra, whereas, for the spanwise averaged spectrum of turbulent kinetic energy, it is observed that the subgrid-scale reconstruction from max-pooling has dissipated more compared to that from sparse-pooling. For further reporting, sparse pooling was chosen since it represents realistic situations for both experimental and numerical data, and it is computationally faster to perform sparse pooling.

Effects of Refinement Direction

Now that the effect of coarse-graining methods is established, we proceed towards investigating refinement direction and its effect on subgrid-scale reconstruction. The DNS dataset is a 3D dataset with data available along x -y, y -z, and x -z 2D planar refinement directions, hence it is essential to demonstrate the deep learning model's performance for subgrid-scale reconstruction along these refinement directions. These investigations would be particularly useful later while demonstrating the trained model's applicability on the PIV dataset which is available only along two planes. The subgrid-scale reconstruction along all three refinement directions shows good agreement with the DNS data. From a finer evaluation, it is observed that the turbulence intensity is slightly under-predicted across all refinement directions, whose effect is then magnified in the turbulence dissipation comparison. Furthermore, streamwise and spanwise spectra are compared for these refinement directions as shown in figure 4.9 and figure 4.10. Both spectra from subgrid-scale reconstruction along all three refinement directions show good accuracy across the range of large and small scales. Overall, a good agreement in measured physical statistics is observed for subgrid-scale reconstructions compared to DNS.

Investigating Successive Refinement

The quality of subgrid-scale reconstruction by the extent of coarse-graining is investigated. Successive refinements are performed until DNS-comparable spatial levels are attained. The present deep learning model is trained for 4× subgridscale reconstruction, which means a DNS comparable complete spatial snapshot is estimated from a 4× spatially coarse-grained snapshot. But, can a 16× spatially coarse-grained snapshot be used to predict a DNS-comparable complete spatial snapshot by using the same deep learning model? Can this be done for 64× coarse-grained snapshot? Essentially, the effect of severe coarse-graining is investigated by successively refining the coarse-grained snapshot by 4× during each refinement. Thus for a DNS comparable solution of subgrid-scale reconstruction, a 64×, 16×, 4× coarse-grained snapshot would need one, two, and three successive refinements, respectively. The subgrid-scale reconstructions for 64×, 16×, 4× coarse-grained snapshots are compared by measuring turbulence intensity as shown in figure 4.11. As expected, the 4× subgrid-scale reconstruction has better accuracy with respect to DNS, as compared to the 64× subgrid-scale reconstruction. As shown in figure 4.12 and figure 4.13, similar trends are observed for turbulence dissipation and enstrophy. It is observed that the 4× and 16× subgrid-scale reconstructions exhibit an LES-like behavior, whereas 64× subgrid-scale reconstruction shows a RANSlike behavior when turbulence intensity, dissipation, and enstrophy are compared . These results show how challenging it is for a deep learning model to get DNSlike solutions from coarse-grained data. When subgrid-scale reconstructions are investigated for streamwise and spanwise turbulence kinetic energy spectra as shown in figure 4.14, the streamwise spectra offer a distinct picture of reconstructions across scales. For streamwise spectra, 64× subgrid-scale reconstruction shows a clear and direct dissipation at larger scales, whereas 4× subgrid-scale reconstruction shows DNS comparable dissipation until smaller scales.

Additionally, root mean squared (RMS) turbulent velocities after subgridscale reconstructions are compared as a function of distance from the channel's wall. Subgrid-scale reconstructions for 64×, 16×, and 4× coarse-grained snapshots are compared. Turbulent velocity components along streamwise, wallnormal, and spanwise directions are compared to DNS data as shown in figure4.15. At higher y + , i.e. away from the wall, the stream-wise velocity component shows expected agreement for all subgrid-scale reconstructions compared to DNS. Near the wall, only 4× subgrid-scale reconstructions show better agreement to that of DNS which is expected and consistent with the previous results. 64× subgridscale reconstructions show unexpected oscillations for all the turbulent velocity components, which could disappear with averaged statistics using more number of snapshots.

Application on Experimental Data

To demonstrate the robustness of the trained deep learning model for subgridscale reconstruction, experimental Particle Image Velocimetry (PIV) data is used. A typical PIV setup consists of flow in a wind tunnel seeded with tracer particles which get illuminated when passed across a laser sheet created in regions of interest. This tracer particle movement is captured with complementary metal-oxide-semiconductor sensors which are cameras with high shutter speeds, from which velocity fields are measured by calculating the speed and direction of tracer particles. High-resolution PIV can measure the flow at a small field-of-view but at a very high spatial resolution, but for studying a flow region with a bigger Coarse-grained data by a factor of 4× is obtained by sparse-pooling as described in previous sections. Subgrid-scale reconstruction is performed by the trained deep learning model by predicting full-field PIV snapshot from the 4× coarse-grained snapshot. Figure 4.17 shows turbulence intensity measured along refinement directions in x -y and z -y planes. A good agreement is seen for subgrid-scale reconstruction when compared with PIV data. Similar trends are seen for turbulence dissipation and enstrophy of turbulence as shown in figure 4.18 and figure 4.19 respectively. Additionally, spectral contributions are examined by comparing streamwise and spanwise spectra as shown in figure 4.20 and figure 4.21, both of which show a good agreement across the range of scales.

Similar to the method described previously, we perform successive refinements on PIV data for 64×, 16×, and 4× subgrid scale reconstruction.

Conclusions

The present study is on the intersection of computer vision, deep learning, and computational as well as experimental turbulence. The primary objective was to explore the state-of-the-art deep learning method to estimate subgrid-scale turbulence from measured large scales or grid-resolved scales. The key contribution of this study is to perform transfer learning by re-training a model trained for non-physics problems to re-learn new physical features. Coarse-grained fields are computed from the reference datasets to make the a priori predictions using the trained deep learning model on the DNS dataset. Subgrid-scale reconstructions are performed and compared with reference DNS as well as PIV datasets. In both datasets, wall-bounded turbulent channel flow datasets are used. Subgrid-scale reconstructions along various refinement directions and various coarse-graining levels are compared with the reference by measuring physical statistics like turbulence intensity, dissipation, enstrophy, and energy spectra.

PIV or HWA being limited by the hardware resolutions, experiments can only capture what is within the limits of their acquisition methods. The DNS or similar numerical simulations are limited by the grid resolutions and computing power. It is almost impossible to increase information transfer beyond this limit, but the features outside the limit can be reconstructed from features based on the global or historical context of the data, demonstrating the need for further work on deep learning to reconstruct grid-scale turbulence. Such an approach may allow going beyond what can be measured, computed, or stored. The results from this work would provide a way to design experiments or numerical simulations in such a way that deep learning facilitates the recovery of a maximum level of information. On the experimental side, time-resolved velocity fields at large field-of-view and high spatial resolution could be estimated with such a deep learning-assisted method.

Introduction

The main factor in turbulent flows is convection, which makes tasks such as flow control and model reduction complex and challenging. These tasks become nonlinear, high-dimensional, multi-scale, and non-convex optimization problems due to the dominance of convection over diffusion. Due to the vast amount of numerical and experimental data available for turbulent flows, data-driven approaches are now gaining popularity in the fluid mechanics community. These approaches use deep learning models to make predictions and represent a valid alternative to traditional methods. This article explores a new data-driven approach based on deep learning to estimate future fluid flow fields from previous ones. The proposed method uses a novel convolutional encoder-decoder transformer model and autoregressive training to achieve long-term spatio-temporal predictions. The approach is tested on two turbulent fluid flow cases, namely a wake-flow past a stationary obstacle and an environmental flow past a tower fixed on a surface. The results show the effectiveness of the proposed method in predicting the fluid flow fields accurately, highlighting the potential of data-driven approaches in solving challenging problems in fluid mechanics.

There are several traditional ways to address temporal estimations, such as Koopman theory and proper orthogonal decomposition, which are suitable for prediction and control [START_REF] Matthew O Williams | A datadriven approximation of the koopman operator: Extending dynamic mode decomposition[END_REF]; [START_REF] Steven L Brunton | Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control[END_REF]; [START_REF] Rowley | Model reduction for flow analysis and control[END_REF]. Additionally, data assimilation schemes are popular, where the model weights are updated to reflect new observations [START_REF] Mons | Reconstruction of unsteady viscous flows using data assimilation schemes[END_REF]. In recent years, supervised learning techniques using neural networks have been applied to capture nonlinear relations between past and future states. For example, a recurrent neural network with long-short term memory was used to predict the chaotic Lorenz system, and convolutional networks were used to predict transient flows [START_REF] Dubois | Datadriven predictions of the lorenz system[END_REF]; [START_REF] Xu | Multi-level convolutional autoencoder networks for parametric prediction of spatio-temporal dynamics[END_REF]. There have also been attempts to approximate the full Navier-Stokes equations using deep neural networks, but prediction accuracy decreased significantly for chaotic and turbulent flows [START_REF] Lusch | Deep learning for universal linear embeddings of nonlinear dynamics[END_REF]; [START_REF] Sirignano | Dgm: A deep learning algorithm for solving partial differential equations[END_REF]; [START_REF] Hs Tang | An exploratory study on machine learning to couple numerical solutions of partial differential equations[END_REF]; [START_REF] Sun | Neupde: Neural network based ordinary and partial differential equations for modeling time-dependent data[END_REF]. Regarding the estimation of flow fields using deep neural networks, several studies have focused on spatial and temporal reconstruction, as well as spatial supersampling [START_REF] Cheng | Deep learning method based on physics informed neural network with resnet block for solving fluid flow problems[END_REF]; [START_REF] Mustafa Z Yousif | High-fidelity reconstruction of turbulent flow from spatially limited data using enhanced super-resolution generative adversarial network[END_REF]; [START_REF] Schmidt | Machine learning accelerated turbulence modeling of transient flashing jets[END_REF]. Hybrid deep neural network architectures have been designed to capture the spatial-temporal features of unsteady flows [START_REF] Han | A novel spatialtemporal prediction method for unsteady wake flows based on hybrid deep neural network[END_REF], and machine learning-based reduced-order models have been proposed for three-dimensional complex flows [START_REF] Nakamura | Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow[END_REF]. A deep learning framework combining long short-term memory networks and convolutional neural networks has been used to predict the temporal evolution of turbulent flames [START_REF] Ren | Predictive models for flame evolution using machine learning: A priori assessment in turbulent flames without and with mean shear[END_REF]. However, despite the significant progress made in the acceleration of flow simulation, these models still suffer from the generalization problem and are sensitive to parameter changes [START_REF] Kochkov | Machine learning-accelerated computational fluid dynamics[END_REF].

New deep learning architectures for temporal problems in unstructured and TURBULENCE structured data are emerging, with transformers being one of the most promising. These models make use of self-attention mechanisms to differentially weight the significance of each part of the input data [START_REF] Vaswani | Attention is all you need[END_REF]; [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], without the need for recurrent network architecture. Inspired by neighborhood-like notions in convolutional neural networks, transformers build features of inputs using a self-attention mechanism to determine the importance of other samples in the dataset with respect to the current sample. The updated features of the inputs are simply the sum of linear transformations of all features weighted by their importance. Transformers avoid recurrence by using the selfattention mechanism, which accounts for the similarity score between elements of a sequence and the positional embedding of these elements, allowing them to account for the full sequence instead of single elements. These models have been successful in natural language processing (NLP) tasks such as translation and text summarization, and are becoming the model of choice for NLP problems, replacing classical recurrent neural network (RNN) models such as long short-term memory (LSTM) [START_REF] Wolf | Transformers: State-of-the-art natural language processing[END_REF]; [START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF]; [START_REF] Radford | Language models are unsupervised multitask learners[END_REF].

Transformers have also been applied to image processing tasks using convolutional neural networks to capture relationships between different portions of an image [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]; [START_REF] Parmar | Image transformer[END_REF]; [START_REF] Touvron | Training data-efficient image transformers & distillation through attention[END_REF]. Hybrid architectures combining convolutional layers with transformers have achieved excellent results in several computer vision tasks [START_REF] Dai | Coatnet: Marrying convolution and attention for all data sizes[END_REF]; Wu et al. [2021a].

In spatio-temporal context, transformers have been used for video-understanding tasks, capturing spatial and temporal information through the use of divided space-time attention [START_REF] Sharir | An image is worth 16x16 words, what is a video worth?[END_REF]; [START_REF] Bertasius | Is space-time attention all you need for video understanding[END_REF]. In fluid mechanics, attention mechanisms have enhanced the reduced-order model to extract temporal feature relationships from high-fidelity numerical solutions Wu et al. [2021b]. Recently, a similar combination of autoregressive transformers and twodimensional homogeneous isotropic turbulence was proposed for spatio-temporal prediction of flow fields [START_REF] Peng | Attention-enhanced neural network models for turbulence simulation[END_REF]. However, transformers have never been used for spatio-temporal prediction of flow fields involving turbulent flows. The present contribution is organized as follows: first, the deep learning method based on the convolutional self-attention transformer is discussed, after which focus is made on the autoregressive training procedure. The following section provides insights into the performance of the proposed approach by considering (i) a turbulent flow case with an obstacle embedded in a rectangular domain, and (ii) a surface-mounted tower in an open flow. This part is followed by a discussion and a conclusion.

Deep Learning Method

The primary focus of this contribution is to address the challenge of learning the spatio-temporal dynamics of turbulent flows, which are known for their high complexity, non-linear behavior, and high dimensionality. There are two main approaches to estimate a reference spatio-temporal field X t : (i) reconstruction, which involves utilizing limited measurements Xt at a specific time t to reconstruct the full X t field at the same time, and (ii) prediction, where a dynamical model is utilized to advance the field in time based on previous estimates. Here, spatio-temporal learning is formulated as a task with a given time-series containing N sequential snapshots x t , x t+∆t , ...., x t+(N -1)∆t , in order to predict the same quantity of interest on M steps ahead in time. The input X of the deep learning model is x t , x t+∆t , ...., x t+(N -1)∆t , and the output Y is x t+N ∆t , ..., x t+N +(M -1)∆t . Each snapshot x t can be a scalar field or a vector field containing multiple features.

Transformer models in deep learning were developed to address natural language processing problems, where sentence completion and translation are performed using a word by word embedding [START_REF] Vaswani | Attention is all you need[END_REF]. The sentencecompleting NLP tasks can be understood as a temporal learning problem, with a time series of words or sentence tensors measured over a time duration. These models have achieved remarkable performances in a variety of other tasks, including learning image patches as sequences, image completion and reconstruction [START_REF] Vaswani | Attention is all you need[END_REF]; [START_REF] Dai | Coatnet: Marrying convolution and attention for all data sizes[END_REF]; Wu et al. [2021a]. As a result, the transformer models have been challenging the classic long short-term memory (LSTM) models, the de facto RNNs, and replacing them with a state-of-the-art approach in a variety of temporal learning tasks.

Like RNNs, transformers are designed to handle sequential input data. However, unlike the latter, they do not necessarily process the data in order. Rather, the attention mechanism provides context for any position in the input sequence, and self-attention itself identifies/learns the weights of attention. In the case of spatio-temporal data, the attention can be applied to the spatial as well as the temporal sequence to attend to or pay attention to. The vanilla transformers in their original form are pure sequence to sequence models, as they learn a target output sequence from an input sequence, i.e. they perform transformation at the sequence level. Their limitations, such as disrupting temporal coherence and failing to capture long-term dependencies, were reached for sentence completion of language generation tasks, where difficulties were noted while generating texts with a model which learns sequences without the knowledge of full-sequences [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF]; [START_REF] Yu | Seqgan: Sequence generative adversarial nets with policy gradient[END_REF]; [START_REF] Guo | Long text generation via adversarial training with leaked information[END_REF]. Several studies were performed, such as that of [START_REF] Dai | Transformer-xl: Attentive language models beyond a fixedlength context[END_REF], to address this inability to capture long-term dependencies by attending to memories from previously learned parameters, yet at the expense of computing costs. To deal with some of these issues, autoregressive transformers were proposed by [START_REF] Katharopoulos | Transformers are rnns: Fast autoregressive transformers with linear attention[END_REF] for sentence and image completion tasks. Although not explicitly stated in some works, the Generative Pre-trained Transformer (GPT) family of models [START_REF] Radford | Improving language understanding by generative pre-training[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF]; [START_REF] Brown | Language models are few-shot learners[END_REF] are in fact autoregressive transformers inspired by the decoder part of the original transformers. In [START_REF] Katharopoulos | Transformers are rnns: Fast autoregressive transformers with linear attention[END_REF], Katharopoulous et al. showed that a self-attention layer trained in an autoregressive fashion can be seen as a recurrent neural network. Transformers can be combined with the classic convolutional encoder-decoder type models to harness their full potential when the input and target output tensors are in a spatio-temporal form. As locality is more important in learning small-scale features, this combination serves as a powerful method for a variety of computervision problems, including video-frame prediction. The self-attention mechanism on convolutional layers not only attends or focuses on a sequence of significance, but it also improves the representation of spatially-relevant regions by focusing on important features and suppressing less-important ones [START_REF] Woo | Cbam: Convolutional block attention module[END_REF].

When a transformer block is applied after a convolutional layer, the model learns to emphasize meaningful features along the channel sequence and spatial dimensions. The input sequences are first appended channel-wise to the input layer and subsequent convolutional operations are performed in the encoder. In the convolutional layers, the intermediate feature maps F ∈ R C×H×W of a given layer are passed through the self-attention convolutional transformer layer conv α , which simultaneously attends to spatial representation and the positional embeddings of the input sequence channels. In conv α , let x, y ∈ R C be the input and output intermediate feature tensors, with C representing the number of intermediate channels. When i, j ∈ R H×W are indices of the spatial nodes, a classical convolution operation is performed such that:

y i = j∈N (i) W i→j x j , ( 5.1)
where N (i) represents the spatial nodes in a local neighborhood defined by a kernel of size k × k centered at node i, i → j represents the relative spatial relationship from i to j, and W i→j ∈ R C×C is the weight matrix. On the other hand, self-attention for intermediate convolutional features has three weight matrices W q , W k , W v ∈ R C×C to compute query, key and value respectively. For each convolution window, the self-attention is given as:

y i = j∈N (i) α i→j W v x j , α i→j = e (Wqx i ) T W k x j z∈N (i) e (Wqx i ) T W k xz = W qk x i [j] z∈N (i) W qk x i [z]
,

(5.2)

where the self-attention α i→j ∈ (0, 1) is a scalar that controls the contribution of values in spatial nodes, with W qk ∈ R C×k 2 , and [j] means j th element of the tensor. α is usually normalized by a softmax operation such that j α i→j = 1. These operations are summarized in figure 5.1.

Combining equations (5.1) and (5.2), one obtains both an input sequence dependent kernel and the learnable convolution filters providing the final output feature map F ′′ by convolutional transformer layer, given as: The current self-attention convolutional transformer layer has a 3 × 3 kernel and incorporates the representation of convolutional features. Combining convolutional neural networks with self-attention thus offers superior learning capabilities of spatio-temporal structures, which would benefit turbulent flows and CFD in general, where one learns spatial filters as well as temporal embeddings and dependencies. In addition to the convolutional transformer layer, the model is trained in an autoregressive fashion. Formally, autoregressive models are those which forecast future sequences from the previously forecasted sequences in a cyclical way, and thus here auto indicates the regression of the variable sequence against itself.

y i = j∈N (i) softmax (α i→j ) W i→j x j i.e. F ′′ = conv α (F) (5.3) TURBULENCE xin space to depth BMM xout similarity values + depth to space Input H, W, Cin H 2 , W 2 , k × k, Cin H, W, Cin H 2 , W 2 , k × k, Cout H 2 , W 2 , k × k, k × k H 2 , W 2 , k × k,
In turbulent flow problems, the high-dimensional state-space is characterized by intricate spatio-temporal dynamics, and therefore, dimensionality reduction techniques can be useful [START_REF] Dubois | Machine learning for fluid flow reconstruction from limited measurements[END_REF]. Reconstruction and prediction problems are therefore equivalent to the estimation of the reduced or latent state, thus making the use of encoder-decoder based architecture a natural choice. The encoder-decoder architecture comprises an encoder that takes input tensors and maps them to a high-dimensional representation by learning which parts of the input tensors are important and converts or passes them to abstract lowdimensional representation. With the addition of a decoder after this encoder, this high-dimensional representation is converted to target output tensors. By chaining the encoder and decoder together, their weight matrices jointly learn the output tensors from input tensors, thus helping to learn the small-scale features. The decoder is comprised of successive up-samplings followed by convolutions, and brings the latent space representation of dimension n z × n z back to the original spatial dimensions of the target output at time x t+∆t . The figure 5.2 offers a global view of the considered deep learning architecture. For a trained model M as shown in figure 5.2, multi-step training is performed for quantity X t in an auto-regressive manner, i.e. X t+∆t is predicted from previously predicted X t , where t is some non-dimensional time. In other words, an initial condition X t is inputted to the model to learn X t+∆t , after what this predicted X t+∆t is then fed back to the model again to learn X t+2∆t and so on, in an autoregressive manner:

               X t+∆t = M(X t ), X t+2∆t = M( X t+∆t ), ... X t+(n-1)∆t = M( X t+(n-2)∆t ),
(5.4)

where t is the time step and X ∈ R C×H×W the input tensor snapshot at instant t. In the following, the autoregressive training sequence length is set equal to two in order to limit the computational cost.

In order to preserve meaningful values at the boundaries, the convolutional filters used in the proposed architecture incorporate a symmetric boundary condition into the padding operation. Classically, padding is used to preserve the spatial dimensions of the field being convoluted, but the standard zero-padding approach doesn't usually represent the expected physical behavior. Indeed, padding with zeros everywhere would violate the representation of existing boundary conditions, for example, the notion of wall boundaries would have lesser significance if a region is padded with zeros on all the sides in a channel flow. To preserve the boundary conditions after multiple successive convolutions, a boundary condition formulation was implemented such that the walls could be padded with zeros if required, while the other sides could be padded with adequate values from the symmetric cells. The resulting model is trained with the Adam Kingma and Ba [2014] optimizer, to iteratively minimize the total equi-weighted mean squared error (MSE) loss defined by:

L = 1 n s ns i=1 (X t+∆t ) i -X t+∆t i 2 + ns i=1 (X t+2∆t ) i -X t+2∆t i 2 + • • • + ns i=1 X t+(n-1)∆t i -X t+(n-1)∆t i 2
.

(5.5)

The ReLU function was used as an activation function, which is known to be an effective tool for stabilizing the weight update in the training process [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF]. The full training dataset is shown repeatedly to the network after a shuffling step during the training, and each pass is referred to as an epoch. An early stopping criterion is used along with a reduction of the learning rate if learning doesn't improve after every 100 epochs. The neural network was implemented using TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF], and trained on Nvidia Tesla V100 GPUs.

Numerical simulation cases and data generation

Governing equations

The evolution of the velocity u and pressure p in an incompressible fluid flow with given positive constant density ρ and dynamic viscosity µ is governed by the Navier-Stokes momentum and continuity equations: (5.6) where ε(u) is the strain-rate tensor, I d is the d-dimensional identity tensor, and f is the additional forcing or source term. Equations (5.6) are supplemented with adequate boundary and initial conditions based on the physical cases. Turbulence is modeled as an additional diffusivity called eddy viscosity µ t which itself proceeds from a model involving one or more turbulent scales, each of which is the solution of a nonlinear convection-diffusion-reaction equation. The turbulence model chosen to compute the eddy viscosity is the one-equation Spalart-Allmaras (SA) model [START_REF] Spalart | A one-equation turbulence model for aerodynamic flows[END_REF], which describes the evolution of the kinematic eddy viscosity like variable ν t by solving a convection-diffusionreaction problem.

ρ (∂ t u + u • ∇u) = ∇ • (-p I d + 2µ ε(u)) + f , ∇ • u = 0,

Case 1 : Wake-flow past a square cylinder

A widely benchmarked turbulent flow past a two-dimensional (2D) square cylinder is considered. The baseline Reynolds number is set to 22 × 10 3 , based on the reference velocity U ∞ and the cylinder lateral size H which is centered at the origin of the domain. The dimensions of the computational domain are [-5H, 15H] × [-7H, 7H] in the streamwise x and crosswise y directions respectively, and the domain is discretized into sufficiently large number of cells to perform a URANS or VMS simulation using an in-house finite-element flow solver [START_REF] Bazilevs | Variational multiscale residual-based turbulence modeling for large eddy simulation of incompressible flows[END_REF]; [START_REF] Takizawa | Stabilization and discontinuity-capturing parameters for space-time flow computations with finite element and isogeometric discretizations[END_REF]; [START_REF] Hachem | Immersed stress method for fluid-structure interaction using anisotropic mesh adaptation[END_REF]; [START_REF] Guiza | Anisotropic boundary layer mesh generation for reliable 3d unsteady rans simulations[END_REF]. The inflow boundary conditions are u = (V in , 0), together with ν = 3ν, which corresponds to a ratio of eddy to kinematic viscosity of approximately 0.2.

For the lateral boundaries, symmetry conditions ∂ y u x = u y = 0 and ∂ y ν = 0 are used. For the outflow, ∂ x u x = ∂ x u y = 0, ∂ x ν = 0 together with p = 0 are prescribed. Finally, no-slip conditions u = 0 and ν = 0 are imposed at the cylinder surface. The time step is ∆t = 0.05 seconds, and the simulation is performed for a total physical time equal to 5000 seconds. Since the wake flow is of interest, around 200 seconds are required for the flow to be established and reach periodic vortex shedding. The data corresponding to the remaining 4980 seconds is stored for training and testing purposes. The data is sampled at each ∆t = 0.25 seconds, thus collecting around 1500 snapshots. In terms of non-dimensional time defined as t * = tU ∞ /H, this sampling rate corresponds to ∆t * = 1. One vortex shedding cycle corresponds to a non-dimensional period T * = 5.23 units, and approximately 24 shedding cycles are observed in simulation data. Given the 70/30 splitting strategy, 16 shedding cycles are observed in training data, which seems reasonable to fully characterize the dynamics of wake turbulent flow past a two-dimensional square cylinder considering its simplicity. Figure shows the sketch of the associated case.

Case 2 : Environmental flow over surface-mounted tower

The turbulent flow past a two-dimensional (2D) rectangular tower on the land surface is considered. The baseline Reynolds number is set to 45 × 10 2 , based on the reference velocity U ∞ and the square tower of sides H which is placed on the surface. The dimensions of the computational domain are [-5H, 30H]×[-H, 7H] in the streamwise x and crosswise y directions respectively, and the domain is discretized into sufficiently large number of cells to perform a URANS or VMS simulation. The inflow boundary conditions are u = (V in , 0), together with ν = 3ν, which corresponds to a ratio of eddy to kinematic viscosity of approximately 0.2.

For the top of the domain, the velocity component normal to the surface is set to zero. No-slip boundary conditions u = 0 and ν = 0 are imposed at the tower surface, as well as the bottom surface at y = -1. The time step is ∆t = 0.01 seconds and 300 seconds are simulated.

Results and Discussion

In this section, the results are discussed as follows: first, the temporal evolutions of the quantities are compared, then the spatial measurements at various times are compared for velocity components. In a second time, temporal propagation of errors and correlation coefficients are compared along with the propagation of phase shifts. Additionally, the contour plots of quantities are also compared to provide qualitative assessments. These comparisons are performed for both the cases and for both the a priori and a posteriori simulations as illustrated in figure 5.4. To compare results, a first a priori simulation is performed by exploiting data samples that were not used during training. The trained model is fed snapshots at instant t, and predicts the next two snapshots at instants t + ∆t and t+2∆t, and the process is repeated by feeding the subsequent snapshots from the dataset, until the same number of snapshots is reached for comparison with the original ground truth time series. As snapshots from the dataset are utilized, this approach is termed a priori deep learning simulation. On the other hand, a posteriori simulation is performed by feeding a snapshot at instant t 0 from the same dataset not used during training, and by predicting the next two snapshots at instants t + ∆t and t + 2∆t. This predicted snapshot at instant t + 2∆t is then injected back into the model to predict snapshots at instants t + 3∆t and t + 4∆t, and the process is similarly repeated until the same number of snapshots are obtained so as to compare with the true snapshots. This way of recycling the model predictions is termed a posteriori deep learning simulation. Once an equal length of time snapshots are obtained, both the a priori and the a posteriori results against the truth from the dataset can now be compared. Figure 5.5 shows the temporal evolution of the ensemble average of velocity magnitude for case 1 and case 2. For case 1, both a priori and a posteriori predictions present a good agreement with respect to the truth, whereas for case 2 the predictions, though fairly accurate, suffer from deterioration. Moreover, the long-term predictions of the model are evident from the accuracy of a posteriori predictions, giving us an indication of global long-term learning while comparing ensemble averages. The accuracy of the predictions is further verified by comparing the values along the various streamwise and cross-streamwise locations. These locations are marked with dashed lines in figure 5.6 for both the cases. For case 1, measurements were made along streamwise directions at cross-streamwise directions at y = -2H, 0, 2H, and similarly for case 2, the measurements were made at x = [-2.5H, 8H, 16H, 32H] and at y = [0.5H, 1.5H, 5H].

As the wake-flows are topic of interest, these locations were chosen based on the region of interest away from the obstacle for both the cases. With regards to temporal evolution, the predictions were compared at a certain percentage of the total predicted snapshots. As a reminder, around 200 snapshots were predicted for case 1 and around 100 snapshots for case 2. The predictions are compared at instants t = [2%, 33%, 66%] to verify the quality of temporal evolution.

Figure 5.8 shows the evolution of temporal predictions of streamwise velocity component u 0 when measured along with cross-streamwise directions. The a priori predictions follow closely the reference values indicating a good agreement with the short-term predictions along with the measured spatial directions. For a posteriori predictions, an increasing deviation from the reference was observed as time evolves, which can be attributed to the accumulation error while making long-term predictions. Similarly, the evolution of the same quantity (u 0 ) when measured along streamwise directions is shown in figure 5.7. A similar trend is observed for the predictions against the reference, where the a posteriori predictions deteriorate as time evolves. It can be noted that the upstream predictions at x = -2.5H are better across times, as it is not affected by the turbulent wake. Overall measurements indicate a decent agreement of both the short-term a priori predictions as well as long-term a posteriori predictions with the reference solutions.

Later, the temporal evolution of prediction error against reference by computing relative mean-squared errors of velocity magnitude for both cases is investigated. These errors are measured along the locations mentioned earlier. Figure 5.9(a) shows the evolution of error for a priori predictions and figure 5.9(b) shows a posteriori predictions for case 1 measured at streamwise locations. As could be expected, the errors accumulate for long-term posterior predictions, leading to a clear distinction when compared to a priori predictions. It is interesting to note that although magnitude increases over time, this evolution also follows the vortex/wake shedding cycles denoting that the trained model performs well for long-term a posteriori predictions. A similar trend is observed for case 2 as shown in figure 5.9(c) and figure 5.9(d), although here the magnitude of accumulated errors is higher than that of case 1. As shown in figure 5.10, a similar trend is observed when measurements of errors were performed at cross-streamwise locations.

The a posteriori predictions are observed to experience high noise conditions caused by error propagation. Hence, extracting the correlation between these two sets (predicted vs. reference) of temporal evolution is important, in particular to assess whether the heavy noise contributions are degrading correlation values. To do so, the Pearson product-moment correlation coefficient R xy of n pairs of time series data {(x 1 , y 1 ), . . . , (x n , y n )} is computed: where n is sample size, x i , y i are the individual sample points indexed with i, and x = 1 n n i=1 x i is the sample mean (analogously for ȳ). In simple terms, R xy is the covariance of the two variables divided by the product of their standard deviations. For our measurements, the two variables are simply the predicted and reference snapshots at the same instants, and computation is performed for both the a priori and a posteriori predictions. Figure 5.11(b) and 5.11(d) show a gradual decrease in the correlation coefficient for the a posteriori predictions for case 1 and case 2 respectively. A steeper degradation of correlation is observed in the measurements at cross-streamwise locations as shown in figures 5.12(b) and 5.12(d) for both cases, while that of the a posteriori predictions remains stable. Since a clear trend is observed in degrading correlations for a posteriori predictions, the phase-shift ϕ(t) were measured for the temporal evolution of velocity magnitude predictions against the reference. Measurements were done along the similar spatial directions as mentioned before, the results of which are shown in figure 5.13. The value ϕ(t) < 0 denotes that predictions are shifted by that value before the reference, and the ϕ(t) > 0 denotes predictions shifted after the reference. For case 1, it is interesting to note that the phase shift goes on increasing in magnitude as time evolves, indicating the model's stability for long-term predictions. However, any clear trend for case 2 when measured at streamwise locations was not observed.

For a qualitative assessment of results, the contours of velocity components for both cases is compared. Figure 5.14 shows the instantaneous snapshots of streamwise velocity contours for case 1 at t = [2%, 33%, 66%] of total predicted snapshots as mentioned earlier, and similar instantaneous snapshots for case 2 are shown in figure 5.15. For both the cases, the a priori, as well as a posteriori predictions, show a fairly accurate agreement with the reference.

Conclusions

A convolutional encoder-decoder-based transformer model has been developed to auto-regressively train on spatio-temporal data of turbulent flows. The method of auto-regressive training works by predicting future fluid flow fields from the previously predicted fluid flow field to ensure long-term predictions without diverging. The model has been validated by demonstrating its applicability to turbulent wake flow past an obstacle and environmental flow past surface mounted obstacle. The work demonstrates a promising model and method for forecasting fluid flow fields where the training data is available. The proposed model trained in an autoregressive way shows significant agreements for a priori evaluations, whereas the posterior predictions show expected deviations after a considerable number of simulation steps. The spatio-temporal complexity of predictions is compara- ble to the target simulations of fully developed turbulence. The autoregressive training and prediction of a posteriori states is the primary step towards the development of more complex data-driven turbulence models and simulations.

  Figure 3.1: 2D square cylinder configuration and mesh used for the study. (5.3a) The cylinder lateral size is denoted H, and is centered at the origin of the domain. The dimensions of the computational domain are [-5H, 15H] × [-7H, 7H] in the streamwise x and crosswise y directions. (3.1b)-(3.1c) The mesh used for CFD computations is refined along with mesh-convergence.

Figure 3 . 2 :

 32 Figure 3.2: Snapshot of velocities u (3.2a), v (3.2b), and turbulent viscosity ν (3.2c) from dataset SqRe22k.

Figure 3 . 4 :

 34 Figure 3.4: Patch extraction from u field. Patches in figure (3.4b) are obtained from the original snapshot (3.4a) For better clarity of the figure, overlapping is only applied in the horizontal direction, and different colors are used to differentiate overlapping patches. Similarly, patches at the same corresponding locations are taken for v and ν fields.

Figure 3 . 5 :

 35 Figure 3.5: Training and validation loss history for the M1 and M2 training methods. The patch-based technique (M2) yields lower error and better generalization than the baseline M1, as evidenced by the negligible gap between validation and training curves. M1 training was performed for 1000 epochs, and the M2 training was stopped after 850 epochs when error stopped improving.

Figure 3 . 6 :

 36 Figure 3.6: Locations of the probe lines used for comparison to CFD reference.

Figure 3 . 7 :Figure 3 . 8 :

 3738 Figure 3.7: Scatter plot and histogram of predicted and expected ν for an out-of-training snapshot of SqRe22k. (3.7a) The plot is a superposition of two scatter plots, namely SA against M1 and SA against M2. (3.7b): The histogram compares the occurence of truth and predictions on a step-type filled histogram.

Figure 3 . 9 :Figure 3 . 10 :

 39310 Figure 3.9: Line plots along x = 10.2 and along y = 0.1 comparing prediction accuracies of M1 and M2 on out-of-training samples from datasets SqRe44k (3.9a)-(3.9e), SqRe88k (3.9b)-(3.9f), CyRe22k (3.9c)-(3.9g) and CyRe88k (3.9d)-(3.9h). M1 and M2 perform similarly on datasets with a square obstacle, even on higher Re values. Yet, M1 consistently fails at predicting accurate ν on samples with cylindrical obstacle, while M2 presents an almost-perfect fit with CFD reference. The small deviation observed for M2 at the top of the square cylinder can be likely attributed to the unstructured-to-structured data sampling, and its study is deferred to a future work.

Figure 4

 4 Figure 4.1: Geometry of 3D turbulent channel flow.

  The basic architecture from ESRGAN by[START_REF] Guo | Long text generation via adversarial training with leaked information[END_REF] is used where most computation is done in the low-resolution feature space. The 'basic blocks' can be designed or chosen such as residual block, dense block, or residual in residual dense blocks (RRDB) for better performance. Image courtesy: Wang et al.[2018]

Figure 4 . 2 :

 42 Figure 4.2: Deep learning architecture used for learning subgrid-scale turbulence

Figure 4 . 3 :

 43 Figure 4.3: Representation of various pooling methods for spatial coarse-graining operations with a sample pool-size of 2×2 from a 4×4 feature space. Max-pooling and mean-pooling refer to taking the maximum and minimum value at each filter respectively and then arranged into a new output with a size of 2 × 2 feature. Sparse-pooling refers to the sparse sampling of the available feature space at each filter.

Figure 4 . 4 :Figure 4

 444 Figure 4.4: Comparison of refinements from max, mean, and sparse sampled coarse-graining methods on the (a) turbulence intensity (b) turbulent dissipation (c) turbulent enstrophy

Figure 4 Figure 4

 44 Figure 4.6: Comparison on the turbulence intensity by refinement direction in (a)XY (b)ZY (c)XZ

Figure 4 CHAPTERFigure 4

 44 Figure 4.11: Comparison of successive refinement on the intensity of turbulence

Figure 4

 4 Figure 4.17: Comparison for PIV on the turbulence intensity by refinement direction in (a)XY (b)ZY

Figure 4 Figure 4 Figure 4

 444 Figure 4.18: Comparison for PIV data on the turbulence dissipation by refinement direction in (a)XY (b)ZY

Figure 4 CHAPTERFigure 4

 44 Figure 4.22: Comparison for PIV data with successive refinement on (a) the intensity of turbulence (b) dissipation of turbulence (c) enstrophy of turbulence

Figure 5 . 1 :

 51 Figure 5.1: The convolutional transformer layer is composed of two blocks: the batched matrix multiplication (BMM) and the self-attention summation. The BMM block corresponds to W i→j x j in equation (5.2), with the batch dimension being the number of spatial locations. It performs k ×k different input-dependent summations with the weights α in equation (5.2). It contains both the learnable filter and the dynamic kernel.

Figure 5 . 2 :

 52 Figure 5.2: Convolutional encoder-decoder transformer architecture Model architecture of the convolutional encoder-decoder transformer to process low and high level features. The canonical four-stage design is utilized in addition to the convolutional transformer blocks or layers. H, W are the input resolutions for each snapshot in T in sequence and T out sequence.

  For a statistically steady state to be reached (periodic vortex shedding to be observed), around 100 seconds are required. The data of the remaining 200 seconds (i.e. approximately 20 × 10 3 time steps) is stored for training and testing purposes. The data is sampled at each ∆t = 0.1 seconds, thus collecting around 2000 snapshots. In terms of non-dimensional time ∂yux = uy = 0, Case 2 setup: Environmental flow over a surface-mounted tower defined as t * = tU ∞ /H, this sampling at each ∆t = 0.1 denotes ∆t * = 1. Figure5.3b shows the sketch of the associated case. Initially, a free separated shear layer expands above the tower and becomes wavy, and then reattaches at the bottom surface of the domain. The shear layer flaps and vortical structures are shed from it. Approximately 18 shedding cycles are observed in simulation data. Given the 70/30 splitting strategy, 12 shedding cycles are observed in training data enough to reasonably characterize the dynamics of environmental flow over the surface-mounted obstacle.

Figure 5 . 5 :Figure 5 . 6 :

 5556 Figure 5.5: Temporal evolution of the ensemble averages for a priori and a posteriori values of velocity magnitude compared to the true values in black. Left: Ensemble mean for spatial values of velocity magnitude for case 1. Right: Ensemble mean for spatial values of velocity magnitude for case 2.

Figure 5 . 8 :Figure 5 . 9 :Figure 5 . 11 :Figure 5

 58595115 Figure 5.8: Comparative predictions of streamwise velocity components (u 0 ) for case 1 (left) and case 2 (right). Figures from top tobottom, denote the predictions at increasing times, i.e. the top row contains instantaneous predictions at t = 0.02 T n , the middle row at t = 0.33 T n , and the bottom row shows the predictions at t = 0.66 T n .

Figure 5 Figure 5 . 14 :

 5514 Figure5.13: Phase-shift evolution for a posteriori values of velocity magnitude with respect to the reference values. The top row shows the evolution for case 1, and the bottom row shows the evolution for case 2. On the left are temporal evolutions measured along with streamwise locations. While on the right are the evolutions measured along with cross-streamwise locations.

Table 3

 3 Two types of obstacles and three different Reynolds numbers are considered, resulting in six different datasets, each holding 3000 snapshots of steady-state velocities and turbulent velocities.

	Dataset name	Re	Obstacle type
	SqRe22k	22 × 10 3	2D square
	SqRe44k	44 × 10 3	2D square
	SqRe88k	88 × 10 3	2D square
	CyRe22k	22 × 10 3	2D cylinder
	CyRe44k	44 × 10 3	2D cylinder
	CyRe88k	88 × 10 3	2D cylinder

.1: Datasets generated for the present study.

ferred to as SqRe22k) composed of 3000 snapshots of steady velocities and SA turbulent viscosities is generated by skipping the transient regime and storing the established regime (i.e. each snapshot is captured only after the flow is estab-

  this article, we have demonstrated the deployment of a robust deep learning model for predicting Spalart-Allmaras eddy viscosities. The method of patchbased training works by dividing the full-scale samples into patches, in order to let the model learn multiple local feature mappings, instead of learning monolithic full-scale features. Applied to an auto-encoder architecture, it was observed that patch-based training led to training and validation errors one order of magnitude lower than standard full-scale training, and was able able to efficiently learn local mappings from multiple datasets with different features, which was not the case of full-scale training method. For practical CFD purposes, a local patch-based model would be of great importance so that any input fluid domain, either full or in parts by region of interest, can be split into patches and passed to the model to predict the quantities of interest. Hence, patch-based training holds an important potential to improve the usability of trained models in the coupling with CFD solvers. Deploying a trained model to solve for turbulent viscosity inside a CFD solver is regarded as a future extension of the present work. Guiza, A Larcher, A Goetz, L Billon, P Meliga, and Elie Hachem. Anisotropic boundary layer mesh generation for reliable 3d unsteady rans simulations. Finite Elements inAnalysis and Design, 170:103345, 2020. Chris Rumsey, Brian Smith, and George Huang. Description of a website resource for turbulence modeling verification and validation. In 40th Fluid Dynamics Conference and Exhibit, page 4742, 2010.

	Elie Hachem, Stephanie Feghali, Ramon Codina, and Thierry Coupez. Immersed Anand Pratap Singh, Shivaji Medida, and Karthik Duraisamy. Machine-learning-
	stress method for fluid-structure interaction using anisotropic mesh adapta-tion. International journal for numerical methods in engineering, 94(9):805-augmented predictive modeling of turbulent separated flows over airfoils. AIAA Journal, pages 1-13, 2017. Chapter 4
	825, 2013. Philippe Spalart and Steven Allmaras. A one-equation turbulence model for
	Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-507, 2006. aerodynamic flows. In 30th aerospace sciences meeting and exhibit, page 439, 1992. Learning Subgrid-scale
	Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Philippe R Spalart. Strategies for turbulence modelling and simulations. Inter-national journal of heat and fluid flow, 21(3):252-263, 2000. Turbulence
	Brendan D Tracey, Karthikeyan Duraisamy, and Juan J Alonso. A machine learn-
	ing strategy to assist turbulence model development. In 53rd AIAA aerospace
	sciences meeting, page 1287, 2015.
	Anikesh Pal. Deep learning emulation of subgrid-scale processes in turbulent
	shear flows. Geophysical Research Letters, 47(12):e2020GL087005, 2020.
	Aakash Vijay Patil and Corentin Lapyere. Development of deep learning methods
	for inflow turbulence generation. arXiv preprint arXiv:1910.06810, 2019.
	Pedro Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for
	scene labeling. In International conference on machine learning, pages 82-90.
	PMLR, 2014.
	Stephen B Pope. Turbulent flows, 2001.

Bibliography

Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation

(OSDI 16)

, pages 265-283, 2016.

Steven R Allmaras and Forrester T Johnson. Modifications and clarifications for the implementation of the spalart-allmaras turbulence model. In Seventh international conference on computational fluid dynamics (ICCFD7), pages 1-11, 2012. Junfeng Chen, Elie Hachem, and Jonathan Viquerat. Graph neural networks for laminar flow prediction around random 2d shapes. arXiv preprint arXiv:2107.11529, 2021. Clement Farabet, Camille Couprie, Laurent Najman, and Yann LeCun. Learning hierarchical features for scene labeling. IEEE transactions on pattern analysis and machine intelligence, 35(8):1915-1929, 2012. Joel H Ferziger, Milovan Perić, and Robert L Street. Computational methods for fluid dynamics, volume 3. Springer, 2002. Kai Fukami, Yusuke Nabae, Ken Kawai, and Koji Fukagata. Synthetic turbulent inflow generator using machine learning. Physical Review Fluids, 4(6):064603, 2019. G SUN Liang, AN Wei, LIU Xuejun, and LYU Hongqiang. On developing datadriven turbulence model for dg solution of rans. Chinese Journal of Aeronautics, 32(8):1869-1884, 2019. Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015. Romit Maulik, Himanshu Sharma, Saumil Patel, Bethany Lusch, and Elise Jennings. A turbulent eddy-viscosity surrogate modeling framework for reynoldsaveraged navier-stokes simulations. Computers & Fluids, page 104777, 2020. Arvind Mohan, Don Daniel, Michael Chertkov, and Daniel Livescu. Compressed convolutional lstm: An efficient deep learning framework to model high fidelity 3d turbulence. arXiv preprint arXiv:1903.00033, 2019. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, pages 285-319, 2010. W Rodi, JH Ferziger, M Breuer, and M Pourquie. Status of large eddy simulation: results of a workshop. Transactions-American Society of Mechanical Engineers Journal of Fluids Engineering, 119:248-262, 1997.

Le présent chapitre se situe à l'intersection de la vision par ordinateur, de l'apprentissage profond et de la turbulence informatique et expérimentale. L'objectif principal était d'explorer l'état de l'art des méthodes d'apprentissage profond pour estimer la turbulence à l'échelle de la maille à partir des grandes échelles mesurées ou des échelles résolues par la maille. La principale contribution de cette étude est d'effectuer un apprentissage par transfert en ré-entraînant un modèle formé pour des problèmes non physiques afin de réapprendre de nouvelles caractéristiques physiques. Les champs à gros grains sont calculés à partir des ensembles de données de référence pour faire les prédictions a priori en utilisant le modèle d'apprentissage profond formé sur l'ensemble de données DNS. Des reconstructions à l'échelle de la maille sont effectuées et comparées aux ensembles de données DNS et PIV de référence. Dans les deux ensembles de données, des ensembles de données d'écoulement de canal turbulent délimité par des murs sont utilisés. Les reconstructions à l'échelle de la maille le long de diverses directions de raffinement et de divers niveaux de grossissement sont comparées à la référence en mesurant des statistiques physiques telles que l'intensité de la turbulence, la dissipation, l'enstrophie et les spectres d'énergie.

  lation (DNS) is limited to flows with low to moderate Reynolds numbers or simple geometries. Complex fluid flows encountered in many engineering and physical applications are computationally demanding to resolve using DNS, hence large eddy simulation (LES) and Reynolds Averaged Navier-Stokes (RANS) modeling are commonly used frameworks that can provide accurate predictions by considering the interaction between the grid-resolved and subgrid scales. On the other hand, experimental measurement techniques such as Particle Image Velocimetry

(PIV)

and Hot-wire Anemometry (HWA) have limited measurement capacity for spatio-temporal resolved velocities. High-frequency tomography PIV has progressed over the last decade, but is still limited to small volumes and low velocities. HWAs provide time-resolved measurements, but combining a large number of HWAs remains challenging and disruptive.

Works involving re-generating turbulence statistics as well as super-resolution has been the focus of multiple recent contributions

Fukami et al. [2019b]

;

Mohan et al. [2019]

;

[START_REF] Beck | Deep neural networks for data-driven les closure models[END_REF]

;

Kim and Lee [2019]

;

  Le modèle proposé peut être optimisé et étendu pour incorporer des conditions physiques et limites supplémentaires, ouvrant ainsi la voie à des simulations plus réalistes de la dynamique des fluides complexes.

	Chapter 5
	Auto-Regressive Learning of
	Spatio-Temporal Turbulence
	064603,
	2019b.
	Kai Fukami, Koji Fukagata, and Kunihiko Taira. Machine learning based spatio-
	temporal super resolution reconstruction of turbulent flows. arXiv preprint
	arXiv:2004.11566, 2020.
	Kai Fukami, Koji Fukagata, and Kunihiko Taira. Machine-learning-based spatio-
	temporal super resolution reconstruction of turbulent flows. Journal of Fluid
	Mechanics, 909, 2021.
	Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-
	Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative ad-
	versarial networks. Communications of the ACM, 63(11):139-144, 2020.
	Julian CR Hunt and David J Carruthers. Rapid distortion theory and the 'prob-
	lems' of turbulence. Journal of Fluid Mechanics, 212:497-532, 1990.
	Chiyu Max Jiang, Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik
	Kashinath, Mustafa Mustafa, Hamdi A Tchelepi, Philip Marcus, Anima Anand-
	kumar, et al. Meshfreeflownet: A physics-constrained deep continuous space-
	time super-resolution framework. arXiv preprint arXiv:2005.01463, 2020.

Kai Fukami, Yusuke Nabae, Ken Kawai, and Koji Fukagata. Synthetic turbulent inflow generator using machine learning. Physical Review Fluids, 4(6):

Junhyuk Kim and Changhoon Lee. Deep unsupervised learning of turbulence for inflow generation at various reynolds numbers. arXiv preprint arXiv:1908.10515, 2019. Un modèle de transformateur basé sur un codeur-décodeur convolutif est proposé pour l'entraînement autorégressif sur des données spatio-temporelles d'écoulements turbulents. La prédiction des champs d'écoulement futurs est basée sur le champ d'écoulement précédemment prédit afin de garantir des prédictions à long terme sans divergence. Une combinaison de réseaux neuronaux convolutifs et d'architecture de transformateur est utilisée pour traiter les dimensions spatiales et temporelles des données. Pour évaluer les performances du modèle, des évaluations a priori sont effectuées et des accords significatifs sont trouvés avec les données de la vérité de terrain. Les prédictions a posteriori, qui sont générées après un nombre considérable d'étapes de simulation, présentent des variances prédites. L'apprentissage autorégressif et la prédiction des états a posteriori sont considérés comme des étapes cruciales vers le développement de modèles et de simulations de turbulence plus complexes basés sur des données. La dynamique hautement non linéaire et chaotique des flux turbulents peut être gérée par le modèle proposé, et des prédictions précises sur de longues périodes peuvent être générées. Dans l'ensemble, cette approche démontre le potentiel des techniques d'apprentissage profond pour améliorer la précision et l'efficacité de la modélisation et de la simulation des turbulences.

of a priori and a posteriori simulations

  x = [-2.5H, 8H, 12H, 16H] and TURBULENCE Left: For a priori simulation, each X t from {X}, the dataset not used in training time, is fed to the model. Right: For a posterioi simulation, the inputs X t are received from its own previous predictions, provided it was initiated with a suitable X t .

	Xt	Xt+∆t Xt+2∆t	Xt	Xt+∆t Xt+2∆t
	Figure 5.4: Illustration

  It is shown that the self-attention transformers incorporated within the convolutional encoder-decoder can predict up to 200∆t time steps with relatively high accuracy, and the proposed data-driven deep learning model remains stable for multiple long time scales, promising a stable and physical deep learning predictive turbulence modeling candidate. Changes to loss function can be done to achieve even longer, stable, physically realistic results. Additional experiments are needed to demonstrate the model's ability on generalizing to local mesh regions as well as longer a posteriori simulation steps. Further investigations on a variety of industrial and academic cases could include training for flow Reynolds numbers, turbulence intensity, and other inlet parameters. Conclusions from this work would also provide valuable insights for the development of new deep learning methods and their deployment for turbulent flows on complex geometries in industrial problems. Deploying a trained model to assist a fluid solver is regarded as a future extension of the present work.

Acknowledgements

Acknowledgements iii

x/H

MOTS CLÉS