text:
Muhammad Shehzad
email: muhammad.shehzad@monash.edu

Bihai Sun

Daniel Jovic

Yasar Ostovan

Christophe Cuvier

Jean-Marc Foucaut

Christian Willert

Callum Atkinson

Julio Soria

Investigation of Large Scale Motions in Zero and Adverse Pressure Gradient Turbulent Boundary Layers Using High-Spatial-Resolution PIV

Keywords: High Spatial Resolution, PIV, Large Scale Motions, Turbulent Boundary Layer, Adverse Pressure Gradient, Zero Pressure Gradient

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Investigation of large scale motions in zero and adverse

pressure gradient turbulent boundary layers using high-spatial-resolution particle image velocimetry

Introduction

The identification and characterization of coherent structures in turbulent flows has been an active area of research for decades. In wall-bounded flows, LSMs are defined as coherent patterns that dominate the log-layer and are characterized as alternating regions of highand low-momentum [START_REF] Kline | The structure of turbulent boundary layers[END_REF][START_REF] Wark | Experimental investigation of coherent structures in turbulent boundary layers[END_REF][START_REF] Ganapathisubramani | Investigation of large-scale coherence in a turbulent boundary layer using two-point correlations[END_REF][START_REF] Del Alamo | Spectra of the very large anisotropic scales in turbulent channels[END_REF][START_REF] Hambleton | Simultaneous orthogonal-plane particle image velocimetry measurements in a turbulent boundary layer[END_REF]. These structures also greatly influence the near-wall region as they superimpose onto the near-wall SSMs and hence, leave their footprints at the wall (Hutchins and Marusic, 2007a). Furthermore, LSMs in the loglayer cause an amplitude modulation of the SSMs in the near-wall region [START_REF] Mathis | Large-scale amplitude modulation of the smallscale structures in turbulent boundary layers[END_REF][START_REF] Harun | Pressure gradient effects on the largescale structure of turbulent boundary layers[END_REF]Hutchins and Marusic, 2007b). These regions of LSMs are elongated up to the order of 20δ in the streamwise direction (Hutchins and Marusic, 2007a), where δ is the boundary layer thickness. They are distinguished from the very thin low-speed streaks in the buffer layer at high Reynolds numbers due to their much larger wall-normal extent [START_REF] Liu | Large-scale modes of turbulent channel flow: transport and structure[END_REF]. A slightly inclined and elongated streamwise velocity correlation function implies that the LSMs are significant contributors to the streamwise turbulent kinetic energy (TKE) as observed by both [START_REF] Grant | The large eddies of turbulent motion[END_REF] and [START_REF] Townsend | The turbulent boundary layer[END_REF].

While there have been a number of studies dealing with the statistical nature of TBLs under the influence of a pressure gradient (see [START_REF] Nagano | Effects of adverse pressure gradients on mean flows and turbulence statistics in a boundary layer[END_REF]; [START_REF] Spalart | Experimental and numerical study of a turbulent boundary layer with pressure gradients[END_REF]; [START_REF] Skaare | A turbulent equilibrium boundary layer near separation[END_REF]; [START_REF] Krogstad | Influence of a strong adverse pressure gradient on the turbulent structure in a boundary layer[END_REF]; [START_REF] Fernholz | The effects of a favourable pressure gradient and of the reynolds number on an incompressible axisymmetric turbulent boundary layer. part 1. the turbulent boundary layer[END_REF]; [START_REF] Na | Direct numerical simulation of a separated turbulent boundary layer[END_REF]; [START_REF] Bourassa | An experimental investigation of a highly accelerated turbulent boundary layer[END_REF]; [START_REF] Kitsios | Direct numerical simulation of a self-similar adverse pressure gradient turbulent boundary layer at the verge of separation[END_REF]; [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF]; [START_REF] Sekimoto | Outer scaling of self-similar adversepressure-gradient turbulent boundary layers[END_REF]; [START_REF] Senthil | Analysis of the spanwise extent and time persistence of uniform momentum zones in zero pressure gradient and adverse pressure gradient turbulent boundary layers[END_REF] among others), literature reveals that only a few characterize the coherent structures [START_REF] Lian | A visual study of the coherent structure of the turbulent boundary layer in flow with adverse pressure gradient[END_REF][START_REF] Zhou | Effect of pressure gradient on coherent structures in a turbulent boundary layer[END_REF][START_REF] Houra | Effects of adverse pressure gradient on quasi-coherent structures in turbulent boundary layer[END_REF][START_REF] Lee | Structures in turbulent boundary layers subjected to adverse pressure gradients[END_REF][START_REF] Drozdz | Detection of coherent structures in a turbulent boundary layer with zero, favourable and adverse pressure gradients[END_REF][START_REF] Kitsios | Direct numerical simulation of a self-similar adverse pressure gradient turbulent boundary layer at the verge of separation[END_REF][START_REF] Sekimoto | Intense focal and reynolds stress structures of a self-similar adverse pressure gradient turbulent boundary layer[END_REF][START_REF] Senthil | Analysis of the spanwise extent and time persistence of uniform momentum zones in zero pressure gradient and adverse pressure gradient turbulent boundary layers[END_REF] and fewer characterize the LSMs [START_REF] Harun | Pressure gradient effects on the largescale structure of turbulent boundary layers[END_REF][START_REF] Hain | Coherent large scale structures in adverse pressure gradient turbulent boundary layers[END_REF][START_REF] Bross | Interaction of coherent flow structures in adverse pressure gradient turbulent boundary layers[END_REF]. Using spectral analysis, [START_REF] Harun | Pressure gradient effects on the largescale structure of turbulent boundary layers[END_REF] studied the effect of pressure gradients on the LSMs in a ZPG-TBL at a Re δ 2 = 8, 160 and in an APG-TBL at a Re δ 2 = 12, 030 and a β = 1.74. They observed that the large scales are energized in the entire APG-TBL and that the amplitude modulation of near-wall small scales by the LSMs increases with an increasing pressure gradient. [START_REF] Bross | Interaction of coherent flow structures in adverse pressure gradient turbulent boundary layers[END_REF] studied the interaction of coherent structures in the near-wall region of an APG-TBL using high-resolution time-resolved 2D and 3D particle tracking velocimetry (PTV). They reported that the high-momentum LSMs in the buffer-and log-layer region were associated with positive fluctuating wall-shear stress +τ ′ w and that low-momentum LSMs were associated with -τ ′ w . Thus, the LSMs in the log-layer significantly manipulate the wall-shear stress, τ ′ w . They also presented a three-layered model of uniform-momentum-zones (UMZs) in a TBL, of which the first zone is associated with the viscous sublayer, the second is formed by the high-and low-speed streaks in the buffer-layer and the third is generated by the highand low-momentum LSMs in the log-layer. When these low-and high-momentum LSMs coincide with ejections or sweeps (i.e. Q2 and Q4 events according to the quadrant analysis of [START_REF] Wallace | The wall region in turbulent shear flow[END_REF]), the number of UMZs strongly increases or decreases, respectively [START_REF] Paakkari | Uniform momentum zones in a turbulent boundary layer above a surface with spanwise traveling wave[END_REF]. Hence, [START_REF] Bross | Interaction of coherent flow structures in adverse pressure gradient turbulent boundary layers[END_REF] conclude that the momentum of log-layer LSMs and their footprint on the wall are related to the number of UMZs in the instantaneous velocity fields.

Proper Orthogonal Decomposition (POD) has been used as a tool to study the coherent structures, including LSMs since its first application in fluid dynamics by [START_REF] Lumley | The structure of inhomogeneous turbulent flows[END_REF]. It is a generalization of the conventionally used Fourier power spectral analysis and is used to investigate the TKE distribution as a function of scale in a TBL flow that is inhomogeneous in the streamwise direction [START_REF] Liu | Large-scale modes of turbulent channel flow: transport and structure[END_REF]. POD has been used to study the randomly distributed counter-rotating eddies as LSMs in a turbulent pipe flow by Bakewell Jr and [START_REF] Lumley | The structure of inhomogeneous turbulent flows[END_REF] at a Re D = U b D/ν = 8, 700, where U b is the bulk velocity and D is the pipe diameter. [START_REF] Liu | Large-scale modes of turbulent channel flow: transport and structure[END_REF] used POD to evaluate the scales contributing towards the events that produce TKE and Reynolds shear stress from the 2D data of two different channel flows at a Re h = U b h/ν = 5, 378 and 29, 935 (where h represents the channel half-height) and concluded that the LSMs contain a large fraction of the Reynolds streamwise stress component and a small fraction of Reynolds wall-normal stress component. [START_REF] Wu | A study of energetic large-scale structures in turbulent boundary layer[END_REF] studied the LSMs in a TBL with a zero pressure gradient (ZPG) at two different configurations, Re δ 2 = 8200 and 12000, and performed POD to establish a connection between the first two dominant POD modes and the instantaneous large scale structures. The authors concluded that the Reynolds streamwise stress, Reynolds shear stress and spatial velocity correlation functions are reduced without the LSMs that significantly contribute to the dominant first POD mode.

By performing POD on a dataset, the LSMs are characterized as structures whose contribution to the dominant spatial mode is above a threshold. The classification of LSMs into high-and low-momentum events is based on the nature of the first POD mode. Conditional averaging of velocity fields based on this classification leads to the analysis of the contribution of the extreme events to the turbulent statistics. Guemes et al. studied the LSMs in a ZPG-TBL by using POD to investigate the effect of high-and low-momentum events on turbulent statistics. They observed that the high-momentum events have a larger influence on the mean flow and Reynolds stresses near the wall when compared to the low-momentum events, and this effect decreases with an increasing distance from the wall. Conversely, the lowmomentum events had a weaker contribution to the inner peak and a stronger contribution to the outer peak in Reynolds streamwise stress profile. To the best of authors' knowledge, the effect of the high and low-momentum events on the turbulent statistics in TBL under the influence of an adverse pressure gradient has not yet been investigated.

Earlier studies on the LSMs in wall-bounded flows were carried out using either singlepoint measurements or low-spatial-resolution 2C-2D PIV measurements. To investigate the effect of high-and low-momentum events near the wall, HSR measurements are of immense importance. In the present paper, HSR 2C-2D PIV measurements have been used to investigate the LSMs and the effect of high-and low-momentum events on the turbulent statistics in both the ZPG-and the APG-TBLs. This paper is organized in the following manner:

The methodology of the POD on the instantaneous fluctuating flow field and classification of LSMs into high-and low-momentum events is presented in Section 2. Section 3 describes the experimental details of the HSR 2C-2D PIV measurements. The first-and second-order statistics, conditionally averaged statistics and sensitivity analysis of different threshold limits along with discussion are presented in Section 4. Lastly, the conclusions are presented in Section 5.

Throughout this paper, we take x, y and z as the streamwise, wall-normal and spanwise directions, respectively. The instantaneous, mean and fluctuating velocities in the x directions are referred to as u, U and u ′ , respectively. Accordingly, the velocities in y and z are represented by v and w.

2. Identification of the Large Scale Motions using POD 2.1. Proper Orthogonal Decomposition (POD) POD using a snapshot method was introduced by [START_REF] Sirovich | Turbulence and the dynamics of coherent structures. parts i-iii[END_REF], briefly discussed in [START_REF] Taira | Modal analysis of fluid flows: An overview[END_REF] and used in this study to extract modes based on optimizing the mean square of the fluctuating velocity. The method is briefly presented below.

Consider a set of velocity fields u ′ (x, t),

u ′ (x, t) = [u ′ (x, t 1 ) u ′ (x, t 2 ) ... u ′ (x, t N )] ∈ R M ×N , M ≫ N (1)
where N is the number of snapshots (i.e. velocity fields) and M is twice the number of grid points in each snapshot and

u ′ (x, t) = u ′ (x, t) v ′ (x, t) (2)
u ′ (x, t) can be decomposed by POD in the following manner

u ′ (x, t) = N i=1 a i (t)φ i (x) (3)
where φ i (x) is the ith spatial mode and a i (t) is the set of the corresponding temporal coefficients. Let X = u ′ . In the regular POD,

Rφ i = λ i φ i , φ i ∈ R M , i = 1, 2, ... , N (4)
where λ i represents the eigenvalue of the ith mode and R is the covariance matrix of vector u ′ (x, t) such that

R = XX T , R ∈ R M ×N (5)
In snapshot-POD, the matrix X T X is used which is much smaller in size as compared to the regular POD matrix XX T and yet has the same nonzero eigenvalues [START_REF] Sirovich | Turbulence and the dynamics of coherent structures. parts i-iii[END_REF]. Hence we can write

X T Xψ i = λ i ψ i , ψ i ∈ R N , (6)
With the eigenvector (i.e. vector of temporal coefficients) ψ i and eigenvalue λ i , the corresponding spatial mode φ i can be computed as

φ i = Xψ i 1 √ λ i (7)
This can also be written as

Φ = XΨΛ -1/2 (8)
where the columns of Φ are the vectors of the spatial modes (Φ = φ 1 φ 2 ... φ N ∈ R M ×N ), the columns of Ψ are the vectors of temporal coefficients corresponding to each POD mode (Ψ = ψ 1 ψ 2 ... ψ N ∈ R N ×N ) and Λ is the vector of eigenvalues corresponding to each POD mode (Λ = λ 1 λ 2 ... λ N ∈ R N ).

The TKE equals to half of the sum of the eigenvalues, i.e.

k = 1 2 u ′ 2 = 1 2 N i=1 λ i (9)
The singular value decomposition (SVD) is defined as

u ′ = ΦΣΨ T (10)
Equation 8 can be written in the form of equation 10 such that Σ = Λ 1/2 and Ψ -1 = Ψ T (as Ψ is an orthogonal matrix), and hence the current POD problem can be solved as an SVD problem [START_REF] Taira | Modal analysis of fluid flows: An overview[END_REF].

Classification of the Large Scale Motions

As described in [START_REF] Wu | A study of energetic large-scale structures in turbulent boundary layer[END_REF], the velocity fields dominated by LSMs are identified as the fields for which the magnitude of the temporal coefficients ψ i j are beyond a threshold Kσ ψ i , where σ ψ i is the standard deviation of the temporal coefficients corresponding to the ith POD mode, and K = {1.0, 1.5, 2.0} is a threshold factor. As the first POD mode contains the largest contribution of the TKE, we take i = 1 to separate the flow field into the velocity fields with LSMs and SSMs.

F (|ψ i | > Kσ ψ i ) -→ F (LSM), F (|ψ i | < Kσ ψ i ) -→ F (SSM) (11)
The LSMs are further classified into the high-momentum (u ′ > 0) and low-momentum (u ′ < 0) events based on the nature of the first POD mode [START_REF] Wu | A study of energetic large-scale structures in turbulent boundary layer[END_REF][START_REF] Guemes | Experimental assessment of large-scale motions in turbulent boundary layers[END_REF]. If the streamwise component of the first mode is positive, i.e. u ′ φ 1 > 0 , the snapshots of the flow field F with their temporal coefficients ψ 1 larger than Kσ ψ 1 are identified as those with dominant high-momentum events (H + ) and those with ψ 1 smaller than -Kσ ψ 1 are identified as those with dominant low-momentum events (L + ).

F (ψ 1 > Kσ ψ 1 ) -→ H + , F (ψ 1 < -Kσ ψ 1 ) -→ L + (12)
Conversely, if u ′ φ 1 < 0, the snapshots of the flow field F with ψ 1 > Kσ ψ 1 are identified as fields with dominant low-momentum events (L -), and those with ψ 1 < -Kσ ψ 1 as fields with dominant high-momentum events (H -)

F (ψ 1 > Kσ ψ 1 ) -→ L -, F (ψ 1 < -Kσ ψ 1 ) -→ H - (13)
The + and -superscripts have been adopted to differentiate between the nature of the first POD mode and subsequent classification in the two data-sets (i.e. of ZPG-and APG-TBLs). The basis of the selection of the optimal value of K is described in section 4.2 and compared with the other values to analyse their effect on the conditionally averaged turbulent statistics in section 4.4.

Experimental methodology

Facility and Apparatus

High-spatial-resolution 2C-2D PIV images were taken in the x -y plane in the LMFL High-Reynolds-Number Boundary Layer Wind Tunnel at Laboratoire de Mécanique des Fluides de Lille (LMFL), Lille, France. This facility has a 2m-wide, 1m-high and 20.6m-long test section. A schematic diagram of the wind tunnel facility is shown in figure 1 with the three different sections indicated; with a zero-pressure-gradient, with a favourable-pressure-gradient (FPG) and finally with an adverse-pressure-gradient. Previously in the EuHIT experiment [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF], 2C-2D PIV measurements of an FPG-TBL with four sCMOS cameras at four stations and an APG-TBL with 16 sCMOS cameras in a 3.466m long continuous field of view (FOV) were obtained to characterize the high Reynolds number FPG-and APG-TBL developing over considerably long regions of the test section before entering the FPG region. The details of the EuHIT experiment are reported in [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF]. 2C-2D Planar PIV and 3C-3D Stereo-PIV (SPIV) measurements of the ZPG-TBL were taken at x = 3.2m and x = 6.8m respectively to characterize the inlet conditions of the TBL.

To study the dynamics of the TBL in the near-wall and log regions, HSR 2C-2D PIV measurements of a ZPG-TBL were taken at x = 6.8m, whereas the measurements of the APG-TBL were taken at s = 5.6m, where s is the curvilinear coordinate along the ramp surface with s = 0 at the beginning of the ramp. The inflow velocity for both measurements is 9m/s. A 29 Megapixel Imperx Bobcat B6640 camera was used to record these images. The camera has a CCD sensor with 6576 x 4384 pixels and a pixel size of 5.5µm. The image magnification was 12.684µm/px for ZPG-TBL and 12.726µm/px for APG-TBL. The FOV was (56.190×46.449)mm for the ZPG-TBL and (47.213×49.656)mm for the APG-TBL. Due to the limited number of high-resolution cameras available, HSR 2C-2D PIV measurements of the whole APG region in the LMFL wind tunnel were not possible. Therefore, APG-TBL measurements were taken only on at one station in the APG region. The FOV was illuminated by a dual cavity, frequency-doubled Innolas Nd:YAG laser for ZPG-TBL and BMI laser for APG-TBL with the maximum energy of 150mJ and 200mJ per pulse at a wavelength of 532nm. The laser sheet was created by a combination of converging and diverging lenses to reduce the thickness in the out-of-plane direction and expand the laser beam to form a sheet in the in-plane direction. Two cylindrical lenses of focal lengths of f = -1000mm at the output of the laser and f = +400mm at about 400mm upstream the FOV were used to create the laser sheet for the ZPG-TBL measurements. For the APG-TBL measurements, a cylindrical lens of f = -40mm and a spherical lens of f = 1500mm were used to create the required laser sheet. The thickness of the laser sheet across the field of view was measured to be about 400µm for the ZPG-TBL and about 200µm for the APG-TBL. The laser sheet for the ZPG-TBL was introduced from underneath the glass-floor of the wind tunnel at x = 6.8m. For the APG-TBL, the laser sheet was introduced from the bottom at a location downstream of the APG region but mirrored to align its bottom edge coincident with the -5 • inclined plate. The elapsed time between the two laser pulses ∆t was set to be 30µs for both TBLs to achieve an optical maximum particle image displacement of 20px on the sensor. The flow was seeded with droplets of a water-glycol mixture with a mean diameter of around 1µm, using a smoke generator. For each TBL, over 10, 000 image pairs were recorded. As the purpose was to take HSR 2C-2D PIV images, the measurements for both TBLs could not capture the whole boundary layer, but instead only a region from the wall up to the beginning of the wake region. The boundary layer parameters for both TBLs which could not be computed from the current HSR 2C-2D PIV data, have been adopted from [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF] as presented in table 1.

Distortion Correction

When measuring with large imaging sensors, lens distortions are introduced into the PIV images. These are usually radial distortions which are a minimum at the centre of the image and a maximum in the corners. To correct for these distortions, the PIV images are dewarped using a second-order rational function (R22) [START_REF] Willert | Stereoscopic digital particle image velocimetry for application in wind tunnel flows[END_REF]

a 33 = 1
where (x i j , ŷi j ) and (x o j , ŷo j ) are coordinates of the image points and the object points in the centred and normalized image space { PI } and object space { PO } respectively. j = [1, 2, 3, ... , N m ] represents a pair of the corresponding image and object points for an individual marker and N m is the total number of markers in the calibration target image.

The locations of all the markers are obtained with a sub-pixel accuracy by cross-correlating the calibration image with a custom marker template and using a peak finding algorithm. This creates the image space, P I . The approximate magnification M is calculated from the average distance between the four markers in the centre of the image space because distortion is minimum there. M is then used to create the object space {P O } with its origin coincident with the origin of the image space. The centred and normalized image space and object space are obtained from their original counter parts P I and P O as follows:

{ PI } = {P I } -P ic x omax (15) { PO } = {P O } -P oc x omax ( 16
)
where P ic and P oc are the image and object points at the middle of the image space and object space respectively and x omax is the largest x coordinate in the object space.

The difference of the centred and normalized object and image points is highlighted in Fig. 2 where the arrows extend from the object points towards their corresponding image points. The maximum difference of the image points and object points are observed to be 6.73px and 5.68px in the ZPG-and APG-TBLs PIV images respectively. As shown in the figure, the distortion in ZPG-TBL images is a combination of a solid body rotation and pincushion type while in APG-TBL, it is mostly the pincushion type distortion.

Complete details of the distortion correction method are given in [START_REF] Sun | Distortion correction in 2c-2d piv measurements of turbulent boundary layer with large sensor size[END_REF]. The characteristic parameters required to compute the dewarping coefficients [a 11 , a 12 , ... , a 16 , a 21 , a 22 , ... , a 26 , a 31 , a 32 , ... , a 36 ] of equation 14 for distortion correction of the current HSR ZPG-and APG-TBLs measurements are listed in table 2.

PIV Analysis

The PIV images were analysed using a multigrid/multipass digital PIV [START_REF] Soria | An investigation of the near wake of a circular cylinder using a video-based digital cross-correlation particle image velocimetry technique[END_REF] with in-house software. The parameters of the PIV analysis are given in terms of viscous units in table 3. In order to compare the first and second-order statistics in the outer layer, profiles of the ZPG-and APG-TBLs from the EuHIT experiment measured at the same streamwise stations and their PIV analysis parameters, are presented in section 4.1. These have been adopted from [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF]. The current measurements are approximately 3 times more resolved in the streamwise direction for both of the TBLs while being about 27 and 8 times more resolved in wall-normal direction for the ZPG-and the APG-TBLs respectively when compared to their EuHIT counterparts from [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF]. As the current PIV measurements have significantly higher resolution in the wall-normal direction, the wall shear stress τ w = µ dU dy y=0

and friction velocity u τ = τ w /ρ for both the TBLs are measured directly from wall-normal gradient of the mean streamwise velocity in the viscous sublayer at every streamwise grid-point. The mean values of the measured friction velocities are given in table 4. These values are in good agreement with those measured in [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF] with a deviation of about 1%.

Results and discussion

First-and second-order statistics

Mean streamwise velocity profiles of ZPG-and APG-TBLs normalized by the viscous units are shown in Fig. 3. The ZPG-TBL has a longer log-law region than that of the APG-TBL.

For comparison, the outer-layer mean velocity profiles of the ZPG-and APG-TBLs from the EuHIT experiment have also been included which are consistent with their respective profile from the current HSR measurements. This is expected because of the same facility and nearly matching experimental conditions used for the two experiments. Moreover, the nearwall profiles of ZPG-TBL [START_REF] Willert | Experimental evidence of near-wall reverse flow events in a zero pressure gradient turbulent boundary layer[END_REF] and APG-TBL [START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF] obtained using time-resolved 2C-2D PIV (TRPIV) have also been included. As the current HSR PIV profiles collapse well with the profiles from the earlier PIV measurements in the respective pressure gradient, this shows sufficient repeatability of the experiments. Reynolds stress profiles scaled in viscous units are shown in Fig. 4. The ZPG-TBL has roughly a 20% lower inner peak in the Reynolds streamwise stress (u ′ u ′ ) profile when compared to the APG-TBL and has no outer peak, which is consistent for TBLs in this Reynolds number regime. Whereas in the APG-TBL, the outer peak located at y + ≈ 800 (where y + denotes the wall viscous units) is almost as strong as the inner peak. The profile of Reynolds wall-normal stress v ′ v ′ has a less pronounced peak at around y + ≈ 500 in the ZPG-TBL but is still more than two times weaker than the peak in the APG-TBL, which is also further away from the wall. Reynolds shear stress u ′ v ′ shows a plateau in the region y + ≈ 100 -300 in the ZPG-TBL and an outer peak at y + ≈ 900 in the APG-TBL. This shows that turbulence moves away from the wall towards the outer region when the TBL is under the influence of an APG. Profiles of Reynolds stresses in the outer layer from the EuHIT experiment have also been included for comparison. These profiles deviate by less than 5% from the HSR measurements in the region y + ≈ 130 -500 which could be the result of their low-spatial-resolution and more averaging due to larger interrogation windows (IWs) used. Above y + = 500, EuHIT profiles are in good agreement with the current HSR measurements. The near-wall profiles of ZPG-and APG-TBL from the TRPIV have also been included. While these profiles are in good agreement with the current HSR measurements for Reynolds wall-normal and shear stresses, they show 2-3% larger inner peak for Reynolds streamwise stress. The terms that significantly contribute to the turbulence production in the current TBLs are -u ′ v ′ ∂U/∂y and v ′ 2 ∂U/∂y. These terms are present in the equations for u ′ 2 and -u ′ v ′ respectively, in the transport equation for the turbulent stress u ′ i u ′ j [START_REF] Skaare | A turbulent equilibrium boundary layer near separation[END_REF]. The profiles of these terms are shown in Fig. 5 where the inner peaks are located at y + ≈ 12 and y + ≈ 18, respectively. The peak heights of these terms in the APG-TBL are roughly 15% and 30% larger than in the ZPG-TBL. This shows a larger turbulence production in the APG-TBL as compared to the ZPG-TBL in the buffer layer. No significant outer peaks are found in either of these terms. For the APG-TBL, this is in contrast to the findings of Skaare and [START_REF] Skaare | A turbulent equilibrium boundary layer near separation[END_REF] who found two distinct peaks in the inner and outer regions for each of these terms. They could not resolve the locations of the inner peaks because of their hot-wire data having low-spatial-resolution in the wall-normal direction. However, their outer peaks were observed at locations where stresses are maximum (y/δ = 0.45). The absence of the significant outer peak in the present study is expected because of the mild APG (β ≈ 2) as compared to strong APG in [START_REF] Skaare | A turbulent equilibrium boundary layer near separation[END_REF] (β ≈ 21). In the current measurements, the inner peaks of -u ′ v ′ ∂U/∂y are almost two times stronger than the those of v ′ 2 ∂U/∂y, while both are higher in the APG-TBL than in the ZPG-TBL.

′ u ′ + , v ′ v ′ + , u ′ v ′ + u ′ u ′ + v ′ v ′ + u ′ v ′ + ZPG EuHIT -ZPG (SPIV) EuHIT -ZPG (TRPIV) APG EuHIT -APG EuHIT -APG (TRPIV)

POD Modes

As aforementioned, POD was used to analyse the HSR measurements of the ZPG-and APG-TBLs. As the EuHIT TBLs do not cover the inner-layer, POD was not performed on those data-sets to avoid erroneous measurements. Therefore, the results from here onwards will rely only on the HSR measurements. The profiles of eigenvalues corresponding to the individual POD modes and their cumulative sum as a fraction of the total energy are shown in Fig. 6. The POD modes are sorted conventionally, such that the contribution to TKE decrease with increasing mode number i i.e. λ 1 λ 2 ...λ N 0. The first modes of the ZPG-and APG-TBLs contain 32% and 42% of the total energy in the flow, respectively. For i 2, the relative contribution of each mode is roughly similar between ZPG-and APG-TBLs. It is also interesting to observe that more than 100 modes are required to obtain 90% of the energy which indicates the highly stochastic (random) nature of the TBL.

The contour plots of the streamwise component of the spatial modes u ′ φ i for i = {1, 2, 3, 4, 5} are shown in figure 7 and for i = {1, 11, 21, 31, 41} in figure 8 for both TBLs. As shown from the two figures, the first mode has the largest scales and hence contributes the most to the TKE. With increasing i, the scales become smaller and smaller which is the reason for their reduced contribution to the total TKE.

Probability density functions (PDFs) of the temporal coefficients corresponding to the first five POD modes for the ZPG-and APG-TBLs are shown in Fig. 9. For each TBL, the first five modes have a similar distribution of their temporal coefficients which appear Gaussian. As all velocity fields (i.e. snapshots) contribute differently to the first and every other POD mode, those fields which have the largest contributions to the large scales of the first mode φ 1 can be characterized as the fields with dominant LSMs. The extent of contribution of each velocity field to the first spatial mode is reflected by the magnitude of its temporal coefficient corresponding to φ 1 . This is an enhanced definition of the basis of the classification described in section 2.2. The threshold K in Eq. 11 is selected based on the distribution of ψ 1 . Given that the temporal coefficients of the first POD mode have a Euclidean norm of 1 (similar to results of Guemes et al.), K = 1 is chosen for the classification of LSMs into high and low-momentum events. A sensitivity analysis of the second-order turbulent statistics without LSMs using the classification based on the different values of the threshold factor K = {1.0, 1.5, 2.0} is presented in section 4.4.

To investigate the effect of high-and low-momentum LSMs on turbulent statistics, the flow fields F of each TBL is divided into three parts: (i) high-momentum LSMs (H), (ii) low-momentum LSMs (L) and (iii) SSMs. Further classification of the fields with LSMs into those dominated by high-or low-momentum events is based on the signs of ψ 1 j and u ′ φ 1 . As u ′ φ 1 < 0 for the ZPG-TBL, the velocity fields of the ZPG-TBL with high-and low-momentum events are H -and L -, respectively. In contrast, as u ′ φ 1 > 0 for the APG-TBL, the velocity fields of the APG-TBL with high-and low-momentum events are H + and L + , respectively. The distributions of the temporal coefficients of velocity fields with LSMs for both TBLs are shown in figure 10 where the red regions correspond to high-momentum events and the blue regions correspond to low-momentum events. These regions are bounded by dotted lines of ψ 1 j = ±σ ψ 1 and solid lines of P DF (|ψ 1 j | > σ ψ 1 ). The dashed lines and dash-dotted lines represent K = 1.5 and K = 2.0 from figure 11, respectively.

∑ i = 1 (λ i )/ N ∑ i = 1 (λ i )
Fractional contribution -ZPG Fractional contribution -APG Cumulative fractional contribution -ZPG Cumulative fractional contribution -APG

Conditional Averaging of the Turbulent Statistics

The conditional averaging of turbulent statistics is performed on the sub-fields H and L as presented in table 5. The -and + superscripts differentiate between the nature of the first POD mode of the TBLs (see sections 2.2 and 4.2). The subscripts h and l represent the conditionally averaged statistics from the H and L sub-fields, respectively. This notation will be followed in the profiles of the conditionally averaged turbulent statistics.

The profiles of these statistics, along with the dominant term in turbulence production -u ′ v ′ ∂U ∂y scaled in viscous units are shown in figure 12 for the ZPG-and APG-TBLs. The profiles from the original ensemble have also been included for comparison. As shown in the mean velocity profiles of the ZPG-TBL (figure 12(a)), high-momentum events have a higher mean velocity when compared to the original ensemble, while the opposite is true for the low-momentum events. A similar trend is observed in the APG-TBL (figure 12(b)), but profiles of high-and low-momentum motions are farther from the original ensemble in the outer region when compared to the ZPG-TBL which shows that the activity of the LSMs is enhanced by the APG in the outer layer.

For the Reynolds streamwise stress in the ZPG-TBL (figure 12(c)), the high-momentum events contribute more near the wall and in the log region when compared to the lowmomentum events while the opposite is the case in the wake region. Near the wall, highmomentum events have positive contributions and low-momentum events have negative contributions to Reynolds streamwise stress. This is in agreement with observations of Guemes et al., who performed similar investigations in a ZPG-TBL. The near-wall peak in u ′ u ′ h is about 22% more pronounced compared to the original ensemble. Both high-and low-momentum events produce an outer peak that is almost as strong as the inner peak in -4 -3 -2 -1 0 1 2 3 4

PDF i = 1 i = 2 i = 3 i = 4 i = 5 -4 -2 0 2 4 Ψ ij /σ Ψi i = 1 i = 2 i = 3 i = 4 i = 5
Ψ 1j /σ Ψ1 0.0 0.1 0.2 0.3 0.4 PDF L - H - ZPG (u ′ ϕ1 < 0) -4 -3 -2 -1 0 1 2 3 4 Ψ 1j /σ Ψ1 H + L + APG (u ′ ϕ1 > 0)
Figure 10: Classification of LSMs in to high-and low-momentum events depicted on the PDFs of temporal coefficients corresponding to the first POD modes. The vertical dotted lines correspond to the ψ 1 /σ ψ1 = ±1.0, dashed lines to ψ 1 /σ ψ1 = ±1.5 and the dash-dotted lines to ψ 1 /σ ψ1 = ±2.0.

High-momentum Low-momentum

Property ZPG APG Property ZPG APG

U h u[H -] u[H + ] U l u[L -] u[L + ] u ′ u ′ h (u ′ [H -]) 2 (u ′ [H + ]) 2 u ′ u ′ l (u ′ [L -]) 2 (u ′ [L + ]) 2 v ′ v ′ h (v ′ [H -]) 2 (v ′ [H + ]) 2 v ′ v ′ l (v ′ [L -]) 2 (v ′ [L + ]) 2 u ′ v ′ h u ′ [H -]v ′ [H -] u ′ [H + ]v ′ [H + ] u ′ v ′ l u ′ [L -]v ′ [L -] u ′ [L + ]v ′ [L + ]
the original ensemble.

Similarly, high-momentum events contribute more to the Reynolds wall-normal stresses than low-momentum from the wall to the end of the log-layer and the opposite is the case in the wake region (see figure 12(e)). The same near-wall distribution is also found in Reynolds shear stress (see figure 12(g)). The Reynolds streamwise and shear stress profiles for highand low-momentum events are up to two times stronger in the wake region compared to the corresponding profiles from the original ensembles. This leads to the conclusion that the LSMs have significantly strong activity in the outer region, especially in the streamwise velocity fluctuations. Another clear observation in the Reynolds stresses is that the cross-over point of the high-and low-momentum profiles happens to be at values higher than the original ensemble which is contradictory to the findings of Guemes et al.. Moreover, Guemes et al. found that the cross-over points in all Reynolds stresses are at nearly the same wall-normal location, which is not the case in the present study. This is probably because the HSR data in this study does not cover the whole boundary layer, which affects the output of POD and hence the classification such that the cross-over point falls at different y location for different Reynolds stresses.

In the APG-TBL case, the contributions of the high-and low-momentum LSMs to Reynolds stresses (see figure 12-{(d),(f),(h)}) show similar trends as in the ZPG-TBL, but their outer peaks are slightly more distant from the original ensemble. This shows the further activation of LSMs in the presence of an APG. Another observable difference is that the cross-over point of the LSM profiles in the APG-TBL moves further from the wall than in the ZPG-TBL.

The profiles of -u ′ v ′ ∂U ∂y show that high-and low-momentum LSMs have inner peaks which are about 25% stronger and weaker than the original ensemble respectively (see figure 12(i)). The cross-over point in these profiles is also above the original ensemble and is located near the wall-normal location of the cross-over point in the Reynolds shear stress profiles. These changes in the turbulence production near the wall are in agreement with the results of Hutchins and Marusic (2007a), who suggest that the LSMs of the log layer have footprints in the near-wall region. In the APG-TBL (see figure 12(j)), this spread between the highand low-momentum LSMs is about 50% larger than in the ZPG-TBL, and their cross-over point is also farther in wall-normal direction. This shows that the TKE production due to the high-and low-momentum events is enhanced and reduced, respectively, by the presence of an APG.

Sensitivity analysis

As described in section 2.2, the flow-field is separated into large-and small-scale motions based on a threshold value of the temporal coefficient of the first (i.e. the most energetic) POD mode, ψ 1 j . As ψ 1 has a unitary Euclidean norm, the threshold factor K is taken as 1. It is, however, worthwhile to investigate the effect of larger values of K (i.e. 1.5 and 2) on the conditionally averaged turbulent statistics to find out what portion of LSMs will be subtracted if the larger values of K are used. Because of the Gaussian nature of the distribution of ψ 1 as shown in figure 9, fewer and fewer number of velocity fields match the criterion |ψ 1 | > Kσ ψ 1 for values of K increasing from 1.0 to 2.0. Table 6 lists the percentage values of the number of the velocity fields matching this criterion for K = {1.0, 1.5, 2.0}. As evident, the effect of a mild APG on these statistics is negligible as these statistics are very similar for both TBLs. The sensitivity of Reynolds stresses to the sorted LSMs based on varying the values of K is analysed below. Figure 13 shows the profiles of Reynolds stresses in a ZPG-and an APG-TBL, computed from ensembles without the fluctuating velocity fields whose ψ 1 values are larger than Kσ ψ 1 where K = {1.0, 1.5, 2.0}. These profiles are normalized with the corresponding profiles from the original ensemble. As a larger contribution of the LSMs are removed, the Reynolds stress values are lower than the original ensemble. As shown in figure 13, the effect of a change in K is felt in all of the Reynolds stress profiles in the inner and outer region, but it is more pronounced in the outer region of the Reynolds streamwise and shear stresses. Reynolds wall-normal stresses show very little response to a change in K. It is found that the reduction in the Reynolds streamwise stress is the highest, roughly 38% in the ZPG and 43% in the APG, for the smallest value of K, which is expected as more instantaneous fluctuating velocity fields with larger ψ 1 are excluded from the ensembles even though they are significant contributors to the TKE. This reduction effect becomes less and less as K increases from 1 to 2 and becomes nearly 10% in the ZPG-TBL and 12% in the APG-TBL at K = 2. Although the reduction of the stresses in the inner region after removing LSMs is minimal, it shows that the near-wall scales are affected by the LSMs of the outer region and this effect is also slightly amplified in a mild APG.

In the Reynolds shear stress, the reduction for different values of K is about 10% and 15% larger compared to the Reynolds streamwise stress in the ZPG and APG, respectively. One can conclude that LSMs based on K = 1 contribute up to 50% of the Reynolds shear stress in the outer region of the boundary layer.

Profiles of the mean streamwise velocity U and the dominant turbulence production term -u ′ v ′ ∂U ∂y are almost insensitive to the change in K and hence, those have not been presented here for the purpose of brevity.

Conclusion

POD analysis has been performed on the fluctuating velocities of the flow field of a ZPGand an APG-TBL obtained using high-spatial-resolution PIV. Based on a threshold value K, the temporal coefficients of the first POD mode are used to classify the flow field into those with large-and small-scale motions. The fields with large-scale structures are further divided into those with high-and low-momentum events based on the nature of the first POD mode and the values of its temporal coefficients. The conditionally averaged turbulent statistics validate the findings of previous studies in which large-scale motions have significant contributions in the outer region and contribute largely to Reynolds streamwise and shear stresses in a ZPG-TBL. These motions become even more energized in the presence of a mild APG. In the near-wall region, the high-and low-momentum events contribute more and less, respectively, to the turbulent statistics when compared to the original ensemble. Their effect is reversed in the outer region following a cross-over point in the Reynolds stress profiles, which always happens to be above the values averaged over the original ensemble. The two notable differences in the APG-TBL are: (i) high-and low-momentum events are more energized and (ii) the cross-over point moves farther from the wall in the APG-TBL than in the ZPG-TBL.

Removing the velocity fields with the LSMs based on K = 1, The estimate of the Reynolds streamwise stress and Reynolds shear stress from the remaining velocity fields in the ZPG-TBL are reduced by up to 42% compared to the original ensemble. This reduction effect is enhanced by roughly 10% in a TBL with a mild APG. This further signifies the conclusion that the LSMs are amplified in the presence of a mild APG.

Figure 1 :

 1 Figure 1: Schematic of the test section in the LML Wind Tunnel. Figure adapted from Cuvier et al. (2017).

Figure 2 :

 2 Figure2: Difference of the centred and normalized object and image points with blue arrows pointing from the object points towards the corresponding image points for the ZPG on top and APG on bottom. The number of arrows (i.e. markers) have been down-sampled to help visualize it better.

Figure 3 :

 3 Figure 3: Mean velocity profiles.

Figure 4 :

 4 Figure 4: Reynolds stress profiles.

Figure 5 :

 5 Figure 5: Profiles of dominant terms in turbulence production.

Figure 6 :

 6 Figure 6: Eigenvalues of individual POD modes and their cumulative sum as a fraction of the total energy of all modes.

Figure 7 :Figure 8 :

 78 Figure 7: Contour plots of the streamwise components of the first five spatial modes of ZPG-TBL on the left and APG-TBL on the right.

Figure 9 :

 9 Figure 9: PDFs of temporal coefficients corresponding to the first five most energetic POD modes for (a) ZPG-TBL and (b) APG-TBL. The vertical dotted lines correspond to the ψ 1 /σ ψ1 = ±1.0, dashed lines to ψ 1 /σ ψ1 = ±1.5 and the solid lines to ψ 1 /σ ψ1 = ±2.0.

Table 1 :

 1 Boundary layer parameters from EuHIT experiment. Source:[START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF]

	Re δ 2	β

TBL U e (m/s) δ(mm) δ 1 (mm) δ 2 (mm) H

  given below: xi j = a 11 xo j + a 12 ŷo j + a 13 + a 14 x2 o j + a 15 xo j ŷo j + a 16 ŷ2 o j a 31 xo j + a 32 ŷo j + a 33 + a 34 x2 o j + a 35 xo j ŷo j + a 36 ŷ2 = a 21 xo j + a 22 ŷo j + a 23 + a 24 ŷ2 o j + a 25 xo j ŷo j + a 26 ŷ2 o j a 31 xo j + a 32 ŷo j + a 33 + a 34 x2 o j + a 35 xo j ŷo j + a 36 ŷ2

		,
	o j
	ŷi j o j	,	(14)

Table 2 :

 2 The characteristic parameters to compute the dewarping coefficients of the equation 14.

	Property	Units	ZPG	APG
	∆x o , ∆y o M	(mm) (px/mm)	1 ± 0.001 78.839	1 ± 0.001 78.580
	P Ic	(px)	(2108.7, 1697.7) (1865.1, 1860.8)
	P Oc	(px)	(2105.8, 1700.6) (1863.7, 1858.0)
	x omax PI min PImax	(px) (px) (px)	4392.1 (-0.467, -0.378) (-0.493, -0.493) 3671.3 (0.521, 0.395) (0.493 , 0.514)
	PO min POmax	(px) (px)	(-0.467, -0.377) (-0.492, -0.492) (0.521, 0.395) (0.492, 0.514)

Table 3 :

 3 PIV analysis parameters. Source for EuHIT data:[START_REF] Cuvier | Extensive characterisation of a high reynolds number decelerating boundary layer using advanced optical metrology[END_REF]

	Measurement	ZPG	APG	ZPG(EuHIT) APG(EuHIT)
	Inflow Velocity (m/s)	9	9	9	9
	Viscous length scale l + (µm)	42	44	42	44
	FOV (l + × l + ) Grid spacing (l + × l + ) IW size (l + × l + ) Number of samples	1, 492 × 1, 090 1, 456 × 1, 115 7, 085 × 4, 265 78, 847 × 5801 5 × 1 5 × 2 26 × 39 34 × 34 19 × 2 19 × 7 57 × 57 58 × 58 10,479 10,479 10,000 30,000
	Vector field size	269 × 593	224 × 633	299 × 180	3, 250 × 238

Table 4 :

 4 Boundary layer parameters of the current HSR 2C-2D PIV measurements.

	Flow condition u τ (m/s)
	ZPG	0.35
	APG	0.33

Table 5 :

 5 The nomenclature of the conditionally averaged turbulent statistics.

Table 6 :

 6 Percentage of velocity fields with |ψ 1 | > Kσ ψ1 for K = {1.0, 1.5, 2.0}.

		ZPG	APG
	K	(%)	(%)
	1.0	32.7	33.0
	1.5	13.6	13.2
	2.0	4.2	4.0

Acknowledgements

The authors would like to acknowledge the support of the Australian Government for this research through an Australian Research Council Discovery grant. C. Atkinson was supported by the ARC Discovery Early Career Researcher Award (DECRA) fellowship. The research was also benefited from computational resources provided by the Pawsey Supercomputing Centre and through the NCMAS, supported by the Australian Government. The computational facilities supporting this project included the NCI Facility, the partner share of the NCI facility provided by Monash University through an ARC LIEF grant and the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE).

Muhammad Shehzad acknowledges the Punjab Educational Endowment Fund (PEEF), Punjab, Pakistan for funding his PhD research. Bihai Sun and Daniel Jovic gratefully acknowledge the support through an Australian Government Research Training Program (RTP) Scholarship.

See caption on the next page. 21


Wafa Aissa

Marin Ferecatu

Michel Crucianu

Multimodal Representations for Teacher-Guided Compositional Visual Reasoning

Keywords: Visual reasoning, Neural module networks, Multi-modality

Neural Module Networks (NMN) are a compelling method for visual question answering, enabling the translation of a question into a program consisting of a series of reasoning sub-tasks that are sequentially executed on the image to produce an answer. NMNs provide enhanced explainability compared to integrated models, allowing for a better understanding of the underlying reasoning process. To improve the effectiveness of NMNs we propose to exploit features obtained by a largescale cross-modal encoder. Also, the current training approach of NMNs relies on the propagation of module outputs to subsequent modules, leading to the accumulation of prediction errors and the generation of false answers. To mitigate this, we introduce an NMN learning strategy involving scheduled teacher guidance. Initially, the model is fully guided by the ground-truth intermediate outputs, but gradually transitions to an autonomous behavior as training progresses. This reduces error accumulation, thus improving training efficiency and final performance. We demonstrate that by incorporating cross-modal features and employing more effective training techniques for NMN, we achieve a favorable balance between performance and transparency in the reasoning process.

Introduction

Visual reasoning, the ability to reason about the visual world, encompasses various canonical sub-tasks such as object and attribute categorization, object and relationship detection, comparison, and spatial reasoning. Solving this complex task requires robust computational models that can effectively capture visual cues and perform intricate reasoning operations. In recent years, deep learning approaches have gained prominence in tackling visual reasoning challenges, with the emergence of foundation models playing a key role in advancing the field. Among the deep learning techniques employed for visual reasoning tasks, integrated attention networks, such as transformers [START_REF] Vaswani | Attention is all you need[END_REF], have demonstrated remarkable success in natural language processing and computer vision applications, including image classification and object detection. These models leverage attention mechanisms to capture long-range dependencies and contextual relationships for language and vision inputs. However, despite their impressive performance, reasoning solely based on integrated attention networks may be susceptible to taking "shortcuts" and relying heavily on dataset bias. There is an increasing need to address the issue of interpretability and explainability, enabling reaserchers to understand the reasoning behind the model's predictions. By leveraging transformer models as a backbone for VQA systems to encode language and visual information, we can adopt modular approaches that enable an improved understanding and transparency in the visual reasoning process. Modular approaches, such as neural module networks (NMNs), break down the problem (question) into smaller sub-tasks which can be independently solved and then combined to produce the final answer. The modular design confers the advantage of greater transparency and interpretability, as the model explicitly represents the various sub-tasks and their interrelationships.

In this paper we aim to bridge the gap between accuracy and explainability in visual question answering systems by providing insights into the reasoning process. To accomplish this, we propose an enhanced training approach for our modular network using a teacher forcing training technique [START_REF] Williams | A Learning Algorithm for Continually Running Fully Recurrent Neural Networks[END_REF], where the ground truth output of an intermediate module is used to guide the learning process of subsequent modules during training. By employing this approach, our module gains the ability to learn its reasoning sub-task both in a stand-alone and end-to-end manners leading to improved training efficiency.

To evaluate the effectiveness of our approach, we conduct experiments using training programs sourced from the GQA dataset. This dataset provides a diverse range of scenarios, enabling us to thoroughly assess the capability of our approach to reason about the visual world. Through experimentation and analysis, our results demonstrate the effectiveness of our proposed method in achieving explainability in VQA systems while maintaining a high degree of effectiveness.

In summary, this work makes two key contributions: first, the utilization of decaying teacher forcing during training, which enhances generalization capabilities, and second, the incorporation of cross-modal language and vision features to capture intricate relationships between text and images, resulting in more accurate and interpretable results.

The remaining sections of the paper are structured as follows: Sec. 2 provides a discussion on related work, Sec. 3 presents our cross-modal neural module network framework, and Sec. 4 introduces our teacher guidance procedure. In Sec. 5, we outline the validation protocol, followed by the presentation of experimental results in Sec. 6. Finally, in Sec. 7, we conclude the paper by synthesizing our findings and discuss potential future developments.

Related work

In this section, we begin by examining integrated and modular approaches employed in visual reasoning tasks. We then introduce the teacher forcing training method and its application to modular neural networks.

Transformer networks. Transformers [START_REF] Vaswani | Attention is all you need[END_REF] have been widely applied as foundation models for various language and vision tasks due to their remarkable performance. They have also been adapted for reasoning problems like Visual Question Answering (VQA). Notably, models such as ViLBERT [START_REF] Lu | Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[END_REF], Visual-BERT [START_REF] Li | Visualbert: A simple and performant baseline for vision and language[END_REF] and LXMERT [START_REF] Tan | LXMERT: Learning cross-modality encoder representations from transformers[END_REF] have demonstrated interesting performance on popular VQA datasets like VQA2.0 [START_REF] Goyal | Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering[END_REF] and GQA [START_REF] Hudson | GQA: A new dataset for real-world visual reasoning and compositional question answering[END_REF]. These frameworks follow a two-step approach: first, they extract textual and image features. Word embeddings are obtained using a pre-trained BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] model, while Faster RCNN generates image region bounding boxes along with their corresponding visual features. Subsequently, a cross-attention mechanism is employed to align the word embeddings with the image features, leveraging training on a diverse range of multi-modal tasks.

Despite the benefits of the integrated approaches, these models also have notable drawbacks. One prominent limitation is their lack of interpretability, making it challenging to understand-and debug, when necessary-the underlying reasoning process. Moreover, these models often rely on "shortcuts" in the reasoning, which means learning biases present in the training data. Consequently, their performance tends to suffer when confronted with out-of-distribution data, as shown on GQA-OOD [START_REF] Kervadec | Roses are red, violets are blue... but should VQA expect them to?[END_REF]. This research also emphasizes the importance of employing high-quality input representations for the transformer model.

To address interpretability concerns, we use features produced by an off-theshelf cross-modal transformer encoder in a step-by-step explainable reasoning architecture. This approach balances the power of the transformer model in capturing relationships between modalities with the ability to understand the reasoning process.

Neural module networks. To enhance the transparency and emulate a human-like reasoning, compositional Neural Module Networks (NMNs) such as those introduced by [START_REF] Hu | Learning to reason: End-to-end module networks for visual question answering[END_REF] and [START_REF] Li | Perceptual visual reasoning with knowledge propagation[END_REF] break down complex reasoning tasks into more manageable subtasks through a multi-hop reasoning approach. A typical NMN comprises a generator and an executor. The generator maps a given question to a sequence of reasoning instructions, known as a program. Subsequently, the executor assigns each sub-task from the program to a neural module and propagates the results to subsequent modules.

In a recent study by [START_REF] Chen | Meta module network for compositional visual reasoning[END_REF], a meta-learning approach is adopted within the NMN framework to enhance the scalability and generalization capabilities of the resulting model. The generator decodes the question to generate a program, which is utilized to instantiate a meta-module. Visual features are extracted through a transformer-based visual encoder, while a cross-attention layer combines word embeddings and image features. Although the combination of a generator and an executor in NMNs may appear more intricate compared to an integrated model, the inherent transparency of the "hardwired" reasoning process in NMNs has the potential to mitigate certain reasoning "shortcuts" resulting from data bias.

A more recent study [START_REF] Aissa | Curriculum learning for compositional visual reasoning[END_REF] has investigated the effects of curriculum learning techniques in the context of neural module networks. The research demonstrated that reorganizing the dataset to begin training with simpler programs and progressively increasing the difficulty by incorporating longer programs (based on the number of concepts involved in the program) facilitates faster convergence and promotes a more human-like reasoning process. This highlights the importance of curriculum learning in improving the training dynamics and enhancing the model's ability to reason and generalize effectively.

Interestingly, [START_REF] Kervadec | Supervising the transfer of reasoning patterns in VQA[END_REF] demonstrated that leveraging the programs generated from questions as additional supervision for the LXMERT integrated model led to a reduction in sample complexity and improved performance on the GQA-OOD (Out Of Distribution) dataset [START_REF] Kervadec | Roses are red, violets are blue... but should VQA expect them to?[END_REF].

Building upon this, our work aims to capitalize on both the transparency offered by NMN architectures and the high-quality transformer-encoded representations by implementing a composable NMN that integrates multimodal vision and language features.

Teacher forcing. Teacher forcing (TF) [START_REF] Williams | A Learning Algorithm for Continually Running Fully Recurrent Neural Networks[END_REF] is a widely used technique in sequence prediction or generation tasks, especially in RNNs with an encoderdecoder architecture. It involves training the model using the true output as novel input, which helps improve prediction accuracy. However, during inference, the model relies on its own predictions without access to ground-truth information, leading to a discrepancy known as exposure bias.

Scheduled sampling (SS) is a notable approach to mitigating the train-test discrepancy in sequence generation tasks [START_REF] Bengio | Scheduled sampling for sequence prediction with recurrent neural networks[END_REF]. It introduces randomness during training by choosing between using ground truth tokens or the model's predictions at each time step. This technique, initially developed for RNN architectures, has also been adapted for transformer networks [START_REF] Mihaylova | Scheduled sampling for transformers[END_REF], aiding to align the model's performance during training and inference.

NMNs, on the other hand, are trained using only the output of a module as input for the next module, which has drawbacks. Errors made by an intermediate module can propagate to subsequent modules, leading to cumulative bad predictions. This effect is particularly prominent during the early stages of training when the model's predictions are close to random.

NMNs can leverage the TF strategy to enhance their training process. Initially, training begins with a fully guided schema, where the true previous outputs are used as input. As training progresses, the model gradually transitions to a less guided scheme, relying more on the generated outputs from previous steps as input. This gradual reduction in guidance and increased reliance on the model's own predictions, named decaying TF, helps NMNs better learn and adapt to the complexity of the task. With decaying TF, modules can conform to their expected behavior for their respective sub-tasks.

Cross-modal neural module network

Our model takes an image, question, and program triplet as input and predicts an answer. We extract aligned language and vision features for the image and question using a cross-modal transformer. The program, represented as a sequence of modules, is used to build an NMN, which is then executed on the image to answer the question (refer to Fig. 1). In the next subsections we detail the feature extraction process and describe the program executor. Cross-modal features. Compositional visual reasoning involves the ability to make logical and geometric inferences on complex scenes by leveraging both visual and textual information. This requires accurate representations of objects and questions. To address this, we employ LXMERT [START_REF] Tan | LXMERT: Learning cross-modality encoder representations from transformers[END_REF], a pretrained transformer model specifically designed for multi-modal tasks. LXMERT has demonstrated impressive performance across various tasks and serves as our feature extractor. In our approach, we discard the answer classification component and freeze the model's weights. To extract cross-modal representations, we process the image I through the object-relationship encoder and the question Q through the language encoder. Then, the Cross-Modality Encoder aligns these representations and produces object bounding box features v j for each object b j in I, as well as word embeddings txt i for each word w i of Q.

Neural modules. Our NMN approach tackles complex reasoning tasks by decomposing them into simpler sub-tasks, inspired by human reasoning skills like object detection, attribute identification, object relation recognition, and object comparison. We developed a library of modules tailored to address specific sub-tasks. These modules are designed to be intuitive and interpretable, using simple building blocks like dot products and MLPs. They are categorized into three groups: attention, boolean, and answer modules. For instance, the Select attention module focuses on detecting object bounding boxes by applying an attention vector to the available bounding boxes within an image. On the other hand, boolean modules like And or Or make logical inferences, while answer modules such as QueryName provide probability distributions over the vocabulary of possible answers. To get a glimpse of the variety within our module library you can refer to Table 1, which showcases an example from each module category.

Modular network instantiation. A program in our framework consists of a sequence of neural modules (Table 1). These modules are instantiated within a larger Neural Module Network (NMN) following the program sequence. Each Table 1: Sample module definitions. S: softmax, σ: sigmoid, r: RELU, W i : weight matrix, a: attention vector (36×1), V: visual features (768×36), t: text features (768 × 1), : Hadamard product.

Name

Dependencies Output Definition Select -

attention x = r(W t), Y = r(WV), o = S(W(Y T x)) RelateSub [a] attention x = r(W t), Y = r(WV), z = S(W(Y T x)) o = S(W(x y z)) VerifyAttr [a] boolean x = r(W t), y = r(W(V a), o = σ(W(x y)) And [b1,b2] boolean o = b1 × b2 ChooseAttr [a] answer x = r(W t), y = r(W(V a), o = S(W(x y)) QueryName [a] answer y = r(W(V a)), o = S(W y)
module has dependencies, denoted as d m , which allow it to access information from the previous modules, and arguments, denoted as txt m , which condition its behavior. For example, the FilterAttribute module, which relies on the output of the Select module, aims to shift attention to the selected objects by considering the attribute that corresponds to the provided text argument. To handle module dependencies, the program executor employs a memory buffer to store the outputs, further used as inputs for subsequent modules. This approach also enables the computation of multi-task losses (see Sec. 4) by comparing the outputs produced by the modules with the expected ground-truth outputs.

Teacher guidance for neural module networks

To achieve explainable reasoning, we use teacher forcing (TF) to guide the modules by providing them with ground-truth inputs. We also employ a multi-task (MT) loss to provide feedback and correct their behaviors towards the expected intermediate outputs. This process is illustrated in Fig. 2. Given a program p, a question q and image I, the modular network executes p on I with the textual arguments txt m encoded in q, producing an answer a. This can be represented as a = p(I, q), where p = m 1 • m 2 • ... • m n denotes the sequential execution of n modules within the program. Each module m t inputs the output of the previous module m t-1 and performs a specific computation or reasoning step to contribute to the final answer. The NMN is trained by minimizing the cross entropy loss L CE over the set of (p, q, I, a) examples. In fact, when a module is provided with its golden input and expected output, it is independently optimized to perform its specific sub-task. However, when modules are jointly trained in a sequential manner, they learn to adapt their behaviors to work together and engage in explicit reasoning without taking shortcuts. This collaborative approach enables the modules to develop a deeper understanding of the task and enables them to perform complex reasoning operations.

From a back-propagation perspective, during the early stages of training, the gradients are computed based on the losses of individual modules when processing correct inputs. As a result, the backward gradient flow of the MT loss is interrupted at the first ground truth input. However, in the case of collaborative module interactions without TF, the full back-propagation can be computed. The intermediate outputs are preserved in continuous form throughout the program execution, enabling the flow of backward gradients between modules. Errors and updates can be propagated through the entire network, facilitating effective learning and enhancing the overall performance of the NMN. We give details about our guidance mechanism in the following subsections.

Input guidance. The modules receive input guidance through decaying teacher forcing (TF). As shown in Fig. 2, at each reasoning step t the executor randomly decides whether to use the predicted output ôt-1 or the ground-truth output o * t-1 from the previous module m t-1 as its input. This decision is made by flipping a coin, where o * t-1 is chosen with a probability of e and ôt-1 is chosen with a probability of 1e . The coin-flipping process for input selection occurs at each reasoning step, allowing the model to train on various sub-programs.

The probability e of selecting o * t-1 depends on the epoch number e. As training progresses and the epoch number increases, e decreases, giving more preference to the module's predictions over the ground-truth intermediate outputs.

Output feedback. We employ a multi-task (MT) loss approach to provide feedback to the modules based on their outputs. The loss consists of a weighted sum L = αL att +βL bool +γL answer of individual losses for the attention modules, boolean modules and answer modules, with α, β and γ scaling factors. Each module is assigned its own average loss, considering its frequency of appearance, to prevent overemphasis on frequent modules at the expense of infrequent ones.

For Boolean modules, we rely on the provided answer to infer the module's output and generate the intermediate Boolean outputs. However, for attention modules, it is necessary to establish correspondences between the bounding boxes in the image graph and those obtained from Faster-RCNN, which brings us to the issue of the ground-truth intermediate outputs, detailed in the following.

Soft matching and hard matching. Two mapping techniques, namely hard matching and soft matching, are employed to align the ground-truth bounding boxes with those obtained from the feature extractor. In the hard matching approach, a ground-truth bounding box b g is matched with the bounding box o * i from the feature extractor that has the highest Intersection over Union (IoU) factor. On the other hand, the soft mapping matches b g with all o * i that have an IoU value above a threshold, resembling a multi-label classification task. The choice of the matching technique directly affects the representation of the attention intermediate output vectors. Hard mapping produces one-hot-like vectors, while soft mapping multi-label vectors, with one(s) for positive matching and zeros for negative matching boxes. It is important to acknowledge that not all modules have ground-truth outputs that can be extracted.

Protocol design

Dataset & metrics. The GQA balanced dataset [START_REF] Hudson | GQA: A new dataset for real-world visual reasoning and compositional question answering[END_REF] consists of over 1 million compositional questions and 113,000 real-world images. The questions are represented by functional programs that capture the reasoning steps involved in answering them. To ensure consistent evaluation, the dataset authors suggest using the testdev split instead of the val split when utilizing object-based features due to potential overlap in training images. In line with the latter and following LXMERT, our model is trained on the combined train+val set. For testing, we evaluate the model's performance on the testdev-all split from the unbalanced set. This allows us to gather additional examples and gain a comprehensive understanding of the NMN's behavior. To simplify the module structure in the GQA dataset, we consolidate specific modules into more general ones based on similar operations. For example, modules like ChooseHealthier and ChooseOlder are combined into ChooseAttribute module, with an argument txt m specifying the attribute to select. This reduces the number of modules from 124 to 32. Our experiments directly utilize the pre-processed GQA dataset programs, with a specific focus on evaluating the teacher forcing training on the Program Executor module. While our system employs a transformer model as a generator to convert the question into its corresponding program, this task is relatively straightforward compared to the training of the executor. As in previous studies [START_REF] Li | Perceptual visual reasoning with knowledge propagation[END_REF][START_REF] Chen | Meta module network for compositional visual reasoning[END_REF], we achieve nearly perfect translation results on testdev-all.

We assess the performance of our approach by measuring answer accuracy. Additionally, we conduct a qualitative evaluation of the intermediate outputs, visualized through plotted images in Sec. 6.

Evaluated methods. As presented in the previous sections, we propose two contributions to improve neural module networks for VQA. First, we use teacher guidance during training, leading to better generalization. Second, we leverage cross-modal language and vision features to capture complex relationships between text and images, resulting in more accurate and interpretable results.

We use the following notations to describe the various experimental setups: -LXV: Employ the cross-modal representations from the LXMERT model [START_REF] Tan | LXMERT: Learning cross-modality encoder representations from transformers[END_REF].

-TF: Apply decaying teacher forcing to guide the inputs of the modules.

-MT: Apply multi-task losses to guide the expected outputs of the modules.

-Soft: Use the soft matching technique described in Sec. 4.

-Hard: Employ the hard matching technique described in Sec. 4.

-BertV: Use unimodal contextual language and vision representions, where contextual text embeddings are extracted by the BERT model [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] and Faster-RCNN bounding boxes features are provided by the GQA dataset [START_REF] Hudson | GQA: A new dataset for real-world visual reasoning and compositional question answering[END_REF].

-FasttextV: Employ unimodal non-contextual fastText embeddings [START_REF] Bojanowski | Enriching word vectors with subword information[END_REF] along with Faster-RCNN bounding boxes features.

Results analysis

In our evaluation, we begin by comparing the different teacher-guided training strategies. We also compare the impact of the addition of the multi-task losses and its correlation with the decaying TF along with the soft and hard matching techniques. Later, we compare the usage of multi-modal representations against uni-modal representations.

Analysis of the teacher guided training. We aim to enable modular reasoning for visual question answering on the GQA dataset. We evaluate the effectiveness of our approach by measuring the answer accuracy of several models (described in Sec. 5), and report the results in Table . 2. Overall, our findings demonstrate that using a combination of input guidance (denoted as TF) and output guidance (MT) achieves the highest accuracy, with a score of 63.2%.

When comparing LXV-TF (decaying teacher forcing) with LXV-MT (multitask loss), we observe that the multi-task loss alone achieves higher accuracy than using decaying teacher forcing alone. This can be attributed to the fact that when using TF alone, the final loss L is solely determined by the answer modules loss L answer and during early training stages, the application of TF limits the backpropagation process, preventing it from reaching the first modules of the programs. As a result, the impact of L answer on initial modules is limited.

Interestingly, the combination of multi-task loss and decaying teacher forcing exhibits complementary effects, leveraging the strengths of both techniques to enhance training dynamics and overall performance.

To assess the effectiveness of the decaying teacher forcing guidance, we compare LXV-MT against LXV-TF-MT. The TF guidance has led to accuracy improvements in both soft and hard matching settings for NMN. This technique can be viewed as a form of curriculum learning, where the model trains on programs of increasing length and complexity. During training, we observed a faster increase in accuracy for the models using TF compared to those without TF, as the answer modules receive ground truth inputs in the early stages. As training progresses, the training performance continues to improve until it reaches a peak, after which it slightly degrades due to the reduced use of TF and the modules adjusting to collaborative functioning. Nonetheless, as training continues, the testing performance surpasses that of the models without TF.

When combining the MT loss with LXV-TF, modules are optimized based on their intermediate outputs losses and they can benefit from the additional guidance provided by the back-propagation of L att and L bool . We reach the best performances outlined by LXV-TF-MT-soft and LXV-TF-MT-hard. The increase in accuracy ranges from +8.2% in the hard matching setting to +9.6% in the soft matching setting.

Unimodal vs cross-modal representations. We measure the impact of different input representations on the performance (see Table . 3). For unimodal embeddings we encode the question with fastText word embeddings or BERT language model, and the image with Faster-RCNN features. For cross-modal representations, we encode the question and the image with LXMERT. The experiments are conducted using our best training strategies from the previous section, i.e. we employ the TF guidance and the MT loss for all the experiments.

When comparing fastText and BERT, empirical observations indicate that BERT tends to achieve better performance when utilizing hard matching, which involves a focused and selective attention mechanism. Conversely, fastText demonstrates improved performance with the soft matching mechanism, enabling a multi-label approach. The choice between these matching mechanisms relies on the inputs of the models and the training strategy, as each model may demonstrate superior performance in different scenarios.

Cross-modal aligned features provided by LXMERT (denoted as LXV) have shown a significant increase in accuracy, with a +12.1% improvement when using soft matching and a +12.4% improvement when using hard matching. This validates our intuition that leveraging cross-modal features pretrained on diverse tasks and large datasets can greatly benefit NMNs. By incorporating these features, the modular reasoning process is performed with a better understanding of Qualitative analysis of the modular approach. In Fig. 3, we illustrate the reasoning process for three different questions. We highlight the bounding boxes with the highest attention values from the attention output vector. For boolean modules, we display the output probability and finally the predicted answer. Taking "Question 2" as an example, the first step successfully selects the skateboard as the object of focus. In the second step, the attention shifts to white objects. Since the skateboard is not white, the attention is then redirected to the white building. The "exist" module assesses if there is an object with a high attention value and produces a probability score based on which the answer is predicted. These examples demonstrate the explainability of our approach and the ability to trace the model's decision-making process.

Conclusion

We have presented a neural module framework trained using a teacher guidance strategy, which has demonstrated several key contributions. First, our approach enhances generalization and promotes a transparent reasoning process, as evidenced by the experimental results on the GQA dataset. Additionally, the utilization of cross-modal language and vision features allows to capture intricate relationships between text and images, leading to improved accuracy. By harnessing our proposed approach, the neural modules acquire the capability to learn their reasoning sub-tasks both independently and in an end-to-end manner.

This not only enhances training efficiency but also increases the interpretability of the system, allowing for a better understanding of the underlying reasoning processes. In addition to the aforementioned contributions, our work paves the way to a better understanding of NMNs for the task of visual reasoning. Promising directions include extending our approach to other visual reasoning datasets for a broader evaluation, and exploring alternative training strategies to enhance performance and efficiency.

Fig. 1 :

 1 Fig. 1: The proposed modular VQA framework. Plain arrows represent the output flow, while dotted arrows represent the Multi-Task loss backward flow.

Fig. 2 :

 2 Fig. 2: The teacher guidance for the program execution process related to the question 'On what is the animal to the right of the laptop sleeping?'. Plain arrows represent input guidance and dotted arrows represent the output feedback.

  -MT-soft 0.511 LXV-TF-MT-soft 0.632 word embeddings and bounding box features, leading to enhanced performance and more accurate predictions.

Fig. 3 :

 3 Fig. 3: Visualization of the reasoning process.

Table 2 :

 2 Performance of various training methods on the testdev-all set.

	Model	accuracy
	LXV-TF-hard	0.548
	LXV-MT-hard	0.598
	LXV-TF-MT-hard 0.630
	LXV-TF-soft	0.536
	LXV-MT-soft	0.563
	LXV-TF-MT-soft 0.632

Table 3 :

 3 Language and vision representations results on testdev-all.

Acknowledgments. We thank Souheil Hanoune for his insightful comments. This work was partly supported by the French Cifre fellowship 2018/1601 granted by ANRT, and by XXII Group.