L Roumiguier
email: lena.roumiguier@unilim.fr

N Pradeilles

G Antou

A Jankowiak

A Maître

Mechanical properties of submicronic and nanometric boron carbides obtained by Spark Plasma Sintering: influence of B/C ratio and oxygen content

Keywords: Boron carbide; Mechanical properties (C), Hardness (C), Carbides (D)

Boron carbide samples exhibiting nanometric and submicronic microstructure were sintered by Spark Plasma Sintering to investigate the effect of grain size on mechanical properties. The mechanical properties of sintered monoliths were characterized at the grain and macroscopic scales. Although nanostructured material exhibits finer grains than the submicronic material (i.e. mean diameter of 82 vs. 474 nm), its apparent rigidity and hardness are found to be reduced by 6.8 % and 8.4 % respectively. This contradiction with the Hall-Petch law is linked to the chemical compositions of both materials, which show significant difference in terms of B/C ratio and higher structural oxygen content especially for nanostructured material.

Introduction

Boron carbide presents high melting point (∽ 2500 °C), low density (2.52 g/cm 3 ), high hardness (30 GPa), high mechanical strength (∽ 350 MPa) and high neutron absorption cross section [START_REF] Thevenot | Boron carbide -a comprehensive review[END_REF][START_REF] Suri | Synthesis and consolidation of boron carbide: a review[END_REF][START_REF] Gosset | Matériaux absorbants neutroniques pour le pilotage des réacteurs[END_REF]. This combination of properties is of high interest for industrial applications, such as armoring systems [START_REF] Vargas-Gonzalez | Flexural Strength, Fracture Toughness, and Hardness of Silicon Carbide and Boron Carbide Armor Ceramics[END_REF], nuclear power plants [START_REF] Gosset | Matériaux absorbants neutroniques pour le pilotage des réacteurs[END_REF], hard metal tools [START_REF] Moshtaghioun | Abrasive wear rate of boron carbide ceramics: Influence of microstructural and mechanical aspects on their tribological response[END_REF], automotive [START_REF] Chapman | Wear-resistant aluminum-boron-carbide cermets for automotive brake applications[END_REF] or even for biomedical [START_REF] Gerbershagen | Measurements and simulations of boron carbide as degrader material for proton therapy[END_REF].

Currently, boron carbide is selected for its very high hardness in comparison with other ceramic materials. Fine grain size boron carbide materials are attractive from a mechanical point of view. Indeed, Hall-Petch effect is usually observed for metals and ceramics, according to which lowering the grain size leads to higher hardness and mechanical strength [START_REF] Rice | Hardness-Grain-Size Relations in Ceramics[END_REF][START_REF] Krell | Grain Size Dependence of Hardness in Dense Submicrometer Alumina[END_REF][START_REF] Krell | Grain Size-Dependent Hardness of Transparent Magnesium Aluminate Spinel[END_REF][START_REF] Wollmershauser | An extended hardness limit in bulk nanoceramics[END_REF]. Consequently, achieving dense boron carbide ceramics with fine grain size could enhance mechanical properties. However, as a result of very strong covalent bonding, the manufacturing of fully dense boron carbide ceramic with limited grain growth remains challenging and requires dedicated sintering process. Indeed, even if the combination of high temperatures and long holding times has been applied to densify boron carbide by pressureless sintering, resulting materials still exhibit residual porosity [START_REF] Lee | Pressureless Sintering of Boron Carbide[END_REF][START_REF] Dole | Microstructural Coarsening During Sintering of Boron Carbide[END_REF]. Near fully dense relative boron carbide specimens have been sintered by hot pressing. However, grain coarsening due to long holding times was observed leading to mean grain size around 16.4 µm, while starting powder showed an initial average particle diameter of 2.9 µm [START_REF] Angers | Hot-pressing of boron carbide[END_REF].

Fully dense boron carbide ceramics with relative densities higher than 98 % associated with micrometric or submicrometric grain size distributions have been manufactured without sintering aids using Spark Plasma Sintering (SPS) [START_REF] Hayun | Static and dynamic mechanical properties of boron carbide processed by spark plasma sintering[END_REF][START_REF] Hayun | Microstructural characterization of spark plasma sintered boron carbide ceramics[END_REF][START_REF] Sairam | Influence of spark plasma sintering parameters on densification and mechanical properties of boron carbide[END_REF][START_REF] Li | Densification behavior and related phenomena of spark plasma sintered boron carbide[END_REF][START_REF] Moshtaghioun | Additive-free superhard B4C with ultrafine-grained dense microstructures[END_REF][START_REF] Moshtaghioun | Enhancing the spark-plasma sinterability of B4C nanopowders via room-temperature methylation induced purification[END_REF][START_REF] Belon | Mechanical behaviour at high temperature of spark plasma sintered boron carbide ceramics[END_REF][START_REF] Réjasse | Influence of chemical composition on mechanical properties of spark plasma sintered boron carbide monoliths[END_REF].

Recently, Moshtaghioun et al. [START_REF] Moshtaghioun | Grain size dependence of hardness and fracture toughness in pure near fully-dense boron carbide ceramics[END_REF] have characterized hardness and fracture toughness of boron carbide samples sintered by SPS with different mean grain sizes in the range 0.12 to 17.2 µm. Fully dense materials were not achieved for finer grain sizes, but a correcting factor was used to take into account the residual porosity for the comparison of mechanical properties. Hardness was found to follow the Hall-Petch law in this range of grain size. Moreover, no evolution in terms of fracture toughness was observed mainly due to the transgranular fracture mechanism. This latter result was also described by Sairam et al. [START_REF] Sairam | Influence of spark plasma sintering parameters on densification and mechanical properties of boron carbide[END_REF].

The purpose of this work was to compare the mechanical properties of same relative density sintered materials exhibiting either nanometric or submicronic mean grain size. The sintered samples were carefully characterized by coupling XRD, Raman, TEM and elemental chemical analyses. These characteristics were correlated with mechanical properties, i.e. elastic properties and hardness characterized at the grain and macroscopic scales.

Experimental procedures

Starting materials were two commercially powders exhibiting submicrometric (grade HD 20, H. C. Starck GmbH, Germany) and nanometric grains (Tekna Advanced Materials Inc., Canada), and named in the following HD and TK, respectively. The grain size distribution was determined using two different techniques according to the expected particle size scale. Laser diffraction particle size analyzer (Horiba-LA-950V2, Japan) was used for HD powder, while TK powder was analyzed by dynamic light scattering (Zetasizer Nano ZS, Malvern Instruments, UK). Specific surface areas were obtained using a volumetric adsorption analyzer (ASAP 2020 V4.00, Micromeritics Instrument Corp., U.S.A.). Structural and morphological characterizations were conducted by Transmission Electron Microscopy (JEM-2100F, JEOL, Japan). Chemical composition was determined by elemental chemical analyses (EMIA-321V for carbon, and EMGA-830 for oxygen and hydrogen from HORIBA, Ltd., Japan). X-ray diffraction (XRD) analyses were conducted using a Bruker D8 Advance device (Germany; LYNXEYE XE-T detector, Cu-Kα radiation, Ni filter). The XRD pattern acquisition conditions are: angular step of 0.02°, time per step of 0.3 s, angle range 10-120°. Finally, Raman spectroscopy analyses (Renishaw Invia Reflex, UK) were conducted to obtain useful information related to chemical bonding and phase distribution in boron carbide compounds. Raman spectra were recorded from 100 to 1800 cm -1 using a wavelength of 532 nm.

Boron carbide powders were previously pressed into a 20 mm diameter graphite die under a uniaxial pressure of 30 MPa. Resulting green densities were 43 ± 2 % and 45 ± 1 % for TK and HD powders, respectively. Then, samples were sintered using a Dr.

Sinter 825 SPS device (from Fuji Electronics Industrial Co. Ltd., Japan). To elaborate samples with various relative densities, sintering temperatures were selected in the range 1300-1650°C, with a dwell time of 5 minutes and a uniaxial pressure of 75 MPa.

A heating rate of 200 °C.min -1 was applied since it was usually reported that boron carbides sintered with heating rates below 100 °C.min -1 were subjected to abnormal grain growth [START_REF] Li | Densification behavior and related phenomena of spark plasma sintered boron carbide[END_REF][START_REF] Moshtaghioun | Additive-free superhard B4C with ultrafine-grained dense microstructures[END_REF]. Density of sintered samples was determined by Archimedes' method in ethanol. Similar chemical analyses than the ones conducted on powders were carried out.

Phases in presence were also determined by XRD and the local order by Raman analyses. Sintered samples were cut axially, and the cross-section was polished up to 1 µm to characterize their microstructure. After being thermally etched, microstructure was observed by Scanning Electron Microscopy (FEG-SEM Quanta 450, FEI, Thermo Fisher Scientific, USA). Grain size distribution was determined by measuring the Feret's diameter on at least 200 grains.

Mechanical properties were characterized at the grain and macroscopic scales.

Nano-indentation measurements were performed by using a XP diamond nanoindenter (Nano Indenter XP™, MTS Nano Instruments, USA) equipped with a Berkovitch tip. The applied load ranged from 0.5 to 1000 mN. At least 30 indentations were performed for each sample and results were averaged. The intrinsic Young's modulus and hardness were estimated using Oliver and Pharr's analysis technique [START_REF] Oliver | An improved technique for determining hardness and elastic modulus using load and displacement sensing indentation experiments[END_REF]. Apparent hardness was measured using a Knoop indenter under a load of 300 g, and results were averaged from at least 10 indentations. Apparent elastic constants (i.e. Young's modulus, shear modulus and Poisson's ratio) were determined by ultrasonic pulse echography using 10 MHz transducers in transmission mode (WC37-10 and SW37-10, Ultran, State College, USA).

Results and discussion

Characterizations of raw powders

First of all, raw materials were carefully characterized. Boron carbide and boric acid phases have been identified in both powders by XRD analyses, as shown in Figure 1.

Free carbon crystallized under the graphite structure is also clearly observed in the HD powder, but not in the TK one. Crystallinity level of the TK powder is also considered too low in order to be detected by XRD technique. In fact, the slight bumps located between the boron carbide peaks around 20-25° and 36° are representative of a partially crystallized powder.

Figure 2 shows Raman spectra of starting powders. The spectra have been normalized from the boron carbide highest intensity band, which is located at 1090 cm - 1 . The HD powder spectrum can be divided in three characteristic zones [START_REF] Jay | Conception in silico d'une nouvelle phase de carbure de bore[END_REF][START_REF] Victor | Etude des modifications structurales induites dans le carbure de bore B4C par irradiation aux ions dans différents domaines d'énergie[END_REF]: (i) two main peaks at 480 cm -1 and 532 cm -1 , which corresponds to the rotation of the chains and the libration of the icosahedra; (ii) a group of peaks from 650 to 900 cm -1 associated to intra-icosahedron vibrations; (iii) high frequency peaks (975, 1002 and 1030 cm -1 ) related to chain stretching, chain-icosahedron and inter-icosahedron vibrations. Regarding TK powder spectrum, the low level of crystallinity leads to band broadening and very low intensity bands [START_REF] Huguet-Garcia | Study of the Ion-Irradiation Behavior of Advanced SiC Fibers by Raman Spectroscopy and Transmission Electron Microscopy[END_REF][START_REF] Gosset | Evidence of amorphisation of boron carbide under slow, heavy ion irradiation[END_REF]. In addition, 1355 and 1575 cm -1 bands corresponding to carbon can be observed in both Raman spectra powders [START_REF] Tuinstra | Raman Spectrum of Graphite[END_REF].

Their high intensity masks the other peaks, which can be distinguished with the zoom in Figure 2. 3 and is centered on 300 nm, but some larger elementary particles are also present up to 1100 nm. In addition, as it can be observed in Figure 4.a, some fine elementary particles of 60 nm diameter can be observed by SEM. Therefore, the HD powder is mainly composed of elementary particles with size ranged from 200 to 600 nm and exhibits a few coarser particles around 1 µm.

Chemical analyses have been carried out for both powders and are depicted in Table 1. The amounts of oxygen, carbon and hydrogen have been directly measured, while boron content was deduced by subtraction. Significant amounts of hydrogen are detected in both powders: 0.19 wt% for HD and 0.35 wt% for TK. These results confirm the presence of the crystallized boric acid detected by XRD, as indicated in Figure 1.

Moreover, a nanometric liquid is observed on particle surface of both powder particles, mentioned with black arrows in Figure 4.f. This layer could be attributed to the existence of native oxide or hydroxide phases. In the literature, it is commonly admitted that boron carbide particles exhibit oxide phases at their surfaces [START_REF] Lee | Pressureless Sintering of Boron Carbide[END_REF][START_REF] Dole | Microstructural Coarsening During Sintering of Boron Carbide[END_REF][START_REF] Rossi | Modeling of Gas-Phase Transport and Composition Evolution during the Initial-Stage Sintering of Boron Carbide with Carbon Additions[END_REF][START_REF] Moshtaghioun | Densification of B4C nanopowder with nanograin retention by spark-plasma sintering[END_REF][START_REF] Asadikiya | Thermodynamic modeling and investigation of the oxygen effect on the sintering of B4C[END_REF], but hydrogen content was not controlled and XRD analysis does not allow to distinguish boron oxide and boric acid. In this work, as hydrogen was detected in both powders, this surface layer is likely to be composed of boric acid. However, some boron oxide may be also a part of this amorphous layer. Besides, higher levels of oxygen and hydrogen were noticed in the TK powder in comparison to the HD one. This observation can be explained by its higher specific surface area (56 and 26 m 2 .g -1 for the TK and HD powders respectively), which generate higher chemical reactivity for surface oxidation and hydroxylation. Also, the carbon content ranges from 17.6 to 22.8 % depending on the powder.

Spark Plasma sintering ability of the raw materials

Finer XRD peaks and Raman bands are observed after sintering, especially for the TK powder, as displayed in Figure 1 and Figure 2 respectively. This refinement allows distinguishing a greater number of Raman bands related to boron carbide. It is assumed that further crystallization occurred in TK samples during sintering as suggested by these refinements and the disappearance of the slight bumps initially detected between the boron carbide XRD peaks for the TK powder. In addition, for both powders, there is no shift of XRD peaks after sintering, which suggests that the stoichiometry of the carbide phase did not vary significantly during SPS treatment. Despite boric acid is almost totally removed during sintering, some residual traces remain in the TK samples, as noted in Figure 1. This is attributed to partial surface oxidization due to the small grain size distribution in the case of the TK specimens.

Besides, according to Table 1, both sintered samples contain some residual oxygen content. As boric acid volatilized, the most part of this remaining oxygen is considered to be structural and corresponds to a boron oxycarbide phase BxCOy in both samples.

The calculated stoichiometries of the oxycarbide phases from elemental chemical analyses of sintered samples are B3.83CO0.04 for HD and B4.59CO0.18 for TK.

Besides, an increase of the Raman band intensities located between 750 and 900 cm -1 in Figure 2 is observed as the B/C ratio increases. This effect has already been observed in the literature [START_REF] Tallant | Boron carbide structure by Raman spectroscopy[END_REF][START_REF] Cheng | Structure and mechanical properties of boron-rich boron carbides[END_REF][START_REF] Xie | Microstructural characterization of boron-rich boron carbide[END_REF] for different boron-rich phases. However, this phenomenon has not been discussed and the amount of oxygen possibly present in their boron carbide structures not studied. Raman bands above 600 cm -1 are associated to icosahedral modes of boron carbide and a substitutional disorder within the icosahedron could involve a broadening of those bands [START_REF] Tallant | Boron carbide structure by Raman spectroscopy[END_REF]. In addition, the broad low frequency band, peaked at around 250 and 320 cm -1 , is usually observed for bulk ceramics and can be attributed to an atomic configuration disorder like twin defects [START_REF] Jay | Carbon-rich icosahedral boron carbide designed from first principles[END_REF]. In fact, twins are commonly observed for boron carbide powders, located at the grain boundaries [START_REF] Moshtaghioun | Exotic grain growth law in twinned boron carbide under electric fields[END_REF].

Figure 5 presents the evolutions of the normalized shrinkage rates during the SPS treatment of both powders. To obtain these curves, the displacements of the plunger were recorded during the sintering of powder compacts, at 1650 °C for the HD powder and at 1550 °C for the TK powder. To remove the thermal expansion contribution of graphite tools on the monitored displacement, the measured displacements have been corrected by subtracting the ones measured on dense boron carbide samples in the same SPS conditions. Then, the deduced shrinkage rates have been normalized by the heights of the powder beds. In the case of the TK powder, the presence of a higher amount of oxide liquid or gaseous phases at the surfaces of boron carbide particles coupled with finer particles seem to initiate the sintering at a lower temperature than the one encountered for the HD powder, as evidenced in Figure 5. In addition, Tekna powder exhibits slower shrinkage rates, displayed in Figure 5. The presence of surface oxides seems to prevent from sintering, in spite of the presence of finer particles.

Moreover, for a same sintering temperature, TK is denser than HD, see Table 2.

The highest density achieved for the TK sintered sample was 2.40 g.cm -3 at 1550 °C, while higher density of 2.47 g.cm -3 was obtained at 1650 °C with the HD powder, as noted in Table 2. The difficulty to obtain fully dense materials from the TK powder was related to the higher amount of boric acid and its premature volatilization during heating This phenomenon could induce the formation of closed porosity (by the development of gas occlusion), which are difficult to eliminate. The determined densities values allow estimations of relative densities to 96.2 and 99.0 % respectively.

Although the chemical compositions of the materials are different, both XRD diffractograms from sintered samples were found to correspond to JCPDS 01-075-0424 card with a radiocrystallographic density of 2.495 g.cm -3 . As a consequence, this latter was chosen as reference for boron oxycarbide instead of the commonly admitted value for boron carbide of 2.52 g.cm -3 .

A typical SEM micrograph of the 99 %TD HD sample sintered at 1650°C is displayed in Figure 6.a. Some graphite layers can be observed in a few pores exhibiting an "onion-like" shape, as represented in Figure 6.b. (white arrow) or Figure 4.e, and as already observed by Belon et al. [START_REF] Belon | Mechanical behaviour at high temperature of spark plasma sintered boron carbide ceramics[END_REF]. Table 3 references grain sizes in the range 100 to 2000 nm, with a mean diameter of 564 nm. Besides, the microstructure of TK sample sintered at 1550 °C (relative density of 96.2 %) exhibits smaller grains ranged from 10 to 400 nm, as seen in Figure 6.c, with a mean diameter of approximately 82 nm.

Consequently, both powders exhibit limited grain growth for these sintering conditions.

Mechanical properties

Mechanical properties (i.e. elasticity and hardness) of sintered samples, which are determined at the macroscopic and microscopic scales, are displayed in Table 3.

The fully dense HD material exhibits a Knoop hardness of 22.8 ± 2 GPa and an apparent Young Modulus of 448 ± 3 GPa. This last value is close to the properties measured by ultrasonic means reported for fully dense boron carbides [START_REF] Hayun | Static and dynamic mechanical properties of boron carbide processed by spark plasma sintering[END_REF][START_REF] Sairam | Influence of spark plasma sintering parameters on densification and mechanical properties of boron carbide[END_REF].

Moreover, apparent hardness was also measured using a Vickers indenter to compare sintered materials with the literature data. In fact, this technique is commonly used in research papers but in this case, due to damage under Vickers indentations, the Knoop indenter was preferred to limit experimental errors. The HD monolith exhibits a Vickers hardness of 31.8 ± 1.4 GPa, which is in good agreement with the literature [START_REF] Hayun | Static and dynamic mechanical properties of boron carbide processed by spark plasma sintering[END_REF][START_REF] Moshtaghioun | Effect of spark plasma sintering parameters on microstructure and room-temperature hardness and toughness of finegrained boron carbide (B4C)[END_REF].

The mechanical properties characterization was carried out sintered materials exhibiting exactly the same relative densities (i.e. 92 and 96 %). At the macroscopic scale, for a relative density of 96 %, the HD sample presents a Knoop hardness of 18.6 GPa, which is higher than the TK one which was found to be 16.9 GPa. In addition, the HD sample is more rigid with an apparent Young's modulus of 395 GPa while only 370 GPa was found for TK. These Young's modulus values are in the range of those measured by Sairam et al. [START_REF] Sairam | Influence of spark plasma sintering parameters on densification and mechanical properties of boron carbide[END_REF] and Hayun et al. [START_REF] Hayun | Static and dynamic mechanical properties of boron carbide processed by spark plasma sintering[END_REF] in 96 %TD boron carbides. The same tendency for both materials has been observed at a lower relative density of 92 %. The elasticity and the hardness were inferior by 11-14 % for TK samples.

Consequently, in spite of the TK significantly lower grain size, HD samples show improved mechanical properties. This observation could be extended to the different relative density levels.

At the microscopic scale, the nanoindentation experiments reveal the bulk mechanical properties of both materials, with a limited effect of the grain boundaries. Indeed, the mechanical response is characterized within the depth range 200-300 nm, which is lower than the grain size of the HD sample and in the same order for the TK sample. As usual, for both materials, the intrinsic elasticity and hardness are higher than the relative apparent properties. The TK material exhibits intrinsic hardness and Young's modulus slightly lower by about 20 % compared to the HD sample.

Discussion

Although TK sintered specimen exhibits a finer grain size, its mechanical properties are lower compared to the HD material. This result is surprising as it disagrees with the Hall-Petch law commonly observed for ceramics, according to which the lower the grain size the higher the mechanical properties (especially hardness).. This dissimilarity in intrinsic properties could hence be related to their different chemical composition. Indeed, the influence of the chemical composition of the material has to be considered as a key parameter in the mechanical response in carbide materials. This difference in chemical composition directly affects their mechanical properties. Indeed, as previously shown by Niihara et al. [START_REF] Niihara | The Effect of Stoichiometry on Mechanical Properties of Boron Carbide[END_REF], a difference in the B/C ratio can affect the hardness. The measured B/C ratios of HD and TK materials are reported on this abacus, displayed in Figure 7.a, as well as a reference material MT [START_REF] Réjasse | Influence of chemical composition on mechanical properties of spark plasma sintered boron carbide monoliths[END_REF]. The hardness of HD and MT samples seems to be in accordance with the evolution of hardness with the B/C ratio. However, the hardness value of the TK material is apart from the values measured by Niihara et al. [START_REF] Niihara | The Effect of Stoichiometry on Mechanical Properties of Boron Carbide[END_REF] for an equivalent B/C ratio. Therefore, the higher oxygen content of TK materials seems to be the main parameter explaining this difference in mechanical behaviour. During sintering, the boric acid is almost totally removed. However, the remaining oxygen in the monoliths is involved in boron oxycarbide ceramics, B3.83CO0.04 and B4.59CO0.18 for HD and TK respectively. The position of the oxygen atom within the boron oxycarbide is unknown from the literature as it could fill a vacancy or, as for Ti-C-O or Zr-C-O, substitutes the carbon [START_REF] Réjasse | Experimental investigation and thermodynamic evaluation of the C-O-Zr ternary system[END_REF][START_REF] Réjasse | Novel insight into the chemical analysis of light elements in oxycarbides[END_REF].

The lower apparent hardness and elasticity of TK material is a consequence of the lower intrinsic properties of the oxycarbide phase. This phenomenon has already been reported by Réjasse et al. [START_REF] Réjasse | Influence of chemical composition on mechanical properties of spark plasma sintered boron carbide monoliths[END_REF] where a decrease of the Young modulus was found when increasing the oxygen amount in fully dense boron carbide specimens having similar grain sizes. Moreover, as depicted in Figure 7.b, it was also observed a decrease in the intrinsic hardness, which is in agreement with the values of this work. So, the higher oxygen content of the TK sample also induces a lower rigidity. Indeed, it is shown that the stoichiometry has a deeper impact than grain size for a similar relative density. Therefore, this work highlights the necessity to control the stoichiometry in order to elaborate boron oxycarbide compounds with improved mechanical properties. To maximize sample hardness, it is necessary to optimize the B/C ratio around 4 and to lower the structural oxygen content within the oxycarbide phase, which have been proved to be detrimental for the mechanical behavior.

Conclusion

This study clearly highlights the importance of the measurement of oxygen content on the mechanical properties on carbides. Two boron carbide powders with different mean grain sizes, one submicronic (HD) and one nanometric (TK), were sintered by SPS. The chemical analyzes reveal that sintered samples are in reality boron oxycarbide ceramics, i.e. B3.83CO0.04 and B4.59CO0.18 HD and TK monoliths respectively. The mechanical properties of SPS sintered specimens were compared at similar relative densities of 92 and 96 % with a significant difference in mean grain size, i.e. 90 and 400 nm for the TK and HD materials respectively. Even if the TK monolith exhibits finer grain size, it was found that the apparent hardness and elasticity were higher for the HD materials. This shift in the Hall-Petch law and more generally in the mechanical properties is explained by the leading effect of structural oxygen content in the boron oxycarbide ceramics. Consequently, to enhance mechanical behavior of boron carbides from the submicronic to the nanometric scale, it is shown the importance to already monitor the stoichiometry at the powder synthesis step as these materials tends easily to form weaker boron oxycarbide phases.

Figure captions

Figure 3

 3 Figure 3 presents particle size distributions of studied powders. These distributions

Figure 1 :Figure 2 : 3 : 5 :

 1235 Figure1: XRD patterns of as-received powders and sintered samples. COLOUR SHOULD BE USED Figure2: Raman spectra of as-received powders and sintered samples. Peak identification reported by[START_REF] Jay | Conception in silico d'une nouvelle phase de carbure de bore[END_REF]. COLOUR SHOULD BE USED Figure 3 : Particle size distribution (in number) for HD and TK powders at 1650 °C (HD) and at 1550 °C (TK). COLOUR SHOULD BE USED Figure 4 : SEM images (a, b) and TEM observations (c, d, e, f) of HD and TK powders. COLOUR SHOULD BE USED Figure 5 : Normalized shrinkage rates measured during the SPS treatment of boron carbide powders. COLOUR SHOULD BE USED Figure 6 : SEM micrographs of a,b) HD sample sintered at 1650 °C and c) TK sample sintered at 1550 °C (after thermal etching). Note: exaggerated grain growth is observed in b) due to the post heat treatment applied to reveal grain boundaries.Figure 7 : Evolution of hardness with a) the B/C content and b) the oxygen content. COLOUR SHOULD BE USED

Figure 7 :

 7 Figure1: XRD patterns of as-received powders and sintered samples. COLOUR SHOULD BE USED Figure2: Raman spectra of as-received powders and sintered samples. Peak identification reported by[START_REF] Jay | Conception in silico d'une nouvelle phase de carbure de bore[END_REF]. COLOUR SHOULD BE USED Figure 3 : Particle size distribution (in number) for HD and TK powders at 1650 °C (HD) and at 1550 °C (TK). COLOUR SHOULD BE USED Figure 4 : SEM images (a, b) and TEM observations (c, d, e, f) of HD and TK powders. COLOUR SHOULD BE USED Figure 5 : Normalized shrinkage rates measured during the SPS treatment of boron carbide powders. COLOUR SHOULD BE USED Figure 6 : SEM micrographs of a,b) HD sample sintered at 1650 °C and c) TK sample sintered at 1550 °C (after thermal etching). Note: exaggerated grain growth is observed in b) due to the post heat treatment applied to reveal grain boundaries.Figure 7 : Evolution of hardness with a) the B/C content and b) the oxygen content. COLOUR SHOULD BE USED

Table 1 :

 1 Elemental chemical analyses carried out on powders and sintered samples.

	Tables
			wt% O	wt% H	wt% C	wt% B *
	HD	Powder	2.51 ± 0.07	0.19 ± 0.01	22.8 ± 0.4	≃ 74
		Bulk	1.31 ± 0.18	0.01 ± 0.01	22.2 ± 0.3	≃ 76
	TK	Powder	6.02 ± 0.16	0.35 ± 0.01	17.6 ± 1.0	≃ 76
		Bulk	4.50 ± 0.15	0.03 ± 0.01	18.6 ± 0.2	≃ 77
					* Obtained by subtraction

Table 2 :

 2 Open porosity and density of sintered samples. The theoretical density used to determine the relative density was the radiocrystallographic density (2.495 g.cm-3 ).

		Sintering temperature	Open porosity	Density	Relative density
		(°C)	(%)	(g.cm -3 )	(%)
		1700	< 1	2.47	99.0
		1650	< 1	2.47	99.0
		1525	< 1	2.45	98.0
	HD	1475	1.9	2.41	96.4
		1450	6.0	2.29	91.9
		1425	15.6	2.11	84.7
		1400	28.8	1.78	71.4
		1550	1.1	2.40	96.2
	TK	1400	2.1	2.29	91.9
		1350	9.6	2.10	84.2

Table 3 :

 3 Microstructure features and mechanical properties of sintered samples.

		HD	HD	HD	TK	TK
	Relative density (%)	> 99	96.4	91.9	96.2	91.9
	Grain size (nm)
	Mean diameter	564 ± 270	474 ± 220	444 ± 160	82 ± 50	74 ± 60
	Range	105 -1940	150 -1570	70 -1270	10 -400	15 -260
	At the grain scale
	Nano-hardness (GPa)		45 ± 2			35 ± 2
	Intrinsic Young Modulus (GPa)		465 ± 23		380 ± 26
	At the macroscopic scale
	Knoop hardness (GPa)	22.8 ± 2	18.6 ± 1	15.0 ± 1	16.9 ± 1	13.9 ± 1
	Apparent Young modulus (GPa)	448 ± 3	395 ± 12	370 ± 10	369 ± 10	317 ± 10
	Apparent shear modulus (GPa)	189 ± 2	167 ± 5	155 ± 4	152 ± 5	132 ± 4

Acknowledgements

The authors gratefully thank Marion Vandenhende from IRCER Institute for her help in carrying out SPS experiments and Dr Gaëlle Gutierrez from CEA Saclay for Raman spectroscopy experiments at the JANNUS-Saclay facility.


Michel Hathout
email: michel.hathout@eivp-paris.fr

Marc Vuillet
email: marc.vuillet@eivp-paris.fr

Claudio Carvajal
email: claudio.carvajal@irstea.fr

Laurent Peyras
email: laurent.peyras@irstea.fr

Youssef Diab
email: youssef.diab@u-pem.fr

Expert judgments calibration and combination for assessment of river levee failure probability

Keywords: structural reliability, subjective probability, levees, expert judgement, elicitation, calibration

In civil engineering, the utilisation of expert judgement is important for evaluating the reliability of structures whose failure behaviour is little known and difficult to quantify. This article presents the application of the approaches of elicitation, calibration and aggregation of expert opinions to evaluate the reliability of river levees. These approaches have seldom been used in the field of French hydraulic structures whereas there is growing interest for them in other countries, concerning structures reliability and other fields such as aerospace industry, nuclear industry, hydrology, statistics, economics, psychology etc. This article proposes a quantitative approach to the use of expert judgement for evaluating river levee failure probability. An application to the case of an existing river levee is presented. The approach developed assesses the expert opinions according to their calibration (statistical accuracy) and their informativeness, then aggregates them in a single calibrated opinion. Applying the proposed weighting and aggregation procedure reduces variability in the probability of failure estimate, then it provides a better probabilistic distribution of aggregated expert opinion than of individual expert opinions. The results allow identifying a trend of aggregated expert opinions that point towards over-or lack of confidence.

Introduction

River levees intended for flood protection are long linear structures of critical importance for people and property. Obtaining better knowledge of their risks and their structural reliability is a major challenge for engineers and the managers of these structures. However, the levees 2 in France and abroad are often old, little known, and poorly maintained and documented [START_REF] Mériaux | Surveillance, entretien et diagnostic des digues de protection contre les inondations[END_REF][START_REF] Ciria | Ministère de l'Ecologie du Développement durable et de l'Energie)[END_REF]. Evaluating their risks and reliability is complex due to the small quantities of data available in relation to the size and heterogeneity of these structures, and the complexity of failure mechanisms. Indeed, there appears to be a general lack of statistical data on levee characteristics (geometric, geotechnical and hydraulic data).

Furthermore some failure mechanisms such as internal erosion, overtopping and scouring are subject to many stability criteria that can be found in the literature according to the type of erosion, materials and levee profiles [2, 3, 4, 5, 6, and 7]. However it is difficult to make formulation choices given the large number of physical laws and parameters potentially combinable as well as uncertainties resulting from these modelling assumptions. There is not really consensus on the limit state conditions for internal erosion, overtopping and scouring in the national regulations, standards and professional recommendations. Only the sliding mechanism is subject to a well-established limit state condition in the regulations and recommendations [START_REF] Peyras | Study on a semiprobabilistic method for embankment hydraulic works -Application to sliding mechanism[END_REF].

In this context, evaluation using different quantitative approaches for risks and the structural reliability of river levees regarding different failure mechanisms -sliding mechanism, internal erosion, overtopping and scouring -is complex and often unfeasible [START_REF] Peyras | Probability-based assessment of dam safety using combined risk analysis and reliability methods -application to hazards studies[END_REF]. This leads to calling on expert judgement in risk analysis studies. Expert engineers must interpret the available data and voice their uncertainties, in order to then evaluate the reliability of levees in terms of failure probability. Moreover, during the assessment of the safety of the French levees, it is necessary to mobilize various areas of expertise and complementary: hydraulic, morpho dynamic, geotechnical, expertise of hydraulic structures, civil engineering, etc. Thus it is necessary to call upon several experts, in charge of deciding on a probability of failure, in a context of high uncertainties: about the physics of the phenomena considered, the representativeness and the variability of the available data.

The ILH (International Levees Handbook) [START_REF] Ciria | Ministère de l'Ecologie du Développement durable et de l'Energie)[END_REF] underlines the advantage and importance of using expert judgement for analysing the risks and reliability of levees when evaluation by quantitative approaches is difficult to perform. However, it does not recommend any specific approach allowing the efficient utilisation of expert judgement in an uncertain context. In the scientific literature, there are few works focused on the use of expert judgement for analysing the risks of levee failures. Perlea and Ketchum [START_REF] Perlea | Impact of Non-Analytical Factors in Geotechnical Risk Assessment of Levees[END_REF] proposed the use of expert judgment to build fragility curves, assessing impact of vegetation, rodents' activity, encroachments, utility penetrations, and erosion of the river bank and levee slopes. Serre et al. [START_REF] Serre | Levee Performance Assessment Methods Integrated in a GIS to Support Planning Maintenance Actions[END_REF] proposed a deterministic model of evaluating the performance of levees based on a multi-criteria decision aid method: the rule-based assignment model. Vuillet et al. [START_REF] Vuillet | Levees performance evaluation based on subjective probability[END_REF][START_REF] Vuillet | Developing a Probabilistic Multicriteria Method for River Levee Performance Evaluation in Support to Diagnostic Analysis[END_REF] proposed a levee evaluation model based on performance indicators, using the judgement of an expert or a consensus of experts and including uncertainties on the data available on the levees. This model is semi-quantitative and does not allow obtaining the failure probability of structures or combining several expert opinions. It appears important to employ a methodology capable of dealing with the latter two points, especially in France where the regulations relating to structural hazard studies systematically require engineers to declare on the failure probability of important structures.

Numerous approaches have been developed proposing probabilistic modelling of expert assessments to study the reliability of structures [START_REF] O'hagan | Uncertain Judgments: Eliciting Experts Probabilities[END_REF][START_REF] Cooke | TU Delft expert judgment data base[END_REF], in various fields such as the aerospace industry, the nuclear industry, resistance of buildings to earthquakes [START_REF] Jaiswal | Estimating structural collapse fragility of generic building typologies using expert judgment[END_REF] and hydraulic structures [START_REF] Vick | Degrees of Belief: Subjective Probability and Engineering Judgment[END_REF]. Cooke [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF] and Cooke and Goossens [START_REF] Cooke | Expert judgement elicitation for risk assessment of critical infrastructures[END_REF] distinguished three approaches for the implementation of expert judgement: i) elicitation, ii) calibration and iii) the aggregation of expert opinions.

The elicitation of expert opinion is a process of collecting expert judgements using specialised verbal and written communication methods that allow quantifying risk [START_REF] Meyer | Eliciting and Analyzing Expert Judgment: A practical Guide[END_REF]. There is general scientific consensus regarding the main steps to be followed for the elicitation of expert opinions, shared by several researchers in the domain like Morgan and Henrion [START_REF] Morgan | Uncertainty -A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis[END_REF]; Cooke [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF]; Clemen and Reilly [START_REF] Clemen | Making hard decisions with decision tools[END_REF], Ayyub [START_REF] Ayyub | Elicitation of expert opinions for uncertainty and risks[END_REF]; Cooke and Goossens [START_REF] Cooke | Expert judgement elicitation for risk assessment of critical infrastructures[END_REF], Garthwaite et al. [START_REF] Garthwaite | Statistical methods for eliciting probability distributions[END_REF] O'Hagan et al [START_REF] O'hagan | Uncertain Judgments: Eliciting Experts Probabilities[END_REF] and Burgman [START_REF] Burgman | Trusting Judgements: How to Get the Best out of Experts[END_REF]. These steps generally correspond to the definition of the problem, the identification and recruitment of experts and then the elicitation of expert judgement.

To help the elicitation of experts in a probabilistic framework, economics researchers have revealed the importance of the expert having access to data describing the context studied and on which a probabilistic estimation can be made [START_REF] Soll | Determinants of overconfidence and miscalibration: The roles of random error and ecological structure[END_REF]. These data can be theoretical or experimental. Furthermore, the authors show the importance of implementing the precise modalities of eliciting expert opinions in a probabilistic format and recommend distinguishing a range of values during elicitation and specifying that which seems most likely [START_REF] Soll | Overconfidence in intervals estimates[END_REF]. Relying on O'Hagan et al [START_REF] O'hagan | Uncertain Judgments: Eliciting Experts Probabilities[END_REF], Mason et al [START_REF] Mason | A Bayesian framework for health economic evaluation in studies with missing data[END_REF] says that with individual elicitation, each expert gives their view independently, and these views are expressed numerically and then appropriately aggregated. The alternative consisting in an elicitation group in the form of an expert panel allows experts to debate and reach consensus about the elicited values, but is more time consuming and limits the number and range of experts included.

The calibration of expert opinions permits evaluating the consistency between the information supplied by the expert and the known values (experimental / observed) [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF]. The approach to calibrating the expert opinions consists in evaluating and weighting the expert opinions in relation to calibration variables whose real values are known. The approach entails performing a quantitative evaluation of the calibration, or statistical accuracy, which refers to how often experts capture the realization into their uncertainty interval and informativeness of the expert elicitations through the use of calibration variables whose true values are known by the analyst. The mathematical calibration models such as the classical model of Cooke [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF] allow determining a performance weight for expert proportional to the product of statistical accuracy and informativeness. The classical model has been used in several applications [START_REF] Cooke | TU Delft expert judgment data base[END_REF][START_REF] Colson | Expert Elicitation: Using the Classical Model to Validate Experts' Judgments[END_REF][START_REF] Martire | What do the experts know? Calibration, precision, and the wisdom of crowds among forensic handwriting experts[END_REF], such as nuclear, banking, volcanoes, underground gas pipelines reliability [START_REF] Cooke | Failure Frequency of Underground Gas Pipelines[END_REF][START_REF] Cooke | Reliability model for underground gas pipelines" Probabilistic Safety Assessment and Management[END_REF][START_REF] Cooke | Reliability Model for Underground Gas Pipelines Case Studies in Reliability and Maintenance[END_REF][START_REF] Forys | A probabilistic model for a gas explosion due to leakages in the grey cast iron gas mains Reliability Engineering & System Safety[END_REF], to quantify internal erosion process of dams [START_REF] Brown | Use of expert opinion elicitation to quantify the internal erosion process in dams[END_REF] and levees failure frequency in Netherland [START_REF] Cooke | Expert Judgment in the Uncertainty Analysis of Dike Ring Failure Frequency Case Studies in Reliability and Maintenance[END_REF]. One of the objectives of the calibration models is to aggregate expert opinions onto a single opinion, however several authors recommend combining and aggregating expert opinions using a simple average of the expert elicitated values, which does not require calibration [START_REF]Principles of Forecasting: A Handbook for Researchers and Practitioners Kluwer Academic[END_REF][START_REF] Larrick | Intuitions about combining opinions: Misappreciation of the averaging principle[END_REF]. In a probabilistic framework, Lichtendahl et al. [START_REF] Lichtendahl | Is-it better to average probabilities or quantiles?[END_REF] proposed a method for aggregating expert opinions based on the mean of quantiles elicited by the experts, permitting aggregation in the form of an uncertainty interval and a most probable value. Colson and Cooke [START_REF] Colson | Cross validation for the classical model of structured expert judgment[END_REF] also proposed aggregating expert elicited distributions constructed using the elicited values values by performance weighting.

The objective of this article is to present an approach for the elicitation, calibration and aggregation of expert opinions for evaluating the probabilities of river levee failures due to failure mechanisms. We wish to propose a method: i) to unify the estimates of several experts and reduce the variability of the expert estimates, ii) to assess the global uncertainty of the expert estimates and the degree of confidence of each expert, iii) and to assess the uncertainties that each expert estimates and finally the degrees of confidence that can be given to the expert estimates given their calibration scores.

The article first presents the development of an approach for eliciting expert opinions in terms of failure probability represented by the slope sliding failure mechanism. It then presents the application of Cooke's [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF] calibration model to levees in order to weight the experts' probabilistic opinions. Lastly, it presents the result of applying the protocol for aggregating expert opinions developed by Lichtendahl et al. [START_REF] Lichtendahl | Is-it better to average probabilities or quantiles?[END_REF] to determine, on the basis of expert judgement, a calibrated and aggregated probabilistic distribution of failure probabilities for several profiles of an existing levee, using an aggregated interval to take into account the uncertainties expressed by the experts.

Methodological approaches for implementing expert judgement proposed for river levees

The studies of risk analyses and evaluations of structural reliability relating to levees must include the evaluation of the probabilities of scenarios leading to failure [START_REF] Ciria | Ministère de l'Ecologie du Développement durable et de l'Energie)[END_REF]. The approaches proposed focus on the elicitation of expert judgement, and the calibration and aggregation of failure probabilities elicited relating to levee failure mechanisms that can lead to a breach of a levee.

Expert opinion elicitation approach

The implementation of the expert opinion elicitation approach starts with the identification of a panel of experts whose activities are directly involved with river levees. Levee risk analysis studies involve a wide range of expert opinions linked to different disciplines: geotechnics, river hydraulics, civil engineering, etc. Several experts from several disciplines with different levels of experience (junior, confirmed, senior) are liable to intervene in these levee risk analysis studies. Thus, we gathered an expert panel by taking into account the interdisciplinary dimension of the different experts and their diversity of opinions [START_REF] Morgan | Uncertainty -A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis[END_REF].

Once the panel has been formed, each expert is questioned individually on the failure probabilities relating to several profiles of the levee to be evaluated regarding different failure mechanisms. A form including a questionnaire and for collecting expert opinions in probabilistic format has been developed for this purpose. The form comprises three main elements for each variable to be elicited: the question, the information and the demand for response. In the example shown in figure 1, regarding the sliding mechanism, these items are set out in the following way:

-The question corresponds to the variable on which the experts are questioned. It contains a detail on the mechanism to be evaluated, the cross-section to be evaluated (downstream or upstream) and the flood for which the failure probability is demanded. This question on the sliding failure mechanism takes the following form: "Given all the information available, what is the probability that the levee will fail due to sliding of the downstream slope in the case of a flood whose level reaches the crest of the levee?" -The information provides all the data required for the experts to formulate their responses and give their judgements. They correspond to the objective cue information presented by [START_REF] Soll | Determinants of overconfidence and miscalibration: The roles of random error and ecological structure[END_REF]. The items of information making up the form are: geometric data, hydraulic data, geotechnical data, the standard cross-section of the profile to be evaluated, the laws of probability of the geotechnical characteristics of the profile to be evaluated and the data resulting from a deterministic analysis represented by the safety factor SF of the levee regarding the slope sliding failure mechanism; -The demand for response corresponds to the way the expert communicates his judgement on the failure probability (P ) of the cross-section to be evaluated. In view to taking into account the uncertainties linked to levees, the approach proposes expert elicitations in the form of uncertainty interval and a most probable value [START_REF] Mériaux | Surveillance, entretien et diagnostic des digues de protection contre les inondations[END_REF][START_REF] Vuillet | Developing a Probabilistic Multicriteria Method for River Levee Performance Evaluation in Support to Diagnostic Analysis[END_REF][START_REF] Soll | Overconfidence in intervals estimates[END_REF]. Thus the expert is first called on to elicit an uncertainty interval [q5%, q50%, q95%], then to specify the median value resulting from the interval elicited proposed by [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF]. The choice of 5% and 95% quantiles is motivated by the practice of using statistical evaluations of resistance and stress parameters in civil engineering [START_REF] Peyras | Study on a semiprobabilistic method for embankment hydraulic works -Application to sliding mechanism[END_REF]. Also these are the quantiles generally used for the application of the classical calibration model [START_REF] Quigley | Elicitation in the classical model[END_REF].

Form 01

Cross-section -Theoretical levee 1

Question

Given all the information available, what is the probability that the levee will fail due to sliding of the downstream slope in the case of a flood reaching the height of the levee crest?

Information

Geometric data

Cf.

Demand for response

Probabilistic analysis Probability (P ) of sliding failure P : quantile q5% P : quantile q95% P : median value (q50%)

Figure 1. Example of a sheet for a levee profile resulting from the questionnaire form developed

The questionnaire is given individually to each of the experts. It contains an introduction briefly presenting the objectives of the study and a short manual for completing the form. Then, each expert freely completes the questionnaire according to his availability within a period of time agreed with the moderator. During this period, each expert may question the moderator if any elements of the questionnaire are unclear. In the same way, each expert may review his answers during this period. A training or dry run with feedback is desirable [START_REF] Cooke | Expert judgement elicitation for risk assessment of critical infrastructures[END_REF].

Approach for calibrating expert opinions

The approach to calibrating the panel experts' opinions consists in evaluating and weighting the panel experts' opinions in relation to calibration variables whose real values are known. The implementation of the approach for calibrating expert opinions is based on the model developed by Cooke [18]. This model is used to evaluate and weight the experts' opinions with the variables of interest and calibration. The variables of interest are those which the analyst seeks to know the values. The expert judgement is therefore used to build a subjective probabilistic distribution regarding the production of these variables. The calibration variables correspond to variables liable to inform the analyst on the capacity of the experts to provide an accurate and precise estimation. They are of the same nature as the variables of interest.

When applying them to levees, it is necessary to define pertinent calibration variables with respect to the variables of interest. Given that the variables of interest are the expert evaluations of risks and of the structural reliability of levees in a probabilistic framework, we adopted them as a calibration variable of the theoretical profiles of levee failure probability with respect to the sliding failure mechanism. This variable can be obtained quantitatively by carrying out a probabilistic reliability analysis, by considering the uncertainties on the input data (cohesion and angle of friction) and by Monte Carlo simulations [START_REF] Carvajal | Probability Modelling of Shear Strength Parameters of RCC Gravity Dams for Reliability Analysis of Structural Safety[END_REF]. The uncertainty of the input data (cohesion and angle of friction) is modelled by laws (see Figure 1). Random draws are made on these values from which stability calculations are made and an evaluation of the probability of failure by Monte Carlo simulation is obtained. This choice of calibration variables has several advantages. The format of the chosen calibration variables is the same as that of the variables of interest: probability of failure of levee profiles. As the calibration variables correspond to the results of a numerical model of theoretical levee profiles, it is possible to construct a base with a large number of calibration variables (using different levee profiles with different geometries, material properties, etc.). It should be noted that the results of a numerical model depend on the assumptions adopted as input data. This allows the elicitation phase of the calibration variables to be easily framed by presenting the input data of the numerical model in the questionnaire.

The elicitations performed by the experts can then be compared with the results of the numerical model for the different calibration variables. Implementing Cooke's [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF] model permits calculating the individual performance weight of each expert, which is also expressed as the relative calibration weight in relation to all the experts of the panel. The calibration weight is determined from the calibration score and the information score as a function of the following relations [18]:

= .

[1]

= . / ∑ . [START_REF] Ciria | Ministère de l'Ecologie du Développement durable et de l'Energie)[END_REF] where: : weight of the expert's calibration e : weight of the relative calibration of expert e in the panel of experts : calibration score of expert e : information score of expert e Z : number of experts Calibration score permits evaluating the quality (statistical accuracy) of the information given by the expert. Calibration measures the statistical likelihood that a set of experimental results corresponds, in a statistical sense, with the experts' assessments. Loosely, the calibration score is the probability that the divergence between the expert's probabilities and the observed values of the seed variables might have arisen by chance. A low score (near zero) means that it is likely, in a statistical sense, that the expert's probabilities are 'wrong' [START_REF] Cooke | Expert judgement elicitation for risk assessment of critical infrastructures[END_REF].The calibration score is evaluated by the following relation 3 below [START_REF] Cooke | Experts in uncertainty: opinion and subjective probability in science[END_REF]. It is determined by the addition of an error probability (" # ) obtained by the " # distribution for large samples size (see [START_REF] Quigley | Elicitation in the classical model[END_REF] for more details). We used Kullback-Laibler (KL) divergence measure which measures the difference between two probability distributions (expected and elicited by expert), noted Ie(ci,pi) within formula 3 and 4 below. : relative information between the theoretical probability vector -and the calibration vector & . It is calculated as follows:

. (&, -) = = & / ln(& / /-/ ) * / [4]
The information score permits measuring the quantity of information contained in the probabilistic distributions given by the experts. It relies on a measure of distances between the

= 1 -?2 * % * . (&, -)B = 1 -" # ?2 * % * . (&, -)B [3]
vector of subjective probabilities and the vector of theoretical probabilities -. The information score is evaluated by the following relation: [START_REF] Mazzoleni | An innovative probabilistic methodology for evaluating the reliability of discrete levee reaches owing to piping[END_REF] where:

= (1/n) = . ,/ (-, ) C /
: vector of subjective probability representing the subjective probabilities of each interquartile interval = ' , # , … , * +. It is used for measuring the distance between the expert's distribution and a non-informative uniform distribution; . (-, )

: the relative information between the vector of subjective probabilities and the vector of theoretical probabilities -.

The vector of subjective probability s and the relative information Ie(p,s) are calculated for each calibration variable by the following formulas:

= D E F -E G E GG -E G , E FG -E F E GG -E G , E HF -E FG E GG -E G , E GG -E HF E GG -E G I [6]
. (p, s) = = -/ ln(-/ / / ) * /

[7]

Expert opinion aggregation approach

Several approaches have been suggested and/or implemented on case studies for the aggregation of expert opinions, concerning unweighted simple average aggregations: elicited probability distributions [START_REF] O'hagan | Uncertain Judgments: Eliciting Experts Probabilities[END_REF] or elicited quantiles [START_REF] Lichtendahl | Is-it better to average probabilities or quantiles?[END_REF], or weighted averages of probabilities distributions constructed from the elicited quantiles via the respective performances of the experts on the calibration variables [START_REF] O'hagan | Uncertain Judgments: Eliciting Experts Probabilities[END_REF]. An interesting discussion of the comparison of the results of these methods can be found in [START_REF] Colson | Expert Elicitation: Using the Classical Model to Validate Experts' Judgments[END_REF]. The expert opinion aggregation approach is implemented in this study with performance weighted quantiles [START_REF] Lichtendahl | Is-it better to average probabilities or quantiles?[END_REF]. The objective is to conserve the initial form of the expert elicitations given in probabilistic format with a value considered as the median and an uncertainty interval. This approach corresponds to a weighted sum of the calibrated quantiles of expert opinions. The aggregated quantiles (E LF% , E LFG% , E LHF% ) are determined by the following relations:

E LF% = ∑ . E F% ( ) [8]
E LFG% = ∑ . E FG% ( ) [START_REF] Peyras | Probability-based assessment of dam safety using combined risk analysis and reliability methods -application to hazards studies[END_REF] E LHF% = ∑ . E HF% ( ) [START_REF] Perlea | Impact of Non-Analytical Factors in Geotechnical Risk Assessment of Levees[END_REF] where : E LF% : aggregated quantile corresponding to a probability of 5%; E LFG% : aggregated quantile corresponding to a probability of 50%; E LHF% : aggregated quantile corresponding to a probability of 95%.

Application of the approach -Results

Presentation of the case study

The levee studied is an earth-fill levee 5500 meters long intended to protect a French city against floods. The height of the levee in the study area varies from 1 m to 6 m. The approach developed was implemented by a panel of 6 expert, all civil engineers with different specialities (geotechnics, river hydraulics, civil engineering structures, risk analysis), all levee safety experts working for the French Ministry of Ecology, and all volunteers. In order to realize the experience in the best conditions, the elicitation was done anonymously.

The questionnaire used in this case study contains a brief introduction (including a notice for completing the form), a section for calibration variables and a section for variables of interest. This questionnaire was given individually to each of the experts. They provided their judgements on the calibration variables and the variables of interest corresponding to the failure probabilities of the levees with respect to the different failure mechanisms. The experts completed the form within one week. Previously, there was a dry run with a limited number of variables, which improved the clarity of the form. Other than this dry run, there was no specific training with feedback on the true values of the calibration variables. This choice was adopted specifically for this case study for further improvements concerning the experts' biases.

In this study, the calibration variables correspond to the probabilities of sliding failure for 30 theoretical levee cross-sections (cf. section 2.2). This number of calibration variables permits a robust statistical analysis [START_REF] Field | Discovering statistics using SPSS for Windows: Advanced techniques for the beginner[END_REF].

The study is applied to 40 variables of interest corresponding to 10 levee cross-sections studied with 4 failure mechanisms per cross-section. The following table summarises the type of calibration variable and the variables of interest used in this study:

Results of the elicitation phase

The following graph presents the elicitations of expert no.1 regarding the failure probabilities ( ) of 4 failure mechanisms. The black squares correspond to the quantiles (5%, 50% and 95%) elicited by the expert, while the grey circles correspond to the "true value" of the calibration variable (evaluated through a reliability analysis). 317 expressing a trend towards underestimating the failure probabilities.

Results of the calibration phase

The calibration phase is performed using 30 calibration variables (n=30). The result of the calibrating the expert opinions is presented as a calibration score, information score and relative calibration weight.

Since elicitation is performed on the q5%, q50% and q95% quartiles, the theoretical probability vector p associated with the interquartile intervals is p={0.05, 0.45, 0.45, 0.05}. The calibration vectors c can be obtained directly from table 2 (for expert 1, c={0.30, 0.37, 0.23, 0.10}). Then, the relative information between c and p vectors, Ie(c,p) is calculated according Eq.4 (for expert 1, Ie(c,p)=0.38), which finally gives the calibration score Ce presented in Table 3 by using Eq.5.

For the calculation of the information score, the elicited failure probability values can be very low, expressed in the format (10 W ). We change the variable in order to conserve a constant difference in absolute value between (10 W ) and ?10 (W ) B, which amounts to applying the common logarithm to the elicited probabilities (log G ( )). This change of variable permits a more pertinent interpretation of the information score. The subjective probability vector s is then evaluated for each expert and for each of the calibration variables. For example, expert 1 elicits the probability values of 10 -4 , 10 -2 and 10 -1 for the quartiles 5%, 50% and 95% respectively for the first calibration variable. With the variable change to base log10, we thus obtain q5%=-4, q50%=-2 and q95%=-1, which makes it possible to evaluate the vector s1={0.24, 0.39, 0.19, 0.18} with Eq.6 and the relative information Ie,1(p,s)=0. [START_REF] Cooke | Failure Frequency of Underground Gas Pipelines[END_REF] with Eq.7. The information score Ke is finally evaluated with Eq.5, and corresponds to the average of the relative information Ie,i(p,s) evaluated for the 30 calibration variables.

Once the calibration score Ce and information score Ke have been evaluated, the individual weight we and the relative weight of calibration w'e are directly evaluated by Eq.1 and Eq.2 respectively. The results of the calibration score Ce, information score Ke, individual weight we, and relative weight of calibration w'e are presented in the following table for each expert. The calculations to obtain these results were carried out in an Excel file. The results obtained from the calibration of expert opinions showed that the values of the calibration scores ( ) varied from 3% to 91%. The highest value was obtained by expert no. 2, meaning that their elicitations contained more true calibration values in comparison to the elicitations of the other panel experts. On the contrary, the lowest calibration score was obtained by expert no. 3, meaning that their elicitations contained the fewest true calibration values.

Regarding the information score, the values varied from 0.44 to 1.04. The highest value was obtained by expert no.4, meaning that the uncertainty intervals elicited by this expert were narrower than those given by the other panel experts. On the contrary, the lowest value of this score was obtained by expert no.5, meaning that the uncertainty intervals elicited by this expert were wider than those given by the other panel experts.

We observed that the calibration score in our study had a more significant influence than the information score on the final result of the expert opinion calibration phase. Indeed, the range of the final calibration score was wider than that of the information score. Thus, expert no.2 with the highest calibration score obtained the best relative calibration weight ( # = 40%) whereas expert no.3 with the lowest calibration score obtained the lowest relative calibration weight ( = 1%). The question of not taking into account his opinion may then arise. In the context of this study, we preferred to keep all the opinions, taking into account the fact that that of some experts does not significantly influence the final results.

Results of the aggregation phase

The result of the expert opinion aggregation phase is shown by one probabilistic distribution relative to each levee cross-section evaluated by the experts. We operated results exposed in table 3 to combine expert assessments, using quantiles combination [START_REF] Lichtendahl | Is-it better to average probabilities or quantiles?[END_REF] to produce results exposed in figure 5 and table 4. The following graph shows the result of the aggregation of the calibrated expert opinions for the 30 calibration variables related to the theoretical levees (continuous lines) where the true values are represented by a circle. The calculations to obtain these results were carried out in an Excel file.

Application to variables of interest

On the basis of the expert calibrations, the approach was applied to the variables of interest. Figure 6 presents the results for 40 variables of interest corresponding to 10 levee crosssections studied for 4 failure mechanisms:

-variables no.31 to no.40 corresponding to the sliding failure mechanism, -variables no.41 to no.50 corresponding to the internal erosion failure mechanism, -variables no.51 to no.60 corresponding to the scouring failure mechanism, -variables no.61 to no.70 corresponding to the overtopping failure mechanism.

Table 5 shows the distribution of true values (in number and in percentage) in the raw interquartile intervals (before calibration and aggregation) elicited by the experts for the sliding failure mechanism for levee cross-sections studied no.31 to no. 40: Regarding the different variables of interest considered in this study, it is possible to perform a quantitative evaluation by analysing the failure probabilities for variables no.31 to no.40 of the sliding failure mechanism, which correspond to the sliding failure probabilities of six cross-sections of the levee studied. These variables allow comparing the results of the calibration and aggregation approach applied to the expert opinions with the results of a quantitative analysis. Table 6 below presents the distribution of true values in the interquartile intervals for the variables of interest no.31 to no.40 at the end of the expert opinion aggregation phase and using the calibration weights determined during the expert opinion calibration phase for the cross-sections of the theoretical levees: The final distributions of the aggregated expert opinions of variables no.31 to no.40 contain the true value in the uncertainty interval [5%, 95%] seven times, giving a percentage of 70% of the true values of the uncertainty values given by the experts. The evaluation of variables 31 and 40 is similar to a cross-validation of the results obtained for the calibration variables (1 to 30). We find that we get similar results. We propose to discuss this in paragraph 3 of section 4.2.

Regarding the other failure mechanisms (internal erosion, overtopping and scouring), the structural limit states were not quantifiable since it was not possible to quantitatively evaluate the failure probability of these failure mechanisms. The true values were therefore unknown for these mechanisms and the approach proposed therefore made it possible to implement the expert judgement to evaluate these failure probabilities in the framework of a levee risk analysis.

Discussion

The approach developed comprises a protocol for using expert judgement in the area of river levees to evaluate their structural reliability regarding the failure probabilities of several failure mechanisms. Our discussion focuses on the approach developed and on the results obtained.

Discussion on the approach developed

The approach proposed answers the issue of evaluating the reliability of levees as raised by French regulations relating to hazard studies and determining the probability of failure on the basis of expert evaluations. The approach permits eliciting, calibrating and aggregating the evaluations of experts in different disciplines. The treatments of the expert elicitations allowed us to obtain an evaluation of failure probability and an evaluation of the uncertainty on this probability by way of an uncertainty interval. The application of this methodological approach led to two significant improvements: i) it combined additional expert opinions from the calibration and information scores to give more weight to the most relevant experts; ii) it used an uncertainty interval to provide quantitative information on the uncertainty on the final failure probability. In addition, the analysis of the uncertainty interval can be completed with the information and calibration scores obtained for the different experts, or, more intuitively, by visualising the quantities of calibration variables falling outside the uncertainty intervals given by the experts.

The major difficulty for implementing the model concerns the construction of the calibration variables. In our work, we chose the sliding failure scenario as the calibration variable of a levee failure. The calibration variables were obtained by analysing probabilistic reliability by taking into account the uncertainties on the input data. However, as we mentioned in introduction within our research study, there was no calibration variable directly involving the failure probability of a levee for internal erosion, overtopping and scouring failure scenarios. Also, the use of calibration variables focused on the sliding mechanism to assign calibration weights to the other mechanisms (internal erosion, overtopping and scouring) could be discussed on the pertinence of the process.

However, we consider that our approach is relevant because the parameters involved in the mechanics of internal erosion, overtopping and scouring are the geotechnical parameters of the soils and of the levee also involved in the sliding mechanism: the density of the materials, cohesion, angle of friction, geometry of the levee, etc. Thus, calibrating the experts on the sliding mechanism amounts to calibrate the experts on the geotechnical parameters involved in the sliding mechanism, and consequently on the other mechanisms of failure. Therefore this approach of calibration of the variables of interest seems well founded and justified.

One perspective for improving the calibration variables of the latter scenarios would be to break these failure scenarios down into elementary functional events [START_REF] Vick | Degrees of Belief: Subjective Probability and Engineering Judgment[END_REF][START_REF] Vuillet | Decision making method for assessing performance of large levee alignment[END_REF], for which it would be easier to obtain calibration variables. It would then be possible to question the experts on the calibration variables linked to elementary events and obtain the failure probability for these scenarios by multiplying the probabilities elicited for the elementary variables.

Discussion on the results

It is considered that there is an overconfidence bias when the intervals estimated by the experts are too narrow [START_REF] Lin | A study of expert overconfidence[END_REF][START_REF] Burgman | Trusting Judgements: How to Get the Best out of Experts[END_REF]. We propose in this study to consider two sub-components of the over-confidence bias, excess of optimism and excess of caution. Situations of bias on excess of optimism and excess of caution can be identified when the true values of the calibration variables fall within the intervals [95%; 100%] and [0%; 5%] in more than 10% of the expert's or the experts' responses. If the value of the calibration variable is in the [0%; 5%] interval, this means that the expert has given a higher failure probability than that of the true value of the calibration variable, in which case the evaluation can be deemed as overcautious, possibly leading to costly structural reinforcement works. On the contrary, if the true value of the calibration variable falls within the [95%; 100%] interval, this means that the expert was overconfident, since they gave a lower failure probability and uncertainty interval than the real failure probability of the structure. This situation is dangerous because it may lead to a poor evaluation of the structure's failure probability liable to place populations in danger.

The results obtained concerning the elicitations of experts before treatment confirm previous observations of the probabilistic evaluation based on expert opinion and allow distinguishing biases of excess of caution and excess of optimism. We observed that the experts tended to be overcautious regarding the evaluation of failure probabilities and to be overconfident regarding the evaluation of uncertainties. Indeed, the results presented in table 2 show that on average 26% of the expert evaluations were cautious, instead of the 5% that should have been obtained if the distribution of the experts were perfectly calibrated. As mentioned above, this bias of caution points towards safety in comparison to a bias of excess of optimism. The bias of excess of optimism obtained reached 19%, which is high for structures critical for the safety of people and property. It is noteworthy that the literature concerning heuristic biases agrees in saying that they can be reduced by implementing several procedures: in the calibration phase as presented in this article, or in the elicitation phase by making the experts aware of biases and their explicit description [START_REF] Vuillet | Levees performance evaluation based on subjective probability[END_REF], or in the framework of calibration exercises prior to elicitation. This provides an interesting perspective of improvement for this study.

The results obtained by applying our calibration approach to calibration variables (nos. 6). To improve this result, it would be interesting to implement specific mathematical procedures to correct these types of bias using inflation factors [START_REF] Clemen | Debiasing expert overconfidence: a Bayesian calibration model[END_REF].

Conclusion

Studying the structural reliability of levees with quantitative approaches is complex and often impossible due to the lack of statistical data or the lack of limit state equations for certain mechanisms. Thus this article presented an approach for evaluating the failure probabilities of levees based on the calibration of expert judgement. The aim of the procedure was to determine a calibration weight that can be used for other levee risk analyses.

The approach developed is pertinent for using expert judgement in the area of river levees. It can be used to evaluate the reliability of structures in terms of failure probability based on this judgement. The approach first proposes eliciting expert opinions in probabilistic format then identifying a calibration weight for the best expert elicitation in view to aggregating them. At the end of an analysis of the accuracy and informativeness of expert opinions, it assigns a high calibration weight to the most accurate expert elicitations and a low calibration weight to inaccurate and thus less informative elicitations.

The results obtained from the application of the approach developed in the area of levees showed the advantage of calibrating expert judgement. Finally, the approach can be used to correct through aggregation expert evaluations of levee reliability and obtain a more informative aggregated evaluation.

The calibration variables used in the case study correspond to probabilities of sliding failure evaluated by a reliability analysis. The accuracy of the expert judgments calibration can be assessed for the sliding mechanism by comparison with the results of a reliability analysis. However, the accuracy of the calibration cannot be evaluated for the other failure mechanisms that have not been assessed by a reliability analysis in this case study. Some research works are available in the literature concerning the reliability analysis of levees with respect to other failure mechanisms such as internal erosion or pipping [START_REF] Mazzoleni | An innovative probabilistic methodology for evaluating the reliability of discrete levee reaches owing to piping[END_REF]. As these works have not been considered herein, one perspective should be to decline the methodology of this paper by also using calibration variables corresponding to other levee failure mechanisms (such as the probability of failure due to internal erosion assessed by a reliability analysis). This would allow the results to be compared in order to further explore the validation/verification of calibration of experts across different failure modes.

Besides, the approach has limitations regarding the quantitative correction of the trends of overconfidence followed by expert opinions. Another perspective for this work would be to improve correction with correction protocols including the treatment of expert biases, such as overconfidence which is one of the most common (and potentially severe) problem in expert judgment [START_REF] Lin | A study of expert overconfidence[END_REF]. In this study, each expert is questioned individually, without allowing discussion or consensus among experts (as in the case of collective elicitation by a panel of experts). Future work could study and compare these approaches (individual and collective elicitation) for application to flood protection levees.

This approach developed for river levees could be adapted and applied in other areas where recourse to expert judgement is the only means of obtaining information usable for carrying out a reliability analysis. The approach will require adjustments specific to each category of civil engineering structure before its implementation.

For m 17

Cross-section P 106

Given all the information available, what is the probability that the levee will fail due to internal erosion in the case of a flood 500 (annual Pflood = 1/500=2 * 10 )? -Terriers: no significant burrows detected.

-Crossing structures: no crossing works identified Probabilis tic analysis Probability (P ) of internal erosion failure P : quantile q5% P : quantile q95% P : median value (q50%) Cross-section -P 140 Given all the information available, what is the probability that the levee will fail due to overtopping in the case of a flood 500 (annual Pflood = 1/500=2 * 10 )? Cross-section P 106 Given all the information available, what is the probability that the levee will fail due to Scouring in the case of a flood 500 (annual Pflood = 1/500=2 * 10 )?

  Figure below -Height = 5 m -Altitude of crest = 205 m NGF -Width of crest = 6 m -Upstream batter = 3/2 (H/V) -Downstream batter = 3/2(H/V) Hydraulic data -Flood level reaching the levee crest -Hydraulic gradient level calculated in steady state foundation) Probabilistic law: normal law truncated at 0 Mean: 10 kPa Standard deviation: 5 kPa, Min = 0

  where : P( ) : probability of a random variable according to a " # law " # ( ) : distribution function of a random variable according to the " # law % : number of calibration variables & : calibration vector representing the portion of true values in each interquartile interval & = '& , & # , … , & * + , : number of interquartile intervals, in this study j=4 -: theoretical probability vector representing the theoretical probabilities of each interquartile interval -= '-, -# , … , -* + . (& / , -/ )

Figure 2 .Figure 3 .

 23 Figure 2. Aerial view of the levee studied (scale: 1/25 000 in A3 format)

Figure 5 . Calibration variables calibrated and aggregated and their uncertainty intervals 398 399 Figure 5

 53995 Figure 5. Calibration variables calibrated and aggregated and their uncertainty intervals

-

  Foundation permeability : = 10 cd 10 F / -Geometric Data : Height 3,24 m -Width of crest 4,61 m -Upstream batter 3H/2V -Downstream batter 1H/1V -flood level is 1.78 m under the crest for Q500 -Water leak : undetected -Sealing of the upstream protection: rip rap of the 50s #,ab Levee body resistance to internal erosion -Materials constituting the levee body: sandy-gravelly, seem locally not very compact. -Materials constituting the foundation of the levee: sand-gravel, seem to present lenses of more or less compact materials. -Cohesion c = 0 kPa -Angle of friction ∅ = 35 ° -Discontinuities of the embankment: not identified by the observation of an in situ geotechnician -The critical hydraulic gradient of materials is icr = 0,1 -The overall hydraulic gradient of this profile (P 106) is of iglob = 0.09 (steady state) The duration of floods 2 days ,ab Irregularit ies in levee body -Trees and roots: significant presence of trees along the entire length of the dyke on the embankment side of the river.

Figure 8 :

 8 Figure 8: example questionnaire sheet for internal erosion

--P

  criteria presenting technical data on the entire section C ,gLevee crest elevation-The embankment height is 4.83 m -The water is above the crest of 5 cm for a Q500 flood C The duration of flood 2 days with overtopping for 3 hours -Height of downstream slope 3.04 m -Materials constituting the body of the dam: sand-gravel, seem locally not very compact.-Materials constituting the foundation of the levee: sand-gravelly, seem to present lenses of more or less compact materials. Permeability. -Cohesion of the body and the foundation & = 0 i -Angle of friction of the levee body ∅=35° -Angle of friction of the levee foundation ∅=32° -Etanchéité de la protection aval : terre végétale -Geometric irregularities: unidentified -Protection of downstream protection: topsoil : median value (q50%)

Figure 9

 9 Figure 9: example questionnaire sheet for overtopping

--P

  Sealing of the upstream protection: pearl of the 50s -Height 3,75 m -Flood level Q500 1.25 m under the crest -The duration of floods 2 days --The average flow velocity of this profile (P 106) for a Q500 flood is 3.42 m / s -En générale, les vitesses d'écoulement dans le lit du Drac sont fortes, supérieures à 3,5 m/s pour le tronçon 1 -Body permeability : = 10 / -Materials constituting the levee body: sandy-gravelly, seem locally not very compact. -Cohesion c = 0 kPa -Angle of friction ∅ = 35 ° -Sealing of the upstream protection: pearl of the 50s -Permeability of the foundation : = 10 à 10 F / -Materials constituting the foundation of the levee: sandy-gravelly, seem to present lenses of more or less compact materials. -Cohesion c = 0 kPa -Angle of friction ∅ = 35 ° : median value (q50%)

Figure 10 :

 10 Figure 10: example questionnaire sheet for scouring

Table 1 . Table summarising the calibration variables and the variables of interest in the application of the approach.

 1

		Calibration variables	Variables of interest
	Failure mechanism	(theoretical levee profiles) Number of variables no. of variable	(profiles of the levee studied) Number of variables no. of variable
	Sliding	30	no.1 -no.30	10	no.31 -no.40
	Internal erosion	-	-	10	no.41 -no.50
	Scouring	-	-	10	no.51 -no.60

Raw elicitations of expert no.1 relating to calibration variables no.1 to no.30 and to variables of interest no.31 to no.40 -sliding failure mechanism, no.41 to no.50 -internal erosion failure mechanism, no.51 to no.60 - scouring failure mechanism), no.61to no.70 -Overtopping failure mechanism.

	Figure 4.
						Pfc
	1.0E-09	1.0E-08	1.0E-07	1.0E-06	1.0E-05	1.0E-04	1.0E-03	1.0E-02	1.0E-01	1.0E+00
	Number of calibration variables and variables of interest

Table 2

 2 gives the distribution of the true values (in number and in percentage) in the interquartile intervals elicited by the experts for theoretical levee cross-sections no.1 to no.30:

Table 2 . Distribution of true values of the calibration variables (no.1-no.30) in the interquartile intervals elicited by the experts and the mean of elicitations.

 2

			Calibration variable (no.1 to no.30)
	Interquartile interval	NO%, P%Q	NP%, PO%Q	NPO%, RP%Q	NRP%, SOO%Q
	(T UV ) (no.1 to no.30)	n %	9 30%	Expert no.1 11 37%	7 23%	3 10%
	(T UV ) (no.1 to no.30)	n %	9 30%	Expert no.2 10 33%	10 33%	1 3%
	(T UV ) (no.1 to no.30)	n %	10 33%	Expert no.3 5 17%	7 23%	8 27%
	(T UV ) (no.1 to no.30)	n %	10 33%	Expert no.4 8 27%	8 27%	4 13%
	(T UV ) (no.1 to no.30)	n %	0 0%	Expert no.5 4 13%	15 50%	11 37%
	(T UV ) (no.1 to no.30)	n %	8 27%	Expert no.6 8 27%	6 20%	8 27%
	(T UV ) (no.1 to no.30)	n %	Means of elicitations (before calibration) 7.67 7.67 8.83 26% 26% 29%	5.83 19%
	The results show that the percentage of true values in the interval [5%, 95%] varies from 40%
	(17%+23%) for expert no.3to 66% (33%+33%) for expert no.2. More exactly for expert 3,
	33% in the interval [0%, 5%] instead of 5%, 17% in the interval [5%; 50%] instead of 45%,
	23% in the interval [50%, 95%] instead of 45, and 27% in the interval [95%, 100%] instead of
	5%, and for expert 2 : 30% in the interval [0%, 5%] instead of 5%, 33% in the interval [5%;

50%] instead of 45%, 33% in the interval [50%, 95%] instead of 45, and 3% in the interval [95%, 100%] instead of 5%. For experts nos. 1, 4 and 6, the true values are concentrated in the interval [0%, 50%], expressing a trend towards overestimating the failure probabilities. On the contrary, for expert no.5, the true values are concentrated in the interval [50%, 100%],

Table 3 . Results of the expert opinion calibration phase in relation to the calibration variables.

 3

	No. expert 1	Calibration score 0.79	Information score 0.74	Weight of individual 0.59	Relative weight ′ 0.28
	2	0.91	0.91	0.83	0.40
	3	0.03	0.79	0.02	0.01
	4	0.46	1.04	0.48	0.23
	5	0.14	0.44	0.06	0.03
	6	0.19	0.52	0.10	0.05
				Total	1

Table 4

 4

	below which presents the true calibration variables

Table 4 . Distribution of the true calibration variable values (no.1-no.30) in the interquartile intervals. Calibrated and aggregated elicitations of calibration variables

 4 (no.1 to no.30) of the aggregation phase we observed that the final distributions of the calibrated and aggregated expert opinions contained true values in the uncertainty interval for 18 calibration variables out of a total of 30 variables, i.e. 60% of the true values belonging to the uncertainty interval [5%, 95%]. Thus, the aggregation of the calibrated expert opinions leads to an uncertainty interval [5%, 95%] containing more true values in comparison to the raw expert uncertainty intervals.

	Interquartile interval n (T UV ) (no.1 to no.30) %	NO%, P%Q 9 30%	NP%, PO%Q 15 50%	NPO%, RP%Q 3 10%	NRP%, SOO%Q 3 10%
	At the end

Table 5 . Distribution of true values of variables of interest (no.31-no.40) in the interquartile intervals elicited by the experts for the sliding failure mechanism.

 5

		Variable of interest (no.31 to no.40) for the sliding failure mechanism
	Interquartile interval	NO%, P%Q	NP%, PO%Q	NPO%, RP%Q	NRP%, SOO%Q
	(T UV ) (no.31 to no.40)	n %	3 30%	Expert no.1 3 30%	4 40%	0 0%
	(T UV ) (no.31 to no.40)	n %	4 40%	Expert no.2 3 30%	3 30%	0 0%

Calibrated and aggregated variables of interest and their uncertainty intervals: no.31 to no.40 -sliding 431 failure mechanism, no.41 to no.50 -internal erosion failure mechanism, no.51 to no.60 -scouring failure mechanism, 432 no.61 to no.70 -overtopping failure mechanism.

						Pfc
	1.0E-09	1.0E-08	1.0E-07	1.0E-06	1.0E-05	1.0E-04	1.0E-03	1.0E-02	1.0E-01	1.0E+00
	Number of calibration variables and variables of interest
	433

Table 6 . Distribution of true values of the variables of interest (no.31-no.40) in the interquartile intervals. Calibrated and aggregated elicitation pf variables of interest

 6 (variables no.25-no.30)

	Interquartile interval n (T UV ) (no.31-no.40) %	NO%, P%Q 2 20%	NP%, PO%Q 5 50%	NPO%, RP%Q 2 20%	NRP%, SOO%Q 1 10%

  1 to 30) showed a marked reduction of variables in the excess of optimism interval [95%; 100 %], thus falling from 19% (mean of individual expert elicitations) to 10% (table4), which is a considerable improvement that goes towards ameliorating the safety of the structures concerned. There is still room for progress to reach the target value of 5% in the excess of optimism interval [95%; 100%]. The results obtained for the variables of interest (no.31 and 40) are analogous with a substantial reduction of variables in the excess of optimism interval [95%; 100 %] reduced from 15% to 10% (cf. table5 and table

Expert no.3 (T UV )

(no. [START_REF] Cooke | Failure Frequency of Underground Gas Pipelines[END_REF] The following four figures present an example questionnaire sheet for each failure mode considered as a variable of interest.

Form 14

Cross-section -P 108 Given all the information available, what is the probability that the levee will fail due to sliding of the downstream slope in the case of a flood 500 (annual Pflood = 1/500=2 * 10 )?

Demand for response

Probabilis tic analysis Probability (P ) of sliding failure P : quantile q5% P : quantile q95% P : median value (q50%)