Anthony Fardet
email: anthony.fardet@inrae.fr

PhD Marion Desquilbet

PhD ; Edmond Rock

The compliance of French purchasing behaviours with a healthy and sustainable diet: a one-year follow-up of regular customers in hypermarkets

Keywords: food choices, hypermarkets, sustainability, ultra-processed foods, animal products, food diversity

The compliance of French purchasing behaviors with a healthy and sustainable diet: a 1-yr follow-up of regular customers in hypermarkets

Anthony Fardet, Marion Desquilbet, Edmond Rock

Introduction

At this time, behavioural change towards the purchase and consumption of healthier and more sustainable food products is a crucial issue to address (Benedetti et al., 2018, Schösler andde Boer 2018). Indeed, most food systems are no longer sustainable at a global level (FAO and WHO 2019). Notably, excessive consumption of animal products and ultra-processed food (UPF), associated with the lack of variety in diets, may threaten human health, animal and plant biodiversity, and the environment as a whole (Fardet and Rock 2018). The qualitative 3V rule provides a simple metric for assessing the healthiness and sustainability of consumer food choices, with the three V's standing in French for "Végétal" (plant-based), "Vrai" ("real" foods, i.e., not ultra-processed) and "Varié" (varied) (Fardet and Rock 2018;Fardet and Rock 2020).

It is based on three indicators of the relation between diet and global health: the plant/animal caloric ratio, with an optimum of ≈15% animal calories/day; the degree of processing, with a maximum of 15% ultra-processed calories/day; the diversity of the diets -if possible organic, local and seasonal.

Through its holistic approach, this three-dimension metric has been built from eight previously identified health protective diets worldwide (including the Mediterranean, Prudent and Okinawa diets), and eight foresight scenarios at horizon 2050 for protecting both human health and food system sustainability (including the EAT-Lancet, INRAE/Agrimonde-Terra, IDDRI/TYFA and Solagro/Afterres scenarios). In the 3V rule, while the impact of animal product consumption and food diversity on health and food systems is globally well known (Fanzo et al., 2013;Johnston et al., 2014, Tilman andClark 2014;[START_REF] Mariotti | Animal and Plant Protein Sources and Cardiometabolic Health[END_REF][START_REF] Bonnet | Viewpoint: Regulating meat consumption to improve health, the environment and animal welfare[END_REF][START_REF] Benton | Food System Impacts on Biodiversity Loss: Three Levers for Food System Transformation in Support of Nature Chatam House[END_REF], the effect of the degree of food processing is rather new and has been under-appreciated in studies about dietary patterns and health (FAO et al., 2019). The UPF concept has first emerged in 2009 [START_REF] Monteiro | Nutrition and health. The issue is not food, nor nutrients, so much as processing[END_REF], and has led to an increasing number of studies in different countries (Pan American Health Organization 2015, FAO et al., 2019;PAHO and WHO 2019), notably in relation to health outcomes [START_REF] Costa | Consumption of ultraprocessed foods and body fat during childhood and adolescence: a systematic review[END_REF][START_REF] Askari | Ultra-processed food and the risk of overweight and obesity: a systematic review and meta-analysis of observational studies[END_REF][START_REF] Chen | Consumption of ultra-processed foods and health outcomes: a systematic review of epidemiological studies[END_REF]Elizabeth et al., 2020;[START_REF] Lane | Ultraprocessed food and chronic noncommunicable diseases: A systematic review and meta-analysis of 43 observational studies[END_REF][START_REF] Pagliai | Consumption of ultra-processed foods and health status: a systematic review and meta-analysis[END_REF] and food system sustainability (Ministry of Health of Brazil 2014; [START_REF] Baker | Ultra-processed foods and the nutrition transition: Global, regional and national trends, food systems transformations and political economy drivers[END_REF]Fardet and Rock 2020;Seferidi et al., 2020). Thus, the 3V rule has the distinction of addressing more clearly the degree of food processing (FAO et al., 2019;Fardet and Rock 2020).

The multicriteria 3V rule adequately addresses nutritional needs while being adaptable to the specificities of each region of the world, such as socio-economic, pedo-climatic and agronomic conditions, culture and culinary traditions (Fardet and Rock 2020). Ultimately, the value of the 3V rule lies primarily in embracing global health with only three generic and simple indicators; and if one dimension were to be missing, the diet would no longer be protective of global health, e.g., with the increasing development of ultra-processed plant-based foods in response to the revegetation of the plate, French vegans and vegetarians consumed more ultraprocessed foods than meat eaters [START_REF] Gehring | Consumption of Ultra-Processed Foods by Pesco-Vegetarians, Vegetarians, and Vegans: Associations with Duration and Age at Diet Initiation[END_REF].

In France, the healthiness of the diet is assessed every seven years by the French Agency for Food, Environmental and Occupational Health & Safety (ANSES), based on representative surveys of French eating habits in different age groups (Dubuisson et al., 2009(Dubuisson et al., , 2019;;[START_REF] Lioret | Trends in food intake in French children from 1999 to 2007: results from the INCA (étude Individuelle Nationale des Consommations Alimentaires) dietary surveys[END_REF]. In particular, between the last two surveys conducted respectively in 2006-2007 and 2014-2015, the content of industrially processed foods on the French plate has increased significantly (ANSES 2017). Current consumption of UPFs is around 31-36% of daily calories (ANSES 2017;[START_REF] Julia | Contribution of ultra-processed foods in the diet of adults from the French NutriNet-Sante study[END_REF]Salomé et al., 2021) and animal products would account for approximately 40% of daily calories (ANSES 2017;Fardet and Rock 2020).

Regarding the places of purchase, convenience stores, discounters, supermarkets and hypermarkets are often the primary suppliers of UPFs worldwide: this is the case in Latin America (Pan American Health Organization 2015); UPFs represented 83% of packaged foods in a sample of New Zealand supermarkets in 2013 [START_REF] Luiten | Ultraprocessed foods have the worst nutrient profile, yet they are the most available packaged products in a sample of New Zealand supermarkets[END_REF], and 67% of packaged foods in a sample of super-and hypermarkets in France in 2019 (Davidou et al., 2020). In France, in 2020, hypermarkets, supermarkets, and hard discounters represented 55%, 38% and 7% of all national employees, respectively (FCD 2020), with the six leading large food retailers accounting for around 89% of the total food market [START_REF] Leclerc | Complete review of 2019 market shares. LSA[END_REF]. 60% of French consumers report going to supermarkets and 58% to hypermarkets for their daily shopping, while diversifying their shopping in smaller food stores (30%) and organic food shops (18%) (Delvallée 2019). With supermarkets, French hypermarkets are therefore the leading place to buy UPFs. The first evaluation based on the NOVA classification according to degree of processing [START_REF] Moubarac | Food Classification Systems Based on Food Processing: Significance and Implications for Policies and Actions: A Systematic Literature Review and Assessment[END_REF] showed in 2012 that approximately 27% of sales were ultraprocessed in French organic specialized stores, compared to approximately 39% in French conventional stores (Desquilbet et al., 2018). In this latter study, the structure of food purchases was only assessed in monetary units and not in calorie units, limiting the dimensions of sustainability included in the analysis. Apart from this publication, purchases in French food retail stores have never been evaluated with a "One Health" approach (Fresco and Centre 2020).

In this context, the main objective of this study is therefore to assess the adequacy of food choices and purchasing behaviours in French hypermarkets according to the three metrics of the 3V rule. To this end, we assessed compliance with the 3V rule in the shopping carts of 708 regular consumers in 122 French hypermarkets, based on actual purchases in four separate months of 2019 covering the four seasons, identified by barcodes provided by a main retailer.

We also assessed potential differences in this compliance by season, according to the presence or absence of children in the customer profile, and by price per 100 calories. In particular, the presence of children was expected to influence parents to change their purchasing habits, especially towards healthier and more sustainable foods. Finally, the study will indirectly give an insight into how the environment of hypermarkets influences the sustainability of consumers' food choices.

Methods

Data and study population

Data collection

The data was provided by one of the four national leaders in food retailing, representing over 10% of the French retail market share. Hypermarkets were chosen as the shopping locations because they provide the largest number of food items, especially compared to supermarkets. The retailer's 122 hypermarkets across France were included in the analysis. The database provided by this retailer contained information on 521,023 food products (including repetitions) purchased by 708 selected regular customers in the 122 hypermarkets over four different months covering the four seasons of 2019. For each purchased food product, anonymous information was available on its customer (generic number and category), the hypermarket where it was purchased, and the season of purchase. Information on foods was also available from the barcodes (including those on packaged products and others directly generated within hypermarkets for non-packaged foods such as bulk and fresh foods weighed on the scale). For each different food (n = 38,168), information was given on its food section (30 options), food category (338 options), food sub-category (1,859 options), unitary price and the number of purchases by customer.

Study population

In order to include customers likely to make a significant proportion of their food purchases in the surveyed hypermarket, the selected customers were all in possession of a loyalty card and were chosen among those having made the greatest number of food purchases in the most varied and representative food sections. On the basis of their purchasing profile, these customers were classified into three generic categories representative of a model defined by Kantar© [START_REF] Housden | Marketing Research and Information[END_REF]): "Optimisers" who are primarily looking for the best prices and want to waste as little time as possible making their purchases, with reasoned and controlled purchases; "Local Traditionals" who rarely change their purchasing habits and are loyal to local and "made in France" products; the "Wealthy Families" who tend to frequent the drive-through hypermarket during the week; their purchases correspond to high quality products and thoughtful purchases for basic necessities.

Individual socio-economic data for each selected consumer (i.e., age, income, dwelling place, and sex) were confidential and protected by the French RGPD law (Le Règlement Général sur la Protection des Données, General data Protection Regulation), and may have been only given as percentages of all customers (Supplementary Table 1). The retailer also checked that these consumers did not purchase for food aid associations that provide free food in France to the most deprived, and whose average expenditure is far higher than the average expenditure of each consumer in this study (i.e., 523 ± 338 € per month).

For each of the 122 hypermarkets, a selection of customer belonging to the three Kantar categories, and with or without children (6 customer types) was attempted. 103 hypermarkets included all six types of customers, 14 hypermarkets included only 5 and 5 hypermarkets included only 4 customer types, leading to 708 consumers. Food expenditure covered all four seasons of 2019, with food purchases in January, April, August and October.

Data formatting for the 3V rule

First, all residual non-food items and non-caloric food items (i.e., nutritional supplements and waters) were removed from the database, resulting in a final database with 511,274 food products purchased in total, with 38,168 different food products purchased after the removal of repeated items. For 19,405 food products, the ingredient list was obtained from the French Open Food Facts database (https://fr.openfoodfacts.org/). Other food products were either present in this database, but without the list of ingredients, or absent from it (e.g., private labels). This open collaborative database of food products marketed in France, under the Open Database License (ODBL), is the most comprehensive database in this respect. To retrieve the list of ingredients of each food item from its bar code, its net weight and the total calorie of the product as sold, a simple algorithm was developed in our laboratory based on the JavaScript Object Notation (JSON: https://fr.openfoodfacts.org/api/v0/produit/'product bar code'.json), which is a data exchange format. In addition, 265 food products were sold under the private label of the French retailer. For these products, the company supplied the list of ingredients, the net weight and the total number of calories of the sold product, giving a total of 19,670 food products with a list of ingredients. We used the available data on these 19,670 food products to calculate the median calorie content per 100 g in each of the 338 food categories (as defined by the French retailer). For the remaining 18,498 food items, neither the retailer nor the Open Food Facts database provided information on the calorie content of the food item purchased. These food items corresponded mainly to niche products (i.e., local, traditional or foreign specialties) or to bulk and fresh products (i.e., unpackaged foods sold in hypermarkets, for which a specific bar code was assigned at the time of purchase in hypermarkets, i.e., 16% of the 18,498 food products) products. For the latter, we applied the median value of the calorie content per 100 g in each of the retailer's 338 food categories (homogeneous in the types of foods that constitute them), determined from the products for which the calorie content per 100 g was available in the Open Food Facts database. After controlling for non-normality of the calorie content of foods within each category (Shapiro-Wilk test, p > 0.05), we chose the median value as indicator of the average calorie content within a category. The impact of this approximation was limited by the high level of detail allowed by the large number of food categories, and the homogeneity of products within each of the 338 food categories.

Next, formatting according to the 3V rule was established for all foods in these 338 categories, as follows.

Rule 1: "Végétal" (Plant)

The final set of 38,168 different food products purchased was classified into three categories for this first rule: 100% plant-based foods, 100% animal-based foods and mixed foods containing both animal and plant-based ingredients. The first two categories were easy to identify (e.g., fresh and bulk products, butchery and bakery, cheese stand, creamery, canned vegetables, sugars, spices…), and we obtained 10,075 different animal food products and 17,285 different plant food products. The 10,808 mixed food products belonged to 127 mixed categories (with an average of about 85 products per category). Within each category, the different mixed foods were fairly homogeneous in terms of formulation. For example, in the "Fruit yogurt" category variations were mainly due to the type of fruit added, or in the "Biscuit", "Viennese pastry" and "Soft bread" categories, variations in recipes were also minor. We selected a representative recipe from each of the 127 mixed categories in order to approximate the shares of animal and plant calories. The percentages of animal and plant calories for each representative recipe were then imputed to all mixed food products in the given category in order to calculate the respective shares of animal and plant calories based on the amount of food purchased and the net weight.

Rule 2: "Vrai" (Real, i.e., non UPF) Ultra-processed foods can be found in plant, animal and mixed products. The degree of processing of each of the 38,168 foods purchased was assessed using the Siga methodology, which identifies UPFs from the ingredient list (Davidou et al., 2020). The Siga score according to the degree of processing (A: not or minimally processed, B: processed, and C: ultraprocessed) was assigned to the 19,670 food items for which ingredient list s ware available. Of the remaining 18,498 food items for which the list of ingredients was not available, all fresh unpackaged foods, bulk foods and catered foods were considered non-UPF (i.e., 60.7% of the 18,498 foods). For the remaining 7,269 packaged industrial foods, we applied the percentages of UPFs found in each of the 126 Siga food categories (defined to be representative of the food assortments of French super and hypermarkets) (Davidou et al., 2020) which we matched to the 338 French retailer food categories for this study. This allowed us to calculate the calorie share of UPFs based on the amount of food purchased and the net weight for each food purchased.

Rule 3: "Varié" (Varied)

The Varied rule applied to non-UPFs, i.e., "real" foods (Fardet and Rock 2020). In line with the study design, we used the Household Dietary Diversity Score (HDDS) (Swindale and Bilinsky 2006). Household dietary diversity, defined as the number of unique foods consumed by household members over a given period, was validated as a measure of household food availability. It measures the total number of these food groups consumed by members of the household, out of a possible twelve. We adapted this simple method for our study as follows.

The French retailer classification considers 293 food categories and 1,152 more detailed food sub-categories (e.g., for the category "citrus fruits", sub-categories are "lemon", "clementine", "orange", "pomelo", etc.). A variety index was first calculated among non-UPFs for each customer based on a combination of retail food categories with retail food sub-categories. This absolute variety index for each consumer is calculated by multiplying the number of food categories purchased by the number of food sub-categories purchased. We then selected the individual with the highest absolute variety index from the 708 consumers, and the Relative Variety Index (RVI) of the remaining 707 consumers was expressed as a percentage of the absolute variety index of this individual. This reference individual, with an RVI of 100%, has children and purchased 309 different non-UPF products over the four months of the year from 178 different food sub-categories and 107 different food categories, or on average 18 different non-UPF products per week per household. This appears to be a moderate level of food diversity among non-UPF products, especially when children are present. However, the number of children in each household was not known for this study, due to the French RGPD law for confidentiality, and only average percentages were given for the presence of 1, 2, 3 or ≥4 children in the household (Supplementary Table 1). For this particular consumer, 58% of the food was UPF. As an indication, in the 3V rule, food diversity corresponds to at least two different food varieties among usual plant and animal food groups, i.e., at least 24 different non-UPF products per week per person, but not per household (Fardet and Rock 2020).

The cost of the average and sustainable shopping cart

The average cost of the shopping cart for all 708 consumers was calculated based on the basis of purchase receipts provided by the retailer. From the prices per 100 kcal of UPF, non-UPF animal and non-UPF plant calories, the cost of an average shopping cart based on the 3V rule was estimated, i.e., with 15% animal calories, 15% UPF plant calories, and 70% non-UPF plant calories.

Statistical and machine learning analyses

The shares of plant and non-ultra-processed calories and the RVIs were determined for each of the 708 consumers over the four months, based on season, consumer category and the presence or absence of children. Based on calories purchased by each consumer, we also calculated the price for 100 ultra-processed and animal calories compared to the prices of non-UPF and plant calories. The effect of the criteria "presence of children" and "season" on the median percentages of animal and UPF calories purchased and on the median RVI in each of the three consumer categories was measured with the non-parametric Wilcoxon test after testing the normality of the data (Shapiro-Wilk test). The effect was considered significant when p < 0.05. Data were also analysed using multivariate analysis (principal component analysis, PCA) and machine learning (decision tree). Decision trees are used in the fields of decision support (e.g., retail or business intelligence) or data mining. They describe how to divide a population of individuals (e.g., customers of a company or users of a social network) into homogeneous groups according to a set of discriminating variables (e.g., age and socioprofessional category) and according to a given objective (also called "variable of interest" or "output variable", e.g., turnover, probability of clicking on an advertisement). The decision tree was applied to the 708 customers to define the purchase rules in the presence and absence of children. According to the 3V rule, the explanatory variables were "share of animal calories purchased (%)", "share of UPF calories purchased (%)", "share of animal expenditure (%, euros)", "share of UPF expenditure (%, euros)", and the RVI. Among the main machine learning algorithms, the Chi-square automatic interaction detector (CHAID) was chosen instead of the classification and regression trees (CART) and C4.5 algorithms because it is more suitable for an exploratory study with a large sample. In all three decision tree analyses, 90% of the customers were used for the learning sample and 10% remained for the test sample (model validation). The similarities and differences between the 708 consumers (illustrative variables) in terms of compliance with the 3V rule and prices of purchased calories (active variables) were also analysed and visualized by PCA.

The Wilcoxon and Shapiro-Wilk tests, PCA analyses and decision trees were all performed using SPAD9.1 software (Coheris © , Suresnes, France).

Results

Due to the lack of significant differences between the three Kantar categories as regards to the three metrics of the 3V rule (results not shown), they were merged, and only the influence of season and the presence or absence of children was studied.

Overall analysis based on the 3V rule

Overall, considering all consumers over four months of 2019 in the 122 hypermarkets, the mean percentages of plant and non-UPF calories were 59±8% (mean±SD) and 39±10%, respectively (Table 1). In relation to the consumer with the highest degree of variety, the mean and median RVI were 28±18% and 25%, respectively (Table 1). For 87% of consumers, plant and animal calories (i.e., Rule 1: Plant) were distributed in a range of 49-69% and 31-51%, respectively. For 76% of consumers, UPF and non-UPF calories (i.e., Rule 2: Real) were in a range of 29-49% and 51-71%, respectively. For 48% of consumers, the relative variety index RVI (i.e., Rule 3: Varied) was in a range of 15-35% (Figure 1).

Specific analysis based on the 3V rule according to variables of interest

The percentages of animal and UPF calories purchased did not vary significantly by season, remaining within a range of 40-42% for animal calories and 60-61% for UPF calories (Figure 2). The presence or absence of children did not imply significant differences in the percentages of animal and UPF calories purchased (Figure 3). For the third rule, Varied, the RVI for each consumer category was largely and significantly lower (-33%, p < 0.05, Wilcoxon test) when purchasers had no children (Figure 3).

Expenditure according to the 3V rule

The average price of 100 calories was 0.351 euros for animal calories versus 0.345 euros for plant calories, and 0.272 euros for UPF calories versus 0.315 euros for non-UPF calories. UPFs accounted for 57% of expenditure and 61% of total calories, while animal foods accounted for 50% of expenditure and 42% of total calories (results not shown). Overall, customers would have spent 1.7% less if they had only purchased plant calories, and 15.6% more if they had only purchased non-UPF calories. Considering only non-UPFs food categories, i.e., mainly fresh and bulk foods, 100 calories cost 0.343 euros, which is 26% more than the price of UPF calories.

In the animal products, 100 calories of UPF cost much less than non-UPF ones, from 0.295 to 0.466 euros respectively.

Altogether, the switch from UPF to non-UPF animal calories in hypermarkets results in an increase in expenditure of 37%, while the difference for plant calories is minor, at only 2%.

Finally, based on 38,168 food products and on the average price per 100 calories, a shopping cart based on the 3V rule, i.e., with 15% animal calories (0.466 euros/100 calorie), 15% UPF calories (0.236 euros/100 calorie) and 70% plant non-UPF calories (0.272 euros/100 calorie), would cost 4.6% less.

Decision tree and multivariate analyses

The decision tree analyses allowed us to define rules for the presence/absence of children irrespective of the three consumer categories (Figure 4). The most discriminating criterion was the RVI, with 34% consumers having an RVI ≥ 31.5%, including 69% of those "with children" (Figure 4). Notably, for the latter, the presence of children was characterized by the following rules: "RVI ≥ 31.5%" → "UPF calorie ≥ 45.5%" → "Animal price ≥ 41.5%".

Then, the customers were compared for their similarities and differences in the 3V rule profile through PCA (Figure 5a-b). The loading plot showed that the first two principal components expressed a high level of variance, i.e., 72.7%, meaning that the PC1 x PC2 plot was a relevant indicator of consumer compliance with the 3V rule (Figure 5a). It also showed that the dimensions of RVI (R = 0.11, p > 0.05) and percent UPF calories (R = -0.03, p > 0.05)

were not significantly correlated with the dimension of percent animal calories, whereas the RVI was inversely and significantly correlated with percent UPF calories (R = -0.21; p < 0.0001). Analysis of the PCA loading plot did not fully discriminate the 708 consumers according to the 3V rule (Figure 5b). Based on a combination of the loading and score plots, approximately 19 consumers (i.e., ≈2.7%) showed a relatively high level of adherence to the three rules combined (green ellipse), and approximately 39 consumers (i.e., ≈5.5%) showed a relatively low level of compliance to the three rules combined (red ellipse).

DISCUSSION

Overall adequacy to the 3V rule

An important finding of this study is the interrelationship between the "Vrai" and "Varié" metrics of the 3V rule with regard to purchasing behaviour, but not with the "Végétal" metric. Indeed, the more diversified foods consumers purchased, the less UPFs they purchased.

However, of the 708 consumers, only 2.7% exhibited a relatively 'high' level of 3V rule adequacy, while about 5.5% has a fairly 'low' level of adequacy. The food choices of the 708 customers are therefore far from the optimum 3V rule, with an average of 41% animal calories and 61% UPF calories, well above the optimum thresholds of 15% for each of these two metrics (Fardet and Rock 2020). Furthermore, on the basis of the RVI, the diversity of foods purchased was not very high, (median RVI of 25% compared to the consumer with the highest RVI set to 100%), which is similar to previous results from the French INCA3 study (2014-2015) on 2,121 adults representative of the French population (ANSES 2017;Dubuisson et al., 2019). In this population, the average daily animal calorie consumption was ≈41% (Fardet and Rock 2020). Furthermore, from 24,932 packaged foods in French super and hypermarkets, using the Siga score revealed that 67% of packaged foods were UPFs (Davidou et al., 2020), close to the 61% found in the present study which included fresh and bulked packaged products. Such a high level of packaged UPFs purchased from food retailers has already been described in other countries. In Norway, from a representative sample of food retailers, UPFs accounted for 59% of purchased and 49% of expenditures (Solberg et al., 2016); similarly, in New Zealand supermarkets, 83% of industrial packaged products were classified as ultra-processed foods [START_REF] Luiten | Ultraprocessed foods have the worst nutrient profile, yet they are the most available packaged products in a sample of New Zealand supermarkets[END_REF].

Alternatively, the study by Desquilbet et al. (2018) examined the plant or animal origin of food products as an indicator of the environmental and health impacts of sales and their degree of processing as an indicator of their health impact (Desquilbet et al., 2018). In French conventional stores and for conventional products, authors reported respectively 29.8%, 44.7% and 25.5% of plant-based, animal-based and mixed products , as a percentage of sales for 7,883 active households and more than 8 million purchases of food products (Kantar Worldpanel database). This corresponds to an average share of animal-based sales of 57% (Desquilbet et al., 2018) compared to 50% in our study.

Finally, considering a reference consumer with the highest degree of variety among non-UPF products, the data showed that 50% of all other consumers had an RVI below 25%. This low food variety at purchase can be related to the evolution of the French diet between 2006-2007 and 2014-2015, which is notably characterized by more industrial processed foods and ready-to-eat meals, and less frequent home food preparations, resulting in a lower diversity of raw or minimally processed foods (ANSES 2017).

Influence of children

The presence of children had no impact on the percentages of animal and UPF calories purchased. For the "Varied" rule, the median RVI for each consumer category was significantly lower when families did not have children, i.e., 33% less, as confirmed by the decision tree analysis. There is therefore a tendency to purchase a greater variety of products among non-UPFs when families have children. The greater variety of foods purchased appears quite logical, since the food supply for children may be different from that for adults; and households with children are larger than households without children, and therefore tend to purchase a wider range of products. One reason for the high level of purchase of UPF calories, i.e., about 61%, may be that UPFs can often be stored for a very long time. It was recently shown in the USA that low-income households participating in the Supplemental Nutrition Assistance Program (SNAP) "selected UPFs because of their familiarity and long shelf life, attributes that mitigated fear of wasting money on foods that may be rejected by children or spoil quickly" [START_REF] Moran | What factors influence ultraprocessed food purchases and consumption in households with children? A comparison between participants and non-participants in the Supplemental Nutrition Assistance Program (SNAP)[END_REF]. In addition, children of parents who were more confident in their cooking skills ate less UPFs [START_REF] Martins | Parents' cooking skills confidence reduce children's consumption of ultra-processed foods[END_REF]. This suggests that the population selected in this study, also purchasing very regularly from hypermarkets, may not be familiar with cooking raw and bulk products at home. Beyond their long shelf life, UPFs are also generally cheaper than non-UPFs [START_REF] Machado | Price and convenience: The influence of supermarkets on consumption of ultra-processed foods and beverages in Brazil[END_REF][START_REF] Gupta | Characterizing ultra-processed foods by energy density, nutrient density and cost[END_REF]Vandevijvere et al., 2020), which makes them very attractive, notably in households with children. Thus, it was reported in the French INCA3 survey that price is the first food choice criterion cited by households (48%), followed by consumption habit (43%), taste (38%) and origin of the product (36%) (ANSES 2017).

The price of animal and ultra-processed calories under the 3V rule

Although healthy diets are generally more expensive (Rao et al., 2013), the present results suggest that purchasing foods more in line with the healthy and sustainable 3V rule would not have been much more expensive in hypermarkets, except considering the additional organic dimension of foods. In the present study, a simulation showed that customers purchasing only plant and non-UPF calories would spend 1.7% less and 15.6% more, respectively. In addition, the cost per 100 UPF calories is approximately 16% lower than that for non-UPF calories, and 26% lower when considering only food categories without UPFs. These differences are lower than those observed in the US, where the cost per calorie was reported to be 64% lower for UPFs than for minimally processed foods (GuptaHawk et al., 2019). One possible explanation may stem from either higher prices for minimally processed and/or fresh foods or lower costs of UPFs in the US.

From a public health perspective, the lower price of UPFs may be a relevant factor in overweight and obesity, especially among low-income families, compared to the cost of a healthier diet [START_REF] Bonaccio | Low income is associated with poor adherence to a Mediterranean diet and a higher prevalence of obesity: cross-sectional results from the Molisani study[END_REF][START_REF] Darmon | Contribution of food prices and diet cost to socioeconomic disparities in diet quality and health: a systematic review and analysis[END_REF]. This statement can be generalized from other recently published data. From 103 countries, it has been shown that "as countries develop economically, overweight prevalence increased substantially among the poorest and stayed mostly unchanged among the wealthiest" (Templin et al., 2019). Similarly, in Brazil, the price of UPFs has been shown to be inversely associated with the prevalence of overweight and obesity (Mendes dos [START_REF] Mendes Dos Passos | Association between the price of ultra-processed foods and obesity in Brazil[END_REF], and supermarket circulars promote the sale of UPFs in their health and wellness sections with the presence of claims of reduced unhealthy or increased healthy nutrient content [START_REF] Botelho | Supermarket Circulars Promoting the Sales of 'Healthy' Foods: Analysis Based on Degree of Processing[END_REF]. More recently, in Belgium, it was also concluded that "diets with a larger caloric share of UPF were significantly cheaper (i.e., 2.3 less) than those with a lower contribution of these products" (Vandevijvere et al., 2020).

Strength and limitations

This study has a number of limitations. First, purchases in French hypermarkets (where 58% of French consumers declare going) are not representative of overall food purchases of the French population. The present study measures purchases, not consumption, and we have no information on food products purchased elsewhere. Indeed, an increasing number of people aer purchasing food outside hypermarkets and supermarkets, where the supply of UPFs may be much lower, e.g., open-air markets, small retailers, organic stores, short food circuits or collective catering. However, the consumers included in this study were regular customers of hypermarkets, i.e., possessing a loyalty card and expected to purchase a significant proportion of their food in the hypermarket. Their average expenses were 523 € per month, a value close to that reported in 2014 for French households with (678 euros) or without (436 euros) children [START_REF] Credoc | Budgets de référence ONPES: Étude réalisée à la demande de l'Observatoire national de la pauvreté et de l'exclusion sociale[END_REF], suggesting that the proportions of foods purchased outside the hypermarket would be relatively low. In addition, purchases on four months in 2019 resulted in an average of 5,885 purchased kcal/week for each buyer at household level (results not shown), i.e., 2,942 kcal/day/person for a family without children, and from 1,962 kcal/day/person for a family with children, which was not so far from the reported 2,102 kcal/day in French adult population (18-79 years) in [START_REF] Credoc | Budgets de référence ONPES: Étude réalisée à la demande de l'Observatoire national de la pauvreté et de l'exclusion sociale[END_REF]-2015(Dubuisson et al., 2019)). Also, the consumers selected in this study are regular and loyal consumers, therefore not representative of all consumers of this specific French retailer. Therefore, the study is only measuring how purchases of these consumers follow the 3V rule. Overall, this study indicates that the food environment faced by regular buyers in hypermarkets is not in line with sustainable and healthy nutritional patterns, which is of concern given the role of food environments in shaping dietary preferences (The High Level Panel of Experts on Food Security and Nutrition 2017).

Another limitation lies in the food products in the French retailer database to which a Siga score could not directly be assigned and in mixed food products for which the shares of animal and plant calories were approximated. These approximations imply that the overall values of 41% of animal calories and 61% of UPF calories found in this study may be slightly underestimated or overestimated. However, these values do not appear unrealistic compared to previous data found in supermarkets for UPF consumption and in the French INCA3 study for animal calories. Moreover, the approximations were made within homogenous food categories for both dimensions (i.e., animal and UPF calories), which limits the risk of under-or overestimations. Despite these limitations, this study has a number of strengths, including a precise picture of purchasing habits of regular French buyers in hypermarkets with respect to health and sustainable dietary patterns, the follow-up of actual food purchases during one year at different seasons, and the high representativeness of the places of purchase including all hypermarkets of the selected French retailer.

Conclusions

The shopping carts of regular customers of the selected French food retailer are far from the sustainable and healthy 3V rule with 26% more animal calories, 46% more UPF calories, and a quite low median RVI. This suggests that part of the French population (i.e., the regular ones) might not go to the hypermarket to consume 'sustainably', but, rather, to purchase long shelflife and emergency foods, with generally a high level of UPFs. The population of customers in this study is not representative of France, but this study reflects an overall habit of at least half of French consumers.

The results also highlighted that it is clearly possible to eat according to the healthy and sustainable 3V rule at a reasonably affordable cost in French hypermarkets, that switching from ultra-processed plant calories to non-ultra-processed plant calories in hypermarkets costs only

  Vandevijvere S, Pedroni C, De Ridder K and Castetbon K (2020) The Cost of Diets According to Their Caloric Share of Ultraprocessed and Minimally Processed Foods in Belgium. Nutrients 12, 2787.

Fig. 1 Fig. 2

 12 Fig. 1 Distribution of the compliance percentages with the 3V rule for each of the 708

Fig. 3

 3 Fig. 3 Adherence to the 3V rule depending on the presence of children or not within the

Fig

  Fig. 5a-b Principal component analysis loading a) and score plots b) derived from the "708

Acknowledgements

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Declaration of competing interest

Anthony Fardet has been a member of the Siga society's scientific committee since 2017.

database online.

2% more, and that replacing the 26% animal calories in excess with plant calories is a relevant strategy to approach the 3V rule without spending much more money. This could be possibly supported by subsidies funded by targeted taxations, and/or a rearrangement of food supply in hypermarkets, e.g., according to the degree of processing. Besides, placing reminders of healthy and sustainable diets in shopping environments where UPF sales are heavily promoted, such as hypermarkets, can be a useful strategy for them to promote healthy and sustainable food choices, and to guide consumer's preferences [START_REF] Botelho | Effect of a health reminder on consumers' selection of ultra-processed foods in a supermarket[END_REF]. Therefore, policies and interventions to promote healthy and sustainable eating aiming at reducing purchases of animal products and UPFs are expected to disrupt routine decisions at the point of purchase in super/hypermarkets, and encourage consumers to establish habits new of purchasing healthier and more sustainable foods [START_REF] Machin | The habitual nature of food purchases at the supermarket: Implications for policy making[END_REF].

Data Availability Statement

The data that support the findings of this study (i.e., the 511,274 purchased food products upon the year 2019 based on consumer's receipts) may be available from AUCHAN Retail © . Therefore, restrictions apply to the availability of these data, being theoretically under the French RGPD law (i.e., "Règlement Général sur la Protection des Données").

Abbreviations

3V rule: "Végétal" (Plant), "Vrai" (Real), "Varié" (Varied) PCA, Principal Component Analysis RVI, Relative Variety Index UPF, Ultra-processed food

Supplementary materials: Supplementary Table:


Erwan Tinturier

Nathalie Fournier-Leblanc

Eric Badel

Jean-Louis Julien

Jean-Louis Julien Mechanical

Mechanical stresses and long-distance signaling in Poplar

Keywords: Poplar, stem bending, electrical signal, hydraulic pressure wave, RNA seq

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.


Cyril Bozonnet

Marc Saudreau

Eric Badel

Thierry Améglio

Guillaume Charrier

Multi-physics modelling of freeze-thaw cycles effects on tree branches

come   L'archive ouverte pluridisciplinaire


Haytame Fallah
email: haytame.fallah@lis-lab.fr

Patrice Bellot
email: patrice.bellot@univ-amu.fr

Emmanuel Bruno
email: emmanuel.bruno@univ-tln.fr

Elisabeth Murisasco
email: elisabeth.murisasco@univ-tln.fr

Adapting Transformers for Multi-Label Text Classification

Keywords: Multi-label classification, Transformers, BERT, French Transformers

Pre-trained language models have proven to be effective in multi-class text classification. Our goal is to study and improve this approach for multi-label text classification, a task that has been surprisingly little explored in the last few years despite its many real world applications. In this paper, our originality is to propose architectures for the classification layers that are used on top of transformers to improve their performance for multi-label classification. Our contribution involves the evaluation of thresholding methods on several transformers, either by computing an individual threshold for each label (IT ) or a global one (GCT ). We also propose two approaches for multi-label text classification. The first consists in adding a parameter for learning the number of labels present for a given example (NHA). The second approach consists in adding a layer to the classification layers in order to learn the features for selecting the relevant labels while avoiding the use of thresholds (TL). We evaluate these approaches on two English corpora of newspaper articles and scientific papers and then on a new multi-label dataset of French scientific article abstracts publicly available. The evaluations show that the performance of our proposals exceeds that of state-of-the-art multi-label text classification methods for the evaluated datasets, and are transposable to any multi-label classification problem.

Introduction

Multi-label classification is a generalization of the multi-class classification problem, where each instance is associated with exactly one label. In multi-label text classification, the goal is to associate one or more labels to the input text sample. It is an important natural language processing task that has many applications in other NLP tasks such as question answering or entity recognition, but also real-world applications such as information retrieval (e.g. metadata enrichment and analysis in digital libraries) or content recommendation, and so on.

Multi-label text classification is a challenging task due to the fact that several factors must be taken into account such as the dependencies that can exist between the labels, the complexity of extracting semantic features from the noisy input text that can contain redundant information, and mapping those features to multiple targets, while also finding the discriminative information that allows the identification of each label of the document.

Several methods have been proposed to tackle the multi-label classification problem, whether it's traditional methods such as Binary Relevance [START_REF] Zhang | Binary relevance for multi-label learning: an overview[END_REF], or deep learning-based approaches such as CNN, RNN (and a combination of both approaches [START_REF] Chen | Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[END_REF]) or the attention mechanism. These methods manage to capture the semantic features of the document but fail to consider the dependencies that can exist between labels. Hierarchical models [START_REF] Yang | Hierarchical Attention Networks for Document Classification[END_REF] and graph neural networks [START_REF] Pal | Multi-Label Text Classification using Attentionbased Graph Neural Network[END_REF] as well as other architectures [START_REF] Yang | SGM: Sequence Generation Model for Multilabel Classification[END_REF] have been introduced to better capture those dependencies. But with the emergence of attention-based transformers [START_REF] Vaswani | Attention is All you Need[END_REF] and their ability to better extract the semantic representations of text documents, and adaptation of these models for multi-label text classification, that has the potential of achieving overall better results, is yet to be explored.

Few multi-label text datasets are popular among the papers treating the multi-label problem. AAPD [START_REF] Yang | SGM: Sequence Generation Model for Multilabel Classification[END_REF] and Reuters [START_REF] Lewis | RCV1: A New Benchmark Collection for Text Categorization Research[END_REF] seem to be the most used datasets in the literature. This is even more true for the french language, only a few studies have involved french datasets in multi-label text classification [START_REF] Bailly | Classification multi-label de cas cliniques avec Camem-BERT (Multi-label classification of clinical cases with CamemBERT )[END_REF][START_REF] Imane | Multi-label Categorization of French Death Certificates using NLP and Machine Learning[END_REF]. We, therefore, introduce in this paper a new french multi-label dataset.

In this article, our main contributions are :

• The creation of a French corpus of multi-label text classification, MFHAD (for Multilabel French HAL Abstracts Dataset), containing the abstracts of scientific articles obtained from the open archive HAL (https://hal.archives-ouvertes.fr) that contains more than one million papers, • The adaptation of available transformer models for multi-label classification, • The study of threshold selection methods for more efficient exploitation of the results of the transformer models, in particular, the choice of a global threshold , or a threshold specific to each label for the individual optimization of the labels, • The proposal of two alternative approaches to thresholding for the selection of relevant labels. The first one consists in introducing a parameter at the last layer of the transformer which will be trained for the calculation of the number of labels present in an example, the value of this parameter will be used to select the labels having the strongest activations;

The second one consists in adding a final layer to the model, which will have the same number of parameters as the second last layer (equal to the number of labels), in order to obtain more discriminating activation values for the given example, i.e. high activation if labels are present, low activation in the opposite case.

The article is organized as follows: the section 2 presents the approaches that address the multi-label classification problem, the sections 3 and 4 describe the proposed approaches and the section 5 is dedicated to experiments.

Related Work

In multi-class classification, each example (instance) 𝑋 in the dataset is associated with a single label. Multi-label classification is furthermore about being able to associate each entry with multiple Y labels, rather than just one.

Multi-label Classification Strategies

Multi-label classification methods can be classified into three categories: problem transformation, adaptation, and ensemble methods.

Problem Transformation (PT)

Problem transformation consists in 'transforming' the dataset to change the problem into a single-label multi-class classification. One such method is to consider all possible unique combinations of labels, label powerset [START_REF] Tsoumakas | Mining Multi-label Data[END_REF], and train a multi-class classifier 𝑀 : 𝑋 → 𝑃 (𝑌 ), where 𝑃 (𝑌 ) is the powerset of Y, the set of unique and distinct subsets of labels. In addition to the high number of possible labels that can reach 2 |𝑌 | , the challenge lies in finding enough examples for each combination of labels. For a large |Y|, the training and inference time of the models is high. It is important to note, that by transforming the problem into a multiclass classification, the dependencies that may exist between the different labels are no longer considered [START_REF] Luaces | Binary relevance efficacy for multilabel classification[END_REF].

Ensemble Methods

A set of multi-class classifiers can be combined to create a multi-label classifier. For a given instance, each classifier will predict a single label and all outputs of these classifiers are then combined via an ensemble method. One of these methods consists in considering a label as present if a percentage of classifiers having predicted this label is reached, also called the discriminative threshold. The 𝑅𝐴𝐾𝐸𝐿 algorithm [START_REF] Tsoumakas | Random k -Labelsets: An Ensemble Method for Multilabel Classification[END_REF] is another variation of this method. Classifiers trained on random subsets of the labels powersets are used for the creation of a multi-label classifier, the predictions of these classifiers go through a voting system for the final prediction. The use of multiple classifiers imposes strong constraints in terms of memory use, as well as the need to optimize a number of models that increase linearly with the number of labels in the dataset.

Problem Adaptation (PA)

Problem adaptation methods do not require a transformation of the dataset but an adaptation of classification algorithms, such as ML-kNN [START_REF] Zhang | ML-KNN: A lazy learning approach to multi-label learning[END_REF] which extends the kNN algorithm for multi-label data, or BP-MLL [START_REF] Zhang | Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization[END_REF] an adaptation of the backpropagation algorithm for neural networks.

The adaptation of deep learning algorithms for multi-labels remains in general an avenue with few contributions. An adaptation of these approaches could contribute to a significant increase in performance. The use of a single model without the need for prior data transformation is an efficient method to try to address the multi-label problem.

Thresholding Methods

Thresholding methods directly impact the choice of a label for the multi-label problem. The threshold can be adjusted in several ways, either to optimize all the labels (a global threshold), or to optimize each label individually (number of thresholds equal to the number of labels). Let 𝑚 be the number of examples in the test dataset (or validation) and 𝑛 𝑦 the number of labels. The four most commonly used strategies for choosing the threshold(s) are:

-SCut: Labels are optimized individually, thresholds are chosen based on the validation set, measured either by maximizing a score or minimizing a cost function and without guaranteeing a global optimum. This method can also be used to obtain a global threshold;

-RCut (Rank Cut) : Labels are ordered according to their score, the 𝑡 first labels are chosen as relevant labels. The 𝑡 parameter is either predefined or set from the validation dataset [START_REF] Yang | A study of thresholding strategies for text categorization[END_REF]; -PCut (Proportion Cut): for each label 𝑦 𝑖 , the instances of the test data set are ordered according to the score obtained for this label. The first 𝑘 𝑖 instances are chosen for the label 𝑦 𝑖 where 𝑘 𝑖 = 𝑃 (𝑦 𝑖 ) × 𝑥 × 𝑛 𝑦 is the number of instances assigned to this label. 𝑃 (𝑦 𝑖 ) is the probability an instance belonging to the label 𝑦 𝑖 (computed from the training set), and 𝑥 the average number of instances to be assigned for any label previously set. If 𝑥 = 𝑛 all instances are taken, for 𝑥 = 0 no instance is considered to be part of the evaluated label [START_REF] Yang | An evaluation of statistical approach to text categorization[END_REF][START_REF] Lewis | A Comparison of Two Learning Algorithms for Text Categorization[END_REF];

-MCut (Maximum Cut): the labels are ordered from the scores obtained for an instance of the dataset, the threshold is equal to the average of the two contiguous labels for which the score difference is the most important [START_REF] Largeron | MCut: A Thresholding Strategy for Multi-label Classification[END_REF].

Variations of these methods aiming to overcome the constraints they may impose have been proposed [START_REF] Yang | A study of thresholding strategies for text categorization[END_REF]. The classification score-based methods are the best among the thresholding approaches [START_REF] Al-Otaibi | Multi-label Classification: A Comparative Study on Threshold Selection Methods[END_REF]. In this paper, the proposed approaches are similar in nature to the Scut and RCut methods, but with different implementations.

Deep learning based approaches

[20] uses a neural network for multi-label classification in spectroscopic multi-composition analysis, a network composed of a classifier to which a parameter is added for learning an activation threshold. This parameter will be optimized according to the threshold computed by applying the model on the training set, the target value of the threshold will thus be different for each iteration of the training phase, rendering the training process much more difficult. On the other hand, MAGNET [START_REF] Pal | Multi-Label Text Classification using Attentionbased Graph Neural Network[END_REF], a graph network implementing the attention mechanism to capture the dependency structure between labels that uses BERT's embeddings, explicitly tries to tackle the Multi-label text classification problem and manages to have good performances in F1 for the AAPD and Reuters datasets (cf. section 5.2). DocBERT [START_REF] Adhikari | DocBERT: BERT for Document Classification[END_REF], which is now the state of the art reference, adds a linear network to the head of BERT, but without subsequent processing of the model outputs that can enhance the performance of the transformer (e.g. thresholding techniques to better choose the correct present labels).

Transformers have been used for the 'extreme' multi-label classification of text where very large corpora of texts with a number of labels that can reach tens of thousands are processed. Such architectures are not well suited for short texts.

Attempts to use neural networks for text classification do not focus on multi-label classification. Those that do deal with this problem generally do not give importance to how the output layer activations are exploited (e.g. use of thresholds).

BERT Adaptation

BERT introduces bi-directionality in the prediction of masked tokens, where both the left and right semantic contexts of the word to be predicted are considered. In addition to masked language modeling, BERT is trained for next-sentence prediction, a task where the model receives a pair of sentences and tries to predict whether the second sentence follows the first. BERT introduces a special classification token [CLS] (also having an identifier and an embedding vector) containing a hidden state of the sentence, updated in each layer of the model.

A feed-forward neural network (FFNN) of 𝐿 dense layers (usually 𝐿=2) is added on top of the last transformer layers of the model. This is done to fine-tune the pre-trained transformer for the desired NLP task (text classification in our case). The token [CLS] is the input of this FFNN, and 𝑛 𝑦 outputs corresponding to the labels of the dataset (similar approach to [START_REF] Adhikari | DocBERT: BERT for Document Classification[END_REF]. Figure 1 shows the architecture of the model.

For multi-label classification, the values of the 𝐴 [𝐿] activations of the 𝐶 [𝐿] output layer can be used to determine the presence of a label. Each activation can be a value between 0 and 1 representing a probability of the presence of the corresponding label. A threshold is then used for the label selection process, the trivial value of 0.5 is usually used for this purpose. The Sigmoid 𝜎 function is the activation function that is suited for this case, alongside the Binary Cross Entropy as a loss function.

This approach cannot be considered completely as a problem transformation, it does not require a transformation of the dataset or the creation of multiple binary classifiers. It also does not a complete adaptation of neural networks for multi-label classification, the results must be processed later on for the final classifications.

… … … C (1) C (n) C (0) ([CLS]) n y activations : a l (n)

#of Labels Parameter

Transformer

MAE / MSE

BCE

(a) Added number of labels parameter architecture (NHA approach)

… … … n y activations C (1) C (2) C (0) ([CLS]) Optimizer-2 Optimizer-1 … C (n)

Thresholding methods

Thresholding approaches can be applied to the transformer if we consider each activation in the final layer to be a representation of a binary classifier of the label it represents. If the activation is high, the label is considered to be a valid label, and vice-versa.

We propose two alternative methods to thresholding in an attempt for more efficient use of the 𝐴 [𝐿] activation values of the output layer. These approaches aim to avoid thresholding by learning text-specific features for a better label selection.

Global Classification Threshold (GCT)

The classification threshold 𝑠 can be chosen to maximize the classification scores. During the training of the model and after each iteration, the value of 𝑠 is varied from 0 to 1, with a step defined beforehand (10 -2 ). The micro-F1 score is then computed for each threshold 𝑠. We finally obtain the global optimal threshold 𝑔𝑐𝑡 which provides the best performance on the training dataset. This threshold is then used for the validation and test datasets. The set of labels 𝐿 present for an example 𝑥 can be expressed as:

𝑌 𝑥∈𝑋 = ∪ 𝑦∈𝑌 {𝑦 𝑖 } : 𝜎(𝑎 [𝐿]) 𝑦 𝑖 ) ≥ 𝑔𝑐𝑡 (1)

𝑎 [𝐿]

𝑦 𝑖 being the activation corresponding to label 𝑦 𝑖 among the activations 𝐴 [𝐿] . A variant of the SCut method (with a global threshold) consists in fine-tuning the optimal threshold from the validation dataset and then applying it to the test dataset. Only the first approach has been studied in this article.

Individual Thresholds (IT)

The use of a shared threshold 𝑠 for all labels assumes that the features associated with the activations 𝐴 [𝐿] are the same for each label, which is not the case, the activation intensity of a neuron in the last layer 𝑎 𝑦 𝑖 , when the label 𝑦 is present, varies from label to label, an effect that is accentuated if the dataset is unbalanced.

We, therefore, propose to evaluate the SCut method (with individual thresholds) by setting a threshold 𝑖𝑡 𝑦 for each label 𝑦 in the dataset, assigned to the 𝑎

[𝐿]

𝑦 𝑖 activations of 𝐶 [𝐿] . The values of 𝑖𝑡 𝑦 are the values that maximize the classification scores for a label 𝑦. These thresholds are computed as for 𝑔𝑐𝑡 during the training phase of the model, and on the training dataset, by varying each threshold to maximize the F1 score for each label.

𝑌 𝑥∈𝑋 = ∪ 𝑦∈𝑌 {𝑦 𝑖 } : 𝜎(𝑎 [𝐿] 𝑦 𝑖 ) ≥ 𝑖𝑡 𝑦 (2)

N Highest Activations (NHA)

Using a threshold to determine the presence of a label leads in several cases to an underclassification (respectively over-classification) of an instance when the predicted number of labels is lower (respectively higher) than the actual number of labels. The first proposed alternative consists in introducing a neuron at the last layer which will be used only for computing the number of labels 𝑦 present for an instance. Let 𝐴 ′[𝐿] be the list of the 𝑁 highest activations, and 𝑁 being the actual number of labels present for a given instance:

𝑌 𝑥∈𝑋 = ∪ 𝑦∈𝑌 {𝑦 𝑖 } : 𝑎 [𝐿]) 𝑦 𝑖 ∈ 𝐴 ′[𝐿]) (3)
The number of labels present for an instance is the target value, the mean absolute error MAE is used as a loss function for this regression problem. We use a single optimizer in the backpropagation, therefore, the error of the regression must be scaled to match the classification error, done by reducing the regression error by a factor of 5. The value of this neuron will be used to recover the 𝑁 highest activations that will be considered as predicted labels, similarly to the RCut method. Figure 2a shows the model architecture for this approach. The objective of this architecture is to extract from the [CLS] token, criteria or features about the number of distinct topics present in the text. This token, the final output of the transformer layers, contains an information-rich representation of the input text.

Threshold Layer (TL)

The second proposed approach is based on the addition of a dense layer, after the output of the classifier, for a total of 𝐿 = 3 layers, having 𝑛 𝑦 neurons to match the last classification layer of the model. This additional layer aims to make the values of the final activations as close as possible to 1 in the case of the presence of the label, or to 0 in the opposite case. Therefore, the trivial threshold of 0.5 can be used for the classification.

The addition of this layer could result in an increase in the recall of the classifier as many activations of the non-present labels will no longer exceed the classification threshold since they will be as close as possible to 0. A gain in precision can also be expected as the activations of the present labels will be emphasized more than the activations of the irrelevant labels.

For this approach, we use two optimizers, the first one which includes and optimizes all the layers of the transformers as well as the first two layers of the classifier, and another one dedicated to the optimization of the last layer that we added. The error function used is the same for both parts of the classifier, the BCE in this case. The final architecture of the classifier is presented in the figure 2b

Experiments and Results

We present in this section the results of the evaluation of non-deep learning baselines as well as all the previously mentioned methods on three multi-label text datasets, including the French dataset "MFHAD" that we have designed and made available. We compare the different methods, i.e. the thresholding methods as well as the proposed alternatives coupled with BERT transformer and its variants, to baseline approaches, all put in perspective with optimal target results (oracle approaches).

Evaluated Transformers

For the evaluation of the proposed methods, we use HugginFace's [START_REF] Wolf | HuggingFace's Transformers: State-of-the-art Natural Language Processing[END_REF] implementation of the uncased-base version of BERT, with 12 transformer layers and an embedding vector of size 768 dimensions, as well as variants of BERT (uncased-base versions):

-RoBERTa [START_REF] Liu | RoBERTa: A Robustly Optimized BERT Pretraining Approach[END_REF] a variant with an optimized training process, where the Next Sentence Prediction part of BERT is removed. The dataset used is ten times larger than the one used for BERT. This has led to a performance gain over the original version on the automatic language processing tasks in the GLUE benchmark;

-DistilBERT [START_REF] Sanh | DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter[END_REF], an efficient variant of BERT that uses knowledge distillation to shrink the size of BERT by 40% while keeping 97% of its performance. DistilBERT is based on the fact that after training a very large model, the output distribution can be approximated by a much smaller neural network, using the Kulback-Leiber divergence [START_REF] Kullback | On Information and Sufficiency[END_REF] as the optimization function ;

-DeBERTa [START_REF] He | DeBERTa: Decoding-enhanced BERT with Disentangled Attention[END_REF], the most recent variant of BERT where words are represented by two vectors that encode their content and their relative position in the sentence. It is also characterized by the optimization of the decoding of the prediction of hidden tokens, which contributes to a significant gain in efficiency in the pre-training phase of the model, but also in the performances concerning the various natural language processing tasks;

-CamemBERT [START_REF] Martin | CamemBERT: a Tasty French Language Model[END_REF] is based on RoBERTa, trained on the French part of OSCAR [START_REF] Suárez | Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures[END_REF]; -FlauBERT [START_REF] Le | FlauBERT: Unsupervised Language Model Pre-training for French[END_REF] trained on various French sub-corpora of different writing styles, from formal writings (e.g. Wikipedia and books), to writings extracted from the internet (e.g. Common Crawl). For this variant, we use the cased-base version.

The comparison of different transformers is important to evaluate the performance of our approaches as well as their re-usability and applicability on different transformer architectures.

Datasets

Multi-label text classification is not present in NLP leaderboards, the GLUE benchmark for example, nor in recent CLEF or Semeval conferences. There are few multi-label corpora frequently used in the literature that can be used to evaluate and compare models, and this is even more true for the French language. To remedy this, we have built our own corpus comprised of abstracts of scientific articles written in French from HAL, a corpus that we make available to the public. In this section, we provide details about the different datasets used 1 for the evaluation of the different models :

-MFHAD2 : Our dataset is comprised of abstracts of french scientific papers collected from 'HAL', an open academic research archive, distributed over three major scientific domains, i.e. computer science, physics, and mathematics. Articles published between 1980 and 2021 that have a french abstract, which represents 15 771 documents distributed over 200 different labels;

-Reuter-215783 is a collection of articles from the Reuters newswire from the year 1987. It is a dataset that has been often used to evaluate models for multi-label text classification. An article can belong to one or more of the 90 domains of the dataset;

-AAPD (or ArXiv Academic Paper Dataset) is, not unlike the "MFHAD", a collection of the "Abstract" of several scientific publications. An article can have one or more classifications among 54 labels. We use the same training, validation, and test distribution as [START_REF] Yang | SGM: Sequence Generation Model for Multilabel Classification[END_REF].

Table 1) and figure 3 present the different characteristics of these datasets in more detail.

Evaluation Method

We will compare the two proposed approaches as well as the coupling of thresholding methods to transformers with methods that are more explicit on the label selection criteria, but also with other adaptations of deep learning models to multi-label text classification. We will also compare all these methods to several upper bounds that represent the optimal results that can be achieved.

Baselines

We will first do a comparison with non-neuronal approaches, with interpretable machine learning classification criteria, using TF-IDF extracted features (with no maximum number of features for each dataset), as inputs: -Decision trees using the "Gini" criterion, without a maximum tree depth, and 2 as the minimum samples for splitting nodes; -Random Forest with 100 estimators (number of trees) and the same parameters used in the previous method for each tree;

-Bagging using decision trees as main estimator (10 estimators); -GradientBoosting with logistic regression as error function, a learning rate of 0.1 and 100 estimators;

-Support Vector machine using RBF as the kernel and a regularization parameter of 1.0.

We also include deep learning approaches as a comparison for the evaluation of our approaches:

-CNN [START_REF] Kim | Convolutional Neural Networks for Sentence Classification[END_REF] and CNN-RNN [START_REF] Chen | Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[END_REF] which use convolutional neural networks to extract textspecific features;

-SGM [START_REF] Yang | SGM: Sequence Generation Model for Multilabel Classification[END_REF] which applies a sequence generation model with a new decoder structure for multi-label classification;

-MAGNET [START_REF] Pal | Multi-Label Text Classification using Attentionbased Graph Neural Network[END_REF] a graph network implementing the attention mechanism to capture dependencies between labels;

-DocBERT [START_REF] Adhikari | DocBERT: BERT for Document Classification[END_REF]: a fine-tuning of the base and large versions of BERT for document classification.

To evaluate the performance gain of our approaches, we compare them to an unchanged version of each transformer model, where no architecture modification is applied, and the classification threshold is the trivial value of 0.5.

Oracle Approaches

The target theoretical optimums (oracle approaches) are represented by the following two approaches:

-The oracle approach for the N highest activation, where we consider the number of labels present for an instance as a given and then take the 𝑁 highest activations as the present labels; -The oracle approach for both SCut thresholding methods, where the calculation of the global threshold and the individual thresholds is done from the test dataset. Considered as the optimal results towards which these methods should come as close as possible.

Results

In this section, we present the performances of all the approaches mentioned in the previous sections, tested on the different transformers presented in section 3. We propose the following notations for these approaches:

-"GCT": The Global Optimal Threshold method (see section 4.1); -"IT": The Individual Threshold method (see section 4.2); -"NHA": The N largest activations approach (see section 4.3); -"TL": Denotes the "thresholding layer" approach (see section 4.4). As for the oracle approaches, they will be designated by adding 𝑜𝑟𝑎𝑐𝑙𝑒 after the corresponding approaches. The maximum length of the sequences used is 512 tokens for all datasets. Tables 2 and4 show the micro-F1 (with precision and recall) and Accuracy scores for the English and French corpus test datasets respectively. The tables 3 and 5 present the optimum scores for the different approaches. The results of the thresholding methods as well as the proposed architectures were obtained from the base versions of the transformers used.

Transformers outperform all other methods, whether it is the classical methods, or other deep learning methods. Our baseline versions of BERT and its variants achieves a better performance than the base version of DocBERT (the current state of the art in multi-label text classification for the AAPD and Reuters corpora), and for Reuters, better than its large version. This is due to the longer training in the fine-tuning process, 150 epochs for the Reuters dataset vs 30 in the case of DocBert, and 40 epochs for the AAPD dataset vs 20. For transformers, a longer training process generally yields better performance with a low risk of over-fitting.

We also note that SVMs and decision trees obtain the highest micro-precision scores, at the expense of recall rate, thus lowering the micro F1 score. But precision is not a reliable factor for performance measurement in multi-label classification. SVM can be considered as the best performing non-neural approach due to its high accuracy score, but it falls short of the other methods tested.

The GCT thresholding techniques seems to be the best among the evaluated approaches, it manages to get a better micro-F1 score than its thresholding counterpart, namely the IT approach, with a score of 91.02 vs 90.70 for the Reuters dataset, and 74.08 vs 74.04 for AAPD (see table 2). For the Reuters Dataset, the thresholding techniques and the proposed alternatives do not manage to achieve a gain in micro-F1 scores compared to the baseline versions, except for the RoBERTa model. This does not apply to the AAPD dataset, gains in performance can be perceived in using thresholding techniques or the TL approach. Especially for the RoBERTa model where the gain is the highest. The large version of DocBERT remains the best performing model for AAPD. The complex nature of the scientific vocabulary of this corpus underlines the possible benefits of increasing the size of the transforming models (adding several layers and increasing the embedding dimension).

The difference in performance gains between the datasets can be due to their complexity level. The proportion of shared tokens/words between the dictionary of the transformers and the Reuters corpus is much higher than AAPD due to its scientifical nature, which makes the training process much more difficult on the latter, therefore rendering the thresholding and alternative approaches more useful to gain performance on the multi-label classification task. Those techniques won't be as useful if the learning process yields a good overall semantic understanding of the dataset, in that case, a threshold of 0.5 is generally more than enough.

The RoBERTa model is the lowest performing variant of BERT. But the performance gain on RoBERTa is more perceivable for all the evaluated datasets. This can be explained by the fact that for this variant of BERT, the Next-Sentence Prediction objective is removed in the pre-training process. The goal of NSP is to learn long dependencies that can exist across sentences, whereas Masked Language Modeling is more focused on understanding relationships on the word level. For multi-label text classification, distinguishing the difference between domains across multiple sentences is vital, and NSP is certainly a good contributing factor for this purpose.

The same conclusions can be drawn from the performance data on the MFHAD dataset, where the baseline version of FlauBERT outperforms its CamemBERT counterpart (65.92 vs 64.47 in the F1-score). This comes as no surprise knowing that the latter is a variant based on RoBERTa that also lacks the NSP objective, but also the fact that FlauBERT is pre-trained, in addition to text crawled from the internet (Common-Crawl), on wikipedia article and books, thus getting more vocabulary coverage of MFHAD than CamemBERT that is pre-trained on a corpus derived from Common-Crawl. The evaluated approaches fail in obtaining a gain in performance for FlauBERT that achieved good performance with its baseline version but allow CamemBERT to get closer to FlauBERT 's performance.

The TL architecture performs better than the base version of the DocBERT and MAGNET models, and matches the large version of the former with a score of 90.70 for the DeBERTa model on the Reuters dataset. The NHA approach does not succeed in reaching its theoretical optimal, even though it manages to obtain a score exceeding the base version of DocBERT for the same dataset (see Transformer Architecture Adaptation part of table 2). These two architectures require less training time than the two thresholding methods (1.5× faster), but have a higher inference cost, especially for TL where it's almost twice that of the other approaches, due to the presence of the extra layer that this architecture requires.

As shown in tables 2 and 4, an increase in the recall is observed for almost every model and every dataset for the two thresholding methods. This is due to the fact that modifying the classification thresholds (especially by lowering them) can lead to more labels being predicted as present, but can also lead to a decrease in precision. The same can be said for the TL alternative method where adding an additional layer contributed in some cases to an increase in the final activations' values, thus using a threshold of 0.5 might include some labels that couldn't otherwise be predicted. The opposite effect is observed for the NHA method, where the model predicts the most frequent number of labels present in the unbalanced dataset which is always the smallest. This leads to an increase in precision (fewer labels are being predicted thus fewer false positives) but this comes at the cost of the recall.

The IT 𝑜𝑟𝑎𝑐𝑙𝑒 approach is the best performing among the evaluated oracle approaches. Computing an individual threshold for each label from the test set leads to a global optimal score for all labels, surpassing the optimum for the GCT method. But this is not the case for its experimental equivalent. GCT remains the method which comes closest to its theoretical optimum (cf. tables 3 and 2). This can be explained by the fact that for the IT method several thresholds must be computed for each label, which does not guarantee a global optimal on all the labels for the test set. This effect is amplified if the number of labels is large.

For the theoretical optimal of the method NHA, providing the actual number of relevant labels allows new labels to be considered as present, which increases true positives and decreases false negatives. But this could lead to an increase in false positives as these new labels are in some cases non-valid predictions. The instances for which this approach reduces the number of what would have otherwise been predicted by the model (using 0.5 as a threshold) are rare. Thus the micro F1 score in some cases may be lower compared to the theoretical optimal of the thresholding approaches. On the other hand, a considerable increase of the accuracy is observed, it is in fact the oracle approach that achieves the highest accuracy scores as shown in tables 3 and 5. The actual implementation of this approach did not achieve the same performance as its theoretical optimum. The hidden state of the sentence contained in the [CLS] token may not be sufficient to estimate the number of existing labels, a task that is made harder by the unbalanced nature of the datasets (regarding the number of labels per instance). Exploiting the attention scores obtained in the different layers of the model could be a more efficient method to accomplish this task.

DeBERTa is the best performing variant of BERT among all the evaluated models, the changes made by this variant regarding the addition of the relative position of the word to the sentence as an input, seem to improve the performance of BERT. This performance improvement comes at the expense of the training speed of the model (18% slower training speed than the other variants on average, but with similar inference speeds). As for DistilBERT, it achieves results at the same level as its original version, despite its reduced size. Knowledge distillation seems to be an efficient method to overcome the major drawback of transformers and neural networks: the necessity of a long training time.

Conclusion and Future Work

As far as the application domain is concerned, multi-label classification is a relevant task. However, multi-label text classification is not a common task and, regrettably, it is not included in the most prominent benchmarks such as GLUE.

Transformer-based language models outperform other deep neural architectures and provide a strong adaptable foundation for a multitude of NLP tasks and text classification, which is the direction we followed in this study.

First, we have tested and shown in this paper that thresholding approaches can achieve a performance gain if the learning process does not yield good results for datasets of a complex nature. Computing a global threshold achieves higher results than computing an individual threshold for each class. The optimization done on each class does not guarantee a general optimal for all labels. We then proposed modifications to the transformer architecture. This is done by adding an additional layer, with a number of activations equal to the number of classes, at the bottom of the model to avoid optimizing thresholds. An approach that, on average, is as effective as thresholding approaches. The analysis of the optimal to be reached showed that the use of the number of classes for the selection of labels significantly increases the performance. However, the difference between the optimal and the actual results of our experiments shows that our proposal needs to be improved in the case of unbalanced data sets. The language of the text corpora, English for AAPD and Reuters, or French for the MFHAD corpus we have built, does not seem to be a factor that impacts the performance of the proposed approaches. These approaches can be used for any multilabel classification problem. Each BERT variant seeks to improve the constraining aspects of the original model. DistilBERT, despite its reduced size, obtains results as high as the original version. DeBERTa is the best performing variant, together with CamemBERT which outperforms FlauBERT.

We have shown the potential of approaches for adapting the transformer model for multi-label text classification. All these approaches are not limited to text classification but can be used for any other multi-label classification task. Our future work will be focused on exploring other methods of adapting neural networks for multi-label text classification. The thresholds can be learned parameters of the model during the training phase. Finding a more effective method for computing the number of classes present for the NHA method, and finding a better way to represent the dependencies between the labels for the TL approach, are other areas we will explore in the future.

Figure 1 :

 1 Figure 1: BERT architecture with a dense classification layer, on top of the transformer layers, connected to the [CLS token].

Figure 2 :

 2 Figure 2: Alternative approaches' architectures

Figure 3 :

 3 Figure 3: Instance count based on the number of labels for all datasets.

Table 1

 1 Dataset used, W is the average number of words per abstract.

		#Train	#Valid	#Test	labels	W
	MFHAD	11035	2366	2370	200	140.75
	AAPD	53840	1000	1000	54	163,16
	Reuter-21578	5827	1943	3019	90	127,76

Table 2

 2 Scores for test dataset from Reuters and AAPD, best scores are in bold blue, the blue up arrow ↑ indicates when an evaluated approach achieves a gain in performance for the associated model. Orig. refers to the original result taken from the corresponding paper.

	Modèles	Pr.	R	Reuters	F1	Acc	Pr.	R	AAPD	F1	Acc
					Baselines
	Decision Tree	78,36	75,05	76,67	74,23	49,67	46,8		48,19	26,6
	Bagging	88,12	79,65	83,67	73,34	77,27	48,16		59,34	26,9
	Random Forest	97,18	57,13	71,96	64,06	94,2	25,49		40,12	20
	GradientBoost	88,06	80,56	84,14	74,23	79,73	46,8		58,98	27,1
	SVM	94,19	79,62	86,29	80,64	80,85	59,98		68,86	36,2
	CNN [30] Orig.	-	-		86,3	-	-	-			66,4	-
	CNN-RNN [2] Orig.	-	-		85,5	-	-	-			66,9	-
	SGM [5] Orig.	-	-			-	-	-	-			71,0	-
	MAGNET [4] Orig.	-	-		89,9	-	-	-			69,6	-
	DocBERT 𝑏𝑎𝑠𝑒 [21] Orig.	-	-		89,0	-	-	-			73,4	-
	DocBERT 𝑙𝑎𝑟𝑔𝑒 [21] Orig.	-	-		90,7	-	-	-			75,2	-
	BERT 𝑏𝑎𝑠𝑒	91.36	90.46	90.91	86.18	76.33	71.95		74.07	41.5
	DistilBERT 𝑏𝑎𝑠𝑒	91.29	90.41	90.84	86.52	80.84	66.41		72.92	41.0
	RoBERTa 𝑏𝑎𝑠𝑒	91.23	87.60	89.38	85.29	74.22	71.50		72.83	40.1
	DeBERTa 𝑏𝑎𝑠𝑒	91.63	90.41	91.02	86.78	75.99	71.00		73.41	41.0
	Thresholding without any specific architecture (GCT and IT)
	BERT 𝑏𝑎𝑠𝑒 + 𝐺𝐶𝑇	90.0	91,.4 ↑	90.86	86.45 ↑	75.56	72.65 ↑ 74.08 ↑ 41.7 ↑
	BERT 𝑏𝑎𝑠𝑒 + 𝐼𝑇	88.89	92.30 ↑	90.56	85.72	75.51	72.61 ↑		74.04	41.3
	DistilBERT 𝑏𝑎𝑠𝑒 + 𝐺𝐶𝑇	90.83	90.78 ↑	90.80	86.29	75.73	72.08 ↑ 73.86 ↑	39.8
	DistilBERT 𝑏𝑎𝑠𝑒 + 𝐼𝑇	88.40	91.66 ↑	90.0	85.29	79.06	68,31 ↑	73.3 ↑	41.0
	RoBERTa 𝑏𝑎𝑠𝑒 + 𝐺𝐶𝑇	90.73	89.90 ↑ 90.31 ↑ 86.28 ↑ 74.49 ↑ 72,49 ↑ 73.47 ↑ 40.3 ↑
	RoBERTa 𝑏𝑎𝑠𝑒 + 𝐼𝑇	89.77	90.49 ↑ 90.13 ↑ 85.92 ↑ 75.35 ↑	71,09	73.15 ↑ 40.2 ↑
	DeBERTa 𝑏𝑎𝑠𝑒 + 𝐺𝐶𝑇	91.63	90.41	91.02	86.78	74.46	73.56 ↑ 74.01 ↑	39.4
	DeBERTa 𝑏𝑎𝑠𝑒 + 𝐼𝑇	90.67	90.92 ↑	90.79	86.61	75.87	72.08 ↑ 73.92 ↑	40.7
		Transformer Architecture Adaptation (NHA et TL)
	BERT 𝑏𝑎𝑠𝑒 + 𝑁 𝐻𝐴	92.33 ↑	85.87	88.98	85.92	73.48	66.38		69.75	40.3
	BERT 𝑏𝑎𝑠𝑒 + 𝑇 𝐿	90.60	90.41	90.50	86.12	73.48	72.20 ↑		72.83	39.5
	DistilBERT 𝑏𝑎𝑠𝑒 + 𝑁 𝐻𝐴	92.08 ↑	86.03	88.95	86.15	73.93	66.29		69.90	41.8 ↑
	DistilBERT 𝑏𝑎𝑠𝑒 + 𝑇 𝐿	90.0	89.66	89.83	85.89	73.63	72.32 ↑ 72.97 ↑	40.2
	RoBERTa 𝑏𝑎𝑠𝑒 + 𝑁 𝐻𝐴	89.43	83.20	86.20	83.40	75.04 ↑	66.58		70.56	42.0 ↑
	RoBERTa 𝑏𝑎𝑠𝑒 + 𝑇 𝐿	91.17	89.05 ↑ 90.09 ↑ 86.02 ↑ 76.48 ↑ 71.58 ↑ 73.94 ↑ 42.5 ↑
	DeBERTa 𝑏𝑎𝑠𝑒 + 𝑁 𝐻𝐴	92.22 ↑	86.41	89.21	86.35	75.32	65.67		70.16	41.9 ↑
	DeBERTa 𝑏𝑎𝑠𝑒 + 𝑇 𝐿	90.97	90.43 ↑	90.70	86.12	75.97	71.70 ↑ 73.78 ↑ 41.8 ↑

Table 3

 3 Scores of the oracle approaches for the test set of the Reuters and AAPD datasets (the best scores are in bold blue).

	Models	Pr.	Reuters R F1	Acc	Pr.	R	AAPD	F1	Acc
	BERT 𝑏𝑎𝑠𝑒 +𝐺𝐶𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	91.46	90.70	91.08	87.01 80.41 68.85	74.18 43.1
	BERT 𝑏𝑎𝑠𝑒 +𝐼𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	93.61	91.24	92.41	87.94	83.0	70.59 76.29 44.9
	DistilBERT 𝑏𝑎𝑠𝑒 +𝐺𝐶𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	91.61	90.17	90.88	86.52 74.50 73.27	73.88 40.0
	DistilBERT 𝑏𝑎𝑠𝑒 +𝐼𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	92.48	92.04	92.26	87.45 82.59 70.34	75.97 44.0
	RoBERTa 𝑏𝑎𝑠𝑒 +𝐺𝐶𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	91.21	89.56	90.38	86.32 74.12 71.83	72.96 39.9
	RoBERTa 𝑏𝑎𝑠𝑒 +𝐼𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	93.18	90.57	91.86	87.78 81.95 70.14	75.58 44.2
	DeBERTa 𝑏𝑎𝑠𝑒 +𝐺𝐶𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	92.74	90.73	91.72	87.41 75.21 71.95	73.55 40.4
	DeBERTa 𝑏𝑎𝑠𝑒 +𝐼𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	93.74 91.56 92.64 88.27 82.82 70.26	76.02 44.0
	BERT 𝑏𝑎𝑠𝑒 +𝑁 𝐻𝐴 𝑜𝑟𝑎𝑐𝑙𝑒	92.55 92.55 92.55 92.45 74.06 74.06	74.06 51.8
	DistilBERT 𝑏𝑎𝑠𝑒 +𝑁 𝐻𝐴 𝑜𝑟𝑎𝑐𝑙𝑒 91.99	91.99	91.99	91.95 75.09 75.09 75.09 54.7
	RoBERTa 𝑏𝑎𝑠𝑒 +𝑁 𝐻𝐴 𝑜𝑟𝑎𝑐𝑙𝑒	91.69	91.69	91.69	91.45 73.36 73.36	73.36 51.5
	DeBERTa 𝑏𝑎𝑠𝑒 +𝑁 𝐻𝐴 𝑜𝑟𝑎𝑐𝑙𝑒	92.31	92.31	92.31	92.41 73.48 73.48	73.48 52.7

Table 4

 4 Scores for the test dataset from MFHAD.

	Models	Pr.	R	MFHAD F1	Acc
		Baselines
	Decision Tree	45.29	42.03	43.60	36.84
	Bagging	74.99	38.86	51.19	35.49
	Random Forest	92.80	32.85	48.52	34.43
	GradientBoost	58.10	40.53	47.75	30.42
	SVM	84.79	46.08	59.71	43.33
	CamemBERT	71.27	58.85	64.47	49.87
	FlauBERT	70.10	62.21	65.92	52.44
	Thresholding without any specific architecture (GCT and IT)
	CamemBERT+𝐺𝐶𝑇	70.14	59.31 ↑	64.27	49.74
	CamemBERT+𝐼𝑇	70.41	59.71 ↑ 64.62 ↑ 50.00 ↑
	FlauBERT+𝐺𝐶𝑇	71.30 ↑	61.40	66.14 ↑ 52.74 ↑
	FlauBERT+𝐼𝑇	65.04	64.52 ↑	64.78	50.71
	Transformer Architecture Adaptation (NHA et TL)
	CamemBERT+𝑁 𝐻𝐴	60.33	53.60	56.76	48.43
	CamemBERT+𝑇 𝐿	71.45 ↑	58.18	64.14	50.97 ↑
	FlauBERT+𝑁 𝐻𝐴	61.64	54.81	58.02	49.16
	FlauBERT+𝑇 𝐿	70.37 ↑	60.95	65.32	51.05

Table 5

 5 Scores of the oracle approaches for MFHAD.

	Models	Pr.	MFHAD R F1	Acc
	CamemBERT+𝐺𝐶𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	71.37	59.68	65.01	51.09
	CamemBERT+𝐼𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	81.05 56.70	66.73	52.24
	FlauBERT+𝐺𝐶𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	75.44	59.12	66.29	52.87
	FlauBERT+𝐼𝑇 𝑜𝑟𝑎𝑐𝑙𝑒	80.54	59.45 68.40 54.55
	CamemBERT+𝑁 𝐻𝐴 𝑜𝑟𝑎𝑐𝑙𝑒 66.85	66.85	66.85	62.83
	FlauBERT+𝑁 𝐻𝐴 𝑜𝑟𝑎𝑐𝑙𝑒	67.49 67.49 67.49 63.37

All datasets can be downloaded here: https://zenodo.org/record/6344750#.Yio1YH_MK-r

The extraction of these abstracts was made using the tool made available by HAL: https://api.archives-ouvertes.fr/ docs/search

https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection