Jean-Yves Chaufray

F Gonzalez-Galindo

M A Lopez-Valverde

F Forget

M Chaffin

Jean-Loup Bertaux

Eric Quémerais

J T Clarke

François Leblanc

Franck Montmessin

Jean-Yves Chaufray

J-Y Chaufray

J-L Bertaux

F Montmes- Sin

R Yelle

N Schneider

Hydrogen escape at Mars : Comparisons from MGCM-LMD simulations and observations from Mars-Express/SPICAM

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

1) Introduction:

The recent observations of an unexpected large seasonal variation of the Martian hydrogen exosphere is important to characterize the water escape along the Martian history. The Martian Global Circulation Model from Laboratoire de Météorologie Dynamique (MGCM-LMD) is a 3D model able to simulate the full hydrogen cycle from the water at the surface to the hydrogen density at the exobase. The hydrogen density is extended to the exosphere using the Liouville's equation. In this presentation, we will present recent simulations of the Martian hydrogen density, taking into account: (1) the observed dayto-day solar and dust load variability during the recent Martian years, and (2) the 3D hydrogen corona at Lyman-alpha performed with a 3D radiative transfer model. These simulations will be compared to observations performed by several instruments (Mars Express/SPICAM

2) Model:

The LMD-MGCM model is a model of the martian atmosphere from the surface to the exobase, including major species in the Martian atmosphere [1,2]. The seasonal variations at different solar activity have been studied by Chaufray et al. [3,4]. The scenario corresponding to the solar activity and dust during the Martian years 28, 29) [5] are considered in this presentation. No mission measured directly the hydrogen density. The hydrgeon density is derived from UV hydrogen lines, mainly the optically thick Lyman-alpha emission. To directly compare our simulations to the observations performed by SPICAM on Mars Express, we use a 3D Radiative transfer model and derive the brightness profile corresponing to the different geometry of observations. Because the 3D radiative transfer model is time expensive, we consider only monthly average hydrogen distribution and focus on the study of the large timescale (season and year to year) (Fig. 1). 3) Observations SPICAM on Mars Express has observed the Martian H corona during several years. [6,7]. We here focus on two martian years (28, 29) At this time, there was an anomalous behavior of the image intensifier characterized by sporadic and spurious changes in the gain during one observation [8] responsible of the spodaric increase of the Lyman-alpha brightness in the vertical profiles presented below.

4) Results

During the Martian year 28, the model is in reasonable agreement with the observations for 160° < Ls < 180° (a,b) but underestimate the brightness for 180 < Ls < 330°, especially near southern summer at Ls = 250°-300°. The agreement becomes better after Ls = 330°. An arbitrary background of 500 R was used for all the observations. Small disagreeemnt could be improved with a better estimate of it, but not the difference near Ls=250°-300°, more likely due to a missing source of hydrogen in the model at this season. For the early Martian year 29 (Ls < 60°), the agreement with the observations is reasonable near equinox though the model slightly overestimates the brightness near Ls = 30-50°.

5) Conclusion

We simulate the monthly average Martian hydrogen corona for Martian years 28, 29. We compare the simulated Lyman-alpha brightness to the observed brightness. The model gives reasonable agreement near equinox : Ls ~ 0° for MY 29 and Ls ~ 180° for MY 28. The model fails to reproduce the large observed brightness between Ls = 210 to 330°, due to a large increase of the hydrogen density confirmed by plasma observations [9]. A possible explanation of this large difference is due to the presence of a larger source of atomic hydrogen at this season not included in the model. The detection of water vapour at higher altitudes (~80 km) than expected at Ls ~ 250° by MEX [10], followed by its photodissociation could increase the abundance of hydrogen in the Martian exosphere and the hydrogen escape flux [11]. Therefore the measurement of the extension of the water vapor in the mesosphere/thermosphere of Mars and its seasonal variation should be needed to confirm this assumption.

Fig. 1

 1 Fig. 1 Simulated escape flux for the Martian years 28 and 29 by the MGCM-LMD model. The escape flux computed with the monthly average hydrogen density and temperature at the exobase are also indicated.

Fig. 2

 2 Fig. 2 Examples of comparison between the model and the SPICAM observations during the Martian year 28.

Fig. 3

 3 Fig. 3 Examples of comparison between the model and the SPICAM observations during the Martian year 29.

Scientific Workshop: "From Mars Express to ExoMars" 27-28 February 2018, ESAC Madrid, Spain Workshop organized by ESA-ESAC & IAA-CSIC, with the support of the ESAC Faculty and the EU H2020 project UPWARDS

Acknowledgements:

This project was partly funded by the Programme National de Planetologie and Programme National Soleil Terre and by the Centre National d'Etudes Spatiales. This work has been partially funded by the European Union Horizon 2020 Programme (H2020 Compet -08-2014) under grant agreement UPWARDS-633127.


Lufei Liu

Xu Sun
email: xu.sun@etu.u-paris.fr

François Veltz

Kim Gerdes

Annotating Discursive Roles of Sentences in Patent Descriptions

Patent descriptions are a crucial component of patent applications, as they are key to understanding the invention and play a significant role in securing patent grants. While discursive analyses have been undertaken for scientific articles, they have not been as thoroughly explored for patent descriptions, despite the increasing importance of Intellectual Property and the constant rise of the number of patent applications. In this study, we propose an annotation scheme containing 16 classes that allows categorizing each sentence in patent descriptions according to their discursive roles. We publish an experimental human-annotated corpus of 16 patent descriptions and analyze challenges that may be encountered in such work. This work can be base for an automated annotation and thus contribute to enriching linguistic resources in the patent domain.

Introduction

Patent applications represent a first step in obtaining exclusive rights over an invention. Analyzing these documents enables inventors to understand technological trends, avoid potential litigation, and assess the competition. The patent description, a substantial part of the patent application, provides detailed information about the invention. Although specific segments have to be present in order to have the application accepted, the information can be provided without any pre-imposed order. Patent descriptions should be well organized in order to communicate the invention's technical details, advantages, and scope with more clarity. This, in turn, helps patent examiners to review applications more efficiently, reduces the likelihood of misinterpretation or ambiguity, and increases the chances of obtaining a patent grant with a well-defined scope of protection.

The contributions of our work are as follows:

• Introducing an annotation scheme based on and adapted for the discursive structure of patent descriptions.

• By focusing on patent descriptions, we aim to contribute to a better understanding of these documents' linguistic characteristics and structures, which have received little attention in patent-related research.

• Set a ground for future patent description analysis, for example, develop automatic methods to apply the annotation to large scale patent datasets. This can contribute to the detection of abnormal patent description and study the patent writing style according to different assignees.

Related work

The analysis of document structure allows for a deeper understanding of the author's thought process and facilitates the retrieval of specific information within the document. Many previous studies have focused on the analysis of technical documents, particularly scientific papers. For example, [START_REF] Fisas | On the discoursive structure of computer graphics research papers[END_REF] created a multi-layered annotated corpus of 40 scientific papers in the domain of Computer Graphics, with each sentence annotated according to its rhetorical role. [START_REF] Dasigi | Experiment segmentation in scientific discourse as clause-level structured prediction using recurrent neural networks[END_REF]) created a corpus by manually annotating 75 articles in the domain of intercellular cancer pathways. Each article was divided into clauses which are classified into one of the following categories: Goal, Fact, Result, Hypothesis, Method, Problem, Implication, None. This corpus was used to develop a discourse tagger for claim extraction and evidence fragment detection [START_REF] Li | Scientific discourse tagging for evidence extraction[END_REF]. Patent applications are another type of technical document that has garnered researchers' interest. A patent application usually consists of various components, including a title, abstract, description, one or more claims, drawings, and classification information. Current patent text analysis mainly focuses on claims or abstracts to improve claim readability [START_REF] Ferraro | Segmentation of patent claims for improving their readability[END_REF][START_REF] Okamoto | Applying information extraction for patent structure analysis[END_REF], such as using them to build an engineering knowledge graph [START_REF] Siddharth | Engineering Knowledge Graph From Patent Database[END_REF][START_REF] Zuo | Patent-kg: Patent knowledge graph extraction for engineering design[END_REF], or to aid in patent classification models [START_REF] Lee | Patent classification by fine-tuning bert language model[END_REF]. However, the less-structured and much longer patent descriptions, an essential part of understanding patents, receive little attention. To our knowledge, only [START_REF] Nakamitsu | Analyzing the structure of u.s. patents using patent families[END_REF] analyzes the structure of patent descriptions, but they focus solely on four content types: Field, Problem, Solution, and Effect. Nevertheless, patent descriptions contain much richer information, including technical term definitions in context, advantages of the invention, and drawing descriptions. Exploring the structure of patent descriptions can be used to acquire patent writing skills -for humans and machines. Writing a good patent description not only requires an understanding of legal knowledge, but also requires expertise in relevant technical fields. Furthermore, mastering the structure of a patent description enables the extraction of reliable features, which may be useful for patent text modeling, specific to domains, assignees, and legal goals of the patent.

The goal of this study is to apply the discourse structure analysis, a common practice in scientific papers, to the whole patent descriptions while considering their unique writing style. To achieve this, we design an annotation scheme to label each sentence in the description according to its discursive role.

Annotation scheme

The patent description is typically divided into several sections. Under the Patent Cooperation Treaty (PCT), the description contains mainly (WIPO, 2022): Title of invention, Technical field, Background art, Summary of invention, Brief description of drawings, and Description of embodiments. The field section specifies the technical domain to which the invention belongs. The background section discusses the prior art related to the invention, identifies previously encountered problems, and explains how the proposed invention may offer solutions to one or more of these issues. The summary section highlights the key features and advantages of the invention. This is commonly followed by a section that provides a concise overview of the content present in each illustrative drawing, if they are included. Lastly, the Detailed Descrip-tion section should encompass greater detail of the claimed invention by way of examples (embodiments), describing figures in detail and defining little known or specially formulated technical terms when necessary, to further clarify the structure and functioning of the invention.

Based on the above essential elements recognized in a patent description and with the assistance of a patent attorney, we initially designed a set of 12 labels corresponding to the bold elements above. key features and examples (embodiments) were combined, represented by the label Embodiment, as the invention is usually described by introducing its features. Following this, two annotators collaboratively annotated two patent descriptions and identified a need to distinguish between the Advantage and Problem labels, to clarify whether these pertain to the invention itself or to existing technologies. In addition, we added the label Other for sentences that don't fall into any of the established 14 categories.

This set of 15 labels was applied by two annotators on the first test dataset of 8 patent applications. We noticed that the Section title label does not cover all kinds of titles within a patent description, since different applicants may introduce subsections with additional titles according to their writing styles. The annotators found it difficult to decide on the class of non-standard titles: Are they section titles or part of the embodiment?

To remedy this difficulty, a 16 th label Section subtitle was added following the annotation of the first dataset. This new label also allows for an elementary encoding of the scope of the main sections, whenever they are indicated by section titles. It is this set of 16 labels that has reached consensus and is deemed operational and representative for annotating the discursive role of sentences within patent descriptions.

Annotation tags

Below is a brief summary of the labels defined for annotating patent descriptions. Additionally, a more detailed annotation guideline has been prepared, offering further explanations, examples, and counterexamples for each label. The guideline was made available to annotators to facilitate their understanding and ensure consistent application of the labeling criteria throughout the annotation process.

Patent title

The title of the patent application.

Example: VEHICLE SPEAKER DISPOSITION STRUCTURE

Section title

The title of each main section of a patent description. Example: BACKGROUND ART

Section subtitle

The title of sub-sections inside the main sections of a patent description, if any. Example: Stability studies

Technical field

Sentences determining the technical scope of the invention. These sentences specify to which field the invention relates and are usually carried out in one single paragraph.

Example: This application relates to the field of electronic materials and component technologies, and in particular, to an embedded substrate and a method for manufacturing an embedded substrate.

Reference

Sentences introducing the state-of-the-art or presenting the context to reach the invention, including related patents or publications, previous techniques, or general knowledge. Example: In Patent Document 1, the acoustic transducer is disposed in the fender located near a front corner of a vehicle cabin, and sound is reproduced from the vicinity of the front corner toward the vehicle cabin.

Reference problem

Sentences stating the disadvantages of prior arts or indicating the technical problem that the invention is designed to solve. Example: It can be learned that according to the existing embedded component packaging process, laser generated when the drill holes 104 are drilled damage the chip.

Reference advantage

Sentences explaining the advantage or quality of the prior arts or known technologies. In the example below, the first sentence provides context and the second sentence should be tagged as Reference advantage.

Example: In Patent Document 1, the acoustic transducer is disposed in the fender located near a front corner of a vehicle cabin, and sound is reproduced from the vicinity of the front corner toward the vehicle cabin. By employing such a structure, an improvement in the reproduction efficiency, of high-quality sound including a low range, with a wide range of directivity in a plan view, is expected.

Embodiment

Sentences describing physical instances or variations of the invention, explaining necessary ways to achieve the desired outcome. These sentences serve to demonstrate the flexibility and applicability of the invention in various contexts. (We keep the original reference numerals such as "104" in the text.) Example: That is, a metal boss maybe disposed on each pad, and then embedded packaging (including drilling, conductive material filling, conductive layer disposing, and the like) is performed on the chip.

Invention advantage

Sentences providing the advantage, quality, or improvement brought about by the invention. Example: The technique disclosed herein achieves both an improvement in the reproduction efficiency of the speaker and a reduction in the noise caused in the vehicle body by the sound generated by the speaker.

Invention problem

Sentences highlighting drawbacks or problems that the invention may cause. Example: In short, since the speaker box 10 needs to have sealing properties in view of improving the reproduction efficiency of the sound including the low range, the drainage performance tends to deteriorate.

Figure description

Some patent applications contain figures which give a visual representation of the invention in the form of drawings, diagrams, or flowcharts. The Figure description tag is assigned to sentences that provide detailed explanations of figures, which usually contain reference numerals of the invention's components. These sentences should allow readers to navigate and understand the various depicted elements.

Example: As seen from Figure 3, for example, in its closed position, the door 20 is received within the recess 10 of the housing base 12 and its lower face 24 lies generally flush with the lower surface 26 of the lip 14.

Definition

The explanation or clarification of technical terms, which could be a specifically formulated term. Context-specific explanations of which are given within the scope of the patent. Example: As used herein, the term "cofactor" refers to a non-protein compound that operates in combination with a ketoredutase enzyme.

Rephrased claim

Sentences repeating portions of claims with nonsubstantive modifications, i.e., without incorporating additional content words that may alter the scope of the claims. Example: A harness system for a power drive unit is disclosed. In various embodiments, the harness system includes an electrical cable having a first end and a second end, a plurality of cover members positioned along a length of the cable and a spring member positioned adjacent the plurality of cover members along the length of the cable. original claim: A harness system for a power drive unit, comprising: an electrical cable (580) having a first end and a second end; a plurality of cover members (581) positioned along a length of the electrical cable; and a spring member (582) positioned adjacent the plurality of cover members along the length of the electrical cable.

Juridical template

Standardized phrases or sentences, which can be used regardless of the patent content. They serve specific purposes, such as facilitating transitions between sections of the description, and extend or narrow down the scope of the claims. Example: The foregoing implementations of the present invention do not constitute a limitation on the protection scope of the present invention.

Technical template

Sentences giving the comprehensive usage of a technical term by providing its closely related synonyms or hyponyms. Example: The first and the second plastic material may also be selected from a third group comprising a High Density Polyethylene, Low Density Polyethylene, Polyethylene, Terephthalate, Polyvinyl Chloride, Polycarbonate, Polypropylene, Polystyrene, Fluorine Treated, Post Con-sumer Resin, K-Resin, Bio-plastic, or combinations thereof.

Other

Sentences belonging to none of the previous categories or contains ambiguity. The following example demonstrates a typical OCR problem that has grouped all the elements of a table of contents together. Given that each title appeared in the corresponding subsection, this sequence is considered as Other to avoid introducing noise into the data.

Corpus annotation

Corpus preparation

To build our annotation corpus, we use patent applications published by the European Patent Office (EPO). These patent applications are classified using the Cooperative Patent Classification (CPC) system, which comprises eight domains (Table 1). We randomly selected 2 applications per domain and divided them into two datasets, each containing one document per CPC class. In cases where an application is classified under multiple domains, we only considered the first one (the primary CPC label). The aim of this dataset separation is to verify whether the inter-annotator agreement remains consistent across the two datasets.

We extracted the description section from each patent application, each description is then segmented into sentences before being annotated. We use scispacy 1 combining with special rules to perform sentence splitting. For example, patent claims, which are usually extremely long sentences separated by semicolons, could be copied into description. In order to balance the size of each sentence, semicolons are also considered as ending punctuation. We chose to stay on the sentence level because delving into a finer-grained level would require not only knowledge in linguistics but also expertise in various technical domains. For instance, the following sentence, which we simply classify as being of type Figure description, could be broken down into sub-sequences that detail the interaction between elements described in the figure: CPU 16 controls first conveyance section 21 to move conveyance pallet 40 to the loading position, and controls multi-joint robot 24 so that robot-side attachment section 27 grips pallet-side attachment section 42 below conveyance pallet 40 (refer to fig. 8).

• Controller action: CPU 16 controls first conveyance section 21; and controls multi-joint robot 24

• Result of the action: to move conveyance pallet 40 to the loading position; so that robotside attachment section 27 grips pallet-side attachment section 42

• Location of the action: below conveyance pallet 40

• Figure reference: refer to fig. 8 As an exploratory study and considering the number of defined labels, we decided to remain at the coarse-grained level. Table 1 shows the number of sentences in each document for both datasets of the corpus.

CPC

Annotation process

In order to measure the consistency across the annotation process, the annotation is conducted in two sessions. For each session, a pair of annotators (with a computational linguists background) independently annotate the same documents. Only one tag is allowed for each sentence. Discussion before the annotation is permitted, in order to allow both annotators to become familiar with the general structure of patent description. During the annotation, discussions are not allowed, instead, annotators have access to the context and any other information necessary for understanding the sentence to be annotated. After the first session, a collective review of the annotation guideline is conducted in order to complete the guideline with newly encountered examples.

Annotation agreement

We employed the pairwise Cohen's kappa to measure inter-annotator agreement. Table 2 shows the scores for each class within each corpus. As explained in Section 3, the label Section subtitle was added after the annotation of the first dataset. This modification has contributed to the perfect agreement concerning the labels associated with titles. It is worth noting that the agreement is relatively low for some labels, which is due to their imbalanced distribution in the corpus (as shown in Figure 1). The matrices in Figure 1 Overall, we observe an improvement in the agreement between the two annotation sessions, particularly for Figure description and Embodiment. The label Other denotes sentences that appear ambiguous or do not belong to any class. Few sentences received this label from both annotators, which shows that our label set is comprehensive enough to cover the entirety of sentences in a patent description.

Labels

As we can see from IAA score and Figure 1, apart from labels concerning titles, the categories that receive higher agreement are those less dependent on the annotator's interpretation of sentence meanings, such as Technical field, Rephrased claim, and Juridical template.

Compared to the work of [START_REF] Nakamitsu | Analyzing the structure of u.s. patents using patent families[END_REF], we have expanded the label set with the intention to encompass the entirety of patent description content, rather than merely focusing on specific parts thereof. We attempted to create an exemplary dataset manually, with the objective of identifying relevant labels and establishing a reli-able sample for future data augmentation. On the whole, our annotation achieved an IAA score of 0.69 following a revision on the first dataset. This result aligns with the performance of similar annotation work [START_REF] Fisas | On the discoursive structure of computer graphics research papers[END_REF], who obtained an IAA score of 0.6567 across eight categories.

However, although we improved the agreement score for some categories, it is worth noting that the level of agreement remains relatively low for some categories, despite post-annotation discussions following the first session. The following section gives examples of pairs of labels that are frequently confused by annotators.

Disagreement analysis

Based on the annotation results, we noticed that certain pairs of labels are often confused. We attempted to analyze the reason for this confusion.

Reference VS Embodiment

The challenge associated with this pair of labels lies in distinguishing whether the subject of the sentence concerns the prior art or the applicant's invention. The reason is that, in the section describing embodiments of the invention, the description of the invention can be mixed with the explanation of prior art. This is particularly the case for patents in the Chemistry; metallurgy domain. In these patents, the disclosure of detailed experimentation is required, which often leads to numerous references when existing components or methods are needed for the experiments. Consequently, the description of the invention example becomes intertwined with that of the prior art, complicating the annotators' comprehension, especially when they lack domain-specific expertise.

Example: Fluorescent nucleotide incorporation by DNA polymerase. As described in the abovereferenced PNAS publication by Braslavsky et al., DNA polymerase may be employed to image sequence information in a single DNA template as its complementary strand is synthesized. The nucleotides are inserted sequentially;

In this example, it is challenging to determine whether the sentence underlined is part of the publication cited in the previous sentence or merely a step in the Fluorescent nucleotide incorporation by DNA polymerase experiment. We have chosen Embodiment as the label for this sentence because the following context explains the experiments related to the invention itself and not the reference.

Reference advantage VS Reference problem

Distinguishing between advantages and problems can be challenging, especially when purely critical or, conversely, commendatory terms are missing.

Example: Furthermore, in regular operation, an auxiliary circuit may be energized and connected to a junction by way of a second current interrupting element. Electrical power can thus be provided from DFIG to auxiliary components, with the electrical power from main power transformer being converted to the appropriate voltage by auxiliary transformer. However, during maintenance operations, the DFIG may be shut down, and the main power circuit may be isolated from the power grid.

In this example, it's difficult to tell if DFIG may be shut down and main power circuit may be isolated are positive characteristics even though the two preceding sentences have provided context. With the help of the following context, we understood that it indeed represents an advantage, especially it was mentioned that this can help to reducing the risk of electrocution during maintenance operations. We thereby decided to annotate it as Reference advantage.

Reference problem VS Invention advantage

Sometimes, the information is presented as a dual statement which requires annotators to interpret the context and infer the intended meaning.

Example: This entails the need to exert a high torque by the motor to carry out the movement quickly.

In this example, it can be inferred that the sentence implies a drawback of the current technique. However, in a patent description, such a sentence exists only to indicates that the mentioned problem will be rectified through the invention, thereby expressing an advantage of the latter. To solve the ambiguity, we added a rule to our annotation guideline, explicitly stating that for such dual statements, the sentence will be annotated as Invention advantage because the presented problem would be solved by the invention.

Embodiment VS Figure description

Despite the introduction of additional specifications after the first annotation session regarding the distinction between Embodiment and Figure description, with a particular emphasis on the functional aspect of the former and the visual aspect of the latter, the differentiation remains challenging. This is because the description of an embodiment often refers to components drawn in the figures.

Example: In atmospheric pressure plasma-generating device 10, processing gas composed only of an inert gas is supplied from first connecting passage 130 to reaction chamber 100 through the inside of holders 72 and 74 of holding member 20.

In this example, the technical terms followed by numbers indicate that these are important components of the invention and that they are illustrated in the drawings. However, the sentence only describes how the processing gas is supplied, which is not depicted in the drawings. Considering the process is not shown in the drawings, we decided to label it as Embodiment and we clarified that it is possible to refer to drawings attached to the patent applications in case of indecision between Embodiment and Figure description.

Embodiment VS Invention advantage

Using comparatives when describing an invention may not always clearly indicate an improvement of the invention. The confusion often caused by insufficient technical knowledge in the respective domain.

Example: It is therefore known that the particle size distribution computed using the profile data about the coke 30 shows larger particle size distribution than the actual particle size distribution.

In this example, it's difficult to decide if larger particle size distribution is an improvement achieved by using the profile data about the coke 30. Thus, in cases where we cannot be certain that the presented feature is an advantage, we annotate it as Embodiment in order to avoid introducing errors.

The analysis of disagreement sheds light on the challenges involved in annotating the discursive roles of sentences in patent descriptions, which are not only related to language complexity but also to individual manner of expression.

Conclusion

In this paper, we have proposed an annotation scheme adapted to the specific writing style of patent applications. As an exploratory work, we defined a set of 16 labels to categorize each sentence in a patent description according to their discursive roles. The initial results show that such work is feasible, since strong agreement is achieved for most categories. However, challenges remain. Considering the aforementioned difficulties, we propose the following improvement to the future work: allow multi-labeling for ambiguous sentences or consider implementing a multi-layer annotation scheme. In the first level, include classes corresponding only to the five common sections of a patent description, followed by additional specific categories in subsequent layers as necessary.

In conclusion, our annotation work is an ongoing process. We plan to expand our dataset once the relevant labels have been established, and to employ active learning methods to streamline the annotation process. We believe that a such linguistic resource in the patent domain could contribute to enhancing the accuracy of tasks such as patent classification, patent novelty detection, patent information retrieval, and, most central to Qatent, computer-assisted patent drafting.

Limitations

Our sample dataset contains only 16 randomly selected documents, which might not be sufficient to contribute to classification model training. Additionally, our work could benefit from having a lead annotator to supervise the annotation process. This would help reduce the time spent on correcting annotation errors and ensure adherence to the annotation guideline.

  Example: I. OverviewII. Description of StepsA. Tissue PreparationB. Distribution of DNA moleculesC. Detection and Quantification1. Digital PCR Methods2. Bead emulsion PCR3. Microfluidic Dilution with PCR4. Single molecule detection and/or sequencingD.

Figure 1 :

 1 Figure 1: The annotation by class across the two datasets. The matrices show, for each annotator, occurrence of labels assigned by each annotator for each dataset. We can observe that the two datasets are mainly composed of Rephrased claim, Figure description, Embodiment, and Invention advantage. In the dataset 2, Reference also represent a significant portion.

Table 1 :

 1 Number of sentences for each domain in the corpus (and the total number of tokens as well as the average tokens per sentence for information).

		dataset 1 dataset 2
	Human necessities (A)	393	228
	Performing operations; transporting (B)	217	101
	Chemistry; metallurgy (C)	349	681
	Textiles; paper (D)	307	106
	Fixed constructions (E)	364	102
	Mechanical engineering;
	lighting; heating;
	weapons;	109	245
	blasting engines
	or pumps (F)
	Physics (G)	224	284
	Electricity (H)	193	221
	Total (nb tokens)	62203	56886
	Average tokens per sentence	28.9	28.9
	Total (nb sentences)	2156	1968

Table 2 :

 2 IAA for each label in each dataset of the corpus. The Cohen's kappa score for the entire dataset is used instead of the mean of scores for each category due to the imbalanced distribution of labels.

		dataset 1 dataseet 2
	Patent title	1.00	1.00
	Section title	0.96	1.00
	Section subtitle		1.00
	Technical field	0.93	0.92
	Definition	0.59	0.46
	Reference	0.77	0.64
	Reference_problem	0.67	0.70
	Reference_advantage	0.45	0.11
	Rephrased claim	0.76	0.84
	Figure description	0.47	0.75
	Embodiment	0.47	0.61
	Invention_advantage	0.64	0.70
	Invention_problem	0.58	0.09
	Juridical template	0.70	0.79
	Technical template	0.22	0.43
	Other	0.19	0.55
	kappa	0.56	0.69

https://allenai.github.io/scispacy/

Acknowledgments

We would like to express our profound thanks to You Zuo for taking the time to read our paper, offering thoughtful comments, and providing enriching ideas. Finally, We are grateful to the anonymous reviewers and for their detailed and helpful feedback.

Ethics Statement

All data used in this study were collected from publicly available sources, with no infringement of intellectual property rights or privacy.


Shuyi Yang
email: shuyi.yang@unito.it

Dino Ienco
email: dino.ienco@inrae.fr

Roberto Esposito
email: roberto.esposito@unito.it

Ruggero G Pensa
email: ruggero.pensa@unito.it

ESA : A Generic Framework for Semi-supervised Inductive Learning

Keywords: semi-supervised learning, graph-based algorithms, inductive methods

A new inductive framework for graph-based semi-supervised classification is proposed.

• The proposed framework combines semi-supervised autoencoders and graph-based pseudo-labeling.

• Two variants based on confidence-aware label propagation and graph attention networks are proposed.

• The framework outperforms state-of-the-art competitors on data with very small amounts of labeled examples.

Introduction

Prediction is one of the most important outcomes of any machine learning algorithm. It supports many time-consuming and tedious tasks once performed by humans and, although classification performances are not yet (always) comparable to humans', especially in noisy domains [START_REF] Geirhos | Generalisation in humans and deep neural networks[END_REF], it is one of the most important tasks deployed throughout the industry. Prediction accuracy strongly depends on the amounts and quality of labeled instances and, unfortunately, labeling is a cost-intensive manual activity requiring time, money, and expertise. Often, labeling (or annotation) tasks are outsourced to external companies (e.g., Amazon Mechanical Turk) that recruit users on the web for doing the job. However, in some sensitive areas, such as X-ray images interpretation [START_REF] Rajaraman | Iteratively pruned deep learning ensembles for COVID-19 detection in chest x-rays[END_REF], experts cannot be replaced by the wisdom of the crowd. Hence, labeling often turns out to be unaffordable for many organizations and, consequently, only small amounts of labeled instances are available for training. This is a major issue, especially for models requiring large amounts of training data such those implementing deep learning architectures.

Semi-supervised learning aims at mitigating the above-mentioned problem by leveraging the so-called smoothness and cluster assumptions: if two data instances are close to each other or belong to the same cluster in the input distribution, then they are likely to belong to the same class [START_REF]Semi-Supervised Learning[END_REF]. Instead of using the few labeled instances to train a classifier, the idea is to propagate the information to other "close" and unlabeled data instances. If labels are of good quality, and clusters are well separated, semi-supervised approaches usually outperform their supervised counterparts.

Among the different algorithmic solutions proposed in literature, graphbased models constitute one of the main families of semi-supervised techniques [START_REF] Van Engelen | A survey on semi-supervised learning[END_REF]. Graph-based methods leverage the manifold assumption: the graphs, typically nearest neighbor graphs built upon the local similarity between data points, provide a lower-dimensional representation of the highdimensional input data. Graph-based semi-supervised learning algorithms typically involve two steps: in the first one, a nearest neighbor graph is constructed using all data points to capture the manifold of the data. Classification is then performed by propagating the information from labeled to unlabeled samples along the edges of the graph.

Unfortunately, graph-based methods are transductive [START_REF] Van Engelen | A survey on semi-supervised learning[END_REF], i.e., they do not construct any classification model and the prediction is limited to exactly those data instances that are already available during the training phase. Therefore, graph-based methods are unable to classify new data examples, unless they are trained again on the augmented dataset. A second limitation concerns the construction of the graph: in general, this phase is completely unsupervised even though, for some instances, labels are available. When the cluster assumption is not completely satisfied, this could lead to poor prediction results. Although some solutions exist [START_REF] Quiles | Label propagation through neuronal synchrony[END_REF][START_REF] Ienco | Enhancing graph-based semisupervised learning via knowledge-aware data embedding[END_REF], they only work in transductive settings. In this paper, we present a novel graph-based semisupervised framework, ESA , that improves in the areas mentioned above: it takes into account the information carried out by labeled instances during the graph construction and is designed to work properly in inductive settings.

Our approach -sketched graphically in Fig. 1 -first constructs a new representation using a semi-supervised autoencoder that takes all labeled and unlabeled training data as input. The representation learnt by the semisupervised autoencoder, for both labeled and unlabeled training data, are then processed by a graph-based semi-supervised algorithm that propagates the label information from labeled to unlabeled data instances. This procedure provides pseudo-labels for the set of unlabeled instances. To this purpose, we will experiment with two different approaches, leading to two variants of our framework: the first, ESA LP , is based on a graph-based label propagation algorithm that exploits homophily and heterophily, as well as the confidence of the inference results [START_REF] Yamaguchi | CAMLP: confidence-aware modulated label propagation[END_REF]; the second one, ESA GAT , exploits a graph convolutional neural network with masked self-attention layers [START_REF] Velickovic | Graph attention networks[END_REF]. All training instances (labeled and unlabeled with pseudo-labels) are then used to train a classification model, which can perform prediction for new unseen examples as well. We show that our approach outperforms state-of-the-art approaches (including ladder networks [START_REF] Rasmus | Semisupervised learning with ladder networks[END_REF] and ICT [START_REF] Verma | Interpolation consistency training for semi-supervised learning[END_REF]), even with extremely small amounts of labeled instances.

The remainder of the paper is organized as follows: a brief related literature review is reported in Section 2; the general framework is introduced in Section 3; the two variants of the label propagation steps are described in Section 3.1 and 3.2 respectively; the results of our experimental validation are discussed in Section 4; finally, we draw conclusions in Section 5.

Related work

Semi-supervised learning algorithms can be characterized, depending on whether they build a general model or not for the underlying data, between transductive and inductive methods.

Transductive methods are mostly based on graphs, with the (dis)similarity between nodes coded as the weight of the graph edges. In these methods, once the graph has been constructed, an inference method is applied to make predictions on unlabeled nodes. For instance, in Confidence-Aware Modulated Label Propagation (CAMLP) [START_REF] Yamaguchi | CAMLP: confidence-aware modulated label propagation[END_REF], an iterative algorithm computes the probability distribution of each node over the classes by combining the prior belief and prediction confidence: a higher number of signals from the neighborhood implies higher confidence. In addition, signals from the neighbors are adjusted to manage both homophily and heterophily networks. Other semi-supervised transductive algorithms that are either confidence-aware or can handle homophily and heterophily networks exist: for instance, belief propagation (BP) [START_REF] Günnemann | Linearized and singlepass belief propagation[END_REF][START_REF] Sen | Collective classification in network data[END_REF] can handle homophily and heterophily networks but it does not include the confidence component, SocNL [START_REF] Yamaguchi | Socnl: Bayesian label propagation with confidence[END_REF], DGR [START_REF] Fang | Confidence-aware graph regularization with heterogeneous pairwise features[END_REF], TACO [START_REF] Orbach | Graph-based transduction with confidence[END_REF], and ReLISH [START_REF] Gong | Relish: Reliable label inference via smoothness hypothesis[END_REF] are confidence-aware but can handle only homophily networks.

Very different approaches have been proposed for inductive semi-supervised methods. Early research focused on wrapper [START_REF] Yarowsky | Unsupervised word sense disambiguation rivaling supervised methods[END_REF][START_REF] Alché-Buc | Semi-supervised marginboost[END_REF][START_REF] Bennett | Exploiting unlabeled data in ensemble methods[END_REF][START_REF] Mallapragada | Semiboost: Boosting for semi-supervised learning[END_REF][START_REF] Wang | A new analysis of co-training[END_REF][START_REF] Zhou | Semi-supervised learning by disagreement[END_REF] and unsupervised [START_REF] Goldberg | Multi-manifold semi-supervised learning[END_REF] preprocessing methods [START_REF] Van Engelen | A survey on semi-supervised learning[END_REF]. Unsupervised preprocessing is often used to provide a better initialization for the training parameters, effectively moving the weights of a neural network closer to the convergence region. For example, in a deep belief network, multiple restricted Boltzmann machines are stacked and trained with unlabeled data [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF], then an output layer is added to the network structure and the entire network is trained on labeled data.

Autoencoders (AE) [START_REF] Hinton | Autoencoders, minimum description length and helmholtz free energy[END_REF], instead, are trained to extract the latent representation of each instance with the goal of feeding the learning algorithm with representations that are conducive to learning better classifiers. Even though an autoencoder is designed to perform an unsupervised task, it can be extended to include a supervised component: in a semi-supervised autoencoder (SSAE) [START_REF] Gogna | Semi supervised autoencoder[END_REF] a further prediction layer is attached to the bottleneck layer. SSAE are then trained with a loss that combines a reconstruction component and a classification component. A significant number of other frameworks also propose to combine unsupervised and supervised components and this approach is neither totally new: in fact, semi-supervised extensions for support vector machines [START_REF] Vapnik | Statistical learning theory wiley[END_REF][START_REF] Chapelle | Optimization techniques for semi-supervised support vector machines[END_REF], probabilistic models based on Gaussian Processes [START_REF] Lawrence | Semi-supervised learning via gaussian processes[END_REF] and density regularization methods [START_REF] Corduneanu | On information regularization[END_REF] have already been developed in the past. What these models have in common is that they all try to maximize the margin by relying on a low-density assumption (i.e., in these models, the decision boundary is assumed to lay through regions of the instance space with low-density data).

In recent years, a particular formulation of the cluster assumption, called smoothness assumption (stating that close elements of an instance space should have similar target variables) has led to the development of a new set of inductive semi-supervised learning algorithms based on the idea of perturbation: predictions of instances that differ only for a small perturbation noise should be similar. A neural network based on this principle is the ladder network (LN) [START_REF] Rasmus | Semisupervised learning with ladder networks[END_REF]. Although, its structure has some similarities with the one of a SSAE, it differs from SSAE in the training process and the cost function. In ladder networks, the network tries simultaneously to reconstruct the input and to denoise the representations built at every layer of the network. In this model, then, the additional denoising autoencoders that are added layer-wise build models for the latent representations, which in turn, help the learning process in both supervised and semi-supervised settings.

Other perturbation-based approaches, instead of injecting the noise in the data, perturb the model itself [START_REF] Bachman | Learning with pseudo-ensembles[END_REF][START_REF] Laine | Temporal ensembling for semi-supervised learning[END_REF]. Another class of semi-supervised learning methods perturbs data by combining feature vectors linearly [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF][START_REF] Berthelot | Mixmatch: A holistic approach to semi-supervised learning[END_REF]. In [START_REF] Verma | Interpolation consistency training for semi-supervised learning[END_REF], Verma et et al. presents Interpolation Consistency Training (ICT), which adopts the mean-teacher [START_REF] Tarvainen | Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] method in addition to feature vector mixup: during the training, the student network is optimized to get correct predictions on labeled samples (by reducing the supervised loss) and to preserve consistency over unlabeled instances. Its prediction on a linear combination of unlabeled samples is compared to the linear combination of predictions made by the teacher network on the same samples (via the consistency loss).

In our work, differently from the reported literature, we propose a framework to tackle the inductive graph-based semi-supervised classification task by combining an ensemble of semi-supervised autoencoders with a graphbased pseudo-labeling process in order to feed a final classifier both with originally labeled instances and pseudo-labeled ones. The pseudo-labeling process provides labels for unlabeled instances and it can be implemented with two different strategies: the first one is based on the confidence-aware label propagation [START_REF] Yamaguchi | CAMLP: confidence-aware modulated label propagation[END_REF], while the second one adopts graph attention networks [START_REF] Velickovic | Graph attention networks[END_REF] to perform convolution operations on graph nodes representing both labeled and unlabeled instances.

X l {x l , y l } X u {x u } X t {x t , y t } Φ {Φ(x l ), y l } {Φ(x u )} {Φ(x u ),

Inductive graph-based semi-supervised learning

In a semi-supervised learning setting, in addition to labeled instances, unlabeled ones are introduced as part of available data during the training phase: let X l ∈ R n l ×f be the matrix of n l labeled samples each with f predictors and y l be the corresponding labels, then a supplementary matrix X u ∈ R nu×f representing n u unlabeled instances is also provided without the corresponding y u labels. Generally, the number n l of labeled instances is limited and much smaller than the number n u of unlabeled instances.

Our framework aims to provide an inductive semi-supervised learning algorithm by leveraging graph-based semi-supervised learning in order to augment the amount of labeled instances to train a supervised classifier.

As shown in Figure 1, our framework consists of different parts: embedding computation, pseudo-labeling of unlabeled instances, and classification. In the embedding computation part, we train an ensemble of neural networks to extract a latent representation for each instance. These representations are used to build a graph over labeled and unlabeled instances so that a graph-based model can be employed to provide a pseudo-label for each unlabeled instance. Finally, labeled instances and pseudo-labeled ones are both used to train a supervised classification model.

In order to extract the data embeddings, an Ensemble of Semi-supervised Autoencoders (ESA) [START_REF] Gogna | Semi supervised autoencoder[END_REF][START_REF] Ienco | Enhancing graph-based semisupervised learning via knowledge-aware data embedding[END_REF] is trained on both labeled and unlabeled data.

An autoencoder (AE) is a neural network that employs a series of fully connected layers which constitute the encoder E : R f → R e . An encoder transforms each instance in a e-dimensional latent representation. When the dimension of the representation layer is smaller than the dimension of the input layer, the representation layer is called bottleneck layer and the autoencoder is defined under-complete [START_REF] Goodfellow | Deep learning book[END_REF]. The bottleneck layer is attached to a second series of fully connected layers (decoder layers) in order to reconstruct the original instance D : R e → R f . Given an instance x r of X l or X u we expect that x r is similar to its reconstructed version via the encoding/decoding process D(E(x r )) and therefore the bottleneck layer provides a latent representation containing all relevant information needed to recover the input despite having much fewer dimensions. Besides the reconstruction task, if we add a classification layer CL : R e → R |C| to the bottleneck one, we can also train the network to learn a bottleneck representation tailored to the custom classification task we are undertaking. The output of the classification layer is a probability distribution over all possible labels in the label set C.

The loss function we use to learn the internal parameters of the SSAE is a combination of reconstruction and classification loss. More formally:

L SSAE = L AE + λL CL (1)
where

L AE = 1 n l + n u x i ∈X l ∪Xu ||x i -D(E(x i |θ E )|θ D )|| 2 , (2)

Algorithm 1 SSAE -Training

Require: X l : set of labeled instances, y l : labels of X l , X u : set of unlabeled instances, f : number of input features, size hidden : size of the hidden layers of Update θ E , θ D and θ CL by descending the gradient of the total loss

∇ θ E ,θ D ,θCL {L AE + λL CL } 6: counter++ 7: end while 8: return θ E , θ D , θ CL L CL = - 1 n l x i ∈X l |C| c=1 y lic • log(CL(E(x i |θ E )|θ CL ) c ), (3)
and θ E , θ D and θ CL are respectively the set of parameters of the encoder, decoder and classification layer, y lic is the c-th element of the i-th row of y l , CL(•) c is the c-th element of the output vector of CL and λ is a parameter that controls the importance of the classification loss. See Algorithm 1 for the training details.

In our architecture (see Figure 2) the encoder has an input layer followed by other two hidden layers; the decoder has one hidden layer of the same size of the first hidden layer of the encoder and an output layer. The size of the input layer, the output layer and the classification layer are respectively fixed to f , f and |C|, while size hidden and size bottleneck (respectively, the size of the hidden layer and that of the bottleneck one) can be varied. In order to get diverse and multi-resolution representations, similarly as in [START_REF] Ienco | Enhancing graph-based semisupervised learning via knowledge-aware data embedding[END_REF], we train K independent SSAEs, each with the sizes of the layers extracted randomly from the intervals f 2 ≤ size hidden < f and f 4 ≤ size hidden < f 2 . Once the ensemble is trained we obtain the new representations Φ(X l ), Φ(X u ) of X l and X u by concatenating the embeddings of these K SSAEs: Given the latent representations, a kNN graph structure can be derived from the data points of X l ∪ X u : embedding representations are nodes and two of them can be considered connected if both of them belong to the top k nearest neighbors of each other, respectively.

Φ(•) = || k=K k=1 E k (•|θ E k ) (4) x1 x2 xn … … … … … ESA x x x L ESA = n i=1 [L AE (x, xi ) + λL CL (y, ỹi )] ỹ1 ỹ2 ỹn X l {x l , y l } X u {x u }
At this point we perform graph-based pseudo-labeling (GBPL) to assign pseudo class labels to unlabeled instances. To this purpose, any graph-based semi-supervised learning algorithm (GBSSL) can be applied to infer the labels of the unlabeled portion of data ỹu by propagating the class information from the labeled data y l over the graph constructed on the embeddings. Successively, a supervised classifier (SC) can be trained leveraging the union of the labeled data (Φ(X l ), y l ) with the pseudo-labeled one (Φ(X u ), ỹu ) as training set. In prediction, we first compute the latent representation of unseen data Φ(X t ) with the trained ESA, then we make predictions with the supervised classifier SC. It is worth pointing out that during the entire process, the transductive GBSSL process is used only during the training phase to provide pseudo-labels of the unlabeled data (as in wrapper methods) in order to help the supervised classifier to generalize better. Therefore, our approach, hereinafter referred as ESA , is inductive1 . See Algorithm 3 and Algorithm 4 for the training and prediction details of the overall framework. In the next two sections, we present two variants adopting different strategies to perform pseudo-labeling based on different graph-based semi-supervised learning approaches.

Pseudo-labeling based on confidence-aware label propagation

In this section we introduce the first variant of our framework for semisupervised learning (see Figure 3). The adopted strategy consists in instantiating the graph-based pseudo-labeling (GBPL) part with a confidence-aware label propagation algorithm working on both homophily and heterophily networks [START_REF] Yamaguchi | CAMLP: confidence-aware modulated label propagation[END_REF]. In the following we provide the details of this strategy, which we name ESA LP .

Given the adjacency matrix A, ESA LP computes the probability distribution over the classes as the solution of:

F iϕ = 1 Z i y iϕ + β j A ij s ji (ϕ) (5)

Algorithm 3 ESA -Training

Require: X l : set of labeled instances, y l : labels of X l , X u : set of unlabeled instances, f : number of input features, |C|: number of classes, epochs: number of training epochs, K: number of SSAEs to be trained, SC: a classifier to be trained Ensure: K SSAEs' encoder weights, parameters of the classifier SC 1: Learn the weights θ E1 , θ E2 , ...,

θ E K of ESA (Algorithm 2) 2: For each instance x i ∈ X l ∪ X u compute the bottleneck representation of each SSAE E 1 (x i |θ E1 ), E 2 (x i |θ E2 ), ..., E K (x i |θ E K ) 3: Stack the K embedding representations of each x i by forming one unique array of real numbers Φ(x i ) = || k=K k=1 E k (x i |θ E k ) 4:
For each Φ(x i ) compute its 20-nearest neighbors in the euclidean space 5: Construct an adjacency matrix A where the (i, j)-element is equal to 1 if and only if Φ(X i ) is in the top 20-nearest neighbors of Φ(X j ) and vice versa 6: Feed a GBSSL algorithm with A, (Φ(X l ), y l ) and Φ(X u ) in order to infer the labels of unlabeled instances ỹu 7: Train a classifier SC(•|θ SC ) with the labeled embeddings (Φ(X l ), y l ) and the pseudolabeled ones (Φ(X u ), ỹu ) 8: where F iϕ is the probability that i-th instance has label ϕ, Z i is a normalization term, y iϕ is the prior belief of i-th instance having label ϕ, 0 < β represents the importance of the neighborhood's influence, A ij is the i, j entry of the adjacency matrix and s ji (ϕ) represents how intense the node j believes that the node i has class ϕ. More formally:

return θ E1 , θ E2 , ..., θ E K , θ SC Algorithm 4 ESA -Prediction Require: θ E1 , θ E2 , ...,
s ji (ϕ) = l F jl H lϕ ( 6
)
where H is the modulation matrix. If H lϕ is low then class l has a low correlation with the class ϕ, on the contrary, if it is high these two classes have a strong correlation. On homophily networks, H is the identity matrix, while on heterophily networks it can be designed empirically. In our experiments we assume that the graph obtained by the embeddings of ESA is a homophily network. We can rewrite the Equation 5 in matrix form and in an iterative way:

Adj matrix

   A 11 . . . A 1n . . . . . . . . . A n1 . . . A nn    F iϕ = 1 Z i   y iϕ + β j A ij s jϕ (ϕ)   num iter    F r 11 . . . . . . F r 1ϕ . . . F r iϕ . . . . . . F r n1 . . . . . . F r nϕ    = F r F numiter {Φ(x u ), ỹu } {Φ(x l ), y l } GBPL-LP
F r+1 = Z -1 (Y + βAF r H) (7)
where Z = I +βD and D is the node degrees diagonal matrix. Once obtained the adjacency matrix A of labeled and unlabeled instances, as described in the previous section, we initialize F 0 as a (n l +n u )×|C| matrix of zeros. Then we apply the Equation 7 num iter times to obtain F num iter , which represents the probability distributions of the instances over the classes. From F num iter we extract only the predictions of X u and keep the original labels for X l . They are then used to feed a classifier SC.

Since the algorithm of our generic framework is already detailed previously, in Algorithm 5 we only report the details of the label propagation phase (Step 6 of Algorithm 3).

Pseudo-labeling based on graph attention networks

For the second variant, we consider a completely different approach leveraging the convolution operation. Many recent works employ the spectral representation of the graph and perform convolution by computing the eigendecomposition of the Laplacian associated to the graph [START_REF] Estrach | Spectral networks and deep locally connected networks on graphs[END_REF][START_REF] Henaff | Deep convolutional networks on graphstructured data[END_REF][START_REF] Defferrard | Convolutional neural networks on graphs with fast localized spectral filtering[END_REF][START_REF] Kipf | Semi-supervised classification with graph convolutional networks[END_REF]. However, since the Laplacian eigenbasis depends on the graph structure, Algorithm 5 GBPL -LP Require: X l : the set of n l labeled instances, y l : labels of X l , X u : the set of n u unlabeled instances, |C|: number of classes, H: modulation matrix, A: adjacency matrix, β: importance of the neighborhood's influence, D: the diagonal matrix of node degrees, num iter : number of iterations Ensure: probability distribution of instances of X l ∪ X u over the classes 1: Initialize F 0 as a (n l + n u ) × |C| matrix with zeros as its entries 2: Initialize Y as a (n l + n u ) × |C| matrix where

Y ij = 1 if the i-th instances of X l ∪ X u has j-th label, Y ij = 0 if the i-th instances of X l ∪ X u is labeled but has not j-th label, Y ij = 1 |C| if the i-th instances of X l ∪ X u is unlabeled 3: Initialize Z = I + βD 4: r = 0 5: while r < num iter do 6: compute F r+1 = z -1 (Y + βAF r H) 7:
r + + 8: end while 9: return F numiter these methods, once trained, can hardly be applied to graphs with different structures. On the other hand, non-spectral methods perform feature extraction from the neighborhood nodes while still maintaining shared weights [START_REF] Duvenaud | Convolutional networks on graphs for learning molecular fingerprints[END_REF][START_REF] Atwood | Diffusion-convolutional neural networks[END_REF][START_REF] Niepert | Learning convolutional neural networks for graphs[END_REF][START_REF] Monti | Geometric deep learning on graphs and manifolds using mixture model cnns[END_REF]. For instance, in [START_REF] Hamilton | Inductive representation learning on large graphs[END_REF] the weights are optimized to output similar representations for nearby nodes and once learned, they can be applied to graphs with different structures. In this models, each neighbor contributes equally to a node's embedded representation. Graph attention networks (GAT) [START_REF] Velickovic | Graph attention networks[END_REF], instead, overcome this limitation by adding a multihead attention mechanism to each embedding layer so that nodes of the same neighborhood can assume different importance. Moreover, the weights of the feature transformation and the multi-head attention are shared and this method can be either transductive or inductive. Hence, GATs are good candidates for our pseudo-labeling process, as they are able to capture different levels of importance of features of neighborhood nodes in the kNN graph built upon the embeddings computed by ESA. We call this strategy ESA GAT and provide the details below (a graphical representation is given in Figure 4).

Given the set of nodes, each represented by a b-dimensional real numbers array obtained from the ESA embedding process or as a result of a previous convolutional layer, we can compute the self-attention on nodes as where h i , h j ∈ R b are the embeddings of instance i and j, W is a b × b shared linear transformation matrix, and a : R b × R b → R is the attentional mechanism consisting in a feedforward layer with weights ℵ and LeakyReLU activation [START_REF] Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF]. Each e ij is computed only for connected nodes (masked attention) so that the graph structure is embedded into the coefficients. The attention coefficients are then normalized using the softmax function:

e ij = a(W h i , W h j |ℵ) (8)
α ij = exp e ij ι∈N (i) exp e iι (9)
with N (i) representing the nodes connected to the node i. The new representation of the node i through the attention layer is then computed as

h i = σ   j∈N (i) α ij W h j   (10)
with σ a non linear transformation. As in ESA, we can concatenate the outputs of different independent attention layers in order to employ a multi-head

Algorithm 6 GBPL -GAT

Require: X l : the set of n l labeled instances, y l : labels of X l , X u : the set of n u unlabeled instances, A: adjacency matrix, num heads : number of heades, h 1 , ..., h n l +nu : ESA embeddings of labeled and unlabeled instances, epochs: number of epochs, a: attentional mechanism, σ, σ : non linear functions Ensure: weights of the GAT 1: Initialize random matrices W 1 , W 2 , ..., W num heads , ℵ 1 , ℵ 2 , .., ℵ num heads , W , ℵ 2: counter = 1 3: while counter ≤ epochs do 4: = 1 5:

while ≤ num heads do 6:

For each couple of instances (i, j) compute e ij = a(W h i , W h j |ℵ ) 7:

For each couple of instances (i, j) compute the attention coefficients αij = exp e ij ι∈N (i) exp e iι

8:

For each instance i compute its new representation h i = σ j∈N (i) α ij W h j 9:

+ + 10:

end while 11:

For each instance i concatenate its representations

h i = || k=num heads =1 h i 12:
For each couple of instances (i, j) compute e ij = a(W h i , W h j |ℵ ) 13:

For each couple of instances (i, j) compute the attention coefficients α ij = exp e ij ι∈N (i) exp e iι

14:

For each instance i compute its new representation h i = σ j∈N (i) α ij W h j 15:

Compute the negative log-likehood loss for the labeled instances L(h i , y l ) 16:

Update W 1 , W 2 , ..., W num heads , ℵ 1 , ℵ 2 , ..., ℵ numheads , W , ℵ by descending the gradient ∇L 17: end while 18: return W 1 , W 2 , ..., W num heads , ℵ 1 , ℵ 2 , ..., ℵ num heads , W , ℵ attention mechanism. To do that, once the ESA embeddings are obtained, we can apply the Equation 10 multiple times: from the point of view of the neural network structure, the mechanism is realized by adding additional convolutional layers, each with its own weights to be trained and the number of nodes of the last layer should be equal to the number of classes. See Algorithm 6 for the training details of a 2-layers GAT.

Even though there are some similarities between the attention mechanism in ESA GAT and the modulation matrix in ESA LP , it is worth noting that while ESA GAT operates on the latent representation of the nodes, ESA LP regards the relationships among classes.

Experiments

To assess the behavior of our framework under different settings and configurations, we conduct four experiments.

In the first experiment, we compare the two versions of our framework: ESA LP and ESA GAT . Our objective is twofold: on the one hand, we want to measure the impact of varying the different components of our framework, on the other hand, we aim at identifying the best combination(s). To this purpose, both ESA LP and ESA GAT variants of our framework are combined with four different classifiers: random forest (RF), multilayer perceptron (MLP), support vector machine (SVM) and discriminative ridge machine (DRM [START_REF] Peng | Discriminative ridge machine: A classifier for highdimensional data or imbalanced data[END_REF]). We name all possible combinations as follows: ESA LP +RF, ESA LP +MLP, ESA LP +SVM, ESA LP +DRM, ESA GAT +RF, ESA GAT +MLP, ESA GAT +SVM and ESA GAT +DRM.

In the second experiment, we compare the two best performing configurations of our framework to four well-known supervised methods (RF, MLP, SVM and DRM) and two recent state-of-the-art semi-supervised approaches: interpolation consistency training (ICT) [START_REF] Verma | Interpolation consistency training for semi-supervised learning[END_REF] and ladder networks (LN) [START_REF] Rasmus | Semisupervised learning with ladder networks[END_REF]. In our experiments, ICT is coupled with fully connected layers, instead of convolutional layers, to deal with tabular shaped dataset, as we do not specifically target image datasets. Our objective is to understand both the strengths and the limitations of our framework and to study under which conditions it outperforms (or is outperformed by) the other approaches.

In the third experiment, we conduct an ablation study for inspecting the contribution of the different components of our framework on the prediction accuracy. We compare the performances of three different inductive semisupervised methods: i) a semi-supervised autoencoder (SSAE), ii) a semisupervised autoencoder ensemble coupled with a classifier (ESA+RF), iii) our complete framework (including the graph-based semi-supervised component).

In the last experiment, we compare the two best performing configurations of our framework to two state-of-the-art transductive semi-supervised approaches: structured graph learning with multiple kernel (SGMK) [START_REF] Kang | Structured graph learning for clustering and semi-supervised classification[END_REF] and robust graph construction (RGC) [START_REF] Kang | Robust graph learning from noisy data[END_REF].

The remainder of the section is organized as follows: we first present the datasets used in our experiments (Section 4.1); then, in Section 4.2 we give the details of our experimental protocol; finally, we report and discuss the results of our four experiments in Section 4.3.

Dataset

Datasets

As shown in Table 1, thirteen publicly available real-world classification datasets, encompassing a wide variety of application scenarios, have been considered in our experiments. They exhibits different sizes (from 178 to 70 000 instances) and dimensionality (from 12 to 1087 features). The datasets include 4 well-known image datasets (USPS [START_REF] Hull | A database for handwritten text recognition research[END_REF], MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], FMNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF], COIL20 [START_REF] Nene | Columbia object image library[END_REF]). However, to be fair when comparing with competing approaches, we do not consider any convolutional filter and handle the images as numeric vectors.

Experiments settings

Each dataset is randomly split into three parts: labeled instances (p% of the dataset), unlabeled instances ((70-p)% of the dataset), and test instances (30% of the dataset). The random split is stratified so that each dataset maintains the same proportion of labels as in the original datasets. Every supervised model (RF, MLP, SVM, DRM) is trained on labeled instances only and evaluated on test instances, while all semi-supervised models are trained both on labeled instances and unlabeled ones and evaluated on the test instances.

During the experiments we vary the percentage p of labeled instances to study how the performances change when the portion of labeled instances increases in both supervised and semi-supervised models. More in details, p takes one of the following values: 0.1, 0.5, 1, 2, 3, 4 or 5. Therefore, in the most adverse situation, the percentage of labeled instances is only 0.1% of the entire dataset while the remaining 69.9% is unlabeled. In any case, we consider at least one labeled instance per class.

To obtain more robust performance indicators, for each combination of dataset, percentage p and model, we evaluate 25 different random splits as described above and then take the average performances. The model is retrained for each of the 25 different splits and new predictions are made on every different test set. For any given percentage p and random split, all models are trained on the same labeled/unlabeled set, and compared on the same test set. As performance index, we consider the micro-averaged F1score computed on the test set.

In Table 2 we report the structures and the hyper-parameters of the models used in our experiments. We have not performed a grid search to find the best combination of hyper-parameters for each model since this would be infeasible for the huge number of experiments required. Instead, when available, we used the default values reported in the original papers of models. When defaults values were unavailable, we made an educated guess based on the considerations found in the papers introducing the algorithms. It is worth pointing out that, in any case, the performances are always measured on an independent test set.

Results and discussion

In all evaluations of the experiments, we compute detailed performance results for each dataset (Figures 6, 7, 8, 9) and a summary of the results (Figure 5). The latter is obtained as follows: for a given percentage p of labeled instances, we compute the average ranks across all the dataset for each algorithm, according to the micro-averaged F1 score, and then we plot them for increasing values of p% (lower ranks are plotted higher in the pictures and mean better performances). This allows us to obtain an overall picture of the relative performances of all competitors considered in our study. In the following, we present and discuss the results for each experiment.

Experiment 1: analysis of the variants

This experiment aims to study different variants of our framework and select two of them as reference for further analysis (Experiments 2 and 3). To this purpose, we compare both ESA LP and ESA GAT variants with a different choice of final classifier (RF, MLP, SVM and DRM). The detailed results are provided in Figure 6 while a summary is presented in Figure 5(a). It can be noticed that, in general, when both variants are coupled with RF, on average, they outperform all other combinations, for all percentages of labeled samples. From the plots it is also evident that ESA GAT +RF performs slightly better than ESA LP +RF when p ≥ 1. Apart from the variants with DRM (which perform poorly on some datasets, probably due to their particularities), it is worth noting that, the overall behavior is similar for the remaining algorithms and all datasets (see Figure 6) except for small datasets (SONAR and WINE) and for COIL20 (the dataset with the largest number of classes). This first batch of experiments suggests that ESA LP +RF and ESA GAT +RF are the best combination for our technique on the chosen datasets, and we shall use these two models for the comparative analysis in the next experiments.

Experiment 2: comparative analysis

In this analysis the objective is to compare our framework to other supervised and semi-supervised methods. From the obtained results (Figure 5(b)), it emerges that, for every percentage of labeled examples, on average, the two variants of our framework outperform all other methods, including the four fully supervised classifiers considered in this study (RF, MLP, SVM and DRM). It is worth noting that, in contrast, the two competing semisupervised methods (ICT and LN) are not able to outperform the supervised competitors with the same consistency. The micro-averaged F1 score of ICT is below the one of RF, for any given value of p. Ladder networks (LN) are ranked third with less than 2% of labeled samples, but RF is still competitive w.r.t. LN despite the fact that it does not take advantage of unlabeled instances. From the Figure [START_REF] Yamaguchi | CAMLP: confidence-aware modulated label propagation[END_REF], we can point out that LN performs well on some datasets (ANTIVIRUS, LANDSAT, PARKINSON, MNIST, FMNIST, USPS), but achieves relatively low accuracy in all other cases. Furthermore, both ICT and LN are ineffective on COIL20: the reason for such deceiving performances could be in the high number of classes of such a dataset. In this case, our methods are the only two capable to exceed the 0.7 threshold of micro-averaged F1 score. Finally, it is worth pointing out that, not surprisingly, when the number of labeled instances increases, the differences between semi-supervised methods and fully supervised ones decrease.

Experiment 3: ablation study

With this evaluation we measure the impact of every individual component of our framework in an inductive setting. In this case, we focus on ESA LP +RF (the variant that considers label propagation and random forest) and analyze RF (the supervised counterpart), ESA+RF (an ensemble of semi-supervised autoencoders coupled with RF), and a simple semisupervised autoencoder. Figure 5(c) summarizes the results by presenting the average rank of each competitor. From this plot, it turns out that using have still comparable overall performances and only the overall framework (in this case, ESA LP +RF) enable us to substantially outperform RF. Hence, providing pseudo-labels for unlabeled instances via the graph-based semisupervised learning step is crucial. Notice that, in this study, we can not consider the graph-based semi-supervised learning step alone, since it is a transductive approach and it does not provide a model that can be deployed on the test data.

Experiment 4: comparative analysis w.r.t. transductive methods

In the last experiment the objective is to compare our inductive framework to recent semi-supervised transductive methods. It is worth pointing out that, while the transductive setting has potential liabilities in terms of applicability, it has advantages in the possibility of leveraging more information than its inductive counterparts (since it can leverage the test set distribution when propagating the labels). As we shall see, while our method is at a disadvantage here, it works quite well nonetheless and we believe that this comparison further clarifies the strength and the weaknesses of the proposed approach.

We compare the variants ESA LP +RF and ESA GAT +RF to the structured graph learning with multiple kernel (SGMK) [START_REF] Kang | Structured graph learning for clustering and semi-supervised classification[END_REF] and to the robust graph construction (RGC) [START_REF] Kang | Robust graph learning from noisy data[END_REF]. Due to computational limitations, the experiments are performed only on a subset of the available datasets, namely: ANTIVIRUS, SONAR, PARKINSON and WINE. From the results shown in Figure 9, our methods outperform the competing ones on ANTIVIRUS and PARKINSON. However, in SONAR and in WINE, RGC has better performances. Overall, with the Figure 5(d), we can conclude that, within our experimental settings, ESA LP +RF and ESA GAT +RF are able to reach competitive performances compared to SOTA transductive methods (i.e. RGC), and, in some cases, even outperform them (i.e. SGMK).

Conclusion

In this paper, we have presented a new inductive semi-supervised learning framework that take the most of two successful approaches: semi-supervised autoencoders and graph-based semi-supervised learning. While the former supports the generation of new data representations improved by labeled instances, the latter spread the label information to unlabeled instances in the new representation space. Thanks to an extensive experimental study, we

Figure 2 :

 2 Figure 2: A graphical representation of ESA (ensemble of semi-supervised autoencoders) step of our framework.

Figure 3 :

 3 Figure 3: A graphical representation of the label propagation strategy of ESA LP .

Figure 4 :

 4 Figure 4: a graphical representation of the graph attention mechanism of ESA GAT .

1 SSAE- 2

 12 layers of sizes 64 and 32; -default values for the rest. SVM -Kernel: Nystroem approximation of an RBF kernel with gamma = 0.2, and number of components = 300 (as in the example given by the authors), -Optimization: Stochastic Gradient Descent; -default values for the rest. DRM -the DRM algorithm based on PPA (DRM-PPA [47]) is used in our experiments; -linear DRM-PPA is used for the datasets MNIST and FMNIST; -alpha = 0.1; -beta = 0.hidden layers of sizes 64 and 32; -autoencoder batch size = 64; -semi-supervised autoencoder batch size = 8; -epochs = 100 ESA 10 SSAEs each one having: -the first hidden layer of size in the interval f 2 , f -the second hidden layer of size in the interval f 4 , f 4 ; these intervals are reduced to f 5 , f 2.5 and f 10 , layers of sizes 128, 64, 32 and 32; -noise standard deviation = 0.3 -denoising cost = [1000, 10, 0.1, 0.1, 0.1, 0.1] -epochs = 100 -default values for the rest.ICT-mixup consistency = 10; -consistency rampup starts = 0; -consistency rampup ends = 100; -mixup sup alpha = 0.2; mixup usup alpha=0.2; -mixup hidden = False; -num mix layer = 2 -in the base model convolutional layers are substituted by full dense layers of sizes 64 and 32; -default values for the rest. for the rest.

Figure 5 :

 5 Figure 5: Average rank according to micro-averaged F1-measure in the experiments.

Figure 6 :

 6 Figure 6: Micro-averaged F1-measure for increasing percentage of labeled instances and for different datasets and variants of our algorithm (Experiment 1).

Figure 7 :

 7 Figure 7: Micro-averaged F1-measure for increasing percentage of labeled instances and for different datasets and competitors (Experiment 2).

  SSAE, size bottleneck : size of the bottleneck layer, |C|: number of classes, epochs: number of training epochs Ensure: weights of the encoder, decoder and classification layers θ E , θ D , θ CL 1: Initialize a SSAE with • the encoder layers of sizes [f, size hidden , size bottleneck ]

• decoder layers of sizes [size hidden , f ]

• the classification layer of size |C| 2: counter = 1 3: while counter ≤ epochs do 4:

Update θ E and θ D by descending the gradient of the autoencoder loss ∇ θ E ,θ D L AE 5:

  E1 , θ E2 , ..., θ E K where || is the stack (concatenation) operator and E k and θ E k are respectively the encoder of the k-th SSAE and its weights. See Algorithm 2 for the training details of ESA.

		(Algorithm 1)
	6:	k + +
	7: end while
	8: return θ

Algorithm 2 ESA -Training Require: X l : set of labeled instances, y l : labels of X l , X u : set of unlabeled instances, f : number of input features, |C|: number of classes, epochs: number of training epochs, K: number of SSAEs to be trained Ensure: K SSAEs' encoder weights 1: k = 1 2: while k ≤ K do 3: sample an integer size hidden in the range f 2 , f 4: sample an integer size bottleneck in the range f 4 , f 2 5: initialize and train the k-th SSAE in order to learn its weights θ E k , θ D k , θ CL k

  θ E K : learned weights of ESA, θ SC : learned parameters of the classifier SC, x new : a new instance Ensure: class prediction of x new 1: Given the weights of ESA (Algorithm 2), compute the embedding representation as Φ(x new ) = || k=K k=1 E k (x new |θ E k ) 2: return SC(Φ(X new )|θ SC )

Table 1 :

 1 List of datasets used during our experiments. This table contains the number of instances and the number of classes of each of them.

		# instances # classes # features
	ANTIVIRUS [54]	373	2	513
	COIL20 [53]	1440	20	576
	FMNIST [52]	70000	10	784
	LANDSAT [54]	6435	6	36
	MADELON [54]	2600	2	500
	MALWARE [54]	6248	2	1087
	MNIST [51]	70000	10	784
	PARKINSON [54]	756	2	753
	SONAR [54]	208	2	60
	SPAMBASE [54]	4601	2	57
	USPS [50]	9298	10	256
	WAVEFORM [54]	5000	3	40
	WINE [54]	178	3	12

Table 2 :

 2 The structures and the hyperparameters of the models used in the experiments. The hyper-parameters are maintained unchanged for the models used in different parts of our experiments: for instance, RF, which is a standalone classifier, is also used as part of ESA LP +RF and ESA GAT +RF.

	RF	ESALP+RF	ESAGAT+RF	ICT	ESA+RF
	MLP	ESALP+MLP	ESAGAT+MLP	LN	SGMK
	SVM	ESALP+SVM	ESAGAT+SVM	SSAE	RGC
	DRM	ESALP+DRM	ESAGAT+DRM
	0.1 0.5 1.0 Percentage of labeled instances (%) 2.0 3.0 4.0	5.0

In the paper, we will use ESA to indicate the block consisting of the ensemble of semi-supervised autoencoders in our framework, while ESA stands for the overall semisupervised learning approach.

The authors wish to thank Mattia Cerrato for stimulating discussions and its critical review of the paper.

counterpart trained only on labeled instances, and also outperforms stateof-the-art semi-supervised competitors. Additionally, we have proposed two variants of our framework: the first one, based on a graph-based confidenceaware label propagation approach, is more effective with very few labeled instances; the second one, leveraging graph attention networks, outperforms the first one when more labeled instances are available.

As possible future work, we will introduce a variant of our framework especially tailored for image data by including convolutional layers in the first stages of the pipeline. Moreover, we will design an end-to-end inductive semi-supervised deep neural network process to strengthen the interaction among the different components of the proposed framework.