Thomas Dany

Clément

Pierre Daivy

Paris : Paul

Charlotte

Johan Léa

Vincent W Henri

Within IRT SystemX, I remember all the doctoral students, especially Julien, Maria, Pascal, Clarisse, with whom we established a doctoral representation association, subsequently taken over perfectly by Tjark, Marina, Emmanuel. I also think of Simo, Kevin, Adrien, and Natkamon. Thanks to everyone for the kindness within this group and the Exploding Kittens and Among Us games during the lockdowns. Thanks also to Selma, Ibrahima, Junkai, and all MPO project partners for insightful scientiÿc discussions

Cÿÿÿÿÿÿÿ

The Fourth Industrial Revolution, often referred to as Industry . , is currently shaping our era with a new phase in the transformation of the industrial sector. Built on the digital revolution (the Third Industrial Revolution), Industry ÿ.ÿ is characterized by a fusion of technologies that blur the lines between the physical, digital, and biological spheres, leading to a systemic transformation of the entire value chain of the manufacturing sector (Schwab ÿÿÿÿ). At the heart of Industry ÿ.ÿ lies a series of technological advancements, represented in Figure ÿ.ÿ , such as the Internet of Things (IoT), Cyber-Physical Systems, Cloud Computing, Digital Twins, and Artiÿcial Intelligence (AI). These advancements have resulted in a paradigm shift from traditional, linear manufacturing processes to complex, integrated systems where machinery and equipment can communicate and cooperate with each other and with humans in real time. This concept is commonly referred to as the smart factory (B. Chen et al. ÿÿÿÿ). To illustrate, in a smart factory environment, an assembly line robot is capable of autonomously communicating with other machinery to adjust its production pace based on real-time demand or even preemptively order replacement parts when a failure is anticipated. Similarly, smart logistics systems in Industry ÿ.ÿ can dynamically reroute shipments based on real-time conditions, reducing delays and enhancing eÿciency. ÿ

One of the most signiÿcant transformations in Industry ÿ.ÿ is the shift towards predictive maintenance. Traditional maintenance policies based on estimated lifetimes are giving way to systems that can predict failures and schedule maintenance in real time. Predictive maintenance, driven by real-time data from various sensors and machines, aims to prevent unplanned downtime, enhance eÿciency, and increase the overall life span of the machinery (Mobley ÿÿÿÿ). The emergence of predictive maintenance systems has been fueled by the massive availability of data from interconnected and intelligent automation systems that Industry ÿ.ÿ puts at the center of global production, particularly through the integration of smart sensors aiming to build global control systems such as Supervisory Control And Data Acquisition (SCADA). The induced challenge and opportunity -that motivate this thesis -lies in exploiting the vast data acquired by these sensors for fault monitoring, diagnosis, and more generally predictive maintenance.

Central to these developments is the role of data, that is the cornerstone of Industry ÿ.ÿ. The interconnected sensors and devices nowadays generate an unprecedented amount of data that embodies a rich source of insight into the functioning, performance, and potential anomalies within the systems, but that are collected from various sources and in various forms, leading to the emergence of complex and often heterogeneous data sources. Consider the example of an automated production line in a smart factory. As part of its operation, it continuously generates multiple types of data through various sensors and systems. For instance, vibration sensors on the machinery provide data on the machine's physical state, indicating its stability or any unusual shaking that could signify a potential issue. Temperature sensors provide another form of data, oÿering insight into the machine's thermal conditions, and cameras installed in strategic locations capture real-time visual data of the machine's operation and the production process. Simultaneously, the system also generates textual data in the form of operational logs or maintenance reports that provide contextual information about the machine's operational status, historical issues, or previous performed maintenances.

The key advantage of considering this multimodal data is that it oÿers a comprehensive and detailed perspective of the system's state. Each modality, whether it be sensor readings, images, or textual reports, captures diÿerent facets of the system's condition, thereby enriching the information available for fault diagnosis or other predictive maintenance tasks. For instance, while real-time sensor data could provide immediate insights about the system's performance parameters such as temperature or vibration, image data could reveal physical anomalies or damages, and textual reports could oÿer context or detailed accounts of previous incidents or interventions. Besides, multimodal data introduces the capability for cross-veriÿcation of faults. An anomaly detected in one modality can be cross-checked and conÿrmed with information from another modality, adding a layer of redundancy and increasing the conÿdence of the fault detection process. This becomes particularly crucial when dealing with complex or subtle faults that may not be readily discernible in a single data modality, but become evident when multiple data types are analyzed collectively. But even more signiÿcantly, the integration of diÿerent modalities allows us to identify faults that might remain hidden when considering each modality in isolation. A minor anomaly in one modality, seemingly insigniÿcant on its own, could be the critical piece of the puzzle when viewed in the context of other modalities, leading to the identiÿcation of a potential fault. Introduction perspective, enriches the information available for predictive maintenance. Particularly, textual data, encapsulating the richness of human language and expert insight, provides subtleties and nuanced patterns that sensor or image data may overlook. Through the integration of these diverse modalities, we aim to design a global representation of a system's state, enhancing the reliability of fault detection and predictive maintenance strategies. Ultimately, this leads to improved operational eÿciency, reduced downtime, and optimized performance within the industry. However, the handling and interpretation of such complex data require advanced methods, which is where deep learning (DL) and other AI techniques come into play.

ÿ . Concrete motivations and context

Jÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿÿÿÿ ÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿ

The ÿeld of AI has seen rapid development since its inception in the ÿÿÿÿs. The historic Dartmouth workshop (McCarthy et al. ÿÿÿÿ), along with Alan Turing's groundbreaking paper Computing Machinery and Intelligence (Turing ÿÿÿÿ), laid the foundations for this exciting ÿeld of study. While the original question -"Can machines think?" -and the pursuit of strong AI, including artiÿcial consciousness, still remains elusive, it has nonetheless inspired the creation of autonomous systems that rival, and sometimes surpass, human performance in speciÿc tasks. Thus, IBM Deep Blue literally beat chess world champion Gary Kasparov in ÿÿÿÿ, while more recently Deepmind reinforcement learning models AlphaGo (Silver et al. ÿÿÿÿ) and AlphaStar (Vinyals, Babuschkin, et al. ÿÿÿÿ) achieved the same performance in more complex games, respectively Go and Starcraft ÿ. In addition to gaming, AI has been instrumental in transforming many industrial sectors. For instance, in healthcare, AI has not only been used for skin cancer detection (Esteva et al. ÿÿÿÿ) but has also demonstrated promising results in diagnosing diabetic retinopathy (Gulshan et al. ÿÿÿÿ).

In biology, apart from revolutionizing protein-structure prediction with AlphaFold (Jumper et al. ÿÿÿÿ), AI has been utilized in drug discovery and development (Stokes et al. ÿÿÿÿ). The aeronautics sector has witnessed the conception of autonomous vehicles powered by AI (Grigorescu et al. ÿÿÿÿ), while in Natural Language Processing (NLP), neural machine translation systems have signiÿcantly improved thanks to AI, notably in ÿÿÿÿ (Sutskever, Vinyals, et al. ÿÿÿÿ), and more recently with the introduction of Transformer architecture (Vaswani et al. ÿÿÿÿ). Furthermore, AI has made signiÿcant strides in predictive maintenance through machine health monitoring (Yiwei Cheng et al. ÿÿÿÿ).

These advancements can be attributed largely to the success of Machine Learning (ML), and more recently, Deep Learning (LeCun et al. ÿÿÿÿ; M. Raghu et al. ÿÿÿÿ). Machine learning, a branch of subsymbolic AI, leverages past experiences, represented by annotated datasets, to build predictive models. This process involves an iterative optimization problem using the available data, placing signiÿcant importance on the representation of the input data. Unlike traditional ML, DL architectures use generic priors to learn a suitable representation of input data through nonlinear transformations (Bengio, Courville, et al. ÿÿÿÿ). This learned representation aims to extract salient features from the raw data structure, which is then used by a classiÿer to make relevant decisions. Over the past years, the community put a lot of emphasis on Representation Learning: the more expressive the representation is, the more eÿective and generalizable the model will be.

ÿ . Concrete motivations and context

However, a key challenge that constrains DL models is the data requirement. These models typically require vast amounts of data to perform optimally, and this prerequisite often outstrips the available labeled data, especially in niche or sensitive domains. While the initial successes of Deep Learning were largely popularized by supervised learning approaches, where models were trained on large labeled datasets, the AI research community therefore quickly recognized the need for more versatile learning paradigms, especially for scenarios where labeled data is scarce or non-existent. This gave rise to the development of multiple learning paradigms to optimize data usage:

• Unsupervised Learning: These approaches, such as clustering (J. Xie et al. ÿÿÿÿ) and dimensionality reduction(Geoÿrey E Hinton et al. ÿÿÿÿ), train models using unlabeled data, discovering hidden patterns and structures without guidance.

• Semi-Supervised Learning: As the name suggests, this technique utilizes a mix of labeled and unlabeled data for training (Zhu ÿÿÿÿ). The idea is to leverage the unlabeled data to enhance the learning process, particularly when labeled data is limited.

• Transfer Learning: This paradigm revolves around the reuse of pre-trained models on new, related tasks. The principle is to leverage the knowledge acquired from one task to improve learning in another, reducing the need for extensive labeled data in the new task (S. J. Pan et al. ÿÿÿÿ).

• Domain Adaptation: This approach aims to adapt models trained on one domain (source) to perform well on a diÿerent but related domain (target), especially when the target domain has limited labeled data (Ganin et al. ÿÿÿÿ). It is a subset of transfer learning that addresses shifts in data distribution between tasks.

• Few-Shot Learning (FSL): FSL (Vinyals, Blundell, et al. ÿÿÿÿ) focuses on training models to make accurate predictions with minimal labeled examples. It leverages techniques that emphasize generalization, enabling models to learn eÿectively from a small sample size.

More recently the AI community has turned towards self-supervised learning, a paradigm in which models are pre-trained on large amounts of unlabeled data and then ÿne-tuned on a smaller labeled dataset. This approach not only makes eÿcient use of the available data but also equips models with a better generalization capacity. The advent of self-supervised learning is complemented by the scaling paradigm (Kaplan et al. ÿÿÿÿ; Rosenfeld et al. ÿÿÿÿ), which posits that model performance can be improved by simply increasing the model size, data size, and the computational resources, given the right model architecture and learning algorithm. This has lately led to the rise of 'Foundation Models', such as GPT-ÿ (OpenAI ÿÿÿÿ), which are large, general-purpose models trained on massive data from the internet. These models can be ÿne-tuned on speciÿc tasks with relatively little data, redeÿning the state of the art in numerous AI applications. As we advance, the focus remains on harnessing these paradigms to build more eÿective, robust, and versatile AI systems.

ÿ ÿÿÿ Oÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

In the evolving landscape of Industry ÿ.ÿ, this thesis takes place in the project Maintenance Prévisionnelle et Optimisation ÿ (MPO) of IRT SystemX. This project aims to overcome the technological and methodological barriers of predictive maintenance and the combination of maintenance policies in production systems, made possible by new technologies and artiÿcial intelligence, and the computing power of the machines, in order to optimize their maintenance in operational condition. In the context of this project, the global objective of this thesis was to study predictive maintenance and more precisely fault diagnosis under the spectra of deep learning and multimodal and heterogeneous data sources. It also includes some works on designing a speciÿc use case, based on a three-tank system, aiming to illustrate the fault diagnosis on a simple applicative example and proposing baselines to tackle the challenges of predictive maintenance data and tasks. The related article (Pellegrain, Batteux, et al. ÿÿÿÿ) was published in a national conference and is relegated to Section ÿ.ÿ. While situated within the highly applied context of Industry ÿ.ÿ and the MPO project, the ambition of this thesis extends beyond the development of models for speciÿc applications. Instead, the main goal is to address the challenges methodologically, intending to introduce novel techniques for the general framework of harnessing multimodal and heterogeneous data. These newly proposed methods aim to unlock the potential of data diversity in Industry ÿ.ÿ, thereby enabling enhanced fault diagnosis and other predictive maintenance tasks. As such, the focus of this thesis lies not in crafting a solution for a speciÿc application, but rather in contributing methodological advancements that can be universally applied in the realm of data exploitation in Industry ÿ.ÿ. However, each of these ambitious goals also presents its unique set of challenges and considerations that requires careful and meticulous addressal.

(i) A ÿrst objective deals with the dynamic and real-time nature of industrial systems.

These systems generate data streams that are continuously acquired, often with heterogeneous acquisition frequencies. For instance, some sensors might collect data at millisecond intervals, while others might gather information every few minutes or even hours. The challenge here is to manage these data streams e ectively, in a time and memory-e cient manner. Due to the real-time demands, it is crucial to devise strategies that are capable of rapidly adapting to changing conditions. These strategies must be able to provide meaningful insights for fault diagnosis while maintaining acceptable computational eÿciency. Within this context, the task of revealing a strong diagnostic signal from potentially weak individual signals becomes even more critical. As an example, an immediate increase in temperature might be less alarming than a slower, yet consistent, increase over a period of time, which could indicate a potential failure or malfunction.

(ii) The second objective emerges from the need to tackle the complexity of integrating data with heterogeneous structures. This data is frequently sourced from various sensors or systems, with each source providing a unique perspective on the system's condition. An example that illustrates this scenario could be a vibration sensor indicating an anomaly. However, when this data is coupled with additional information such as system's images ÿ https://www.irt-systemx.fr/en/projets/mpo/ ÿ . Objectives and challenges

or noise, the diagnostic potential becomes far more precise and insightful. It is clear that considering such heterogeneity in data sources and their representations is crucial in improving the accuracy and reliability of fault diagnosis. This challenge is not unique to industrial systems but common to many domains, which has led to the introduction of multimodal learning and fusion paradigms. Many approaches have been proposed under these paradigms, aiming to capture the richness of these multiple perspectives and translate them into robust decision-making strategies. These strategies seek to consolidate data from diÿerent modalities, each contributing uniquely to the overall understanding of the system. However, a closer look reveals an under-explored aspect within these strategies: the interactions among features from diÿerent data sources. While these interactions can bring critical insights, they are often not explicitly considered in the fusion models. Therefore, we do not fully control how they inÿuence the decision-making process. Further, when these interactions are taken into account, it is typically the redundant interactions that are most often considered. The complementary interactions, that amplify or reÿne the understanding of a system when considered together, are frequently overlooked. Therefore, the second objective is twofold: ÿrstly, to better integrate multisource heterogeneous data; secondly, to reinforce our understanding and control the interactions among these data sources. This poses a broader question: how can we design fusion models that not only e ectively integrate data from multiple sources and modalities but also take advantage of the redundancy and complementarity among these features?

(iii) The third objective focuses on leveraging the wealth of information captured in textual data, particularly in maintenance reports. These documents, often written by experts, encapsulate rich, contextual information about the system's state, historical issues, and previous maintenance activities. The growing interest in exploiting all modalities far addressing Industry ÿ.ÿ tasks results in more open-access resources (Akhbardeh et al. ÿÿÿÿ). However in real-world, the scarcity of such reports, combined with the highly specialized and industryspeciÿc vocabulary, makes their processing and understanding a challenging task. Traditional methods of training DL models require many annotated data to understand and adapt to this speciÿc language use. Given the rarity and speciÿcity of these maintenance reports, applying usual supervised learning paradigms becomes unrealistic. Recent advancements in language models provide a promising direction for interpreting these reports, yet their application is not straightforward. How can we e ectively harness the expressiveness of human language encapsulated in these reports, especially when they are scarce? How can we adapt these advanced language models to the speci c language used in these maintenance reports? Moreover, the usage of these models should not compromise the privacy and conÿdentiality of sensitive information, adding another layer of complexity.

Addressing these challenges forms the core of this thesis. By exploring novel strategies and techniques, we aim to help in surmounting these obstacles and reveal the full potential of multimodal and textual data in predictive maintenance.

ÿ ÿÿÿ Oÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ

In line with the previously deÿned challenges, this thesis presents two distinct contributions, each devoted to a speciÿc area of research: Multimodal Learning and FSL in NLP. This also deÿnes the outline of the thesis, divided in two primary parts.

I. Exploiting multimodal data for fault diagnosis.

The ÿrst part begins with a clear, pragmatic need from the industrial ÿeld to diagnose faults in complex, multimodal systems. This concrete motivation led us towards the development of a more abstract theoretical framework based on multimodal learning, which is inherently motivated by the multimodal nature of our realworld environment. In Chapter ÿ, we revisit related established concepts such as multimodal fusion and representation. We analyze the evolution of these paradigms, from their early stages to the advent of DLbased multimodal representations. This comprehensive review also includes an analysis of the few attempts that have applied ML for fault diagnosis, focusing on the pragmatic constraints of fault diagnosis that have not been addressed by previous multimodal approaches. Speciÿcally, the challenges of handling arbitrarily long data streams in a memory and time-eÿcient manner, and performing inferences in streaming mode, are examined in depth. Bridging the gap between theory and application, in Chapter ÿ we introduce "StreaMulT," a Streaming Multimodal Transformer. This innovative algorithm oÿers a unique solution to the challenges posed by Industry ÿ.ÿ systems' complexity. By employing cross-modal attention and a memory bank, StreaMulT is capable of processing arbitrarily long input sequences during training. Further, it operates in a streaming mode during inference, thereby managing the temporal unalignment of multimodal data and balancing the diÿerences in data acquisition frequency. This contribution led to the article (Pellegrain, Tami, et al. ÿÿÿÿ), published in the Conférence Nationale d'Intelligence Arti cielle . Chapter ÿ extends the discussion to the theoretical realm, presenting an exploration of multimodal representation and fusion and highlighting the need for further research in datasets and architectures for eÿective multimodal learning.

II. Leveraging scarce and speci c textual data in a realistic setting

The second part of the thesis begins with Chapter ÿ, oÿering an extensive overview of NLP methodologies, starting with early techniques centered on feature engineering and statistical word properties and transitioning towards DL approaches and recent Foundation models. Furthermore, the chapter examines associated works in Few-shot learning, shedding light on the latest progress and challenges in this research area. In light of these developments, we notice a gap in the ÿeld when dealing with scenarios where labeled data are rare. Current FSL methods in NLP, mainly based on the prompting strategy, show limitations, especially for realistic classiÿcation tasks with a large number of classes. These limitations are primarily due to engineering eÿorts required to make these methods work eÿectively in such situations. To cope with these issues, in Chapter ÿ we revisit transductive learning in the NLP ÿeld, trying to reproduce the success encountered in computer vision. This paradigm, unlike inductive learning, enables the eÿective utilization of limited labeled data by taking advantage of the statistics of unlabeled data.

ÿ . Outline and contributions

Then, we consider the increasing prevalence of proprietary and closed Application Programming Interfaces (APIs) for Large Language Models (LLM) in Chapter ÿ. We introduce a new parameterfree regularizer based on the Fisher-Rao loss, which demonstrates its eÿectiveness and applicability in this setting. This diÿers from current methods and provides a novel way to tackle FSL problems. In such a scenario, our transductive approach enables fast and eÿcient predictions without the need to share sensitive label information, thus adapted data-privacy constraints. This not only paves the way for improved performance but also opens new research ideas for practical applications in the ÿeld of FSL. The article that emerged from this contribution is currently under review for publication in an international journal.

Finally, Chapter ÿ concludes this thesis and proposes perspectives for both parts.

ÿ Pÿÿÿ I Pÿÿÿÿÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿ Lÿÿÿÿÿÿÿ Sÿÿÿÿÿÿÿÿÿ ÿÿÿ Iÿÿÿÿÿÿÿÿÿ Fÿÿÿÿ Dÿÿÿÿÿÿÿÿ ÿÿ ÿ Bÿÿÿÿÿÿÿÿÿ ÿÿÿ Rÿÿÿÿÿÿ Wÿÿÿ Cÿÿÿÿÿÿ'ÿ Sÿÿÿÿÿÿ
In this chapter, we give the reader the background needed to motivate and understand the ÿrst part of this thesis. We start by presenting fundamentals of Fault diagnosis theory in Section ÿ.ÿ and we review existing strategies to tackle this problem, focusing on ML approaches and exploring the few attempts that considered data from heterogeneous modalities. In Section ÿ.ÿ, we introduce the multimodal learning paradigm, with a particular emphasis on multimodal fusion. From there, we propose an overview of developed methodologies, beginning with older works relying on simple fusion strategies such as concatenation, and more focused on which level to realize the fusion. We then point out the advantages of building expressive data representations, which is mostly feasible by the mean of Deep-Learning-based architectures, and the closeness between multimodal fusion and multimodal representation. We therefore explore approaches on Multimodal Representation Learning, which are nowadays mainly based on the Transformer architecture.

ÿÿÿ Aÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿ

Due to plenty of causes -both internal and external -industrial machines are likely to suÿer a fault at some point (e.g., corrosion). If not detected, these faults can lead to the incidence of failures (e.g., leakage). That is a major issue since it means a ÿnancial loss for the company and sometimes much more when human lives are at stake. To address this problem, it is common to perform fault diagnosis. Following (Isermann ÿÿÿÿ) terminology, we properly deÿne these previous terms.

Dÿÿÿÿÿÿÿÿÿ

De nition . A fault is an uppermitted deviation of at least one characteristic property (feature) of the system from the acceptable, usual, standard condition.

A failure is a permanent interruption of a system's ability to perform a required function under speciÿed operating conditions. Fault monitoring refers to the detection of a fault occurrence. Fault diagnosis consists in determining the type, size and location of the most possible fault, as well as its time of detection.

ÿÿ

Background and Related Work

In practice though, in the literature it is common to write fault diagnosis to refer to both fault detection and its diagnosis as deÿned above. The pioneer series of three articles of Venkatasubramanian et al. (Venkatasubramanian et al. ÿÿÿÿ) is one of the ÿrst works to list and categorize the diÿerent methods of fault diagnosis; and therefore constitutes the starting point of our review. This series classiÿes fault diagnosis approaches depending on both the a priori knowledge one has on eventual faults, along with how they would be expressed through the acquired data of the system (i.e. fault symptoms). Two diÿerent kinds of strategies can be distinguised in the litterature. A ÿrst family of approaches, named model-based, uses the a priori knowledge by the system by a physical model. On the other side, the approaches only relying on the history of acquired data are called data-based. While model-based methods can be well suited when one has a nice a priori understanding of physical laws governing the system, they become less relevant otherwise. Thus, when the considered system reaches a certain level of complexity, inter-components interactions can less easily be modelled. To address this, data-based approaches provide a viable alternative: the designed model aims to learn these components dependencies from the data history. We mostly focus on data-based works in our review, and more precisely on ML ones.

In the next section, we ÿrst introduce the paradigm of Machine Learning (ML) and Deep Learning (DL) through the lens of Statistical Learning theory and the popular supervised learning framework. In a second time, we review the diÿerent approaches that make use of ML and DL to tackle Fault diagnosis.

ÿÿÿÿÿ Bÿÿÿÿÿÿÿÿÿÿ Pÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ

Given a set of observations of a phenomenon, the aim of Statistical Learning (V. Vapnik ÿÿÿÿ) is to build a model of this phenomenon than can then perform inference on new data, that is, make predictions. Machine Learning is a framework that tries to automate this learning process using algorithms to design a function that maps input observations to desired outputs. Based on statistics and optimization problems, this procedure selects the function that both best ÿts the observed data, and stays generalizable to unobserved data.

Formally, we consider an input space X and an output space Y. We consider tuples of observations (x, y) 2 X å Y, that are viewed as realizations of the random variables X and Y respectively. Considering a dataset D = (x i ; y i ) n i=1 containing n independent and identically distributed data pairs sampled from a distribution density p X,Y , unknown but such that:

p X,Y (x, y) = p Y |X (y|f å (x))p X (x)
we seek to address the related task, that is learning a function (i.e. a model) f : X ! Y approaching f å , the true unknown mapping of the task.

To assess for the quality of the model f , we consider a loss function L : Y å Y ! R + , such that L(f (x), y) measures the point-wise error when the model predicts f (x) instead of y.

ÿÿ . Addressing fault diagnosis with machine learning and deep learning

To fulÿll the objective of learning a model close to the true mapping f å , the usual learning paradigm is to minimize the population risk R(f ) deÿned in the following:

Dÿÿÿÿÿÿÿÿÿ De nition . Population Risk Let D = (x i ; y i ) n
i=1 a dataset of i.i.d. data pairs sampled from a distribution p X,Y , f : X ! Y a prediction function, L a loss function. The population risk associated to f is deÿned as the expected loss:

R(f ) = E p X,Y [L(f (X), Y )] (ÿ.ÿ)
where E p X,Y is the expectation associated to distribution p X,Y . As we usually cannot access the true distribution p, a common surrogate is to minimize the Empirical Risk.

Dÿÿÿÿÿÿÿÿÿ

De nition . Empirical Risk Let D = (x i ; y i ) n i=1 a dataset of i.i.d. data pairs sampled from a distribution p, f : X ! Y a prediction function, L a loss function. The empirical risk Rn (h) is deÿned as the empirical mean loss measured on the dataset:

Rn (f ) = 1 n n X i=1 L(f (x i ), y i ) (ÿ.ÿ)
Hence, for a class of functions F, the Empirical Risk Minimization (ERM) algorithm consists in ÿnding f := arg min f 2F Rn (f ). The hypothesis space F represents the family of models (for instance linear functions) on which to minimize the Empirical Risk, and is usually chosen by the learner in preamble of the procedure, following inductive biases regarding the input data and the task. From then, we note our model f ψ , where ψ 2 Ψ are learnable parameters, and Ψ is the parameter space deÿned by the chosen family of models (for instance, vectors of weights and biases deÿning the linear models). The learning objective is now to ÿnd ψ that minimizes the empirical risk:

ψ := arg min ψ2Ψ Rn (f ψ )
Finally, we can decompose f as f = h g, in which g : X ! Z represent a feature extraction module, that maps input observations to a latent space Z and h : Z ! Y a predictor that maps the latent representations to the output space. By writing g and h as parametric functions, and noting ψ = (θ, φ) 2 Θ å Φ, we note f ψ = h φ g θ . In the classical shallow ML setting, the manually designed feature extraction module g is ÿxed, and the ERM thus consists in optimizing only the predictor h φ on parameter space Φ, that is, ÿnd φ such that:

φ := arg min φ2Φ Rn (h φ g) ÿÿ

Background and Related Work

Dÿÿÿ ÿÿÿÿÿÿÿÿ

Over the last decade, the advent of DL architectures (LeCun et al. ÿÿÿÿ), demonstrated their superiority over classical ML approaches in numerous application ÿelds and their related tasks. At the heart of this paradigm: deep neural networks. A deep neural network of L layers is a function

f : X ! Y such that: 8x 2 X , f (x) = h g(x) = h g 1 . . . g L (x)
Compared to previously formalized shallow models f = h g considered in classical ML approaches, deep neural networks' feature extraction modules g are composed of L stacked layers, that will also be optimized during the learning procedure, and thus not manually designed. These layers essentially characterize the whole network as they condition the learned representations of input data, which, if expressive enough, only needs a simple predictor h to eÿectively address a task.

A traditional neural network architecture is for instance the Multi-Layer Perceptron (MLP), that is composed of stacked linear functions, followed by non-linear activations, i.e. for i = 1, . . . , L, g i (x) = a i (w T i x + b i ) with w i and b i being the i th layer associated weights and bias, and a i being the non-linear activation function, usually hyperbolic tangent, sigmoid function, softmax function, or rectiÿed linear unit function (Goodfellow et al. ÿÿÿÿ). This breakthrough in AI quest is mainly explicable in DL ability to learn good representations from input data (Bengio, Courville, et al. ÿÿÿÿ), compared to classical feature extraction modules. Their design consisting of stacked modules followed by non-linear activations oÿers the possibility to learn hierarchical and distributed representations in which last layers thus represent concepts of a higher abstraction, expressed as a combination of simpler components learned in ÿrst layers. These properties tend to facilitate the encoding of factors of variation of the input data, while being more invariant to meaningless noise. Therefore, many current works focus their energy on the design of representation learning algorithms that integrate such generic properties.

From there, we can rewrite the true mapping from inputs to outputs f å such as f å = h å g å with g å : X ! Z being the true mapping from input to latent space and h å : Z ! Y the true mapping from latent to target space. The sampling distribution p X,Y of the considered dataset D can now be written as:

p X,Y = p Y |X (y|h å g å (x))p X (x) (ÿ.ÿ)
and we now aim to ÿnd f that minimizes the associated empirical risk:

ÿÿ

. Addressing fault diagnosis with machine learning and deep learning

f : = arg min f 2F Rn (f ) (ÿ.ÿ) = arg min f 2{h g|g2G,h2H} Rn (f ) (ÿ.ÿ)
with F, G, H the class functions deÿning the hypothesis spaces. Using the parametric notation, we aim to ÿnd ( θ, φ) such that

( θ, φ) = arg min (θ,φ)2ΘåΦ
Rn (g θ h φ )

When solving this optimization problem, we hope that the learned parameters ( θ, φ) also minimize the population risk over unseen new samples, so that the inference function can be reliably used to solve the task of interest. To realize an eÿective learning procedure, the learner should then, based on priors regarding the task of interest and input data:

• Deÿne an adequate hypothesis space deÿning the family of considered models (for instance Convolutional Neural Networks), through the parameter space Ψ = Θ å Φ;

• Specify an adequate loss function L to measure the pointwise prediction error of the model (for instance the Cross-Entropy loss);

• Select an appropriate learning procedure to solve the optimization problem induced by ERM (for instance, using Stochastic Gradient Descent algorithm or derivatives such as SGD with momentum (Sutskever, Martens, et al. ÿÿÿÿ) or Adam optimizer (Kingma et al.

ÿÿÿÿ));

• Design a testing procedure to evaluate the model's performance on unseen data and therefore get insight on its generalization ability.

ÿÿÿÿÿ Mÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ

Several reviews (cited thereafter) list the diÿerent ML architectures designed for tackling the fault diagnosis problem. Some of these reviews adopt an industrial-domain-speciÿc position: while (Nor et al. ÿÿÿÿ) expose fault diagnosis methods that have been used for chemical process systems, (S. Zhang et al. ÿÿÿÿ) focus on bearing faults, whereas (Rogers et al. ÿÿÿÿ) only consider residential air conditioning systems. These studies mainly motivate their approach by the consequences of fault occurrences in their relative ÿelds, such as the over-consumption of electricity and the induced economic costs (Rogers et al. ÿÿÿÿ). Besides, the methods listed in these reviews are presented as relevant for dealing with data relative to these applicative ÿelds. Therefore, (S. Zhang et al. ÿÿÿÿ) essentially consider vibration and stator current data, as contained in the Paderborn dataset ÿ ; whereas (Rogers et al. ÿÿÿÿ) rather present models calibrated for thermostat and humidity data, with for each of these approaches an important and non-scalable work that consists in ÿ Available online: https://mb.uni-Paderborn.de/kat/forschung/datacenter/bearing-datacenter ÿÿ Background and Related Work designing speciÿc hand-crafted feature extractor g for the speciÿc applications. By contrast, other works adopt a more methodological position regarding their reviews of stateof-the-art algorithms for addressing fault diagnosis (Palade et al. ÿÿÿÿ). This is more in adequation with our positioning. Most recent ones (Angelopoulos et al. ÿÿÿÿ; Z. Li ÿÿÿÿ; Reis et al. ÿÿÿÿ) motivate their work by the emergence of new practical challenges induced by the arrival of Industry ÿ.ÿ era, such as notably the ability to handle massive and multi-sources data with a short-time response. These reviews qualify ML methods as more eÿective compared to model-based approaches when fault proÿles are complex, such as (S. Zhang et al. ÿÿÿÿ), which mention the limits of model-based approaches for the early detection of faults, due to symptoms that are untraceable by this kind of models. They also point out model-based approaches' diÿculty to disentangle the simultaneous occurrences of diÿerent faults.

Although some articles only consider fault detection (Luo et al. ÿÿÿÿ; Wen et al. ÿÿÿÿ), the vast majority also considers fault isolation and identiÿcation ÿ . However as emphasized by (Reis et al. ÿÿÿÿ), in practice two methodologies co-exist. On the one hand, Statistical Process Control community sequentially processes fault detection and fault isolation and identiÿcation. On the other hand, ML community often processes these two tasks in a simultaneous fashion, in the form of a (C + 1)-classes classiÿcation, decomposed into one class of normal functioning mode and C distinct faulty functioning modes.

As presented in (Z. Li ÿÿÿÿ), ML models used for fault diagnosis are generally composed of a feature-extraction module and a diagnosis module. In that conÿguration, the former feeds the latter relevant elements computed from raw data. Some feature-extraction modules focus on time domain to catch and characterize information contained within time series acquired from the system sensors, using for instance neural networks (Zarei et al. ÿÿÿÿ). It is also common to use signal processing tools in order to exploit features from the time series in the frequency domain. (Yukun Liu et al. ÿÿÿÿ) and (Taj et al. ÿÿÿÿ) thus respectively use Fourier and Laplace transforms to this purpose. Finally, other approaches choose to work in the time-frequency domain, through the usage of wavelet transforms for instance (Z. Zhang et al. ÿÿÿÿ). The choice of feature-extraction module is strongly inÿuenced by the structure of input data and the subsidiary task, therefore by the a priori knowledge of its designer. The very diagnosis module is then composed of:

• either a ÿrst detection submodule aiming to perform fault monitoring, followed by a second classiÿcation submodule performing fault isolation and identiÿcation;

• either a unique classiÿcation module carrying out simultaneously both fault detection, and fault isolation and identiÿcation.

In a supervised setting, the unique classiÿcation module fed with extracted features is free to use any ML model: Support Vector Machine (Konar et al. ÿÿÿÿ), Random Forest (B.-S. Yang et al. ÿÿÿÿ), shallow neural networks (Jafar et al. ÿÿÿÿ), Recurrent Neural Networks (RNN) (Yam et al. ÿÿÿÿ), and so on. This scheme of performing simultaneously fault detection and classiÿcation has however been sometimes criticized (Reis et al. ÿÿÿÿ), as it might lead to practical issues:

• fault occurrences that might lead to failures and dreaded event are often scarce in real datasets.

This results in an imbalanced dataset problem, exacerbated the more faulty classes one considers.

ÿ note that (Angelopoulos et al. ÿÿÿÿ) • For this kind of tasks, a prediction error will have the same weight during the learning phase, regardless of which misclassiÿcation has been made. However, depending on the system criticality, one would like to put a lot more emphasis on the fault detection rather than on its proper identiÿcation.

To cope with these issues, a prior monitoring task can be realised using anomaly detection methods (Goldstein et al. ÿÿÿÿ). Similarly to the architectures designed in Statistical Process Control community's works, these semi-supervised methods model the normal functioning mode of the system during the learning stage, and classify as fault the datapoints which deviate signiÿcatively from this model's prediction at test time. These approaches are more robust to imbalanced datasets and can then be coupled with a classiÿcation model to perform the isolation and identiÿcation task. Lastly, if the normal functioning mode conditions are unknown (i.e. in an unsupervised setting), it is also possible to design the diagnosis module by using clustering approaches (Diaz Rozo et al. ÿÿÿÿ).

Similarly to model-based methods, classical ML approaches faced some limitations induced by growing complexity of industrial system data. As described in (Z. Li ÿÿÿÿ; Y. Peng et al. ÿÿÿÿ; S. Zhang et al. ÿÿÿÿ), classical feature-extraction-based models based on a certain a priori knowledge on input data structure, may no longer be eÿective to perform a correct fault diagnosis. Indeed, with a growing complexity in studied systems, the manual feature engineering struggles in designing representations encompassing all the expressiveness and complexity of input data. As such, these approaches are less prone to model more abstract inter-dependencies between data signals and to be robust to noise. To answer these challenges, DL models are designed, as they integrate a representation learning part in the layers g 1 , . . . , g L . This part aims to automatically extract the most salient features for a subsidiary task (here the fault diagnosis), with no -or few -a priori knowledge on input data structure required (Bengio, Courville, et al. ÿÿÿÿ; LeCun et al. ÿÿÿÿ). Thus, numerous articles have shown the superiority of DL models over classical ML ones for fault diagnosis, using as representation learning algorithms either discriminative models (like Convolutional Neural Networks (CNN) (J. Pan et al. ÿÿÿÿ; Wen et al. ÿÿÿÿ; Xia et al. ÿÿÿÿ), deep RNN (Abed ÿÿÿÿ; L. Guo et al. ÿÿÿÿ), Transformers (B. Wu et al. ÿÿÿÿ), etc.) or generative models (like Probabilistic Graphical Models (PGM) (T. Liang et al. ÿÿÿÿ; K. Yu et al. ÿÿÿÿ), autoencoders (Jia et al. ÿÿÿÿ; Shao et al. ÿÿÿÿ; J. Sun et al. ÿÿÿÿ), GANs (Han Liu et al. ÿÿÿÿ; Y. Xie et al. ÿÿÿÿ)). However, all these works consider unimodal data (namely sensors measurements), and therefore do not address the multimodal input challenge.

ÿÿÿÿÿ Fÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿ

The complexity of industrial systems and of the relative acquired datasets, reaches nowadays a new level, with sensors producing multimodal data. While some previous works tackled the challenge of fault diagnosis from various unimodal data such as thermal images (Choudhary et al. ÿÿÿÿ; Janssens et al. ÿÿÿÿ; Taheri-Garavand et al. ÿÿÿÿ), x-ray data (Reid et al. ÿÿÿÿ), photographs (J. Wang et al. ÿÿÿÿ; Sen Wang et al. ÿÿÿÿ) or textual maintenance reports (Sipos et al. ÿÿÿÿ; F. Wang et al. ÿÿÿÿ), the application of such models to multimodal data (i.e. of heterogeneous natures) is still in its infancy. Most previous works addressing the fault diagnosis task and mentioning "multimodal" ÿÿ Background and Related Work data actually refer to the diÿerent functioning modes of the considered system (such as an air conditioner functioning in eco-mode or in normal mode) (Sipple ÿÿÿÿ). For (F. Zhou et al. ÿÿÿÿ), the word "multimodal" refers to the diÿerent orders of derivatives of the input time series. To the best of our knowledge, only two articles properly consider multimodal data (as of heterogeneous natures) in a perspective of industrial maintenance. (Mian et al. ÿÿÿÿ) fuse numerical time series of vibration signals with thermal images in order to improve classiÿcation performances in the context of bearing fault diagnosis of rotating machine. They use a classical ML approach, with an Hilbert transform module for feature extraction and a concatenation module for data fusion. Yang et al. (Zhe Yang et al. ÿÿÿÿ) design a multimodal architecture to address failure prognostics, a related task. The aim of this challenge is to forecast the Remaining Useful Life (RUL) of a system, that is the duration before the system encounters failure. In that sense, the ultimate task is a regression, but the studied framework can be transferred to the one we consider. Their approach handle three modalities (sensors numerical measurements, images and texts) as three distinct blocks, learning respective unimodal representations using either convolutive layers (images and texts) or linear layers (numerical measurements). These unimodal representations are then concatenated and eventually fused using a regression layer. While these approaches are interesting and are close of our objective, they suÿer some important limitations. A ÿrst limitation is the fact that they are focused on their speciÿc application, rather than interesting in providing general methods for handling multimodal data in predictive maintenance related tasks. As a consequence their results are diÿcult to generalize to other systems. For instance, a strong limitation is related to their datasets. While in (Mian et al. ÿÿÿÿ), the dataset is not publicly available thus preventing the community to compare one's work to theirs, in (Zhe Yang et al. ÿÿÿÿ) the dataset is synthetic, which implies a lack of richness and diversity, especially for the textual modality. Indeed, the numerous appearances of the exact same sentences in diÿerent examples make the usually unstructured nature of raw text less prominent and representative in that case. Besides, the considered images are actually only curve plots corresponding to the acquired numerical measurements. Hence, they do not represent actual visual captures of the system, which have a much diÿerent local structure and would have brought additional information.

T

A large body of works has been proposed regarding ML learning approaches for fault diagnosis with two main strategies: sequentially processing fault detection then fault identiÿcation or processing the two tasks simultaneously. However, as in other domains these ML approaches have been limited by the hand-crafted feature engineering part and has open an avenue for DL models, that enable to automatically learn an expressive representation that can more easily and eÿectively be processed. While the fault diagnosis in Industry ÿ.ÿ is multimodal by nature, only few approaches have taken interest in this challenge yet, handling either private or synthetic data. These observations emphasize an important and critical point for the study of multimodality in the context of industrial system monitoring: the unavailability of real multimodal dataset in the Industry ÿ.ÿ community. As for the MPO project, we did not either access multiÿÿ . Multimodal learning modal datasets, as a main part of the objectives was rather to structure the data acquisition pipeline. As a consequence, in our study, while still motivated by the challenges that come from the predictive maintenance ÿeld, we will mostly consider datasets coming from alternative ÿelds. Therefore, we hereby invite industrial actors to provide such public representative data, in order to encourage the development of future works on these high-stakes challenges. Building upon this clear need for enhanced multimodal analysis in the realm of industrial systems, we delve deeper into the speciÿc methodologies and potential applications of multimodal learning in the subsequent section.

ÿÿÿ Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿ Fÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ Human beings perceive the world through a multimodal lens, integrating various sensory inputs to better understand and interact with their environment. Multimodal perception encompasses the following:

• Situating ourselves in space and navigating using sight to generate images of our surroundings,

• Communicating with one another through speech, thus producing and interpreting sounds,

• Smelling odors,

• Tasting ÿavors,

• Experiencing diÿerent temperatures and textures, and more.

Hence, from a cognitive perspective, the term "multimodal" here refers to the nature of diÿerent sensory stimulations that we, human beings, receive when engaging with the environment. The ÿeld of multisensory processing, also known as multisensory integration, investigates how distinct parts of the nervous system and brain process and combine these stimuli to form accurate beliefs about the environment. According to (Maragos et al. ÿÿÿÿ), this whole process can be divided into three stages:

• Sensation: The electrical signal generated by a speciÿc organ in response to a stimulus,

• Perception: The more complex process of ÿltering, aggregating, and organizing sensations,

• Cognition: The ultimate comprehension and decision-making component.

Although the boundaries between these stages are often blurred, the term multimodal perception is commonly employed to describe sensory-based reasoning about the environment, particularly the reverse path of inferring the world state from various stimuli. The accuracy and robustness of Human multimodal perception are either innate (determining the localization of a speaker ÿÿ Background and Related Work using sight and sound) or learned over time through repeated exposure to similar situations. To demonstrate the signiÿcance of this phenomenon, consider a person strolling alone on a cloudy beach. They can smell the aroma of meat cooking on a barbecue at a nearby restaurant. Suddenly, they hear a rumble of thunder. In this instance, the individual experiences three unimodal stimuli:

• The sight of the desolate, cloudy beach,

• The sound of thunder,

• The smell of barbecue.

These stimuli activate diÿerent sensory organs and their associated acquisition systems, namely the visual, auditory, and olfactory systems. Thus, the person's brain and nervous system will associate visual and auditory modalities as both indicate the presence of a storm (the sight of clouds and the sound of thunder), while ÿltering out irrelevant information, such as the smell of barbecue. Drawing from past experiences or learned information, the person will recognize this multimodal situation as dangerous by combining visual and acoustic complementary modalities (beaches are unsafe during thunderstorms due to the risk of lightning strikes). This example is illustrated in Figure ÿ.ÿ. As (Lachs ÿÿÿÿ) highlight, multisensory integration not only aggregates relevant unimodal stimuli or ÿlters out irrelevant ones but also enhances the strength of neural responses when processing multimodal events compared to unimodal ones. This phenomenon, known as multimodal enhancement, means that the measured response to a multimodal event exceeds the sum of measured responses when experiencing the same event unimodally. The enhancement capacity is even greater when the strongest response to unimodal stimuli is weak: this is the Principle of Inverse Eÿectiveness (Stein et al. ÿÿÿÿ). (Lachs ÿÿÿÿ) illustrate this principle by considering a task of speech comprehension in a crowded place. If the environment is excessively noisy, the auditory modality alone may not suÿce for proper comprehension. Simultaneously, lipreading (the visual modality) can help decipher some words but is generally insuÿcient to understand an entire sentence. However, the combination of visual and acoustic cues can provide the listener with a general understanding of the conversation. Therefore, although both unimodal responses are relatively weak ÿÿ . Multimodal learning for this task, the enhancement resulting from multisensory integration is substantial. Conversely, in a quiet environment, the listener only requires the auditory modality, which will generate a strong response, and the multimodal enhancement will be minimal. As a result, in the beach example, multisensory integration led the person to take the decision to seek safety. Meanwhile, if they had processed only unimodal signals independently, they would not have arrived at this conclusion, as none of the unimodal information (sight of a beach, sound of thunder, or smell of meat) typically suggests the need to urgently ÿnd shelter. This scenario intuitively demonstrates the advantages of processing multisensory signals over unimodal ones: the human brain ingeniously gathers relevant modalities to exploit redundant and/or complementary information, resulting in an improved decision-making capacity. The primary motivation behind multimodal learning is to emulate the role of the human nervous system in its biological ability to aggregate pertinent data from diÿerent modalities in such a way that it enhances knowledge for a downstream task.

The parallel with our application is obvious. Indeed, in our industrial system case, the auditory stimuli can be replaced by sensors measurements that are continuously acquired, while the visual stimuli can be replaced by images of a part of the system that are regularly acquired (see Figure ÿ.ÿ). The challenge lies in the fact that while the human nervous system naturally converts stimuli from various modalities into electrical signals through receptors from corresponding sensory organs and integrates them via multisensory neurons, numerical data from diÿerent modalities exist in distinct mathematical spaces and possess inconsistent distributions. For example, considered modalities can be either continuous (analog signals like audio recordings) or sparse and discrete (one-hot encoding vectors of raw text, i.e., a symbolic modality). This issue is referred to as the heterogeneity gap and constitutes one of the main challenges of multimodal learning. In other words, as depicted in In particular, (Baltrusaitis et al. ÿÿÿÿ) identify ÿve primary challenges within the Multimodal Learning ÿeld:

• Representation, i.e. learning how to represent and summarize multimodal data in a way that exploits the complementarity and redundancy of multiple modalities

• Translation, i.e. mapping a data point from a source modality space to a corresponding point in a target modality space

• Alignment, i.e. identifying elements from diÿerent modalities related to the same semantic concepts or generative temporal events

• Fusion, i.e. determining the most eÿective and robust method of combining relevant information from diÿerent unimodal signals to enhance a decision-making procedure for a downstream task

• Co-learning, i.e. applying knowledge learned in one modality space to enhance inference in another modality with limited resources From a pragmatic perspective, in the ÿrst part of this thesis we focus on Multimodal Fusion for predictive maintenance downstream tasks such as fault diagnosis. Successfully addressing this task is intrinsically linked to the challenges of Multimodal Representation and Alignment ÿÿ . Multimodal learning challenges. Indeed, tackling these two challenges implicitly helps narrow the heterogeneity gap, making it a valuable preliminary step for Multimodal Fusion. In contrast, we do not explicitly prioritize Multimodal Translation, nor Multimodal Co-learning. These challenges are nonetheless once again ultimately dealing with the heterogeneity gap issue and very linked to the previous ones. For instance, the recent DALL-E ÿ model (Ramesh et al. ÿÿÿÿ)), which addresses the popular challenge of text-to-image generation, relies on the CLIP (Radford, Kim, Hallacy, et al. ÿÿÿÿ) pretrained model, which aims to learn a joint text-image representation using contrastive learning (Bachman et al. ÿÿÿÿ; Hjelm et al. ÿÿÿÿ).

In the next section we essentially review previous works on Multimodal Fusion approaches, that sometimes thus also address other issues as stated above, and especially Multimodal Representation learning.

ÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿ

We begin by formalizing the Multimodal Fusion framework by extending the setting introduced in Subsection ÿ.ÿ.ÿ, in a similar way as (Y. Huang et al. ÿÿÿÿ).

We now consider that a datapoint x = (x 1 , . . . , x M ) is composed of M modalities and thus lives in a multimodal input space X = X 1 å X 2 å . . . å X M , i.e. 8 1 ÿ α ÿ M, x α 2 X α , with X α the deÿnition space of the modality α, with its speciÿc dimension. Tuples (x, y) 2 X å Y are viewed as realizations of the random variables X = (X 1 , . . . , X M ) and Y respectively. We still consider a dataset D = (x i ; y i ) n i=1 containing independent and identically distributed data pairs sampled from a distribution density p X,Y , unknown but factorizing as: p X,Y (x, y) = p Y |X (y|f * (x))p X (x), with f * the true mapping from input to output space. We still seek to learn a function f ψ : X ! Y, with f ψ = h φ g θ approaching f * = h * g * . The main diÿerence with unimodal framework is that functions g θ and h φ shall now be designed in a way such that they are able to eÿectively fuse information from input modalities.

Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿ ML ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿÿÿÿ Historically, ML research primarily focused on determining the optimal level for data fusion. Consequently, various approaches were categorized into three distinct groups: early fusion (or feature-level fusion), late fusion (or decision-level fusion), and hybrid fusion methods. Figure ÿ.ÿ represents the diÿerent fusion strategies: early (d), late (e) and hybrid (f), with the help of Analysis Units (AU), Feature Fusion (FF) and Decision Fusion (DF) units, represented in schemas (a), (b) and (c), respectively. As their names suggests, FF and DF units represent the diÿerent fusion modules, while AU units aim to output a decision from an input vector. In the decomposition f ψ = h φ g θ , g θ operates at a feature level, transforming raw inputs into exploitable features, that can then be exploited by h φ to produce decisions. In that sense, we can see Feature-Fusion units as part of g θ , rendering features in a well-suited structure, while Analysis Units and Decision-Fusion units as part of predictor h φ , in charge of producing decisions.

ÿÿ

Background and Related Work

In early fusion (d) (or feature-level fusion), the fusion mechanism operates within g θ : input unimodal features are combined within the FF unit and sent to an AU to produce a ÿnal decision. Conversely, in the late fusion scheme (e), the fusion mechanism operates within h φ : unimodal features are passed through unimodal AU to produce respective unimodal decisions, which are then fused within a DF unit to output the ÿnal decision. Lastly, the hybrid fusion technique (f) involves repeating either early or late fusion strategies on diÿerent sets of unimodal features, ultimately fusing intermediate decisions with a ÿnal DF unit, followed by a ÿnal AU to produce the ultimate decision. In that sense, in this strategy the fusion mechanism operates partly in g θ and partly in h φ .

ÿÿ . Multimodal learning

Early fusion presents the opportunity to model inherent correlations between diÿerent modalities, expressed through low-level features, in order to capture inter-modality dependencies. However, due to the heterogeneity gap, modeling relationships between inconsistent distributions is a nontrivial task and is scarcely achievable when using standard feature fusion strategies, such as simple vector concatenation Overall, these early works on diÿerent fusion levels generally emphasized the advantages of late fusion over early fusion:

• it does not deal with heterogeneity gap between low-level features as it combines unimodal decision scores;

• it does not need to consider diÿerent acquisition times between modalities;

• it is usually more robust when one of the modalities is missing.

Nonetheless, late fusion does not exploit correlation at low-level features between modalities, and thus is not ideally suited for modeling multimodal complementarity. Additionally, from a practical standpoint, early fusion also requires to train only one model (for fusion), as opposed to late fusion.

To leverage the strengths of both approaches, some architectures adopt a hybrid strategy. (Z.-z. Lan et al. ÿÿÿÿ) for instance addresse video event detection by ÿrst training n + c + 1 classiÿers in a early-fusion scheme, in which n is the number of extracted features (individual classiÿers), c is the number of categories for which features have been combined in an early-fusion fashion, and the last classiÿer is fed with all input features. After the training of these classiÿers, their outputs (score vectors) are combined at test time to produce the ÿnal prediction. This double-fusion architecture hence beneÿts from the correlation modelized by early classiÿers and robustness of late classiÿer to eventually provide better results than a single fusion method.

Tÿÿ ÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿ ÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ When considering deep neural networks, the feature-extraction module g φ becomes a representation learning module composed of stacked layers:

g φ = g 1 • • • g L . When
η(g) = inf h∈H [R(h g) R(h * g * )]
(ÿ.ÿ) with g * and h * the true mappings from input to latent space and from latent space to output space, respectively. Here inf h∈H R(h g) is the best achievable population risk with the ÿxed latent representation g. Thus, to a certain extent, η(g) measures the loss induced by the distance between g and g * . Using this deÿnition of the representation quality and extending it to a multimodal framework as deÿned in the beginning of this Subsection, (Y. Huang et al. ÿÿÿÿ) theoretically showed that the diÿerence of population risks of models fN = ĥN ĝN and fM = ĥM ĝM learned on two diÿerent modalities subsets N and M was bounded by the diÿerence of the corresponding latent representation qualities on these subsets. This directly suggests that an adequate proxy to ensure better performances on a multimodal learning task is to build a latent representation closer to the true mapping, as long as the sample size is suÿcient. Besides, they also show that considering more modalities, with a suÿcient sample size, implies a better latent representation quality, hence better learning performances. The intuition, depicted on Figure ÿ.ÿ, is that for two subsets of modalities M and N such that N ã M, the representation learning module ĝM , minimizing empirical risk on M has a more suÿcient space to explore than ĝN . As in other Deep Learning ÿelds, focus is therefore on designing the most expressive and generalizable representation, thus on ÿnding the best architecture for function class G, while the classiÿers considered when deÿning H are often common architectures such as Multi-Layer Perceptrons. In that sense, the boundary between Multimodal Fusion and Multimodal Representation ÿÿ .

Multimodal learning

Learning has become fuzzy. We thus focus on the following on multimodal representation learning.

Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ

Multimodal representation learning strategies are mainly divided into Joint Representation Learning and Coordinated Representation Learning. These two frameworks are illustrated in Figure ÿ.ÿ. The aim of the former is to embed unimodal representations together into a shared multimodal representation. Diÿerently, Coordinated Representation Learning approaches learn distinct unimodal representations that are coordinated, using constraints during training, such as similarity maximization for close concepts. Contrastive approaches (Bachman et al. ÿÿÿÿ; Hjelm et al. ÿÿÿÿ; Radford, Kim, Hallacy, et al. ÿÿÿÿ) are examples of strategies learning coordinated representations. ÿÿÿÿÿ Dÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ

In this section, we present the diÿerent neural architectures that have been proposed in the litterature to implement multimodal representation learning and the two strategies described in the previous section. These architectures can be divided into model-agnostic or speciÿc architectures.

The most straightforward strategy for addressing deep multimodal representation learning is Model-agnostic approaches, like early additive or multiplicative fusion (Bruni et al. ÿÿÿÿ; Zadeh, Minghai Chen, et al. ÿÿÿÿ). These methods design a shared subspace for joint representation learning using a shared hidden layer. Here, encoded data from various modalities are either concatenated, added, or multiplied before activation, thus enabling the fusion of semantics. Figure ÿ.ÿ illustrates such concatenation and multiplication from diÿerent modalities.

In contrast to these architectures, typical models used for Deep Multimodal Representation Learning notably include Probabilistic Graphical Models (PGM), Autoencoders or Attentionbased models. We recall the principles of these diÿerent models in the following.

Aÿÿÿÿÿÿÿÿÿÿÿ

Similarly, autoencoders provide another unsupervised learning approach as they aim to encode input data in a condensed representation, while ensuring the preservation of essential semantic features through input reconstruction. Multimodal adaptations have been proposed (Ngiam et al. ÿÿÿÿ; Silberer et al. ÿÿÿÿ), with hidden representation layer taking as input both modalities, subsequently attempting to reconstruct them (see Figure ÿ.ÿ). However, training solely depends on the reconstruction loss, which results in a task-agnostic representation. Constraints (such as the corruption of the input) or subsequent supervised objective need to be set up to add desired properties (like robustness) to the multimodal representation (Silberer et al. ÿÿÿÿ).

Aÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ

Attention-based models are now well-known models that possess the ability to focus on a speciÿc part of the input, depending on the context. They are widely used since apart from increasing performance, they bring some interpretability to decisions, evaluating the importance of features.

Regarding multimodality, they have some interesting properties. weight to the ones with greater importance (Long et al. ÿÿÿÿ). This two-level impact is illustrated in Figure ÿ.ÿ. The advent of the Transformer architecture (Vaswani et al. ÿÿÿÿ) marked a signiÿcant shift in this domain. This type of encoder-decoder model has gained a massive interest, with numerous derivatives and impressive performances on applicative tasks across diÿerent modalities, e.g. in NLP (Devlin et al. ÿÿÿÿ) or in computer vision (Dosovitskiy et al. ÿÿÿÿ). Their building block, Multi-head Self-Attention mechanism, aims to learn a contextual representation Z of an input sequence X.

ÿÿ

Background and Related Work

Each attention head ÿrst maps the input X to a set of key K, value V and query Q matrices. The queries and keys are combined through a matrix product to produce attention weights (through a softmax function), representing the contextual interdependencies of the input elements. The values elements are ÿnally multiplied by these weights to produce the output representation Z. Formally:

Z = softmax 7 QK T √ d k ç V (ÿ.ÿ) = softmax 7 XW Q W T K X T √ d k ç XW V (ÿ.ÿ)
Here, d k denotes the dimension of queries and keys, while W Q , W K , W V represent weight matrices. Transformer encoder blocks are commonly used to learn contextual representations that can afterward be used for subsidiary tasks.

Remark (Preprocessing). It is essential to note that the variable X in Equation ÿ.ÿ is not typically raw input data, but rather the initial embedding of tokenization of X:

X = E(T (X)) (ÿ.ÿ)
where E is an embedding block and T a tokenizer. The considered input data hence does not need to be initially a sequence (as for textual data) to be processed by the Transformer: this sequential formatting is the task of a designed tokenizer. For instance, Visual Transformer (ViT) (Dosovitskiy et al. ÿÿÿÿ) uses small patches as tokens to represent an image (see Figure ÿ.ÿÿ).

ÿÿ . Multimodal learning

The intent of the embedding block is to map the sequence into an initial expressive latent space, typically achieved through a linear projection (Dosovitskiy et al. ÿÿÿÿ; Vaswani et al. ÿÿÿÿ). It is commonplace to fuse several types of embeddings at the token level, thereby injecting pertinent information into the model. For example, the Self-Attention mechanism, being invariant to the positioning of tokens within the sequence, can utilize absolute positional embeddings added to the initial token embeddings as an inductive bias for positional relevance information. Numerous works have sought to ascertain the most eÿective and eÿcient methodologies for computing these positional embeddings. The original transformer proposed in (Vaswani et al. ÿÿÿÿ) employs either learnable vectors or sinusoidal functions to oÿer absolute embeddings, with little noticeable variation in performance outcomes. The approach of absolute positional encoding through a learnable vector has been adopted widely in subsequent works (Devlin et al. ÿÿÿÿ; Z. Lan et al. ÿÿÿÿ; Radford, Narasimhan, et al. ÿÿÿÿ; Radford, J. Wu, et al. ÿÿÿÿ). (Shaw et al. ÿÿÿÿ) proposed to encode relative positions, predicated on the intuition that the distance between two tokens holds more signiÿcance than their absolute positions. In that case, learned relative positional embeddings based on the token distances are added to keys and values matrices during attention calculation (Equation ÿ.ÿ). This methodology has been replicated in subsequent studies (He et al. ÿÿÿÿ; Z. Huang et al. ÿÿÿÿ; Ke et al. ÿÿÿÿ; Raÿel, Shazeer, et al. ÿÿÿÿ).

For the multimodal framework, segment encoding may be incorporated at the token level in a similar manner to positional encoding, thereby informing the model of the token modality (G. Li et al. ÿÿÿÿ; L. H. Li et al. ÿÿÿÿ). This embedding fashion is illustrated in Figure ÿ.ÿÿ. (audio, visual and textual) to model cross-modal interactions in all modality pairs. In contrast, (R. Li et al. ÿÿÿÿ) proceed to a later fusion (c) by initially encoding intermediate modality-speciÿc representations for music pieces and ÿÿÿ-frames seed motion sequences, which are subsequently fused in a cross-modal transformer to learn the correspondence between both modalities and generate the future motion sequences. Advantages and limitations of studied approaches are summarized in Table ÿ.ÿ.

Pÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ

Apart from initial encodings at the token level and architecture design (Figure ÿ.ÿÿ), eÿective modeling of cross-modal interactions can be achieved through the pre-training objective. Originally developed for NLP and related tasks, transformers have revolutionized the ÿeld by enabling eÿective modeling of contextual dependencies within sequences. The trend of pre-training transformerbased models on vast quantities of unlabeled data using self-supervised objectives to learn general language knowledge has led to the development of large foundation models (see Section ÿ.ÿ).

These models demonstrate impressive performance across diverse tasks and exhibit robust generalization ability.

ÿÿ

. Multimodal learning Requires careful tokenizer and embedding layer designs.

Requires consequent computational power.

Self-supervised learning is a learning setting in which the objective is deÿned by the data themselves. Aside from having the advantage not to need labeled data, these methods force the models to learn representations that leverage the structure of the data, as it constrains the learning objective. For instance, it is common to try to predict a hidden part of the input:

• The seminal work (Devlin et al. ÿÿÿÿ) introduced the Masked Language Modelling loss (MLM), that needs the model to predict a masked token in a sentence.

• That approach has been adapted to other modalities. (Dosovitskiy et al. ÿÿÿÿ) hence explored a Masked Patched Prediction pre-training objective, consisting in predicting the mean color of corrupted image patches, while (Junkun Chen et al. ÿÿÿÿ) similarly mask some frames of speech inputs, and tries to reconstruct the initial sequence from the corrupted data.

These modality-speciÿc learning objectives have been quite straightforwardly extended to the multimodal framework, especially for unlabeled and unaligned datasets. Although the corresponding losses remain unimodal, the associated learning process leverages the cross-modal dependencies between multimodal inputs to gain information from the other modalities. It is also frequent to consider a general loss composed of the sum of modality-speciÿc losses. For instance, the VideoBERT (C. Sun, Myers, et al. ÿÿÿÿ) model's training consists in encoding textual and visual sequences and to predict masked tokens (either textual or visual) using modality-speciÿc input sequences and the MLM objective. Besides the text-only and video-only objectives, a third crossmodal objective is tackled: after encoding a bimodal sequence formed by the concatenation of textual and video sequences (see Figure ÿ.ÿÿ), the model shall predict if the two sequences are temporaly aligned using as input the CLS token. The global pre-training objective is composed of the sum of the three objectives.

In the case in which we possess aligned modalities however, we can use this alignment as a selfsupervised objective. A popular framework of SSL that is suited for aligned modalities is contrastive learning, which encourages representations of input data and their augmented views to be ÿÿ close in latent space, while pushing apart representations of diÿerent inputs. This way, the learned representations should be invariant to small perturbations, while encoding salient features. Given a context vector c, the popular InfoNCE loss (Oord et al. ÿÿÿÿ) uses categorical cross-entropy to identify the positive sample x drawn from the distribution p(x|c) from unrelated noises x 0 . This loss optimizes the negative log probability of classifying the positive sample correctly:

Background and Related Work

L InfoNCE = E ÿ log f (x, c) P x 0 2X f (x 0 , c) (ÿ.ÿÿ)
where f (x, c) estimates the density ratio p(x|c) p(x) .

ÿÿ . Multimodal learning

Despite the simplicity of the architecture, this self-supervised setting enabled to pre-train CLIP on ÿÿÿ million unlabeled image-text pairs. This results in impressive results, the model achieving for instance the same performances in Zero-Shot setting on ImageNet as a fully supervised ResNet ÿÿ. Besides CLIP, many works have explored the contrastive framework to pre-train multimodal architectures (Alayrac, Recasens, et al. ÿÿÿÿ; Bachman et al. ÿÿÿÿ; J. S. Chung et al. ÿÿÿÿ; Hjelm et al. ÿÿÿÿ; Miech et al. ÿÿÿÿ; C. Sun, Baradel, et al. ÿÿÿÿ).

In summary, the popularized paradigm consisting in pre-training transformers on self-supervised objectives has also been explored intensively in the multimodal paradigm. Taking inspiration from the BERT introduced Masking Language Modeling loss, these architectures' pre-training objectives mainly consist in reconstructing masked tokens from inputs, in cross-modal or modalityconditional fashions (J. Lu et al. ÿÿÿÿ; C. Sun, Myers, et al. ÿÿÿÿ). Moreover, in these approaches the alignment between diÿerent modalities also constitutes an interesting supervision. This alignment is even the main self-supervised objective of contrastive methods.

Cÿÿÿÿÿÿÿÿÿ

In conclusion, there has been extensive work in the realm of machine learning and deep learning for unimodal fault diagnosis. However, the area of multimodal diagnosis has been less thoroughly explored, reÿecting a signiÿcant gap in research that is waiting to be addressed. Notably, the ÿeld of fusion and multimodal representation learning has witnessed considerable development. These advancements are mostly driven by challenges in text/image tasks, with recent trends highlighting the supremacy of transformer-based architectures. However, the general framework for these multimodal transformer architectures can still be improved and adapted for new scenarios, such as multimodal fault diagnosis. Despite these advancements, the distinctive properties of the corresponding data in industrial applications, including the presence of unaligned and long temporal streams, add a layer of complexity. Furthermore, the multimodal nature of these streams has not been considered enough within the realm of fault diagnosis, hence there is a rich opportunity for exploration and development. In addressing this complex problem, it is vital to note that the heterogeneity gap remains a signiÿcant challenge. Given this, there is a pressing need to deÿne a new setting for this kind of data, an undertaking we will focus on in the subsequent chapter. In parallel, we will also introduce a new architecture, named StreaMulT, speciÿcally designed to confront these emerging challenges.

ÿÿ ÿ SÿÿÿÿMÿÿTÿ A Sÿÿÿÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿ Tÿÿÿÿÿÿÿÿÿÿ Fÿÿ Hÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ Aÿÿÿÿÿÿÿÿÿÿ Lÿÿÿ Sÿÿÿÿÿÿÿÿÿ Dÿÿÿ Cÿÿÿÿÿÿ'ÿ Sÿÿÿÿÿÿ
In this chapter, we tackle the new challenges posed by the rising complexity of Industry ÿ.ÿ systems, and their relation to fault detection and diagnosis tasks. We explore these challenges in a realistic environment that involves multi-source data streams from various modalities, including time series sensor measurements, machine images, and textual maintenance reports. These heterogeneous multimodal streams also diÿer in their acquisition frequency, may embed temporally unaligned information and can be arbitrarily long, depending on the considered system and task. Building on the previous chapter, wherein we examined principal approaches to multimodal fusion, we broaden our scope to this setting. We consider arbitrarily long multimodal streams in conjunction with related tasks, such as prediction across time. ÿÿÿ Iÿÿÿÿÿÿÿÿÿÿÿ

As explained in the previous chapters, the availability of massive amounts of data, coupled with recent ML breakthroughs oÿers great potential in numerous domains and particularly for the industry. More speciÿcally, in Industry ÿ.ÿ era, one major challenge is to exploit all information sources related to a system in order to perform data-driven diagnosis for corrective and predictive maintenances. To represent a typical example of studied industrial system, we consider an aircraft engine that is continuously running and from which we acquire feedback data of diÿerent modalities (numerical time series, raw text, images, sound, etc.) over time. For example, these modalities ÿÿ StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data can correspond to sensors measurements, textual maintenance reports, system photographs, system audio recordings, and so on. From these data, our goal is, depending on the task, either to detect if the system is in a faulty mode (fault monitoring) or to determine which fault is occurring (fault diagnosis). This setting is illustrated in Figure ÿ.ÿ. This paradigm comes with diÿerent challenges, from which we decide to consider the following:

• Heterogeneous modalities: The diÿerent sources of acquired data can come in diÿerent modalities, hence resulting in a heterogeneity gap issue when combining them. It is therefore relevant to develop methods aiming to narrow this gap, to exploit redundant and complementary information across modalities (see Section ÿ.ÿ).

• Heterogeneous acquisition frequencies: Despite their heterogeneous nature (end therefore structures), diÿerent sources of data generally possess their own acquisition frequencies. For instance on illustrated Figure ÿ.ÿ, numerical sensors measurements of physical quantities such as temperature, pressure, vibration or current signals, can be acquired at a regular high frequency, in the order of few seconds. On the other hands, system photographs, are also obtained at a regular acquisition frequency but with a greater period (say hours). Eventually, textual maintenance reports are acquired only every time following a maintenance, that is at a low and sporadic frequency.

• Unaligned modalities: The diÿerent acquired streams are generally not aligned on the temporal axis. Indeed, as illustrated on Figure ÿ.ÿ, a fault occurring at a speciÿc time step may be highly correlated with very recent sensors measurements or system photographs, while the related relevant information for the textual modalities would be contained in a much former report.

• Arbitrarily long input sequences: As introduced in the previous point, depending on the fault, the relevant part of the input data to perform fault monitoring or diagnosis can ÿÿ . Introduction be located far back in time for one modality, relatively to another one. Thus we consider that the acquired streams can be arbitrarily long and we do not bound them.

• Streaming mode: Depending on the level of criticality of the system, it can be imperative to perform the fault monitoring/diagnosis task with a relatively short response time. Plus, we can also imagine industrial systems that have to run uninterruptedly. In both cases it is either not desirable or not feasible to wait for the system to stop before executing the diagnosis module. Consequently, we consider as mandatory the ability to the designed approach to work in a streaming fashion, that is processing the input streams as they are acquired over time.

These diÿerent challenges have been tackled in the literature but separately to the best of our knowledge. If a large avenue of research exists in multimodal learning, and from now on recently mainly based on the Transformer architecture (see Subsection ÿ.ÿ.ÿ), the quadratic dependency of space and time complexities of the architecture with the input length limits its use for arbitrarily long inputs or streaming inference. By the mean of StreaMulT, we thus propose to tackle these ÿve problems jointly.

Cÿÿÿÿÿÿ'ÿ Cÿÿÿÿÿÿÿÿÿÿÿÿ

In this chapter, our contributions are threefold:

• Motivated by this industrial application and its key challenges, we formally deÿne a new applicative paradigm, in which one aims to solve a prediction task across time, from heterogeneous (by nature and acquisition frequency) multimodal sequential data and in a streaming fashion, hence handling arbitrarily long input data at both training and inference time.

• We then introduce StreaMulT, a Streaming Multimodal Transformer architecture based on cross-modal attention and conveying a memory bank to tackle these issues and deal with unaligned input streams.

• Due to the lack of a either public or private (within the MPO project) datasets adapted to our task, we propose to evaluate our model with the CMU-MOSEI dataset on a multimodal sentiment analysis task, in order to compare StreaMulT performances with previous approaches. It includes both multimodal and unaligned streams. We show that our model can deal with arbitrarily long sequences without suÿering from performance loss. When improving the textual pre-trained embedding, we even improve the state-of-the-art metrics on this dataset.

In Section ÿ.ÿ we formalize the multimodal setting with arbitrary long sequential data and we deÿne the positioning we decided to adopt to tackle the task of industrial diagnosis in this setting. We then review the connected works that brought us to develop the architecture of StreaMulT in Section ÿ.ÿ. We introduce the StreaMulT model in Section ÿ.ÿ, and we ÿnally conduct experiments on CMU-MOSEI dataset and ablation study in Section ÿ.ÿ. In order to avoid confusion between modality, sample, feature dimension and time indices, we use greek letters to index the modalities.

Let M 2 N be the number of considered modalities. For each modality µ, with µ 2 J1, M K, we consider the corresponding time series X α , indexed by time according to its own acquisition times and lying in its own deÿnition space:

X α := (X α (t)) t2Tα and 8t 2 T α , X α (t) 2 X α
where T α and X α are respectively the countable (possibly not ÿnite) set containing acquisition times of modality µ and its associated deÿnition space. We can for instance suppose real components without loss of generality, i.e. X α = R dα with d α the feature dimension.

Let X be the set deÿned as:

X := {X(t), t 2 R} where X(t) := (X 1 (s 1 )) s 1 ÿt s 1 2T 1 å . . . å (X M (s M )) s M ÿt s M 2T M (ÿ.ÿ)
The elements of X are basically M-tuples whose the µ th term is composed of the elements of the sub-sequence X α up to a speciÿc time step t that is common to all modalities.

A label space Y and the corresponding set of ground truth time steps T y are deÿned depending on the considered speciÿc task and on the input data. From these elements, one can construct a dataset D = x i , y i i=1,...,n composed of realizations of previously introduced random variables: 8i 2 J1, nK 8 < :

t i := T y [i], where T y [i] denotes the i th element of T y x i := X(t i ) y i := y(t i )
The global objective of this setting is thus to perform a supervised prediction task (classiÿcation or regression) on this dataset. Hence, given L : Y å Y ! R a loss function and F a function class, we aim to ÿnd f 2 F minimizing the associated empirical risk (see Equation ÿ):

f å = arg min f 2F Rn (f ) = arg min f 2F 1 n n X i=1 L(f (x i ), y i ) (ÿ.ÿ)

Example . Ideal fault diagnosis

In an ideal setting of fault diagnosis, one would like to be able to give a prediction of the state of the system in real time, that is, every time one acquires a new data point, from any modality. Hence in that case,

Y = ã {0, 1} for fault monitoring J1, CK for fault diagnosis , T y = S 1ÿαÿM T α ÿÿ .

Multimodal learning with heterogeneous and arbitrarily long sequential streams

The elements of the dataset D = (x i , y i ) are thus the M-tuples composed of subsequences of all modalities up to a time step t i , associated with y(t i ), the state of the system at time t i , where t i takes all possible values of data acquisition times among all modalities.

Example . Fault diagnosis with resources or user constraint

The ideal setting of fault diagnosis described above is nonetheless in general not realistic for inference, as the acquisition frequencies and the available resources can make the diagnosis module impossible to run in real time. In such a case, or if the user wants to put a speciÿc constraint on the time to output a prediction, T y can be constructed iteratively:

Algorithm Creation of custom T y with constraints.

t i 0 while data_acquisition do t i t i + 1 if condition_on_t i then T y T y [ {t i } end if end while
In Algorithm ÿ, the condition "data_acquisition" refers to the state of the data acquisition process and can be seen as the upper bound of the value of t i . Namely, this condition is set to True while t i has not reached the last time step of acquired time series for the training set, and can indeÿnitely be set to True for inference in streaming. The "condition_on_t i " includes all diÿerent constraints deÿned by the task or the user. For instance, in the current case of industrial diagnosis, with a resource constraint imposing a minimum of ÿÿ time steps between two predictions:

"condition_on_t i " = "t i 2 8 < : [ 1ÿαÿM T α 9 = ; " ^("T y = ;" _ "t i T y [ 1] 10")
in which ^represents the logical AND and _ represents the logical OR.

Remark. When the inference is realized in a streaming fashion, the construction of T y and the execution of the diagnosis module are simultaneous.

Example . Multimodal Sentiment Analysis

If we consider now a sentiment analysis task in which the objective is to assign a score from -ÿ to ÿ to each sentence contained in a long sequence (keeping past sentences in input), then for a sequence of s multimodal sentences, the associated ground truth time steps are the last acquisition time steps of each sentence:

Y = [ 3, 3], T y = ã max 7 M S α=1 T j α ç , 1 ÿ j ÿ s ÿÿ
StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data where j is the sentence index and T j α are the acquisition time steps of modality µ for sentence j.

We now describe the positioning we decided to adopt regarding this global problem.

ÿÿÿÿÿ Pÿÿÿÿÿÿÿÿÿÿ

To the best of our knowledge, the previous framework has never been introduced as such, hence the related task of Multimodal Fault diagnosis (addressing the ÿve challenges) has never been dealt with. Therefore, there is no existing and available public dataset to evaluate models on this task.

f = arg min f 2{h g|g2G,h2H} Rn (f ) (ÿ.ÿ)
Following the discussion pointing out the importance of the quality of a multimodal representation in Subsubsection ÿ.ÿ.ÿ, and its link to the subsidiary prediction performances, we mainly focused our research work on ÿnding an architecture dealing with data presented in Section ÿ.ÿ and maximizing their multimodal representation, in a task-agnostic manner. The only assumption we make on the considered task is that we are in a supervised setting.

ÿÿÿ Rÿÿÿÿÿÿ ÿÿÿÿ While Transformers architectures have been widely used on numerous applicative tasks, we show in Subsection ÿ.ÿ.ÿ that their complexity prevents them to cope with long inputs or to run in a streaming fashion as such. We present some variants addressing this limitation in Subsection ÿ.ÿ.ÿ.

ÿÿÿÿÿ Tÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ Classical approaches dealing with multimodal sequential data, such as RNN-based architectures, do not tackle the unalignment issue (Zadeh, P. P. Liang, et al. ÿÿÿÿ; Zadeh, Poria, et al. ÿÿÿÿ), and ÿÿ . Related work hence consider the input multimodal data are temporally aligned. Furthermore, the autoregressive nature of these architectures generally implies to consider same acquisition timesteps along diÿerent modalities. Multimodal Transformer (Tsai et al. ÿÿÿÿ) adresses both these issues, taking advantage of its crossmodal transformer modules, that aims to learn a contextual and cross-modal representation of unaligned input sequences as depicted in Figure ÿ.ÿ. At the heart of this module, cross-modal attention blocks indeed express a target modality µ with raw features from a source modality . Formally, considering our input sequences X α and X β from modalities µ and , the cross-modal attention for X α attending to X β , denoted X β!α is computed as:

X β!α : = softmax Q α K T β p d k ! V β (ÿ.ÿ) = softmax X α W Qα W T K β X T β p d k ! X β W V β (ÿ.ÿ)
with Q α the query matrix for modality µ, K β , V β the key and value matrices for modality ; W Qα , W K β , W V β being learned weights, and d k being the common embedding dimension for query and key matrices. Unalignment is mainly handled by the matrices product

ã Q α K T β ;
which sets the receptive ÿeld of cross-modal interactions to the entire input sequences X α and X β , hence enabling longrange dependencies modeling, whereas prior works ÿrst realign multimodal sequences with the same length and then use autoregressive nature of a model (such as RNN) to iteratively fuse crossmodal information. This makes these approaches inadequate for asynchronous modalities, and less eÿective, as intermodal interactions are only computed through a compressed hidden state, resulting in a loss of information for long-range dependencies. This cross-modal alignment can be viewed as a step diagonal activation in the cross-modal attention matrix, as pictured in Fig-

ÿÿ

StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data ure ÿ.ÿ, hence as a temporal monotonic attention. Another drawback of these models lies in their autoregressive nature, making it diÿcult to parallelize. However, due to the arbitrarily long size of input sequences in our setting, Multimodal Transformer architecture faces two main issues. Training is intractable due to its space and time complexities, and inference cannot be done in a streaming way, as the vanilla model needs the whole sequence as input to compute the relative matrix product. The construction of eÿcient transformers is actually a well studied subject, as stated in a recent survey (Tay et al. ÿÿÿÿ). Indeed, the self-attention module essentially implies to compute the product of two l å l matrices (where l is the length of the input sequence), and hence has a complexity in O(l 2 ). Thus, many works try to reduce this quadratic complexity, up to a linear one, in order to speed up computation time or to enable longer history for input data. These approaches approximate the full quadratic-cost attention matrix by adding some sparsity to it, using essentially either Low-rank methods (Sinong Wang et al. ÿÿÿÿ), ÿxed (Child et al. ÿÿÿÿ) or learned (Kitaev et al. ÿÿÿÿ) sparsiÿcation patterns, side memory modules (Zaheer et al. ÿÿÿÿ), kernalization (Katharopoulos et al. ÿÿÿÿ), or recurrence (Dai et al. ÿÿÿÿ).

Remark. From this point, and until the end of the chapter, for the sake of clarity we adopt new notations:

• X i will denote the i th segment of input X, whereas X i,α will refer to the i th segment of modality µ of input X ÿÿ . Related work

• X l will refer to the value of variable X at the layer l

• M α will be used to refer to the memory bank of modality µ, whereas M still refers to the number of modalities

• n will be used to refer to the number of segments, rathen than the number of samples ÿÿÿÿÿ Sÿÿÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿ

To our knowledge, none of the previous papers (Tay et al. ÿÿÿÿ) (so-called Eÿcient Transformers) yet considered arbitrary long or streaming data frameworks. This is an issue, as even a matrix whose computation complexity is linear in the input length becomes intractable for very long sequences. In the same way, for input sequences acquired on the ÿy, modeling inter-modalities dependencies with a cross-modal Transformer requires to recompute the whole attention matrix, which is also intractable. On the other side, some prior works focus on dealing with streaming scenarios, although unimodal. That is the case of papers addressing Automatic Speech Recognition (ASR), or Simultaneous Machine Translation (SMT) tasks, for which there is a need to keep a relevant temporal information ÿow, coupled with a necessary low latency at inference. This results in a quality-latency trade-oÿ, in which the model needs to produce an output with only a partially available input sequence to ensure low latency. If some works choose to mask previous and future contexts using a sliding window (Moritz et The main drawback of the former strategy is that the receptive span of the self-attention is linearly growing with the number of transformer layers (see Figure ÿ.ÿ), implying more latency ; while the issue of the latter strategy is on the contrary that the relation between diÿerent chunks is lost, undermining the performances of the model as long-range dependencies cannot be computed. ÿÿ

StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data

To alleviate the issue of chunk-wise methods, (C. Wu et al. ÿÿÿÿ; Yeh et al. ÿÿÿÿ) add a memory bank of multiple slots to this architecture, aiming to store salient history from long-range history. Thus, whereas a recurrent-connection-based approach such as Transformer-XL (Dai et al. ÿÿÿÿ) can only attend to a segment that is k steps away after O(k) steps, Augmented-memory Transformer (AM-TRF) (C. Wu et al. ÿÿÿÿ) can already attend to previous segments embeddings through attention performed on memory bank. Formally, input sequence is ÿrst chunked into non-overlapping smaller segments (C i ) i≥0 , which are concatenated with left L i and right R i context blocks to prevent boundary eÿects, hence forming contextual segments X i := [L i : C i : R i ]. Considering a contextual segment X i and a memory bank M i = [m 1 , . . . , m i-1 ] containing compressed information from previous segments, the output X n+1 i of the n-th layer is computed as:

Xn i = LN(X n i ) K n i = W k [M n i , Xn i ] V n i = W V [M n i , Xn i ] Q n i = W Q Xn i [Z n L,i : Z n C,i : Z n R,i ] : = Attn(Q n i , K n i , V n i ) + X n i Xn+1 i = FFN LN [Z n L,i : Z n C,i : Z n R,i ] X n+1 i = LN ã Xn+1 i + [Z n L,i : Z n C,i : Z n R,i ]
;

m n i = Attn(W Q s n i , K n i , V n i )
ÿÿ

. Proposed model where s n i is the mean of C n i and LN, FFN, Attn respectively correspond to Layer Normalization, Feed-Forward and Attention layers. After passing through all N layers, outputs corresponding to left and right contexts are discarded to keep only center segments representations

(C N i ) i≥0 . Fig- ure ÿ.ÿ illustrates this architecture.
Emformer architecture (Shi et al. ÿÿÿÿ) is an improved version of AM-TRF, in the sense that it performs attention on the memory bank from the lower layer, hence dumping its autoregressive nature and becoming parallelizable during training. Besides, it considerably reduces the amount of computation by caching Key and Value matrices from previous segments, and optimizes global performance by cutting oÿ some dependencies during self-attention computation. The two approaches mainly diÿer in the content of M n i , that contains summarized information from lower layer in Emformer (enabling to parallelize the computations on all layers); and in the cached keys and values from previous segments. All these optimized changes render the architecture more eÿcient and prone to work in a streaming scheme. Figure from (Shi et al. ÿÿÿÿ).

ÿÿÿ Pÿÿÿÿÿÿÿ ÿÿÿÿÿ

We propose StreaMulT, a Streaming Multimodal Transformer architecture, taking advantages of both a cross-modal attention mechanism and a block processing approach to tackle the diÿerent challenges of this framework. Finally, we optimize the training scheme of the model to lower space complexity, training time and enabling inference short-time response at the same time.

Our global end-to-end architecture combines beneÿts from block processing and cross-modal attention. The architecture is illustrated in Figure ÿ.ÿ. We describe here the processing of the data of modality µ, with 1 ÿ µ ÿ M . X α is ÿrst passed through a ÿD convolutional layer aiming to model some local temporal structure, and map all modalities to a common feature dimension d. Segment bounds are then ÿxed. Extending the block processing method to input data with heterogeneous sampling rates, we deÿne hard segment bounds with respect to the temporal axis, hence producing shared segments across modalities, as illustrated in Transformer. Diÿerent colors represent heterogeneity nature of diÿerent modalities, and shadings represent cross-modal features. Each modality-speciÿc time series is passed through a ÿDconvolutional layer, and then through a unimodal Emformer block to initialize its modalityspeciÿc memory bank. Cross-modal interactions are then captured through SCT blocks, that express a target modality with the help of source modalities' features and memory banks. Target modalities representations computed from diÿerent source modalities are then concatenated and passed through modality-speciÿc Transformer encoders, that output contextual crossmodal representations, summarizing the whole sequences. These outputs are then processed by a ÿnal FFN to produce the prediction. contextual segments X α,i = [L α,i : C α,i : R α,i ] are processed in a parallel way. They are ÿrst given to a modality-speciÿc Emformer module to initialize its own modality memory bank M α . Then, each source modality / target modality ( / µ) pair ( 6 = µ) is processed by its own Streaming Cross-modal Transformer (SCT) module. Speciÿcally, each segment from the target modality X α,i is expressed using the same temporal segment from the source modality X β,i along with the source modality memory bank M β,i . For each layer n:

h Ĉn α,i , Rn α,i i = LN( å C n α,i , R n α,i å ) (ÿ.ÿ) h Ĉn β,i , Rn β,i i = LN( å C n β,i , R n β,i å ) (ÿ.ÿ) K n β,i = å K n M,β→α,i , K n L,β→α,i , K n C,β→α,i , K n R,β→α,i å (ÿ.ÿ) V n β,i = å V n M,β→α,i , V n L,β→α,i , V n C,β→α,i , V n R,β→α,i å (ÿ.ÿ) Z n C,β→α,i = Attn(Q n C,β→α,i , K n β,i , V n β,i ) + C n β→α,i (ÿ.ÿÿ) Z n R,β→α,i = Attn(Q n R,β→α,i , K n β,i , V n β,i ) + R n β→α,i (ÿ.ÿÿ) h Ĉn+1 α,i , Rn+1 α,i i = FFN(LN([Z n C,β→α,i , Z n R,β→α,i ])) (ÿ.ÿÿ) å C n+1 α,i , R n+1 α,i å = LN( h Ĉn+1 α,i , Rn+1 α,i i + [Z n C,β→α,i , Z n R,β→α,i ]) (ÿ.ÿÿ) ÿÿ . Proposed model Figure ÿ.ÿ:
Block processing for Multimodal learning in a streaming scheme. For modality µ: X α , C α,i , L α,i and R α,i respectively correspond to the full input sequence, the initial i-th block, and the left and right contexts associated to this block to form the contextual i-th segment. s α,i corresponds to the mean of current segment C α,i . Blue area represents an initial block for modality while the pink one represents a contextual segment for modality .

where,

å K n M, →µ,i , K n C, →µ,i , K n R, →µ,i å = W k, →µ h M ,i , Ĉn ,i , Rn ,i i (ÿ.ÿÿ) å V n M, →µ,i , V n C, →µ,i , V n R, →µ,i å = W v, →µ h M ,i , Ĉn ,i , Rn ,i i (ÿ.ÿÿ) å Q n C, →µ,i , Q n R, →µ,i å = W q, →µ å C n →µ,i , R n →µ,i å (ÿ.ÿÿ) and ã K n L,β→α,i , V n L,β→α,i
; are the key and value copies (cached) corresponding to previous segments, up to left context size. This module is illustrated in Figure ÿ.ÿ.

After the last layer N , right contexts representations (R N β→α,i ) i>0 are discarded. (C N β→α,i ) i>0 are concatenated to form the ÿnal cross-modal representation X β→α . We then concatenate along the feature dimension all cross-modal outputs corresponding to the same target modality µ in a

vector Z α := 0 B B B B B B @ X 1→α • • • X α-1→α X α+1→α • • • X M →α 1 C C C C C C A
, that is given as input to a Transformer Encoder exploiting sequential nature of data, to produce modality output y α . All modality outputs are eventually concatenated and passed through a ÿnal fully-connected layer to output prediction ŷ.

Tÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ

The main motivation to design StreaMulT architecture is to handle the arbitrarily long nature of considered multimodal input sequences. In that sense, the block processing mechanism we use aims to alleviate the quadratic complexity of cross-modal attention modules, similarly to several speech recognition works (Shi et al. ÿÿÿÿ; C. Wu et al. ÿÿÿÿ). However, these applications focus on getting short-time response at inference to perform simultaneous speech translation or recognition and hence essentially diÿer from our framework. Indeed, to handle very long sequences we are at least as concerned about space complexity as time complexity. We thus cannot train our model in the same fashion as these approaches, that is by parallelizing on all input segments the cross-modal attention computation. This indeed still implies a quadratic space complexity to store cross-modal attention weights matrix. To fulÿll both space capacity and eÿcient training time constraints, we introduce a ÿexible training scheme. This is illustrated in Figure ÿ.ÿÿ. More speciÿcally, at training time we parallelize operations of Memory bank initialization and Streaming Cross-modal Transformer modules on subsequences of h consecutives segments. h is chosen in an empiric way, as the highest integer enabling one's memory capacity to run the model. This training scheme enables StreaMulT to run arbitrarily long sequences by only storing limited-size matrices, while still beneÿting from simultaneous computations through parallelization. Space and time complexities for diÿerent layer types are derived in Section ÿ.ÿ. Note that we do not change the segment length but rather concatenate them in a single matrix product. This enables to keep short segments at inference and thus still work in a short-time response for streaming application. VAREP (Degottex et al. ÿÿÿÿ). Textual features are also extracted from words transcripts, using GloVe (Pennington et We could not reproduce the results shown in the paper, hence we present the results we obtained, that are not as good as the given ones. All scores from our experiments are averaged on ÿ runs. The corresponding results are represented in the upper part of following Table ÿ.ÿ. This shows that our architecture globally reproduces the results of Multimodal Transformer (even performs a little bit better on some metrics), which highlights the availability of its memory bank to properly convey salient information through time, as StreaMulT receptive ÿeld only attends to segments of length ÿÿ, while MulT attends to whole sequence of length ÿÿ. We then decided to use contextual pre-trained embedding layers for textual modality, namely BERT (Devlin et al. ÿÿÿÿ) and BART (Lewis et al. ÿÿÿÿ). The corresponding results are described in the lower part of Once again, the usage of a contextual pre-trained embedding layer signiÿcantly improves performances. The Multimodal Transformer architecture coupled with a BERT embedding layer now equals the performances of SOTA MMIM model on several metrics, questioning the real ÿÿ . Experiments improvement on the Multimodal Sentiment Analysis task over the last three years. Besides, it emphasizes the power of language models, which is supported by the performances of MulT-BART, deÿning a new SOTA for several metrics on this dataset.

We ÿnally simulated arbitrarily long sequences by concatenating all video clips related to the same speaker and considering these as inputs streams. In this setting, StreaMulT architecture successfully parallelizes its training along segments and handles long sequences at inference in a streaming way. On the other side, Multimodal Transformer faces memory issue. To qualitatively validate our architecture, we also plot the heatmap of the diÿerent attention weights of the model in Figure ÿ.ÿÿ. This plot represents the diÿerent attention weights of the Streaming Cross-modal Transformer related to the visual/textual modalities, for a multimodal sequence of length ÿÿ. For consistence with previous notations, we call µ the visual modality and the textual modality. On the x-axis, the key matrix K β is organized as: [memory bank; right contexts; segments utterances]. On the yaxis, the query matrix Q α is organized as: This ÿgure ÿrst reminds us, as stated in (Tsai et al. ÿÿÿÿ), that language sequences are unaligned across modalities. This is indeed shown by the several activations on vertical lines (diÿering from a temporal monotonic diagonal line), corresponding to speciÿc word embeddings correlated to many visual frames. If some of these unalignments remain in the scope of the same temporal segment, as illustrated in ÿÿ StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data the fourth segment by the green box, the access to the memory bank enables the model to attend beyond the current segment and to catch unalignments at longer range, as illustrated in the third segment by the yellow boxes. The yellow box on the right witnesses the unaligned dependencies within the third segment, while the left yellow box illustrates that some textual features of the past history activate the visual frames of the current segment. These diÿerent behaviors show the ability of the StreaMulT architecture to adapt its strategy depending of the context, attending to unaligned data from past history via memory bank when necessary.

ÿÿÿÿÿ Aÿÿÿÿÿÿÿ ÿÿÿÿÿ

We conducted some ablation experiments to assess for the importance of speciÿc parts of the model or of the data. The results of these experiments are displayed in Table ÿ.ÿ. Speciÿcally, we tried to highlight the importance of each modality for the considered MSA task by sequentially leaving it out.

While omitting sound or images streams does not aÿect much the performances of the model (less than 1% loss in binary accuracy and F 1-score), the absence of textual modality results in an impressive drop of more than 15% in binary accuracy and Fÿ-score, that cannot be compensated by visual and audio modalities. Table ÿ.ÿ derives the diÿerent time and space complexity classes for diÿerent types of layers, along with the number of sequential operations. Vanilla self-attention layers have a quadratic complexity both in time and in space, which is problematic for handling long sequences. Similarly, cross-modal attention, as deÿned in (Tsai et al. ÿÿÿÿ) also has a quadratic complexity in the sequence length. More precisely, the complexity class depends of the product of the two modalities lengths n α , .n β , as they can diÿer. Streaming Cross-modal Attention modules trained in regular fashion for blocks processing (as in ÿÿ . Implementation details (Shi et al. ÿÿÿÿ)) have the same space and time complexity classes, which make them intractable for arbitrarily long sequences. This indeed requires to compute the matrix product of

Q α 2 R dq µ ×d and K β 2 R d k ×d , with d qµ = n seg .(R α + C α + 1) and d k = n seg .(R β + C β + l mem ). n seg
is the number of segments of the input sequence, R α and R β correspond to the length of right contexts for modalities µ, , and C α and C β to the length of their central segments. Last, l mem corresponds to the length of a memory cell. We suppose that R and l mem are negligible before C, and noting that n seg = nµ Cµ = n C , one obtains the results mentioned above. If we train this layer in the ÿexible scheme as described in Section ÿ.ÿ, for each subsection of h consecutive segments we need to handle the product of matrices

Q α 2 R dq µ ×d and K β 2 R d k ×d , with now d qµ = h.(R α +C α +1) and d k = h.(R β +C β +l mem ), which has a time complexity class of O(h 2 .C α .C β .d).
As mentioned in the third column, to process the whole sequence we need to perform nµ hCµ sequential operations, which also derives the whole time complexity class. Note that the space complexity now only depends on h, C and d, as we only need to store a subsequence at a time. At inference, one can thus choose h = 1 to process the input sequence in streaming, enabling a short-time response with time and space complexity classes being respectively

O(C α .C β .d) (for one segment) and O(C α .C β + C α .d + C β .d).
ÿÿÿ Iÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ

We now describe the diÿerent parts of the implementation of the StreaMulT algorithm for the example of Multimodal Sentiment Analysis. Require: text, audio, vision, state

X t , X a , X v [ConvÿD t (text), ConvÿD a (audio), ConvÿD v (vision)] state, X t , X a , X v Emformer t (X t ), Emformer a (X a ), Emformer v (X v ) Z a→t , state SCT a→t (X t , X a , state) Z v→t , state SCT v→t (X t , X v , state) Z t [Z a→t : Z v→t ] Z t resize_segments(Z t ) Z t TransformerEncoder t (Z t ) Z t→a , state SCT t→a (X a , X t , state) Z v→a , state SCT v→a (X a , X v , state) Z a [Z t→a : Z v→a ] Z a resize_segments(Z a ) Z a TransformerEncoder a (Z a ) Z t→v , state SCT t→v (X v , X t , state) Z a→v , state SCT a→v (X v , X a , state) Z v [Z t→v : Z a→v ] Z v resize_segments(Z v ) Z v TransformerEncoder v (Z v ) Z [Z t [ 1] : Z a [ 1] : Z v [ 1]] preds projection_layer(Z) return preds, state
In Algorithm ÿ, the function sequence_to_segments_batches splits the input batches of long sequences into smaller segments batches whose size is controlled by the parameter memory_batch_size, depending on the available memory of the hardware (this is illustrated by the batches of h parallelized segments in Figure ÿ.ÿÿ). The variable "state" is initialized as None and will contain the diÿerent memory banks, along with the cached left contexts (keys and values). The forward loop of the model is detailed in Algorithm ÿ.

In Algorithm ÿ, the diÿerent unimodal segment_batches are passed through unimodal Emformers to initialize memory banks and get a ÿrst intramodal representation. All cross-modal representations Z α→β are then obtained through related SCT modules, which also update the content of the variable "state". The function "resize_segments" splits the diÿerent segments_batches into segments, from which contextual representations are learned thanks to a modality-speciÿc Transformer Encoder. A last projection module composed of feed-forward layers with residual connections and dropout regularization (for training) produces the ÿnal representations, from which the predictions related to these segments are obtained thanks to an usual classiÿer (a linear layer).

ÿÿ . Implementation details

: central_segments β ])] Q β→α X β→α W Q →µ K β , V β split(X β W KV ) K β [K β [: mem_size+rc_size] : cached_K β : K β [ central_segments_size :] V β [V β [: mem_size+rc_size] : cached_V β : V β [ central_segments_size :] Q β→α , K β , V β reshape_multihead_scaling(Q β→α , K β , V β ) a weights attention_mask(Q β→α (K β + rpe k ) T ) a probs dropout(softmax(a weights )) output a probs (V β + rpe_v) X β→α , state after_attention_operations(output) end for Z β→α X β→α return Z β→α , state
In Algorithm ÿ, a cross-modal representation Z β→α is computed from unimodal input streams X α , X β , along with the variable "state" that contains global information such as memory banks or cached keys and values from previous segments (used as left context). Therefore, at the beginning of each layer, right context blocks and central segments are extracted from the input streams used for queries and keys/values. Summary vectors are then computed for query stream as a temporal average pooling of each segment. The queries and keys/values input streams are then reordered on temporal axis, respectively as [right_context_blocks, central_segments, summary] and [memory_bank, right_context_blocks, central_segments], in order to compute all attention weights in a single matrix product. Matrices Q β→α , K β and V β are thus computed thanks to linear projection layers (K and V are computed as once and split in two halves), and cached keys and values are concatenated at the relevant time steps on temporal axis. As its name suggests, the function "reshape_multihead_scaling" reshapes these matrices along the feature dimension axis to perform multihead-attention, and rescales their corresponding elements by the factor p d k (see Equation ÿ.ÿ). The queries/keys matrix product is then computed, with an additive term "rpe_k" in the key matrix corresponding to relative positional embeddings, implemented in the same way as in (Shaw et al. ÿÿÿÿ). "rpe_k" and "rpe_v" are obtained as linear projections of a distance matrix "rpe", global for the whole StreaMult architecture. An attention mask is also applied to ensure the fact that queries attend to relevant keys. At the end, the output representation of X β→α is fed to several output layers (feed-forward layers, residual connections, layer normalizations; see Equation ÿ.ÿÿ and Equation ÿ.ÿÿ) and is given as input for the next SCT layer. The training has also been realized with Adam optimizer (Kingma et al. ÿÿÿÿ), early stopping procedure and gradient clipping. Dropout is frequently used throughout the network, mostly with a weight of 0.1.

Cÿÿÿÿÿÿÿÿÿ

In this chapter, we introduced StreaMulT, a model that merges the power of cross-modal attention for multimodal representation and the eÿciency of the block processing approach to manage arbitrarily long sequences in a streaming manner. In doing so, Strea-MulT eÿectively responds to the novel challenges of Multimodal Learning with heterogeneous and arbitrarily long sequential streams-a task that previous approaches have struggled with. Experiments carried out on the CMU-MOSEI dataset demonstrated promising results, with a notable enhancement in state-of-the-art metrics and a demonstrated capacity to handle arbitrarily long data during training and process sequences in a streaming manner during inference. The paradigm has numerous applications such as Industrial Monitoring, which necessitates an adapted dataset for benchmarking future related works. A main drawback of StreaMulT and similar multimodal architectures though, is that we do not control how the cross-modal interactions are captured through the learned representations. Thus, in the next chapter we present some thoughts on the characterization of relevant information across modalities. ÿÿ ÿ Tÿÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ Iÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

Cÿÿÿÿÿÿ'ÿ Sÿÿÿÿÿÿ

This chapter presents a discussion of diverse multimodal interactions, rather than advancing a speciÿc contribution. It begins by decomposing the relevant content into redundant and complementary types of information. Subsequently, it delves into the exploration of research focused on maximizing redundant information, predominantly within the multiview setting, and the frameworks employed therein. The ÿnal section attempts to broaden these approaches to encapsulate the characterization of complementary information, and articulates critiques of both existing methodologies and the deÿcit of evaluation benchmarks. This analysis oÿers a comprehensive understanding of the ongoing challenges and potential paths forward in the ÿeld of multimodal learning.

ÿÿÿ Iÿÿÿÿÿÿÿÿÿÿÿ

The preceding chapter delved into the development and understanding of StreaMulT, a streaming multimodal transformer capable of managing arbitrarily long, unaligned heterogeneous data streams. This innovative model, like its multimodal counterparts, attempts to model relationships between diÿerent modalities. It does so in a supervised manner, employing the powerful backpropagation algorithm to devise meaningful and insightful latent multimodal representations. The pragmatic capability of the StreaMulT architecture has been underscored, particularly in relation to handling voluminous, unaligned, and diverse data streams. However, as we turn the page onto this chapter, our focus shifts subtly, yet signiÿcantly. While previous models, including StreaMulT, have oÿered valuable contributions to multimodal learning, an under-explored area has emerged -the lack of models that rely on well-deÿned theoretical tools and assumptions, such as mutual information losses, to leverage and control complementary information between modalities.

In a multimodal setting, various modalities often bring forward information that may appear redundant on the surface. Many multimodal models, accordingly, tend to focus primarily on multi-view settings where the redundant information is the primary target. This, indeed, is a valid and essential task, as redundant information is assumed to essentially be relevant for downstream prediction tasks. However, in doing so, we should not lose sight of another equally crucial aspect the potential complementarity that exists between di erent modalities. Indeed, diÿerently to multi-view settings in which inputs generally consist in variations of a same scene (such as data augmentations or diÿerent point of views), we may also be interested in diÿerent multimodal ÿÿ Thoughts on the characterization of Information across modalities settings, in which input may describe a scene at diÿerent scales or times. In that conÿguration, diÿerent modalities may share less (down to none) information, making the exploitation of complementary information crucial.

Complementarity, in this context, refers to the unique and supplementary information that diÿerent modalities may bring to the table, which could be key to building a more comprehensive understanding of the data at hand. Complementary information, when eÿectively utilized, may not only enhance the richness of multimodal representations but also bring insights that could potentially be overlooked otherwise.

The signiÿcance of harnessing the potential of complementarity in multimodal learning is clear. However, how to integrate such a notion into our existing models in a theoretically robust and practical way is a challenge yet to be fully tackled.

This chapter reÿects our exploration into this very challenge, without delving into speciÿc experiments. It encapsulates an important question that has persistently emerged throughout the course of this thesis work: How can we e ectively integrate complementarity into multimodal learning, and how should we measure the performance of such endeavors? This exploration is crucial and deserves to be highlighted here, as it forms the groundwork for future investigations and implementations.

ÿÿÿ Tÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿ For the sake of clarity, we limit the scope of this chapter to a setting with only two modalities. However, the discussions and conclusions outlined here are applicable to any number of modalities. Thus, we restrict the setting introduced in Subsection ÿ.ÿ.ÿ to M = 2 modalities, where data points are represented as x = (x (1) , x [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF] ) and modeled by random variables (X 1 , X 2 ) 2 X = X 1 å X 2 . We denote F i , G i , H i as the restriction of the classes of functions F, G, H (respectively) to the unimodal input space X i , for i 2 {1, 2}. The multimodal fusion framework is motivated by the fundamental assumption:

min f ∈F R(f ) ÿ min 7 min f 1 ∈F 1 R(f 1 ), min f 2 ∈F 2 R(f 2 ) ç (ÿ.ÿ)
that is, that considering more modalities is beneÿcial for a task. Noting ( ĥ, ĝ) and ( ĥi , ĝi ) the empirical risk minimizers learned on (H, G) and (H i , G i ) for i 2 {1, 2}, respectively, (Y. Huang et al. ÿÿÿÿ) show that :

R( ĥ ĝ) ÿ min i=1,2 " R( ĥi ĝi ) + ;(ĝ) ;(ĝ i ) + O( r 1 n ) # . (ÿ.ÿ)
where ;(g) is the latent representation quality introduced in Equation ÿ, and n is the sample size of the training dataset. As X = X 1 å X 2 , for i 2 {1, 2} any candidate g i 2 G i can be retrieved in G, and thus ;(ĝ) ;(ĝ i ) ÿ 0. In essence, this proposition argues that for a sufÿcient sample size n, the inclusion of more modalities enhances performance on learning tasks,

ÿÿ

. Theoretical background and this enhancement is measured by the quality of its latent representation. This hypothesis appears quite intuitive: augmenting the number of modalities leads to an increased data availability, which can help the model in reÿning its predictions. The additional modality either reinforces the model's current belief (thereby increasing its predictive conÿdence) or introduces a novel element to the input data. This new element, coupled with information from other modalities, may alter the model's belief, reducing its conÿdence in the prior prediction and perhaps even producing a change in the prediction itself. Therefore, we can classify this ÿux of information as either redundant or complementary. The information theory, as proposed by (Shannon ÿÿÿÿ), provides a solid framework to formalize these concepts.

ÿÿÿÿÿ A ÿÿÿÿÿ ÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

Considering two random variables X and Y , the mutual information between X and Y is deÿned by :

I(X; Y ) = H(X) H(X|Y )
as the diÿerence between the entropy of X and the conditional entropy of X given Y . In the context of information communication, I(X; Y ) quantiÿes the average reduction in bits required to encode X given knowledge of Y , compared to the scenario where Y is unknown. As entropy measures the uncertainty of a random variable's value, I(X; Y ) can also be interpreted as the reduction in uncertainty about one variable's value when the other is observed. In our multimodal context, we use the mutual information operator to measure the interdependencies between modalities X 1 and X 2 , and between these modalities and the prediction task at hand represented by the random variable Y .

From there, one can deÿne redundancy between modalities, as in (Federici et al. ÿÿÿÿ):

Dÿÿÿÿÿÿÿÿÿ

De nition (Redundancy). X 1 is redundant with respect to X 2 for Y if and only if I(Y ; X 1 |X 2 ) = 0. If we also have I(Y ; X 2 |X 1 ) = 0, we say that X 1 and X 2 are mutually redundant.

The redundancy between modalities X 1 and X 2 for Y can thus be measured by I(X 1 ; X 2 ; Y ). It corresponds to the quantity of predictive information shared by both modalities.

Inversely, we deÿne the complementarity of one modality relative to another as follows:

Dÿÿÿÿÿÿÿÿÿ De nition (Complementarity). X 1 is complementary with respect to X 2 for Y if and only if I(Y ; X 1 |X 2 ) > 0.
The complementarity between modalities X 1 and X 2 for Y can thus be measured by

I(X 1 , X 2 ; Y ) I(X 1 ; X 2 ; Y ) = I(Y ; X 1 |X 2 ) + I(Y ; X 2 |X 1 )
. It corresponds to the ÿÿ Thoughts on the characterization of Information across modalities quantity of predictive information that is modality-speciÿc, hence not shared by both modalities.

In the rest of the chapter, we focus on these two parts of the information. We ÿrst review works that focused on maximizing the redundancy across modalities, and then discuss the limitations of current multimodal approaches when characterizing the complementarity.

ÿÿÿ Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ

Numerous studies in the ÿeld of multimodal learning have aimed to exploit redundant information across modalities to construct more expressive latent representations. This is particularly the case of all works concentrating on multi-view scenarios, where redundancy is inherently assumed between the two views. In the multi-view learning paradigm, the input variable is partitioned into two diÿerent views X 1 and X 2 and there is a target variable Y of interest. As a consequence, it is highly connected to our multimodal setting in which X 1 and X 2 are two diÿerent modalities of a same observed phenomenon.

As formulated by (Sridharan et al. ÿÿÿÿ):

Assumption (Multi-view assumption). There exists an / inf o > 0 such that:

I(Y ; X 2 |X 1 ) ÿ / inf o and I(Y ; X 1 |X 2 ) ÿ / inf o
The Multi-view assumption states that (on average) if we already know X 1 , then there is little more information that we could gain about Y from observing X 2 (and vice-versa). This small potential gain is quantiÿed by / inf o . This hypothesis is however generally not assumed (for a small / inf o ) in the multimodal setting, as diÿerent modalities, compared to diÿerent views of a same scene, may contain a non-negligible quantity of modality-speciÿc information that is of use for prediction. Building on this assumption, various frameworks have been developed to capitalize on this information without requiring supervision. Many studies, for instance, employ the self-supervised paradigm and particularly the contrastive learning framework, conjecturing that "a powerful representation is one that models view-invariant factors" (Y. Tian, Krishnan, et al. ÿÿÿÿ). These works, driven by the InfoMax principle (Linsker ÿÿÿÿ), aim to bring representations of diÿerent views closer to each other and hence maximize mutual information between them (Bachman et (Alayrac, Recasens, et al. ÿÿÿÿ) extend this framework to the multimodal setting, using Info NCE loss (Oord et al. ÿÿÿÿ) between the modality representations of videos (audio, visual, textual modalities) in a shared latent space.

Concurrently, alternative strategies have been proposed to reÿne this approach by discarding superÿuous information. These strategies mainly build on the concept of a suÿcient representation (Achille et al. The mutual information between X and its representation Z can then be decomposed as follows:

I(X; Z) = I(Y ; Z) + I(X; Z|Y ) (ÿ.ÿ)
Proof. Using the multivariate mutual information chain rule (Cover ÿÿÿÿ), we have:

I(X; Z|Y ) = I(X; Z) I(X; Y ; Z) = I(X; Z) I(Y ; Z) I(Y ; Z|X)
As Z is a representation of X, we have I(Y ; Z|X) = 0, which concludes the proof.

The ÿrst term represents the predictive information we seek to preserve for eÿective prediction, while the second term, devoid of predictive power, is considered as superÿuous for the task at hand. The information bottleneck principle (Tishby et al. ÿÿÿÿ) provides a suitable approach to construct expressive representations in a supervised manner. This principle seeks to minimize I(X; Z), while simultaneously maximizing I(Y ; Z). In other words, it constraints Z to be a minimal suÿcient statistics (Soatto et al. ÿÿÿÿ) of X to predict Y . Given the complexities associated with computing mutual information, proxies such as variational lower bounds are often used (Alemi et al. ÿÿÿÿ).

The information bottleneck principle was further adapted to the multi-view setting by (Qi Wang et al. ÿÿÿÿ), and to an unsupervised framework by (Federici et al. ÿÿÿÿ). The key theoretical contribution of their work is Corollary ÿ.

Corollary . Let X 1 and X 2 be two mutually redundant views for a target Y and let Z 1 be a representation of X 1 . If Z 1 is suÿcient for X 2 (i.e. I(X 1 ; X 2 |Z 1 ) = 0) then Z 1 is as predictive for Y as the joint observation of the two views (I(X 1 , X 2 ; Y ) = I(Z 1 ; Y )). In that case:

I(X 1 ; Z 1 ) = I(X 2 ; Z 1 ) + I(X 1 ; Z 1 |X 2 )
In the latter equation, the ÿrst term is predictive for X 2 , while the second term represents superÿous information for the task (because of the mutual redundancy of the views). This result suggests an unsupervised learning objective: to maximize the ÿrst term while simultaneously minimizing the second one. By doing that, we force the representation Z 1 to be suÿcient for X 2 (hence conserving its predictive power following the corollary), while discarding superÿuous information to make the representation more robust. The global objective simultaneously optimizes the same tradeoÿ by symmetrically decomposing I(X 2 ; Z 2 ). These quantities can be approximated using lower bounds on mutual information (Hjelm et Y . The amount of information conveyed by X 1 and X 2 are represented by red and blue areas, respectively, while the purple area represents the amount of information share by both modalities. The amount of predictive information, conveyed by the variable Y , is represented by the green area. The only amount of predictive information that is accessible is I(Y ; X 1 ; X 2 ). This piece of information is shared by both modalities (mutual redundancy), hence its representation area on the diagram is encapsulated in the purple area, representing I(X 1 ; X 2 ). It is worth noting that we generally lack access to the entirety of the information conveyed by Y ; this unavailable quantity is

H(Y |X 1 ; X 2 ).
learning in a multi-view setting.

The goal of maximizing redundant information between views in the latent representations is largely driven by the mutual redundancy assumption intrinsic to the multi-view scenario. Indeed, from a multi-view standpoint, where the same object is observed from diÿerent angles, viewspeciÿc data is often treated as noise that does not contribute to prediction. Figure ÿ.ÿ illustrates a setting of total mutual redundancy between tho modalities X 1 and X 2 for a target Y , with the help of an information diagram, a type of Venn diagram. As a result of this assumption, methods that learn representations to maximize this information perform well on multi-view downstream tasks (Tosh et al. ÿÿÿÿ). This framework has been extended to the multimodal setting, in which the modalities are considered as the diÿerent views. The related works essentially rely on contrastive methods to tackle the multimodal coordinated representation learning in a self-supervised manner (Alayrac, Recasens, et However, by focusing solely on shared factors, these approaches neglect the complementary part of the information, thereby failing to harness all synergies between modalities.

ÿÿ . Characterizing complementary information ÿÿÿ Cÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ

When considering modalities that comprise complementarity, the general setting is the one depicted in Figure ÿ.ÿ, where the predictive information available is distributed across both modalities, with some information being shared and some being speciÿc to each modality. In an attempt to preserve modality-speciÿc information, (Y.-C. Liu et al. ÿÿÿÿ) propose to directly contrast multimodal input tuples describing the same scene, as opposed to learning a crossmodal embedding space by contrasting distinct modalities. This strategy enables the model to retain unique information associated with each modality. This approach is further reÿned by (Yunze Liu et al. ÿÿÿÿ), who enhance negative sampling and positive sample generation, ensuring equal weight is given to each modality during the process of learning representations. Subsequent research, such as (W. Han et al. ÿÿÿÿ; W. Yu et al. ÿÿÿÿ), seek to exploit modality-speciÿc information in conjunction with shared information. Nevertheless, these studies primarily rely on backpropagation to leverage this information, rather than exploiting theoretical insights to develop representations that accurately depict and manage the complementarity between modalities.

These work hence aim to leverage modality-speciÿc information through supervision, assuming that predictive information I(Y ; X 1 |X 2 ) and I(Y ; X 2 |X 1 ) will be retained in the learned representation, facilitated by backpropagation. However, the acquisition of substantial annotated data is costly and not always feasible. In such scenarios, an unsupervised approach is more suitable to eÿectively and aÿordably leverage predictive information. The task becomes more challenging when we relax the redundancy assumption, as it becomes harder to diÿerentiate relevant information from noise and superÿuous information within modality-speciÿc content. Besides, by using only backpropagation to guide the learning, there is no real control over the type of information embedded in that representation that aims to leverage modality-speciÿc content, ÿÿ Thoughts on the characterization of Information across modalities for instance whether redundant information is also included (W. Han et To address this latter limitation, and taking inspiration from (M. Lee et al. ÿÿÿÿ) we tried to implement an architecture that aims to build shared and private (i.e. modality-speciÿc) representations that are also disentangled. The proposed model was based on a Variational AutoEncoder (VAE) architecture that produced for each bimodal input a shared representation and modalityspeciÿc (private) representations. A global learning objective aimed to simultaneously minimize the reconstruction and disentanglement losses. The framework was appealing:

• it leveraged self-supervised framework through reconstruction loss;

• it was motivated by theoretical assumptions, using mutual information estimators for disentangling shared and private representations;

• it would have provided an accessible latent space (to observe learned patterns) and easy sampling process.

Unfortunately we never succeeded in training the model, either the representations did not carry relevant information or they were not disentangled.

The balanced setting of Figure ÿ.ÿ might also be unrealistic. There could be situations where one modality signiÿcantly inÿuences the prediction due to possessing more information relevant to the target Y . Contrary to the assumption in (Yunze Liu et al. ÿÿÿÿ), which aims to give more weight to weaker modalities, the ideal model should prioritize the dominant modality. If we go one step further, we can imagine a setting in which all the predictive information is contained in a single modality, as illustrated in Figure ÿ.ÿ. In an extreme case, all predictive information could be modality-speciÿc, rendering the other modality superÿuous and approaches based on maximizing redundant information ineÿective. In that setting, modality X 1 has a much bigger impact than modality X 2 , which does not encompass any modality-speciÿc predictive information, i.e. I(Y ; X 2 |X 1 ) = 0 (left). In the extreme case, all predictive information is made unavailable from the perspective of X 2 view, that is I(Y ; X 2 ) = 0 (right).

ÿÿ . Characterizing complementary information

On the other hand, we can also envisage a setting in which both modalities are predictive but do not share any predictive information, as illustrated in Figure ÿ.ÿ. Observations drawn from these scenarios reveal a potential problem in the approach to multimodal learning tasks, which might originate from an ill-deÿned problem statement. While some research has started to leverage modality-speciÿc information, they mostly create representations that are developed through supervised backpropagation. However, the global complementary setting actually encompasses many diÿerent conÿgurations, therefore the objective seems ambitious as to design representations that are robust to these diÿerent situations only using supervision. This problem is dual, as the considered tasks and related public datasets do not always represent the diÿerent situations depicted above. If some recent datasets aims to address tasks that require to combine modality-speciÿc information, such as sarcasm detection (Castro et al. ÿÿÿÿ), or multimodal disambiguation (Talmor et al. ÿÿÿÿ), there is no (to our knowledge) public dataset or benchmark that focuses on explicitly evaluating models on their ability to design representations that leverage complementary information across modalities and that are robust to speciÿc conÿgurations depicted above. (P. P. Liang, Y. Lyu, et al. ÿÿÿÿ) however show a promising direction by gathering many datasets and related tasks, that for some require the model to have an ability to leverage a certain level of complementarity to perform well.

Finally, the interaction between modalities can ÿuctuate based on the speciÿc requirements of a task. Some classiÿcation tasks only necessitate the additive interaction of data from multiple modalities, as the labeling is dependent on elements that are only jointly available in these modalities. For instance, in the case of identifying a "green pencil" one modality might provide the visual representation of a pencil, while another may furnish the color information, namely, green. Conversely, certain tasks demand a more sophisticated integration of the modalities, i.e. a proper ÿÿ Thoughts on the characterization of Information across modalities reasoning step. These tasks require the combination of elements from diÿerent modalities in an insightful manner that utilizes the content from both. For instance, consider a medical diagnostic AI system that uses three modalities: medical imaging (like CT scans), patient medical history, and real-time vital sign data. The medical imaging modality oÿers visual evidence of potential physical abnormalities. The patient's medical history provides context on past health issues, family history, etc. The real-time vital sign data delivers immediate health information, like heart rate, blood pressure, and oxygen levels. Diagnosing a complex condition like a lung disease might involve reasoning across all three modalities. A CT scan might reveal a lung nodule, the patient's medical history could indicate a long history of smoking, and the real-time vital sign data might show low oxygen levels in the blood. The AI system must then reason that the lung nodule might be cancerous, potentially exacerbated by the patient's smoking history, and the low oxygen levels could be due to impaired lung function from the cancer. This reasoning process creates a possible diagnosis like "Lung Cancer -Identiÿed through CT scan, corroborated by smoking history and low oxygen levels". This diagnosis involves a nuanced understanding and combination of data across all three modalities. Very recent research attempts to tackle this complex challenge. For example (P. P. Liang, Yun Cheng, et al. ÿÿÿÿ) propose to decompose multimodal interactions into redundancy, uniqueness and synergy. This approach acknowledges the varying complexity of tasks and the diÿerent types of interplay that may exist between modalities. The future of multimodal learning research will likely involve further exploration of these dynamics, working towards more sophisticated models that can adaptively handle a range of scenarios and tasks.

Cÿÿÿÿÿÿÿÿÿ

In this chapter, we delved into the diÿerent natures of multimodal interactions, distinguishing the distinct redundant and complementary information. Through a review of contemporary works, we have underscored the importance of maximizing redundant information within the multiview setting, while concurrently highlighting the important role that complementary information plays in multimodal landscape. Nevertheless, current methodologies overwhelmingly rely on backpropagation as the central tool for learning modality-speciÿc representations. This strategy, while eÿective in certain contexts, tends to undermine the development of truly robust and versatile multimodal representations that can adapt to a wide array of scenarios. The lack of evaluation benchmarks stresses this issue, preventing accurate assessments of models' proÿciency in leveraging complementary information. As a conclusion, the task of adequately leveraging and understanding multimodal interactions remains a formidable challenge. The redundancy-complementarity dichotomy provides a useful lens through which to approach the problem, but it is clear that more sophisticated methods and robust evaluation measures are needed to tackle the diverse and complex nature of multimodal interactions.

From the experiments conducted on StreaMulT architecture in Subsection ÿ.ÿ.ÿ, textual modality appeared to be the most informative one, as its ablation leads to the biggest perÿÿ . Characterizing complementary information formances drop. This observation endorses our hypothesis that the semantics of a precise and detailed textual maintenance report, coupled with the expressive power of highdimension pre-trained textual encoders, can place the text as the predominate modality for a fault diagnosis task. Thus, in the second part we decide to put a special emphasis on text. In the following chapter, we give the reader some background on NLP research directions, from the classic tasks and architectures, up to recent interest for large foundation models and their application to FSL tasks. This chapter oÿers an overview of Natural Language Processing (NLP) methodologies, up to the development of recent large Foundation Models, and then transitions towards Few-shot learning, a strategy for learning from limited labeled data, before culminating in a discussion of FSL applied to NLP.

The initial section of this chapter outlines the progression of NLP research in understanding human language. This includes early rule-based or feature engineering methods, the utilization of word embeddings to create distributed, meaningful representations, and the development of various architectures for eÿective Language Models. In Section ÿ.ÿ, we investigate the prevailing approach to addressing NLP tasks, which involves large pre-trained transformer-based Language Models and their subsequent evolution towards creating versatile central models capable of handling a diverse range of tasks, despite their distinct nature. Finally, we explore in Section ÿ.ÿ the realm of Few-Shot Learning, examining its principal techniques and intersection with current NLP paradigms, while shedding light on the latest progress and challenges in this research area.

ÿÿÿ Iÿÿÿÿÿÿÿÿÿÿÿ

Natural Language Processing is a crucial subdomain of computer science and AI, focused on enabling computers to comprehend, interpret, and generate human languages. NLP methods have evolved over the years to handle the messiness of textual data. The primary challenges in NLP indeed stem from the inherent complexity of natural language, which is often ambiguous, contextdependent, and unstructured (Manning and Schütze ÿÿÿÿ). To tackle these challenges, NLP encompasses a wide range of tasks:

• low-level tasks, such as tokenization (K. (Woods ÿÿÿÿ), that draw from this analysis to perform complex language understanding and generation.

This section traces the history of NLP advances, beginning with early rule-based methods and feature engineering techniques. We then explore the development of word representation methods, focusing on word embeddings, which have become a crucial component in modern NLP systems. The next part delves into language models, from count-based approaches such as N -grams to neural language models based on RNN, encoder-decoder architectures, and attention mechanisms. Finally, we discuss the recent emergence of transformer-based models and large Foundation models, which bridge the gap between word embeddings and language models by leveraging contextual word representations.

ÿÿÿ Eÿÿÿÿ NLP ÿÿÿÿÿÿÿ

Rÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ

Rule-based methods, which originated in the early days of NLP and AI in the ÿÿÿÿs, relied on manually crafted rules and expert knowledge to process and analyze text. These methods were based on a set of predeÿned linguistic rules or patterns that were applied to the text to extract or manipulate information (William John Hutchins ÿÿÿÿ). Some popular rule-based NLP techniques included phrase structure grammars, and context-free grammars (Chomsky ÿÿÿÿ). Techniques such as regular expressions and ÿnite-state automata (Mohri ÿÿÿÿ) were also used to identify patterns and perform basic text processing tasks, such as tokenization and stemming. Rule-based methods were widely used in early machine translation systems, such as the ÿÿÿÿ Georgetown-IBM experiment (W. John Hutchins ÿÿÿÿ), and natural language interfaces (Androutsopoulos et al. ÿÿÿÿ; Woods ÿÿÿÿ). Rule-based methods have limitations, such as scalability and adaptability to new languages or domains, that respectively require the developments of new and complex rules. The manual creation of rules is time-consuming and requires signiÿcant domain knowledge, making these methods less eÿcient compared to more recent data-driven approaches.

Feature engineering is a process of extracting relevant features from raw data that can be used to build eÿective ML models. In the context of NLP, feature engineering often involved using expert knowledge to design features based on linguistic properties and domain-speciÿc knowledge (Jurafsky ÿÿÿÿ). Part-of-speech (POS) tagging was used as a preprocessing step in early NLP systems, identifying the grammatical role of each word in a sentence (Marcus et al. ÿÿÿÿ). This information could then be used as input for other NLP tasks, such as parsing or information extraction. Named entity recognition (NER) is another example of feature engineering in early NLP systems, where the goal is to identify and classify proper nouns, such as people, organizations, and locations, within a text (Bunescu et al. ÿÿÿÿ). Dependency parsing extracts the syntactic structure of a sentence by identifying the relationships between words (i.e., subject, object, modiÿers). Like POS tagging and NER, dependency parsing was used as a feature in other NLP tasks (Y. Zhang et al. ÿÿÿÿ). Similarly to rule-based methods, feature-engineering-approaches face several major limitations, such as the need for time-consuming expert knowledge for designing eÿective feature ÿÿ . Early NLP methods extraction methods, that may not be generalizable to new tasks and do not scale eÿciently to large datasets or long sequences. While feature engineering and expert knowledge played a signiÿcant role in early NLP tasks, another approach that emerged for handling unstructured textual data was the use of vector space models.

Vÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿ Bÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ TFÿIDF

In these models based on linear algebra, documents and words are represented as vectors (Salton et al. ÿÿÿÿ) with the aim of leveraging some similarity between them. Bag of Words (BoW) (Harris ÿÿÿÿ) is a simple and widely-used method for representing text data in NLP tasks. BoW converts text into a ÿxed-size vector by counting the frequency of words in a document and disregarding the order of words. BoW represents each document as a vector with the same length as the vocabulary size. Each element in the vector corresponds to a word in the vocabulary and contains the frequency of that word in the document. The main limitation of BoW is that it ignores word order and contextual information, making it less eÿective for capturing semantic relationships between words. Additionally, BoW can lead to high-dimensional and sparse representations, which can be computationally expensive for large vocabularies. Term Frequency-Inverse Document Frequency (TF-IDF) is a technique that extends the BoW approach by incorporating the importance of words in a document relative to their importance in the entire corpus. TF-IDF is calculated as the product of the term frequency (TF) (Luhn ÿÿÿÿ), which is the number of times a word appears in a document, and the inverse document frequency (IDF) (Sparck Jones ÿÿÿÿ) , which is the logarithm of the ratio of the total number of documents in the corpus to the number of documents containing the word. Hence, for a word w and a document d from a corpus C:

TF-IDF(w, d, C) = TF(w, d) × IDF(w, C) = Card({x ∈ d|x = w}) × log Card(C) Card({c ∈ C|w ∈ c})
where Card(C) denotes the cardinality of set C. The IDF weighting scheme assigns higher weights to words that are less frequent in the entire corpus, eÿectively reducing the impact of common words and emphasizing the importance of more informative words for a given document. Although TF-IDF provides a more sophisticated representation of text data compared to the BoW approach, it still has limitations. Similar to BoW, TF-IDF does not capture word order or contextual information.

While vector space models such as BoW and TF-IDF have proven eÿective in capturing documentlevel information and enabling the application of ML techniques to textual data without requiring engineering or expert knowledge, they do not inherently account for the sequential and structured nature of language. To address this shortcoming, researchers have turned to probabilistic frameworks that can model the dependencies and relationships between words in a sequence.

ÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models

Pÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

Probabilistic frameworks, such as Conditional Random Fields (CRFs) and Hidden Markov Models (HMMs), have been widely used in early NLP tasks to model sequences and dependencies between elements in a text. HMMs are generative probabilistic models that represent the joint probability distribution of observed and hidden variables (Rabiner ÿÿÿÿ). CRFs, on their side, are discriminative probabilistic models that directly model the conditional probability of the hidden variables given the observed variables (Laÿerty et al. ÿÿÿÿ). These probabilistic models have been used in tasks like POS tagging, NER, and shallow parsing, among others (Finkel et al. ÿÿÿÿ; Sha et al. ÿÿÿÿ). While they have proven to be eÿective in capturing relationships and dependencies in sequential data, some limitations remain, such as their lack of scalability (when dealing with long sequences or datasets, CRF are computationally expensive, whereas HMM struggle in capturing long-range dependencies due to the Markov assumption) or the lack of semantic representation (these models operate at the level of individual words), preventing them to leverage the deep semantic structure of natural language.

Methods

Scalability to large datasets Adaptability

Expert

T

Despite the success of early NLP methods in addressing various language processing tasks, these early techniques struggle in capturing the rich semantic and syntactic information present in natural language. The BoW and TF-IDF models, for example, lack the ability to represent the semantic relationships between words and fail to account for word order, which is crucial for understanding the meaning of a text. Similarly, while probabilistic frameworks like HMMs and CRFs oÿer a way to model sequences and dependencies,

ÿÿ

. Word embeddings they still rely on hand-crafted features and do not scale well to large vocabularies or complex dependencies. The limitations of each of these methods are synthesized in Table ÿ.ÿ.

As the ÿeld of NLP evolved, researchers recognized the need for better word representations that could capture both the syntactic and semantic information in text. The development of word embeddings, which are continuous vector representations of words, emerged as a promising solution to address these limitations. In the next section, we delve into the world of word embeddings, exploring the various techniques that have been proposed to learn these representations, from count-based to prediction-based methods, and how they have signiÿcantly advanced the state-of-the-art in NLP.

ÿÿÿ Wÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

The limitations of early NLP methods led to the development of word embeddings as a way to better represent and capture semantic and syntactic information about words. Word embeddings are continuous and dense vector representations that map words from a large vocabulary into a lower-dimensional space. These embeddings are based on the distributional hypothesis, which states that words that occur in similar contexts tend to have similar meanings (Firth ÿÿÿÿ; Harris ÿÿÿÿ) ÿ . They can be generated using various techniques, broadly categorized into count-based and prediction-based methods.

Cÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

Count-based word embedding techniques take the idea to put information about contexts into word vectors literally, by manually designing a word-context matrix M in which columns represent potential contexts and rows represent words. In a second step, a dimension reduction technique is applied to the matrix to produce dense embeddings. As their name suggests, these approaches are based on global corpus statistics, and in that sense share some similarities with BoW and TF-IDF. However, those latter methods are not considered as count-based word embeddings because they represent documents rather than individual words and produce sparse vectors instead of dense embeddings. From there, the diÿerent count-based word embeddings strategies diÿer in the way to consider what is context (hence deÿning what represent the matrix columns) and how to compute matrix elements. A simple co-occurrence-based approach is for instance to consider as contexts the surrounding words contained in a ÿxed-size sliding window, and to deÿne M as a word-word matrix with M ij being the number of times word w i appears in context w j (Lund et al. ÿÿÿÿ). Based on the same deÿnition of contexts, information theoretic measures such as Pointwise Mutual Information (PMI) (K. W. Church et al. ÿÿÿÿ) and Positive Pointwise Mutual Information (PPMI) (Bullinaria et al. ÿÿÿÿ) have been used to deÿne word representations in matrix M . PMI of a words pair (w i , w j ) is deÿned as the log ratio between joint probabilities and product of marginal probabilities: P M I(w i , w j ) = log P (w i ,w j ) P (w i )P (w j ) . Intuitively, designing the matrix M such that ÿ Also found as "You shall know a word by the company it keeps"

ÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models M ij = P M I(w i , w j ) will associate positive values to word pairs (w i , w j ) that appear more frequently in a same context than if they were independent, and negative values to word pairs that appear less frequently than being independent. (Bullinaria et al. ÿÿÿÿ) extend this idea by considering only positive values, that is deÿning M ij = P P M I(w i , w j ) = max(P M I(w i , w j ), 0). Finally, a popular count-based word embedding technique is Latent Semantic Analysis (LSA) (Deerwester et al. ÿÿÿÿ). Alternatively, LSA considers diÿerent documents from a corpus C as contexts, and hence designs matrix M as a word-document matrix, with M ij = TF-IDF(w i , d j , C).

The second step is then to reduce the dimensionality of the term-document matrix through a singular value decomposition (SVD) to capture latent semantic relationships between words and documents. By doing so, LSA can identify and represent synonyms, polysemes, and other linguistic relationships in the reduced-dimensional space.

While count-based word embeddings capture dependencies between words and semantic relationships through their term-context matrix, constructing and factorizing such large matrices may undermine their scalability. Besides, count-based models generally struggle with out-of-vocabulary words since they are based on direct observation of the training corpus.

Pÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

Prediction-based word embeddings are generated by training models to predict words or their contexts based on the local context information, which is generally a sliding window surrounding the target word. This approach aims to learn word representations that can eÿectively capture semantic and syntactic information while exploiting the co-occurrence patterns of words in their prehensive representation of word meaning. However, it requires explicit construction of the co-occurrence matrix, which can be computationally expensive for larger corpora, and it can be sensitive to the choice of hyperparameters, such as the window size and weighting scheme.

Interestingly, using similarity to build rich word representations is not reÿected only in quantitative metrics of subsidiary tasks. (Tomas Mikolov et al. ÿÿÿÿ) indeed qualitatively analyzed the learned vector space and pointed out geometrical patterns based on meanings similarity (see Figure ÿ.ÿ). Thus, the diÿerence between the representation vectors of many country/capital pairs seem to produce the same vector. Another example (Tomás Mikolov, Le, et al. ÿÿÿÿ) shows the similar distribution of embedding vectors from a language to another one, suggesting a simple linear mapping for translation.

T

Word embeddings have become an essential tool in NLP, capturing semantic and syntactic relationships between words and providing a foundation for more advanced techniques. However, despite their ability to capture word relationships, word embeddings have limitations, particularly in representing context-dependent word meanings. Indeed, these representations are pre-computed in a static corpus, which may not be convenient when using a word in a diÿerent context afterwards (this is notably the case for polysemous words that have in this framework only one representation). Besides, long sequences can be handled well as the window size can be varied, but distant dependencies might be missed. The comparison of approaches is thus updated in Table ÿ.ÿ.

We now delve into language models, which oÿer a comprehensive approach to capture the structure and context of language. Their development have led to powerful and versatile models capable of handling complex linguistic phenomena and signiÿcantly improving performance on a wide range of tasks, such as machine translation, speech recognition, and text generation.

ÿÿÿ Lÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

Language models play a critical role in various NLP tasks by predicting the likelihood of a sequence of words, represented as a probability distribution over words. Given a sequence of words ÿÿ . Language models (w 1 , w 2 , ..., w n ), a language model assigns a probability P(w 1 , w 2 , ..., w n ) to this sequence. This can be used for numerous applications such as machine translation (Bahdanau et (Graves ÿÿÿÿ). In this section, we explore the evolution of language modeling techniques, from early count-based approaches to more sophisticated neural models that have driven signiÿcant advances in the ÿeld of NLP.

Cÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

The early days of language modeling were dominated by count-based methods, with N -gram models being one of the most widely-used approaches (Jelinek ÿÿÿÿ). N -grams are simply contiguous sequences of N words, where N is a ÿxed integer. An N -gram language model predicts the probability of a word given its preceding N -1 words by estimating the frequency of Ngrams in a large corpus. Thus, an N -gram model makes a Markov assumption, which states that the probability of a word depends only on the previous N -1 words:

P(w n |w 1 , . . . , w n-1 ) ≈ P(w n |w n-N +1 , . . . , w n-1 )
N -gram probabilities P(w n |w n-N +1 , . . . , w n-1 ) can be estimated by counting in a corpus the occurrences of N -gram (w n-N +1 , . . . , w n-1 , w n ) and normalizing by the number of occurrences of (w n-N +1 , . . . , w n-1 ).

Despite their simplicity, N -gram models suÿer from several limitations, such as data sparsity, which occurs when certain N-grams do not appear in the training corpus, leading to inaccurate probability estimates. To overcome this issue, various smoothing techniques have been proposed (S. F. Chen et al. ÿÿÿÿ). Other drawbacks of N -gram models are their inability to capture longrange dependencies, as they only consider a ÿxed number of preceding words to predict the next word, or the curse of dimensionality they may face when considering large vocabulary (Bengio, Ducharme, et al. ÿÿÿÿ).

While count-based language models have provided a foundation for early NLP research, their limitations have led to the development of more advanced techniques such as neural language models (Bengio, Ducharme, et al. ÿÿÿÿ), that afterwards leveraged the power of deep learning to better understand and represent natural language.

Nÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

Neural language models aim to provide a continuous representation of words and capture semantic and syntactic information in dense vector space. They have demonstrated their ability to overcome some of the limitations of count-based language models, such as the curse of dimensionality and the sparsity of N -grams. One of the ÿrst neural language models was a feedforward neural network (FFN) language model (Bengio, Ducharme, et al. ÿÿÿÿ). This model aimed to predict the next word in a sequence by concatenating word embeddings of previous words and feeding them into the FFN. The output models the word probability given a context. The model's architecture is illustrated in Recurrent Neural Networks were introduced as an extension to feedforward neural language models to better capture long-range dependencies in natural language data (Elman ÿÿÿÿ). RNNs are designed to process sequences of variable length by maintaining a hidden state that can store information from previous time steps (Tomás Mikolov, Karaÿát, et al. ÿÿÿÿ). However, RNNs have some limitations, such as the vanishing gradient problem that makes learning long-range dependencies diÿcult (Hochreiter, Bengio, et al. ÿÿÿÿ). To overcome the vanishing gradient problem in RNNs, Long Short-Term Memory (LSTM) networks were proposed (Hochreiter and Schmidhuber ÿÿÿÿ). LSTMs introduce a gating mechanism that helps to maintain and propagate information over long sequences, making them more eÿective for learning long-range dependencies. LSTMs have thus been used as building blocks for Language Models (Sundermeyer et al. ÿÿÿÿ). Finally, Gated Recurrent Units (GRU) are another variant of RNNs that simplify the LSTM architecture while retaining its ability to model long-range dependencies (Cho et al. ÿÿÿÿ). GRUs use update and reset gates to control the ÿow of information in the hidden state, making them computationally more eÿcient than LSTMs, however they may not capture long-term dependencies as well as LSTM.

ÿÿ

. Encoder-decoder architecture ÿÿÿ Eÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ Many NLP tasks require not only an understanding of the input text but also the generation of a meaningful output sequence, such as in neural machine translation and text summarization. To tackle these challenges, a new class of models has emerged: encoder-decoder architectures, also known as sequence-to-sequence models (Sutskever, Vinyals, et al. ÿÿÿÿ). The encoder-decoder architecture is composed of two main components: the encoder and the decoder. The encoder processes the input sequence and generates a ÿxed-length context vector that encapsulates the essential information of the input. The decoder, in turn, takes this context vector and generates an output sequence, conditioned on the input sequence. These architectures split the model into two parts, with one component (the encoder) focusing on processing the input sequence and the other (the decoder) generating the output sequence (Cho et al. ÿÿÿÿ). In the early encoder-decoder models, both the encoder and decoder were typically implemented as RNNs, LSTMs, or GRUs. The encoder processes the input sequence one token at a time, updating its hidden state at each step. The ÿnal hidden state of the encoder is then used as the initial hidden state of the decoder, which generates the output sequence one token at a time. An illustration of this family of architectures is given in Figure ÿ.ÿ . sequentially fed to the encoder module, that stores the input information in a context S. Using this context and the previous generated token (starting with a special token), the decoder module sequentially generates the output.

While the encoder-decoder architecture was a signiÿcant improvement over the previous models, it still faced some limitations. One of the main challenges was that the encoder had to compress the entire input sequence into a single ÿxed-size context vector, which could result in loss of information, especially for long input sequences (Bahdanau et al. ÿÿÿÿ). This limitation prompted researchers to explore more sophisticated ways to better capture and leverage the information in the input sequence, leading to the development of attention mechanism.

Aÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ

The key idea behind attention mechanism (Bahdanau et al. ÿÿÿÿ) is that the decoder should be able to focus on diÿerent parts of the input sequence at diÿerent time steps, rather than relying solely on a single context vector. This allows the model to weight the importance of diÿerent input tokens and selectively retrieve information from the input sequence. In an attention-based encoder-decoder model, the encoder produces a sequence of hidden states, one for each input token. The decoder, at each time step, computes a weighted sum of these hidden states, where the weights are determined by the attention mechanism. These weights, also known as attention scores, indicate how much the decoder should "attend" to each input token when generating the output token at a given time step. The attention mechanism computes attention scores using a scoring function that takes as input the current hidden state of the decoder and the hidden states of the encoder. There are several variants of the scoring function, such as dot product, additive, and multiplicative attention (T. Luong et al. ÿÿÿÿ). The introduction of attention mechanisms signiÿcantly improved the performance of encoder-decoder models on a wide range of NLP tasks, including neural machine translation (Bahdanau et al. ÿÿÿÿ), text summarization (Rush et al. ÿÿÿÿ), and speech recognition (Chorowski et al. ÿÿÿÿ). The success of attention mechanisms in these tasks paved the way for further advancements in NLP, such as the development of transformers.

ÿÿÿ Tÿÿÿÿÿÿÿÿÿÿÿ

Despite the success of attention mechanisms in improving the performance of encoder-decoder models, researchers continued to explore ways to further enhance the capabilities of NLP models. One signiÿcant drawback of the RNN-based models was their sequential nature, which makes it diÿcult to parallelize the computations and exploit the full potential of modern hardware, such as GPUs. In response to this challenge, (Vaswani et al. ÿÿÿÿ) introduced the Transformer architecture, which replaces the recurrent layers in encoder-decoder models with self-attention mechanisms. This groundbreaking innovation has become the foundation for many state-of-the-art models in NLP, including BERT (Devlin et al. ÿÿÿÿ), GPT (Radford, Narasimhan, et al. ÿÿÿÿ), and their variants, as well as in other domains (vision (Dosovitskiy et al. ÿÿÿÿ), speech (Radford, Kim, T. Xu, et al. ÿÿÿÿ), etc.). The self-attention mechanism is at the core of the Transformer architecture. Unlike the attention mechanism used in encoder-decoder models, self-attention operates within a single sequence, allowing each token to attend to all other tokens in the sequence. This mechanism enables the model to capture long-range dependencies more eÿectively and allows for parallel computation across tokens. See Subsubsection ÿ.ÿ.ÿ for a more detailed overview of the self-attention mechanism. The Transformer architecture is built upon a stack of self-attention layers and feed-forward layers, with residual connections and layer normalization applied throughout the model. The original Transformer model proposed in (Vaswani et al. ÿÿÿÿ) consists of an encoder and a decoder, similar to the earlier encoder-decoder models. The encoder is composed of a stack of identical layers, each containing a multi-head self-attention mechanism followed by a position-wise feedforward network. The decoder has a similar structure, with an additional layer of cross-attention that attends to the encoder's output. The global architecture is presented in Figure ÿ.ÿ.

Transformers can also be designed as standalone encoders or decoders for various NLP tasks, depending on the nature of the problem and the desired model architecture. For instance, BERT (Devlin et al. ÿÿÿÿ) is built upon a stack of Transformer encoder layers, while GPT (Radford, Narasimhan, et al. ÿÿÿÿ) former architecture can be more suitable for tasks that require a ÿxed-length representation of the input sequence, such as sentence classiÿcation. The Transformer encoder processes the input sequence and produces a contextualized representation for each token, which can be aggregated or pooled to generate a ÿxed-length vector. On the other hand, using only the decoder part of the Transformer can be advantageous for tasks that involve generating text or predicting the next token in a sequence, such as language modeling, text generation, and summarization. The Transformer decoder is designed to handle autoregressive decoding, where the model generates one token at a time and feeds the generated tokens back as input for the subsequent steps. This architecture enables the model to leverage the self-attention mechanism for capturing dependencies between generated tokens, while still beneÿting from the parallelizability and eÿcient handling of long-range dependencies oÿered by the Transformer architecture.

ÿÿ

Background

T

Driven by the diverse requirements of NLP tasks and the inherent pursuit of comprehending and generating human language automatically, numerous frameworks and methodologies have been pursued and reÿned, successively diminishing the constraints of preceding methods (see Table ÿ.ÿ). The advent of word embedding methods marked a signiÿcant milestone, providing dense, vector-based semantic representations that proved invaluable for a multitude of downstream tasks. Recurrent Neural Networks, particularly LSTM, advanced this paradigm by capturing distributed, contextually-dependent representations via their hidden state. They led to the introduction of a new architectural framework: the Encoder-Decoder model. This approach is exceptionally suitable for tasks requiring contextual generation, such as machine translation.

The colossal breakthrough came with the advent of Transformer models, inspired by the Encoder-Decoder architecture and the introduction of the Attention Module. These models oÿer outstanding semantic and context-aware representations through their selfattention module, directly capturing all types of dependencies across sequence elements, rather than compressing pertinent information within a hidden state as is the case with ÿÿ LSTM. Furthermore, the ability of Transformer models to parallelize eÿciently permits impressive scaling, aligning seamlessly with the capabilities of modern hardware. This has resulted in Transformers becoming the cornerstone for the vast majority of today's architectural designs in NLP and other applications of Deep Learning.

ÿÿÿ Fÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

Transformers have signiÿcantly impacted the ÿeld of NLP, and their introduction came with a change of paradigm in the ÿeld. Rather than using an end-to-end supervised framework composed of task-speciÿc neural networks, most works in the recent years follow the pre-training and ÿne-tuning paradigm to achieve state-of-the-art performance across a wide range of NLP tasks. This has today led to the Foundation models era, that aim to unify all kind of NLP tasks within a single architecture.

Remark. Following the Center for Research on Foundation Models of Standford University ÿ , we refer to Foundation models (Bommasani et al. ÿÿÿÿ) as the following: "In recent years, a new successful paradigm for building AI systems has emerged: Train one model on a huge amount of data and adapt it to many applications. We call such a model a foundation model.". These models are based on Pre-trained Language Models (PLMs) architectures (see thereafter), and as they become larger and larger, are often referred to as Large Language Models. The interchange of these terms is hence frequent in the literature.

Pÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ

The pre-training and ÿne-tuning paradigm has emerged as a successful approach for building Pre-trained Language Models in NLP. The idea is to ÿrst train a large neural network (mainly transformer-based one) on a massive amount of unsupervised text data (such as the Cÿ dataset (Raÿel, Shazeer, et al. ÿÿÿÿ)), and then ÿne-tune the pre-trained model on a speciÿc supervised task (Howard et al. ÿÿÿÿ; Peters et al. ÿÿÿÿ). This approach leverages the ability of DL models to learn rich and meaningful representations from large-scale data, which can then be adapted to speciÿc tasks with relatively small amounts of labeled data (see Figure ÿ.ÿ).

Transfer learning is a key concept underlying the pre-training and ÿne-tuning paradigm. It refers to the process of transferring knowledge learned in one task or domain to another, usually related, task or domain (S. J. Pan et al. ÿÿÿÿ). In NLP, transfer learning has been shown to be highly eÿective, as the knowledge learned from large-scale unsupervised text data can be generalized to a wide range of tasks (Ruder et al. ÿÿÿÿ). The beneÿts of transfer learning in NLP are numerous. Firstly, it allows for more eÿcient learning and better generalization, as the pre-trained model has already learned meaningful language representations (Bengio, Courville, et al. ÿÿÿÿ). Secondly, it reduces the need for labeled data in the target task, as the pre-trained model can be ÿne-tuned with relatively small amounts of labeled data (Peters et al. ÿÿÿÿ). Finally, it leads to faster convergence

Pÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

As we discussed, Foundation models aim to acquire a vast amount of knowledge by pre-training on massive unsupervised corpora. The choice of pre-training tasks and associated losses is therefore crucial in enabling these models to gain the general linguistic knowledge necessary for eÿective downstream task performance. By carefully designing the pre-training objective, we can encourage the model to learn valuable patterns, structures, and relationships within the data that can be eÿectively transferred to a wide range of downstream tasks. In this context, pre-training losses play a pivotal role in guiding the learning process of foundation models and shaping their ability to generalize and adapt to various NLP challenges.

In the initial stages, ELMo (Peters et al. ÿÿÿÿ) was developed to obtain context-sensitive word representations by ÿrst pre-training a bidirectional LSTM (biLSTM) network (rather than acquiring ÿxed word representations). Subsequently, the biLSTM network was ÿne-tuned to cater to particular downstream tasks.

BERT (Devlin et al. ÿÿÿÿ) is a powerful model based on the Transformer encoder architecture. BERT is pre-trained on a large corpus of text using a Masked Language Modeling (MLM) objective, which enables it to learn bidirectional contextual representations. In this objective, a certain percentage of the input tokens are randomly masked (literally replaces by a MASK token), and the model is trained to predict the original token based on the context provided by the surrounding unmasked tokens. The MLM loss is calculated by comparing the predicted probabilities for the masked tokens with the true tokens using cross-entropy. This objective allows BERT to learn BERT is also pre-trained using a Next Sentence Prediction (NSP) loss, in which the model shall predict if a sequence is subsequent to another one (but the NSP loss appeared to have low impact on performance). (Yamaguchi et al. ÿÿÿÿ) explored other cheaper pre-training objectives, similar to MLM, and showed comparable performance (see Figure ÿ.ÿ). Context-aware word representations of BERT and its variants (such as RoBERTa (Yinhan Liu et al. ÿÿÿÿ)) have demonstrated state-of-the-art performance on a wide range of NLP predictive tasks, such as sentiment analysis, named entity recognition, and question-answering. Fine-tuning BERT on task-speciÿc datasets allows it to adapt its powerful pre-trained representations to the target task, often with minimal additional training. GPT (Radford, Narasimhan, et al. ÿÿÿÿ) is another signiÿcant milestone in contextual word representations. GPT models are based on the Transformer decoder architecture and are pre-trained using a unidirectional autoregressive Language Modeling (LM) objective. The primary goal of GPT is to predict the next token in a sequence given its preceding context. The LM loss is computed by comparing the predicted probabilities for the next token in the sequence with the true next token using cross-entropy. The unidirectional nature of GPT allows it to learn powerful contextual representations, capturing the left context of each token. However, due to their autoregressive loss, these models are especially suitable for generative tasks such as dialogues and document summarization. There have been several iterations of the GPT model, with GPT-ÿ (Radford, J. Wu, et al. ÿÿÿÿ) and GPT-ÿ (Brown et al. ÿÿÿÿ), especially diÿering by their sizes, both in number of parameters and training corpora. More recently, GPT-ÿ (OpenAI ÿÿÿÿ) was released, once again crushing its previous version size with now ÿ trillion (10 12 ) parameters, and now being multimodal, as it can process both text prompts and images as input. Like BERT, GPT models can be ÿne-tuned on task-speciÿc datasets to adapt their pre-trained representations to the target tasks.

ÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models Conditional Language Modeling (CLM) objective is another type of pre-training loss used in some foundation models. Unlike standard LM loss used in GPT, which focuses on predicting the next word in a sequence given the previous words, or the MLM loss used in BERT that concentrates on predicting randomly masked words within a sentence, the CLM loss aims at reconstructing the input sequence after a speciÿc kind of perturbations. A prominent encoder-decoder architecture that employs CLM objective is Tÿ (Raÿel, Shazeer, et al. ÿÿÿÿ), that adopts a text-totext transfer learning approach, where both input and output sequences are represented as text strings. It is pre-trained on a denoising autoencoder task, which involves reconstructing the original text from a corrupted version. During pre-training, Tÿ introduces noise to the input text by applying transformations such as token masking or deletion. The model then learns to recover the original input sequence from the perturbed version. By learning to reconstruct the original sequence, Tÿ captures bidirectional context and adapts well to various NLP tasks. Another notable architecture that uses CLM loss is BART (Lewis et al. ÿÿÿÿ). BART also adopts a denoising autoencoder setup, applying transformations such as token masking, token deletion, or text shufÿing. The combination of bidirectional context and autoregressive nature allows both Tÿ and BART to excel in a wide range of tasks, taking advantage of both LM and MLM frameworks.

The diÿerent pre-training objectives are listed in Table ÿ.ÿ. For each objective, the considered network aims to model the conditional probability p. It can be trained with maximum likelihood estimation.

Objective

Loss

MLM L M LM = - X w∈m(w) log p( w|w \m(w) ) LM L LM = - T X t=1 log p(w t |w <t ) CLM L CLM = - T X t=1 log p(w t | w, w <t )
Table ÿ.ÿ: Pre-training objectives and their respective loss functions for a sentence w = (w 1 , . . . , w T ).

w <t := (w 1 , . . . , w t-1 ), while m(w) designs masked words of w, w \m(w) designs the unmasked elements of w and w designed corrupted sentence.

In summary, the introduction of PLM have revolutionized the ÿeld of NLP, providing generalpurpose contextual word representations that have signiÿcantly improved performance across various tasks. Building on this success, following works developed larger architectures to still improve performances on downstream tasks.

Lÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

Several studies (Hoÿmann et al. ÿÿÿÿ; Kaplan et al. ÿÿÿÿ; Rosenfeld et al. ÿÿÿÿ) have demonstrated the advantages of scaling up language models in terms of model size, dataset size, and computational resources, by introducing scaling laws in terms of loss reduction. This led to the emergence of Large Language Models (LLMs) tures with hundreds of billions or more parameters, are trained on extensive text datasets. These scaled-up models, despite adopting similar Transformer architectures and pre-training objectives as smaller PLMs, beneÿt signiÿcantly from increased model size, data size, and computational power. Over the last years, several tech resource-rich organizations launched their own LLM, with for instance Google's PaLM (Chowdhery et al. ÿÿÿÿ) and LaMDA (Thoppilan et al. ÿÿÿÿ), OpenAI's GPT-ÿ (OpenAI ÿÿÿÿ), DeepMind's Chinchilla (Hoÿmann et al. ÿÿÿÿ), or Meta's LLaMA (Touvron et al. ÿÿÿÿ). In parallel, a team of researchers released BLOOM (Scao et al. ÿÿÿÿ), a ÿÿÿB-parameter open-access language with the aim to make this kind of models publicly accessible. ÿÿ

Background and Related work in NLP: from Symbolic methods to Foundation Models

Interesting learning abilities

LLMs exhibit strong capacities to understand natural language, generate text, and display emergent abilities, that "are not present in small models but arise in large models" (Wei, Tay, et al. ÿÿÿÿ).

These abilities include In-context learning (ICL) and Instruction formatting.

Introduced by GPT-ÿ (Brown et al. ÿÿÿÿ), ICL allows language models to generate outputs at test time, given demonstrations of task, without requiring additional ÿne-tuning or gradient updates. While the ÿÿÿB GPT-ÿ model exhibits strong ICL abilities, the GPT-ÿ and GPT-ÿ models do not. Besides, when ÿne-tuned on multi-task datasets using instructions (natural language descriptions), LLMs show considerable performance on unseen tasks that are also described by instructions (Ouyang et

Some limitations

Whereas LLMs have demonstrated impressive performance across a broad spectrum of NLP tasks, they sometimes produce unexpected outputs, or hallucinations, that may cause harm or mislead the user. To prevent this behavior, the concept of human alignment has been introduced to ensure LLMs outputs align with human expectations (Glaese et al. ÿÿÿÿ; Ouyang et al. ÿÿÿÿ). Reinforcement learning from human feedback (RLHF) (Christiano et al. ÿÿÿÿ; Ziegler et al. ÿÿÿÿ) for instance uses a policy-gradient RL algorithm to adjust LLMs based on human feedback. The integration ÿÿ . Few-shot learning in NLP of human preferences via instructions, combined with training on both code and natural text segments, resulted in the development of the GPT-ÿ.ÿ series. After undergoing a conversation-like training process, the widely-adopted chatbot ChatGPT was introduced, signiÿcantly inÿuencing future AI research and underscoring the potential of human-like AI systems. Google similarly then released their chatbot BARD, aligned on human preferences with their own instruction ÿnetuning method FLAN (Wei, Bosma, et al. ÿÿÿÿ). Anthropic's Claude chatbot has on its side been aligned with human moral behavior using a technique called Constitutional AI (Bai et al. ÿÿÿÿ), providing a principle-based approach to produce harmless outputs.

T

Large Language Models have made remarkable strides in the ÿeld of NLP by employing the pre-training and ne-tuning paradigm. This approach has enabled these models to achieve impressive results on a wide range of NLP tasks, even though the tasks themselves are quite diverse. While these models are yet subject to hallucinations, human alignment appeared as ÿrst step to ensure more control on their output. However, the ÿnetuning process needs sizable labeled datasets for adapting the model to a new task, given the signiÿcant number of parameters involved. The challenge of gathering annotated data is ampliÿed by the expenses involved and the scarcity of such data across diÿerent languages and domains. Consequently, there is a pressing need to develop eÿective methods for learning with limited annotated data. In parallel, LLMs show emergent abilities, such as In-Context Learning, that may be suitable for addressing this challenge. This leads us to the next section, which focuses on Few-Shot Learning (FSL) techniques for NLP.

ÿÿÿ Fÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿ NLP

Fÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ

Few-Shot Learning (FSL) refers to the ability to learn tasks with limited annotated examples. This ability of humans, that are able to use their previous experience to adapt fastly to new context, has been largely studied recently in the context of machine learning algorithms (Lake et al. ÿÿÿÿ). As illustrated in Figure ÿ.ÿÿ, it can concern many tasks: classiÿcation , generation, etc. Historically, Meta-learning -or learning to learn (Thrun et al. ÿÿÿÿ)-approaches have for quite long stood as the de-facto paradigm for FSL (K. Lee et al. ÿÿÿÿ; A. Raghu et al. ÿÿÿÿ; A. Rusu et al. ÿÿÿÿ; Snell et al. ÿÿÿÿ; Q. Sun et al. ÿÿÿÿ; Sung et al. ÿÿÿÿ). Meta-learning refers to the process of improving a learning algorithm with multiple learning episodes (episodic training). These learning episodes are a distribution of tasks and not data samples. This improved learning ability has then been applied to the FSL realm. For instance, MAML (Antoniou et al. ÿÿÿÿ; Finn et al. ÿÿÿÿ), arguably the most popular meta-learning method, tries to train a model such that it can be ÿne-tuned endto-end using only a few supervised samples while retaining high generalization ability. of model parameters, such that they can be ÿne-tuned eÿciently with minimal supervision data (Finn et al. ÿÿÿÿ; Ravi et al. ÿÿÿÿ). Model-based approaches involve learning a model that can generate or adapt parameters for new tasks with the help of limited examples, often by using memoryaugmented networks or modular architectures (Graves et al. ÿÿÿÿ; N. Mishra et al. ÿÿÿÿ). Lastly, metric-based methods rely on learning a similarity metric between instances, such that classiÿcation can be performed by comparing the relationships between few-shot examples and new instances in a latent space (Snell et al. ÿÿÿÿ; Vinyals, Blundell, et al. ÿÿÿÿ). Semi-supervised learning methods with few annotations also contribute to the FSL landscape, combining a small amount of labeled data with a larger pool of unlabeled data to improve performance on speciÿc tasks (Oliver et al. ÿÿÿÿ; Rasmus et al. ÿÿÿÿ).

The majority of these methodologies have primarily been developed and tested within the realm of computer vision. Nonetheless, certain articles have shown that straightforward techniques rooted in transfer learning can competently compete with meta-learning approaches (Jiaxin Chen et al. ÿÿÿÿ; Y. Tian, Yue Wang, et al. ÿÿÿÿ). As a result, a signiÿcant number of modern investigations are centered around the pre-training and e cient ne-tuning paradigm as a means of developing eÿective methods for FSL (Jiaxin Chen et al. ÿÿÿÿ). Similarly, in state-of-the-art NLP, FSL is predominantly executed through strategies that harness the power of Pre-trained Language Models.

Fÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿ NLP ÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ

A signiÿcant body of research has addressed the challenge of FSL in NLP by leveraging Pre-trained Language Models (PLMs) (Devlin et al. ÿÿÿÿ; Yinhan Liu et al. ÿÿÿÿ; Radford, J. Wu, et al. ÿÿÿÿ; Zhilin Yang et al. ÿÿÿÿ). These approaches can be broadly categorized into three primary groups:

parameter-e cient tuning, prompt-based learning, and in-context learning. Parametereÿcient tuning aligns with methods in the ÿeld of computer vision, introduced at the end of ÿÿ . Few-shot learning in NLP previous paragraph, drawing heavily on the principles of transfer learning. On the other hand, the approaches of prompt-based learning and in-context learning are speciÿc to the domain of NLP. They innovatively restructure tasks into natural language "prompts" and take advantage of Pre-trained Language Models (PLMs) to ÿll in these prompts.

Parameter-e cient tuning: These methods, such as adapters (Houlsby et al. ÿÿÿÿ) have emerged as a promising solution for transfer learning and FSL in NLP tasks. These approaches involve adding lightweight, task-speciÿc adapter layers to pre-trained transformer models, which allow for ÿne-tuning on limited labeled data while keeping the majority of the pre-trained model's parameters ÿxed (see Figure ÿ.ÿÿ). Examples of such methods include AdapterHub (Pfeiÿer et al. ÿÿÿÿ), a framework for adapting transformers, and (D. Guo et al. ÿÿÿÿ), referred to as "Diÿ-Pruning", accomplishing a similar objective by incorporating a sparse, task-speciÿc diÿerence vector to the original parameters. Moreover, in some cases, ÿne-tuning just a small fraction of the pre-trained model has proven to be eÿective. For instance, BitFit (Ben Zaken et al. ÿÿÿÿ) only ÿne-tunes the bias parameters, which account for less than 1% of the total model parameters, yet it achieves competitive results on downstream tasks. More recently, T-FEW (Haokun Liu et al. ÿÿÿÿ) proposed an approach consisting in adding learned vectors that rescale the network's internal activations.

Prompt-Based Few-Shot Learning:

In recent years, Pre-trained Language Models (PLMs) have been used to solve FSL tasks in NLP, notably using a prompting strategy. The idea is to frame the task as a language modeling problem by designing a template that guides the model towards generating a desired output. The seminal work (Schick et al. ÿÿÿÿ) formalizes the prompt setting ÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models by deÿning the template as pattern-verbalizer pairs, in which the pattern is a function mapping a set of input sentences to a cloze question. Verbalizers, on the other hand, are injective functions that map discrete labels into natural language phrases or tokens. This association leverages the generation capability of PLMs to perform classiÿcation tasks using a template, allowing the classiÿcation task to be formatted in a way that is intelligible to the PLM (Ding et al. ÿÿÿÿ; P. Liu et al. ÿÿÿÿ). This framework is illustrated in Figure ÿ.ÿÿ. By varying the patterns and verbalizers, it is then possible to annotate a larger unlabeled dataset with soft labels, on which a classic classiÿer will be ÿne-tuned. cellent pizza!" as good or bad. The pattern is ÿrst transforming the input as a cloze question P (x). P (x) is then fed to a PLM that outputs prediction scores for the masked word. Eventually, the verbalizer v converts the token prediction scores as classiÿcation logits. ÿ

In-Context Learning: GPTÿ (Brown et al. ÿÿÿÿ), GPTÿ (OpenAI ÿÿÿÿ) and related chatbot ChatGPT based on InstructGPT model (Ouyang et al. ÿÿÿÿ) showed that PLMs were also eÿcient for in-context FSL tasks. In this setting, the prompt is composed of the task description, but also some support input examples with their corresponding outputs and a query input with the objective to predict the query output (Wei, Xuezhi Wang, et al. ÿÿÿÿ). ICL hence requires no parameter update, produces a new prediction model for each new prompting, and therefore quickly adapts to a new task (see Figure ÿ.ÿ).

Iÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ

Learning an inductive classiÿer on embeddings generated by a pre-trained model, as proposed by (Snell et al. ÿÿÿÿ), is a common baseline for performing FSL. This approach is prevalent in NLP, where a parametric model is trained on data to infer general rules that are applied to label new, unseen data (known as inductive learning (V. N. Vapnik ÿÿÿÿ)). However, in FSL scenarios with limited labeled data, this approach can be highly ambiguous and lead to poor generalization. Transduction oÿers an attractive alternative to inductive learning (Sain ÿÿÿÿ). Unlike inductive learning, which infers general rules from training data, transduction involves ÿnding rules that work speciÿcally for the unlabeled test data. By utilizing more data, such as unlabeled test instances, and

Figure ÿ.ÿÿ:

Inductive vs transductive settings. In the inductive setting (left), the model aims to learn general rules from labeled data, that will then serve to classify all unlabeled test samples, one by one. In the transductive setting (right), the model leverages information from both labeled data and all available unlabeled samples to adapt its classiÿcation to these samples. In this example, the same datapoint represented by a red circle is not classiÿed the same way by the two approaches.

Transductive methods yield substantially better performance than their inductive counterparts by leveraging the statistics of the unlabeled data (such as batch normalization statistics (Nichol et al. ÿÿÿÿ)). While (R. Hou et al. ÿÿÿÿ; Yanbin Liu et al. ÿÿÿÿ) use graphs or cross-attention modules to perform label propagation from support to query samples, other main strategies consist in minimizing the entropy of query samples predictions (Dhillon et al. ÿÿÿÿ), using prototype rectiÿcation (J. Liu et al. ÿÿÿÿ), Laplacian regularization (Ziko et al. ÿÿÿÿ), optimal transport (Y. Hu et al. ÿÿÿÿ), or maximizing Mutual Information measures (Boudiaf et al. ÿÿÿÿ; Y. Guo et al. ÿÿÿÿ; Veilleux et al. ÿÿÿÿ). However, despite their success experienced in the vision community, this framework has not yet been explored in the context of textual data.

Cÿÿÿÿÿÿÿÿÿ

In conclusion, this chapter provided a comprehensive overview of the evolution and current state of NLP, delving into the various methodologies and techniques that have shaped the ÿeld. We began with early NLP approaches, including rule-based methods, vector space models, and probabilistic frameworks, before moving on to the groundbreaking deÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models velopment of word embeddings that signiÿcantly advanced the state-of-the-art. The chapter then explored the emergence of language models and the attention mechanism, which have led to the transformative introduction of transformer architectures. Large PLMs have revolutionized NLP by providing general-purpose contextual word representations that have greatly improved performance across a wide range of tasks. The pretraining and ÿne-tuning paradigm has proven highly successful, and has further pushed the boundaries of what is possible in NLP. However, these advancements based on the scaling paradigm require huge computational resources and available annotated data for ÿne-tuning. To handle this challenge, an interest in Few-shot Learning for NLP has grown. If universal eÿcient transfer-learning-based have been explored, new NLP-speciÿc FSL paradigms have been developed, based on natural language prompts, and leveraging PLMs generation ability. Yet, they may not be suitable for realistic assumptions. A possible solution could be the use of transductive paradigm, that has not been explored in NLP. This is the main focus of Chapter ÿ. ÿÿÿ ÿ A ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ NLP

Cÿÿÿÿÿÿ'ÿ Sÿÿÿÿÿÿ

In this chapter, we explore the potential of transductive methods for textual classiÿcation in the context of ew-shot learning, aiming to address the limitations of current FSL methods in NLP, speciÿcally the engineering eÿorts required for realistic classiÿcation tasks with a large number of classes. We ÿrst discuss the limitations of current FSL methods, such as prompt-based strategies or in-context learning. Then, in Section ÿ.ÿ we explore the application of transductive approaches, which have shown promising results in computer vision, to NLP classiÿcation. Finally, in Section ÿ.ÿ we evaluate the performance of traditional transductive regularizers in comparison to inductive techniques on textual fewshot classiÿcation tasks and investigate the impact of diÿerent factors, such as the number of backbone parameters and ÿne-tuning strategies, on the performance of transductive methods. The results indicate that transductive methods have diÿculty outperforming inductive cross-entropy-based ÿne-tuning when there is some ÿexibility in the pre-trained feature extractor parameters. However, by ÿxing all parameters of the feature extractor, the transductive approach ÿnally rivals the inductive one.

ÿÿÿ Iÿÿÿÿÿÿÿÿÿÿÿ

As discussed in previous chapter, Few-Shot Learning (FSL) has gained signiÿcant attention in the ÿeld of NLP due to its ability to rapidly adapt to new tasks using limited labeled data. Current FSL methods, such as prompting and ICL, have demonstrated promising results in a wide range of NLP tasks. However, as the complexity of the classiÿcation problem grows, especially in cases with a large number of classes, these methods are confronted with inherent limitations, such as the need for extensive engineering to achieve practical results. This chapter aims to address these limitations by exploring the potential of transductive methods for textual classiÿcation in the context of fewshot learning. Transductive methods, which have been successfully applied in other domains, oÿer a promising alternative to traditional FSL techniques by leveraging the structure of the input data to make predictions for the unseen data points. By adapting these methods for textual tasks, we seek to harness their potential to tackle the challenges posed by the ever-increasing complexity and scale of classiÿcation problems in NLP, hence meeting more realistic assumptions.

ÿÿÿ

A transductive approach for performing few-shot classi cation in NLP

Cÿÿÿÿÿÿ'ÿ Cÿÿÿÿÿÿÿÿÿÿÿÿ

The primary contributions of this chapter are three-fold:

• We provide an analysis of the limitations of current FSL methods in NLP, speciÿcally in terms of the engineering eÿorts required for realistic classiÿcation tasks with a large number of classes, and we formulate the textual few-shot classiÿcation problem.

• We propose a novel adaptation of transductive methods for textual classiÿcation in the context of FSL, enabling eÿective utilization of limited labeled data.

• We present a series of research questions and their related experiments conducted to validate or rebut the eÿectiveness of our proposed methods, comparing their performance to the inductive techniques in FSL for NLP.

ÿÿÿ Pÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ

The main assumption of FSL in modern NLP paradigm supposes the availability of a large pretrained backbone model. The objective is to leverage this model's learned representations to adapt to a novel classiÿcation task when only a handful of annotated samples are at our disposal.

ÿÿÿÿÿ Cÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ

While previous works on NLP-FSL present promising results, they mainly focus on datasets with a reduced number of classes (i.e. always less than ÿÿ classes and often less than ÿ classes) (Mahabadi et al. ÿÿÿÿ; Perez et al. ÿÿÿÿ). However, when considering realistic setting, a few-shot classiÿer shall be able to classify among much more unseen classes, or to have a generalization ability that makes it prone to quickly adapt to a new set of classes. Under this consideration, current NLP-FSL strategies face practical limitations:

• Using a prompt-based approach demands a cumbersome handcraft engineering to design every Pattern-Verbalizer pairs. Thus, recent studies have questioned the beneÿts of promptbased learning due to the high variability in performance caused by the choice of prompt (Haokun Liu et al. ÿÿÿÿ). As the number of classes increases, crafting appropriate prompts and verbalizers becomes increasingly diÿcult, and the resulting prompts may not be equally eÿective for all classes. This can lead to a performance degradation in complex classiÿcation problems. Besides, this engineering is mainly validated on held-out labeled examples, which could not be available in general (Perez et al. ÿÿÿÿ). The prompting setting is therefore hardly scalable for tasks with realistic settings. To cope with these limitations, recent NLP-FSL approaches try to alleviate the importance of template design (Logan IV et al. ÿÿÿÿ), or to break with prompt paradigm (Fei et al. ÿÿÿÿ).

• Several works have shown that in-context-learning design, along with the choice and ordering of training samples, is highly sensitive and not robust to the choice of PLM (Y. Lu et al.

ÿÿÿ

. Problem statement ÿÿÿÿ; Z. Zhao et al. ÿÿÿÿ). Second, as the number of classes increases, the need for longer contexts to provide suÿcient examples for all classes can exceed the maximum input length of the models. This can result in the truncation of important information or the inability to adequately represent the full range of classes. These drawbacks prevent the usage of such strategy for realistic NLP-FSL tasks.

• Finally, parameter-eÿcient tuning methods shall be considered on a case-by-case basis. While T-FEW (Haokun Liu et al. ÿÿÿÿ) additionally requires a set of manually created prompts for each dataset making it hard to use in practice, Diÿ-Pruning (D. Guo et al. ÿÿÿÿ) considers an inconsistent set of parameters that change values across diÿerent tasks, which may prevent us to use it on highly variable number of test classes for hardware practical reasons. Nonetheless, some approaches such as (Houlsby et al. ÿÿÿÿ), or BitFit (Ben Zaken et al. ÿÿÿÿ) (consisting in ÿne-tuning only bias terms in transformer-encoder layers) seem not to present speciÿc drawback for our setting, hence we will compare the latter with transductive approaches in the conducted experiments.

ÿÿÿÿÿ Tÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ

In response to the constraints inherent in NLP-speciÿc methodologies such as prompt-based and ICL strategies, we propose using the episodic framework popularized by meta-learning and mostly used to formalize few-shot learning setting in computer vision, and we adapt it to the NLP paradigm.

Let Ω be the considered vocabulary, we denote Ω å its Kleene closure. The Kleene closure corresponds to sequences of arbitrary size written with tokens in Ω, i.e., Ω å = 1 S i=0 Ω i . Given an input space X with X 7 Ω å , a latent space Z and a label space Y, we consider a pre-trained backbone model g θ : X ! Z = R d , where θ 2 Θ represents the parameters of the encoder and d is the embedding dimension size.

The objective of few-shot classiÿcation is to learn a classiÿer h φ : Z ! Y from limited labeled data and generalize to new, unseen tasks or classes. To accomplish this, we consider transferlearning-based strategies that are evaluated on an episodic testing setting. In such setting, randomly sampled few-shot tasks are created from a test dataset D test := {(x i , y i )} Ntest i=1 that has a set of classes Y test , unseen by the backbone during pre-training. To follow the nomenclature of the FSL literature, each few-shot classiÿcation task is deÿned by the number of targeted classes K and is composed of a support set S and a query set Q. For each class 1 ÿ k ÿ K, N S labeled samples from the class k are randomly sampled from D test to compose S, while N Q diÿerent and unlabeled samples from the class k are randomly sampled from D test to compose Q. Thus, S = x i , y i i2I S with Card(S) = N S å K, and

Q = x i i2I Q , with Card(Q) = N Q å K.

Remarks.

• Contrary to the works of computer vision, there is no necessary distinction between the dataset used to pre-train the backbone g θ and the test dataset D test . Indeed, as the current pre-training corpora are mostly composed of pages of the entire internet (or a large part of it), it seems diÿcult to check that the model did not see test samples during pre-training stage. However, in NLP the backbone is pre-trained using self-supervised objectives (rather than supervised tasks), therefore there is no risk of overlap between pre-training and testing tasks.

• Episodic testing is slightly diÿerent than the original episodic training introduced in metalearning approaches. In the latter, a single model is incrementally trained or ÿne-tuned on the diÿerent tasks, improving its robustness and generalization task after task. Diÿerently, we use the episodic setting as an evaluation protocol, meaning that a diÿerent model is initialized for each generated few-shot task, and all tasks are compiled independently in parallel. This approach allows to compute more reliable performance statistics by evaluating the generalization capabilities of each method on a more diverse set of tasks. Finally, as we want to evaluate the performance of NLP-FSL approaches for larger number of classes, in this very chapter we ÿx the number of ways to be equal to the number of classes of the test dataset, i.e. K = Card(Y test ).

ÿÿÿ Tÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿ FSL ÿÿ NLP

To alleviate the drawbacks of few-shot approaches using prompting strategies, and especially the extensive manual engineering needed for designing all verbalizers for multiclass classiÿcation, we ÿÿÿ . Transductive approaches for FSL in NLP explore transductive approaches that achieved promising results in computer vision community such as TIM (Boudiaf et al. ÿÿÿÿ), and their application to NLP classiÿcation. Speciÿcally, we train a classiÿcation head h φ : Z ! R K mapping the representations features to the posterior distribution space to perform prediction. To simplify the equations for the rest of the paper, we use the following notations for the posterior predictions of each i 2 I S [ I Q and for the class marginals within Q:

p i k = h φ (g θ (x i )) k = P(Y = k|X = x i ; θ, φ) and b p k = 1 |Q| X x i 2Q p i k = P(Y Q = k; θ, φ)
where X and Y are the random variables associated with the raw features and labels, respectively, and where Y Q means restriction of the random variable Y to set Q.

The global classiÿer f φ * ,θ * = h φ * g θ * is obtained by simultaneously training the classiÿcation head and ÿne-tuning the feature extractor such that they solve the following objective:

(φ å , θ å ) = arg min φ,θ CE λ å R Q (ÿ.ÿ) with CE:= 1 |S| X i2I S K X k=1 y i k log(p i k )
being the cross-entropy supervision on the support set (in which y i k is the k th coordinate of the one-hot encoded label vector associated to sample i) and R Q being a transductive loss on the query set Q. The exact deÿnition of R Q depends on the transductive approach. It is worth noting that transductive regularization has been introduced in literature, grounded in the InfoMax principle (Cardoso ÿÿÿÿ; Linsker ÿÿÿÿ). In the upcoming paragraph, we provide an overview of the transductive techniques presented in prior works.

Entropic Minimization An eÿective regularizer for transductive FSL can be derived from the ÿeld of semi-supervised learning, drawing inspiration from the approach introduced in (Grandvalet et al. ÿÿÿÿ). This regularizer, proposed in (Dhillon et al. ÿÿÿÿ), utilizes the conditional Shannon Entropy (Cover ÿÿÿÿ) of forecast results from query samples during testing to enhance model generalization. Formally:

R H Q = 1 |Q| X i2I Q K X k=1 p i k log(p i k ) (ÿ.ÿ)
Mutual Information Maximization A promising alternative to the entropic minimization for addressing the challenges of transductive FSL is to adopt the Info-max principle. (Boudiaf et al. ÿÿÿÿ) extended this idea, introduced in (W. Hu et al. ÿÿÿÿ), and proposed as regularizer a surrogate of the mutual-information R I Q (β):

R I Q (β) = K X k=1 pk log pk + β 1 |Q| X i2I Q K X k=1 p i k log(p i k ) (ÿ.ÿ) = Ĥ(Y Q ) + β( Ĥ(Y Q |X Q )) (ÿ.ÿ) ÿÿÿ
A transductive approach for performing few-shot classi cation in NLP where Ĥ(Y Q ) and Ĥ(Y Q |X Q ) are Monte-Carlo estimators of the marginal entropy of the query set and the negative conditional entropy over labels given features on the query set, respectively. Hence the maximization of the second term (when minimizing R I Q (β)) in Equation ÿ.ÿ makes the classiÿer more conÿdent, making its posterior distribution more spiky, while the maximization of the ÿrst term prevents the model to degenerate by always predicting the same class. The balance between the two terms of the loss is controlled by the hyperparameter β.

The α-TIM method (Veilleux et al. ÿÿÿÿ) extends the TIM setting by considering imbalanced datasets, hence non-uniform labels distributions. The corresponding R Iα Q loss is in that sense based on empirical Tsallis α entropy Ĥα rather than on Shannon entropy:

R Iα Q = 1 α 1 0 @ 1 |Q| X i2I Q K X k=1 (p i k ) α K X k=1 pα k 1 A (ÿ.ÿ) = Ĥα (Y Q ) Ĥα (Y Q |X Q ) (ÿ.ÿ) (Veilleux et al. ÿÿÿÿ)
empirically show that using estimators of Tsallis entropy is indeed better suited to handle imbalanced classes than Shanon entropy.

We ÿnally compare these methods with an inductive baseline:

Linear probing The inductive baseline loss can be obtained by assigning λ = 0. We refer to this approach as Linear Probing: ÿne-tuning a linear head on top of a pre-trained model is a popular approach to learn a classiÿer for various classiÿcation tasks and was originally proposed in (Devlin et al. ÿÿÿÿ).

ÿÿÿ Eÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿ NLP ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

In this section we describe the experimental protocol and results to compare the performances of these diÿerent transductive methods for the task of few-shot text classiÿcation in realistic settings.

ÿÿÿÿÿ Lÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

Previous studies on textual few-shot classiÿcation (Gao, Fisch, et al. ÿÿÿÿ; Mahabadi et al. ÿÿÿÿ; Schick et al. ÿÿÿÿ; Schick et al. ÿÿÿÿ; Tam et al. ÿÿÿÿ) have predominantly assessed their algorithms on classiÿcation tasks with a restricted number of labels (typically less than ÿve). The statistics of mostly used datasets in these works are depicted in Table ÿ.ÿ. Real-world problems yet often comprise larger multi-class classiÿcation tasks, which could undermine current FSL methods due to the ÿÿÿ . Experimental study of transductive few-shot inference for NLP classi cation signiÿcant required handcraft engineering. We take a step forward and consider datasets that are more representative of real-world scenarios. Hence, we decided to run our tests on the following datasets:

• Tweet eval (Barbieri et al. ÿÿÿÿ) contains english tweets annotated with ÿÿ diÿerent emojis.

• Bankingÿÿ (Casanueva et al. ÿÿÿÿ) contains online banking customer service queries annotated with their intents, distributed among ÿÿ classes.

in Equation ÿ.ÿ). The diÿerent plots in Figure ÿ.ÿ represent the classiÿcation accuracy on test set for diÿerent values of N S , consider K = 20 classes. These speciÿc plots correspond to the Tweet_eval dataset, with BERT as the pre-trained backbone, and a classiÿcation head composed of two linears layers (ÿÿÿåÿÿÿ and ÿÿÿåÿÿ) separated by a relu activation. For each bar, the accuracy is averaged on ÿ diÿerent seeds and a ÿÿ% conÿdence interval is given. While the method consisting in minimizing Shannon entropy for conditional output distributions struggles to compete with other strategies for one-shot setting, none of the presented approaches clearly has an edge over the other ones and especially not signiÿcatively on the inductive baseline consisting in ÿne-tuning a classiÿcation head with cross-entropy (CE). From there, we try to explore the diÿerent reasons that could explain the ineÿciency of transductive methods over inductive ones on NLP tasks, as the performance improvement claimed on vision tasks was promising. Speciÿcally, we focused on comparing the inductive baseline only with the TIM approach, as it was proven to be eÿective on the vision tasks.

RQ : Does the number of parameters of backbone have an impact ?

A possible way to explain the fact that transductive methods struggle to beat inductive ÿnetuning on few-shot textual classiÿcation may reside in the quality of representations learned by the pre-trained backbone. Thus, we try here to compare the diÿerence of performances between a pre-trained BERT-base architecture (ÿÿÿM parameters) and a RoBERTa-large architecture (ÿÿÿM ÿÿÿ . Experimental study of transductive few-shot inference for NLP classi cation parameters). In the meantime, we focus on the Bankingÿÿ dataset for evaluation, as its test set is balanced with ÿÿ samples per class. Indeed, the Tweet_eval set is unbalanced, which may undermine TIM performances, as the intuition of this approach is to push the label distribution towards a uniform distribution. The results are illustrated in Figure ÿ.ÿ. This plot clearly denies this hypothesis: if improving the initial representation by increasing the capacity of the pre-trained backbone clearly results in a performance improvement, the transductive method does not compete with its inductive counterpart.

Finally we also compare the performances of such architecture with a diÿerent classiÿcation head. Namely, as in (Boudiaf et al. ÿÿÿÿ), we suppose that:

p i k / exp ã τ 2 kφ k z i k 2 ; (ÿ.ÿ)
where Φ := [φ 1 , . . . , φ K ] denotes learnable classiÿer weights, z i = g θ (x i ) kg θ (x i )k 2 are the normalized representations produced by pre-trained backbone and τ is a temperature parameter. In this setting, classication head weights Φ are initialized as the prototypes of the support set, as introduced in (Snell et al. ÿÿÿÿ):

φ (0) k = P i2I S y i k z i P i2I S y i k
The results of the experiment are illustrated in As we can see, initializing the weight matrix of the classiÿcation head according to the prototypes of the support set also does not help the transductive method, which faces a decrease in performance across all few-shot regimes (with only an accuracy similar to CE for N S = 100).

RQ : Which ne-tuning strategy improve results?

Eventually, we try diÿerent ÿne-tuning strategies to improve accuracy on the few-shot classiÿcation task:

• Freezing all the weights of the pre-trained backbone, and only ÿne-tuning the classiÿcation head. This strategy is referred as "Frozen LM" on the plots.

• Freezing all the weights of the pre-trained backbone except the parameters controlling the layer normalization procedures, and the classiÿcation head. This strategy is referred as "Lay-erNorm" on the plots.

• Freezing all the weights of the pre-trained backbone except the bias parameters, and the classiÿcation head. This strategy is referred as "BitFit" (Ben Zaken et al. ÿÿÿÿ) on the plots.

• Fine-tuning all parameters of the model. This strategy is referred as "Complete" on the plots.

The detailed results are reported in Table ÿ.ÿ with relative gains of TIM regularizer over inductivebased method, while Figure ÿ.ÿ illustrates them as bar plots.

Our analysis reveals that exhaustive ÿne-tuning of all model parameters does not necessarily guarantee superior outcomes when juxtaposed with alternative strategies like BitFit or LayerNorm. Interestingly, these strategies oÿer a more cost-eÿective approach to ÿne-tuning, and in certain data regimes, they even surpass the performance of complete ÿne-tuning. It is noteworthy (but ÿÿÿ . Experimental study of transductive few-shot inference for NLP classi cation not surprising) that maintaining frozen weights for the pre-trained feature extractor g θ consistently resulted in inferior performance, as the model ability to adapt to unseen classes is restricted to the classiÿcation head parameters. However, if we focus on the Gain column in Table ÿ.ÿ, we observe that this ÿne-tuning conÿguration is the one in which TIM regularizer most competes (and sometimes slightly surpasses) CE-based ÿne-tuning. This is more coherent with the results obtained in original TIM work (Boudiaf et al. ÿÿÿÿ), for which the parameters of the visual feature extractor are frozen.

Cÿÿÿÿÿÿÿÿÿ

In this chapter, we delved into the utilization of transductive losses as supplementary objectives for textual few-shot classiÿcation, aiming to address the limitations of promptingbased and in-context-learning-based approaches in real-world few-shot scenarios with a vast number of classes. Throughout our experiments, we evaluated the performance of traditional transductive regularizers applied to textual few-shot classiÿcation. We discovered that transductive methods have diÿculty outperforming inductive cross-entropybased ÿne-tuning when there is some ÿexibility in the pre-trained feature extractor g θ parameters, regardless of g θ 's capacity or the classiÿcation head h φ 's initialization. Last but not least, we found that by ÿxing all parameters of g θ , the transductive approach ÿnally rivals the inductive one. Building on this insight, the next chapter will focus on examining textual few-shot classiÿcation in an API-based setting.

ÿÿÿ ÿ Tÿÿÿÿÿÿ FÿÿÿSÿÿÿ Cÿÿÿÿÿÿÿÿÿÿÿÿÿ Fÿÿ APIÿÿÿÿÿÿ Mÿÿÿÿÿ Cÿÿÿÿÿÿ'ÿ Sÿÿÿÿÿÿ
In this chapter, we address the increasing prevalence of proprietary and closed APIs for large language models like GPT-ÿ and ChatGPT, which have signiÿcant implications for practical applications of NLP, including few-shot classiÿcation. Few-shot classiÿcation entails training a model to execute a new classiÿcation task with minimal labeled data. Our investigation presents three key contributions. Firstly, we introduce a situation in which a pre-trained model is made accessible through a gated API, taking into account computecost and data-privacy constraints. Secondly, we delve deeper into the application of transductive inference, a learning paradigm that has been relatively underexplored within the NLP community. As opposed to traditional inductive learning, transductive inference takes advantage of the statistics of unlabeled data. In this context, we also introduce a new parameter-free transductive regularizer based on the Fisher-Rao loss, demonstrating its applicability and eÿectiveness in the gated API embedding setting. This approach fully leverages unlabeled data, avoids sharing any label information with third-party API providers, and could serve as a baseline for future research. Finally, we propose an enhanced experimental setting and compile a benchmark of eight datasets encompassing multiclass classiÿcation in four diÿerent languages, with up to ÿÿÿ classes. We evaluate our methods using eight backbone models and an episodic evaluation across ÿ,ÿÿÿ episodes, which demonstrate the superiority of transductive inference over the standard inductive setting.

ÿÿÿ Iÿÿÿÿÿÿÿÿÿÿÿ

Recent advances in NLP have been largely driven by the scaling paradigm (Kaplan et al. ÿÿÿÿ; Rosen- feld et al. ÿÿÿÿ), where larger models with increased parameters have been shown to achieve state-ofthe-art results in various NLP tasks (Radford, J. Wu, et al. ÿÿÿÿ; Touvron et al. ÿÿÿÿ). This approach has led to the development of foundation models such as ChatGPT Despite the success of the scaling paradigm, signiÿcant challenges still exist especially when the many practical constraints of real-world scenarios have to be met: labeled data can be severely limited (i.e., few-shot scenario (Y. Song et al. ÿÿÿÿ; Ye et al. ÿÿÿÿ)), data privacy is critical for many industries and has become the subject of increasingly many regulatory pieces (Commission ÿÿÿÿ; Com- ÿÿÿ Textual Few-Shot Classi cation For API-based Models mission ÿÿÿÿ), compute costs need to be optimized (Strubell et al. ÿÿÿÿ). Furthermore, these challenges are made even more complex as stronger foundation models are now available only through APIs (e.g., OpenAI's GPT-ÿ, GPT-ÿ or ChatGPT, Anthropic's Claude or Google's PaLM (Chowdhery et al. ÿÿÿÿ)) which has led to some of their parameters being concealed, presenting new challenges for model adaptation (Solaiman ÿÿÿÿ). This chapter is still centered on the fundamental task of few-shot text classiÿcation, but with a speciÿcal focus on cloud-based/API access, as their ease of integration, reduced infrastructure overhead, and the ability to leverage cutting-edge models is likely to become the standard approach for numerous enterprises looking to implement few-shot NLP classiÿcation tasks. Speciÿcally, we formulate three requirements for API-based FSL (see Figure ÿ.ÿ):

(R ) Black-box scenario. We focus on learning from models that are opaquely deployed in production to the end-user, who only has access to the end-point of the encoder, i.e., the resulting text embedding produced by the ÿnal layer of the network.

(R ) Low resources / computation time. AI systems are often required to make rapid predictions at high frequencies in various real-world applications. Therefore, any few-shot classiÿer used in such scenarios should have a low training and inference time, as well as require minimal computational resources.

(R ) Limited Data Sharing. When utilizing API models, data sharing becomes a major concern. In the current landscape, providers are increasingly oÿering less transparent procedures for training their networks. As a result, users prefer sharing as little information as possible, such as labeling schema and annotated data, to safeguard their data privacy.

While numerous previous studies have addressed the popular few-shot classiÿcation setting, to our knowledge no existing line of work adequately satisÿes the three API requirements described above. In particular, prompt-based FSL (Schick et al. ÿÿÿÿ) and parameter-eÿcient ÿne-tuning FSL (Houlsby et al. ÿÿÿÿ) both require access to the model's gradients, while In-Context learning scales poorly with the task's size (e.g number of shots, number of classes) (Brown et al. ÿÿÿÿ; Y. Chen et al. ÿÿÿÿ; S. Min, Lewis, et al. ÿÿÿÿ; S. Min, X. Lyu, et al. ÿÿÿÿ) and requires full data sharing. Instead, in this work, we focus on methods that can operate within API-based constraints. Under R , R , and R requirements, the standard inductive learning (Haokun Liu et al. ÿÿÿÿ) may be quite limiting. To mitigate the labeled data scarcity while retaining API compliance, we once again explore transduction (V. N. Vapnik ÿÿÿÿ) in the context of textual few-shot classiÿcation. Speciÿcally, in the context of few-shot learning, transductive FSL (Yanbin Liu et al. ÿÿÿÿ) advocates leveraging unlabeled test samples of a task as an additional source of information on the underlying task's data distribution in order to better deÿne decision boundaries. Such additional source essentially comes for free in many o ine applications, including sentiment analysis for customer feedback, legal document classiÿcation, or text-based medical diagnosis. For this API-based setting, our ÿndings corroborate the recent ÿndings in computer vision (Boudiaf et al. ÿÿÿÿ; Y. Hu et al. ÿÿÿÿ; Lichtenstein et al. ÿÿÿÿ; Yanbin Liu et al. ÿÿÿÿ; Ziko et al. ÿÿÿÿ), that substantial gains can be obtained from using transduction over induction, opening new avenue of research for the NLP community. This is in adequation with the last ÿndings of Chapter ÿ, ÿÿÿ when we considered a frozen backbone. We discuss the links between the two chapters in Subsection ÿ.ÿ.ÿ. However, the transductive gain usually comes at the cost of introducing additional hyperparameters, and carefully tuning them. Motivated by Occam's razor principle, we propose a novel hyperparameter-free transductive regularizer based on Fisher-Rao distances and demonstrate the strongest predictive performances across various benchmarks and models while keeping hyperparameter tuning minimal, thereby emphasizing its eÿectiveness and practicality in the current context. We believe that this parameter-free transductive regularizer can serve as a baseline for future research.

Cÿÿÿÿÿÿ'ÿ Cÿÿÿÿÿÿÿÿÿÿÿÿ

In this chapter, our contributions are threefold:

A new textual few-shot scenario: We present a new scenario for FSL using textual API-based models that accurately captures real-world constraints. Our novel scenario opens up new research avenues and opportunities to address the challenges associated with FSL using API-based models, paving the way for improved performance and practical applications in the ÿeld. We show that current NLP FSL approaches all face limitations to tackle classiÿcation in this setting.

A novel transductive baseline:

We propose a transductive FSL algorithm that utilizes a novel parameter-free Fisher-Rao based loss. By leveraging only the network's embedding (R ), our approach enables fast and eÿcient predictions (R ) without the need to share the labeling schema or the labels of few-shot examples making it compliant with (R ). This innovative method marks a signiÿcant step forward in the ÿeld of few-shot learning, oÿering improved performance and practicality for real-world applications.

A truly improved experimental setting: Previous studies on textual few-shot classiÿcation (Gao, Fisch, et al. ÿÿÿÿ; Mahabadi et al. ÿÿÿÿ; Schick et al. ÿÿÿÿ; Schick et al. ÿÿÿÿ; Tam et al. ÿÿÿÿ) have predominantly assessed their algorithms on classiÿcation tasks with a restricted number of labels (typically less than ÿve). In line with the previous chapter, we take a step forward and create a benchmark that is more representative of real-world scenarios. Our benchmark relies on a total of eight datasets, covering multiclass classiÿcation tasks with up to ÿÿÿ classes, across four diÿerent languages. Moreover, we further enhanced the evaluation process by not only considering ÿÿ classiÿers trained with ÿÿ diÿerent seeds (Logan IV et al. ÿÿÿÿ; Mahabadi et al. ÿÿÿÿ), but also by relying on episodic evaluation on ÿ,ÿÿÿ episodes (Hospedales et al. ÿÿÿÿ). Our results clearly demonstrate the superiority of transductive methods.

ÿÿÿ Textual Few-Shot Classi cation For API-based Models ÿÿÿ API ÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿ Pÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ As in the framework deÿned in Subsection ÿ.ÿ.ÿ, we consider a vocabulary Ω, an input space X with X 7 Ω * and a latent space Z. We then seek to learn a classiÿer from limited labeled data and generalize to new, unseen tasks or classes by adapting a pre-trained backbone model g θ : X ! Z, by the mean of few-shot tasks created from a test dataset D test . Each task has a support set S composed of N S å K labeled examples and a query set

Q composed of N Q å K unlabeled examples, sampled between K unseen classes.
Setting the values of N and K in textual FSL is not standardized. Therefore, in all of our experiments, we have relied on setting (N, K) 2 {5, 10} 2 . In the API-based setting, the main diÿerence is that we assume that we are unable to access the exact structure of g θ as mentioned in R . However, we do have access to the last embedding of the encoder which is available for our use (see R ) ) with an API is not possible due to the need to compute gradients of the encoder (as per R ) and the requirement to send both the labeling schema and the labels, which violates R . In Context Learning. A signiÿcant drawback of this approach (Wei, Xuezhi Wang, et al. ÿÿÿÿ) is that the user must supply the input, label examples, and task description, which is both slow (Haokun Liu et al. ÿÿÿÿ) (R ) and raises data privacy concerns (as highlighted in R ). Additionally, the inability to reuse text embeddings for new tasks or with new labels without querying the model's API limits practicality and scalability, making reusable encoding unfeasible for ICL models. Meta-learning. Unlike the three previous lines of work, meta-learning methods operate by modifying the pre-training procedure and therefore assume access to both the training data and the model, which wholly breaks both R and R .

ÿÿÿÿÿ Pÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿ

As in Chapter ÿ, our goal is to learn a classiÿer f φ * ,θ * = h φ g θ . However, as in the API-based setting we cannot access backbone parameters θ, we aim to train only the classiÿcation head h φ by solving the related objective:

φ * = arg min φ CE λ å R Q (ÿ.ÿ) with CE= 1 |S| X i∈I S K X k=1 y i k log(p i k )
being the cross-entropy supervision on the support set and R Q being a transductive loss on the query set Q. In the conducted experiments, we chose to compare the transductive methods based on Entropic Minimization (H) and TIM algorithm (I), associated to respective regularizers R H Q and R I Q (β) (as introduced in Equation ÿ.ÿ and Equation ÿ.ÿ) with the Linear probing inductive baseline (CE). All three methods were introduced in Chapter ÿ. We ÿnally consider another inductive baseline: Prototypical Networks (P T ). Prototypical Networks learn a metric space where the distance between two points corresponds to their degree of similarity. During inference, the distance between the query example and each class prototype is computed, and the predicted label is the class with the closest prototype. Prototypical networks have been widely used in NLP and are considered as a strong baseline (Gao, X. Han, et al. ÿÿÿÿ; Snell et al. ÿÿÿÿ; S. Sun et al. ÿÿÿÿ).

Limitation of existing transductive strategies: Despite its eÿectiveness, the TIM method implies the need to ÿne-tune the weight of diÿerent entropies using the hyperparameter β. This hyperparameter-tuning process can be time-consuming and may require extensive experimentation to achieve optimal results. Additionally, recent studies have shown that relying solely on the ÿ The cost of API queries is determined by the number of input tokens that are transmitted.

ÿÿÿ

Textual Few-Shot Classi cation For API-based Models ÿrst entropic term, which corresponds to the Entropic Minimization scenario in Equation ÿ.ÿ, can lead to suboptimal performance in FSL.

ÿÿÿÿÿ A FÿÿÿÿÿÿRÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ

In the FSL scenario, minimizing parameter tuning is crucial. Motivated by this, in this section we introduce a new parameter-free transductive regularizer which ÿts into the InfoMax framework. Additionally, our loss inherits the attractive properties of the recently introduced Fisher-Rao distance between soft-predictions q := (q 1 , . . . , q K ) and p := (p 1 , . . . , p K ), which is given by (Picot et al. ÿÿÿÿ) and (Gomes et al. ÿÿÿÿ):

d FR (q, p) := 2 arccos K X k=1 p q k å p k ! . (ÿ.ÿ)
The proposed transductive regularizer denoted by R FR Q , for each single few-shot task, can be described as measuring the Fisher-Rao distance between pairs of query samples:

R FR Q := 1 |Q| X i∈Q log X j∈Q K X k=1 q p i k å p j k = 1 |Q| X i∈Q log X j∈Q cos 7 d FR (p i , p j ) 2 ç , (ÿ.ÿ)
where d FR (p i , p j ) is the Fisher-Rao distance between pairs of soft-predictions (p i , p j ). Furthermore, it is shown that expression (ÿ.ÿ) yields a surrogate of the Mutual Information as shown by the following proposition. This result to the best of our knowledge is new, as far as we can tell.

Proposition . (Fisher-Rao as a surrogate to maximize Mutual Information) Let (q i ) i∈Q be a collection of soft-predictions corresponding to the query samples. Then, it holds that:

R FR Q + log |Q| ÿ R I Q (1) ÿ R I Q (α), 8 0 ÿ α ÿ 1. (ÿ.ÿ)
Proof: Further details are relegated to Section ÿ.ÿ.

Advantage of R FR Q over R I Q (β): Similarly to R I Q (β), R FR
Q can be exploited to maximize the Mutual Information. However, R FR Q is parameter free and thus, it does not require to tune β.

ÿÿÿ Aÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ

In this chapter, we put a special emphasis on the experimental setting, which builds upon the limitations from prior studies and observations outlined in Chapter ÿ. Speciÿcally, we underscore the diversity in our evaluation datasets. These datasets are characterized by a broad range of classes and varied label distributions, further enhancing their robustness. Moreover, drawing from the performance disparities observed between the BERT and RoBERTa backbones in the previous chapter, we initiate an in-depth exploration involving multiple pre-trained backbones, spanning both monolingual and multilingual scopes. Finally, we direct our attention towards the capacity for generalization and adaptability in this chapter. As such, we integrate a greater number of tasks that contain fewer sampled classes per task.

ÿÿÿ . An enhanced experimental setting ÿÿÿÿÿ Dÿÿÿÿÿÿÿ

Benchmarking the performance of FSL methods on diverse set of datasets is critical to evaluate their generalization capabilities in a robust manner as well as their potential on real-world applications. As mentioned in Subsection ÿ.ÿ.ÿ, previous work on FSL (Mahabadi et ÿÿ Go Emotion (Demszky et al. ÿÿÿÿ) ÿÿ Tweet_eval (Barbieri et al. ÿÿÿÿ) ÿÿ Bankingÿÿ (Casanueva et al. ÿÿÿÿ) ÿÿ Clinc (Larson et al. ÿÿÿÿ) ÿÿÿ Speciÿcally, besides Tweet_eval (Barbieri et al. ÿÿÿÿ), Bankingÿÿ (Casanueva et al. ÿÿÿÿ) studied in Chapter ÿ, we consider:

• Multilingual Amazon Reviews Corpus (MARC) (Casanueva et al. ÿÿÿÿ), that consists of reviews extracted from diÿerent Amazon marketplaces. The reviews comprise six languages: English, German, French, Spanish, Japanese, and Chinese.

• Go Emotion (Demszky et al. ÿÿÿÿ), which contains Reddit comments extracted from popular English-language subreddits and labeled with emotion categories.

• Clinc (Larson et al. ÿÿÿÿ), that consists of thousands of annotated examples of natural language queries and responses, covering ÿÿÿ intent classes over ÿÿ domains, and one out-ofscope class.

These datasets cover a wide range of text classiÿcation scenarios and are of various diÿculty. A summary of the datasets used can be found in Table ÿ.ÿ. They are all available in Dataset (Lhoest et al. ÿÿÿÿ).

ÿÿÿÿÿ Mÿÿÿÿ ÿÿÿÿÿÿ

The selection of an appropriate backbone model is a critical factor in achieving high performance in few-shot NLP tasks. To ensure the validity and robustness of our ÿndings, we have included a diverse range of transformer-based backbone models in our study, including:

• Three diÿerent sizes of RoBERTa-based models (Yinhan Liu et al. ÿÿÿÿ). Similar to BERT, RoBERTa is pre-trained using the cloze task (W. L. Taylor ÿÿÿÿ). We consider two diÿerent sizes of the RoBERTa model, namely RoBERTa (B) with ÿÿÿM parameters and RoBERTa (L) with ÿÿÿM parameters and DistilRoBERTa, a lighter version of RoBERTa trained through a distillation process (Geoÿrey E. Hinton et al. ÿÿÿÿ), for a total of ÿÿM parameters.

ÿÿÿ Textual Few-Shot Classi cation For API-based Models

• Three sentence-transformers encoder (Reimers et al. ÿÿÿÿ). Following the recommendation of (Muennighoÿ et • Multilingual models. To address realistic scenarios, we do not restrict our study to the English language. We rely on three sizes of XLM-RoBERTa (Conneau et al. ÿÿÿÿ): base (B) with ÿÿÿM, large with ÿÿÿM (L) and XL (XL) with ÿ.ÿB of parameters.

• GPT-ÿ.ÿ model: to mimic the typical setting of API-based models, we also conduct experiments on GPT-ÿ.ÿ (Brown et al. ÿÿÿÿ), only accessible through OpenAI's API.

Preliminary Experiment. In our experiments, the backbone models are of utmost importance. Our objective in this preliminary experiment is to assess the eÿcacy of these models when ÿnetuning only the model head across a variety of datasets. Through this evaluation, we aim to gain insight into their generalization abilities and any dataset-speciÿc factors that may inÿuence their performance. This information is used to analyze the performance of diÿerent models in the fewshot scenario, as described in Section ÿ.ÿ. We present the results of this experiment in Table ÿ.ÿ, noting that all classes were considered, which diÿers from the episodic approach detailed in Section ÿ.ÿ. ÿÿÿÿÿ Eÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ

Prior research in textual FSL typically involves sampling a low number of tasks, generally less than ÿ, of each dataset. In contrast, we utilize an episodic testing framework that generates a large number of N-shots K-ways tasks. To account for the model's generalization ability, we average the results for each dataset over ÿÿÿÿ episodes, with the K considered classes varying in every episode.

For each experiment, we consider the Fÿ-Score as the evaluation metric.

ÿÿÿ

. In order to thoroughly analyze the performance of each method, we conducted a per-dataset study, beginning with a focus on the mono-lingual datasets. In this experiment, we investigated the performance of diÿerent loss functions under varying conditions of 'ways' and 'shots'. As shown in Figure ÿ.ÿ, we observed that increasing the number of classes ('ways') led to a decrease in Fÿ-score while increasing the number of examples per class ('shots') led to an improvement in Fÿ-score. This can be explained by the fact that having more data enables the classiÿer to better discern the unique characteristics of each class. Interestingly, the relationship between the number of shots and classiÿcation Fÿ-score may not be the same for all classes or all loss functions. Figure ÿ.ÿ shows that diÿerent loss functions (e.g. FR on Bankingÿÿ) beneÿted greatly from adding a few shots, while others did not show as much improvement. However, this variability is dependent on the speciÿc dataset and language being used, as diÿerent classes may have diÿerent levels of complexity and variability, and some may be inherently easier or harder to classify than others. tive regularization technique using FR outperforms other methods on GPT-ÿ.ÿ. This highlights the eÿectiveness of FR in improving the performance of the model and suggests that transductive regularization may be a promising approach for optimizing language models.

ÿÿÿÿÿ A ÿÿÿÿ ÿÿÿÿ GPTÿÿÿÿ ÿÿÿÿÿÿÿ GPT-ÿ.ÿ appears to be the backbone providing the most informative a priori embeddings in Table ÿ.ÿ and could be considered as the prime model for API-based FSL, showcasing the current requirements in this area. It is thus a typical candidate for application uses that must meet the following criteria (R ) -(R ). Therefore, we put a special emphasis on its related results. (CE and PT), the underperformance of the entropic-minimization-based strategy (H), and the higher amount of information conveyed by GPT-ÿ.ÿ learned embeddings over other backbones, resulting in higher Fÿ-scores on all datasets.

These phenomena still occur in the multi-lingual setting, as illustrated in Figure ÿ.ÿ (right), stressing the superiority of transductive (and especially FR) over other approaches for presumably universal tasks, beyond english-centered ones, and without the need of using language-speciÿc engineering as for prompting-based strategies.

ÿÿÿ Textual Few-Shot Classi cation For API-based Models Note that for both of these settings, the entropic-minimization-based strategy (H) seems to be capped at a ÿÿ% Fÿ-score, thus with no improvement over other backbones embeddings, and independently of the dataset diÿculty. ÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ To provide an exhaustive analysis, we report the same experiment that is made in Subsection ÿ.ÿ.ÿ for multi-lingual models on the MARC dataset.

The observations made in Subsection ÿ.ÿ.ÿ are not speciÿc to GPT-ÿ.ÿ backbone and extend to the other multi-lingual encoders (that is XLM-RoBERTa-based ones). While both latin languages (French and Spanish) share almost identical results, with a trend very similar to the one of English language (an Fÿ-score gain of around ÿ% for FR over CE), the results on German language exhibit an Fÿscore increased by more than ÿ% when switching from inductive CE to transductive FR, ÿirting with performances obtained on English tasks.

ÿÿÿÿÿ Iÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ

In this section, we report the results of our experiment aggregated per backbone. The goal is to understand how the diÿerent losses behave on the diÿerent backbone. The results are presented in the previous charts are retrieved for the majority of backbones, some of these models are exceptions. For example, while transductive methods perform generally better than inductive methods, the CE-based method seems to perform slightly better than I for XLM-RoBERTa-xl. Additionally, while FR is the most eÿective method for the majority of backbones, it is surpassed by I for the all-distilroberta-vÿ model. Furthermore, the inverse-scaling-law details are found for the RoBERTa(B/L) and XLM-RoBERTa (B/L) models per dataset. In general, it is interesting to note that although model performance is constrained by dataset diÿculty, the performance order of each method is consistent across all ÿ datasets for each considered backbone.

Rÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿ

In this experiment, we report the performance of diÿerent losses on the Amazon dataset by averaging the results over the number of shots, ways, and model backbones.

ÿÿÿÿÿ Pÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ

In this experiment, we adopt a practical standpoint and aim to evaluate the eÿectiveness of an API model, speciÿcally GPT-ÿ.ÿ. In Subsection ÿ.ÿ.ÿ, we report the training speed of one episode on a MAC with CPU. Overall, we observed that the transductive loss is slower as it necessitates the computation of the loss on the query set, whereas PT is faster as it does not involve any optimization. Furthermore, we note that FR is comparable in speed to I. To provide a better understanding of these results, we can compare our method with existing approaches (in the light of R ). For instance, PET (Schick et al. ÿÿÿÿ) ÿÿÿÿÿ Lÿÿÿÿ ÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿ Cÿÿÿÿÿÿ ÿ

In the previous chapter, our comparison between transductive and inductive methods under different ÿne-tuning conditions yielded mixed results. When the backbone parameters were accessible for ÿne-tuning, our experiments indicated an advantage for inductive methods. However, when the backbone parameters were frozen, the performance of the two methods was comparable, with a slight advantage for transductive methods in some data regimes, although the performance diÿerence was not statistically signiÿcant.

The results obtained in the current chapter conÿrm and expand upon these initial promising observations. We ÿnd that transductive methods not only perform at least as well as the inductive ones when the backbone parameters are frozen, but also exhibit even better performance in this setting. One potential explanation for this superior performance of transductive methods in the API-based setting is the adoption of episodic evaluation, where we consider a ÿxed number of classes during inference. This evaluation approach diÿers from the one used in the previous chapter, where all classes were considered simultaneously. Indeed, the ÿxed number of classes during inference reduces the complexity of the problem, allowing transductive methods to better exploit the structure and relationships among the few-shot examples, which is one of the key strengths of transductive learning. Furthermore, episodic evaluation does not discredit the generalization ability of the studied approach, as the reported performances are averaged over ÿÿÿÿ parallel episodes, with diÿerent classes sampled for each episode.

In summary, our ÿndings in this chapter provide strong empirical evidence that transductive methods are a serious candidate for few-shot classiÿcation in an API-based setting, where the backbone parameters are unavailable.

Cÿÿÿÿÿÿÿÿÿ

In this chapter, we have presented a novel few-shot learning framework that eÿectively leverages API models while adhering to the critical constraints of real-world applications (i.e., R , R , R ). The R constraint is particularly relevant and crucial, as current competitive models are only accessible via API, preventing access to model parameters. Our approach is especially appealing as it shifts the computational requirements R , eliminating the need for heavy computations for the user, and enables training classiÿers on-the-ÿy in web browsers without sharing labels of the data R . Building upon the mixed results from the previous chapter, we have demonstrated the signiÿcant advantages of using transductive losses to perform NLP FSL in this API-setting, ÿÿÿ Textual Few-Shot Classi cation For API-based Models that exhibit better performances than inductive ones when the backbone parameters are frozen, with a signiÿcant power of generalization across a large number of new classes and at a consequently aÿordable cost. The regularizer based on the Fisher-Rao distance provides a promising candidate, which is parameter-free and could serve as a straightforward baseline in future studies. In conclusion, in this chapter we successfully addressed the initial motivations for developing an eÿcient and eÿective FSL framework that meets realworld constraints. By shedding light on the potential of transductive losses and demonstrating their practicality in various use-cases, we hope to inspire further exploration and reÿnement of these methods, ultimately contributing to the advancement of FSL in the ÿeld of NLP. ÿÿÿ ÿ Cÿÿÿÿÿÿÿÿÿ ÿÿÿ Pÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ Sÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ

The ÿrst part of this thesis addresses the challenge of exploiting multimodal data for fault diagnosis in the context of Industry . systems. Motivated by a tangible need in the industrial ÿeld, we dove into the exploration of complex multimodal systems. Our journey begins in Chapter ÿ with the development of a theoretical framework based on multimodal learning, motivated by the intricate multimodal nature of our real-world environment. In this framework, we examined established concepts such as multimodal fusion and representation, taking a comprehensive view of the evolution of these paradigms from their early stages to the advent of DL-based multimodal representations. Our analysis, enriched by a focus on previous few attempts to apply Machine Learning for fault diagnosis, highlights the practical constraints of this application that have been overlooked by previous multimodal approaches. In Chapter ÿ, our investigation led to the identiÿcation of ÿve signiÿcant challenges arising from the considered setting. In response, we developed StreaMulT, a Streaming Multimodal Transformer. This architecture employs cross-modal attention and a memory bank to process arbitrarily long input sequences during training and operate in a streaming mode during inference. This approach uniquely addresses the complexity posed by Industry ÿ.ÿ systems, eÿciently managing the temporal unalignment of multimodal heterogeneous data and diÿerences in data acquisition frequency. Despite an access to an adapted industrial dataset, its evaluation on the connected multimodal sentiment analysis task revealed that our model can manage arbitrarily long sequences without a loss in performance. With a carefully selected textual embedding module, StreaMulT surpassed existing methods, setting a new state-of-the-art performance on the CMU-MOSEI dataset. Coupled with the ablation study, this underscored the signiÿcant inÿuence of the textual modality, thus justifying the emphasis placed on it in the second part of the thesis. In Chapter ÿ, we investigated the various interactions within multimodal data, which are categorized into redundant and complementary information types. We underscored the crucial role of complementary information, while simultaneously noting the scarcity of robust methodologies and benchmarks to evaluate the capacity of models to exploit this type of information.

The second part of the thesis is dedicated to harnessing the unique value of textual data in the realm of Industry . , o ering a rich, contextual understanding of system operations, past incidents and expert knowledge. Such insights are crucial for fault diagnosis and predictive maintenance. However, these reports are scarce and use industry-speciÿc language, presenting challenges in processing and interpretation. To overcome this challenge, we adopt the few-shot learning paradigm. As detailed in Chapter ÿ, our exploration dives into the realm of Natural Language Processing, investigating its progression from early methodologies to DL approaches and large Foundation Models. We further highlight the function of FSL and the primary frameworks ÿÿÿ

Conclusion and Perspectives

that facilitate the application of large PLMs within this paradigm. In Chapter ÿ, the limitations of current FSL methods, speciÿcally the engineering eÿorts required for realistic classiÿcation tasks with a large number of classes, are explored. In response, we propose a novel adaptation of transductive techniques for textual classi cation. The study demonstrates that transductive methods rival inductive ones when all parameters of the feature extractor are ÿxed. Finally, in Chapter ÿ we take into consideration the increasing prevalence of proprietary and closed APIs for LLMs. A new scenario for FSL using textual API-based models is presented, highlighting the constraints related to computation cost and data privacy. The chapter introduces a new parameter-free transductive regularizer based on the Fisher-Rao loss, demonstrating its eÿectiveness in the gated API embedding setting. Moreover, it proposes an enhanced experimental setting for compiling a benchmark of datasets encompassing multi-clasThiss classiÿcation in diÿerent languages.

G T

In summary, this thesis we have provided two methodological contributions in two major areas:

. the proposal of the StreaMulT architecture (Pellegrain, Tami, et al. ÿÿÿÿ), a multimodal approach that serves as a pivotal contribution to the evolution of fault diagnosis methodologies,

. the introduction of novel transductive methods for Few-Shot Learning in Natural Language Processing, framed in a realistic and API-based context (under review for publication in an international journal).

Apart from these methodological contributions, we have also proposed:

• a signiÿcantly expanded state-of-the-art in fault diagnosis, with an illustrative designed case study (Pellegrain, Batteux, et al. ÿÿÿÿ).

• important discussions regarding the formalization and characterization of multimodal interactions, particularly the roles of redundancy and complementarity in multimodal representation learning.

This thesis, while oÿering signiÿcant advancements, is a stepping stone in a continually evolving ÿeld of research. As we move forward, it is important to remember that the applications and methodologies described here will need to be tested further and reÿned in response to the challenges and opportunities presented by new developments in Industry ÿ.ÿ. In the following section, we critically examine the contributions of this thesis, and propose possible directions for future research, further strengthening the impact of this work on the broader landscape of Industry ÿ.ÿ systems.

ÿÿÿ

. Perspectives ÿÿÿ Pÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ Cÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ

While StreaMulT presents an eÿective framework in handling multi-modal data processing, it falls short in certain areas that demand closer examination. We thereby list its limitations, focusing on the lack of robustness in performance, issues with missing modalities and imbalanced datasets, and the fully supervised approach it embodies.

Performance Limitations First and maybe most importantly, we did not conduct an exhaustive study of running and latency time. Even though the architecture theoretically allows the handling of arbitrarily long inputs during training and can be deployed in a streaming fashion at inference, the latency becomes a crucial factor. In the case that the system's speed is less than optimal, it undermines the capability of real-time deployment, which is a crucial aspect of the streaming aspect of StreaMulT. This oversight is a signiÿcant drawback, particularly for industrial applications, where timeliness often equals eÿciency. Our choice has been to use a chunk-wise approach with augmented memory, but diÿerent strategies exist, such as monotonic attention Consequently, the model does not account for the adaptation of training in streaming, a critical requirement for maintaining the system's robustness. By not taking these aspects into account, the model could potentially be unsuitable for deployment in a dynamic and ever-evolving industrial setting where data is rarely perfect or consistent. A straightforward perspective is therefore to consider strategies that tackle concept drifts (Souza et al. ÿÿÿÿ), for instance leveraging continual learning (Kirkpatrick et al. ÿÿÿÿ).

Fully Supervised Approach and Its Implications The assumptions made when designing StreaMulT raise some critical questions as well. The model presumes a fully supervised approach and relies heavily on the availability of numerous datapoints from all modalities. This assumption is somewhat contradictory with the assertions made in Part ÿ of the thesis, where we advocate that textual data are scarce. In addition, StreaMulT relies heavily on supervision and backpropagation. These methods are the precise limitations pointed out in Chapter ÿ. StreaMulT hence does not exploit complementary information eÿectively to provide control over representation. Therefore, it is essential to acknowledge that StreaMulT, while being a promising tool, remains bound by the constraints and limitations characteristic of current approaches. Weakly supervised and unsupervised settings could be handled using anomaly detection approaches, if the fault occurrences are rare in the considered setting.

In conclusion, while StreaMulT presents a step forward in multimodal data processing, it faces several critical areas, which must be addressed for its successful implementation in real-world scenarios. Further research is necessary to address these limitations and explore possible solutions that would allow StreaMulT to fully fulÿll its potential.

ÿÿÿ

Conclusion and Perspectives

Our transductive approach introduced in Chapter ÿ to perform textual classiÿcation in FSL paradigm with API-based language models introduces an innovative methodology. Nevertheless, there are several key criticisms that should be addressed in order to truly judge the applicability and eÿcacy of this approach.

Method Speci city and Inference Latency One primary concern is that this method performs optimally for API-based LLMs. The eÿcacy of transductive methods as compared to their inductive counterparts, in settings not based on API, was shown to be signiÿcantly lower in Chapter ÿ. This creates a strong limitation on the scope and utility of this method. Moreover, the inherent latency of transductive methods surpasses that of inductive methods. While the training time has been studied thoroughly, the inference time, a critical factor for real-world applications, was not analyzed exhaustively. For this methodology to be applicable in an industrial setting, where realtime responses are often crucial, this aspect needs to be studied in depth. Dependence on Annotations and Application to Multi-Source Data This approach, even though it employs few annotations, still relies on them. In an industrial setting, one might access a labeled multimodal dataset of faults, but the alignment of these labels with the associated maintenance reports is not guaranteed. Consequently, the challenge arises of how to ensure an appropriate annotation scheme for these reports. Similarly, this approach concentrates solely on textual data. Given the initial goal of incorporating data from multiple sources, a question arises: how can this methodology be implemented in a broader multimodal framework? Some recent works such as (Alayrac, Donahue, et al. ÿÿÿÿ) extend the paradigm of pre-trained LLMs to other modalities, such as images, to perform multimodal FSL with In-Context learning paradigm. Exploring an adaptation of transductive framework to these architectrure thus constitutes an interesting perspective. Privacy Concerns and Dependence on Contemporary Framework Finally, while this approach addresses privacy concerns by leveraging API-based models, the underlying assumption may seem unrealistic. It assumes that labels carry the most sensitive data, rather than the input data. In practice, the input data are often also sensitive and thus of higher concern from a privacy perspective, and anonymization of textual documents might undermine expressive content that is already scarce. As a result, the approach might not be as eÿective in scenarios where privacy of the input data is crucial. In summary, while our transductive approach oÿers a promising solution for NLP few-shot classiÿcation with API-based LLM, several criticisms highlight the areas where further work is needed. This includes the inference latency, reliance on annotations, suitability for multimodal data integration, and privacy concerns related to input data. Addressing these issues could potentially expand the applicability and usefulness of this approach in varied settings.

ÿÿÿÿÿ Lÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ

At the time of AI is preponderant and completely questioning the perspectives of the society, mainly through the Large Language Models breakthrough, the multimodal quest strikes back as a mean of grounding world concepts (Girdhar et al. ÿÿÿÿ). As the last section of this manuscript, we chose to discuss what can appear as a philosophical yet central question: "What is a modality?".

ÿÿÿ . Perspectives

Despite the many previous works studying the best ways to perform multimodal data fusion, mainly through representation learning, there is no formal deÿnition of what is called a modality, and therefore how two diÿerent sources of data can be considered as coming from either same or diÿerent modalities. Indeed, (Baltrusaitis et al. ÿÿÿÿ) informally deÿne a modality as "the way in which something happens or is experienced" and adds that "a research problem is characterized as multimodal when it includes multiple such modalities". Besides, most of previous works follow this kind of informal deÿnitions, and often give as examples of multimodal data the human experience of the world, through human multisensory integration (see Subsection ÿ.ÿ.ÿ).

In light of the heterogeneity gap paradigm, a ÿrst attempt to deÿne the modalities α, β of two data sources X µ and X could be through their deÿnition domain. For instance, a RGB picture (of height h and width w) of a dog in R h×w×3×256 and a text describing a dog, encoded as l different d-length one-hot vectors, lie in vastly diÿerent spaces even though they share redundant information and can be semantically close (they both embed the concept of a dog). However, this deÿnition seems insuÿcient, as two images of diÿerent resolutions exist in diÿerent spaces, yet intuitively do not exhibit heterogeneity gap. On the other hand, some studies treat similar structured inputs, such as RGB and LIDAR images, as distinct modalities under a multimodal framework. Yet, no framework provides tools to determine if these modalities are closer to each other than an image and a text of a dog are.

The heterogeneity gap paradigm supports the previous informal deÿnitions of modality and multimodality, in the sense that all these considerations are human-centered. The challenge it presents is that a prediction model designed for a modality α may not perform eÿciently when applied to a diÿerent modality β, as the data structure diÿers. However, this model design choice is determined according to the assumptions the human learner makes on the input data: namely, an inductive bias. In that sense, we propose to deÿne a modality through the lens of inductive biases.

Considering the multimodal fusion framework of Subsection ÿ.ÿ.ÿ, aside from training algorithm selection, optimization, and loss selection, the model choice is essentially determined by the hypothesis space F. This parameter space is where an inductive bias can be added, particularly in response to the nature and structure of input data, hence their modality. For example, assuming spatial structures in images such as locality or translation invariance, CNNs are a popular choice due to their ability to share weights locally in space. Similarly, RNN are employed to manage the presumed recurrent structure of text. By writing F = H G where H is the representation's hypothesis space and G is the classiÿer's hypothesis space, one can deÿne the notion of modality in a relational way: two sources of data X µ and X are said to be from the same modality if and only if they are processed with the same representation's hypothesis spaces, i.e. G µ = G . This means that the learner applies the same bias when encoding them prior to classiÿcation. With this consideration, we could deÿne a distance between two modalities using a distance between their representation's hypothesis spaces.

Eventually, if this section is just a discussion and a proposition, we truly believe that the definition and characterization of a modality, and understanding its distance with other modalities ÿÿÿ In this Appendix, we prove the inequality (Equation ÿ.ÿ) provided in Proposition ÿ. The righthand side of (Equation ÿ.ÿ) follows straightforwardly from the deÿnition of R I Q (β) and the nonnegativity of the Shannon entropy. In order to prove the ÿrst inequality, we need to introduce the following intermediate result.

For any arbitrary random variable X and countable random variable Y , and any real number β, let

I (X; Y ) := E X ? Y log E X ÿ P (Y |X) P (Y |X ? ) ,
where the random variable X ? follows the same distribution than X. Notice that it is obvious that I 1 (X; Y ) = I(X; Y ), where I(X; Y ) is Shannon Mutual Information.

Lemma . For any arbitrary random variable X and countable random variable Y , we have

I(X; Y ) I (X; Y ), for 0 ÿ β ÿ 1.
Proof of the lemma: We must show that the diÿerent of I(X; Y ) I (X; Y ) is nonnegative. To this end, we write this diÿerence as:

I(X; Y ) I (X; Y ) = E X ? Y log P 1-(Y |X ? )E X P (Y |X) E X P (Y |X) (ÿ.ÿ) log E X ? Y P 1-(Y |X ? )E X P (Y |X) E X P (Y |X) (ÿ.ÿ) = log X y∈Y E X ? P (y|X ? ) P 1-(y|X ? )E X P (y|X) E X P (y|X) (ÿ.ÿ) = log X y∈Y E X ? P (y|X ? )E X P (y|X) E X P (y|X) (ÿ.ÿ) = log X y∈Y E X P (y|X) (ÿ.ÿ) = 0, (ÿ.ÿ)
where the ÿrst inequality follows by applying Jensen's inequality to the function t 7 ! log(t).

ÿÿÿ Appendix Proof of Proposition : From Lemma ÿ, using Jensen's inequality, we have

I(X; Y ) = E X ? Y log E X ÿ P (Y |X) P (Y |X ? ) , (ÿ.ÿ) E X ? Y log E X ÿ P (Y |X) P (Y |X ? ) (ÿ.ÿ) E X ? log E X E Y |X ? ÿ P (Y |X) P (Y |X ? ) (ÿ.ÿ) = E X ? log E X X y∈Y P (Y |X)P 1-(Y |X ? ), (ÿ.ÿÿ)
where inequality (ÿ.ÿ) follows by applying Lemma ÿ and inequality (ÿ.ÿ) follows by exploiting the convexity of the function t 7 ! log(t) for any 0 ÿ β ÿ 1. Finally, it is not diÿcult to check from the deÿnition of the Fisher-Rao distance given by expression (ÿ.ÿ) that Using the identity given by (ÿ.ÿÿ) in expression (ÿ.ÿÿ) setting β = 1/2, we obtain the desired inequality

I(X; Y ) E X ? log E X cos 7 d FR (P (y|X), P (y|X ? )) 2 ç . (ÿ.ÿÿ)
The inequality (ÿ.ÿ) immediately follows by replacing the distribution of the random variable X with the empirical distribution on the query and P (y|x) with the soft-prediction corresponding to the feature x, which concludes the proof of the proposition.

ÿÿÿ Pÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿÿÿÿÿ ÿÿ ÿÿÿ MPO ÿÿÿÿÿÿÿ

In the next pages, we provide the reader the publication (Pellegrain, Batteux, et

Mots-clefs 4 Surveillance, Diagnostic, RŽseau de neurones rŽcurrents, LSTM Abstract4

Managing the maintenance of industrial plants is an important factor of competitiveness. Different technics can be used to ensure maintenance strategies: fault monitoring and diagnosis, for instance, to detect and identify a failure after it occurs. Works presented within this publication show the application of a monitoring algorithm to detect occurrences of failures on an applicative example. These works are realized within the MPO project (Predictive maintenance and Optimization) at IRT SystemX. The example is the 3-Tanks system, already presented in previous works. A machine learning algorithm was implemented, based on data generated by simulation.

Keywords 4 Monitoring, Diagnosis, Recurrent neural network, LSTM

I. INTRODUCTION

La gestion de la maintenance d'installations industrielles de production est un facteur important de compŽtitivitŽ. En effet, de tels systèmes sont composés d9une multitude de composants hŽtŽrog•nes en interactions les uns avec les autres : des composants physiques, des actionneurs, des capteurs, des calculateurs de contr™le/commande. Ajoutons que certains composants embarquent en eux-m•mes de tels ŽlŽments logiciels de contr™le/commande, comme les capteurs dits 8intelligents9. De plus certains de ces syst•mes peuvent •tre distribuŽs en diffŽrents endroits physiques, demandant de ce fait des liens de connexions par rŽseaux (internet par exemple). De tels syst•mes combinant des composants physiques, logiciels et en rŽseaux sont Žgalement appelŽs des 8syst•mes cyber-physiques9 [START_REF] Khaitan | Design Techniques and Applications of Cyberphysical Systems: A Survey[END_REF].

Les composants et parties de ces syst•mes sont naturellement sujets ˆ des dŽfaillances (qui se nomment Žgalement fautes dans la communautŽ du diagnostic), pouvant mener ˆ des dysfonctionnements ou pannes du syst•me. Certaines de ces dŽfaillances peuvent avoir des consŽquences nŽgligeables, m•me si le syst•me ne remplit plus sa fonction : par exemple l9oxydation d9un c‰ble de haut-parleur, qui occasionne soit un mauvais son, soit pas de son, sortant du haut-parleur, et impactant le confort de l9utilisateur. D9autres dŽfaillances peuvent, au contraire, avoir des consŽquences catastrophiques : par exemple l9usure de joints d9étanchéité de durites de freinage, qui am•ne ˆ un dysfonctionnement, voire m•me une perte d9un syst•me de freinage. Dans ce cadre et suivant la sŽvŽritŽ des dysfonctionnements et pannes du syst•me considŽrŽ, il est nŽcessaire de mettre en Suvre des solutions de maintien en conditions opŽrationnelles du système. Même si l9amélioration de la fiabilité des composants, ou les techniques de redondances matŽrielles, peuvent •tre des solutions, elles ne sont nŽanmoins pas suffisantes. En effet, tout composant physique est lié à l9usure matŽrielle et m•nera ˆ des dysfonctionnements ou des pannes. La maintenance joue donc un r™le important pour rŽduire les risques d9occurrence de pannes, en particulier pour des syst•mes dont la panne peut impacter la sŽcuritŽ des personnes.

DiffŽrentes stratŽgies de maintenances existent, et sont rŽsumŽes en Figure 1. Les maintenances correctives se rŽalisent ˆ la suite des occurrences des dŽfaillances. Ë l9inverse les maintenances préventives anticipent les dŽfaillances en se rŽalisant avant leurs occurrences. La maintenance prŽventive est un levier important pour rŽduire les risques de panne et les cožts de maintenance. Cependant, 23ème Congrès Lambda Mu de l9IMdR 10 au 13 octobre 2022, EDF Lab Paris Saclay réaliser trop d9actions de maintenance préventives pourrait se rŽvŽler plus cožteux que nŽcessaire. Il existe un Žquilibre entre l9investissement en maintenance préventive et le risque de dŽfaillance. Une analyse de la fiabilitŽ du syst•me, ˆ travers l9étude des données historique de panne et/ou l9élicitation d9expert de son fonctionnement, permettra de calculer des indicateurs d9aide à la décision permettant de trouver un Žquilibre optimal.

Figure 1 : Les diffŽrents types de maintenances

Différentes techniques existent afin d9assurer au mieux les stratŽgies de maintenance. La surveillance et le diagnostic permettent de détecter et d9identifier un comportement anormal du système (défaillance de l9un des composants) avant que cela ait un impact important. Le pronostic permet d9estimer la durée avant l9occurrence de la défaillance ou la panne (nous reviendrons sur ces notions en section II.C). La premi•re technique est principalement utile dans le cadre de maintenances conditionnelles, et la seconde l9est principalement dans le cadre des maintenances prŽvisionnelles. NŽanmoins quelle que soit la technique, il est nécessaire d9avoir une connaissance du fonctionnement et des dysfonctionnements du syst•me.

Dans le cadre des travaux prŽsentŽs dans cette publication, nous nous intéressons à l9application de techniques et d9algorithmes de surveillance et diagnostic pour dŽtecter des occurrences de dŽfaillances sur un exemple applicatif. Ces travaux sont rŽalisŽs au sein du projet de recherche MPO, pour Maintenance Prévisionnelle et Optimisation, de l9IRT SystemX 1 . Ce projet, en partenariat avec plusieurs acteurs industriels et académiques, porte sur l9optimisation des stratŽgies de maintenance des syst•mes de production. L9exemple applicatif considéré est un syst•me virtuel construit durant ce projet : le syst•me 3-RŽservoirs prŽsentŽ dans [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF]. Nous avons appliquŽ un algorithme d9apprentissage automatique sur des donnŽes gŽnŽrŽes par simulation du syst•me 3-RŽservoirs.

La suite de cette publication est organisŽe de la mani•re suivante. La section II fera un rappel de l9état de l9art sur la surveillance et le diagnostic. Cette section II nous permettra de justifier d9une part la définition de dŽfaillances du syst•me 3-Réservoirs, ainsi que le choix d9un algorithme de diagnostic basŽ sur les donnŽes. La section III fera une prŽsentation succincte du syst•me 3-RŽservoirs, issu des travaux prŽsentŽs dans [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF]. Les sections IV et V montreront l9implémentation de l9algorithme de surveillance du système 3-RŽservoirs, ainsi que les premi•res expŽrimentations rŽalisŽes. La section VI discutera des perspectives envisageables sur ces travaux. Enfin la derni•re section VII conclura cette publication.

Learning9) apprŽhendent Žgalement la problŽmatique de la surveillance et du diagnostic [START_REF] Hochreiter | <Long Short-Term Memory=[END_REF]. De plus rŽcents travaux, [START_REF] Angelopoulos | Tackling Faults in the Industry 4.0 Era4A Survey of Machine-Learning Solutions and Key Aspects=[END_REF] et [START_REF] Reis | <Industrial Process Monitoring in the Big Data/Industry 4.0 Era: from Detection, to Diagnosis, to Prognosis=[END_REF] par exemple, motivent d9ailleurs leur dŽmarche par l9apparition de nouveaux challenges pratiques liŽs ˆ l9arrivée de l9industrie dite 84.09, comme notamment la capacitŽ ˆ gŽrer des quantitŽs massives de donnŽes multi-sources en temps rapide. Ces Žtudes prŽsentent les approches de ML comme plus adaptŽes lorsque les profils de dŽfaillances sont complexes. Les approches utilisent des rŽseaux de neurones, des outils de traitement du signal (transformŽes de Fourier et de Laplace), etc.

C. Les notions de dŽfaillances, dysfonctionnements, et pannes

Quelles que soient les approches de surveillance et diagnostic basŽes mod•les ou basŽes donnŽes, nous considŽrons des dŽfaillances pouvant mener ˆ des dysfonctionnements ou des pannes. Nous prŽsentons donc ces notions, que nous reprenons de [START_REF] Isermann | <Fault-Diagnosis Systems=[END_REF] :

"

B. Dysfonctionnements du syst•me 3-RŽservoirs

Les trois Žv•nements redoutŽs considŽrŽs sur ce syst•me 3-RŽservoirs sont les suivants :

" Le rŽservoir L3 a dŽbordŽ ;

" Le rŽservoir L3 est vide ;

" La tempŽrature dans L3 a dŽpassŽ un niveau critique.

Les dŽfaillances menant aux dysfonctionnements du syst•me 3-RŽservoirs sont multiples. Des dŽfaillances intempestives des vannes : une vanne peut soit se coincer dans l9état dans lequel elle se trouve au moment de la défaillance, soit changer brusquement d9état, c9est-ˆ-dire s9ouvrir si elle est fermŽe ou se fermer si elle est ouverte. Les dŽfaillances intempestives des pompes ont les m•mes comportements que celles des vannes. Enfin pour chaque rŽservoir, une fuite qui appara"t ˆ la suite d9une fissure de la paroi.

C. ModŽlisation et gŽnŽration de donnŽes du syst•me 3-RŽservoirs

Dans [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF], nous prŽsentions la modŽlisation et la gŽnŽration de donnŽes simulŽes, c9est-ˆ-dire des sŽries temporelles, pour ce syst•me 3-RŽservoirs. Nous avons en effet gŽnŽrŽ des sŽries temporelles en fonctionnement normal, et des sŽries temporelles avec les dŽfaillances.

Le système a été modélisé par un PDMP pour 8Piecewise Deterministic Markov Process9 (voir [START_REF] Davis | <Piecewise Deterministic Markov Processes: A general class of non-diffusion stochastic models[END_REF] et [START_REF] Devooght | <Dynamic Reliability=[END_REF]), et les sŽries temporelles ont ŽtŽ gŽnŽrŽes en simulant, par Monte-Carlo, ˆ l9aide de l9outil PyCATSHOO (PythoniC Object Oriented Hybrid Stochastic AuTomata) dŽveloppŽ par EDF R&D [START_REF] Chraibi | <PyCATSHOO: Toward a new platform dedicated to dynamic reliability assessments of hybrid systems=[END_REF].

IV. IMPLEMENTATION DE L9ALGORITHME DE SURVEILLANCE

Nous avons utilisŽ les sŽries temporelles en fonctionnement normal et avec les dŽfaillances des composants afin de produire un outil de surveillance de ce syst•me 3-RŽservoirs. La partie surveillance est donc celle qui permet d9établir si le système est en bon fonctionnement ou en fonctionnement dŽgradŽ ˆ la suite de l9occurrence d9une dŽfaillance d9un des composants. Nous avons utilisŽ une approche basŽe sur les donnŽes avec un algorithme d9apprentissage machine.

La construction de l9outil de surveillance s9est réalisée en trois Žtapes. La premi•re Žtape a consistŽ ˆ prŽtraiter les donnŽes. La deuxi•me Žtape a consistŽ ˆ dŽfinir et entrainer un modèle d9apprentissage. Enfin la troisième étape a consisté à construire l9outil de surveillance par rapport au mod•le d9apprentissage entrainé. Cette implŽmentation est inspirŽe de [START_REF] Park | <RNN based Time-series Anomaly Detector Model Implemented in Pytorch=[END_REF].

A. PrŽtraitement des donnŽes

Les sŽries temporelles, issues de la base de donnŽes gŽnŽrŽes dans [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF], ont ŽtŽ prŽtraitŽes afin de concatŽner les valeurs des capteurs, les valeurs des manSuvres sur les actionneurs (c9est-ˆ-dire les ouverture et fermetures des vannes, et les dŽmarrages et arr•ts de pompes) et les valeurs des dŽfaillances des vannes et des pompes. Certaines modifications ont Žgalement ŽtŽ rŽalisŽes sur ces sŽries temporelles. Dans la suite, une dŽfaillance correspond ˆ une défaillance d9un des composants (vannes ou pompes) et pas à la dŽfaillance du syst•me.

D9abord une Žtiquette (label) a ŽtŽ ajoutŽe pour rŽaliser la surveillance. Cette Žtiquette est ˆ la valeur 0 quand le syst•me n9a pas eu de défaillance à l9instant de temps courant considŽrŽ. Cette Žtiquette est ˆ la valeur 1 ˆ partir de l9instant de temps d9occurrence d9une défaillance (quelconque). visibles ont ŽtŽ ŽtiquetŽes avec la valeur 1. En effet, les dŽfaillances dites invisibles ne sont pas visibles via les mesures des capteurs ; il est ainsi impossible de rŽaliser la tâche de détection car il n9y a pas d9information sur l9occurrence de cet événement dans les données.

B. Entrainement du mod•le de prŽvision

Le mod•le de prŽvision est un rŽseau de neurone rŽcurrent de type LSTM, pour 8Long Short-Term Memory9 [START_REF] Hochreiter | <Long Short-Term Memory=[END_REF], modŽlisant la dŽpendance temporelle des capteurs. Ce mod•le a ŽtŽ entrainŽ de fa•on semi-supervisŽe sur les sŽries temporelles saines, c9est-ˆ-dire celles pour lesquelles aucun dysfonctionnement n9a été généré. L9entraînement est réalisé sur une fen•tre glissante de taille L. Au temps t, le mod•le estime les valeurs de capteurs des temps + 1 ˆ + . Ainsi, pour un m•me pas de temps , on peut obtenir prŽvisions d9horizons temporels variables (de 1 ˆ ), selon si on se place ˆ 2 1 ou jusqu9à 2 . Ces prŽvisions sont stockŽes dans un vecteur de taille , et on peut calculer le vecteur d9erreur correspondant : = 2 . 1

en notant 1 le vecteur de taille L ne contenant que des 1. De lˆ, on peut calculer la moyenne et la variance empiriques de ces vecteurs d9erreur correspondant à un comportement sain du syst•me :

= 1 3 =1 , £ = 1 3( 2 ) ( 2 )
=1 qui seront utiles pour l9outil de surveillance. L9indice it•re sur l9ensemble des pas de temps de l9ensemble des trajectoires d9entraînement du modèle, pour un total de points.

Pour l9entraînement du modèle de prévision, les données d9entrée correspondent aux séries temporelles des 4 capteurs (3 de niveaux d9eau et 1 de température), de la première jusqu9à l9avant-derni•re mesure (incluse). Les labels ˆ prŽdire correspondent à ces mêmes séries temporelles décalées d9un pas de temps dans le futur : de la deuxi•me mesure jusqu9à la derni•re (incluse) ; le but Žtant de prŽdire la prochaine mesure de capteurs ˆ partir des prŽcŽdentes. Ces valeurs sont enfin standardisées (centrées et réduites). L9outil de surveillance utilisant les prŽvisions du mod•le de prŽvision LSTM sur un horizon temporel variable (majorŽ par le param•tre L), il est nécessaire d9entraîner ce LSTM à réaliser des prévisions rŽcursives prŽcises (multi-step). Cet objectif pouvant impliquer un comportement instable lors de l9entraînement (les erreurs de prŽvision s9accumulent au fil des étapes), on ajoute ˆ la fonction de perte multi-step une fonction de perte one-step, pénalisant l9erreur du modèle sur un seul pas de temps. Ceci est rŽalisŽ en suivant une procŽdure de teacher forcing [START_REF] Williams | learning algorithm for continually running fully recurrent neural networks=[END_REF], redonnant la vŽritŽ terrain au mod•le ˆ chaque pas de temps pour guider son apprentissage. Enfin, pour renforcer cette notion de guidage, on ajoute une derni•re fonction de perte, visant à minimiser l9écart entre les états cachŽs du LSTM, entre la prŽvision multi-step ou la prŽvision one-step.

La fonction de perte utilisŽe est la Mean Squared Error (MSE), et l9outil d9optimisation utilisé pour la descente de gradient est la méthode d9Adam [START_REF] Diederik | <Adam: A method for stochastic optimization=[END_REF].

C. Construction de l9outil de surveillance

L9algorithme de surveillance consiste en la comparaison d9un score d9anomalie à un seuil, permettant de discriminer entre les comportements normaux et les comportements avec les dŽfaillances.

Pour chaque trajectoire, le score d9anomalie , correspondant au pas de temps , est calculŽ en fournissant le vecteur d9erreur

ˆ un mod•le gaussien multivariŽ, paramŽtrŽ par et £ : = ( 2 )£ 21 [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF] Ce score d9anomalie est ensuite comparŽ ˆ un seuil (hyperparam•tre optimisŽ sur un espace de validation) pour obtenir la prŽdiction du mod•le sur la prŽsence de dŽfaillance au temps :

= 1 > avec 1 la fonction indicatrice.

V. EXPERIMENTATIONS

Les expŽrimentations sont ŽvaluŽes via le calcul de plusieurs mŽtriques dŽpendant de la valeur du seuil , comme la prŽcision, le rappel, et le score F1. Afin de garantir le meilleur Žquilibre entre faux-positifs et faux-nŽgatifs, nous avons choisi cette derni•re comme mŽtrique de dŽcision pour la valeur du seuil . Sur la Figure 3, on observe l9évolution de ces mŽtriques sur un ensemble de validation, selon la valeur du seuil choisi. Sur notre jeu de test, l9outil de diagnostic affiche un score F1 de 0.9555

A. Introduction d9une notion de temporalité

Une premi•re perspective de poursuite serait d9intégrer la notion de temporalité dans la détection d9une dŽfaillance. En effet une exigence communŽment dŽfinie pour un tel outil de surveillance et de diagnostic concerne la temporalitŽ : c9est-ˆdire le dŽlai entre l9instant où la dŽfaillance apparait, et l9instant où elle est dŽtectŽe, puis isolŽe et identifiŽe.

Il y a des cas o• ce dŽlai doit •tre court afin de mettre le syst•me dans un mode sžr. Ce dŽlai doit, bien entendu, •tre 23ème Congrès Lambda Mu de l9IMdR 10 au 13 octobre 2022, EDF Lab Paris Saclay mis en relation avec la sŽvŽritŽ de la dŽfaillance et la dynamique de ses consŽquences. Cette exigence de dŽlai de dŽtection peut de plus impacter le maintien des performances du syst•me. En effet, un tel outil con•u avec une exigence de dŽlai de dŽtection rapide sera tr•s certainement sensible aux bruits ou perturbations furtives (courtes et temporaires), ce qui impliquera une augmentation potentielle des fausses alarmes en fonctionnement normal et impactera ainsi les performances du syst•me.

Dans l9état actuel de l9implémentation de l9algorithme de surveillance, il est nécessaire d9y apporter des modifications complŽmentaires.

B. ConsidŽration de donnŽes complŽmentaires Ç en l9état È

Une deuxi•me perspective de poursuite serait de tester l9algorithme sur d9autres données du système 3-RŽservoirs, mais sans changer ce syst•me 3-RŽservoirs, plus prŽcisŽment sans changer le mod•le de simulation. En effet, les travaux rŽalisŽs ont considŽrŽ un ensemble de donnŽes gŽnŽrŽes initialement pour une problŽmatique de pronostic (voir [START_REF] Batteux | GŽnŽration de donnŽes pour le diagnostic et le pronostic : un exemple applicatif[END_REF]) ; ce qui a potentiellement un impact sur la pertinence des donnŽes dans un cadre de dŽtection et diagnostic de dŽfaillances, et qui demanderait ˆ •tre ŽvaluŽ.

Pour le moment, et comme expliquŽ dans la partie IV, le jeu de donnŽes a ŽtŽ divisŽ en deux parties : une partie servant à l9entrainement du modèle et une autre partie servant de tests, ce qui est d9ailleurs une approche classique. L9ajout d9autres données simulées, suivant bien sûr d9autres consignes de fonctionnement du syst•me 3-RŽservoirs, devraient ajouter de la prŽcision dans le mod•le de surveillance. Cette deuxi•me poursuite nŽcessiterait de rŽaliser de nouvelles simulations du mod•le du syst•me 3-RŽservoirs.

C. Prise en compte du diagnostic

Une troisi•me perspective de poursuite serait d9implémenter la partie diagnostic, plus précisément les étapes d9isolation et d9identification d9une dŽfaillance. En effet, en l9état seule la partie surveillance, c9est-ˆ-dire la dŽtection des occurrences de dŽfaillances, est implŽmentŽe. De plus comme le mod•le 3-Reservoirs et les simulations générées n9ont pas été initialement construits pour une approche de diagnostic, le passage au diagnostic nŽcessite des travaux complŽmentaires ˆ plusieurs niveaux : au niveau du modèle, au niveau des simulations, et au niveau de l9outil de surveillance/diagnostic.

D. Modifications du mod•le du syst•me 3-RŽservoirs

Au niveau des modifications du mod•le du syst•me 3-RŽservoirs, nous pouvons envisager diffŽrentes perspectives. En premier lieu l9ajout de dŽfaillances ou de pannes. Par exemple un encrassement dans les tuyaux ou les pompes ou encore les vannes m•nerait ˆ de mauvais débits qu9il serait possible de modéliser sous la forme d9ajouts d9aléas dans ces calculs de dŽbits dans le mod•le. Par exemple encore des fuites des rŽservoirs qui seraient causŽes par des fissures sur les parois de ces rŽservoirs modŽlisŽes (les fissures) au moyen d9un processus Markovien pour la taille de la fissure et sa hauteur sur le rŽservoir.

Il serait Žgalement possible de rendre des dŽfaillances qui ne sont pas diagnosticables actuellement en dŽfaillances qui deviendraient diagnosticables par l9ajout de tests dans le mod•le. Cela Žquivaudrait ˆ rajouter une instance virtuelle d9un outil de surveillance dans le mod•le afin de fournir les informations de tests.

Ces perspectives nŽcessitent donc de modifier le mod•le de diffŽrentes mani•res :

" Soit en ajoutant de nouveaux observateurs dans le modèle, c9est-ˆ-dire des variables d9intérêt qui n9ont pas d9impact sur les phénomènes physiques reprŽsentŽs ; " Soit en modifiant les phŽnom•nes physiques reprŽsentŽs au moyen de nouvelles variables et de nouvelles relations liant ces variables, avec potentiellement des impacts sur les variables et relations existantes ;

Ces modifications signifient par la suite de rŽaliser de nouvelles simulations, comme nous allons l9expliquer dans la sous-partie suivante.

E. Modifications au niveau des simulations du syst•me 3-RŽservoirs

Au niveau des simulations, nous pouvons envisager soit la rŽalisation de nouvelles simulations, soit la modification des simulations existantes.

La rŽalisation de nouvelles simulations sera nŽcessaire dans le cas o• le mod•le a ŽtŽ modifiŽ, comme expliquŽ dans les perspectives indiquŽes dans les sous-parties prŽcŽdentes. Dans le cas où le modèle n9intègre que de nouveaux observateurs, ce pourront •tre les simulations existantes qui seront rejouŽes, afin de capturer, dans les donnŽes, ces nouvelles observations. Dans le cas o• le mod•le int•gre de nouveaux phénomènes, il faudra d9une part définir les plans de simulation, c9est-ˆ-dire spŽcifier quelles sont les consignes et trajectoires ˆ simuler, car les simulations existantes seront obsolètes et ne pourront être rejouées, et il faudra d9autre part rŽaliser ces nouvelles simulations suivant ces nouveaux plans de simulation.

Pour la modification des simulations existantes, il s9agit par exemple de supprimer certaines valeurs ou ensembles de valeurs. Ces suppressions peuvent •tre soit suivant les observateurs, c9est-ˆ-dire de supprimer les données d9un ou plusieurs observateurs, soit suivant une plage temporelle de fonctionnement. Il peut Žgalement s9agir de modifier certaines valeurs, par exemple en ajoutant une valeur alŽatoire pour représenter du bruit, ou encore d9ajouter des nouvelles donnŽes construites via les donnŽes existantes.

Enfin ˆ la suite de la production de nouvelles simulations, ou la modification des simulations existantes, il sera nécessaire d9en faire un prétraitement, c9est-ˆ-dire de les mettre au bon format, afin que l9algorithme de surveillance et de diagnostic puisse les considŽrer.

F. Implémentation de l9algorithme de diagnostic

Au niveau de l9outil de surveillance/diagnostic, nous pouvons envisager l9implémentation d9algorithmes dédiés pour le diagnostic. Les algorithmes abordant une vision Machine Learning se distinguent selon s9ils traitent de la dŽtection et de l9isolation/identification de mani•re simultanŽe, ou de mani•re sŽquentielle.

Pour le cas sŽquentiel, les donnŽes d9entrée du module de diagnostic correspondent aux plages temporelles des donnŽes ayant conduit ˆ une prŽvision de dŽfaillance de la part du module de dŽtection. Dans ce cas-lˆ, il serait possible de réutiliser l9approche de détection déjà implŽmentŽe comme premi•re brique du mod•le de diagnostic global. Dans le cas 23ème Congrès Lambda Mu de l9IMdR 10 au 13 octobre 2022, EDF Lab Paris Saclay d9une détection et isolation/identification simultanŽes, la majoritŽ des algorithmes se placent dans un cadre supervisŽ, et con•oivent un mod•le de classification en N+1 classes, composée d9une classe correspondant à un fonctionnement normal du syst•me, et de N classes de dŽfaillances diffŽrentes.

Les mod•les de classification sont en gŽnŽral prŽcŽdŽs d9un module d9extraction de 8features9 permettant de représenter les données d9entrée sous une forme exploitant leurs caractŽristiques pertinentes pour faciliter la t‰che de classification. Cela peut •tre rŽalisŽ de fa•on automatique ou sur la base de comprŽhension du phŽnom•ne physique, et est communément appelé 8feature engineering9.

Dans le cadre du jeu de donnŽes 3-RŽservoirs, il pourrait être possible d9utiliser des outils de traitement du signal, tels que prŽsentŽs en sous-partie II.B de l9état de l9art (transformŽes de Fourier, transformŽes de Laplace, ou en ondelettes dans le domaine temps-frŽquence). Le module de classification pourra ensuite exploiter ces 8features9, notamment via l9utilisation de SVM, de rŽseaux de neurones peu profonds, ou de for•ts alŽatoires. Des mŽthodes d9apprentissage profond, intŽgrant la phase d9apprentissage de reprŽsentation de mani•re automatique, peuvent Žgalement s9appliquer à ce jeu de données : des rŽseaux de neurones convolutifs, des rŽseaux de neurones rŽcurrents profonds, des transformers, ou des auto-encoders.

VII. CONCLUSION

Dans cette publication, nous avons montrŽ l9implémentation d9un algorithme de surveillance sur un exemple virtuelle du projet MPO de l9IRT SystemX. Cet exemple, nommŽ syst•me 3-RŽservoirs, est constitué d9un ensemble de composants (pompes, vannes, rŽservoirs, capteurs) sujets ˆ des dŽfaillances. De prŽcŽdent travaux ont montrŽ la modŽlisation et la gŽnŽration de donnŽes, plus prŽcisŽment des sŽries temporelles, sur cet exemple.

Nous nous sommes donc servis de ces donnŽes gŽnŽrŽes pour dŽfinir et implŽmenter un outil de surveillance de ce syst•me 3-RŽservoirs. Nous avons utilisŽ un mod•le d9apprentissage de type réseau de neurone récurrent (plus prŽcisŽment de type LSTM), qui a ŽtŽ entrainŽ sur les sŽries temporelles sans les dŽfaillances. L9algorithme implémenté de surveillance a consistŽ en un vecteur d9erreur, issus du mod•le appris, fourni ˆ un mod•le Gaussien afin de produire un score d9anomalie. Ce score est ensuite comparŽ ˆ un seuil permettant de discriminer entre les comportements normaux et les comportements avec les dŽfaillances.

Nous avons ensuite prŽsentŽ diffŽrentes perspectives permettant de complŽmenter ces travaux dans diffŽrentes directions, soit en augmentant l9ensemble des données gŽnŽrŽes ˆ partir du mod•le du syst•me 3-RŽservoirs, soit en modifiant le mod•le du syst•me 3-RŽservoirs, enfin soit en modifiant l9algorithme implémenté. Les objectifs principaux de ces compléments étant d9une part de traiter la partie diagnostic, c9est-ˆ-dire d9identifier la défaillance apparue, et d9autre part d9ajouter des défaillances à diagnostiquer, ou ˆ minima d9en rendre certaines actuelles diagnosticables.

REMERCIEMENTS

Ces travaux ont ŽtŽ rŽalisŽs dans le cadre du projet MPO de l9Institut de Recherche Technologique SystemX. Ils ont ŽtŽ soutenus par le gouvernement Fran•ais au travers du programme 88France 203099.

RƒFƒRENCES

ÿÿÿ Rÿÿÿÿÿ ÿÿ ÿÿ ÿÿÿÿÿ ÿÿ Fÿÿÿÿÿÿÿ La Quatrième Révolution Industrielle, également appelée Industrie ÿ.ÿ, marque une transformation profonde du secteur industriel en fusionnant les domaines physiques, numériques et biologiques. Se bâtissant sur une transformation numérique, elle est caractérisée par des avancées telles que l'Internet des Objets, l'Intelligence Artiÿcielle, et les systèmes cyber-physiques. Au coeur de cette révolution se trouve la smart factory (usine intelligente), où les machines interagissent en temps réel avec l'homme et d'autres équipements. Un des enjeux majeurs de l'Industrie ÿ.ÿ est la maintenance prévisionnelle, visant à prévenir les pannes des systèmes industriels. L'objectif principal de cette thèse de doctorat est d'étudier la maintenance prévisionnelle, en particulier le diagnostic des défauts, à travers le prisme de l'apprentissage profond et des sources de données multimodales et hétérogènes de l'Industrie ÿ.ÿ. Ces données, qu'elles proviennent de capteurs de vibration, de température, de caméras ou de rapports de maintenances, oÿrent une perspective multimodale (séries temporelles, images, texte...) riche et détaillée de l'état des systèmes de production. L'analyse intégrée de ces modalités distinctes permet non seulement une détection plus précise des défauts, mais aussi une vériÿcation croisée pour une plus grande ÿabilité, révélant parfois des anomalies qui pourraient rester inaperçues si chaque modalité était considérée isolément. L'utilisation de données multimodales oÿre de nombreux avantages, mais l'importance des données textuelles est également particulièrement remarquable. Ces données, tirées principalement de rapports de maintenance ou de journaux opérationnels (logs), oÿrent une vue approfondie des opérations des systèmes et des incidents précédents. Elles sont uniques car elles contiennent des détails nuancés provenant de l'expertise humaine, essentiels pour diagnostiquer les défauts. Ces données textuelles relient diverses modalités, ajoutant du contexte et une interprétation aux données numériques et visuelles. Toutefois, leur rareté représente un déÿ pour leur exploitation optimale dans les analyses.

Bien ancrée dans le contexte très appliqué de l'Industrie ÿ.ÿ et du projet "Maintenance Prévisionnelle et Optimisation" de l'IRT SystemX, l'ambition de cette thèse va au-delà du développement de modèles pour des applications spéciÿques et vise à répondre méthodologiquement aux déÿs considérés. Le premier objectif concerne la nature dynamique et en temps réel des systèmes industriels, générant des ÿux de données continus avec des fréquences d'acquisition hétérogènes. Le second objectif émerge du besoin de gérer la complexité d'intégration de données à structures hétérogènes, en soulignant l'importance des interactions entre les caractéristiques de diÿérentes sources de données. Le troisième objectif se concentre sur l'exploitation de la richesse des données textuelles, en particulier dans les rapports de maintenance. Ces documents encapsulent une information contextuelle riche, mais leur rareté et le vocabulaire spéciÿque qu'ils contiennent rendent leur traitement diÿcile.

Conformément aux déÿs précédemment déÿnis, cette thèse présente deux contributions distinctes, chacune dédiée à un domaine de recherche spéciÿque : l'Apprentissage Multimodal et l'Apprentissage avec peu de données (Few-Shot Learning) en TAL (Traitement Automatique du Langage). Ceci déÿnit également la structure de la thèse, divisée en deux parties principales.

ÿÿÿ

. Résumé de la thèse en Français La première partie commence par un besoin pragmatique clairement déÿni dans le domaine industriel pour diagnostiquer des pannes dans des systèmes multimodaux complexes. Cette motivation concrète nous a orientés vers le développement d'un cadre théorique basé sur l'apprentissage multimodal, intrinsèquement motivé par la nature multimodale de notre environnement réel. Dans le Chapitre ÿ, nous fournissons au lecteur les bases nécessaires pour motiver et comprendre la première partie de cette thèse. Nous commençons par présenter les fondamentaux de la théorie du diagnostic de pannes et nous passons en revue les stratégies existantes pour aborder ce problème, en nous concentrant sur les approches basées sur l'apprentissage et en explorant les rares tentatives ayant pris en compte des données issues de modalités hétérogènes. Nous introduisons ensuite le paradigme de l'apprentissage multimodal, avec un accent particulier sur la fusion multimodale. De là, nous proposons un aperçu des méthodologies développées, en commençant par les travaux plus anciens reposant sur des stratégies de fusion simples comme la concaténation, et en se concentrant davantage sur le niveau auquel réaliser la fusion. Nous soulignons ensuite les avantages de construire des représentations de données expressives, qui sont principalement réalisables grâce aux architectures basées sur l'apprentissage profond, et la proximité entre la fusion multimodale et la représentation multimodale. Nous explorons donc les approches d'apprentissage de représentation multimodale, qui sont aujourd'hui principalement basées sur l'architecture Transformer. Dans le Chapitre ÿ, nous abordons les nouveaux déÿs posés par la complexité croissante des systèmes Industrie ÿ.ÿ et leur relation avec les tâches de détection et de diagnostic de pannes. Nous explorons ces déÿs dans un environnement réaliste qui implique des ÿux de données multi-sources provenant de diverses modalités, incluant des mesures de capteurs en séries temporelles, des images de machines et des rapports de maintenance textuels. Ces ÿux multimodaux hétérogènes diÿèrent également dans leur fréquence d'acquisition, peuvent intégrer des informations temporellement non alignées et peuvent être arbitrairement longs, en fonction du système et de la tâche considérés. S'appuyant sur le chapitre précédent, où nous avons examiné les principales approches de fusion multimodale, nous élargissons notre champ d'application à ce contexte. Nous considérons des ÿux multimodaux arbitrairement longs conjointement avec des tâches associées, telles que la prédiction dans le temps. Pour relever ce déÿ, nous proposons StreaMulT, un Transformer multimodal. StreaMulT utilise un mécanisme d'attention cross-modale et une banque de mémoire pour traiter des séquences d'entrée arbitrairement longues pendant l'entraînement et fonctionne au ÿl de l'eau à l'inférence. Le Chapitre ÿ présente une discussion sur les diverses interactions multimodales. Nous commençons par décomposer le contenu pertinent des données en tant qu'information redondante et complémentaire. Par la suite, nous nous plongeons dans l'exploration des recherches axées sur la maximisation de l'information redondante, principalement dans le cadre multi-vues, et les outils utilisés dans ce domaine. La dernière section tente d'élargir ces approches pour incorporer la caractérisation de l'information complémentaire, et formule des critiques à la fois sur les méthodologies existantes et sur le manque de repères d'évaluation. Cette analyse oÿre une compréhension exhaustive des déÿs actuels et des pistes potentielles dans le domaine de l'apprentissage multimodal.

La deuxième partie de la thèse se concentre sur l'exploitation de données textuelles rares et spéciÿques dans un contexte réaliste. Elle débute avec le Chapitre ÿ, qui oÿre un aperçu des méthodologies de Traitement Automatique du Langage (TAL), jusqu'au développement des récents grands ÿÿÿ Appendix modèles dits "fondateurs", puis se tourne vers l'apprentissage à partir de peu d'exemples (Few-shot learning), une stratégie pour apprendre à partir de données étiquetées limitées, avant de conclure par une discussion sur l'application du FSL au TAL. La première section de ce chapitre décrit la progression de la recherche en TAL pour comprendre le langage humain. Cela inclut les premières méthodes basées sur des règles établies ou sur de l'ingénierie des caractéristiques (feature engineering), l'utilisation des plongements de mots pour créer des représentations distribuées et signiÿcatives, et le développement de diverses architectures pour des modèles de langage eÿcaces. Nous étudions ensuite l'approche dominante pour traiter les tâches du TAL, qui implique de grands modèles de langage basés sur des transformateurs pré-entraînés et leur évolution ultérieure vers la création de modèles centraux polyvalents capables de gérer une gamme variée de tâches, malgré leurs natures distinctes. Enÿn, nous explorons le domaine de l'apprentissage à partir de peu d'exemples, en examinant ses principales techniques et son intersection avec les paradigmes actuels du TAL, tout en mettant en lumière les derniers progrès et déÿs de ce domaine de recherche. Dans le Chapitre ÿ, nous explorons le potentiel des méthodes transductives pour la classiÿcation textuelle dans le contexte de l'apprentissage à partir de peu d'exemples, dans le but de pallier les limites des méthodes actuelles de FSL en TAL, notamment les eÿorts d'ingénierie nécessaires pour des tâches de classiÿcation réaliste avec un grand nombre de classes. Nous discutons d'abord des limites des méthodes actuelles de FSL, telles que les stratégies basées sur des prompts ou de l'apprentissage en contexte. Puis, nous explorons l'application des approches transductives -qui ont montré des résultats prometteurs en vision par ordinateur -à la classiÿcation en TAL. Enÿn, nous évaluons la performance des régularisateurs transductifs traditionnels par rapport aux techniques inductives sur des tâches de classiÿcation textuelle avec peu d'exemples et étudions l'impact de diÿérents facteurs, tels que le nombre de paramètres du modèle principal et les stratégies de ÿnetuning, sur la performance des méthodes transductives. Les résultats indiquent que les méthodes transductives ont du mal à surpasser le ÿne-tuning inductif basé sur la cross-entropie lorsqu'il y a une certaine ÿexibilité dans les paramètres de l'extracteur de caractéristiques pré-entraîné. Cependant, en ÿxant tous les paramètres de l'extracteur de caractéristiques, l'approche transductive rivalise ÿnalement avec l'approche inductive. Enÿn, dans le Chapitre ÿ nous abordons la prévalence croissante des API propriétaires et fermées pour les grands modèles de langage tels que GPT-ÿ et ChatGPT, qui ont des implications signiÿcatives pour les applications pratiques du TAL, y compris la classiÿcation avec peu d'exemples. La classiÿcation avec peu d'exemples implique de former un modèle pour exécuter une nouvelle tâche de classiÿcation avec un minimum de données étiquetées. Notre investigation présente trois contributions clés. Premièrement, nous introduisons une situation dans laquelle un modèle préentraîné est accessible via une API protégée, en tenant compte des contraintes de coût de calcul et de conÿdentialité des données. Deuxièmement, nous approfondissons l'application de l'inférence transductive, un paradigme d'apprentissage qui a été relativement peu exploré au sein de la communauté du TAL. Contrairement à l'apprentissage inductif traditionnel, l'inférence transductive tire parti des statistiques des données non étiquetées. Dans ce contexte, nous introduisons également un nouveau régularisateur transductif sans paramètre basé sur la perte de Fisher-Rao, démontrant son applicabilité et son eÿcacité dans le cadre de l'incorporation via une API protégée. Cette approche exploite pleinement les données non étiquetées, évite de partager toute information d'étiquette avec les fournisseurs d'API tiers et pourrait servir de référence pour les recherches futures. Enÿn, nous proposons un cadre expérimental amélioré et compilons un benchmark de ÿÿÿ

  Concrete motivations and context . . . . . . . . . . . . . . . . . . . . . . . . ÿ ÿ.ÿ Objectives and challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿ ÿ.ÿ Outline and contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿ I Pÿÿÿÿÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿ Lÿÿÿÿÿÿÿ Sÿÿÿÿÿÿÿÿÿ ÿÿÿ Iÿÿÿÿÿÿÿÿÿ Fÿÿÿÿ Dÿÿÿÿÿÿÿÿ ÿÿ ÿ Bÿÿÿÿÿÿÿÿÿ ÿÿÿ Rÿÿÿÿÿÿ Wÿÿÿ ÿÿ ÿ.ÿ Addressing fault diagnosis with machine learning and deep learning . . . . . . ÿÿ ÿ.ÿ.ÿ Background: Principles of the machine learning and deep learning paradigms ÿÿ ÿ.ÿ.ÿ Machine learning and deep learning models for fault diagnosis . . . . . ÿÿ ÿ.ÿ.ÿ Fault diagnosis approaches using multimodal data . . . . . . . . . . . ÿÿ ÿ.ÿ Multimodal learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ From multimodal perception to multimodal learning . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Multimodal fusion: an overview . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Deep neuronal architectures for multimodal representation learning . . ÿÿ Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ SÿÿÿÿMÿÿTÿ A Sÿÿÿÿÿÿÿÿ Mÿÿÿÿÿÿÿÿÿ Tÿÿÿÿÿÿÿÿÿÿ Fÿÿ Hÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ Aÿÿÿÿÿÿÿÿÿÿ Lÿÿÿ Sÿÿÿÿÿÿÿÿÿ Dÿÿÿ ÿÿ ÿ.ÿ Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Multimodal learning with heterogeneous and arbitrarily long sequential streams ÿÿ ÿ.ÿ.ÿ Problem formalization . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Positioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Transformer architectures and unaligned modalities . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Streaming input data . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Proposed model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Dataset and evaluation task . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Experimental setting and results . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Ablation study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ v Contents ÿ.ÿ Time and space complexities study . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ Tÿÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ Iÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿ.ÿ Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Theoretical background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ.ÿ Multimodal learning provably performs better than unimodal . . . . . ÿÿ ÿ.ÿ.ÿ A brief recap on information theory . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Maximizing redundant information . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Characterizing complementary information . . . . . . . . . . . . . . . . . . ÿÿ Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ II Tÿÿÿÿÿÿ Rÿÿÿÿÿÿÿÿ FÿÿÿSÿÿÿ Tÿÿÿÿÿÿ Cÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿ Bÿÿÿÿÿÿÿÿÿ ÿÿÿ Rÿÿÿÿÿÿ ÿÿÿÿ ÿÿ NLPÿ ÿÿÿÿ Sÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿ Fÿÿÿÿ ÿÿÿÿÿÿ Mÿÿÿÿÿ ÿÿ ÿ.ÿ Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Early NLP methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Word embeddings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Language models .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Encoder-decoder architecture . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Foundation models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Few-shot learning in NLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ A ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿ NLP ÿÿÿ ÿ.ÿ Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Current methods limitations . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Textual classiÿcation in few-shot setting . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Transductive approaches for FSL in NLP . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Experimental study of transductive few-shot inference for NLP classiÿcation . . ÿÿÿ ÿ.ÿ.ÿ Limitations of existing benchmarks . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Research questions and related results . . . . . . . . . . . . . . . . . ÿÿÿ Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ Tÿÿÿÿÿÿ FÿÿÿSÿÿÿ Cÿÿÿÿÿÿÿÿÿÿÿÿÿ Fÿÿ APIÿÿÿÿÿÿ Mÿÿÿÿÿ ÿÿÿ ÿ.ÿ Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ API based few-shot learning . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Limitations of current methods . . . . . . . . . . . . . . . . . . . . ÿÿÿ vi Contents ÿ.ÿ.ÿ Proposed transductive approaches and baselines . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ A Fisher-Rao based regularizer . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ An enhanced experimental setting . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Model choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Evaluation framework . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Overall results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Study under diÿerent data regimes . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Ablation study on backbones . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ A dive into GPT-ÿ.ÿ results . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Multilingual experiment . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Importance of model backbones on monolingual experiment . . . . . ÿÿÿ ÿ.ÿ.ÿ Importance of model backbones on multilingual experiment . . . . . ÿÿÿ ÿ.ÿ.ÿ Practical considerations . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Links with the obervations of Chapter ÿ . . . . . . . . . . . . . . . . ÿÿÿ Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ Cÿÿÿÿÿÿÿÿÿ ÿÿÿ Pÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿ.ÿ Summary of the contributions . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Criticism and short-term perspectives . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ.ÿ Long-term perspectives . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ Aÿÿÿÿÿÿÿ ÿÿÿ ÿ.ÿ Proof of Theorem ÿ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Publication in the context of the MPO project . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Résumé de la thèse en Français . . . . . . . . . . . . . . . . . . . . . . . . . Pillars of Industry ÿ.ÿ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿ ÿ.ÿ Example of a corrective maintenance report. . . . . . . . . . . . . . . . . . . ÿ ÿ.ÿ Example of multimodal perception of the environment. . . . . . . . . . . . . ÿÿ ÿ.ÿ Example of multimodal data acquirement in Industry ÿ.ÿ setting. . . . . . . . ÿÿ ÿ.ÿ Illustration of the heterogeneity gap. . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Diÿerent fusion techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Multimodal representations mappings. . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Joint and coordinated representations learning. . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Early fusion techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Multimodal autoencoder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Intra-modality and inter-modality impacts of attention mechanism. . . . . . . ÿÿ ÿ.ÿÿ Visual Transformer architecture. . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Multimodal segment embeddings. . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Multimodal transformers variants. . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ VideoBERT architecture and pre-training. . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ CLIP Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Typical example of fault diagnosis task in the context of Industry ÿ.ÿ. . . . . . . ÿÿ ÿ.ÿ Cross-modal attention block. . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Cross-modal attention matrices. . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Linearly growing receptive ÿeld linearly growing. . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Forward step for the Augmented Memory Transformer. . . . . . . . . . . . . ÿÿ ÿ.ÿ Comparison of AM-TRF with Emformer. . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Streaming Multimodal Transformer architecture. . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Block processing for Multimodal learning in a streaming scheme. . . . . . . . . ÿÿ ÿ.ÿ Streaming Cross-modal Transformer module. . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Training and inference ÿexible scheme. . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Heatmap of StreaMulT attention weights. . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Information diagram of two redundant modalities. . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Information diagram of two mutually complementary modalities. . . . . . . . ÿÿ ÿ.ÿ Information diagram of a modality-domination setting. . . . . . . . . . . . . ÿÿ ÿ.ÿ Information diagram of two modalities with no redundancy. . . . . . . . . . . ÿÿ ÿ.ÿ Comparison of CBoW and Skip-Gram approaches . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Qualitative results for WordÿVec embeddings. . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Neural language model architecture. . . . . . . . . . . . . . . . . . . . . . . ÿÿ ix List of Figures ÿ.ÿ Sequence-to-sequence architecture. . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Original transformer architecture. . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Pre-training and ÿne-tuning paradigm. . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Masked language modeling and similar pre-training objectives. . . . . . . . . . ÿÿ ÿ.ÿ Timeline of released large language models. . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ In-context learning and instruction ÿne-tuning. . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Few-shot learning paradigm. . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Parameter-eÿcient tuning with adapter. . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Prompt-based few-shot learning. . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿÿ Inductive vs transductive settings. . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ N -shot K-way tasks example. . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Comparison of cross-entropy-based and transductive-based approaches for different numbers of shots. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Comparison of cross-entropy-based and TIM-based approaches for BERT and RoBERTa backbones. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Comparison of BERT and RoBERTa backbones performances. . . . . . . . . ÿÿÿ ÿ.ÿ Comparison of cross-entropy-based and TIM-based approaches for diÿerent ÿnetuning strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ API-based few-shot learning scenario. . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Performance of the diÿerent pre-trained encoders on the monolingual datasets. ÿÿÿ ÿ.ÿ Eÿects of diÿerent ways and shots on test performance. . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Impact of model size on performances. . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Diÿerent losses when training a on GPTÿ.ÿ embeddings. . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Performance of the diÿerent losses on multilingual datasets. . . . . . . . . . . ÿÿÿ ÿ.ÿ Performance of diÿerent pre-trained encoder on the monolingual datasets. . . . ÿÿÿ ÿ.ÿ Performance of diÿerent pre-trained backbones on multilingual Amazon dataset. ÿÿÿ x Lÿÿÿ ÿÿ Tÿÿÿÿÿ ÿ.ÿ Comparison of diÿerent approaches for multimodal representation learning. . . ÿÿ ÿ.ÿ StreaMulT results on CMU-MOSEI aligned. . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ StreaMulT results on CMU-MOSEI unaligned. . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Ablation study on CMU-MOSEI aligned. . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Time and space complexities. . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Optimal hyperparameters for StreaMulT. . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Summary of limitations of early NLP methods. . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Summary of limitations of early NLP methods and word embeddings techniques.ÿÿ ÿ.ÿ Summary of advantages and limitations of general NLP methods and word embeddings techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Pre-training objectives and their respective loss functions. . . . . . . . . . . . . ÿÿ ÿ.ÿ Overview of diÿerent transformer-based models. . . . . . . . . . . . . . . . . ÿÿ ÿ.ÿ Overview of the various datasets. . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Comparison of CE-based and TIM-based approaches performances for diÿerent ÿne-tuning strategies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Statistics of considered NLP datasets. . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Preliminary experiment results. . . . . . . . . . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Aggregated performances over the diÿerent datasets and considered backbones. ÿÿÿ ÿ.ÿ Global results for multilingual Amazon dataset. . . . . . . . . . . . . . . . . . ÿÿÿ ÿ.ÿ Training time for ÿ episode on a Mÿ-CPU. . . . . . . . . . . . . . . . . . . . ÿÿÿ xi ÿ Iÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ Cÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿ Pÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿ ÿÿÿ ÿÿÿ ÿÿ Iÿÿÿÿÿÿÿ ÿÿÿ

Figure

  Figure ÿ.ÿ: Pillars of Industry ÿ.ÿ. Figure from (Ryalat et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: An example of a corrective maintenance report of a climate research facility. Figure from (Teske et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Example of multimodal perception of the environment.

Figure

  Figure ÿ.ÿ: Example of multimodal data acquirement in Industry ÿ.ÿ setting.

  Figure ÿ.ÿ, vectorial representations of semantical close concepts from ÿÿ Background and Related Work diÿerent modalities are generally also heterogeneous, which lead to the diÿculty to measure the content similarity between diÿerent modalities.

Figure

  Figure ÿ.ÿ: Illustration of the heterogeneity gap. Image from (W. Guo et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Diÿerent fusion techniques: early (d), late (e) and hybrid (f). Figure adapted from (Atrey et al. ÿÿÿÿ). AU, FF and DF represent Analysis, Feature Fusion and Decision Fusion units, respectively.

Figure ÿ. ÿ :

 ÿ Figure ÿ.ÿ: Diÿerent multimodal representations mappings ĝN and ĝM for relative modalities subsets N ã M. These mappings produce respective images z N and z M , for which the latter is closer to z * , the image corresponding to the true mapping g * . Figure from (Y. Huang et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: (left) Joint representation learning, (right) Coordinated representation learning. Figure from (W. Guo et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Intra-modality (left) and inter-modality (right) impacts of attention mechanism. On the leftside picture, the attention mechanism forces the model to focus on speciÿc parts of the image input, conditioned by the textual input to perform VQA. On the right-side picture, the attention mechanism balances the weight of each input modality depending on their relevances for identifying a scene. Figures from (Zichao Yang et al. ÿÿÿÿ) (left) and (Long et al. ÿÿÿÿ) (right).

Figure

  Figure ÿ.ÿÿ: Visual Transformer architecture. Input images are tokenized in small patches that are encoded with positional embeddings. Figure from (Dosovitskiy et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿÿ: Multimodal segment embeddings. Besides the positional embeddings, each image patch and textual input is encoded with a special segment embedding indicating its input modality. Figure from (G. Li et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿÿ: Diÿerent variants of Transformer for processing multimodal data. Colors represent modalities. Figure from (P. Xu et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿÿ: VideoBERT architecture and pre-training. Figure from (C. Sun, Myers, et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿÿ: CLIP Architecture. Figure from (Radford, Kim, Hallacy, et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Typical example of fault diagnosis task in the context of Industry ÿ.ÿ: case of an aircraft engine. Each modality present fault symptoms through acquired data (red circles), that, if fused together, can enable the fault detection (and identiÿcation).

ÿÿ

  StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data ÿÿÿ Mÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿ Pÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿ

Figure

  Figure ÿ.ÿ: Cross-modal attention block between sequences X α , X β from distinct modalities (left) and cross-modal transformer module (right). Figures found in (Tsai et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Examples of cross-modal attention matrices: explicitly aligned data on top and unaligned data on bottom. Orange boxes correspond to cross-modal pairs the model attends to, with higher weights on brighter boxes. Figure from (Tsai et al. ÿÿÿÿ).

  al. ÿÿÿÿ; Tripathi et al. ÿÿÿÿ; Q. Zhang et al. ÿÿÿÿ), a strategy so called time-restricted attention, other ones segment input sequences in smaller chunks before performing self-attention on those chunks (Z. Tian et al. ÿÿÿÿ; C. Wang et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Examples of receptive ÿeld linearly growing with the number of layers: context masking for the y 7 position (left=ÿ, right=ÿ). Figure from (Q. Zhang et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Illustration of one forward step for the augmented memory transformer on the n-th segment. B refers to the segment length. Figure from (C. Wu et al. ÿÿÿÿ).

  Figure ÿ.ÿ sums up the main diÿerences between both architectures.

Figure

  Figure ÿ.ÿ: Comparison of AM-TRF (left) with Emformer (right).The two approaches mainly diÿer in the content of M n i , that contains summarized information from lower layer in Emformer (enabling to parallelize the computations on all layers); and in the cached keys and values from previous segments. All these optimized changes render the architecture more eÿcient and prone to work in a streaming scheme. Figure from(Shi et al. ÿÿÿÿ).

  Figure ÿ.ÿ: Streaming Multimodal Transformer architecture. SCT stands for Streaming Cross-modalTransformer. Diÿerent colors represent heterogeneity nature of diÿerent modalities, and shadings represent cross-modal features. Each modality-speciÿc time series is passed through a ÿDconvolutional layer, and then through a unimodal Emformer block to initialize its modalityspeciÿc memory bank. Cross-modal interactions are then captured through SCT blocks, that express a target modality with the help of source modalities' features and memory banks. Target modalities representations computed from diÿerent source modalities are then concatenated and passed through modality-speciÿc Transformer encoders, that output contextual crossmodal representations, summarizing the whole sequences. These outputs are then processed by a ÿnal FFN to produce the prediction.

ÿÿ

  Figure ÿ.ÿ: Streaming Cross-modal Transformer module.

  Figure ÿ.ÿÿ: Flexible scheme. At training time (left), subsequences of h consecutive segments are created to parallelize cross-modal attention operations. At inference (right), one can still process segments one by one to obtain a short-time response.

Figure

  Figure ÿ.ÿÿ: Heatmap of StreaMulT attention weights for the Visual/Textual cross-modal module. The sequence of length ÿÿ is chunked into segments of length ÿÿ, with left and right contexts of respectively lengths ÿÿ and ÿ.

  [right contexts; segments utterances; summary vectors]. Diÿerent blocks are delimited on Figure ÿ.ÿÿ by vertical and horizontal blue lines.

  Suÿcient representation). A representation Z of X is suÿcient for Y if and only if I(X; Y |Z) = 0.

  Figure ÿ.ÿ: Information diagram of two modalities X 1 , X 2 that are mutually redundant for a given targetY . The amount of information conveyed by X 1 and X 2 are represented by red and blue areas, respectively, while the purple area represents the amount of information share by both modalities. The amount of predictive information, conveyed by the variable Y , is represented by the green area. The only amount of predictive information that is accessible is I(Y ; X 1 ; X 2 ). This piece of information is shared by both modalities (mutual redundancy), hence its representation area on the diagram is encapsulated in the purple area, representing I(X 1 ; X 2 ). It is worth noting that we generally lack access to the entirety of the information conveyed by Y ; this unavailable quantity is H(Y |X 1 ; X 2 ).

  al. ÿÿÿÿ; J. S.Chung et al. ÿÿÿÿ; Miech et al. ÿÿÿÿ; Radford, Kim, Hallacy, et al. ÿÿÿÿ; C. Sun, Baradel, et al. ÿÿÿÿ), where modalities are for instance videos, text or sound. The contrastive framework implicitly rely on the same assumption, as several works have shown the parallel between used contrastive losses and the maximization of mutual information between views (Y. Tian, C. Sun, et al. ÿÿÿÿ; M.Wu et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Information diagram of two modalities (X 1 , X 2 ) that are mutually complementary for a given target Y . The amounts of information conveyed by X 1 , X 2 and Y are still represented by red, blue and green areas, respectively, while the purple area still represents the amount of information shared by both modalities. While some predictive information is shared by both modalities, i.e. I(Y ; X 1 ; X 2 ) > 0, there is modality-speciÿc predictive information, i.e. I(Y ; X 1 |X 2 ) > 0 and I(Y ; X 2 |X 1 ) > 0, represented by the intersections of red/green and blue/green areas that are outside of the purple area.

Figure

  Figure ÿ.ÿ: Modality-domination setting. In that setting, modality X 1 has a much bigger impact than modality X 2 , which does not encompass any modality-speciÿc predictive information, i.e. I(Y ; X 2 |X 1 ) = 0 (left). In the extreme case, all predictive information is made unavailable from the perspective of X 2 view, that is I(Y ; X 2 ) = 0 (right).

Figure

  Figure ÿ.ÿ: Information diagram of two modalities (X 1 , X 2 ) that do not share predictive information for a target Y , i.e. I(Y ; X 1 ; X 2 ) = 0. The model shall thus combine modality-speciÿc contents to provide correct prediction.

ÿ

  Bÿÿÿÿÿÿÿÿÿ ÿÿÿ Rÿÿÿÿÿÿ ÿÿÿÿ ÿÿ NLPÿ ÿÿÿÿ Sÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿ Fÿÿÿÿÿÿÿÿÿ Mÿÿÿÿÿ Cÿÿÿÿÿÿ'ÿ Sÿÿÿÿÿÿ

  Figure ÿ.ÿ: Comparison of CBoW and Skip-Gram approaches. CBoW projects context words to predict a central word (left), while Skip-Gram inversely projects a unique word to predicts its context (right). Figure from (Tomás Mikolov, Le, et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Qualitative results for WordÿVec embeddings. Subtracting capital vector to its related country vector produces similar vector among all country/capital pairs (left). Learned embeddings of number and animal words have very similar spatial distribution in English and Spanish (right).

  Figure ÿ.ÿ: Neural Language Model architecture. The input sentence (w t-n+1 , . . . , w t-1 ) is converted to feature vectors stored in a matrix C, which are then fed to a neural network g represented by the green plain lines. The output of g estimates the probability of each word in the vocabulary, conditioned the input context. Figure from (Bengio, Ducharme, et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Sequence-to-Sequence architecture ÿ . Every words of the input sentence are embedded and thensequentially fed to the encoder module, that stores the input information in a context S. Using this context and the previous generated token (starting with a special token), the decoder module sequentially generates the output.

ÿ

  Figure from https://www.guruÿÿ.com/seqÿseq-model.html ÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models

  Figure ÿ.ÿ: Original Transformer architecture. to encoder-decoder models, the embedded input is ÿrst encoded in a speciÿc module before the decoder module generates the output autoregressively. The main diÿerence is the use of Self-attention modules that make possible to model contextual dependencies between all parts of the sequences. The masking process in the decoder modules enables to parallelize the training. Figure from (Vaswani et al. ÿÿÿÿ).

ÿ

  Figure ÿ.ÿ: Pre-training and ÿne-tuning paradigm. ÿ Large Language Models are ÿrst trained in an unsupervised fashion on massive textual corpora, and then ÿne-tuned on a speciÿc supervised dataset for a related task.

ÿ

  Figure from https://ai.stanford.edu/blog/linkbert/ ÿÿ . Foundation models deep bidirectional representations, capturing both the left and the right context of each token.

Figure

  Figure ÿ.ÿ: Masked Language Modeling and similar pre-training objectives. In each scenario, |C| represents the number of classes of the pre-training objective, which considerably impacts computational eÿciency. Figure from (Yamaguchi et al. ÿÿÿÿ).

  Figure ÿ.ÿ provides an overview of the main LLM released over the last years.

Figure

  Figure ÿ.ÿ: Timeline (left-to-right) of the released LLMs (bigger than ÿÿB parameters) over the last years. The models marked in yellow are the ones made available for public use. The ÿgures along the timeline represent the month of release. Figure from (W. X. Zhao et al. ÿÿÿÿ).

  Figure ÿ.ÿÿ: Few-shot learning paradigm. The objective is to leverage information from one or few annotated examples in order to perform many downstream tasks such as classiÿcation (i), generation of new examples (ii), segmentation and parsing (iii), new concepts generation (iv). Figure from (Lake et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿÿ: Adapter architecture (right) and its integration in Transformer (left). The Adapter consists in few-parameter modules that are inserted after Transformer FFN. When ÿne-tuning the modiÿed architecture on a downstream tasks, only green modules (within Adapter and Layer Normalization) are updated. Figure from (Houlsby et al. ÿÿÿÿ).

Figure

  Figure ÿ.ÿÿ: Prompt-based few-shot learning. The considered objective is to classify the input sentence "Excellent pizza!" as good or bad. The pattern is ÿrst transforming the input as a cloze question P (x). P (x) is then fed to a PLM that outputs prediction scores for the masked word. Eventually, the verbalizer v converts the token prediction scores as classiÿcation logits. ÿ

  I S and I Q represent the drawn indices during the sampling process for support set and query set, respectively. The task is thus named a N S -shot K-way task. Pre-trained models use few-shot techniques and the labeled support sets to adapt to ÿÿÿ A transductive approach for performing few-shot classi cation in NLP the tasks at hand and are evaluated based on their performances on the unlabeled query sets. This setting is illustrated in Figure ÿ.ÿ for a computer vision application.

Figure

  Figure ÿ.ÿ: 3 shots 3 ways tasks example for a computer vision task. Figure from (Ouali ÿÿÿÿ).

Figure

  Figure ÿ.ÿ: Comparison of cross-entropy-based and transductive-based approaches for diÿerent N S values on Tweet_eval dataset. We consider K = 20 classes.

Figure

  Figure ÿ.ÿ: Comparison of cross-entropy-based and TIM-based approaches for BERT and RoBERTa backbones on the Bankingÿÿ dataset, considering K = 77 classes.

  Figure ÿ.ÿ: Comparison of BERT (left) and RoBERTa (right) backbones performances on Bankingÿÿ when initializing classiÿcation head as support set prototypes. We consider K = 77 classes.

Figure

  Figure ÿ.ÿ: Comparison of cross-entropy-based and TIM-based approaches for diÿerent ÿne-tuning strategies on the Bankingÿÿ dataset (K = 77).

  . The other desiderate R and R are represented in the schema of the API-based FSL setting depicted in Figure ÿ.ÿ.

Figure

  Figure ÿ.ÿ: API-based few-shot learning scenario. The black-box API is providing embeddings from the pre-trained encoder g θ . The black-box scenario discards existing inductive approaches and ICL methods due to inaccessible of model's parameters (R and privacy concerns (R ). This scenario allows to tune a classiÿcation head h φ (using induction or transduction) at low computational cost (R ), while retaining all support labels locally.

Figure ÿ. ÿ :

 ÿ Figure ÿ.ÿ: Performance of the diÿerent pretrained encoders on the monolingual datasets.

  Figure ÿ.ÿ reveals that the global trends observed in

  Figure ÿ.ÿ: The eÿect of diÿerent ways and shots on test performance. Monolingual experiments are shown on top, and multilingual experiments on bottom.

  Figure ÿ.ÿ (left) details the GPT-ÿ.ÿ results of the experiments conducted on the mono-lingual datasets. These plots highlight the consistency of the tendencies emerged in

  Figure ÿ.ÿ: The diÿerent losses when training a on GPTÿ.ÿ embeddings.

  Figure ÿ.ÿ. While the trends observed in ÿÿÿ . Experiments

  Figure ÿ.ÿ: Performance of diÿerent pre-trained encoder on the monolingual datasets.

  Conclusion and Perspectivescould beneÿt the multimodal learning ÿeld by oÿering frameworks to address the heterogeneity gap challenge more eÿectively. ÿÿÿ ÿ Aÿÿÿÿÿÿÿ ÿÿÿ Pÿÿÿÿ ÿÿ Tÿÿÿÿÿÿ ÿ

cos 7 d

 7 FR (P (y|X = x), P (y|X = x ? )) 2 ç = X y∈Y p P (y|X = x)P (y|X = x ? ). (ÿ.ÿÿ)

  Ensuite les diffŽrentes dŽfaillances ont ŽtŽ scindŽes en deux ensembles. Un ensemble des dŽfaillances visibles des vannes et des pompes : 8blocage en position fermée d9une vanne ouverte 9 et 8blocage en position ouverte d9une pompe fermŽe 9 pour les vannes, et 8blocage en position dŽmarrŽe d9une pompe arrêtée9 et 8blocage en position arrêtée d9une pompe en fonctionnement9 pour les pompes. Un ensemble des dŽfaillances invisibles : 8blocage en position ouverte d9une vanne ouverte9 et 8blocage en position fermée d9une vanne fermée9 pour les vannes, et 8blocage en position arrêtée d9une pompe arrêtée9 et 8blocage en position dŽmarrŽe d9une pompe en fonctionnement9 pour les pompes. Seules les dŽfaillances 23ème Congrès Lambda Mu de l9IMdR 10 au 13 octobre 2022, EDF Lab Paris Saclay

Figure 3 :

 3 Figure 3 : Evolution des mŽtriques selon la valeur du seuil VI. PERPESPECTIVES Les travaux prŽsentŽs dans cette publication concernent la mise en place d9un outil de surveillance, et donc de détection de dŽfaillances du syst•me virtuelle 3-RŽservoirs. Ces travaux peuvent •tre Žtendus et poursuivis suivant diffŽrentes orientations afin d9atteindre un niveau plus élevŽ de polyvalence, de performance et de gŽnŽricitŽ.

  Table ÿ.ÿ: Comparison of diÿerent approaches for multimodal representation learning.

	Approach	Advantages		Drawbacks
	Model-Agnostic	Simple and ÿexible.		May miss complex dynamics.
	PGM	Handles missing modalities.	Computationally expensive.
		Unsupervised training.
	Autoencoders	Unsupervised. Captures es-	Task-agnostic	representa-
		sential semantics.		tion.
	Attention-based Models	Interpretable.	Balance	Performance varies with task
		modalities.		complexity.
	Transformers	High performance. Some-
		times interpretable through
		attention maps. Oÿers many
		architecture variants.

  There exist many multimodal datasets, for other diÿerent applications ÿelds, like Visual Question Answering (Microsoft COCO(Lin et al. ÿÿÿÿ)), Aÿective Computing (CMU-MOSEI (Zadeh, P. P. Liang, et al. ÿÿÿÿ)), Healthcare (MIMIC-iii(Johnson et al. ÿÿÿÿ)) and so on, but none of these datasets possess the ÿve desired properties of our challenge. Recently, (P. P.Liang, Y. Lyu, et al. ÿÿÿÿ) proposed a uniÿed benchmark spanning ÿÿ datasets, ÿÿ modalities, ÿÿ prediction tasks, and ÿ research areas. Among these datasets, the ones related to the aÿective computing ÿeld appeared

to us as the closest to our problem, as they present a sequential setting, with unaligned streams from diÿerent modalities and with diÿerent acquisition frequencies. We thus chose to conduct experiments on the CMU-MOSEI dataset, addressing a Multimodal Sentiment Analysis task, as introduced in Example ÿ. This decision -by lack of dataset considering Multimodal Fault Diagnosis task -is compatible with the lens we see our challenge through: by seeing the tasks of fault monitoring and diagnosis as a unique classiÿcation in C + 1 classes as in Subsection ÿ.ÿ.ÿ, and by writing the function f in Equation ÿ.ÿ as f = h g, we now seek f that minimizes the related empirical risk as deÿned in ÿ.ÿ:

  We compared StreaMulT performances with Multimodal Transformer (MulT) and other models addressing Multimodal Sentiment Analysis challenge, among which the recent SOTA methods (W.Han et al. ÿÿÿÿ; W. Yu et al. ÿÿÿÿ). We strongly emphasize that the added value of StreaMulT is its ability to deal with arbitrarily long unaligned multimodal inputs, and that it does not intend to address Multimodal Sentiment Analysis speciÿc task. Hence at ÿrst we only reported Multimodal Transformer metrics scores given in(Tsai et al. ÿÿÿÿ) for a fair comparison, as both approaches use GloVe embeddings for text modalities whereas most recent works (W.Han et al. ÿÿÿÿ; W. Yu et al. ÿÿÿÿ) use BERT embeddings. We also used the available oÿcial code ÿ for Multimodal Transformer architecture to run the experiments, with hyperparameters given in(Tsai et al. ÿÿÿÿ).

	StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long
	Sequential Data
	ÿ https://github.com/yaohungt/Multimodal-Transformer
	ÿÿ

al. ÿÿÿÿ) pre-trained embeddings. This produces an unaligned version of the dataset, which is used to create a word-aligned version, using PÿFA algorithm (J.

Yuan et al. ÿÿÿÿ)

. All aligned sentences are padded to a ÿxed length of ÿÿ time steps. The related task aims to perform sentiment analysis on these clips, labeled by human annotators with a sentiment score from -ÿ to ÿ. As in

(Tsai et al. ÿÿÿÿ)

and previous works, we evaluate model performances using various metrics: ÿ-class-accuracy, binary accuracy (positive or negative statements), Fÿ-Score, MAE and correlation between model's predictions and labels.

ÿÿÿÿÿ Eÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿ

To highlight StreaMulT added value, we conduct experiments in diÿerent settings. We ÿrst consider input video clips as our whole input sequences, and observe StreaMulT performances when dividing these clips into smaller segments. As we need to deÿne hard segment temporal bounds, which are not given in the unaligned version of CMU-MOSEI, we conduct this experiment with the aligned version of the dataset. For StreaMulT, we choose to divide the input sentences into ÿ segments of length ÿÿ.

  Table ÿ.ÿ, with a signiÿcant improvement in all metrics, StreaMult-BART achieving now the best results on the aligned version of CMU-MOSEI dataset. We then trained the Multimodal Transformer and StreaMulT architectures on unaligned version of CMU-MOSEI dataset and reported the results in Table ÿ.ÿ.

	Metric		MAE l Corr h	Acc h 7	Acc h 2	Fÿ h
	MulT ‡		0.580 0.703	51.8	82.5	82.3
	MulT *		ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ
	StreaMulT *		ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ
	MulT-BERT *		ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ
	StreaMulT-BERT *	ÿ.ÿÿÿ	ÿ.ÿÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ	ÿÿ.ÿÿ
	MulT-BART *	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ
	StreaMulT-BART *	.	.	.	.	.
	Metric	MAE l Corr h	Acc h 7		Acc h 2	Fÿ h
	TFN ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿ	-/ÿÿ.ÿ	-/ÿÿ.ÿ
	LMF ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿ	-/ÿÿ.ÿ	-/ÿÿ.ÿ
	MFM ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿ	-/ÿÿ.ÿ	-/ÿÿ.ÿ
	ICCN ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿ	-/ÿÿ.ÿ	-/ÿÿ.ÿ
	MulT \	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿ	-/ÿÿ.ÿ	-/ÿÿ.ÿ
	MISA ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	-	ÿÿ.ÿÿ/ÿÿ.ÿÿ ÿÿ.ÿÿ/ÿÿ.ÿÿ
	MAG-BERT ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	-	ÿÿ.ÿ/ÿÿ.ÿ	ÿÿ.ÿ/ÿÿ.ÿ
	Self-MM ‡	ÿ.ÿÿÿ	ÿ.ÿÿÿ	-	ÿÿ.ÿÿ/ÿÿ.ÿÿ	ÿÿ.ÿÿ/ÿÿ.ÿÿ
	MMIM ‡	.	ÿ.ÿÿÿ	.	ÿÿ.ÿÿ/ÿÿ.ÿÿ ÿÿ.ÿÿ/ÿÿ.ÿÿ
	MulT-BERT	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ ÿÿ.ÿÿ/ÿÿ.ÿÿ	ÿÿ.ÿÿ/ÿÿ.ÿÿ
	MulT-BART	ÿ.ÿÿÿ	.	ÿÿ.ÿÿ		. / .	. / .
	StreaMulT-BERT	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ/ÿÿ.ÿÿ	ÿÿ.ÿÿ/ÿÿ.ÿÿ
	StreaMulT-BART	ÿ.ÿÿÿ	ÿ.ÿÿÿ	ÿÿ.ÿÿ ÿÿ.ÿÿ/ÿÿ.ÿÿ ÿÿ.ÿÿ/ÿÿ.

Table ÿ.ÿ: Results on CMU-MOSEI aligned. Best results are marked in bold. ‡: results from (Tsai et al. ÿÿÿÿ). *: Own implementation or reproduced from oÿcial code with provided hyperparameters. ÿÿ Table ÿ.ÿ: Results on CMU-MOSEI unaligned. Best results are marked in bold. ‡: results from (W. Han et al. ÿÿÿÿ). \: results from (Tsai et al. ÿÿÿÿ).

Table ÿ

 ÿ

	Layer Type	Time Complexity by layer	Space Complexity by layer	Sequential Operations
	Self-Attention	O(n 2 .d)	O(n 2 + n.d)	O(1)
	Cross-modal Attention	O(nα.n β .d)	O(nα.n β + nα.d + n β .d)	O(1)
	Streaming Cross-modal Attention (regular training scheme)	O(nα.n

.ÿ: Ablation study on CMU-MOSEI aligned. Best results are marked in bold. ÿÿÿ Tÿÿÿ ÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿ β .d) O(nα.n β + nα.d + n β .d) O(1) Streaming Cross-modal Attention (ÿexible training scheme) O(nα.h.C β .d) O(h 2 .Cα.C β + h.Cα.d + h.C β .d) O( nα hCα ) Table ÿ.ÿ: Time and space Complexities for diÿerent layer types.

  Algorithm StreaMulT Training loop.

	StreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long
	Sequential Data
	Algorithm StreaMulT forward loop.
	Require: train_loader, model, text_encoder, optimizer, criterion
	for i = 1, . . . , nb_sequences_batches do
	sequences, labels	iterate(train_loader)
	raw_text, audio, vision	sequences
	text	text_encoder(raw_text)
	segments_batches	sequence_to_segments_batches(text, audio, vision, labels,
			segment_size, memory_batch_size,
			left_context, right_context)
	state	None
	for j = 1, . . . , nb_segments_batches do
	text, audio, vision, labels	segments_batches[j]
	preds, state	model(text, audio, vision, state)
	loss		MAE(preds, labels)
	model	backward_propagation(loss, model)
	model	update(model, optimizer)
	end for
	end for
			ÿÿ

  Algorithm Streaming Cross-modal Transformer ( ! µ) forward loop. Require: X α , X β , state, rpe X β→α X α for i = 1, . . . , nb_layers do rc_blocks β→α , central_segments β→α X β→α summary β→α summarize(rc_blocks β→α , central_segments β→α ) rc_blocks β , central_segments β X β X β→α [LN([rc_blocks β→α : central_segments β→α ])] : summary β→α ] X β [memory β : LN([rc_blocks β

  The ÿÿStreaMulT: A Streaming Multimodal Transformer For Heterogeneous and Arbitrarily Long Sequential Data "after_attention_operations" function also contains update steps for the variable "state".We realized an hyperparameters tuning when training the StreaMulT model, evaluating diÿerent parameters conÿgurations on a validation set. The optimized hyperparameters are listed in Tableÿ.ÿ, alongside with their diÿerent values.

	Hyperarameter	Value
	batch size	ÿÿ
	nb layers Emformer	ÿ
	nb layers SCT	ÿ
	nb heads attention	ÿ
	embedding dimension	ÿÿ
	segment size	ÿ
	memory size	ÿ
	left context	ÿ
	right context	ÿ
	keep raw	False
	ÿne-tune text encoder	True
	learning rate	ÿe-ÿ
	learning rate text encoder	ÿe-ÿ

Table ÿ.ÿ: Optimal hyperparameters conÿguration for StreaMulT on CMU-MOSEI aligned.

  al. ÿÿÿÿ; Henaÿ ÿÿÿÿ; Ji et al. ÿÿÿÿ; Y. Tian, Krishnan, et al. ÿÿÿÿ). Similarly,

  al. ÿÿÿÿ; Y.-C. Liu et al. ÿÿÿÿ; Yunze Liu et al. ÿÿÿÿ; Wan et al. ÿÿÿÿ; W. Yu et al. ÿÿÿÿ).

  Church et al. ÿÿÿÿ), ÿltering(Manning, Raghavan, et al. ÿÿÿÿ), and stemming(Porter ÿÿÿÿ), which prepare and process raw text,• intermediate-level tasks, like part-of-speech tagging(Marcus et al. ÿÿÿÿ) and named entity recognition (Bunescu et al. ÿÿÿÿ), which analyze and label the data, ÿÿ Background and Related work in NLP: from Symbolic methods to Foundation Models • and high-level tasks, including machine translation (Sutskever, Vinyals, et al. ÿÿÿÿ), sentiment analysis (Pang et al. ÿÿÿÿ), and question answering

  al. ÿÿÿÿ; Koehn et al. ÿÿÿÿ; Sutskever, Vinyals, et al. ÿÿÿÿ), speech recognition (G. Hinton et al. ÿÿÿÿ; Jelinek ÿÿÿÿ), and text generation

  and Related work in NLP: from methods to Foundation Models

	Methods	Scalability to large datasets	Adaptability	Expert Knowledge	Robustness to Unknown Words	Dependencies Between Words	Long Sequences Scalability	Semantic Representation	Context-dependent representations
	Rule-based	-	-	+	-	-	-	-	-
	Feature-Engineering Based	-	+/-+ +/-	-	-	-	-
	Vector Space Models (BoW, TF-IDF)	+/-+	-	-	-	+/-	-	-
	Probabilistic Frameworks (HMMs, CRFs)	-	-	+ +/-+	-	-	-
	Count-Based Word Embeddings	+/-+	-	-	+ +/-+	-
	Prediction-Based Word Embeddings	+	+	-	+	+ +/-+	-
	Count-based Language Models	+/-+/--	-	+/-	-	-	-
	Recurrents Neural Networks (LSTM,GRU) +	+	-	+	+ +/-+	+
	Transformers	+	+	+	+	+	+ ++ ++

Table ÿ.ÿ: Summary of advantages and limitations of general NLP methods and word embeddings techniques. "+" denotes signiÿcant presence/requirement of the criterion, "-" denotes signiÿcant lack/limitation, and "+/-" denotes moderate presence/requirement.

  . LLMs, typically composed of Transformer-based architec-

				. Foundation models
	Model	Architecture	Pre-training Loss	Corpus
	ELMo	LSTM	biLM	WikiText-ÿÿÿ
	GPT	Transformer Decoder	LM	BookCorpus
	BERT	Transformer Encoder	MLM & NSP	WikiEn+BookCorpus
	RoBERTa Transformer Encoder	MLM	BCOS
	BART	Transformer	CLM	BCOS
	Tÿ	Transformer	CLM	Cÿ
	Table ÿ.ÿ: Overview of diÿerent Transformer-based models.	BCOS stands for BookCor-
	pus+CCNews+OpenWebText+STORIES. biLM is a bidirectional LM loss.
	ÿÿ

  al. ÿÿÿÿ; Sanh et al. ÿÿÿÿ), without necessarily giving the model explicit examples, improving generalization abilities. Some studies (H. W. Chung et al. ÿÿÿÿ; Wei, Bosma, et al. ÿÿÿÿ) showed that this phenomenon induced by instruction-formatting essentially appears once a sufÿcient size has been reached. Some models such as Galactica (R. Taylor et al. ÿÿÿÿ) even include Instruction formatting within the pre-training stage to achieve superior performance and better generalization capacity. These emergent abilities are illustrated in Figure ÿ.ÿ.

Figure ÿ.ÿ: In-context learning (left): The model is given a prompt containing k input-label pairs (here k = 3) alongside with a test input (in the same prompt), and is asked to predict in response the test label. The model leverages the information contained in the demonstrations to eÿectively generate the label with no gradient update. Figure from (S. Min, X. Lyu, et al. ÿÿÿÿ). Instruction ÿne-tuning (right): The model is ÿne-tuned by providing Natural language descriptions of the task in preamble. It can also contain labeled examples in the prompt (bottom). Figure from (H. W. Chung et al. ÿÿÿÿ).

  Figure from http://timoschick.com/explanatory%ÿÿnotes/ÿÿÿÿ/ÿÿ/ÿÿ/pattern-exploiting-training.html ÿÿ . Few-shot learning in NLP aiming for a more localized rule rather than a general one (see Figure ÿ.ÿÿ), transductive learning has shown promise and practical beneÿts in FSL for computer vision (Dhillon et al. ÿÿÿÿ; Y. Guo et al. ÿÿÿÿ; R. Hou et al. ÿÿÿÿ; S. X. Hu et al. ÿÿÿÿ; Y. Hu et al. ÿÿÿÿ; J. Liu et al. ÿÿÿÿ; Yanbin Liu et al. ÿÿÿÿ; Yaoyao Liu et al. ÿÿÿÿ; Qiao et al. ÿÿÿÿ; Veilleux et al. ÿÿÿÿ; Yikai Wang et al. ÿÿÿÿ; Ling Yang et al. ÿÿÿÿ; Ziko et al. ÿÿÿÿ).

ÿ

  Table ÿ.ÿ: Results of the diÿerent ÿne-tuning methods for the Bankingÿÿ dataset (K = 77), along with the relative gain of TIM against CE method: Complete ÿne-tuning (C), BitFit (BF), Frozen LM (FLM), LayerNorm (LN).

		ÿ			ÿ			ÿÿ			ÿÿÿ
		CE	TIM	Gain	CE	TIM	Gain	CE	TIM	Gain	CE	TIM	Gain
	C	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ #-ÿ.ÿÿ
	BF	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ ÿÿ.ÿÿ	#-ÿ.ÿÿ
	FLM	ÿÿ.ÿÿ	ÿÿ.ÿÿ	"ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ #-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	"ÿ.ÿÿ
	LN	ÿÿ.ÿÿ ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ ÿÿ.ÿÿ	"ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ	ÿÿ.ÿÿ	ÿÿ.ÿÿ	#-ÿ.ÿÿ

  (KocoE et al. ÿÿÿÿ; Lehman et al. ÿÿÿÿ), GPT-ÿ (OpenAI ÿÿÿÿ), GPT-ÿ(Brown et al. ÿÿÿÿ), Tÿ(Raÿel, Shazeer, et al. ÿÿÿÿ), and BERT(Devlin et al. ÿÿÿÿ), which have achieved unprecedented performance in text classiÿcation(Yinhan Liu et al. ÿÿÿÿ), language modeling, machine translation(Fan et al. ÿÿÿÿ), and coding tasks

(Mark Chen et al. ÿÿÿÿ).

  al. ÿÿÿÿ; Perez et al. ÿÿÿÿ) mainly focus on datasets with a reduced number of classes (i.e., K < 5).

		Motivated by
	practical considerations we choose to build a new benchmark composed of datasets with a larger
	number of classes.
	Dataset	Number of classes
	Multilingual Amazon Reviews Corpus (Keung et al. ÿÿÿÿ)

  Table ÿ.ÿ: Statistics of the considered datasets.

  al. ÿÿÿÿ), we consider MPNET-base (K. Song et al. ÿÿÿÿ) (ÿÿÿM parameters), MiniLM (ÿÿM parameters) (W. Wang et al. ÿÿÿÿ), and Albert Small Vÿ (ÿÿM parameters) (Z. Lan et al. ÿÿÿÿ).

  Table ÿ.ÿ: Preliminary experiment results. Accuracy of the diÿerent backbone trained on each training set.

	Model	Params Emotion Twitter Clinc Bankingÿÿ		Amazon
			en	en	en	en	en	fr	es	de
	Albert Small Vÿ (XS)	ÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	X	X	X
	MiniLM (S)	ÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	X	X	X
	MPNET-base (B)	ÿÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	X	X	X
	DistilRoBERTa (S)	ÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	X	X	X
	RoBERTa (B)	ÿÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	X	X	X
	RoBERTa (L)	ÿÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	X	X	X
	XLM-RoBERTa (B)	ÿÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ
	XLM-RoBERTa (L)	ÿÿÿM	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ
	XLM-RoBERTa (XL) ÿ.ÿÿB	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ
	GPT-ÿ.ÿ	ÿÿÿB	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ	ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ ÿÿ.ÿ

Table ÿ

 ÿ

	Textual Few-Shot Classi cation For API-based Models
	ÿÿÿÿÿ Sÿÿÿÿ ÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿÿ ÿÿÿÿÿÿÿ
	.ÿ remain consistent across datasets of vary-
	ing diÿculty levels. Notably, we observed consis-
	tent improvements achieved by transductive regu-
	larizers (such as I or FR) over CE. However, the
	relative improvement is highly dependent on the
	speciÿc dataset being evaluated. Speciÿcally, FR
	achieves +ÿ.ÿ% Fÿ-score on Bankingÿÿ, but only a
	shy +ÿ.ÿ% on Tweet_eval. A strong baseline gener-
	ally suggests highly discriminative features for the
	task, and therefore a strong upside in leveraging ad-
	ditional unlabeled features, and vice versa. There-
	fore, we hypothesize that the potential gains to be
	obtained through transduction correlate with the
	baseline's performance. Additional results can be
	found on Subsection ÿ.ÿ.ÿ multilingual experiments (i.e., on es, de, fr) which exhibit the same
	behavior.
	ÿÿÿ

  Table ÿ.ÿ, Table ÿ.ÿ and Figure ÿ.ÿ, namely: the superiority of transductive approaches (FR and I) over inductive ones

  The results are presented in Table ÿ.ÿ. Our observations indicate that the transductive regularization improves the results for two languages over the inductive baseline (i.e., CE). Additionally, we note that the observed improvements for FR are more consistent. This further demonstrates that the transductive loss can be useful in few-shot NLP. Table ÿ.ÿ: Global results for multilingual Amazon dataset.

		fr	de	en	es
	FR	.	.	.	.
	I	ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ
	H	ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ
	CE ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ
	PT ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ ÿÿ.ÿÿ

  entails a training time of ÿÿ minutes on Aÿÿÿ, while ADAPET (Tam et al. ÿÿÿÿ) necessitates ÿÿ minutes on the same hardware.Table ÿ.ÿ: Training time for ÿ episode on a Mÿ-CPU.

		. Experiments
	Loss CPU Time
	CE	ÿ.ÿÿs
	FR	ÿ.ÿÿs
	H	ÿ.ÿÿs
	I	ÿ.ÿÿs
	PT	ÿ.ÿÿs
	ÿÿÿ

  (Arivazhagan et al. ÿÿÿÿ; X. Ma et al. ÿÿÿÿ; Raÿel, M. T. Luong, et al. ÿÿÿÿ), in which one should alternate between reading the input and writing the output. Robustness and Handling of Missing Modalities Further, StreaMulT does not address the issue of missing modalities which can impact the functionality of cross-modal attention modules. It also does not explicitly tackle problems related to imbalanced datasets and concept drifts.

  La gestion de la maintenance d'installations industrielles de production est un facteur important de compŽtitivitŽ. Différentes techniques existent afin d9assurer au mieux les stratŽgies de maintenance, par exemple la surveillance et le diagnostic permettant de détecter et d9identifier une défaillance à la suite de son occurrence. Les travaux prŽsentŽs dans cette publication consistent à montrer l9application d9un algorithme de surveillance pour dŽtecter des occurrences de dŽfaillances sur un exemple applicatif virtuel du projet de recherche MPO, pour Maintenance Prévisionnelle et Optimisation, de l9IRT SystemX. L9exemple est le syst•me 3-RŽservoirs, dŽjˆ prŽsentŽ dans une prŽcŽdente communication, et nous y avons appliquŽ un algorithme d9apprentissage automatique afin de construire un outil de surveillance de dŽfaillances.

	Démonstration de surveillance de défaillances sur un exemple applicatif
	Fault monitoring demonstration on an applicative example
	PELLEGRAIN Victor	BATTEUX Michel	LAIR William	KACZMAREK Michel
	IRT SystemX	IRT SystemX	EDF R&D	Airbus Protect
	2, boulevard Thomas	2, boulevard Thomas	7 BD Gaspard Monge	1 Bd Jean Moulin
	Gobert	Gobert	91120 Palaiseau	CS 70562
	91120 Palaiseau	91120 Palaiseau	william.lair@edf.fr	78996 Elancourt Cedex
	victor.pellegrain@irt-	michel.batteux@irt-		Michel.Kaczmarek@apsys
	systemx.fr	systemx.fr		-airbus.com
	RŽsumŽ 4

al. ÿÿÿÿ). ÿÿÿ 23•me Congrès Lambda Mu de l9IMdR 10 au 13 octobre 2022, EDF Lab Paris Saclay

  1 www.irt-systemx.fr/projets/mpo II. RAPPEL D9ETAT DE L9ART SUR LA SURVEILLANCE ET LE DIAGNOSTIC Comme indiquŽ en introduction, les dŽfaillances de composants ou parties d9un système ne peuvent être complŽtement Žviter. Un levier pour limiter le risque d9occurrence de défaillance du système ou de sa conséquence est de mettre en place des techniques permettant de dŽtecter au plus vite une anomalie Ces techniques sont connues sous les termes de 8surveillance9 et 8diagnostic9. Il y a deux principales approches pour la surveillance et le diagnostic[START_REF] Isermann | <Fault-Diagnosis Systems=[END_REF] : les approches dites 8basŽes mod•les9 et les approches dites 8basŽes donnŽes9.A. Les approches basŽes mod•lesLes approches ˆ base de mod•les consistent ˆ comparer le comportement rŽellement observŽ du syst•me ˆ un comportement prŽdit, issu d'un mod•le de fonctionnement nominal et avec dŽfaillances du syst•me. Les mod•les utilisŽs par ces mŽthodes peuvent •tre de deux types : les mod•les quantitatifs et les mod•les qualitatifs. Les approches par mod•les quantitatifs sont celles issues de la communauté de l9automatique, et classiquement nommées par l9acronyme FDI pour 8fault detection and isolation9. L'utilisation d'un mod•le de fonctionnement nominal du syst•me permet d'engendrer des incompatibilitŽs entre le comportement rŽel du syst•me et celui prŽdit par le mod•le. Ces incompatibilitŽs, appelŽes 8rŽsidus9, sont gŽnŽrŽes ˆ partir des mesures effectuŽes sur le syst•me et de calculs fondŽs sur le mod•le du syst•me. Ces rŽsidus sont des signaux devant reflŽter la cohŽrence des donnŽes mesurŽes du syst•me par rapport au mod•le de fonctionnement. L'objectif d'un rŽsidu est d'•tre sensible aux dŽfaillances : c'est-ˆ-dire qu'il doit reflŽter l'Žventuelle prŽsence d'une dŽfaillance. Cela signifie donc qu'un rŽsidu est en gŽnŽral proche d'une valeur de rŽfŽrence si aucune dŽfaillance n'affecte le syst•me, et qu'il est dŽviŽ vers une valeur diffŽrente d•s l9occurrence d'une dŽfaillance. Les mod•les qualitatifs permettent d'abstraire, ˆ un certain degrŽ, le comportement du syst•me ˆ travers des mod•les de type symbolique. Ces mod•les dŽcrivent d'une mani•re qualitative l'espace d'Žtat continu du syst•me et ne reprŽsentent pas la physique du syst•me, contrairement aux mod•les quantitatifs, car ils le dŽcrivent en termes de mode de fonctionnement. Les mŽthodes ˆ base de mod•les qualitatifs peuvent •tre classifiŽes soit selon le niveau d'abstraction considŽrŽ du syst•me ˆ diagnostiquer (les graphes causaux pour les syst•mes continus, les syst•mes ˆ ŽvŽnements discrets, ou encore les syst•mes hybrides dynamiques) ; soit selon la prise en compte, ou non, des dŽfaillances (les mod•les de dysfonctionnement comme dans les techniques de propagation des dŽfaillances ou pour les graphes causaux, ou les mod•les de bon fonctionnement dans le cas du diagnostic ˆ partir des principes premiers ou par simulation qualitative). B. Les approches basŽes donnŽes Contrairement aux mŽthodes ˆ base de mod•les, celles ˆ base de donnŽes reposent sur un nombre important de donnŽes 23ème Congrès Lambda Mu de l9IMdR 10 au 13 octobre 2022, EDF Lab Paris Saclay qui sont supposŽes reprŽsenter convenablement le syst•me. Les seules informations disponibles sont les signaux issus des capteurs du syst•me, ce qui implique que ces approches prŽsupposent donc que ce syst•me puisse •tre compl•tement dŽcrit par ses observations passŽes et prŽsentes. L'objectif de ces approches est alors de construire un mod•le ajustŽ sur les donnŽes collectŽes, et la principale difficultŽ va donc •tre de dŽfinir non seulement la structure appropriŽe du mod•le, mais aussi le calage appropriŽ entre ce mod•le et le syst•me. Les mŽthodes par reconnaissance de formes ont pour objectif de classifier des objets, nommŽs des 8formes9, qui sont reprŽsentŽes par des donnŽes, dans des classes prŽdŽterminŽes en les comparant ˆ des prototypes. Ces mŽthodes reposent donc sur une description compl•te de ces formes et de chacune des diffŽrentes classes prototypes. Un probl•me de diagnostic peut ainsi se dŽfinir comme un probl•me de reconnaissance de formes o• les classes sont les modes de fonctionnement du syst•me (nominal ou sous la prŽsence de dŽfaillances) et les formes sont reprŽsentŽes par les observations du syst•me. Les mŽthodes par syst•mes experts sont utilisŽes dans des applications o• l'expertise humaine y est importante et le dŽveloppement de mod•les y est faible. Ce sont des syst•mes ˆ base de r•gles du type 8si9, 8et9, 8ou9, 8alors9 qui utilisent une information heuristique pour lier les sympt™mes aux dŽfaillances, Žtablissant ainsi des associations empiriques entre effets et causes des dŽfauts. Ces associations sont gŽnŽralement fondŽes sur l'expŽrience de spŽcialistes, dits 8experts9, plut™t que sur une connaissance de la structure et/ou du comportement du syst•me. Leur fonctionnalitŽ est de trouver la cause de ce qui a ŽtŽ observŽ en parcourant, par un raisonnement abductif, les r•gles prŽalablement Žtablies. Enfin les mŽthodes par apprentissage machine (ML pour

	Les approches basŽes sur les mod•les qualitatifs sont celles issues de la communauté de l9intelligence artificielle
	(communautŽ historique, et pas celle actuelle liŽe ˆ l9apprentissage automatique), et nommées par l'acronyme DX pour 8Data eXtraction9.

  , une dŽfaillance peut •tre spŽcifiŽe par trois caractŽristiques : son comportement, son effet et sa consŽquence. Le comportement d'une dŽfaillance qui dŽtermine son instant d'occurrence dans le temps, sa force d'apparition ainsi que sa durŽe de prŽsence. L'instant d'occurrence peut •tre alŽatoire, systŽmatique ou dŽpendant d'un Žv•nement interne ou externe au syst•me. La force d'apparition peut •tre brusque ou progressive. La durŽe de prŽsence d'une dŽfaillance peut •tre permanente, transitoire ou intermittente. L'effet d'une dŽfaillance dŽtermine sa prise en compte dans le syst•me. Il s'agit de dŽterminer sa localisation dans le syst•me ainsi que la ou les perturbations induites. 'eau circule de la fa•on suivante. Les deux rŽservoirs L1 et L2 sont alimentŽs par deux sources froides indŽpendantes gr‰ce aux deux pompes P1 et P2. Ces rŽservoirs L1 et L2 alimentent en eau le troisi•me rŽservoir L3 dans lequel se situe la source de chaleur. L'alimentation de L3 par L1 est gŽrŽe par la vanne V1, et l'alimentation de L3 par L2 est gŽrŽe par la vanne V2. Enfin l'Žvacuation de l'eau de L3 est gŽrŽe par la vanne V3. Initialement, les deux pompes P1 et P2 fonctionnent et les vannes V1, V2 et V3 sont ouvertes. Les fonctionnements des ouvertures et fermetures des vannes V1, V2 et V3 dépendent du niveau d9eau dans le rŽservoir L3. Les vannes V1 et V2 se ferment quand la hauteur d9eau dans le réservoir L3 dŽpasse une certaine valeur seuil maximum, correspondant ˆ un niveau maximum dans les rŽservoirs, et elles s9ouvrent quand la hauteur est infŽrieure ˆ une certaine valeur seuil minimum. La vanne V3 s9ouvre quand la hauteur dŽpasse la valeur de seuil maximum, et se ferme quand la hauteur est infŽrieure ˆ la valeur de seuil minimum. Les fonctionnements des dŽmarrages et arr•ts des pompes sont similaire aux fonctionnements des ouvertures et fermeture des vannes. La pompe P1, respectivement P2, dŽmarre quand la hauteur d9eau dans le réservoir L1, respectivement L2, est infŽrieure ˆ une valeur seuil minimum ; et elle s9arrête quand cette hauteur d9eau est supérieure à une valeur seuil.

		syst•me. Enfin une dŽfaillance peut initier un
		dysfonctionnement ou une panne du syst•me.
		" Un dysfonctionnement est une irrŽgularitŽ
		intermittente dans la rŽalisation d'une fonction
		dŽsirŽe du syst•me. Un dysfonctionnement est
		donc une interruption temporaire de la fonction
		du syst•me, et il s'agit d'un Žv•nement rŽsultant
		d'un ou plusieurs dŽfauts.
		" Enfin une panne est une interruption permanente
		de la capacitŽ du syst•me ˆ exŽcuter une fonction
		requise sous des conditions opŽrationnelles
	Figure 2 : ReprŽsentation schŽmatique du syst•me 3-RŽservoirs	spŽcifiŽes. Comme pour un dysfonctionnement, une panne est un Žv•nement rŽsultant d'un ou
	A. Fonctionnement du syst•me 3-RŽservoirs L9objectif du système 3-RŽservoirs est de refroidir la source de chaleur dans le réservoir L3 avec de l9eau dans les réservoirs L1 et L2. Pour cela il est nécessaire d9assurer un certain niveau de température et de hauteur d9eau dans ce rŽservoir L3.	plusieurs dŽfauts. DiffŽrents types de pannes peuvent •tre distinguŽs suivant leurs nombres (panne simple ou pannes multiples) et leurs prŽvisions (panne alŽatoire donc non prŽvisible, panne dŽterministe donc prŽvisible sous certaines conditions, panne systŽmatique ou causale dŽpendant de conditions connues).
		Selon [3]Enfin la consŽquence engendrŽe par une dŽfaillance, sur le
		syst•me lui-m•me et/ou son environnement, sont ˆ dŽterminer
		suivant les pertes potentielles (matŽrielles et/ou humaines)
		qu'il peut gŽnŽrer. Ces caractŽristiques permettent de bien
		dŽfinir une dŽfaillance afin de la modŽliser si nŽcessaire.
		III. LE SYSTéME 3-RƒSERVOIRS
		Le syst•me 3-RŽservoirs, prŽsentŽ dans [2], est un syst•me
		dynamique hybride, au sens o• ils combinent des phŽnom•nes
		qui seront dŽcrits par des Žvolutions continues et des
		phŽnom•nes qui seront dŽcrits par des Žvolutions discr•tes.
		Comme montrŽ en Figure 2, ce syst•me est constituŽ de
		diffŽrents composants : deux rŽservoirs amonts L1 et L2 et un
		rŽservoir aval L3, deux pompes P1 et P2, trois vannes V1, V2
		et V3, ainsi que trois capteurs CH1 CH2 et CH3 de hauteurs d9eau dans chaque réservoir, et un capteur de tempŽrature CT3
		dans le rŽservoir L3. Le rŽservoir aval L3 contient une source
		de chaleur qui fonctionne en continu et qui doit •tre refroidie par de l9eau froide venant des deux réservoirs L1 et L2 en
	Une dŽfaillance, également nommée 8faute9 par	amont.
	la communautŽ du diagnostic, est une dŽrive non-permise d9au moins une propriétŽ caractŽristique
	du syst•me par rapport aux conditions standard et
	acceptables de fonctionnement du syst•me. Une
	dŽfaillance est un Žtat anormal de
	fonctionnement du syst•me pouvant causer une
	rŽduction, voire une perte de la capacitŽ de l'unitŽ
	fonctionnelle ˆ exŽcuter sa fonction requise. Une
	dŽfaillance est indŽpendante du fait que le
	syst•me soit opŽrationnel ou non et peut tr•s bien
	ne pas affecter le fonctionnement normal du
	23ème Congrès Lambda Mu de l9IMdR	10 au 13 octobre 2022, EDF Lab Paris Saclay

L

Dataset

Task ÿÿÿÿÿ Rÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿ ÿÿÿ ÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿ

RQ : Do transductive methods improve few-shot classi cation performances over classic transfer learning?

To answer this question, we trained diÿerent transductive methods presented in Section ÿ.ÿ, and we compare their performances with the linear probing inductive baseline (by setting λ = 0 ÿÿÿ huit ensembles de données englobant la classiÿcation multi-classes dans quatre langues diÿérentes, avec jusqu'à ÿÿÿ classes. Nous évaluons nos méthodes à l'aide de huit modèles principaux et d'une évaluation épisodique sur ÿ ÿÿÿ épisodes, qui démontrent la supériorité de l'inférence transductive par rapport au cadre inductif standard. Kevin Zhou, and Tengchao Lv (ÿÿÿÿ). "Hierarchical attention prototypical networks for few-shot text classiÿcation". In: Proceedings of the conference on empirical methods in natural language processing and the th international joint conference on natural language processing. Sundermeyer, Martin, Ralf Schlüter, and Hermann Ney (ÿÿÿÿ). "LSTM neural networks for language modeling". In: Thirteenth annual conference of the international speech communication association. Sung, Flood, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales (ÿÿÿÿ). "Learning to compare: Relation network for few-shot learning". In: Proceedings of the IEEE conference on computer vision and pattern recognition. Sutskever, Ilya, James Martens, George Dahl, and Geoÿrey Hinton (ÿÿÿÿ). "On the importance of initialization and momentum in deep learning". In: Proceedings of the th International Conference on Machine Learning. Sutskever, Ilya, Oriol Vinyals, and Quoc Le (ÿÿÿÿ) Open and eÿcient foundation language models". arXiv preprint arXiv: . . Tripathi, Anshuman, Jaeyoung Kim, Qian Zhang, Han Lu, and Hasim Sak (ÿÿÿÿ). "Transformer transducer: One model unifying streaming and non-streaming speech recognition". arXiv preprint arXiv: . . Tsai, Yao-Hung Hubert, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov (ÿÿÿÿ). "Multimodal transformer for unaligned multimodal language sequences". In: Proceedings of the conference. Association for Computational Linguistics. Meeting. Turing, Alan M (ÿÿÿÿ). Computing machinery and intelligence. Vapnik, Vladimir (ÿÿÿÿ). The Nature of Statistical Learning Theory. Statistics for Engineering and Information Science. Springer. Vapnik, Vladimir N (ÿÿÿÿ). "An overview of statistical learning theory". IEEE transactions on neural networks. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Aukasz Kaiser, and Illia Polosukhin (ÿÿÿÿ). "Attention is all you need". Advances in neural information processing systems.

ÿÿÿ