Pierre Colombo

Victor Pellegrain

Malik Boudiaf

Victor Storchan

Myriam Tami

Ismail Ben Ayed

Céline Hudelot

Pablo Piantanida

Celine Hudelot

George Zha

He He 2021b Karypis

Aakanksha Chowdhery

Sharan Narang

Jacob Devlin

Maarten Bosma

Gaurav Mishra

Adam Roberts

Hyung Paul Barham

Won Chung

Charles Sutton

Sebastian Gehrmann

Parker Schuh

Kensen Shi

Sasha Tsvyashchenko

Joshua Maynez

Abhishek Rao

Parker Barnes

Yi Tay

Noam Shazeer

Vin- Odkumar Prabhakaran

Emily Reif

Nan Du

Ben Hutchinson

Reiner Pope

James Bradbury

Jacob Austin

Michael Isard

Guy Gur-Ari

Pengcheng Yin

Toju Duke

Anselm Levskaya

Sanjay Ghemawat

Sunipa Dev

Henryk Michalewski

Xavier Garcia

Vedant Misra

Kevin Robinson

Liam Fedus

Denny Zhou

Daphne Ippolito

David Luan

Hyeontaek Lim

Barret Zoph

Alexander Spiridonov

Ryan Sepassi

David Dohan

Shivani Agrawal

Mark Omernick

An- Drew M Dai

Thanumalayan Sankaranarayana

Marie Pellat

Aitor Lewkowycz

Erica Moreira

Rewon Child

Oleksandr Polozov

Katherine Lee

Zongwei Zhou

Xuezhi Wang

Brennan Saeta

Mark Diaz

Orhan Firat

Michele Catasta

Jason Wei

Kathy Meier-Hellstern

Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

Recent advances in Natural Language Processing (NLP) have been largely driven by the scaling paradigm [START_REF] Kaplan | Scaling laws for neural language models[END_REF][START_REF] Rosenfeld | A constructive prediction of the generalization error across scales[END_REF], where larger models with increased parameters have been shown to achieve state-of-the-art results in various NLP tasks [START_REF] Touvron | Llama: Open and efficient foundation language models[END_REF][START_REF] Radford | Language models are unsupervised multitask learners[END_REF]. This approach has led to the development of foundation models such as Chat-GPT [START_REF] Lehman | Do we still need clinical language models?[END_REF][START_REF] Kocoń | Chatgpt: Jack of all trades, master of none[END_REF] * These authors contributed equally to this work [START_REF] Kaplan | Scaling laws for neural language models[END_REF], GPT-4 (OpenAI, 2023), GPT-3 (Brown et al., 2020), T5 [START_REF] Raffel | Exploring the limits of transfer learning with a unified text-to-text transformer[END_REF], and BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], which have achieved unprecedented performance in text classification (Liu et al., 2019b), language modeling, machine translation [START_REF] Fan | Beyond english-centric multilingual machine translation[END_REF], and coding tasks (Chen et al., 2021a).

Despite the success of the scaling paradigm, significant challenges still exist especially when the many practical constraints of real-world scenarios have to be met: labeled data can be severely limited (i.e., few-shot scenario [START_REF] Song | A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities[END_REF][START_REF] Ye | Crossfit: A few-shot learning challenge for cross-task generalization in nlp[END_REF]), data privacy is critical for many industries and has become the subject of increasingly many regulatory pieces (Commission, 2020(Commission, , 2016)), compute costs need to be optimized [START_REF] Strubell | Energy and policy considerations for deep learning in nlp[END_REF]. Furthermore, these challenges are made even more complex as stronger foundation models are now available only through APIs (e.g., Ope-nAI's GPT-3, GPT-4 or ChatGPT, Anthropic's Claude or Google's PaLM (Chowdhery et al., 2022)) which has led to some of their parameters being concealed, presenting new challenges for model adaptation [START_REF] Solaiman | The gradient of generative ai release: Methods and considerations[END_REF]. This paper is centered on the fundamental task of fewshot text classification, specifically focusing on cloud-based/API access. Specifically, we formulate three requirements for API-based few-shot learning (FSL) (see Fig. 1): (R1) Black-box scenario. We focus on learning from models that are opaquely deployed in production to the end-user, who only has access to the end-point of the encoder, i.e., the resulting text embedding produced by the final layer of the network. (R2) Low resources / computation time. AI systems are often required to make rapid predictions at high frequencies in various real-world applications. Therefore, any few-shot classifier used in such scenarios should have a low training and inference time, as well as require minimal computational resources.

(R3) Limited Data Sharing. When utilizing API models, data sharing becomes a major concern. In the current landscape, providers are increasingly offering less transparent procedures for training their networks. As a result, users prefer sharing as little information as possible, such as labeling schema and annotated data, to safeguard their data privacy. Shortcomings of Existing Works. While numerous previous studies have addressed the popular few-shot classification setting, to our knowledge no existing line of work adequately satisfies the three API requirements described above. In particular, prompt-based FSL (Schick and Schütze, 2020a) and parameter-efficient fine-tuning FSL [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF] both require access to the model's gradients, while in-context learning scales poorly with the task's size (e.g number of shots, number of classes) (Chen et al., 2021b;[START_REF] Min | Metaicl: Learning to learn in context[END_REF][START_REF] Min | Rethinking the role of demonstrations: What makes in-context learning work?[END_REF][START_REF] Kaplan | Scaling laws for neural language models[END_REF] and requires full data sharing. Instead, we focus on methods that can operate within API-based constraints.

Under R1, R2, and R3 requirements, the standard inductive learning [START_REF] Liu | Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning[END_REF] may be quite limiting. To mitigate the labeled data scarcity while retaining API compliance, we revisit transduction [START_REF] Vladimir | An overview of statistical learning theory[END_REF] in the context of textual few-shot classification. Specifically, in the context of FSL, transductive FSL (Liu et al., 2019a) advocates leveraging unlabeled test samples of a task as an additional source of information on the underlying task's data distribution in order to better define decision boundaries. Such additional source essentially comes for free in many offline applications, including sentiment analysis for customer feedback, legal document classification, or text-based medical diagnosis.

Our findings corroborate recent findings in computer vision (Liu et al., 2019a;[START_REF] Ziko | Laplacian regularized few-shot learning[END_REF][START_REF] Lichtenstein | Tafssl: Task-adaptive feature sub-space learning for fewshot classification[END_REF]Boudiaf et al., 2020;Hu et al., 2021b), that substantial gains can be obtained from using transduction over induction, opening new avenue of research for the NLP community. However, the transductive gain comes at the cost of introducing additional hyperparameters, and carefully tuning them. Motivated by Occam's razor principle, we propose a novel hyperparameter-free transductive regularizer based on Fisher-Rao distances and demonstrate the strongest predictive performances across various benchmarks and models while keeping hyperparameter tuning minimal. We believe that this parameter-free transductive regu-larizer can serve as a baseline for future research.

Contributions

In this paper, we make several contributions to the field of textual FSL. Precisely, our contributions are threefold: A new textual few-shot scenario: We present a new scenario for FSL using textual API-based models that accurately capture real-world constraints. Our new scenario opens up new research avenues and opportunities to address the challenges associated with FSL using API-based models, paving the way for improved performance in practical applications.

A novel transductive baseline. Our paper proposes a transductive FSL algorithm that utilizes a novel parameter-free Fisher-Rao-based loss. By leveraging only the network's embedding (R1), our approach enables fast and efficient predictions (R2) without the need to share the labeling schema or the labels of few-shot examples making it compliant with (R3). This innovative method marks a significant step forward in the field of FSL. A truly improved experimental setting. Previous studies on textual few-shot classification (Schick andSchütze, 2022, 2020b;Mahabadi et al., 2022;[START_REF] Tam | Improving and simplifying pattern exploiting training[END_REF][START_REF] Gao | Making pre-trained language models better few-shot learners[END_REF] have predominantly assessed their algorithms on classification tasks with a restricted number of labels (typically less than five). We take a step forward and create a benchmark that is more representative of realworld scenarios. Our benchmark relies on a total of eight datasets, covering multiclass classification tasks with up to 151 classes, across four different languages. Moreover, we further enhanced the evaluation process by not only considering 10 classifiers trained with 10 different seeds [START_REF] Robert L Logan | Cutting down on prompts and parameters: Simple few-shot learning with language models[END_REF]Mahabadi et al., 2022), but also by relying on episodic evaluation on 1,000 episodes [START_REF] Hospedales | Meta-learning in neural networks: A survey[END_REF]. Our results clearly demonstrate the superiority of transductive methods.

Related Work

Few-shot learning in NLP

Numerous studies have tackled the task of FSL in Natural Language Processing (NLP) by utilizing pre-trained language models [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]Liu et al., 2019b;[START_REF] Radford | Language models are unsupervised multitask learners[END_REF][START_REF] Yang | Xlnet: Generalized autoregressive pretraining for language understanding[END_REF]. These methods can be classified into three major categories: prompt-based, parameterefficient tuning, and in-context learning. R3)). This scenario, allows tuning a classification head g ϕ (using induction or transduction) at low computational cost (R2) while retaining all support labels locally.

Prompt-based FSL: Prompt-based FSL involves the use of natural language prompts or templates to guide the model to perform a specific task [START_REF] Ding | Openprompt: An open-source framework for prompt-learning[END_REF][START_REF] Liu | Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing[END_REF]. For example, the seminal work (Schick and Schütze, 2020a) proposed a model called PET, which uses a pre-defined set of prompts to perform various NLP tasks as text classification. They also impose a choice of a verbalizer which highly impacts the classification performances [START_REF] Cui | Prototypical verbalizer for prompt-based few-shot tuning[END_REF][START_REF] Hu | Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification[END_REF]. However, recent studies have questioned the benefits of prompt-based learning due to the high variability in performance caused by the choice of prompt [START_REF] Liu | Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning[END_REF]. To address this issue, researchers have proposed prompt tuning which involves a few learnable parameters in addition to the prompt [START_REF] Lester | The power of scale for parameter-efficient prompt tuning[END_REF]. Nevertheless, these approaches face limitations when learning from API: (i) encoder access for gradient computation is infeasible (as in R1), (ii) prompting requires to send data and label which raises privacy concerns (as in R3), and (iii) labeling new points is time-consuming (see in R3) and expensive due to the need to send all shots for each input token1 . Parameter-efficient fine-tuning. These methods, such as adapters [START_REF] Houlsby | Parameter-efficient transfer learning for nlp[END_REF][START_REF] Pfeiffer | Adapterhub: A framework for adapting transformers[END_REF], keep most of the model's parameters fixed during training and only update small feedforward networks that are inserted within the larger model architecture. A recent example is T-FEW [START_REF] Liu | Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning[END_REF], which adds learned vectors that rescale the network's internal activations. Additionally, it requires a set of manually created prompts for each dataset making it hard to use in practice. Relying on parameter-efficient fine-tuning methods with an API is not possible due to the need to com-pute gradients of the encoder (as per R1) and the requirement to send both the labeling schema and the labels, which violates R3. In Context Learning (ICL). In-context learning models are models that utilize input-to-output training examples as prompts to make predictions, without any parameter updates [START_REF] Wei | Chain of thought prompting elicits reasoning in large language models[END_REF]. These models, such as text-davinci, rely solely on the provided examples to generate predictions, without any additional training. However, a significant drawback of this approach is that the user must supply the input, label examples, and task description, which becomes prohibitively expensive when the number of classes or shots increases, is slow [START_REF] Liu | Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning[END_REF] (R2) and raises data privacy concerns (as highlighted in R3). Additionally, the inability to reuse text embeddings for new tasks or with new labels without querying the model's API limits practicality and scalability, making reusable encoding unfeasible for in-context learning models2 . Meta-learning. Meta-learning approaches have for quite long stood as the de-facto paradigm for FSL [START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Rusu | Meta-learning with latent embedding optimization[END_REF]Sung et al., 2018b;[START_REF] Lee | Meta-learning with differentiable convex optimization[END_REF][START_REF] Raghu | Rapid learning or feature reuse? towards understanding the effectiveness of maml[END_REF]Sun et al., 2019a)). In meta-learning, the objective is to provide the model with the intrinsic ability to learn in a data-efficient manner. For instance, MAML ( (Finn et al., 2017a;Antoniou et al., 2018)), arguably the most popular meta-learning method, tries to train a model such that it can be fine-tuned end-to-end using only a few supervised samples while retaining high generalization ability. Unlike the three previous lines of work, meta-learning methods operate by modifying the pre-training pro-cedure and therefore assume access to both the training data and the model, which wholly breaks both R1 and R3.

Inductive vs transductive learning

Learning an inductive classifier on embeddings generated by an API-based model, as proposed by [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], is a common baseline for performing FSL. This approach is prevalent in NLP, where a parametric model is trained on data to infer general rules that are applied to label new, unseen data (known as inductive learning [START_REF] Vladimir | An overview of statistical learning theory[END_REF]). However, in FSL scenarios with limited labeled data, this approach can be highly ambiguous and lead to poor generalization.

Transduction offers an attractive alternative to inductive learning [START_REF] Stephan R Sain | The nature of statistical learning theory[END_REF]. Unlike inductive learning, which infers general rules from training data, transduction involves finding rules that work specifically for the unlabeled test data. By utilizing more data, such as unlabeled test instances, and aiming for a more localized rule rather than a general one, transductive learning has shown promise and practical benefits in computer vision (Boudiaf et al., 2020(Boudiaf et al., , 2021;;[START_REF] Ziko | Laplacian regularized few-shot learning[END_REF]. Transductive methods yield substantially better performance than their inductive counterparts by leveraging the statistics of the query set [START_REF] Guneet S Dhillon | A baseline for few-shot image classification[END_REF]. However, this approach has not yet been explored in the context of textual data.

API-based Few-shot Learning

Problem Statement

Let Ω be the considered vocabulary, we denote Ω * its Kleene closure. The Kleene closure corresponds to sequences of arbitrary size written with tokens in

Ω, i.e., Ω * = ∞ i=0 Ω i .
Given an input space X with X ⊆ Ω * and a latent space Z, we consider a pretrained backbone model f θ : X → Z = R d , where θ ∈ Θ represents the parameters of the encoder and d is the embedding dimension size. In the API-based setting, we assume that we are unable to access the exact structure of f θ as mentioned in R1. However, we do have access to the last encoder embedding which is available for our use (see R1).

The objective of few-shot classification is to learn a classifier from limited labeled data and generalize it to new, unseen tasks or classes. To accomplish this, randomly sampled few-shot tasks are created from a test dataset

D test := {(x i , y i )} Ntest i=1
that has a set of unseen classes Y test . Each task involves a few labeled examples from K different classes chosen at random among Y test . These labeled examples constitute the support set S = {x i , y i } i∈I S , with a size of |S| = N S × K. Additionally, each task has an unlabeled query set

Q = {x i } i∈I Q composed of |Q| = N Q × K un-
seen examples from each of the K classes. I S and I Q represent the drawn indices during the sampling process for support set and query set, respectively. Pre-trained models use few-shot techniques and the labeled support sets to adapt to the tasks at hand and are evaluated based on their performances on the unlabeled query sets.

Remark Setting the values of N and K in textual FSL is not standardized, as discussed in Sec. 3.1. Therefore, in all of our experiments, we have relied on setting (N, K) ∈ {5, 10} 2 .

Proposed Transductive Method

NLP few-shot classifiers rely only on inductive inference, while computer vision has shown significant performance improvements using transductive inference for FSL. Transductive inference succeeds in FSL because it jointly classifies all unlabeled query samples of a single task, leading to more efficient and accurate classification compared to inductive methods that classify one sample at a time. Let us begin by introducing some basic notation and definitions before introducing our new transductive loss based on the Fisher-Rao distance.

In the API-based few-shot classification setting, our goal is to train a classification head g ϕ : Z → R K that maps the feature representations to the posterior distribution space for making predictions. To simplify the equations for the rest of the paper, we use the following notations for the posterior predictions of each i ∈ I S ∪ I Q and for the class marginals within Q:

p ik = g ϕ (f θ (x i )) k = P(Y = k|X = x i ; θ, ϕ) and p k = 1 |Q| i∈I Q p ik = P(Y Q = k; θ, ϕ)
where X and Y are the r.v.s associated with the raw features and labels, respectively, and where Y Q means restriction of the r.v. Y to set Q.

For training the classification head in the transductive setting, prior research aims at finding ϕ

such that ϕ = arg min CE -λ × R Q 3 , with CE:= -1 |S| i∈I S K
k=1 y ik log(p ik ) being the cross-entropy supervision on the support set (in which y ik is the k th coordinate of the one-hot en-coded label vector associated to sample i) and R Q being a transductive loss on the query set Q.

Note that this transductive regularization has been proposed in the literature based on the Info-Max principle (Cardoso, 1997;[START_REF] Linsker | Self-organization in a perceptual network[END_REF], and the inductive loss can be found by setting λ = 0. In what follows, we review the regularizers introduced in previous work.

Entropic Minimization (H) An effective regularizer for transductive FSL can be derived from the field of semi-supervised learning, drawing inspiration from the approach introduced in [START_REF] Grandvalet | Semisupervised learning by entropy minimization[END_REF]. This regularizer, proposed in [START_REF] Guneet S Dhillon | A baseline for few-shot image classification[END_REF], utilizes the conditional Shannon Entropy (Cover, 1999) of forecast results from query samples during testing to enhance model generalization. Formally:

R H Q = 1 |Q| i∈I Q K k=1 p ik log(p ik ). (1)
Mutual Information Maximization (I) A promising alternative to entropic minimization for addressing the challenges of transductive FSL is to adopt the Info-max principle. (Boudiaf et al., 2020) extended this idea, introduced in [START_REF] Hu | Learning discrete representations via information maximizing self-augmented training[END_REF], and propose as regularizer a surrogate of the mutualinformation R I Q (α) =:

- K k=1 pk log pk + α 1 |Q| i∈I Q K k=1 p ik log(p ik ). (2)
Limitation of existing strategies: Despite its effectiveness, the previous method has a few limitations that should be taken into account. One of these limitations is the need to fine-tune the weight of different entropies using the hyperparameter α. This parameter-tuning process can be time-consuming and may require extensive experimentation to achieve optimal results. Additionally, recent studies have shown that relying solely on the first Entropic term, which corresponds to the Entropic minimization scenario in Equation 1, can lead to suboptimal performance in FSL.

A Fisher-Rao Based Regularizer

In the FSL scenario, minimizing parameter tuning is crucial. Motivated by this, in this section, we introduce a new parameter-free transductive regularizer that fits into the InfoMax framework. Additionally, our loss inherits the attractive properties of the Fisher-Rao distance between soft-predictions q := (q 1 , . . . , q K ) and p := (p 1 , . . . , p K ), which is given by [START_REF] Picot | Adversarial robustness via fisher-rao regularization[END_REF]:

d FR (q, p) := 2 arccos K k=1 √ q k × p k . (3)
The proposed transductive regularizer denoted by R FR Q , for each single few-shot task, can be described as measuring the Fisher-Rao distance between pairs of query samples:

R FR Q := 1 |Q| i∈I Q -log j∈I Q K k=1 p ik × p jk (4) = 1 |Q| i∈I Q -log j∈I Q cos d FR (p i , p j ) 2 , (5)
where d FR (p i , p j ) is the Fisher-Rao distance between pairs of soft-predictions (p i , p j ). Furthermore, it is shown that expression (4) yields a surrogate of the Mutual Information as shown by the following proposition. This result to the best of our knowledge is new, as far as we can tell.

Theorem 1. (Fisher-Rao as a surrogate to maximize Mutual Information) Let (q i ) i∈I Q be a collection of soft predictions corresponding to the query samples. Then, it holds that ∀ 0 ≤ α ≤ 1:

R FR Q + log |Q| ≤ R I Q (1) ≤ R I Q (α), (6)
Proof: Further details are relegated to Ap. A.

Advantage of R FR Q over R I Q (α): Similarly to R I Q (α), R FR
Q can be exploited to maximize the Mutual Information. However, R FR Q is parameter-free and thus, it does not require tuning α.

Additional Few-shot Inductive Baseline

In addition to the transductive methods of Sec. 3.2, we will explore three additional inductive methods for few-shot classification: prototypical networks, linear probing, and a semi-supervised classifier.

Prototypical Networks (PT) PT learn a metric space where the distance between two points corresponds to their degree of similarity. During inference, the distance between the query example and each class prototype is computed, and the predicted label is the class with the closest prototype. PT has been widely used in NLP and is considered as a strong baseline [START_REF] Snell | Prototypical networks for few-shot learning[END_REF]Sun et al., 2019b;[START_REF] Gao | Hybrid attention-based prototypical networks for noisy few-shot relation classification[END_REF]. Linear Probing (CE) Fine-tuning a linear head on top of a pretrained model is a popular approach to learn a classifier for classification tasks and was originally proposed in [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF].

Semi-supervised Baselines (SSL). We additionally propose two semi-supervised baselines following two steps. In the first step, a classifier is trained using the support set S and used to label Q. In the second step, the final classifier is trained on both S and Q with the pseudo label obtained from the first step.

An Enhanced Experimental Setting

Datasets

Benchmarking the performance of FSL methods on diverse sets of datasets is critical to evaluate their generalization capabilities in a robust manner as well as their potential for real-world applications. Previous work on FSL (Karimi Mahabadi et al., 2022;[START_REF] Perez | True few-shot learning with language models[END_REF] mainly focuses on datasets with a reduced number of classes (i.e., K < 5). Motivated by practical considerations we choose to build a new benchmark composed of datasets with a larger number of classes. Specifically, we choose Go Emotion [START_REF] Demszky | GoEmotions: A Dataset of Fine-Grained Emotions[END_REF]), Tweet Eval (Barbieri et al., 2020), Clinc [START_REF] Larson | An evaluation dataset for intent classification and out-ofscope prediction[END_REF]), Banking (Casanueva et al., 2020) and the Multilingual Amazon Reviews Corpus [START_REF] Keung | The multilingual Amazon reviews corpus[END_REF]. These datasets cover a wide range of text classification scenarios and are of various difficulty 4 . A summary of the datasets used can be found in Tab. 1.

Model Choice

The selection of an appropriate backbone model is a critical factor in achieving high performance in few-shot NLP tasks. To ensure the validity and robustness of our findings, we have included a diverse range of transformer-based backbone models in our study, including 1. Three different sizes of RoBERTa based models (Liu et al., 2019b). Similar to BERT, RoBERTa is 4 Datasets are available in Dataset [START_REF] Lhoest | Datasets: A community library for natural language processing[END_REF] pretrained using the closed task [START_REF] Wilson | cloze procedure": A new tool for measuring readability[END_REF]. We consider two different sizes of the RoBERTa model, namely RoBERTa (B) with 124M parameters and RoBERTa (L) with 355M parameters and Distil-RoBERTa, a lighter version of RoBERTa trained through a distillation process [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], for a total of 82M parameters.

2. Three sentence-transformers encoder [START_REF] Reimers | Sentence-BERT: Sentence embeddings using Siamese BERTnetworks[END_REF]. Following [START_REF] Muennighoff | Mteb: Massive text embedding benchmark[END_REF], we consider MPNET-base [START_REF] Song | Mpnet: Masked and permuted pretraining for language understanding[END_REF], MiniLM [START_REF] Wang | Minilm: Deep selfattention distillation for task-agnostic compression of pre-trained transformers[END_REF], and Albert Small V2 [START_REF] Lan | Albert: A lite bert for self-supervised learning of language representations[END_REF].

3. Multilingual models. To address realistic multilingual scenarios, we rely on three sizes of XLM-RoBERTa [START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF][START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF]: base (B), large (L) and XL (XL).

4. text-davinci model: to mimic the typical setting of API-based models, we also conduct experiments on text-davinci, only accessible through OpenAI's API.

Evaluation Framework

Prior research in textual FSL typically involves sampling a low number of tasks, typically less than 10, of each dataset. In contrast, we utilize an episodic learning framework that generates a large number of N-shots K-ways tasks. This framework has gained popularity through inductive metalearning approaches, such as those proposed by (Finn et al., 2017b;[START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Vinyals | Matching networks for one shot learning[END_REF]Sung et al., 2018a;[START_REF] Mishra | A simple neural attentive metalearner[END_REF][START_REF] Rusu | Meta-learning with latent embedding optimization[END_REF][START_REF] Oreshkin | Tadam: Task dependent adaptive metric for improved few-shot learning[END_REF], as it mimics the few-shot environment during evaluation and improves model robustness and generalization. In this context, episodic training implies that a different model is initialized for each generated few-shot task, and all tasks are compiled independently in parallel. This approach allows to the computation of more reliable performance statistics by evaluating the generalization capabilities of each method on a more diverse set of tasks. To account for the model's generalization ability, we average the results for each dataset over 1000 episodes, with the N considered classes varying in every episode. For each experiment, we consider the F1-Score.

Experiments

Case Study of text-davinci

In this experiment, we investigate the performance of text-davinci in both its language model and embedding-based model forms. We assess its classification capabilities using the aforementioned baseline and explore the language model's performance when applied in an in-context learning (ICL) setup with prompting.

Takeaways. From Tab. 2, we observe that SSL performs comparably to CE, which is simpler to use and will be considered as the baseline in the next part of our study. Although ICL slightly outperforms CE, its implementation comes at a significant cost. In ICL, each class requires N shots, forcing the user to send a long input query with additional instructions. This query length becomes prohibitive as the number of classes increases, and on average, it is 58 times longer than using the embedding base API in our benchmark. The lengthy input and ICL approach make it time-consuming for generation (violating R1), require the user to provide labels (violating R2), and prevent the reuse of embeddings for future use (e.g., retrieval, clustering). Additionally, ICL is 60 times more expensive than CE. Thus, we will discard ICL for the subsequent part of this study.

Overall Results

Global results: To evaluate the effectiveness of various few-shot methods, we conducted a comprehensive analysis of their classification performance across all datasets, all backbones, and all considered N-shots/K-ways scenarios. Results are reported in Tab. 3. An interesting observation is that transductive approaches I and FR outperform their inductive counterparts (CE and PT). Notably, we found that vanilla entropy minimization, which solely relies on H, consistently underperforms in all considered scenarios. Our analysis revealed that FR surpasses traditional fine-tuning based on cross-entropy by a margin of 3.7%.

Mono-lingual experiment: In order to thoroughly analyze the performance of each method, we conducted a per-dataset study, beginning with a focus on the mono-lingual datasets. Fig. 2 reveals that the global trends observed in Tab. 3 remain consistent across datasets of varying difficulty levels. Notably, we observed consistent improvements achieved by transductive regularizers (such as I or FR) over CE. However, the relative improvement is highly dependent on the specific dataset being evaluated. Specifically, FR achieves +6.5% F1-score on Banking, but only a shy +1.5% on Tweet. A strong baseline generally suggests highly discriminative features for the task, and therefore a strong upside in leveraging additional unlabeled features, and vice versa. Therefore, we hypothesize that the potential gains to be obtained through transduction correlate with the baseline's performance.5

Study Under Different Data-Regime

In this experiment, we investigated the performance of different loss functions under varying conditions of 'ways' and 'shots'. As shown in Fig. 3, we observed that increasing the number of classes ('ways') led to a decrease in F1 while increasing the number of examples per class ('shots') led to an improvement in F1. This can be explained by

N10-K10 N10-K5 N5-K10 N5-K5 N10-K10 N10-K5 N5-K10 N5-K5 N10-K10 N10-K5 N5-K10 N5-K5 N10-K10 N10-K5 N5-K10 N5-K5
T ask -N-Shot -K-Way

N10-K10 N10-K5 N5-K10 N5-K5 N10-K10 N10-K5 N5-K10 N5-K5 N10-K10 N10-K5 N5-K10 N5-K5 N10-K10 N10-K5 N5-K10 N5-K5
T ask -N-Shot -K-Way the fact that having more data enables the classifier to better discern the characteristics of each class. Interestingly, the relationship between the number of shots and classification F1 may not be the same for all classes or all loss functions. Fig. 3 shows that different loss functions (e.g. FR on banking) benefited greatly from adding a few shots, while others did not show as much improvement. However, this variability is dependent on the specific dataset and language being used, as different classes may have different levels of complexity and variability, and some may be inherently easier or harder to classify than others.

Ablation Study On Backbones

In this experiment, we examined how different loss functions perform when increasing the number of parameters in various models. The results, presented in Fig. 4, show the average performance across the experiments and are organized by the loss function. We observed an inverse scaling law for both the RoBERTa and XLM-RoBERTa family of models, where increasing the number of parameters led to a decrease in performance for the losses tested. However, within the same family, we observe that the superiority of FR remains consistent. An interesting finding from Fig. 4 is that the transductive regularization technique using FR outperforms other methods on text-davinci. This highlights the effectiveness of FR in improving the performance of the model and suggests that transductive regularization may be a promising approach for optimizing language models.

Practical Considerations

In this experiment, we adopt a practical standpoint and aim to evaluate the effectiveness of an API model, specifically text-davinci. In Tab. 4, we report the training speed of one episode on a MAC with CPU. Overall, we observed that the transductive loss is slower as it necessitates the computation of the loss on the query set, whereas PT is faster as it does not involve any optimization. Furthermore, we note that FR is comparable in speed to I. To provide a better understanding of these results, we can compare our method with existing approaches (in the light of R2). For instance, PET (Schick and Schütze, 2020a) entails a training time of 20 minutes on A100, while ADAPET [START_REF] Tam | Improving and simplifying pattern exploiting training[END_REF] necessitates 10 minutes on the same hardware.

Conclusions

This paper presents a novel FSL framework that utilizes API models while meeting critical constraints of real-world applications (i.e., R1, R2, R3). This approach is particularly appealing as it shifts the computational requirements (R2), eliminating the need for heavy computations for the user and reducing the cost of embedding. To provide a better understanding, embedding over 400k sequences cost as low as 7 dollars. In this scenario, our research highlights the potential of transductive losses, which have previously been disregarded by the NLP community. A candidate loss is the Fisher-Rao distance which is parameter-free and could serve as a simple baseline in the future.

We are optimistic that our research will have a positive impact on society. Nonetheless, it is essential to acknowledge the limitations of API-based fewshot classification models despite their promising results in various tasks. Firstly, the performance of the introduced methods is heavily dependent on the quality of available API models. If the API models do not provide sufficient information or lack diversity, the introduced methods may struggle to accurately classify input texts. Secondly, the blackbox nature of the backbone limits the interpretability of API-based few-shot classification methods, which may hinder their adoption. Ultimately, the aim of this work is to establish a baseline for future research on transductive inference. As a result, not all existing transductive methods are compared in this study.

A Proof of Proposition 1

In this Appendix, we prove the inequality (Eq. 6) provided in Proposition 1. The right-hand side of (Eq. 6) follows straightforwardly from the definition of R I Q (α) and the non-negativity of the Shannon entropy. In order to prove the first inequality, we need to introduce the following intermediate result.

For any arbitrary random variable (r.v) X and countable r.v Y , and any real number β, let

I β (X; Y ) := -E X ⋆ Y log E X P (Y |X) P (Y |X ⋆ ) β ,
where the r.v X ⋆ follows the same distribution than X. Notice that it is obvious that I 1 (X; Y ) = I(X; Y ), where I(X; Y ) is Shannon Mutual Information. Lemma 1. For any arbitrary r.v. X and countable r.v. Y , we have

I(X; Y ) ≥ I β (X; Y ), for 0 ≤ β ≤ 1.
Proof of the lemma: We must show that the different of I(X; Y ) -I β (X; Y ) is nonnegative. To this end, we write this difference as:

I(X; Y ) -I β (X; Y ) (7) = -E X ⋆ Y log P 1-β (Y |X ⋆ )E X P (Y |X) E X P β (Y |X) (8) ≥ -log E X ⋆ Y P 1-β (Y |X ⋆ )E X P (Y |X) E X P β (Y |X) (9) = -log y∈Y E X ⋆ P (y|X ⋆ ) P 1-β (y|X ⋆ )E X P (y|X) E X P β (y|X) (10) = -log y∈Y E X ⋆ P β (y|X ⋆ )E X P (y|X) E X P β (y|X) (11) = -log y∈Y E X P (y|X) (12) = 0, (13)
where the first inequality follows by applying Jensen's inequality to the function t → -log(t).

Proof of Proposition 1: From Lemma 1, using Jensen's inequality, we have

I(X; Y ) = -E X ⋆ Y log E X P (Y |X) P (Y |X ⋆ ) , (14)
≥

-E X ⋆ Y log E X P (Y |X) P (Y |X ⋆ ) β (15) ≥ -E X ⋆ log E X E Y |X ⋆ P (Y |X) P (Y |X ⋆ ) β (16) = -E X ⋆ log E X ( 17
) y∈Y P β (Y |X)P 1-β (Y |X ⋆ ), (18)
where inequality (15) follows by applying Lemma 1 and inequality (16) follows by exploiting the convexity of the function t → -log(t) for any 0 ≤ β ≤ 1. Finally, it is not difficult to check from the definition of the Fisher-Rao distance given by expression (3) that

cos d FR (P (y|X = x), P (y|X = x ⋆ )) 2 = ( 19
) y∈Y P (y|X = x)P (y|X = x ⋆ ). (20)

B Additional Experimental Results

B.1 Preliminary Classification Results

Preliminary Experiment. In our experiments, the backbone models are of utmost importance. Our objective in this preliminary experiment is to assess the efficacy of these models when fine-tuning only the model head across a variety of datasets. Through this evaluation, we aim to gain insight into their generalization abilities and any dataset-specific factors that may influence their performance. This information can be utilized to analyze the performance of different models in the few-shot scenario, as described in Sec. 5. We present the results of this experiment in Tab. 5, noting that all classes were considered, which differs from the episodic training approach detailed in Sec. 5.

B.2 A Dive Into text-davinci results

text-davinci appears to be the backbone providing the most informative a priori embeddings in Tab. 5 and could be considered as the prime model for API-based FSL, showcasing the current requirements in this area. It is thus a typical candidate for application uses that must meet the following criteria (R1) -(R3). Therefore, we put a special emphasis on its related results. Fig. 6 (top) details the text-davinci results of the experiments conducted on the mono-lingual datasets. These plots highlight the consistency of the tendencies that emerged in Tab. 5, Tab. 3 and Fig. 2, namely: the superiority of transductive approaches (F R and I) over inductive ones (CE and P T ), the underperformance of the entropic-minimization-based strategy (H), and the higher amount of information conveyed by text-davinci learned embeddings over other backbones, resulting in higher F1 scores on all datasets.

These phenomena still occur in the multi-lingual setting, as illustrated in Fig. 6 (bottom), stressing the superiority of transductive (and especially FR) over other approaches for presumably universal tasks, beyond English-centered ones, and without the need for using language-specific engineering as for prompting-based strategies.

Note that for both of these settings, the entropic-minimization-based strategy (H) seems to be capped at a 15% F1 score, thus with no improvement over other backbones embeddings, and independently of the dataset difficulty.

B.3 Multilingual Experiment

To provide an exhaustive analysis, we report the same experiment that is made in Sec. 5.2, for multi-lingual models on the Amazon dataset. While both Latin languages (French and Spanish) share close results, with an F1 gain of 2.8% for FR over CE, the results in the German and English language exhibit an F1 increased by almost 4%.

B.4 Importance of Model Backbones on Monolingual Experiment

In this section, we report the results of our experiment aggregated per backbone. The goal is to understand how the different losses behave on the different backbones. The results are presented in Fig. 10. While the trends observed in the previous charts are retrieved for the majority of backbones, some of these models are exceptions. For example, while transductive methods perform generally better than inductive methods, the CE-based method seems to perform slightly better than I for XLM-RoBERTa-xl. Additionally, while FR is the most effective method for the majority of backbones, it is surpassed by I for the alldistilroberta-v1 model. Furthermore, the inverse-scaling-law details are found for the RoBERTa(B/L) and XLM-RoBERTa (B/L) models per dataset. In general, it is interesting to note that although model performance is constrained by dataset difficulty, the performance order of each method is consistent across all 4 datasets for each considered backbone.

B.4.1 Results Per Language

In this experiment, we report the performance of different losses on the Amazon dataset by averaging the results over the number of shots, ways, and model backbones. The results are presented in Tab. 6.

Our observations indicate that the transductive regularization improves the results for all languages over the inductive baseline (i.e., CE), with a substantially higher gain for the German language. Additionally, we note that the observed improvements for FR are more consistent. This further demonstrates that the transductive loss can be useful in few-shot NLP. In the future, we would like to explore the application of transductive inference to other NLP tasks such as sequence generation [START_REF] Pichler | A differential entropy estimator for training neural networks[END_REF][START_REF] Colombo | Affect-driven dialog generation[END_REF]Colombo et al., , 2021d,b) ,b) and classification tasks (Chapuis et al., 2020;Colombo et al., 2022d,b;[START_REF] Himmi | Towards more robust nlp system evaluation: Handling missing scores in benchmarks[END_REF] as well as NLG evaluation (Colombo et al., 2021e, 2022c(Colombo et al., 2021e, , 2021c,a,b) ,a,b) and Safe AI (Colombo et al., 2022a;Picot et al., 2022a,b;[START_REF] Darrin | Rainproof: An umbrella to shield text generators from out-of-distribution data[END_REF][START_REF] Darrin | Unsupervised layer-wise score aggregation for textual ood detection[END_REF]

Figure 1 :

 1 Figure1: API-based FSL scenario. The black-box API provides embeddings from the pretrained encoder f θ . The black-box scenario discards existing inductive approaches and in-context learning methods due to the inaccessible of the model's parameters ((R1)) and privacy concerns ((R3)). This scenario, allows tuning a classification head g ϕ (using induction or transduction) at low computational cost (R2) while retaining all support labels locally.

Figure 2 :

 2 Figure 2: Performance on the monolingual datasets.

Figure 3 :Figure 4 :

 34 Figure 3: The effect of ways and shots on test performance on monolingual (left) and multilingual (right) datasets.

Figure 10 :

 10 Figure 10: Performance on monolingual datasets.

Table 1 :

 1 Datasets Statistics.

	Dataset Classes (K)
	Tweet	20
	Emotion	25
	Amazon	30
	B77	77
	Clinc	151

Table 2 :

 2 Aggregated performance over K, N, the different datasets for text-davinci. |x| stands for the averaged input length.

	N-shots	10		5	|x|
	K-ways	10	5	10	5
	FR	69.83 77.46 66.70 75.03 14.2
	H	10.00 20.00 10.01 20.04 14.2
	I	68.38 75.82 65.15 73.06 14.2
	CE	68.21 75.47 64.92 72.70 14.2
	PT	67.95 75.41 64.60 72.50 14.2
	SSL	68.27 75.55 64.99 72.75 14.2
	ICL	68.9 76.24	65.2	74.3 900

Table 3 :

 3 Aggregated performance over K,N, the different datasets and considered backbone.

	N-shots	10		5
	K-ways	10	5	10	5
	FR	52.09 61.99 48.71 56.55
	I	50.07 59.17 46.42 55.74
	H	15.07 27.39 15.33 25.84
	CE	48.31 56.87 45.27 53.94
	SSL	50.39 58.78 47.33 55.85
	PT	47.29 56.05 44.32 53.20
	25 30 35 40 45 50 55 60 65 70 75 80 85 90 95

Table 4 :

 4 Training time for 1 episode on a M1-CPU.

	Loss CPU Time
	CE	0.45s
	FR	0.83s
	H	0.75s
	I	0.83s
	PT	0.01s
	SSL	0.80s

Table 6 :

 6 . Global Results for multilingual Amazon

		fr	de	en	es
	FR	29.36 33.98 53.89 28.47
	I	27.74 31.41 51.75 26.79
	H	15.04 15.13 15.04 15.04
	CE	27.15 30.24 50.89 26.21
	PT	26.37 29.16 50.34 25.44
	SSL 27.20 30.29 50.94 26.26

The cost of API queries is determined by the number of input tokens that are transmitted.

Furthermore, as the number of considered classes increases, the fixed size of the transformer limits the number of possible shots that can be fed to the model. Previous studies have often neglected this limitation by focusing on a few numbers of labels.

λ is set to 1 in all the experiments.

Additional multilingual results (i.e., on es, de, fr) can be found on Sec. B.3. They exhibit the same behavior.

Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grants 2022-AD01101838, 2023-103256 and 2023-101838).

Antreas Antoniou, Harrison Edwards, and Amos Storkey. 2018. How to train your maml. arXiv preprint arXiv:1810.09502. Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020. TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification. In Proceedings of Findings of EMNLP. Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. 2021. Few-shot segmentation without meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13979-13988. Malik Boudiaf, Imtiaz Ziko, Jérôme Rony, José Dolz, Pablo Piantanida, and Ismail Ayed. 2020. Information maximization for few-shot learning. Advances in Neural Information Processing Systems, 33:2445-2457. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901. J-F Cardoso. 1997. Infomax and maximum likelihood for blind source separation. IEEE Signal processing letters, 4(4):112-114. Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. Emile Chapuis, Pierre Colombo, Matteo Manica, Matthieu Labeau, and Chloé Clavel. 2020. Hierarchical pre-training for sequence labelling in spoken dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2636-2648, Online. Association for Computational Linguistics. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021a. Evaluating large language models trained on code. arXiv preprint

Using the identity given by ( 19) in expression ( 18), and setting β = 1/2, we obtain the following lower bound on I(X; Y ):

-E X ⋆ log E X cos d FR (P (y|X), P (y|X ⋆ )) 2

The inequality (6) immediately follows by replacing the distribution of the r.v. X with the empirical distribution on the query and P (y|x) with the soft-prediction corresponding to the feature x, which concludes the proof of the proposition.


Malik Boudiaf
email: ma-lik.boudiaf.1@etsmtl.net

Éts Montreal

Etienne Bennequin
email: etienneb@sicara.com

Myriam Tami Mics

Antoine Toubhans Sicara

Pablo Piantanida

Celine Hudelot

Ismail Ben

Ayed Éts Montreal

Open-Set Likelihood Maximization for Few-Shot Learning

We tackle the Few-Shot Open-Set Recognition (FSOSR) problem, i.e. classifying instances among a set of classes for which we only have a few labeled samples, while simultaneously detecting instances that do not belong to any known class. We explore the popular transductive setting, which leverages the unlabelled query instances at inference. Motivated by the observation that existing transductive methods perform poorly in open-set scenarios, we propose a generalization of the maximum likelihood principle, in which latent scores down-weighing the influence of potential outliers are introduced alongside the usual parametric model. Our formulation embeds supervision constraints from the support set and additional penalties discouraging overconfident predictions on the query set. We proceed with a blockcoordinate descent, with the latent scores and parametric model co-optimized alternately, thereby benefiting from each other. We call our resulting formulation Open-Set Likelihood Optimization (OSLO). OSLO is interpretable and fully modular; it can be applied on top of any pre-trained model seamlessly. Through extensive experiments, we show that our method surpasses existing inductive and transductive methods on both aspects of open-set recognition, namely inlier classification and outlier detection. Code is available at https://github.com/ebennequin/fewshot-open-set.

Introduction

Few-shot classification consists in recognizing concepts for which we have only a handful of labeled examples. These form the support set, which, together with a batch of unlabeled instances (the query set), constitute a few-shot task.

Most few-shot methods classify the unlabeled query samples of a given task based on their similarity to the support instances in some feature space [START_REF] Snell | Prototypical networks for few-shot learning[END_REF]. This implicitly assumes a closed-set setting for each task, i.e. query instances are supposed to be constrained to the set of classes explicitly defined by the support set. However, the real world is open and this closed-set assumption may not hold in practice, especially for limited support sets. Whether they are unexpected items circulating on an assembly line, a new dress not yet included in a marketplace's catalog, or a previously undiscovered species of fungi, open-set instances occur everywhere. When they do, a closed-set classifier will falsely label them as the closest known class.

This drove the research community toward open-set recognition i.e. recognizing instances with the awareness that they may belong to unknown classes. In large-scale settings, the literature abounds of methods designed specifically to detect open-set instances while maintaining good accuracy on closed-set instances [START_REF] Bendale | Towards open set deep networks[END_REF][START_REF] Walter | Toward open set recognition[END_REF][START_REF] Zhou | Learning placeholders for open-set recognition[END_REF]. Very recently, the authors of [START_REF] Liu | Few-shot open-set recognition using meta-learning[END_REF] introduced a Few-Shot Open-Set Recognition (FSOSR) setting, in which query instances may not belong to any known class. The study in [START_REF] Liu | Few-shot open-set recognition using meta-learning[END_REF], together with other recent follow-up works [START_REF] Huang | Task-adaptive negative envision for few-shot open-set recognition[END_REF][START_REF] Jeong | Few-shot open-set recognition by transformation consistency[END_REF], exposed FSOSR to be a difficult task.

To help alleviate the scarcity of labeled data, transduction [START_REF] Vapnik | The nature of statistical learning theory[END_REF] was recently explored for few-shot classification [START_REF] Liu | Learning to propagate labels: Transductive propagation network for few-shot learning[END_REF], and has since become a prominent research direction, fueling a large body of works, e.g. [START_REF] Boudiaf | Few-shot segmentation without meta-learning: A good transductive inference is all you need?[END_REF][START_REF] Boudiaf | Transductive information maximization for few-shot learning[END_REF][START_REF] Singh Dhillon | A baseline for few-shot image classification[END_REF][START_REF] Hu | Leveraging the feature distribution in transfer-based few-shot learning[END_REF][START_REF] Liu | Prototype rectification for few-shot learning[END_REF][START_REF] Martin | Towards practical few-shot query sets: Transductive minimum description length inference[END_REF][START_REF] Veilleux | Realistic evaluation of transductive few-shot learning[END_REF][START_REF] Wang | Instance credibility inference for few-shot learning[END_REF][START_REF] Ziko | Laplacian regularized few-shot learning[END_REF], among many others. By leveraging the statistics of the query set, transductive methods yield performances that are substantially better than their inductive counterparts [START_REF] Boudiaf | Transductive information maximization for few-shot learning[END_REF][START_REF] Veilleux | Realistic evaluation of transductive few-shot learning[END_REF] in the standard closed-set setting.

In this work, we seek to explore transduction for the FSOSR setting. We argue that theoretically, transduction has the potential to enable both classification and outlier detection (OD) modules to act symbiotically. Indeed, the classification module can reveal valuable structure of the This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.

Except for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. inlier's marginal distribution that the OD module seeks to estimate, such as the number of modes or conditional distributions, while the OD part indicates the "usability" of each unlabelled sample. However, transductive principles currently adopted for few-shot learning heavily rely on the closed-set assumption in the unlabelled data, leading them to match the classification confidence for open-set instances with that of closed-set instances. In the presence of outliers, this not only harms their predictive performance on closed-set instances, but also makes prediction-based outlier detection substantially harder than with simple inductive baselines. Contributions. In this work, we aim at designing a principled framework that reconciles transduction with the open nature of the FSOSR problem. Our idea is simple but powerful: instead of finding heuristics to assess the outlierness of each unlabelled query sample, we treat this score as a latent variable of the problem. Based on this idea, we propose a generalization of the maximum likelihood principle, in which the introduced latent scores weigh potential outliers down, thereby preventing the parametric model from fitting those samples. Our generalization embeds additional supervision constraints from the support set and penalties discouraging overconfident predictions. We proceed with a block-coordinate descent optimization of our objective, with the closed-set soft assignments, outlierness scores, and parametric models co-optimized alternately, thereby benefiting from each other. We call our resulting formulation Open-Set Likelihood Optimization (OSLO). OSLO provides highly interpretable and closed-form solutions within each iteration for both the soft assignments, outlierness variables, and the parametric model. Additionally, OSLO is fully modular; it can be applied on top of any pre-trained model seamlessly.

Empirically, we show that OSLO significantly surpasses its inductive and transductive competitors alike for both outlier detection and closed-set prediction. Applied on a wide variety of architectures and training strategies and without any re-optimization of its parameters, OSLO's improvement over a strong baseline remains large and consistent. This modularity allows our method to fully benefit from the latest advances in standard image recognition. Before diving into the core content, let us summarize our contributions: 1. To the best of our knowledge, we realize the first study and benchmarking of transductive methods for the Few-Shot Open-Set Recognition setting. We reproduce and benchmark five state-of-the-art transductive methods.

2. We introduce Open-Set Likelihood Optimization (OSLO), a principled extension of the Maximum Likelihood framework that explicitly models and handles the presence of outliers. OSLO is interpretable and modular i.e. can be applied on top of any pre-trained model seamlessly.

Related Works

Few-shot classification (FSC) methods. Many FSC works involve episodic training [START_REF] Vinyals | Matching networks for one shot learning[END_REF], in which a neural network acting as a feature extractor is trained on artificial tasks sampled from the training set. This replication of the inference scenario during training is intended to make the learned representation more robust to new classes. However, several recent works have shown that simple fine-tuning baselines are competitive in comparison to sophisticated episodic methods, e.g. [START_REF] Chen | A closer look at few-shot classification[END_REF][START_REF] Goldblum | Unraveling metalearning: Understanding feature representations for few-shot tasks[END_REF], motivating a new direction of few-shot learning research towards the development of model-agnostic methods that do not involve any specific training strategy [START_REF] Singh Dhillon | A baseline for few-shot image classification[END_REF].

Transductive FSC. Transductive FSC methods leverage statistics of the query set as unlabeled data to improve performance, through model fine-tuning [START_REF] Singh Dhillon | A baseline for few-shot image classification[END_REF], Laplacian regularization [START_REF] Ziko | Laplacian regularized few-shot learning[END_REF], clustering [START_REF] Lichtenstein | Tafssl: Task-adaptive feature sub-space learning for few-shot classification[END_REF], mutual information maximization [START_REF] Boudiaf | Transductive information maximization for few-shot learning[END_REF][START_REF] Veilleux | Realistic evaluation of transductive few-shot learning[END_REF], prototype rectification [START_REF] Liu | Prototype rectification for few-shot learning[END_REF], or optimal transport [START_REF] Bennequin | Bridging few-shot learning and adaptation: New challenges of support-query shift[END_REF][START_REF] Hu | Leveraging the feature distribution in transfer-based few-shot learning[END_REF][START_REF] Lazarou | Iterative label cleaning for transductive and semi-supervised few-shot learning[END_REF], among other transduction strategies. The idea of maximizing the likelihood of both support and query samples under a model parameterized by class prototypes is proposed by [START_REF] Yang | Prototype mixture models for few-shot semantic segmentation[END_REF] for few-shot segmentation. However, their method relies on the closed-set assumption. Differing from previous works, our framework leverages an additional latent variable, the inlierness score.

Open-set recognition (OSR). OSR aims to enable classifiers to detect instances from unknown classes [START_REF] Walter | Toward open set recognition[END_REF]. Prior works address this problem in the large-scale setting by augmenting the SoftMax activation to account for the possibility of unseen classes [START_REF] Bendale | Towards open set deep networks[END_REF], generating artificial outliers [START_REF] Ge | Generative openmax for multi-class open set classification[END_REF][START_REF] Neal | Open set learning with counterfactual images[END_REF], improving closed-set accuracy [START_REF] Vaze | Open-set recognition: A good closed-set classifier is all you need[END_REF], or using placeholders to anticipate novel classes' distributions with adaptive decision boundaries [START_REF] Zhou | Learning placeholders for open-set recognition[END_REF]. All these methods involve the training of deep neural networks on a specific class set. Therefore, they are not fully fit for the few-shot setting. In this work, we use simple yet effective adaptations of OpenMax [START_REF] Bendale | Towards open set deep networks[END_REF] and PROSER [START_REF] Zhou | Learning placeholders for open-set recognition[END_REF] as strong baselines for FSOSR.

Few-shot open-set recognition. In the few-shot setting, methods must detect open-set instances while only a few closed-set instances are available. [START_REF] Liu | Few-shot open-set recognition using meta-learning[END_REF] use meta-learning on pseudo-open-set tasks to train a model to maximize the classification entropy of open-set instances. [START_REF] Jeong | Few-shot open-set recognition by transformation consistency[END_REF] use transformation consistency to measure the divergence between a query image and the set of class prototypes. [START_REF] Huang | Task-adaptive negative envision for few-shot open-set recognition[END_REF] use an attention mechanism to generate a negative prototype for outliers. These methods require the optimization of a separate model with a specific episodic training strategy. Nonetheless, as we show in sec:experiments, they bring marginal improvement over simple adaptations of standard OSR methods to the few-shot setting. In comparison, our method doesn't require any specific training and can be plugged into any feature extractor without further optimization.

Few-Shot Open-Set Recognition

Model training. Let us denote the raw image space X . As per the standard Few-Shot setting, we assume access to a base dataset D base = {(x i , y i )} i=1...|Dbase| with base classes C base , such that x i ∈ X and y i ∈ C base . We use D base to train a feature extractor ϕ θ . Our method developed later in sec:open s et l ikelihood, f reezesϕ θ and performs inference directly on top of the extracted features for each task.

Testing.

Given a set of novel classes C novel disjoint from base classes i.e. C novel ∩ C base = ∅, a K-way FSOSR task is formed by sampling a set of

K closed- set classes C CS ⊂ C novel , a support set of labeled instances S = {(x i , y i ) ∈ X × C CS } |S| i=1 and a query set Q = {x i ∈ X } |S|+|Q| i=|S|+1 .
In the standard few-shot setting, the unknown ground-truth query labels {y i } |S|+|Q| i=|S|+1 are assumed to be restricted to closed-set classes i.e. ∀i, y i ∈ C CS . In FSOSR, however, query labels may also belong to an additional set

C OS ⊂ C novel of open-set classes i.e. ∀ i > |S|, y i ∈ C CS ∪ C OS with C CS ∩ C OS = ∅.
For easy referencing, we refer to query samples from the closed-set classes C CS as inliers and to query samples from open-set classes C OS as outliers. For each query image x i , the goal of FSOSR is to simultaneously assign a closed-set prediction and an outlierness (or equivalently inlierness) score.

Transductive FSOSR. As a growing part of the Few-Shot literature, Transductive Few-Shot Learning assumes that unlabelled samples from the query set are observed at once, such that the structure of unlabelled data can be leveraged to help constrain ambiguous few-shot tasks. Transductive methods have achieved impressive improvements over inductive methods in standard closed-set FSC [START_REF] Boudiaf | Transductive information maximization for few-shot learning[END_REF][START_REF] Singh Dhillon | A baseline for few-shot image classification[END_REF][START_REF] Hu | Leveraging the feature distribution in transfer-based few-shot learning[END_REF][START_REF] Ziko | Laplacian regularized few-shot learning[END_REF]. We expect that transductive methods can help us improve overall open-set performance. While we find this to generally hold for closed-set predictive performance, we empirically show in sec:experiments that accuracy gains systematically come along significant outlier detection degradation, indicating that transductive methods are not equipped to handle openset recognition. In the following, we take up the challenge of designing a transductive optimization framework that leverages the presence of outliers to improve its performance.

= " > A A A C E X i c b V D L S s N A F J 3 4 r P U V d e l m s A g V p C R S V H B T d O O y g n 1 A E 8 J k M m m H T B 7 M T M Q S 8 g t u / B U 3 L h R x 6 8 6 d f + O k z a K 2 H h j m c M 6 9 3 H u P m z A q p G H 8 a E v L K 6 t r 6 5 W N 6 u b W 9 s 6 u v r f f F X H K M e n g m M W 8 7 y J B G I 1 I R 1 L J S D / h B I U u I z 0 3 u C n 8 3 g P h g s b R v R w n x A 7 R M K I + x U g q y d H r S d 1 y Y + a J c a i + 7 D E / h c E V n J W s M M 2 d 4 M T R a 0 b D m A A u E r M k N V C i 7 e j f l h f j N C S R x A w J M T C N R N o Z 4 p J i R v K q l Q q S I B y g I R k o G q G Q C D u b X J T D Y 6 V 4 0 I + 5 e p G E E 3 W 2 I 0 O h K P Z T l S G S I z H v F e J / 3 i C V / q W d 0 S h J J Y n w d J C f M i h j W M Q D P c o J l m y s C M K c q l 0 h H i G O s F Q h V l U I 5 v z J i 6 R 7 1 j D P G 8 2 7 Z q 1 1 X c Z R A Y f g C N S B C S 5 A C 9 y C N u g A D J 7 A C 3 g D 7 9 q z 9 q p 9 a J / T 0 i W t 7 D k A f 6 B 9 / Q L Z 3 Z 2 t < / l a t e x i t > p(x, k; µ k ) < l a t e x i t s h a 1 _ b a s e 6 4 = " b m Y o 6 S t m x c T T B U H J D 1 c U B r 2 p F I o = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l E 1 G P R i 8 e K 9 g P a U D b b S b t 0 s w m 7 G 7 G E / g Q v H h T x 6 i / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S A T X x n W / n c L K 6 t r 6 R n G z t L W 9 s 7 t X 3 j 9 o 6 j h V D B s s F r F q B 1 S j 4 B I b h h u B 7 U Q h j Q K B r W B 0 M / V b j 6 g 0 j + W D G S f o R 3 Q g e c g Z N V a 6 7 z 7 x X r n i V t 0 Z y D L x c l K B H P V e + a v b j 1 k a o T R M U K 0 7 n p s Y P 6 P K c C Z w U u q m G h P K R n S A H U s l j V D 7 2 e z U C T m x S p + E s b I l D Z m p v y c y G m k 9 j g L b G V E z 1 I v e V P z P 6 6 Q m v P I z L p P U o G T z R W E q i I n J 9 G / S 5 w q Z E W N L K F P c 3 k r Y k C r K j E

Open-Set Likelihood

In this section, we introduce OSLO, a novel extension of the standard likelihood designed for transductive FSOSR. Unlike existing transductive methods, OLSO explicitly models and handles the potential presence of outliers, which allows it to outperform inductive baselines on both aspects of the open-set scenario.

Observed variables. We start by establishing the observed variables of the problem. As per the traditional setting, we observe images from the support set {x i } |S| i=1 and their associated labels {y i } |S| i=1 . The transductive setting also allows us to observe images from the query set. For notation convenience, we concatenate all images in X = {x i } |S|+|Q| i=1 . Latent variables. Our goal is to predict the class of each sample in the query set Q, as well as their inlierness, i.e. the model's belief in a sample being an inlier or not. This naturally leads us to consider latent class assignments z i ∈ ∆ K describing the membership of sample i to each closed-set class, with

∆ K = {z ∈ [0, 1] K : z T 1 = 1} the K-dimensional simplex.
Additionally, we consider latent inlierness scores ξ i ∈ [0, 1] close to 1 if the model considers sample i as an inlier. For notation convenience, we consider latent assignments and inlierness scores for all samples, including those from the support, and group everything in

Z = {z i } |S|+|Q| i=1 and ξ = {ξ i } |S|+|Q| i=1
. Note that support samples are inliers, and we know their class. Therefore ∀i ≤ |S|, the constraints z i = y i and ξ i = 1 will be imposed, where y i is the one-hot encoded version of y i Parametric model. The final ingredient we need to formulate is a parametric joint model over observed features and assignments. Following standard practice, we model the joint distribution as a balanced mixture of standard Gaussian distri-butions, parameterized by the centroids µ = {µ 1 , . . . , µ K }:

p(x, k; µ) = p(k)p(x|k) ∝ exp(- ∥ϕ θ (x) -µ k ∥ 2 2 ) (1)
As mentioned in sec:fsosr s etting, thef eatureextractor ′ sparametersθ are kept frozen, and only µ will be optimized.

Objective. Using the i.i.d. assumption, we start by writing the standard likelihood objective:

p(X, Z; µ) = |S|+|Q| i=1 K k=1 p(x i , k; µ) z ik (2)
Without loss of generality, we consider the log-likelihood:

log(p(X, Z; µ)) = |S|+|Q| i=1 K k=1 z ik log(p(x i , k; µ)) (3)
Eq. ( 3) tries to enforce a high likelihood of all samples under our parametric model p. This becomes sub-optimal in the presence of outliers, which should ideally be disregarded. fig:likelihood illustrates this phenomenon on a toy 2D drawing. To downplay this issue, we introduce Open-Set Likelihood Optimization (OSLO), a generalization of the standard likelihood framework, which leverages latent inlierness scores to weigh samples:

L O (X, Z, ξ; µ) = |S|+|Q| i=1 ξ i K k=1 z ik log (p(x i , k; µ)) (4)
Eq (4) can be interpreted as follows: samples believed to be inliers i.e. ξ i ≈ 1 will be required to have high likelihood under our model p, whereas outliers won't. Note that ξ is treated as a variable of optimization, and is co-optimized alongside µ and Z. Finally, to prevent overconfident latent scores, we consider a penalty term on both Z and ξ:

L soft = |S|+|Q| i=|S|+1 λ z H(z i ) + λ ξ H(ξ i ) (5)
where

ξ i = [1 -ξ i , ξ i ],
and H(p) = -p ⊤ log(p) denotes the entropy, which encourages smoother assignments.

Optimization. We are now ready to formulate OSLO's optimization problem: 6) is strictly convex with respect to each variable when the other variables are fixed. Therefore, we proceed with a block-coordinate ascent, which alternates three iterative steps, each corresponding to a closed-form solution for one of the variables. Proposition 1. OSLO's optimization problem (6) can be minimized by alternating the following updates, with σ denoting the sigmoid operation:

max µ,Z,ξ L O (Z, ξ, µ) + L soft (Z, ξ) s.t z i ∈ ∆ K , ξ i ∈ [0, 1] ∀ i (6) z i = y i , ξ i = 1, i ≤ |S| Problem (
ξ (t+1) i =      1 if i ≤ |S| σ 1 λ ξ K k=1 z (t) ik log p(x i , k; µ (t) )) else z (t+1) i ∝      y i if i ≤ |S| exp ξ (t+1) i λ z log p(x i , • ; µ (t) ) else µ (t+1) k = 1 |S|+|Q| i=1 ξ (t+1) i z (t+1) ik |S|+|Q| i=1 ξ (t+1) i z (t+1) ik ϕ θ (x i )
The proof of proposition 1 is performed by derivation of L O (Z, ξ, µ)+L soft (Z, ξ) and deferred to the supplementary material. The optimal solution for the inlierness score ξ i appears very intuitive, and essentially conveys that samples with high likelihood under the current parametric model should be considered inliers. We emphasize that beyond providing a principled outlierness score, as 1ξ i , the presence of ξ i allows to refine and improve the closed-set parametric model. In particular, ξ i acts as a sample-wise temperature in the update of z i , encouraging outliers (ξ i ≈ 0) to have a uniform distribution over closed-set classes. Additionally, those samples contribute less to the update of closed-set prototypes µ, as each sample's contribution is weighted by ξ i .

Experiments

Experimental setup

Baselines. One goal of this work is to fairly evaluate different strategies to address the FSOSR problem. In particular, we benchmark 4 families of methods: (i) popular Outlier Detection methods, e.g. Nearest-Neighbor [START_REF] Ramaswamy | Efficient algorithms for mining outliers from large data sets[END_REF], (ii) Inductive Few-Shot classifiers, e.g. SimpleShot [START_REF] Wang | Simpleshot: Revisiting nearest-neighbor classification for few-shot learning[END_REF] (iii) Inductive Open-Set methods formed by standard methods such as OpenMax [START_REF] Bendale | Towards open set deep networks[END_REF] and Few-Shot methods such as Snatcher [START_REF] Jeong | Few-shot open-set recognition by transformation consistency[END_REF] (iv) Transductive classifiers, e.g. TIM [START_REF] Boudiaf | Transductive information maximization for few-shot learning[END_REF], that implicitly rely on the closed-set assumption, and finally (v) Transductive Open-Set introduced in this work through OSLO. Following [START_REF] Jeong | Few-shot open-set recognition by transformation consistency[END_REF], closed-set few-shot classifiers are turned into open-set classifiers by considering the negative of the maximum probability as a measure of outlierness. Furthermore, we found that applying a centernormalize transformation ψ υ : x → (xυ)/||x -υ|| 2 on the features extracted by ϕ θ benefited all methods. Therefore, we apply it to the features before applying any method, using an inductive Base centering [START_REF] Wang | Simpleshot: Revisiting nearest-neighbor classification for few-shot learning[END_REF] for inductive methods υ Base = Table 1. Standard Benchmarking. Evaluating different families of methods on the FSOSR problem on mini-ImageNet and tiered-ImageNet using a ResNet-12. For each column, a light-gray standard deviation is indicated, corresponding to the maximum deviation observed across methods for that metric. Best methods are shown in bold. Results marked with ⋆ are reported from their original paper.

mini-ImageNet

Strategy Method 1-shot 5-shot Acc AUROC AUPR Prec@0.9 Acc AUROC AUPR Prec@0.9

AUPR

Strong baseline OpenMax (+-0.9) SnatcherF (+-0.18) OSLO (+4.67) TIM (+-4.97) Figure 2. OSLO improves open-set performances on a wide variety of tasks. Relative 1-shot performance of the best methods of each family w.r.t the Strong baseline using a ResNet-12, across a set of 5 scenarios, including 3 with domain-shift. Each vertex represents one scenario, e.g. tiered→Fungi (x) means the feature extractor was pre-trained on tiered-ImageNet, test tasks are sampled from Fungi, and the Strong Baseline performance is x. For each method, the average relative improvement across the 5 scenarios is reported in parenthesis in the legend. The same charts are provided in the supplementary materials for the 5-shot setting and using a WideResNet backbone.

Task centering [START_REF] Hu | Leveraging the feature distribution in transfer-based few-shot learning[END_REF] υ T ask = 1 |S∪Q| x∈S∪Q ϕ θ (x) for all transductive methods. Since features are normalized, we empirically found it beneficial to re-normalize centroids µ k ← µ k /||µ k || 2 after each update from Prop. 1, which we show in the Appendix remains a valid minimizer of Eq. ( 6) when adding the constraint ||µ k || 2 = 1. Hyperparameters. For all methods, we define a grid over salient hyper-parameters and tune over the validation split of mini-ImageNet. To avoid cumbersome per-dataset tuning, and evaluate the generalizability of methods, we then keep hyper-parameters fixed across all other experiments. Architectures and checkpoints. To provide the fairest comparison, all non-episodic methods are tuned and tested using off-the-shelf pre-trained checkpoints. All results except fig:barplots are produced using the pre-trained ResNet-12 and Wide-ResNet 28-10 checkpoints provided by the authors from [START_REF] Ye | Fewshot learning via embedding adaptation with set-to-set functions[END_REF]. As for episodically-finetuned models required by Snatcher [START_REF] Jeong | Few-shot open-set recognition by transformation consistency[END_REF] and FEAT [START_REF] Ye | Fewshot learning via embedding adaptation with set-to-set functions[END_REF], checkpoints are obtained from the authors' respective repositories. Finally, to challenge the model-agnosticity of our method, we resort to an additional set of 10 ImageNet pre-trained models covering three distinct architectures: ResNet-50 [START_REF] He | Deep residual learning for image recognition[END_REF] for CNNs, ViT-B/16 [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] for vision transformers, and Mixer-B/16 [START_REF] Ilya O Tolstikhin | Mlpmixer: An all-mlp architecture for vision[END_REF] for MLP-Mixer. These models are taken from the excellent TIMM library [START_REF] Wightman | Pytorch image models[END_REF]. Datasets and tasks. We experiment with a total of 5 vision datasets. As standard FSC benchmarks, we use the mini-ImageNet [START_REF] Vinyals | Matching networks for one shot learning[END_REF] dataset with 100 classes and the larger tiered-ImageNet [START_REF] Ren | Meta-learning for semi-supervised fewshot classification[END_REF] dataset with 608 classes. We also experiment on more challenging cross-domain tasks formed by using 3 finer-grained datasets: the Caltech-UCSD Birds 200 [START_REF] Welinder | [END_REF] (CUB) dataset, with 200 classes, the FGVC-Aircraft dataset [START_REF] Maji | Fine-grained visual classification of aircraft[END_REF] with 100 classes, and the Fungi classification challenge [START_REF] Schroeder | Fgvcx fungi classification challenge[END_REF] open-set classes. We follow this setting for a fair comparison with previous works [START_REF] Jeong | Few-shot open-set recognition by transformation consistency[END_REF][START_REF] Liu | Few-shot open-set recognition using meta-learning[END_REF] which sample open-set query instances from only 5 classes. We also report results in supplementary materials for a more general setting in which open-set query instances are sampled indifferently from all remaining classes in the test set.

Results

Simplest inductive methods are competitive. The first surprising result comes from analyzing the performances of standard OOD detectors on the FSOSR problem. Fig. 1 shows that k-NN and PCA outperform, by far, arguably more advanced methods that are OCVSM and Isolation Forest. This result contrasts with standard high-dimensional benchmarks [START_REF] Zhao | Pyod: A python toolbox for scalable outlier detection[END_REF] where k-NN falls typically short of the latter, indicating that the very difficult challenge posed by FSOSR may lead advanced methods to overfit. In fact, Fig. 2 shows that across 5 scenarios, the combination SimpleShot [START_REF] Wang | Simpleshot: Revisiting nearest-neighbor classification for few-shot learning[END_REF]+ k-NN [START_REF] Ramaswamy | Efficient algorithms for mining outliers from large data sets[END_REF] formed by the simplest FS-inductive classifier and the simplest inductive OOD detector is a strong baseline that outperforms all specialized open-set methods. We refer to this combination as Strong baseline in Figures 2 and4. Additional results for the Wide-ResNet architecture are provided in the supplementary material. Transductive methods still improve accuracy but degrade outlier detection. As shown in Table 1, most transductive classifiers still offer a significant boost in closed-set accuracy, even in the presence of outliers in the query set. Note that this contrasts with findings from the semi-supervised literature, where standard methods drop below the baseline in the presence of even a small fraction of outliers [START_REF] Chen | Semisupervised learning under class distribution mismatch[END_REF][START_REF] Killamsetty | Retrieve: Coreset selection for efficient and robust semi-supervised learning[END_REF][START_REF] Saito | Openmatch: Open-set semi-supervised learning with open-set consistency regularization[END_REF][START_REF] Yu | Multitask curriculum framework for open-set semi-supervised learning[END_REF]. We hypothesize that the deliberate under-parametrization of few-shot methods -typically only training a linear classifier-, required to avoid overfitting the support set, partly explains such robustness. However, transductive methods still largely underperform in outlier detection, with AUROCs as low as 52 % (50% being a random detector) for LaplacianShot.

Note that the outlierness score for these methods is based on the negative of the maximum probability, therefore this result can be interpreted as transductive methods having artificially matched the prediction confidence for outliers with the confidence for inliers.

OSLO achieves the best trade-off. Benchmark results in Fig. 1 show that OSLO surpasses the best transductive methods in terms of closed-set accuracy, while consistently outperforming existing out-of-distribution and open-set detection competitors on outlier detection ability. Interestingly, while the gap between closed-set accuracy of transductive methods and inductive ones typically contracts with more shots, the outlier detection performance of OSLO remains largely superior to its inductive competitors even in the 5shot scenario, where a consistent 3-6% gap in AUROC and AUPR with the second-best method can be observed. We accumulate further evidence of OSLO's superiority by introducing 3 additional cross-domain scenarios in Fig. 2, corresponding to a base model pre-trained on tiered-ImageNet, but tested on CUB, Aircraft, and Fungi datasets. In such challenging scenarios, where both feature and class distributions shift, OSLO remains competitive in closed-set accuracy and largely outperforms other methods in outlier detection.

OSLO benefits from more query samples. A critical question for transductive methods is the dependency of their performance on the size of the query set. Intuitively, a larger query set will provide more unlabeled data and thus lead to better results. We exhibit this relation in fig:variate-query- [START_REF] Ilya O Tolstikhin | Mlpmixer: An all-mlp architecture for vision[END_REF]. For each architecture, we include different types of pre-training, including Supervised (Sup.), Semi-Supervised, Semi-Weakly Supervised (SW Sup.) [START_REF] Zeki Yalniz | Billion-scale semi-supervised learning for image classification[END_REF], DINO [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF], SAM [START_REF] Chen | When vision transformers outperform resnets without pre-training or strong data augmentations[END_REF], MIIL [START_REF] Ridnik | Imagenet-21k pretraining for the masses[END_REF].

Improvements over the baseline are consistently significant and generally higher than those observed with the ResNet-12 in fig:spidercharts.

scenario on the * → Fungi benchmark. Without any tuning, OSLO remains able to leverage the strong expressive power of large-scale models, and even consistently widens the gap with the strong baseline, achieving a remarkable performance of 79% accuracy and 81% AUROC with the ViT-B/16. This set of results testifies to how easy obtaining highly competitive results on difficult specialized tasks can be by combining OSLO with the latest models.

Ablation study. We perform an ablation study on the important ingredients of OSLO. As a core contribution of our work, we show in Tab. 2 that the presence and optimization of the latent inlierness scores is crucial. In particular, the closed-form latent score ξ yields strong outlier recognition performance, even at initialization (i.e. after the very first update from Prop. 1). Interestingly, refining the parametric model without accounting for ξ in Z and µ's updates (i.e. standard likelihood) allows the model to fit those outliers, leading to significantly worse outlier detection, from 71.76% to 64.97%. On the other hand, accounting for ξ, as proposed in OSLO, improves the outlier detection by more than 3% over the initial state, and closed-set accuracy by more than 5%. In the end, in a fully apples-to-apples comparison, OSLO outperforms its standard likelihood counterpart by more than 2% in accuracy and 10% in outlier detection.

We strengthen this ablation study in the Appendix.

Discussion

Limitations. Unlike inductive methods, transductive methods are inevitably affected by the amount of unlabelled data provided, which in real-world scenarios cannot necessarily be controlled. OSLO is no exception, and fewer query samples tend to decrease its performance. In the extreme case with only 1 sample per class, OSLO's performance comes close to our inductive baseline. In those scenarios where unlabelled data is particularly scarce, the benefits brought by transduction remain therefore limited. As a second limitation, poorer representations appear to diminish OSLO's competitive advantage in closed-set accuracy. In particular, OSLO's closed-set accuracy stands more than 6% above the baseline on the tiered → tiered scenario but reduces to 2% in the most challenging-domain scenario tiered → Aircraft. Fig. 4 further corroborates this hypothesis, with OSLO's accuracy outperforming the baseline's by 9% with the best transformer, but by only 2.7% on the least performing model.

Conclusion.

We presented OSLO, the first transductive method for FSOSR. OSLO extends the vanilla maximum likelihood objective in two important ways, First, it accounts for the constraints imposed by the provided supervision. More importantly, it explicitly models the potential presence of outliers in its very latent model, allowing it to co-learn the optimal closed-set model and outlier assignments. Beyond FSOSR, we believe OSLO presents a general, conceptually simple, and completely modular formulation to leverage unlabelled data in the potential presence of outliers. That, of course, naturally extends to other classification settings, such as large-scale open-set detection, but to other tasks as well, such as segmentation in which background pixels could be viewed as outliers with respect to closed-set classes. We hope OSLO inspires further work in that direction.

  t e x i t s h a 1 _ b a s e 6 4 = " 3 6 2 E u c A l P T b 7 n J l 8 Z S j o t m e 9 U s w

Figure 1 .

 1 Figure 1. Intuition behind OSLO. Standard transductive likelihood (left) tries to enforce high likelihood for all points, including outliers. OSLO (right) instead treats the outlierness of each sample as a latent variable to be solved alongside the parametric model. Besides yielding a principled outlierness score for open-set detection, it also allows the fitted parametric model to effectively disregard samples deemed outliers, and therefore provide a better approximation of underlying class-conditional distributions.

  with 1394 classes. Following standard FSOSR protocol, support sets contain |C CS | = 5 closed-set classes with 1 or 5 instances, or shots, per class, and query sets are formed by sampling 15 instances per class, from a total of ten classes: the five closed-set classes and an additional set of |C OS | = 5

Figure 3 .

 3 Figure 3. OSLO improves performance even with few queries. We study the closed-set (accuracy) and open-set (AUROC) performance of transductive methods depending on the size of the query set on tiered-ImageNet in the 1-shot and 5-shot settings. The total size |Q| of the query set is obtained by multiplying the number of queries per class NQ by the number of classes in the task (i.e. 5) and adding as many outlier queries e.g. NQ = 1 corresponds to 1 query per class and 5 open-set queries i.e. |Q| = 10. We add the inductive method k-NN + SimpleShot to compare with a method that is by nature independent of the number of queries. The results for mini-ImageNet are provided in the supplementary materials.

  3. Through extensive experiments spanning five datasetsand a dozen of pre-trained models, we show that OSLO consistently surpasses both inductive and existing transductive methods in detecting open-set instances while competing with the strongest transductive methods in classifying closed-set instances.

Table 2 .

 2 OSLO's ablation study along two factors described in sec:ablationstudy.Resultsareproducedonthe1shotscenarioonmini -ImageN et, withaResN et -12. OSLO's improvement is consistent across many architectures and training strategies. To evaluate model-agnosticity, we compare OSLO to the Strong baseline on challenging 1-shot Fungi tasks. We experiment across 3 largely distinct architectures: ResNet-50 (CNN)[START_REF] He | Deep residual learning for image recognition[END_REF], ViT-B/16 (Vision Transformer)[START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF], and Mixer-B/16 (MLP-Mixer)

	(i) Inlierness latent	Acc	AUROC
	Ignore (3)	69.42	64.97
	Leverage (4)	71.73	74.92
	(ii) Optimization steps	Acc	AUROC
	At initialization	66.63	71.76
	After optimization	71.73	74.92
	tiered by spanning the number of queries per class from
	1 to 30. We observe that the closed-set accuracy of most
	transductive methods is stable across this span in the 5-shot
	scenario. In the 1-shot scenario, OSLO gains from addi-
	tional queries but stays above the baseline even with a small
	number of queries. Interestingly enough, OSLO is the only
	transductive method to improve its outlier detection ability
	when the number of queries increases.
	OSLO steps toward model-agnosticity. We evaluate
	OSLO's model-agnosticity by its ability to maintain con-
	sistent improvement over the Strong Baseline, regardless
	of the model used, and without hyperparameter adjustment.
	In that regard, we depart from the standard ResNet-12 and
	cover 3 largely distinct architectures, each encoding differ-
	ent inductive biases. To further strengthen our empirical
	demonstration of OSLO's model-agnosticity, for each ar-
	chitecture, we consider several training strategies spanning
	different paradigms -unsupervised, supervised, semi and
	semi-weakly supervised -and using different types of data
	-image, text-. Results in fig:barplots show the relative im-
	provement of OSLO w.r.t the strong baseline in the 1-shot

|D base |x∈D base ϕ θ (x), and a transductive