Mathieu Chalvidal
email: mathieu_chalvid@brown.edu

Thomas Serre
email: thomas_serre@brown.edu

Rufin Vanrullen
email: rufin.vanrullen@cnrs.fr

Learning Functional Transduction

Keywords:

Research in machine learning has polarized into two general approaches for regression tasks: Transductive methods construct estimates directly from available data but are usually problem unspecific. Inductive methods can be much more specific but generally require compute-intensive solution searches. In this work, we propose a hybrid approach and show that transductive regression principles can be meta-learned through gradient descent to form efficient in-context neural approximators by leveraging the theory of vector-valued Reproducing Kernel Banach Spaces (RKBS). We apply this approach to function spaces defined over finite and infinite-dimensional spaces (function-valued operators) and show that once trained, the Transducer can almost instantaneously capture an infinity of functional relationships given a few pairs of input and output examples and return new image estimates. We demonstrate the benefit of our meta-learned transductive approach to model complex physical systems influenced by varying external factors with little data at a fraction of the usual deep learning training computational cost for partial differential equations and climate modeling applications.

Introduction

Transduction vs. induction ˛In statistical learning, transductive inference [START_REF] Vapnik | Estimation of dependences based on empirical data[END_REF] refers to the process of reasoning directly from observed (training) cases to new (testing) cases and contrasts with inductive inference, which amounts to extracting general rules from observed training cases to produce estimates. The former principle powers some of the most successful regression algorithms, from k-Nearest Neighbors [START_REF] Cover | Nearest neighbor pattern classification[END_REF] to Support Vector Machines [START_REF] Boser | A training algorithm for optimal margin classifiers[END_REF] or Gaussian Processes [START_REF] Williams | Gaussian processes for regression[END_REF]. A major advantage of such systems is their wide applicability and straightforward construction. In contrast, deep learning research has mostly endeavored to find inductive solutions by relying on the empirical evidence that stochastic gradient descent can faithfully encode functional relationships described by large datasets into the weights of a neural network. Although generic, inductive neural learning with gradient descent is compute-intensive, necessitates large amounts of data to approximate a single functional map, and poorly generalizes outside of the training distribution [START_REF] Jin | Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness[END_REF] such that a slight modification of the problem might require retraining and cause "catastrophic" forgetting of the previous solution [START_REF] Mccloskey | Catastrophic interference in connectionist networks: The sequential learning problem[END_REF]. This may be particularly problematic for real-world applications where data has heterogeneous sources, or only a few examples of the target function are available.

Meta-learning to regress functions ˛In this work, we meta-learn a regression program in the form of a neural network able to approximate instantaneously an infinity of functions defined on finite or infinite-dimensional spaces through a transductive formulation of the solution. Namely, our model is meta-trained to take as input any dataset D O of pairs pv i , Opv i qq iďI of some target function O together with a query element v 1 and produces directly an estimate of the image Opv 1 q. After meta-training, our network is able to perform regression of unseen operators O 1 from varying dataset sizes in a single feedforward pass, such that our model can be interpreted as performing in-context functional learning. In order to build such a model, we leverage the theory of Reproducing Kernel Banach Spaces (RKBS) [START_REF] Micchelli | A function representation for learning in banach spaces[END_REF][START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF][START_REF] Lin | On reproducing kernel banach spaces: Generic definitions and unified framework of constructions[END_REF] and take inspiration from the Transformer's [START_REF] Vaswani | Attention is all you need[END_REF] attention mechanism interpreted as a parametric vector-valued reproducing kernel. While kernel regression might be plagued by the "curse of dimensionality" [START_REF] Bellman | Dynamic programming[END_REF][START_REF] Aggarwal | On the surprising behavior of distance metrics in high dimensional space[END_REF], we show that our meta-learning approach can escape this pitfall, allowing, for instance, to perform instantaneous regressions over spaces of operators from a few example points, by building solutions to regression problem instances directly from the general reproducing kernel associated with such spaces.

Contributions ˛We introduce the Transducer, a novel meta-learning approach leveraging reproducing kernel theory and deep learning methods to perform instantaneous regression of an infinity of functions in reproducing kernel spaces.

• Our model learns an implicit regression program able to identify, in a single feedforward pass, elements of specific functional spaces from any corresponding collection of input-output pairs describing the target function. Such ultra-fast regression program, which bypasses the need for gradient-based training, is also general and can be applied to functions either defined on finite dimensional spaces (scalar-valued function spaces) or infinite dimensional spaces (function-valued operator spaces).

• In particular, we demonstrate the flexibility and efficiency of our framework for fitting function-valued operators in two PDEs and one climate modeling problem. We show that our transductive approach allows for better generalization properties of neural operator regression, better precision when relevant data is available, and can be combined with iterative regression schemes that are too expensive for previous inductive approaches, thus holding great potential to improve neural operators applicability.

• To the best of our knowledge, our proposal is the first to marry vector-valued RKBS theory with deep meta-learning and might also shed new light on the in-context learning abilities observed in deep attentional architectures.

Problem formulation

Let V and U be two (finite or infinite-dimensional) Banach spaces, respectively referred to as the input and output space, and let B a Banach space of functions from V to U. We also note LpU, Bq (resp. LpUq) the set of bounded linear operators from U to B (resp. to itself). We consider the meta-learning problem of creating a function T able to approximate any functional element O in the space B from any finite collection of example pairs D O " tpv i , u i q | v i P V, u i " Opv i qu iďn . A prominent approach in statistical learning is empirical risk minimization which consists in predefining a class B Ă B of computable functions from V to U and subsequently selecting a model Õ as a minimizer (provided its existence) of a risk function L : B ˆD Þ Ñ R:

T pD O q P argmin ÕP B Lp Õ, D O q (1)
For instance, the procedure consisting in performing gradient-based optimization of objective ( [START_REF] Lu | Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators[END_REF] or Fourier Neural Operator (FNO) (Li et al., 2020a). As previously discussed, for every regression problem instance, evaluating T with these approaches requires a heavy training procedure. Instead, we show in this work that for specific spaces B, we can meta-learn a parametric map T θ that transductively approximates (in a certain functional sense) any target function O P B given a corresponding dataset D O such that:

@v P V, T pD O qpvq " T θ pv 1 , Opv 1 q, . . . , v n , Opv n q, vq « Opvq (2)
3 Vector-valued Reproducing Kernel Banach Space regression

In order to build T θ , we leverage the structure of reproducing kernel Banach spaces of functions B and combine it with the universal approximation abilities of deep networks. As we will see in the experimental section, RKBS are very general spaces occurring in a wide range of machine learning applications. We start by recalling some elements of the theory of vector-valued RKBS developed in [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF]. Namely, we will consider throughout uniform Banach spaces S (such condition guarantees the unicity of a compatible semi-inner product x., .y S : S ˆS Þ Ñ R, i.e. @s P S, xs, sy S " ||s|| 2 S and allows to build a bijective and isometric dual space S ˚). Theorem 1 (Vector-valued RKBS [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF]). A U-valued reproducing kernel Banach space B of functions from V to U is a Banach space such that for all v P V, the point evalutation δ v : B Þ Ñ U defined as δ v pOq " Opvq is continuous. In this case, there exists a unique function K : V ˆV Þ Ñ LpUq such that for all pv, uq P V ˆU:

$ & % v 1 Þ Ñ Kpv, v 1 qpuq P B @ O P B, xOpvq, uy U " xO, Kpv, .qpuqy B @ v 1 P V, }Kpv, v 1 q} LpU q ď }δ v } LpB,U q }δ v 1 } LpB,U q (3)
Informally, theorem [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF] states that RKBS are spaces sufficiently regular such that the image of any element O at a given point v can be expressed in terms of a unique function K. The latter is hence called the reproducing kernel of B and our goal is to leverage such unicity to build the map T θ . Let D be the set of all datasets D O previously defined. The following original theorem gives the existence of a solution to our meta-learning problem and relates it to the reproducing kernel. Theorem 2 (RKBS representer map). Let B be a U-valued RKBS from V to U, if for any dataset D O P D, Lp., D O q is lower semi-continuous, coercive and bounded below, then there exists a function T : D Þ Ñ B such that T pD O q is a minimizer of equation [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF]. If L is of the form Lp., D O q " L ˝tδ vi u iďn with L : U n Þ Ñ R, then the dual T pD O q ˚is in spantKpv i , .qpuq ˚, i ď n, u P Uu. Furthermore, if for any D O , Lp., D O q is strictly-convex, then T is unique.

While theorem (2) provides conditions for the existence of solutions to each regression problem defined by [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF], the usual method consisting in solving instance-specific minimization problems derived from representer theorems characterizations is generally intractable in RKBS for several reasons (non-convexity and infinite-dimensionality of the problem w.r.t to variable u, nonadditivity of the underlying semi-inner product). Instead, we propose to define image solutions T pD O q " ř iďn K θ pv i , .qp ũi q where K θ and p ũi q are respectively the learned approximation of the U-valued reproducing kernel K and a set of functions in U resulting from a sequence of deep transformations of image examples pu i q that we define below.

Transformers attention as a reproducing kernel ˛We first need to build K. Several pieces of work have proposed constructions of K in the context of a non-symmetric and nonpositive semi-definite real-valued kernel (Zhang et al., 2009a;[START_REF] Georgiev | Construction of pairs of reproducing kernel banach spaces[END_REF][START_REF] Lin | On reproducing kernel banach spaces: Generic definitions and unified framework of constructions[END_REF][START_REF] Xu | [END_REF]. In particular, the exponential key-query function in the popular Transformer model [START_REF] Vaswani | Attention is all you need[END_REF] has been interpreted as a real-valued reproducing kernel κ θ : V ˆV Þ Ñ R in [START_REF] Wright | Transformers are deep infinite-dimensional non-mercer binary kernel machines[END_REF]. We extend below this interpretation to more general vector-valued RKBS: Proposition 1 (Dot-product attention as U-valued reproducing kernel). Let pp j q jďJ a finite sequence of strictly positive integers, let pA j θ q jďJ be applications from V ˆV to R, let V j θ be linear applications from LpU, R pj q and W θ a linear application from Lp ś jďJ R pj , Uq, the (multi-head) application κ θ : V ˆV Þ Ñ LpUq defined by

κ θ pv, v 1 qpuq fi W θ ˆ"..., A j θ `v, v 1 ˘¨V j θ puq, ... ‰ jďJ ˙(4)
is the reproducing kernel of an U-valued RKBS. In particular, if U " V " R p , for p P N ànd A j θ " exp `1 τ pQ j θ vq T pK j θ v 1 q ˘{σpv, v 1 q with pQ j θ , K j θ q jďJ applications from LpV, R d q, κ θ corresponds to the dot-product attention mechanism of [START_REF] Vaswani | Attention is all you need[END_REF].

Note that in (4), the usual softmax normalization of the dot-product attention is included in the linear operations A j θ through σ. We show in the next section how such kernel construction can be leveraged to build the map T θ and that several variations of the kernel construction are possible, depending on the target space B and applications. Contrary to usual kernel methods, our model jointly builds the full reproducing kernel approximation K θ and the instance-specific parametrization pũ i q iďI by integrating the solutions iteratively over several residual kernel transformations. We refer to our system as a Transducer, both as a tribute to the Transformer computation mechanism from which it is inspired and by analogy with signal conversion devices.

The Transducer

Model definition ˛We define T θ as the sum of L residual kernel transformations tκ θ u ďL whose expression can be written:

@ v P V, T θ pD O qpvq " ÿ iďI K θ pv i , vqp ũi q " ÿ iďI ÿ ďL κ θ pv i , v qpu i q (5)
where pv i , u i q iďn,lďL and pv q lďL refer to sequences of representations starting respectively with pv 1 i , u 1 i q iďn " D O , v 1 " v and defined by the following recursive relation:

# v `1 i " F θ pv i q , v `1 " F θ pv q u `1 i " ũ i `řj κ θ pv `1 j , v `1 i qp ũ j q where ũ i " G θ pu i q (6)
where pF θ , G θ q ďL correspond to (optional) parametric non-linear residual transformations applied in parallel to representations pv i , v i q iďn while pκ θ q ďL are intermediate kernel transformations of the form κ : V ˆV Þ Ñ LpUq such as the one defined in equation (4). Breaking down kernel estimation through this sequential construction allows for iteratively refining the reproducing kernel estimate and approximating on-the-fly the set of solutions pũ i q iďI . We particularly investigate the importance of depth L in the experimental section. Note that equations ( 5) and (6) allow to handle both varying dataset sizes and efficient parallel inference by building the sequences pv q ďL with pv i q iďn, ďL in batches and simply masking the unwanted cross-relational features during the kernel operations. All the operations are parallelizable and implemented on GPU-accelerated tensor manipulation libraries such that each regression with T θ is orders of magnitude faster than gradient-based regression methods.

Discretization ˛In the case of infinite-dimensional functional input and output spaces V and U, we can accommodate, for numerical computation purposes, different types of function representations previously proposed for neural operator regression and allowing for evaluation at an arbitrary point of their domain. For instance, output functions u can be defined as a linear combination of learned or hardcoded finite set of functions, as in [START_REF] Lu | Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators[END_REF] and [START_REF] Bhattacharya | Model reduction and neural networks for parametric pdes[END_REF]. We focus instead on a different approach inspired by Fourier Neural Operators (Li et al., 2020a), by applying our model on the M first modes of a fast Fourier transform of functions pv i , u i q iďn , and transform back its output, allowing us to work with discrete and finite function representations.

Meta-training ˛In order to train T θ to approximate a solution for all problems of the form (1), we jointly learn the kernel operations pκ θ q ďL as well as transformations pF θ q ďL . Let us assume that L is of the form LpO 1 , D O q " ř j LpO 1 pv j q, Opv j qq, that datasets D O are sampled according to a probability distribution D over the set of possible example sets with finite cardinality and that a random variable T select the indices of each test set

D test O " tpv i , u i q | pv j , u j q P D O , j P Tu such that the train set is D train O " D O zD test O .
Our meta-learning objective is defined as:

J pθq " E D,T " ÿ jPT LpT θ pD train O qpv j q, Opv j qq ı (7)
which can be tackled with gradient-based optimization w.r.t parameters θ provided L is differentiable (see S.I for details). In order to estimate gradients of ( 7), we gather a meta-dataset of M operators example sets pD Om q mďM and form, at each training step, a Monte-Carlo estimator over a batch of k datasets from this meta-dataset with random train/test splits pT k q. For each dataset in the batch, in order to form outputs T θ pD train O qpv j q defined by equation ( 5), we initialize the model sequence in ( 6) by concatenating

D train O with D query O " tpv i , 0 U q | v i P D test O u and obtain each infered output T θ pD train O qpv j q as ř viPD train O K θ pv i , v j qp ũi q .
Since each regression consists in a single feedforward pass, estimating gradients of the meta-parameters θ with respect to L for each batch consists in a single backward pass achieved through automatic differentiation.

Numerical experiments

In this section, we show empirically that our meta-optimized model is able to approximate any element O of diverse function spaces B such as operators defined on scalar and vector-valued function spaces derived from parametric physical systems or regression problems in Euclidean spaces. In all experiments, we use the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] to train for a fixed number of steps with an initial learning rate gradually halved along training. All the computation is carried on a single Nvidia Titan Xp GPU with 12GB memory. Further details can be found in S.I. First, we examine the problem of regressing operators O associating functions v from V Ă Cpr0, 1s, Rq to their solutions u " Opvq Ă Cpr0, 1s, Rq with respect to advection-diffusionreaction equations defined on the domain Ω " r0, 1s ˆr0, ts with Dirichlet boundary conditions sp0, tq " sp1, tq " 0. We consider the space B of operators O pδ,ν,k,tq specifically defined by vpxq " spx, 0q, upxq " spx, tq and s follows an equation depending on unknown random continuous spatially-varying diffusion δpxq, advection νpxq, and a scalar reaction term k " Ur0, 0.1s:

Regression of Advection-Diffusion Reaction PDEs

B t spx, tq " ∇ ¨pδpxq∇ x spx, tqq looooooooooomooooooooooon diffusion `νpxq∇ x spx, tq looooooomooooooon advection `k ¨pspx, tqq 2 loooooomoooooon reaction (8)
Eq. ( 8) is generic with components arising in many physical systems of interest, leading to various forms of solutions spx, tq. (We show examples for three different operators in figure 2.) Several methods exist for modeling such PDEs, but they require knowledge of the underlying parameters pδ, ν, kq and often impose constraints on the evaluation point as well as expensive time-marching schemes to recover solutions. Here instead, we assume no a priori knowledge of the solution and directly regress each operator O behavior from the example set Baselines and evaluation ˛We meta-trained our model to regress 500 different operators O pδ,ν,k,1q with t " 1 fixed and varying number of examples n P r20, 100s with images evaluated at 100 equally spaced points px k q kPrr0,100ss on the domain r0, 1s and meta-tested on a set of 500 operators with new parameters δ, ν, k and initial states v. Although not directly equivalent to existing approaches, we compared our method with standard regression methods as well as inductive neural operator approximators. We applied standard finite-dimensional regression methods, K-Nearest-Neighbors [START_REF] Fix | Discriminatory analysis. nonparametric discrimination: Consistency properties[END_REF], Decision Trees [START_REF] Quinlan | Induction of decision trees[END_REF] and Ridge regression with radial basis kernel [START_REF] Hastie | The elements of statistical learnin[END_REF] to each discretized problems `tOpv j qpx k q " u j px k qu j,k q as well as two neural-based operators to each dataset instance: DeepONet [START_REF] Lu | Learning nonlinear operators via deeponet based on the universal approximation theorem of operators[END_REF] and FNO (Li et al., 2020a). For these approaches, an explicit optimization problem is solved before inference in order to fit the target operator. On the other hand, after meta-training of the Transducer, which takes only a few minutes to converge, each regression is solved in a single feedforward pass of the network, which is orders of magnitude faster and can be readily applied to new problems (Table 1).

D O . METHOD RMSE TIME (S) GFLOPS FNO 2.96e ´4 1.72e 2 1.68e 2 DEEPONET 2.02e ´2 7.85e 1 1.54e 2 TRANSDUCER 2.39e ´4 3.10e ´3 1.06e ´1
Results ˛We first verified that our model approximates well unseen operators from the test set (Table 1). We noted that our model learns a non-trivial kernel since the estimation produced with 2 -Nearest Neighbors remains poor even after 1e 3 examples. Moreover, since our model can perform inference for varying input dataset sizes, we examined the Transducer accuracy when varying the number of examples and found that it learns a converging regression program (Figure 2) which consistently outperforms other instance-specific regression approaches with the exception of FNO when enough data is available (ą 60). We also found that deeper Transducer models with more layers increase kernel approximation accuracy, with untied weights yielding the best performance (figure 2.)

Extrapolation to OOD tasks ˛We further tested the Transducer ability to regress different operators than those seen during meta-training. Specifically, we varied the correlation length (C.L) of the Gaussian processes used to generate functions δpxq and νpxq and specified a different target time t 1 ‰ 1. We showed that the kernel meta-optimized for a solution at t " 1 transfers well to these new regression problems and that regression performance degrades gracefully as the target operators behave further away from the training set (figure 3), while inductive solutions do not generalize.

Outliers detection on 2D Burgers' equation

We further show that our regression method can fit operators of vector-valued functions by examining the problem of predicting 2D vector fields defined as a solution of a two-dimensional Burgers' equation with periodic spatial boundary condition on the domain Ω " r0, 1s 2 ˆr0, 10s:

B t sp v, tq " ν∆ v ¨sp v, tq loooooomoooooon diffusion ´sp v, tq∇ x sp v, tq loooooooomoooooooon advection (9)
Here, we condition our model with operators of the form, vp xq " sp x, tq, up xq " sp x, t 1 q such that our model can regress the evolution of the vector field v starting at any time, with arbitrary temporal increment t 1 ´t ď 10 seconds and varying diffusion coefficient ν P r0.1, 0.5s. We show in figure (4) and table (2) that our model is able to fit new instances of this problem with unseen parameters ν. Fast and differentiable regression ˛Since fitting with respect to training data is orders of magnitude faster than other operator regression approaches and fully differentiable, we can quickly execute expensive schemes requiring multiple regressions. This can have several applications, from bootstrapping or producing confidence intervals by varying the example set D train O , or performing inverse problems using Monte-Carlo Markov Chain in the dataset space. We showcase an example of this potential with an outlier detection experiment: We use the Transducer to identify outliers of a dataset of Burgers' equation with coefficient ν 1 artificially contaminated with elements from another dataset ν 2 ą ν 1 at 5% level. We identify outliers by estimating RMSEs over 5000 different regressions using random 50 % splits with outliers potentially present in both training and testing sets. This technique takes only a few seconds to estimate while outliers are clearly identified as data points with significantly higher RMSE than the dataset average (figure 5). As a comparison, performing Spectral Clustering [START_REF] Yu | Multiclass spectral clustering[END_REF] on the FFT of elements pu i q yields very poor precision (

Climate modeling with seasonal adaptation

One advantage of our approach is the ability to select the data that is most relevant with respect to a certain prediction task and subsequently adapt the model response. For instance, robust and precise Similar to [START_REF] Pathak | Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators[END_REF], we modify a ViT backbone to incorporate a kernel transduction layer before every patch attention and compare our model to an unmodified ViT baseline with a matching number of parameters. We additionally compare with a fully transductive Nearest Neighbors approach. In Figure 6 and Table 3, we present results obtained on training a Transducer with data from 2010 to 2014 and testing it on data from 2016 to 2019. We trained our model by predicting 5 random days sampled from random 20-day windows and present two test configurations: We either condition the Transducer with a window centered at the previous year's same date (P.Y) or with a 15 days window lagging by a week (P.W) (see SI for details). Both cases outperform transductive and inductive baselines with fast inference time, confirming that our solution can scale to large problems and be combined with other deep learning modules.

Finite-dimensional case: MNIST-like datasets classification

We finally confirm the generality of our approach in the case of finite-dimensional spaces U and V by studying the meta-learning problem presented in [START_REF] Kirsch | General-purpose in-context learning by meta-learning transformers[END_REF] which consists in regressing classification functions from the 784-dimensional space of MNIST-like images to a 10-dimensional space of one-hot class encoding (i.e functions considered are O : r0, 1s 784 Þ Ñ r0, 1s 10 ). We meta-train a 2-layer Transducer to classify consistently pixel-permuted and class-permuted versions of MNIST.

We then meta-test the Transducer to classify the unpermuted MNIST dataset and how the regression map transfer to Fashion MNIST and KMNIST. We show that without particular fine-tuning, the Transducer outperforms previous meta-learning approaches on both the original MNIST classification task as well as Fashion MNIST and K-MNIST.

Related work

Transductive Machine learning ˛Principles of transductive statistical estimation have been formally described in [START_REF] Gammerman | Learning by transduction[END_REF]; [START_REF] Vapnik | The nature of statistical learning theory[END_REF]. Algorithms relying on relational structures between data points such as K-nearest neighbors [START_REF] Cover | Nearest neighbor pattern classification[END_REF] and kernel methods [START_REF] Nadaraya | On estimating regression[END_REF][START_REF] Watson | Smooth regression analysis[END_REF] build estimates by weighing examples with respect to a certain metric space. Further, the "kernel trick" allows to embed possibly infinite-dimensional features [START_REF] Ferraty | Nonparametric functional data analysis: theory and practice[END_REF] into finite Gram matrix representations that are also well-suited for multitask regression [START_REF] Evgeniou | Learning multiple tasks with kernel methods[END_REF][START_REF] Caponnetto | Universal multi-task kernels[END_REF]. Distinctively, Gaussian processes regression [START_REF] Williams | Gaussian processes for regression[END_REF] combines transduction with Bayesian modeling to estimate a posterior distribution over possible functions. These techniques might suffer from the so-called "curse of dimensionality": with growing dimensionality, the density of exemplar point diminishes, which increases estimators' variance. More recent work combining deep learning with transductive inference has shown promising results even in high-dimensional spaces for few-shot learning [START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Sung | Learning to compare: Relation network for few-shot learning[END_REF] or sequence modeling [START_REF] Jaitly | A neural transducer[END_REF], but the vast majority of neural networks still remain purely inductive.

Neural operator learning ˛The universal approximation abilities of neural networks have been generalized to infinite-dimensional function spaces: Chen and Chen (1995) showed that finite neural parametrization can approximate well infinite-dimensional operators. More recent work using neural networks to perform operator regression has shown strong results [START_REF] Lu | Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators[END_REF], especially when mixed with tools from functional analysis and physics [START_REF] Raissi | Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations[END_REF]Li et al., 2020a;[START_REF] Gupta | Multiwavelet-based operator learning for differential equations[END_REF][START_REF] Li | Multipole graph neural operator for parametric partial differential equations[END_REF][START_REF] Nelsen | The random feature model for input-output maps between banach spaces[END_REF][START_REF] Wang | Learning the solution operator of parametric partial differential equations with physics-informed deeponets[END_REF][START_REF] Roberts | Rethinking neural operations for diverse tasks[END_REF] and constitutes a booming research direction in particular for physical applications [START_REF] Goswami | Physics-informed neural operators[END_REF][START_REF] Pathak | Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators[END_REF][START_REF] Vinuesa | Enhancing computational fluid dynamics with machine learning[END_REF][START_REF] Wen | U-fno-an enhanced fourier neural operator-based deep-learning model for multiphase flow[END_REF][START_REF] Pickering | Discovering and forecasting extreme events via active learning in neural operators[END_REF]. Recently, the Transformer's attentional computation has been interpreted as a Petrov-Galerkin projection [START_REF] Cao | Choose a transformer: Fourier or galerkin[END_REF] or through Reproducing Kernel Hilbert Space theory [START_REF] Kissas | Learning operators with coupled attention[END_REF] for building such neural operators, but these perspectives apply attention to fit a single target operator.

Meta-learning and in-context learning ˛Promising work towards more general and adaptable machines has consisted in automatically "learning to learn" or meta-learning programs [START_REF] Schmidhuber | Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement[END_REF][START_REF] Vilalta | A perspective view and survey of meta-learning[END_REF], by either explicitly treating gradient descent as an optimizable object [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF], modeling an optimizer as a black-box autoregressive model [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] or informing sequential strategies via memorization [START_REF] Santoro | One-shot learning with memory-augmented neural networks[END_REF][START_REF] Ortega | Meta-learning of sequential strategies[END_REF] More recently, converging findings in various domains from reinforcement learning [START_REF] Mishra | A simple neural attentive meta-learner[END_REF][START_REF] Laskin | In-context reinforcement learning with algorithm distillation[END_REF], natural language processing [START_REF] Brown | Language models are few-shot learners[END_REF][START_REF] Xie | An explanation of in-context learning as implicit bayesian inference[END_REF][START_REF] Olsson | In-context learning and induction heads[END_REF] and functional regression [START_REF] Garg | What can transformers learn in-context? a case study of simple function classes[END_REF] have established the ability of set-based attentional computation in the Transformer [START_REF] Vaswani | Attention is all you need[END_REF] for in-context learning by flexibly extracting functional relationships and performing dynamic association such as linguistic analogy or few-shot behavioral imitation. We show that the theory of RKBS can help interpret such property and extends it to function-valued operators regression.

Discussion

We proposed a novel transductive model combining kernel methods and neural networks that is capable of performing regression over entire function spaces. We based our model on the theory of vector-valued Reproducing Kernel Banach Spaces and showcased several instances where it learns a regression program able, in a single feedforward pass, to reach performance levels that match or outperform previous instance-specific neural operators or meta-learning systems. Our approach holds potential to create programs flexibly specified by data and able to model entire families of complex physical systems, with particular applications in functional hypothesis testing, dataset curation or fast ensemble learning. However, one limitation is that our model relies on meta-training, which requires collecting a sufficiently diverse meta-dataset to explore the kernel space. In future work, we plan to investigate methods such as synthetic augmentation to reduce meta-training costs.

Learning Functional Transduction: S.I.

S.8 Theoretical analysis

We propose below the proofs of the results presented in the main text. Most of the arguments are adapted from the development proposed in [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF] which goes beyond real or complex-valued RKBS developed in [START_REF] Zhang | Reproducing kernel banach spaces for machine learning[END_REF][START_REF] Song | Reproducing kernel banach spaces with the -1 norm[END_REF] to develop the notion of vector-valued RKBS. In addition, we note that assumptions regarding the properties of the RKBS of interests such as uniform Fréchet differentiability and uniform convexity have been further relaxed in other works [START_REF] Xu | [END_REF][START_REF] Lin | On reproducing kernel banach spaces: Generic definitions and unified framework of constructions[END_REF] but are here sufficient for our discussion since they guarantee the unicity of a semi-inner product x., .y B compatible with the norm ||.|| B [START_REF] Giles | Classes of semi-inner-product spaces[END_REF].

S.8.1 Theoretical results

Theorem 1 ˛Theorem 1 gathers for the sake of compactness the definition of a vector-valued reproducing kernel Banach space with the properties of existence and unicity of the kernel K.

Proof. For any v P V and u P U, the mapping O Þ Ñ xOpvq, uy U is a bounded linear form in LpBq. By Theorem 7 of [START_REF] Giles | Classes of semi-inner-product spaces[END_REF], we have the bijectivity of the duality mapping in U, hence there exists a unique element K v,u P B such that: xOpvq, uy U " xO, K v,u y B (10) Hence, this defines a unique function K : V ˆV Þ Ñ LpUq such that: @pv, v 1 q P V 2 , @u P U, Kpv, v 1 qpuq " K v,u pv 1 q (11) By construction K is unique, furthermore we have that (i) the functional v 1 Þ Ñ Kpv, v 1 q is an element of B (ii) it verifies the reproducing relation @pv, uq, xOpvq, uy U " xO, Kpv, .qpuqy B . Finally, property (iii) follows from the following bound on the norm of v 12) and ( 13) allows to write:

||Kpv, v 1 qpuq|| B ď ||δ v 1 || LpB,U q .||Kpv, .qpuq|| B ď ||δ v 1 || LpB,U q .||δ v || LpB,U q .||u|| U (14)
Observing that in particular for all u P U\0 U :

||Kpv, v 1 qpuq|| B ||u|| U ď ||δ v 1 || LpB,U q .||δ v || LpB,U q (15)
concludes the proof.

Theorem 2 ˛We first show the existence of a solution for any problem of the form ( 1) and then characterize the solution in terms of the data points.

Proof. We first show the existence of the map T : D Þ Ñ B. Let us take D P D, by assumption the function L D : Õ Þ Ñ LpO, Dq is weakly-lower semi-continuous, coercive and bounded below. Let us take a sequence pO k q kPN of elements in B such that L D pO k q Ñ L " inf OPB L D pOq. Since L D is coercive, the sequence is bounded in B, so there is a weakly-convergent subsequence pO ki q such that pO ki q Ñ O 0 . Finally, by property of weakly-lower semi-continuity, we have that L ď L D pO 0 q ď lim inf L D pO k q which shows that for any D, there exists a minimizer of L D .

We now turn to the characterization of the solution O 0 when we have that L D " L ˝tδ vi u iďn with L :

U n Þ Ñ R.
This assumption allows to exhibit a characterization of the solution in terms of annihilator and pre-annihilitors in B as in previous work [START_REF] Zhang | Vector-valued reproducing kernel banach spaces with applications to multi-task learning[END_REF][START_REF] Xu | [END_REF]. Let us consider the set S " tO P B, Opv i q " u i , i ď Iu. It is clearly a closed convex subset of B. Since B is uniformly convex, the problem inft||O|| B , O P Su (16) admits a best approximation in S (?). Furthermore, O 0 is the minimizer of ( 16) if and only if for all O P S 0 " tO P B, Opv i q " 0 U , i ď Iu, we have:

||O `O0 || B ě ||O 0 || B ( 17
)
which by [START_REF] Giles | Classes of semi-inner-product spaces[END_REF] is equivalent to O 0 P pS 0 q K . Finally, we note that O P S 0 if and only if xO, Kpv j , .qpuqy B " xOpv j q, uy U " 0, @j ď n, @u P U (18) which allows us to say that

O P K pKpv j , .qpuqq ˚, j ď n, u P U ( (19)
Finally, we obtain the following characterization: O P `K pKpv j , .qpuqq ˚, j ď n, u P U (˘K . Since B is reflexive, we have further that @S Ă B, p K Sq K " spanS, which concludes the proof for the characterization of T pDq.

Finally, if for all D, the function L D is strictly-convex, then it guarantees the unicity of a minimizer over B for every problem, which in turn defines an unique map T .

Proposition 1 ˛The result is direct by considering the feature map characterization of vector-valued RKBS (Corollary 3.2 of Zhang ( 2013)) that we recall hereafter: We first define for any linear operator T P LpS 1 , S 2 q between two Banach spaces S 1 , S 2 , the generalized adjoint T : P LpS 2 , S 1 q as the application verifying xT s, s 1 y S1 " xs, T : s 1 y S2 for all ps, s 1 q P S 1 ˆS2 .

Let F be a uniform Banach space and Φ : V Þ Ñ LpF, Uq a feature map such that: @pv, v 1 q P V 2 , Φpv 1 qpΦ : pvqq " Kpv, v 1 q (20)

spantpΦ : pvqpuqq ˚, v P V, u P Uu " F ˚(21)
with Φ : : V Þ Ñ LpU, Fq is defined by: @v, Φ : pvq " pΦpvqq : . then the vector space B " tΦp.qpwq|w P Fu endowed with the norm ||Φp.qpwq|| B compatible with the following semi-inner product: xΦp.qpwq, Φp.qpw 1 qy B " xw, w 1 y F (22) is a U-valued RKBS with reproducing kernel K given in (20).

Proof. We show our result in the case J=1 and can be directly extended to any cardinality J. By hypothesis, V and U are a uniform Banach space and so is LpV, Uq. We hence define the feature map Φ as defined by equations ( 20) and ( 21) Φ and noting here that F " LpV, Uq:

Φ : V Þ Ñ LpLpV, Uq, Uq (23) v Þ Ñ Φpvq " ˆl Þ Ñ lpvq ˙(24)
In particular, by considering the uniform space F " tl P LpV, Uq | D v 1 P V, u P U l " W θ `A1 θ `., v 1 ˘.V θ puq ˘u Ă F, we have the following relation:

@pv, lq P V ˆF, D v 1 P V, u P U s.t Φpvqplq " W θ `A1 θ `v, v 1 ˘.V θ puq ˘, (25)
identifying the adjoint

Φ : pvq : U Þ Ñ LpV, Uq as Φ : pvq : u Þ Ñ `v1 Þ Ñ W θ `A1 θ `v, v 1 ˘.V θ puq ˘˘and verifying the kernel relation: @pv, v 1 q P V 2 , Φpv 1 qpΦ : pvqq " Kpv, v 1 q " W θ `A1 θ `v, v 1 ˘.V θ p.q ˘(26)
Furthermore, by bijectivity of the duality map on F Ă LpV, Uq that spantpΦ : pvqpuqq ˚, v P V, u P Uu " F˚. The application of the feature map characterization of K on F allows to conclude.

S.9 Numerical implementation S.9.1 Loss functions and evaluations Definition of loss function ˛In the case of operator regression, we meta-train models with respect to the Mean-Squarred error (MSE) over I test pairs pv i , u i q iďI of the meta-train set and K evaluation points px k q kďK of the domain of the output functions in V:

Lp Õ, D O q " 1 I ÿ iďI Lp Õpv i q, u i q " 1 I.K ÿ iP ÿ kďK || Õpv i qpx k q ´ui px k q|| 2 2 (27)
In the case of experiment 1 (ADR equation), px k q kďK corresponds to equally spaced points px k q kPrr0,100ss on the domain r0, 1s. For experiment 2 (2D Burgers equation), px k q kďK corresponds to uniform 2D mesh px k,p q kPrr0,64ss,pPrr0,64ss discretizing the domain r0, 1s ˆr0, 1s. For experiment 3 (Climate modeling), as stated in the main text, px k q kďK corresponds to 2D mesh px k,p q kPrr0,720s,pPrr0,720sss spanning the domain r0, 180 ˝s ˆr0, 360 ˝s. Finally for the final finitedimensional experiment (MNIST-like datasets), evaluation points pkq kPrr0,10ss corresponds to indices of 10-dimensional vectors of one-hot class encodings such that L corresponds to:

Lp Õ, D O q " 1 I.K ÿ iP ÿ kď10 | Õpv i qpkq ´ui pkq| 2 (28)
Definition of RMSE ˛Similarly, in the case of operator regression, we report average Relative Mean-Squarred Errors (RMSEs) defined as:

RMSEp Õ, D O q " 1 I.K ÿ iP ÿ kďK || Õpv i qpx k q ´ui px k q|| 2 2 ||u i px k q|| 2 2 (29)
Note that for meta-training and meta-evaluation, MSEs and RMSEs are further averaged over batches of J 1 elements pO j q jPJ 1 . S.9.2 Discussion on multi-head reproducing kernels Kernel definition ˛In coherence with [START_REF] Wright | Transformers are deep infinite-dimensional non-mercer binary kernel machines[END_REF], we show that different expressions of the kernel κ θ can be proposed. Specifically, we tested three expressions:

• Exp. dot product: A θ pv, v 1 q " expp K θ pvqq T pQ θ pv 1 qq τ q • RBF: A θ pv, v 1 q " expp ||K θ pvq´Q θ pv 1 q|| 2 2 τ q • 2 -norm: A θ pv, v 1 q " ||K θ pvq ´Qθ pv 1 q|| 2 2
Note that for each kernel expression, we still perform a normalization operation v Þ Ñ A θ pv,viq ř iďI A θ pv,viq over the entire set pv i q iďI without loss of generality. We report below regression RMSE for the ADR experiment with the different expressions for the linear function A θ pv, v 1 q for different dataset sizes. The two first expressions yield similar result in the ADR experiment at an equal compute cost. For coherence, we present all other results with the "exponentiated dot product" kernel definition.

S.9.3 Details on model hyperparameters and architecture

Discretization ˛As mentionned in the main text, in order to manipulate functional data, our model can accomodate previous forms of discretization. We particularly tested two different forms of discretization discussed in (Li et al., 2020a) and [START_REF] Lu | Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators[END_REF].

• In most of our experiments, we apply the Transducer model after performing a Fast Fourier transforms (FFT) of the considered input and output functions, and transform the Transducer's output back to form estimates at arbitrary resolution. More specifically, we apply our model on the d-dimensional finite vector formed by the first modes of the Fourier transform, and discard the rest of the function spectrum. For experiments with 2D fields, we describe more precisely in section S.10.2 how we combine the 2D FFT with our model. • We also tried a 'branch' and 'trunk' networks formulation of the model as in DeepONet [START_REF] Lu | Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators[END_REF]. Specifically, the branch network g : V Þ Ñ K P correspond to the Transducer network which outputs the weight parameters pk p q pďP for the functional basis learned by the 'trunk' networks f : D Þ Ñ K P where D corresponds to the domain of U. Hence, the transducer model reads:

@x P D T pD O qpvqpxq " ÿ pďP g p pD O qpvq.f p pxq (30)
We tested this approach in the ADR experiment by directly feeding the functions values pv i px k qq kď100 and pu i px k qq kď100 of the uniformly discretized domain of V and U. We noted that performance was slightly worse than the Fourier method as we did not perform additional tuning such as feature augmentation for the branch network. For coherence, we kept the Fourier transform for the other experiments.

Feedfoward networks definition ˛For F θ and G θ , we use a simple feedfoward network architecture defined as Layer normalization [START_REF] Ba | Layer normalization[END_REF] followed by one layer perceptron with GeLU activation and did not performed architectural search on this part of the network.

Architecture hyperparamters ˛We present in the following table the particular architectural choices for each experiment.

S.9.4 Details on meta-training

As stated, we used for all experiments, the same meta-training procedure. We optimized Transducer models using the Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] for a fixed number of epochs with learning rates halved multiple times across meta-training.

S.10 Experiments

In this section, we provide additional details with respect to data generation and model evaluation for each experiments discussed in section (5) of the main text.

S.10.1 Advection-Diffusion-Reaction operators

Data generation -For our experiment, we collect a meta-dataset of N " 500 datasets of the advection-diffusion-reaction trajectories on the domain Ω " r0, 1sˆr0, 1s by integrating the following equations:

@n P rr1,
We use an explicit forward Euler method with step-size 1e ´2, storing all intermediate solutions on a spatial mesh of 100 equally spaced points. Hence, our discretized reference trajectories are of dimensions 100 ˆ100. For each operator O n we generate spatially varying diffusion and advection coefficients as random function δ n pxq : r0, 1s Þ Ñ R and ν n pxq : r0, 1s Þ Ñ R as well as a random scalar reaction coefficient k n . Defining Gp0, k l px 1 , x 2 qq the one-dimensional zero-mean Gaussian random field with the covariance kernel:

k l px 1 , x 2 q " e ´}x 1 ´x2 } 2 2l 2 (32)
and lenght-scale parameter l " 0.2, as well as a boundary mask function m : r0, 1s Þ Ñ r0, 1s, mpxq " 1 ´p2x ´1q 10 (to comply with Dirichlet boundary condition and preserve numerical computation stability), we sample δ n pxq and ν n pxq according to the following equations:

• diffusion δ n pxq " 0.01 ˆun pxq 2 ˆmpxq where u n " Gp0, k 0.2 px1, x2qq

• advection ν n pxq " 0.05 ˆyn pxq ˆmpxq where y n " Gp0, k 0.2 px1, x2qq

• reaction k n " Upr0, 0.3sq. Furthermore, we collect for each dataset i " 100 trajectories with each different initial state spx, 0q " v i pxq, where functions v i pxq are sampled according to the following:

• initial state v i pxq = mpxq ˆui pxq where u i " Gp0, k 0.2 px1, x2qq. For meta-testing, we sample N " 500 new datasets of the same generic advection-diffusion-reaction equation with new parameters δ n pxq, ν n pxq, k n pxq, for up to 1000 different initial states v i pxq. We present below example of function profiles present in the meta-datasets.

Training ˛We train Tranducers for 200K gradient steps. At each training step, we randomly draw a single operator O n from the meta-training set and isolate the pairs pv i , u i q iďI " ps i px, 0q, s i px, 1qq iďI to form the set E On . We sample a "query" subset Q of J " 10 pairs from E On to be regressed and form the input to our model by concatenating pairs of the query set Q (with output elements pu i q iPQ set to zero), with a non-overlaping set of I P rr20, 100ss example elements drawn from pv i , u i q iRQ . We train our model to minimize the sum of L 2 error between each output function of the set Q and its corresponding ground truth upxq " O n pvqpxq " spx, 1q at the 100 discretized positions.

Baselines -In order to implement the baseline regression algorithms, we use the scikit-learn library [START_REF] Pedregosa | Scikit-learn: Machine learning in Python[END_REF] for decisions trees, K-nearest neighbours and Ridge regression. We specifically tuned Ridge regression using cross-validation and selected the best-performing 'RBF' kernel with regularisation lambda " 1e ´3. For FNO (Li et al., 2020a), we use the official PyTorch implementation provided by authors and defined for each regression, a 4-layer deep 1-dimensional FNO network with 16 modes and 64-dimensional 1 ˆ1 convolutions. For DeepOnet [START_REF] Lu | Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators[END_REF], we implement our own PyTorch version with 4 hidden layers of 50 hidden units with ReLU activation for the branch and trunk networks.

Extrapolation experiment -In this task, we modify the generative process of the considered operators by changing the lenght-scale parameter l used to produce functions δpxq and νpxq, as well as the target time t used to define the operator output. Generation ˛In order to produce the meta-datasets of our second experiment, we use the ΦFlow library [START_REF] Holl | Learning to control pdes with differentiable physics[END_REF] that allows for batched and differentiable simulations of fluid dynamics and available at https://github.com/tum-pbs/PhiFlow. Following the same methodology as experiment 1, we generate batches of the state evolution of random functions pv i q : R 2 Þ Ñ R 2 defined on the domain Ω " r0, 1s 2 at a resolution of 64 ˆ64 through different parametrization of equation ( 4). We Spectral clustering ˛As a baseline for the outlier detection experiment, we used the spectral clustering algorithm [START_REF] Yu | Multiclass spectral clustering[END_REF] implemented in the Scikit-learn on the same FFT preprocessing transformation of the output elements pu i q iďI that is discussed above and specifying the number of clusters C " 2. We tried to tune the clustering algorithm in the embedding space either using K-means or a kernel formulation. The tested variations yielded no significant difference in performance.

S.10.3 Climate modeling

ViT modification ˛In order to tackle the high-resolution climate modeling experiment, we take inspiration from [START_REF] Pathak | Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators[END_REF], which combines neural operators with the patch splitting method of Vision Transformer (ViT) [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]. Specifically, we split input and output functions into patches of size 40 ˆ40. Since both models operations preserves dimensionality, we interleaves Transducer layers that apply kernel transformations κ θ along the batch dimension with ViT layers performing spatial attention on the set of patched output function representations pu i q. We drop positional encoding but reduce spatial attention to the neighboring patches for each patch position through masking. We compare this bi-attentional model to a vanilla ViT model that learns by induction a single map from temperature V to pressure U. We double the depth of this baseline to L " 12, in order to match number of trainable parameters.

Data ˛We take our data from ERA5 reanalysis [START_REF] Hersbach | The era5 global reanalysis[END_REF], that is freely available on the Copernicus https://cds.climate.copernicus.eu/cdsapp#!/dataset/ reanalysis-era5-land?tab=overview. Surface and temperature pressure are re-gridded from a Gaussian grid to a regular Euclidean grid using the standard interpolation scheme provided by the Copernicus Climate Data Store (CDS) to form 2D fields that we further interpolate in the longitude dimension to obtain images of size 720 ˆ720. Although the ERA5 possess hourly estimates, we subsample the dataset by considering only measurement at 12:00am UTC every day.

Training ˛As mentioned in the main text, we trained our model to predict variables for 5 days randomly sampled from a 20-day window and condition the Transducer with remaining 15 days. We do not explore larger settings due to GPU memory constraints.

S.10.4 MNIST-like dataset classification

Training ˛We report results from [START_REF] Kirsch | General-purpose in-context learning by meta-learning transformers[END_REF] for baselines and train and evaluate our model on datasets versions provided by the torchvision library. For this version, we directly treat the images inputs pv i q i as 784-dimensional vectors and the outputs pu i q i as 10-dimensional vectors.

We do not perform intermediary non-linear transformations G θ for the outputs representations. We haven't performed extensive hyper-parameter search for this experiment in terms of learning rate, head dimensions or kernel expression but simply noted that a deeper 4-layer version of the model was giving similar performance results.

Figure 1 :

 1 Figure 1: Batches of functional images T θ pDO i qpvjq « Oipvjq " ujpx, tq P Cpr0, 1s 2 , Rq obtained with the same Transducer model T θ but conditioned, at each row, by a different dataset pDO i qiď3 during feedfoward computation. Each underlying operator Oi corresponds to a different advection-diffusion-reaction equation (defined in Sec. 5.1) with spatially varying advection, diffusion, and reaction parameters unseen during training, and functions pvjqjď9 correspond to initial conditions. While usual neural regression approaches learn a single target function (one row), our model learns to approximate instantaneously an infinity of them.

Figure 2 :

 2 Figure 2: Left: RMSEs (and 95% C.I) on unseen operators as a function of the dataset size. The grey area corresponds to dataset cardinalities seen during the Transducer meta-training. To provide comparison, we train baselines from scratch with the corresponding number of examples. Middle: Training losses of Transducers with different depths. Applying several times the kernel improves performance, with untied weights yielding the best performance. Right: 3 examples of the evolution of spx, tq for different ADR equations and spatial MSEs of intermediate representations pu q colored by iteration . The decreasing error, consistent with the MSE reduction of deeper models, suggests that network depth allows for progressively refining function estimates.

Figure 3 :

 3 Figure 3: Example of Transducer regression extrapolation and RMSEs on OOD tasks with n " 100 examples. Color code corresponds to different correlation lengths used to generate the random functions δpxq and νpxq. Much of the result remains below 1% error despite never being trained on such operators.

Figure 4 :

 4 Figure 4: Illustrative example of initial pt " 0q, target pt " 10q and Transducer estimation of the vector field sp x, tq discretized at resolution 64 ˆ64 over the domain r0, 1s 2 for the Burgers' equation experiment. The last panel represents absolute error to ground truth.

Figure 5 :

 5 Left: Meta-test regression and outlier detection results at two target times. RMSEs on Burgers' equations averaged over 200 different parameter conditions ν P r0.1, 0.5s each with 100 train examples. Precision/Recall in outlier detection of the Transducer versus Spectral clustering. Right: RMSE distributions of each element in the contaminated dataset over the 5000 regressions. Outliers are clearly identified.

Figure 6 :

 6 Figure 6: Up -Illustrative examples of 720 ˆ720 temperature (left) and pressure (right) fields of the ERA5 dataset. Bottom -Estimated pressure field from conditioning the Transducer with 15 days data dating 1 week before the target date. Insets show recovered details of the estimation (blue) compared with ground truth (red).

Figure 7 :

 7 Figure 7: Comparison of meta-test accuracies of MNIST-like datasets classification task presented in Kirsch et al. (2022) against the Transducer.

Figure S. 8 :

 8 Figure S.8: Examples of sampled functions δpxq and νpxq used to build operators On.

Figure S. 9 :

 9 Figure S.9: Examples of advection-diffusion-reaction datasets (different operators by row) present in the meta-test set.

Figure S. 10 :

 10 Figure S.10: Examples of the spatial function sampled with carying lenght scale parameter l P r0.1, 0.2, 0.3s

  form a meta training set of 200 operator datasets for different parameters ν P r0.1, 0.5s each of cardinality I " 100, and meta testing set of 200 different operator datasets with the same cardinality. Here, we consider vector fields input functions vp xq whose coordinates pv 1 p xq, v 2 p xqq are drawn each from a two-dimensional zero-mean Gaussian random fields with uniform exponential covariance function and correlation length l " 0.125. Discrete Fourier representation -Since we are dealing with high-dimensional inputs, we perform kernel regression on the 2D fast Fourier transforms of our model. To reduce further dimensionality, since the FFT of a real signals is Hermitian-symmetric, we pass as input to our model only the flattened 10 ˆ10 upper and lower quadrants of the Fourier transform coefficients, since we verified that those are sufficient to reconstruct the signal at relative error level of 1e ´5. (We present examples of the 2D FFT of our signal.) After regression, we reconstruct our model estimate in the spatial domain at the desired 64 ˆ64 resolution and train for the L 2 distance against ground truth.

Figure S. 13 :

 13 Figure S.13: Illustrative examples of initial pt " 0q, target pt " 10q and Transducer estimation of the vector field sp x, tq discretized at resolution 64 ˆ64 over the domain r0, 1s 2 for the Burger's equation experiment. The last panel represents absolute error compared to ground truth.

  1) over a parametric class B of neural networks defines implicitly such a function T . Fundamentally, this technique works by induction: It captures the statistical regularities of a single map O into the parameters of the neural network Õ such that D O is discarded for inference. Recent examples of gradient-based optimization of neural networks for operator regression (i.e when V and U are infinite-dimensional) are DeepOnet

Table 1 :

 1 RMSE

and compute costs of regression over 50 unseen datasets with n " 50 examples. Note that DeepONet and FNO are optimized from scratch while the Transducer has been pre-trained. GFLOPs represent the total number of floating point operations for regression.

table 2 )

 2

		t = 5s	t = 10s
	RMSE (test sets)	2.2e ´3	5.9e

´3

Outliers (Pre./Rec.) 100%{100% 100%{100% S.C. (Pre./Rec.) 6%{85% 7%{85%

Table 2 &

 2

Table 3 :

 3 Latitude-weighted mean-square error (in hectopascals) and inference time for the earth surface pressure prediction task.

  Theoretical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Loss functions and evaluations . . . . . . . . . . . . . . . . . . . . . . . . . 14 S.9.2 Discussion on multi-head reproducing kernels . . . . . . . . . . . . . . . . . 14

	Contents
	S.8 Theoretical analysis	11
	S.8.1 S.9 Numerical implementation	14
	S.9.1

S.9.3 Details on model hyperparameters and architecture . . . . . . . . . . . . . . . 15 S.9.4 Details on meta-training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 S.10 Experiments 16 S.10.1 Advection-Diffusion-Reaction operators . . . . . . . . . . . . . . . . . . . . 16 S.10.2 Burger's equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 S.10.3 Climate modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 S.10.4 MNIST-like dataset classification . . . . . . . . . . . . . . . . . . . . . . . . 20

  Further, we have by continuity of the point evaluation δv : O Þ Ñ Opvq that: |xOpvq, uy U | ď ||δ v || LpB,U q .||v|| U

	(13)
	Combining (

1 Þ Ñ Kpv, .qpuq: ||Kpv, .qpuq|| B ď sup OPB,||O|| Bď1 |xO, Kpv, .qpuqy B | " sup OPB,||O|| Bď1 |xOpvq, uy U | (12)

  Results from variation of the Transducer kernel constructions in the ADR experiment. Note that contrary to other definitions, the 2-based kernel does not generalize to dataset cardinalities beyond those seen in the meta-training set.

	KERNEL EXPRESSION	S=10	S=100	S=500
	EXP. DOT PRODUCT	2.71e ´3	2.39e ´4	1.79e	´4
	RBF	8.71e ´3	3.46e ´4	3.22e ´4
	Table S.4:

  Summary of the architectural hyperparameters used to build the Transducer in the four experiments. 'Depth' corresponds to network number of layers, 'MLP dim' to the dimensionality of the hidden layer representation in F θ and G θ , d to the dimension of the discrete function representations.

	EXPERIMENT	DEPTH	MLP DIM	DIM d	#HEADS	DIM HEADS
	ADR	1-16	100	50	32	16
	BURGERS	10	800	800	64	16
	CLIMATE	6	512	512	40	16
	MNIST	2	256	784	32	32
	Table S.5:

  Summary of the meta-learning hyperparameters used to meta-train the Transducer in our four experiments.

	EXPERIMENT	# OF EPOCHS	LEARNING RATE	DIM HEADS
	ADR	200	1e ´4	50
	BURGERS	200	1e ´4	800
	CLIMATE	200	1e ´4	512
	MNIST	500	1e ´4	784
	Table S.6:

  500ss, B t spx, tq " ∇ ¨pδ n pxq∇ x spx, tqq loooooooooooomoooooooooooon

diffusion `νn pxq∇ x spx, tq loooooooomoooooooon advection `kn ¨pspx, tqq 2 looooooomooooooon reaction

-NORM 1.71e ´2 6.98e ´4 7.33e5