Philippe Bich

On the continuous representation of quasi-concave mappings by their upper level sets

Keywords:

we provide a continuous representation of quasi-concave mappings by their upper level sets. A possible motivation is the extension to quasi-concave mappings of a result by Ulam and Hyers, which states that every approximately convex mapping can be approximated by a convex mapping.

Introduction

Since a seminal paper of Hyers and Ulam (1), several papers have studied the problem of the stability of functional inequalities. In particular, the first issue given (and solved) by Hyers and Ulam was roughly the following: is it possible to approximate approximately convex mappings by convex mappings ?

Now, an important extension of convexity, especially for mathematical economics or game theory, is quasi-concavity. In the following, let X be a convex subset of some vector space. Definition 1.1 A mapping f : X → R is quasi-concave if the following property is true: ∀(x, y) ∈ X × X, ∀λ ∈ [0, 1], f (λ.x + (1 -λ)y) ≥ min{f (x), f (y)}.

Equivalently, f : X → R is quasi-concave if its upper level sets S λ (f ) = {x ∈ X, f (x) ≥ λ} are convex subsets of X for every λ ∈ R. Clearly, concave and convex mappings are quasi-concave. Besides, non-increasing or non-decreasing mappings from R to R are quasi-concave.

In order to raise the issue of the stability of the quasi-concavity property, one defines the notion of approximately quasi-concave mapping as follows: Definition 1.2 For every > 0, the mapping f : X → R is -quasi-concave if the following property is true:

∀(x, y) ∈ X × X, ∀λ ∈ [0, 1], f (λ.x + (1 -λ)y) ≥ min{f (x), f (y)} -.
A natural question, in the vein of Hyers and Ulam's stability result, is to know whether it possible to approximate an -quasi-concave mapping f : X → R by a quasi-concave mapping. Such question could have applications in game theory or in mathematical economics, but it also has an interest for its own sake.

A possible answer to this question could be to build the quasi-concave envelop f of f as follows: first, associate to the -quasi-concave mapping f its upper level sets S λ (f ) = {x ∈ X, f (x) ≥ λ}. [START_REF] Hyers | Approximately Convex Functions[END_REF] Then convexify the upper level sets by defining their convex hulls coS λ (f ). Last, try to find a quasi-concave mapping f whose upper level sets are the coS λ (f ).

A first objective of the present paper is to clarify the previous construction. [START_REF] Hadjisavvas | Handbook of Generalized Convexity and Generalized Monotonicity[END_REF] In particular, one could hope that the mapping f is a good approximation of f if is small enough. In view of the construction of f , a question which naturally arises is the continuity of the representation of each mapping f : X → R by its upper level sets. This is the main purpose of this paper, and this study is given in Section 2. In particular, it requires the definition of a good topology on the set of upper level sets. Besides, the stability problem presented above suggests to study the continuity properties of the convex hull operator in the set of upper level sets, which is done in Section 3. The stability of the quasi-concave property is given in Section 4, as a immediate byproduct of the previous sections.

Continuous representation of a quasi-concave mapping by its upper level sets

The purpose of this section is to prove that the standard representation of quasi-concave mappings by their upper level sets can be made continuous by defining adapted topologies.

In the following subsection, we recall the standard representation result.

2.1 Definition of u.l.s. multivalued mappings and the standard representation of a mapping by its upper level sets.

Definition 2.1 Let X be a vector space. A multivalued mapping T from R to X is said to be a u.l.s. (upper level sets) multivalued mapping if the following properties are satisfied:

i) Monotony For every (λ, µ) ∈ R × R, λ ≤ µ implies T µ ⊂ T λ . ii) Left Continuity For every λ ∈ R, one has n∈N * T λ-1 n = T λ .
iii) Limits at +∞ and -∞ One has λ∈R

T λ = X and λ∈R T λ = ∅.
Throughout this paper, T denotes the set of all multivalued mappings from R to X satisfying the Assumptions i) and iii) above, and T u.l.s. the set of all u.l.s. multivalued mappings from R to X. Besides, F denotes the set of all mappings from X to R.

We recall the following well known result:

Theorem 2.2 (See, for example, Crouzeix (2), chapter 3.)

The mapping Φ : F → T u.l.s. defined by

∀f ∈ F, ∀λ ∈ R, Φ(f )(λ) = S λ (f ) := {x ∈ X, f (x) ≥ λ} is bijective and Ψ = Φ -1 is defined by ∀T ∈ T u.l.s. , Ψ(T )(x) = sup{λ ∈ R | x ∈ T λ }.
Remark 2.3 Condition iii) of Definition 2.1 guarantees that f takes finite values. In ( 2) is given a version of Theorem 2.2 for quasiconvex mappings whose values can be infinite, so that Condition iii) is not useful. The case of Theorem 2.2 here given is straightforward.

Now, we would like to refine the previous representation result, in order to obtain some continuity of this representation.

Definition of a topology on T

In the following Proposition 2.4, whose proof is given in the appendix, we define a generalized metric structure on T u.l.s. . Recall that for every set E, a mapping δ : E × E → R + ∪ {+∞} is a generalized pseudo metric if these three conditions are true:

(i) for every x ∈ E, d(x, x) = 0.

(ii) for every (x, y) ∈ E × E, d(x, y) = d(y, x).

(iii) for every (x, y, z) ∈ E 3 , one has d(x, z) ≤ d(x, y) + d(y, z).

The generalized pseudo metric d is a generalized metric if in addition, for every (x, y) ∈ E 2 , d(x, y) = 0 implies x = y.

In the following proposition, by convention, the infimum of an empty set is +∞.

Proposition 2.4 The mapping δ : T × T → R + ∪ {+∞} defined for every (T, T ) ∈ T × T by δ(T, T ) = sup λ∈R {max{inf{a ≥ 0 | T λ ⊂ T λ-a }, inf{a ≥ 0 | T λ ⊂ T λ-a }}}
is a generalized pseudo-metric, and its restriction to T u.l.s. × T u.l.s. is a generalized metric. Remark 2.5 It is easy to build a metric from the generalized metric δ, for example by defining d : T u.l.s. × T u.l.s. → R + by

We now define a topology on T , thanks to the generalized pseudo metric δ defined above. Definition 2.6 Let T op be the topology on T induced by the previous generalized pseudo metric δ. More precisely, this topology is generated by the collection of open sets B(x, r) = {y ∈ T , δ(x, y) < r} where x ∈ T and r > 0.

Throughout this paper, T is equipped with the topology T op, and the topological space such defined will be denoted (T , δ). Notice that from the properties satisfied by the generalized pseudo-metric δ, the collection of sets B(x, r) is a basis of T op: it means that every open set for T op is a union of some B(x, r).

Since we aim at representing continuously quasi-concave mappings from X to R by elements of T u.l.s. , we now have to define a topology on F.

Definition 2.7 For every (f, g) ∈ F, define f -g ∞ = sup x∈X | f (x) -g(x) |.
Let T op be the topology on F generated by the collection of open sets B(f, r) = {g ∈ F, f -g ∞ < r}, where f ∈ F and r > 0.

Remark that . ∞ is not a norm. Nevertheless, (F, . ∞ ) will denote the set F equipped with the topology U. Notice that the collection of sets B(x, r) is a basis of T op .

The continuous representation

Let F q be the set of all quasi-concave mappings in F, and for every > 0, let F q be the set of all -quasi-concave mappings in F. The sets corresponding to F q and F q through our representation will be T conv and T conv : the set T conv is the set of all multivalued mappings of T u.l.s. with convex (possibly empty) values; the set T conv is defined as follows:

Definition 2.8 The set T conv is the set of all multivalued mappings of T u.l.s. such that for every λ ∈ R, one has coT λ ⊂ T λ-.

We will see in the next section the interpretation of this last condition in terms of distance, which is roughly that the multivalued mapping T and its convex hull coT are not too far. Throughout this paper, all the subsets of F and of T are equipped with the topologies induced by T op and T op.

We now state the following refinement of Theorem 2.2:

Theorem 2.9 i) The mapping Φ : (F, . ∞ ) → (T u.l.s. , δ) defined by

∀f ∈ F, ∀λ ∈ R, Φ(f )(λ) = S λ (f ) := {x ∈ X, f (x) ≥ λ} is an isometric isomorphism, i.e. for every (f, g) ∈ F × F, one has f - g ∞ = δ(Φ(f ), Φ(g)).
ii) Φ is an isometric isomorphism from F q to T conv , and also from F q to T conv .

Proof. See the appendix.

The Convex hull operator

In this section, one defines the convex hull operator co from the set of all multivalued mapping from R to X to itself as follows: for every multivalued mapping T from R to X and for every λ ∈ R, (coT ) λ is the convex hull of T λ .

We have seen in the introduction that a natural method to build the quasiconcave envelop of a mapping f is to take the convex hull coΦ(f ) of the upper level sets of f , then to define a new quasi-concave mapping Φ -1 (coΦ(f )), if it is possible. A problem is that the multivalued mapping coΦ(f ) may not be in T u.l.s. , so that Φ -1 (coΦ(f )) could be not well defined. The following example provides a u.l.s. multivalued mapping T from R to R whose convex hull is not u.l.s.

Example 1 Let T be the multivalued mapping from R to R defined by

T x = [x, -x] ∪ ([0, 1[∩ Q) if x ≤ 0 T x = ([0, 1 -x[∪[x, 1[) ∩ Q if x ∈ [0, 1[ T 1 = {0} T x = ∅ if x > 1
It is easy to check that T ∈ T u.l.s. . Yet, the multivalued mapping coT is defined by

coT x = [x, -x] if x ≤ -1 coT x = [x, 1[ if x ∈] -1, 0] coT x = [0, 1[ if x ∈ [0, 1[ coT 1 = {0} coT x = ∅ if x > 1
and clearly, coT does not satisfy Property ii) of Definition 2.1 at x = 1 (left continuity), so is not u.l.s.

Example 2 Let f be the mapping from R 2 to R defined as follows: for every (x, y) ∈ R 2 , f (x, y) is the opposite of the Euclidean distance between (x, y) and the set

K = {(0, 0)} ∪ {(x, 1 x ) ∈ R 2 , x < 0}.
The mapping f is continuous, Φ(f )(0) = K and coΦ(f )(0) = coK is not closed. Now, for every integer n > 0, one has

coK ⊂ coΦ(f )(- 1 n ), so that n∈N * coΦ(f )(- 1 n ) = coΦ(f )(0),
and so the multivalued function coΦ(f ) is not u.l.s.

So, the first task to avoid this problem is to extend Ψ = Φ -1 on a larger set. We prove that Ψ is, as a matter of fact, an isometry on all the set T :

Proposition 3.1 ∀T ∈ T , Ψ(T )(x) = sup{λ ∈ R | x ∈ T λ }
is well defined, and Ψ is an isometry. More precisely, for every (T, T ) ∈ T × T , one has Ψ(T ) -Ψ(T ) ∞ = δ(T, T ).

Proof. See the appendix.

Unfortunately, there could remain some multivalued mapping T ∈ T u.l.s. such that coT is not in T , as shown in the following example:

Example 3 Let T be the multivalued mapping from R to R defined by

T x =] -∞, -x] ∪ [x, +∞[ if x ≥ 0 T x = R if x ≤ 0 It is clear that T ∈ T u.l.s. , but coT / ∈ T , because x∈R coT x = R.
So, the second task is to find a subset T of T such that for every T ∈ T , coT ∈ T . This is the aim of the following definition:

Definition 3.2 Define T the set of multivalued mappings of T such that

λ∈R coT λ = ∅.
It is clear from the previous definition that co(T ) ⊂ T . Remark that for every bounded mapping f :

X → R, Φ(f ) ∈ T , because Φ(f )(n) = ∅ for n ∈ N large enough.
The following proposition relates this new set T to T conv : Proposition 3.3 For every > 0, T conv ⊂ T .

Proof of Proposition 3.3 For every > 0, let T ∈ T conv . One has λ∈R coT λ ⊂ λ∈R T λ-(from the definition of T conv ). This last set is also λ∈R T λ = ∅ from Property iii) of the definition of T u.l.s. .

To finish, notice that each point of T conv is almost stabilized by the convex hull operator:

Proposition 3.4 For every T ∈ T conv , δ(co(T ), T ) ≤ .
Proof. This is clear from the definition of T conv .

Stability of quasi-concavity

To illustrate the utility of the previous results, one proves the stability of the quasi-concave property.

Let f ∈ F q an -quasi-concave mapping from X to R. By Theorem 2.9, Φ(f ) ∈ T conv , so Φ(f ) ∈ T from Proposition 3.3. Since co(T ) ⊂ T , one has co(Φ(f )) ∈ T . Hence, from Proposition 3.1, f :

= Ψ(co(Φ(f ))) is well defined. Besides, since Ψ(Φ(f )) = f , one has f -f ∞ = Ψ(co(Φ(f ))) - Ψ(Φ(f )) ∞ . Now, from Proposition 3.1, Ψ is an isometry on T ; thus, one has Ψ(co(Φ(f ))) -Ψ(Φ(f )) ∞ = δ(co(Φ(f )), Φ(f )) ≤ from Proposition 3.4 applied to T = Φ(f ) ∈ T conv .
Thus, one obtains a result similar to Hyers-Ulam's one for quasi-concave mappings:

Proposition 4.1 For every -quasi-concave mapping f : X → R there exists f : X → R a quasi-concave mapping such that f -f ∞ ≤ .

Appendix

Proof of Proposition 2.4. First notice that by definition, δ(T, T ) is positive for every (T, T ) ∈ T × T , and may be infinite. Besides, δ(T, T ) = 0 for every T ∈ T , and δ is clearly symmetric. To prove the triangular inequality, let (T, T , T ) ∈ T 3 such that δ(T, T ) < +∞ and δ(T, T ) < +∞ (if one of these two inequalities are not true, then the inequality δ(T, T ) ≤ δ(T, T ) + δ(T , T ) is clearly true). First, from the definition of δ(T, T ) and from the monotony assumption satisfied by T and T , one has

∀λ ∈ R, ∀ > 0, T λ ⊂ T λ-δ(T,T )-. (1)
Second, one can apply Equation 1 to T and T , replacing λ by λ-δ(T, T )-. One obtains

∀λ ∈ R, ∀ > 0, T λ-δ(T,T )-⊂ T λ-δ(T,T )-δ(T ,T )-2 . (2)
Combination of Equations 1 and 2 gives, for every λ ∈ R and every > 0,

T λ ⊂ T λ-δ(T,T )-δ(T ,T )-2 coS λ (f ) ⊂ S λ-(f ), one finally obtains x ∈ S λ-(f ), which is equivalent to f (x) ≥ min{f (x 1 ), ..., f (x n )} -. This proves that f is -quasi-concave.
Finally, Φ is an isometric isomorphism from F q to T conv . Proof of Proposition 3.1 Notice that Ψ is clearly well defined on T , and not only on T u.l.s. . First prove that δ(T, T ) ≤ Ψ(T ) -Ψ(T ) ∞ .

By definition of δ(T, T ), there exist two sequences (λ n ) n∈N and (µ n ) n∈N in R, where (µ n ) n∈N converges to δ(T, T ), and such that for every integer n, T λn is not contains in T λn-µn . It means that for every integer n, there exists x n ∈ X such that x n ∈ T λn and x n / ∈ T λn-µn . Thus, by definition, for every integer n one has Ψ(T )(x n ) ≥ λ n and Ψ(T )(x n ) ≤ λ n -µ n .

Thus, Ψ(T ) -Ψ(T ) ∞ ≥ µ n , and one obtains at the limit Ψ(T ) -Ψ(T ) ∞ ≥ δ(T, T ). Notice that this proof works if δ(T, T ) = +∞. Now prove that δ(T, T ) ≥ Ψ(T ) -Ψ(T ) ∞ .

Suppose this last inequality is false, i.e. δ(T, T ) < Ψ(T ) -Ψ(T ) ∞ . Thus, there exists x ∈ X such that for > 0 small enough, one has Ψ(T )(x) -Ψ(T )(x) > δ(T, T ) + (switching T and T is necessary). Thus, from the definition of Ψ(T )(x), one has x / ∈ T Ψ(T )(x)-δ(T,T )-.

But from the definition of Ψ(T )(x), one also have

x ∈ T Ψ(T )(x)-2 .
From the definition of δ(T, T ), it implies δ(T, T ) ≥ (Ψ(T )(x)-2 )-(Ψ(T )(x)δ(T, T ) -) = δ(T, T ) + 2 , a contradiction.

The representation of a mapping by its upper level sets is a well nown process (see, for example, Crouzeix (2), chapter 3.) In this paper, we are more interested in the continuity of such representation.

Other constructions of the quasi-concave envelop could be given. But we think the construction here given, which is very natural, raises interesting questions we would like to answer in this paper. 2 Document de Travail du Centre d'Economie de la Sorbonne -2009.62

Document de Travail du Centre d'Economie de la Sorbonne -2009.62

and similarly, by symmetry, T λ ⊂ T λ-δ(T,T )-δ(T ,T )-2 .

Thus, from the definition of δ(T, T ), for every > 0 one has δ(T, T ) ≤ δ(T, T ) + δ(T , T ) + 2 which entails the triangle inequality.

Thus, δ is an generalized pseudo metric, that is to say a pseudo metric that can take infinite values.

Besides, if δ(T, T ) = 0 for some (T, T ) ∈ T u.l.s. × T u.l.s. , then for every λ ∈ R, one has

But from Property ii) of Definition 2.1, this last set is equal to T λ ; similarly, one obtains, for every λ ∈ R, T λ ⊂ T λ , and finally T = T . Thus, the restriction of δ to T u.l.s. × T u.l.s. is a generalized metric. Proof of Theorem 2.9 Statement i.) One has to prove that for every (f, g)

First case: finite. We want to prove that δ(Φ(g), Φ(f )) = .

First, prove the inequality δ(Φ(g), Φ(f )) ≤ . This would be a consequence of the fact that for every λ ∈ R,

To prove the first inclusion, let

The second inclusion is proved similarly. Now, prove the converse inequality δ(Φ(g), Φ(f )) ≥ . If this inequality is false, then there exists a > 0 such that

From the definition of Ψ, for every x ∈ X there exists a sequence (λ

which is equivalent by definition to g(x) ≥ λ n -+ a.

Passing to the limit, one obtains g(x) ≥ f (x) -+ a, and similarly, one proves f (x) ≥ g(x) -+ a. Thus, f (x) -g(x) ∞ < , a contradiction with the definition of .

Second case: = +∞. We want to prove that δ(Φ(g), Φ(f )) = +∞; suppose that δ(Φ(g), Φ(f )) < A for some A ∈ R. From the definition of Ψ, for every x ∈ X, there exists a sequence (λ n

or equivalently, for every integer n, g(x) ≥ λ n -A.

Passing to the limit, one obtains g(x) ≥ f (x) -A, and similarly, one proves f (x) ≥ g(x) -A. Thus, f (x) -g(x) ∞ ≤ A, a contradiction with the definition of . Statement ii.) It is well known that f is quasi-concave if and only if its upper level sets are convex; thus, Φ is an isometric isomorphism from F q to T conv . Now, let us prove that f is -quasi-concave if and only Φ(f ) ∈ T conv . First suppose that f is -quasi-concave. Let λ ∈ R and x ∈coS λ (f ). Thus, there exists an integer n and x 1 , ..., x n in S λ (f ) such that x = n i=1 µ i x i , with µ i ≥ 0 for every i = 1, ..., n and n i=1 µ i = 1. From the definition of S λ (f ), for every i = 1, ..., n one has f (x i ) ≥ λ; since f is -quasi-concave, one obtains f (x) ≥ λ -, or equivalently x ∈ S λ-(f ). We have proved that for every λ ∈ R, one has coS λ (f ) ⊂ S λ-(f ), or equivalently, Φ(f ) ∈ T conv .

Conversely, suppose coS λ (f ) ⊂ S λ-(f ) for every λ ∈ R. Let x = n i=1 µ i x i , with µ i ≥ 0 for every i = 1, ..., n and n i=1 µ i = 1. Define λ = min{f (x 1 ), ..., f (x n )}, so that one has f (x i ) ≥ λ for every i = 1, ..., n; equivalently, this means x i ∈ S λ (f ) for every i = 1, ..., n. Consequently, one has x ∈ coS λ (f ). From


H Ouertatani
email: houssem.ouertatani@irt-systemx.fr

C Maxim

S Niar

E-G Talbi

Bayesian optimization for NAS with pretrained deep ensembles

Introduction

Neural architecture search (NAS) is the use of a search algorithm to find, according to one or many criteria, the most suitable neural architecture for a given task, within a search space of neural network architectures. Among prominent search methods used in NAS literature, we can find reinforcement learning (RL) [START_REF] Hsu | MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning[END_REF], evolutionary algorithms (EAs) [START_REF] Elsken | Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution[END_REF][START_REF] Lu | NSGA-Net: neural architecture search using multi-objective genetic algorithm[END_REF], and bayesian optimization (BO) [START_REF] White | BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search[END_REF][START_REF] Cai | TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks[END_REF].

In this paper, we present a BO-based method, where instead of using Gaussian Processes which are usually associated with BO, we use deep ensembles as performance predictors for candidate neural networks. Specifically, we explore the potential of pretraining the ensemble networks as a way to mitigate their need for more data compared to GPs

The idea is to use pretraining to accelerate the training of the deep ensemble, in order to obtain a good prediction performance early on in the optimization process. In the case of NAS, we have access to zero-cost metrics, which can be calculated quickly and without the need to train the candidate networks, by far the most costly part of evaluating them.

By using existing and widely used NAS benchmarks, we show the improvement pretraining brings to the deep-ensemble based method, as a result accelerating the search process.

Bayesian optimization for NAS with deep ensembles

At a glance, the description of the procedure is as follows. At each iteration, the best points to evaluate are selected, according to a criteria calculated using the prediction of the ensemble, and which balances exploration and exploitation, known as the acquisition function. In our work, we used probability of improvement, ie. the probability of improving over the current best observed value. Subsequently, these selected candidates are observed or evaluated, ie. fully trained and tested on the task. Their evaluation data is then added to the training data for the deep ensemble, which is updated for the next iteration. Figure 1 describes the overall procedure of using a deep ensemble for BO. Before starting the search, the deep ensemble is first pretrained on zero-cost metrics. In the following section, we describe the pretraining procedure for the deep ensemble.

Pretraining step

In order to increase the sample efficiency obtained with the deep ensemble, we pretrain them on zero-cost metrics, ie. metrics whose measurement doesn't require the prior training of the candidate networks. Among such metrics we mention FLOPs, number of trainable parameters, or the average time of a forward pass (or latency).

The idea is to force the last layer of the deep ensemble networks to have meaningful representations of the architectures in the search space. The embeddings generated by these last layers for input architectures need to hold relevant information that helps in predicting the eventual performance of these architectures once the BO iterations begin.

In order to achieve this, we use a separate prediction head for each of the metrics we are using as targets. Then, the weights of the deep ensemble networks are updated using backpropagation. The forward pass and optimization steps are performed for all the different metrics at the same time, as we found this achieved better performance. Our intuition in explaining this is it ensures the last layer doesn't specialize on one of the target metrics, instead being as general as possible.

Figure 2 describes the pretraining procedure.

First results

Our tests were performed on NASBench101, NASBench201, and NATS-Bench-size benchmarks. These benchmarks can be queried to obtain the accuracies, as well as other measurements, for the networks described in their search spaces. One of the most used ways to compare predictor-based NAS methods is the correlation or ranking correlations between the predictions, and the real ranking of the networks. Table 1 shows the difference in ranking correlations measured by the Kendall-Tau and Spearman-Rho ranking correlations, and the Pearson-R correlation of the accuracies. The first two columns also report the best found accuracy on CIFAR10 and CIFAR100 within the evaluations budget. To situate the correlations with regards to other methods in the literature, we can mention the paper describing BossNAS, which reports a Kendall-tau measure on the same benchmark of τ = 0.59, a Spearman-Rho of ρ = 0.76, and a Pearson-R of R = 0.79.

We can also plot the evolution of the best observed value, and of the Spearman-Rho correlation, as the number of evaluations performed increases.

Conclusion and further perspectives

In this work, we have presented a method to use bayesian optimization powered by deep ensembles for the NAS problem. One of the advantages of using a deep ensemble is the ability to easily pretrain it and keep the weights for use in the bayesian optimization procedure. Pretraining allows us to gain in sample efficiency, and enables the deep ensemble to produce good quality predictions from fewer datapoints.

This work is a first step which will be built upon for multi-objective NAS, to produce energyefficient and low-latency models. The deep ensemble can be used in a similar fashion as it was used in the pretraining step described previously: the weights of the initial parts of the network could be shared and used to produce predictions for the different objectives, potentially unlocking further sample efficiency by the inclusion of additional information. Our future work will focus on multiobjective aspects, both in terms of improving the search method for multi-objective optimization, and in terms of the objectives that will be optimized.

Fig. 1 .

 1 Fig. 1. Bayesian optimization for NAS with deep ensembles

Fig. 2 .

 2 Fig. 2. Pretraining procedure for one network of the deep ensemble networks

  Results obtained on NAS-Bench-101, NAS-Bench-201 and NATS-Bench-Size are competitive with state-of-the art predictor-based methods, getting close to or better results in most datasets offered by the benchmarks (CIFAR10, CIFAR100 and ImageNet). Further improvement of the architecture of the ensemble networks or their training hyperparameters can potentially offer further performance gains.

Table 1 .

 1 Comparison of results on NATS-Bench-size with and without pretraining

		C-10 C-100	τ	ρ	R
	No pretraining	92.99 69.92 0.46 0.65 0.61
	With pretraining 93.65 70.72 0.73 0.90 0.938