Yujia Liu

Mana Kigure

Riho Okawa

Nobuhiro Takeda

Masafumi Unno

Armelle Ouali

Synthesis and characterization of tetrathiol-substituted double-decker or ladder silsesquioxane nano-cores

come   L'archive ouverte pluridisciplinaire

Tetra(3-mercaptopropyl)-silsesquioxanes with double-decker (DDSQ) or ladder nano-cores were easily prepared from the corresponding tetraallyl derivatives through fast and convenient thiol-ene reactions. An additional tetrathiol-DDSQ with more flexible arms was also synthesized in high yield from the corresponding tetrachloro-DDSQ derivative. The three novel tetrathiol silsesquioxanes described represent versatile building blocks for the preparation of hybrid organic-inorganic materials.

Hybrid molecules combining the thermostability of inorganics and the chemical flexibility of organics at the nanometer scale attract growing interest in various areas of chemistry. Along these lines, polyhedral oligomeric silsesquioxanes (POSSs) composed of an inner rigid inorganic Si-O-Si core surrounded by organic reactive groups constitute a versatile platform to design various types of functional hybrid materials. [1][2][3] Among them, the cubic silsesquioxanes (T 8 -cages) have been widely studied and shown to have a great potential in numerous fields including catalysis, 4 photoactive materials, 5 nanocomposites based on POSS and biomass 6 or coating materials 7 to name a few. [7][8][9] Among functionalizable T 8 -cages, those decorated with thiol functions revealed key building blocks to design multifunctional materials. Indeed, the SH group can react with readily available olefins through thiol-ene reactions that are expedited synthetic processes with quantitative yields and short reaction times. 10 These "click reactions" occur in the presence of a radical initiator and/or upon irradiation.

Although the preparation of mono-or di-substituted cages bearing one (or two) SH function(s) have been reported, their application seems still limited. [11][12][13] On the contrary, the related octathiol T 8 cage 14,15 received great attention. Indeed, the presence of several reactive thiol groups confers multivalency that constitutes a crucial parameter in many applications. For example, octathiol-T 8 -POSS has been used as dendrimers cores 16 to reach molecular materials relevant in the fields of energy 17 or optics. 18 Multivalency is also a strong asset in the field of nanomedicine. 19,20 The octathiol POSS was also used to access disulfide-based organic/inorganic hybrid dynamic networks with self-healing 21 or sensing 22 properties. In most of the reported cross-linked systems, the vertices of the cage are connected by polymers chosen depending on the targeted application. The POSS units confer high thermal and mechanical stabilities to the corresponding hybrid materials that find applications in areas as diverse as UV-curable nanocomposite coatings, hydrogels for tissues engineering scaffolds, conducting elastomers for lithium batteries, chromatography or sensing. Most of these hybrids were prepared through thiol-ene photopolymerization that constitutes an attractive alternative technology since faster than sol-gel chemistry, free radical or ring-opening polymerization and less sensitive towards oxygen. 48 Besides, POSSs cores have been shown to be important building blocks to construct and drive the assembly of amphiphilic hybrid POSSs. 49 The outstanding properties of the resulting materials find their origin in the self-organisation of the molecular fragments including the inorganic T 8 -head and the organic tails. Along these lines, octathiol-T 8 -POSSs occupy a key place and were functionalized with hydrophilic [50][51][52][53] or hydrophobic fragments [54][55][56][57][58][59] to reach materials employed for sensing, in photocurable systems. or for preparing hybrid microcapsules.

60-61

Also, carbazole substituted T8-POSSs prepared from the corresponding thiol derivatives and 9vinylcarbazole exhibit very good thermal, mechanical and photochemical properties. 62 To sum up, the versatility offered by reactive thiol groups present on each corner of T 8 -POSS gave rise to a feature-rich platform opening opportunities in various application fields. Given the increasing interest of hybrid materials with good mechanical and thermal properties in science and industry, the design of functionalizable POSSs with different topological structures appears as a research hotspot. Indeed, varying the number of reactive groups and the inorganic core structure will give access to new nanocomposites displaying different properties.

Scheme 1. Synthesis of the tetra-(3-mercaptopropyl)-DDSQ 3.

In this work, the synthetic routes to three unprecedented tetrathiol silsesquioxanes are described. Two of them are based on D 2 T 8 -double decker silsesquioxane (DDSQ) structure and the third one involved ladder oligosilsesquioxanes with defined structures (called "laddersiloxanes"). The resulting compounds were prepared in fair to very good yields from inexpensive precursors and both DDSQ derivatives could be prepared in gram scale. These three functionalizable nanohybrids are complementary to those derived from the T 8cage and are particularly relevant for preparing a wide range of materials. Difunctional D 2 T 8 and tetrafunctional M 4 T 8 DDSQ bearing two or four functional groups respectively have been widely described and used to prepare various materials with appealing properties. 63 More recently, we reported the first D 2 T 8 structures bearing four reactive substituents including vinyl, allyl, chloro, iodo or azido groups. Therefore, the previously described tetraallyl-DDSQ 1 was allowed to react with thioacetic acid in the presence of AIBN as the radical initiator (Scheme 1). After 16h at 60 °C in toluene, quantitative functionalization occurred and the expected tetrakis(3-propyl ethanethioate) derivative 2 could be isolated in 92% yield after washing crude product with hexane. On reduction with lithium aluminium hydride followed by workup with hydrochloric acid, the corresponding tetrathio-DDSQ 2 was isolated in 60% yield (Scheme 1). 66 This yield could be improved to 65% by using 15 equivalents of LiAlH 4 instead of 10 and by increasing the reaction time from 10 to 15 minutes. Compound 2 was characterized by multinuclear NMR and mass spectroscopies (see supporting information). The 29 Si spectrum of 3 (Figure 1a) displayed two singlets for the non-equivalent T-unit silicon atoms of the core.

The resonance at -79.44 ppm was assigned to the internal Tgroup silicon atoms (Si in green) and that at -78.59 was assigned to the T-group silicon atoms (red color) nearest to the D-group silicon atoms (blue color). The latter gave rise to a singlet at around -20.08 ppm consistent with chemical shifts previously reported for related D 2 T 8 -DDSQs with alkyl substituents. 64, 65 1 H NMR spectrum showed the four resonances expected for the mercaptopropyl substituents with chemical shifts and multiplicity in accordance with those reported for octa(3-mercaptopropyl)-silsesquioxane (-CH 2 (as compared to the D-group silicon atom) at 0.81 ppm; SH at 1.06 ppm appearing as a triplet ( 3 J H-H = 7.9 Hz); -CH 2 at 1.71 ppm and -CH 2 at 2.41 ppm). 14,15 The data obtained from 1 H, 13 C NMR and MALDI-TOF mass spectroscopy were in accordance with the molecular structures (see supporting information).

Single crystals were grown and the structure of 3 determined by X-ray crystallography (Figure 2). This structure displays similar framework to the parent diallyl-DDSQ 1, with Si-O bond lengths in the ranges of 1.600(3)-1.633(3) Å (average 1.617(6) Å) for 3 as compared to 1.607(9)-1.634(8) Å (average 1.619(9) Å) for 1. Si-C lengths, in the ranges of 1.842-1.855 Å (average 1.848 Å), were also consistent with those found for related compound 1 (in the ranges of 1.845(6)-1.869(2) Å (average 1.852(6) Å)). The C-S lengths were found in the range of 1.807(3)-1.827(3) Å (average 1.817(3) Å). Generally, the Si-O, Si-C and C-S bond lengths in 3 were comparable with those reported for the related octakis(3-mercaptopropyl)-T 8 -POSS. 15 The internal Si…Si distances were also measured (see Table S2). The interatomic Si…Si distances were found in the range of 67 For the DDSQ cage 3, higher interatomic

Please do not adjust margins Please do not adjust margins Si…Si distances and larger deviations are thus observed as compared to the T 8 -cage. These observations may be rationalized by the larger size and the enhanced flexibility of the DDSQ nanocore as compared to the rigid cubic cage. 68 In addition, in 3, the Si-O-Si bond angles vary from 136.44 to 176.62° which corresponds to the range encountered for 1 (138.8(8)°-166.3(7)°) and previously reported DDSQ resulting from the hydrosilylation of the C=C bond. [64][65] The large deviation of the bond angles as compared to most of T 8scaffolds (standard deviation < 15 °) indicates the flexible structure of the frameworks. 1 In the case of the octathiol-T 8cage, adjacent molecules were found to be associated in onedimensional supramolecular chains due to intermolecular S-H…S hydrogen bonds. S3). In an endeavour to access tetramercapto-DDSQ with longer arms and increased flexibility, previously described DDSQ 4 decorated with four chloro substituents and obtained quantitatively from 1 in one step 65 was allowed to react with potassium thioacetate. The resulting tetra-thioacetate DDSQ 5 was isolated in 91% yield after 16h at 22 °C (Scheme 2). 69 DDSQ 5, prepared in gram scale, characterized by multinuclear NMR and mass spectroscopies (see supporting information) was used without further purification and converted into the expected tetra-mercapto-DDSQ 6 by using typical reducing conditions with lithium aluminium hydride (Scheme 2). 66 DDSQ 6, obtained in 86% yield, was fully characterized. By 29 Si NMR and as for compound 3 (Figure 1b), the non-equivalent silicon atoms of the core gave rise to two distinct singlets at -79.15 (Si in green) and -78.34 (Si in red) ppm. The signal at -19.27 ppm was assigned to the external D-group silicon atoms (Si in blue) while the singlet at 3.87 corresponded to the carbosilane (Si in pink). The C NMR, elemental analysis and MALDI-TOF mass spectroscopy also confirmed the structure and the purity (see supporting information), with for example an experimental mass found at 1733.79 g.mol -1 for [M+Na + ] in accordance with the calculated one at 1734.00 g.mol -1 . Crystals could not be grown from 6 which is likely due to the inherent flexibility of this compound that was designed for this purpose. Moreover the carbosilane arms enabled to tune the solubility since 6 was more soluble in nonpolar solvents such as hexane or 1-hexene than the related (3-mercaptopropyl) derivative 3. Given the importance and the versatility of the thiol reactive groups and in order to create novel polymercapto-POSS with different core structure, we next planned to prepare tetra(3mercaptopropyl)-laddersiloxanes. 70-72 To this end, the previously described tetraallyl laddersiloxane 7 72 was quantitatively transformed into the corresponding tetrakis(3propyl ethanethioate) 8 derivative in the presence of thioacetic acid and AIBN as the radical initiator (Scheme 3). After washing the crude product with hexane, 8 was isolated in 91% yield, fully characterized and used without further purification in the next step. The latter, consisting in a reduction with lithium aluminium hydride followed by an acidic work-up, yielded the expected tetrathiol 9 in 38% yield after column chromatography. The lower yield in the case of this ladder siloxane as compared to the DDSQ might be explained by the higher sensitivity of the core towards the reducing agent. Indeed, the temperature and reaction time had to be finely tuned to complete the deacetylation while limiting the degradation of the ladder framework. The use of milder reducing agents might enable to improve the yield. 73 Multinuclear NMR spectroscopies were performed to characterize laddersiloxane 9. By 29 Si NMR spectrum (Figure 1c), the singlet corresponding to the T-group silicon atoms was found at -67.49 ppm (Si in red). The 29 Si resonance at -6.67 ppm has been assigned to D-groups silicon atoms (Si in blue) and this value was found to be consistent with the one reported for related ladder-like structures with four alkyl substituents and resulting for example from the hydrosilylation of 7 with dimethylphenylsilane. 72

Conclusions

In this work, tetrakis(3-mercaptopropyl)-silsesquioxanes with double-decker (DDSQ) (3) or ladder (9) nano-cores were successfully prepared from the corresponding tetraallylderivatives through straightforward thiol-ene reaction with thioacetic acid and subsequent reduction. An additional tetrathiol-DDSQ with more flexible arms (6) was also synthesized from the corresponding tetrachloro-DDSQ derivative through nucleophilic substitution with potassium thioacetate followed by a reduction step. To the best of our knowledge, 3, 6 and 9 represent the first examples of welldefined functionalizable DDSQ and ladder silsesquioxanes bearing thiol reactive groups possibly further functionalized through thiol-ene reaction but also by resorting to rarely used yet also attractive and complementary thiol-yne reactions 36,37 or Michael additions. 24 They display significantly different topological structures than the well-described and widely used octakis-(3-mercaptopropyl)-octasilsesquioxane (T 8 -cage). With their different core structures, we believe that the three tetrathiol-silsesquioxanes reported here will present new opportunities in all the numerous areas where the related octathiol-T 8 -cage found applications. They thus constitute promising building blocks for the preparation of unprecedented hybrid organic-inorganic materials including cross-linked materials, 23-48 challenging self-assembly systems, 49,75 dendrimers or multifunctional silicon-based drug. 66 It might also be possible to take advantage of the thiol groups to stabilize quantum dots or metal surfaces.

76-77

Moreover, all tetrathiol-silsesquioxanes reported here displayed a very high solubility in a wide range of solvents including chloroform, dichloromethane, tetrahydrofuran, ether and toluene. Enhanced solubility is of particular importance to design optic devices for which solution-processable photoluminescent hybrid polymers are needed.

The functionalization of compounds 3, 6 and 9 to reach hybrid catalytic materials is for example currently underway in our laboratories.

Figure 1 .

 1 Figure 1. 29 Si NMR spectra (in CDCl3) for the DDSQs 3 and 6 (a, b respectively) and for Ladder 9 (c). Chemical shifts are in ppm unit.

  3.0984-4.6241 Å (average 3.3883 Å) for the o-Si…Si, 4.2069-5.2974 Å (average 4.7371 Å) for the m-Si…Si and 5.0421-6.6830 Å (average 5.8303 Å) for the p-Si…Si. These average values reported for related T 8 -cages are found in the range of 3.11-3.16 Å, 4.40-4.45 Å and 5.39-5.43 Å for the o-, m-and p-Si…Si respectively.

15 There

 15 are no remarkable intermolecular interactions in the crystal structure of 3, although some C-H•••S, C-H•••O, and C-H•••C distances are within the sum of van der Waals radii (see Table

Figure 2 .

 2 Figure 2. Crystal structure of tetra-(3-mercaptopropyl)-DDSQ 3. Black: carbon; blue: silicon; red: oxygen; yellow: sulphur.

1 H 3 J

 13 NMR spectrum displayed a triplet at 0.93 ppm ( H-H = 7.1 Hz) integrating for 4 protons that clearly showed the presence of the thiol SH groups. The four methylene groups gave rise to four resonances at 0.59, 0. 81, 0. 94 and 1.41 ppm. Other analytical techniques including13

Acknowledgments

The authors acknowledge the Gunma University Initiative for Advanced Research (GIAR) for funding the Open International Laboratory and the grant to Y. L, CNRS and New Energy and Industrial Technology Development Organization (NEDO) for funding, and JASSO for M. K. travel grant to Montpellier.

Conflicts of interest

There are no conflicts to declare.


Nathanaël Carraz Rakotonirina

Andry Rasoanaivo
email: r.andry.rasoanaivo@gmail.com

Laurent Najman
email: laurent.najman@esiee.fr

Petr Kungurtsev
email: pkungurtsev@enthought.com

Jeremy Rapin ¶d

Fabien Teytaud
email: teytaud@univ-littoral.fr

Baptiste Roziere

Olivier Teytaud
email: oteytaud@fb.com

Markus Wagner
email: markus.wagner@adelaide.edu.au

Pak-Kan Wong
email: pkwong@link.cuhk.edu.hk

Vlad Hosu
email: vlad.hosu@uni-konstanz.de

Many-Objective Optimization for Diverse Image Generation

In image generation, where diversity is critical, people can express their preferences by choosing among several proposals. Thus, the image generation system can be refined to satisfy the user's needs. In this paper, we focus on multi-objective optimization as a tool for proposing diverse solutions. Multiobjective optimization is the area of research that deals with optimizing several objective functions simultaneously. In particular, it provides numerous solutions corresponding to trade-offs between different objective functions. The goal is to have enough diversity and quality to satisfy the user. However, in computer vision, the choice of objective functions is part of the problem: typically, we have several criteria, and their mixture approximates what we need. We propose a criterion for quantifying the performance in multi-objective optimization based on cross-validation: when optimizing n-1 of the n criteria, the Pareto front should include at least one good solution for the removed n th criterion. After providing evidence for the validity and usefulness of the proposed criterion, we show that the diversity provided by multiobjective optimization is helpful in diverse image generation, namely super-resolution and inspirational generation.

Introduction

Diversity in image generation is critical. Problems of insufficient representation of some classes can be mitigated by presenting several possibilities to the user, if there is enoough diversity. In the present paper, we propose multi-objective optimization as a tool for facilitating diverse image generation.

Multi-objective optimization (MOO [START_REF] Chand | Evolutionary many-objective optimization: A quick-start guide[END_REF][START_REF] Emmerich | A tutorial on multiobjective optimization: Fundamentals and evolutionary methods[END_REF]) is the simultaneous optimization of several objective functions (OFs). For example, given a request, an internet search method typically ranks a list of possible answers using a combination of OFs. OF can include the relevance of the answer, price, geographical distance, reliability of the source, aesthetics, the technical quality of an image, etc. MOO can assist with this as it obtains a Pareto set of solutions, which is a set of solutions that are not dominated by other solutions (Eq. 1). Then, we can subsample that set and get a list of proposals [START_REF] Carmel | Multi-objective ranking optimization for product search using stochastic label aggregation[END_REF] (Sec. 2.2). The user can then select a proposal. The goal of MOO is to help a user find the best solution according to their unknown OF. It can suggest either a huge but navigable set of solutions, or a small set that the user can exhaustively check: we focus on the latter.

While benchmarking single-objective optimization is already a complicated task, MOO raises additional issues: how to compare two MOO methods? The hypervolume (HV) indicator [START_REF] Auger | Hypervolume-based multiobjective optimization: Theoretical foundations and practical implications[END_REF] (Eq. 2), which measures the dominated volume up to a reference point, is a simple criterion. However, the choice of the reference point is often arbitrary and can completely change the conclusion. In addition, this is not directly related to user satisfaction, and numerical comparisons based on HV might be biased in favor of methods using HV as a proxy.

The fact that we do not have an exact measure of user satisfaction is critical in MOO: if we had a preference function for guessing if the user prefers solution A or solution B, we would optimize that function. MOO precisely depends on having several proxies of the user satisfaction, e.g., cost, quality, maintainability, speed, etc., but not the exact measure. With an exact measure of satisfaction, we would not need MOO in the first place: a principled, practical comparison between MOO must take into account this approximate nature of the OFs.

We present several algorithms for MOO (Section 2.1), including specifically MOO methods, HV-based methods, and MOO variants of Differential Evolution (DE). We run all algorithms from Nevergrad [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF] that can deal with MOO plus our new contributions. We also include several methods for subsampling Pareto sets. We propose a new method termed cross-validation of criteria, based on cross-validating objective functions, for benchmarking MOO (Section 3.1), specifically in the many-objective cases when the OFs are a redundant and partial approximation of the real user preference, so that cross-validating criteria makes sense.

We present computer vision benchmarks (Section 3.2), with, in some cases, a representation of images based on latent variables, and we report on extensive comparative experiments in computer vision (Section 4).

Finally, in Section 4.2, we visualize results for conditional image generation such as inspirational generative adversarial networks [START_REF] Berthelot | BEGAN: boundary equilibrium generative adversarial networks[END_REF] (Section 4.2.1) and super-resolution (Section 4.2.2).

Multi-objective optimization

In this section, we start by presenting MOO and several algorithms that can deal with it (overview of algorithms in Table 1). We then present different methods for subsampling Pareto fronts (Section 2.2). Table 1: Top: different settings considered in the experimental comparisons. Second col: in some cases, we use a latent representation of variables using a GAN (Section 3.3). Each row corresponds to several optimization settings (e.g., one for u =K512 and one for u =Brisque in the first row). k refers to the number of objective functions. u is the ground truth and M is the number of proposed solutions. IQA (image quality assessment) methods are presented in Section 3.2. The number of settings (6 th col) is the cardinal of F (4 th row), i.e., the number of possible u (see Section 3.1). ||z|| refers to the norm of the latent variable: this represents relevance to the domain as a large z means an image far from the center of the domain. D refers to the discriminator. S is a similarity defined in PytorchGanZoo (combining several measures). Bottom: an overview of our optimization methods. Each algorithm is equipped with one of the subsampling methods that extracts a Pareto front of limited size (Section 2.2). Unless stated otherwise, this is IGD.

Multi-objective optimization

Classical numerical optimization is the search for x * ∈ D minimizing some function f : D → R. Given N OFs f 1 , . . . f N , MOO is the (approximate) search for the Pareto front defined by {x * ∈ D such that x ∈ D, x x * }, where (in our minimization context) x x * stands for Pareto-dominance

∀1 ≤ i ≤ N, f i (x) ≤ f i (x * ) ∧ ∃ j, f j (x) < f j (x * ). (1)
Our study does not include all existing MOO. For example, we do not include approaches based on a representation by µ × d variables of a Pareto approximation of cardinal µ in a domain of dimension d [START_REF] Wang | Hypervolume indicator gradient ascent multi-objective optimization[END_REF]. Also it does not consider preference elicitation or interactive MOO, for which the reader is referred to [START_REF] Belton | Multiple criteria decision analysis: an integrated approach[END_REF][START_REF] Greco | Multiple criteria decision analysis[END_REF][START_REF] Xin | Interactive multiobjective optimization: A review of the state-of-the-art[END_REF]. However, [START_REF] Emmerich | A tutorial on multiobjective optimization: Fundamentals and evolutionary methods[END_REF] distinguishes three categories of MOO, all of them represented here: (i) based on indicators, such as the HV (Section 2.

Methods based on the HV

Given n points x 1 , . . . , x n and a reference point y ∈ R N , the HV indicator measures how they are distributed in the objective space. It is defined as

HV (x 1 , . . . , x n , y) = (2) µ({t ∈ R n ; ∀1 ≤ i ≤ n, ∀1 ≤ j ≤ N, f j (x i ) ≤ t j ≤ y j }),
with µ the Lebesgue measure. One can use the HV indicator for converting MOO into single-objective optimization. Importantly, the OF becomes dynamic: the objective value depends on previous points. With HV, we can use many existing algorithms. We refer to [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF] for the full list of methods considered in the present paper; we mention Particle-Swarm Optimization (PSO [START_REF] Kennedy | Particle swarm optimization[END_REF]), Differential Evolution (DE [START_REF] Storn | Differential evolution -a simple and efficient heuristic for global optimization over continuous spaces[END_REF]), Covariance Matrix Adaptation [START_REF] Hansen | Completely derandomized self-adaptation in evolution strategies[END_REF] (F127CMA refers to FCMA version 1.2.7), Diagonal-CMA (DCMA [START_REF] Ros | A simple modification in cma-es achieving linear time and space complexity[END_REF]), the (1 + 1)-ES with one-fifth rule [START_REF] Schumer | Adaptive step size random search[END_REF], NGOpt12 the optimization wizard from Nevergrad automatically selecting a method depending on dimension/budget/type of variables. Inspired by [START_REF] Riviere | Inspirational adversarial image generation[END_REF], we also include algorithms from discrete optimization adapted to the continuous domains such as (1+1)EA. Describing all considered algorithms would be beyond the scope of the present paper: we only give an overview of the (1+1)EA in Alg. 1 because it is unusual in a continuous context. A specificity of methods based on random mutations (such as the one in Alg. 1) is that they prefer flat stable basins rather than peaked small basins of attraction [START_REF] De | Genetic algorithms are not function optimizers[END_REF]. For this reason, they can be expected to generalize better from a proxy of objective function to the real objective function, which might be useful for MOO.

Multiple single-objective runs

When we want to find distinct trade-offs but do not want to bother playing with multiple OFs, we might use:

• MSRH (multiple single runs with handcrafted coefficients): use a human-defined trade-off, and optimize m times, assuming that the presence of multiple local minima will be enough for ensuring diversity so that the user will be happy with at least one of the results. • MSR (multiple single runs): optimize, m times, a randomly chosen linear combination of weights. With NGOpt12 being the default optimizer in Nevergrad, we consider NGOpt12 m the method running m times a single objective optimization with each of the original OFs weighted by a randomly uniformly drawn coefficient in [0, 1].

Methods based on NSGA-II

NSGA-II [START_REF] Deb | A fast and elitist multiobjective genetic algorithm: Nsga-ii[END_REF] is a well-known generic method for transforming an optimization method based on selection and/or crossover into a MOO method. NSGA-II uses the crowding distance as a comparison tool: it prefers non-dominated solutions, but (if needed) it will also prefer solutions that occupy the largest cuboids in the objective space, where the size of the solution's cuboid is defined with respect to its neighbouring solutions.

Variants of differential evolution adapted to MOO

We also use a variant of differential evolution specifically dedicated to MOO. Our DE is based on the differential evolution code in [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF], i.e., a rand-to-best variant, modified using ideas from [START_REF] Robič | DEMO: Differential evolution for multiobjective optimization[END_REF] and [START_REF] Abbass | PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems[END_REF]. DE is adapted to MOO as follows:

• For comparing a point with its parent: each time DE requests such a comparison, DE randomly chooses one of the k objective functions and returns True if the child outperforms its parent for this objective function.

• Selection of the best: each time we need "the" best point in DE, our DE randomly selects one of the Pareto optimal solutions in the population. Nevergrad contains an optimization wizard which automatically chooses an optimization method. It turns out that when it detects that the problem is MOO (i.e., when it receives some objective values), then it switches to our implementation of DE. Therefore, it randomly samples the domain before switching to DE. This is termed DE+ in our results, and performs close to DE.

Approximating Pareto-fronts

To extract 1 ≤ m ≤ n points from an approximate Pareto set {x 1 , . . . , x n }, a range of approaches can be used:

• Random subset: just pick up m of the x i , uniformly at random and without replacement.

• HV: pick up {x j 1 , . . . , x j m } such that their Hypervolume C h is maximal.

• Loss-covering, also known as IGD (inverted generational distance, [START_REF] Sato | Local dominance using polar coordinates to enhance multiobevolutionary algorithms[END_REF]): pick up {x j 1 , . . . ,

x j m } such that C l = ∑ n i=1 inf j≤m ||F(x i ) -F(x i j )|| 2 is minimal, where F(x) = ( f 1 (x), . . . , f N (x)). • Domain-covering: pick up {x j 1 , . . . , x j m } such that C d = ∑ n i=1 inf j≤m ||x i -x i j || 2 is minimal. • Additive epsilon approximation (EPS, [35]): pick up {x j 1 , . . . , x j m } such that C e = max n i=1 inf j≤m ||F(x i ) - F(x i j )|| ∞ is minimal, where F(x) = ( f 1 (x), . . . , f N (x)).
Finding optimal subsets for some of those OFs is known to be NP-hard [START_REF] Bringmann | Maximum Volume Subset Selection for Anchored Boxes[END_REF]. While subset selection is an important research area, it is outside of the scope of this article. We therefore resort to a simple, unbiased approach: we randomly draw 30 subsets and pick the best for the chosen criterion.

Experimental setup: a new criterion and image generation

In the present section, we present tools necessary for the rest of the paper. Section 3.1 presents a new criterion for comparing MOO. Section 3.2 presents our computer vision benchmarks: some of those benchmarks use a latent representation of images, presented in Section 3.3.

Methodology for comparing MOO methods: cross-validating objective functions

There are two main families of MOO. First, sometimes, MOO is interactive [START_REF] Xin | Interactive multiobjective optimization: A review of the state-of-the-art[END_REF]: the user provides feedback, possibly through preference elicitation [START_REF]Multiobjective Optimization: Interactive and Evolutionary Approaches[END_REF], during the MOO run. Second, purely offline MOO: the algorithm is given some OFs c 1 , . . . , c k , where for all i, C i : X → R and outputs a finite set {p 1 , . . . , p M } ⊂ X. Then the user selects p i * ∈ {p 1 , . . . , p M }. We postulate that there exists a u which quantifies the user preference: ∀i ∈ {1, . . . , M}, u(p i * ) ≤ u(p i ). The function u is related to the c 1 , . . . , c k but is not one of them. We focus on this second scenario and propose a method for benchmarking performance in such a case. Many methods not directly connected to user satisfaction have been proposed for quantifying the performance of MOO [START_REF] Bigaud | Evaluating multiobjective evolutionary algorithms using MCDM methods[END_REF]: our criterion is centered on the satisfaction of an unknown preference, related but not equal to the objective functions. More precisely, we will use an auxiliary objective function u : X → R as a ground truth, and use it as follows. Cross-validated criterion (CVC (F, F )) for MOO: given an integer M, a set F of objective functions and a subset F ⊂ F of possible targets, we will consider, for each target u ∈ F , a scenario scenario u,F\{u} as follows.

Given F a finite set of OFs, u one of them, we consider {c 1 , . . . , c k } the set F after removing u. In scenario u,(c 1 ,...,c k ) , we multi-optimize (c 1 , . . . , c k ). The output of the algorithm is then a finite approximation (p 1 , . . . , p M ) of the Pareto set. Our loss is inf 1≤i≤M u(p i )u(p * ), with p * the best solution, for u, met in all our runs in scenario u,(c 1 ,...,c k ) .

Typically, F contains the OFs in F to be used as ground truth. For example, Koncept512 [START_REF] Hosu | Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment[END_REF] is the best approximation of opinion scores provided by humans in terms of quality assessment, so it is in F , whereas pure Blurriness is not. Table 1 presents several such contexts used in our experiments.

Diverse computer vision

Cheng et al. [START_REF] Cheng | Test problems for large-scale multiobjective and manyobjective optimization[END_REF] mention how artificial problems studied in the MOO literature are easier than real-world ones, which entails misleading benchmark results. We consider computer vision tasks combining some of the following categories of objective functions: quality assessment (often combining several scores), discriminator score, distance to the target, user feedback, and distance to the domain of latent variables. We consider the following similarity measures: S = {distance between histograms, L1-distance, L2-distance, Lpips-AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF], Lpips-Vgg [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]}. We consider the following quality measures: Q = {Blurriness, Brisque [START_REF] Mittal | No-reference image quality assessment in the spatial domain[END_REF], Koncept512 [START_REF] Hosu | Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment[END_REF]}. To compare MOO algorithms, we use either the HV or our proposed crossvalidated criterion (Section 3.1). We consider many settings, all for computer vision, as described in Table 1: • Image generation assisted by image quality assessment, as in e.g. [START_REF] Roziere | Evolgan: Evolutionary generative adversarial networks[END_REF]. The idea here is to improve the quality of the images generated by a generative adversarial network (GAN). The image quality assessment (IQA) in such tools is sometimes performed by deep networks [START_REF] Hosu | Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment[END_REF] or by interaction with a human (HEVOL in [START_REF] Riviere | Inspirational adversarial image generation[END_REF]): this automatization of quality assessment removes tedious manual search over thousands of generated images [START_REF] Spratt | Creation, curation, and classification: Mario klingemann and emily l. spratt in conversation. XRDS: Crossroads[END_REF]. • Inspirational generation as in e.g. [START_REF] Riviere | Inspirational adversarial image generation[END_REF]. Inspirational generation consists of generating an image, in a given domain typically represented by a dataset, close to another given image, by searching the latent space. The OF of similarity to a given image is the difference with a classical GAN. For example, consider toonme. com, ranked #1 on the app store at the time of the present writing: this app takes as input a face image, and generates multiple images as an output (i) similar to the input (ii) in a cartoon domain, for "toonifying" that face. This follows the tradition of applying GANs for design [START_REF] Sbai | DesIGN: Design Inspiration from Generative Networks[END_REF][START_REF] Zhu | Be your own prada: Fashion synthesis with structural coherence[END_REF]. Compared to [START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF][START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF][START_REF] Reed | Learning what and where to draw[END_REF], this method searches a latent space and provides an output in the training domain. Compared to [START_REF] Mao | Mode seeking generative adversarial networks for diverse image synthesis[END_REF], this does not modify the training procedure. • Inspirational generation assisted by image quality assessment. For adding constraints on the generated image of a GAN, style transfer constrains the training [START_REF] Gatys | Image style transfer using convolutional neural networks[END_REF][START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF]. Other approaches analyze the latent space [START_REF] Riviere | Inspirational adversarial image generation[END_REF]: a z matching some constraints or optimizing some OF is used, in lieu of a randomly generated z. This means that we optimize simultaneously both the quality and the similarity to some example. Such an approach is directly useful for artists or designers [START_REF] Wolf | Unsupervised creation of parameterized avatars[END_REF], but also for inpainting, facial composites, anonymization or photo edition [START_REF] Bontrager | Deep interactive evolution[END_REF][START_REF] Gafni | Live face de-identification in video[END_REF][START_REF] Ulyanov | Deep image prior[END_REF][START_REF] Yeh | Semantic image inpainting with deep generative models[END_REF].

We have cases in which we work on raw images and cases in which we work on latent variables as detailed in Section 3.3.

Image representation by GAN latent variables

Generative models, and in particular adversarial ones [START_REF] Goodfellow | Generative adversarial nets[END_REF], are becoming prevalent in computer vision as they enhance artistic creation [START_REF] Elgammal | Creative adversarial networks[END_REF][START_REF] Zhu | Unpaired image-to-image translation using cycleconsistent adversarial networks[END_REF][START_REF] Park | Semantic image synthesis with spatially-adaptive normalization[END_REF], inspire designers [START_REF] Sbai | DesIGN: Design Inspiration from Generative Networks[END_REF][START_REF] Zhu | Be your own prada: Fashion synthesis with structural coherence[END_REF], or prove useful in semi-supervised learning [START_REF] Donahue | Adversarial feature learning[END_REF][START_REF] Frid-Adar | GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification[END_REF][START_REF] Nie | Medical image synthesis with context-aware generative adversarial networks[END_REF]. GANs typically generate random images. More precisely, a random latent vector z is generated, and the generator outputs G(z). Much like in some previously mentioned works, we use z as a smaller and more structured representation of an image.

Experimental results

For most experiments, we rely on the open-source project Nevergrad1 . We did some small modifications to PytorchGanZoo2 and Tarsier 3 . In numerical minimization, the loss means the objective value of the best candidate, minus the minimum possible objective value. In all experimental results, we use plots as provided by Nevergrad [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF], namely average (over settings) median (over replicas) loss. Before averaging, the loss is linearly normalized to [0, 1]. The definition of loss (a.k.a. simple regret) refers to an optimal value: this is replaced by the best-known value when the optimal value is unknown. Light-colored lines refer to ± standard deviation. For our new CVC (Section 3.1), we plot the average loss for this CVC. Other plots are based on the HV and we plot the average loss for the HV. The budget refers to the number of candidate points evaluated by the OFs. In all plots, the lower, the better. The quality assessment tools can fail: Figure 4 shows that PytorchGanZoo can produce aberrant data: using MOO precisely helps for mitigating such issues by considering several trade-offs. As we use the median, missing data means that at least 50% of replicas failed.

Computer vision benchmarks: HV and CVC

Figures 1-3 present experimental results. Roughly speaking, the MOO variant of DE we propose, based on [START_REF] Abbass | PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems[END_REF] and [START_REF] Robič | DEMO: Differential evolution for multiobjective optimization[END_REF], performs well in many cases. However, in the case of CDC for Many-OO (Figure 3, middle), MSR is sometimes better. These facts are consistent with our observations on PytorchGanZoo (Section 4.2.1) and Tarsier (Section 4.2.2). In some difficult ill-defined cases such as in Figure 1, HV with (1+1)EA is best: random mutations are known for focusing on wide, flat, stable basins. This confirms [START_REF] De | Genetic algorithms are not function optimizers[END_REF] and, closer to our field, a remark in [START_REF] Riviere | Inspirational adversarial image generation[END_REF] and in Section 2.1.1: with (1 + 1)-EA (also known as Discrete-(1 + 1)) we get solutions which, though not better numerically, are preferred by humans. For subsampling methods, IGD performs best overall (Figure 3, right).

Examples of applications: diverse inspirational generation and diverse super-resolution 4.2.1 Inspirational generation

Inspirational generation consists in the following: given a target image t, generating an image i(t), which is in a given domain D while having similarities with t (e.g. we look for a human face i(t) with some similarities with Casimir (t) the gentle orange dinosaur). We use the code from [START_REF] Riviere | Inspirational adversarial image generation[END_REF]. A popular application of diverse image generation inspired by a target image is toonme.com (based on [START_REF] Karras | Analyzing and improving the image quality of StyleGAN[END_REF]). [START_REF] Riviere | Inspirational adversarial image generation[END_REF] uses a weighted sum of 3 OFs, namely image similarity to t, relevance to D (quantified by the norm of the latent injection), and realism according to the GAN discriminator. We test:

• multiobjective methods equipped with the various Pareto sampling tools in Section 2.2;

• MSRH, as it turns out that the problem is so multimodal that there is quite a lot of diversity. We select the two methods that seem most promising, namely MSRH and DE. Then, we compare them through a double-blind human study. Results of the human study show that the latter (and simpler) method performs best: 68.57% ± 7.8% (35 generated groups of 8 images by MSRH vs 8 images generated by DE Figure 1: (Bigger version in the Supplementary Material) Various experiments: the (1 + 1)-EA is best (as in [START_REF] Riviere | Inspirational adversarial image generation[END_REF]) when there is a risk of optimization-based artifacts (quality optimization), and MOO DE is best overall in other cases. Left, similarity optimization on raw images: MOO-optimizing the five similarity measures. We compare algorithms using the HV. The best methods, namely NGOpt12/9 (i.e. a MSR) and variants of DE based on MOO-specific operators as in Section 2.1.4, are actually not based on HV. Middle, similarity optimization through PGAN [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF]: CVC. Average similarity (renormalized as detailed in Section 4) for one of the similarity measures when MOO-optimizing the four other similarity measures (see Section 3.2) with a latent representation. The MOO DE from Section 2.1.4 performs among the best for many values of the budget. Right, quality optimization: CVC. We have 12 contexts made of two settings and 6 budget values: we compute the best K512 (resp. Brisque in the second setting) score among the 16 images obtained by MOO of Blurriness and Brisque (resp. K512 in the second setting). We present the frequency at which non-trivial (finite) values were obtained over those 12 cases. This case is challenging, as the quality OFs are quite different from each other (Koncept512, based on a neural net for IQA; Brisque; and pure Blurriness) -for example, in many cases, we get a failure as Brisque value for the 16 output images of MOO optimizing Koncept512 and Blurriness, hence missing results. The only method which provided a non-trivial median result in all cases, consistently with results in [START_REF] Riviere | Inspirational adversarial image generation[END_REF], is (1+1)-EA: this algorithm, originally from the discrete optimization community, prefers optima with stable flat basins (Section 2.1.1). Fischer's exact test: p-value < 1.4% for (1+1)-EA vs any other.

with IGD: the best image is one of the MSRH in 24 cases -p-value 0.05 for the exact Clopper-Pearson test). Figure 4 shows results for this repeated single objective optimization.

Super-resolution

Super-resolution is known to be an ill-posed problem, because a low-resolution image can correspond to multiple high-resolution versions. We apply MOO to image super-resolution in order to generate several super-resolved images instead of just one. Tarsier [START_REF] Roziere | Tarsier: Evolving noise injection in super-resolution gans[END_REF] is a GAN-based and perceptual-driven [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF][START_REF] Rakotonirina | Esrgan+ : Further improving enhanced super-resolution generative adversarial network[END_REF] super-resolution model that uses noise injections in its architecture. At inference time, the injected noise is optimized according to two OFs along with an l 2 penalization: image quality score based on the IQA Koncept512 [START_REF] Hosu | Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment[END_REF] and realism score based on the discriminator. We therefore have three objective functions. According to our previous results with CVC, our MOO DE might perform better than DCMA (Diagonal-CMA): We run the two methods to experiment on images of the Set14 [START_REF] Zeyde | On single image scale-up using sparse-representations[END_REF] and Set5 [START_REF] Bevilacqua | Low-complexity single-image super-resolution based on nonnegative neighbor embedding[END_REF] datasets which are benchmark datasets for super-resolution. As presented in Figure 5, the images obtained by DE are more diverse compared to those obtained using DCMA, and even more compared to images obtained by MSRH which are essentially identical: here MOO, using classical Pareto-dominance, does provide diversity. This is further confirmed quantitatively by the perceptual metric [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF] of the Set5/Set14 images presented in Tabs 2 and DCMA (based on HV) and check if the good performance of DE is confirmed here. PSNR is to be maximized and we see that in all cases but one (head) the best value for DE's outputs is better than the best value for DCMA's outputs. Perceptual is to be minimized: the best value (min) for DE's outputs is always better than the best value for DCMA's outputs. Consistent with observations in the rest of this paper, we get a better best value among the nine outputs of DE than among the nine outputs of DCMA with HV. Neither PSNR nor Perceptual were optimized for obtaining those images: this is an application of CVC, i.e., we check on other criteria than those that were optimized. Human assessment (Figure 5) confirms DE's superior performance. Conclusions: the MOO variant of DE does a good job here, and the variance of results over the selected Pareto front is significant; therefore, MOO does work for bringing diversity in super-resolution. and 3. The peak signal-to-noise ratio (PSNR) is also provided for reference. This approach, which encourages diversity, could improve fairness [START_REF] Kenfack | On the fairness of generative adversarial networks (gans)[END_REF][START_REF] Mehrabi | A survey on bias and fairness in machine learning[END_REF][START_REF] Sattigeri | Fairness gan: Generating datasets with fairness properties using a generative adversarial network[END_REF] in generative models.

Conclusion

We advocate multi-objective optimization (MOO) as a means for generating diverse solutions to computer vision problems. The approach improves user satisfaction and fairness. Multiple solutions stand a better chance of satisfying the more demanding users. After performing experiments in Nevergrad, we validate our results in Super-Resolution (Tabs 2-3 and Figure 5) and Inspirational generation (Figure 4). Depending on the case, we advocate one of two different MOO strategies: pareto fronts and MSR.

Many MOO problems arising in computer vision use ill-defined criteria, which do not exactly match human preference. In such cases, defining solutions based on Pareto fronts is questionable. For instance, if a dominates b it does not always follow that a is better than b. Typically, image quality, as a proxy for user satisfaction, might favor solutions that represent the most frequent categories, i.e., objects/people, leading to We see that only MSR (and, interestingly, all variants of MSR) provides stable results in this inspirational generation context. This is consistent with previous results: whereas HV incorrectly predicts better results for DE. Right: impact of the subsampling method extracting M non-dominated points. For each algorithm, we compute the CVC loss, normalized and averaged. For each algorithm, we show the rank of the best result obtained by IGD (resp. EPS and Random) as subsampling methods. IGD significantly outperforms EPS (Wilcoxon p-value < 0.05).

an unfair representation of the others. This makes the MSR strategy (and MSRH when a tuned combination of criteria is available and multiple local optima are present) particularly competitive as they aim to optimize locally. This subtle point is critical for increasing diversity in inspirational generation, as shown by Figs. 3 and4. However, MSR does not always produce the best results: for conditional GANs such as those used for super-resolution, MSR leads to nearly identical images, and in this context "real" "Pareto-style" MOO helps (Figs. 5 and Tab. 2). Next, we propose a new principled methodology, termed CVC, for comparing MOO methods by cross-validation of objective functions. In Section 3.1 we advocate this methodology for the many-objective cases. Compared to the HV method, CVC is not, and by design can not be based on an indicator used by some algorithms. Therefore, it facilitates an unbiased comparison between methods.

For the many-objective cases, with partially redundant objectives designed as proxies for user preference, CVC helps for investigating the sufficiency of a set of objective functions. This is particularly relevant in computer vision, where we often use multiple similarity measures, which are but rough approximations of the criteria implemented by the human visual system. Results obtained with CVC are consistent with human inspection and diversity measures: Sections 4.2.1 and 4.2.2 show that when a method is "good" according to the CVC criterion, it is typically also preferred by users. Results using CVC in Tabs 2-3 are consistent with human inspection in Figure 5 and Figs. 3 and4 are also in accordance. Note that human studies in scientific papers are a form of CVC: we optimize for some OF and validate with another OF, namely the user preference. We open source a large family of benchmarks for MOO in computer vision. They are merged in a maintained platform and can be run as one-liners. Limitations. We compare numerous methods in our comparative results (Section 4.1), reproducible in one-line in Nevergrad. However, the field is wide and, as discussed in [START_REF] Chand | Evolutionary many-objective optimization: A quick-start guide[END_REF], the best algorithms are very problem-dependent so we could include even more algorithms. Our open sourced MOO DE is heavily inspired by [START_REF] Robič | DEMO: Differential evolution for multiobjective optimization[END_REF][START_REF] Abbass | PDE: a Pareto-frontier differential evolution approach for multi-objective optimization problems[END_REF] (Section 2.1.4). Our results in super-resolution do not include datasets focused on human faces yet, even though diversity is particu- In the context of inspirational generation (generating an image similar to a target), it turned out that the local optima of the original trade-off optimization from [START_REF] Riviere | Inspirational adversarial image generation[END_REF], obtained by repeated optimization runs (MSRH method), offer more diversity and quality than other multiobjective methods (see text: OF are valid only locally, leading to a diversity loss when applying Pareto-dominance globally, hence the success of MSR variants). MSR is already the best for CVC in Figure 3. In these two cases above, the target image is the top left one. We present two hard cases, on which the original PytorchGanZoo code (without MOO) frequently fails, and PytorchGanZoo with Pareto-dominance generates very little diversity. Top left: the original inspirational GAN tends to generate images of women whereas the target is male; in contrast to this, over the eight obtained images, our MOO code generates male faces. Top right: in this difficult case, we look for a face with similarities with Casimir [START_REF] Wikipedia | Casimir from l'ile aux enfants[END_REF], the gentle orange dinosaur. Two of the faces have the orange color and the big dark-circled white eyes. Three of the eight generated faces are complete failures, but with the diversity obtained by MOO the user can select the best of the eight generated images. Bottom, unsuccessful inspirational generation using Pareto dominance, compared to top left: we get very little diversity (all female). Conclusions: there is an intrinsic variance in inspirational generation so that MSRH does work quite well: using MSRH rather than applying Pareto-dominance means using objective functions only locally and avoids discarding dominated parts of the domain.

larly important in that case for fairness reasons. Our proposed CVC criterion is meaningful only for many redundant misspecified OFs. . We present the frequency at which non-trivial (finite) values were obtained over those 12 cases. This case is challenging, as the quality OFs are quite different from each other (Koncept512, based on a neural net for IQA; Brisque; and pure Blurriness) -for example, in many cases, we get a failure as Brisque value for the 16 output images of MOO optimizing Koncept512 and Blurriness, hence missing results. The only method that provided a non-trivial median result in all cases, consistently with results in [START_REF] Riviere | Inspirational adversarial image generation[END_REF], is (1+1)-EA: this algorithm, originally from the discrete optimization community, prefers optima with stable flat basins. Fischer's exact test: p-value < 1.4% for (1+1)-EA vs. any other. MSR is already the best for CVC in Fig. 3. In these two cases above, the target image is the top left one. We present two hard cases, on which the original PytorchGanZoo code (without MOO) frequently fails, and PytorchGanZoo with Pareto-dominance generates very little diversity. Top: the original inspirational GAN tends to generate images of women whereas the target is male; in contrast to this, over the eight obtained images, our MOO code generates male faces. Middle: in this difficult case, we look for a face with similarities with Casimir [START_REF] Wikipedia | Casimir from l'ile aux enfants[END_REF], the gentle orange dinosaur. Two of the faces have the orange color and the big dark-circled white eyes. Three of the eight generated faces are complete failures, but with the diversity obtained by MOO the user can select the best of the eight generated images. Bottom, unsuccessful inspirational generation using Pareto dominance, compared to top left: we get very little diversity (all female). Conclusions: there is an intrinsic variance in inspirational generation so that MSRH does work quite well: using MSRH rather than applying Pareto-dominance means using objective functions only locally and avoids discarding dominated parts of the domain.

Figure 2 :

 2 Figure 2: Other experiments, also good performance for MOO DE. Left: quality optimization through PGAN, HV evaluation. Aggregation (as detailed in Section 4) of two settings, K512 (resp. Brisque) score of an image obtained by MOO of Blurriness and Brisque (resp. K512), with latent variables as a representation. The y-axis is the (normalized and averaged, as explained in Section 4) HV: once more, DE performs best, though it is not based on the HV. Right: similarity and quality optimization: CVC. Best Lpips-Alex obtained by various algorithms in budget up to 5e4 when MOO-optimizing the similarity measures and the following quality measures: Lpips-Vgg, Bluriness, Koncept512. For clarity and ease of reading, only the best performing methods are presented on each plot.

Figure 3 :

 3 Figure 3: (Bigger version in the Supplementary Material) Settings close to inspirational generation (hence diversity is critical): MSR performing best, consistent with later experiments. Many-objective similarity and quality optimization, through PGAN. Images are represented by latent variables of a GAN. Left, median HV (log HV divided by max) obtained for each algorithm with budget 1e4: DE variants perform best, as in our artificial experiments. Middle, comparison based on the CVC methodology: best K512 score over images obtained by various MOO treatments of several similarity metrics and Brisque and Blurriness.We see that only MSR (and, interestingly, all variants of MSR) provides stable results in this inspirational generation context. This is consistent with previous results: whereas HV incorrectly predicts better results for DE. Right: impact of the subsampling method extracting M non-dominated points. For each algorithm, we compute the CVC loss, normalized and averaged. For each algorithm, we show the rank of the best result obtained by IGD (resp. EPS and Random) as subsampling methods. IGD significantly outperforms EPS (Wilcoxon p-value < 0.05).

Figure 4 :

 4 Figure 4: (Extended version in the Supplementary Material) MSRH for inspirational generation (projection onto the celebrities model): two examples in which (1) MOO optimization provides diversity (2) MOO by MSRH outperforms MOO by criteria using Pareto-dominance (3 versions).In the context of inspirational generation (generating an image similar to a target), it turned out that the local optima of the original trade-off optimization from[START_REF] Riviere | Inspirational adversarial image generation[END_REF], obtained by repeated optimization runs (MSRH method), offer more diversity and quality than other multiobjective methods (see text: OF are valid only locally, leading to a diversity loss when applying Pareto-dominance globally, hence the success of MSR variants). MSR is already the best for CVC in Figure3. In these two cases above, the target image is the top left one. We present two hard cases, on which the original PytorchGanZoo code (without MOO) frequently fails, and PytorchGanZoo with Pareto-dominance generates very little diversity. Top left: the original inspirational GAN tends to generate images of women whereas the target is male; in contrast to this, over the eight obtained images, our MOO code generates male faces. Top right: in this difficult case, we look for a face with similarities with Casimir[START_REF] Wikipedia | Casimir from l'ile aux enfants[END_REF], the gentle orange dinosaur. Two of the faces have the orange color and the big dark-circled white eyes. Three of the eight generated faces are complete failures, but with the diversity obtained by MOO the user can select the best of the eight generated images. Bottom, unsuccessful inspirational generation using Pareto dominance, compared to top left: we get very little diversity (all female). Conclusions: there is an intrinsic variance in inspirational generation so that MSRH does work quite well: using MSRH rather than applying Pareto-dominance means using objective functions only locally and avoids discarding dominated parts of the domain.

Figure 5 :

 5 Figure 5: Comparison between DCMA (diagonal CMA optimizing HV) and MOO DE. Zooms on high-resolution images obtained by MOO-Tarsier using DE (a) and DCMA (b) on one image in Set14. In many cases, as in the present example, the images generated by DE are more diverse. For DE, the first image is blurrier and darker than the second one. The textures are all different for the four DE images.

Figure 6 :

 6 Figure6: Various experiments: the (1 + 1)-EA is best (as in[START_REF] Riviere | Inspirational adversarial image generation[END_REF]) when there is a risk of optimization-based artifacts (quality optimization), and MOO DE is best overall in other cases. Top, similarity optimization on raw images: MOO-optimizing the five similarity measures. We compare algorithms using the HV. The best methods, namely NGOpt12/9 (i.e. a MSR) and variants of DE based on MOO-specific operators, are actually not based on HV. Middle, similarity optimization through PGAN[START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF]: CVC. Average similarity (renormalized as detailed in the main text) for one of the similarity measures when MOO-optimizing the four other similarity measures (see text) with a latent representation. The MOO DE performs among the best for many values of the budget. Bottom, quality optimization: CVC. We have 12 contexts made of two settings and 6 budget values: we compute the best K512 (resp. Brisque in the second setting) score among the 16 images obtained by MOO of Blurriness and Brisque (resp. K512 in the second setting). We present the frequency at which non-trivial (finite) values were obtained over those 12 cases. This case is challenging, as the quality OFs are quite different from each other (Koncept512, based on a neural net for IQA; Brisque; and pure Blurriness) -for example, in many cases, we get a failure as Brisque value for the 16 output images of MOO optimizing Koncept512 and Blurriness, hence missing results. The only method that provided a non-trivial median result in all cases, consistently with results in[START_REF] Riviere | Inspirational adversarial image generation[END_REF], is (1+1)-EA: this algorithm, originally from the discrete optimization community, prefers optima with stable flat basins. Fischer's exact test: p-value < 1.4% for (1+1)-EA vs. any other.

Figure 7 :

 7 Figure 7: Settings close to inspirational generation (hence diversity is critical): MSR performing best, consistent with later experiments. Many-objective similarity and quality optimization, through PGAN. Images are represented by latent variables of a GAN. Top, median HV (log HV divided by max) obtained for each algorithm with budget 1e4: DE variants perform best, as in our artificial experiments. Middle, comparison based on the CVC methodology:best K512 score over images obtained by various MOO treatments of several similarity metrics and Brisque and Blurriness. We see that only MSR (and, interestingly, all variants of MSR) provides stable results in this inspirational generation context. This is consistent with previous results: whereas HV incorrectly predicts better results for DE. Bottom: impact of the subsampling method extracting M non-dominated points. For each algorithm, we compute the CVC loss, normalized and averaged. For each algorithm, we show the rank of the best result obtained by IGD (resp. EPS and Random) as subsampling methods. IGD significantly outperforms EPS (Wilcoxon p-value < 0.05).

Figure 8 :

 8 Figure 8: MSRH for inspirational generation (projection onto the celebrities model): two examples in which (1) MOO optimization provides diversity (2) MOO by MSRH outperforms MOO by criteria using Paretodominance (3 versions).In the context of inspirational generation (generating an image similar to a target), it turned out that the local optima of the original trade-off optimization from[START_REF] Riviere | Inspirational adversarial image generation[END_REF], obtained by repeated optimization runs (MSRH method), offer more diversity and quality than other multiobjective methods (see text: OF are valid only locally, leading to a diversity loss when applying Pareto-dominance globally, hence the success of MSR variants). MSR is already the best for CVC in Fig.3. In these two cases above, the target image is the top left one. We present two hard cases, on which the original PytorchGanZoo code (without MOO) frequently fails, and PytorchGanZoo with Pareto-dominance generates very little diversity. Top: the original inspirational GAN tends to generate images of women whereas the target is male; in contrast to this, over the eight obtained images, our MOO code generates male faces. Middle: in this difficult case, we look for a face with similarities with Casimir[START_REF] Wikipedia | Casimir from l'ile aux enfants[END_REF], the gentle orange dinosaur. Two of the faces have the orange color and the big dark-circled white eyes. Three of the eight generated faces are complete failures, but with the diversity obtained by MOO the user can select the best of the eight generated images. Bottom, unsuccessful inspirational generation using Pareto dominance, compared to top left: we get very little diversity (all female). Conclusions: there is an intrinsic variance in inspirational generation so that MSRH does work quite well: using MSRH rather than applying Pareto-dominance means using objective functions only locally and avoids discarding dominated parts of the domain.

Table 2 :

 2 Peak signal-to-noise ratio (PSNR) and perceptual score of images in Set5. We compare DE (MOO variant)

		DCMA		DE
		PSNR	Perceptual	PSNR	Perceptual
		max (best) std min (best) std	max (best) std min (best) std
	baby	28.5013 0.0584 0.1258 0.0012	28.5942 0.0284 0.1170 0.0016
	bird	28.4378 0.1615 0.0487 0.0019	28.5327 0.0920 0.0464 0.0013
	butterfly 23.1786 0.0208 0.0540 0.0003	23.1997 0.065 0.0534 0.0007
	head	26.8428 0.0098 0.1190 0.0006	26.7854 0.0496 0.1150 0.0018
	woman 25.4202 0.0231 0.0818 0.0003	25.4 513 0.1113 0.0814 0.0019

Table 3 :

 3 Peak signal-to-noise ratio (PSNR) and perceptual score of images in Set14. We get a significant diversity, whereas running MSRH leads to several times the exact same image. Here, overall and contrarily to Tab. 2, Diagonal-CMA performs slightly better than DE.

https://github.com/facebookresearch/nevergrad

https://github.com/facebookresearch/pytorch_GAN_zoo

https://github.com/ncarraz/ESRGANplus

A Supplementary material

The following pages contains extended versions of some of the experiments described in the paper.

Target:

MSHR with (1 + 1)-EA.

MSHR with BFGS.

Next rows: three distinct runs of (1 + 1)-EA with HV during the optimization run.


Jun Guan

Zejun Sun

Ramin Ansari

Yujia Liu

Aimi Endo

Masafumi Unno

Armelle Ouali

Shahrea Mahbub

Joseph^^c Furgal

Nuttapon Yodsin

Siriporn Jungsuttiwong

Daniel Hashemi

John Kieffer

Prof Richard^^m

Laine

Conjugated Copolymers That Shouldn't Be

Keywords:

Unexpected excited-state conjugation was observed in a series of copolymers derived from ladder silsesquioxane, vinyl(Me/Ph)Si(O 0.5 ) 2 [PhSiO 1.5 ] 4 (O 0.5 ) 2 Si(Me/Ph)vinyl, which exhibited 30--60^^nm red-shifted emission relative to doubledecker derived analogues with equal or larger degrees of polymerization.

Further studies hint at their potential application as semiconducting polymers.

Main text

There is growing impetus to explore silsesquioxanes (SQs) due to their ease of synthesis and purification, well-defined 3D nanostructures, robust nature as well as high compatibility with multiple types of organics leading to large families of hybrid materials. The major story still centers on properties that can be typified as organic decorated silica moieties. In contrast, we find that cage SQs {[RSiO 1.5 ] 8,10,12 (R=R'phenyl, R'vinyl, R=conjugated moiety, R'=H, Me, MeO, NH 2 , CN, etc.)} offer photophysical properties not at all reflective of organic decorated silicas but commensurate with cage centered

LUMOs. [1--5] We recently extended this concept of excited-state 3D conjugation to sets of incomplete and modified cages including corner-missing T 8 , [R'StilbeneSiO 1.5 ] 7 [O 1.5 SiMe/nPr],

[R'StilbeneSiO 1.5 ] 7 [O 0.5 SiMe 3 ] 3 , [6] Ar} that unexpectedly exhibit conjugation that appears to involve two vinyl(Me)Si(O-) 2 bridges evidenced by significant emission red-shifts from corresponding model compounds. [9] Schemes^^1^a<schr1>,b provide general synthetic routes and structures. In all systems, conjugation presents in the form of emission red-shifts of 50--120^^nm from model compounds without cage components. Such novel combinations of structural, thermal, mechanical and photophysical properties, such as tunable broad-band UV/Vis fluorescence, may be important for applications as emissive layers in OLEDs, [10] multi-functional nanodrugs, in photothermal and photodynamic therapies for cancer, etc. [11--13] With the recent discovery of routes to ladder SQs (vinyl-LL-vinyl), [14] we can now explore vinyl-LL-vinyl equivalent systems of the type suggested in Scheme^^1^c<xschr1>. The motivation is to ascertain the nature of LUMO formation in ladder SQs where the structure appears to be approximately one-half that of relatively complete double-deckers. Our initial presumption was that these compounds should represent the limiting case where not only is there no 3D cage, but there are also two vinyl(Me/Ph)Si(O-) 2 bridges per cage in any oligomers/polymers we might synthesize. Contrary to our original thoughts, the resulting compounds actually offer emission further red-shifted than any analogous compounds previously studied, [1--7,^9] seeming to imply conjugation and semiconducting behavior where there should be none. In the following sections, we begin by characterizing a set of ladder bithiophene and thienothiophene as reported in our previous paper. [9] Similar ICT also occurred in 10^^mol^%F 4 TCNQ doped LLco-thiophene, bithiophene and thienothiophene using the mixedsolution method. The original orange-red color becomes dark green immediately on mixing. The literature reports that poly(3-hexyl-thiophene-2,5-diyl) regioregular (P3HT) exhibits integer charge transfer interactions with F 4 TCNQ and coincident nitrile-stretching mode shifts from a neutral value ν 0 =2227^^cm <M->1 to ν 1 =2194^^cm <M->1 on doping. [16,^17] In marked contrast, however, quaterthiophene (4T) forms partial charge transfer with F 4 TCNQ and thus shows only a small shift. [18] Shifts of characteristic cyano-vibrational bands from the neutral value to the anion value are observed in the FTIR for LL-co-bithiophene mixing with F 4 TCNQ as shown in Figure^^3<figr3> indicating integer charge transfer (ICT).

The literature also reports that the electronic structure of F 4 TCNQ anion includes doublet states because of the presence of an unpaired single electron. [19] The UV/Vis spectrum of The photophysical behavior of both DD/LL-co-phenyl reappear after zinc debromination, [20] with absorption λ max around 300^^nm Furthermore, LL SQs linked with longer or more complex aromatic bridges, LL-co-biphenyl, terphenyl, stilbene and thiophene, show 30--60^^nm emission red-shifts beyond those seen with the DD analogs and progressively red-shifted emissions with respect to longer chains were also observed.

Integral charge transfer occurs in 10^^mol^% F 4 TCNQ-doped LLco-thiophene, bithiophene, thienothiophene, which also occur on mixing F 4 TCNQ with DD analogous polymers. These results

suggest not only a new perspective on the ease of formation of LUMOs in even LL SQs, but also more efficient communication of the LL SQ with conjugated organic moieties in the excited state as compared to full and double-decker SQs that potentially offer access to a wide variety of semiconducting polymers. In addition, brominating vinyl as well as phenyl groups in DD/LL derived copolymers and then debrominating vinyls not only restores through chain conjugation but also provides the potential to further peripherally functionalize these systems via a variety of catalytic cross-coupling reactions to explore 3D conjugation in "hairy SQ polymers". DP [a] Abs. λ max [nm] Em. λ max

[nm] [b] Φ F Vinyl(Me)DD(Me)vinyl 1 264 281

Table^^2 MALDI-TOF and GPC data for DD-co-thiophene, short LLco-thiophene isolated by TLC and long LL-co-thiophene.

GPC MALDI-TOF

M n M w Ð [a] Oligomers [b] M/z LL-co-thiophene, DP≈3 2540 3630 1.12 dimer 1896 [c] LL-co-thiophene, DP≈7 5600 8000 1.42 trimer 2797 [c] DD-co-thiophene, DP≈20 22^540 43^250 1.92 decamer 13^010 [c] [a]^^Polydispersity.

[b]^^Largest oligomers identifiable.

[c]^^As Ag + adduct.

  photophysical behavior are essentially identical as expected, thus only data for vinyl-LL(Me)-vinyl co-polymers are presented. Spectra for vinyl-LL(Ph)-vinyl derived copolymers are shown in Figures^^S9--14. Both co-polymers were characterized by MALDI-TOF, GPC, FTIR, 1 H and 13 C^^NMR, and the starting ladder SQs were also characterized by 29 Si NMR as recorded in Tables^^S1--3 and Figures^^S1--8. MALDI-TOF shows the expected m/z, consistent with GPC determined molar masses of the ladder SQs. The GPC retention time for vinyl-LL(Ph)vinyl is ≈0.3^^min earlier than that for vinyl-LL(Me)-vinyl due to the slightly bulkier phenyl groups. Table^^1<tabr1> GPC data for the ladder copolymers indicates degrees of polymerization (DPs) of 3--18 units. MALDI-TOF analyses always show peaks every m/z for the co-monomer unit vinyl-LL-vinyl-

F 4

 4 Figure^^4<figr4>clearly shows spectral signatures from F 4 TCNQ˙< M-> at 600--900^^nm and LL-co-bithiophene at 350--500^^nm, strongly suggesting ICT, consistent with the FTIR data.

  Scheme^^1Heck cross-coupling of a)^^model compounds, b)^^vinylDDvinyl derived copolymers and c)^^vinyl-LL-vinyl derived copolymers.

Figure^^5

  Figure^^5 a)^^Bromination and debromination of LL-co-phenyl. b)^^Steady-state absorption and emission spectra in CH 2 Cl 2 .

Acknowledgements

We thank NSF for support of this work through Chem. Award No. 1610344. J.C.F. thanks Bowling Green State University for Startup Funding. S.J. thanks the Center of Excellence for Innovation in Chemistry (PERCH-CIC), Ministry of Higher Education, Science, Research and Innovation, the Human Resources Development in Science Project Science Achievement Scholarship of Thailand (SAST) for support. J.K. thanks NSF-DMR-1435965 for support.

Conflict of interest

The authors declare no conflict of interest.

Author Contributions

J.G. Conceptualization: Equal; Data curation: Equal; Formal analysis: Equal; Writing----original draft: Equal; Writing---review & editing: Equal.