Guillaume Baudart
email: guillaume.baudart@inria.fr

Martin Hirzel
email: hirzel@us.ibm.com

Kiran Kate
email: kakate@us.ibm.com

Parikshit Ram
email: parikshit.ram@ibm.com

Avraham Shinnar
email: shinnar@us.ibm.com

Jason Tsay
email: jason.tsay@ibm.com

Pipeline Combinators for Gradual AutoML

Automated machine learning (AutoML) can make data scientists more productive. But if machine learning is totally automated, that leaves no room for data scientists to apply their intuition. Hence, data scientists often prefer not total but gradual automation, where they control certain choices and AutoML explores the rest. Unfortunately, gradual AutoML is cumbersome with state-of-the-art tools, requiring large non-compositional code changes. More concise compositional code can be achieved with combinators, a powerful concept from functional programming. This paper introduces a small set of orthogonal combinators for composing machinelearning operators into pipelines. It describes a translation scheme from pipelines and associated hyperparameter schemas to search spaces for AutoML optimizers. On that foundation, this paper presents Lale, an open-source sklearn-compatible AutoML library, and evaluates it with a user study.

Introduction

Automated machine learning can make data scientists more productive by lowering the barrier to entry for novices as well as reducing the tedium for experts. With total automation, the data scientist just provides data and the AutoML tool does the rest. While this yields a convenient starting point, it gives users little control. Instead, gradual automation, where the user tweaks some choices by hand, leverages user expertise for better outcomes and wastes less compute time. Unfortunately, gradual automation is not easy with existing AutoML tools and their programming interfaces. Existing AutoML tools tend to have different interfaces from those for manual machine learning, preventing a gradual transition between the two. Furthermore, with existing tools, code changes for gradual automation tend to be large and non-compositional. The programming interface of existing tools is often coupled to a given optimizer. And finally, tools often support controlling some aspects of automated pipeline search only with a much lower-level programming interface, if at all. We hypothesize that gradual AutoML can be made easier via combinators. Combinators were first discovered by Schönfinkel in the 1920s in an attempt to find elementary building blocks of logic [START_REF] Schönfinkel | Über die Bausteine der mathematischen Logik[END_REF]. Combinators are functions that compose functions and play an important role in functional programming. For instance, Lisp provides the combinators map and reduce [START_REF] Guy | Common Lisp: The Language[END_REF], which inspired the MapReduce programming model [START_REF] Dean | MapReduce: Simplified data processing on large clusters[END_REF][START_REF] Lämmel | Google's MapReduce programming model -revisited[END_REF]. Combinators enable a tacit programming style, where data remains unnamed, keeping function compositions concise and minimizing edits when re-arranging them. We argue that sklearn's make_pipeline and make_union [START_REF] Buitinck | API design for machine learning software: Experiences from the scikit-learn project[END_REF] are combinators: they compose machine-learning operators into pipelines without naming datasets such as X or y during composition. These two combinators enable the two primary execution modes, training and prediction, of sklearn pipelines. This paper streamlines them with infix notation, adds a choice combinator, and adds AutoML search as a third execution mode.

Enabling search as a third execution mode requires a translation from combinators to AutoML search spaces. To solve this challenge, this paper describes a novel translation scheme, which takes a combinator-based pipeline and generates a search space suitable for a given AutoML optimizer. The translation first normalizes and combines the hyperparameter schemas of the operators in the pipeline, which specify the space of hyperparameter values. Next, depending on the optimizer at hand, it flattens and discretizes the search space. This translation is essential to bridge from the modular and tacit combinators syntax to an optimizer's syntax, and may re-arrange components and introduce (possibly mangled) names as needed. The translation scheme also supports round-tripping back to combinators, needed for evaluating and understanding results as well as for iterative refinement. This paper introduces Lale, a Python AutoML library implementing the combinators and the translation scheme. Lale enjoys active use both in the open-source community (https://github.com/ibm/lale/) and in IBM's AutoAI product. It is sklearn-compatible, giving users access to many operators and familiar functionality. Lale is gradual, letting users specify only what they want while reusing and automating the rest. For instance, Lale comes with an extensive library of reusable hyperparameter schemas for many popular operators, so users rarely need to write their own schemas; but it also makes it easy to customize schemas when desired. There are Lale optimizer backends for Hyperopt [START_REF] Bergstra | Hyperopt: a Python library for model selection and hyperparameter optimization[END_REF] (with 4 solvers tpe, atpe, rand, and anneal); sklearn's GridSearchCV and HalvingGridSearchCV [START_REF] Buitinck | API design for machine learning software: Experiences from the scikit-learn project[END_REF]; ADMM [START_REF] Liu | An ADMM based framework for AutoML pipeline configuration[END_REF] (with 5 continuous and 5 combinatorial solvers); SMAC [START_REF] Feurer | Efficient and robust automated machine learning[END_REF]; and Hyberband [START_REF] Li | Hyperband: A novel bandit-based approach to hyperparameter optimization[END_REF].

The contributions of this paper are:

• Combinators for gradual automated machine learning.

• A translation scheme for mapping combinators to a broad variety of optimizers.

• The Lale library and a user study for validating combinators and gradual automation. This paper addresses the problem of simplifying gradual AutoML. Our solution, Lale, is easy to use for data scientists with moderate sklearn experience, as demonstrated by the user study. Lale uses combinators for modularity, and uses schemas both for search spaces and for type-checking. It is more expressive than the high-level interfaces of prior AutoML tools. Lale gives users fine-grained control over AutoML without requiring them to be AutoML experts.

Pipeline Combinators

This paper proposes three combinators to compose machine-learning operators into pipelines. To support hyperparameter tuning, each operator has an associated hyperparameter schema. Thus, when combinators compose operators, then under the hood, that implies weaving together the associated hyperparameter schemas. This section describes the syntax both for combinators and for schemas, leaving it to Section 4 to describe the translation that weaves them together for AutoML.

Combinators. The three combinators are >>, &, and |. The pipe combinator, p >> q, connects the output of p to the input of q, similar to sklearn's make_pipeline. The and combinator, p & q, composes p and q side-by-side without creating dataflow between them. It is similar to sklearn's make_union, except that make_union(p, q) implicitly concatenates the output features of p and q whereas p & q does not. Instead, if users want to combine features after &, they have to explicitly add an operator for that. Making this explicit makes code more understandable and also provides flexibility to pick different data-combining operators when desired, for instance, vertical stacking vs. relational joining [START_REF] Sahni | RASL: Relational algebra in scikit-learn pipelines[END_REF]. The or combinator, p | q, introduces a choice between p and q. While it has no direct equivalent in sklearn, it can be encoded in sklearn's GridSearchCV, as well as in other optimizers such as Hyperopt.

Figure 1 shows the syntax for combinator-based pipelines. The start symbol, pipeline, can be an individual operator or it can nest other pipelines by using one of the three combinators. An individualOp has a name and an optional configuration for zero or more hyperparameters. For example, in Figure 2, the operators Project and AdaBoostClassifier are explicitly configured with hyperparameters, whereas the operators PCA, NoOp, etc. are not. When the user does not manually configure hyperparameters, then AutoML automatically tunes them. A hyperParam configuration uses an expression, which can be a simple Python value or a nested pipeline. For example, in Figure 2, Project is configured with a simple Python dictionary, whereas AdaBoostClassifier is configured with a nested pipeline. An operator that takes another operator as an argument is higher-order, and besides AdaBoostClassifier, there are plenty of other examples, such as sklearn's RFE, OneVsRestClassifier, or ColumnTransformer, or fairness mitigation operators with preprocessing operators as an argument [START_REF] Hirzel | Engineering fair machine learning pipelines[END_REF].

What makes the combinator-based pipeline syntax powerful is that everything nests. A pipeline is itself an operator in the sense that it implements the methods fit and transform or predict. Unlike other AutoML tools, Lale can thus easily express search spaces around higher-order operators. The syntax is concise, as it focuses just on operators and is tacit about data. The Lale library implements this syntax as pure Python code. That has the advantage that users need not learn a new programming language, and can instead make full use of Python's existing features. For example, Figure 2 uses a Python assignment (pre_n = ...) to improve readability. Furthermore, existing Python tooling just works, such as auto-completing editors, syntax and type checkers, and interactive notebooks.

Execution modes. Pipelines have three execution modes: predict or transform; fit; and AutoML search. The first two modes come from sklearn. Their semantics are based on the dataflow graph of a pipeline, which is a directed acyclic graph whose vertices are operators and whose edges are given by the >> combinator (note that & and | do not induce dataflow edges, and pipelines with | only support the third execution mode). The first execution mode of a pipeline, predict or transform, simply invokes predict or transform on the vertices in topological order, guaranteeing that the results from all predecessors of an operator are available before that operator fires. The second execution mode, fit, also processes vertices in topological order, calling first fit and then transform or predict. The result of fit on a pipeline is a new pipeline that is the same as the original except that all operators are replaced by trained versions, i.e., their learned coefficients are bound. Higher-order operators have the flexibility to invoke the methods of nested operators as needed for the outer operator.

The third execution mode of a pipeline, AutoML search, is implemented by translating the planned pipeline to a search space for an AutoML optimizer. The search space encodes joint operator selection and hyperparameter tuning. Given a planned pipeline such as pplan from Figure 2, users access this execution mode by invoking the auto_configure method:

1 best_found = pplan.auto_configure( 2 train_X, train_y, optimizer=Hyperopt, cv=3, max_opt_time=300, max_eval_time=30)

The result (in this case best_found) is the trained pipeline that yielded the best score (here, crossvalidated with cv=3) during the search. Section 4 discusses the details of the translation. Schemas. Besides combinators, which specify a space of pipelines to search, AutoML also needs hyperparameter schemas, which specify a space of hyperparameter values to search. Here, we adopt the value-set definition of a schema (or type) as a space (or set) of values [START_REF] Parnas | Abstract types defined as classes of variables[END_REF]. While data scientists frequently write code with combinators to exercise control, they write schemas less often. As of today, Lale includes schemas for 216 operators, sufficient for many common machine learning tasks.

Figure 3 shows the most important parts of the schema syntax (eliding some details, such as defaults, integers, and array schemas, that are not needed for examples in this paper). The start symbol, schema, can be an enumeration, a range, an object, or it can nest other schemas by using one of three logic connectives. An enum schema specifies a categorical hyperparameter with a choice between one or more values. maximum (most AutoML optimizers do best with bounded ranges) and an optional distribution distr . Figure 4 includes two examples of ranges: (0..1) and (0..0.5). An object schema specifies a dictionary of zero or more hyperparameter schemas. For example, Figure 4 specifies one hyperparameter for the PCA operator from sklearn [START_REF] Buitinck | API design for machine learning software: Experiences from the scikit-learn project[END_REF] and two each for J48 from Weka [START_REF] Hall | The WEKA data mining software: An update[END_REF] and for LR from sklearn.

The schema syntax supports three logic connectives disjunction (∨), conjunction (∧), and negation (¬). Disjunction describes a union type; for example, in Figure 4, the N hyperparameter of PCA can specify the number of components with either a range (0..1) or an enum [mle]. Conjunction describes an intersection type and negation can be used to describe an implication. These connectives help express constraints for ruling out combinations of hyperparameters that are individually valid but collectively invalid. For example, with Weka's J48 operator, if R is true (use reduced error pruning), then C must be 0.25 (cannot set the pruning confidence to other values). Figure 4 specifies this via ⇒, which is syntactic sugar for the basic logic connectives. Hyperparameter constraints are useful for pruning invalid areas from the search space, so AutoML search can focus on valid areas instead.

To make schemas easier to use, we decided not to invent our own new schema language, but instead to adopt a mature existing language with a broad user base and tooling ecosystem. One choice we considered was Python 3 types [START_REF] Guido Van Rossum | PEP 484 -Type Hints[END_REF][START_REF] Rak-Amnouykit | Python 3 types in the wild: A tale of two type systems[END_REF]. Unfortunately, Python 3 types are not expressive enough for specifying constraints. But fortunately, there is another language that also works well with Python and is used extensively by a large community well beyond that of AutoML: JSON Schema [START_REF] Pezoa | Foundations of JSON schema[END_REF]. Most constructs from Figure 3 have exact equivalents in JSON Schema. For instance, a range (a..b) can be written as {"type": "number", "minimum":a, "maximum":b}. Lale adds a few extensions to JSON Schema. For instance, it supports an optional "distribution" keyword in a "number" schema; and it supports an "operator" type for hyperparameters that accept a nested machine-learning operator.

Gradual Automation

The previous section described pipeline combinators and hyperparameter schemas. This section builds on that foundation to describe a programming model for gradual AutoML. Here, gradual means that users get to use total automation if they so desire, but can also exercise fine-grained manual control over any aspects of the pipeline. The programming model design was guided by three principles. First, progressive disclosure lets users start by learning just a subset of the constructs, only requiring them to learn more constructs as needed to accomplish more tasks. Second, orthogonality minimizes the number of independent constructs to make the programming model easy to learn, while maximizing the ways these constructs can be plugged together to make the programming model expressive [START_REF] Mailloux | Report on the Algorithmic Language ALGOL 68[END_REF]. Third, the principle of least surprise makes each construct look and behave the way most users would expect. That is why we started from well-established programming models sklearn [START_REF] Buitinck | API design for machine learning software: Experiences from the scikit-learn project[END_REF] and JSON Schema [START_REF] Pezoa | Foundations of JSON schema[END_REF] and try to be faithful to their expectations.

Figure 5 show the learning curve of gradual AutoML from zero to mastery. The rest of this section describes each set of constructs, starting at the bottom and working upwards one layer at a time.

Use total automation. Lale provides an AutoPipeline operator. From the outside, it looks and feels just like any sklearn operator with fit and predict methods. Internally, fit runs AutoML on a predefined planned pipeline, which users need not dwell on. Hence, total automation with Lale is trivial to learn for users who already know sklearn (least surprise).

Inspect returned pipeline. One level up from total automation is to look at its result: even users who have little interest in controlling the pipeline may want to see it for transparency. Lale provides a visualize method and a round-trippable pretty_print method that returns a pipeline's Python code. Plan pipeline from scratch. Learning about the combinators and the auto_configure method from Section 2 brings users to the next level. To further assist in using Lale's collection of 216 operators, Lale generates documentation from schemas and uses schemas for error checking [START_REF] Habib | Finding data compatibility bugs with JSON subschema checking[END_REF][START_REF] Rak-Amnouykit | Extracting hyperparameter constraints from code[END_REF].

Refine returned pipeline. Data science, even with automation, is often iterative. After an AutoML search (whether with full automation or with a hand-planned pipeline), users may want to tweak the pipeline for another search. Lale supports this with functions for breaking down a pipeline into smaller pieces, which can then be composed back into a new pipeline using combinators (orthogonality).

Control search space details. To exercise even more control in a localized and composable way, users can disable AutoML search for a sub-pipeline with freeze_trainable, or they can disable refitting with freeze_trained. Furthermore, users may want to locally tweak the schema for some hyperparameters of an operator, such as adjusting a range. Given an operator A, calling A.customize_schema(..) returns a new operator that is the same as A except for the given schema tweaks. Again, these constructs are orthogonal to combinators: frozen or customized operators can be used like any others.

Add new operators. Given an operator without a schema, Lale infers a simple schema that just sets all hyperparameters to their defaults. Users can then call customize_schema to make them selectively tunable. Or they can write a comprehensive schema and attach it to the operator using make_operator.

Write pipeline grammars. Some AutoML tools [START_REF] Olson | Automating biomedical data science through tree-based pipeline optimization[END_REF][START_REF] Alex | RECIPE: A grammar-based framework for automatically evolving classification pipelines[END_REF][START_REF] Drori | Automatic machine learning by pipeline synthesis using model-based reinforcement learning and a grammar[END_REF] explore not a rigid template for pipeline topologies but a possibly unbounded space of topologies, including some that define a context-free grammar using Lale's combinators [START_REF] Katz | Exploring context-free languages via planning: The case for automating machine learning[END_REF][START_REF] Marinescu | Searching for machine learning pipelines using a context-free grammar[END_REF]. Lale lets users define such search spaces by declaring recursive operators, which are self-referential definitions (e.g., g.p = g.p >> q). Recursive operators are orthogonal to the other constructs such as combinators, with which they can nest. The methods sample and unfold turn these into normal Lale pipelines suitable for auto_configure. Add new optimizers. Only few users get so close to mastery that they desire to add their own optimizers. While the details will vary on a case-by-case basis, the next section describes a principled translation scheme that helped Lale target several existing optimizers.

Translation Scheme

This paper adds a third execution mode to sklearn-style pipelines: the first two modes are prediction and training, and the third mode is AutoML search. Lale's API for this mode is auto_configure, which first translates the pipeline to a search space for a given optimizer and then runs that optimizer.

Figure 6 details that interaction to illustrate how the translation scheme (encode and decode) interacts with an optimizer and how it uses the first two execution modes (predict and fit). The starting point is a planned pipeline, which has operators whose hyperparameters need to be tuned or operator choices (| combinator) whose operators need to be selected.

Step "encode search space" translates the planned pipeline to an optimizer-specific search space. Once called, the optimizer loops over trials, which are search points it selects based on its solver, e.g., TPE [START_REF] Bergstra | Algorithms for hyperparameter optimization[END_REF].

Step "decode search point" turns an optimizer-specific search point (i.e., a specific value combination in the search space) into the configuration needed for instantiating a trainable pipeline. In a trainable pipeline, all hyperparameters and operator choices are bound to specific values. Fitting the trainable pipeline to training data (folds) yields a trained pipeline, where the learnable coefficients of all operators are also bound. The next step is to call predict on the trained pipeline and to score the result to compute a loss for the optimizer. After wrapping up its loop, the optimizer returns the best trial, and auto_configure returns the corresponding trained pipeline.

The translation scheme is one of the contributions of this paper. Its job is to enable the modular combinator-based syntax for users, by weaving together the operators and their hyperparameter schemas and rewriting them to a form suitable for a given optimizer. Users can thus make local changes using orthogonal constructs without worrying about their impact on the rest of the pipeline. To target a variety of optimizers, we structured the translation scheme as a series of rewriting passes between different forms. Each form has its own syntax restricting how a search space can be expressed. Figure 3 shows the starting form. Each pass is designed to be semantics-preserving, in the sense that the search space encodes the same set of pipeline instances before and after the pass. The running example for this section is the pipeline PCA >> (J48 | LR); Figure 4 shows its schemas in starting form.

Normalize. Figure 7 shows the normal form, which is a disjunction of objects. This form is crafted to simplify later rewriting passes by letting them start from a less feature-laden form. While nesting is still possible in normal form, it is restricted to objects. Also, once in normal form, schemas contain no conjunctions or negations, which would inhibit local reasoning. Normalization works in a bottom-up pass. At each level, it performs disjunction hoisting using rewrites such as De-Morgan's law. Negation and conjunction are pushed into primitive schemas. This can temporarily give rise to uninhabited or trivial nested schemas, which the pass simplifies away with another set of rewrites.

Our translation scheme normalizes the schema of each operator in a pipeline separately. Combine. Figure 8 shows the combined form, which represents the entire pipeline together instead of each operator separately. Combining encodes combinators as follows. The >> and & combinators turn into an object schema (dict), where each operator becomes a step s i . The | combinator turns into a disjunction schema (∨), with an additional discriminant D to track what was chosen. Higher-order operators yield nested object schemas (dict). The encoding preserves sufficient information to enable later decoding. After this pass, the encoding is conceptually similar to that of Hyperopt [START_REF] Bergstra | Hyperopt: a Python library for model selection and hyperparameter optimization[END_REF], so our implementation provides a simple backend translating it into a Hyperopt search space.

Flatten. Most other optimizers require a less hierarchical search space than the combined form. Figure 9 shows the flattened form, which is a disjunction of objects without further nesting. The flatten pass hoists properties from nested objects, renaming them as needed for uniqueness (in a way that enables later decoding). The flattened form is close to what SMAC [START_REF] Feurer | Efficient and robust automated machine learning[END_REF] needs, except that objects of different disjuncts have different properties. In SMAC, all disjuncts have the same properties, with conditionals on the side indicating which variables are relevant. The SMAC backend thus generates these conditionals, and renders the space in SMAC's parameter configuration space format [START_REF]SMAC Parameter Configuration Space (PCS)[END_REF].

The ADMM backend uses a partially flattened form. ADMM [START_REF] Liu | An ADMM based framework for AutoML pipeline configuration[END_REF] supports a pipeline of operator choices. The backend thus leaves the top-level spine of a pipeline intact as a pipeline of choices. If one of the choices contains a complex operator (e.g. a nested pipeline), it is flattened as for SMAC.

Discretize. The optimizers GridSearchCV and HalvingGridSearchCV from sklearn [START_REF] Buitinck | API design for machine learning software: Experiences from the scikit-learn project[END_REF] require continuous ranges to be made categorical. Discretization for a hyperparameter first includes the default and then samples a user-configurable number of additional values from the range based on the prior distribution. Figure 10 shows the discretized form when generating two values per range.

Results

This section presents experiments evaluating Lale's usability and expressiveness. While the experiments focus on tabular data and do not use deep learning, Lale also works with other data modalities such as text [START_REF] Chaudhary | AutoText: An endto-end AutoAI framework for text[END_REF], code [START_REF] Mavroudeas | Learning GraphQL query cost[END_REF], and time-series [START_REF] Roy | A semi-supervised deep learning algorithm for abnormal EEG identification[END_REF], including with pipelines that contain deep-learning operators.

RQ1: How usable is gradual AutoML with combinators? To explore this research question, we compare Lale to the popular sklearn machine learning framework in a user study with 18 participants. The study design is between-subjects [START_REF] Creswell | Research design: Qualitative, quantitative, and mixed methods approaches[END_REF], where participants are randomly assigned to a Lale or sklearn version of the study. We asked participants to perform four tasks involving sample machine learning pipelines: (T1) inspecting and understanding an existing pipeline, (T2) refining the given pipeline by replacing operators and hyperparameters, (T3) debugging a failing pipeline, and (T4) searching across a space of potential pipelines. The tasks are based on the principle of progressive disclosure in that we provide a sample pipeline (Figure 11) using common manual constructs in either sklearn or Lale format and ask participants to perform tasks that use a progressively expanding set of constructs. All participants self-reported having at least some sklearn experience, worked at IBM, and were not further compensated for their voluntary participation. The supplemental material describes the detailed study design and methodology, including the sklearn and Lale notebooks used.

One surprising result of the user study is that four (44%) of the sklearn participants were not able to correctly identify the branching structure of the preprocessing subpipelines of the sample pipeline during T1 (seen in Figure 11), compared to a single Lale participant (11%). Two of the Lale participants also used inspection features in Lale such as the visualize and pretty_print methods. In T4, two (22%) of the sklearn participants were either unable to correctly (or at all) implement some method of searching across a space of potential pipelines whereas all of the Lale participants were able to complete the task. When implementing T4, most of the Lale participants (6 or 67%) used the or combinator (|) and the AutoML execution mode to perform automated search as opposed to more manual methods. This is compared to two (22%) sklearn participants using automated search via GridSearchCV. The relatively high success rate of Lale participants both in understanding existing pipelines that use the and combinator (&) and in composing new planned pipelines that use the or combinator (|) suggests that new Lale users are able to pick up and use these independent constructs. This supports the principle of orthogonality in the Lale programming model. Automated search also tends to be more concise than manual implementations for the search task, perhaps contributing to Lale participants on average implementing the task in 25 fewer lines of code (LoC) (59% less) compared to sklearn participants. Lale participants also on average performed all of the tasks faster than sklearn participants, over 2.5 minutes faster total (13% faster). (In reality, the time difference would likely be even higher, since we stopped participants who take a manual approach early, see supplement for details.) For T2 and T3, participants performed equally well in both versions of the study. We note that some Lale participants were able to refine pipelines without referring to documentation, relying on their sklearn knowledge to replace operators and define their hyperparameters. This supports the principle of least surprise in that Lale operators work in ways that experienced sklearn users expect. Table 1 summarizes the results from the user study. Answer to RQ1: In our user study, gradual AutoML with combinators was usable enough to increase task correctness by more than 20% while reducing lines of code by more than half.

RQ2: Does the translation scheme work for diverse planned pipelines? To explore this research question, we used Lale's auto_configure on several planned pipelines with Hyperopt and its TPE solver. We used auto-sklearn [START_REF] Feurer | Efficient and robust automated machine learning[END_REF] as a state-of-the-art baseline; it uses SMAC for joint algorithm selection and hyperparameter tuning similar to our work, but also uses meta-learning, which we do not yet do. We used a 2.0GHz virtual machine with 32 cores and 128GB memory and gave each search a 1 hour time budget with a timeout of 6 minutes per trial, which corresponds to the default setting of auto-sklearn. We chose 14 datasets from OpenML [START_REF] Vanschoren | OpenML: Networked science in machine learning[END_REF] (CC-BY license) that allow for meaningful optimization (as opposed to just the initial few trials) within that 1-hour budget.

Our datasets are drawn from the AutoML Benchmark [START_REF] Gijsbers | An open source AutoML benchmark[END_REF] and four of them were also used in the auto-sklearn paper [START_REF] Feurer | Efficient and robust automated machine learning[END_REF]. (The goal of our evaluation is orthogonal to the sizes of the datasets.) We used a 66:33% train:test split with 5-fold cross validation on the train set during optimization.

Table 2 summarizes the results. Column AUTOSKL shows the auto-sklearn baseline; LALE-AUTO uses Lale's AutoPipeline operator from Section 3; LALE-AD3M and LALE-TPOT use Lale grammars inspired by AlphaD3M [START_REF] Drori | Automatic machine learning by pipeline synthesis using model-based reinforcement learning and a grammar[END_REF] and TPOT [START_REF] Olson | Automating biomedical data science through tree-based pipeline optimization[END_REF] unfolded with a maximal depth of 3; LALE-ADB uses a pipeline with a choice of preprocessors and the AdaBoostClassifier higher-order operator; and ASKL-ADB is auto-sklearn [START_REF] Feurer | Efficient and robust automated machine learning[END_REF] with the set of classifiers limited to AdaBoost with only preprocessing. (See supplement for details.) All searches succeed and most yield performance in the same ballpark as auto-sklearn (aside from shuttle, which triggers an auto-sklearn issue that other users have also encountered). Considering all 14 datasets, LALE-AUTO performs on average 0.7% better than AUTOSKL.

Excluding shuttle, Lale performs on average 0.3% worse than AUTOSKL. These 0.3% are smaller than the standard deviation on most datasets, and based on a t-test, they are not statistically significant. Answer to RQ2: The translation scheme works for pipelines with complex topologies, including those generated by grammars inspired by AlphaD3M and TPOT, as well as pipelines including ensembles, producing competitive results.

RQ3: Does the translation scheme work for diverse optimizer backends? To explore this research question, Table 3 lists test accuracies using 9 different optimizers for the OpenML phoneme dataset. The planned pipeline uses a scaler, a polynomial feature generator, and a choice between XGBoostClassifier and LGBMClassifier. The other experimental settings are the same as above. All translations succeed, and most yield reasonable accuracy, except for the ones that timed out. The translated search spaces for GridSearchCV and HalvingGridSearchCV were correct but too large to explore within the time budget. Most people would only use grid-based search for very small search spaces, but then, suffer from exploring the space less thoroughly than "proper" AutoML tools. Answer to RQ3: The translation succeeds for Hyperopt (3 optimizers), Hyperband, ADMM (3 optimizers), GridSearchCV, and HalvingGridSearchCV.

Related Work

Combinators, which were discovered in the 1920s [START_REF] Schönfinkel | Über die Bausteine der mathematischen Logik[END_REF], have long been prominent in functional programming languages such as Lisp [START_REF] Guy | Common Lisp: The Language[END_REF]. The advantage of their tacit style has been recognized more broadly; for instance, the Unix pipe [START_REF] Mcilroy | A research UNIX reader: Annotated excerpts from the programmer's manual, 1971-1986[END_REF] can be viewed as a combinator. Combinators have also enabled some sophisticated innovations such as parser combinators [START_REF] Leijen | Parsec: Direct style monadic parser combinators for the real world[END_REF].

Combinators and data science. MapReduce [START_REF] Dean | MapReduce: Simplified data processing on large clusters[END_REF] is a successful programming model for largescale data processing named after two combinators [START_REF] Lämmel | Google's MapReduce programming model -revisited[END_REF]. Allison [START_REF] Allison | Models for machine learning and data mining in functional programming[END_REF] describes Haskell combinators for converting between different machine-learning model types, but does not discuss AutoML. Sklearn [START_REF] Buitinck | API design for machine learning software: Experiences from the scikit-learn project[END_REF] provides combinators (make_pipeline and make_union) as well as some basic AutoML (GridSearchCV, HalvingGridSearchCV, and RandomizedSearchCV) but only integrates combinators with AutoML loosely, instead requiring users to perform manual name-mangling and non-compositional repetition. Lale builds on sklearn, extending and streamlining its combinators and tightly integrating them with AutoML. Pilat et al. [START_REF] Pilat | Asynchronous evolution of data mining workflow schemes by strongly typed genetic programming[END_REF] describe a combinator language suitable as a target for automatic code generation with genetic programming, but not intended to be written by hand. Unlike our work, they focus on total (not gradual) automation and do not provide an or combinator (|).

AutoML programming models. Auto-WEKA demonstrated how to support conditional hyperparameters on multiple solvers, but the paper focused on a single planned pipeline, not gradual automation [START_REF] Thornton | Auto-WEKA: Combined selection and hyperparameter optimization of classification algorithms[END_REF]. While both hyperopt-sklearn [START_REF] Komer | Hyperopt-sklearn: Automatic hyperparameter configuration for scikit-learn[END_REF] and auto-sklearn [START_REF] Feurer | Efficient and robust automated machine learning[END_REF] support some gradual automation, instead of tightly integrating gradual automation with combinators, they require users to drop to a bespoke lower-level API: Hyperopt for hyperopt-sklearn and SMAC's parameter configuration space format [START_REF]SMAC Parameter Configuration Space (PCS)[END_REF] for auto-sklearn. We argue that JSON Schema [START_REF] Pezoa | Foundations of JSON schema[END_REF], being a language with a track record of wide-spread use by programmers who are not AutoML experts, is less low-level; e.g., it is the backbone of OpenAPI [START_REF]OpenAPI specification (fka swagger restful api documentation specification)[END_REF]. We also built a tool to extract schemas from sklearn docstrings [START_REF] Baudart | Mining documentation to extract hyperparameter schemas[END_REF]. In Optuna, users write imperative code that first samples values, then uses either if-statements for operator choice or passes values to constructors for hyperparameter tuning [START_REF] Akiba | Optuna: A next-generation hyperparameter optimization framework[END_REF].

In contrast, combinators are more concise, more localized, and less surprising to sklearn users. OBOE [START_REF] Yang | OBOE: Collaborative filtering for AutoML initialization[END_REF] proposes an alternative meta-learning approach from that of auto-sklearn; it requires an initial discrete set of models with hyperparameters to choose from, and Lale could potentially be used to specify this set of models. In PyGlove, users start from a manually written neural network which they then "hyperify" by replacing a component or hyperparameter with a "hyper value" [START_REF] Peng | PyGlove: Symbolic programming for automated machine learning[END_REF]. Their notion of "hyper" is similar to our notion of "planned", but they do not use combinators, and they focus only on neural architecture search. AutoGluon-Tabular focuses on total automation and the paper does not discuss gradual automation [START_REF] Erickson | AutoGluon-Tabular: Robust and accurate AutoML for structured data[END_REF]. In AutoGOAL, users specify a hierarchy of pre-defined "indicator" types, then the system tries out any pipeline of operators that gets from the input to the output [START_REF] Estevez-Velarde | Solving heterogeneous automl problems with AutoGOAL[END_REF]. AutoGOAL's API deviates from sklearn conventions, and while it uses the syntax of Python 3 types, it does not capture constraints, nor is it compatible with Python 3 type checkers.

Limitations and Societal Impacts

A limitation of the user study is that it used only 18 participants (9 for the sklearn treatment plus 9 different participants for Lale), and that all participants worked at IBM. Another potential limitation is that researcher bias may be present in the design of the user study tasks. We designed the tasks to showcase and validate aspects of the Lale programming model which may potentially bias tasks in favor of Lale. We attempt to mitigate this by only selecting participants who are experienced in sklearn and by only allowing Lale newcomers to participate in the Lale version of the study.

One limitation of the translation scheme is that it is not guaranteed to handle all JSON Schema features; in fact, we encountered and fixed some limitations early during development. Fortunately, there were no new limitations recently, and there is theoretical work that could be used to tackle them if they arise [START_REF] Baazizi | Not elimination and witness generation for JSON Schema[END_REF]. A limitation of schema-based constraints is that they currently do not capture interdependence between operators. We mitigate that by providing not just hyperparameter schemas but also dataset schemas, which enable us to check whether data flowing from one operator to another in a pipeline is compatible [START_REF] Habib | Finding data compatibility bugs with JSON subschema checking[END_REF]. A limitation of the programming model is that, while it achieves a high level of sklearn compatibility, Python code that explicitly reflects over dynamic types can still expose differences. We mitigate this by testing that Lale interoperates well with a broad set of sklearn features including metrics, cross-validation, operators, etc.

One potential societal impact is that enabling automation of searching machine learning pipelines may encourage computationally-heavy approaches to data science tasks and raise CO 2 emissions. On the other hand, we hope gradual automation will encourage data scientists to use their domain knowledge to more selectively and efficiently automate their machine learning work. In future work, we are planning to adopt caching techniques [START_REF] Li | Exploiting reuse in pipelineaware hyperparameter tuning[END_REF][START_REF] Xin | Helix: Accelerating human-in-the-loop machine learning[END_REF] to further reduce CO 2 emissions.

Conclusion

This paper presents combinators for gradual AutoML, along with a novel translation scheme from combinators to existing optimizers. We have implemented our combinators in an open-source library called Lale, while ensuring they interoperate smoothly with the sklearn ecosystem. Besides being open-source, Lale is also actively used in a commercial AutoML product, highlighting the real-world value of the underlying concepts. Our ultimate goal is to make AutoML as accessible as manual ML by offering a gradual combination of both.

  pipeline ::= individualOp | pipe | and | or individualOp ::= NAME | NAME (hyperParam* ) hyperParam ::= NAME = expression expression ::= VALUE | pipeline pipe ::= pipeline >> pipeline and ::= pipeline & pipeline or ::= pipeline | pipeline

Figure 1 : 4 >>

 14 Figure 1: Pipeline syntax. Non-terminals are in lower-case italics , literal tokens ((, ), =, >>, &, |) are in type-writer font, other terminals are in all-caps ITALICS, and the meta-syntax (::=, |, *) indicates rules, choice, and Kleene star.

Figure 2 :

 2 Figure 2: Pipeline example with two preprocessing sub-pipelines. Features from both are concatenated and piped to a boosted ensemble, whose base estimator is a choice of classifiers.

Figure 3 :Figure 4 :

 34 Figure3shows the most important parts of the schema syntax (eliding some details, such as defaults, integers, and array schemas, that are not needed for examples in this paper). The start symbol, schema, can be an enumeration, a range, an object, or it can nest other schemas by using one of three logic connectives. An enum schema specifies a categorical hyperparameter with a choice between one or more values. Figure 4 includes multiple examples of enums: [mle], [true, false], [0.25], [linear, sag, lbfgs], etc. A range schema specifies a continuous hyperparameter with a minimum and

Figure 5 :

 5 add new optimizers new search space encoder / search point decoder write pipeline grammars recursive operator definition, sample, unfold add new operators make_operator(impl, schemas) control search space details freeze, customize_schema refine returned pipeline functions that decompose pipelines plan pipeline from scratch >>, |, &, operator library, auto_configure inspect returned pipeline visualize, pretty_print use total automation AutoPipeline, fit, predict mastery 0 Gradual automation with progressive disclosure.

Figure 6 :

 6 Figure 6: Interaction with an optimizer.

Figure 7 :

 7 Figure 7: Normalized schemas: syntax (left) and example (right).

s 0 :Figure 8 :

 08 Figure 8: Combined search space: syntax (left) and example (right).

Figure 9 :

 9 space ::= object (∨ object)* object ::= dict { (NAME : prop)* } prop ::= enum | range dict{N : (0..1), D: [J48], R: [false], C: (0..0.5)} ∨ dict{N : (0..1), D: [J48], R: [true, false], C: [0.25] } ∨ dict{N : [mle], D: [J48], R: [false], C: (0..0.5)} ∨ dict{N : [mle], D: [J48], R: [true, false], C: [0.25] } ∨ dict{N : (0..1), D: [LR], S: [linear], P : [l1, l2] } ∨ dict{N : (0..1), D: [LR], S: [linear, sag, lbfgs], P : [l2] } ∨ dict{N : [mle], D: [LR], S: [linear], P : [l1, l2] } ∨ dict{N : [mle], D: [LR], S: [linear, sag, lbfgs], P : [l2]} Flattened search space: syntax (left) and example (right).

Figure 10 :

 10 Figure 10: Discretized search space: syntax (left) and example (right).

Figure 11 :

 11 Figure 11: Sample pipeline used in user studies.

Table 1 :

 1 Summary of user study results.

	VERSION	COUNT	T1 CORRECT	T4 CORRECT	T4 LOC MEDIAN (ALL)	TOTAL TIME MEAN (STDDEV)
	Lale	9	89%	100%	10 (7, 8, 9, 9, 10, 10, 10, 14, 75)	17:36 (5:03)
	sklearn	9	56%	78%	23 (12, 15, 16, 22, 24, 43, 100, 100)	19:26 (6:34)
	Total	18	72%	89%	14	18:54 (5:45)

Table 2 :

 2 Test accuracy for 14 OpenML classification tasks, using 4 Lale search spaces with Hyperopt.

			Absolute accuracy mean (and stddev) over 5 runs		100 * (LALE/AUTOSKL -1)
	DATASET	AUTOSKL LALE-AUTO LALE-TPOT LALE-AD3M	LALE-ADB	ASKL-ADB	AUTO	TPOT	AD3M	ADB
	australian	85.1(0.4)	86.2 (0.0)	85.9 (0.6)	86.8 (0.0)	86.0 (1.6)	84.7 (3.1)	1.3	0.9	2.0	1.1
	blood	77.9(1.4)	75.3 (0.0)	77.5 (2.5)	74.7 (0.7)	77.1 (0.7)	74.7 (0.8)	-3.3	-0.5	-4.0	-1.0
	breast-cancer	73.0(0.6)	73.0 (0.0)	71.4 (1.1)	69.5 (3.3)	70.9 (2.0)	72.4 (0.5)	0.0	-2.3	-4.9	-2.9
	car	99.4(0.1)	97.7 (0.0)	99.1 (0.1)	92.7 (0.6)	98.3 (0.3)	98.2 (0.2)	-1.6	-0.2	-6.7	-1.1
	credit-g	76.6(1.2)	75.7 (0.0)	74.1 (0.5)	74.8 (0.4)	76.1 (1.3)	76.2 (1.0)	-1.1	-3.2	-2.4	-0.7
	diabetes	77.0(1.3)	76.3 (0.0)	76.4 (1.1)	77.9 (0.2)	76.0 (0.5)	75.0 (1.0)	-0.9	-0.8	1.1	-1.3
	jungle-chess	88.1(0.2)	92.4 (0.0)	88.9 (2.0)	74.1 (2.0)	89.4 (2.3)	86.9 (0.2)	4.9	0.9	-15.8	1.5
	kc1	83.8(0.3)	83.4 (0.0)	83.5 (0.5)	83.6 (0.2)	83.3 (0.4)	84.0 (0.3)	-0.5	-0.4	-0.2	-0.6
	kr-vs-kp	99.7(0.0)	(0.0)	99.4 (0.0)	96.8 (0.1)	99.5 (0.1)	99.5 (0.2)	-0.2	-0.3	-2.9	-0.2
	mfeat-factors	98.7(0.1)	97.1 (0.0)	97.9 (0.5)	97.5 (0.1)	97.5 (0.4)	97.9 (0.1)	-1.6	-1.5	-1.2	-1.2
	phoneme	90.3(0.4)	89.5 (0.0)	89.6 (0.4)	76.6 (0.0)	90.1 (0.4)	91.4 (0.2)	-0.8	-0.8	-15.2	-0.2
	shuttle	87.3(11.6)	100.0 (0.0)	99.9 (0.0)	99.9 (0.0)	100.0 (0.0)	100.0 (0.0)	14.5	14.5	14.4	14.6
	spectf	87.9(0.9)	87.7 (0.0)	88.4 (2.2)	83.6 (6.9)	88.4 (2.6)	89.7 (2.9)	-0.2	0.6	-4.9	0.6
	sylvine	95.4(0.2)	95.0 (0.0)	94.4 (0.7)	91.3 (0.1)	95.1 (0.2)	95.1 (0.1)	-0.4	-1.1	-4.3	-0.3

Table 3 :

 3 Test accuracy (mean and stddev) for the phoneme dataset using multiple optimizer backends.

		Hyperopt		Hyperband		ADMM		GridSearchCV	Halving-
	TPE	Anneal	Rand		RND	BOBa	GPRND		GridSearchCV
	88.75 (0.2)	88.94 (0.3)	88.75 (0.5)	87.94 (0.2)	89.00 (0.2)	89.14 (0.2)	88.61 (0.3)	Timed out	Timed out

(c) Did you include any new assets either in the supplemental material or as a URL?

[Yes] The Lale library is open-source at https://github.com/ibm/lale/. In addition, the supplemental material includes a copy of the code, based on a snapshot from the time when we ran the experiments for this paper. (d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We used open datasets and open-source software. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We used standard datasets from OpenML [START_REF] Vanschoren | OpenML: Networked science in machine learning[END_REF] that contained neither personally identifiable information nor offensive content.


A Ougazzaden

D J Rogers

F Hosseini Teherani

G Orsal

T Moudakir

S Gautier

V E Sandana

F Jomard

M Abid

M Molinari

F Hosseini

M Troyon

P L Voss

D Mcgrouther

J N Chapman

Epitaxial MOVPE growth of highly c-axis oriented InGaN/GaN films on ZnO-buffered Si (111) substrates

Keywords: metalorganic vapour phase epitaxy, InGaN, pulsed laser deposition, ZnO, Si substrate, lift-off

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Recently, the Gallium Nitride (GaN) and Zinc Oxide (ZnO) materials systems have attracted considerable attention because of their use in a broad range of emerging applications including lightemitting diodes [1][2] and solar cells [3][4]. GaN and ZnO are similar in that they have direct wide bandgaps, wurtzite crystal structure, high thermal stability and comparable thermal expansion coefficients. GaN has the advantage, however, of a mature know-how for p-type doping, while ZnO has proven to be more crystallographically compliant to non-native substrates than GaN. With a view to exploiting both of these strong points, ZnO layers have been proposed as buffer layers for the regrowth of GaN-based p-n devices on amorphous and mismatched substrates [START_REF] Rogers | Proc. SPIE[END_REF][START_REF] Rogers | [END_REF][7][8][9].

Alloying with In is important for engineering a reduced bandgap in GaN and thus facilitating quantum wells and devices which can function over the whole visible range. Previously, epitaxial growth of InGaN on ZnO has been reported on sapphire (Al 2 O 3 ) substrates [10][11]. However, Al 2 O 3 substrates are insulating, limited in size and relatively expensive. Silicon (Si) substrates, on the other hand, are cheaper, available in larger formats, have higher thermal/electrical conductivities and offer the potential for integration with mature Si-based electronics [12][13]. In this paper, we investigate the growth of InGaN/GaN on ZnO-buffered Si (111) substrates using metalorganic vapour phase epitaxy (MOVPE), which is the dominant growth technology for InGaN-based materials and devices. One serious obstacle to the epitaxy of InGaN on ZnO is the stability of ZnO in the MOVPE process. In particular, exposure to ammonia (NH 3 ) precursor and H 2 carrier gas at growth temperatures over about 650°C results in the decomposition of the ZnO [14][15]. This can lead to the formation of interfacial layers/compounds, poor epitaxy and the unintentional incorporation of liberated Zn and O into the InGaN epilayer, which is detrimental for the optical and electrical properties.

EXPERIMENT

ZnO thin film templates were grown on 2-inch diameter Si (111) substrates using Pulsed Laser Deposition (PLD) with a Coherent KrF excimer laser (248nm) [START_REF] Rogers | Proc. SPIE[END_REF]. PLD was chosen because the high adatom mobility in this growth technique allows the ZnO to crystallise readily on amorphous substrates, such as the native SiO 2 passivation layer on the surface of the Si substrate. 40 nm thick GaN and InGaN layers (of various thicknesses) were then grown on the ZnO films using a low pressure, T-shaped, MOVPE reactor [16] with Trimethylgallium (TMG) as the Ga source and Trimethylindium (TMIn) as the In source. A combination of NH 3 and Dimethylhydrazine (DMHY) was used as a source of atomic nitrogen. DMHY was added because of its' low decomposition temperature, which enhances the concentration of atomic nitrogen in the gas phase compared to NH 3 alone, allowing GaN growth at lower temperatures. Nitrogen was employed as a carrier gas [17] during the whole growth process, in order to avoid etching of the ZnO surface and ensure In incorporation [18]. The impact of In content was studied by adjusting the TMIn flow at constant TMGa flow. The surface roughness and growth rate of InGaN layers were monitored during the growth using insitu reflectometry. Images of fracture cross-sectional samples were obtained using a Hitachi S-4800 field-emission scanning electron microscope (SEM). For transmission electron microscopy (TEM) investigation, cross-sectional samples were prepared using the in-situ liftout technique in a Dualbeam FIB (FEI Nova Nanolab 200). High resolution (HR-TEM) imaging was performed using an FEI Tecnai F20 equipped with a field emission gun source operated at 200 keV. The crystalline quality and indium composition of the InGaN layers were characterized by high resolution x-ray diffraction (HR-XRD), using Cu K α radiation in Panalytical MRD-Pro system equipped with a hybrid monochromator. Concentration profiles of In, Ga, N and Zn along the growth direction were obtained by MCs+ Secondary Ion Mass Spectroscopy (SIMS) in a Cameca IMS4F system. The primary (incident) ions were 133Cs+ with an impact energy of 5.5 keV. Positively charged secondary ions were detected. The optical properties of the InGaN/GaN/ZnO were studied using a home-made cathodoluminescence (CL) system in an SEM [19]. The image shows continuous lattice fringes across the InGaN/GaN/ZnO interfaces, with no evidence of a misfit layer, or any interfacial compounds. The FFT confirms that both the lattice plane alignment and lattice spacing are similar in the InGaN/GaN and ZnO layers. Thus the TEM studies suggest that there has been epitaxial growth. Figure 4 shows the XRD 2θ-ω scans as a function of TMIn/III ratio during growth.

RESULTS & DISCUSSION

InGaN ZnO

XRD study revealed the InGaN/GaN/ZnO layers to have single-phase wurtzite structure with strong c-axis orientation. The scans in Figure 4 present two distinct peaks, which were indexed as the (0002) peaks of the ZnO and the InGaN. An increase of the TMIn/III ratio led to a shift of the InGaN peak to lower angle and thus a higher out-of-plane "c" lattice parameter. The scans in Figure 4 also exhibited a small shoulder on the right of the ZnO peak, which was not visible in the XRD scans for the ZnO films prior to InGaN/GaN growth. This shoulder might be from the GaN layer. The concentration of In in the InGaN layer with higher In content (peak on left) was calculated from the (0002) InGaN peak position via Vegard's law to be about 21.5%. The (0002) ZnO peaks for all samples grown on Si were observed to be shifted to a higher 2θ angle than expected for bulk ZnO, which suggests that they were under tensile strain in the a-plane. This has been reported elsewhere for ZnO grown on Si substrates [20]. After InGaN/GaN growth, the ZnO peaks shifted slightly higher in 2θ (Figure 4). Thus the ZnO films absorbed some stress (tensile in this case) from the InGaN/GaN layers (as was observed in previous experiments on the MOVPE growth of GaN on ZnO/c-Al 2 O 3 [START_REF] Rogers | [END_REF]). Figure 5 shows the SIMS concentration profiles for Ga, In, N and Zn along the growth direction (in a sample with a thicker ZnO layer than that shown in Figure 1). The profile for Zn indicates some diffusion of Zn into the InGaN/GaN near the GaN/ZnO interface [8][9]. The simulation suggested that an incident electron beam, with an energy of 5kV, stimulated a response from most of the InGaN/GaN layer depth. The spectrum itself shows two peaks located at about 445 nm (2.79 eV) and 565 nm (2.20 eV). The first of these peaks was indexed as InGaN with an In concentration of about 17.5 + 5 % [21], which is coherent with the value calculated from the XRD scan. The second (565 nm) peak has been observed in InGaN grown on bulk ZnO in the past and has been attributed to emission from Zn-related impurity centers in the InGaN [22]. This is coherent with the slight diffusion of Zn into the InGaN/GaN near the interface with ZnO, which was observed by SIMS. Finally, the InGaN/GaN was lifted-off the substrate by immersion in dilute HNO 3 . After several minutes, the ZnO dissolved completely and the InGaN/GaN separated from the substrate.

CONCLUSIONS

InGaN/GaN films were grown on ZnO-buffered Si (111) substrates by MOVPE. The dissociation of ZnO observed during conventional MOVPE growth of InGaN/GaN was combated through the use of a low pressure/temperature MOVPE approach with N 2 as a carrier gas and DMHY added to the NH 3 in order to facilitate atomic nitrogen production at lower temperatures. XRD, SEM, TEM, SIMS and CL characterisations showed smooth, highly c-axis oriented InGaN/GaN films having a well-defined interface with the ZnO underlayer, continuous lattice fringes across the interfaces and no evidence of any back-etching of ZnO. XRD and CL suggest that between 17.5 and 21.5% of In was integrated in the GaN, without any phase separation. Slight Zn diffusion into the InGaN/GaN was observed near the interface with ZnO. Thus it was demonstrated that use of a PLD-grown ZnO buffer layer allows highly c-axis oriented wurtzite InGaN/GaN to be grown on Si substrates without removal of the amorphous SiO 2 layer. Such an approach may open the way to the integration of InGaN based devices with Si-based electronics. The feasibility of chemical lift-off of the InGaN/GaN from the substrate was also demonstrated via dissolution of the ZnO in dilute HNO 3 . Such a wet-etching approach would allow wafer bonding of the InGaN/GaN onto alternative substrates, with advantages such as superior transparency, and/or electrical/thermal conductivity.

Figure 1

 1 Figure 1 shows an SEM image of a fracture cross-section of an InGaN/GaN/ZnO/Si sample with approximately 21.5% In. The sample shows continuous InGaN/GaN and ZnO layers. A well-defined interface between ZnO and GaN can be clearly observed but the InGaN/GaN interface is not obvious. The thicknesses of the InGaN/GaN and ZnO are about 180nm and 140nm, respectively.

Figure 1 .

 1 Figure 1. SEM fracture cross-section image of the InGaN/GaN/ZnO/Si heterostructure.

Figure 2

 2 Figure2shows a HR-TEM image of the ZnO/Si interface. In this image, the regular lattice fringes in the Si substrate are interrupted at the interface with the ZnO due to the presence of the amorphous SiO 2 overlayer (~3nm thick). The subsequent ZnO layer shows established lattice fringes after only 2 nm of growth, confirming the capacity of PLD-grown ZnO to crystallise on amorphous substrates.

Figure 2 .

 2 Figure 2. A HR-TEM image of the interface between the PLD-grown ZnO layer and the Si substrate.

Figure 3

 3 Figure3shows a HR-TEM image of the interfaces between the InGaN, GaN and ZnO layers plus the result of a Fast Fourier Transform (FFT) analysis.

Figure 3 .

 3 Figure 3. A HR-TEM image of the InGaN/GaN/ZnO interfaces plus a FFT analysis of the image (the arrows, in the former, suggest the position of GaN/ZnO interface).

Figure 4 .

 4 Figure 4. XRD 2θ-ω scans of the (0002) peak for ZnO/Si and the InGaN/GaN/ZnO/Si with varying In contents. The dashed line shows the peak position (34.43°) for a ZnO powder reference sample.

Figure 5 .

 5 Figure 5. SIMS elemental concentration depth profiles for the Zn, Ga, N and In signals for the InGaN/GaN/ZnO-buffered Si (log. scale). It should be noted that the tails in the concentration profiles for In, Ga, and N, which extend into the ZnO, are probably artifacts due to residual particles of InGaN in the probe crater.

Figure 6

 6 Figure6shows a normalized, room temperature (RT), CL spectrum plus a Monte Carlo simulation of the respective primary and back-scattered electron diffusion.

Figure 6 .

 6 Figure 6. Normalized, RT, CL spectrum under 5kV accelerating voltage, plus a Monte Carlo simulation of the primary (red) and back-scattered (blue) electron diffusion.

ACKNOWLEDGEMENTS

The authors would like to thank the French "Agence National de la Recherche" for financial support.


Viviana Del Rocío Hernández-Castañón

Steven Le Cam

Radu Ranta

Sparse EEG source localization in frequency domain

Keywords: EEG source localization, frequency domain, sparse inverse problem

This work presents an approach for EEG source localization when strong priors on predominant frequencies in the activities of the source are available. We describe the fundamentals of the used source reconstruction method based on a greedy approach, which can be applied indifferently in the time or frequency domain. The method is evaluated using simulated data reproducing realistic recorded activities in the context of fast periodic visual stimulation. In particular the advantage of reconstructing the source in the frequency domain against time domain is quantified in a realistic setup. Finally, the performances of the method are illustrated on real EEG signals recorded during a fast periodic visual stimulation task.

I. INTRODUCTION

One of the most widely used techniques for exploring the electrical brain activity is the electroencephalogram (EEG). The EEG electrodes, placed on the surface of the head, capture potentials generated by current sources situated inside the brain. EEG source estimation and localization implies a wide panel of techniques to reconstruct the source map and their activities [START_REF] Michel | EEG source imaging[END_REF]. Most importantly, when determining the sources that generate the EEG signal, constraints have to be incorporated in order to constrain and regularize this highly ill-posed inverse problem. In particular, we are interested, for reasons explained below, in approaches enforcing sparsity, for which the main hypothesis is that the recorded signals are generated by few active regions, possibly organized in a network.

While classical source localization methods are developped for time-domain signals, in some situation the relevant information lies in specific frequency bands. Very few approaches were proposed directly in the frequency domain ( [START_REF] Muñoz-Gutiérrez | Localization of active brain sources from EEG signals using empirical mode decomposition: A comparative study[END_REF]- [START_REF] Lütkenhöner | Frequency-domain localization of intracerebral dipolar sources[END_REF]), see also [START_REF] Michel | EEG source imaging[END_REF] for a brief review. Besides sparsity, we narrow down our interest to this particular situation, i.e., when sources of interest are band-limited or even pure oscillations.

Indeed, our target application is source reconstruction during fast periodic visual stimulation of the brain (FPVS).

FPVS is a privileged technique to study the brain face recognition system, as it is well controlled and provides highly reproducible responses [START_REF] Jonas | A face-selective ventral occipitotemporal map of the human brain with intracerebral potentials[END_REF] with strong SNR. This kind of protocol consists in a periodic presentation of natural images to the observer at a given base frequency (6Hz for the protocol used in this work), periodically interleaved with face images (at an oddball frequency: 1.2 Hz in this study, All Authors are with the Centre de Recherche en Automatique de Nancy (CRAN UMR 7039, University of Lorraine-CNRS, F-54000).

radu.ranta@univ-lorraine.fr i.e., every 5 images). Combined with invasive electrophysiological measurements (intracerebral EEG, also known as SEEG), these protocols allowed to identify the main areas recruited for face recognition along the ventral occipitotemporal cortex with a right predominance [START_REF] Jonas | A face-selective ventral occipitotemporal map of the human brain with intracerebral potentials[END_REF]. These areas synchronize their activities with the visual stimuli, both at the base and the oddball frequencies as illustrated in fig. 3.

We address in this paper the problem of reconstructing the responsive sources to such protocols from scalp EEG measurements, by taking benefit of the oscillatory nature of the sources of interests and of the fact that they are well localized in specific brain areas. We focus thus on frequency localized and sparse approaches. More precisely, we adopt in this work a method derived from the Matching Pursuit principle used for sparse localization in EEG called Single Best Replacement (SBR) [START_REF] Soussen | From Bernoulli-Gaussian deconvolution to sparse signal restoration[END_REF] already applied on EEG for time-frequency based localizations [START_REF] Korats | A space-time-frequency dictionary for sparse cortical source localization[END_REF].

In the present work, the reconstruction performances are analyzed from two angles: by applying the sparse source localization methodology in the time domain after filtering in the frequency bands of interests, as well as directly in the frequency domain after a FFT transform of the recorded channels.

II. METHODOLOGY/PERIODIC SOURCE LOCALIZATION

The EEG forward model consists in calculating the electrical potential observed on a given sensor when the configuration of the underlying electrical sources and the biophysical properties and geometry of the propagation environment are known [START_REF] Hallez | Review on solving the forward problem in EEG source analysis[END_REF]. A general linear forward model is based on a gain matrix, known as lead-field, which encodes the propagation coefficients between a freely oriented source at any position in the head and the measuring electrodes. A simplified model considers fixed orientation sources (i.e. for every possible position only one orientation is considered). Although restrictive, this simplified lead-field can be justified by biological considerations: dipolar current sources represent population of pyramidal neurons, which are not freely oriented but mostly orthogonal to the gray matter surface. Thus, a linear forward model projecting N sources to M electrodes expresses scalp recordings X as a linear combination of source amplitudes S through the propagation coefficients stored in the lead-field M × N matrix A, plus noise ε. Every column of A corresponds to a specific source position (i.e., contains the propagation coefficients between that specific position and the electrodes).

X = AS + ε (1)
The linear forward model in ( 1) is generally expressed in the time domain, meaning that X and S are, respectively, measurements and source amplitudes at different time instants (matrices M × T and S × T , with T the number of time samples). But model ( 1) is valid under any linear transformation of the signals such as the Fourier transform or linear filtering in specific frequency bands. Moreover, the model is valid for any specific time instant, when considered in time, as well as for any specific frequency bin (or band after band-pass filtering), when considered in frequency.

Since the lead-field or gain matrix A (M × N, M < N ) has a number of electrodes M much smaller than the number of possible source locations N , the system does not have a unique solution and constraints need to be imposed in order to reduce the solution space. Under sparsity regularization, the estimates of the source amplitudes and of their localization are given by ( 2):

Ŝ, Ĵ = argmin S,J X -A J S 2 2 + λN J ( 2
)
where J contains the indices of the selected columns from the lead-field A, representing source positions, N J represents the number of selected columns (cardinal of J), S are the magnitudes of the (selected) sources and A J is the submatrix containing the J indexed columns from A (hats indicate estimates of the indices J and source amplitudes Ŝ; estimating J is equivalent to choosing an estimated submatrix ÂJ ).

In [START_REF] Soussen | From Bernoulli-Gaussian deconvolution to sparse signal restoration[END_REF], the minimization problem ( 2) was addressed by a penalized Orthogonal Least Squares (OLS) regression called Single Best Replacement (SBR): the first term decreases with every new added source (index in Ĵ, thus column in A J and respective magnitude in Ŝ), while the (second) penalization term increases with a fixed amount defined by the user parameter λ. The algorithm, equivalent to classical OLS for λ = 0, iterates by adding or removing sources until convergence to a more or less sparse solution depending on the user choice of λ. While the original SBR is applied on vectors (i.e., X and S are column vectors), we present here two versions applied on matrices:

• Time domain. In this case, X is a matrix constructed as follows:

(1) band-pass filter the EEG signals (either at 1.2Hz or at 6Hz, see above) using narrow band (0.2Hz) 2nd order Butterworth filters; (2) define epochs based on the FPVS frequencies (base or oddball, i.e., 1/1.2 or 1/6 seconds) and average them to reduce the noise.

The final X matrix contains on every row a band-pass averaged signal having the length of one epoch; • Frequency domain. After Fourier transforming the signals of every electrode, select the complex values of the FFT at every frequency of interest (e.g., 1.2 Hz and harmonics or 6 Hz) to construct the frequency domain X complex matrix. Note that both positive and negative frequencies need to be considered, as the FFT has hermitian symmetry and both are needed in order to avoid phase indetermination. Applying the matrix version of the SBR will then yield real time or complex frequency amplitudes Ŝ and, most importantly, the set of indices Ĵ that will allow source localization. In order to test the feasibility of the source localization method, EEG scalp potentials were simulated by using a realistic head model with three comparments (Colin 27) extracted from Brainstorm toolbox [START_REF] Baillet | BrainStorm: a Matlab Toolbox for the Processing of MEG and EEG Signals[END_REF]. The cortical layer mesh (inner shell) has 7292 nodes, the outer skull and the scalp mesh (outer shell) have 1922 nodes. The source space consisted of a regularly spaced grid (7 mm) constructed inside the inner shell, which yielded N =6184 source positions. The scalp electrodes (M =64) were simulated according to the approximate 10-10 system of a Biosemi Cap. The leadfield matrix A was constructed using fixed orientations for all sources, orthogonal to the surface of the head. The EEG scalp potentials were simulated by projecting the sources on the sensors using a boundary element model (BEM) implemented in the Helsinki toolbox [START_REF] Stenroos | A Matlab library for solving quasi-static volume conduction problems using the boundary element method[END_REF].

III. RESULTS

A. Data simulation

Realistic EEG signals during FPVS contain peaks at oddball and base frequencies, i.e., 1.2Hz and 6Hz (and harmonics), being thus a mixture of sources at all these frequencies. But for our purposes, as the localization is performed either after band-pass filtering or directly on the peaks of interest after Fourier transform, the actual combination (the respective contributions) of oddball and base frequencies is irrelevant, the important difficulties coming from the interactions between sources having the same frequency. We have simulated different configurations of sources as follows (see fig. stream, about 8cm far from the first 1.2Hz source and deeper in the brain. This second source was delayed with respect to the first 1.2Hz source (phase delay π/6). C3 Two sinusoidal sources (same frequency f b = 6Hz, same phase) were placed symmetrically in left and right hemispheres, approximately in the primary visual areas in the posterior part of the brain. The distance between them was 8 cm. C4 Four f b = 6Hz sources, the two from C3 and two more, also symmetric between the two hemispheres, but closer one to the other (5 cm) and each one relatively close to the 6Hz sources used in C3 ( 3cm). They were also synchronized between them (same phase) and slightly delayed with respect to the other 6Hz sources (phase delay π/10). The amplitudes of all sources were identical (but their contribution to the scalp EEG varied of course due to the lead-field coefficients).

We have simulated increasingly complex signals by combining projections of the source configurations described above (4 types of EEG, from one 1.2 and two 6Hz sources to two 1.2 and four 6Hz sources). For each combination, white Gaussian noise was added to the simulated scalp potentials. Seven noise power levels have been used, computed so as to reach SNRs in the set {20dB, 10dB, 3dB, 0dB, -3dB, -10dB, -20dB} with respect to the projection of the first 1.2Hz source (C1). One hundred simulations per type of EEG and noise level were performed using the set up described previously.

B. Performance criteria on simulated EEG data

To compare the performance of the localization procedures, two criteria were used: the Goodness of Fit (GOF, computed with respect of the no-noise simulation) and a weighted distance of localization error (wDLE). The GOF is defined as:

GOF = 1 - A J S -ÂJ Ŝ 2 A J S 2 (3)
where A J S are the simulated electrical potentials without noise (see 1) and ÂJ Ŝ are the estimated potentials calculated as the projection of the reconstructed sources on the EEG electrodes. The higher the GOF, the closest is the estimation to the originally simulated data. We also propose an original weighted distance criterion derived from the Distance of Localization Error (DLE) [START_REF] Becker | A performance study of various brain source imaging approaches[END_REF], where the source distances are weighted by their contributions to the reconstructed potentials:

wDLE = j∈J α j min i∈I r j -r i 2 j∈J α j + i∈I α i min j∈J r i -r j 2 i∈I α i (4)
J is the set of the N J reconstructed sources at positions {r j } j∈ [1..Nj ] , and I is the set of the N I simulated sources placed at positions {r i } i∈ [1..Ni] . The coefficient α j is computed as the norm of the projection of the j th estimated source on the electrodes divided by the norm of the reconstructed scalp map. Similarly, α i is the weight given by the ratio of the norm of the potential due to the i th simulated source with respect to the norm of the potential of the simulated data. The first term of equation ( 4) penalizes the presence of false positive sources, while the second term penalizes true sources that are not recovered by the method.

C. Simulation results

The figure 2 sums up the performance in term of true positives (TP)/false positives (FP) and GOFs, and table I provides averages values of wDLE for each configuration. When a single superficial source is simulated for a given frequency (C1), the method is robust and tends to fail for strong noise only: the localization in frequency domain is accurate up to -10dB SNR, with the low wDLE and high GOFs (near 1), very few to no FP, while in the time domain the performance becomes weaker from the -3dB case (about 2 FP and wDLE over 1cm on average for this noise level). When two f o simulated sources are introduced (C2), the method clearly struggles in recovering the deeper (second) source, the frequency approach being accurate up to -3dB, while in the time domain the performances are mild from 3dB SNR, where the number of FP is near 2 with associated GOF values over 30% on average. It is worth noticing that the localization based on FFT produces globally much less FP in these two configurations of f o sources compared to the time domain.

Considering the reconstruction of the f b = 6Hz sources, the inversion in frequency and time domain do not differ significantly. When two f b sources are simulated (C3), the reconstruction is accurate over all SNRs (putting the -20dB case aside). It can be seen that about two FP are consistently found, but with very low amplitude and thus with unsignificant impact on the reconstructed data (low FP GOFs and low wDLE). However when four f b sources are simulated (C4), the method is failing even in the most favorable 20dB SNR case. Let recall that the first and second groups of two sources are synchronized two by two and slightly delayed between the two groups, and that they are all placed in the occipital areas. This becomes infeasible for the method to distinguish between them as they are relatively close in a region where the EEG sensors are only partly covering the projection of the dipoles. A closer look at the algorithm first iterations reveals that the algorithm is misled from the beginning by choosing false sources explaining a high portion of the data (see the high GOFs for the FP on fig. 2.C4, bottom), and the SBR strategy is not able to discard them afterwards.

To sum up, when the method is able to recover the sources, a clear advantage is given to the inversion in the frequency domain (C1 and C2 configurations), as the method focused on the frequency value fully supporting the information and as the temporal filtered version of the data still collects some noise residuals. Such strategy is helpful to recover profound sources, an important aspect when the reconstruction is carried out from the EEG measurements. In some case of source configurations where the source map is not identifiable by the method (e.g., close and synchronised activities), the inversion in the frequency domain is not helpful in disentangling the contributions of the sources.

D. Real signals

An example of the frequency based localization method is given here, on a 64-channels real EEG recording (BioSemi Cap) at a sampling rate of 256 Hz. Two approximately 1 minute long sessions were recorded using the protocol described in the introduction [START_REF] Jonas | A face-selective ventral occipitotemporal map of the human brain with intracerebral potentials[END_REF]. After average re-referencing, two sessions averaging and Fourier Transform, one can obtain spectra like the one presented fig. 3, with peaks at the oddball frequency of 1.2 Hz and harmonics (2.4 Hz, 3.6 Hz, etc.). This particular spectrum was recorded on the PO8 electrode (see also the power scalpmap at 1.2 Hz, same figure). One can also notice an important peak in the spectrum at the base frequency (6 Hz).

The frequency localization method was performed after selecting in the X matrix (1) the complex values of the FFT corresponding to the peaks at 1.2, 2.4 Hz and 3.6 Hz and their negative counterparts. We used the same head model as for the simulated signals (lead-field matrix A). The result, after tuning λ to the norm of X divided by 20, are shown fig 3. Three sources were reconstructed (GOF=0.88 with respect to the measured data X) in the posterior brain region and both hemispheres (results to be compared for example with the fMRI images from [START_REF] Kietzmann | The occipital face area is causally involved in facial viewpoint perception[END_REF].fig. 2). Of course, these results should be interpreted with caution. For example, the leadfield matrix that we used is designed for the simulation, but not optimized for the subject on which the EEG recordings were performed (nor actually optimized for sources in the gray matter with more carefully selected orientations).

IV. CONCLUSION

In this paper, we have proposed a sparse source estimation method in the context of oscillatory source activities, in response to periodic stimulation as observed e.g., in FPVS. The method can be applied on either an epoch-averaged band-pass filtered version of the signal centered on the frequency of interest or directly on the FFT of the signal, by considering the complex values associated with the peaks at these frequencies.

We have demonstrated on simulation the higher robustness of this former version, as the information of interest is maximally concentrated in these peaks. In particular, this approach is likely to provide more reliable reconstruction of deep brain sources from the EEG surface measurements. However, our simulation results also illustrate that the reconstruction of synchronized sources concentrated in a limited area of the brain is a highly challenging task for the proposed iterative method, whatever the chosen reconstruction domain. The method fails and converges to an equivalent solution with high GOF but wrong source estimates, which brings us back to the initial question of the regularization of this severely ill-posed inversion problem. In order to solve such indeterminacy, it appears to be necessary to introduce into the method additional neurophysiological or anatomical constraints on the position of the sources.

We finally illustrate our methodology on an example of real data recorded during an FPVS protocol. The results, which need to be further analysed and evaluated using a brain atlas and a wider panel of subjects, is nevertheless coherent with the present knowledge on brain face responsive areas and is a motivation to pursue our research in this direction.

Fig. 1 :

 1 Fig. 1: Simulation setup. In red, oddball sources (1.2Hz); in blue, base frequency sources (6Hz). The filled circles correspond to C1/C3, empty circles to supplementary sources in C2/C4 (see text). The 64 electrodes are represented as dots.

  1): C1 A single sinusoidal source at f o = 1.2Hz, placed in the right hemisphere, at the approximate location of one of the face responding regions on the posterior ventral visual stream. C2 Two sources at f o = 1.2Hz, the one used in C1 plus a second one, approximately corresponding to a face responding region in the right anterior ventral visual

Fig. 2 :

 2 Fig. 2: Source localization performances. Top row: number of true positives (correctly localized sources, within 1 cm from the simulated source). Bottom row: number of false positives (falsely-localized sources). The results are displayed for the four source configurations, for different levels of SNR. For every SNR, the first bar shows the performance in frequency domain and the second bar in time domain. The color represents the accuracy of reconstruction (GOF), see colorbar.

Fig. 3 :

 3 Fig. 3: Frequency source localization in real signals. Three sources were reconstructed (red dots) from a data vector with the peaks at the frequencies 1.2, 2.4 and 3.6 Hz. The scalpmap at 1.2Hz is given at the left, centered on the PO8 electrode, for which the spectrum is displayed nearby.

TABLE I :

 I Mean ± std wDLE values of the source localization in frequency and time domain for the four source configurations (in mm; 100 simulations per SNR)

		C1		C2
	SNR (dB)	Frequency	Time	Frequency	Time
	20	0 ± 0	0 ± 0	0.02 ± 0.01	0.08 ± 0.07
	10	0 ± 0	0 ± 0	0.13 ± 0.40	1.36 ± 2.57
	3	0 ± 0	0.75 ± 1.52	1.19 ± 2.22	4.25 ± 4.09
	0	0 ± 0	3.22 ± 3.90	1.81 ± 3.03	7.13 ± 4.16
	-3	0 ± 0	10.07 ± 8.12	3.02 ± 3.76	11.11 ± 4.67
	-10	1.21 ± 2.24	35.75 ± 11.71	7.07 ± 4.08	25.07 ± 7.68
	-20	34.93 ± 11.69	47.57 ± 8.29	24.11 ± 6.90	38.54 ± 4.75
		C3		C4
	SNR (dB)	Frequency	Time	Frequency	Time
	20	0.04 ± 0.02	0.05 ± 0.02	18.06 ± 0.01	18.06 ± 0.01
	10	0.14 ± 0.05	0.16 ± 0.07	18.06 ± 0.02	18.06 ± 0.04
	3	0.30 ± 0.12	0.35 ± 0.15	18.03 ± 0.24	17.92 ± 0.61
	0	0.42 ± 0.16	0.48 ± 0.21	17.96 ± 0.67	17.87 ± 1.02
	-3	0.58 ± 0.22	0.65 ± 0.28	17.98 ± 0.90	17.96 ± 1.04
	-10	2.62 ± 3.39	4.84 ± 5.72	18.17 ± 1.31	18.66 ± 1.44
	-20	25.79 ± 10.82	30.38 ± 12.19	19.82 ± 2.00	21.28 ± 5.24

ACKNOWLEDGMENT

The authors would like to thank Angélique Volfart for her cooperation in providing the real EEG data and for her comments. This work was partly supported by funding from the National Council of Sciences and Technology of Mexico (CONACyT).