Keywords: Parallel programming, multi-bsp, ml, Programming language, Parallel execution safety iii Résumé Programmation parallèle, multi-bsp, ml, Langage de programmation, Sûreté des langages v multi-bsp, ml, Langage

Functional abstraction for programming multi-level architectures: Formalisation and implementation

From personal computers using an increasing number of cores, to supercomputers having millions of computing units, parallel architectures are the current standard. The high performance architectures are usually referenced to as hierarchical, as they are composed from clusters of multi-processors of multi-cores. Programming such architectures is known to be notoriously difficult. Writing parallel programs is, most of the time, difficult for both the algorithmic and the implementation phase. To answer those concerns, many structured models and languages were proposed in order to increase both expressiveness and efficiency. Among other models, multi-bsp is a bridging model dedicated to hierarchical architecture that ensures efficiency, execution safety, scalability and cost prediction. It is an extension of the well known bsp model that handles flat architectures.

In this thesis we introduce the multi-ml language, which allows programming multi-bsp algorithms "à la ml" and thus, guarantees the properties of the multi-bsp model and the execution safety, thanks to a ml type system. To deal with the multi-level execution model of multi-ml, we defined formal semantics which describe the valid evaluation of an expression. To ensure the execution safety of multi-ml programs, we also propose a typing system that preserves replicated coherence. An abstract machine is defined to formally describe the evaluation of a multi-ml program on a multi-bsp architecture. An implementation of the language is available as a compilation toolchain. It is thus possible to generate an efficient parallel code from a program written in multi-ml and execute it on any hierarchical machine.

la programmation parallèle et est également adapté à la vérification de programmes. Il est possible de générer un programme parallèle certifié à partir d'une preuve de programme écrite via l'assistant de preuve coq. Cependant, bsml est défini pour programmer des architectures plates de type bsp. Il semble naturel de vouloir proposer une approche similaire à bsml permettant la programmation efficace d'architectures hiérarchiques, basée sur le modèle multi-bsp.

Cette thèse s'articule naturellement autour de cette idée et propose de définir multi-ml, un langage basé sur le modèle logico-matériel multi-bsp, garantissant efficacité, prédiction de performances, sécurité d'exécution et passage à l'échelle. Ce nouveau langage de programmation peut être vu comme une extension de bsml. En effet, multi-ml utilise des primitives empruntées à bsml ainsi que la notion de vecteur parallèle, tout en ajoutant une syntaxe particulière permettant d'exprimer une récursivité sur les différents niveaux des machines hiérarchiques. Chaque noeud de la machine est donc une unité qui peut effectuer des calculs locaux mais également distribuer données et calculs sur ses fils. L'approche multi-ml consiste à interpréter une machine multi-bsp comme un arbre où les calculs vont être propagés de la racine (composant le plus haut) vers les feuilles (composants les plus bas). Cette approche top-down permet une programmation récursive et élégante exprimant un parallélisme multi-niveau. Cette notion de récursion permettant de parcourir une machine hiérarchique est au coeur de multi-ml, et est implémentée sous forme d'une fonction récursive particulière : la multi-fonction. Une multi-fonction est composée de deux parties, l'une visant à manipuler un noeud et l'autre une feuille. Le code du noeud permet de manier et distribuer des données ainsi que de propager des calculs sur ses fils. Au contraire, le code des feuilles est exclusivement dédié au calcul intensif. Lors de l'évaluation d'une multi-fonction, la récursion est initiée sur la racine de l'arbre multi-bsp, en exécutant le code du noeud. La récursion est effectuée via l'appel récursif de la multi-fonction à l'intérieur d'un vecteur parallèle. En effet, un vecteur parallèle défini dans un noeud manipule directement la mémoire de ses fils. Ainsi, la récursion se propage sur toute l'architecture : de la racine vers les feuilles. Un noeud étant un niveau de l'architecture représentant une sous-machine bsp, il peut utiliser certaines primitives empruntées à bsml. La construction des vecteurs parallèles peut se faire par l'intermédiaire de la syntaxe << >> (section parallèle) et il est possible d'utiliser le sucre syntaxique $v$ afin d'accéder à la valeur du vecteur v à l'intérieur de la section parallèle. Afin de pouvoir faire référence aux valeurs définies dans la mémoire d'un noeud à partir d'une section parallèle, nous proposons un nouveau sucre syntaxique. En effet, une valeur d'un noeud doit être explicitement copiée dans la mémoire de ses fils afin d'y faire référence dans une section parallèle. Pour ce faire, annoter une variable x par #x# , permet d'exprimer une communication explicite de la variable x (contenue dans la mémoire d'un noeud) vers la mémoire des fils. En multi-ml, il est également possible de créer des vecteurs parallèles en utilisant la primitive mkpar . Cependant, son comportement est différent de sa version bsml. Ici, la fonction donnée en argument de mkpar ne va pas être exécutée sur chacun des fils d'un noeud, mais va l'être localement. La fonction est donc exécutée autant de fois que le noeud a de fils, et l'ensemble des résultats générés sont ensuite distribués, un à un aux fils, afin de créer un vecteur parallèle. L'intérêt de cette modification est de minimiser les communications lors de la création d'un vecteur parallèle. Par exemple, le découpage d'une liste via mkpar se fait localement et chaque sous-partie est communiquée. Au contraire, avec l'utilisation de la section parallèle, il aurait été nécessaire de répliquer la liste sur tous les fils pour finalement la découper localement. Des primitives de communication synchrones (impliquant une barrière de synchronisation) sont également disponibles. La primitive proj , qui est le dual de la création d'un vecteur, est utilisée pour rendre les éléments d'un vecteur parallèle disponible, sous forme fonctionnelle, au niveau du noeud appelant. put permet d'effectuer des communications entre tous les processeurs disponibles à un noeud de l'architecture. L'utilisation des primitives de communication synchrone proj et put est similaire à leur version bsml.

Afin de pouvoir exploiter les différents niveaux de mémoire de l'architecture, nous proposons une nouvelle structure de données parallèle, nommée tree (arbre), représentant la distribution des données à travers tous les niveaux. Un ensemble restreint de primitives permet de créer et de manipuler de telles structures. Par exemple, l'opérateur at permet, à un composant de l'architecture, d'accéder à l'élément d'un arbre le concernant.

Contrairement au modèle multi-bsp, il est possible d'exécuter des calculs sur les noeuds d'une architecture. Cette petite dissemblance s'explique dans le fait qu'il paraît plus naturel d'avoir la possibilité de distribuer des données directement dans le code des noeuds concernés. Cependant, une implémentation de multi-ml qui s'apparente effectivement au modèle multibsp est tout à fait possible. Il suffit de simuler l'exécution des calculs des noeuds sur les feuilles de l'architecture.

Le mécanisme d'exécution d'un programme parallèle, et à fortiori d'un programme multiml, est particulier. En effet, l'exécution en parallèle de nombreux processus est généralement difficile à appréhender. Afin de clarifier les comportements acceptables d'un programme multiml, nous formalisons une sémantique opérationnelle (grand pas). Celle-ci permet de décrire en détail les comportements d'une expression (ou programme) exprimée dans la syntaxe multi-ml. Par exemple, nous décrivons le mécanisme de distribution des calculs et des données d'un noeud vers ses fils. Par l'intermédiaire d'une sémantique grand pas co-inductive, il nous est également possible de décrire les évaluations qui divergent, en plus des évaluations qui terminent.

De plus, comme énoncé précédemment, le modèle multi-bsp propose un modèle de coût. Nous définissons donc une sémantique annotée décrivant les différents coût des différentes constructions du langage. Ainsi, nous décrivons le comportement des composants engagés dans l'évaluation d'une expression, tout en quantifiant les calculs et les communications nécessaires. Il est ainsi possible de raisonner sur le coût algorithmique d'un programme écrit en multi-ml via une sémantique formelle.

Comme en ml, nous proposons un système de type permettant de nous assurer que les programmes soient bien formés. Bien que proche du système de type d'ocaml, celui de multiml doit prendre en charge un degré de complexité supplémentaire lié au parallélisme. En effet, la possibilité d'évaluer des expressions sur différents niveaux implique une gestion fine de ces niveaux à l'intérieur du système de type. Pour ce faire, nous avons choisi de développer un système de type avec effet, annotation de types et contraintes. Ces annotations reflètent les capacités d'un niveau de l'architecture. Nous avons : m pour une évaluation concernant tous les composants de la machine ; b pour un noeud, c'est-à-dire une machine pouvant exécuter du code bsml ; c pour les expressions communicables dans un vecteur parallèle et l pour ces mêmes expressions lorsqu'elles sont non communicables ; finalement, s concerne les exécutions séquentielles des feuilles. Une expression dite communicable est une expression qui a un sens dans la mémoire d'une autre unité. Typiquement, les pointeurs (ou références) sur les données d'un noeud ne sont pas exploitables à un autre niveau de l'architecture, ils sont donc annotés l (pour local). Un type est donc annoté par une localité représentant l'endroit où l'expression a été évaluée et donc, où une donnée réside. Par exemple, le type int_m dénote un entier qui est défini au niveau m. De plus, la notion d'effet est intégré à notre système de type via, notamment, l'annotation des types flèches. La notation -(b)-> représente un effet latent, c'est-à-dire, un effet qui sera émis lors de l'application d'une fonction. Dans ce cas, l'exemple représente une fonction effectuant des calculs parallèles devant être effectués au niveau b. Ainsi, l'application d'une telle valeur à un autre niveau doit être prohibée.

Le système de type conçu dans cette thèse utilise ces localités afin de déterminer si une expression est valide à un niveau en particulier de l'architecture mais également afin de vérifier si une expression peut être communiquée vers un autre niveau. Il est également possible de prévenir l'imbrication inappropriée des structures parallèles comme l'imbrication de vecteurs parallèles (comme en bsml) ou l'imbrication d'arbres. De plus, le système de type garantit la cohérence répliquée afin d'éviter les inter-blocages ainsi que les comportements non déterministes. Par exemple, l'exécution d'un code menant à l'exécution d'une barrière de synchronisation sur une sous-partie des processeurs d'un noeud n'est pas acceptée. Le système de type de multi-ml garantit donc la sécurité d'exécution de tels programmes sur des architectures hiérarchiques. Afin de décrire formellement le modèle d'exécution d'un programme multi-bsp, nous proposons une machine abstraite. Celle-ci décrit en détail les suites d'instructions nécessaires à l'exécution d'un programme multi-ml et permet de lier le comportement d'un langage de haut niveau avec celui de la machine physique, dit bas niveau. Ce langage intermédiaire est utilisé pour dérouler, étape par étape, l'exécution d'un programme sans se soucier des détails d'implémentation liés aux composants physiques (hardware). L'utilisation d'un schéma de compilation générique pour une machine abstraite parallèle permet également de garantir une certaine confiance par rapport à l'implémentation d'un compilateur. Dans un premier temps, nous proposons une machine abstraite permettant une implantation rapide et relativement efficace. Afin de simuler une exécution hiérarchique, un processus démon est disponible pour chaque noeud et feuille de l'architecture. Ces processus vont attendre des ordres consistant à lancer l'exécution de différents codes. Les communications entre processus, bien que hiérarchiques, sont effectuées en utilisant des routines de communication asynchrones point à point (de type send/receive). Les ordres d'exécutions consistent, principalement, en une fermeture (contenant une fonction à exécuter) et en un ensemble de valeurs (nécessaires pour l'évaluation de la fermeture). Un ensemble de briques élémentaires est ainsi utilisé pour décrire l'ensemble des comportements et actions nécessaires pour l'exécution d'un programme multi-ml.

Grâce au modèle de coût proposé par le modèle multi-bsp, il est possible de définir formellement un coût d'exécution pour un programme multi-ml. Pour ce faire, nous proposons une sémantique naturelle (ou grand pas) qui intègre un coût pour l'évaluation de chaque expression. Ainsi, il est possible de comptabiliser l'ensemble des coûts engendrés par l'utilisation des primitives asynchrones lors des phases de synchronisation. L'utilisation d'une algèbre de coût, utilisant les paramètres multi-bsp liés à l'architecture permet de proposer une prédiction de coût réaliste, pour un algorithme multi-ml donné.

Actuellement, les sources du langage multi-ml (disponible sous licence open source) représentent approximativement 8.000 lignes de codes, réparties dans environ 100 fichiers (la vaste majorité étant écrite en ocaml). La distribution est structurée en différentes parties afin de proposer un langage facilement extensible. Nous pouvons notamment distinguer les différents étages de compilations, le typage, les modules de communications ainsi que le support d'exécution séquentiel (et toplevel) et parallèle. Afin de pouvoir expérimenter la programmation d'algorithmes en multi-ml, nous proposons une implémentation du langage qui peut être vue comme une extension du langage de programmation fonctionnelle ocaml. Une chaîne de compilation est disponible afin de pouvoir transformer un code multi-ml en exécutable. La chaîne de compilation est structurée en trois phases principales : [START_REF] Alaniz | MBSPDiscover: An Automatic Benchmark for MultiBSP Performance Analysis[END_REF] Le frontend (ou "partie dépendante du langage") permet de générer un arbre de syntaxe abstrait (ast) à partir d'un programme écrit en multi-ml. Pour ce faire, cette première phase doit effectuer une analyse lexicale et syntaxique, afin de vérifier que la norme du langage est respectée. De plus, le front end comprend la phase de typage, utilisée pour vérifier que le programme est bien formé et ainsi, garantir sa sécurité d'exécution. Cette phase permet également de récupérer de nombreuses informations comme les types, les erreurs ou les avertissements. [START_REF] Alexandrov | LogGP: Incorporating Long Messages into the LogP Model-One Step Closer Towards a Realistic Model for Parallel Computation[END_REF] Le middle-end (ou "partie commune") permet de manipuler l'ast produit par le frontend afin d'y effectuer des transformations et optimisations.

Il produit un code intermédiaire générique. (3) Le backend utilise le résultat du middle-end afin de terminer la phase de compilation et de générer un code exécutable. Le produit du backend est donc un exécutable qu'il est directement possible d'utiliser sur une architecture concrète. Pour la mise au point de multi-ml, nous avons choisi de reprendre les sources d'ocaml et de modifier directement l'analyse syntaxique (lexer) et l'analyse lexicale (parser). Ainsi, nous avons pu retirer les constructions qui ne sont pas encore disponibles en multi-ml (structures, modules, etc.), et qui seront intégrées au fur et à mesure de l'évolution du langage. Ensuite, nous avons ajouté les constructions et sucres syntaxiques propres à multi-ml (primitives de communication, multi-fonctions, etc.). Ces modifications nous permettent de reconnaître un ensemble d'expression limité, afin de générer un arbre de syntaxe abstrait (ast) valide. La validité de cet arbre est vérifiée par le système de type. Une fois que l'ast validé est généré, le middle-end se charge d'effectuer des transformations permettant d'obtenir un code intermédiaire adapté à l'architecture ciblée. Actuellement, nous sommes en mesure de générer un ast adapté à une machine séquentielle ainsi qu'une version dédiée aux architecture hiérarchiques (en utilisant des primitives mpi). C'est à partir de ce code intermédiaire que le backend se charge de générer une exécutable, en utilisant le compilateur ocaml. En effet, l'ast produit par le middle-end est structuré à la manière d'un programme ocaml. La phase de compilation d'un programme multi-ml consiste en la transformation d'un code multi-ml en un code ocaml contenant une ensemble de primitives de communication. Le code généré est adapté et optimisé à l'architecture ciblée. L'exécutable final est obtenu via l'utilisation du compilateur ocaml optimisé. 1.11 The difference between the multi-bsp and bsp models for a multi-core architecture. 1.12 A cluster of multi-core modeled with multi-bsp . . . . . . . . . . . . . . . . . . The main topic of this thesis is the design and the implementation of a high-level parallel language designed for programming hierarchical architectures in a functional way.

List of Figures

In this introduction, we provide a general background on high-level parallel programming (Section 1.1), and present different hpc architectures (and thus, forms of parallelism) in Section 1.2. We will then introduce some technologies that are the foundations of our work in Section 1.3: the Bulk Synchronous Parallelism model (bsp) and the bsml language which is a bsp extension of the high-level, general purpose and mainly functional ml language. We finally show, in Section 1.4, their inherent limitations and introduce our solution, which is the core and the purpose of this thesis.

Context of work: generalities and background

Nowadays, it is accepted that parallel and distributed programming is the norm in many areas [START_REF] Hendrickson | Computational Science: Emerging Opportunities and Challenges[END_REF] such as meteorology, molecular biology, geology, etc. Personal computers are using an increasing number of cores and even some smartphones now include octo-core processors and beyond. On another scale, supercomputers reach new records of performance using a growing number of hierarchical architectures [6] such as gpus, multi processors of multi-cores, clusters of multi-cores, etc.

In this context of "Think Parallel or Perish", easy and safe programming of all parallel architectures should be of paramount importance. Moreover, the additional costs of erroneous computation induced by deadlocks or unexpected messages on large-scale simulations can be very important. Today, popular parallel and distributed programming methodologies are sadly dominated by low-level techniques. High-level approaches offer many possible advantages by abstracting implantation details. Such structured approaches have a key role in scalable, portable and ubiquitous parallelism.

Unfortunately, it is a hard task to conceive such structured approaches and there is no universal solution. In fact, the design of a parallel language is a trade-off between two key features. First, the possibility, for the programmer, to finely control the parallel aspects that are necessary for efficiency (but which makes programs harder to write, to prove correct and to make portable). Then, the abstraction of such features which are necessary to make parallel programming easier (but which hampers efficiency and performance prediction).

Background

In [START_REF] Gorlatch | Send-Receive Considered Harmful: Myths and Realities of Message Passing[END_REF], Gorlach demonstrates that programming distributed and parallel architectures1 is a hard task and that only using "send" and "receive" primitives (to transmit data between processes) is not a good way of programming. He compares this practice to the "Goto statement" [START_REF] Edsger | Letters to the Editor: Go to Statement Considered Harmful[END_REF] programming and "spaghetti code" developing. Most parallel programming tasks are still performed using low-level tools and languages, leading to poor quality code, high debugging and upkeep costs [START_REF] Gorlatch | Send-Receive Considered Harmful: Myths and Realities of Message Passing[END_REF]. Architectures are also becoming more and more complicated by aggregating thousands of processors and cores with different levels of memory and networks. Thus, a perfect and universal programming language to rule them all seems inconceivable. We recall why parallel programming is such a hard task:

• Additional complexity It is not easy to move from a sequential language to a parallel one: the way of thinking parallel algorithms is completely different. Transform a sequential program into a parallel one incrementally is not always possible. Moreover, tools for automatic parallelisation mainly work on trivial loops and basic data structures. There is no magic method to decompose a task and distribute data. If the wrong decomposition is made, one might not see any performance improvement on hpc applications. The only thing that really helps is the experience and/or the cost analysis, which is also an arduous task with complex algorithms.

• Parallel programming is error-prone even for "Gurus". Some subtle errors (e.g. deadlocks, data-races, etc.) can be introduced and may arise after a large number execution of a given program. Furthermore, compilers are not tested as well as their sequential counterparts, mainly because of the lack of standards and portability of programs. Due to an endless quest for efficiency, the implementation of mpi [[ 1 ]] does not always meet the specifications -even if these specifications are not always very accurate. Indeed, in the mpi standard, there are many "unspecified cases" (at least dependent on the manufacturer); This lack of "understanding" makes the "rigorous" requirement difficult to satisfy. In fact, many libraries and programming interfaces (e.g. mpi, openmp, etc.) allows to test your parallel programs on your personal computer by emulating parallelism using the operating system's processes. However, not all interleaving can be tested that way and some combinations that generate deadlocks may be missed. Regarding hpc, testing and debugging such errors (detailed in Section 1.2.2.1) are usually a challenge.

• Powerful abstractions are scarce in the languages used for parallel programming, as described in [START_REF] Lee | The Problem with Threads[END_REF]. They are often too low-level and each communication or synchronisation operations has to be finely managed by the programmer. Programming becomes way easier, when the programmer can rely on powerful libraries that encapsulate complex behaviours. Works based on high-level patterns date back to several decades but there is still not much knowledge of innovative parallel programming languages and tools for verifying such codes.

Parallel programming is thus a strenuous task. When every detail of parallelism is left to the programmer, the program complexity becomes excessive and large sections of code deal with purely technical issues.

Generalities concerning solutions proposed in this thesis

A solution proposed in this document is the use of a functional programming language with structured parallelism. Functional languages [START_REF]Research Directions in Parallel Functional Programming[END_REF] have the advantages to be high-order, safe and code composing friendly. This is also the case with structured parallelism which composes patterns of communications and computations.

We are also exploring thoroughly the intermediate position of the paradigm of structured parallelism and algorithmic skeletons [15,[START_REF] Cole | Algorithmic Skeletons: Structured Management of Parallel Computation[END_REF][START_REF] Darlington | Parallel Programming Using Skeleton Functions[END_REF][START_REF] Skillicorn | Optimising Data-Parallel Programs Using the BSP Cost Model[END_REF] in order to obtain universal parallel languages where execution cost can easily be determined from the source code (in this context, cost means the estimated parallel execution time). This last requirement forces the use of explicit processes corresponding to the processors of the parallel machine.

Using structured parallelism also eases to describe, more formally, the behaviours of the programs. For example, by designing operational semantics and abstract machines.

Another important point is safety [START_REF] Hoare | The Verified Software Initiative: A Manifesto[END_REF]. Parallel libraries and languages are generally well documented for the common cases. Nevertheless, even experienced programmers can misunderstand those apis, especially when they are informally described and concern complex operations leading to multiple sources of potential errors. By designing a type system and by limiting (not too much) the the way of programming, one can prove that programs can be safely executed and do not crash the whole machine (or stop it in a inconsistent state). Types can also help to understand codes. Because parallel code is the norm in many areas, having parallel programs that can be executed without any error is necessary. Indeed safety seems essential when considering the growing number of parallel architectures (gpus, multi-cores, etc. [6]), the complexity of such architectures and the cost of conducting large-scale simulations (the losses due to faulty programs, unreliable results, unexpected crashing simulations, etc.). This is especially true when parallel programs are executed on architectures which are expensive and consume many resources, and also on critical systems.

In this thesis, we keep in mind all those considerations to design a parallel extension of a ml (functional) language. This extension is based on a structured model of parallelism that also allows to estimate the execution time of programs. Given the strong heterogeneity of massively parallel architectures and their complexity, a frontal attack of the problem is a daunting task which is unlikely to materialise. Therefore, it seems more appropriate to design languages that limits the way of using such architectures but also preserve the performances. In fact, designing a programming language requires to choose a trade-off between the possibility to control parallel aspects -necessary for predictable efficiency but which makes programs more difficult to write, to prove and to port -and the abstraction of such features -which makes programming easier but which may hamper efficiency and performance prediction. The language designed in this thesis aims at have both efficient and safe program execution for hierarchical architectures. The chosen solution uses a bridging model that abstract efficiently many hpc architectures. Using a functional language (ml) allows us to have the required programming safety.

In the rest of this introduction, we will explain what are hpc architectures, what is a bridging model (by presenting one of them: bsp) and how programming algorithms with such a model (using a extension of ml call bsml). We will also show some limitations and how to overcome them: which is the main goal of this thesis.

Parallel and distributed architectures

In this section, we first recall some concepts of parallel computing in order to define a common vocabulary that will be used along this thesis. Indeed, in the parallel and hpc community, several terms are not used in a proper way and can lead to misunderstandings. We will also recall the CHAPTER 1. INTRODUCTION traditional problems of parallel and distributed programming. These difficulties will allow us to reveal the philosophy of the solutions that are proposed in this thesis.

The different architectures

A parallel computer, or a multi-processor system, is a computer composed by more than one processor (or unit of computation). It is common to classify parallel computers by distinguishing them by the way they access to the system memory. Indeed, the memory access scheme influences heavily the programming method of a given system.

Flynn's taxonomy

Flynn [START_REF] Michael | Some Computer Organizations and Their Effectiveness[END_REF] defines a taxonomy of computer architectures that are based upon the number of concurrent instructions (or controls) and data streams available in the architecture [START_REF] Duncan | A Survey of Parallel Computer Architectures[END_REF] where:

• SISD is "Single Instruction, Single Data stream" that is a sequential machine;

• SIMD is "Single Instruction, Multiple Data streams" that is mostly array processors;

• MISD is "Multiple Instruction, Single Data stream" that is pipeline of data;

• MIMD is "Multiple Instruction, Multiple Data streams" that is clusters of cpus.

Two major classes of parallel computers can be distinguished: distributed memory and shared memory systems. The shared memory model allows multiple processes to read and write data from the same location. Otherwise, in distributed memories, the message passing model, where each process sends messages to the other processes, is used. Those different approaches are respectively adapted to various domains of application. For example, distributed memory systems are needed for computations using a large amount of data which does not fit in the memory of a single machine.

Shared memory model

In the shared memory model, a program often starts as a single process (known as a master thread) which is executed on a single processor. We consider a program where some code regions are defined as parallel regions. When the master thread reaches a parallel region, a fork operation is executed. It creates a group of threads which execute the parallel region on multiple processors. At the end of this parallel computation phase, a join operation terminates the fork, leaving the master thread as a lonely unit that continues to execute the program.

Most of the time, when programmers talk about parallel architecture or concurrent programming languages, they refer to the shared memory models. Some multi-core systems or specific supercomputers (such as Cray ones) are part of this category. In the following, we introduce some well known examples of libraries for the shared memory programming model. openmp [START_REF] Chapman | Using OpenMP: Portable Shared Memory Parallel Programming[END_REF][START_REF] Wang | OpenMP Compiler for Distributed Memory Architectures[END_REF] is a directive-based programming interface for the shared memory programming model. It consists of a set of directives and runtime routines for fortran, and a corresponding set of pragmas for c and c++ (since 1998). openmp directives take the form of special code annotations that are interpreted by the compiler. The advantage of such directives is that the code is still sequential and can be executed on sequential machines, by ignoring the directives (pragmas). The main benefit of this approach is to avoid to maintain both a sequential and a parallel version.

intel Threading Building Blocks (Intel tbb [[ 2 ]] ) [START_REF] Kukanov | The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks[END_REF] is a c++ library for task parallelism. It allows to write programs that take full advantage of multi-core performance, that are portable, composable and scalable. It also proposes features such as parallel algorithms, parallel data structures, task scheduling and scalable memory allocation. Different applications are developed using the library and are evaluated in terms of their usability and expressibility, as shown in [START_REF] Joseph | Evaluating Threading Building Blocks Pipelines[END_REF].

Graphic Processing Units (gpu) are used in hpc for a short while. It is mainly due to their tremendous computing power, favourable cost effectiveness, and energy efficiency. The Compute Unified Device Architecture (cuda) [[ 3 ]] [START_REF] Nickolls | Scalable Parallel Programming with CUDA[END_REF] and Open Computing Language (opencl) have enabled gpus to be explicitly programmed as general-purpose shared-memory multi-core processors with a high-level of parallelism. The General-Purpose computing on Graphics Processing Units (gpgpu) was born. In a short period, gpus have moved from graphical-centred computing units to programmable and parallel computing devices. In cuda, a gpu can be exploited by the programmer as a set of general-purpose shared-memory simd multi-core processors.

Distributed memory model and the Message Passing Interface (MPI)

Distributed memory No Remote Memory Access (norma) computers (or shared nothing) do not have any special hardware to support a direct access to another machine's local memory. The nodes are only connected through a computer network (which can be specific to a supercomputer and therefore be very efficient). Processors can obtain data from a remote memory by exchanging messages, through the network, between the requesting and the supplying node. Such computers are sometimes also called Network Of Workstations (now). When shared memory systems are used, we talk about Remote Memory Access (rma) computers. They grant access to remote memory via specific operations that may be implemented by hardware. However the hardware does not provide a global addressing space.

The major advantage of distributed memory systems is their ability to scale to a very large number of nodes. A shared memory architecture provides a global addressing space defined by the hardware, i.e. all memory locations can be accessed via load and store operations. Such a system may appear to be much easier to program. However, it is hard to handle a high number of processes, as the concurrent read and write makes the program difficult to design. Furthermore, there may be unexpected performance loss due to Non Uniform Memory Access (numa) shared memory models.

The mimd streams model, and especially the so-called Single Program Multiple Data (spmd) model, is widely used for programming parallel computers. In the spmd model, the same program is executed on each processor but the computations are done on different pieces of data that are distributed over the processors.

The message passing model is based on a set of processes with private data structures. Processes communicate by exchanging messages with special send and receive operations. It is widely used for programming distributed memory machines but it can be also used on shared memory computers. The most popular message passing technology is the Message Passing Interface (mpi) [START_REF] Snir | MPI-The Complete Reference. 2nd[END_REF]. It is a c and fortran message passing library. mpi is an industry standard and is implemented on a wide range of parallel computers. Details of the underlying network protocols and infrastructure are hidden from the programmer. This helps to achieve portability while enabling programmers to focus on writing parallel code rather than networking code. It includes routines for point-to-point communication, collective communication, one-sided communication, parallel io and dynamic task creation.

Message passing and distributed memory models also subsume the well-known master-slave paradigm. In a sense, mobile agents (and nomadic computations) can also be referenced as this model. Concurrent programming such as the famous π-calculus [START_REF] Milner | Communicating and Mobile Systems: The Pi-Calculus[END_REF] (or its java version [START_REF] Peschanski | Parallel Computing with the Pi-Calculus[END_REF]) can also be referenced as message passing models. Some more general models, such as in [START_REF] Cardelli | Mobile Ambients[END_REF], were defined in order to unify those models. Nevertheless, they appear to be unsuitable to hpc and do not provide a structured enough cost model.

Hybrid architectures

In addition to shared-memory and distributed models, modern parallel architectures now provide hybrid models. Many architectures are composed by clusters of clusters of … of multiprocessors of multi-cores with gpus. Such architectures are referenced as hybrid due to the diversity of its components. Furthermore, as the computations taking place on these machines are organised as a tree, they are also mentioned as hierarchical architectures. For example, we give the top five 4 of the 500 most powerful supercomputers in the world:

1. (China) Sunway TaihuLight(NRCPC): 10.649.600 cores @1.45Ghz with 1.31PiB of memory 2. (China) Tianhe-2(NUDT): 3.120.000 cores @2.2Ghz with 1TiB of memory 3. (USA) Titan(Cray): 299.008 cores @2.2Ghz with 584TiB of memory and 18.688 gpus with 109TB of memory 4. (USA) Sequoia(IBM): 1.572.864 cores @2.3Ghz with 1.5PiB of memory 5. (USA) Cori(Cray): 64.128 cores @2.3Ghz with 203TiB of memory and 632.672 Xeon Phi @1.4Ghz with 1PiB of memory

To illustrate this, we choose to describe the Sequoia supercomputer of the Lawrence Livermore National Laboratory in Livermore, California, usa. This is a Bluegene version q model of supercomputer which is offered by ibm. The Figure 1.12 shows, in detail, the hierarchy of all the components that are used to build such a machine. A 16 core chip is associated with a memory in a compute card and put together in a node card containing 32 compute cards; 16 node cards are assembled inside a midplane; a rack contains 2 midplanes. Finally, the whole system is composed by 96 racks. Thus, we have 16 × 32 × 16 × 2 × 96 = 1.572.864 computing units that are hierarchically organised.

Clusters of multi-processors of multi-cores have become the de-facto standard in parallel processing due to their high performance to price ratio. Symmetric Multi-Processing (smp) clusters are also gaining popularity, mainly because of the fast interconnection networks and memory buses. smp clusters can be seen as a hierarchical two-level parallel architecture, since they combine the features of both shared and distributed memory machines. Aggregating smp clusters (clusters of multi-cores) can combine the efficiency of shared-memory multiprocessors with the scaling advantages of distributed-memory architectures. The result of such a combination leads to very large computer architectures with a good cost/efficiency ratio. Nevertheless, programming a machine combining both shared and distributed memories induces a more complex programming model. As a consequence, hybrid architectures are interesting in terms of performance but they require a new way of programming.

Intuitively, a paradigm using memory access for intra-node communication and message passing for inter-node communication seems to fully exploit smp clusters [START_REF] Drosinos | Performance Comparison of Pure MPI vs Hybrid MPI-OpenMP Parallelization Models on SMP Clusters[END_REF]. Hybrid models have already been used for scientific computations [START_REF] Henty | Performance of Hybrid Message-Passing and Shared-Memory Parallelism for Discrete Element Modeling[END_REF], including more symbolic ones such as probabilistic model-checking [START_REF] Hamidouche | Three High Performance Architectures in the Parallel APMC Boat[END_REF].

These kinds of architectures are those that we are targeting in this thesis. However, as gpus require particular programming instructions [16,[START_REF] Ouwehand | GPU Programming in Functional Languages A Comparison of Haskell GPU Embedded Domain Specific Languages[END_REF], they are excluded from the core of the thesis.

Critical point of parallel programming

We recall that writing parallel/distributed/hybrid programs (for hpc architectures) is known to be notoriously difficult. Most of the time, programmers need to reason in terms of messagepassing algorithms or shared data structures, which could be hard to deal with.

One can say that models based on shared memory (generally, concurrent languages) are easier to program. As such models provide a single address space that is shared, it seems easier to access to any pieces of data. However, it is mandatory to control simultaneous access to resources. To do so, it requires to write programs carefully and manage expensive system locks. Most of the time, implementing shared-memory abstractions requires to devote numerous resources of a machine to communication and coherence maintenance task. As shown in [START_REF] Lee | The Problem with Threads[END_REF] such an abstraction is not easy to handle and introduces costs.

CHAPTER 1. INTRODUCTION

Concerning the distributed architectures, the control of message-passing algorithms could also be a hard task. For example, managing communication of different processes executing different codes doing different tasks could be hazardous.

Hence, both shared memory and message passing models introduces complexity and bugs. In the following, we recall the most common of them.

Common bugs

Here, we give a non exhaustive list of the most common bugs that parallel programmers are facing.

Data race condition

When evaluating a sequential program, the execution order of the instructions is quite obvious (assuming no compilation or execution optimisation). On the contrary, the evaluation of a parallel program where two processors share data is not that simple. For example, one can imagine that one processor might progress faster than the other one. We illustrate with the following example:

Process 0 Process 1 x=0 x=0 x=x+1 x=x+1 y=read(x ,1) y=read(x,0) print(y) print(y)
Here, read(x,i) returns (or read) "immediately" the value of x at processor i . In this example, every processor executes the same sequence of instructions. This is consistent using the spmd model of parallelism. Indeed, if the processors execute always the same instruction simultaneously, they would both store in x the result of the computation, get the result from the other processor, and, then, display y . However, in parallel programming, it is (almost) impossible to ensure that all processors are executing the same statement at the same time.

In this example, if the processor 0 finishes its computations faster than the processor 1, he might execute the instruction y=read(x,1) while the processor 1 is still computing. The value of x is still 0 . Then, the output will be 0 for processor 0 and 1 for processor 1.

In this example the problem was fairly obvious. However, in a complex code, this kind of bug can be very difficult to spot. The behaviour of such programs is often non-deterministic, which make the bug even harder to track and correct. Concurrent analysers can be used to try to identify obvious bugs.

A similar behaviour can occur when using the famous mpi library. As the timing of a message arrival can impact the execution logic of a program, it is very common to introduce nondeterministic behaviour using message-passing libraries. A typical form of non-determinism occurs in a code where multiple concurrent incoming messages are handled in an unspecified arrival order ( using MPI_Waitany for example). The following code illustrates this non-deterministic behaviour: Here, both process 0 and 1 are going to send a value (0 and 42) to process 2. The values are received and used in order to perform a division. As the values are received without any explicit reception order (which is possible in mpi) the process 2 has two different behaviours: performing a valid computation or raising a division by 0 exception. This kind of non-determinism is difficult to spot and requires meticulous code writing to avoid such behaviour.

Deadlock

A deadlock is a state where at least two processes are waiting for each others. This problem mainly appears when adding synchronisation primitives in a language to avoid data races. Using a shared data structure, the classical deadlock example is the following: Where Lock(x) forces all other processes to wait until the variable x (a mutex) is unlocked using UnLock(x) . In this example, process 0 locks x whereas process 1 locks y . Later on, process 0 is blocked when accessing to y and, reciprocally, process 1 is waiting for process 0 to free x .

Concerning message passing, this situation may also appear in a similar way. For example: Process 0 Process 1

SendBlock(1,mess0) SendBlock(0,mess1) x=recv(1) x=recv(0) ... ...
Where SendBlock(i,mess) is a synchronous send of a message mess to process i , where the process waits until it receives the notification of complete reception. In this case, both process 0 and 1 are waiting for each other to receive the two messages.

Another common situation is when using a communication pattern involving all the processes: a collective operation. For example:

if pid<>0 then broadcast(0,buffer) else asynchronous_computations
Where broadcast(i,buffer) makes the process i broadcast its values to all other processes, into their respective buffers. In this case, the processors with a non-zero identifier are waiting for a value broadcasted by processor 0. As processor 0 is executing some asynchronous_ computations , we are facing a deadlock concerning the used partition of the parallel machine.

Many other examples of erroneous parallel computation exist, such as: bad size/ number of messages, sending data to a nonexistent processor and wrong memory accesses relying on pointers. Using a programming language that strongly limits these incorrect uses of parallelism would increase the confidence concerning parallel codes.

Coveted properties

Here, we list the most important properties that makes parallel programmer's life easier.

Determinism.

The main thing that a programmer is expecting when executing a program is to always get the same output for a given input -excepted for randomised algorithms. Determinism is a property assuring that: what ever the execution order of the processes is, the final value will be the same. This property is close to confluence of sequential programs. This feature is uncommon in the world of parallel languages [START_REF] Blelloch | Internally Deterministic Parallel Algorithms Can Be Fast[END_REF]; nevertheless it is a worthwhile property [START_REF] Bocchino | Parallel Programming Must Be Deterministic by Default[END_REF].

For many computational problems, there is no inherent non-determinism in the problem statement. As said in [START_REF] Cappello | On Communication Determinism in Parallel HPC Applications[END_REF] about hpc applications: it is typically the case that the outcome of the computation will not depend on the arrival order: The execution of the process will consist of multiple sections, where there is non-determinism within each section, but the state at the end of each section is deterministic. Furthermore, non-determinism "does not propagate": the order and content of sent messages is not affected by receive order. In [START_REF] Cappello | On Communication Determinism in Parallel HPC Applications[END_REF], this property was called: "send-determinism". Otherwise, the notion of "internally deterministic", studied in [START_REF] Blelloch | Internally Deterministic Parallel Algorithms Can Be Fast[END_REF], concerns evaluations that generates a unique trace for each input (which depends on the level of abstraction of trace's equivalence).

Determinism is convenient for safety: no deadlock nor data-race can occur; a deterministic program is free of the problem of execution interleavings (and complex memory consistency models). Furthermore, the composition of separately developed deterministic codes is easier as their behaviour is similar, regardless of the execution context. Determinism also allows the programmer to reason on an algorithms (the behaviour is completely defined by the sequential equivalent) and their optimisations without dealing with the previously defined safety problems. The programmer can reason "as" for sequential computation and add, incrementally, parallelisation strategies instead of sequential computations. Generally speaking, this method is simple but does not always offer the best performance.

Therefore, it seems to be a growing consensus on the fact that determinism is worthwhile [START_REF] Bocchino | Parallel Programming Must Be Deterministic by Default[END_REF].

Reasoning about execution time Another pleasant property is the ability of prediction concerning the execution time of hpc programs. For sequential algorithms, the common solution uses the ram/VonNeumann model and describes the complexity of a problem according to the input size n and relate it with a O function (worst case) or θ function (average case), etc. Based on these complexities analysis, some tools are able to accurately predict the execution time of programs by estimating their Worst Case Execution Time. For example, merasa was used to analyse parallel computation on multi-cores [START_REF] Rochange | WCET Analysis of a Parallel 3D Multigrid Solver Executed on the MERASA Multi-Core[END_REF][START_REF] Ungerer | Parallelizing Industrial Hard Real-Time Applications for the parMERASA Multicore[END_REF].

Concerning parallel and distributed algorithms, such a property is obviously valuable. Curiously, in hpc, the cost measurement is not based on the parallel complexity of an algorithm but is rather based on the execution time (measured using empirical benchmarks). Programmers are benchmarking load balancing, communication (size of data), etc. Using such a technique, it is very difficult to explain why a code is faster than another one and which one is more suitable for one or another architecture.

Even if the pram cost analysis (which is detailed in Chapter 2) proposes a method to estimate the work of a parallel algorithm, such a technique is not widespread. One explanation of such a lack can be found in [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF]: it seems that too many programmers still prefer using asynchronous send/received primitives (with potential data-races, deadlocks, etc.) instead of a structured model. Thus, measuring the overlap of computations and communications is hard to determine. Let's have a look to the following situation: Here, both processes 1 and 2 are going to send values to process 0. Each time process 0 receives a value, it uses it to make some computations. We assume synchronous sending (a process is blocked until the value has been received), constant size of data and constant computations.

One can ask how is it possible to determine the cost of this code. We assume that we just measure the efficiency of the communications and the latency of the network. As there are O(n) local steps, process 0 will receive n pieces of data and other processes will send n/2 pieces of data. If we assume that the overlap is possible, it is possible to both send and receive values while computing. If process 0 computes while receiving both the values of process 1 and 2, one value is directly available when the computation of 0 is done, for example 1. Then, both process 0 and 1 can compute; the value of process 2 is thus available for the next computation of 0 while 1 is sending a new one. The problem here is that we cannot distinguish the network efficiency of two distinct cases: receiving one value and receiving two values. Furthermore, the latency is hard to determine. Indeed, we should distinguish the latency of a value that has just been received and the latency of a value already received.

Therefore, it seems hard to reason about costs of message-passing algorithms. Reasoning, on a combination of high-level patterns to deduce the cost of a parallel algorithm seems wiser. It also allows to give some measures that takes into account worst cases (at least average) execution of programs on a given architecture.

A solution: a bridging model for abstracting HPC architectures

As we have just seen, one inherent difficulty in the development of scientific software is to have codes that are correct, easy to develop and efficient. Today, the number of parallel languages may exceed the number of different architectures available. Most languages are inadequate to such purposes because they are not made for portability, correctness and performance. Furthermore, the performances of such programs are typically difficult to predict because of the interaction of a large number of individual data transfers. On another side, placing too much emphasis on abstraction may result in inefficient code. Nevertheless, striving for optimal efficiency can run the risk of compromising software correctness and can result in the employment of architecturespecific programming models.

A solution is the use of a bridging model. This term was introduced in Leslie Valiant's 1990 paper: "A Bridging Model for Parallel Computation" [START_REF] Valiant | A Bridging Model for Parallel Computation[END_REF]. This paper argued that the strength of the von Neumann model was largely responsible for the success of computing.

Figure 1.2 schemes the idea of a bridging model where the signification of the directed arrow is (from the tail to the head) "can efficiently be simulated on". We can distinguish a sequential version in Figure 1.2a, based on the von neumann model; and a parallel one, Figure 1.2b. In computer science, a bridging model is an abstract model of a computer which provides a conceptual bridge between the physical implementation of the machine and the abstraction available to a programmer for the corresponding machine; in other words, it is intended to provide a common level of understanding between hardware and software engineers. A bridging model provides to software developers a way to escape architecture-dependent development.

A flourishing bridging model is able to be efficiently implemented on real architectures and efficiently targeted by programmers. In particular, it should also be possible for a compiler to produce efficient code from a high-level programming language. To evaluate a bridging model and a programming language based on this model, it is necessary to consider them in terms of all its properties: (1) its benefits as a basis for the design and analysis of algorithms in order to obtain a relevant complexity; (2) its ability to be applied onto the whole range of general-purposes architectures while providing efficiency and scalability; (3) its support of a fully-portable design; and (4) software engineering tools able to check correctness or debug programs based on a given model. For example, an efficient gpu-like program could be interesting in terms of performance and scalability, nevertheless it is not fully portable as it cannot be directly used on a cluster.

Von

The main advantage of a bridging model is that if an algorithm is efficient, then it is also efficient on "all" physical architectures. It will always be possible to have models adapted to particular architectures that are more detailed than a bridging model. Nevertheless, such a detailed model raises much more difficulties concerning cost analysis and portability. The usage of a bridging model also enables to write an algorithm, once; then, for all inputs and all architectures, an efficient and seasoned version is available.

Programming BSP algorithms in ML

As a partial solution to the aforementioned problems, some researchers have designed bridging models such as the bsp (Bulk Synchronous Parallelism) [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF][START_REF] Valiant | A Bridging Model for Parallel Computation[END_REF] model of computation. Many people are working on the way of programming bsp algorithms; and some of them program them in a functional way (using ml). In this section we are going to introduce the bsp model and its functional implementation: the bsml language. We also present the "philosophy" of bsp and bsml which are part of the foundation of this thesis.

The next section is dedicated to the limitations of bsp and bsml on modern architectures. It justifies the recursive and hierarchical extension of the bsp model called the multi-bsp model, and hence, the work of this thesis which is to propose a way to program multi-bsp algorithms in ml.

The BSP model of computation

The bsp model (Bulk Synchronous Parallelism) [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF][START_REF] Valiant | A Bridging Model for Parallel Computation[END_REF] is a bridging model designed for parallel architectures. We recall that the aim of a bridging model [START_REF] Maggs | Models of Parallel Computation: A Survey and Synthesis[END_REF] is to ease the way of programming various parallel architectures using a certain level of abstraction. It allows to reduce the gap between an abstract execution (programming an algorithm) and concrete parallel systems (using for example a compiler).

The initial assumption of the bsp model is to have a portable and scalable performance prediction for parallel programs. Without dealing with low-level details of parallel architectures, the programmer can focus on the design of an algorithm, its complexity, correctness, etc. A nice introduction concerning the bsp "philosophy" can be found in [START_REF] Skillicorn | Questions and Answers about BSP[END_REF] and a complete book of numerical algorithms is also available [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF]. A comparison of the bsp model with other different models can be found in [START_REF] Kielmann | Bandwidth-Latency Models (BSP, LogP)[END_REF].

The bsp bridging model describes (1) a parallel architecture, (2) an execution model ( 3) and a cost model in order to allows the performance prediction of a bsp algorithm on a given architecture. In the following, we present the three key points of the bsp model.

The BSP architecture

A bsp computer is structured by three main components:

1. A set of homogenous pairs of processors-memories; A wide range of current architectures can be seen as a bsp computers. For example, shared memory machines could be used in a way such that each processor only accesses a sub-part of the shared memory (which is then "private") and communications could be performed using a dedicated part of the shared memory. Moreover the synchronisation unit is very rarely a hardware feature but is rather software [START_REF] Hill | Practical Barrier Synchronisation[END_REF]. Supercomputers, clusters of computing units [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF], multi-cores [START_REF] Yzelman | An Object-Oriented Bulk Synchronous Parallel Library for Multicore Programming[END_REF] and gpus [START_REF] Hou | BSGP: Bulk-Synchronous GPU Programming[END_REF], etc. can be thus considered as bsp computers. Figure 1.3 shows a representation of a bsp computer which is composed by a set of pairs of processor/memory connected by a network.

The execution model

The execution of a bsp program is a sequence of supersteps (Figure 1.4) where each one is divided into three successive disjoint phases: This structured model enforces a strict separation of communication and computation: during a superstep, no communications between the processors are allowed, only requests of transfer; information exchanges only occur at the synchronisation barrier. Note that a bsp library3 can send data during the computation phase of a superstep, but this is hidden from the programmers.

The cost model

The performance of a bsp computer is characterised by four parameters:

1. The local processing speed r;

The number of processors p;

3. The time L required for a barrier; 4. The time g for collectively delivering a 1-relation.

Where r, g and L can be expressed using FLoating-point Operations Per Second (flops).

A 1-relation is a collective exchange where every processor receives/sends at most one word. The network can deliver an h-relation in time g × h. To accurately estimate the execution time of a bsp program, these 4 parameters can be easily benchmarked [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF]. The execution time (cost) of a superstep s is the sum of the maximal local processing time, the data delivery and the global synchronisation times. The total cost (execution time) of a bsp program is the sum of its supersteps's costs. It is expressed by the following formula:

Cost(s) = max 0≤i<p w s i + max 0≤i<p h s i × g + L Where w s i
is the local processing time on processor i during superstep s and h s i is the maximal number of words transmitted or received by processor i during superstep s.

The bsp model considers homogeneous processors only. Indeed, with heterogeneous processors, due to global synchronisation, the speed r is lowered to the slowest processor. In practice, many hpc architectures have heterogeneous networks and memories but a homogeneous set of processors. Moreover, the design of algorithm made for heterogeneous architectures is known to be much more difficult. For example, managing many parameters to load-balance computations and data automatically introduces complexity in the designing of algorithms.

We now present how to program bsp algorithms in a functional way using the the bsml language. It is an attempt at providing a good balance between the two aforementioned opposite approaches of parallel programming: low-level (and subject to concurrency issues), and highlevel (with loss of flexibility and efficiency).

The BSML language

1.3.2.1 The essence of bsml bsml combines the high degree of abstraction of ml (with performances relatively close to c) with the scalable and predictable performance of the bsp model. On another side, we can find libraries such as mpi [START_REF] Snir | MPI-The Complete Reference. 2nd[END_REF], generally used with fortran or c, which are unsafe approaches leaving the programmer responsible for deadlock or non-determinism issues. Otherwise, algorithmic skeletons [START_REF] Cole | Bringing Skeletons out of the Closet: A Pragmatic Manifesto for Skeletal Parallel Programming[END_REF] proposes a safe programming model, but it is limited to a restricted set of algorithms. Discussions concerning those approaches are available in Chapter 2.

bsml follows the bsp paradigm to structure the computations and communications between the processors in a data-parallel fashion. All communications in bsml are collective (they involve all processes) and deadlocks are avoided thanks to global synchronisation barriers.

An important feature of bsml is its confluent semantics: whatever the order of execution of the processes, the final value will be the same. Confluence is convenient for debugging since it allows to get an interactive loop (also know as toplevel), with the exact same execution as in parallel. It also eases the incremental transformation of a ml program to a parallel version written in bsml. Last but not least, it is possible to reason about bsml programs using the coq ([[ 5 ]]) proof assistant [START_REF] Gava | Formal Proofs of Functional Bsp Programs[END_REF][START_REF] Gesbert | Systematic Development of Correct Bulk Synchronous Parallel Programs[END_REF] and to extract actual bsp programs from proofs. This feature is uncommon in the world of parallel languages [START_REF] Blelloch | Internally Deterministic Parallel Algorithms Can Be Fast[END_REF], even if it is a worthwhile property.

bsml [START_REF] Gesbert | Bulk Synchronous Parallel ML with Exceptions[END_REF] uses a small set of primitives and is currently implemented as a library [[ 6 ]] for the ml programming language ocaml [[ 7 ]]). Using a safe high-level language as ocaml to program bsp algorithms allows performance, scalability and expressiveness. The ocaml language was chosen among different variants of ml because of its close to c performance. As the evaluation strategy of ocaml is eager, it eases the cost analysis of a program. Moreover, the ocaml community is large and a generous set of libraries and tools are available.

CHAPTER 1. INTRODUCTION

The bsml primitives work over a parallel data structure called a parallel vector. To handle this parallel data structure, the language proposes two functions to manipulate it. As well as four constants to access the bsp parameters of the underlying architecture. By comparison, the standard bsplib [[ 8 ]] [START_REF] Jonathan | BSPlib: The BSP Programming Library[END_REF] library for bsp programming in c offers about twenty primitives, and mpi more than a hundred. The bsml approach is uncommon since the majority of parallel (or distributed) programming languages (or libraries) offers a lot of primitives. Most of the time, the programmer must choose between basic and optimised (and complex) primitives to program an algorithm. By reducing the primitives, bsml aims to simplify and structure the way of programming parallel algorithms. As bsml is conformant to the bsp cost model, it enables a seamless cost analysis of algorithms.

Model of execution

A bsml program is written as a ml one where it is possible to express parallel computation using parallel vectors. The core syntax of bsml is alike the ocaml one, with few restrictions. bsml programs can be read as ocaml ones. In particular, the execution order should not appear unexpected to a programmer used to ocaml, even if the program is parallel. Moreover, most ocaml programs can be considered as bsml programs that do not use parallelism. Such a program would be executed sequentially on each processor of the parallel machine and would return their results, as usual. This behaviour allows the parallelisation to be done incrementally from a sequential program.

The ml type of parallel vectors is 'a par . A vector expresses that each of the p processors embeds a value of any type 'a . The processors are labelled with ids from 0 to p -1 where p is the number of processors. It is important to note that the number of processors is constant throughout the whole execution of the program. p can be accessed in bsml using the integer constant bsp_p . The other bsp parameters are accessible as float values through the constants bsp_g , bsp_l and bsp_r .

We use the following notation to describe a (parallel) vector: ⟨v 0 , v 1 , . . . , v p-1 ⟩. We distinguish a vector from a usual array because each local value, that will be called local values, are not aware of others. It is possible to access the local value v i of a vector locally: on processor i (using a specific syntax); or after some communications. The nesting of parallel vectors is not allowed in bsml-see Section 4.1.3 of Chapter 4 for discussions and a type system that prevents this forbidden use of vectors. Thus, the vector ⟨x 0 , x 1 , . . . , x p-1 ⟩ holds the value x i on processor i.

Since a bsml program deals with the whole parallel machine and individual processors at the same time, a distinction between the 3 existing levels of execution is needed:

1. Replicated or r is the default level of execution. Code that does not involve bsml primitives is run by the parallel machine as it would be by a single processor. Replicated code is executed at the same time by every processors, and leads to the same result everywhere.

2.

Local execution (l) represent an evaluation within the scope of a parallel vector. Each processor uses its local data to do their own computations.

3. Global execution (g) concerns the set of all the processors together, as for bsml communication primitives.

The distinction between local and replicated is strict: the replicated code cannot depend on local information. If such a thing happened, it would lead to a replicated inconsistency. A predefined vector: i on processor i

$v$ l 'a (if v: 'a par ) v i on processor i, assumes v ≡ ⟨ v 0 , . . . , v p-1 ⟩ proj g 'a par-> (int-> 'a) ⟨x0, . . . , xp-1⟩ → ( fun i → xi) put g (int-> 'a)par-> (int-> 'a)par ⟨f0, . . . , fp-1⟩ → ⟨ fun i → fi 0, . . . , fun i → fi (p-1)⟩
Table 1.1: Summary of the bsml primitives.

Parallel primitives and their informal semantics

Parallel vectors are handled through the use of different communication primitives. Table 1.1 summarises the bsml primitives. Informally, the bsml primitives works as follow: let << e >> be the vector holding e everywhere (on each processor), the << >> piece of notation indicates that we enter the scope of a parallel vector and, thus, we switch to the local level. Here, replicated values are directly available inside the vector. To access to the local value of a vector named x inside the scope of a parallel vector, we add the syntax $x$ . The ids can be accessed with the predefined vector pid . Note that, in the bsml implementation, ids are accessible through $this$ . For the sake of coherence with the rest of the thesis, we choose to replace the keyword this by pid . Nevertheless, they do have the same meaning.

To illustrate the previous primitives with a small piece of code, we use the bsml toplevel that simulates a bsp machine with 3 processors: # let vec1 = << "Hello" >> in << $vec1$^", proc "^(string_of_int $pid$) >>;; -: string par = <"Hello, proc 0", "Hello, proc 1", "Hello, proc 2"> The # symbol is the prompt that invites the user to enter an expression to be evaluated. Then, the toplevel gives the evaluated value with its type. Thanks to bsml confluence, it is ensured that both the results of the toplevel and the distributed implementation are identical.

In this code, we create a vector vec1 containing a string and we built a new vector by concatenating vec1 with the vector identifier (using $pid$ ).

The proj primitive is the only way to extract local values from a vector. Given a vector, proj returns a function such that, applied to a valid identifier of a processor, it returns the corresponding value of the vector. proj performs communications to make local values available globally (a single-broadcast), and ends the current superstep. One can say that proj is a parallel vector destructor. For example, if we want to convert a vector into a replicated list, we write: In this code example we map the function produce by (proj vec) on procs , a list of ids [0; 1; …; p -1]. The function list_of_par extracts, for each process, all the values of vec . Thus, the evaluation of list_of_par on a parallel vector containing the pids produces the list of pids (on each processes). The bsp cost of such a function is (p-1)×s×g+L where s is the size of the biggest sent value. Indeed, each processor will receive all the values of other processors (to be replicated) but only one processor will send the biggest value (p-1) times.

It is interesting to note that the result of proj is of type (int -> 'a) . It could have returned an array or a list of size p instead, but the interface is more functional. Furthermore, as seen in the examples, the conversion between one style to the other is easy.

The put primitive is another communication primitive. It allows any local value to be transferred to every other processor. It is also synchronous, and ends the current superstep. The parameter of put is a vector of functions (of type (int -> 'a) ) that describes, for each processor, the values that one processor wants to send to each other. Thus, the function returns the data to be sent to processor j when applied to j. The result of put is a new vector of functions: at a processor j the function, when applied to i, yields the value received from processor i by processor j. The canonical use of put is: (put << fun sendto -> e($pid$, sendto, $x$) >>) where expression e computes (or usually, selects) the data that should be sent depending on sender to sendto . For example, a total_exchange could be written:

Some simple examples

In the following, we give three examples that are useful in bsp programming and some are already available in additional bsml libraries. As they are typical of the way of programming bsml algorithm, it is interesting to detail them. Firstly, it is often needed to scatter a data set among the processors. The following function is a way to select a sub-part of a list on every processor of a bsp computer:

#(* select_list:'a list -> 'a list par *) let select_list l = let len = List.length l in <<cut_list l ($pid$ * len / bsp_p) (($pid$ + 1) * len / bsp_p))>>
where cut_list l a b returns the sub-list of the elements of l from index a (inclusive) to index b (exclusive). Now, in order to map a function, in parallel, on a distributed list (which is a classical data-parallel skeleton [3,[START_REF] Cole | Bringing Skeletons out of the Closet: A Pragmatic Manifesto for Skeletal Parallel Programming[END_REF]), we can write this simple code:

(* parmap : ('a->'b)->'a list par -> 'b list par *) #let parmap f parlist = << (List.map f) $parlist$ >>
Secondly, reducing and scanning are also important operators which are both implemented as mpi collective operations. A simple one-superstep reduce (having ⊕ p-1 k=0 v k on each processor from the parallel vector ⟨v 0 , v 1 , . . . , v p-1 ⟩) can be done with:

(* simple_reduce: ('a->'a->'a)->'a->'a par -> 'a let simple_reduce op e v = List.fold_left op e (proj_list v)
here e is a neutral element and op the associative operator ⊕. This reduce function does not use the capabilities of the parallel machines. If the combination operator ⊕ has an important cost, we may use a multi-step reduce (in a logarithmic way), by applying the operator locally. The following algorithm combines the values of processors i and i + 2 n at processor i for every superstep n from 0 to ⌈log 2 (p)⌉: The third example concerns the parallel sorting by regular sampling algorithm (psrs) [START_REF] Li | On the Versatility of Parallel Sorting by Regular Sampling[END_REF][START_REF] Shi | Parallel Sorting by Regular Sampling[END_REF] in its bsp version [START_REF] Tiskin | The Design and Analysis of Bulk-Synchronous Parallel Algorithms[END_REF]. Given a distributed list, the psrs algorithm proceeds in four steps:

(1) The p sub-lists of each processor are sorted locally by a sequential algorithm. The problem now consists in merging the p sorted lists.

(2) To do so, we must choose globally p -1 pivots in order to split the data into p sub-sets.

(3) We now need to communicate those sub-sets: processor i will receive, from every other process, the sub-list corresponding to the elements from (i -1) th to i th pivot.

(4) Finally, each processors must merge the received sub-lists.

After these steps, the result consist in a distributed list ordered through the p processes. The efficiency of the algorithm relies on a suitable selection of the pivots, done in step (2). This selection, called regular sampling, can be done in three steps: where vlengths is the vector of list sizes, extract_n n extracts n -1 evenly distributed elements; slice_p transforms a sorted list to a list of list depending on the samplings and p_merge merges sorted lists. The cost of steps 1 to 2(b) is thus:

n p ×log( n

Low-level primitives and the Coq proof assistant

Before [START_REF] Bousdira | Functional Parallel Programming with Revised Bulk Synchronous Parallel ML[END_REF], bsml had only the put and proj communication primitives combined with: where mkpar aims to create a parallel vector as an array and apply allows the local application of a vector of functions onto a vector of values. Those primitives are the ones that are used in the current implementation [START_REF] Gava | Formal Proofs of Functional Bsp Programs[END_REF][START_REF] Gesbert | Systematic Development of Correct Bulk Synchronous Parallel Programs[END_REF], and inside the Coq [START_REF] Bertot | Interactive Theorem Proving and Program Development -Coq'Art: The Calculus of Inductive Constructions[END_REF] proof assistant. Introduced in [START_REF] Bousdira | Functional Parallel Programming with Revised Bulk Synchronous Parallel ML[END_REF], $v$ and << e >> is a syntactic sugar of mkpar and apply that aims to ease the way of programming algorithm in bsml. It is not suitable for formal purposes. The syntax of vectors ( << e >> ) and low-level primitives are equivalent. However, the use of those primitives can sometimes become awkward. Indeed, every operation inside the scope of a parallel vector has to call a primitive and define an "ad hoc" function. This gets worse when working with multiple vectors and nested calls to apply .

In Coq, we also used these low-level primitives as the syntactic transformation is not available. Moreover, studying the semantics without the syntactic sugar is easier. For example, we do not have to differentiate variables annotated with a $ and the ones which are not inside the scope of a parallel vector.

From BSP to multi-BSP 1.4.1 Pro and Cons of BSP and BSML

Benefits

The bsp's execution policy has two main advantages. First, it removes non-determinism (no concurrent accesses) and guarantees the absence of deadlocks, as there is no shared memory and communications are always collective. This is a worthwhile point for proving the correctness of algorithms, as it is done in [START_REF] Fortin | BSP-Why: A Tool for Deductive Verification of BSP Algorithms with Subgroup Synchronisation[END_REF]. Secondly, it proposes an accurate model of performance prediction based on the bandwidth and the latency of the interconnection network, and on the speed of processors. This performance prediction model can even be used at runtime to dynamically make decisions, for example, choose whether or not to communicate a data in order the re-balance the workload.

On many distributed architectures, synchronisation barriers are often expensive when the number of processors dramatically increases (more than 1 000). Even if proprietary architectures usually provide much faster synchronisation systems, this cost may be important. However, efficient bsp algorithms tend to reduce the use of synchronisation barriers, making them almost negligible (in term of speed-up constraint). In spite of the performances issues, synchronisation barriers have several non negligible benefits: it is harder to introduce the livelock, since barriers do not create circular data dependencies; barriers also permit new forms of fault tolerance, as shown in [START_REF] Skillicorn | Questions and Answers about BSP[END_REF].

The bsp model also considers communication actions en masse. This is less flexible than asynchronous messages, but easier to debug since there are many simultaneous communication actions in a parallel program, and their interactions are typically complex. Bulk sending also provides better performance since, from an implementation point of view, grouping communication together in a separate program phase allows global optimisations of the data exchange by the communications library. Moreover it is easy to measure, during the execution of a bsp program, the time spent to communicate and to synchronise (by adding timers before and after the primitive of synchronisation). This capability is mainly uses to compare different algorithms. Since bsp programs are portable and cost predictions can lead to power consumption estimation, they are interesting in cloud-computing [5]. Thus it is not surprising to read that both the famous Google's MapReduce [START_REF] Dean | MapReduce: Simplified Data Processing on Large Clusters[END_REF] and the Pregel framework [START_REF] Malewicz | Pregel: A System for Large-Scale Graph Processing[END_REF] (for large graph processing) are based on the bsp model of execution.

In respect to the level of abstraction of the chosen language, the programmer can gain in programming simplicity but lose in terms of flexibility and performance. Indeed, it is known that a skilled programmer can produce an efficient and rather optimised code using a low-level language combined with a message passing interface. Nevertheless, to write such a code is not always possible with very complicated simulation running on complex architectures. Moreover, it is often a long task (that may introduce bugs) which is rarely profitable when using a wide range of different machines [START_REF] Gorlatch | Send-Receive Considered Harmful: Myths and Realities of Message Passing[END_REF][START_REF] Lee | The Problem with Threads[END_REF].

Another advantage of bsp is the way it eases debugging. As the computations during a super-step are independent, they can be debugged independently. This capability is used to formally prove the correctness of algorithms.

The simplicity of programming, debugging and proving, but also the efficiency of the bsp model makes it a good framework for teaching: a few hours are needed to teach bsp programming and algorithms.

The bsp model has also been used with success in a wide variety of problems. In the following, we propose a non-exhaustive list of the most recent works from our knowledge: scientific computing [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF], string search and genetic algorithms [START_REF] Deo | Coarse-Grained Parallelization of Distance-Bound Smoothing for the Molecular Conformation Problem[END_REF][START_REF] Krusche | New Algorithms for Efficient Parallel String Comparison[END_REF], graphs [START_REF] Chan | CGMGRAPH/CGMLIB: Implementing and Testing CGM Graph Algorithms on PC Clusters and Shared Memory Machines[END_REF], search engine (queries to textual databases) [START_REF] Costa | A Parallel Search Engine with BSP[END_REF], 3-sat solver [13], algebra [START_REF] Salem | Parallel Methods for Absolute Irreducibility Testing[END_REF], scheduling threads [START_REF] Da | Observing the Impact of Multiple Metrics and Runtime Adaptations on Bsp Process Rescheduling[END_REF], multi-agent services [START_REF] Jiang | Resource Load Balancing Based on Multi-Agent in ServiceBSP Model[END_REF], image processing [START_REF] Hou | BSGP: Bulk-Synchronous GPU Programming[END_REF][START_REF] R. D R. Righi | MigBSP: A Novel Migration Model for Bulk-Synchronous Parallel Processes Rescheduling[END_REF], etc. bsp was adopted because it represents a common model for writing successful parallel programs that exhibit phase-based computational behaviours [START_REF] Hendrickson | Computational Science: Emerging Opportunities and Challenges[END_REF].

In addition to being convenient in practice, bsp allows the design of what could be called "immortal algorithms". Proposed by Leslie Valiant and Bill McColl in the 1990's, an immortal algorithm is an algorithm that is written once and can be run optimally on any parallel machine of today or tomorrow, because its bsp algorithm cost is optimal. For example, for sequential computing, the merge sort algorithms is known to be of time complexity of O(n log(n)); such a complexity will be valid forever. For parallel computing, because the bsp cost model is a realistic cost model for standard parallel architectures, the cost (formulae) of bsp algorithms will also remain the same in the future. Thus, algorithm designers can formulate immortal algorithms that are proved optimal regardless of the values of (p, g, L): algorithms that need to be written just once, and remain forever optimal. In the absence of a proof of optimality (but with an algorithm that still reduce and balance both computation and communication, while avoid synchronisation), the cost model allows derivation of the asymptotic behaviour of the algorithm to determine its scalability. Such a property allows the design of parallel algorithms for long-time support.

Disadvantages

All the benefits of the bsp model are possible because the runtime system knows precisely which computations are independent. In an asynchronous message-passing system like mpi, the independent sections tend to be smaller and identifying them is harder. Nevertheless, using bsp, programmers and designers have to keep in mind that some parallel patterns are not really bsp friendly. For example, bsp does not handle with ease pipeline and master/slave paradigms (also known as farm of processes). Nevertheless, it is possible to get a not too inefficient bsp version of such patterns, as we will see later in this document. Therefore, some parallel computations and other optimisations would never be bsp [START_REF] Eisenbiegler | BSP, LogP, and Oblivious Programs[END_REF]. This is the drawback of bsp that also concern all the restricted models of computations as well.

In parallel programming, it is convenient to stop a computation if an exceptional case arrives (typically an error). It seems natural that one processor could warn other processors that an error was raised. Using the strict bsp model, it is impossible to do so until the end of the supersteps. Such a behaviour may let other processors to continue their computations for nothing.

Another example is the pipe-lining pattern (where each processor sends its value to the next processor and waits for the data of its preceding processor). As in bsp the communication cannot be asynchronous, such a code is clearly not adapted to the bsp model. Nevertheless, with some restrictions on the operations on the data, it is possible to get a not too inefficient bsp algorithm to solve this problem (and more) [START_REF] Da | Observing the Impact of Multiple Metrics and Runtime Adaptations on Bsp Process Rescheduling[END_REF].

The bsp cost model reveals two increasing and unavoidable sources of costs that are ruled by the physical realities of the architectures: a cost g for bandwidth and a cost L for latency.

As architectures are growing bigger, those cost are becoming more and more expensive. Such concerns are important but quite complicated to handle with bsp. The bsp model considers homogeneous processors only. Indeed, with heterogeneous processors, due to global synchronisation, the speed r is lowered to the slowest processor. In practice, many hpc architectures have heterogeneous networks and memories but a homogeneous set of processors. Moreover, the design of algorithms made for heterogeneous architectures is known to be much more difficult. For example, managing many parameters to load-balance computations and data automatically introduces complexity in the designing of algorithms.

The worst case concerns is the congestion problem that arises when using bsp algorithms on a cluster of multi(many)-cores [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF]. In fact, the design of a bsp algorithm makes the hypothesis that communications are nearly identical over all the network (the idea of the h-relations). But this is not accurate when comparing the communication between the cores of the same machine (shared memory) and cores of different machines (over the network). Let as see this problem with a practical example.

The problem of congestion using a cluster of many-cores

As said above, one of the main advantages of the bsp model is its cost model: it is quite simple and quite accurate on a cluster or on multi-cores. Moreover, the cost model allows to reason on algorithm efficiency. One might wonder if the bsp model is also accurate with clusters of multi-core. Many works [START_REF] Gerbessiotis | Extending the BSP Model for Multi-Core and out-of-Core Computing: MBSP[END_REF][START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF] think not; therefore we would like to try to answer the question using bsml. To do so, we use a bsml version of a program from [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF] that aims to benchmark and determine the bsp parameters of a parallel machine. It measures the time to perform some random collective all-to-all operations of growing size; then a least squares methods is applied to compute the values g and L.

The benchmark were done on two parallel architectures named Mirev2 and Mirev3 of the lifo: "Laboratoire d'Informatique Fondamental d'Orléans, Orléans, France" [[ 9 ]]. Here are the main specifications of these machines:

• Mirev2: 8 nodes with 2 quad-core (AMD 2376) at 2.3Ghz with 16Gb of memory per node and a 1Gb/s network.

• Mirev3: 4 nodes with 2 hyper-threaded octo-core (16 threads)(Intel XEON E5 -2650) at 2.6Ghz with 64Gb of memory per node and a 10Gb/s network. On both machines, the version 4.02.1 of ocaml, gcc 4.9 and mpich 3.1 were used. The g and L bsp parameters are given in Figure 1.5. The number of cores growth in abscissa and we distribute the computations on cores with two different strategies: (1) cycling on nodes in a round-robin fashion; (2) using as much as possible of cores on each node. One can notice that until 64 cores g evolves as expected: (1) the round robin distribution lacks from its intense network access in the first time; (2) on the contrary, the distribution by nodes achieves better performances as it minimises network accesses. However, when an equivalent number of processes are accessing the network the performances of both distributions are similar. With more than 64 processes, which is the number of physical cores of our architecture, the performances reach a plateau. Too many cores are accessing the network, resulting in a bottleneck which severely hindering the performance. Then, the value of g is not accurate making impossible to use the bsp cost model correctly.

Regarding the L parameter, we can observe an important growth of the parameter as the number of processing grows.

To show this problem of accuracy and hence of bottleneck, we can grow the size of the data exchanged during a total-exchange. The bsp cost prediction will exhibit a linear growing:

p × n × g + L.
Where n is the constant size of data on each processor. On Figure 1.6 we show the computation time of a total-exchange (with a fixed data size) concerning a growing number of processes. The reader can see that the actual execution time differs from the cost prediction: the bsp cost prediction is not precise enough.

Without the cost prediction, we lost one of the main advantage of a bridging model such as bsp. Different solutions have been proposed in the past. We compare them in the State of the art, Chapter 2. We have chosen Valiant's solution [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF] because of its elegance: it seems to be a simple model which is precise and close enough to typical and modern hpc machines; in short the best bridging model from our point of view.

The Multi-BSP model of computation

Modern hpc architectures are made of hundreds of interconnected nodes each, with thousand of cores. As we have just seen, the bsp model is not truly adapted for these architectures that breaks the bridging model. An updated of the bsp model that is able to handle such architectures is needed. This model needs to be defined with the same level of abstractions and must bridge the architecture as the original bsp model. Furthermore, it must be adapted to clusters of multi-cores architectures. Valiant made a proposition of such a model and he called it the multi-bsp model [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF]. The goal of Valiant is: with the comparison with the previous literature, our goal here is that of finding a bridging model that isolates the most fundamental issues of multi-core computing and allows them to be usefully studied in some detail.

The Multi-BSP architecture

The multi-bsp model introduces a representation where a hierarchical architecture is a tree structure of nested components (sub-machines) where the lowest stage (leaves) are processors and every other stage (nodes) only contains memory. The tree is of depth d and is assumed balanced and homogeneous.

In the following, we use a particular vocabulary to describe the multi-bsp architecture. A component stands for an entity of the multi-bsp architecture, which can be a node or a leaf. A stage designates a given depth of the multi-bsp architecture.

At the ith stage there are a specified number of components, each of them manages a number of i-1 th level components, as illustrated in Figure 1. 7 (left). This number of sub-components only depends on the depth i of the multi-bsp architecture. Finally, for the stage of depth d: the leaves, we find the hardware threads (unit of computations) as illustrated in Figure 1.7 (right). Each processor can access a small memory (usually caches). Figure 1.8 shows the multi-bsp model of an architecture with its parameters (which are explained below).

Stage i (Node) Note that a sub-component (child) can access the memory of its top component of stage i (parent). However, they cannot directly access to the memory of their "brothers" without doing an explicit copy through the memory of the component i.

M i S i-1 0 S i-1 p i L i g i g i Stage i (Leaf) M i P rocessor g i L 1 = 0
The multi-bsp model is assumed to be homogeneous concerning the processors but not with the network capacities (bandwidth and latency) and memory sizes: they are identical for all the We have made some minor modifications compared to the original multi-bsp model. Those modifications do not change the way to write or analyse the algorithms, they aim to simplify the approach and be more consistent with the rest of the thesis. In particular, we assume that the processors have access to a local memory whereas the original model assumes an additional abstract stage. Otherwise, the bandwidth parameter of the memory will concern the current stage, instead of the upper stage as described in the original version of multi-bsp (see [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF]).

The execution model

The execution model of multi-bsp is similar to the bsp one. The notion of superstep is the key and those supersteps are hierarchically organised in order to manage sub-synchronisations.

At a stage i, a superstep is composed by the following steps:

• The sub-components of stage i -1 execute some code independently (until they reach a barrier)

• There are exchanges of informations with the m i memory

• Synchronisation of the sub-components (stage i -1)

Then, a new superstep can start at stage i. Note that the definition requires synchronisation of the sub-components of a component, but no synchronisation across branches that are separated in the component hierarchy. The synchronisation of a superstep concerns all the nodes of the sub-components (stage i -1) and there is no sub-group synchronisations.

Figure 1.9 illustrates the sequence of local supersteps of the multi-bsp model. We can observe that the architecture is composed by a multi-core decomposed in two cores with respectively four threads per core. The black rectangles stand for the computations and the dotted lines symbolise the communications. Regarding the stages that are only composed by a memory capacity (that is to say nodes), all the memory manipulations is done by one of the processors of the branch (which is chosen depending of the implementation). The processor is charged to perform the data exchanges such as scattering or gathering data, etc. To do so, the most part of the workload must be done by the stage 1, that is to say, in the cache memory. Otherwise, the computations that are done in the top memories must focus on data management (such as scattering or gathering data).

The main observation done by Valiant in [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF], is that certain recursive algorithms are well suited to computational models with multiple parameters. Programming such algorithms is an onerous task, however, the use of a bridging model enables one to make one big effort, once and for all, to write a program that is efficient for all inputs and all machines.

The cost model

An instance of a multi-bsp architecture is defined by the depth d of a tree and four parameters; four parameters for each stage i:

• p i is the number of sub-components inside the i -1 stage.

• g i is the bandwidth between stages i and i -1: the ratio of the number of operations to the number of words that can be transmitted in a second (illustrated in Figure 1.10). The difference is that, in the basic bsp model, direct communications are allowed horizontally, between units at the same stage. On the contrary, in multi-bsp, such communication must be simulated via a memory of a higher stage; For the sake of simplicity, horizontal communications are also allowed (as suggested by Valiant in [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF]) with this same bandwidth parameter g i 4

• L i is the synchronisation cost of all sub-components of a component of i -1 stage.

• m i is the amount of memory available at stage i for each component of this stage.

CHAPTER 1. INTRODUCTION

Finally, there is the local processing speed r of each of the computing units (leaves).

Stage i

Stage i - We consider p 0 as a basic computing unit (a processor) where a step on a word is considered as the unit of time. Therefore, it does not share any memory with any other component and does not have sub-component making p 0 equal to 1 and L 0 equal to 0. On another side, the stage 1 memory is accessed by the level 0 processors, and thus the data rate (corresponding to the notional g 1 ) has value 1. Figure 1.11 illustrates the difference between the bsp model and the multi-bsp one for a multi-core architecture with threads5 .

1 n n.1 … … n.p i g i+1 g i m i L i

Notation:

In the following, when representing multi-bsp architecture, a node will be symbolised by a circle: ; a leaf by a square: . Compared to other models, the d = 0 case with (p 0 = 1, g 0 = 1, L 0 = 0, m 0 ) (where m 0 is the size of the RAM) gives the von Neumann model. The bsp model with parameters (p, g, L) where the basic unit has memory m would be modeled with [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF]

d = 1 and (p 0 = 1, g 0 = 1, L 0 = 0, m 0 = m), (p 1 = p, g 1 = g, L 1 = L, m 1 )
where m 1 is, for example, the size of a slower but bigger memory. Another value for m 1 could be n, the size of the problem (implemented as a virtual memory with a distributed file system). Assuming a specific architecture with data streams as input, m 1 could also be ∞ and g 1 could rely on the acquisition stream data rate. We can relate multi-bsp parameters to existing multi-core architectures. For example, a cluster of p multi-processors of multi-cores with d = 3 may have the following multi-bsp parameters6 : The MBSPDiscover tool of Alaniz et al. [START_REF] Alaniz | MBSPDiscover: An Automatic Benchmark for MultiBSP Performance Analysis[END_REF] is able to determine the multi-bsp parameters (for c codes) in order to get realistic values of the multi-bsp parameters. A case study of measurement of multi-bsp parameters can be found in [START_REF] Savadi | Measurement of the Latency Parameters of the Multi-BSP Model: A Multicore Benchmarking Approach[END_REF].

multi-bsp

We would like to add a few remarks on these parameters, proposed in [START_REF] Valiant | A Bridging Model for Multi-Core Computing[END_REF]. First, the L i parameters (except for the stage d) are the latency parameters given by the chip manufacturer specifications, rather than the actual synchronisation costs. Second, the caches of the chip (L1 or L2 memories) are controlled internally by a specific protocol and cannot be explicitly used as usual, like other memories. Third, in a chip each physical computing unit runs threads, and groups of threads share a common arithmetic unit (or even a gpu). For all these reasons, while the relative magnitudes of the various g i and L i values shown may be meaningful, their absolute values may be hard to figure out, as shown in [START_REF] Alaniz | MBSPDiscover: An Automatic Benchmark for MultiBSP Performance Analysis[END_REF].

We saw that when a node executes some code on its nested components (children), it has to wait for the end of the sub-computation to do the communications and synchronise the children.

Consider the communication cost C i j of a superstep j at stage i as following:

C i j = h j ×g i +L i
where h j the maximum size of the exchanged messages at superstep j, g i the communication bandwidth with stage i and L i the synchronisation cost. We can express the cost T of a multibsp algorithms as following:

T = d-1 ∑ i=0 ( N i -1 ∑ j=0 w i j + C i j )
where d is the depth of the multi-bsp architecture, N i is the number of supersteps at stage i, w i j is the maximum computational cost of the superstep j within stage i.

We can also recursively express the cost T of a multi-bsp algorithm as following:

T = N root-1 ∑ j=0 (T root-1 + C root j + w root j )
with:

         T 0 = w 0 j T i = N i -1 ∑ j=0 (T i-1 + C i j + w i j )
where T i is the multi-bsp cost at stage i and root is the root node of the multi-bsp architecture such that root = d -1.

Finally, we have three main observations about the multi-bsp architecture.

First, and unlike the bsp model, the multi-bsp model controls the storage explicitly. Nevertheless, it is not clearly specified how the various cache protocols are supported by the model.

Second, we only need that machines support the features of this model; no constraint is implied about what else they might support. Thus there is no objection to machines being able to have additional mechanisms such as external disks, gpu, etc. Valiant proposes a new bridging model: a minimal description on which architects, algorithm designers and programmers can come together.

Third, the multi-bsp model does not propose a way to handle heterogeneous and unbalanced architectures. However, it is known that programming homogeneous architecture is already a challenging task and adding another level of difficulty may lead to a non-programmable model.

Outline

Chapter 2 is dedicated to the state of the art. We compare our language with other approaches and previously published works.

In Chapter 3, we introduce the multi-ml language: an extension of ml for programming multi-bsp algorithms, as bsml is for bsp algorithms. We present the main primitives and illustrate them with simple examples. We also explain the underlying ml types needed to contain the specific information added to manage the multi-levels of execution. We also detail the arbitrary choices made in the design of our language. Thus, we extract the essence of our language in order to propose it as a "nutshell".

In Chapter 4, we first present a core language of multi-ml which will be used for the formal work of this thesis. As traditionally, we then present the formal operational semantics using 1.5. OUTLINE a natural semantics (big-steps) for both terminating and diverging programs. Now that the dynamic part have been presented, we show the static analysis of our core-language, which is the type system, and how we infer types. To do so, we first present the grammar of the types (how ml types has been extended) to explain how and why they are used; then the rules of the type system are given. Secondly, we show how the type inference works, as a traditional constraints resolution. We finally give some results, and the proof of the technical lemmas finishes this chapter. During this chapter, we illustrate the rules and the definitions with some simple examples.

In Chapter 5, we give an overview of the current implementation of the multi-ml language. To do so, we describe an abstract compilation scheme from a multi-ml parallel code to an ocaml sequential version with low level communication primitives. This scheme allows us to propose a generic and efficient implementation. We then explain how to use this model to make a modular implementation. We finally give the benchmarks of some algorithms.

Chapter 6 concludes this thesis and gives an overview of the results; some possible (near) future works and what could be done in a more expectative way.

This chapter provides an overview of the parallel programming methods that are available in the literature. We introduce the parallel programming models that aim to structure the way of programming parallel algorithms and some of their implementations. We will present the advantages and drawbacks of the ones that are the most important for our concern.

Parallel programming models

In this section we present a non-exhaustive list of parallel computing models that aim to describe various parallel architectures. We try to describe how the models evolve to match new architectures and new expectations.

The PRAM family

The pram family is the oldest way of structuring parallel algorithms. Nevertheless, it is still an accurate way of studying the intrinsic parallelism of algorithmic problems. In the following, we describe the different evolutions of the pram model.

PRAM

The pram model [START_REF] Fortune | Parallelism in Random Access Machines[END_REF], first of its name, was introduced to conceive algorithms for parallel machines and measure the degree of parallelism of parallel algorithms. The Parallel Random Access Machine is made up by an unbounded number of p processors that can access to a shared memory. Using this shared memory is the only way to communicate between processors. At each evaluation step, each processor can read a value from the global memory, write a value to the global memory or compute in its own registers. The pram model allow the definition of a policy concerning simultaneous access to the global memory. These restrictions are: erew (Exclusive Read Exclusive Write), crew (Concurrent Read Exclusive Write) and crcw (Concurrent Read Concurrent Write). The pram model does not provide a way to handle data locality and synchronisation to ease writing algorithms. With this model, it is possible to estimate the work performed by the computation units only. Nevertheless, there is no way to consider latency and communication bandwidth, which are the main concerns using hpc architectures.

CHAPTER 2. STATE OF THE ART

APRAM

This is an asynchronous variant of the pram model called apram [START_REF] Gibbons | A More Practical PRAM Model[END_REF]. Its goal is to be close to the mimd architectures. In the apram model, each processor has its own local memory and can access a shared global memory. Unlike pram, the processors can run asynchronous computations. Nevertheless, a memory access policy is used on top of synchronisation steps. The model is composed by four types of instructions: (1) Read the global memory to store data in local memory; [START_REF] Alexandrov | LogGP: Incorporating Long Messages into the LogP Model-One Step Closer Towards a Realistic Model for Parallel Computation[END_REF] Local computations where data and results are stored into the local memory; (3) Write data from the local memory to the global memory; (4) Synchronisation steps. A synchronisation step consists of a logical point in a computation where each processor of a set waits for the other. Processors can read and write to shared memory asynchronously. Nevertheless, a processor cannot read a memory location that was written by another processor before a synchronisation step involving those processors. There exist variants of apram that introduce the notion of synchronisation latency costs [START_REF] Martel | Asynchronous PRAMs with Memory Latency[END_REF]. This approach is still incomplete regarding current hierarchical architectures.

HPRAM

The Hierarchical pram [START_REF] Heywood | A Practical Hierarchical Model of Parallel Computation I. The Model[END_REF] is proposed to allow distinct asynchronous pram computations to be synchronised within a single algorithm. hpram can build a disjoint subset of pram processors. Each subset receives different instructions that are executed asynchronously. Synchronisations are done independently, within the subsets. As expected, the partitioning instruction can be executed inside each subset. The hpram model proposes two variants. First, private hpram divides the memory among processors. Each subset has its own private memory block from the global memory. Secondly, shared hpram does not divide the shared memory. Every processor subset can access the global memory. hpram introduces latency and synchronisation costs. The latency cost is proportional to the size of the processor subsets. Concerning the synchronisations, it distinguishes synchronisations between the subsets and within the subsets. This model is closer to the hierarchical machines that are today standard, however such a dynamic hierarchy does not take into account the actual architecture. Thus it is not possible to benefit from the advantages of fast hierarchical memories.

The LogP family

Close to bsp, these models are, most of the time, used to study network capabilities and low-level libraries such as mpi.

LogP

The logp model [START_REF] Culler | LogP: Towards a Realistic Model of Parallel Computation[END_REF] is an asynchronous model that describes point-to-point communications for distributed computers with distributed memories. It aims to reflect the critical technology trends underlying parallel computers and networks. It was intended to serve as a basis for developing fast, portable parallel algorithms and to offer guidelines to machine designers. The model provides a way to handle communication costs using various parameters: (l, o, g, P ). The communication's latency upper bound is described by L. The incompressible send and receive cost is denoted by o. The parameter g represent the minimum gap between two consecutive messages, it can be interpreted as the inverse of the bandwidth per processors. Finally, P is the number of processors.

The logp model does not provide a way to handle costs induced by hierarchical architecture.

LogGP

The loggp [START_REF] Alexandrov | LogGP: Incorporating Long Messages into the LogP Model-One Step Closer Towards a Realistic Model for Parallel Computation[END_REF] model is an extension of the logp model for parallel computation which abstracts the communication of fixed-sized short messages through the use of four parameters. Like in logp, those parameters are L : the communication latency, o the overhead, g the bandwidth and P the number of processors. The logp model can predict communication cost of short messages accurately. Nevertheless, as many parallel machines have special capabilities concerning bandwidth optimisation of long messages, loggp has an additional parameter, G, which considers the bandwidth for long messages. This simple extension can quite accurately predict communication performances for both short and long messages.

Those additional parameters are not able to handle different bandwidth of today's architectures. Indeed, hpc architectures are mainly hierarchical and require a model that manages multi-levels of parameters.

LoPC

The lopc model [START_REF] Frank | LoPC: Modeling Contention in Parallel Algorithms[END_REF] aims to extend the logp model to deal with contention between nodes. The authors claim that the costs of network latency of both fine-grain message passing algorithms and shared memory are dominated by the network contention. lopc takes the L, o and P parameters directly from the logp model and uses them to predict the cost of contention C. The lopc model makes the assumption that the hardware message buffers are infinite and do not induce interconnection contention. The main goal of the lopc model is to write algorithms that minimises contention at runtime, with a limited set of parameters, and give a more accurate prediction of cost. A computation consists in a sequence of supersteps, with synchronisation taking place between supersteps.

The lopc model is close to the bsp model concerning its supersteps execution and cost model. Nevertheless, the point-to-point communication of the model is not structured enough and can lead to complicated communication schemes.

The sub-synchronous family

Even if sub-synchronisations can introduce problems [START_REF] Hains | Subset Synchronization in BSP Computing[END_REF], it brings a lot of benefits, especially in terms of performance concerning clusters of multi-cores. Indeed, as shown in Figure 2.1, subsynchronisation allow to split the machine (S) in order to synchronise, independently, different groups of processors (S1 and S2). The extension of the bsp model with sub-synchronisation is thus logical. We will introduce some of the well-know models.

E-BSP

The e-bsp model [START_REF] Ben | The E-BSP Model: Incorporating General Locality and Unbalanced Communication into the BSP Model[END_REF] (standing for Extended bsp) extends the bsp model in two ways. First, it aims to deal with unbalanced communication patterns where the amount of data sent or received by each processor is different. Secondly, it adds the notion of general locality to the bsp model by supplying a non-linear delay when accessing a remote memory. Like a bsp computer, the e-bsp model consists of several attributes: a set of processors/memory nodes, a point-to-point message router and a facility to synchronise the nodes. The only difference between bsp and e-bsp is the cost of communication within a superstep. The h-relation denotes the communication pattern in which each processor sends and receives at most h messages. In bsp, the cost of a super step is g * h + l (where g is the bandwidth and l the synchronisation cost). As the bsp model assumes that each processor receives, exactly, h messages, this estimation is overestimated with unbalanced communications: corresponding to the worst-case estimation. The e-bsp model proposes a data routing related to data localities with a better routing distribution (treated as a routing problem) to give a more accurate h-relation.

The e-bsp model trades simplicity for efficiency and complicates the design and analysis of parallel algorithms more than the bsp model. However, it does not handle multi-level architecture, that is the standard of today's architectures, and is not enough structured with the architectures itself.

dBSP

The dbsp model [START_REF] De | Submachine Locality in the Bulk Synchronous Setting[END_REF], for decomposable-bsp, extends the bsp model to take advantage of sub-machine locality. We denote dbsp (p, g(p), l(p)) a dbsp computer with p processors and particular router performance functions g(p) and l(p). During a superstep, all the processors can execute a split instruction to partition the computer into sub-machines (clusters). All the sub-machines compute independently as separate computers in the subsequent superstep. All the processors of all sub-machines must execute the join instruction to re-synchronise. After, a join, the computation of the sub-machine is suspended until a join is performed in all the sub-machines. Then, all the processors work together as in bsp. It is important to note that no partial synchronisation of the sub-machines are allowed.

As expected, the cost of a dbsp computation which is not decomposed is the same as the bsp one. Otherwise, the time between a pair of corresponding spilt and join is a maximum of the time complexities of computations performed by sub-machines. Assuming that the number of supersteps performed by the sub-machine i is S i , we have

T = max i {T i } = max i { ∑ S i k=1 (w k,i + h k,i g(p i ) + l(p i ))}.
A dbsp computer is, basically, a bsp computer where independent sub-group synchronisations are allowed. The cost model is thus extended with a few parameters to handle subsynchronisations. The cost analysis of parallel algorithms is more accurate but also more complicated due to the g(p) parameters that are difficult to measure. Furthermore, the dbsp model does not provide a way to manage hierarchical architectures with its own physical specifications.

It is interesting to note that the multi-bsp model is, in a way, a sub-synchronisation model. Nevertheless, it is more structured and does not allow dynamic group creation: it is based on the architectures itself.

The hierarchical family

HiHCoHP

The Hierarchical Hyper Clusters of Heterogeneous Processors (HiHCoHP) [18,[START_REF] Cappello | An Algorithmic Model for Heterogeneous Hyper-Clusters: Rationale and Experience[END_REF] claims to be the successor of the homogeneous logp model in a hierarchical way. The aim of HiHCoHP is to propose a detailed enough architecture with an abstraction layer, in order to make the model algorithmically tractable. The HiHCoHP model characterises a hypercluster via parameters that reflect its tri-axial heterogeneity. There is the processor's heterogeneity: the individual processor's message-processing and memory access times; the bandwidth heterogeneity due to the interconnected clusters of a hierarchy of networks; and size heterogeneity of each level of clusters. The computing power of a HiHCoHP resides in N heterogeneous computing nodes composed by a pair of processor and memory. The P i computing nodes intercommunicate via an l-level hierarchy of networks of successively higher latencies, lower bandwidth and higher capacities. The nodes of a HiHCoHP are aggregated into disjoint level -1 clusters. Two nodes communicate over the lowest-level network that they "share". The communication within a HiHCoHP is done by successive message processing and message transmissions between the P i nodes. This message passing encounters three sources of delay: network latency, network bandwidth and network capacity (maximum number of packets that can be handle at a time).

The HiHCoHP model proposes a very detailed cost model that distinguishes, for each level k and processor a: the communication initialisation cost σ k a ; the packet emission cost on the network π k a ; the maximum capacity of the network κ k ; the network latency λ k and the packet transmission cost β k . As we have a hierarchy of clusters, we have ∀a,

σ k a ≤ σ k+1 a , π k a ≤ π k+1 a , λ k a ≤ λ k+1 a , β k a ≤ β k+1 a
. Thus, the total cost for sending p packets from P a to P b is:

(σ k a + σ k b ) + (π k a + π k b ) × p + λ k + ∆(p) where ∆(p) = (p -1)/β k .
The numerous of parameters of the HiHCoHP model make the conception of algorithms difficult. Compared to multi-bsp, the execution model of HiHCoHP is not structured and can lead to deadlocks.

H-BSP

The h-bsp model [START_REF] Cha | H-BSP: A Hierarchical BSP Computation Model[END_REF] aims to add a hierarchical concept to bsp. An h-bsp program consists of a number of bsp groups which are dynamically created at runtime and executed in a hierarchical fashion. Each group behaves as an independent bsp system and they communicate asynchronously. The working mechanism of h-bsp is similar to the process tree concept of unix, in the sense that fork and join functionalities are available. Nevertheless, the group creation is done on existing processors and there are no process spawn. Thus, h-bsp provides a group-based programming paradigm and naturally supports divide-and-conquer algorithms.

The overall cost of a h-bsp algorithm comes from the bsp root, when the machine is not split. Hence, the cost of a h-bsp algorithm is obtained by combining the cost of the bsp root algorithm plus the sum of the costs of every group split. A group split reflects the time from the split to the join and it depends on the most costly group. Let p be the number of processors at root, f the number of groups at root, G i the numbers of bsp groups created by the i th split (1 ≤ i ≤ f ), gid the created group's id (0 ≤ gid ≤ G i -1) and GT (i, gid) the cost of the gid-th group resulting from the i th group split. Then, the cost of the i th group split is

F T i = max{GT (i, gid) where 0 ≤ gid ≤ G i -1} + S H (G i , p).
Where S H (G, p G ) is the synchronisation cost depending the number of groups G and the number of processors p G in the parent group. Thus, the total cost of a split at root is

∑ f i=1 F T i . Finally, the overall cost of an h-bsp algorithm A is T h-bsp (A) = N root comp + N root comm × g + S root × l + ∑ f i=1 F T i . h-bsp
inherits the bsp principles but its hierarchical structure reflects the locality of processor communication and synchronisation. h-bsp also provides cost analysis and the authors claim that the model predictions are better than the bsp model, thanks to its locality-preserving nature. Unlike the multi-bsp model, the h-bsp cost model is not fine enough to take into account the various bandwidth and latencies that belong to hierarchical architectures. Moreover, the ability to create groups dynamically makes the algorithmic analysis harder.

SGL

The sgl [START_REF] Li | SGL: Towards a Bridging Model for Heterogeneous Hierarchical Platforms[END_REF], standing for Scatter-Gather Language, aims to propose a programming language (as a set of primitives) adapted to multi-level and heterogeneous architectures. It is based on the idea of scattering data, which corresponds to a phase of data and computation distribution, followed by a gathering step, to collect all the computation results. sgl is very close to the multi-bsp model and the multi-ml language in its recursive bsp vision.

The sgl model defines a set of processors composed by both a processing unit and a memory. The processors are arranged in a tree structure with the root node called "master" and its children that are either masters themselves or "leaf-workers". The number of children is not limited so that the bsp/pram concept of flat p-vector of processors is easily simulated in sgl.

A sgl machine can be a sequential machine: only one worker without master; a bsp computer: master with several workers; a hierarchical machine of any shape. Nevertheless, a sgl computer must have only one root-master, a master coordinates its children through communication, a worker is controlled by only one master and communications is always between master and children. The execution model is composed by a sequence of supersteps composed of 4 phases: (1) A scatter from the master; (2) An asynchronous computation performed by the children (this phase can also be a nested sequence of sgl supersteps); (3) A gather communication phase centred on the master; (4) A local computation phase on the master.

Like all the bsp-inspired systems, the cost T of an sgl algorithm is the sum of the costs of its supersteps S. The cost of a single superstep is divided into two independent terms: computation cost and communication cost. Thus T = ∑ S where S = Comm scatter + maxComp children + Comm gather +Comp master , Comm scatter = k scatter ×g scatter +l, Comm gather = k gather ×g gather +l and k is the number of words, g the bandwidth and l the synchronisation cost. We can express a more detailed cost formula by defining g ↓ as the time interval for transmitting one word from master to its children and g ↑ for children to master, k ↓ the number of words that the master scatters to children and k ↑ the number of words that master gather from children, c the computation speed of processors (c 0 for the master and c i (i = 1...p) on children) and p the number of children processors that master has. Then,

Cost master = Cost child i i=1...p + w 0 × c 0 + k ↓ ×g ↓ +k ↑ ×g ↑ +2l and Cost worker = w i × c i . In case of symmetric communication, we have Cost superstep = w × c + ( max i=1...p (Cost child i ) + (k ↓ +k ↑) × g + 2l).
The sgl model proposes a simple execution model with a lightweight cost model to program multi-level architectures. However, there is no programming language implemented over this model. As the sgl communication scheme resides in scatter and gather operations, it is not conceivable to express point-to-point communications. The multi-bsp cost model is simpler, as there are fewer parameters, but it is accurate enough for hpc architectures. Regarding its approach and behaviour, sgl is really close to algorithmic skeletons.

The BSP like models

BSP2

The bsp2 [START_REF] Jeremy | BSP Modelling of Two Tiered Architectures[END_REF] model was proposed to reduce the difference between real hardware and the bsp model. The observation states that 1) Local memory access times are variable and are affected by the usual hierarchy of memory, cache and registers. 2) Access times to remote memory may vary because of non-uniformity within the interconnection network. Difference 1 may be absorbed into the bsp model by allowing to set the processor speed as an applicationspecific value. The difference 2 could also be concealed using random placement of processes to processors and random message routing. It involves diverting all messages via arbitrary intermediate locations but it will removes any possibility of performance improvements using local communications. To avoid those inefficient solutions, the authors propose an extension of the bsp model which allows two levels of synchronisation: the bsp2 model.

A bsp2 computer consists of a number of bsp units interconnected by a communication network. Each bsp unit consists of a fixed number of processor/memory pairs, connected by a local network. Each processor has a fast access to its local memory and a uniformly slow access to other memories, through the network. Each bsp unit has, also, a uniformly slower access to the memory within other bsp units. The execution of a bsp2 program resides in super-superstep, separated by global synchronisation. During each super-superstep, each bsp unit performs a complete bsp computation and/or communicates data with other bsp units.

The cost model of a bsp2 computer is modelled by seven parameters : (s, p, q, l, L, g, G). Where s is the processor speed, p the number of processors in each bsp unit, q the number of bsp units (total processor count is p × q), l represents the latency for a bsp unit, L is the latency for all bsp units, g is the bandwidth within a unit and G is the bandwidth between two units. Thus, the cost of a super-superstep is C l + L + n c G where C l is the max cost of bsp computation performed by any unit, n c the max number of values communicated by all the processors of a bsp unit to/from other units.

To conclude, the authors state that the bsp2 model does not show any significant benefit from its approach, even when local communication is several orders of magnitude faster than the remote communication. The bsp2 model fails to provide any major performance gain and encourages the usage of flat bsp on architectures with a good global communication network.

Dynamic BSP

The Dynamic bsp model [START_REF] Jeremy | Dynamic BSP: Towards a Flexible Approach to Parallel Computing over the Grid[END_REF] was born on the assumption that bsp was not adapted to grid computing. Dynamic bsp aims to adapt bsp to deal with heterogeneity issues introduced by grid computing. It is a significant extension of bspgrid [START_REF] Vasil | Bspgrid: Variable Resources Parallel Computation and Multiprogrammed Parallelism[END_REF], a grid-based model for parallel algorithms. The Dynamic bsp is a flexible model allowing to spawn, when it is required, additional processes (virtual bsp processes) within a superstep. It allows to deal with heterogeneity issues as well as fault-tolerance. To do so, the model consists in: (1) a master processor (the task server), ( 2) worker processes and (3) a data server. At each superstep, a set of virtual processors is executed on the physical processors available on the architecture. The master processor is responsible for task scheduling and memory management. Each virtual process has the responsibility to retrieve local data from the data server, perform the required computations, write back the modified data, and then inform the master processor that it has finished the task. The master processor maintains a queue of pending virtual processors and dynamically assigns them to waiting physical processors. As the virtual processes need to access data, the data server must be implemented over a global shared memory. The principal barrier to scalability of Dynamic bsp is related to the bottleneck induced by such a global memory.

A cost model is also available in the Dynamic bsp model. However, as the g (bandwidth) and l (latency) parameters may vary significantly between grid nodes, the cost model difficult to handle.

Multi-bab and Multi-dac

In [START_REF] Savadi | A Bridging Model for Branch-and-Bound Algorithms on Multi-Core Architectures[END_REF] and [START_REF] Savadi | Multi-DaC Programming Model: A Variant of Multi-BSP Model for Divide-and-Conquer Algorithms[END_REF] the authors proposes two approaches that are, respectively, dedicated to branch-and-bound (bab) and divide-and-conquer (dac) algorithms. These approaches aim to program architectures that are based on a tree hierarchy of memories.

The multi-bab model [START_REF] Savadi | A Bridging Model for Branch-and-Bound Algorithms on Multi-Core Architectures[END_REF] maps branch-and-bound algorithms on a cache hierarchy of multi-cores. The model provides a cost model to predict the upper bounds for communication and synchronisation.

The multi-dac model [START_REF] Savadi | Multi-DaC Programming Model: A Variant of Multi-BSP Model for Divide-and-Conquer Algorithms[END_REF] deals with divide-and-conquer algorithms. It also proposes a cost model dedicated to this particular type of algorithms.

Currently, those models are not implemented over a programming language.

MBSP

The mbsp model [START_REF] Gerbessiotis | Extending the BSP Model for Multi-Core and out-of-Core Computing: MBSP[END_REF], or multi-memory bsp, is an extension of bsp that abstracts and models parallelism in the presence of multiple memory hierarchies and multiple cores. The aim of mbsp is to retain the properties of bsp and, in addition, to abstract not only traditional external memory-supported parallelism (that uses another level of slower memory) but also multi-level cache-based memory hierarchies such as those present in multi-core systems. To do so, the model is parameterised by the septuplet (p, l, g, m, L, G, M ). In addition to the bsp parameters (p, l, g), the collection of core/processor components has m alternative memory units distributed in an arbitrary way. The size of the fast memory is composed by M words. It is important to note that the model makes no assumption whether an alternative memory unit resides on every processor/core or if a unit is controlled by one of the p processors or not. Whatever the case is, the cost of memory unit-related accesses is modeled by the pair (L, G). L and G are, respectively, similar to the bsp parameters l and g. That is to say, G expresses the unit transfer bandwidth and L a denominate the memory latency time.

With the mbsp model, the cost of a superstep is max(l, L, w) + gh + Gb, where w is the maximum computational work of any process, h is the maximum size of sent or received messages by any processor in a superstep and b is the maximum amount of information read or written into an alternative memory-unit.

The mbsp model could be viewed as a two-level version of a multi-bsp instance. However, the model is not able to finely handle several levels of memory, latencies or bandwidths.

High-level parallel programming

Today, popular parallel and distributed programming methodologies are dominated by lowlevel techniques. High-level approaches offer many possible advantages by abstracting implantation details. Such structured approaches have a key role in scalable, portable and ubiquitous parallelism. We now present the main paradigms and try to list the pros and cons compared to our approach.

Algorithmic skeletons

Proposed in [START_REF] Cole | Algorithmic Skeletons: Structured Management of Parallel Computation[END_REF], algorithmic skeletons are high-level parallel patterns that are build to solve a particular problem. They aim to be seen, by the programmer, as high order functions that are part of a collection of algorithmic tools from which a problem specific program must be fashioned. Meanwhile, the system implementor sees each skeleton as a generic computational pattern for which an efficient parallel code must be defined. An algorithmic skeleton must hide all the implementation and optimisation details in order to provide a simple way to use it.

Even if many parameters may be required, using an algorithmic skeleton allows to avoid low-level considerations, such as data decomposition, computation scheduling and communications. They also aim to be reusable in various domains and applications while being highly optimised for parallel computations. We can easily distinguish general purpose skeletons, that appear in a wide range of applications from different domains, and domain specific skeletons, that are specialised to particular domains. The aim of this section is to present the most common skeletons (we do not intend to be exhaustive) and some of the well-known libraries.

We can classify skeletons into four principal categories.

1. First, we have task parallel skeletons. Here, the parallelism is expressed by different "tasks" that are available as an input stream. All the tasks are distributed and scheduled to the different workers. The most common task parallel skeletons are the following:

• Farm (Figure 2.2a) is a basic task distribution that is done on all the available computing units. When the tasks are completely independent, we talk about pure farms.

• Pipeline (Figure 2.2b) is an algorithmic skeleton that splits a complex computation process into many simple steps. Each step is attributed to a stage S, that is a computation unit. The result of the stage S i is the input of the stage S i+1 . 2. Then, we have data parallel skeletons that compute a single task, in parallel, on distributed pieces of data. Those skeletons are often adapted to various data structures [START_REF] Karasawa | A Parallel Skeleton Library for Multi-Core Clusters[END_REF] such as lists, arrays, matrices, trees, etc. The principal skeletons are:

• Map is a well known pattern that consists in applying a function f on a distributed collection of data x 1 , …, x n . The resulting data collection y 1 , …, y n is obtained by

y i = f (x i ).
• Reduce is also a common operator that "sums up" all the items of a data collection x 0 , …, x n given a commutative operator.

Figure 2.3 shows how the map-reduce skeleton works. From a distributed set of data, we apply an operator op on every value. Then, a resulting value is generated using a reduce function. The map-reduce concept is well known for a long time, especially in functional programming. The map-reduce skeleton should not be confused with the Map-reduce frameworks proposed by google [START_REF] Dean | MapReduce: Simplified Data Processing on Large Clusters[END_REF], as explained in the following section. 3. The control (service) parallel skeletons are used to control a flow of data using parallel operators. For example:

• Divide & Conquer (see Figure 2.4) which starts by dividing tasks into sub-tasks recursively until the sub-tasks are easily solved (conquer phase). Then, the partial results are recursively combined to compute the final result.

• While is a simple skeleton that represents a conditional iteration where a given skeleton is executed until a condition is valid.

4. Finally, specialised parallel skeletons are dedicated to a specific task and are mainly used in particular domains. For example:

• Stencil maps a function computing

y i = f (x i , N (x i ))
for some neighbourhood function N , as shown in Figure 2.5. 9 2 8 7 7 5 6 6 5 7 4 8 3 9 2 1

× 2 1 1 1 0 -1 -1 -1 -2 = 7 -9 10 8
Figure 2.5: The stencil skeleton.

• Wavefront [4] is a pattern close to the stencil where each element of a grid computes a value that depends on the computation of a set of previous elements. The computation typically flows from one region to another.

• Orbit-skeleton [START_REF] Janjic | Space Exploration Using Parallel Orbits: A Study in Parallel Symbolic Computing[END_REF] is a pattern close to the state-space. The finite state-space construction problem is computing the explicit graph representation (also known as Kripke structure) of a given model from the implicit one. This graph is constructed by exploring all the states reachable through a successor function succ (which returns a set of states) from an initial state s0 . Generally, during this operation, all the explored states must be kept in memory in order to avoid multiple explorations of a same state.

There are many algorithmic skeleton frameworks and libraries. They mainly differ in their skeleton set and their implementation language such as c, c++, java, etc. A nice classification could be found on the dedicated wikipedia page: [[ 10 ]]. It is important to note that most of the skeleton libraries do not provide a way of nesting computations; that is to say nesting of algorithmic skeletons. Furthermore, the type safety is not always available either.

One can say that algorithmic skeletons are an easier way of programming parallel algorithms. Nevertheless, as algorithmic skeletons are derived from a set of parallel patterns [START_REF] Cole | Bringing Skeletons out of the Closet: A Pragmatic Manifesto for Skeletal Parallel Programming[END_REF], it is not always possible to find patterns that meets our needs. In this case, it is possible to extend the pattern set, but it requires a lot of work for the library implementer and complicates the user side by adding new skeletons. Otherwise, skeleton libraries mostly provide a special skeleton in which the programmer is free to use whats he wants (mostly c,c++, java and mpi) with the drawback of adding unsafe features. It also requires a deep knowledge of how the skeleton work to be able to interact with them.

We now introduce some skeleton frameworks and libraries that are common to the domain. We also describe them and compare their capabilities.

SKLML

The sklml [[ 11 ]], formerly called ocamlp3l, is a functional parallel skeleton compiler and programming system for ocaml programs. This framework embeds an innovative compositional skeleton algebra into the ocaml language. Thanks to its skeleton algebra, sklml provides two ways of program evaluation: (1) a regular sequential evaluation (merely used for prototyping and debugging) and (2) a parallel evaluation obtained via a recompilation of the same source program in parallel mode. sklml was specifically designed to show that the sequential and parallel evaluation regimes coincide. To do so, the framework proposes several skeletons such as farm, pipeline (denoted ||| ), product (denoted *** ), sum (denoted +++ ), etc. The sklml allows to write elegant functional codes. For example, to compute f (x) = g(x) + h(x) with the *** skeleton, we can write the following code [[ 12 ]]:

let split_skl = skl () -> fun x -> (x, x) in let merge_skl = skl () -> fun (x, y) -> x + y in let f_skl = (split_skl ()) ||| (g_skl *** h_skl) ||| (merge_skl ()) in
The skl () syntactic construction allows the programmer to transfer a regular ocaml function into the universe of skeletons. Then, g_skl and h_skl stands for the computational functions which are used with the *** skeleton, surrounded by the split_skl and merge_skl phases.

D3 in D2 in D1 in D0 in Input data W 3map W 2map W 1map W 0map Map phase D2 int D1 int D0 int Intermediate data W 2red W 1red W 0red
Reduce phase

D2 out D1 out D0 out Output data

The MapReduce framework

The MapReduce parallel computation framework was introduced in [START_REF] Dean | MapReduce: Simplified Data Processing on Large Clusters[END_REF] and widely inspired by the bsp model and algorithmic skeletons. The authors describe a programming model with an associated implementation for processing and generating large datasets (the famous BigData) that is amenable to a broad variety of real-world tasks. The user must specify a map and a reduce function first. Then, the runtime system will automatically run parallel computation across the available computation units: the workers. To do so, the computation takes a set of input (key/value pairs), and produces a set of output (key/value pairs). The map function takes an input pair and produces a set of intermediate key/value pairs. This set of intermediate values, identified with the key I, is then given to the reduce function. As expected, the reduce function takes an intermediate key I and a set of values corresponding to that key. It merges these values together to form a possibly smaller set of values. Figure 2.6 summarises the behaviour of the MapReduce framework, where square nodes stand for data, circles for the workers and the communications (merges) are symbolised by directed arrows. Many implementation of this framework are available, such as the hadoop framework [START_REF] White | Hadoop: The Definitive Guide. 1st[END_REF].

For example, we consider the problem of counting the number of occurrences of each word in a large collection of documents. The pseudo-code of the map and reduce function that the user would write is the following code [START_REF] Dean | MapReduce: Simplified Data Processing on Large Clusters[END_REF] (Figure 2.7):

The map function emits, for each word, a pair identified by the word and a count of one. The reduce function sums all counts emitted for each words. At runtime, all the elements produced by the map function are given to the reduce function in order to compute the final list of counts. The MapReduce framework is easy to use, even for programmers who are not familiar with parallel and distributed programming. A large variety of problems are easily expressible as MapReduce computations; an in-depth comparison with bsp is available in [START_REF] Felice | BSP vs MapReduce[END_REF]. However, there is no way to exploit data locality in order to optimise computations. The performance mostly relies on the implementation of the framework and the communication scheme is limited. The bsp cost and scalability analysis for the MapReduce framework is proposed in [START_REF] Senger | BSP Cost and Scalability Analysis for MapReduce Operations[END_REF]. parallelism, targeting heterogeneous platforms composed of clusters of shared-memory platforms, possibly equipped with computing accelerators. The main design philosophy of fastflow is to provide applications to designers with key features for parallel programming (e.g. time-tomarket, portability, efficiency and performance portability) via suitable parallel programming abstractions and a carefully designed run-time support. The fastflow run-time support uses several techniques to efficiently support fine grain parallelism (and very high frequency streaming), such as non-blocking multi-threading with lock-less synchronisations; Zero-copy network messaging; Asynchronous data feeding for accelerator offloading.

For example, the code [[ 13 ]] in Figure 2.8 shows how to construct a 3-level pipeline using fastflow.

The firstStage produces the data stream. The secondStage is the task that processes the data stream. Then, the thirdStage finalises the pipeline computation. Finally, we call the ff_Pipe skeleton with the three stages to execute the pipeline. The pipeline is created and evaluated within the fastflow runtime, transparently.

Muesli

The Muenster Skeleton Library [START_REF] Ciechanowicz | Enhancing Muesli's Data Parallel Skeletons for Multi-Core Computer Architectures[END_REF][START_REF] Ciechanowicz | The Münster Skeleton Library Muesli -A Comprehensive Overview[END_REF], called muesli, is a c++ template library enabling fluent programming of multi-node, multi-core cluster computers by implementing the concept of algorithmic skeletons. muesli supports most of the task and data parallel skeletons. It proposes high scalability across nodes and cores by simultaneously using mpi and openmp. The user can also build complex skeletal topologies by arbitrary nesting both task and data parallel skeletons.

For example, the code [[ 14 ]] of Figure 2.9 shows how to construct a 3-level pipeline using the muesli library.

As expected, init produces the input data of the pipeline. The compute function is the computation stage of the pipeline and fin is the finalisation function that compute the resulting value. To execute the pipeline, we must construct the Pipe skeleton with the previously defined stages and start the execution using pipe.start() .

CHAPTER 2. STATE OF THE ART

HIGH-LEVEL PARALLEL PROGRAMMING

CHAPTER 2. STATE OF THE ART

Skepu

skepu [START_REF] Enmyren | SkePU: A Multi-Backend Skeleton Programming Library for Multi-GPU Systems[END_REF] is a skeleton programming framework for multi-core cpus and multi-gpu systems. It is a c++ template library with six data-parallel and one task-parallel skeleton, two generic container types, and support for execution on multi-gpu systems both with cuda and opencl. It also supports hybrid execution, dynamic scheduling and data load balancing thanks to the starpu runtime system. It also provides a modern c++ syntax and an efficient implementation.

The following code[ The map sequence is defined by the add function while the reduce phase is defined by the mult function. The skeleton MapReduce is instantiated using the map and reduce functions. Finally, the skeleton istance is used as a regular function. The evaluation of the skeletons is transparently handled by the skepu runtime.

Quaff

quaff [START_REF] Falcou | Quaff: Efficient C++ Design for Parallel Skeletons[END_REF] is a skeleton-based parallel programming library. Its main originality is to rely on c++ template meta-programming techniques to achieve high efficiency. It performs most of the skeleton instantiation and optimisation at compile-time, and keeps the overhead traditionally associated with object-oriented implementation of skeleton-based parallel programming libraries very small. quaff provides an implementation of the main skeletons, such as pipeline, farm and pardo, on both shared and distributed memories. Apart from these parallel skeletons, it also provides constructs allowing tasks or skeletons to be composed sequentially or conditionally in applications. Furthermore, skeleton nesting is naturally supported in quaff. Indeed, once instantiated, all skeletons are valid functors and consequently can be used as arguments to other skeleton templates. For example, the following code [START_REF] Falcou | Quaff: Efficient C++ Design for Parallel Skeletons[END_REF] In this example, we intend to read an image, apply a Sobel filter and write the output image. The program is defined as a pipeline. The second stage is a farm. The 3 first definitions contain the specific sequential tasks. The farm skeleton is defined by giving the tasks to be computed by workers and the amount of workers. The pipeline uses the farm as its second stage to, respectively, read, compute and write the resulting image.

The parallel libraries which we just introduced aim to make parallel programming easier. To do so, they propose skeletons that have been designed to execute specific parallel patterns. The programmer does not have to worry about optimisations, everything is done within the skeleton. However, some algorithms are too complex to be modeled by a simple skeleton. In such cases, some programmers want to be able to write a specific optimised code that is structured enough, thanks to an expressive parallel programming model.

BSP programming

The Message Passing Interface

The Message Passing Interface (mpi for short) library [START_REF] Snir | MPI-The Complete Reference. 2nd[END_REF] is a standard api for communication in distributed-memory parallel applications. There are numerous implementations of this api, both commercial and open source (e.g. open-mpi, mpich, etc.). mpi aims to provide a tool for high-performance, scalability and portability. mpi is the de facto standard for highperformance computing today. It is the main library that is used for implementing parallel languages. The mpi standard defines the informal semantics of routines that are useful to a wide range of users writing both shared memory and distributed parallel programs in fortran and c. However, there are many stubs codes for languages such as java, ocaml, etc.

As mpi contains a huge number of routines, it is not possible to present an exhaustive list. We thus only present the routines that are close to bsp such as collective operations and operations for the communicators manipulation. That is to say subgroup creation and, thus, subgroup synchronisation. We also present some of the primitives that are used in the multi-ml implementation, see Chapter 5.

After the initialisation of mpi it is possible to access the initial communicator MPI_COMM_WORLD which contains all the processes participating to the computation. A collective communication is defined as communication that involves a group of processes, i.e. a communicator.

All processes in a group identified by a communicator must call the same collective routine. We do not give explanation of all the arguments but resume the most important ones for our purpose. The details concerning several mpi routines and its arguments could be found in Table 2.1. In many cases, collective communication can occur "in place" for communicators, with the output buffer being identical to the input buffer. This is specified by providing a special argument value, MPI_IN_PLACE , instead of the send buffer or the receive buffer argument, depending on the performed operation.

In mpi, using point-to-point communications is a usual practice. To do so, the routine MPI_Send can be used to send data to another process. To receive the data, the routine MPI_Recv must catch the sent value. It is important to note that those two routines perform a blocking send/receive. mpi also provides a non blocking send: MPI_Isend , a synchronous one: MPI_Ssend and many other optimised alternatives.

The collective operations are for broadcasting, scattering, gathering and all-to-all exchanges. The main arguments are the buffer to send (respectively, to receive the data) with the number of data elements and their types ( MPI_INT , MPI_DOUBLE , etc.) to ensure portable programs. As many c routines, they return an integer that indicates if the operation succeeded. The mpi primitives are often annotated with a suffix to specify its behaviour. For example, MPI_Alltoall is the all-to-all communication primitive. It sends a data of a fixed size to all concerned processes. MPI_Alltoallv stands for a all-to-all communication primitive that can send data with various sizes. Finally, MPI_Alltoallw is the most general form of all-to-all, it allows the communication of various types of data of different sizes. For example, by making all processes have recvcounts[i] = 0 , the primitive achieves a MPI_scatterw , which perform a data distribution on processes. This is a powerful routine but it is hard to use. The sendrecv routine involves only two processes which exchange data in a synchronous way as a barrier. In fact, this is a subgroup of only two processors that are communicating synchronously. Collective operations are not really bsp communications since they are not synchronous. For example, using a broadcast, if the emitter changes the buffer to send after the call of the routine, the value received by other processors is unspecified. A call to a MPI_barrier() can be used to avoid such an undesirable behaviour. In practice, most mpi libraries implement collective operations in a synchronous way but this is not standard.

There are many routines for managing communicators (we count 35 in mpi-2). However, the main routine is MPI_comm_split which creates a new communicator by partitioning the group into disjoint sub-groups using a set of colours. Each subgroup contains all processes with the same colour. Within each sub-group, the processes are ranked in the order defined by the value of an argument key, with ties broken according to their rank in the old group. This is a collective call, but each process is permitted to provide different values for colour and key. This mechanism is very powerful for dividing a single communicating group of processes into k sub-groups and many routines, such as MPI_comm_create , are similar to a specific call to MPI_comm_split . Unlike the pub (see thereafter) function, this mpi routine requires communications between the processors of the group. In fact, exchanging the colours can be seen as a call to the MPI_Allgather collective operation. Communicators in mpi are really flexible but they are complex. Unlike in pub, there is almost no restriction to the way groups are formed. Processes can be in more than one group, as they have an unique rank within each group. A traditional example of use of such communicators would be in linear algebra, where it is often useful to split processors by rows and by columns.

mpi also provides routines for asynchronous drma accesses. Thus a programmer can manage a window. It is a synchronous operation that every processor performs. The, processors can write or read on remote processors using that window. There is a specific barrier, the fence , for processors that have been registered to the window. Using the Win_complete , we can force a processor to wait until all remote readings and writings have been performed on a window. Furthermore, mpi obviously provides asynchronous sends/receives of data. Such primitives are intentionally ignored in this thesis.

mpi can be seen as the assembly of the communication libraries. Most programming languages using communication are based on mpi, or could rely on it. mpi is very powerful, however, it is quite complicated to handle and may lead to complex code and deadlocks.

The BSPlib

The bsplib [START_REF] Jonathan | BSPlib: The BSP Programming Library[END_REF] is a c-library of communication routines. It aims to support the development of parallel algorithms based on the bsp model. It offers routines for both message passing (bsmp) and remote memory access (drma). The library offer two implementations over tcp/ip and mpi. Its implementation aims at distributed-memory super-computing and thus explicitly starts separate processes on supercomputer nodes. The main routines of the bsplib are (1) Core;

(2) Synchronisation; (3) Message passing (4) and drma routines. It is interesting to note that about 20 routines are available, compared to the more than 200 that composed mpi. The bsplib is thus easier to understand and to use, without lack of expressivity for programming hpc architectures. As the bsplib can be seen as an extension of the pub, the main primitives are jointly detailed in Table 2.2. We now introduce the main routines: Core Routines. As in standard mpi, we first need to initialise our parallel computation, using the function bsp_init . Then, we can have different bsp computations, each beginning with bsp_begin and terminating with bsp_end . Within a bsp computation, we can query some information about the machine: bsp_nprocs returns the number of processors p and bsp_pid returns the processor identifier which belongs to 0, ..., p -1.

Synchronisation.

According to the bsp model, all messages are received during the synchronisation barrier and cannot be read before. The barrier is done using bsp_sync which blocks the node until all other nodes have called bsp_sync and all messages sent to it, in the current superstep, have been received.

Message Passing.

With bsmp, a non-blocking send operation is provided to deliver (a packet of) messages to a system buffer associated with the destination process. It guarantees that the message will be available in the destination buffer at the beginning of the subsequent superstep. Furthermore, the message can be accessed by the destination process only during that superstep. If the message is not accessed within the superstep it is removed from the buffer. With the bsplib, the destination buffer of a processor may be viewed as a queue where incoming messages are enqueued in an arbitrary order.

Sending a packet (in a buffering mode) is done using bsp_send . The routine is based on the idea of a two-part message. A fixed-length part carries tag information that will help the receiver to interpret the message; a variable-length part contains the main data payload. The length of the tag is required to be fixed during any particular superstep, but can vary between supersteps.

Choosing a tag is done using bsp_set_tagsize where the user specifies the size of the fixedlength portion of every message in the subsequent supersteps. It allows the user to set the tag size to enable the use of tags that are appropriate for the communication requirements of each superstep. This is particularly useful in the development of subroutines either in user programs or in libraries. The procedure must be called collectively by all processes. A change in tag size takes effect in the following superstep; it then becomes valid. The programmer can know the number of received messages as well as the total size of received data (in bytes) using the routine bsp_qsize . This routine works on the queue of received messages. To receive a message, the user should use the procedures bsp_get_tag and bsp_move . bsp_get_tag returns the tag of the first message in the queue and the size of the corresponding payload (status is -1 if the queue is empty). bsp_move copies the payload of the first message of the system queue, i.e. the buffer call payload, and removes it from the queue. Then, the system will advance to the next message. drma routines. Registering or deleting a variable from global access is done using: void bsp_push_reg(ident,size) and bsp_pop_reg(ident) (see Table 2.2). Due to the spmd structure of bsp programs, if p instances share the same name, they will not, in general, have the same physical address. To allow bsp programs to be executed correctly, the bsplib provides a mechanism for relating these various addresses by creating associations called registrations. A registration is created when each process calls void bsp_push_reg . If different variables have to be registered/unregistered, all processors must call the functions in the same order. Both the address and the extent of a local area of memory must be provided. The registration takes effect at the next synchronisation barrier and newer registrations replace older ones. This scheme does not impose a strict nesting of push-pop pairs. For example: The variable x on processor 0 would be associated with y on processor 1. Note that this example is clearly not a good way of programming in bsp. In the same manner, a registration association is destroyed when each process calls bsp_popregister and provides the address of its local area participating in that registration. A run-time error will be raised if these addresses (i.e. one address per process) do not refer to the same registration association. Un-registration takes effect at the next synchronisation barrier.

The two drma operations (Figure 2.12) are the following:

• bsp_get(int pid,const void *src, int offset,void *dst,int nbytes) stands for global reading access. It copies n bytes to the local memory address dst from the variable src at offset offset of the remote processor pid ;

• bsp_put(int pid,const void *src,void *dst,int offset,int nbytes) stands for global writing access. It copies n bytes from local memory src to dst at offset offset on remote processor pid .

It is important to note that the get and put operations are executed during the synchronisation step and all get are served before a put overwrites a value. It is interesting to note that the bsplib also provides high-performance variants of the drma and bsmp primitives. These are bsp_hpput , bsp_hpget , and bsp_hpsend , and allow for communication to occur somewhere between the call to the hp-primitive and the end of the next bsp_sync . The usage of such an hp-primitive is twofold. Firstly, hp-primitives allow to overlap the computation and communication whenever it is possible. Secondly, it avoids the inefficiency of buffering communication. When those primitives are used, the user must guarantee that the source and destination memory areas remain unchanged until the end of the current superstep. Non-deterministic behaviour can easily be introduced using such hp-primitives.

The Paderborn University BSP library

The Paderborn University bsp (pub) library [14] is a c communication library based on the bsp model. The pub's routines are close to the bsplib ones. The main differences remains in the availability of subgroup synchronisation features. The way that values are stored with concurrent access is also particular using the pub. The main pub routines are available in Table 2.2. In the following, we describe the main differences between the bsplib and the pub. The drma routines are comparable except their subgroup capabilities.

Message Passing. Sending a single message can be done using bsp_send or bsp_sendmsg . After calling one of these routines the buffer (or the message) may be overwritten or freed. The bsp_send routine is recommended with a code with a small amount of communication during a superstep. On the contrary, with a lot of data with separated memory locations, it is better to use the bsp_sendmsg routine. As for drma operations, these routines work in the scope of a subgroup.

Subgroup Primitives. The main functions involved in the sub-grouping process are bsp_dup , bsp_partition and bsp_done . The bsp_dup function creates a subgroup that is a duplicate of the current bsp computer. This is useful to organise algorithms in a compositional manner, since a code working on the duplicate will not be affected by previous messages that were already in the queue, waiting to be sent. The synchronisation on the duplicate subgroup will only complete the communications that were requested with that subgroup. It is thus possible to define a function for a sub-algorithm, starting with a call to bsp_dup so that it can be called from anywhere in a parallel program, without interfering.

The bsp_partition function is the way to create proper subgroups. The subgroups are described as a partition of the current bsp computer in contiguous subsets. The starting indexes of these subsets are given as argument in the partition integer array. To be a correct partition, the array has to be sorted. Finally, the bsp_done function indicates that we have finished the work with the subgroup. In the pub library, it is impossible to synchronise or create other subgroups from the parent object until the current subgroup is released with the routine bsp_done . However, it is possible to create different new subgroups of the current subgroup. The organisation of subgroups is similar to a stack, with only the lowest subgroup being allowed to create new subgroups. Here is an example of a program, in the pub library, with subgroup synchronisation: Communicators are thus present in both mpi and the pub library as group which are called bsp objects. Although they play a similar role, there are notable differences between both. The pub library allows to partition the set of processors into pairwise disjoint subsets. Each subset acts like an independent bsp computer. Subgroups are created by a call to the bsp_partition function. A call to the function creates a new t_bsp object, that can be used in any bsp call. It is also still possible to use the parent bsp object. A call to bsp_partition does not require communications between the processors, since each processor has to explicitly give the whole partition table. However, there are limitations. It is not possible to create other subgroups of the parent bsp object, until the current subgroup subbsp has been disposed of using the bsp_done function. The subgroups work like a stack: it is only possible to create subgroups of the lowest subgroup. Another limitation comes from the way the subgroups are defined. The partition has to be made of subgroups of consecutive processors. It is not possible to combine the processors 1 and 3 in a group, and 2 and 4 in another one, for a parallel machine of four computers.

t_bsp subbsp; int part[2]; part[0] = 2;
The pub and the bsplib can introduce synchronisation problems such as deadlocks. For example, the code if pid = 0 then sync() else {} makes the code unsafe. Incomplete synchronisation of groups of processes also leads to unwanted behaviours. It is thus necessary to use sophisticated tools [START_REF] Fortin | BSP-Why: A Tool for Deductive Verification of BSP Programs[END_REF] to avoid such problems.

MulticoreBSP

The Multicorebsp for c programming interface is directly derived from the bsplib. It adds two new high-performance primitives and updates the interface of existing ones. Whereas bsplib focuses on distributed architectures, the Multicorebsp targets shared-memory computing specifically and thus employs thread-based parallelism. As expected, Multicorebsp provides a set of functions that are used to control the spmd sections in bsp programs. The bsplib like drma and bsmp functions are available, as well. Furthermore, Multicorebsp also provides the high-performance variants of the drma and bsmp primitives. In addition to the existing hpprimitives, Multicorebsp introduced the bsp_direct_get primitive. Its semantics is exactly that of the bsp_hpget , except that it guarantees that communication is done after the call to the primitive has finished. However, the user must also ensure that the data remains unchanged during the current superstep.

Like the pub library, Multicorebsp supports hierarchical execution of bsp programs. This means that bsp processes may call the bsp_init and bsp_begin within a spmd section. When a bsp process does so, it is considered as the initialising process for the upcoming nested bsp run. It is ruled by the same rules as the original bsp run. It is important to note that, after the spawn of a nested bsp run, the processes have no knowledge of the bsp processes that spawned them and previous variable registrations are no longer valid. It is also the case in bsml as there is no free variable inside the scope of parallel vectors ( << e >> ).

The last version of Multicorebsp improved support for nested bsp runs, which benefits code explicitly following the multi-bsp model. The full support of the multi-bsp model is the next logical step of the evolution of Multicorebsp.

The Multicorebsp programming model assumes a down-top execution. Such an evaluation model is less secure than a top-down approach as the synchronisations are not implicitly managed and must be handled by the programmer. The new multi-bsp (synchronous) primitive of Multicorebsp are the following: bsp_up() is used to move upward in the multi-bsp tree; bsp_down() to move downward in the multi-bsp tree; bsp_sync() is used to explicitly synchronise. To inspect the multi-bsp tree information, Multicorebsp proposes bsp_lid() to get the leaf identifier of this SPMD program; bsp_leaf() to know if the SPMD program runs on a leaf; bsp_sleaves() returns the number of leaf nodes in this subtree; bsp_nleaves() gives the total number of leaf nodes in the full tree.

To illustrate the way of programming Multicorebsp algorithms, we propose the (pseudo-) code [[ 16 ]] Figure 2.13 that broadcast a value.

MigBSP

The migbsp framework [START_REF] R. D R. Righi | MigBSP: A Novel Migration Model for Bulk-Synchronous Parallel Processes Rescheduling[END_REF] aims to control processes rescheduling in bsp applications. The main idea of migbsp is to adjust the process location in order to reduce the super steps times, while waiting for the slowest process. To do so, migbsp allows to create more bsp processes than the total amount of bsp processors. migbsp manages load balancing issues, where the load grain is represented by a bsp process. The model considers memory migration costs in order to evaluate the process transferring viability. The migration candidates are evaluated thanks to the non-trivial computation of a scalar metric P M (Potential Migration). The architecture is assembled with Sets: different sites. Set Managers are responsible for scheduling, capturing data from a specific Set and exchanging it among other managers. migbsp considers data from both processes and resources to generate the scalar metric P M . Its evaluation is available on both regular (same amount of data and same computation pattern for each bsp process) an irregular (potentially different amounts of data and computation patterns) applications. The model applies adaptation on rescheduling launching in order to reduce its impact on application runtime.

<< f $v$ >> f(v)

BSP++

bsp++ [START_REF] Hamidouche | A Framework for an Automatic Hybrid MPI+OpenMP Code Generation[END_REF] is a meta-programming library that aims to benefit from the different levels of parallelism of parallel architectures. It provides a high level interface that is simple to apprehend. To guarantee simplicity and abstraction, the bsp++ library is based on the bsml interface and uses almost identical primitives, as shown in Table 2.3 . Furthermore, it natively handles several widespread high-performance architectures. To take advantage of hierarchical architectures, and to avoid global synchronisation of a machine with a large amount of processes, bsp++ is based on a hierarchical model. Likewise dbsp, it is possible to decompose the machine into sub-machines. Furthermore, bsp++ exploits the localities of the machine. Thus, it is possible to benefit from shared memory available in cores. The bsp++ implementation uses openmp to communicate inside a shared memory and mpi for inter-nodes communications. For example, the code [START_REF] Hamidouche | Three High Performance Architectures in the Parallel APMC Boat[END_REF] of Figure 2.14 gives the implementation of the inner product using bsp++.

The computation of the inner product starts at line 5. As the data is distributed using a parallel vector, the inner product is computed on the local values of each processors. Then, the bsml-like proj primitive is used to gather the local results. Finally, in line 11, the inner product is finalised.

The bsp++ programming library is highly optimised and has been designed to obtain the maximum performance on very large clusters counting hundreds of cores. In [START_REF] Hamidouche | Parallel Smith-Waterman Comparison on Multicore and Manycore Computing Platforms with BSP++[END_REF] and [START_REF] Hamidouche | Parallel Biological Sequence Comparison on Heterogeneous High Performance Computing Platforms with BSP++[END_REF] the authors describe the implementation and benchmarks of some algorithms on very large clusters. However, there is no support of the multi-bsp model in bsp++. It is possible to nest parallel vectors on 2 levels, because of the implementation that uses mpi and openmp, but the library is not based on multi-level abilities. As there is no typing system, the nesting of parallel vectors which exceeds 2 levels leads to an error. Furthermore, there is no mechanism to ensure valid sub-synchronisations.

BSP Python

bsp python [START_REF] Hinsen | Using BSP and Python to Simplify Parallel Programming[END_REF] is a way to program bsp algorithms in python. It is based on the idea that it is advantageous to use interpreted high-level languages in scientific programming. To obtain the maximum efficiency, the time-critical code is therefore written using an optimised compiled language. bsp python is close to the bsml language on many points. A bsp python program has two levels: local (any one process) and global (all processes). As expected, local objects (or values) are standard python objects that exist in a single process. Global objects are available by the parallel machine, as a whole. The simplest way of creating a global value is using the ParConstant routine which represents a constant that is available on all processors. In the same way, ParData can be used to create a global value that are computed on every processor.

According to the bsp model, communication takes place at the end of a superstep. To do so, bsp python proposes a set of communication patterns: (1) put(pid_list) sends the local value to all processors in pid_list and returns a global object whose local value is a list of all the values received from other processors in an unspecified order; (2) fullExchange() sends the local value of each processor to all other processors and returns a global object whose local value is a list of all the received values in a unspecified order; (3) accumulate(operator,zero) performs an accumulation where the result is a global object where the local value is the reduction of the values from all processors with lower or equal identifier.

The Figure 2.15 shows how to sort a distributed list using the PSRS [START_REF] Shi | Parallel Sorting by Regular Sampling[END_REF] algorithm.

BSGP

bsgp [START_REF] Hou | BSGP: Bulk-Synchronous GPU Programming[END_REF] is a bsp programming language for gpu. It extends the c language with primitives for spawning threads and performing barriers. The code within barriers is automatically deduced as a superstep and is translated into a gpu kernel by the bsgp compiler. As bsgp variables are visible and shared across supersteps, data dependencies are implicitly defined. The main ambition of bsgp is, as for gpu, stream processing and especially streams of pixels for image processing. However the bsgp programming model does not directly match the gpu 's stream processing architecture, and the compiler must convert bsgp programs into efficient stream programs. First, the compiler automatically adds synchronisation barriers that are based on registers and thread usages. Secondly, due to the presence of shared variables across the supersteps, the compiler uses a dependency graph to automatically allocate and minimise temporary streams, used to store such values. bsgp also allows to create and destruct threads. It is also possible to manage load balancing and thread reassignment across the kernels. Remote variable access intrinsics for efficient communications between threads and collective primitives are also available.

The bsgp compiler generates a c code with cuda calls. A current limitation is the inability to handle control flow across barriers, such as loops or if statements.

Pregel

pregel [START_REF] Malewicz | Pregel: A System for Large-Scale Graph Processing[END_REF] is a powerful c++ (proprietary) parallel language dedicated to graph algorithms that are intended to be run efficiently on big clusters. A free version is available via the giraph project: [[ 17 ]]. The approach centers around computations on the vertices of the graph and the computation are expressed using specific bsp primitives. One can see pregel as a bsp specialisation for graphs. The implementation of pregel relies on low-level Google libraries with checkpoint at each barrier to enable fault tolerance. Each vertex of the graph has a unique "id", an associated value and a list of outgoing weighted edges. The computation is organised using a master/slave architecture and the input data is arbitrarily partitioned and stored on a distributed storage system. The pregel system maintains the vertices and edges stored on the node that will perform the computation. Nevertheless, the computation is migrated to where the data are stored.

On each superstep, each worker node invokes a procedure Compute() for each active vertex that is under its control. This procedure is responsible for the execution of the algorithm and is allowed, among other actions, to invoke other methods, compute new values for the vertex, add or remove vertices and edges, and send messages to other vertices. These messages are exchanged directly among the vertices, even if the vertices are being executed on different machines of the platform. The messages are sent asynchronously in order to allow the overlapping of computation and communication, but are delivered to the destination vertex only on the beginning of the next superstep. If a vertex declares that all its processing was done, it sends a message informing all the other nodes and deactivates itself. The master node stops the execution of the application after receiving this message from all the participants. An important feature required by graph algorithms is the ability to change the topology of the graph. For example, a clustering algorithm might replace each cluster with a single vertex, and a minimum spanning tree algorithm might keep only the tree edges. In order to avoid conflicting changes on the topology (for instance, multiple vertices could issue a command to create a vertex V with different initial values), pregel uses two deterministic mechanisms: (1) a partial ordering: removals are performed first, with edge removal before vertex removal and additions follow removals, with vertex addition before edge addition; (2) a handler for user-defined conflict-resolution policies. Any change on the topology is only performed on the next superstep, before the invocation of the Compute procedure.

pregel is considered as the main reference of graph applications on cloud and BigData processing on large clusters. It is used by Google for their own applications. Due to its proprietary nature, it is not possible to obtain the source code nor the full api. For example, a pregel implementation [START_REF] Malewicz | Pregel: A System for Large-Scale Graph Processing[END_REF] of Google's page-rank algorithm can be found in Figure 2. 16.

The example works as following: The graph is initialised so that in superstep 0, the value of each vertex is 1/NumVertices() . In each of the first supersteps, each vertex sends, along each outgoing edge, its page-rank proposition divided by the number of outgoing edges. Each vertex sums up the values arriving on messages into sum and sets its own page-rank proposition. Note that, a complete page-rank algorithm would run until a convergence was found.

NestStep

neststep [START_REF] Kessler | NestStep: Nested Parallelism and Virtual Shared Memory for the BSP Model[END_REF] is a parallel programming language for the bsp model. It extends the classical bsp model and supports dynamic nested parallelism with nested supersteps and introduces a hierarchical processor group concept. It also adds a virtual shared memory where the memory consistency is relaxed (but deterministic) during supersteps. It is also supports shared arrays. A prototype of neststep was designed as an extension of java [START_REF] Kessler | NestStep: Nested Parallelism and Virtual Shared Memory for the BSP Model[END_REF] and c [START_REF] Kessler | Managing Distributed Shared Arrays in a Bulk-Synchronous Parallel Programming Environment[END_REF] but it also possible to implement it over c++ or fortran. The neststep-c implementation consist in a source-to-source compiler but it does not support nested supersteps yet. The neststep-c run-time system is available through two versions: (1) Cluster-neststep-c using mpi for inter-processor communication [START_REF] Sohl | A Scalable Run-Time System for NestStep on Cluster Supercomputers[END_REF]; (2) cell-neststep-c running on the cell be [START_REF] Johansson | Bulk-Synchronous Parallel Computing on the CELL Processor[END_REF](which is a heterogeneous multi-core processor). It is interesting to note that blocklib is a skeleton programming library of data-parallel skeletons working on cell-neststep-c distributed arrays.

The neststep processes are organised in groups. The main method of a program is executed by all available processors of the partition of the parallel computer. A step statement represent both a step statement and a neststep statement. A step statement denotes supersteps that are executed by an entire group of processors in a bsp way. The step statements control the shared memory consistency within the current group. When a superstep is nested, we talk about neststep statements. A neststep statements splits the current group into disjoint subgroups. Each subgroup executes a statement, independently of the others, as a supersteps. The parent group has the ability to resume when all subgroups finish the execution of a neststep. neststep also offers a mechanism for load balancing the computation between groups. The main primitives of neststep are the following:

• step : executes a statement in parallel;

• neststep(k) : splits the current group (of size p) in k subgroups;

• neststep(k,weight) : here weight is a shared replicated array of k non-negative floats where the sum is equal to one. The k sub-groups created get a fraction of the available processors as close as possible as the fraction in weight[i] (at least 1).

• neststep(k,@=intexpr) : creates k sub-groups where intexpr is evaluated on each processors. The processor joins the group with the id intexpr (if it ranges [0...k -1], it skips the statement otherwise);

• seq : the group leader only is concerned by this execution. No synchronisation is implied;

The neststep syntax also introduces several symbols to manage parallelism: # is the number of processors in the current group; $ is the processor rank in the current group. @ is the id of the group the processor is part of.

To illustrate a neststep computation, the code [START_REF] Kessler | NestStep: Nested Parallelism and Virtual Shared Memory for the BSP Model[END_REF] in Figure 2.17 describes a way to compute a parallel prefix sum using the neststep-c implementation:

neststep gives the possibility to nest bsp computation in a hierarchical way. Nevertheless, this hierarchy is not directly related to the architecture of a given machine. Such a limitation does not allow to fully take advantage of the hierarchical characteristics of a machine.

grid-neststep [START_REF] Mattsson | Towards a Bulk-Synchronous Distributed Shared Memory Programming Environment for Grids[END_REF] proposes to adapt the bsp model with the neststep language to enable grid computing. To do so, a precompiler translates the neststep constructs in order to explicitly decompose the program into supersteps. Each supersteps is a virtual bsp computation that matches the grid environment. As supersteps are synchronised, all grid processors work on the same supersteps. Following the hierarchical structure of grids, the computation work of a superstep is split into workpackages of suitable sizes. The workload is managed by a scheduler which manages the grid environment. An implementation of this extension is available via the grid-modelica project [START_REF] Fritzson | Principles of Object-Oriented Modeling and Simulation with Modelica 2[END_REF].

The grid-neststep approach is closer to the architecture and is adapted to hpc computing. Nevertheless, as the load-balancing is managed by a scheduler, it is difficult to determine a cost for such programs. It is also difficult to design an algorithm based on the cost analysis, as those costs are hidden by the run-time implementation.

Other functional parallel languages

There exist a lot of parallel languages or parallel extensions of sequential languages. In this section, we try to point out functional parallel languages that were the most important in our view. Notice that there is a lack of comparisons between parallel languages. As efficiency, scalability, expressiveness, etc. must be taken into account, it is rather hard to compare them.

Two nice introductions (with many references) to parallel functional programming can be found in [START_REF] Hammond | Parallel Functional Programming: An Introduction[END_REF] and [START_REF]Research Directions in Parallel Functional Programming[END_REF]. They have been used as a basis for the following classification. The authors explain the three main reasons to use functional languages in parallel programming. First, they ease the partitioning of parallel programs (task or data decomposition). Second, most of them are deadlock free. As a value is computed independently of the evaluation order, a sequential run will deliver the same value when run in parallel. Third, they have a straightforward semantics that is similar in both sequential and parallel evaluation. This is very useful for testing and debugging.

Data-parallel Languages

To our knowledge, the first important data-parallel functional language was nesl [START_REF] Blelloch | NESL: A Nested Data-Parallel Language[END_REF]. This language allows to create specific arrays and nested computations within these arrays. The distribution of both the data and the computations over the available computing units is done at compile time. The code [[ 18 ]] in Figure 2.18 shows how the nesting of parallelism is expressed with nesl, on the quick sort algorithm.

The quick sort algorithm written in nesl is really easy to understand. Assuming that #a returns the size of the list a , the algorithm proceeds has follows. First it returns a if the list consists of one element. Otherwise, it determines the pivot , builds the list of elements lesser , equal and greater that contain the elements related to their names, compared to the pivot. Then it calls, in parallel, the algorithm on the lesser an greater elements. Finally, the result consist in the concatenation of sorted element that where computed in parallel. This example shows that the nested parallelism in nesl is not controlled as it is in multi-bsp. The parallel execution is not related to a specific memory or computing unit, everything is abstracted by the syntax. Two extensions of nesl for ml programming are nepal [START_REF] Manuel | Nepal -Nested Data Parallelism in Haskell[END_REF] and manticore [START_REF] Fluet | Manticore: A Heterogeneous Parallel Language[END_REF]; the latter is clearly a mix between nesl and Concurrent ml [START_REF] Fluet | Implicitly Threaded Parallelism in Manticore[END_REF]. Concurrent ml is a language of creation of asynchronous threads and send/receive messages in ml.

sac (Single Assignment c) [START_REF] Grelck | Classes and Objects as Basis for I/O in SAC[END_REF] is a lazy functional language (with a syntax close to c) for array processing. The language provides high-order operations on multi-dimensional arrays and the compiler is responsible for the generation of efficient parallel code.

An extension of the famous lazy functional language haskell is Data Parallel haskell [START_REF] Manuel | Data Parallel Haskell: A Status Report[END_REF]. It allows to create data arrays that are distributed across the processors. And some specific operations permit to manipulate them.

The main drawback of these languages is that cost analysis (for comparing algorithms) is hard to do since the system is responsible for the data distribution.

Data-flow and Coordination languages

Coordination languages [START_REF] Gelernter | Coordination Languages and Their Significance[END_REF] (and data-flow languages, which are similar to them) add primitives to a sequential language in order to describe how to coordinate processes over a parallel architecture. To do so, they mainly generate a data flow graph. caliban [START_REF] Paul | Functional Programming for Loosely-Coupled Multiprocessors[END_REF] is on of the first language that allows declarative description of a processes network and the mapping of processes to processors. The compiler determines statically code for all processes and their interconnections. caliban is an extension of a purely functional languages.

One of the most popular coordination languages is sisal (Streams and Iteration in a Single Assignment Language) [START_REF] Feo | A Report on the Sisal Language Project[END_REF]. It is close to most statement-driven languages, but variables should be assigned once. This allows the compiler to identify easily the inputs and outputs. The two main approaches are linda [START_REF] Nicholas J Carriero | The Linda Alternative to Message-Passing Systems[END_REF] and pcn (Program Composition Notation) [START_REF] Foster | Productive Parallel Programming: The PCN Approach[END_REF], which have been implemented in many programming languages. These languages are inserted as notations in sequential languages, such as c,c++, lisp or java, to generate and manage parallelism. A complex runtime system needs to adequately synchronise concurrent memory access of the processes.

Explicit process creation

In this category we have two extensions of haskell: eden [START_REF] Loogen | Parallel Functional Programming in Eden[END_REF] and gph (Glasgow Parallel haskell) [START_REF] Hammond | Glasgow Parallel Haskell (GpH)[END_REF][START_REF] Loidl | Comparing Parallel Functional Languages: Programming and Performance[END_REF]. They both extend haskell by adding a small set of syntactic constructs for explicit process creation. Those extensions allow fine-grain parallelism while providing enough control to implement efficient parallel algorithms. The usage of communications based on lazy shared data also frees the programmer from the tedious task of managing low-level communication details. Threads are automatically managed by a sophisticated runtime system for both shared or distributed memories. It is interesting to note that algorithmic skeletons are also proposed in both languages [START_REF] Berthold | Parallel FFT with Eden Skeletons[END_REF]. As explained previously, cost analysis of programs written with those languages is hard. Indeed, as the runtime handles data distribution [START_REF] Scaife | Empirical Parallel Performance Prediction from Semantics-Based Profiling[END_REF], it may introduce too much communication and, thus, a lack of scalability.

The hume language [START_REF] Hammond | The Dynamic Properties of Hume: A Functionally-Based Concurrent Language with Bounded Time and Space Behaviour[END_REF] is another distributed language that proposes a cost analysis for real-time purpose programs. Nevertheless, allow such an analysis the language expressiveness is arbitrarily limited.

Distributed functional languages

In addition to parallel functional languages, there are many concurrent extensions of functional languages such as erlang, concurrent clean or jocaml [START_REF] Mandel | Programming in JoCaml (Tool Demonstration)[END_REF]. jocaml is a concurrent extension of caml based on the join-calculus. It adds explicit generation of parallel processes and distribution directives across processes using "resources". Processes synchronisation is done using specific patterns called join-patterns.

The communication with futures is proposed in alice ml [START_REF] Rossberg | Typed Open Programming : A Higher-Order, Typed Approach to Dynamic Modularity and Distribution[END_REF]. A future is a placeholder for an undetermined result of a concurrent computation. When the computation delivers a result, the associated future is eliminated by globally replacing it by the result value. The concept of promises that explicitly handle futures is also available.

multimlton [START_REF] Sivaramakrishnan | MultiMLton: A Multicore-Aware Runtime for Standard ML[END_REF] is a multi-core aware runtime for standard ml, which is an extension of the mlton compiler. It effectively manages composable and asynchronous events using, in particular, safe-futures [START_REF] Ziarek | Isolating Determinism in Multi-Threaded Programs[END_REF]. It provides an analysis to automatically claim the resulting value of a future F by avoiding manual calls to the touch(F) operation, which blocks until the execution of F is complete.

scala is a functional extension of java which provides concurrency, using the actor model. oz-mozart [START_REF] Smolka | The Oz Programming Model[END_REF] is a multi-paradigm language (originally functional) with explicit creation of threads and communications are performed using "ports". It also provides data-flow computation over lazy lists, but the generated code is not very efficient.

Such languages are able to simulate or implement the multi-ml language. Nevertheless, they are not adapted to hpc computations.

Multi-ML in a Nutshell

The multi-ml language is based on the idea of executing a bsml-like code on every stage of the multi-bsp architecture, that is on every sub-machine. It is therefore possible to program multi-bsp algorithms while maintaining the bsml programming convenience. multi-ml consist in a lightweight specific syntax added to ml allowing to express multi-level parallelism. In this chapter, we talk about the philosophy of the multi-ml language and its syntax. Using several examples, we give some intuitions of its semantics, which will be developed in the next chapters.

This chapter is dedicated to a language that is an extension of ocaml. Thus, we will use some ocaml features such as pairs, operation on lists, labels, etc. The reader unfamiliar with ocaml can find the required information in [[ 19 ]] and [[ 20 ]].

The Multi-ML concepts

What Multi-ML is not

multi-ml aims to program hpc architectures, that is to say clusters of multi-cores with, potentially, both shared and distributed memories. This language does not intend to be generalist, with multi-paradigm programming features. multi-ml has been designed to program specific architecture that are hierarchically organised and it is intrinsically dedicated to it. The programming model does not assume that the code must be written for a specific architecture, it is rather disconnected from the runtime support. As multi-ml proposes a high level of abstraction, it is not possible to spawn or manage threads manually. All the parallelism is controlled by the language syntax, which is directly related to the multi-bsp model.

The execution levels

As multi-ml deals with multi-bsp architectures, we can distinguish several levels of execution. An expression is thus evaluated at a specific stage of the multi-bsp architecture, depending on the level of parallelism expressed. Those levels are useful to identify where the computations take place, but also where a data is stored. In multi-ml, an expression is always annotated with a locality representing where the data belongs to. With this locality information, the programmer can easily know if a value can be communicated to another level or if it is already available in another memory. This notion of locality is embedded in the multi-ml 66 CHAPTER 3. MULTI-ML IN A NUTSHELL type system to guarantee safe communications through the multi-bsp levels (see Chapter 4 for more details) and optimised memory managements. The memory localities of multi-ml are the following:

m: The multi, or global, memory of the multi-level architecture. At this level, all the nodes and leaves of the multi-bsp architecture are concerned by any execution. A value defined at level m is accessible "anywhere" on the machine. Some restrictions concerning parallel vectors will be discussed later on.

b: The bsp memory of a node. At this level, a bsml code can be executed in order to manage communications and parallel computations. It is possible to point toward the underlying memory via the parallel vectors. A value defined at level b is only accessible in the concerned node and can be communicated with the respect of certain limitations.

l: The local memory of a sub-node (inside a parallel vector). This is the underlying memory that can be managed inside a parallel vector from the upper node . This memory can contain pointers to the upper memory via a specific syntax explained below. It is important to note that the expressions of this level cannot be communicated because they are specific to a node and may have no sense anywhere else.

c: Like l, this memory is managed inside parallel vectors but it is free of references to the upper level. It is possible to transfer such communicable values through the multi-bsp levels.

s: The sequential memory of a leaf. It can contain specific values such as non-deterministic expressions. At this level, there is only sequential (traditional ml) code and the execution takes place in the lower level of the memory. A value defined at level s can be communicated under certain constraints.

It is important to note that the evaluation of a function that is specific to the locality s, such as random functions, is not possible inside a parallel vector. Like in bsml, the multi-ml implementation could be distributed. In this case, the values generated by a non-deterministic function on different components of the architecture may differ and then break the replicated coherency. Such a behaviour could introduce deadlocks, something we want to avoid. Of course, one can imagine a random operator that generates a random value that is coherent in a distributed implementation. An in-depth discussion on this topic is available in Chapter 5. Figure 3.1 shows how the different localities are mapped on the multi-bsp memory hierarchy. Such an architecture can correspond to multi-core with two physical threads per core. Inside the nodes (at bsp level), we can see the notation << >>. It denotes the scope of a parallel vector. As explained previously, this section represent the underlying memories of a node, which are divided in two categories: c and l (respectively communicable and purely local).

A short introduction to the syntax of Multi-ML types

We saw that the multi-ml language manipulates expressions that belongs to a specific locality. This locality information is attached to every expression of a multi-ml program. To keep track of those locality informations, we choose to extend the regular ocaml type system by adding locality annotations to types. Thus, the multi-ml types are annotated or tagged by their definition localities, representing the memory level where they belongs to.

This section is an overview of the multi-ml types. The typing system is widely discussed and detailed in Chapter 4.

For example, the execution of let x =1 in x at the multi-bsp level returns a value of type int_m . As the multi-bsp level of execution deals with the whole architecture, the value

x is computed and available in all the memories. Then, it is annotated with the locality m .

This type annotation is also attached to various type constructions such as pairs:

(int_m * string_m)_m meaning that the pair and its elements are defined at the multi level.

Function types are also annotated by their level of definition. Moreover, as functions can hide parallelism inside their definitions, we must add an extra annotation called a latent effect. A latent effect [START_REF] Talpin | The Type and Effect Discipline[END_REF] is a locality annotation that gives information on the nature 1 of the code that will be evaluated during the function application. This information is particularly useful to avoid malformed expressions, such as manipulating a parallel vector at the multi-bsp level.

For example, the evaluation of the code let f = fun () -> << 42 >> at the multi level will produce an expression typed (unit_`z -(b)-> int_c par_b)_m with [`z <|b] . The function f is thus defined at the multi-bsp level, it takes a value of type unit_`z (where `z refers to a polymorphic locality) and returns a parallel vector int_c par_b . The annotated arrow -(b)-> stands for the latent effect of the function, that is to say the locality where the function must be evaluated. In this example, as a parallel vector is build inside the function, its locality must refer to the bsp level. Finally, the with [`z <|b] element concerns the remaining constraints that must be respected. Here, we specify that the locality `z (of the first argument) must be valid (or acceptable) in locality b . The constraint operator ◁ checks that a locality makes sense in another locality. This notion of acceptability is widely discussed in Chapter 4.

CHAPTER 3. MULTI-ML IN A NUTSHELL

Note that the multi-ml type system allows the definition of bsp codes at the multi level. This is particularly interesting for the design and use of bsml libraries.

Localities are not always fixed. For example, the identity function cannot be typed with a fixed locality. Indeed, the code let id = fun x -> x in id << id 1 >> is perfectly valid. The application of the function id is valid regardless of its locality. Therefore, we introduce a notion of locality polymorphism using the `(backquote) notation, analogue to the ' (quote) notation for type polymorphism in ocaml. The identity function is thus typed: id : 'a_`z -(`z)-> 'a_`z .

Note that the latent effect of the arrow is `z , as both input and output localities. Indeed, the locality of the values manipulated by a function must appear in its latent effect. For example, the application of the identity function on a parallel vector must have the latent effect b as it manipulates parallel data.

To ease the understanding of types, a convention is used for the polymorphic types and localities: they are alphabetically, respectively reversed alphabetically, enumerated from a set of letters deprived of m, b, c, l and s.

The Multi-ML features

In the following section, we choose to illustrate the multi-ml language using code examples. To ease the pen and paper execution of codes, we define a small multi-bsp architecture that will be recurrent through all the examples. This architecture is composed by 3 levels and every inner node has 2 sub-nodes. To identify each component, we use a regular tree based numbering. Therefore, this 3-leveled (d = 2) architecture is composed by one root node (0), two nodes (0.0 and 0.1) and four leaves (0.0.0, 0.0.1, 0.1.0 and 0.1.1). Figure 3.2 shows a representation of this small multi-bsp architecture where, as before, circles and squares respectively stand for nodes and leaves.

The multi-functions definitions

The main functionality added by multi-ml, compared to bsml, is the ability to define multifunctions. A multi-function is a particular kind of recursive function that is adapted to the recursion through a multi-bsp tree. This kind of function is used to execute some codes all over the architecture thanks to a simple mechanism based on both parallel vectors and recursive calls. The multi-function is identified by the keyword multi and the body declaration of this function is split in two specific parts: the node code and the leaf code. The declaration structure of a multi-function is the following: As expected, the node code on lines 2 -3 is going to be executed on the nodes2 . Within this section, the programmer must write a bsml code to manage parallel computations and communications. The leaf code on lines 4 -5 is the sequential code executed on the leaves. The type of such a multi-function is (unit_`z -(m)-> string_m)_m with [`z <|m] . The multifunction mf takes an unit_`z as argument and returns a string of location m . The annotated arrow -(m)-> implies that the evaluation of the multi-function must be done at the multi-bsp level. It is important to note that the definition of a multi-function is possible at the multibsp level only. As the multi-function body code must be known by the whole architecture, a definition during a recursion through the multi-bsp architecture is not conceivable. It must be defined in all memories (depending on the implementation: see Chapter 5 for details).

THE MULTI-ML FEATURES

One can notice that there is a slight difference between the multi-bsp model and the multiml implementation. Indeed, multi-bsp describes an execution model where the computations take place on the leaves only and nodes are memories, without computing capabilities. In multiml, it is possible to express computation on nodes to manage parallel computations. However, the underlying implementation can propose an execution in a multi-bsp fashion, where node computations are simulated by leaves. To guarantee the efficiency of a multi-ml program, it is important to concentrate the intensive computing tasks to leaves and favour lightweight management code for nodes.

The recursive multi-functions calls

Multi-functions are very powerful to recurse through a tree and are easy to handle. In multiml, the code that is executed inside a parallel vector of a node is actually evaluated on the sub-nodes. Based on this simple idea, the evaluation of a multi-function inside a parallel vector will launch a multi-function recursion on the sub-nodes. That is to say: recurse through the multi-bsp architecture. When a multi-function is applied to a value at the multi-bsp level, the evaluation is initiated on the root node of the architecture. In this example, the multi-function mf is called inside a parallel vector (line 3) to initiate the recursion on the sub-nodes. As the evaluation of a multifunction starts from the root node, the recursion will spread from the root toward the leaves. The Figure 3.3 shows how the multi-function is distributed on the sub-nodes using the parallel vector construction. Thanks to a typing mechanism that will be detailed in Chapter 4, the parallel vector rec_call is consequently typed string_c par_b (as the string is fully communicable and the parallel vector is restricted to the level b). The Figure 3.4 shows how the values transit through the different levels of the multi-bsp architecture. It is important to note that the values returned by both the node and the leaf code (symbolised by arrows in Figure 3.4) are not copied in the upper memory, they are just available through the parallel vector. It is possible to manipulate the parallel vector rec_call as a regular one, using the bsml primitives3 . For example, the bsml primitive proj can be used to extract, to the bsp level of a node, the values of this parallel vector. When the evaluation is done, the final value computed by the root node is given as the result of the multi-function. This value is pushed to the multi-bsp level. It is then available for every unit of the architecture. Concerning our example, the multi-function recursion will transit from the root node to the leaves, and then, up from the leaves to the root node. The string "hello node" is therefore the result of the multi-function application. The recursive multi-function calls are only accepted within the scope of a parallel vector. Calling a multi-function at the bsp level would introduce a behaviour that is difficult to handle in both the typing system and the implementation. The main difficulty concerns the introduction of tree structures at bsp level dues to multitree-functions (see Section 3.2.4).

The tree structure

As multi-ml deals with multi-bsp architectures, we choose to create a data type that represents values that are distributed all over the architecture. This particular type of data is called a tree. As the values of a tree are available in all the memories, they can be easily referenced by a single identifier. Such a data structure allows to keep previously computed values in the concerned memories, to avoid memory transfers. The easiest way to create a tree is using the mktree primitive. This primitive takes an expression e and evaluates it on every level of the architecture. let t = mktree "hello tree"

This code builds a tree t that contains the string "hello tree" at each level of the multi-bsp architecture. In accordance to the multi-ml type system, the value t is typed string_c tree_m . The type string_c means that the string contained in the tree has no restrictions concerning its communication. As expected, the tree is annotated with the locality m , as its definition takes place at the multi-bsp level. One can say that the tree concept is similar to the parallel vectors of bsml, in a multi-bsp way.

The multitree-functions

Unlike the multi-function that only generates a communicable sequential value at the multibsp level, a multitree-function aims to build a tree. However, the multitree-functions are much more powerful than the mktree primitive in the sense that it is possible to execute a code according to the execution level to generate a specific value.

Let us have a look to the following example: The multitree-function mtf build a tree that contains the value "node" on every node of the multi-bsp architecture (lines 2 -4) and "leaf" on every leaf (lines 5 -6). This simple example shows that, in multitree-functions, the node code must return a value constructed by the operator named finally . This operator is a constructor of a value of type Final and is typed This constructor highlights the fact that particular operations are going to be executed during a tree construction. As expected, the first argument, sub , is a parallel vector of trees. It represents the sub-trees or branches that we want to use in order to construct the resulting tree. Indeed, as the type of the multitree-function mtf is (unit_`a -(m)-> int_c tree_m)_m with [`a<|m] , the result of the recursive call of the multitree-function inside a vector is typed int_c tree_l par_b . It is important to note that the location of the tree inside the vector is l because we do not want to communicate such a value. Thus, a tree can be annotated by the localities m or l. The value rec_call contains the sub-trees or branches that have been constructed on the sub-levels. The second argument, head , is the value that is kept, at the current node. It is the top node of all the sub-branches. Figure 3.5 shows how the construction is done, by successively combining all the branches of the levels. Of course, the trees are not copied between the levels, they are only accessible through the parallel vector structure.

v 0.0.0 v 0.0.1 v 0.1.0 v 0.1.1 v 0.0 v 0.0.0 v 0.0.1 v 0.1 v 0.1.0 v 0.1.1 v 0 v 0.0 v 0.0.0 v 0.0.1 v 0.1 v 0.1.0 v 0.1.1 Figure 3.5: Tree construction.
With the combination of multitree-functions and tree recursion it is possible to take advantage of the multi-bsp architecture with a small syntax overhead.

Tree manipulations

We saw that both multitree-functions and the mktree primitive are useful to build a tree. Such a parallel data structure can be accessed using a particular operator in order to use, one by one, the values distributed at every level of the multi-bsp architecture. We propose the primitive at to perform this operation. at can be executed at different levels and its locality of execution determines its behaviour. Currently, we have one at operator for each level: at_b , at_c , at_l and at_s . This operator gives an access to its own memory, that is to say, values of trees that are relative to the current level. It is not possible to read the memory of an other level without explicit communication. Furthermore, there is no at operator defined for level m as it does not make sense to access to the value of a tree at this level. Figure 3.6 shows that the application of at_b on a tree t at level 0.1 returns the value contained at level 0.1 in t : t 0.1 . To illustrate the at primitive, we consider the following example: In this code, the multi-function sum_t computes the sum of all the elements contained in a tree given as argument. Concerning the node code, at line 4 the parallel vector rec_call contains all the sub-sums computed during the recursion. Using the proj primitive (line 5) in combination of a list operator (lines 6 -7), we can compute the sum of the elements (where procs is a list containing the processors ids [0, ..., p -1]). Finally, the result of the node code is the sub-sum of the sub-nodes plus the value contained in the tree at this level (line 8). The leaf code is pretty simple, we just need to return the value of the given tree (line 10).

It is interesting to note that the at primitive can also be applied on a tree contained in a parallel vector. For example, if v : int_c tree_l par_b , << at_l $v$ >> is valid. In this case, the resulting value is the local value of v that is stored on the sub-node. As this value is in the sub-memory, it cannot be manipulated outside the parallel vector without explicit communication.

Concerning tree composition, the multi-ml languages impose several restrictions. For example, let A be a built tree. During the construction of a tree B, one can imagine taking a whole branch of A to build B. This is not allowed as the tree manipulations are highly restricted. Nevertheless, building B with the values of A using the at operator is possible. To guarantee safe executions, the programmer is not allowed to manually nest trees.

A global identifier

In bsml, the keyword this (also known as pid ) is used to get the identifier of a processor, inside a parallel vector. It is basically a built-in parallel vector containing the values 0, …, p -1.

As multi-ml deals with trees, we propose a new type of identifier adapted to this structure. The gid (Global Identifier) is a built-in tree that contains those identifiers. As shows in Figure 3.7, the positions are encoded by a top-down path of positions. For example, 0 stands for the root node, 0.0 and 0.1 for, respectively, its first and second child, etc. For the ith component of a vector at node n, the identifier is n.i.

Communications

Vertical communications

In bsml, and therefore in multi-ml, we can express horizontal communication using the put primitive. In both bsml and multi-ml, vertical communications can be performed using the proj primitive. In this case, the communication is upwards: from the leaves to the node. In multi-ml we propose a notation to explicitly communicate data downwards: from the node to the leaves, directly inside the scope of a parallel vector. As the bsml implementation is replicated on every process of the architecture, a value used inside a parallel vector is implicitly available in every memory. Nevertheless, in multi-ml, the memories are all distinct. Thus, a value defined at an arbitrary level is not available in the sub-nodes memories: we need to explicitly distribute it in order to make it available in the sub-memories. By annotating a value with # , we explicitly communicate a value to the sub-memory, in order to be able to use it inside the scope of a parallel vector.

For example, the following code is valid: We explicitly specify that the value x , defined in the node memory line 3, must be communicated downwards using the notation #x# (line 4). Therefore, x is available inside the parallel vector. As the code is known by all the processes of the architecture, one can say that the value of x is implicitly known at runtime. Nevertheless, the following code is not valid: Indeed, the evaluation of x on the sub-nodes will lead to an unbound value because x is only bound in the current scope: the node environment. Furthermore, as the boolean cond (line 3) is only known in the node memory, the sub-node cannot know the actual value of x . Thus, we cannot know the value of x in the sub-nodes. Such a code justifies the usage of this annotation to get reliable communications.

Communication friendly vector construction

As explained previously, it is mandatory to communicate data to the sub-nodes to use it inside parallel vectors. One can imagine that splitting a list through the sub-nodes is inef-ficient if we need to copy the whole list everywhere and, then, split it. We assume that, omitting the locality annotation that is not significant in this example, we have a function split : int -> int -> int list -> int list that takes an integer n, an integer i, a list of integers and returns the ith list of the n split lists. When we want to split the list lst using the vector notation, we could write the following code: Thanks to the # operator, this code is perfectly valid and its execution leads to a valid list split. However, it implies an unnecessary amount of communication: the whole list is, first, distributed (because of the # operator) and, then, split. To avoid superfluous communications, we choose to redefine the mkpar primitive that exists in bsml. Its aim remains the same: distribute values to the sub-nodes, but its execution is different. By omitting the type annotations, the ocaml type of mkpar is: (int -> 'a) -> 'a par . Its usage is similar to the bsml one. However, the function provided to mkpar to build the parallel vector is going to be executed sequentially on the node. And then, the results will be distributed on the leaves. Figure 3.8 shows the execution of the code mkpar (fun i -> f i) . The first step is the sequential computation: the evaluation of f 0 and f 1 produces respectively v0 and v1 . The second step is the communication of v0 and v1 to the sub-nodes. The evaluation of the mkpar primitive is computationally more intensive for the node, compared to the regular vector construction, but it is more efficient in terms of communication. For example, using the mkpar primitive and the previously defined split function, we can write the following code: In this example, the list lst is split on the node and only the required sub-lists are communicated. At level j, the estimated cost of mkpar (fun i -> split n i lst) is:

p j-1 ∑ k=0 (w k ) + n × g j + L j ;
Where p j-1 is the number of processes of the sub-node of j, n is the size of list lst , g j is the communication bandwidth, L j the synchronisation cost and w j-1 the work done by level j -1. Otherwise, the cost of << split #n# #lst# >> can be estimated with the following formula:

p j-1 ∑ k=0 (n) × g j + L j + w j-1 .
Compared to the code without mkpar that communicates p j-1 times the whole list, here, the whole list is communicated only once. If the computations implied by the evaluation are not intensive and the amount of data is important, the usage of mkpar is more efficient. Such a behaviour can be dynamically chosen during the execution of an algorithm. For example, given the multi-bsp parameters and the data size, one can choose to favour the usage of mkpar when the amount of data is important and the bandwidth is low: at the top of the architecture; Otherwise, when the amount of data is limited by the memory size and the bandwidth is high: at the bottom of the architecture (next to the cache memories), it seems legitimate to use the << >> primitive.

A complete example

In this section, we are going to analyse a small code example that uses most of the multi-ml features presented above. Let us say that we have a big list available in a file and we want to sum all its elements and keep the sub-sums at all levels of the multi-bsp architecture. As this list is huge, it can only fit in the memory of the root node and it must be split to fit the sub-memories. To simplify the algorithm, we assume that when the list is split in a node in p sub-lists, the element will fit the sub-memories. This assumption avoids to check the memory size of the sub-nodes and iterates through a set of well calibrated lists. To simplify the illustration of the example, we chose the list [0;1;2;3;4;5;6;7] as input value. We also have, at our disposal, the split: int -> int -> int list -> int list function that, given a number of slices, an element i and a list, returns the i th slice of the list; nprocs is the number of sub-processes; a flatten: 'a par -> 'a list function that simply transforms a parallel vector into a sequential list; and a sum_seq: int list -> int function that computes the sum of all the elements of a list, sequentially. The type of the sum_list multitree-function is sum_list: (int_`z list_`y -(m)-> int_c tree_m)_m . As expected, this function takes a list of integers that does not require a specific locality. It

A COMPLETE EXAMPLE

77 returns a tree of integers that are communicable. The latent effect m annotates the arrow in order to indicate the multi-bsp nature of the value.

The first step of this multi-function node code, lines 5 -6, is to read the list from the file if the current level is the root ( gid is 0). In the other cases we take, as input, the list given as argument: line 8. Note that the list_of_file function is defined in accordance to the node memory limits. The next important step is the distribution of the list, line 10. One can note that we choose to use the mkpar primitive to avoid useless communications. Therefore, v is a parallel vector that contains all the slices of the given list l . Now, we need to initiate the recursion of the multitree-function on the sub-nodes by calling sum_list on the vector v , line 11. Just before the execution of the leaf code, the history of the distribution of the list through the memories is illustrated in Figure 3.9. Afterwards, we are going to compute the sum of all the sub-sums that are available thanks to the recursive call of the multi-function, line 12. They are available on the tree rc and the sub-sums of the sub-nodes can be accessed using at_l .

<< at_l $rc$ >> is therefore a vector containing the sub-sums of the sub-nodes. We just need to flatten it and sum it to get the sum of the sub-sums. Finally, line 13, we returns the vector of branches (to construct the tree of sub-sums) with the value s , freshly computed, to conclude the node computation. Concerning the leaf code (line 15), it is pretty simple: we just need to compute the sequential sum of the given list and return it as a result. The result of the execution of the multi-function sum_list on the list [0;1;2;3;4;5;6;7] is thus, a tree containing the sum of all the elements on the root node and the sub-sums of the sub-lists on the other nodes. The Figure 3.10 shows the values of the resulting tree. As it is said in [START_REF] Milner | A Theory of Type Polymorphism in Programming[END_REF]: "Well-typed programs cannot go wrong". The typing system that we will define for multi-ml ensures that a program is well-formed. In this context of multi-bsp programming we add a degree of complexity, compared to the ocaml type system. We now need to handle new problems such as memory coherency, deadlocks, safe data communications, etc.

To ease the understanding of the multi-ml typing system, we choose to split it in several parts that are highly related. After introducing the main problematic issues of multi-bsp program safety in general and how it is related in bsml and multi-ml, we present first, a semantics and then, a type system for a core version of multi-ml. Then, we introduce the management of references and exceptions. The last section of this chapter is dedicated to the type inference system, which is an algorithm that finds the (most general) type for a given expression. The proofs of lemmas are available in the Annex 6.2.6 at the end of the chapter.

Safety in parallel and distributed programming

In this section we are going to talk about the pathological cases raised by the parallel capabilities of the multi-ml language.

Replicated coherency

In multi-ml, values can be defined at various levels. In addition to designating a component of a given depth, a level adds a notion of parallelism. A level is associated to a locality which stands for the quantity of parallelism within a component of the multi-bsp architecture. For example, the operational semantics of the language says that a value declared at the multi-bsp level is accessible from every process. This value is thus stored inside a memory that can be accessed from each process of the architecture. However, according to the implementation of the language, this value is replicated in the memory of every process. In this case, we need to keep all these value instances (potentially stored in different memories) to the same value to preserve the replicated coherency. Otherwise, it is possible to introduce incoherency and unpredictable behaviour. For example, if a value v defined at level m is different in some memories, a code of the form if v = … then (*multi-function call*) else (*sequential code*) may lead to an incoherent state where only a sub-part of the machine will execute a multi-function. They can have 80 CHAPTER 4. TYPE SYSTEM unpredictable results, especially when global barriers are performed without all the processors.

We also need to prevent the execution of random operators1 . For example, let random_bool be an operator that returns a random boolean. When executed inside a node, the following code is not valid:

if random_bool () then (proj v) else (fun x -> x)
Here, this code clearly leads to an undefined behaviour: some processors would execute the proj synchronisation primitive and the others will not. Depending of the implementation (e.g. with different communication libraries), we could face a deadlock or an erroneous message signal.

Level compatibility

Some values are intended to be used at a specific level. For example, the parallel vector construction ( << (*code*) >> ), cannot be executed outside a node. As a parallel vector manages the memory of sub-nodes, it only makes sense at the bsp level. In our model it is absurd to execute this primitive on the leaves (sequential level) because there are no sub-processes. The evaluation of a parallel vector at the multi-bsp level does not make sense either; the multibsp level cannot express parallelism in this way, it only represents a replicated execution of a sequential code or the call of a multi-function. Furthermore, such a code inside a parallel vector (at local level) would lead to parallel vector nesting: something we want to avoid, like in bsml (see next section). We want to avoid the execution of bsp code outside the scope of a node. As the access to the sub-component of a level is possible from a node only, it is meaningless to express bsp parallelism anywhere else.

Because of those reasons, we need to define rules concerning the accessibility (where the data can be accessed) and definability (where a data can be declared) of values, to check whether or not a value makes sense at a given level. For example, the definition of a bsp code outside the scope of a node is possible. However, such a value will be usable on a node only.

Parallel structure nesting

At first, we want to avoid parallel vector nesting. Like in bsml, we do not want to accept a code such as let v = << $this$ >> in << v >> which leads to type int par par . We can imagine that this vector nesting could stand for the manipulation, as a global structure, of the sub-nodes, directly inside a parallel vector. But such a structure is not convenient in terms of programming and also introduces implicit communication cost through levels, as we manipulate several levels of memory. Therefore, the vector nesting is prohibited by the typing system. Some languages allow this kind of nested parallelism, such as nesl [START_REF] Blelloch | NESL: A Nested Data-Parallel Language[END_REF], but this implies difficulties concerning implementation, portability and predictions (due to a more complicated cost model). Note that experiences on parallel vector nesting (within two levels) were proposed in the Departmental Metacomputing ml library (dmml) [START_REF] Gava | A Functional Language for Departmental Metacomputing[END_REF], but this approach was too complex to handle.

It is also necessary to prohibit trees nesting. Indeed the type 'a tree tree is not acceptable because it means that a complete tree (representing the amount of values distributed all over the nodes and leaves of the architecture) is available at every level of the architecture. In other terms, we can say that a complete tree has been serialised and distributed all over the architecture. Such a structure is unreasonable in multi-ml and is not acceptable.

Then, we need to manage cross-nesting such as 'a par tree and 'a tree par .

The first nesting is obviously a nonsense. A type 'a par tree means that the value stored at each node and leaf is a parallel vector. However, it is impossible (and inconceivable in the multi-ml language) to have a parallel vector on leaves. This type is therefore rejected by the multi-ml typing system.

Concerning the type 'a tree par , it is a bit different. Having trees inside parallel vectors seems to be a bad idea, but it makes sense under certain conditions. During the construction of a tree, we need to make a recursive call of a multi-function inside a parallel vector. This execution will lead to a parallel vector containing all the sub-trees of the current level: a vector of branches. In other terms, we have a parallel vector that contains the pointers to branches (sub-trees) that are going to be used during the construction of the final tree. Such a type is not a problem in our type system because its construction is highly restricted (by the multitree-function mechanism) and cannot be communicated outside the level where it belongs to. This is the reason why this kind of nesting is permitted in the multi-ml language, during the construction of trees only.

Operational semantics

To ensure that a program "cannot go wrong", we first need to define how a program "goes", which consists in defining the operational semantics. This formal definition of the multi-ml language will be used to prove several properties relative to parallelism, such as determinism. To agree with the typing system's proof, we introduce the natural (also called big step) semantics. To do so, and to avoid being overwhelmed by the details of a complete language, we describe the core-language and the definitions that will be used for the semantics. Then, we propose the set of evaluation rules that describe the behaviour of the multi-ml language.

To formalise the multi-ml semantics, ott [START_REF] Sewell | Ott: Effective Tool Support for the Working Semanticist[END_REF], [[ 21 ]] has been used. ott is a tool for writing definitions of programming languages and calculi. It takes as input a definition of a language syntax and semantics, in a concise and readable ASCII notation that is close to what one would write in informal mathematics. It generates many outputs, such as a coq version of the definition, which is very useful to formally prove the correctness of the semantics.

A core language

We chose to select a subset multi-ml and form the µmulti-ml language, to ease the understanding and analysis of the semantics. This core-language relies on a minimal set of ocaml constructions. This small amount of programming features is sufficient enough to express all the parallel behaviour that is used in multi-ml. Thus, features such as records, modules, pattern matching, sum types are excluded from µmulti-ml.

The grammar of the µmulti-ml expressions is defined in Figure 4.1.

In this grammar, x and f range over an infinite set of identifiers. We also find the typical ml-like language constructors such as let for bindings and also fun and rec for, respectively, function and recursive functions. As expected, the application is denoted (e e). For the sake of readability, we take the liberty to use the familiar infix notation for binary and ternary operators, as well as the usual precedence and associativity rules. When the context is clear, we can avoid the usage of parentheses. The multi-function definition is written with the keyword multi and * ,/,fst,snd,… Basic operators takes two arguments separated by the † symbol. Then, we have the parallel primitive mkpar followed by the synchronous parallel primitives used for communications: proj and put. The grammar also contains the syntactic sugar which eases the way of manipulating parallel vectors and the core parallel primitives which are used in the µmulti-ml languages.

::= Values | op Operator | cst Constant | (fun x → e )[E] Closure | (rec f x → e )[E] Recursive closure | (multi f x → e † e)[E] Multi-function closure | (v, v) Pair | < v, …, v > Parallel vector op ::= Operators | +,
E ::= {x 1 → v 1 , …x n → x n } Environment
Then, the values v stands for the standard operators, such as common computations on integers or the fst and snd projections, and also constants (which might refer to integer values, booleans, characters, strings, etc.), functional, recursive and multi-function closures, pairs and also parallel vectors.

An environment E is interpreted as a partial mapping (informally called "finite mapping") with finite domain from identifiers to values. The mapping associates v i to x i , for all i from 1 to n, and is undefined on other identifiers. The empty mapping is written {∅}. We also write Dom(E) for the domain of E, that is {x 1 , . . . , x n }, and CoDom(E) for its range, that is {v 1 , . . . , v n }. If x belongs to Dom(E), {x → v} ∈ E denotes the variable v associated to x in E. Finally, we define the extension of E by v in x, written E ⊕ {x → v}, by:

{x 1 → v 1 , …, x n → v n } ⊕ {x → v} = {x 1 → v 1 , …, x n → v n , x → v} if x ̸ ∈ {x 1 , …, x n } {x 1 → v 1 , …, x n → v n } ⊕ {x → v} = {…, x i-1 → v i-1 , x → v, x i+1 → v i+1 , …} if x = x i
Hence, we have as expected:

Dom(E ⊕ {x → v}) = Dom(E) ∪ {x} {x → v} ∈ (E ⊕ {x → v}) = v {y → v} ∈ (E ⊕ {x → v}) = {y → v} ∈ E for all y ∈ Dom(E), y ̸ = x
The set of expressions available in the multi-ml language is pretty straightforward and reflects the constructions available in the language. However, it is important to distinguish the parallel syntactic sugar (which can be used to program a multi-ml algorithm) and the Core parallel primitives (which are only used in the semantics). Indeed, the syntactic sugar eases the way of programming but it is not suitable for the semantics as it introduces implicit assumptions. Thus, we must transform and abstract the syntactic sugar proposed using the core parallel primitives. This transformation is trivial and produces an equivalent expression.

The informal description of this new set of primitives is the following:

• apply e e is similar to the original bsml primitive. It takes a two values: a parallel vector of functions and a parallel vector of values, and produces a parallel vector consisting in the application of functions on values. apply is useful to apply a function on each element of a parallel vector. Its informal semantics is:

apply < f 0 , …, f p-1 >< v 0 , …, v p-1 > →< (f 0 v 0 ), …, (f p-1 v p-1 ) >
• replicate (fun _ → e ) is used to replicate a value on every component of a sub-node.

To do so, it takes a function (fun _ → e ), which takes any arguments and returns e.

Then, the function is evaluated on a unit argument, on each sub-component. Its informal semantics is:

replicate (fun _ → e ) →< (fun _ → e )(), …, (fun _ → e )() >
• down x simply takes a value x and returns a parallel vector containing x on each compo-84 CHAPTER 4. TYPE SYSTEM nents. Its informal semantics is:

down x →< x, …, x >
replicate, down and apply are core primitives which have been introduced in the semantics to perform the transformation. Thus, those keywords are not available in the syntax of the multi-ml language. The transformation applied to switch from the syntactic sugar to the core parallel primitives is straightforward.

The parallel vector scope, denoted << e >>, is transformed using the replicate core primitive. Thus, << e >> is transformed into replicate (fun _ → e ).

The open syntax ( $ ) is transformed using the apply primitive. Like in bsml, the transformation is simple and does not require a complicated expression analysis. To do so, we build a vector of functions that takes, as argument, the dollar's annotated value. Using the apply primitive, we can apply this vector of functions on the vector of values. For example, the expression << e $x$ >> is transformed into apply (replicate (fun _ x → e x )) x.

The copy syntax ( # ) stands for a communication of a value to the sub-nodes. The primitive down is used to perform such a transfer. As the down primitive aims to distribute a value to all the subnodes, the transformation is obvious. For example, the expression << e #x# >> is transformed into apply (replicate (fun _ x → e x ))(down x). As the sharp annotated value is given as argument of the vector of functions, there are no redundant copies. The expression << #x# + #x# >> is tranformed into a code that copy x to the sub-nodes, only once.

To sum up, the informal description of the primitives and their (ml) types are available in Table 4.1. It is important to note that p, which stands for the number of processes of a bsp architecture, describes, here in multi-bsp, the number of sub-nodes of a node. Table 4.1: The informal description of µmulti-ml primitives.

In this core language we do not propose the grammar for trees. The multi-tree-function definition, its application and the usage of trees are thus not available in both the semantics and the typing system. Indeed, the usage of the tree structure introduces a large number of annotations resulting in technical and rather unreadable rules. For these reasons, we do not introduce them in this thesis and we postpone their presentation to future work.

However, we will informally introduce the types of the tree primitives, as they are available in the current multi-ml implementation.

Semantics rules

The big-step semantics (or natural semantics) is used to describe the evaluation of an expression to its final value using inductive rules. On the contrary, the small-steps semantics apply one-step reductions on terms. Even if big-step semantics may be more convenient in various cases (especially to prove some properties by induction), small-step semantics is widely used because of its expressiveness concerning both terminating and non-terminating programs. Thanks to [START_REF] Leroy | Coinductive Big-Step Operational Semantics[END_REF], it is also possible to describe both terminating and diverging evaluations with a big-step semantics. Thus, we choose to use this approach to describe the multi-ml semantics. Unlike the method chosen in [START_REF] Fortin | Towards Mechanised Semantics of HPC: The BSP with Subgroup Synchronisation Case[END_REF], here, there is no need to handle processors interleaving. As a consequence, it is easier to model the semantics using coq. First, we will propose the terminating semantics for terms and then give rules for diverging ones. In the context of ml programming, we only consider eager (or call-by-value) evaluation strategy. Thus, we do not consider lazy (or call-by-need) evaluations.

Terminating semantics rules

The inference rules are written:

P M ⊢ e ⇓ L p v
and P

M ⊢ e ⇓ L p ∞ On the left-hand side, we have the inductive inference rule. It can be read: under the premises (or antecedents) P, the inference rule concludes that, on the component p of locality L, within a multi-environment M, the expression e is evaluated into a value v. This rule leads to a finite derivation tree. On the right-hand side, the co-inductive inference rule is written similarly, but using a double-bar. It stands for a case where e diverges (∞). Otherwise, it is an error: no applicable rules are defined.

The environment of evaluation is denoted by M and is called a multi-environment. It is composed by two elements: (1) a multi-bsp environment and (2) a tree of environments.

The first element represents the multi-bsp memory. This memory contains values that are accessible by any processes. Depending on the implementation and the architecture, it can be simulated by a virtual unified memory2 or by a physical global memory, for example, when simulating an execution in the toplevel. The virtual memory is not available in the current implementation of multi-ml, we leave it for future work. In practice, the memory is replicated on each component of the multi-bsp architecture. We choose to create a single environment that is used during multi-bsp evaluations. An evaluation taking place at the multi-bsp level, denoted by multi, is then evaluated within the environment ||M||@multi. The notation ||M|| denotes the extraction of an environment from the memory M and the @multi refers to the targeted memory. As expected, every unit of the architecture can access this multibsp environment. It is possible to lookup in it at any time, from any level p, that is to say, from each component of the multi-bsp architecture. Nevertheless, the modifications of this environment are possible at the multi-bsp level only.

The second element represents all the environments of the architecture. It takes the shape of a tree which maps all the environments of the different levels of the architecture, that is to say all the nodes and leaves. Referring to the environment of a particular component p of this tree is denoted ||M||@p. We recall that the components are identified using a regular tree numbering where 0 stands for the root node, and 0.0 to 0.n identify its n sub-components. Thus, it contains its environment only, deprived of the multi-bsp memory. As the multi-bsp memory is always available, the notation ||M|| p stands for the environment composed by ||M||@multi ∪ ||M||@p, where the subscript p stands for the targeted memory. Now, we introduce the symbol of rule evaluation. The piece of notation ⇓ L p denotes an evaluation taking place at a specific level L, also parametrised by the position p. The evaluation level L can take the values m, b, l or s, to specify on which level the evaluation must occur. The locality c, which is proposed in the multi-ml type system and denotes values that are communicable to any contexts, is not used here. This locality is important and makes sense in the type system but, it is not necessary in the semantics. We use the locality l (local) to refer to the scope of a parallel vector, and we do not pay attention to its communication ability through an explicit annotation. Nevertheless, the communicability of a value is still important in the semantics. Thus, we propose a predicate to make this verification: Comm s (v). By abuse of the notation, this predicate is ⊤ if the value v is communicable, ⊥ otherwise, within environment s. The definition of the predicate can be found in Figure 4.2. The subtlety of this notation comes with the communicability of x. We assume that, if x / ∈ s, then the value x is communicable. Indeed, if x does not belong to s, it means that x is a free variable which comes from the application of a closure, as the environment used for a closure is ε, which is the environment of the closed term. Thus, during the evaluation of an application (e 1 e 2 ), we assume that x is communicable in e 1 as its communicability will be checked with e 2 .

The position p stands for the identifier of the component where an evaluation takes place. We use the notation p.i to refer to the ith sub-unit of a component that is accessible through a parallel vector. The identifier alias root stands for the root node and, as explained previously, multi is used when the whole architecture is concerned by an evaluation.

As the multi-ml language proposes different levels associated with different behaviours, we propose several rules that are either generic or specific to m, b, l. In Figure 4.3, we define all the evaluation rules which are applicable independently of the evaluation level. Those generic rules are valid everywhere and represent the functional kernel of the language. These rules are pretty straightforward and are really close to the ml semantics. However, some of them introduce new notations that will be explained along the rules description.

The values and op_eval rules a straightforward. The var rule introduces the predicate lookup(x, M, p, L), which is defined in Figure 4.7. As the var rule gives the value corresponding to a binding, we must lookup for it in a particular way. The value x must be seek, first, in ||M||@p, which is the memory of the evaluation at level p. Then, if the value is not found there, we must look for it in the multi-bsp environment: ||M|| multi . If the value is not found anyway, the identifier is unbound in the current expression, which is an evaluation error ("unbound value"). As expected, if the evaluation takes place at the multi-bsp level (multi), the variable x is only searched for in the environment ||M|| multi . Furthermore, we add the locality of the evaluation as an input of the lookup() predicate to ensure that the values defined in the m memory won't be accessed within the scope of a parallel vector (of locality l).

values M ⊢ cst ⇓ L p cst op_eval M ⊢ op ⇓ L p op var {x → v} ∈ lookup(x, M, p, L) M ⊢ x ⇓ L p v closure E = select(M, F(fun x → v ), p, L) v ≡ (fun x → e )[E] M ⊢ fun x → e ⇓ L p v rec_closure E = select(M, F(rec f x → v ), p, L) v ≡ (rec f x → e )[E] M ⊢ rec f x → e ⇓ L p v fun_app M ⊢ e 1 ⇓ L p (fun x → e )[E] M ⊢ e 2 ⇓ L p v M ⊕ p E ⊕ p {x → v} ⊢ e ⇓ L p v ′ M ⊢ (e 1 e 2 ) ⇓ L p v ′ rec_fun_app M ⊢ e 1 ⇓ L p (rec f x → e )[E] M ⊢ e 2 ⇓ L p v M ⊕ p E ⊕ p {x → v} ⊕ p (rec f x → e )[E] ⊢ e ⇓ L p v ′ M ⊢ (e 1 e 2 ) ⇓ L p v ′ op_app M ⊢ e 1 ⇓ L p op M ⊢ e 2 ⇓ L p v op v ≡ v ′ M ⊢ (e 1 e 2 ) ⇓ L p v ′ let_in M ⊢ e 1 ⇓ L p v 1 M ⊕ p {x → v 1 } ⊢ e 2 ⇓ L p v 2 M ⊢ let x = e 1 in e 2 ⇓ L p v 2 Figure 4.3: Generic evaluation rules. 88 CHAPTER 4. TYPE SYSTEM replicate ∀i ∈ p M ⊕ p i {f → (fun _ → e )} ⊢ f () ⇓ l p i v i Comm ||M||p i (fun _ → e ) M ⊢ replicate (fun _ → e ) ⇓ b p < v 0 , …, v n > down M ⊢ x ⇓ b p v Comm ||M||p (v) M ⊢ down x ⇓ b p < v > mkpar M ⊢ e ⇓ b p (e ′ )[E ′ ] ∀i ∈ p M ⊕ p E ′ ⊢ e ′ i ⇓ b p v i Comm ||M||p (v i ) M ⊢ mkpar e ⇓ b p < v 0 , …, v n > apply M ⊢ e 1 ⇓ b p < (e i )[E i ] > M ⊢ e 2 ⇓ b p < v 0 , …, v n > ∀i ∈ p M ⊕ p.i {f i → (e i )[E i ]} ⊢ f i i ⇓ l p.i v ′ i M ⊢ apply e 1 e 2 ⇓ b p < v ′ 0 , …, v ′ n > proj M ⊢ e ⇓ b p < v 0 , …, v n > ∀i ∈ p M ⊕ p.i {f → (e ′ )[E ′ ]} ⊢ f i ⇓ b p.i v i Comm ||M||p (v i ) M ⊢ proj e ⇓ b p (e ′ )[E ′ ] put M ⊢ e ⇓ b p < (e 0 )[E 0 ], …, (e n )[E n ] > ∀i, j ∈ p M ⊕ p.i {f i → (e i )[E i ]} ⊢ f i j ⇓ l p.i v ij M ⊕ p.j {f ′ j → (e ′ j )[E ′ j ]} ⊢ f ′ j i ⇓ l p.j v ij M ⊢ put e ⇓ b p < (e ′ 0 )[E ′ 0 ], …, (e ′ n )[E ′ n ] > f
select(M, V, p, L) = let E ′ = ∅ ∀α ∈ V, lookup(α, M, p, L) ⊕ E ′ where V ≡ α 0 , …, α n return E ′
The closure and rec_closure rules are used to enclose all the needed variables that are available in M. To do so, we use the predicate select(M, V, p, L) which is defined in Figure 4.8. This predicate selects all the variables of the set of variables V from the multi-environment M. As the predicate uses lookup(), we must provide the current level (p) and its locality (L). The set of variables V is determined using F(e), which stands for the set of free variables of the expression. The set of free variables F(e) is defined in Figure 4.9.

Definition 4.1 (Free variables)

A free variable stands for a variable which is referenced in a context but which is not bound within the context.

A term e is referenced as closed if it does not contain any free variable: F(e) = ∅.

The final and minimal environment E produced by select() is then attached to the final closure.

The fun_app is used to evaluate functional constructions on values. The rule introduces the notation ⊕ p , which is a concatenation operator allowing to enrich the environment of p with bindings. Here, we add the closure environment E and the binding {x → v} to ||M|| p to the environment of p. The rec_fun_app rule is similar to fun_app but it also adds the associated recursive closure into the environment. The op_app is also close to the fun_app rule as it deals with basic operators instead of closures.

The let_in rule is necessary to bind values within a context. The rule is straightforward as it only adds {x → v 1 } to the environment of p. In Figure 4.4 we present all the rules that can be evaluated at the bsp level.

The replicate rule is used to create parallel vectors. We recall that this primitive is introduced by the code transformation and is not available in the syntax of the multi-ml language. The replicate primitive requires, syntactically, a function definition as an argument. This function takes any argument, denoted with _ (similarly to the ocaml syntax), which is unbound in the enclosed term. Such function definition allows to delay the evaluation of the expression e when replicate is evaluated. This is necessary for the multi-ml runtime system, as explained in Section 5.1. The notation ∀i ∈ p allows to reference the i sub-components of the current node p, from 0 to n. Thus, the closure is added to the environment of each leaf of the current node, identified by p i , if the term is communicable. Then, the function is evaluated on (), which is a unit term, on each sub-components. As the evaluation takes place within the scope of a parallel vector, its evaluation locality is l. The resulting value is, as expected, a parallel vector which contains the results of the evaluation of e on each sub-component.

The down rule is used to distribute data to the sub-nodes. From a variable x, it builds a parallel vector containing the corresponding value v, if v is communicable. The distributed values are identical on the sub-nodes, thus the resulting parallel vector is < v >. mkpar also aims to build a parallel vector. Nevertheless, the evaluation of the ith elements of the resulting vector takes place at level p, sequentially. This evaluation is done within a single environment that is common to every evaluation. Thus, one evaluation could impact the environment of another one with a code using side effects, even if there is no way of doing any side effects in the current semantics. It is important to note that the big-step semantics does not allow to describe precisely the evaluation order of the i evaluations. We arbitrarily choose to do it in accordance to the ascending order: from 0 to n. Thus, the expression e ′ is applied on i, which corresponds to the processor identifier (pid) of the i st component. It is important to check that every value v i is communicable. Indeed, we need to communicate the values v 0 , .., v n to the sub-nodes of the level p and we must check the validity of such a transfer.

As expected, the apply rule applies a parallel vector of functions (e i ) on a parallel vector of data (v i ) and returns a parallel vector of values. Here, we compute, in parallel, the evaluations of f i i within the environment of each sub-component i.

The proj rule must be applied on a parallel vector available at the current level p. The resulting value is a closure which contains the content of the given vector. As expected, the values are accessible via their original identifiers. It is also necessary to check whether or not the values of the given vector are communicable in order to build a communicable closure.

The put rule works as expected. It takes a parallel vector of communication functions, referenced as f i within environment p i . Then, the received values are accessible via f ′ j . Here, both i and j stand for the sub-components of the current node p. They are denoted as distinct variables to show the all-to-all exchange.

The rules presented in Figure 4.5 are specific to the locality l. We recall that this locality corresponds to the scope of a parallel vector. With the core language, only the multi-functions recursive calls are particular within the local context.

The multi_node and multi_leaf rules are relative to the recursive multi-function calls, which corresponds to the application of multi-function function on a value within the scope of a parallel vector. multi_node and multi_leaf are applied regarding to the type of the current component. The operators isN ode(p) and isLeaf (p) are used to make such a distinction. Informally speaking, isN ode(p) returns an error if p has no sub-components. On the contrary, isLeaf (p) returns an error if p has sub-components. Concerning the multi_node rule, the environment of evaluation of e ′ 1 is enriched by the argument given to the multi-function and the closure of the multi-function. Indeed, the sub-node must be able to recall the multi-function. We can observe that, after enriching the environment, the evaluation of e ′ 1 goes from l to b. This locality change is due to the fact that, when a recursive call of a multi-function (done within the scope of a parallel vector) is evaluated, we must initiate the recursion on the sub-nodes: which corresponds to evaluate e ′ 1 at level b. On the contrary, the multi_leaf rule is not concerned by this environment enrichment because of its status of leaf. A similar behaviour can be observed in the multi_leaf rule, where we go from l to s.

Finally, in Figure 4.6, we propose the rules that are specific to the multi-bsp level. It mainly deals with the multi-functions.

The multi_def rule aims to create the closure of a multi-function and make it available in the multi-bsp environment. To do so, we simply call the predicates select(||M|| multi , F(multi f x → e 1 † e 2 )) in order to built the environment M ′ from the multi-bsp environment and the free variables of the multi-function itself. As the closure v aims to be communicated to the root node in order to initiate the recursion, we must check that v is communicable.

The multi_call rule is used to initiate the multi-function evaluation. As expected, the multi-function evaluation starts on the root node, with an environment enriched by the given argument and the multi-function closure. Thus, the evaluation of e ′ 1 , corresponding to the node code, is evaluated at level b, on the root node. As the resulting value of the evaluation of the multi-function is going to be transferred to the multi-bsp level, we also need to check if v ′ is communicable. Indeed, the final value must be available in the ||M multi || environment. To do so, in case of a distributed multi-bsp memory, we must broadcast v ′ from the root node to all the components of the multi-bsp architecture.

We recall that, for now, we do not describe the usage of multi-tree-functions nor trees. To do so, we must define a new kind of environment which is, basically, a tree of environments. Thus, at each evaluation step, it produces a new tree of environment with a resulting value. Such a definition introduces a large amount of notation which makes the semantics rules barely readable. 4.14: multi-bsp coinductive evaluation rules. evaluate in general. Without "random" operators, programs that neither evaluate nor diverge according to the rules above are said to "go wrong". For instance, the program 0 0 (zero applied on zero) goes wrong since, for any environment E and value v, neither E ⊢ (0 0) ⇓ v nor

replicate ∃i ∈ p M ⊕ p i {f → (fun _ → e )} ⊢ f () ⇓ l p i ∞ M ⊢ replicate (fun _ → e ) ⇓ b p ∞ mkpar-e M ⊢ e ⇓ b p ∞ M ⊢ mkpar e ⇓ b p ∞ mkpar-v M ⊢ e ⇓ b p (e ′ )[E ′ ] ∃i ∈ p M ⊕ p E ′ ⊢ e ′ i ⇓ b p ∞ M ⊢ mkpar e ⇓ b p ∞ apply-l M ⊢ e 1 ⇓ b p ∞ M ⊢ apply e 1 e 2 ⇓ b p ∞ apply-r M ⊢ e 1 ⇓ b p < (e i )[E i ] > M ⊢ e 2 ⇓ b p ∞ M ⊢ apply e 1 e 2 ⇓ b p ∞ apply-f M ⊢ e 1 ⇓ b p < (e i )[E i ] > M ⊢ e 2 ⇓ b p < v 0 , …, v n > ∀i ∈ p M ⊕ p.i {f i → (e i )[E i ]} ⊢ f i i ⇓ l p.i ∞ M ⊢ apply e 1 e 2 ⇓ b p ∞ proj-e M ⊢ e ⇓ b p ∞ M ⊢ proj e ⇓ b p ∞ proj-f M ⊢ e ⇓ b p < v 0 , …, v n > ∃i ∈ p M ⊕ p.i {f → (e ′ )[E ′ ]} ⊢ f i ⇓ b p.i ∞ M ⊢ proj e ⇓ b p ∞ Figure 4.11: bsp coinductive evaluation rules -part 1. put-e M ⊢ e ⇓ b p ∞ M ⊢ put e ⇓ b p ∞ put-ij M ⊢ e ⇓ b p < (e 0 )[E 0 ], …, (e n )[E n ] > ∀i ∈ p M ⊕ p.i {f i → (e i )[E i ]} ⊢ f i j ⇓ l p.i ∞ M ⊢ put e ⇓ b p ∞ put-ji M ⊢ e ⇓ b p < (e 0 )[E 0 ], …, (e n )[E n ] > ∀i, j ∈ p M ⊕ p.i {f i → (e i )[E i ]} ⊢ f i j ⇓ l p.i v ij M ⊕ p.j {f ′ j → (e ′ j )[E ′ j ]} ⊢ f ′ j i ⇓ l p.j ∞ M ⊢ put e ⇓ b p ∞ Figure 4.12: bsp coinductive evaluation rules -part 2. multi_node-r isN ode(p) M ⊢ e 1 ⇓ l p (multi f x → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ l p ∞ M ⊢ (e 1 e 2 ) ⇓ l p ∞ multi_node-b isN ode(p) M ⊢ e 1 ⇓ l p (multi f x → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ l p v M ′ ⊕ p {x → v} ⊕ p {f → (multi f x → e ′ 1 † e ′ 2 )[M ′ ]} ⊢ e ′ 1 ⇓ b p ∞ M ⊢ (e 1 e 2 ) ⇓ l p ∞ multi_leaf-r isLeaf (p) M ⊢ e 1 ⇓ l p (multi f x → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ l p ∞ M ⊢ (e 1 e 2 ) ⇓ l p ∞ multi_leaf-s isLeaf (p) M ⊢ e 1 ⇓ l p (multi f x → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ l p v M ′ ⊕ p {x → v} ⊢ e ′ 2 ⇓ s p ∞ M ⊢ (e 1 e 2 ) ⇓ l p ∞ Figure 4.13: Local coinductive evaluation rules. multi_call-r M ⊢ e 1 ⇓ m multi (multi f x → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ m multi ∞ M ⊢ (e 1 e 2 ) ⇓ m multi ∞ multi_call-b M ⊢ e 1 ⇓ m multi (multi f x → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ m multi v M ⊕ root {x → v} ⊕ root {f → (multi f x → e ′ 1 † e ′ 2 )[M ′ ]} ⊢ e ′ 1 ⇓ b root ∞ M ⊢ (e 1 e 2 ) ⇓ m multi ∞ Figure
E ⊢ (0 0) ⇓ ∞ holds.
In the following, we propose a system of typing rules that allows to identify such programs. A type system is an over-approximation which forbid programs that "go wrong".

A typing system for µmulti-ml

In this section, we propose a typing system for the core language µmulti-ml. This minilanguage offers a subset of the ocaml features that are necessary to program a multi-ml algorithm. We choose to reduce the expressivity of this toy language for conciseness reasons.

Type definitions

A lot of effect based type systems or flow analyses use type annotations [START_REF] Simonet | Type Inference with Structural Subtyping: A Faithful Formalization of an Efficient Constraint Solver[END_REF]. Considering a regular type system, annotating types with an effect allows to represent the level of information carried in addition to the structure of the values. These informations embedded with the type are controlled by relations defined, most of the time, in lattices. As a type describes the value that an expression may return, effects describe the side-effect that an expression may have, that is to say its locality of evaluation.

In multi-ml, a type annotated by its locality information is denoted: τ π . This means that a traditional ml type τ is complemented with the locality of use: π. The locality of a type is, basically, an effect associated to a type that keeps its location information. The evaluation of a lambda term may also produce an effect. We talk about a latent effect (usually symbolised with ε instead of π) and we denote it ε -→. This annotation announces the presence of a particular effect ε inside the closure of the lambda term. This effect is emitted during the evaluation of the function and we need to deal with it according to the current evaluation level. When an expression e is evaluated, it also emits an effect which represents the locality requirement of e. We write τ π /ε when the evaluation of an expression leads to a value typed τ π with an emitted effect ε.

Type variables are denoted by α and β (and its derivatives α 1 , …, α ′ , …, β 1 , …, β ′ ). The locality variable is usually denoted by π (and its derivatives π 1 , …, π ′ , …). When representing a locality, λ denotes a variable effect and ε refers to a latent effect.

The definition of the grammar of annotated types is given in Figure 4.15. The meaning of the different localities can be expressed as follow:

τ ::= α π type variable | Base π Basic type (int , Bool, …) | (τ π -→ τ ) π arrow type | (τ × τ ) π product type | (τ par) π parallel vector | (τ tree) π tree
m : The multi-bsp level. An expression evaluated at this level concerns every stage of the multi-bsp architecture. Thus, the generated value is available for every process of the machine. This level is useful to run some code that is going to be executed through the whole architecture and can handle particular constructions such as multi-functions and trees. Depending on the implementation and the architecture, this memory can be a specific memory accessible from every level; it also could be simulated by the replication of the values in every memory.

b : The bsp memory of a node. At this level, we can execute bsml code, manage multifunction recursion through parallel vectors and also execute communications and synchronisations. It is also possible to point towards the underlying memory of a node via the parallel vector construction.

c : The communicable memory of the underlying memory which is managed through a parallel vector. This locality is manly used inside the scope of a parallel vector to denote a value that makes sense in any context. Thus, this memory is free of references to the upper level and context sensitive constructions.

l : The local memory of the underlying memory that is managed inside a parallel vector. Like c, this is the underlying memory that can be managed inside a parallel vector from the upper node. This memory can contain references to the upper memory via a specific syntax and context sensitive constructions. This is the reason why it makes no sense to communicate values of such locality to another stage of the architecture.

s : The sequential memory of a leaf. This memory is used to execute sequential computations. As it is purely sequential, it can contain specific values, such as random values.

To sum up, each locality is related to a memory level which refers to a level of execution. These levels are defined by the multi-bsp architecture and are distinguished by their parallel capabilities. We have m for multi-bsp, b for bsp, s for sequential, c for communicable and l for local.

We recall that, as effects are not displayed in standard ocaml, we need to update the ocaml type syntax in order to show the locality information on types. For example, the annotated type int m is naturally displayed int_m . Concerning functional values, the latent effect l -→ is represented -(l)-> . Finally, to denote polymorphism on localities we choose the backquote ': `z .

As the multi-ml type system deals with polymorphic types, we need to add several definitions. Here, as the definition concerns type polymorphism, the type annotations are not required. Thus, we do not consider them in the following definitions, for conciseness purposes.

Let F(τ ) denote the set of types variables that are free (also known as Free Type Variables ) in the type τ which is defined by :

F(Base π ) = ∅ F(α π ) = {α} F(τ 1 → τ 2 ) = F(τ 1 ) ∪ F(τ 2 ) F(τ 1 × τ 2 ) = F(τ 1 ) ∪ F(τ 2 )
Here and elsewhere, an environment Γ is interpreted as a partial mapping (improperly called finite mapping) with finite domain from identifiers to values: the mapping that associates v i to x i , for all i from 1 to n, and that is undefined on the other identifiers. The empty mapping is written {}. We write Dom(Γ) for the domain of Γ, that is {x 1 , . . . , x n }, and CoDom(Γ) for its range, that is {v 1 , . . . , v n }. If x belongs to Dom(Γ), {x → v} ∈ Γ denotes the type v associated to x in Γ. Finally, we define the extension of Γ by v in x, written Γ ⊕ {x → v}, by:

{x 1 → v 1 , . . . , x n → v n } ⊕ {x → v} = {x 1 → v 1 , . . . , x n → v n , x → v} if x ̸ ∈ {x 1 , . . . , x n } {x 1 → v 1 , . . . , x n → v n } ⊕ {x → v} = {. . . , x i-1 → v i-1 , x → v, x i+1 → v i+1 , . . .} if x = x i
As expected, we have:

Dom(Γ ⊕ {x → v}) = Dom(Γ) ∪ {x} {x → v} ∈ (Γ ⊕ {x → v}) = v {y → v} ∈ (Γ ⊕ {x → v}) = {y → v} ∈ Γ for all y ∈ Dom(Γ), y ̸ = x
The substitutions considered here are finite mappings from type variables to type expressions. They are denoted φ, ψ:

φ, ψ ::= [α 1 → τ 1 , . . . , α n → τ n ]
A substitution φ is defined as follows for a type expression:

φ(Base) = Base φ(α) = τ ′ if [α → τ ′ ] ∈ φ φ(τ 1 → τ 2 ) = φ(τ 1 ) → φ(τ 2 ) φ(τ 1 × τ 2 ) = φ(τ 1 ) × φ(τ 2 )
A substitution φ is also naturally defined on a set of variables. The following property shows the effect of a subscript over the free variables of a type:

Proposition 4.5

For a type τ and a substitution φ, we have:

F(φ(τ )) = ∪ α∈F (τ )

F(φ(α))

Proof : By structural induction over τ :

• Case τ = Base. F(φ(Base)) = ∅ and there is no α ∈ F(τ ), hence the result.

• Case τ = α. If α ̸ ∈ Dom(φ) then φ(α) = α and φ(τ ) = τ and hence the result. Otherwise, there exists β such that β = φ(α). Thus, F(φ(τ )) = β and only α ∈ F(τ ) ; hence the result. • Case τ = τ 1 → τ 2 (resps. τ 1 × τ 2 ). Both by straightforward applications of the inductive hypothesis.

To handle polymorphism, we now define the set type schemes, with the typical element σ, by the following grammar: σ ::= ∀α 1 , . . . , α n .τ In this syntax, the quantified variables α 1 , . . . , α n are considered as a set of variables: their relative order is not significant and they are assumed to be distinct. When the context is obvious, we do not distinguish the type τ and the trivial type scheme ∀.τ . Thus, we write τ for both. We identify two kinds of type schemes that differ modulo the renaming of the variables bound by the ∀ quantifier (α-conversion), and by the introduction, or deletion, of quantification variables that are not free in the type. More precisely, the type schemes are equated as following:

∀α 1 , . . . , α n .τ = ∀β 1 , . . . , β n .[α 1 → β 1 , . . . , α n → β n ](τ ) ∀αα 1 , . . . , α n .τ = ∀α 1 , . . . , α n .τ if α ̸ ∈ F(τ )
The free variables in a type scheme are:

F(∀α 1 , . . . , α n .τ ) = F(τ ) \ {α 1 , . . . , α n }
This definition is naturally compatible with the two equations over type schemes. Then, we extend substitutions to type schemes as following: φ(∀α 1 , . . . , α n .τ ) = ∀α 1 , . . . , α n .φ(τ ) if {α 1 , . . . , α n } ̸ ⊆ F(τ ) Thus, assuming that, after renaming the α i , there are out of reach of φ which are: none of the α i is in the domain of φ; and none of the α i is free in one of the types in the range of φ. That is: (1) φ(α) = α (the substitution does not modify α); (2) if α is not free in τ then α is not free in φ(τ ) (φ does not introduce α in its result). We can easily check that this definition is compatible with the two equations over schemes. The Proposition 4.5 also holds with the type τ replaced by a scheme σ: Proposition 4.6

For a type σ and a substitution φ, we have:

F(φ(σ)) = ∪ α∈F (σ)

F(φ(α))

Proof : Take σ = ∀α 1 , . . . , α n .τ such that ∀i the α i are out of reach of φ. By definition, F(σ) = F(τ )\{α 1 , . . . , α n } and φ(σ) = ∀α 1 , . . . , α n .φ(τ ). Then F(φ(σ)) = F(φ(τ )). Since the α i are out of reach of φ then

F(σ) = F(τ ). Thus ∪ α∈F (σ) F(φ(α)) = ∪ α∈F (τ )
F(φ(α)) and then by application of the Proposition 4.5, we get the result.

Proposition 4.7

For a substitution φ and an environment E, we have:

F(φ(E)) = ∪ α∈F (E)
F(φ(α))

A TYPING SYSTEM FOR µMULTI-ML 101

Proof : By induction on E and applying the Proposition 4.6 for each β ∈ Dom(E).

A typing environment, written E, is a finite mapping from identifiers to type schemes:

E ::= {x 1 → σ 1 , . . . , x n → σ n }
The mapping definitions (Dom, CoDom, ∈, ⊕) are also naturally defined for a typing environment. The image φ(E) of an environment E by a substitution φ is naturally defined as following:

φ({x 1 → σ 1 , . . . , x n → σ n }) = {x 1 → φ(σ 1 ), . . . , x n → φ(σ n )}
The operator F is extended to the typing environment, by assuming that a type variable α is free in E if and only if for some x, {x → σ} ∈ E, α is free in σ:

F(E) = ∪ x∈Dom(E) F(σ) if {x → σ} ∈ E
Finally, we say that the type τ is an instance of the type scheme σ = α 1 , . . . , α n .τ ′ and we write τ ≤ σ, if and only if there exists a substitution φ whose domain is a subset of {α 1 , . . . , α n } such that τ is equal to φ(τ ′ ):

τ ≤ ∀α 1 , . . . , α n .τ ′ if and only if ∃τ 1 , . . . , τ n such as τ = [α ← τ 1 , • • • α ← τ n ](τ ′ )
The relation τ ≤ σ is obviously compatible with the two equations over schemes (α-conversion and useless quantified variable elimination).

For constants and primitives, we assume a function T C that assign type schemes to constants and operators. For instance, we propose the following list of non exhaustive assignments:

T C(i) = int (if i is an integer) T C(b) = Bool (if b equals True or False) T C(+) = int → int → int (if + is addition on integer) T C(IfThenElse) = ∀α.Bool → α × α → α

Type constraints definition

As our type system is highly inspired by the ocaml one, and is basically an extension of it, we are faithful to the standard Damas and Milner principles. In this section, our type system is related to the Hindley-Milner system relying on a X constraint system: hm(x) [START_REF] Odersky | Type Inference with Constrained Types[END_REF]. In our context of parallel programming, the X constraint system will handle all the multi-ml constructions. The type inference system will rely on pcb(x). As explained in [START_REF] Pottier | The Essence of ML Type Inference[END_REF], hm(x) and pcb(x) are really close in their concepts and definitions.

To manage effects dues to the parallel features of the multi-ml language, we choose to introduce a set of constraints. The grammar of the constraints can be found in Figure 4.17.

c ::= constraints | True truth | False falsity | c ∧ c conjunction | τ = τ type equality | λ = λ locality equality | λ ◁ λ accessibility of locations | λ ◀ λ definability of locations | λ = Propgt(ε, ε ′ ) effect propagation | τ = Seria Λ (α π )
type serialisation As expected, a constraint can be True or False and is equated using regular conjunctions. The constraint language allows to equate both types and localities. It also defines several predicates (Propgt, Seria) and relations (◁,◀) that are necessary to handle the annotated types associated with the particular multi-ml hierarchical architecture. We describe them in the following.

Accessibility

As the locality refers to the evaluation level of a value, all the localities are not accessible everywhere. Indeed, the locality of a value stands for, in a way, its "quantity of parallelism". Thus, all localities are not compatible with each other. In a multi-ml way of speaking, we can say that the notion of accessibility is similar to an access to a level by reading in its memory. In practice, it is basically how it works. For example, declaring a parallel vector in the scope of a leaf must be forbidden. As a leaf is a purely sequential unit, it is a nonsense to use a parallel data structure at this level. Accessing a b value at level s must also be blocked by the accessibility relation.

Definition 4.8 (Locality accessibility: ◁)

The piece of notation λ1 ◁ λ2 means that the locality λ1 is accessible (by read access) in locality λ2, that is to say «locality λ1 can be read from locality λ2».

Therefore, we need to define which read accesses are acceptable between two locality levels. The valid accessibility relations are described in Figure 4.18a. As expected, a variable annotated with the locality m is accessible from everywhere. On the contrary, level b and s are accessible in their own context only. Finally, l and c are mutually accessible in the body of a parallel vector.

The

Definability

Values that deal with special localities cannot be defined at every level of the architecture. The definition of a value that is not usable in its context and which cannot be communicated outside its context shall not be permitted. For example, it is important to reject the definition of a multi-function within a level which is not the multi-bsp (m) one. As we said previously, the multi-function is a particular construction which concerns the whole machine. It is thus senseless to define it the context of a single unit. In a sense, it is important to avoid the definition of values that are pointless.

Nevertheless, it is important to be able to define constructions that are not usable in their definition context, but that will be afterwards. For example, the definition of a function that manages parallel vectors at the multi-bsp level is not usable in its definition context. As this code deals with the bsp level, it is suitable on nodes, for the design of bsp algorithms. Such a cross-definition is particularly useful for multi-ml libraries and functions managing parallel vectors. We can interpret the notion of definability as a write access from a locality to another3 .

Definition 4.9 (Locality definability: ◀)

The notation λ 1 ◀ λ 2 means that it is valid to define a value of locality λ 1 within a context of locality λ 2 ; in other word, λ 1 can "write" in memory λ 2 .

We also need to define which "write accesses" are conceivable between two memory levels. The valid definability relations are described in Figure 4.19a. As expected, s and b are valid within m. The locality s is restricted to its own context and both c and l are inter-compatible. One can note that the definability relations c ◀ m and c ◀ b are not defined. Indeed, as the c locality comes from the scope of a parallel vector, it is not possible to explicitly declare a value with such a locality in m and b.

A ocaml code would have an unbound locality ('z for instance), which may be bound during the application.

The Figure 4.19b shows the relations between localities using directed arrows, from λ 1 to λ 2 , to specify that « λ 1 is definable in λ 2 ». In order to handle side effects, that is to say references, we need to add several constraints to guarantee safe executions. Those considerations are detailed in Section 4.4.1.

s, b, m ◀ m b ◀ b l, c ◀ c l, c ◀ l s ◀ s (

Propagation

To handle the multiple effects that can be generated by the evaluation of an expression, we need to have a relation of propagation. This relation allows to select the prevailing effect among several of the emitted effects of an evaluation. For example, the rule app (Figure 4.24) may generate different effects. Without going into the details of this rule, it is obvious that the evaluation of e 1 e 2 (i.e. e 1 applied to e 2 ) may emit two different effects; respectively ε 1 and ε 2 for the evaluation of e 1 and e 2 . In order to determine the effect of the whole expression, it is necessary to choose the prevailing effect between ε 1 and ε 2 .

Definition 4.10 (Propagation

: Propgt(ε 1 , ε 2 ))
This relation returns the prevailing effect amongst ε 1 and ε 2 .

The Figure 4.20 shows both the table (Figure 4.20a) and the lattice (Figure 4.20b) of the relations of propagation, between all localities. We can easily see a relation of superiority between levels. Some combinations are marked with a ⊥ when the relation is not acceptable. For example, Propgt(s, l) means that we are going to mix up what have been detected to be a sequential code with local code: which is not conceivable.

Serialisation

Serialisation is a safeguard assuring that all the values that are going to be communicated will be safe in any context. If we refer to the implementation, we must check that the serialisation mechanism of ocaml ( Marshal.to_bytes ) is suitable for each communicated value. Thus, it is necessary to use it with every communication primitive and also with several multi-ml constructions.

Definition 4.11 (Serialisation: Seria

Λ (τ π ))
This relation takes an annotated type τ π and returns a potentially transformed type which is safe to communicate to the given level Λ. Otherwise, it fails with an error.

The serialisation procedure is quite easy to understand: when a value of type τ π need to be transferred from a level to another, its type and locality must be inspected to approve such a communication. This verification is used to forbid values that are not communicable by essence, but also values that have no sense in a locality Λ. The serialisation also allows to transform values in order to make them acceptable in another locality. For example, an integer declared in a leaf will be typed int_s and seems to be noncommunicable. Thanks to the serialisation, and to the harmlessness of such a value, it will be transformed into int Λ . That is to say, a perfectly communicable value.

The serialisation relation Seria is defined in Figure 4.21.

It is important to note that the serialisation rules are ordered. It is important to respect the order to be able to communicate harmless values. For example, the predicate Seria c (int s ) is called when we need to communicate a integer value from a leaf to its upper node (see Section 4.3.3 for details concerning the typing rules). In such a case, as the type int can

4.3. A TYPING SYSTEM FOR µMULTI-ML 105 (1) Seria Λ (Base π ) = Base Λ if Base = int, string, float, Bool, … (2) Seria Λ (Base π ) = Fail if Base = i/o, … (3) Seria Λ (τ π ) = { τ Λ , if π ◁ Λ Fail, otherwise (4) Seria Λ (τ π par b ) = Fail (5) Seria Λ (τ π tree π ′ ) = Fail (6) Seria Λ (_ l -→ _) = Fail (7) Seria Λ (_ b -→ _) = Fail (8) Seria Λ (_ m -→ _) = Fail (9) Seria Λ (_ s -→ _) = Fail (10) Seria Λ ((τ 1 π 1 ε -→ τ 2 π 2 ) δ ) =            (τ ′ 1 π ′ 1 ε -→ τ ′ 2 π ′ 2 ) Λ , if ε / ∈ {m, b, s, l}
and τ

′ 1 π ′ 1 = Seria Λ (τ 1 π 1 ) ̸ = Fail and τ ′ 2 π ′ 2 = Seria Λ (τ 2 π 2 ) ̸ = Fail Fail, otherwise (11) Seria Λ ((τ 1 π 1 × τ 2 π 2 ) δ ) =        (τ ′ 1 π ′ 1 × τ ′ 2 π ′ 2
) Λ , and τ be communicated to any level without any worries, the rule saying that: a base type can be serialised to any locality prevails. Here, the notion of base type stands for a basic or primitive type provided by ocaml as a built-in type. In the first rule, we give a non exhaustive list of the so called base types that are harmless. On the contrary, the second rules avoids the serialisation of context sensitive base types, such as input/output channels. To complete the first two rules, the third one restricts, recursively, the communications using the accessibility relation between the locality of the serialised type π and the destination locality Λ. The rules (4) and ( 5) are useful to avoid the communication of parallel data structures: parallel vectors and trees. From rule (6) to (9), we forbid the communication of functional values using emits context sensitives effect (l, b, m and s). The tenth rule is used to serialise a functional type. To do so, we need call recursively the serialisation predicate on the left and right hand side of the functional value. If and only if each side is recursively valid, the value can be communicated. Otherwise, it fails. Finally, rule [START_REF] Blelloch | Internally Deterministic Parallel Algorithms Can Be Fast[END_REF] is used to serialise pairs and works similarly to the previous rule.

′ 1 π ′ 1 = Seria Λ (τ 1 π 1 ) ̸ = Fail and τ ′ 2 π ′ 2 = Seria Λ (τ 2 π 2 ) ̸ = Fail Fail, otherwise
As the serialisation is used to restrict the communication of harmful values, some safe code may be rejected as well.

Weakening

We need an overlay of generalisation to handle harmless values. To do so, we propose the notion of weakening. In addition to the standard type generalisation, this predicate is used to erase the harmless effects of an expression. Otherwise, the effect is kept. First, we recall the definition of a type scheme, extended with effects: Definition 4.12 (Type scheme)

A type scheme σ is the generalisation of a type in a polymorphic type system. We write σ = The weakening of the type τ π and the effect ε is performed according to the the nature of the effect in order to erase it, if it is harmless. Otherwise, the locality π of the type is kept and the effect of the type is set to π.

According the specificities of the localities, the Weakening rules are defined in Figure 4.22.

Weak(τ 1 l -→ τ 2 , ε) = τ 1 l -→ τ 2 /l Weak(τ 1 s -→ τ 2 , ε) = τ 1 s -→ τ 2 /s Weak(τ 1 b -→ τ 2 , ε) = τ 1 b -→ τ 2 /b Weak(τ 1 m -→ τ 2 , ε) = τ 1 m -→ τ 2 /m Weak(τ 1 ε -→ τ 2 , ε ′ ) = τ ′ 1 ε -→ τ ′ 2 /Propgt(ε 1 , ε 2 )
where τ We can see that the weakening of latent effects of locality l, s, b or m are spread to keep track of them.

1 ′ /ε 1 = Weak(τ 1 , ε ′ ) and τ 2 ′ /ε 2 = Weak(τ 2 , ε ′ ) Weak((τ tree) π , m) = (τ tree) π /m Weak((τ par) b , b) = (τ par) b /b Weak(τ, ε) = τ /c
For example, the code << let x = #y# in fun _ -> x >> is perfectly valid in multi-ml and the value is communicable. Let say that y is a value defined previously and typed y : int b . Even if #y# is going to emit an effect of locality l that is going to be spread to the binding of x . x is no longer dangerous inside the fun _ -> x because it is available in the environment so the closure can store the value of x . Here, the weakening is going to erase the effect of x . The type of this expression is thus

( ′ a 'z c -→ int c ) par b .
On the contrary, the code << fun _ -> let x = #y# in x >> is also valid in multi-ml but the value is not communicable. In this case, x has the effect l, because of #y# , and will be free of effect inside the right part of the let binding. Nevertheless, the effect l is spread inside the closure in order to specify that this value is not communicable. The type of this expression is thus

( ′ a 'z l -→ int l ) par b .
The communication of closures is widely discussed in Section 4.3.5.

Typing rules

The typing rules describe how our type system attributes types and effects to a given expression. In this section we define and explain all the rules that are used in the multi-ml type system.

Definition 4.14 (Typing context (environment))

The typing context Γ is a function that associate a type schemes and an effect to variables. Γ(x) denote the type scheme of x in Γ. ∅ denotes an empty environment and ; is used to

4.3. A TYPING SYSTEM FOR µMULTI-ML 107 Γ(proj) = ∀(τ, τ ′ , π, π ′ )[τ ′ π ′ = Seria b (τ π )].(τ π par b b -→ (int c c -→ τ ′ π ′ ) b ) m /b Γ(put) = ∀(π, π 1 , π 2 , π 3 , τ, τ ′ , ε)[π ◁ c ∧ ε ◁ c ∧ π 1 ◁ c ∧ τ ′ c = Seria c (τ π 1 ) ∧ π 3 ◁ c ∧ π 2 ◁ c]. ((int π ε -→ τ π 1 ) π 3 par b b -→ (int π 2 c -→ τ ′ c ) c par b ) m /b Γ(mkpar) = ∀(π, π 1 , π 2 , τ, τ ′ , ε).[π ◁ b ∧ ε ◁ b ∧ π 1 ◁ b ∧ π 2 ◁ b ∧ τ ′ c = Seria c (τ π 1 )] ((int π ε -→ τ π 1 ) π 2 b -→ τ c par b ) m /b Γ(fst) = ∀(α, β, π, π 1 , π 2 )[ ].((α π × β π 1 ) π 3 π 3 -→ α π ) m /π 3 Γ(fst) = ∀(α, β, π, π 1 , π 2 )[ ].((α π × β π 1 ) π 3 π 3 -→ β π 1 ) m /π 3 Γ(+) = ∀(π, π 1 , π 2 )[ ].((int π c -→ int π 1 ) m c -→ int π 2 ) m /c] … Figure 4
.23: Types of primitives and basic operators.

concatenate environments.

Definition 4.15 (Typing Judgement)

A judgement Γ, Λ ⊢ e : τ π /ε[c] denotes that, within an environment Γ, when evaluated at a level Λ, the expression e is typed τ π under the constraints c and emit the effect ε.

The types of the multi-ml primitives and basic operators that are needed in the environment are defined in Figure 4.23.

The application of proj now emits an effect of locality b and generates a function that associates, for each i (i ∈ p i ), the value contained in the given vector. This output value needs to be transformed by the predicate Seria as the value is going to be transferred from the body of a parallel vector to the scope of a node. This validation is necessary, for example, to avoid the projection of parallel vectors containing branches; that is to say vectors of trees.

Likewise, the output value of the put primitive must be transformed by the predicate Seria to ensure the communication of the value between the different member of the parallel vector. This transformation is particularly important with references, see Section 4.4.1. As the values are used inside the scope of a parallel vector, localities ε, π, π 1 , π 2 and π 3 must be acceptable in c. Finally, as the function that describes the received data is generated through a parallel vector, its latent effect is c.

The execution of the mkpar primitive is different compared to bsml but its type behaviour is almost similar. We recall that, in multi-ml, the function given to the mkpar primitive is executed sequentially on a node, then the computed values are distributed to the sub-nodes. Thus, the evaluation of the construction function is evaluated within locality b. The locality of the effects of the first argument must be acceptable in b. The results of the sequential executions are going to be serialised in order to be communicated. The output value is thus a parallel vector with the type τ ′ c .

The behaviour of the fst and snd operators is pretty straightforward: the latent effect is directly related to the locality of the given pair (constrained by the typing rules).

To illustrate common operators, we describes the (+) operator used for integer additions. Here, the latent effects are arbitrarily set to c as the operator is harmless. The localities of the integers given to the function are left as free variables as they will be constrained during the evaluation of the application (app) rule. Now, we need to define the typing rules that are used to guarantee the safe evaluation of a multi-ml expression. The typing rules are defined in Figure 4.24, 4.25 and 4.26. The figures are, respectivelly, corresponding to generic rule (rules that can be evaluated everywhere such as let-bindings, application, constants, etc.), bsp rules (evaluation takes place at level b or c/l) and multi-bsp rules (corresponding to the m level).

First, we describe the generic rules given in Figure 4.24. The var rule allows to type a value already bound in the environment. The locality π of the variable must be acceptable in the current environment Λ as there is no re-binding of the type τ π to the current environment.

The cst and op rules works in a similar way.

The pair and if_then_else rules are pretty straightforward. The pair rule introduces a constraint that is used to determine the locality of the final type. To do so, we use the Propgt predicate to get the prevailing locality. The if_then_else rule works in a similar way, the Propgt is used to determine the effect of the expression using both the effect produce by the condition e 1 and the expression e 2 and e 3 . As expected, the types and locality of e 2 and e 3 must be identical. let in is used to generalise the value e 1 inside e 2 . To do so, we call the predicate Weak to generalise the type and to try to erase harmless effects, or spread harmful ones. As explained in the Definition 4.13, this manipulation is used to relax the let binding rule and to avoid zealous restrictions. However, the effects emitted by the evaluation of e 1 and e 2 are propagated with Ψ. let rec is the standard rule for recursive bindings. This rules behave like the regular let in except that it adds the type of f in the environment of e 2 , to be able to make the recursive call. Likewise let in, we try to weaken the type of f to relax the binding.

deffun is going to escape the current locality Λ by evaluating e in a free locality: Λ ′ . This locality bypassing allows to declare function that is not applicable in its definition context. As said previously, it is important to be able to declare bsp code outside a node code (level b), to declare, for example, a library of bsp communication functions. However, the function definition is constrainted by ε ◀ Λ to avoid inconsistent definition such as declaring a multi-function inside a parallel vector. The output arrow type is annotated by the effect generated by the evaluation of the body of the function and the value is bound to the current context: Λ.

The app rule generate a lot of constraints in favor of the respect of the localities of the operands. We need to assure that the locality of the given argument is valid regarding the type of var f using π 3 ◁ π 1 . The effect thrown by the application must be acceptable in the current locality (ε 1 ◁ Λ). This constraint is very important and rejects, for example, the application of bsp primitives outside the scope of a node. Obviously, we also check that the locality π 2 of the type resulting in the application of the function is acceptable in the current locality Λ. The constraint ε 1 ◀ Λ is, in particular, used to check if the application of the function is possible. For example, it avoid to evaluate a multi-function (annotated -(m)-> ) in a leaf, as m ◀ b is False. This definability constraint is also necessary with references as detailed in Section 4.4.1.1. Finally, Ψ is the propagation of the effects emitted by the evaluation of e, f and its arrow effect. It is also important to note that the locality of the outgoing type is bound to the current locality Λ.

Γ(x) = τ π /ε [c] c 1 = [π ◁ Λ, c] Λ, Γ ⊢ x : τ π /ε [c 1 ] cst ⊢ cst : τ π /ε [c] c 1 = [π ◁ Λ, c] Λ, Γ ⊢ cst : τ π /ε [c 1 ] op Γ(op) = τ π /ε [c] c 1 = [π ◁ Λ, c] Λ, Γ ⊢ op : τ π /ε [c 1 ] pair Λ, Γ ⊢ e 1 : τ 1 π 1 /ε 1 [c 1 ] Λ, Γ ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ] c 3 = [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 ] Λ, Γ ⊢ (e 1 , e 2 ) : (τ 1 π 1 × τ 2 π 2 ) Ψ /Ψ [c 3 ] if then else Λ, Γ ⊢ e 1 : Bool π 1 /ε 1 [c 1 ] Λ, Γ ⊢ e 2 : τ π 2 /ε 2 [c 2 ] Λ, Γ ⊢ e 3 : τ π 2 /ε 2 [c 3 ] c 4 = [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 , c 3 ] Λ, Γ ⊢ if e 1 then e 2 else e 3 : τ π 2 /Ψ [c 4 ] let in Λ, Γ ⊢ e 1 : τ 1 π 1 /ε 1 [c 1 ] Λ, Γ; x : Weak(τ 1 π 1 , ε 1 ) ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ] c 3 ≡ [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 ] Λ, Γ ⊢ let x = e 1 in e 2 : τ 2 π 2 /Ψ [c 3 ] let rec Λ ′ , Γ; f : (τ 1 π 1 ε 1 -→ τ 2 π 2 ) Λ ′ /ε 2 ; x : τ 1 π 1 /ε 3 ⊢ e 1 : τ 2 π 2 /ε 1 [c 1 ] Λ, Γ; f : Weak((τ 1 π 1 ε 1 -→ τ 2 π 2 ) Λ ′ , ε 2 ) ⊢ e 2 : τ 3 π 3 /ε 4 [c 2 ] c 3 = [Ψ = Propgt(ε 2 , ε 4 ), c 1 , c 2 ] Λ, Γ ⊢ let rec f x = e 1 in e2 : τ 3 π 3 /Ψ [c 3 ] deffun Λ ′ , Γ; x : τ 1 π 1 /ε 1 ⊢ e : τ 2 π 2 /ε 2 [c 1 ] c 2 = [ε 2 ◀ Λ, π 1 ◁ ε 2 , c 1 ] Λ, Γ ⊢ fun x → e : (τ 1 π 1 ε 2 -→ τ 2 π 2 ) Λ /ε 2 [c 2 ] app Λ, Γ ⊢ f : (τ 1 π 1 ε 1 -→ τ 2 π 2 ) Λ ′ /ε 2 [c 1 ] Λ, Γ ⊢ e : τ 1 π 3 /ε 3 [c 2 ] c 3 = [Ψ = Propgt(ε 1 , Propgt(ε 2 , ε 3 )), π 3 ◁ π 1 , ε 1 ◁ Λ, π 2 ◁ Λ, ε 1 ◀ Λ, c 1 , c 2 ] Λ, Γ ⊢ f e : τ 2 Λ /Ψ [c 3 ] Figure 4.24: Inductive rules -Generic level. vector c, Γ ⊢ e : τ π /ε [c 1 ] c 2 = [ε ◁ c, π ◁ c, c 1 ] b, Γ ⊢<< e >>: τ π par b /b [c 2 ] copy Γ(x) = τ b /ε[c 1 ] c 2 = [τ π = Seria c (τ b ), c 1 ] c, Γ ⊢ #x# : τ π /l [c 2 ] open Γ(x) = τ π par b /ε[c 1 ] c, Γ ⊢ $x$ : τ π /l [c 1 ] op' M mlP rimitives(op ′ ) Γ(op ′ ) = (τ 1 π 1 ε -→ τ 2 π 2 ) Λ ′ /b [c 1 ] b, Γ ⊢ e : τ 1 π 3 /ε 2 [c 2 ] c 3 = [π 3 ◁ b, ε ◁ b, π 2 ◁ b, c 1 , c 2 ] b, Γ ⊢ op ′ e : τ 2 π 2 /b [c 3 ] replicate b, Γ ⊢ f : (unit π 1 ε 1 -→ τ 2 π 2 ) Λ ′ /ε 1 [c 1 ] c 2 = [ε 1 ◁ c, π 2 ◁ c, c 1 ] b, Γ ⊢ replicate f : τ 2 π 2 par b /b [c 2 ] down Γ(x) = τ b /ε[c 1 ] c 2 = [τ π = Seria c (τ b ), c 1 ] c, Γ ⊢ down x : τ π par b /l [c 2 ] apply b, Γ ⊢ e 1 : (τ 1 π 1 ε -→ τ 2 π 2 ) b /b [c 1 ] b, Γ ⊢ e 2 : τ 1 π 3 par b /b [c 2 ] c 3 = [π 3 ◁ π 1 , ε ◁ c, π 2 ◁ c, c 1 , c 2 ] b, Γ ⊢ apply e 1 e 2 : τ 2 π 2 par b /l [c 3 ] Figure 4.25: Inductive rules -bsp level. multi fun b, Γ; x : τ 1 b /ε; f : (τ 1 π 1 l -→ τ 2 c ) l /l ⊢ e 1 : τ 3 π 3 /ε 1 [c 1 ] s, Γ; x : τ 1 s /ε ⊢ e 2 : τ 3 π 4 /ε 2 [c 2 ] c 3 = [τ 3 m = Seria m (τ 3 π 3 ), τ 3 c = Seria c (τ 3 π 4 ), c1, c2] m, Γ ⊢ multi f x → e 1 † e 2 : (τ 1 m m -→ τ 3 m ) m /m [c 3 ]
Then, we describe the bsp rules given in Figure 4.25. The vector rule is necessary to generate well formed vectors. We need to check that the locality of τ and its effect are acceptable inside the vector (i.e. c). As parallel vectors can only take place at a bsp level (b), the rule forces the evaluation locality to be b. As expected, the output type is a parallel vector of type τ π par b where τ π is directly obtained by the evaluation of e. copy is the operator related to the syntactic sugar #x#. It allows to access, within the scope of a parallel vector (c), a value which is defined in the memory of the upper-node. As expected, the value annotated by the sharps must be of locality b. As the value x is communicated from b to c, we must serialise it using the Seria predicate.

Similarly, open is related to the $x$ piece of notation. It is used to access, within the scope of a parallel vector, to an element of a parallel vector. Thus, x must be of type τ π par b .

Both copy and open emit an effect of locality l to restrict their communication.

The op' rule is close to the op rule but is dedicated to the multi-ml primitives. It aims to reduce the number of typing rules, by introducing a generic one, dedicated to multi-ml primitives. The M mlP rimitives(op ′ ) match the set of synchronous and asynchronous parallel primitives: proj, put and mkpar. The definition of M mlP rimitives(e) can be found in The replicate, down and apply rules are part of the core primitives of the multi-ml language. Those rules are necessary to prove the coherency of our typing system. We recall that those rules are not available in the multi-ml syntax and thus, cannot be used by the programmer.

The replicate rule is the core primitives which builds a parallel vector from an expression f of the form (fun _ → e ). We recall that this particular expression is generated by the syntactic sugar transformation. As the function f aims to be evaluated inside the scope of a parallel vector, it must be acceptable within c. As expected, the output type is a parallel vector typed: τ 2 π 2 par b . The down rule is the core primitives which builds a parallel vector from a variable x. It aims to "copy" x inside the memory of all the sub-nodes of the current node. As expected, it works similarly to the copy rule.

The apply rule is the core primitives which aims to manipulate the construction of parallel vectors using both replicate and down. Like in bsml, apply takes e 1 and e 2 , where e 1 is a parallel vector of functions which is applied on a parallel vector of values: e 2 . multi tree fun b, Γ; x : 4.28: The multi tree fun inference rule.

τ 1 b /ε; f : (τ 1 π 1 l -→ τ 2 c tree l ) l /l ⊢ e 1 : τ 2 π 2 tree l par b × τ 2 π 3 /ε 2 [c 1 ] s, Γ; x : τ 1 s /ε ⊢ e 2 : τ 2 π 4 /ε 3 [c 2 ] c 3 = [τ 2 c = Seria c (τ 2 π 4 ), c1, c2] m, Γ ⊢ multi tree f x → e 1 † e 2 : (τ 1 m m -→ τ 2 c tree m ) m /m [c 3 ] Figure
Finally, we describe the multi-bsp rule given in Figure 4.26, which is basically the multifunction definition.

multi fun is the rule that defines a multi-function. It behaves like the declaration of a recursive function: by injecting the type of the function inside the environment of the node code. This recursive multi-function has the latent effect l to prevent communications and to force the application to be within the scope of a parallel vector. Its type is similar to the type of the multi-function itself, expect the value of the latent effect and the locality of the types. As expected, the level of evaluation of e 1 (i.e. the node code) is b (bsp) and e 2 (i.e. the leaf code) is s (sequential). The rule constraints the evaluation of the multi-function to be at level multi, in order to restrict the definition context of multi-functions. Concerning the values returned by both nodes and leaves, we need to transform them using the predicate Seria. Indeed, it is important to verify that the values that are going to be given as a result are communicable to the upper level. Namely, from s to c (when the value is communicated from a leaf to the body of the vector defined in the upper node), from b to c (when the value is communicated from a node to the body of the vector defined in the upper node, i.e. during the recursive call of the multi-function) and from b to m (when the value is communicated from the root node to the multi level).

As said previously, the definition of the rules handling trees are not detailed. However, we give the intuition on the rule used to build a tree, the multi tree fun, as the multi-ml language currently implements the tree construction. The inference rule can be found in Figure 4.28.

The multi tree fun rule allows to define multi-tree-functions, and is similar to the multi fun rule. The principal difference remains in its capacity of building a tree. To do so, the node type (e 1 ) must the be a pair containing the sub-tree (τ 2 π 2 tree l par b ) and the value of the current node (τ 2 b ). As expected, the resulting value is a function that build a tree, typed: (τ

1 m m -→ τ 2 c tree m ) m .

Examples of rule derivation

In this section, we propose several rule derivation to show how the inference rules behave. To do so, we chose some typical expressions which illustrate the specificites of the multi-ml language. To lighten the rules, we assume that the unsolved constraints of the premises are implicitly kept into the conclusion's constraints. In the final conclusion, we inject the unsolved constraints of the premises to get the complete type representation.

First, we derive an expression which aims to define a function dealing with parallel vectors within the multi-bsp level. We recall that such a feature is essential to define bsp libraries that can be used on node codes. The multi-ml typing system allows to write such crossdefinitions under some constraints. In Figure 4.29, we propose the derivation of the expression let f = fun x -> << 1 >> in f . The type inference of the expression takes place, at level m. As expected, the type inference holds as the expression is valid. The type of the given expression is thus :

(τ 1 π 1 b -→ int c par b ) m /b [π 1 ◁ b]
where π 1 ◁ b stands for an unsolved constraint which is attached to the type, until the application of the function raise a value for π 1 . This derivation illustrates how the definability predicate works.

Then, we propose the derivation of the syntactic sugar introduced in the multi-ml syntax: the $ notation. We recall that, within the scope of a parallel vector, this notation aims to copy a value defined in the memory of a node into the memory of its sub-nodes. The derivation of the expression << #x# >> can be found in Figure 4.30. To reduce the size of the derivation, we assume that the expression is directly inferred within the locality of a node (b) with an environment containing {x : int b /b}. We observe that the Seria c (int b ) is used to serialise an int b to an int c . The type infered by the rules is, as expected, int c par b /b.

Finally, a more complex derivation is proposed in Figure 4.31 to show how the attempt to communicate a non-communicable value is rejected by the typing system. To do so, we try to find a type for the expression proj << fun x -> #y# >> . As explained in Section 4.3.5, the expression fun x -> #y# is not communicable. Thus, the application of the proj primitive on such an expression must be forbidden. As previously, we begin the evaluation within the scope of a node with an initial environment containing {y : int b /b}. On the left hand part of the rule, we yield the type of the proj operator and its constraints. On the right hand side of the rule, we derive the expression << fun x -> #y# >> , which is similar to the Figure 4.30, expect from the environment. We can see that the expression is rejected during the application of proj on the parallel vector, because of the constraint of proj which consists in Seria b ((τ 3

π 3 l -→ int c ) c ).
Here we try to serialise a functional value with a latent effect l to the b level: something prohibited. Indeed, the latent effect l was introduced to prevent the communication of such values. The type inference fails and produce typing error.

Discussions on closure communication

A closure (or lexical closure) refers to an expression where free variables have been bound in the environment, resulting to a closed expression. The communication of such a term forces to serialise all the values needed by the closure, that is to say: copy the minimal environment inside the closed term. Even if it is quite complex to implement, there are no problem to use this kind of techniques on regular values. On the contrary, with parallel values, it is not the same thing.

For example, the code << fun x -> #y# >> where y is a well defined value in the node memory. Here, we have the case of a closure that uses a value, y , which is not defined at the same level. The mechanism of closure could work by binding y inside the closure. As y is not at the same level of the function declaration, the binding the value of y implies a communication: from the memory of the node to the memory of the sub-node. In terms of implementation, there is no problem for doing such a thing. However, it adds hidden communications which are disturbing for the programmer and it is not satisfying regarding the cost system. Furthermore, as the code is not executed where the closure is made but where it is evaluated; such a code could be ambiguous concerning the evaluation. The closure of this expression explicitly says that the copy of a value can be executed somewhere else, regardless of its locality. As it is ambiguous, we choose to refuse the communication of copy (and also open) by adding the effect l when the primitive is evaluated.

Type soundness

To respect the well known slogan: well-typed programs cannot "go wrong" [START_REF] Milner | A Theory of Type Polymorphism in Programming[END_REF], we must prove that programs containing a type error are not typable. That is to say, if an expression e is irreducible due to a type error, the type system must not be able to assign a type to such an

let fun vector cst ⊢ 1 : int c /c [ ] c, {x : τ 1 π 1 /ε 1 } ⊢ 1 : int c /c [c ◁ c] b, {x : τ 1 π 1 /ε 1 } ⊢<< 1 >>: int c par b /b [c ◁ c, c ◁ c] m, {} ⊢ fun x → << 1 >> : (τ 1 π 1 b -→ int c par b ) m /b [b ◀ m, π 1 ◁ b] var Γ(f ) ≡ (τ 1 π 1 b -→ int c par b ) m /b [ ] m, {f : Weak((τ 1 π 1 b -→ int c par b ) m , b)} ⊢ f : (τ 1 π 1 b -→ int c par b ) m /b [m ◁ m] m, {} ⊢ let f = fun x → << 1 >> in f : (τ 1 π 1 b -→ int c par b ) m /Ψ [Ψ = Propgt(b, b), π 1 ◁ b]
vector copy Γ(x) ≡ int b /b [ ] b, {x : int b /b} ⊢ #x# : τ 1 π 1 /l [τ 1 π 1 = Seria c (int b )] b, {x : int b /b} ⊢<< #x# >>: int c par b /b [l ◁ c, c ◁ c]
Γ(proj) = (τ 1 π 1 par b b -→ (int c c -→ τ 2 π2 ) b ) m /b [τ 2 π 2 = Seria b (τ 1 π 1 )] vector fun copy Γ(y) ≡ int b /b [ ] c, {y : int b /b; x : τ 3 π 3 /ε 3 } ⊢ #y# : τ 3 π 3 /l [τ 3 π 3 = Seria c (int b )] c, {y : int b /b} ⊢ fun x → #y# : (τ 3 π 3 l -→ int c ) c /l [l ◀ c, π 3 ◁ l] b, {y : int b /b} ⊢<< fun x → #y# >>: (τ 3 π 3 l -→ int c ) c par b /b [l ◁ c, c ◁ c] b, {y : int b /b} ⊢ proj << fun x → #y# >>: unsound/b [π 3 ◁ l, c ◁ b, b ◁ b, b ◁ b, τ 2 π 2 = Seria b ((τ 3 π 3 l -→ int c ) c )] = Fail

4.3.

A TYPING SYSTEM FOR µMULTI-ML 115 expression. We choose to follow the syntactic approach proposed in [START_REF] Wright | A Syntactic Approach to Type Soundness[END_REF] to establish the type soundness.

In the following, we propose the proof body of the type soundness. It is important to note that the proof was not mechanised nor formally proved using the coq proof assistant. It is an ongoing work. First, we prove the validity of expression within their context, which is necessary to ensure safe locality manipulation and avoid unsound parallel structure nesting, for example.

Lemma 4.16 (Validity of localities within the evaluation context)

If Λ, Γ ⊢ e : τ π /ε [c] then [c] holds. Proof : See Appendix A.2.1.
The aim of the type system of a programming language is not to ensure termination, but to guarantee the structural conformity of data. We are therefore going to show that the evaluation of a term with type τ π , if it terminates, returns a value that, semantically, does belong to the type τ π . We must therefore define precisely what does it means for a value to semantically belong to a type. To do so, we define the following predicates:

(1) |= v : τ π the value v belongs to the type τ π , (2) |= v : σ the value v belongs to the scheme σ,

(3) |= M : Γ all the values contained in the evaluation environment M belong to the corresponding type schemes in Γ.

These predicates are defined as follows:

• |= const : Base and T C(const) = Base

• |= op : σ and σ = T C(op)

• |= v : σ if for all types τ π such that τ π ≤ σ, we have |= v : τ π • |= M : Γ if Dom(M) = Dom(Γ)
and for all x ∈ Dom(M) we have |= M(x) : Γ(x) In the following, to ease the reasoning on type scheme, we use the notation ∀ -→ α -→ λ [c].τ π to stand that, for all sets of type α and for all sets of locality λ, the type τ π respects the constraint c. Furthermore, the notation ∃ -→ α -→ λ .c denotes the existence of a constraint c based on -→ α -→ λ . Those constructions are not available in the grammar of the constraints, they just aims to make the proofs more readable.

• |= (fun x → e )[M] : (τ 1 π 1 ε -→ τ 2 π 2 ) Λ if there exists a typing environment Γ such that |= M : Γ and Λ, Γ ⊢ (fun x → e ) : (τ 1 π 1 ε -→ τ 2 π 2 ) Λ • |= (multi f x → e 1 † e 2 )[M] : (τ 1 m m -→ τ 2 m ) m if there exists a typing environment Γ such that |= M : Γ and m, Γ ⊢ (multi f x → e 1 † e 2 ) : (τ 1 m m -→ τ 2 m ) m • |= << e, …,

Hypothesis 4.17 (Well typed operators and constants)

H0 For all operators op within locality

Λ, T C(op) is of the form ∀ -→ α -→ λ .(τ π ε -→ τ ′ π ′ ) Λ ′ with F(T C(op)) = ∅;
For all constants cst and for all localities, T C(cst), is a valid type of Base

H1 If Λ, Γ ⊢ v : τ π /ε and, within locality Λ, T C(op) = ∀ -→ α -→ λ .(τ 1 π 1 ε 1 -→ τ 2 π 2 ) Λ ′ , τ π ≤ ∀ -→ α -→ λ .τ 1 π 1 , ε 1 ◁ Λ and Λ ′ ◁ Λ then there exists v ′ ≡ op v such that |= v : ∀ -→ α -→ λ .τ 2 π 2 /ε 2 Lemma 4.

(Typing is stable under substitution)

Under the following hypothesis which defines the substitution:

• Λ, Γ; x : ∀ -→ α -→ λ [c 1 ].τ 1 π 1 ⊢ e : τ 2 π 2 /ε [c 2 ; ∃ -→ α -→ λ .c 1 ] • Λ, Γ ⊢ v : τ 1 π 1 /ε [c 1 ; c 2 ] • -→ α -→ λ ∩ F(c 2 ) ∩ F(Γ) = ∅ We have: Λ, Γ ⊢ {v/x}e : τ 2 π 2 /ε [∃ -→ α -→ λ .c 1 ; c 2 ]
where {v/x}e stands for the substitution of x by v in expression e.

Proof : See Appendix A.2.1.

Proposition 4.19 (Semantic generalisation)

Let v be a value and τ π be a type such that |= v : τ π . Then, for all substitution φ, we have

|= v : φ(τ π ). Proof : See Appendix A.2.1.
We now need to prove that an expression is well-formed after each evaluation steps. Thus, after a step, the previous set of constraint of a typing judgement is simplified. By definition, as Λ, Γ ⊢ e : τ π /ε [c] implies that [c] holds at the current evaluation step, we do not need to handle the constraints. If a set of unsolved constraints remains (not enough information on types or localities) they are considered holding, as there they are not unsatisfied.

Lemma 4.20 (Evaluation progress)

Let e be an expression, τ π be a type, Γ be a typing environment, M be an evaluation environment such that Λ, Γ ⊢ e :

∀ -→ α -→ λ [c].τ π and |= M : Γ. If there exists a value v such that M ⊢ e ⇓ L p v then |= v : τ π Proof : See Appendix A.2.1.

Lemma 4.21 (Co-evaluation progress)

Let e be an expression, τ π be a type, Γ be a typing environment, M be an evaluation environment such that Λ, Γ ⊢ e : τ π and |= M : Γ. Let v be a value; if it is impossible to obtain M ⊢ e ⇓ L p v; Then M ⊢ e ⇓ L p ∞.

Proof : See Appendix A.2.1.
To provide a more pleasant way to characterise "programs that do not go wrong", we now use the relation M ⊢ e⇓ L p Saf e: Let e be an expression, τ π be a type, Γ be a typing environment, M be an evaluation environment such that Λ, Γ ⊢ e : τ π and |= M : Γ. Then:

∃v such that M ⊢ e ⇓ L p v M ⊢ e⇓ L p Saf e M ⊢ e ⇓ L p ∞ M ⊢ e⇓ L p Saf e
M ⊢ e⇓ L p Saf e
Proof : By application of Lemma 4.2, we have two cases: (1) there exists a value v as a result;

(2) there is no possible resulting value. By application of Lemma 4.20 with the resulting value v, we prove the first case. Otherwise, by application of Lemma 4.21, we obtain an infinite evaluation and thus prove the second case.

Finally, using Lemma 4.4, we can also prove that the two cases are mutually exclusive that is: any well-typed program can evaluate to a value that semantically belongs to the type or diverge; so without "going wrong". Note that we do not have any information on ill-typed programs: some of them may diverge or evaluate well.

Extensions to imperative features and exceptions

In this section we propose two extensions of the µmulti-ML typing system. We propose, first, a way to handle a very useful imperative feature: references; and then, a way of managing exceptions.

Note that no proofs of the typing system's extensions has been written. As those features are standard to ml, we believe that the proofs are almost direct and mainly deals with technical issues.

A typing system for µmulti-ML ref

In pure functional programming, it is not allowed to perform side effects. It is thus not possible to change the state of a value. However, in ocaml, the concept of reference is one of the most used imperative (or impure) features. Basically, a reference is a pointer. It is possible to define a reference to a value, and change this value along the program execution.

To declare a reference, we use the keyword ref which takes any type α and produce a value of type α ref.

To modify the value of a given reference, we use the set operator, usually written As expected, references are not safe regarding to the different levels of locality of multi-ml. Indeed, changing the state of a value shared by many processes may cause unexpected behaviours without strong limitations. Thus, we need to handle references in the context of both parallel programming and multi-level definitions.

A reference cannot be modified regardless of its locality and the locality of the modification request. For example, the following code: In this code, we are going to change the value of the reference rm (declared at level m) from a parallel vector (level l). Such a reference modification is not valid. In this case we are losing the replicated coherency and the value rm is corrupted because it is impossible to know, without global communications and synchronisations, the value written in rm . To avoid dealing with such concerns, we propose some restrictions regarding references management. The same problem is also existing in the bsml language and was study and detailed, with code examples, in [START_REF] Gesbert | Développement Systématique et Sûreté d'exécution En Programmation Parallèle Structurée[END_REF].

let rm = ref

Definitions

As said previously, we need to prohibit the modification requests of a reference bound in a level different from the request locality. To keep track of the references declarations localities, we choose to add a locality annotation to the reference itself. It is necessary to keep this locality declaration unchanged during the management of the references. Otherwise, we cannot keep in mind where the reference can be read or written. Thus, a reference is now written: τ π ref Λ , where Λ stands for the locality of definition of the reference which point to a value of type τ π .

The types of the operators that manipulate references are defined in Figure 4.32.

Γ(ref) = ∀(α, π, Λ).(α π Λ -→ α π ref Λ ) m [π ◁ Λ] Γ(get) = ∀(α, π, Λ 1 , Λ 2 ).(α π ref Λ 1 Λ 2 -→ α Λ 2 ) m [π ◁ Λ 2 ] Γ(set) = ∀(α, π 1 , π 2 , π 3 , Λ 1 , Λ 2 ).(α π 1 ref Λ 1 Λ 2 -→ (α π 2 Λ 1 -→ unit π 3 ) Λ 2 ) m [π 1 ◁ Λ 2 , π 2 ◁ π 1 ]

EXTENSIONS TO IMPERATIVE FEATURES AND EXCEPTIONS 119

The references constructor ref takes any type α annotated with locality π and produces a reference, with the same type, which is bound at level Λ, i.e. the current locality. The constraint forces the given argument to be acceptable in the current location.

Operator get reads the value contained in a reference α π ref Λ 1 and returns the pointed value, bound to the current level. As previously, the constraint forces the given argument to be acceptable in the current location.

The set operator is a little more complicated. We need to check if the locality of the reference is accessible from the level where the operator is applied. For example, let rm be a reference on an integer value at the multi-bsp level: rm : int_m ref_m . This reference cannot be modified from a level which is not the multi-bsp level, otherwise the replicated coherency will not be respected. To preserve the coherency, the set rule unifies the locality declaration of the reference with the pending effect of the function that effectively sets a value to the given reference. In other words, when applied to a reference, the set operator returns a function that takes a value and set it. The latent effect of this function is equal to the locality of the reference given in argument. Thus, this function must be applied at an acceptable locality only (thanks to the app rule). As expected, the constraints force that the locality of the value sets in the reference to be acceptable: π 2 ◁ π.

It is important to note that the set operator endorses the usage of the definability predicate, written λ 1 ◀ λ 2 , used in the type constraints. In the context of typing with references, this predicate makes perfect sense. For example, in the APP rule (seeFigure 4.24), the constraint ε ◀ Λ is used to avoid an incoherent behaviour that could be raised by the set operator. Let rm be of type int_m ref_m , if we apply the set operator to the reference rm inside a node (level b): (:=) rm , we obtain a value of type (int_`z -(m)-> unit_`y)_b with the constraints ([m <| b, `z <| m]) . If we want to apply this function on a value on the node, we need to respect the constraint of acceptability of the pending effect of the function, which is m ◁ b; here, the constraint is valid. Thus, if we do not add another constraint, we have the ability to modify, from a node, a reference defined at the multi-bsp level: something unacceptable. To avoid such a case, the constraint of definability of the app rule checks if the function could have been declared, regularly, at this level. In this case, the definition of a functional value with a latent effect of locality m within the bsp level is rejected: the predicate m ◀ b is False.

Generalisation and references

It is necessary to avoid the generalisation of the type of an expression which is able to generate references. This problem has been widely studied and is not specific to multi-ml.

With ocaml, a typical code exposing the problem of polymorphic references is the following: We can easily see that the generalisation of the type of r leads to inconsistencies. The standard rule will generalise the type of r to ('a -> 'a) ref . On line 2, r := (fun x -> +(x,1)) is well typed: r : (int -> int) ref . On the third line, (!r) true is also well typed:

(bool -> bool) ref .
The whole code is well typed, but the reduction of the last expression leads to +(true,1) , which is not a valid reduction.

To make it simple, we choose to use a standard solution which involves limiting the generalisation to non-expansive expression only. The definition of a non-expansive expressions (denoted e ne ) is defined in the grammar of Figure 4.33. Less restrictive solution exists, as explained in [START_REF] Garrigue | Relaxing the Value Restriction[END_REF][START_REF] Wright | Typing References by Effect Inference[END_REF], but they generate a important modification of the typing rules. In addition to the non-expansive original let-in rule, we only need to add the let-in-ref rule for expansive expression. This rule is similar to the let-in but does not generalise the type. The updated rules are defined in Figure 4.34.

e ne ::= non-expansive expression | x identifier | c constant | op operators | fun x -→ e functions | (e ′ ne , e ′′ ne ) non-expansive pairs | op(e ne ) if op ̸ = ref | let x = e ′
ne in e ′′ ne let binding The application of the ref operator is excluded from the non-expansive expression. It is also the case for function application because we don't know if they are going to create new references (without using complex mechanisms).

let-in

Λ, Γ ⊢ e 1 : τ 1 π 1 /ε 1 [c 1 ] Λ, Γ; x : Weak(τ 1 π 1 , ε 1 ) ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ] c 3 ≡ [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 ] when e 1 is non-expansive Λ, Γ ⊢ let x = e 1 in e 2 : τ 2 π 2 /Ψ [c 3 ] let-in-ref Λ, Γ ⊢ e 1 : τ 1 π 1 /ε 1 [c 1 ] Λ, Γ; x : τ 1 π 1 ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ] c 3 ≡ [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 ] when e 1 is expansive Λ, Γ ⊢ let x = e 1 in e 2 : τ 2 π 2 /Ψ [c 3 ]
Concerning the previous example, as (fun x -> x) is not non-expansive, the code is rejected. First, the value r is typed 'a -> 'a ref is 'a not generalised. Then, thanks to the line 2, 'a is unified with int . Finally, at third line, the application of (!r) true is not valid: an expression was expected of type int , not bool .

References and communications

As references are strongly linked to a locality, their communication must be controlled meticulously. We need to extend the definition of the predicate Seria in order to re-instantiate a reference after its communication. In Figure 4.35, we can found the additional rules that need to be added to the previous definition of the Seria predicate (Figure 4.21). The communication of a references lead to its re-instantiation in a potentially different locality. Thus, we need to create a new memory zone that contains the serialised value, in the new locality, and make the reference point to it. This transformation is necessary to keep the replicated coherency concerning references. The data pointed by the reference remains unchanged but its locality is re-located. For example, at the end of a multi-function, the value returned by the root node is going to be communicate from the node (level b) to the level multi (level m). If this value is identified as a reference, it is important to re-locate it from the level b to level m.

EXTENSIONS TO IMPERATIVE FEATURES AND EXCEPTIONS

121 … Seria c (τ π ref c ) = Seria c (τ π ) ref c Seria c (τ π ref b ) = Seria c (τ π ) ref c Seria b (τ π ref c ) = Seria b (τ π ) ref b Seria m (τ π ref b ) = Seria c (τ π ) ref m Seria c (τ π ref s ) = Seria c (τ π ) ref c otherwise = Fail

A typing system for µmulti-ML exn

In ocaml, exceptional situations are handled with a system of exceptions. For example, the evaluation of 1/0 leads to an exception: Exception: Division_by_zero . In the world of sequential programming, it is easy to catch exception with a simple system of try … with … . In parallel computing, if one processor triggers an exception during a computation, we need to deal with it to prevent a crash of the whole system. Some work has been done with bsml in [START_REF] Gesbert | Bulk Synchronous Parallel ML with Exceptions[END_REF]. In this subsection, we propose an sketchy approach of multi-ml with embedded exceptions.

The main issue introduced by exceptions is when an exception escapes the scope of a parallel vector. Indeed, when an exception is thrown from the local level by, at least, one processor, the execution flow continues on other processors. When a synchronisation barrier occurs, a global barrier is missed by the processors in an exceptional state. Such a situation can quickly lead to a dead-lock. To prevent this kind of situation, a specific handler was introduce in [START_REF] Gesbert | Bulk Synchronous Parallel ML with Exceptions[END_REF]. It consist of a scope that captures exceptions, as usual in ocaml. The parallel exception handler is written trypar ... withpar ... . Within this particular structure, all exceptions are communicated between the processes during the synchronisation barrier. Then, a decision regarding the next execution step is taken, in accordance, by all the concerned processes.

Furthermore, any access to a parallel vector that is in an incoherent state, that is to say with a processor in an exceptional state, will raise the exception, repeatedly. Thus, a set of processes containing at least one process in an exceptional state, remains in that state while the exception is not handled.

In multi-ml, as the execution structure is shaped like a tree, an uncaught exception at any stage must be propagated through the architecture. Any uncaught exception thrown at a node of the multi-bsp tree is propagated to the upper level, that is to say, toward the root node during the synchronisation barrier. If an exception is not handled correctly during the evaluation of a multi-function, the exception is propagated to the multi-bsp level and all the architecture is impacted. When a node, or a leaf, is left aside in such an exceptional state, any access to this tree leads to an exception. Indeed, we want to avoid partially evaluated trees. For example, the code << if random () then raise Error else f () >> , where f is a multi-function, leads to a tree where some branches (sub-trees) are never instantiated. Accessing a partially evaluated tree immediately leads to a exceptional case, as for the parallel vectors.

A purely constraint-based type system for µmulti-ml

Constraint based type systems have been widely studied: from the Damas and Milner (dm) [START_REF] Damas | Principal Type-Schemes for Functional Programs[END_REF] which introduces a standard style were typing judgements are defined inductively by a collection of typing rules; to the well known Hindley and Milner's parametrised type system (hm(x)) [START_REF] Odersky | Type Inference with Constrained Types[END_REF], a constraint based extension of dm. Here, we concentrate on pcb(x) (Purely Constrained Based with X), which is a way to reduce type inference problems to constraint solving problems. Type inference is therefore done in two steps: constraint generation and constraint solving. This partitioning allows us to add new features to the language by only adding new constraint generation rules. As the constraint solver is independent, there is no need to modify it after adding new constructions. It is also possible to call an external solver to take care of the constraint solving. We may have used the Dalton solver proposed in [START_REF] Simonet | Flow Caml in a Nutshell[END_REF]. The current section is based on the remarkable chapter The Essence of ml Type Inference, written by François Pottier and Didier Rémy [START_REF] Pottier | The Essence of ML Type Inference[END_REF], from [START_REF] Pierce | Advanced Topics in Types and Programming Languages[END_REF]. Most of their notations and proposals are used and adapted to match the needs of the multi-ml type system.

pcb(x)

As explained in [START_REF] Pottier | The Essence of ML Type Inference[END_REF], relating pcb(x) and hm(x) is pretty straightforward. Basically, pcb(x) is an alternative presentation of hm(x).

As the pcb(x) system is based on constraints, we now use the constraint language proposed in Figure 4.36.

Constraint generation

In the standard ml, a type inference problem consists of an expression e and a type τ . We want to determine if e is well-typed with the type τ , that is to say, if the constraint C is satisfiable; then C ⊢ e : τ holds. A constraint solving problem must determine if the constraint C is satisfiable. To reduce a type inference problem to a constraint solving problem, we must produce a constraint C that is both sufficient and necessary for C ⊢ e : τ to hold.

A constraint C, written Γ ⊢ e : τ , is sufficient if Γ ⊢ e : τ ⊢ e : τ , that is to say: the constraint is specific enough to guarantee that e has type τ . Therefore, the constraint generation is sound. We write C 1 ⊩ C 2 , and say that C 1 entails C 2 , if and only if, any valid type assignment of

C 1 implies C 2 .
A Constraint is necessary if, for every constraints C, C ⊢ e : τ implies C ⊩ Γ ⊢ e : τ . Then, the constraint guaranties that e has type τ is as least as specific as Γ ⊢ e : τ . We say the constraint generation is complete.

So, if C ⊢ e : τ holds for a satisfiable constraint C, then, by completeness, C ⊩ Γ ⊢ e : τ holds. Then Γ ⊢ e : τ is satisfialbe; And conversely.

With µ-multi-ml typing system, a constraint is define as follow:

Γ ⊢ e : τ π /ε Λ . Where e is the expression that we want to type, τ π is the output annotated type, ε is the effect raised by the evaluation of e and Λ is the level where the evaluation takes place. The main constraint generation rules are described in The constraint generation rules are pretty straightforward and are similar in many points to the typing rules previously defined. Thus, we do not describe the rules in details. One can note that the let construct introduces a principal type scheme for x constrained by the constraint generated by the examination of e 1 . This is necessary to give the most general type for x.

Constraints solving

In this section we are going to talk about the unification algorithm and the constraint solver which are used in the multi-ml type inference. We need to define a unification algorithm on types and a unification algorithm on localities. In fact, the type and locality unifications are related (some localities are constrained by types) but are solved almost independently. Finally, we will present a constraint solver that is build on top of the unification algorithms.

Type unification

The unification algorithm that we present below is due to [START_REF] Huet | Résolution d'équations Dans Des Langages d[END_REF]. This algorithm is, basically, a (non-deterministic) system of constraint rewriting rules. Based on the union-find data structure, we update the syntax of constraints by replacing equation with multi-equations.

But first, we need to introduce several definitions and notations.

Definition 4.24

The notion of kind, written κ, denotes an element of the set of type variables. Roughly speaking, it is a particular type among all types.

Definition 4.25

For every kind κ, the n-ary equality predicate = n κ τ 1 …τ n is written τ 1 = … = τ n , and is called a multi-equation. A multi-equation ϵ of kind κ can be viewed as a finite multi-set of types of kind κ. The concatenation of ε and ε ′ is written ε = ε ′ .

Definition 4.26

For every kind κ, V κ is disjoint and enumerable set of type variables.

Definition 4.27

Let α, β range over the set V of all type variables. We write αβ for the set α ∪ β and α for S-ExAnd gather all the multi-equations into a single one. This rule makes the application of S-Fuse and S-Cycle possible. S-Fuse takes two multi-equations with the same variable α and fuses them into a single one. S-Stutter wipes out all the redundant variables. S-Decompose decomposes an equation between type variables with two terms where head symbols are matching. It produces a conjunction of equation between their sub-terms: ⃗ α = ⃗ β. One of the two terms remain in the original multi-equation. S-Decompose is not applicable when both terms at hand have a non-variable sub-term. S-Name-1 remedies this problem by introducing a fresh variable standing for such a sub-term. The repeated application of S-Name-1 yields a unification problem composed of small-terms. S-Clash handles equations with different head symbols. This kind of expression leads to a failure. S-Single and S-True suppress multi-equations that are tautologies. S-Cycle is the occurs check that signals a failure if the constraint is cyclic. S-Fail propagates the failure where U is a unification constraint context.

Γ ⊢ x : τ π /ε Λ = Γ(x) = τ π /ε Γ ⊢ cst : τ π /ε Λ = Γ(cst) = τ π /ε Γ ⊢ op : τ π /ε Λ = Γ(cst) = τ π /ε Γ ⊢ let x = e 1 in e 2 : τ 2 π 2 /Ψ Λ = let x : ∀τ 1 [ Γ ⊢ e 1 : τ 1 π 1 /ε Λ ].τ 1 π 1 /ε in Γ ⊢ e 2 : τ 2 π 2 /ε ′ Λ ∧ Ψ = Propgt(ε, ε ′ ) Γ ⊢ fun x → e : τ Λ /ε Λ = let x : τ 1 π 1 /ε 1 in Γ ⊢ e : τ 2 π 2 /ε 2 Λ ′ ∧ ε 2 ◀ Λ ∧ π 1 ◁ ε 2 ∧ τ Λ = (τ 1 π 1 ε 2 -→ τ 2 π 2 ) Λ Γ ⊢ e 1 e 2 : τ 2 Λ /Ψ Λ = Γ ⊢ e 1 : (τ 1 π 1 ε 1 -→ τ 2 π 2 ) Λ ′ /ε 2 Λ ∧ Γ ⊢ e 2 : τ 1 π 3 /ε 3 Λ ∧ π 3 ◁ π 1 ∧ π 2 ◁ Λ ∧ ε 1 ◁ Λ ∧ ε 1 ◀ Λ ∧ Ψ = Propgt(ε 1 , Propgt(ε 2 , ε 3 )) Γ ⊢<< e >>: τ π par b /b b = Γ ⊢ e : τ π /ε c ∧ ε ◁ c ∧ π ◁ c Γ ⊢ #x# : τ π /l c = Γ(x) = τ b /ε ∧ ε ◁ c ∧ τ π = Seria c (τ b ) Γ ⊢ $e$ : τ π /l c = Γ(x) = τ π par b /ε ∧ ε ◁ c Γ ⊢ op ′ e : τ 2 Λ /b b = Γ ⊢ e 1 : (τ 1 π 1 ε 1 -→ τ 2 π 2 ) Λ ′ /ε 2 b ∧ Γ ⊢ e 2 : τ 1 π 3 /ε 3 b ∧ π 3 ◁ b ∧ ε ◁ b ∧ π 2 ◁ b Γ ⊢ multi f x → e1 † e2 : τ m /m m = let f : (τ 1 π 1 l -→ τ 2 π 2 ) l /l in let x : τ 1 b /ε in Γ ⊢ e 1 : τ 3 π 3 /ε 1 b ∧ let x : τ 1 s /ε in Γ ⊢ e 2 : τ 3 π 4 /ε 2 s ∧ τ 3 m = Seria m (τ 3 π 3 ) ∧ τ 3 c = Seria c (τ 3 π 4 ) ∧ τ m = (τ 1 m m -→ τ 3 m ) m

Locality unification

To deal with localities constraints generated during the constraint generation phase and type unification, we need a dedicated unification algorithm on localities.

Considering an environment L that contains constraints on localities, defined in Figure 4.39, the unification algorithm will simply tell whether or not the constraint is coherent. To do so, we need to resolve all the locality equalities and propagate effects. Then we have to check if the location accessibility and definability are valid. The unification holds only if the constraint is valid. There is no need to have an algorithm as complex as the type unification one, in the sense that localities are bounded. Indeed, polymorphism on types must deal with any kind of types. On the contrary, locality unification must deal with a finite set of localities, namely m, b, c, l, s. It is important to note that, in some cases, a pending locality is not unified with non-variable locality and the solver admits that there is not enough information to solve it. The variable locality is returned as a result and the constraints concerning the locality are kept. Thereby, the annotated type is given with a constraint that indicates the nature of the accepted locality.

The principle of the unification algorithm on localities is pretty straightforward. L-Fuse takes two multi-equations with the same locality and fuses them into a single one. L-Stutter wipes out redundant variables. L-Clash raises a failure when different localities are equated. L-Loc-1 and L-Loc-2 check whether or not a locality is acceptable in another one. It raises an error when the localities are not compatible. L-Def-1 and L-Def-2 process similarly using the definability check. L-Prop computes the propagation of two localities and injects it as a new constraint.

π = ϵ ∧ π = ϵ ′ → π = ϵ = ϵ ′ (L-Fuse) π = π = ϵ → π = ϵ (L-Stutter) π = π 1 = ϵ → False if π̸ =π 1 (L-Clash) λ 1 ◁ λ 2 ∧ L → L if λ 1 ◁λ 2 =True
(L-Loc-1)

λ 1 ◁ λ 2 ∧ L → False if λ 1 ◁λ 2 =False (L-Loc-2)
λ 1 ◀ λ 2 ∧ L → L if λ 1 ◀λ 2 =True
(L-Def-1)

λ 1 ◀ λ 2 ∧ L → False if λ 1 ◀λ 2 =False (L-Def-2) λ = Propgt(ε 1 , ε 2 )∧ → λ = ε W here ε=Propgt(ε 1 ,ε 2 ) (L-Prop)

A constraint solver

We now define a constraint solver that is independent from the unification algorithms. All the type or locality unification is explicitly triggered from the constraint solver. These algorithms are inter-dependant to build a full constraint solver and their implementations are not related to each other. Definition 4. [START_REF] Ciechanowicz | The Münster Skeleton Library Muesli -A Comprehensive Overview[END_REF] We assume that every type can be branded with a kind. For every kind κ, let V κ be a disjoint, denumerable set of type variables. We write f tv(o) for the set of free type variables of an object o. The notation A#B stands for A ∩ B = ∅. The subtyping predicate ⪯ is interpreted as equality in the following rules.

We define the notion of stack, denoted S, to represent the information that is analysed by the constraint solver. A stack is basically a list of frames that are added and deleted at the inner end of a stack. The syntax of a stack is defined as following:

S ::= [] | S[[] ∧ C] | S[∃α.[]] | S[let x : ∀α[[]].T in C] | S[let x : σ in []]
A state of the constraint solver is a quadruplet S; U ; L; C where S is the stack, U is a unification constraint on types, L is a unification constraint on localities and C is an external constraint. In other words, C is the constraint that the solver will solve up next. U and L contains all the information on types, respectively localities, that have been considered since the beginning of the solver execution. S is the head of the stack which is examined. The solver's initial state is []; True; True; C where C is the whole constraint.

Implementation

In this chapter we describe the elements relative to the implementation of the multi-ml language. In Section 5.1 we propose a formalisation of the compilation scheme of a multi-ml program. Then, in Section 5.2 we introduce a natural semantics with costs which aims to be used in order to predict the evaluation cost of a multi-ml program. Section 5.3 describes the compilation toolchain of multi-ml and describes the way of implementing each step. Finally, in Section 5.4, we propose some experimentation and benchmarks that were done, with multi-ml, on real architectures.

Formalisation of the compilation

Abstract machines [START_REF] Danvy | Abstracting Abstract Machines : Technical Perspective[END_REF] bridge the gap between the high level of a programming language and the low level of a real (physical) machine. They are thus very useful to provide an intermediate language for compilation. They are called "machines" because they allow step-by-step execution of programs; and they are "abstract" as they omit details of real architectures (hardware).

The design of a generic compilation scheme for a parallel abstract machine, based on any sequential machine, has already been used for the implementation of functional languages. Regarding hierarchical machines, it ensures a greater confidence for the implementation of the compiler and thus, for the safety of the language. Such a compilation scheme can be easily extended, or may be a sub-part of a more complex machine.

In the following, we describe the abstract compilation scheme dedicated to the multi-ml language.

Current implementation

The current implementation of the multi-ml language relies on mpi. The implementation works in a spmd way and the compiler generates ocaml + mpi code from a source program written in multi-ml.

Currently, there is one mpi process for each node and leaf of the multi-bsp architecture, except for the root which executes the global part of the code. Each process can be seen as a "daemon", which is waiting for orders, to execute them. Basically, an order consists in a piece of code which must be evaluated by the concerned process. The communication between the daemons is done using asynchronous send and receive routines. The communicated values This "naive" implementation has the advantage to be quite simple to implement and relatively efficient in most cases. However, it has several drawbacks: [START_REF] Alaniz | MBSPDiscover: An Automatic Benchmark for MultiBSP Performance Analysis[END_REF] The closure to be sent can be large if it contains several ml codes; in a spmd model, one might want the necessary values to be sent only (here, the value of x ); [START_REF] Alexandrov | LogGP: Incorporating Long Messages into the LogP Model-One Step Closer Towards a Realistic Model for Parallel Computation[END_REF] In case of a tree with many levels, every node above the leaves (of the multi-bsp tree) is a process. In addition to an unnecessary scheduling, these processes are also executed by the computing units (the cores) as mpi processes. Thus, for nested processes executed of the same unit, an unnecessary sending is performed (the value is already available as they share the same computing unit).

The goals of the multi-ml virtual machine are: (1) having only one process on each computing unit, we will use continuations (threading) in place of multiple mpi processes; (2) minimise the number of daemons (and thus the extra processes); (3) abstract the communications (like in bsml) in order to have only a module of routines to implement in the future (using mpi or any low level library).

The main goal is to abstract the compiler. To do so, we will extend the traditional compilation scheme of ml with multi-ml routines. Thus, we will describe the compilation scheme of multi-ml codes on any abstract machine.

A core language

We propose, in Figure 5.1, a syntax which extends the popular core-ml.

A program is a list of expressions which are executed one after the other. Expressions contain variables, constants (integers, etc.), operators (≤, +, etc.), pairing, let, if, fun statements as usual in ml, rec for recursive calls, the bsml primitives (replicate, put, proj, mkpar, apply), local copy (copy) of a parent's variable to its child, and gid the tree of identifiers.

Finally, we define "let-multi" as a particular function construction with codes for both the nodes and the leaves. Only the expressions without free variables in replicate (fun g → e) are valid. Furthermore, only programs without free variables are valid. Free variables are defined in Figures 5.2. A replicate is well defined when the g in replicate (fun g → e) corresponds to a multi-function. We note WF _ (e) a well formed expression (induction definition in Figure 5.3) which uses WF f (e) to check if replicate is well formed or not. This is used to make sure that g is a multi-function. For example, it forbids codes like (fun f -> replicate (fun f -> e)) or the definition of (replicate (fun f -> e)) outside the scope of multi-function (where it is impossible to communicate downward the identifiers of the multi-function).

F(x) = {x} F(cst) = ∅ F(op) = ∅ F((
We do not use the (common) syntactic sugars of bsml and multi-ml and we suppose that they have been transformed into primitives calls, first:

(1) < e >≡ replicate (fun _ → e);

(2) every access $x$ (inside a vector) to the local value of a vector x has been replaced by a call of apply, e.g. < $x$ + 1 >≡ apply (replicate (fun _ x → x + 1)) x;

(3) in the same way, each local copy #x# of a parent's variable x is replaced by a call to the copy primitive.

We recall that those primitives are available in the multi-ml-core syntax; Only the syntactic sugars are available to programmers, for sake of programming simplicity.

Compilation

To get a generic compilation of multi-ml, we need a compilation that does not depend on the compilation of the sequential parts of the language. Thus we are able to generate pure ml codes with some low level communication routines. We abstract such a compilation with the notation [[e]] s , corresponding to the compilation of the term e with the standard ml compiler. This compilation generates instructions of the standard virtual machine of ml (ocaml in our current implementation). In general, such instructions manipulate a stack and some

WF f (x) = true WF f (cst) = true WF f (op) = true WF f ((e 1 , e 2 )) = WF f (e 1 ) ∧ WF f (e 2 ) WF f (let x = e 1 in e 2 ) = WF f (e 1 ) ∧ (WF _ (e 2 )) if (x ≡f ) WF f (let x = e 1 in e 2 ) = WF f (e 1 ) ∧ (WF f (e 2 )) otherwise WF f (if e 1 then e 2 else e 3 ) = WF f (e 1 ) ∧ WF f (e 2 ) ∧ WF f (e 3 ) WF f ((fun x → e)) = WF _ (e) if (x ≡f ) WF f ((fun x → e)) = WF f (e) otherwise WF f ((rec f x → e)) = WF _ (e) if (x ≡f ∨ g ≡f ) WF f ((rec f x → e)) = WF f (e) otherwise bsml-like primitives WF f (replicate (fun g → e)) = true if (g ≡f ) WF f (replicate (fun g → e)) = false WF f (copy x) = true WF f (mkpar e) = WF f (e) WF f (put e) = WF f (e) WF f (proj e) = WF f (e) WF f (apply e 1 e 2 ) = WF f (e 1 ) ∧ WF f (e 2 )
multi-functions environments (memories). We denote them E with the pid of each core of the machine (assuming an unique pid for each core of the machine) as subscript.

WF _ ((multi f x → e 1 † e 2 )) = WF f (e 1 ) ∧ WF _ (e 2 )
We note [[e]] _ m the compilation of a multi-ml expression. This compilation may have called [[e]] s , if a sub-term is free of multi-ml primitives. Without considering the type system of multi-ml, it is hard to detect such expressions statically. In our case, and without lack of generality, we prefer to apply such a compilation to e after the pass of

[[e]] _ m 1 .
The compilation of a term e is thus a compilation which transforms all multi-ml features into standard ml ones and then, compile them using the standard ml compilation. The compilation of a program P ≡ e 1 ; ; • • • ; ; e n is the compilation of all the expressions, that is,

[[P ]] m = [[e 1 ]] _ m ; ; • • • ; ; [[e n ]] _ m .
The compilation and execution can be expressed as following:

⟨⟨{E 0 , P ′ }, . . . , {E p c , P ′ }⟩⟩
where p c stands for the total number of cores of the multi-bsp machine and where

P ′ = [[[[P ]] m ]] s .
The execution scheme of a program P is based on the following ideas:

1. The code is duplicated to be executed on each core; the hierarchical architecture is thus "flattened" with the hypothesis that on the same core, only nodes and leaves of the same branch may appear. Moreover, each node is managed by one leaf. Figure 5.4 illustrates the "flattening" of the multi-ml tree on cores, using two different methods. Depending on the flattening, we can observe variations of g 0 , corresponding to the communication bandwidth between nodes 1,2 and 3. This is mainly due to the hardware capabilities of the machine;

2. The code outside multi-functions is executed sequentially, one time, for each core; 3. A multi-function is a special function which initialises some data (basically, it is defined by an identifier corresponding to the codes of nodes and leaves, stored in a global hash table of multi-functions) and run a scheduler with the appropriate code of the root node;

4. Each scheduler is waiting for two kinds of instructions: spawning a daemon and continue the execution or terminate a multi-function execution; Daemons are thus independent, which allows to have different daemons running on the same physical core; each daemon corresponds to a node or a leaf during both the "up" and "down" phase of the recursive call of the multi-function;

5. Each daemon is waiting for instructions of its parent (upper-node):

• Perform an asynchronous bsml routine such as a replicate or apply;

• Perform a synchronous routine such as proj, mkpar, copy or put;

• Complete the computation, since the parent has also completed the node code.

6. Leaf codes are running sequentially;

7. A multi-function recursive call is "just" the execution of the code corresponding to the identifier of a multi-function.

The main difficulty comes from parallel vectors. From the node point of view, a parallel vector contains a code which manages the memories of distant cores. To avoid the communication of serialised codes (which are known and shared by each core), the construction of a parallel vector is based on the following idea: we give a static identifier for each parallel vector in the program (syntactically, the code following replicate). When created, a parallel vector uses a global hash table that relates those identifiers to the corresponding codes. Then a dynamic identifier is shared between a node (parent) and its sub-nodes (children), allowing to reference a parallel vector between the two processes. It is thus possible to remove a parallel vector when it is detected as obsolete by the parent's node (using a garbage collection procedure).

A multi-function is not a simple function since it can be propagated through the whole hierarchical machine and can perform communications. A simple closure is thus not satisfying. The system must keep the trace of which the multi-function is currently running, using a global variable call CurrentIdM. We write CodeId for the identifier of each core of the machine.

A multi-function is thus compiled as following: The parameter is the static identifier used to build the parallel vector. We will explain, later, how we obtain them. Here, two identifiers are necessary: (1) the dynamic identifier of the current vector and (2) the identifier of the current (running) multi-function.When a parallel vector construction order is received (Rpl), three identifiers must be communicated to the children that are, by construction, daemons which are waiting orders. The identifier of the multi-function is used to bind the call of the multi-function in the body of the replicate. As the replicate routine is asynchronous, the Signal routine is an asynchronous send which involves insignificant costs for modern architectures and many cores. Finally the generated dynamic identifier is returned to complete the replicate. The code of apply works in the same manner, except for the use of two vectors:

let apply = fun idD 1 idD 2 -> let idD 3 = newIdD() in Signal i App(idD 1 ,idD 2 ,idD 3 ); idD 3
The code for the proj is also close, except for the use of a synchronous routine to communicate the values of the children upwards: The routine fromChildren generates an array containing all the received values of every process. The order Prj thus forces the children to communicate their values. The resulting value is thus, for each i, a function mapping of those values.

The code of the put primitive is as follow:

let put = fun idD i -> let idD o = newIdD() in Signal i Pt(idD o ,idD i ); idD i
There are two parallel vectors: one for the input, another one for the output. The Pt order forces the children to synchronise and exchange their values (similarly to the bsml semantics). On the other hand, the parent just returns the identifier of the new parallel vector since it does not directly access to the values stored on its children.

The code for copy is:

| Rpl(idD,idS,idM) -> VD[idD]:=(VS[idS] VM[idM]); daemon i () | App(idD1,idD2,idD3) ->V D[idD3]:=(VD[idD1] VD[idD2]); daemon i () | Prj(idD) -> up i VD[idD]; daemon i () | Pt(idDi,idDo) -> VD[idDo]:=(flatSend i VD[idDi]); daemon i () | Cpy (idD) -> VD[idD]:=down i ; daemon i () | Mk (idD) -> VD[idD]:=down i ; daemon i () | EndNode -> ()
In this code, we have:

• VD, which is the hash table of the dynamic identifier of parallel vectors. Respectively, VS is the shared hash table of static identifiers (from identifiers to codes) and VM the hash of multi-function identifiers;

• up which sends the value to the parent and thus synchronises the children;

• flatSend which works similarly to the bsml Send (also for the put primitive);

• down which allows to receive the values of the parent in a synchronous way.

The code works as follow. When the replicate order is received, we create a new parallel vector with the given information. The apply order works similarly. Regarding the proj order, we communicate upwards the value of the vector, by reading the hash table. For the put order, we perform the all-to-all exchanges. Finally, for the copy and mkpar orders, we receive the values sent by the parent in order to build, and update, the hash table of parallel vectors.

Note that using only one daemon for simulating the execution of upper nodes seems impossible. Indeed, if a process must manage two nodes, the daemon simulating the first node may be waiting for the results of its children, which could be itself. Moreover the second node could also wait for the results of its own children, making the core in a situation of deadlock if no more than one thread is active. To avoid such problems, we chose a solution where, on each core, only one thread is active during all the execution of the multi-ml code.

The compilation [[e]] _

m of an expression e is defined in Figure 5.5 and it works as follows: • Most sequential constructions are left as they are or are build over trivial inductions. This is the case of constants, operators, pairing, conditional, binding and functions;

• At the beginning, we are not within the scope of a multi-function binding. But after the declaration of a multi-function, we must select the appropriate identifier which is given to the compilation function; • When executing the replicate primitive, g stands for the current evaluated multi-function and thus, it is used to define where the call to the CurrentIdM must be done. Moreover, we update the hash table of vectors: H.

Finally, the compilation of a program is performed with H ≡ ∅. Note that our core-language does not allow the definition of multiple multi-function at the same time. It is thus possible to declare one multi-function at a time. To exceed this limitation, an appropriate set of bindings must be given as arguments of replicate, that is all the potential multi-function calls. This feature is discussed in future works.

Execution and Correctness

The execution of a compiled program, as a virtual machine, is done using a small-steps semantics. It consists of a predicate between an expression and another expression defined by a set of rules called steps.

As usual, we have two kinds of reductions. One for the cores and one for the whole machine. We note ⇒ c the local reduction of one expression on a single core and ⇒ the reduction of an expression for the whole machine. For both rules, ⇒ * is a reflexive and transitive closure (defined using an induction) and ⇒ ∞ is used for infinite programs (defined using a co-induction). The reduction of a program P = e 1 ; ; • • • ; ; e n is the reduction e 1 followed by the reduction of e 2 etc.

For sake of simplicity, we do not give the reduction rules of ocaml expressions, but we focus one the reduction of communication routines needed for the implementation of multi-ml. Note that sequential codes do not modify the communication environments nor messages queues. An environment E contains at least two independent queues of orders, one (denoted F s ) for the schedulers and another one (denoted F d ) shared by the daemons. We write e for a thread It is easy to observe that we cannot always make a head reduction of an expression. We have to reduce, in depth, the sub-expressions. To define such a depth reduction, we define two kinds of contexts: (1) An expression Γ i with a "hole" denoted [] which has the abstract syntax given in Figure 5.6 where i is the gid of the expression; [START_REF] Alexandrov | LogGP: Incorporating Long Messages into the LogP Model-One Step Closer Towards a Realistic Model for Parallel Computation[END_REF] We then define Γ i j as the context of each thread where the reduction holds. For the core j at gid i, {E j , Γ i j [e]} stands for {E j , e ′ , . . . , Γ i [e] , . . . e ′′ }.

Γ i ::= [] head evaluation | Γ i e application | vΓ i application | let x = Γ i in e let | (Γ i , e) left pair | (v, Γ i ) right pair | if Γ i then e else e conditional
From the previously defined low level routines, we have the following rules: toChildren and down On the contrary, we also have a synchronous operation to communicate downward a value (from a parent to children) and thus, we have the following rule:

⟨⟨. . . , {E 0 , Γ i 0 [down()]}, . . . , {E p , Γ i p [down v]}, {E, Γ ′i j [ToChildren v()]}, . . .⟩⟩ ⇒ ⟨⟨. . . , {E 0 , Γ i 0 [v]}, . . . , {E p , Γ i p [v]}, {E, Γ ′i j [v]}, .
. .⟩⟩ where {0, . . . , p} are the gid i of the branch of the children of the parent on core j.

flatSend This synchronous low level primitive is the one used for implementing the put primitive. flatSend takes an array of (functional) values to send and returns an array of received values: As it is purely asynchronous, one daemon may be in a situation where it has received all the messages before executing the orders. Thus we define the rule of reception:

⟨⟨. . . , {E 0 , Γ i 0 [
⟨⟨. . . , {E j ⊕ | . . . v | i Γ i ′ j • • • Γ i j [rcv i ()] • • • , . . .⟩⟩ ⇒ c ⟨⟨. . . , {E j ⊕ | . . . | i Γ i ′ j • • • Γ i j [v] • • • , .
. .⟩⟩ On core j, the daemon of gid i (in a thread) reads the message in its own queue. Note that those rules are not global since they only rely on one thread of one unique core instead of synchronous rules that need the whole machine to work.

WakeUpChildren, WakeUpAll and wake They are working in the same way, but where the queue F s of the schedulers and WakeUpAll sends the message to all cores, WakeUpChildren sends the messages to its Children, similarly to toChildren. We do not give the rule for sake of conciseness.

run Finally, we have a routine to run threads (daemons):

⟨⟨. . . , Γ i ′ j • • • Γ i j [run d k ] • • • , . . .⟩⟩ ⇒ c ⟨⟨. . . , Γ i ′ j • • • Γ i j [()] d k • • • , .
. .⟩⟩ Where, on core j, for gid i, we run the daemon for gid k.

When a thread ends, it may have produced an orphan value. When such a value is not communicated upward using the proj primitive, there is no need to keep them. Thus:

⟨⟨. . . , e • • • v • • • , . . .⟩⟩ ⇒ c ⟨⟨. . . , e • • • , . . .⟩⟩
Finally, the asynchronous reduction ⇒ c (for standard ocaml expressions) can only work in the context of a call of the ⇒ reduction on the whole machine, using the following rule:

⟨⟨. . . , {E j , • • • Γ i j [e] • • • }, . . . , ⟩⟩ ⇒ ⟨⟨. . . , {E j , • • • Γ i j [e ′ ] • • • }, . . . , ⟩⟩ If Γ[e] ⇒ c Γ[e ′ ]
uses a standard reduction rule of ocaml (matching, if-then-else, application, binding, pairing, etc.).

Note that all the low level communication routines only works when sending valid values, as in the operational semantics. We do not introduce a validity test for values to simplify the formalisation. Now we have the following results that could be proved by induction and co-induction. The generated code works similarly to the multi-ml's big step semantics. If all the expressions give values, then the code finally gives a final value. Otherwise, if one expression diverges, then the whole machine is considered to be diverging.

Distributed Garbage collector

Distributed garbage collection is known to be non trivial. In multi-ml, the main concern about garbage collection is related to the parallel vectors. Indeed, when a node builds a parallel vector, the values are actually stored inside the memory of the sub-nodes. Thus, after each recursion of the multi-functions, the hash table which contains the dynamic identifier (linked to values) are growing bigger. As the hash table cannot be automatically freed by the ocaml garbage collector, we must clean the table manually. To do so, the simpler solution may consist in a system which can determine the maximal life time of a parallel vector. In the context of tree construction (using multi-tree-function) we cannot say that a parallel vector's life is over when the node code that built it ends. Indeed, the final tree may need this particular value, at the end, to build the resulting tree. In this case, we have to wait until the end of the multitree-function itself to delete the values which were used to build a tree. The other parallel vectors can be deleted at the end of the execution of the node which built them. Then, all the parallel vectors built using an identifier which is "older" than the last multi-tree-function call are declared as garbage collectable. During the recursion of the next multi-functions, we can imagine a mechanism, on each process, that updates and cleans its own memory. We could also use the finalize function of the ocaml garbage collector to warn the processes when a value is garbage collected.

As such a feature is considered as an implementation detail which aims to optimise the memory footprint of a multi-ml program, it is not yet implemented in the current multi-ml version.

Limitations of the implementation

Currently, the multi-ml implementation has several limitations. In the following, we present two of the principal restrictions.

To reference a value defined in the multi-bsp memory from the body of a parallel vector, we must declare it as following: If the values are not declared as a list of expressions, it is not possible to reference them within the scope of a parallel vector. The value must be completely evaluated otherwise, it is not possible to determine which identifier corresponds to a particular value. Indeed, in the case of imbricated bindings, it is not possible to determine a global identifier to reference a value. It is thus impossible to tell to a process which value to use.

The multi-function nesting is not yet available. Indeed, as we said previously, a multi-function is not as simple as a function. It implies additional runtime systems to be executed all over the multi-bsp architecture. A multi-function needs an initialisation phase in order to be executed. Thus, the following code is not valid: The multi-function g is not initialised in the expression << f()+g() >> , as the expression is in the body of f . This multi-function composition has been studied and we propose, as future work in Section 6.2.6, solutions to be able to do it.

Modular Implementation

As an abstract machine is defined onto elementary blocks which aims to be dedicated to simple tasks, it is easy to have a modular implementation of the multi-ml language. Such an abstract and generic design is interesting for the evolution of the language. The development is easier and it is possible to propose different implementations of the language to be able to execute multiml programs on a wide variety of architectures. For example, as it is detailed in Section 5.3.4.1, it is possible to propose communication modules relying on different communication libraries.

Big step semantics with costs

Thanks to the multi-bsp cost model, it is possible to compute the evaluation cost of a given expression. This cost gives an estimation of the execution time of an expression according to the machine specifications. It is even possible to propose a static analysis of a program which gives an approximation of its execution time.

The combination of the big-step semantics and a simple cost algebra allow to make such a performance prediction. In this section we add costs to the big-step semantics introduced in Chapter 4. Thus, the rules behaves as previously.

An inference rule with costs annotations can be written as following:

P M ⊢ e ⇓ L p e ′ /C p and P M ⊢ e ⇓ b
p e ′ /C p / < C p.i > We can distinguish two ways of annotating rules by costs. First, the left hand side rule stands for evaluations which do not deal with bsp computation, that is to say: (1) generic rules (which can be executed a any level of the architecture); (2) local rules (which can be executed inside the scope of a parallel vector) and (3) multi-bsp rules (which are executed globally, at the multi-bsp level). Here, C p describes the amount of cost a component itself, at level p. When dealing with rules which can be evaluated at level m, we consider the whole machine as a single component where its cost is C p_multi . This cost is the sum of all the costs of all the components of the multi-bsp architecture.

The rule on the right hand side handles bsp execution: nodes evaluations. As a node is a particular component where it is possible to express bsp parallelism, we must consider the asynchronous costs generated by bsp computations. To do so, in addition to the cost inherent to the node itself, we keep track of the sub-nodes costs (that were engenders by the evaluations taking place at level p) by using a parallel vector of costs. This vector, denoted < C p.i >, lists all the cost of the i components of p: < C 0 , …, C n >. We recall that, for readability, we use n to refer to the number of sub-nodes of a component at level p.

Apart from the cost annotations, the semantics rules behave in a way which is similar to the standard rules (defined in the Section 4.2 of Chapter 4)

To manipulate the costs within the rules, we use a simple cost algebra which is defined in Figure 5.7.

Definition 5.2 (Cost algebra)

The cost algebra is used to accumulate the cost of evaluation of the semantics rules. The cost algebra relies on integer values and uses simple operators.

The constants g and l, correspond to the standard multi-bsp parameters that are used to, respectively, quantify the bandwidth and the synchronisation cost. To lighten the rules, we assume that g and l are standing for the multi-bsp parameters corresponding to the level of evaluation of the rule. Thus, at level p, they correspond to g p and l p . We also use the notation S ε (v) to compute the size of a value v within environment ε. This function is useful when data transfers are required. A simple data size computation is given in Figure 5.8. We can imagine a more realistic version with allocation costs, garbage collections costs and where the closures costs are propotional to the size of the enclosed environment. It is important to note that, as closures are intended to be communicated, the operator is not defined on parallel primitives, parallel vectors nor trees. For sake of simplification, this function is an approximation of the real size of data. We also consider that the environments of closures are minimal environments, by construction. For example, the environment E ′ of ((fun x → e) [E ′ ]) only contains free variable of e. In Figure 5.9 we propose the generic rules that can be evaluated regardless of the locality.

C ::= | 1 Arbitrary unit cost | g g parameter | l l parameter | C ⊕ C Addition | C ⊗ C Multiplication | n max i=0 (C i ) Maximum | n ∑ i=0 (C i ) Sum | S ε (v) Data size (a) Grammar. C 1 ⊕ C 2 ≡ C 2 ⊕ C 1 C 1 ⊗ C 2 ≡ C 2 ⊗ C 1 n max i=0 (C i ) ≡ maximum(C 0 , …, C n ) n ∑ i=0 (C i ) ≡ C 0 ⊕ … ⊕ C n (
S(c) = 1 S(op) = 1 S E (x) = S(v) if {x → v} ∈ E S E (e 1 e 2 ) = S E (e 1 ) ⊕ S E (e 2 ) ⊕ 1 S E (e 1 , e 2 ) = S E (e 1 ) ⊕ S E (e 2 ) ⊕ 1 S E ([e o , …, e p ]) = p ∑ i=0 (S E (e i )) S E (fun x → e ) = S x•E (e) ⊕ 1 S(E) = p ∑ i=0 (S(v)). ∀x ∈ E : {x → v} S E ((fun x → e) [E ′ ]) = S x•E ′ (e) ⊕ 1 ⊕ S(E) S E (multi f x → e 1 † e 2 ) = S x•f •E (e 1 ) ⊕ S x•E (e 2 ) ⊕ 1 S E ((multi f x → e 1 † e 2 )[E ′ ]) = S x•f •E ′ (e 1 ) ⊕ S x•E ′ (e 2 ) ⊕ 1 S E (if
The rules values and op_eval does not generate any additional cost as their evaluations are almost syntactical. With the var rule, we need to lookup for variables in the environment. The lookup() predicate is defined in Figure 4.7. For sake of conciseness, we choose to evaluate the cost of lookup() as an atomic cost of one.

The closure and rec_closure rules only rely on the size of the final closure. According to the size of the environment, we approximate the evaluation cost of the closure. One can imagine a finer approach taking into account the number of variables stored in the enclosed environment. In this semantic, such a coarse cost approximation is plenty enough.

The rest of the generic rules are simple to handle. We just compute the final cost using the cost of the premises. We add an extra cost when the rule needs an additional operation for the evaluation. For example, the fun_app requires to push values to the top of the memory stack to proceed the evaluation. Thus, we estimate such a cost to one. The Figure 5.10 gives an overview of the bsp rules. Those rules are the core of the multi-ml language and they concentrate most of the costs relative to parallelism. As we are dealing with bsp rules, the cost is now expressed with two quantifiers: (1) the cost of the current node and (2) the cost of the sub-nodes, generated by asynchronous computations.

The cost of the replicate rule is highly related to computation time of the sub-nodes. Each ith sub-node of the current node p performs the computation for a cost C i . Thus, the cost engendered by such an asynchronous computation is added to the current sub-nodes cost C p.i . A cost of 1 is added and correspond to the signal of evaluation which is sent to the sub-nodes. This operation is widely discuss in the implementation chapter (Chapter 5) and is necessary to handle multiple implementations.

The down rule cost is simple. The value v is communicated to the sub-nodes with a cost relative to its size, namely S S (v). As the down primitive must communicate a value from a node to its sub-nodes, we must execute a synchronisation barrier. In addition to the cost due to the bandwidth and the latency, we propagate the asynchronous computation costs of the sub-nodes by adding the n max i=0 (C p.i ). Thereafter, the sub-node's cost is set to zero. On the top of that, we just add lookup cost of one concerning the variable evaluation.

The mkpar rule is similar to the replicate rule in the sense that it creates a parallel vector. Nevertheless, its cost is quite different. It is composed by the sum of the i computation costs (C 0 to C n ) plus the sum of the v i transfers costs. Indeed, as the v i values are computed sequentially on the node before, we need to take into account both the i computations costs plus their communication cost. The asynchronous costs are propagated, as a synchronisation barrier is done. It is conceivable to imagine a finer cost estimation by considering the overlapping of both communications and computations.

As the evaluation of the apply rule is done asynchronously, we simply add the evaluation costs of e 1 and e 2 . Then, asynchronous cost are added on each sub-components of the node p.

The cost of proj is related to the evaluation cost of e. As all the values of the parallel vector < v 0 , …, v n > must be communicated to the memory of the node p, the cost consists in the sum of communication costs of those values. In addition to that, as proj is synchronous, we add the n max i=0 (C p.i ), and set the sub-node's costs to zero.

The put cost is quite dense because of the number of communications which are done during the evaluation of the primitive. Thus, the expression of C put is the following:

C put = C p ⊕ 1 ⊕ C 1 ⊕ n max i=0 (C 1.i ) ⊕ n max i=0 (C i ) ⊕ n max i=0 (C j ) ⊕ n max i=0 (max( n ∑ j=0 (S M (v ij )), n ∑ j=0 (S M (v ji )))) ⊗ g ⊕ l
First, we need to add the evaluation cost of e. Then, C i and C j describes, respectively, the cost of building the value to send and handling the received value. As such computations are done in parallel, we took the maximum of C i and C j costs. We also need to take into account the communication costs of all those values. To do so, for each i, we select the maximum value between the cost of sending a value from i to j and the cost of sending a value from j to i. Finally, we take the maximum of all the ij exchanges.

The Figure 5.11 gives the local rules. Here, we compute the cost of the multi-function's recursive call cost.

Here, only the multi_node and multi_leaf rules can be evaluated. The costs of the multifunction recursive call taking place on both the node and the leaf is simple. We just add the evaluation cost of e 1 and e 2 , plus the multi-function call cost, resulting in the recursive call. The multi_node rule adds the C i costs which result from the potential asynchronous computations done on the node. Thus, we collect all the costs engendered by multi-function recursion. As expected, this mechanism is not necessary on the multi_leaf rule, as there is no parallel computation at this level.

Finally, Figure 5.12 describes the rules that are evaluated at the multi-bsp level.

The cost of multi_def is directly related to the size of the multi-function closure. As before, for conciseness reason, we estimate this cost at one.

Concerning the multi_call rule, we simply add the evaluation cost of e 1 and e 2 , plus the multi-function call cost. Like the multi_node rule, we collect all the costs engendered by multifunction recursion at the root node. It is important to note that, as the multi-bsp memory may be distributed, we must take into account the cost of broadcasting the value v ′ to all the component of the multi-bsp architecture. In practice, the broadcast(v ′ ) cost can be done in a logarithmic way.

We now have the description of a semantics with costs. However, we do not propose an abstract machine relying on this cost. As it seems interesting and as it does not appear to introduce difficulties, it is reserved for future work.

The compilation toolchain

The compilation toolchain structure is, most of the time, structured by three blocks. A simple toolchain could be structured as follow:

(1) The frontend checks the syntax and the semantics of the language, according to its specifications. It could also perform type checking and code analysis to collect informations such as types, errors and warnings. The output of this phase results in an abstract syntax tree (ast).

(2) The middle-end takes the ast produced by the front end and performs transformations or optimisations.

(3) The backend takes the output of the middle-end to produce a machine code, suitable for a particular architecture. It is then possible to execute this code on a machine.

In the context of the multi-ml language, the frontend performs the parsing and lexing phase, to make sure that the multi-ml syntax is respected. Then, the type checker is called to ensure the execution safety of the program. The middle-end consists in a code transformation which takes the abstract syntax tree as input. Then, depending on the requested operation, it transforms the ast by adding parallel or sequential instructions. Finally, the backend compiles the given ast to produce a parallel (or sequential) executable. Figure 5.13 summarises the compilation steps.

To build the multi-ml toolchain we choose to start from the ocaml 4.02.1 sources. Then, we modified the ocaml parser and lexer, by reducing its expressivity and adding the multi-ml constructions (see Section 5.3.1). Then we choose not to modify the typing system of ocaml. Indeed, it is a twisted and rather complicated piece of code which is difficult to handle. We choose to implement our own type checker, as explained in Section 5.3.2. Finally, the code transformation is described Section 5.3.3 and the way that multi-ml programs are executed in parallel or sequential is explained in Section 5.3.4.

One may wonder why we did not use the ocaml tools which aims to extend the language. For example, camlp5 is a preprocessor and pretty-printer for ocaml programs. Its preprocessor allows to extend the syntax of ocaml and redefine the whole syntax of the language. The pretty printer allows to convert one syntax to another and check the results of syntax extensions. Such a tool seems to be convenient. However, it is not that easy to use. For example, bsml is build over camlp4 and its maintenance was difficult. Indeed, camlp4 introduced a backward-incompatible update which was very laborious to adapt in order to get a working version of bsml. To avoid such inconvenience, we choose not to use those tools. Of course, as multi-ml is based on a particular version of ocaml, all the new syntax or type features introduced by new versions would not be available in multi-ml. As our language does not allow the cutting edge ocaml features, it does not matter. Moreover, if compilation optimisations involving performance gains are added to a future version of the ocaml compiler, we will benefit from them, as the ocaml compiler is directly used, in the backend.

One unfortunate thing of this approach is the impossibility of bootstrapping multi-ml, that is, to be able to recompile multi-ml sources with the newly created compiler.

Lexing and parsing

The first phase of the syntactical analysis is lexing. Lexing is a simple process which tokenizes a stream of text: it breaks a program in words or phrases. Basically, the lexer identifies all the tokens defined in the language (such as let, fun, if, mkpar, etc.) and generates a new stream of tokens: the parse tree. The definition of the multi-ml lexer can be found in the file lexer.mll. Thanks to ocamllex, which produces a lexical analyser from a set of regular expressions, we obtain a standalone lexer. Then, we must parse our program, using the lexical analyser generated by ocamllex. To do so, we use the ocamlyacc command, which produces a parser from a context-free grammar specification with attached semantic actions, in the style of yacc. From the definition of the grammar, which can be found in file parser.mly, we can produce a parser, written in ocaml.

As previously explained, we took the original ocaml lexer and parser and we modify them to be able to recognise the µmulti-ml language.

Without going into details, we must also define all the structures needed to handle the multi-ml particular constructions. For example, we enrich the parse tree structure (which is declared in file parsetree.mli) by adding expressions such as trees, multi-functions and all the multi-ml primitives. Thus, the ast produced as an output of the parsing phase is now available for type checking.

Type system implementation

As explained in Chapter 4, the type inference consists in two phases: constraint generation and constraint solving.

From a freshly produced ast, we are able to generate a set of constraints. This set is composed by the constraints on types and their localities, directly obtained by the constraint generation rules, derived from the inference rules.

The constraint set produced is then given, as input, to the constraint solver. It aims to decompose the constraint and, thanks to a constraint solving algorithm, a type unification algorithm and a locality unification algorithm, solve the constraints.

As the ast produced in the frontend respects the syntax, there is no reason for the failure of the constraint generation phase. However, the type errors will be raised during the constraint solving phase.

It is known that generating a "good" error message from a type error is an hard task. As explained in [START_REF] Charguéraud | Improving Type Error Messages in OCaml[END_REF], there exist many ways to improve the message related to an error. Here, we do not aim to improve the ocaml type error messages. However, as typing multi-bsp programs introduces a large amount of potential errors related to parallelism, we have tried to give specific error messages. Thus, to ease the understanding of typing errors and help the programmer to resolve them, we have introduced a new set of errors. In addition to the regular ocaml typing errors (such as Unbound Values, Wrong types, etc.) we have defined:

• Wrong_Locality: raised when a locality is not acceptable. It is the case if the accessibility or definability predicated are not respected;

• Loc_Unification: raised when the unification between two localities is not acceptable (incoherent localities);

• Vector_Imbrication: raised when a nested parallel vector is detected. It is, basically, a specialisation of the Wrong_Locality error;

• Non_Communicable: raised when a noncommunicable value tries to be communicated.

Those type error messages are not always very accurate, due to the fact that the locality unification may occur at an unexpected time, and thus rely on a locality constraint introduced by an expression far away from the location of the displayed error. However, this set of errors gives significant information helping to correct a piece of multi-ml code.

Code transformation

The code transformation is an important step occurring during the middle-end phase. It consists in the transformation of the (intermediate) ast produced by the frontend. As explained in Chapter 4, we need to transform the program written using the multi-ml syntactic sugar into a core language. Thus, the code is expressed with a small set of primitives which are implemented depending on their purpose (parallel or sequential code generation). For the parallel code transformation, we obtain an multi-ml program, written in ocaml, with a communication module using the mpi routines. This code can be compiled by the standard ocaml compiler and the c compiler for mpi: mpicc.

Code execution

In this section we briefly explain how the current multi-ml implementation works and how it could be improved.

Parallel execution

As multi-ml aims to be executed on any hierarchical architectures, we propose a system of generic functor which takes a configuration file as input. This file contains all the information of an architecture, shaped like a tree. We can distinguish the depth, the components and their siblings and also the capabilities of each level, based on the latency, bandwidth and memory size.

To be compatible with a wide set of parallel systems, it is also possible to use different communication libraries. To do so, the communication module is based on a generic functor which takes a communication module as a parameter. For now, the mpi version is the only communication module available. Nevertheless, as the communication primitives are based on a minimal set of instructions, it is easy to write several communication modules based on, for example, the pub or tcd/ip, as it is done in bsml.

After the compilation of a multi-ml code using the multimlopt.mpi compiler, it is possible to execute the program on the hierarchical architecture. To do so, we need to execute one instance of the produced executable on each computing unit of the machine. As the binary produced is a spmd program, the same binary is executed everywhere. Only their initialisation values distinguish them. The mpi tools and this configuration files handle this initial step, seamlessly.

To simplify the multi-ml code generation we use closures to communicate values through components. Thus, when a multi-function is called recursively within the scope of a parallel vector, we communicate, to its sub-nodes, a closure containing the multi-function code plus its arguments. As the generated code is spmd and the type system ensures that the multi-function is declared at the multi-bsp level, there is no need to do such a thing: the multi-function code is already known by components, only the arguments need to be communicated. As explained previousely, a solution to avoid the multi-function communication can consist in referencing all the multi-functions with static identifiers. Then, only the identifiers will be communicated. The same idea can also be used to reference all the multi-bsp value. Furthermore, it is also possible to avoid the communication of the code within the scope of parallel vectors. To do so, we need to generate, at runtime, dynamic identifiers to reference parallel vectors between a node and its sub-nodes. In combination with the static identifiers, we avoid useless code communications. Such a technique would drastically decrease the size of the communicated values, and would avoid to communicate values that are already known by the receiver. As this technique requires to develop a global addressing (to reference values with common identifiers) with dynamic addressing between two levels, its implementation is not trivial. To ease the implementation of multi-ml, we choose not to develop this ingenious solution.

In the current multi-ml implementation, we do not use the shared memory as it could be. Indeed, all the communications are done using the serialisation of ocaml: Marshal. This operation takes an expression and produces a closed term which is possible to communicate to another process. This operation imposes to copy a piece of data and communicate it. When a communication is done between two components accessing the same memory, such an operation is useless. Without side effects, we could simply give, to the receiver, a pointer to the data. The usage of mmap, a posix api which allows to share a memory through processes using a memorymapped file I/O, could drastically increase the performance of the components accessing a shared memory.

Another concern is the garbage collection. The ocaml garbage collection is designed to be executed with sequential programs. In this context of multi-bsp programming, we must deal with parallel data structures which cannot be collected by the standard ocaml garbage collector. Ideally, we could have a dedicated parallel garbage collector. However, writing such an algorithm is not a trivial task [START_REF] Doligez | Portable, Unobtrusive Garbage Collection for Multiprocessor Systems[END_REF]. For example, we should be able to clean all the parallel vector. To do so, a node must notify all its sub-nodes that a parallel vector will not be used any more. It requires a fine collection mechanism and a lightweight system of notification between nodes. Currently, the multi-ml implementation does not deal with the parallel garbage collection.

As the current implementation of multi-ml relies on a set of daemons which are running for each components of the multi-bsp architecture, the number of process instances is high. This method simplifies the way of implementing multi-ml but could be improved. For example, we can imagine that all the leaves will run only one process (instead of n > 1). This process will schedule a set of components composed by the leaf itself, plus some of the top nodes. Thus, the leaf process would dynamically switch its execution contexts to evaluate several stages of the architecture. Such an implementation may allow communication optimisations and better performances. However, such a method may introduce problems such as process interleaving and potential deadlocks.

The interactive loop

The interactive loop, commonly referenced as toplevel, is a very useful ocaml feature. In this mode, the system repeatedly reads ocaml phrases from the input, then typechecks, compiles and evaluates them, then prints the inferred type and result value, if any. Such a feature is very interesting to test and debug auxiliary functions of a more complex program. It is possible to check that the type inferred by ocaml is the one we expect, and then, we can test the behaviour of a piece of code.

As parallel programming may introduce more complex behaviour, the toplevel seems to be almost essential. In bsml, an interactive loop is available to test bsp programmes. It seems mandatory to propose the same feature with multi-ml.

Consequently, we propose a toplevel to interactively evaluate multi-bsp algorithms. As multi-ml can deal with complex hierarchical architectures, we propose to run the toplevel with a given architecture. Thus, it is possible to test the behaviour of a multi-ml program on a virtual hpc machine, or on any imaginary architectures. For example, we can trace the data load balancing of our algorithm on various architectures, in the blink of an eye. To declare a new architecture, we just need to define a tree using the ocaml style, as shown in Figure 5.14. Here, for each node and leaf, we give a triplet to denote the latency (in flops), bandwidth (in flops) and memory capacities (in bytes): (l, g, m). In this example, we describe an architecture of depth 3 with: The tree structure can be visualised through the interactive loop. As the toplevel point of view is directed to the multi-bsp level, all the executions start within locality m. In Figure 5.15, we combine the use of the mktree primitive (which builds a tree from an expression) and the gid (which returns the global identifier of a component) to build a tree of gids. The output value t is thus displayed, in the toplevel, with the shape of a tree where the nodes are represented by a circle ( o ) and the leaves by an arrow head ( > ). The depth and hierarchy of the tree is symbolised using separation characters and indentation.

Other standard bsml, and thus ocaml, features work as expected.

Experimentation and benchmarks

In this section, we propose several examples of multi-bsp algorithms. As expected, they are written using the multi-ml language and are executed on real architectures.

The execution platforms

To test the performances of the algorithms, we use two clusters. Those machines are thus composed by both shared and distributed memories. The machines used are named mirev2 and As previously, the version 4.02.1 of ocaml, gcc 4.9 and mpich 3.1 were used. To estimate the multi-bsp parameters of both architectures, we use the probe method described in [START_REF] Bisseling | Parallel Scientific Computation: A Structured Approach Using BSP and MPI[END_REF]. It measures the time to perform some random collective all-to-all operations of growing size and, then, a least squares methods is applied to compute the values g and L. As the probe method was designed for bsp architectures, we recursively call it on each stage of the multi-bsp architecture. Thus, we have an acceptable approximation of the multi-bsp parameters, even if the configuration of the machines may produce some noise. The resulting values can be found in Table 5.1. We can observe that the latency L 1 and L 2 (regarding respectively the ram and cache memories) are in the same order of magnitude. On the contrary, the L 0 latency, corresponding to network accesses, are way slower. The same behaviour can be observed with the bandwidth g. As the probe method is quite sensitive to noise (due to the machine configuration), the multibsp parameters may be inaccurate. However, they illustrate well the hierarchy of memory with their different capabilities and can be used to predict the execution time of an algorithm. In Section 1.4.1.3, we give the evolution of the multi-bsp parameters on each architecture, depending on the thread distribution (Figure 1

p 0 = 0 g 3 = 3 L 3 = 0 m 3 = 0 p 1 = 16 g 2 = 6 L 2 = 1800 m 2 = 20M b p 2 = 2 g 1 = 89 L 1 = 1100 m 1 = 64Gb p 3 = 4 g 0 = ∞ L 0 = 149000 m 0 = 0 p 0 = 0 g 3 = 5 L 3 = 0 m 3 = 0 p 1 = 4 g 2 = 6 L 2 = 800 m 2 = 6M b p 2 = 2 g 1 = 14 L 1 = 472 m 1 = 16Gb p 3 = 8 g 0 = ∞ L 0 = 195400 m 0 = 0

Eratosthenes's sieve

Algorithm The sieve of Eratosthenes is a simple algorithm used to find all prime numbers up to any given limit. It works by iteratively marking as composite (i.e. not prime) the multiples of each prime, starting with the multiples of 2. The multiples of a given prime number are generated as a sequence of numbers starting from that prime, with constant difference between them that is equal to that prime.

Here, we present a simple and rather naive implementation of the sieve of Eratosthenes. To do so, it is possible to use a scan with a particular operator. The sieve of Eratosthenes is thus done by the propagation of the prime numbers, thanks to the scan. The multi-ml code of a scan is given in Figure 5.16.

The scan algorithm (also known as prefix sums algorithm) implementation is standard functional and high order programming. The scan algorithm takes a binary associative operator ⊕ and a set of n elements [a 0 , a 1 , …, a n-1 ] and returns the sums of the prefixes: [a 0 , (a 0 ⊕a 1 ), …, (a 0 ⊕ a 1 ⊕ … ⊕ a n-1 )]. The sequential version of the scan algorithm is trivial and its implementation can be easily found in published works. Moreover, it is available in most of the standard library of high order languages, such as ocaml. Here, we describe the behaviour of the multi-ml scan. First, on line 1, we define the multi_ml_scan value which aims to compute the scan on a multi-bsp architecture. It takes, as arguments, an associative operator op, a neutral element e and a list l. We then define (on line 2) the multi-tree, function which is going to be used to compute the scan. The function takes three arguments: 1) a flag which is used to control the recursion, 2) the list ( lst ) on which the operator must be applied, 3) and a tree t which is used to build the final tree. As every multi-tree-function, the body of m_scan is divided in two parts: the node and the leaf code.

First, when the evaluation takes place on a node, we check the value of the flag (line 4). It allows to distinguish two phases: 1) the first recursion level which distributes the list toward the leaves, 2) the second and following recursion level which only propagate the prefix-sum to the right. To distribute the list, we use the mkpar primitive and a split function, on line 5. The split: int -> int -> int list -> int list function takes a number of slices, an element i and a list, and returns the i th slice of the list. We recall that nprocs returns the number of children of the current node. When the distribution is complete, we can call recursively the m_scan function to continue the distribution of the list toward the leaves (line 6). When the recursion is terminated, the parallel vector r1 contains nprocs trees, where the scan algorithm was successfully applied. Now, we need to retrieve the roots of the sub-branches (which contain the values of the prefix-sums of the sub-levels) and propagate them. The retrieving of the value to propagate downward is done by using a bsp scan ( scan_direct ) onto the root of the sub-branches (line 7). Then, on line 8 -11, we recall the m_scan onto the sub-nodes, with a new list composed by the freshly computed prefixes. We use a conditional statement to avoid the recursive call of the multi-tree-function onto the left-most node, which is useless as its prefix-sum is already computed. To end the node code, we take the right-most element of the r2 parallel vector, which is the prefix that must be propagated at the next step, on the upper node. Finally, on line 13, we return the value v and the vector of branches containing the computed prefix-sum of the sub-levels. Thus, we build the branch of the current level.

The leaf code is much easier. When the flag is true (which correspond to the first time we reach the leaves: the end of the list distribution), we compute the local scan, in sequential. Otherwise (line 22), we must take into account the new element given in parameter, which correspond to the propagation of a prefix, from a upper node. In both cases, we return, as a value of the leaf, the computed value.

One can note that the usage of the flag is only used to distinguish two phases: the list distribution and the prefix propagation. In the presented algorithm, we choose to mix those two phases to reduce the size of the code. However, it is absolutely possible to write two multi-treefunctions, computing each phases independently, and call them, one after the other.

The sieve of Eratosthenes is easy to implement using a simple operator and the scan algorithm (Figure 5.17). The function op_erato aims to eliminate all the values of a given list, with the elim function. The erato function computes the Eratosthenes's sieve on a given list of values ( vals ) using the multi-bsp version of scan ( multi_ml_scan ), the Eratosthenes's operator ( op_erato ) and a list of initial prime numbers. Thanks to the scan algorithm, the op_erato is applied on all the distributed values "from the left to the right" with a growing set of values to eliminate.

Cost Analysis

The multi-bsp cost of the scan algorithm is the following: 100_000 500_000 1_000_000 3_000_000 multi-ml bsml multi-ml bsml multi-ml bsml multi-ml bsml p i ). The cost C i at level i is the cost of a bsp scan, of the diffusion of the computed values toward the leaves, in addition to the sequential cost of a map on list held by leaves. Let s be the size of the partial sum, the cost of bsp scan at level i is s * p i * g i-1 + L i , the diffusion cost is ∑ j∈]i...d] g j * s + l j and the final map cost is O(s d ). The size s may be difficult to evaluate: for a sum of integers it will simply be the size of an integer, but for Eratosthenes sieve, the size of exchanged lists varies depending on which data are held by the node.

Benchmarks

We measured the time to compute the Eratosthenes sieve without taking into account the time to generate the initial list of values. The experiments have been done on Mirev2 and Mirev3 using bsml (mpi version) and multi-ml over lists of increasing size on an increasing number of processors. The processes have been assigned to machines in order to scatter as much as possible the computation among the machines, i.e. a 16 process run will use one core on each processor of the 8 machines of Mirev3. Tables 5.2 and 5.3 show the results of our experimentation, where × stands for unfinished computations.

We notice that, on both Mirev2 and Mirev3, the efficiency of multi-ml on small lists is poor. It can be explained by the cost overhead introduced by the initialisation of the multi-ml "system", which is heavier that the bsml one. On Mirev3, as the number of processes grows, the execution time variance is smaller, but is still in favour of bsml.

Nevertheless, as the size of the data grows, multi-ml exceeds bsml. For example, on Mirev3 with 1.000.000 elements and 64 processors, multi-ml is 13 times faster than bsml.

The same behaviour can be observed on Mirev2, using the same parameters, with a speedup of 10. This difference is due to the fact that bsml communicates through the network at every super step; while multi-ml is focusing on communications through local memories and finally communicates through the network. Thus, the bsml algorithm is going to saturate the network with a large amount of communications. This congestion drastically decrease the performances. On the contrary, the multi-ml algorithm will communicate through local memories, and only a few communications will take place on the network. The network is not saturated and the performances are maximised.

Parallel computations are used almost everywhere. They offer the possibility to compute data faster than sequentially by taking advantage of parallel architectures ranging from small embedded chips to cutting edge hyper-clusters. However, designing and programming parallel algorithms is challenging and a rather not straightforward task. Several complex parameters must be taken into account in order to obtain performance and scalability and portability. Furthermore, parallel algorithms are prone to errors (such as deadlocks, data-races, etc.) and we must ensure safe executions. To do so, structured programming models and bridging models have been proposed in published materials. We choose to explore the multi-bsp model, which is an extension of the bsp model. bsp has proven itself a highly reliable bridging model that can tackle all the parallel programming problematics. Nevertheless, bsp was designed for flat architectures which was the standard of yesterday's parallel machines. As today's machines tend to be more and more hierarchical, we think that multi-bsp is the logical answer to this evolution. Its multi-stage capabilities allow us to take advantage of the properties of today's architectures with a simple bridging model to unify them all.

In this thesis we introduce multi-ml as a new programming language "à la ml", which is based on the multi-bsp model. multi-ml is an extension of bsml, a language conceived to program bsp algorithms "à la ml". Its aim is to draw strength from the multi-bsp model and enable us to program hierarchical architectures with ease. To do so, we offer a way to manage distant memories through the use of parallel vectors. We also define trees as a new parallel data structure that handle multi-stage data, where the data distribution is managed by the programmer. Furthermore, we introduce the concept of multi-function, which has the ability to recursively go through the hierarchical architecture.

In the following, we summarise the contributions regarding multi-ml semantics, a type system and its implementation. Finally, we discuss the perspectives around the evolution of multi-ml.

global code of a bsml program is replicated, every global value is available in every memory, through parallel vectors. This is not the case with multi-ml.

The multi-ml language aims to propose a way to program hierarchical architectures more efficiently than the bsp model. Even if its implementation does not rely on a low-level and high performance language commonly used in hpc, its strength resides in its structured programming model. multi-bsp is disconnected from the implementation details and places the emphasis on a way to exploit hierarchical architectures by designing efficient algorithms instead of writing bare metal optimisations.

Compared to bsp, multi-bsp adds complexity in its model in order to enhance its expressivity. Its ability to handle multi-stage memories and allow sub-synchronisations is attractive but, as multi-bsp is a young model, it has to prove itself. One can ask if multi-bsp is too complex to be adopted in place of bsp. As expected, it seems harder to design multi-bsp algorithms instead of bsp ones. Thus, programming such multi-bsp algorithms (using multi-ml) seems more complicated than programming bsp algorithms (using bsml). This overhead may discourage algorithmicists and programmers. Even if it seems harder to program with multi-ml, the language can be seen as a tool for experts who want to develop skeletons for turnkey solutions.

The promise of unifying various hierarchical architectures under a bridging model that ensures efficiency, safe execution, performance prediction and scalability must be thoroughly researched.

Semantics and typing

To get a formal description of the multi-ml language, we choose to write an operational semantics. It allows us to describe the behaviour of the evaluation of an expression. As the semantics is disconnected from the current implementation, it allows us to reason on well founded terms. In addition to that, and thanks to the multi-bsp cost model, we propose an operational semantics with costs. This annotated semantics is useful to determine the cost of all constructions of the language. It also gives hints on the way the implementation must be done in order to respect the multi-ml concepts. Furthermore, it could be used to predict performance of a given algorithm.

As well as in ml, type safety ensures that a program is well-formed and thus, "cannot go wrong". To guarantee the evaluation safety of multi-ml programs, we have defined a specific type system. This type system guarantee parallel program safety of multi-ml algorithm by rejecting unsound constructions. It mainly deals with replicated coherency, parallel structures nesting and memory level compatibility. To do so, we choose to enrich the ml type system with type annotations, effects and constraints. Thanks to the conjunction of a purely constraint based type system and a constraint solver, we are able to propose an algorithm that can reject unsound expressions. For example, we can ensure replicated coherency and avoid programs with erroneous communications or unauthorised parallel structure nesting. Currently, the type system is quite limited. All the ocaml features are not yet available and the multi-functions have high-order limitations. Thus, it is not yet possible to compose multi-functions or make partial applications. In Section 6.2.6, we propose solutions to overcome those limitations.

Implementation

We propose a formalisation of an abstract compilation scheme for a multi-ml program. Thus, we provide a detailed formal description of a parallel runtime system for multi-ml. To do so, we have defined a set of elementary blocks that describes the essential features which are disconnected from the low-level implementation details. Those elementary blocks are used to propose a compilation scheme for each multi-ml primitive. On top of that, we define the execution scheme of a multi-ml program execution. This formalised abstract compilation scheme also ensures a greater confidence for the design of the multi-ml compiler. multi-ml is based on ml and is implemented over ocaml. Thus, we choose to modify the ocaml sources from the parsing and lexing phase to the abstract syntax tree generation. The multi-ml type inference system is based on the ocaml type system, however it is not directly based on its implementation. As we choose to start the multi-ml implementation by using a limited core language: µ-multi-ml, the type system works on a few ocaml constructions. New features can be added, as new constraints, to the type system, gradually.

To complete the compilation toolchain, we take a sound abstract syntax tree in order to generate a valid ocaml parallel code. Currently, the parallel implementation relies on mpi. The usage of mpi allows good performances on a wide variety of architectures. However, as the implementation relies on generic modules, it is possible to use any communication library. For example, one can imagine an implementation dedicated to shared-memory models. A sequential version of multi-ml is also available to simulate the execution on various hierarchical architectures, in sequential. A toplevel based on the sequential implementation is available to provide an interactive loop for test and debug.

Benchmarks and results

Through practical experimentation, we have shown the benefits of the multi-bsp model. This approach takes advantage of the hierarchical architectures by allowing the sub-synchronisations of each machine of the cluster. Conversely, the bsp model interprets the whole machine as a flat set of processors communicating through a common network. When algorithms require communications between all the processors at the same time, bsp algorithms induce a network congestion due to the network bottleneck. On the contrary, the multi-bsp model avoids the network congestion by allowing communications on sub-groups of components. Thus, the communications are first done locally, between a few components; and finally, the last communications are done through the network.

In addition to avoid network congestion, the multi-bsp model favours communications between components sharing a low-level and high performance memory. It is thus possible to take advantage of the hierarchical memories of an architecture.

Our benchmark shows that, in practice, the performances of bottleneck sensitives algorithms are better using multi-ml than bsml.

Perspectives

The multi-ml language provides many perspectives of future work. In the following, we present the most promising research directions to our mind.

Certified parallel programming

As bsml is a purely functional extension of ml, it is possible to use a theorem prover (coq) to get an axiomatization of the bsml primitives. Thus, a certified parallel program can be extracted, directly from the program proof written in coq [START_REF] Gava | Formal Proofs of Functional Bsp Programs[END_REF][START_REF] Gesbert | Systematic Development of Correct Bulk Synchronous Parallel Programs[END_REF]. Such a property is quite rare in the world of parallel programming and is worthwhile. As multi-ml is an extension of bsml, it seems legitimate to wish the same property, for multi-bsp algorithms. To do so, we need a syntactic extension to be able to handle the multi-function concept in coq. This syntactic sugar is necessary to express the multi-ml primitives in pure coq. To do so, we may use shallow embedding, as it is done in bsml, for a fragment of multi-ml. The recursive calls due to the recursion through the hierarchical architecture may introduce tricky proof of termination. The formalisation of multi-ml in coq seems to be a difficult and rather non trivial task. Such a tool would lead to systematic development of certified multi-bsp algorithms, and thus, to certified parallel programs.

Alternatively, it is also interesting to imagine a way of programming multi-bsp algorithms with a skeleton. We could use hylomorphisms [START_REF] Meijer | Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire[END_REF] to capture our recursion schemes in order to generalise them. Intuitively, for a simple divide-and-conquer algorithm, hylo T f g is a recursive algorithm where T is the recursive call tree, f is the scatter part and g the gather part. The required "muscles" of such a skeleton could be: 1) down for data distribution; 2) leaf for leaves computations; 3) up for results scattering; 4) and control to control recursion. Even if the expressivity of this approach is similar to the multi-ml language, the simulation is not trivial.

GPU integration

As Graphical Processing Units (or gpu's) are widespread on hierarchical architectures, it seems interesting to handle them in the multi-bsp language. Indeed, General Purpose Graphics Processing Units (or gpgpu) is a non negligible way of computing in the world of hpc. The support of gpu's in multi-ml is thus a promising feature which could be achieved using spoc [16]: a set of tools for gpgpu programming in ocaml.

Basically, a gpu is a computing unit composed by many computational units accessing a very high bandwidth memory. A gpu can be seen as a shared memory architecture. Nevertheless, as gpus work in a very particular way (due to their graphical capabilities), they need a particular programming scheme. It is not possible to program them as usual. A kernel is a code written in a specific language that can be run on a gpu. We can imagine an extension of the multi-ml multi-function which consists in adding a kernel code, next to the node and leaf codes. The kernel code would be executed on gpu, when it is available. However, adding a gpu into the multi-bsp tree is not a that trivial extension. In the following figures, we represent a gpu as a triangle, where the thick lines stand for the special link needed to integrate this new component. The gpu integration can be seen in many ways, for example: 1) A simple unit of computation (Figure 6.2a), that is to say a leaf. In this case, the gpu is accessible at the same stage of the leaves;

2) A particular node available at an arbitrary stage of the architecture that runs a kernel code (Figure 6.2b);

3) A node with many leaves corresponding to the gpu threads or streaming multiprocessors (Figure 6.2c). It is thus possible to finely program the gpu.

Those proposals give two principal points of views. The first and second one interpret the gpu as a black box where the gpu execution is done seamlessly, without fine control. On the contrary, the third approach aims to allow a precise way of programming gpus by decomposing it as a tree. Those solutions introduce an unbalanced tree, something which is not considered by the multi-bsp model and must be studied in details.

As it seems non trivial to modify the multi-bsp model in its essence, we may favour a balanced approach. Thus, we can image the following methods: 1) At a given stage, a gpu is accessed by several nodes of the architecture, at the same time (Figure 6.3a); 2) All the nodes of a given stage can access their own gpu (Figure 6.3b). We can imagine an implementation where the gpu is virtually split;

3) All the leaves have access to their own gpu (Figure 6.3c);

Those approaches are balanced and seem in accordance to the multi-bsp model. Moreover, one can imagine that accessing a gpu must be done from a node with ram memory in order to achieve better performance. On the contrary, a gpu accessed from a node (or leaf) with a small amount of memory may be inefficient.

All those considerations may lead to an extended version of multi-ml that handles gpus, thanks to spoc. Thereby, we will propose a language able to program hybrid and hierarchical architectures, which is the standard of today's cutting edge machines.

Examples

multi-bsp is a young model which is not yet widespread. There are few algorithms based on this model in published materials. Moreover, there is no complete implementation that are available for testing. It seems important to write multi-bsp algorithms in order to demonstrate the benefits offered by the model. As a proof of concept, writing multi-bsp algorithms using multi-ml would justify the existence of such a language. Then, it will be possible to benchmark a given algorithm on various architectures to experience the scalability of multi-bsp algorithms. Comparing the results with the cost predictions of the multi-bsp model would also point out the benefits of such a structured model.

Costs analysis

The cost analysis of an algorithm is one of the key features of the multi-bsp model. Reasoning on algorithmic costs allows to develop efficient algorithms running on multi-bsp architectures. Moreover, writing a multi-bsp algorithm with an optimal cost allows to obtain an immortal algorithm that can be run optimally on every hierarchical architecture. As the cost model is embedded in the semantics of the language, it seems interesting to propose a way to extract automatically the cost of a given program. One can imagine a compilation tool that gives a runtime estimation of a program, based on the computation and network capabilities of an architecture. Furthermore, one can imagine to combine the resource bound analysis proposed in [START_REF] Hoffman | Towards Automatic Resource Bound Analysis for OCaml[END_REF] with multi-ml in order to estimate the amount of resources exploited by a program at runtime. In terms of resources, it is also possible to consider a way to estimate the power consumption of a system, based on its specifications. Thus, it would be possible to obtain the efficiency of an algorithm, based on its execution time and energy consumption.

Multi-ML implementation

Currently, multi-ml relies on a relatively un-optimised implementation. As we use closures to communicate data, the communication between components is not optimised. The usage of closures has simplified the development of multi-ml and will also ease its maintenance. However, one can imagine a unified memory addressing that would avoid closure communications. Thus, the communications would only rely on pure data communication with shared function identifier. Moreover, the implementation of an efficient parallel garbage collector would improve the memory management of multi-ml programs.

Multi-ML extensions

Language A certain amount of work is also needed to manage parallel input/output (i/o), as it was done in [START_REF] Gava | Parallel I/O in Bulk-Synchronous Parallel ML[END_REF] with bsml. Indeed, i/o at the multi-bsp level (when all the computing units are concerned) is not trivial to handle. We must be able to synchronise all the computing units on a single channel. Such consideration introduces many questions on determinism, coherency and performances. It seems interesting to propose a multi-ml library to ease i/os. However, it also raises many questions regarding the design and implementation of such features.

Exceptions is an important feature that must be embedded in the multi-ml language. The multi-ml implementation does not propagate exceptions in a proper way, yet. It seems natural to propose, as in bsml, a way to handle exceptions with a pattern matching of the form: trypar … withpar … . This pattern matching involves the definition of a small language of regular expressions to be able to match parallel exceptions easily, as it is the case in bsml. It seems quite trivial to do the same thing in multi-ml.

It also seems very interesting to propose a primitive of superposition, at it is the case in bsml. The superposition is a purely functional construction that fuses the communications and synchronisations of the evaluation of two expressions given in argument. Such an executions would preserve the multi-bsp model and would allow to overlap the communications and synchronisations of two multi-stage executions, resulting in better performance. However, the implementation of superposition was a very difficult task in bsml and could be even more difficult in multi-ml.

Type system As we said previously, multi-ml does not implement all the ocaml features. In the first time, we may add the variants (or sum types). This construction seems trivial but it introduces many locality constraints. Therefore, a way to match patterns in parallel vectors, define records and modules may be considered. However, modules introduces difficulties regarding exception handling. Indeed, as exceptions can be defined in modules, their definition must be known by the whole architecture to be considered.

Regarding the multi-ml type system itself, we can imagine a way to exceed the limitations of multi-functions composition. As a multi-function defined outside the scope of another one is not directly usable by the latter (because of its locality), we may imagine a way to relax this constraint. For example, we can introduce a new keyword to be able to re-bind, locally, a multifunction inside the definition context of another one. To do so, we just need to add where f = f to the definition of a multi-function, where f is a predefined multi-function. The multi-function f is thus re-bound during the initialisation of another multi-function. This solution is simple and does not involve important changes. We could also update the grammar of constraints in order to be able to have multiple latent effects on arrows types. Adding the disjunctions to the grammar would allow us to have the latent effect ϵ -→ where ϵ = m ∨ l. Thus, we could use multi-functions in a high-order way.

We may also use the type system's informations to offer code optimisations. For example, by proposing a smarter way to serialise values, based on their types and localities.

APPENDIX

• Case app. The expression e is of the form f e ′ . By induction hypothesis, we have

f : (τ 1 π 1 ε 1 -→ τ 2
π 2 ) Λ ′ /ε 2 and e ′ : τ 1 π 3 /ε 3 , which are both valid in Λ. Thus, we have π 3 ◁ π 2 to ensure the validity of the argument, ε 1 ◁ Λ ensures the validity of evaluation of f within Λ and π 2 ◁ Λ forces the the produced value to be valid locally. The constraint • Case replicate. By induction hypothesis on f and by ensuring that the evaluation and the value produced is valid within b : ε 1 ◁ c and π 2 ◁ c.

• Case down. Similarly to the copy rule.

• Case apply. By induction hypothesis on e 1 and e 2 . Then, similarly to the app rule, by ensuring that π 3 ◁ π 1 , ε ◁ c and π 2 ◁ c.

• Case multi_fun. The expression e is of the form (τ 1 • Case var: we define e = x ′ , thus we have: • Case deffun: we define e = fun x ′ → e ′ , thus, we have τ 2 π 2 = (τ

var Γ x (x ′ ) = ∀ -→ α ′ -→ λ ′ [c 0 ].τ π ′ /ε [c 0 ] c = [π ′ ◁ Λ, c 0 ] Λ, Γ x ⊢ x ′ : τ 2 π 2 /ε [c] -If x ′ ̸ = x,
′ 1 π ′ 1 ε ′ -→ τ ′ 2 π ′ 2
) Λ . The derivation rule is:

deffun Λ ′ , Γ x ; x ′ : τ ′ 1 π ′ 1 /ε 1 ⊢ e ′ : τ ′ 2 π ′ 2 /ε ′ [c 1 ] c 2 = [ε ′ ◀ Λ, π 1 ◁ ε ′ , c 1 ] Λ, Γ x ⊢ fun x ′ → e ′ : (τ ′ 1 π ′ 1 ε ′ -→ τ ′ 2 π ′ 2 ) Λ /ε ′ [c 2 ]
-If x ̸ = x ′ , similarly to the let-in rule, we have:

Γ x ; x ′ : τ ′ 1 π ′ 1 /ε 1 = Γ x ; x ′ : τ ′ 1 π ′ 1 /ε 1 ; x : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].τ 1 π 1 .
By induction of the first premise, Λ, Γ; x ′ : τ • Case multi_fun. Similar to the deffun case.

• Other cases: the remaining rules are resolved directly as they do not deal with the environment Γ.

Proposition A.19 (Semantic generalisation)

Let v be a value and τ π be a type such that |= v : τ π . Then, for all substitution φ, we have |= v : φ(τ π ).

Proof : By structural induction over v.

• Case where v ≡ cst and τ = Base. With Hypothesis H0 4.17 π 2 ) Λ ′ ). It remains to prove that |= M : φ(Γ). For any y ∈ Dom(M), we write Γ(y) as ∀α 1 , …, α n λ 1 , …, λ n [c].τ y with the α i and λ i chosen out of reach of φ. We then have φ(Γ(y)) = ∀α 1 , …, α n λ 1 , …, λ n [c].φ(τ y ). We need to prove that M(y) : τ ′ π ′ for all instances of τ ′ π ′ of φ(Γ(y)). Let τ ′ π ′ be such an instance. We therefore have τ ′ π ′ = ψ(φ(τ y )) for some substitution ψ. We also have |= M(y) : τ y , by definition of |= over type schemes. We can therefore apply the inductive hypothesis to the value M(y), to the type τ y and to the substitution ψ •φ.

1. 1

 1 The Sequoia super computer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 The scheme idea of a bridging model. . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 A bsp computer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 A bsp superstep . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 g and L bsp parameters of Mirev2 in flops. . . . . . . . . . . . . . . . . . . . . . 1.6 Times of a total exchange on Mirev2. . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 The multi-bsp components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 The multi-bsp computer model. . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.9 Sub-synchronisation capabilities of multi-bsp . . . . . . . . . . . . . . . . . . . 1.10 The multi-bsp parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. 1

 1 The sub-synchronisation scheme. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Data parallel skeletons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 The mapreduce skeleton. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 The divide and conquer skeleton. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 The stencil skeleton. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 The Google's mapreduce skeleton. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 A MapReduce example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 A fastflow example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 A muesli example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 A skepu example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.11 A quaff example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.12 drma bsp operations: "put" and "get". . . . . . . . . . . . . . . . . . . . . . . . . 2.13 A Multicorebsp example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.14 A bsp++ code example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.15 A bsp python example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.16 A pregel code example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.17 A neststep example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.18 A nesl code example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 The multi-ml levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 The small architecture used as example. . . . . . . . . . . . . . . . . . . . . . . . 3.3 The multi-function recursion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 The multi-function evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Tree construction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 The at operator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 The gid tree. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi 1 Introduction

2 CHAPTER

 2

Figure 1 . 1 :

 11 Figure 1.1: The Sequoia super computer.

  Phi (Many-core) (b) A parallel bridging model.

Figure 1 . 2 :

 12 Figure 1.2: The scheme idea of a bridging model.

Figure 1 . 3 :

 13 Figure 1.3: A bsp computer.

  2. A communication network to exchange messages between pairs;3. A global synchronisation unit to execute global synchronisation barriers.

Figure 1 . 4 :

 14 Figure 1.4: A bsp superstep

1 . 2 .

 12 Each processor only uses its local data to perform sequential computations and to request data transfers to other nodes (thick arrows); The network delivers the requested data (dotted arrows); 3. A global synchronisation barrier occurs, making the transferred data available for the next superstep (empty rectangle).

#

  let list_of_par vec = List.map (proj vec) procs;; -: val list_of_par : 'a par -> 'a list = <fun> # list_of_par << $pid$ >>;; -: int list = [0; 1; 2]

#

  let total_exchange vec = let msg = put <<fun dst -> $vec$>> in << List.map $msg$ procs >>;; -: val total_exchange : 'a par -> 'a list par = <fun> # total_exchange << $pid$ >>;;

(

  * scan': int->('a->'a->'a)->'a->'apar->'apar *) let rec scan' step op e v = if step >= bsp_p then v else let comm = put << fun j -> if (j mod (2*step) = 0) && ($pid$ = j + step) then $v$ else e >> in let v' = << if $pid$ mod (2*step) = 0 then if $pid$ + step < bsp_p then op $v$ ($comm$ ($pid$ + step)) else $v$ else e >> in scan' (step*2) op e v' (* reduce: ('a->'a->'a)->'a->'a par->'a *) let reduce op e v = (proj (scan' 1 op e v)) (bsp_p-1)

  (a) Each processor extracts p -1 pivots from its ordered local list, in a regular way. (b) Those pivots lists are merged globally, in order to form an ordered list of size p(p -1). (c) p -1 pivots are chosen regularly spaced in the list. A bsml version of the corresponding algorithm can be coded as follows: 20 CHAPTER 1. INTRODUCTION (* psrs: int par -> 'a list -> 'a list *) let psrs lvlengths lv = (* step 1: local sorting *) let locsort = << List.sort compare $lv$ >> in (* step 2(a): selection of the (p-1) primary samples *) let regsampl = << extract_n bsp_p $lvlengths$ $locsort$ >> in (* step 2(b): total exchange of the primary samples *) let glosampl = List.sort compare (gather_list regsampl) in (* step 2(c): selection of the secondary samples *) let pivots = extract_n bsp_p (bsp_p*(bsp_p-1)) glosampl in (* step 3 : building the communicated lists of values *) let comm = << slice_p $locsort$ pivots >> in (* step 3: send the values *) let recv = put << List.nth $comm$ >> in (* step 4: local fusion of the received data*) << p_merge bsp_p (List.map $recv$ procs_list) >>

  mkpar: (int -> 'a) -> 'a par apply: ('a -> 'b) par -> 'a par -> 'b par

  For example: <<f ($pid$+1) >> is transformed by bsml into: let pid=mkpar (fun i ->i);; apply (mkpar (fun _ -> f)) (apply (mkpar (fun _ pid -> pid+1)) pid)

  g bsp parameters of Mirev2.

  L bsp parameters of Mirev2.

Figure 1 . 5 :

 15 Figure 1.5: g and L bsp parameters of Mirev2 in flops.

Figure 1 . 6 :

 16 Figure 1.6: Times of a total exchange on Mirev2.

Figure 1 . 7 :

 17 Figure 1.7: The multi-bsp components.

Figure 1 . 8 :

 18 Figure 1.8: The multi-bsp computer model.

Figure 1 . 9 :

 19 Figure 1.9: Sub-synchronisation capabilities of multi-bsp

Figure 1 . 10 :

 110 Figure 1.10: The multi-bsp parameters Figure 1.10 illustrates the multi-bsp parameters on a multi-bsp architectures.We consider p 0 as a basic computing unit (a processor) where a step on a word is considered as the unit of time. Therefore, it does not share any memory with any other component and does not have sub-component making p 0 equal to 1 and L 0 equal to 0. On another side, the stage 1 memory is accessed by the level 0 processors, and thus the data rate (corresponding to the notional g 1 ) has value 1. Figure1.11 illustrates the difference between the bsp model and the multi-bsp one for a multi-core architecture with threads 5 .

Figure 1 . 11 :

 111 Figure 1.11: The difference between the multi-bsp and bsp models for a multi-core architecture.

1. 4 . 29 •Figure 1 . 8 P 2 = 4 Figure 1 .

 42918241 Figure 1.12 illustrates such a multi-bsp architecture. Stage 4

Figure 2 . 1 :

 21 Figure 2.1: The sub-synchronisation scheme.

Figure 2 . 2 :

 22 Figure 2.2: Data parallel skeletons.

Result

Figure 2 . 3 :Figure 2 . 4 :

 2324 Figure 2.3: The mapreduce skeleton.

Figure 2 . 6 :

 26 Figure 2.6: The Google's mapreduce skeleton.

FastFlow fastflow [ 37 }Figure 2 . 7 :

 3727 Figure 2.7: A MapReduce example.

Figure 2 . 8 :

 28 Figure 2.8: A fastflow example.

47 #Figure 2 . 9 :

 4729 Figure 2.9: A muesli example.

Figure 2 .

 2 Figure 2.10: A skepu example.

Figure 2 .

 2 Figure 2.11: A quaff example.

  Collective communicating operations: int MPI_Barrier(MPI_Comm comm) int MPI_Send(const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm) int MPI_Recv(void *buf, int count, MPI_Datatype datatype, int source, int tag, MPI_Comm comm, MPI_Status *status) int source, int recvtag, MPI_Comm comm, MPI_Status *status) int MPI_Bcast(void* buffer, int count, MPI_Datatype datatype, int root, MPI_Comm comm) int MPI_Gather(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) int MPI_Scatter(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPI_Comm comm) int MPI_Allgather(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) int MPI_Alltoall(void* sendbuf, int sendcount, MPI_Datatype sendtype, void* recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm) Collective reducing operations: int MPI_Reduce(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, int root, MPI_Comm comm) int MPI_Allreduce(void* sendbuf, void* recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) int MPI_Reduce_scatter(void* sendbuf, void* recvbuf, int *recvcounts, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm) Communicator manipulation: int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *newcomm) Duplicates an existing communicator int MPI_Comm_free(MPI_Comm *comm) Deallocation of the communicator int MPI_Comm_split(MPI_Comm comm,int color,int key,MPI_Comm *newcomm) Creates new communicators based on colors and keys MPI_Comm peer_comm,int remote_leader,int tag,MPI_Comm *newintercomm) Creates a communicator from two others Windows for DRMA: int MPI_Win_create(void *base,MPI_Aint size,int disp_unit,MPI_Info info,MPI_Comm comm,MPI_Win *win) Creation of a window int MPI_Win_fence(int assert,MPI_Win win) Fence synchronisation int MPI_Put(void *origin_addr,int origin_count,MPI_Datatype origin_datatype,int target_rank, MPI_Aint target_disp,int target_count,MPI_Datatype target_datatype,MPI_Win win) Put data into a memory window on a remote process int MPI_Get(void *origin_addr,int origin_count,MPI_Datatype origin_datatype,int target_rank, MPI_Aint target_disp,int target_count,MPI_Datatype target_datatype,MPI_Win win) Get data from a memory window on a remote process

Figure 2 . 12 :

 212 Figure2.12: drma bsp operations: "put" and "get".

part[ 1 ]

 1 = bsp_nprocs(bsp); bsp_partition (bsp, &subbsp, 2, part); if(my_pid<2) {...;bsp_sync(&subbsp);...} else {...} bsp_done(&subbsp);

  Figure 2.13: A Multicorebsp example. bsml bsp++ 'a par

Figure 2 .

 2 Figure 2.14: A bsp++ code example.

Figure 2 .

 2 Figure 2.15: A bsp python example.

classFigure 2 .

 2 Figure 2.16: A pregel code example.

}Figure 2 .

 2 Figure 2.17: A neststep example.

Figure 2 .

 2 Figure 2.18: A nesl code example.

Figure 3 . 1 :

 31 Figure 3.1: The multi-ml levels.

Figure 3 . 2 :

 32 Figure 3.2: The small architecture used as example.

69 let

 69 multi mf () = where node = (*node code*) "hello node" where leaf = "hello leaf" (*leaf code*)

Figure 3 . 3 :

 33 Figure 3.3: The multi-function recursion.

Figure 3 . 4 :

 34 Figure 3.4: The multi-function evaluation.

  finally : ~sub:'a_c tree_l par_b -(b)-> ~head:'a_`z -(b)-> Final ('a_c tree_l par_b, 'a_`z)_m .

at_b t t 0. 1 Figure 3 . 6 :

 136 Figure 3.6: The at operator.

  rec_call = << sum_t () >> in let p = proj rec_call in let res = List.fold_left (fun x y -> (p x) + y ) 0 procs in res + (at_b t)

1 Figure 3 . 7 :

 137 Figure 3.7: The gid tree.

  : int list) in let n_lst = << split #n# #lst# >> in ...

Figure 3 . 8 :

 38 Figure 3.8: mkpar execution steps.

  i_lst = mkpar (fun i -> split n i lst) in ...

(

  *List summing with intermediate sums *) let multi tree sum_list lst = mkpar (fun i -> split nprocs i l) in let rc = << sum_list $v$ >> in let s = sum_seq (flatten << at_l $rc$ >>) in finally rc s where leaf = sum_seq l

Figure 3 . 9 :

 39 Figure 3.9: End of the distribution.

Figure 3 . 10 :

 310 Figure 3.10: Resulting tree.

  | let x = e in e Let binding | fun x → e Function | rec f x → e Recursive function | multi f x → e † e Multi-function | (e e) Application | if e then e else e Conditional | (e, e) Pair | mkpar e Parallel primitives | proj e | put e Synchronous parallel primitives | << e >> | #x# | $x$ Parallel syntactic sugar | replicate (fun _ → e ) Core parallel primitives | down x | apply e e Core parallel primitives v

Figure 4 . 1 :

 41 Figure 4.1: The grammar of the µmulti-ml expressions.

  a -> 'a par down x → ⟨x, …, x⟩ replicate replicate : (unit -> 'a) -> 'a par replicate (fun _ → e ) → ⟨(fun _ → e )(), …⟩ apply apply : ('a -> 'b) par -> 'a par -> 'b par apply⟨f0, …, fp-1⟩⟨v0, …, vp-1⟩ → ⟨(f0 v0), …, (fp-1 vp-1)⟩

Figure 4 . 2 :

 42 Figure 4.2: The communication predicate.

Figure 4 . 4 :Figure 4 . 7 :

 4447 Figure 4.7: The lookup predicate.

Figure 4 . 8 :

 48 Figure 4.8: The select predicate.

FFigure 4 . 9 :

 49 Figure 4.9: The free variable predicate.

Figure 4 . 15 :Figure 4 . 16 :

 415416 Figure 4.15: Annotated types definition.

Figure 4 . 17 :

 417 Figure 4.17: Constraints definition.

Figure 4 .

 4 18b show the relation between all the levels using directed arrows, from λ 1 toλ 2 , to denote the notion of « λ 1 can read in memory λ 2 ». m, c ◁ m m, b ◁ b m, l, c ◁ c m, l, c ◁ l m, s ◁ s(a) Relations of accessibility. Relations of accessibility.

Figure 4 . 18 :

 418 Figure 4.18: The accessibility relation.

4. 3 .

 3 A TYPING SYSTEM FOR µMULTI-ML 103

  a) Relations of definability. Relations of definability.

Figure 4 . 19 :

 419 Figure 4.19: The definability relation.

  The lattice of propagation.

Figure 4 . 20 :

 420 Figure 4.20: The propagation predicate.

Figure 4 . 21 :

 421 Figure 4.21: Serialisation rules.

∀α 1

 1 …α n λ 1 …λ m [c].τ π to denote the set the types obtained by the instantiation of α 1 …α n λ 1 …λ m by types and localities respecting the constraint c. Where α 1 …α n are type variables and λ 1 …λ m are effect variables. We write τ for a scheme ∀∅[True].τ . Definition 4.13 (Weakening: Weak(τ π , ε))

Figure 4 . 22 :

 422 Figure 4.22: Weakening rules.

Figure 4 . 26 :

 426 Inductive rulesmulti-bsp level.

  Figure 4.27: The MmlPrimitives(e) predicate.

Figure 4 . 29 :

 429 Figure 4.29: bsp function derivation.

Figure 4 . 30 :

 430 Figure 4.30: Parallel vector derivation.

Figure 4 . 31 :

 431 Figure 4.31: Derivation of a non-communicable vector projection.

  e >>: τ π par b if there exists a typing environment Γ such that |= M : Γ and b, Γ ⊢<< e, …, e >>: τ π par b

Theorem 4 . 22 (

 422 Type soundness)

#

  let r = ref 1;; val r : int ref = {contents = 1} #let f = fun x -> r:=x in f 42

Figure 4 . 32 :

 432 Figure 4.32: Operators of references.

Figure 4 .

 4 Figure 4.33: Non-expansive expressions grammar.

Figure 4 . 34 :

 434 Typing rules for references.

Figure 4 . 35 :

 435 Figure 4.35: Serialisation with references.

Figure 4 . 36 :

 436 Figure 4.36: Constraints and type scheme syntax.

4. 5 . 123 Definition 4 . 23

 5123423 A PURELY CONSTRAINT-BASED TYPE SYSTEM FOR µMULTI-ML

Figure 4 .

 4 37, and are directly obtained from the rules of Figure 4.24, Figure 4.25 and Figure 4.26.

Figure 4 . 37 :

 437 Constraints generation rules.λ = Propgt(ε, ε ′ ) effect propagation

Figure 4 . 39 :

 439 Figure 4.39: Constraints on localities.

Figure 4 . 40 :

 440 Figure 4.40: Locality unification.

132 CHAPTER 5 Figure 5 . 1 :

 132551 Figure 5.1: Syntax of core-multi-ml.

Figure 5 . 2 :

 52 Figure 5.2: Free variables of core-multi-ml.

Figure 5 . 3 :

 53 Figure 5.3: Well formed terms of core-multi-ml.

Figure 5 . 4 :

 54 Figure 5.4: Flattening the multi-ml tree into cores.

  let replicate = fun idS -> let idD= newIdD() in let idM = CurrentIdM in Signal i Rpl(idD,idS,idM); idD

  let proj = fun idD -> Signal i Prj(idD); let tab=fromChildren i () in (fun j -> tab[j])

  let copy = fun x -> let idD= newIdD() in Signal i Do(idD); toChildren i x; idD Basically, it simply signals, to the children, the construction of a new parallel vector and waits for the corresponding value. The generated code for mkpar is close, except for the sending of values to the children: let mkpar = fun g -> let idD= newIdD() in Signal i Mk(idD); toChildren i (Array.create f p); idD Finally, the code of a daemon consists in waiting for orders, executing them and waiting until the execution ends.let rec daemon i () = match rcv i with

Figure 5 . 5 :

 55 Figure 5.5: Compilation scheme of core-multi-ml.

Figure 5 . 6 :

 56 Figure 5.6: Abstract context syntax.

  fromChildren and up First, to communicate synchronously a value upward (from children to the parent), we have the following rule:⟨⟨. . . , {E 0 , Γ i 0 [up v 0 ]}, . . . , {E p , Γ i p [up v p ]}, {E, Γ ′i j [fromChildren()]}, . . .⟩⟩ ⇒ ⟨⟨. . . , {E 0 , Γ i 0 [()]}, . . . , {E p , Γ i p [()]}, {E, Γ ′i j [[|v 0 , . . . , v p |]]}, .. .⟩⟩ where {0, . . . , p} are the gid i of the branch of the children of the parent on core j.

Theorem 5. 1

 1 For a program P = e 1 ; ; • • • ; ; e n and if WF _ (e i ),• If M ⊢ e i ⇓ L p v i for each e i then: ⟨⟨{E 0 , [[P ]] m }, . . . , {E p c , [[P ]] m }⟩⟩ ⇒ * ⟨⟨{E ′ 0 , v 0 }, . . . , {E ′ p c , v p c }⟩⟩ • If M, ⊢ e i ⇓ L p ∞ for one e i then: ⟨⟨{E 0 , [[P ]] m }, . . . , {E p c , [[P ]] m }⟩⟩ ⇒ ∞Then, we write ⟨⟨e, ..., e⟩⟩ ⇒ saf e , where ⇒ saf e ≡ ⇒ * ∪ ⇒ ∞ .

  let pi=3.14;; let multi f () = where node = .... << 2+pi >> where leaf = 2+pi

  let multi g () = ...;; let multi f () = where node = .... << f()+g() >> where leaf = ...

  b) Rules equivalence.

Figure 5 . 7 :

 57 Figure 5.7: Cost algebra definitions.

e 1 then e 2 1 Figure 5 . 8 :

 2158 Figure 5.8: Computation of the size of data.

Figure 5 . 13 :

 513 Figure 5.13: The multi-ml compilation toolchain.

•Figure 5 .

 5 Figure 5.14: A multi-ml toplevel configuration file. #let t = mktree (to_string gid);; -: val t : string tree # t;; o "0" | --o "0.0" | |--> "0.0.0" | |--> "0.0.1" --o "0.1" | |--> "0.1.0" | |--> "0.1.1"

Figure 5 . 15 :

 515 Figure 5.15: Interactive tree visualisation.

  .5 and Figure 1.6)

letFigure 5 .

 5 Figure 5.16: A multi-ml scan.

∑

  i∈[0...d[ V sp (i) + O(|l d |) + ∑ i∈[0...d[ C i . 5.4. EXPERIMENTATION AND BENCHMARKS 159 (* elim: int list -> int -> int list *) let elim l m = let rec aux l accu = match l with [] -> List.rev accu | hd::tl -> if (hd mod m=0) then aux tl accu else aux tl (hd::accu) in aux l [] (*op_erato: int list -> int list -> int list*) vals (*erato: (int list -> int list -> int list) -> int list -> int list -> int list*) let erato op vals = multi_ml_scan op_erato [2;3;5] vals

Figure 5 . 17 :

 517 Figure 5.17: The sieve of Eratosthenes.

1 p

 1 (i) is the total cost of splitting the list toward the leaves (at depth d -1); O(|l d |) is the time needed to compute the local sums in the leaves; and ∑ i∈[0...d[ C i corresponds to the cost of diffusing partial sum back to leaves and to add these values to the values held by leaves. This diffusion is done once per node. V sp (i) is the work done at level i to split the locally held chunk l i and scatter it among children nodes. Splitting l i in p i chunks costs O(|l i |) where |l i |, the size of l i , is n * ∏ j∈[0...i[ j where n is the size of the initial list held by the root node. Scattering it among children nodes costs p i * g i-1 + n i p i + L i . The sequential list scan cost at leaves is O(|l d |) = O(n * ∏ i∈[0...d[ 1

( a )

 a Single gpu leaf. (b) Single gpu node. ...... (c) Detailed gpu node.

Figure 6 . 2 :

 62 Figure 6.2: Unbalanced gpu integration.

  (a) Shared gpu. (b) Shared gpu node. (c) Shared leaf gpu.

Figure 6 . 3 :

 63 Figure 6.3: Balanced gpu integration.

ε 1 ◀

 1 Λ is used to ensure the validity of references. • Case vector. The expression e is of the form << e ′ >>. By induction hypothesis on e ′ , π is valid in c. Then, by definition, ε ◁ c and π ◁ c ensure the validity of the term. • Case copy. By definition of the Seria predicate, τ π = Seria c (τ b ) thus π ◁ c. • Case open. The rule is direct. • Case op'. By definition, Γ(op ′ ) leads to a valid type. Similarly to the app rule, π 3 ◁ b ensures the validity of the argument and ε ◁ b ensures the validity of evaluation of op ′ within Λ. Hence the result.

-→ τ 3 m

 3 ) m /m. By induction hypothesis, e 1 : τ 3 π 3 /ε 1 and e 2 : τ 3 π 4 /ε 2 . By definition of the Seria predicate, π 3 is valid in m and π 4 is valid c.

Lemma A. 18 (

 18 Typing is stable under substitution)Under the following hypothesis which defines the substitution:• Λ, Γ; x : ∀ -→ α -→ λ [c 1 ].τ 1 π 1 ⊢ e : τ 2 π 2 /ε [c 2 ; ∃ -→ α -→ λ .c 1 ] • Λ, Γ ⊢ v : τ 1 π 1 /ε [c 1 ; c 2 ] • -→ α -→ λ ∩ F(c 2 ) ∩ F(Γ) = ∅ We have: Λ, Γ ⊢ {v/x}e : τ 2 π 2 /ε [∃ -→ α -→ λ .c 1 ; c 2 ]where {v/x}e stands for the substitution of x by v in expression e.Proof : We define Γ x = Γ; x : ∀ -→ α -→ λ [c 1 ].τ 1 π 1 and c = c 2 ; ∃ -→ α -→ λ .c 1 for conciseness reasons. The first hypothesis is Λ, Γ x ⊢ e : τ 2 π 2 /ε [c].The proof is made by induction on the derivation of this typing judgement.

2 /Ψ [c 3 ]-

 23 then {v/x}e = e and Γ x (x ′ ) = Γ(x). Hence the result.-If x ′ = x, then {v/x}e = v and, by hypothesis: Λ, Γ ⊢ {v/x}e :τ 1 π 1 /ε [c 1 ; c 2 ]. Then, we have Γ x = ∀ -→ α -→ λ [c 1 ].τ 1 π 1 = ∀ -→ α ′ -→ λ ′ [c 1 ].τ 2 π 2 .Thus, τ 1 π 1 and τ 2 π 2 are equals modulo renaming (alpha-conversion).• Case op. Similar to var.A.2. TYPING193• Case let-in: we define e = let x ′ = e 1 in e 2 . The derivation rule is thus the following:let in Λ, Γ x ⊢ e 1 : τ 1 π 1 /ε 1 [c ′ 1 ] Λ, Γ x ; x ′ : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 1 π 1 , ε 1 ) ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ] c 3 ≡ [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 ] Λ, Γ x ⊢ let x ′ = e 1 in e 2 : τ 2 π If x = x ′ , then Λ, Γ x ; x ′ : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 1 π 1 , ε 1 ) = Λ, Γ; x ′ : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 1 π 1 , ε 1 ). Thus, Λ, Γ; x ′ : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 1 π 1 , ε 1 ) ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ]. By using the let-in rule, we get Λ, Γ ⊢ let x ′ = {v/x}e 1 in e 2 : τ 2 π 2 /Ψ [c 3 ] and, by definition of the substitution, Λ, Γ ⊢ {v/x}letx ′ = e 1 in e 2 : τ 2 π 2 /Ψ [c 3 ]. -If x ̸ = x ′ , then Γ x ; x ′ : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 1 π 1 , ε 1 ) = Γ; x ′ : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 1 π 1 , ε 1 ); x : ∀ -→ α ′ -→ λ ′ [c ′ 1 ].Weak(τ 3 π 3 ,ε3 ). By using the let-in rule, we get Λ, Γ ⊢ let x ′ = {v/x}e 1 in {v/x}e 2 : τ 2 π 2 /Ψ [c 3 ] and thus Λ, Γ ⊢ {v/x}(let x ′ = e 1 in e 2 ) : τ 2 π 2 /Ψ [c 3 ]. • Case let rec. Similar to the let in case.

′ 1 π ′ 1 /ε 1 = 2 / 1 and•

 1121 Γ x ; x ′ : τ ε ′ [c 2 ]. Hence the result.-If x = x ′ , then Γ x ; x ′ : τ Case copy. Similar to the var case, whereΓ x (x ′ ) = ∀ -→ α ′ [c 0 ].τ b /ε [c 0 ] • Case open. Similar to the var case, where Γ x (x ′ ) = ∀ -→ α ′ [c 0 ].τ par b /ε [c 0 ] • Case op'.Where e ≡ op ′ e ′ , similarly to the var case and by induction hypothesis on e ′ .• Case down. Similarly to the var case with the locality of τ constrained to b.

1 π 1 ε-→ τ 2 π 2 ) 1 ε-→ τ 2 π 2 ) 1 ε-→ τ 2

 1121212 , we have φ(τ ) = τ . Hence the result. • Case where v ≡ op and σ = T C(op). With Hypothesis H0 4.17 and α of T C(op) out of reach of φ. Hence the result. APPENDIX • Case where v ≡ (fun x → e )[M] and τ π = (τ Λ ′ . Let Γ be a typing environment such that |= M : Γ and Γ ⊢ (fun x → e ) : (τ 1 π Λ ′ . By Lemma 4.18, we have φ(Γ) ⊢ (fun x → e ) : φ((τ 1 π

2 2 A 1 ε-→ τ 2 π 2 -→ τ 2 m( 1 ) 1 π 1 ε-→ τ 2 π 2 1 ε-→ τ 2 π 2 ) 3 }-→ τ 2 m ) m -|= v : τ 1 m-→ τ 2 m

 22122111212312 It follows that |= M(y) : τ ′ π ′ (this holds for all instances τ ′ π ′ of φ(Γ(y))). Hence |= M(y) : φ(Γ(y)) (this holds for all y ∈ Dom(e)). Hence |= M : φ(Γ) and the expected result. • Case where v ≡ (rec f x → e )[M]. As previously. • Case where v ≡ (multi f x → e 1 † e 2 )[M]. As previously.• Case where v ≡ (v 1 , v 2 ) and τ π = τ 1 π 1 × τ 2 π 2 . We apply the induction hypothesis to both v 1 and v 2 . It follows |= v 1 : φ(τ 1 π 1 ) and |= v 2 : φ(τ 2 π 2 ). Hence the result. • Case where v ≡<< v >> and τ π = τ 1 π 1 par b . By induction hypothesis on v and by definition of |= we have |= v : φ(τ 1 π 1 par b ). Hence the result.Lemma A.20 (Evaluation progress)Let e be an expression, τ π be a type, Γ be a typing environment, M be an evaluation environment such that Λ, Γ ⊢ e :∀ -→ α -→ λ [c].τ π and |= M : Γ. If there exists a value v such that M ⊢ e ⇓ L p v then |= v : τ π Proof :By induction on M ⊢ e ⇓ L p v and by case analysis of e (and thus on the concluding rule used in the typing tree derivation).• Case cst. e can only be reduced into itself. We must ensure that |= cst : T C(cst) for all constants cst. By definition, it is true for the constants and type assignment T C. Hence the result.• Case op. As previously.• Case var. The only applicable typing rule is var. The typing system guarantees thatx ∈ Dom(Γ) and thus, by hypothesis on |= M : Γ, x ∈ Dom(M). Thus, M ⊢ x ⇓ L p v is the only evaluation possible. By hypothesis |= M : Γ, we have |= M(x) : Γ(x) and thus |= M(x) : τ π by definition of |= over type schemes.• Case let_in. The let in typing rule is the following:let in Λ, Γ ⊢ e 1 : τ 1 π 1 /ε 1 [c 1 ] Λ, Γ; x : Weak(τ 1 π 1 , ε 1 ) ⊢ e 2 : τ 2 π 2 /ε 2 [c 2 ] c 3 ≡ [Ψ = Propgt(ε 1 , ε 2 ), c 1 , c 2 ] Λ, Γ ⊢ let x = e 1 in e 2 : τ 2 π 2 /Ψ [c 3 ] We thus have that Λ, Γ ⊢ e 2 : τ 2 πFollowing the let_in semantics rule, the last evaluation step is:let_in M ⊢ e 1 ⇓ L p v 1 M ⊕ p {x → v 1 } ⊢ e 2 ⇓ L p v 2 M ⊢ let x = e 1 in e 2 ⇓ L p v on e 1 , we have |= v 1 : τ 1 π 1 . Using Lemma 4.19, |= v 1 : Weak(τ 1 π 1 , ε 1 ). We define M 1 = M ⊕ {x → v 1 } and Γ 1 = Γ ⊕ {x → Weak(τ 1 π 1 , ε 1 )}. Therefore, we have |= M 1 : Γ 1 .We apply the induction hypothesis on e 2 with M 1 and Γ 1 . Thus, we have |= v 2 : τ 2 π 2 . • Case let_rec. Similar to the let_in case. • Case deffun. The only applicable typing rule is deffun. Thus M ⊢ (fun x → e ) ⇓ L p (fun x → e [M]). By definition of |=, we have |= (fun x → e [M]) : (τ 1 π ) Λ .• Case multi_fun. The only applicable typing rule is multi_fun as the evaluation locality is constrained to m. Thus M ⊢ (multif x → e 1 †e 2 ) ⇓ m multi (multi f x ′ → e ′ 1 † e ′ 2 )[M ′ ]. By definition of |=, we have |= (multi f x ′ → e ′ 1 † e ′ 2 )[M ′ ] : (τ 1 m m ) m .• Case app. Here, we have two possible evaluation: (1) the application of a function closure and (2) the application of a multi-function. For the first case (function closure) we have:fun_app M ⊢ e 1 ⇓ L p (fun x ′ → e ′ )[M ′ ] M ⊢ e 2 ⇓ L p v M ⊕ p M ′ ⊕ p {x ′ → v} ⊢ e ′ ⇓ L p v ′ M ⊢ (e 1 e 2 ) ⇓ L p v ′ By induction hypothesis, we get: -|= (fun x → e )[M] : (τ ) Λ ′ -|= v : τ 3 π 3 (with π 3 ◁ π 1 ) By definition of |=, there exists a typing environment Γ ′ such that |= M ′ : Γ ′ and Λ, Γ ′ ⊢ (fun x ′ → e ′ ) : (τ 1 π Λ ′ .As there is only on typing rule that can be derived, its premises must hold: Λ, Γ ′ ⊕ {x → τ 1 π 3 } ⊢ e ′ : τ 2 π 2 . Using the following environments:-M 1 = M ′ ⊕ {x → v} -Γ 1 = Γ ′ ⊕ {x → τ 1 πWe have |= M 1 : Γ 1 and Λ, Γ 1 ⊢ e ′ : τ 2 π 2 (by definition of |= M ′ : Γ ′ ). By applying the induction hypothesis to the evaluation of e ′ , we get : |= v ′ : τ 2 π 2 . (2) For the second case (multi-function closure) we have:fun_app M ⊢ e 1 ⇓ m multi (multi f x ′ → e ′ 1 † e ′ 2 )[M ′ ] M ⊢ e 2 ⇓ m multi v M ⊕ root M ′ ⊕ root {x ′ → v} ⊢ e ′ ⇓ b root v ′ M ⊢ (e 1 e 2 ) ⇓ m multi v ′ By induction hypothesis, we get: -|= (multi f x ′ → e ′ 1 † e ′ 2 )[M ′ ] : (τ 1m m By definition of |=, there exists a typing environment Γ ′ such that |= M ′ : Γ ′ and Λ, Γ ′ ⊢ (multi f x → e 1 † e 2 ) : (τ 1 m m ) m . As there is only on typing rule that can be derived, its premises must hold: Λ, Γ ′ ⊕ {x → τ 1 m } ⊢ e ′ 1 : τ 2 m . Using the following environments: -M 1 = M ′ ⊕ {x → v} -Γ 1 = Γ ′ ⊕ {x → τ 1 m } We have |= M 1 : Γ 1 and Λ, Γ 1 ⊢ e ′ 1 : τ 2 m (by definition of |= M ′ : Γ ′ ). By applying the induction hypothesis to the evaluation of e ′ 1 , we get : |= v ′ : τ 2 m .

Table 2 .

 2 1: The mpi primitives.

Table 2 . 2

 22

: The pub/bsplib primitives.

Table 2 .

 2

3: Comparison between bsp++ and bsml primitives.

  The processor n sends the value v m n to processor m. After the communication, the processor m gets the value v n m from processor n.Signal and rcvThe Signal function is a low level asynchronous communication routine which aims to send a message to the queue F d of another (distant or not) thread. Then the rcv routine can read it. We write E ⊕ | . . . | i to highlight the use of the queue of daemons (allocated to gid i) without modifying the rest of the environment.First we define the sending of a message:⟨⟨. . . , {E 0 ⊕ | . . . | i , e i 0 }, . . . , {E p ⊕ | . . . | i , e i p }, {E, Γ i j [Signal i v]}, . . .⟩⟩ ⇒ c ⟨⟨. . . , {E 0 ⊕ | v . . . | i , e i 0 }, . . . , {E p ⊕ | v . . . | i , e i p }, {E, Γ i j [()]}, . . .⟩⟩Where v is one of the possible messages presented above and where {0, . . . , p} are the gid i of the branch of the children of the parent on core j. Note the message could be processed later. For example, such a case may occur when a daemon received different messages manipulating a parallel vector:

	let v1=replicate (fun g -> 1) in
	let v2=replicate (fun g -> 2) in
	let vplus=replicate (fun g -> (+) ) in
	(apply (apply vplus v1) v2)
	flatSend [|v 0 0 , . . . , v 0 k , . . . , v 0 p |]]}, . . . , {E p , Γ i p [flatSend [|v p 0 , . . . , v p k , . . . , v p p |]]}, . . .⟩⟩
	⇒
	⟨⟨. . . , {E 0 , Γ i 0 [[|v 0 0 , . . . , v k 0 , . . . , v p 0 |]]}, . . . , {E p , Γ i p [[|v 0 p , . . . , v k p , . . . , v p p |]]}, . . .⟩⟩

Table 5 .

 5 1: Multi-BSP parameters of Mirev3 (left) and Mirev2 (right). mirev3 and belong to the lifo: "Laboratoire d'Informatique Fondamental d'Orléans, Orléans, France". Their specifications are the following: • Mirev2: 8 nodes with 2 quad-core (AMD 2376) at 2.3Ghz with 16Gb of memory per node and a 1Gb/s network. • Mirev3: 4 nodes with 2 hyper-threaded octo-core cpus with 16 threads(Intel XEON E5-2650) at 2.6Ghz with 64Gb of memory per node and a 10Gb/s network.

Table 5 .

 5 Table5.2: Execution time (in seconds) of Eratosthenes using multi-ml and bsml on Mirev3. 3: Execution time (in seconds) of Eratosthenes using multi-ml and bsml on Mirev2.

	8	0.7	1.8	22.4	105.0	125.3	430.7	×	×
	16	0.5	0.8	13.3	50.3	68.1	331.5	1200.0	×
	32	0.3	0.5	2.6	18.9	11.3	122.2	173.2	×
	48	0.5	0.4	1.75	14.5	5.5	88.4	69.3	×
	64	0.3	0.3	1.3	8.7	4.1	56.1	51.1	749.9
	96	0.3	0.38	1.6	6.3	3.9	30.8	38.1	576.1
	128	0.5	0.45	2.1	5.2	4.7	24.3	30.6	443.7
			100_000 multi-ml bsml multi-ml bsml multi-ml bsml 500_000 1_000_000
		8	1.5	1.7	64.5	106.1	402.9	1538.1
		16	0.45	0.93	16.0	49.3	91.4	631.7
		32	0.14	0.45	4.1	18.7	21.1	219.7
		48	0.13	0.40	2.6	11.0	10.8	123.5
		64	0.11	0.34	1.89	7.5	8.2	80.5

Where ∑ i∈[0...d[ V sp

Bien que nous avions la possibilité de reprendre le système de type d'ocaml afin de l'étendre, nous avons choisi une autre solution. En effet, à cause de la complexité des sources du système de typage d'ocaml, nous avons préféré construire le système de type ainsi que son inférence, à partir de zéro. Pour ce faire, nous avons choisi de mettre en place un système d'inférence basé sur un système de contraintes : pcb(x). Ce système d'inférence de type est composé de deux phases principales : (1) la génération de contraintes et (2) la résolution de contraintes. La génération de contraintes est effectuée à partir de l'ast produit pendant la phase de compilation. Les contraintes générées sont directement tirées des règles d'inférence de types proposées pour multi-ml. Les contraintes sont ensuite données à un algorithme de résolution de contraintes qui est composé de trois principales phases : (1) Unification de types, (2) unification de localités et (3) résolution des contraintes. C'est lors de la résolution de contraintes qu'il est possible de détecter les erreurs de programmation qui pourraient remettre en cause la sûreté d'exécution d'un programme multi-ml. Ainsi, nous sommes capable de détecter des erreurs spécifiques à multiml, comme l'imbrication de vecteurs parallèles, la manipulation de données avec des localités de définition incompatibles ou la tentative de communication de données non communicables.En ce qui concerne l'implémentation mpi, nous avons choisi de proposer une approche modulaire. Celle-ci repose sur un module de communication qu'il est possible de choisir, en fonction du contexte d'exécution. Ce module décrit un ensemble de primitives élémentaires de bas niveau, qui permettent d'exprimer l'ensemble des constructions du langage. Actuellement, nous proposons un module de communication basé sur la librairie mpi, du fait de ses bonnes performances et de sa compatibilité avec de nombreuses architectures. Cependant, nous pouvons facilement imaginer l'utilisation d'une autre librairie, adaptée à un type d'architecture en particulier. De plus, cette approche modulaire nous permet de proposer une implémentation séquentielle. Celleci peut être utilisée à des fins de débugage ou pour simuler l'exécution de programmes sur des architectures imaginaires. Une boucle interactive (toplevel), à la manière d'ocaml et bsml, est également disponible.

We will explain in details, in the next section, what parallel and distributed architectures are.

LLNL. Lawrence Livermore National Laboratory: BlueGene/Q, 2012. https://computing.llnl.gov/tutorials/ bgq/.

bsp libraries are generally implemented using mpi[START_REF] Snir | MPI-The Complete Reference. 2nd[END_REF] or low-level routines of the given specific architectures.

In the worst case, an horizontal communication is just the succession of a write and a read operation in the upper memory of the component; the cost is thus 2 × gi.

The use of hyperthreading technology just adds an extra stage to specify physical threads.

The measurements of gi are hypothetical and aim to illustrate the slowdown of the bandwidth (in flops/words).

In this context, it stands for the level of parallelism.

Note that this code does not actually lead to a multi-level execution. As explained in the next section, we must recursively call a multi-function to do so. In practice, this code is just executed on the node code and then, returns the final value.

In this example, the rec_call value is ignored.

We can imagine a pseudo-random operator, available in the standard library, initialised with the same seed in order to keep the replicated coherency using communications, or not. However, it is not the topic here.

A memory which is updated and unified globally by a synchronous mechanism.

This interpretation makes perfect sens with references (see Section

4.4.1).

:= in ocaml. The access to a reference is done using the get operator, denoted by ! in ocaml. For example, we can write the following code, using the ocaml toplevel:

We left the possibility of proof of optimisations for future work.

Remerciements CHAPTER 6. CONCLUSION AND PERSPECTIVES

Bibliography

CHAPTER 4. TYPE SYSTEM Comm s (mkpar e) = ⊥ Comm s (proj e) = ⊥ Comm s (put e) = ⊥ Comm s (apply e 1 e 2 ) = ⊥ Comm s

Comm s (e) Comm s (e 1 e 2 ) = Comm s (e 1 ) ∧ Comm s (e 2 ) Comm s (let x = e 1 in e 2 ) = Comm s (e 1 ) ∧ Comm s (e 2 ) Comm s (if e 1 then e 2 else e 3 ) = Comm s (e 1 ) ∧ Comm s (e 2 ) ∧ Comm s (e 3 ) Comm s (x) = ⊤, if x / ∈ s Comm s (v), if {x → v} ∈ s Comm s (cst) = ⊤ Comm s (op) = ⊤

multi_def

Terminating semantics is deterministic

In this section, we give the proof of the terminating semantics rules. To do so, we prove that the evaluation of an expression following the proposed rules is deterministic.

Lemma 4.2 (Evaluation or not)

Let e be an expression, M be an evaluation environment, L an evaluation locality and p a position (referencing to a particular unit). Then, using classical logic with the excluded middle:

• It is impossible to obtain a value v such that M ⊢ e ⇓ L p v • or there exists a value v such that M ⊢ e ⇓ L p v

Proof : (Using classical logic). The excluded middle holds over M ⊢ e ⇓ L p v.

Lemma 4.3 (Evaluation is deterministic)

Let e be a program, let M be an environment, L an evaluation locality, p a position and v 1 and v 2 be values. If M ⊢ e ⇓ L p v 1 and M ⊢ e ⇓ L p v 2 then v 1 = v 2 .

Proof : See Appendix A.1.1.

Diverging semantics rules

Following [START_REF] Leroy | Coinductive Big-Step Operational Semantics[END_REF], we define the divergent rules (with infinite evaluations) by the coinductive interpretation of the previous inference rules. The relation of divergence is written e ⇓ L p ∞ where e is an expression which diverges, when evaluated at level p of locality L.

In Figures 4.10,4.11,4.12, 4.13 and 4.14 we describe the diverging rules, written directly from the terminating semantics rules defined in Section 4.2.3. They, respectively, describe the coinductive rules of generic, bsp (divided in two figures), local and multi-bsp. As the principle of the rules is straightforward, we do not describe the rules one by one. Informally, the main idea of the diverging rules is to describe all the states where the evaluation of an expression can diverge. The rules where an application e 1 e 2 takes place and the diverging term is e 1 (corresponding to rules fun_app, rec_fun_app, multi_node, multi_leaf and multi_call) were fused into a single rule: app-l.

Diverging semantics properties

In this section, we use the properties of the co-inductive semantics to prove that an evaluation produces a value or diverges.

Lemma 4.4 (Evaluation and divergence exclusivity)

Let e be a program, let M be an environment, L an evaluation locality, p a position and v a value. If M ⊢ e ⇓ L p v and M ⊢ e ⇓ L p ∞ then there is a contradiction.

Note that this lemma does not prove that any program can be evaluated. For example, this lemma is clearly false when "random" operators are allowed: one can write a code like if randomBool then finiteComputation else infiniteComputation , which neither diverge nor 94

(S-Name-1) the singleton set {α}.

Definition 4.28

Let α and β range over finite sets of type variables.

Definition 4.29

Let ⃗ α be a vector of distinct type variables.

The constraint language used in this section is defined as follow:

We now introduce the unification algorithm in Figure 4.38. It is interesting to note that a multi-equation encode both state and control. Indeed, a multi-equation τ 1 = τ 2 = ϵ may be viewed as the equivalence class τ 1 = τ 2 with τ 2 = ϵ as a pending request to solve. The specification of this algorithm is non deterministic and several rules can be applied at the same time. As the type unification and the locality unification are separated, the Figure 4.38 does not deal with localities. The solver is based on the proposal made in [START_REF] Pottier | The Essence of ML Type Inference[END_REF]. We choose to remove some of the rules that are not decisive in our context, for example recursive types. And we add the ability to process annotated types, a requirement for the multi-ml typing system. This solver is a non deterministic rewriting system performed modulo α-conversions. S-Unify-T, respectively S-Unify-L, makes the type unification, respectively locality unification, a component of the solver by allowing to invoke it at any time. S-Solve-Eq inject an equation in the typing environment U . S-Solve-Id match an instanciation constraint x ⪯ τ and replace it with σ ⪯ τ , where σ = S(x) is the type scheme carried in the nearest environment frame that defines x in the stack S. The stack examination is defined in Figure 4.42.

S; U

S-Solve-And pushes, arbitrarily, the left element of the constraint on the stack. S-Solve-Ex introduces a new existential frame in the stack. S-Solve-Let handle the let binding of the stack. First it examines the left-hand side to simplify the type scheme instanciation. Indeed, starting with the right-hand side involves a new frame where the type scheme is not simplified: ∀α[D].T . On the contrary, we can inject a simplified type scheme: ∀α[U ].α. S-Pop-And deals with the remaining conjunction that was partially examined by s-Solve-And. S-Pop-Let retrieve the type constraint of the let binding to introduce it in the type scheme and start to explore the right-hand side of the expression. S-Pop-Env remove the unnecessary frame composed by a type scheme that is not occurring in U any more and a solved right-hand side (turned into a constraint in U ). The current typing system of the multi-ml language is implemented on top of those algorithms.

Notice that, due to the execution model of the multi-ml, all cores are running the exact same code and thus, all of them are registering the same multi-functions in order to execute them (in a spmd mode). To do so, an identifier is first created. Then, we create a specific function with an argument i that will be the gid (global identifier) of the current level of execution. The CurrentIdM is then modified by the identifier of the multi-function called. If the process is a node, the node code of the multi-function is executed, after notifying the sub-nodes to wait for instructions. Then daemons will be used to manipulate the vectors (construction, projection, communication, etc.). Otherwise, when the process is a leaf, the leaf code is executed. When the multi-function starts, the root node is active and all other processes are awakened, waiting for orders, as daemons. Finally, the multi-function is un-registered (globally by all the cores) and we return the computed value.

The code of the scheduler is:

Where the wake is a passive-wait function which waits until an order is received. Note that each core has its own ordered queue of messages. That could be easily implemented using a library such as mpi. Now, we have to handle parallel vector's manipulations. We recall that the idea is to have a static identifier which correspond to the code to execute and another identifier (dynamic identifier) which is used during the life-time of a parallel vector. As static identifiers are shared by all cores, the codes must be the same on each core (spmd). On the contrary, dynamic identifiers will be shared by a node and its children (which could be nodes or leaves). The content of the vector (data and expressions) is handled by the child, using a hash table of vectors where the identifiers are the keys. From the parent point of view, a parallel vector is just a dynamic identifier. Here is the generated code for replicate:

.9: Generic evaluation rules with costs. 5.10: bsp evaluation rules with costs. multi_def (1) The typing of expression e generates a valid set of constraints;

BIG STEP SEMANTICS WITH COSTS

(2) The typing system and the semantics ensure a safe evaluation;

(3) The abstract compilation scheme produces a generic code;

(4) The cost model gives a performances prediction;

(5) The implementation allows the execution of a multi-ml program.

Figure 6.1: Summary of this thesis.

6.1. CONTRIBUTIONS 165

Contributions

Figure 6.1 aims to summarise all the contributions of this thesis. The reading of this figure is similar to the manuscript. From the left hand side, we describe the founding elements of multiml: the bsp model and bsml, its ml implementation (unfilled arrows). Both have inspired the multi-ml language. The dashed arrows symbolise hierarchical extensions. Then, we unfold (using plain arrows) the main steps of the conception of multi-ml: from the type system, the semantics and the abstract compilation scheme, to the implementation and testing phase. We also describe all the properties resulting from our approach.

In [START_REF] Alaniz | MBSPDiscover: An Automatic Benchmark for MultiBSP Performance Analysis[END_REF] we can see that we develop an algorithm of type inference for multi-ml. The typing system (2) ensures the safe execution of multi-ml programs thanks to a formal semantics. We can observe that the locality constraints dues to the multi-bsp model influences both the typing system and the semantics. Thus, with (3) , we are able to compile a multi-ml program. The cost model of multi-bsp (4) allows to make performance predictions. In association with (5) , we propose an implementation relying on communication modules such as mpi.

If a type system gives a valid type for e and if the evaluation of e produces a value then the compilation of e is valid. Then, we propose the following result:

Final result 6.1 (Correctness of a multi-ml program)

Let e be an expression standing for a multi-ml program, Γ a type environment and C a set of constraints generated from e, then:

Finally, if WF _ (e) we have ⟨⟨e, …, e⟩⟩ ⇒ saf e .

The Multi-ML language

multi-ml has been designed to program hierarchical architectures with multiple levels of memory. As it is inspired by the multi-bsp model, its programming model is based on a tree structure with nested components. The multi-ml language slightly differs from the original idea of multi-bsp in the sense that nodes have computation capabilities. While multi-bsp trees are composed by nodes with storage capacities and leaves with computation capabilities, in multiml it is possible to execute codes on nodes. The node codes must be as lightweight as possible and focus on data scattering and gathering. This ability is mainly used to manage parallel computation during the multi-stage execution. Such a parallel computation is written using the bsml primitives plus some additional multi-bsp primitives. Moreover, we propose a new type of recursion by introducing the multi-functions. Thanks to the combination of parallel vectors and multi-functions, it is easy to recurse through the different stagess of a hierarchical architecture. To strengthen its hierarchical abilities, we also propose a new parallel data structure that handles hierarchically distributed values: the tree structure.

Such hierarchical architectures are composed by layers of memory that tend to decrease in size as we draw closer to the leaves. As all those memories are physically different in terms of both size and bandwidth, we must take it into account. To do so, multi-ml proposes a system of locality that gives information on the memory where the data belongs. It is useful to manage data in order to get the best performance and avoid expensive communication. The locality information is also used as a safeguard to ensure well founded communications.

We could almost say that multi-ml is an extension of bsml with hierarchical capabilities. Nevertheless, some concepts that are the essence of bsml differ from multi-ml. As the execution model of multi-ml is dedicated to hierarchical machines, memories are separated. Thus, a value defined in a node must be explicitly communicated to a sub-node in order to be used on that sub-node. This behaviour differs from the global execution that is used in bsml. Indeed, as the Let e be a program, let M be an environment, L an evaluation locality, p a position and v 1 and v 2 be values.

TABLE OF NOTATIONS

Proof : By induction on the derivation of M ⊢ e ⇓ L p v 1 and case analysis on M ⊢ e ⇓ L p v 2 . For each rule concluding the derivation tree, we have:

• Case values (values). For all locality of evaluation and for all position, the only applicable evaluation rule is values, and thus, the two evaluation judgement are identical. Hence, the result.

• Case op_eval (op). As previously.

• Case var (x). As previously.

• Case closure (the expression e is of the form fun x → e ). As previously.

• Case rec_closure (rec f x → e ). As previously.

• Case fun_app. The expression e is of the form (e 1 e 2 ). By definition of the fun_app rule, e 1 is evaluated into

as it is the only valid case. The derivation rule is the following:

Then, as a conclusion of the fun_app rule, v 1 ≡ v 2 . • Case rec_fun_app. As previously, with e 1 ≡ (rec f x → e ′ )[E].

• Case op_app. As previously, with e 1 ≡ op.

• Case let_in. With e ≡ (let x = e 1 in e 2 ), as there is only one applicable rule, by induction hypothesis, all the premises are the valid. Hence the result.

APPENDIX

• Case replicate. As previously.

• Case down. As previously, using the var rule.

• Case mkpar. As previously, by induction M ⊢ e ⇓ L p e ′ . • Case apply. As previously.

• Case proj. As previously.

• Case put. As previously.

• Case multi_node. Similar to the fun_app rule where the expression e is of the form (e 1 e 2 ) and, by induction hypothesis, e 1 is of the form (multi

as it is the only applicable rule at locality l. As the predicates isN ode(p) and isLeaf (p) are mutually exclusive, isN ode(p) is valid within the application context.

• Case multi_leaf. Similar to the multi_node rule but where the predicate isLeaf (p) is valid.

• Case multi_def. Similar to the closure rule where the expression e is of the form

) at locality m. • Case multi_call. Similar to the multi_node rule, where the locality of evaluation is m.

A.1.2 Diverging semantics properties

Lemma A.3 (Evaluation and divergence exclusivity)

Let e be a program, let M be an environment, L an evaluation locality, p a position and v a value. If M ⊢ e ⇓ L p v and M ⊢ e ⇓ L p ∞ then there is a contradiction.

Proof : By induction on the derivation of M ⊢ e ⇓ L p v and case analysis on M ⊢ e ⇓ L p ∞, using Lemma 4.3. We have:

• Case values (values). For all locality of evaluation and and for all position, there is no coinductive rule for constants. Hence the contradiction.

• Case op_eval (op). As previously.

• Case var (x). As previously.

• Case closure, the expression e is of the form (fun x → e ). As previously.

• Case rec_closure (rec f x → e ). As previously.

• Case fun_app. The expression e is of the form (e 1 e 2 ). By definition of the fun_app rule,

By induction hypothesis on the premises, we have:

-M ⊢ e 1 ⇓ L p ∞ (rule app-l). Using the previous inductive hypothesis, we get a contradiction.

as it is the only valid case, and M ⊢ e 2 ⇓ L p ∞ (rule fun_app-r). Using the previous inductive hypothesis, we get a contradiction.

we have:

M ⊢ (e 1 e 2 ) ⇓ L p ∞ By induction, we obtain:

p ∞, then we have a contradiction. • Case rec_fun_app. As previously, with e 1 ≡ (rec f x → e ′ )[E].

• Case op_app. As previously, with e 1 ≡ op.

• Case let_in. When e ≡ let x = e 1 in e 2 , the case is direct as there is a single applicable rule.

• Case replicate. By induction hypothesis, if at least one of the i (i ∈ p) evaluations leads to a diverging evaluation, then we have a contradiction.

• Case down. There is no rule for down, similarly to values. Hence the contradiction.

• Case mkpar. As previously, as the premises are mutually exclusives.

• Case apply. As previously.

• Case proj. As previously.

• Case put. As previously.

• Case multi_node. Similarly to the fun_app rule, if

, by induction hypothesis, it excludes diverging rules.

• Case multi_leaf. As previously.

• Case multi_call. As previously. • Case var. The result is explicit in the derivation rule: π ′ ◁ Λ.

A.2 Typing

A.2.1 Type soundness

• Case cst. As previously.

• Case op. As previously.

• Case pair. By induction hypothesis on e 1 and e 2 , hence the result.

• Case if then else. By induction hypothesis on e 1 , e 2 and e 3 , hence the result.

• Case let in. By induction hypothesis on e 1 and e 2 , hence the result.

• Case let rec. As previously.

• Case deffun. The expression e is of the form fun x → e ′ and is typed using the rule: • Case open. Similar to the var case.

• Case op'. Similar to the app case as the rule consists in the application of a multi-ml primitive on an expression.

• Case replicate. The last evaluation step of the rule is: • The remaining rules (pair and if then else) are direct.

Lemma A.21 (Co-evaluation progress)

Let e be an expression, τ π be a type, Γ be a typing environment, M be an evaluation environment such that Λ, Γ ⊢ e : τ π and |= M : Γ. Let v be a value; if it is impossible to obtain M ⊢ e ⇓ L p v; Then M ⊢ e ⇓ L p ∞.

Proof : (Using classical logic) The proof is done by coinduction and case analysis over e:

• Cases e ≡ cst (cst), e ≡ op (op), e ≡ (fun x → e ) (deffun) and e ≡ (multi f x → e 1 † e 2 ) (multi_fun) lead to a contradiction, since they can be reduced to themselves or into a value from the environment M.

• Case where e ≡ x (var). We have a contradiction if Λ, Γ ⊢ x : τ π and ⊭ M : Γ.

• Case where e ≡ (e 1 e 2 ) (app). Using the excluded middle, either e 1 evaluates to a value f 1 , or not. In the latter case, M ⊢ e ⇓ L p ∞ follows from one of the rule of the application and from M ⊢ e 1 ⇓ L p ∞, which was obtained by the coinduction hypothesis. Thus, f 1 is typed (τ 1

π 2 ) Λ ′ by Lemma 4.20. Using, again, the excluded middle, either e 2 evaluates to a value v, or not. In the latter case M ⊢ e ⇓ L p ∞ follows from one of the rule on of the rule of the application and by the coinduction hypothesis. In the former case, we have Λ, Γ ⊢ e 2 : τ 1 π 1 . The case when f 1 is an operator clearly leads to a contradiction since, by Hypothesis H1 4.17 • Case where e ≡ let x = e 1 in e 2 (let_in). Similarly to the app case.

• Case where e ≡ let rec f x = e 1 in e 2 (let_in). As previously.

• Case where e ≡<< e ′ >> (vector). Using the excluded middle, ∀i ∈ p, e ′ can be evaluated into a value v i ( M ⊢ e ′ ⇓ c p i v i ), or not. In the latter case, ∃i ∈ p such as M ⊢ e ′ ⇓ L p ∞. Hence the result. • Case e ≡ copy copy. Similar to the var case.

• Case e ≡ openopen. Similar to the var case.

• Case e ≡ op ′ op'. Similar to the app case as the rule consists in the application of a multi-ml primitive on an expression.

• Case e ≡ replicate replicate. Using the excluded middle, replicate (fun _ → e ′ ) can be evaluated into a value or not. In the latter case, by coinduction hypothesis f → (fun _ → e ′ )} : (unit π 1 ε 1 -→ τ 2 π 2 ) Λ ′ and M ⊢ e ′ ⇓ c p i ∞. Thus, ∃i ∈ p such that M ⊕ p i {f → (fun _ → e )} ⊢ f () ⇓ l p i ∞. Hence the result. • Case e ≡ down down. Similar to the copy case.

• Case e ≡ apply apply. Similar to the app case where the only possible evaluation of e 1 is a parallel vector of functions (fun x → e )[M].

• The remaining rules (pair and if then else) are direct.