Henri Casanova
email: henric@hawaii.edu

Martin Quinson
email: martin.quinson@ens-rennes.fr

Arnaud Legrand
email: arnaud.legrand@imag.fr

Frédéric Suter
email: frederic.suter@cc.in2p3.fr

SMPI Courseware: Teaching Distributed-Memory Computing with MPI in Simulation

Keywords: High Performance Computing Education, Parallel Computing Education, Message Passing Interface, Simulation

It is typical in High Performance Computing (HPC) courses to give students access to HPC platforms so that they can benefit from hands-on learning opportunities. Using such platforms, however, comes with logistical and pedagogical challenges. For instance, a logistical challenge is that access to representative platforms must be granted to students, which can be difficult for some institutions or course modalities; and a pedagogical challenge is that hands-on learning opportunities are constrained by the configurations of these platforms.

A way to address these challenges is to instead simulate program executions on arbitrary HPC platform configurations. In this work we focus on simulation in the specific context of distributed-memory computing and MPI programming education. While using simulation in this context has been explored in previous works, our approach offers two crucial advantages. First, students write standard MPI programs and can both debug and analyze the performance of their programs in simulation mode. Second, large-scale executions can be simulated in short amounts of time on a single standard laptop computer. This is possible thanks to SMPI, an MPI simulator provided as part of SimGrid. After detailing the challenges involved when using HPC platforms for HPC education and providing background information about SMPI, we present SMPI Courseware. SMPI Courseware is a set of in-simulation assignments that can be incorporated into HPC courses to provide students with hands-on experience for distributed-memory computing and MPI programming learning objectives. We describe some these assignments, highlighting how simulation with SMPI enhances the student learning experience.

I. INTRODUCTION

Many challenges are involved in teaching High Performance Computing (HPC) concepts and practices, as indicated by the establishment of workshop series focused on HPC education. Although part of the HPC curriculum can be taught at theoretical or conceptual levels, it is widely acknowledged that a large portion of it is taught more effectively if students are provided with hands-on experience opportunities. This is typically achieved by providing students with access to one or more HPC platforms for them to design, implement, execute, and analyze the execution of programs developed as part of lab activities and/or homework assignments. Unfortunately, using real-world platforms to teach HPC courses comes with logistical challenges as well as, perhaps counter-intuitively, pedagogical challenges. We review both types of challenges hereafter.

Logistical Challenges -The most obvious logistical challenges is that it is not always possible to obtain access to a representative HPC platform in the first place. For some institutions, it may be that no HPC platform is available whatsoever. In this case, one can resort to building lowcost HPC platforms for teaching purposes [START_REF] Ludin | LittleFe: The high performance computing education appliance[END_REF]- [START_REF] Abuzaghleh | Implementing an Affordable High-performance Computing for Teaching-oriented Computer Science Curriculum[END_REF], to using virtualization and/or container technology to emulate an HPC platform, e.g., in cloud environments [START_REF] Ivica | StarHPC -Teaching parallel programming within elastic compute cloud[END_REF]- [START_REF] Johnson | Designing, Constructing and Implementing a Low-Cost Virtualization Cluster for Education[END_REF], or to gain access to public HPC testbeds. Although a lot can be achieved pedagogically on such platforms, they are not necessarily representative of production HPC platforms (in terms of capabilities, scale, and/or performance behaviors), which can be problematic for performance-oriented learning objectives. Even if an institution hosts HPC platforms, there may be no straightforward mechanism to use these platforms for education purposes. Throughout the careers of the authors of this paper, in their past and present institutions, although production HPC resources were available for research purposes, providing access to students in a course has often times proved more challenging than expected. In many cases, access can be secured, but only to a small subset of the platform, thus making it difficult for students to experience scaling effects. If access is secured to a large platform, either hosted in-house or as part of some community HPC testbed, this platform is typically used in production, and thus is shared among students and regular platform users. This often raises social issues in which regular platform users, e.g., researchers working toward conference deadlines, are then impacted by an influx of new (student) users, which can create tensions. Furthermore, these student users are novices, and can make mistakes that lead to resource waste. For instance, when teaching HPC courses, we have seen students asking for long batch allocations for a program that does not terminate, bringing down compute nodes due to creating too many processes, filling up local disks instead of writing files to a distributed file systems, etc. Although such mistakes can occur with any user, "unleashing" dozens of novices on a production platform at once tends to, unsurprisingly, increase the number of user complaints significantly. Even if a platform is dedicated to the students in a course, another logistical challenge is that students share this platform among themselves, which leads to possibly unhealthy competition (e.g., batch queue competition in the hours before an assignment due date). The above can result in non-trivial effort and time commitments by the course instructor (e.g., platform monitoring, interaction with platform administrators, managing user interactions), which could be better invested into course preparation or delivery.

Pedagogical Challenges -Even assuming that access to one or more representative HPC platforms is provided, using these platforms still raises challenges from a pedagogical standpoint. First, students must be trained to use the platform, which entails learning how to use various software infrastructures, the most complex of which is often a batch scheduler, as well as various usage policies. Some courses could deliberately devote a large portion of the syllabus to teaching the practice and the details of the software infrastructures used in typical HPC environments. But for other courses (e.g., for a course focused on parallel algorithms and MPI programming), the time and effort spent learning the software infrastructure and usage policies, which is necessary for providing students with hands-on experience, could likely be better spent on more course-relevant learning objectives.

Second, and more importantly, students are exposed to platform configurations that are specific and limited to those of the HPC platform(s) provided to them. Unless several (large) platforms are available, many interesting phenomena simply cannot be experienced hands-on by students. The instructor is then forced to mention what would happen in hypothetical, but relevant, scenarios that are simply not available to students. For instance, in our experience instructors often find themselves make statements such as "if the network topology were different, this algorithm would behave differently", "if the platform were heterogeneous, there would be many interesting issues with data/load distribution", etc. These statements, albeit valid, fall short of deepening student interest, and thus understanding, as several important concepts may never be experienced hands-on. Finally, from a logistical perspective, when the platform in use changes from one incarnation of the course to the next, there can be a large burden placed on the instructor to ensure that previous assignments can still help students achieve the required learning objectives.

Third, real-world platforms are known to be "noisy" [START_REF] Hunold | Reproducible MPI Benchmarking Is Still Not As Easy As You Think[END_REF]- [START_REF] Tuncer | Diagnosing Performance Variations in HPC Applications Using Machine Learning[END_REF]. The observed performance of a program is subject to many effects, some deterministic and some non-deterministic, which make performance difficult to understand without deep knowledge about hardware/software infrastructures and without solid experimental and analytical skills. This is problematic since most students initially lack such knowledge and skills. Therefore, in order for them to acquire knowledge about fundamental HPC concepts and practices, the possible sophistication and depth of course assignments, and thus the learning experience, are diminished. At the simplest level, it is difficult to observe small performance variations due to platform noise, and thus it is difficult to teach effectively those techniques that result in marginal, but still desirable, performance improvements (e.g., using asynchronous communication to overlap some amount of computation with communication). In our experience, in some cases platform noise can even make it difficult for an instructor to grade programming assignments fairly based on observing executions of student's programs.

Fourth, it is by running experiments that students can develop their own understanding of the material, e.g., by validating/invalidating their own intuitions. Yet, running experiments on an HPC platform is not free as it requires time, electrical power, and in some cases funds. The number of experiments that students can perform is thus typically limited, as are their opportunities to deepen their independent understanding of the subject matter.

Fifth, a popular way to reach students is to develop and make available Massive Open Online Courses (MOOCs), so that students can acquire skills from anywhere and at their own pace. Developing an HPC course as a MOOC that provides students with valuable hands-on experience would require that access to a representative HPC platform be guaranteed for a large, diverse, and distributed student population, which can be done effectively only by a few institutions.

One approach to overcome the above logistical and pedagogical challenges is to use simulation, i.e., instead of running programs on a few actual HPC platforms, simulate the execution of programs on a software realization of arbitrary HPC platform configurations. Using simulation for HPC education was pioneered in the early 1990's [START_REF] Luque | A Quantitative Approach for Teaching Parallel Computing[END_REF]- [START_REF] Pears | Using the DiST Simulator to Teach Parallel Computing Concepts[END_REF]. More recently, the work in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF] demonstrates the benefit of using simulation for teaching HPC, with student surveys showing increases in both satisfaction and acquired knowledge due to the use of simulation. In this work we describe SMPI Courseware, a set of assignments that can be incorporated into any HPC course. These assignments are structured as progressive steps with well-defined learning objectives. The scope is distributedmemory parallel computing with the Message Passing Interface (MPI). The foundational technology is SMPI [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF], an MPI simulator provided as part of the SimGrid simulation framework [START_REF] Casanova | Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms[END_REF], [START_REF]The SimGrid Project[END_REF]. SMPI makes it possible to simulate accurately the execution of unmodified MPI applications on arbitrary platform configurations. Furthermore, simulations can be executed quickly on a standard laptop computer. The above is made possible thanks to the scalable and validated simulation models provided by SimGrid. SMPI Courseware thus does not require access to any HPC resource, and has been used already in graduate-level HPC courses as a complement to or completely replacing the use of HPC platforms.

In this work, we describe some of the assignments in SMPI Courseware, and showcase how simulation with SMPI makes it possible to achieve popular HPC learning objectives effectively. The rest of this paper is organized as follows. Section II discusses related work. Section III provides necessary background information on SMPI. Section IV describes some of SMPI Courseware assignments and how they capitalize on the simulation capabilities of SMPI. Finally, Section V concludes with a summary and future work directions.

II. RELATED WORK

Several authors have addressed the logistical challenge of procuring an HPC platform for educational purposes by establishing low-cost such platforms [START_REF] Ludin | LittleFe: The high performance computing education appliance[END_REF]- [START_REF] Abuzaghleh | Implementing an Affordable High-performance Computing for Teaching-oriented Computer Science Curriculum[END_REF] or by using virtualization and/or container technology [START_REF] Ivica | StarHPC -Teaching parallel programming within elastic compute cloud[END_REF]- [START_REF] Johnson | Designing, Constructing and Implementing a Low-Cost Virtualization Cluster for Education[END_REF]. Although these solutions offer compelling capabilities, such as exposing students to real-world software infrastructures, they impose limits on platform scale and/or may not be representative of production HPC platforms, especially in terms of performance behaviors. In this work, instead, we resort to simulation so as to be completely free of the need for a hardware HPC platform and can simulate arbitrary platform configurations. In what follows we review previous works that have also used simulation for the purpose of HPC education.

Simulation is used routinely as a pedagogic tool in many areas of the computer science curriculum. For instance, it is traditional to use simulation frameworks for teaching computer architecture and network concepts, since without simulation it can be extremely challenging to create hands-on experience opportunities in these domains. By comparison, the use of simulation as a pedagogic tool is relatively rare in the HPC domain, likely because in many cases it is possible to use some HPC platform for teaching purposes, albeit facing the logistical and pedagogical challenges identified in Section I. Several works in the early 1990's proposed using simulation for the purpose of parallel computing education. In [START_REF] Luque | A Quantitative Approach for Teaching Parallel Computing[END_REF] the authors argue that using a simulator via which students implement and execute parallel algorithms enhances the learning experience. A parallel algorithm is described as an abstract model, and a large part of the proposed approach is a graphical user interface with execution visualization capabilities (e.g., execution Gantt charts). A similar approach is proposed in [START_REF] Pears | Using the DiST Simulator to Teach Parallel Computing Concepts[END_REF], but in this approach programs are written in an actual programming language (Modula-P) and compiled to run in "simulation mode", which is also what we propose in this work. The author reports on increased student motivation and satisfaction. A textbook was developed that comes with a simulator and programming exercises in simulation [START_REF] Lester | The Art of Parallel Programming[END_REF]. In [START_REF] Hartman | Teaching parallel processing using free resources[END_REF], it is proposed to teach parallel computing using a SIMD architecture simulator as one of the options for teaching parallel computing at low cost. In all the above, the authors motivate their work by some of the same rationales as those outlined in Section I, namely: (i) acquiring a platform may not be possible (due to cost); (ii) learning is limited because experimental scenarios are limited by the platform at hand; and (iii) simulation makes it possible to easily observe and understand deterministic program executions that are not subject to any platform "noise."

More recently, the authors in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF] describe the parallel computing module of a M.S. degree in HPC, which relies on simulation as a foundational technology. They report on improvements in student grades and self perception of acquired knowledge, attributing these improvements to the use of simulation. Although there is overlap between the authors' motivations and our own motivations as outlined in Section I, there are two significant differences between this work and that in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF]. First, the work in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF] is based on the OPNET Modeler (now known as Riverbed Modeler [START_REF]Riverbed Modeler[END_REF]) simulation framework. This framework uses packet-level network simulation, which simulates the behavior of individual network packets in a discrete event simulation fashion. Although typically considered accurate, packet-level simulation of full-fledged applications is time-consuming (in part because the number of packets increases with the communicated data size). This problem is acknowledged by the authors in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF], and is the reason why they provide students with a software tool to run simulations on an remote HPC platform. By contrast, our approach relies on the SimGrid simulation framework, which uses scalable "macroscopic" network simulation models and provide high accuracy and orders of magnitude simulation speed improvements over packet-level network simulators [START_REF] Velho | On the Validity of Flow-level TCP Network Models for Grid and Cloud Simulations[END_REF]. As a result, full-fledged simulations of large-scale HPC applications can be executed in a as little as a few seconds on a student laptop. In our approach there is thus no need for an actual HPC platform. The second significant difference is that while with the approach in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF] students work with parameterizable predefined simulation scenarios to understand performance issues, instead with our approach students write standard MPI code that is then executed in simulation mode transparently. Thus not only can students achieve the learning objectives in [START_REF] Zarza | An Innovative Teaching Strategy to Understand High-Performance Systems through Performance Evaluation[END_REF] (and can be provided with predefined simulation scenarios as already implemented MPI applications if so desired), but they can also learn how to use MPI and can debug their MPI code in simulation mode.

Another recent work, Paralab teachware, is presented in [START_REF] Kozinov | Learning Parallel Computations with ParaLab[END_REF], which uses the Paralab system [START_REF] Gergel | ParaLab System for Investigating the Parallel Algorithms[END_REF] to teach parallel computing concepts and algorithms using simulation. Paralab itself is available for Windows and, based on what can be learned in [START_REF] Gergel | ParaLab System for Investigating the Parallel Algorithms[END_REF], implements idealized simulation models that may not be representative of real-world HPC platforms. For instance, network simulation seems to be based on a "latency plus data size divided by bandwidth" model. While simple models can have pedagogic value as they make understanding performance trends easier, these models can be far from the realities of HPC platforms. By contrast, in this work we use SMPI, which implements sophisticated validated simulation models that incorporate many features encountered in realworld platforms (e.g., protocol switching based on message size, network congestion). The Paralab teachware provides students with pre-packaged applications and a set of network topologies (with configurable parameters) for students to experience application performance for learning purposes. By contrast, this work allows students to write standard MPI programs and to simulate their executions on arbitrary platform configurations. Therefore, the scope of this work, and thus of the learning objectives that can be achieved, is larger. It would be straightforward to re-implement Paralab teachware entirely using our approach. III. SMPI: SIMULATED MPI SMPI [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF], which comes with the SimGrid [START_REF] Casanova | Versatile, Scalable, and Accurate Simulation of Distributed Applications and Platforms[END_REF], [START_REF]The SimGrid Project[END_REF] distribution, makes it possible to simulate the execution of MPI applications. SMPI works seamlessly with unmodified MPI programs written in C, C++, or FORTRAN. For instance, assuming the program is written in C, one simply compiles it with smpicc, instead of mpicc, and execute it with smpirun, instead of mpirun. The only difference between smpirun and mpirun is that smpirun takes one extra command-line argument, -platform. This argument is used to specify a description of the hardware platform on which the execution of the MPI program is to be simulated. This description is in XML and allows the specification of arbitrary platform configurations [START_REF] Bobelin | Scalable Multi-Purpose Network Representation for Large Scale Distributed System Simulation[END_REF]. At the most basic level, the platform description specifies a network topology between hosts, where network links have specified latencies and bandwidths, and hosts have specified hostnames, numbers of cores, and per-core compute speeds. The XML description schema provides higher-level constructs (e.g., a homogeneous cluster with a single switch) so that these descriptions can be easily written by hand if so desired. The SimGrid distribution comes with many example XML platform descriptions. Given an MPI program, simulating and comparing its executions on different platforms is easily done by running smpirun multiple times, each time with a different XML platform description file as specified by the -platform commandline argument. Because SMPI simulations are deterministic, differences between simulated executions of the same MPI program, provided this program is itself deterministic, are guaranteed to be solely due to the differences in the platforms. However, non-deterministic programs are subject to the same issues when simulated with SMPI as when executed with MPI on real-world platforms (e.g., race conditions caused by nondeterministic message receptions from multiple senders due to using MPI_ANY_SOURCE).

The basic principle behind SMPI simulations is as follows. The code of the MPI program is executed as is, but the MPI processes actually execute as threads in a single process, and thus share the same address space. This is depicted in Figure 1. Each time an MPI API function is called, control is handed off to SMPI where network operations are replaced by simulated delays. These delays are computed using the simulation models at the core of SimGrid. Each block of code in between two MPI API calls is benchmarked on the machine used to execute the simulation. This is possible because, with SMPI, MPI processes execute as threads in mutual exclusion.

The benchmarked execution times of each block of code can then be scaled and simulated as compute delays that correspond to the compute speeds of the hosts in the simulated platform. In this way, both communication and computation operations are simulated as computed delays by SMPI.

As described above, the SMPI simulation approach would have two severe drawbacks. First, the simulation of an MPI application with p processes in which each process computes for t seconds on machines of compute speed s, would run in time at least pts/s on a machine with compute speed s . Second, the simulation of an MPI application with p processes in which each process allocates m bytes of RAM would need to be executed on a machine with pm bytes of RAM. As a result, simulating long and/or large MPI application executions would be intractable.

To address the long execution time problem, SMPI can avoid executing the entire application code. While executing the entire code of an MPI application is required to obtain valid application output, the objective of most SMPI users is only to reproduce the performance behavior of the application. Once encountered again on the same or a different MPI process, the execution of a basic block of code can be simulated by SMPI without executing the code itself, but merely as a delay based on previously benchmarked executions of it. This is done by inserting code in the application at compile time. While the modified code likely produces incorrect application output, performance behavior can be preserved and simulation time is vastly reduced.

To address the large memory footprint problem SMPI makes it possible for MPI processes to share allocated memory regions (e.g., malloc may return a pointer to a memory region already allocated by another MPI process). Furthermore, SMPI implements a mechanism that allows the same memory page to be used multiple time as part of a single allocate zone or RAM. As a result, scenarios with large memory footprints can be simulated on a computer with small RAM capacity. Here again the application output is compromised but performance behavior can be preserved.

These two techniques, which are activated via straightforward and succinct code annotations, can only be viably applied to non-data-dependent applications, i.e., for applications in which branching does not depend on application data. In summary, SMPI makes it possible to simulate small-scale executions of any MPI application for validation/debugging purposes, but also possible to simulate large-scale executions of data-independent applications (at the expense of application output correctness) for performance/scalability analysis purposes, all on a single machine.

SMPI implements the MPI-2 standard and a portion of the MPI-3 standard. An important performance consideration when using an implementation of the MPI standard is the algorithms used for collective communication operations. This is because production MPI applications use these operations extensively, and their performance often determines overall parallel efficiency. The design space for collective communication algorithms is large, and thus many algorithms exist, exhibiting different performance characteristics depending on the network topology, the message size, and the number of communicating processes [START_REF] Faraj | STAR-MPI: self tuned adaptive routines for MPI collective operations[END_REF]. MPI implementations thus often select at runtime which algorithm to employ, which leads to relatively complex collective communication implementations. For instance, in OpenMPI, the MPI_Allreduce operation amounts to about 2,300 lines of code. It is necessary for an accurate MPI simulation framework to capture such complexity. To this end, SMPI implements all the collective algorithms and selection logic of both OpenMPI and MPICH [START_REF] Thakur | Optimization of Collective Communication Operations in MPICH[END_REF] as well as a few other collective algorithms and selectors from Star-MPI [START_REF] Faraj | STAR-MPI: self tuned adaptive routines for MPI collective operations[END_REF], the Intel vendor implementation of MPI, and MVAPICH2 [START_REF] Panda | The MVAPICH Project: Evolution and Sustainability of an Open Source Production Quality MPI Library for HPC[END_REF]. Overall, SMPI currently provides more than 120 different collective communication algorithms. Users can activate these implementations via command-line arguments passed to smpirun, which makes it straightforward to compare collective communication implementations for arbitrary application and platform scenarios.

To date, SMPI has been validated for representative applications and benchmarks [START_REF] Degomme | Simulating MPI applications: the SMPI approach[END_REF], and used to simulate largescale MPI application executions on a single machine [START_REF] Cornebize | Capacity Planning of Supercomputers: Simulating MPI Applications at Scale[END_REF]. SMPI has also been used to successfully simulate 51 unmodified MPI applications included in standard HPC application benchmarks/suites (ECP Proxy Applications, CORAL, Trinity, PRACE CodeVault, HPL, and Meteo-France), which are included as part of SMPI's nightly integration testing [START_REF]SMPI Integration Testing of MPI Proxy applications[END_REF].

Given the above capabilities and accomplishments, SMPI provides a compelling foundation for addressing the challenges outlined in Section I in the context of distributedmemory parallel computing and MPI programming education.

IV. SMPI COURSEWARE A. Objectives and Availability

The HPC community has developed sets of learning objectives as well as entire curricula to meet these objectives. The goal of SMPI CourseWare is not to define or provide a curriculum. Rather, its goal is to provide ready-to-use assignments through which students can achieve learning objectives in the context of distributed-memory parallel computing and MPI programming. Capitalizing on the capabilities of SMPI (see Section III), these assignments can be integrated into an HPC course without any need for an HPC platform. These assignments can and have been used with several modalities. These include in-class sessions during which instructors and teaching assistants provide instructional scaffolding, students working in groups outside of the classroom, or students working individually outside of the classroom.

To date, SMPI Courseware is simply available as a Web site [START_REF]SMPI Courseware[END_REF]. Instructors can either point their students to that site, or download and modify assignments as they wish. A first section of the site provides instructions on how to install and test SMPI. In the near future, this site will also provide pre-installed Docker containers with all necessary software and assignment files. Currently no mechanisms is available for retrieving assignment solutions, but they can be requested directly via e-mail to henric@hawaii.edu.

B. Sample Assignments and Usefulness of Simulation

In this section we showcase four of the assignments provided in SMPI Courseware. While we hope that these assignments have some pedagogic merit, our main focus here is to explain how, in the context of these assignments, simulation with SMPI enhances the student learning experience. In several cases, the use of simulation unlocks learning opportunities that would remain out of reach otherwise. We refer readers interested in the details of the assignments to the SMPI Courseware site [START_REF]SMPI Courseware[END_REF]. In what follows we also report on our own experience using SMPI in our courses. No formal user studies have been conducted to date. But we believe that what we observed in the classroom, although for now anecdotal, is nevertheless enlightening.

1) Parallel Julia set computation: SMPI Courseware provides an introductory assignment in which students implement a parallel Julia set computation. The learning objectives include (i) to understand the principle of SPMD (Single Program Multiple Data) programming; (ii) to understand and implement the concept of data distribution in distributed memory programming; (iii) to understand and implement some process communication/synchronization feature; (iv) to achieve mastery of essential MPI calls.

The first (optional) step of the assignment is to implement a sequential program that generates a jpeg image of a Julia set with dimensions provided as command-line arguments. The rest of the assignment consists in parallelizing this program in a sequence of baby steps. Students are led to discover the concepts of partitioning the work across MPI ranks in an SPMD fashion.

In this assignment, student must understand and implement the initially difficult notion of data distribution across MPI ranks. As expected, the assignment imposes the constraint that if the Julia set consists of n pixels, then each process in a p-process execution can allocate memory for no more than n/p pixels. As a result, students are exposed to the (initially quite difficult) notions of local indices and global indices: computing a pixel in the Julia set is based on the pixel's coordinates in the global image, but these pixels must be stored in a local array using some local indexing scheme. Students first use a 1-D strip data distribution and then a 2-D block distribution. This is likely the most challenging learning objective in this assignment. In fact, the rationale for having the program compute a Julia set is that students can detect and understand data distribution bugs quickly by visually inspecting the output image. As part of this assignment, therefore, the students must generate the output image. This requires that students use some communication in their MPI program since pixels blocks must be written in the right order to a single file. Students are free to implement some tokenpassing scheme in which each process writes its "piece" of the image to the file, or have a master process do all the writing after receiving image pieces from other processes. This assignment places very little emphasis on performance (beyond common-sense "it goes faster with more processors" or "it goes faster with faster processors" observations), and students are mostly focused on producing correct programs.

Even for this introductory assignment, the use of SMPI affords some pedagogical advantages. First, students can hit the deck running without having to learn all the logistics of using an HPC resource. Learning these logistics can be completely decoupled from learning parallel computing and MPI programming concepts (it can come later in the course if desired, or not be part of the course at all). Furthermore, even if acquired by the students, these logistics tend to lengthen the implement-test-debug cycle, which is particularly frustrating for an introductory assignment. For this particular assignment for instance, the image of the Julia file is created on the student's own laptop. It thus can be opened in some image viewer that automatically updates the image when it changes, i.e., each time the MPI program execution is simulated. This is to be contrasted with having students view the image on a remote HPC platform (often requires, e.g., multiple scp hops, ssh X forwarding). In our experience, these seemingly small details can end up getting in the way of the initial learning experience, requiring extra time and effort. Furthermore, "technical difficulties" do occur on HPC platforms in spite of the instructor's best effort, especially for a large class size.

A consequence of the above is that it is possible to hold an effective, efficient, and relatively short in-class development session in which students perform the full assignment under the guidance of the instructor. This makes it possible to address the inevitable initial difficulties with MPI programming in the classroom, focusing 100% on MPI programming issues. This is by contrast with our experience using real-world HPC platforms at the introductory level, in which attempts at such sessions are hindered by the logistics of using these platforms. One option that is sometimes used for introductory assignment is to have each student run MPI on their laptop in class (using a hostfile with only localhost entries). But in this case no performance benefit due to parallelism can be observed beyond the number of cores, which we have found to be an underwhelming first parallel computing experience for the students. With SMPI, instead, students are able to experiment right away with using arbitrary numbers of hosts, and witness the performance benefits. As a result, we have observed that well-prepared students who go through the assignment relatively quickly are able to discover on their own several parallel computing principles simply by modifying platform descriptions and execution conditions. For instance, in the last instance of the graduate-level "Principles of High Performance Computing" course at the University of Hawai'i at Mānoa (UHM), during the 1h15min in-class development session in which students go through this assignment with help from the instructor, several students were able to independently discover the fundamental notions of weak scaling and strong scaling. We contend that such invaluable independent learning would have been difficult to achieve if an actual HPC platform had been used for this session.

2) Physical topologies and communication patterns: One assignment in SMPI Courseware has students implement several versions of a broadcast. The learning objectives include: (i) to be proficient with a large subset of the MPI API for pointto-point communication; (ii) to be able to implement moderately complex communication patterns efficiently; (iii) to experience and understand the effect of network topology on communication patterns; (iv) to experience and understand the trade-offs between message size, network bandwidth, and network latency; (iv) to be able to conduct experimental performance comparisons of competing implementations and draw sensible conclusions; and (v) to become aware of the large size of the design space for collective communication operations and of the challenges involved.

After implementing a naive broadcast in which a message is sent by one process to n other processes via n MPI_Send calls, the students implement a broadcast first using a ring communication pattern, and then using a binary tree communication pattern. This assignment, if done entirely, is challenging as many students are not necessarily used to thinking about, let alone implementing, communication patterns. Throughout the assignment students compare their implementations with each other, and with MPI_Bcast as provided by several implementations of the MPI standard. They perform these comparisons using various network topologies, varying message size, and varying network bandwidth and latency values. They thus get to experience relevant performance effects (e.g., for small messages and/or high bandwidth the broadcast is latency-bound, if the communication pattern matches the physical topology then the broadcast is more efficient).

The obvious way in which simulation makes this assignment effective, or even possible, is that the HPC platform, and more specifically its network topology, is entirely simulated. As a result, it is possible to provide students with many network topology configurations. These configurations can be hypothetical and used for confirming expectations (e.g., the ring, resp. tree, broadcast perform well on a platform whose physical topology is actually a ring, resp. tree), or can be representative of actual network topologies used in HPC platforms (e.g., a single switch, a shared backplane, a fat-tree structure, a k-ary n-cube). Also, thanks to simulation, students can tweak platform descriptions to test hypotheses. For instance, it is possible to vary network latencies to understand the impact of network latency as message size varies. The scale of the platform can also be changed at will to explore scalability limits of the implementations. Finally, because SMPI provides many implementations of collective operations from popular implementations of the MPI standard (see Section III), students can experiment with all these implementations. This allows students to see first-hand that these implementations are not all equal, and that some of them work better than others in some regimes. This brings home the message that the design space for collective communication algorithms and their implementations is large.

All the above would be at best very difficult if teaching with real-world HPC platforms (i.e., changing network topologies is not possible, installing several MPI versions on a production platform is typically problematic, scale is limited). Instead, simulation makes it possible to provide students with a wide range of relevant scenarios, makes it possible to students to design their own scenarios, and in the end makes the learning objectives easier to achieve because the entire gamut of relevant effects can still be experienced hands-on.

3) Pipelined and asynchronous communication: One sobering (on purpose) conclusion from the assignment above is that popular MPI distributions provide broadcast implementations that are orders of magnitude faster that the implementations developed by the students. Another part of that assignment is thus focused on increasing broadcast performance. The additional learning objectives include: (i) to understand how splitting a message into chunks can increase throughput; (ii) to understand how asynchronous communication can increase throughput; (iii) to be able to enhance a communication patterns and implement a version of it that achieves high throughput using message splitting and asynchronous communication; and (iv) to be suitably impressed by the performance of production broadcast implementations provided by various MPI implementations.

In this phase of the assignment the students enhance their implementations by first splitting the message into chunks. Via experiments they draw conclusions about the best chunk size to use. They then use asynchronous communications to increase throughput further by overlapping sending and receiving operations. This is typically a challenging part of the assignment, as many students find it difficult to use asynchronous communication primitives in a way that actually achieves increase in throughput as well as correctness. In this assignment, student can also witness first-hand how impressive some production MPI_Bcast implementations can be performance-wise. For instance, once they have developed a ring-based broadcast that splits the message into chunks and uses asynchronous communication to benefit from communication pipelining, students experimentally determine the best chunk size to use for a given message size, a given total number of processes, and a physical platform that is an actual ring. In this case, their naive implementation does outperform the best MPI_Bcast implementations, but this improvement is typically marginal. And yet, the MPI_Bcast implementations do not know the topology structure and do not get to pick a good chunk size based on experiments. Furthermore, when changing the topology, the number of processes, and the message size, students then see that their own implementation can be outperformed by the adaptive MPI_Bcast implementations by orders of magnitude. In the graduate-level "Principles of High Performance Computing" at UHM, this assignment provides the motivation to explore what algorithmic and implementation techniques are used by these "Swiss army knife" MPI_Bcast implementations, often resulting in some students extending this assignment into their capstone course project, all in simulation with SMPI.

Most of the considerations in Section IV-B2 on why simulation is key to achieving learning objectives apply for the above learning objectives as well. Another consideration here, however, is that simulated executions are "clean". In other terms, they are reproducible and well-structured because they do not suffer from idiosyncratic effects seen in real-world platforms [START_REF] Hunold | Reproducible MPI Benchmarking Is Still Not As Easy As You Think[END_REF]- [START_REF] Tuncer | Diagnosing Performance Variations in HPC Applications Using Machine Learning[END_REF]. Therefore, it is possible to observe clearly performance effects of variations in algorithms, implementations, and platform configurations. As an example, consider Figure 2, which depicts Gantt charts of 2 real-world executions of an MPI program that performs a sequence of 92-process Binomial tree broadcasts, on 92 cores of a real-world cluster and on 4 cores of a single laptop computer, and one simulated execution of the same program on a specification of another real-world cluster. As seen in the figure, while the simulated execution shows clear, expected patterns, the real-world executions experience software/hardware effects that make the pattern more chaotic and difficult to understand. Furthermore, not shown on the figure, there are differences between multiple runs of the same program on the same platform. We include the results on a single laptop since this is one option when teaching an HPC course without giving access to an actual HPC platform and have student instead run MPI programs on their own computers. Even on a single computer, due to time-sharing of cores, the patterns are also complex and nondeterministic. Understanding the effects the impact real-world executions is interesting from a research and development perspective. But they impede learning for novice students by obfuscating fundamental principles. Finally, another problem with these platform effect is that even if an instructor verifies that some learning objective is easily achievable on a specific platform used for a course, in spite of platform effects, there is no guarantee their the same holds on other platforms. This makes it difficult to share assignments among instructors and institutions, and places a large burden on the instructor if the platform changes from one course instance to the next.

4) Scalability studies: SMPI Courseware provides an assignment in which students study the performance of a matrix multiplication application. The learning objectives include: (i) to be able to determine an application's performance bottleneck; (ii) to understand both strong and weak scaling concepts and be able to perform an experimental scalability study; (iii) to be able to develop algorithmic and implementation techniques for pushing scalability further.

Student actually implement the matrix multiplication implementation in another assignment (they can of course be provided with the implementation directly for this assignment, if so desired). That implementation is based on an outer product algorithm and uses a 2-D (non-cyclic) block distribution. The algorithm entails performing multicasts between rows and columns of processes, which are logically arranged in a 2-D grid. This is done by creating MPI communicators and performing broadcasts within these communicators. In terms of learning objectives, that assignment is a step up from the assignment described in SectionIV-B1.

In this assignment students first instrument their MPI program to measure communication and computation times separately. They then perform various scalability studies by keeping the matrix sizes fixed and increasing the number of processors, and by keeping the number of processors fixed and increasing the matrix sizes. They also experiment with various platform configurations that are more or less network-bound. Throughout, students draw conclusions regarding the weak and strong scaling properties of the program. Given the O(n 3 ) computation and O(n 2 ) data features of matrix multiplication, students can observe good strong scaling behavior but poor weak scaling behavior in several relevant scenarios (e.g., slow network, poor broadcast implementations).

The benefit of using simulation for the above is that largescale executions can be simulated on each student's laptop. This is because, as explained in Section III, SMPI provides techniques to reduce the CPU and RAM footprints of MPI applications. These techniques compromise the output of the simulated application (i.e., the computed matrix product is erroneous) but preserve the performance behavior when the application execution is non-data-dependent, which is the case for a matrix multiplication. As a result, students can push their scalability studies up to large scales, well beyond scales that could be achieved on HPC platforms used for teaching purposes in the majority of institutions.

V. CONCLUSION

Teaching HPC topics on a real-world HPC platform comes with many logistical and pedagogical challenges. In this work we have described SMPI Courseware, a set of ready-to-use assignments that target part of the HPC curriculum, namely distributed-memory parallel computing and MPI programming, and uses simulation to enhance the learning experience. More specifically, using SMPI, SimGrid's MPI application simulator, it is possible to have students implement MPI programs and run them unmodified in simulation mode for arbitrary platform configurations on their own computers. We have illustrated the pedagogic benefits of these capabilities by describing the content of and our experience with some of the assignments included in SMPI Courseware. SMPI Courseware has already been used in university courses, namely in several instance of the "Principles of High Performance Computing" graduate course at the University of Hawai'i at Manoa (USA) and of the "Parallel Systems" graduate course at the University of Grenoble (France). In some instances in-simulation assignments were used in addition to assignments on a realworld HPC platform, while in other instances all assignments in the course were in simulation. Although no user studies have been conducted, feedback in the classroom has been overwhelmingly positive. Simple questionnaires revealed that students did not find it frustrating to perform assignments in a "fake" environment. No student has complained that, instead of experiencing actual execution times first-hand, they merely see in the simulation output what the execution time would have been in the real-world. In those courses in which students used both real-world and simulated HPC platforms, students felt they learned a lot in simulation because they could tweak experimental conditions at will to explore ideas. Students also acknowledged that although the typical frustrations with realworld platforms (e.g., being "stuck" in the batch queue for a log time, being subjected to announced but regular platform down times, being forced to carefully plan job submissions) are valuable experiences, they do get in the way of learning if encountered for every assignment. A clear future direction for this work is to perform a thorough pedagogic evaluation in the classroom to obtain evidence-based qualitative and quantitative measures of the merit of using our approach for teaching parallel computing and MPI programming.

To date, SMPI Courseware contains seven assignments, most of which have already been used in university courses. We plan to augment the set of assignments in SMPI Courseware. While current assignments hit fundamental MPI programming and parallel computing learning objectives, many more assignments can be designed that capitalize on SMPI for the same or other learning objectives. For instance, current assignments do not include any master-worker programming, any stencil application programming, and only have students use the most basic collective communication operations. There are also several ways in which to further capitalize on simulation to achieve learning objectives that are often out of reach in traditional non-simulation-based courses. For instance, one important topic is heterogeneous computing. Many parallel algorithms for heterogeneous architectures are known, and in a traditional course one often merely describes their principles without students gaining any hands-on experience. With SMPI it is straightforward to have students experiment with arbitrarily heterogeneous (simulated) platform configurations. More generally, many compelling assignments are proposed/used in the HPC education community that could be part of SMPI Courseware (e.g., those presented at the eduPar and eduHPC workshop series, those available in popular HPC training opportunities such as Blue Waters and XSEDE). Developing in-simulation versions of such assignments would be straightforward in many cases and SMPI Courseware welcomes contributors. Although currently modest in size and scope, with both our own and outside contributions SMPI Courseware has the potential to become an invaluable resource for many HPC curricula world-wide.

A broader future direction is to use simulation for HPC education beyond the scope of MPI programming and traditional parallel computing. For instance, we are involved in a simulation project focused on scientific workflows [START_REF]The WRENCH Project[END_REF]. In a nutshell, this project uses SimGrid as a foundation to develop a simulation framework for quickly prototyping and simulating the execution of Workflow Management Systems. We are currently developing a set of pedagogic modules with reusable in-simulation assignments that target learning objectives centered around fundamental distributed computing principles (data locality, load balancing, resource management and scheduling, fault-tolerance). The use of simulation in this context is if anything even more compelling than in the context of distributed-memory parallel computing and MPI programming. Ultimately, we aim at developing a large pedagogic corpus that provides a wealth of reusable assignments that can be integrated across the HPC curriculum for both undergraduate and graduate courses.

Fig. 1 .

 1 Fig. 1. Realization of a multi-process MPI execution (left) as a multi-threaded simulated execution with SMPI (right).

Fig. 2 .

 2 Fig.2. Execution Gantt charts of 11 consecutive 92-process Binomial tree broadcasts of 400,000-byte messages implemented using MPI point-to-point communication primitives and with different broadcast roots. (a) Execution on six 16-core nodes (2.40GHz cores) connected via 10Gbps Ethernet in the Paravance cluster of the Grid'5000 testbed[START_REF]Grid'5000 Testbed[END_REF] using OpenMPI 2.0.2; (b) Execution on one 4-core laptop computer (1.30GHz cores) using OpenMPI 2.1.1; and (c) Execution on 92 nodes of the Griffon cluster of the Grid'5000 testbed as simulated using SMPI. Time spent in MPI_Send, MPI_Recv, and MPI_Finalize are indicated with different colors, and black arrows match sends to their respective receives.

ACKNOWLEDGMENTS

Some of the experiments in this work were carried out on the Grid'5000 experimental testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER, and other Universities and organizations (see https://www.grid5000.fr). Part of this work was supported through the HAC SPECIS Inria Project Laboratory.


Keywords: SThM technique, SPM thermoresistive wire probe, spatial resolution, ICs temperature measurement

Surface temperature measurements were performed with a Scanning Thermal Microscope mounted with a thermoresistive wire probe of micrometric size. A CMOS device was designed with arrays of resistive lines 0.35µm in width. The array periods are 0.8 µm and 10µm to study the spatial resolution of the SThM. Integrated Circuits with passivation layers of micrometric and nanometric thicknesses were tested. To enhance signal-to-noise ratio, the resistive lines were heated with an AC current. The passivation layer of nanometric thickness allows us to distinguish the lines when the array period is 10μm. The results raise the difficulties of the SThM measurement due to the design and the topography of ICs on one hand and the size of the thermal probe on the other hand.

I. INTRODUCTION

Semiconductor devices are continuously reduced in size for integration. They must withstand ever higher electric fields which induce severe degradation or total failure. Measurements of power sources with sub-µm scales have then become fundamental for microelectronics.

While far field optical methods such as laser reflectance thermometry and micro-Raman thermometry are limited by diffraction, the Scanning Thermal Microscope (SThM) is a relatively new technique allowing for such investigations [START_REF] Majumdar | Scanning Thermal Microscopy[END_REF]. The principle of the method is based on the control of the thermal near field interaction between a small thermal probe and a surface. The technique uses Atomic Force Microscopy (AFM) to keep the probe-sample interatomic force constant and to accurately scan the surface when imaging is performed. Its key element is the thermal probe. For fifteen years, various thermal probe prototypes (thermocouple or thermoresistive Manuscript received January 23, 2006. This work was supported in part by the French National Centre of Scientific Research (CNRS).

S. Gomès is with the Centre de Thermique de Lyon, UMR CNRS 5008, Institut National des Sciences Appliquées de Lyon, Université Claude Bernard Lyon 1, 20 Avenue Albert Einstein, 69621-F Villeurbanne cedex, France (phone: 334-7243-6428 ; fax: 334-7243-8811; e-mail: severine.gomes@insalyon.fr). As the other authors of this paper, she is member of the French CNRS Group (Groupement de Recherches CNRS 2503) called "Heat Transfer at Micro-and Nano-scales", coordinated by S. Volz (Groupement de Recherche Micro et Nanothermique, GDR CNRS 2503, Ecole Centrale Paris, Grande Voie des Vignes, 92295 Châtenay Malabry, France ; phone: 331-4113-1049; e-mail: volz@em2c.ecp.fr).

elements integrated at the apex of AFM probes) [START_REF] Majumdar | Scanning Thermal Microscopy[END_REF] have been proposed. We use the thermoresistive wire probe developed by Dinwiddie et al in 1994 [START_REF] Dinwiddie | Thermal conductivity contrast imaging with a scanning thermal microscope[END_REF]. Thermal heat transfers [START_REF] Gomès | Thermal Microscopy : Study of the thermal exchange between a probe and a sample[END_REF][START_REF] Gomès | Scanning Thermal Microscopy: Characterisation and Interpretation of the Measurement[END_REF][START_REF] Lefèvre | Thermal conductivity calibration for hot wire based DC Scanning Thermal Microscopy[END_REF][START_REF] Lefèvre | Nanoscale heat transfer at contact between a hot tip and a substrate[END_REF] and materials thermophysical properties [START_REF] David | Characterisation of thin films using scanning thermal microscopy[END_REF] were analysed with a submicrometric spatial resolution when the probe is heated by Joule effect . However its ability to measure quantitative temperatures on ICs at submicrometric scale are not yet proven. Fiege et al. [START_REF] Fiege | Front-and backside investigations of thermal and electronic properties of semiconducting devices[END_REF] have demonstrated that this tool can allow for the localization of failures in integrated devices at saturation or in breakdown condition but on larger scales. The SThM has also been used to investigate the mechanisms of electric degradation inside a metal-oxidesilicon (MOS) structure submitted to an increasing electric stress [START_REF] Gomès | Investigation of the electrical degradation of a metal-oxide-silicon capacitor by Scanning Thermal Microscopy[END_REF]. More recently, quantitative temperature variations on PN thermoelectric couples [START_REF] Patino-Lopez | Characterization of the thermal behavior of PN thermoelectric couples by scanning thermal microscope[END_REF] were measured.

The spatial resolution of scanning probes is determined by the characteristic length of the predominant physical mechanism governing the probe-sample interaction (and then implicitly by the probe size). In the case of the thermoresistive probe, Figure 1 illustrates the heat transfers operating between the probe and a sample: solid-solid conduction, conduction via the water meniscus and through the surrounding gas and near field radiation. Solid-solid contact and water meniscus with averaged radii 20-50 nm and 100-200 nm respectively [START_REF] Gomès | Thermal Microscopy : Study of the thermal exchange between a probe and a sample[END_REF][START_REF] Gomès | Scanning Thermal Microscopy: Characterisation and Interpretation of the Measurement[END_REF][START_REF] Lefèvre | Thermal conductivity calibration for hot wire based DC Scanning Thermal Microscopy[END_REF][START_REF] Lefèvre | Nanoscale heat transfer at contact between a hot tip and a substrate[END_REF] have been shown to favor the tip-sample heat transfer [START_REF] Dinwiddie | Thermal conductivity contrast imaging with a scanning thermal microscope[END_REF][START_REF] Gomès | Scanning Thermal Microscopy: Characterisation and Interpretation of the Measurement[END_REF][START_REF] Lefèvre | Thermal conductivity calibration for hot wire based DC Scanning Thermal Microscopy[END_REF][START_REF] Lefèvre | Nanoscale heat transfer at contact between a hot tip and a substrate[END_REF]. The detection of submicrometric area can then be expected in spite of the micrometric size of the considered tip. What is the real ability of the SThM in terms of spatial resolution and sensitivity, to detect a local heat dissipation on ICs devices ?

This work tackles the problem of the SThM response on active integrated structures by modeling and measuring several ICs configurations. A dedicated CMOS device was designed with arrays of resistive lines with periods of 0.8 µm and 10µm. The resistive lines are 0.35µm in width. ICs covered by passivation layers with thicknesses of 3,6µm and of a few nanometers were tested. In the last case, the topography of the arrays under the layer was shown. After the presentation of the experimental technique in the following part, the description of the samples and their used conditions are given and the modeling used to predict the thermal behavior of the component with thick passivation layers is presented. The results of the measurements with the SThM are next addressed and analyzed in section IV. Results obtained for a sample with the thinner passivation layer are also discussed. Finally, some concluding remarks and recommendations are given in section V. The experimental set up used in this study results from the combination of a SThM working in the DC mode and a sample submitted to an AC electric current (see Figure 2).

The SThM is a scanning probe microscope marketed by Veeco Corporation. As introduced before, the thermal probe is a thermoresistive element consisting in a platinum wire 5µm in diameter and 200µm in length. The wire is bent to form a loop [START_REF] Dinwiddie | Thermal conductivity contrast imaging with a scanning thermal microscope[END_REF]. The thermal sensor is used as a thermometer. It constitutes one of the legs of a Wheastone bridge. As the temperature of the probe varies, the corresponding change in the probe electric resistance leads to a modification of the voltage balance of the bridge and then of the output voltage (V AB ) of the circuit. The control and the amplification of the output voltage during the scan allow for generating the thermal image. The contrast of this image corresponds to the variations of the probe electric resistance and the temperature averaged on the length of the platinum wire. Those variations are due to modifications of the temperature of the sample surface. This contrast only provides qualitative information about the temperature field. The calibration of the technique is required for quantitative measurements. There actually exists no general calibration protocol because the measure depends on the tip-sample contact as well as on the geometry and thermal properties of the sample. The calibration of the technique constitutes one of the objectives of our further works. The thermal control unit being independent of the AFM system, topographical and thermal images can be obtained simultaneously.

When studying a resistive sample, a second probe is put on the electric contact of the considered specimen to inject a current in the sample to heat it by Joule effect. The sample can be submitted to a sinusoidal current I at a frequency ( )

/ 2 f ω π = : 0 s sin( t) I I ω =
. In this configuration, the internal heating of resistive elements is proportional with

2 2 0 0.5 (1 sin(2 t-/2)) I I ω π = +
and occurs at f s =2f. The contrast of SThM images is obtained by measuring the amplitude and the phase of V AB at the 2f harmonic by using a Lock-In Amplifier (LIA).

Let us notice that because the thermal behavior of the probe follows a low-pass filter law and because the time response in air is found to be about 200-300µs [START_REF] Buzin | Application of a Wollaston wire probe for quantitative thermla analysis[END_REF], a.c. operations at low frequencies are usually required to enhance the SThM sensitivity (<800Hz). In the experiments, presented in section IV of this paper, the frequencies of the thermal excitation of samples were chosen to be lower than the frequency cut-off of the tip so that the thermal response of probe follows the thermal response of the sample surface with no dephasing compared to the excitation.

III. THE ICS SAMPLES

A. Samples Design

The sub-micrometric ICs investigated consist in 0.35µm in width polysilicone strips resistors. As shown in Figure 3, the studied structures are composed with nine resistors in parallel and spaced of 10µm or 0.8µm. Resistors are insulated from their polysilicone substrate by a SiO 2 layer of about 300 nm in thickness (gate and field oxides in Figure 4). In a first and original configuration ('samples 1'), the resistors are covered by a 3.6µm in thickness SiO 2 passive layer. To obtain a second configuration ('samples 2'), the passivation layer of the original configuration has been chemically etched so that a thin layer of some nanometers in thickness covers the resistors (spaced of 10µm only in this case). Table 1 synthesizes these data and gives the topography aspect for each of the investigated samples .

In order to deal with the samples surface, a study of the topography of samples were performed by AFM. Figure 5 and Figure 6 give AFM images and topographical profiles representative of results obtained on the surface of samples of type 'samples 1' and 'samples 2' respectively. Very small features of a few nanometers in height that are the signature of the resistors locations under the 3.6µm in thickness passivation layer were measured on 'samples 1'. Given that so small topographical features are negligible facing the roughness of the micrometric probe, 'samples 1' were used to model flat chips including buried heating sources.

In Figure 6, large and deep holes are placed in a prominent position at the strips locations. 'Samples 2' were then used to simulate components with surface heating sources and topography.

B. Samples Heating Conditions

In all the experiments given in this paper, the resistive samples were submitted to a sinusoidal current (see in Table 2) to allow for SThM measurements in the ac regime. In the ac regime, LIA indeed provides a bandwidth reduction which yields higher signal-to-noise ratio than dc measurements.

C. Thermal Model

In order to predict the thermal response of the SThM for measurements on the samples of type 'samples 1', a model was created to simulate the a.c. thermal behavior of these samples. In this model, the geometry of the heating sources is assumed of such a kind that a simplification into a twodimensional problem can be performed (case of an infinite wires array). As shown in Figure 7, sample is described as a multiplayer constituted up to down of an oxide layer and a polysilicon substrate.

In ac regime, the heat propagation in each layer, indexed ' n' (n = 1 for the oxide and 2 for the substrate), is described by the harmonic Fourier equation:

( ) ( ) ( ) , , , , , , n n n n n S x y z T x y z i T x y z K ω α - Δ + = (1)
where the following T n (x,y,z)=T n (x,y,z).exp(-iωt) phase convention has been retained. In this equation, T n (x,y,z) is the temperature a.c. component; x, y, z are the space coordinates; t andω are respectively the time and the time angular frequency of the heating source (ω=2πf s ); α n and K n are respectively the thermal diffusivity and conductivity of the layer n and S n is the power density of the source in the layer n.

According with the configuration of our samples, S 2 =0 in the substrate. In the oxide layer, the heat sources distribution (nine heating lines spaced by a period of 0.8 µm and 10µm) is modeled by describing each heating line section as a limited flat heat source with power density P. This heating sources distribution generates from its plane (in z=z e in Figure 7) the thermal waves in the whole system. In our approach, the induced thermal waves satisfy the following conditions: -Continuity of both temperature and heat flow at the interface between the oxide and polysilicon substrate (in z = z S in Figure 7). -A.C. thermal bath (T(x,y,z=e)=0 at the back of the sample).

-Adiabatic sample surface (in z=0). This last assumption is supported by the fact that the convective and radiative heat transfer coefficient of a hot horizontal slab is less than 10W.m - 2 K -1 . This includes linearized radiation. The 1D heat transfer coefficient of the material between the resistances and the surface is thermal conductivity divided by thickness: 1.28/(3,6.10 -6 )=3,6.10 5 W.m -2 K -1 . The heat flux will be reduced 3,6.10 4 times when crossing the boundary. The boundary limit can therefore be assumed as adiabatic to describe the temperature field in the oxide above the resistances.

In each homogeneous layer of our sample description, the a.c. temperature field may be expanded as a double plane wave spectrum [12]:

( ) ( ) ( ) ( ) ( ) 1 , , ( , exp 2
, exp

n x y nx y n n x y n T x y z dk dk T k k i z T k k i z χ χ +∞ +∞ + -∞ -∞ - = + - ∫ ∫ % % (2) with 1/ 2 2 2 n x y n i k k ω χ α ⎛ ⎞ = -- ⎜ ⎟ ⎝ ⎠ ( ) 0 Re > n χ (3)
and where k x and k y are the Fourier space coordinates;

+ n T ~and - n
T ~are respectively the forward and the backward components of the 2D Fourier transform of the ac temperature field in the layer n. Each spectrum is related with the others by the boundary conditions given above. The development of these boundary conditions in the Fourier space then permits to precise the a.c. temperature field in each part of the system. At the sample surface, i.e. in z = 0, the 2D Fourier transform of the ac temperature field can be written as:

( ) ( ) ( )( ) 1 e 1 1 , , ,0 exp -i z 1 2 x y x y n iS k k T k k B K χ χ = + % % (4) with 1 1 exp( 2 ) 1 s A B i z A χ - = + (5)
where

( ) ( ) 2 2 2 2 1 1 2 2 exp( ) exp( 2 ) exp( ) exp( 2 ) s s s s i z i e z K A K i z i e z χ χ χ χ χ χ + - = - - (6) and ( ) ( ) , ( , )
x y S k k FFT S x y = % is the 2D Fourier transform of the ac heat source in z=z e . Equations ( 2) and ( 6) give access to the ac temperature field on the sample surface. In all the numerical simulations of this temperature field, given in this paper, we used 2D fast Fourier transform (FFT) algorithms (a bounded window and a bounded number of points; the 2D areas were limited here to 1200 X 1200 points). We here used a Gaussian distribution of the heating power for each of the limited flat heat sources describing the heating lines. This Gaussian approximation was made with an effective radius r at 1/e equal to the half of the line width (r=175mm) and an integrated value of power which is the double of the experimental one integrated on a cube of area 2r x 2r x b where b is the polysicon P1 resistor thickness (b=300nm). As in the experiments presented in this paper, the modulation frequency of heat sources was set to 140Hz in our calculi. In Table 3 are listed thermophysical properties and geometrical characteristics involved in the samples description.

Figure 8 gives the simulated amplitude of the a.c. temperature field at the surface of the sample with the resistors spaced of 0.8µm and heated by an input power of 70mW. It is clearly shown that the wires can not be thermally revealed by probing the sample surface. This is due to the heat spreading from the resistors in the SiO 2 layer which is too thick compared to the space between the heat sources.

Similar simulations were made for the structure with resistors spaced of 10µm under a power of 275mW. Results are given in Figure 9. In this case, the distance between the resistive lines appears large enough so that the a.c. temperature profile at the surface indicates the contribution of each resistors.

IV. STHM MEASUREMENTS, RESULTS AND DISCUSSION

The experimental set-up presented in section II was used to investigate the a.c. component of the temperature field on the surface of each sample described in section III. Measurements were performed in ambient.

A.'Samples 1': With Thick Passivation Layer

Figure 10 reports the thermal contrast in amplitude obtained at 140Hz and the thermal signal corresponding with two scan lines drawn in the middle of the structure surface and along but perpendicularly to the resistors lines for the wire array with a 0.8µm period. It clearly appears that the wires cannot be distinguished. As found by simulation with the model in the section III, the wires array can be modelled as a single linear heat source along which no axial temperature gradient occurs at the level of the scanned area.

Figures 11 and12 were obtained for the array with the 10µm period. They reveal a temperature gradient due to the presence of the cold border of the structure. Contrary to the numerical simulations (see Figure 9) no local heating allows for distinguishing individual wires. A combination of effects related to the measurement technique can explain such results:

(i) As reported in the introduction of this paper, thermal conduction through the air contributes to the heat transfer between the sample and the probe. Its contribution is not negligible and the more so as the scanned sanple is thermal insulating material, as it is the case for our sample. Its radius of action is larger than the solid-solid contact area and the water meniscus width. The corresponding contact radius can be several micrometers due to the size and the geometry of the tip [START_REF] Lefèvre | Thermal conductivity calibration for hot wire based DC Scanning Thermal Microscopy[END_REF]. As a consequence, SThM is sensitive to variations of the heat rate (exchanged between the tip and the sample) averaged on a micrometric area. This may lead to a smoothing of the detected temperature field on the sample surface. (ii) A local variation of the heat loss from the sample to the tip is detected only if the SThM sensitivity allows for it. We determined the order of magnitude of the sensitivity corresponding to our experimental conditions by use of the following expression [13]:

( ) ( ) 1 1 ( ) AB p p t h ch mL V s E T L mc hm L SK Rm s h m L ω ω α - ∂ = = ∂ - (7)
where m²=jω.a -1 +h.p.(K p .S p ) -1 , and E=CVR 1 R p /(R 1 +R p )² is a function which depends on the electronic components used in the amplification chain stage. V ABω and T ω are respectively the measurement system output voltage and the top sample surface temperature at the modulation pulsation ω. α, a, K p R th , S p , L, p and h are respectively the temperature coefficient of the probe electrical resistance, the thermal diffisivity and the thermal conductivity of the platinum probe tip, the total thermal tip-sample contact resistance, the section of the probe, the half-length and the circular perimeter of the platinum wire, and the platinum wire convection-radiation coefficient. We calculated this sensitivity for the frequency of 140Hz. In Table 4 are listed the all used SThM thermal resistive probe properties and acquisition circuit characteristics used in this determination. For the discussed experiment, the sensitivity varies between 10 to 70µV.K -1 on the range of the possible values of the thermal resistance of the tip-sample contact: 5.10 4 -10 6 K.W -1 [START_REF] Gomès | Scanning Thermal Microscopy: Characterisation and Interpretation of the Measurement[END_REF][START_REF] Lefèvre | Thermal conductivity calibration for hot wire based DC Scanning Thermal Microscopy[END_REF][START_REF] Lefèvre | Nanoscale heat transfer at contact between a hot tip and a substrate[END_REF]. The used lock-in amplifier provides a stable signal up 10µV with our bridge. A local temperature increase of about 1K above each resistor is predicted from numerical simulations (see Figure 9) but no thermal resistance between the resistive lines and the passivation layer was taken into account in our model. The effective presence of such thermal resistances confines heat in the sample oxide. Our simulations then certainly overestimate the amplitude of the surface ac temperature field. In these conditions, the different heated lines could not be to be detected.

A smoothing of the detected temperature field on the sample surface due to a thermal exchange on a micrometric area, a poor thermal conductance between the used probe (of which the apex surface is not well known) and a lack of sensitivity of our set up can then explain the presented results. The limit of detection of the technique would then be reached in our experiments. As a result, the spatial resolution of the technique seems here larger than 10µm.

B. 'Samples 2': With Thin Passivation Layer

Figure 13 gives the topography and the thermal contrast obtained with the SThM above one of the nine resistive lines for a sample of type "samples 2". The array was submitted to a 11 mA sinusoidal current at f s = 110Hz. The contrast of the thermal image clearly reveals the heating of the resistor. But the comparison of topography profiles given by Figure 13.a and Figure 6.b shows that the thermal probe cannot follow the surface due to its large curvature radius.

As illustrated by Figure 14, this leads to surface dilatation phenomena in the topographical image obtained with the thermal probe. The contrast of the thermal image is also affected by this dilatation effect. The heat rate exchanged between the tip and the sample is indeed directly proportional to the thermal contact area. An artifact is then included in the figure 13.b due to a topography-thermal coupling. Note that the dips on both sides of the wires are insulators. If the surface was flat, heat spreading might also hide the real size of the wires.

For this sample, the SThM mounted with the 5µm wire allows to detect submicrometric elements but the artefact due to topography-thermal coupling enlarges the size of the wires by a factor of 10.

IV. CONCLUSION

AC surface temperature fields of several ICs including arrays of resistive lines with submicrometric sections were investigated with a SThM. The results for flat chips including buried heat sources in one hand and components with apparent heat sources in the other hand highlight the irrelevance of the SThM to handle complex ICs design and topographies, especially due to the size of the thermal probe itself.

When an IC passivation layer is too thick compared to the space between submicrometric integrated heat sources to permit a surface temperature contrast, submicrometric elements resolution will not of course be achieved due to heat spreading in this passivation layer. In the opposite case, the large area of the thermal contact between the probe and the sample, resulting from the thermal conduction through the surrounding gas, may limit the spatial resolution. In order to improve this resolution, one of our further works will concern measurements in vacuum.

The poor resolution obtained in our experiments may also be explained by the lack of sensitivity of our set up. Parameters defining this sensitivity, such as probe-sample thermal conductance and the apparatus for signal generation and processing, limit the minimum measurable heat rate between the probe and the sample and temperature variation. The spatial resolution is then strongly linked with these parameters to be specified.

Finally, we also show that the spatial resolution may also be affected by topography-thermal coupling. The large curvature radius of the probe may particularly lead to large dilatation of the ICs topography and surface temperature.

ICs appear as interesting tools to benchmark the spatial resolution of thermal metrologies such as SThM. The thermal design and the composition of the structure has however to be precisely defined and controlled to make quantitative temperature measurements. Flat samples with various and well known physical properties and dimensions are notably needed to develop an accurate thermal model of the probe-sample system in order to finally achieve the calibration of the technique.

First A. Author (M'76-SM'81-F'87) and the other authors may include biographies at the end of regular papers. Biographies are often not included in conference-related papers. This author became a Member (M) of IEEE in 1976, a Senior Member (SM) in 1981, and a Fellow (F) in 1987. The first paragraph may contain a place and/or date of birth (list place, then date). Next, the author's educational background is listed. The degrees should be listed with type of degree in what field, which institution, city, state or country, and year degree was earned. The author's major field of study should be lowercased.

The second paragraph uses the pronoun of the person (he or she) and not the author's last name. It lists military and work experience, including summer and fellowship jobs. Job titles are capitalized. The current job must have a location; previous positions may be listed without one. Information concerning previous publications may be included. Try not to list more than three books or published articles. The format for listing publishers of a book within the biography is: title of book (city, state: publisher name, year) similar to a reference. Current and previous research interests ends the paragraph.

The third paragraph begins with the author's title and last name (e.g., Dr. Smith, Prof. Jones, Mr. Kajor, Ms. Hunter). List any memberships in professional societies other than the IEEE. Finally, list any awards and work for IEEE committees and publications. If a photograph is provided, the biography will be indented around it. The photograph is placed at the top left of the biography. Personal hobbies will be deleted from the biography.

The"

  Micro and Nano Scales Heat Transfer" French CNRS research network was launched in 2002 and has been coordinated by Dr. Volz, CNRS researcher at the Ecole Centrale Paris in France. This network gathers several French laboratories, namely: -The CEntre de THermIque de Lyon, INSA-Lyon, Université Claude Bernard, Villeurbanne, -The LOP Lab of Ecole Supérieure de Physique Chimie Industrielle de Paris, -The FEMTO-ST Institute in the Université de Franche Comté, -The TIMA Lab in Grenoble, -The CPMOH in the university of Bordeaux I -The EM2C Lab in Ecole Centrale Paris, working together to solve engineering and scientific thermal issues at micro and nanometers scales.

ACKNOWLEDGMENT

The authors are thankful to the laboratory TIMA: Techniques of Informatics and Microelectronics for computer Architecture (Grenoble, France) for supplying the samples.


Anthony Randriamampianina

Pierre Maubert

Wolf G Früh

Peter L Read

Direct Numerical Simulation of structural vacillation in the transition to geostrophic turbulence

Keywords: Structural Vacillation, Barotropic instability, Geostrophic turbulence

The onset of small-scale fluctuations around a steady convection pattern in a rotating baroclinic annulus filled with air is investigated using Direct Numerical Simulation. In previous laboratory experiments of baroclinic waves, such fluctuations have been associated with a flow regime termed Structural Vacillation which is regarded as the first step in the transition to fully-developed geostrophic turbulence.

Résumé :

Le développement de fluctuations de petite échelle dans un écoulement convectif permanent d'air en cavité tournante barocline est analysé par simulation numérique directe. Des travaux expérimentaux antérieurs ont associé de telles fluctuations au régime de vacillation structurelle, considéré comme la première étape vers la turbulence géostrophique.

Introduction

The transition to disordered behaviour in the form of 'Baroclinic Chaos' provides an important prototypical form of chaotic transition in fluid dynamics. This is of particular geophysical relevance in the context of understanding the origins of chaotic behaviour and limited predictability in the large-scale atmospheres of the Earth and other terrestrial planets, such as Mars, and in the oceans (e.g [START_REF] Pierrehumbert | Baroclinic instability[END_REF]; [START_REF] Read | Wave interactions and baroclinic chaos: a paradigm for long timescale variability in planetary atmospheres[END_REF]; [START_REF] Read | Transition to geostrophic turbulence in the laboratory, and as a paradigm in atmospheres and oceans[END_REF]). For many years, the differentially-heated, rotating cylindrical annulus has proved a fruitful means of studying fully-developed, nonlinear baroclinic instability in the laboratory [START_REF] Fowlis | Thermal convection in a rotating annulus of liquid : effect of viscosity on the transition between axisymmetric and non-axisymmetric flow regimes[END_REF], [START_REF] Früh | Wave interactions and the transition to chaos of baroclinic waves in a thermally driven rotating annulus[END_REF]). Transitions within the regular wave regime follow canonical bifurcations to low-dimensional chaos, but disordered flow appears to emerge via a different mechanism involving small-scale secondary instabilities. Not only is the transition to geostrophic turbulence less well understood than those within regular waves, but also the classification and terminology for the weakly turbulent flows is rather vague. Various terms applied include Structural Vacillation or Tilted-Trough Vacillation which both refer to fluctuations by which the wave pattern appears to change its orientation or structure in a roughly periodic fashion. These fluctuations have been explained by the growth of higher order radial mode baroclinic waves [START_REF] Weng | Wave structure and evolution in baroclinic flow regimes[END_REF]), by barotropic instabilities, or by small-scale secondary instabilities or eddies, which lead to erratic modulations of the large-scale pattern [START_REF] Hignett | A comparison of laboratory measurements and numerical simulations of baroclinic wave flows in a rotating cylindrical annulus[END_REF]). Subsequent development within this so-called 'transition zone' leads to the gradual and progressive breakdown of the initially regular wave pattern into an increasingly disordered flow, ultimately leading to a form of 'geostrophic turbulence' [START_REF] Read | Transition to geostrophic turbulence in the laboratory, and as a paradigm in atmospheres and oceans[END_REF].

The numerical model

The physical model comprises a body of air contained between two vertical, coaxial cylinders held at constant temperatures and two horizontal insulating rigid endplates separated by a distance of d. The inner cylinder at radius r = a is cooled (T a ) and the outer cylinder at radius r = b is heated (∆T = T b -T a = 30K). The whole cavity rotates at a uniform rate around the axis of the cylinders. The geometry is defined by an aspect ratio, A = d/(ba) = 3.94 and a curvature parameter, R c = (b + a)/(ba) = 3.7, corresponding to the configuration used by [START_REF] Fowlis | Thermal convection in a rotating annulus of liquid : effect of viscosity on the transition between axisymmetric and non-axisymmetric flow regimes[END_REF] in their experiments with liquids. Following a long-established tradition (e.g. [START_REF] Fowlis | Thermal convection in a rotating annulus of liquid : effect of viscosity on the transition between axisymmetric and non-axisymmetric flow regimes[END_REF]), the analysis of the flow in a fixed geometry for a given fluid, here air at ambient temperature with P r = 0.707, is made by varying two main parameters: the Taylor number T a and the thermal Rossby or 'Hide' number Θ,

T a = 4Ω 2 (b -a) 5 ν 2 d , Θ = gdα∆T Ω 2 (b -a) 2 .
The Navier-Stokes equations coupled with the energy equation via the Boussinesq approximation are solved using a pseudo-spectral collocation-Chebyshev Fourier method associated with a second order time scheme.

The onset of SV regime

Figure 1 shows the equilibrated amplitude of the dominant azimuthal mode against the Taylor number. For steady solutions, the equilibrated amplitude of that wave mode is shown while for time-dependent solutions, the maximum and minimum as well as the mean amplitude are given. As such, the figure is a bifurcation diagram representing the sequence of three basic solutions, the axisymmetric solution on the x-axis and the m = 2 and m = 3 solutions pro- jected onto the same plane. The instability of the axisymmetric solution and the m = 2 solution branch are reproduced from the earlier study in [START_REF] Randriamampianina | Direct Numerical Simulation of bifurcations in an air-filled rotating baroclinic annulus[END_REF]. The present study focusses on evolution of the m = 3 solution branch when increasing progressively the rotation rate up to T a = 5. × 10 6 . A transition in the flow structure is observed, where the three-dimensional structure of the flow responds to a shift in the balance between gravity and the centrifugal force, quantified by the local Froude number, F r = Ω 2 r/g, and the dominant physical process changes from baroclinic instability to convection due to radial buoyancy. The transition of this convection to chaotic behaviour is fundamentally different from that observed in the transition to the chaotic flow observed at lower rotation rates. Rather than via a sequence of low-dimensional, quasi-periodic states, the large-scale convection developed small-scale instabilities, which have been previously suggested as the origin of Structural Vacillation (SV) within the transition to geostrophic turbulence.

The Froude number taken at the inner radius F r a ≡ Ω 2 a/g was used as the secondary axis on top of the graph in Figure 1 and the location of F r a = 1 is highlighted by the dotted vertical line at T a = 414, 000. The first vertical line, at T a = 239, 000, indicates where the centrifugal force equals gravity at the outer cylinder, F r b = 1. Below that Froude number or Taylor number, gravity is the stronger of the two forces everywhere in the annulus. It can be seen that F r a = 1 corresponds to a place on the solution branch where the radial structure of the steady wave changes. At the bifurcation, a higher radial mode of the same azimuthal wave grows to a finite amplitude while the first radial mode appears saturated around T a ∼ 10 6 and even decays. In terms of the physical quantities, this development results in strong velocity and temperature gradients near the boundaries and the concentration of the radial transport of heat and momentum in narrow plumes or jets. At the highest values of Taylor numbers explored, 2 to 5 × 10 6 , that flow then developed temporal fluctuations.

This change from a regular steady wave dominated by baroclinic instability to the transition zone characterized by small-scale fluctuations can be seen from the variations of eddy variables at two different values of rotation rates in the azimuth-height maps at mid-radius in Figure 2. At T a = 0.235 × 10 6 , the solution shows a pressure field which tilts westward with height and the temperature and vertical velocity fields which tilt eastward in such a way that the strongest upwelling is located around the strongest temperature gradient and strongest pressure gradient near the lower boundary. This behaviour is that of a typical baroclinic steady wave. At higher rotation rate T a = 4. × 10 6 , the pressure field clearly shows very little phase tilt with height while the temperature field exhibits highly concentrated plumes of hot and cold air which have very little slope with height, interspersed with broad regions with relatively very weak horizontal thermal gradients. The thermal plumes are aligned azimuthally with the regions of strong azimuthal pressure gradient, consistent with optimising the correlation between radial (geostrophic) velocity and temperature perturbations. Similarly, the regions of strong upward vertical velocity are also concentrated close to the strongest positive temperature anomalies, and vice versa, consistent with an optimisation of w ′ T ′ (where w ′ and T ′ represent departures in vertical velocity and temperature from their azimuthal mean values). Eventually, the decrease of the tilt results in a virtually vertical, 'barotropic' structure. Regions of strong downward motion (necessary to satisfy mass conservation) are concentrated in plumes or jets adjacent to the strong upwelling jets, forming 'cross-frontal' circulations which show some similarities with those inferred for atmospheric frontal regions in developing cyclones [START_REF] Hoskins | The mathematical theory of frontogenesis[END_REF]). The overall impression is that the flow in this region of parameter space is strongly nonlinear and much modified from the simple, linearly unstable Eady solution found at much lower Taylor number.

The SV characteristics

The spatial structure associated especially with the onset of structural vacillation is of particular interest, since it may be associated with bifurcations involving quite different spatial modes. As a means of isolating the dominant patterns in the flow, we have made use of a form of Empirical Orthogonal Function (EOF) analysis [START_REF] Preisendorfer | Principal Component Analysis in Meteorology and Oceanography[END_REF]), in which time sequences of spatial maps e.g. of temperature were analysed to obtain the covariance matrix of every spatial point with every other in the two-dimensional field. Figures 3 show the first six pairs of EOFs at T a = 4.0 × 10 6 , which together account for a little over 98% of the temperature variance. However, the first six EOFs appear as three conjugate pairs and represent 93.2% of the variance, indicating a slightly broader spread into the higher order patterns. The first two EOFs, e.g. Fig. 3(a), correspond to pattern dominated by m = 3, where the radial structure seems somewhat more concentrated towards the side boundaries, with only weak amplitude throughout most of the interior radii. EOFs 3 and 4, Fig. 3(b), show a pattern dominated by m = 6, but with even more contorted phase shifts with radius. The overall impression is of a pattern represented by m = 6 in azimuth and l = 4 in radius (where l is a nondimensional radial wavenumber). Subsequent EOFs display patterns with a reasonably simple l = 1 radial structure (except for (c), which also shows evidence for l = 4 dependence on r in EOFs 3 and 4), and increasing azimuthal harmonics of m = 3. The higher order EOFs in Figs 3(c) and (d) generally have a much simpler radial structure, with a simple maximum in amplitude near mid-radius and only weak phase-tilts with radius. They again appear in conjugate pairs, and represent other azimuthal harmonics of m = 3. The complex phase variations with radius in EOFs 3 and 4 demonstrate the growth of higher radial modes as previously supposed in Figure 1, though the patterns in (c) and (d) of Figs 3 would seem to suggest such anti-correlations ought to be strongest for m = 6 rather than the dominant m = 3.

Spatially averaged power spectra from the temperature residuals, after removing the steady wave, at mid-height and three radial positions, at mid-radius and around 15% from each wall, are shown in Figure 4. The main frequency (∼ 0.03T -1 , with the time scale T = 1/2Ω) is that of the emission of the perturbations described by EOFs 3 and 4. The spectrum falls off at all radial positions, with some distinctive peaks over the general decay. While the decay at higher frequencies is largely consistent with a f -3 law at all postions, a slight flattening in the frequency range between that of the main perturbation and about 0.1T -1 can be observed near the outer wall. The behaviour of the spectrum in that range appears to be closer to a f -5/3 law.

The spectral evidence suggests that the flow investigated here consists of a fairly steady large-scale convection pattern in the form of three pairs of hot and cold radial jets. From those jets, smaller perturbations are emitted at relatively regular intervals where a jet approaches a wall. The overall flow field responds in a cascade of faster (and presumably smaller) fluctuations which appear consistent with two-dimensional, quasi-geostrophic turbulence over a wide range of frequencies similar to results of DNS of geostrophic turbulence. Recently, similar energy spectra were observed for geostrophic turbulence in a square box [START_REF] Lindborg | The kinetic energy spectrum of the two-dimensional enstrophy turbulence cascade[END_REF]; [START_REF] Waite | The transition from geostrophic to stratified turbulence[END_REF]).

Conclusions

In the present work, we find the emergence of a new steady wave solution with complex radial structure, associated with a change in the balance of forces to a centrifugally-dominated regime, followed (at high Taylor number) by a further transition to a form of structural vacillation associated with regular m = 3 eddies driven by centrifugal buoyancy. The computations have been carried out on the NEC SX-5 of the IDRIS (CNRS, Orsay, France). The authors are grateful to the British Council and the CNRS for funding the collaboration between the French and UK partners in a joint programme Alliance.

Figure 1 :

 1 Figure1: Summary of the regimes obtained by the numerical model for a temperature difference of ∆T = 30K. The axis at the bottom shows the Taylor number while the axis at the top shows the ratio of the centrifugal term to gravity at the inner radius.

Figure 2 :

 2 Figure 2: Azimuth-height maps of eddy fields (with azimuthal flow removed) of a wave-3 simulation at (i) T a = 0.235 × 10 6 ; (ii) T a = 4. × 10 6 . (a) Pressure (contour interval = 0.25 dimensionless units), (b) Temperature (contour interval = 0.05, normalised to an imposed temperature difference of ±1.0) and (c) Vertical velocity (contour interval = 0.5 dimensionless units).

Figure 3 :

 3 Figure 3: Maps of the odd EOFs of the first six pairs EOFs of temperature at mid-height from a simulation of baroclinic waves in air at Ta = 4.0 × 10 6 . EOFs are shown in order of decreasing contribution to the total temperature variance at this level.

Figure 4 :

 4 Figure 4: Power spectra from the time series of the temperature after removing the steady wave: (a) near the inner wall, (b) at mid-radius, and (c) near the outer wall. All show a line ∼ f -3 but (c) also shows a line ∼ f -5/3 at the lower frequencies.

ème Congrès Français de MécaniqueGrenoble, 27-31 août 2007