Implementing and Evaluating Multi-Resource Scheduling in Slurm

Chapter 1

Project Overview

The exponential growth in computing power has enabled High Performance Computing(HPC) Systems to attack problems that are much larger and more complex. One example would be Data-Intensive applications. To meet intense I/O demand for these applications, Burst buffers are often employed in production systems. This heterogeneity forces HPC Schedulers which are mainly CPU Centric to consider multiple resources like Burst Buffers, network bandwidth in order to make optimized decisions for efficient performance [START_REF] Fan | Trade-Off Between Prediction Accuracy and Underestimation Rate in Job Runtime Estimates[END_REF][START_REF] Qiao | Joint Effects of Application Communication Pattern, Job Placement and Network Routing on Fat-Tree Systems[END_REF][START_REF] Qiao | Preliminary Interference Study About Job Placement and Routing Algorithms in the Fat-Tree Topology for HPC Applications[END_REF][START_REF] Fan | Exploiting Multi-Resource Scheduling for HPC[END_REF][START_REF] Yu | System-wide Trade-off Modeling of Performance, Power, and Resilience on Petascale Systems[END_REF].

In this Project, we consider Slurm, a well-known Scheduler in HPC Systems and incorporate multi-resource scheduling scheme. This is done by formulating the scheduling problem into a Multi-Objective optimisation problem and solving it using a multi-objective genetic algorithm [START_REF] Fan | Scheduling beyond cpus for hpc[END_REF]. Existing Slurm Scheduler supports Burst-Buffer Scheduling and allocates jobs from the waiting queue until either CPU or burst buffer is exhausted. This approach has a limited efficiency as the depletion of one resource can prevent the queued jobs from allocation and thus causing the under-utilisation of the other resources [START_REF] Fan | Job Scheduling in High Performance Computing[END_REF][START_REF] Li | The Effect of System Utilization on Application Performance Variability[END_REF][START_REF] Allcock | Experience and Practice of Batch Scheduling on Leadership Supercomputers at Argonne[END_REF]. As such, we implement a plugin called BBSched which when configured allows Slurm to perform multi-resource scheduling. The implementation is tested using Slurm Simulator [START_REF] Simakov | Slurm simulator github[END_REF].

This Project has been performed for a period of 12 weeks starting from January 14 2019 -April 26 2019 under the supervision of Dr.Zhiling Lan and her doctoral student Yuping Fan. The status of the project and any clarifications were discussed during the weekly meetings which are held every Thursday from 2:00 -2:30 PM. Besides for every two weeks,a Biweekly report has been submitted which summarises the work done in the past two weeks and the goal for the next two weeks. The reports can be viewed from the webpage at http://www.cs.iit.edu/ ~lan/moo.html.

Week

Project Tasks

Understanding Slurm

The main Objective of this project is to implement multi-resource scheduling scheme (named as "BBSched") in Slurm. For this, the first task was to understand Slurm. This is done by reading the Slurm design from [START_REF] Jette | Slurm design[END_REF] and going through the tutorials which have been delivered by the Slurm Programmers at various conferences and can be viewable at [START_REF] Jette | Slurm video tutorials[END_REF]. These tutorials and the design manual gave a good insight of Slurm by describing what Slurm is and what it is not; it's architecture which includes detailed description of Slurmd, Slurmctld, Command Line Utilities, Plugins, Communication layer and Security model; the design of Slurm Control daemon, it's subsystems: Node manager,Partition manager and Job manager, it's configuration and fault tolerance; the design Slurmd daemon; the working of each of the following command line utilites: scancel, scontrol, squeue, sinfo and srun; and the Job Initiation design.

This deep understanding of Slurm and its plugins helped in knowing that the multiresource scheduling scheme needs to be incorporated as a new scheduling plugin so as to keep the existing functionality undisturbed and configure the new scheduling scheme only when it is required by the system administrator. As such, the programmer guide and plugin implementation details of Slurm are read from [START_REF] Schedmd | Slurm programmer guide[END_REF] and [START_REF] Schedmd | Slurm plugin guide[END_REF]. Further, the Slurm code from github [START_REF] Schedmd | Slurm github[END_REF] has been analysed by installing it in the local machine. Specifically, the prevailing scheduling plugins namely sched/builtin and sched/backfill have been explored so as to know how the slurm control daemon (precisely the Job Manager) communicates with the scheduling plugins. Besides, various Slurm scheduling design papers [START_REF] Fan | Deep Reinforcement Agent for Scheduling in HPC[END_REF] [17] [START_REF] Fan | DRAS-CQSim: A Reinforcement Learning based Framework for HPC Cluster Scheduling[END_REF] [19] [START_REF] Fan | ROME: A Multi-Resource Job Scheduling Framework for Exascale HPC System[END_REF] have been examined.

Understanding Slurm Simulator

Time accelerated simulation of workloads is needed to verify the implementation of Multiresource scheduling in Slurm. As such, an Open source Slurm Simulator based on [START_REF] Jokanovic | Evaluating slurm simulator with real-machine slurm and vice versa[END_REF], [START_REF] Simakov | Slurm simulator: Improving slurm scheduler performance on large hpc systems by utilization of multiple controllers and node sharing[END_REF] and [START_REF] Simakov | A Slurm Simulator: Implementation and Parametric Analysis[END_REF] is used in this project. As, it is important to understand the work flow of the simulator before using it, the aforementioned papers have been read. Further, the code of the simulator from the github [START_REF] Simakov | Slurm simulator github[END_REF] has been analysed line by line to get a clear overview of its working.

Design of Multi-resource Scheduling in Slurm

The design of Multi-resource Scheduling in Slurm is shown in the figure below. Since, the simulator uses the Slurm Code for scheduling, the change in configuration automatically reflects to the Simulator and as such no specific design changes are needed in Simulator.

Implementation of BBSched in Slurm

The implementation of the new plugin "BBSched" is done and tested on docker container running on a virtual machine on Lightning Server of IIT. The IP Address of lightning server is 216.47.142.240 and virtual machine is 192.168.122.99. Inside the docker container, all the modified changes related to Slurm and Slurm Container are present in /home/slurm folder. The Step by Step manual written as a part of Appendix would give more details on Installation. Here, we specify only the implementation part.

Adding BBSched Plugin to Slurm

The Slurm related code is present in /home/slurm/slurm sim ws/slurm simulator and slurm simulator code is present in /home/slurm/slurm sim ws/slurm sim tools. Inorder to add the new plugin to Slurm code, a new directory called bbsched is created and added to Makefile.am both of which are located at /src/plugins/sched. Inside bbsched folder, four files are added namely bbsched.c, bbsched.h, bbsched wrapper.c and Makefile.am. Once all the files have been added, ./autogen.sh is run on the Slurm's top level directory so as to update the existing Makefile.in files and create the new Makefile.in file for the bbsched plugin. Besides, configure.ac is modified to identify the new Makefile to be built at Slurm configure time. Precisely the following line: src/plugins/sched/bbsched/Makefile is added to configure.ac inorder for the plugin to be built at Slurm Configure time.

Coding for Multi-resource Scheduling in Slurm

Here, the functionality added in each file of the new plugin: src/plugins/sched/bbsched is described.

• bbsched wrapper.c :

This file has the functions which are mandatory for Scheduler plugins and directly follows the Slurm Scheduler Plugin Api documentation [START_REF] Schedmd | Slurm scheduler api plugin[END_REF]. It includes: 1. Entry point function init() where the global initialisation is placed. This includes creation of threads specific to the bbsched plugin, acquiring locks and code specific to simulation mode like running the bbsched plugin from main simulation loop. 2. fini() function which releases the acquired mutexes, stops the plugin agent and suspends the execution of the bbsched thread.

3. slurm sched p reconfig function which rereads the configuration files in case on any changes. 4. All the other functions which have no specific implementation have been stubbed.

• bbsched.c

This file has the function which implements Multi-resource scheduling. Inorder to implement the code, the base paper [START_REF] Fan | Scheduling beyond cpus for hpc[END_REF] has been thoroughly read. Besides, CQSim [START_REF] Fan | Application Checkpoint and Power Study on Large Scale Systems[END_REF] [26] [27] [START_REF] Yang | Integrating dynamic pricing of electricity into energy aware scheduling for hpc systems[END_REF] has been installed and the Multi-Objective Optimisation (MOO) Scheduler in CQSim has been examined so as to code Slurm in a Similar way. As such the following functions are implemented: 1. run genetic algorithm function: This is the entry point of the Multi-objective optimization algorithm. It is called by the sort job queue function to update the priority ordering of the jobs in the waiting queue. The basic functionality includes, considering the jobs only pertaining to the window size and performing population initialization, parent selection, crossover, mutation and survivor selection. This process is done for around 500 iterations (generations) and the population after the last generation is returned.

2. init population function: This function generates a random binary value for each of the jobs in the window. So, For example if the number of jobs in window is 5 and random ordering is 10110, it implies that 1,3 and 4 jobs are selected; 2 and 4 are not.

We then check if this ordering of jobs is actually feasible. This is done by computing CPU Utilisation(described in detail in compute fitness value function). All the feasible orderings of jobs are returned at the last, the size of which is equal to the population size considered.

3. parent selection function: This function returns two randomly selected orderings from the populations initialised above. 4. crossover function: This function uses the parent list generated above and swaps the ordering at random position.It then checks for the feasibility of the swapped ordering and returns the feasible list as result. 5. mutation function: This function is used to introduce diversity so that the algorithm would not be trapped in the local optima. Specifically, it looks over all the orderings in the feasible population and flips the binary value at a random position. It them checks for the feasible orderings in the flipped list and returns them. 6. survivor selection function: This function returns the best among the recently generated orderings and separates them into two sets. Pareto Solutions in Set1 and rest in Set2. A solution is chosen as a Pareto solution, if improving one of its objectives would deteriorate at least one other objective [START_REF] Fan | Scheduling beyond cpus for hpc[END_REF]. If the set1 has less number of orderings than the population size considered, all the orderings in set1 are returned and the best among set2 are appended. 7. compute fitness value function: For each of the job orderings, this function takes into account the jobs which have a binary value of 1 (implies those that are selected) and sums up the number of nodes required by each of the jobs. It then looks for the total number of nodes using assoc mgr functions and also gets the currently allocated nodes. Then utilisation is computed as (total nodes -available nodes + required nodes)/total nodes. Here available nodes = total nodes -allocated nodes and required nodes is the sum of number of nodes required by each of the jobs in the ordering. 8. sort job queue function: This function takes the waiting list of Jobs updated each time by the Job Manager of Slurm and calls the run genetic algorithm function described above. It then takes the best among the orderings returns and the jobs which have binary value of 1 are added to the queue.

• bbsched.h This is a header file for the bbsched plugin

• Makefile.am This is written using the existing Makefile.am of sched/builtin as a model. It basically specifies the top source directory of Slurm and the files inside the current directory which need to be built at compile time.

Further, slight modifications are made to /src/slurmctld/simulator.c file to accomodate for bbsched plugin. These modifications include calling the plugin from Simulator main loop for job scheduling and including bbsched output in the log file. Further, bbsched job scheduling statistics are added to /src/slrumctld/slurmctld.h and debug flags to /slurm/slurm.h.in.

Chapter 3

Results

Here, Slurm is configured such that it manages 10 compute nodes and 5 users with 2 accounts. The Job trace consisting of 500 jobs is generated using a R Script and is similar to the log from Theta machine.

Experiment 1

Here, SchedulerType is configured as sched/bbsched in slurm.conf present in /home/slurm/slurm sim ws/sim/m folder. The simulation time for 500 Jobs was 23 seconds.

• Average Job Slow down: Between 5-7 Seconds

• CPU Utilisation -70-77%

Experiment 2

Here, SchedulerType is configured as sched/builtin in slurm.conf. The simulation time for 500 Jobs was 15 Seconds.

• Average Job Slow down: Around 6 Seconds

• CPU Utilisation -73% Note: The results with builtin (FIFO) scheduling are not direct as the simulator doesn't seem to use the builtin plugin of the Slurm and it only schedules 50 jobs out of 500 jobs given as input. The above result is an extrapolated value.

Observations

Here, we are optimising a single resource, so the effect of the Multi Objective Optimisation algorithm doesn't seem much. But if we add more resources like Burst Buffer, the efficient utilisation of multiple resources would be seen.

Besides, the following is the graph that shows the Simulation time Vs Number of generations by BBSched Scheduler.

Conclusion

In this project, Slurm and Slurm Simulator have been deeply explored and their installation is done on Docker. Further, Multi-resource scheduling has been implemented in Slurm by adding a new plugin called BBSched to the Scheduler Plugins. The workload trace to test the implementation has been generated based on theta log and the results were examined for 500 Jobs. Since, the CPU optimisation is considered in this project pertaining to time limitations, the utilisation seems to be nearly same. The future work would involve fixing the issue with slurm simulator when scheduler type is builtin and to incorporate burst buffer management in Simulator so as test the Multi-resource scheduling. To brief the directory structure which is based on [START_REF] Simakov | Slurm simulator github[END_REF], the actual slurm code is present in slurm directory and the simulator source code is in slurm sim tools folder. All the compiled Slurm code is present in bld opt folder and as such if any change in slurm code is made, we need to run the command: " make -j install " in this folder. slurm opt folder has the command line utilities in bin folder and Slurm control daemon, Slurm DB daemon and Slurmd daemon runnable files in sbin folder.

Figure 2 . 1 :

 21 Figure 2.1: Design of Multi-Resource Scheduler in Slurm

Figure 3 . 1 :

 31 Figure 3.1: Simulation Time Vs No. of Generations

Figure A. 1 :

 1 Figure A.1: Directory Structure of Installation

Table 1 . 1 :

 11 Overview of the Work specified Biweekly.

	No.	Task
	1 and 2	Understanding Slurm and MOO Scheduler
	3 and 4	Installation of Slurm and Running Slurm Controller with Slurmd daemons
	5 and 6	Coding for the new Scheduler Plugin "BBSched"
	7 and 8	Understanding Slurm Simulator,installing it on Docker
	9 and 10	Integrating the plugin with Slurm Code and Slurm Simulator
	11 and 12 Generating the workload trace, debugging to fix errors and result Analysis

  Perform Simulation Modify Scheduler type as SchedulerType=sched/bbsched in slurm.conf present at /home/slurm/slurm sim ws Run: /home/slurm/slurm sim ws/slurm sim tools/src/run sim.py -e \ /home/slurm/slurm sim ws/sim/micro/baseline/etc -s \ /home/slurm/slurm sim ws/slurm opt -d Note : Results are present in /home/slurm/slurm sim ws/slurm sim tools/src/results folder. Trace is present at /home/slurm/slurm sim ws/slurm sim tools/tutorials/results/micro cluster/test.trace.

	3. Installing Dependencies: e. Installing Depending Packages of R 12.
	a. Install MySQL (Maria DB) command: yum -y install mariadb-server The following commands are to be run inside R. So in the command Prompt type 'R' 9. Create a top level directory for Simulation This takes to the R Workspace where in the following commands needs to be executed Command: install.packages("ggplot2") cd /home/slurm/slurm sim ws install.packages("gridExtra") mkdir -p /home/slurm/slurm sim ws/sim/micro install.packages("cowplot") yum -y install mariadb-devel install.packages("lubridate") 10. Copy existing Slurm configuration to Simulation Directory systemctl enable mariadb install.packages("rPython") Command: systemctl start mariadb install.packages("rstudioapi") cd /home/slurm/slurm sim ws mysql secure installation slurm sim tools/src/cp slurm conf dir.py -o -s slurm opt \
	Note : slurm sim tools/tutorials/micro cluster/etc sim/micro/baseline
	b. Install Python command: yum -y install epel-release If the install package command asks to select a domain, please select the one corre-11. Populate SlurmDB sponding to the country you are located at. This would give a faster download speed. Run Slurmdbd in foreground mode
	yum -y install python34 python34-libs python34-devel python34-numpy python34-scipy 4. Create a workspace in /home/slurm:
	python34-pip command: Commands:
	yum install python34-setuptools cd /home/slurm export SLURM CONF=/home/slurm/slurm sim ws/sim/micro/baseline/etc/slurm.conf
	sudo easy install-3.4 pip mkdir slurm sim ws
	pip3 install pymysql cd slurm sim ws Run :
	pip3 install pandas
	/home/slurm/slurm sim ws/slurm opt/sbin/slurmdbd 5. Copy the Slurm Simulator code from Virtual machine: Note : Command: If there is an error saying "Broken toolchain: cannot link a simple C program" then Command : cd slurm simulator use the following command to install gcc: sudo docker cp slurm simulator 〈containerid〉:/home/slurm/slurm sim ws/ In a separate terminal populate SlurmDB using Slurm sacctmgr utility:
	yum install python-devel export SLURM CONF=/home/slurm/slurm sim ws/sim/micro/baseline/etc/slurm.conf
	yum -y install gcc-c++ 6. Prepare Building directory
	Command: export SACCTMGR=/home/slurm/slurm sim ws/slurm opt/bin/sacctmgr
	Note : cd ..
	If there is an error related to installation of numpy packages while installing pandas. mkdir bld opt $SACCTMGR -i modify QOS set normal Priority=0
	Do the following: cd bld opt $SACCTMGR -i modify QOS Name=supporters Priority=100
	cd /usr/lib64/python3.4/site-packages $SACCTMGR -i add cluster Name=micro Fairshare=1 QOS=normal,supporters
	rm -rf numpy rm -rf 〈numpy-egg-info〉 $SACCTMGR -i add account name=account1 Fairshare=100 7. Run slurm simulator configure command Command: $SACCTMGR -i add account name=account2 Fairshare=100
	/home/slurm/slurm sim ws/slurm simulator/configure \
	c. Install R -prefix=/home/slurm/slurm sim ws/slurm opt \ $SACCTMGR -i add user name=user1 DefaultAccount=account1 MaxSubmitJobs=1000
	yum -y install wget -enable-simulator -enable-pam -without-munge -enable-front-end \ $SACCTMGR -i add user name=user2 DefaultAccount=account1 MaxSubmitJobs=1000
	yum -y install R R-Rcpp R-Rcpp-devel -with-mysql-config=/usr/bin -disable-debug CFLAGS="-g -O3 -D NDEBUG=1" $SACCTMGR -i add user name=user3 DefaultAccount=account1 MaxSubmitJobs=1000
	yum -y install texlive-* $SACCTMGR -i add user name=user4 DefaultAccount=account2 MaxSubmitJobs=1000
	d. Install R Studio $SACCTMGR -i add user name=user5 DefaultAccount=account2 MaxSubmitJobs=1000 Command to compile the Slurm code: make -j install $SACCTMGR -i modify user set qoslevel="normal,supporters"
	wget https://download2.rstudio.org/server/centos6/x86 64/rstudio-server-rhel-1.2.1335-
	x86 64.rpm unset SLURM CONF
	yum -y install rstudio-server-rhel-1.2.1335-x86 64.rpm 8. Copy Slurm Simulator Code Terminate the first Slurmdbd
	sudo docker cp slurm sim tools 〈containerid〉:/home/slurm/slurm sim ws/

Appendix A

Manual for Multi-resource Scheduling in Slurm

The following Steps for installation of Slurm and Slurm Simulator can be directly executed on Centos distribution of Linux. If not, docker needs to be installed. If Docker is used, the following steps need to be followed before the actual installation [START_REF] Simakov | Slurm simulator github[END_REF].


Bernd Ammann

Nadine Gro E

Victor Nistor

ANALYSIS AND BOUNDARY VALUE PROBLEMS ON SINGULAR DOMAINS: AN APPROACH VIA BOUNDED GEOMETRY

We prove well-posedness and regularity results for elliptic boundary value problems on certain singular domains that are conformally equivalent to manifolds with boundary and bounded geometry. Our assumptions are satisfied by the domains with a smooth set of singular cuspidal points, and hence our results apply to the class of domains with isolated oscillating conical singularities. In particular, our results generalize the classical L 2 -well-posedness result of Kondratiev for the Laplacian on domains with conical points. However, our domains and coefficients are too general to allow for singular function expansions of the solutions similar to the ones in Kondratiev's theory. The proofs are based on conformal changes of metric, on the differential geometry of manifolds with boundary and bounded geometry, and on our earlier geometric and analytic results on such manifolds.

. Dans la suite, M sera une variété riemannienne lisse à bord de dimension m et E → M sera un fibré vectoriel hermitien équipé d'une connexion. Pour nos résultats, nous allons aussi supposer que M ait une géométrie bornée. Soit a une forme sesquilinéaire lisse sur T * M ⊗ E et P a : H 1 (M ; E) → H 1 (M ; E) * défini par la formule P a u, v := M a(∇u, ∇v)d vol g , pour u, v ∈ H 1 (M ; E). Nos espaces de fonctions seront les espaces de Sobolev pondérés de type Kondratiev, voir Équation (1). Nous supposons donée une partition ∂M = ∂ 0 M ∂ 1 M du bord en deux sous-ensembles disjoints et ouverts, ainsi que des conditions au bord B j d'ordre j, B j sur ∂ j M . Nos résultats sont alors:

(i) P satisfait la régularité dans les espaces pondérés f K ,2 (ρ) de l'équation (1) si, et seulement si les conditions au bord B = (B 0 , B 1 ) satisfont la condition de régularité de Shapiro-Lopatinski uniforme. Ces conditions sont satisfaites pour les opérateurs satisfaisant les conditions de Legendre fortes avec des conditions au bord mixtes (Dirichlet/Neumann). On obtient en particulier des résultats de régularité pour l'opérateur de Laplace avec conditions au bord mixtes, Théorème 6.

B.A. and N.G. have been partially supported by SPP 2026 (Geometry at infinity), funded by the DFG. B.A. has also been partially supported by the DFG SFB 1085 (Higher Invariants). V.N. has been partially supported by ANR-14-CE25-0012-01 (SINGSTAR). AMS Subject classification (2010): 58J32 (primary), 35J57, 58J05, 35B65.

(ii) Si, en plus des conditions de (i), P satisfait une inégalité de Hardy-Poincaré, alors le problème au bord associé à P est également bien posé.

En principe, la classe des domaines à laquelle nos résultats s'appliquent est assez large, mais pour des raisons d'espace et afin de réduire au minimum les détails techniques, nous considérons dans cette note principalement les exemples de domaines cuspidaux et wedge. L'ensemble des points singuliers de ces domaines est une sous-variété lisse compacte.

Introduction. We prove well-posedness and regularity results for systems of partial differential equations satisfying the strong Legendre condition with mixed Dirichlet-Neumann boundary conditions on certain singular domains. Our class of singular domains includes the class of domains with isolated conical singularities and thus they generalize the classical well-posedness result of Kondratiev [START_REF] Kondrat Ev | Boundary value problems for elliptic equations in domains with conical or angular points[END_REF]. Unlike Kondratiev's theory, singular functions expansions are not possible in our setting.

Let us briefly state our main result. Here are first our assumptions. Throughout this paper, (M, g) will be a smooth, m-dimensional Riemannian manifold with boundary and E → M will be a hermitian vector bundle with connection ∇ such that its curvature R E and all its covariant derivatives ∇ j R E , j ≥ 1, are bounded. For our results, we shall also assume that M has bounded geometry (a concept recalled below, see, however [START_REF] Ammann | The strong Legendre condition and the wellposedness of mixed Robin problems on manifolds with bounded geometry[END_REF][START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF][START_REF] Brezis | Functional analysis, Sobolev spaces and partial differential equations[END_REF][START_REF] Große | Uniform Shapiro-Lopatinski conditions and boundary value problems on manifolds with bounded geometry[END_REF] for the concepts not recalled in this paper). Let a be a smooth, sesquilinear form on T * M ⊗ E and P a :

H 1 (M ; E) → H 1 (M ; E) * be defined by the formula P a u, v := M a(∇u, ∇v)d vol g , for u, v ∈ H 1 (M ; E).
Recall that if a is uniformly positive definite, then a is said to satisfy the strong Legendre condition. If P = P a + Q, where Q is of order ≤ 1, we shall say that P satisfies the strong Legendre condition if, and only if, a does. This implies that P is strongly elliptic. For scalar operators, the condition that P satisfies the strong Legendre condition is actually equivalent to P being uniformly strongly elliptic. A smooth function f : M → (0, ∞) will be called an admissible weight if f -1 df has bounded covariant derivatives of all orders. Let f and ρ be admissible weights on M . If g 0 := ρ 2 g and ∇ 0 is the Levi-Civita connection associated to g 0 , then we can describe our function spaces as the following Kondratiev-type weighted Sobolev spaces

f K ,p (ρ) (M, g 0 ; E) := ψ | ρ j ∇ j 0 (f -1 ψ) ∈ L p (M, g 0 ; T * M ⊗j ⊗ E), (∀) j ≤ . ( 1
)
In our applications and in some of our results, the weight ρ is bounded. For simplicity, we will assume this throughout the paper. We will also assume that we have a partition ∂M = ∂ 0 M ∂ 1 M of the boundary in two disjoint, open subsets and that we are given boundary conditions B j of order j, B j on ∂ j M , satisfying the boundedness and smoothness conditions stated before Theorem 6. Our results are then as follows (for (M, g) with bounded geometry):

(i) P satisfies regularity in the weighted spaces f K ,2 (ρ) of Equation (1) if, and only if B = (B 0 , B 1 ) satisfies the uniform Shapiro-Lopatinski regularity conditions. These conditions are satisfied for operators satisfying the strong Legendre conditions with mixed (Dirichlet/Neumann) boundary conditions. In particular, the Laplace operator satisfies regularity for mixed boundary conditions (Theorem 6).

(ii) If, in addition to the conditions of (i), P satisfies a Hardy-Poincaré inequality, then P also satisfies a well-posedness result. We provide several examples of how to obtain the Hardy-Poincaré inequality. In principle, the class of domains to which our results apply is pretty large, but for reasons of space and in order to keep the technicalities to a minimum, we mostly consider the examples of canonical cuspidal and wedge domains introduced by H. Amann [START_REF] Amann | Function spaces on singular manifolds[END_REF], whose definition is recalled below. The set of singular points of such domains is smooth and compact (without corners). It is even a finite set for cuspidal domains. Some very general and nice results were obtained in [START_REF] Dauge | Strongly elliptic problems near cuspidal points and edges[END_REF][START_REF] Kozlov | Elliptic boundary value problems in domains with point singularities[END_REF] for certain domains with isolated point cusp singularities. Our methods are quite different, relying more on differential geometry, and thus allowing us to treat a large class of domains. Moreover, our coefficients are less regular than the ones in the references, but we lose the Fredholm properties and the singular function expansions obtained in [START_REF] Dauge | Strongly elliptic problems near cuspidal points and edges[END_REF][START_REF] Kozlov | Elliptic boundary value problems in domains with point singularities[END_REF] and in other papers. Algebras of pseudodifferential operators on manifolds with cuspidal points were considered in [START_REF] Schulze | An operator algebra on manifolds with cusp-type singularities[END_REF]. The index problem on such manifolds was considered in [START_REF] Lesch | On index formulas for manifolds with metric horns[END_REF]. We thank Herbert Amann for useful comments.

Manifolds with boundary and bounded geometry.

In this paper, (M, g) will always be a smooth, m-dimensional Riemannian manifold with boundary and E → M will be an vector bundle with metric and metric preserving connection. A smooth function f : M → (0, ∞) will be called a g-admissible weight if f -1 df has bounded covariant derivatives of all orders. We shall say that E has totally bounded curvature if its curvature and all of its covariant derivatives are bounded. We endow T M with the Levi-Civita connection ∇ associated to g. Recall that M is said to have bounded geometry if its injectivity radius r inj (M ) > 0 is positive and if T M has totally bounded curvature. We assume from now on that E is complex and it has totally bounded curvature.

Let us consider a codimension one submanifold H ⊂ M (i.e. hypersurface). Assume that H carries a globally defined unit normal vector field ν and let exp ⊥ (x, t) := exp M x (tν x ) be the exponential in the direction of the chosen unit normal vector. By II H we denote the second fundamental form of H. The following two definitions are from [START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF]. Definition 1. Let ( M , g) be a Riemannian manifold with bounded geometry. We say that

H ⊂ M is a bounded geometry hypersurface in M if it is a closed subset of M , if ∇ k II H L ∞ < ∞ for all k ≥ 0, and if here is r ∂ > 0 such that exp ⊥ : H × (-r ∂ , r ∂ ) → M is a diffeomorphism onto its image.
Definition 2. A Riemannian manifold (M, g) with boundary is said to have bounded geometry if there is a Riemannian manifold M with bounded geometry containing M as an open subset such that ∂M is a bounded geometry hypersurface in M . Remark 3. In [START_REF] Amann | Function spaces on singular manifolds[END_REF], Amann has introduced the class of "singular manifolds." A singular manifold (M, g 0 , ρ) is a Riemannian manifold with boundary (M, g 0 ) together with a singularity function ρ satisfying suitable properties. In particular, (M, ρ -2 g 0 ) is assumed to be a manifold with boundary and bounded geometry. Conversely, if (M, g) has bounded geometry and ρ is a g-admissible weight, then (M, g 0 := ρ 2 g) is a singular manifold with singularity function ρ. In the boundaryless case, this was first noticed in [START_REF] Disconzi | Some remarks on uniformly regular Riemannian manifolds[END_REF] (see also [START_REF] Amann | Uniformly regular and singular Riemannian manifolds[END_REF].) The singularity function ρ is seen to be a g-admissible weight. For manifolds with boundary, this result follows from [START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF] or [START_REF] Große | Sobolev spaces on Riemannian manifolds with bounded geometry: general coordinates and traces[END_REF]. The results of [START_REF] Amann | Function spaces on singular manifolds[END_REF] apply therefore to the setting of manifolds with boundary and bounded geometry endowed with an admissible weight. A triple (M, g, ρ) consisting of a manifold with boundary and bounded geometry and a bounded g-admissible weight ρ will be called an Amann triple.

Conformal changes of metric

. If h 1 , h 2 : X → (0, ∞), we shall write h 1 ∼ h 2 if h 1 /h 2
and h 2 /h 1 are both bounded. Let g 0 be a second Riemannian metric on M , whose Levi-Civita connection is denoted ∇ 0 . Let ρ, f : M → (0, ∞) be measurable functions and p ∈ [1, ∞]. Recall then from Equation (1) the definition of the spaces f K ,p (ρ) (M, g 0 ; E), which reduce to the usual Sobolev spaces if ρ, f ∼ 1. More precisely, if g := ρ -2 g 0 and if ρ is a g-admissible weight and f is continuous, then the weighted and classical spaces are related by

f K ,p (ρ) (M, g 0 ; E) = f ρ -m/p W ,p (M, g; E), 1 ≤ p ≤ ∞. (2)
(see [START_REF] Amann | Function spaces on singular manifolds[END_REF][START_REF] Amar | Sobolev embeddings with weights in complete riemannian manifolds[END_REF][START_REF] Ammann | Sobolev spaces on Lie manifolds and regularity for polyhedral domains[END_REF] and Remark 3). We drop the superscript p for p = 2:

K (ρ) (M, g 0 ; E) := K ,2
(ρ) (M, g 0 ; E) and so on. We assume from now on that g = ρ -2 g 0 . Example 4. A typical example is when M ⊂ R m is the closed unit ball, g 0 is the euclidean metric, and ρ = r λ , where r is the distance to the origin. Then (M, g := ρ -2 g 0 ) has bounded geometry if, and only if, λ ≥ 1. Moreover, We shall assume from now that (M, g, ρ) is an Amann triple (see Remark 3) and that f : M → (0, ∞) be a second g-admissible weight. In particular, ρ is a bounded g-admissible weight. We have seen in Equation (2) how the Sobolev spaces change with respect to conformal changes of metric. Recall that g 0 = ρ 2 g. For differential operators, a simple calculation based on L ∞ (M ; E ⊗ T M ⊗p ⊗ T M * ⊗q , g) = ρ p-q L ∞ (M ; E ⊗ T M ⊗p ⊗ T M * ⊗q , g 0 ) and the fact that ρ is bounded gives: Lemma 5. Let P be an order k differential operator on M and P 1 := ρ k P . We have that P satisfies the strong Legendre condition with respect to the metric g 0 if, and only if, P 1 satisfies the strong Legendre condition with respect to the metric

f := e -( r ) -, if λ = 1 + > 1, r = ρ, if λ = 1.
g = ρ -2 g 0 . If P has coefficients in W ∞,∞ (g 0 ), then P 1 has coefficients in W ∞,∞ (g).
A similar result is valid for the boundary differential operators appearing as boundary conditions.

Regularity and well-posedness. Let P be a second order differential operator. We assume from now on that we are given a partition of the boundary ∂M = ∂ 0 M ∂ 1 M into two disjoint, open subsets, as in [START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF], and order i differential boundary conditions B i on ∂ i M . See, for example, [START_REF] Brezis | Functional analysis, Sobolev spaces and partial differential equations[END_REF][START_REF] Lions | Non-homogeneous boundary value problems and applications[END_REF] for general results on boundary value problems on smooth domains, [START_REF] Dauge | Elliptic boundary value problems on corner domains[END_REF][START_REF] Kozlov | Elliptic boundary value problems in domains with point singularities[END_REF][START_REF] Nazarov | Elliptic problems in domains with piecewise smooth boundaries[END_REF][START_REF] Nazarov | Self-adjoint and skew-symmetric extensions of the Laplacian with singular Robin boundary condition[END_REF] for the case of non-smooth domains, and [START_REF] Ammann | The strong Legendre condition and the wellposedness of mixed Robin problems on manifolds with bounded geometry[END_REF][START_REF] Große | Uniform Shapiro-Lopatinski conditions and boundary value problems on manifolds with bounded geometry[END_REF] for more general boundary conditions involving projections. We assume that ρ 2 P , ρB 1 , and B 0 have coefficients in W ∞,∞ (g). The typical assumption is that P , B 1 , and B 0 have coefficients in W ∞,∞ (g 0 ), which means that they "stabilize" towards the singular points, as in [START_REF] Kozlov | Elliptic boundary value problems in domains with point singularities[END_REF], and this is a necessary condition for the existence of singular function expansions. In view of Lemma 5, our assumptions are thus weaker, but singular functions expansions are no longer available in general in our setting. Our more general setting may be needed in applications to non-linear PDEs and uncertainty quantification. Also, recall from [START_REF] Große | Uniform Shapiro-Lopatinski conditions and boundary value problems on manifolds with bounded geometry[END_REF] the uniform Shapiro-Lopatinski regularity conditions and that they are invariant with respect to conformal changes of metric. Combining this property with Equation (2) and with Lemma 5, we get: Theorem 6. Let P be a g 0 -uniformly elliptic second order differential operator acting on sections of E → M and B = (B 0 , B 1 ) be a boundary differential operator. We assume that P and B satisfy the g 0 -uniform Shapiro-Lopatinski regularity conditions. Then, for any ∈ N, there exists C > 0 such that, for all

u ∈ f K 1 (ρ) (M, g 0 ; E) u f K +1 (ρ) (M,g0;E) ≤ C P u f ρ -2 K -1 (ρ) (M,g0;E) + u f K 1 (ρ) (M,g0;E) + B 0 u f ρ -1 2 K + 1 2 (ρ) (∂0M,g0;E) + B 1 u f ρ -3 2 K -1 2 (ρ) (∂1M,g0;E)
.

In particular, we can take P to be a uniformly strongly elliptic scalar operator (such as the Laplacian P = ∆ g0 ), B 0 u = u| ∂ D M (the restriction) and B 1 u = ∂ a ν u. Let (M, g 0 ) be a Riemannian manifold with boundary. We now turn to the wellposedness on (M, g 0 ). Let h : M → (0, ∞), and A ⊂ ∂M be a measurable subset. We shall say that (M, A, E, g 0 , h) satisfies the Hardy-Poincaré inequality if there exists a constant C > 0 such that, for any u ∈ H 1 loc (M, g 0 ; E), u = 0 in L 2 (A), we have M |du| 2 g0 dvol g0 ≥ C M h -2 u 2 dvol g0 . The Hardy-Poincaré inequality implies coercive estimates, and hence well-posedness also for the associated parabolic and hyperbolic equations, as in [START_REF] Lions | Non-homogeneous boundary value problems and applications[END_REF]. The Hardy-Poincaré inequality is related to the Poincaré inequality, and hence to the concept of "finite width." If A ⊂ ∂M , recall that (M, A) is said to have finite width if dist(x, A) is uniformly bounded on M [START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF] (the distance between two disjoint connected components of M is +∞). Typically in our results, the set A will be an open and closed subset of ∂M .

Example 7. Again, a typical application is when g 0 is the euclidean metric on R m , r is the distance to the origin, and λ > 0, as in Example 4. However, in this case, we let ρ = r λ only for r < 1/2, but set ρ = r for r > 1 and M ⊂ R m is a closed, infinite cone with base a smooth domain of the unit sphere and with vertex at the origin. Then again, (M, g) has bounded geometry if, and only if, λ ≥ 1. Also, (M, ∂M, g) has finite width if, and only if λ = 1. Finally, (M, ∂M, g 0 , ρ) satisfies the Hardy-Poincaré inequality (for ρ) if, and only if, λ ≤ 1.

Recall that we have assumed (M, g) to be a Riemannian manifold with boundary and bounded geometry, ρ, f : M → (0, ∞) to be g-admissible weights, and g 0 := ρ 2 g. We define P a by (P a u, v) g0 := M a(∇u, ∇v)d vol g0 , with a sesquilinear form a satisfying the strong Legendre condition with respect to g 0 . Let ∂ a ν be the conormal derivative associated to P , see [START_REF] Große | Uniform Shapiro-Lopatinski conditions and boundary value problems on manifolds with bounded geometry[END_REF]. Combining Theorem 6 with the Lax-Milgram Lemma and with the fact that the Dirichlet and Neumann boundary conditions satify the uniform Shapiro-Lopatinski regularity conditions [START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF][START_REF] Große | Uniform Shapiro-Lopatinski conditions and boundary value problems on manifolds with bounded geometry[END_REF], we obtain: Theorem 8. We assume that (M, ∂ 0 M, E, g 0 , ρ) satisfies the Hardy-Poincaré inequality. Let P = P a satisfy the strong Legendre condition with all ∇ j a bounded. Then there exists η a,f > 0 such that, for |s| < η a,f and ≥ 1, we have an isomorphism

P a : ρf s K +1 (ρ) (M, g 0 ; E) ∩ {u| ∂0M = 0, ∂ a ν u| ∂1M = 0} → ρ -1 f s K -1 (ρ) (M, g 0 ; E).
In particular, we can take P = ∆ g0 , the Laplacian associated to g 0 . For = 0 the result remains true, once one reformulates it in a variational (i.e. weak) sense.

Examples. We include some basic examples.

Two dimensional domains. We consider a (disjoint) partition of the boundary ∂M = ∂ 0 M ∂ 1 M as above (so ∂ 0 M and ∂ 1 M are open and closed). Recall that P a is a second order differential operators on E → M with coefficients in W ∞,∞ (g) and satisfying the strong Legendre condition with respect to g 0 . For dimension two domains M , the Poincaré inequality for (M, A, g) is equivalent to the Poincaré inequality for (M, A, g 0 ) (same proof as the conformal invariance of the Laplacian in two dimensions). The Poincaré inequality of [START_REF] Ammann | Well-posedness of the Laplacian on manifolds with boundary and bounded geometry[END_REF] then gives: Theorem 9. Assume that (M, ∂ 0 M, g) has finite width and m := dim(M ) = 2. Let P = P a satisfy the strong Legendre condition with all ∇ j a bounded. Then there exists η a,f > 0 such that, for |s| < η a,f and ∈ Z + , we have an isomorphism

P a : ρf s K +1 (ρ) (M, g 0 ; E) ∩ {u| ∂0M = 0, ∂ a ν u| ∂1M = 0} → ρ -1 f s K -1 (ρ) (M, g 0 ; E).
In particular, we can take P a = ∆ g0 . Canonical cuspidal and wedge domains. We continue with some concrete examples.

The simplest examples in higher dimensions are those of "model cuspidal and wedge domains." We follow the presentation in [START_REF] Amann | Function spaces on singular manifolds[END_REF]. Let 1 < α < ∞ and B ⊂ R m-1 a compact submanifold, possibly with boundary, and

K m α (B) := {(r, r α y) ∈ R m | 0 < r < 1, y ∈ B}, (3)
which will be called a model canonical cusp of order α. For α = 1, we take B a subset of the unit sphere. A domain with canonical cuspidal singularities is a bounded domain Ω ⊂ M in a Riemannian manifold ( M , g 0 ) such that, around each singular point P of the boundary, it is locally diffeomorphic to K m α P (B P ) via a diffeomorphism defined in a neighborhood of the ambient manifold. Let V be the set of singular points of the boundary, then V is finite and we let M := Ω V. If α P = 1 for all P ∈ V, we obtain a domain with conical points. If we replace K m α (B) with

K m-k α (B) × [0, 1] k , k ≥ 0,
we obtain domains with canonical wedge singularities, in which case, of course, the set V of singular points of ∂Ω will no longer be finite.

Let us fix λ P ≥ 1 for each singular point P ∈ V. The weight functions ρ and f are then chosen, around each P ∈ V as in Example 4 for λ = λ P . Let g := ρ -2 g 0 , as before. If λ P ≥ α P for all P , then (M, g) has bounded geometry (proved in [START_REF] Amann | Uniformly regular and singular Riemannian manifolds[END_REF] if λ P = α P for all P ∈ V) and consequently, we have regularity in the weighted spaces for the mixed Dirichlet-Neumann problem for operators satisfying the strong Legendre condition. If λ P ≤ α P and ∂ 0 M intersects each V P , then (M, ∂ 0 M, g) satisfies the Hardy-Poincaré inequality. This follows from the usual Poincaré inequality on each {r} × r α B by also rescaling in r. We work in generalized spherical coordinates (r, y) ∈ (0, ∞) × S m-1 , so dx = r m-1 drdy. This gives r -2α

r α B |u(r, y)| 2 dy ≤ C r α B |∇ y u(r, y)| 2 dy the Hardy-Poincaré inequality K m α (B) r -2α |u(r, y)| 2 dx ≤ C K m α (B) |∇u(r, y)| 2 dx for u = 0 on ∂ 1 B.
Let us fix λ P = α P . By considering the Amann triple (M, g := ρ -2 g, ρ), we obtain that our domain with canonical wedge singularities satisfies the conclusion (isomorphism) of Theorem 9. For canonical cuspidal domains and constant coefficient operators, this theorem was first proved in [START_REF] Kozlov | Elliptic boundary value problems in domains with point singularities[END_REF]. See also [START_REF] Cardone | Asymptotics of solutions of the Neumann problem in a domain with closely posed components of the boundary[END_REF][START_REF] Costabel | Analytic regularity for linear elliptic systems in polygons and polyhedra[END_REF][START_REF] Dauge | Strongly elliptic problems near cuspidal points and edges[END_REF][START_REF] Kamotski | On the third boundary value problem in domains in cusps[END_REF][START_REF] Munnier | Asymptotic analysis of a Neumann problem in a domain with cusp. Application to the collision problem of rigid bodies in a perfect fluid[END_REF].

Other examples. Certain simple examples are not "canonical." Let g := ρ -2 g E , where g E is the standard (flat) Euclidean metric. We then have that (M, g), M := Ω {O, A i , i = 1, 4}, is a manifold with boundary and bounded geometry. Assume that ∂ 0 M touches each singular point (where O is considered as a double point as above). We can then prove that (M, ∂ 0 M, g E ) satisfies the Hardy-Poincaré inequality as in the previous example, and hence Theorem 9 applies. See also [START_REF] Kamotski | On the third boundary value problem in domains in cusps[END_REF][START_REF] Munnier | Asymptotic analysis of a Neumann problem in a domain with cusp. Application to the collision problem of rigid bodies in a perfect fluid[END_REF].

Example 10. Let Ω = {(x, y) ∈ R 2 | x, y > 0, (y -1) 2 + x 2 > 1} ∩ [-2, 2] × [0,
Example 11. Let f 0 , f 1 : R → (0, 2π) satisfy f (k) i ∞ < ∞, k ≥ 0, and f 1 -f 0 ≥ > 0. Let Ω f0,f1 := {(r cos θ, r sin θ) | f 0 (log r) < θ < f 1 (log r)} and Ω be a bounded domain, smooth away from a finite number of points, at which it coincides, up to a diffeomorphism, with a neighborhood of 0 in a set of the form Ω f0,f1 . Then Ω is a domain with oscillating conical singularities, similar to the ones studied by [START_REF] Rabinovich | C * -algebras of singular integral operators in domains with oscillating conical singularities[END_REF]. Theorem 9 holds for this domain with ρ = f = r = (x 2 + y 2 ) 1/2 . In general, a domain with oscillating conical points is not a domain with conical points. Finally, in [START_REF] Băcuţă | Interface and mixed boundary value problems on n-dimensional polyhedral domains[END_REF], it was proved that the assumptions and the conclusions of Theorem 8 are fulfilled by any polyhedral domain Ω ⊂ R m (defined as a suitable stratified space) with the Euclidean metric g 0 for a suitable g-admissible weight ρ ∼ the distance to the singular points of the boundary (g = ρ -2 g 0 ).

  is a g-admissible weight. This example is adapted to a domain with conical points (for instance, a polygonal domain) with set of vertices V by taking f (x) = ρ(x) := P ∈V |x -P | and λ = 1. The extra weight f becomes then unnecessary (for λ = 1) and the weighted Sobolev spaces K (ρ) (M ) := {u | ρ |α| ∂ α u ∈ L 2 (M ), (∀) |α| ≤ } are the spaces introduced by Kondratiev [17].

  2] (see the picture). The corners {A i } 4 i=1 of Ω are conical points and are treated as above. Close to O we have two cuspidal open sets (U and its mirror image), similar to the ones treated in the previous subsection, but not canonical. We have λ = 2 for these open sets. (We double O, in a certain sense.) Let r be the distance to O. Close to O, we then choose ρ ∼ r 2 , and f ∼ e -r -1 . More precisely, ρ(x) = r 2 4 i=1 |x -A i | and f = e -r -1 4 i=1 |x -A i |. Note that we could have also chosen ρ ∼ x 2 near O.