Marine Jaouen
email: marine.jaouen@culture.gouv.fr

Olivia Hulot
email: olivia.hulot@culture.gouv.fr

Eric Rieth
email: eric.rieth.cnrs@gmail.com

Digitising Wrecks on the Foreshore: The Case of a Seventeenth-Century Wreck in Brittany, France

Keywords: Wreck, coaster, 17th century, foreshore, digitisation

Studying archaeological sites on the foreshore gives rise to its own set of specific problems. Historical wrecks buried in the sands at intertidal zones usually come to our notice through natural erosion or human activities, or both. Such threats affecting the sites can also accelerate their deterioration, even their obliteration. Intertidal wrecks are evidently a precious and irreplaceable testimony of maritime heritage, and yet they are very fragile. It is a real challenge for researchers to find adequate ways to study them.

One such study undertaken in France focused on the small tonnage ship known as Erquy-les-Hôpitaux on the Brittany coast. This was a small coaster, which carried mainly lime mortar and foundered in the seventeenth century.

In 2015 DRASSM, France's Underwater Archaeology Research Department, conducted a special study of this wreck in order to define and assess various approaches to collecting archaeological data. The first phase of the study looked at three-dimensional digitisation techniques. Generating a 3D model of the wreck enabled the archaeologists to continue their studies after the excavation, and in greater detail. It also proved to be a valuable asset for presenting the project to the general public.

Most foreshore sites of archaeological interest are subject to the action of the tide, a constraint which requires archaeologists to rethink their usual methods for working on land or underwater. They also have to define technological tools that allow them to document artefacts effectively and in a short period of time, in particular, the merits of 3D laser scanning compared to photogrammetry.

Introduction

The foreshore is a zone situated at the interface between the land and the sea. While its configuration may vary from one coast to the next, its value as an abundant repository of our maritime heritage is considerable. It contains sites from every age, such as submerged megalithic tombs, Palaeolithic settlements, fish weirs, cemeteries for the drowned whose religion was unknown, wrecks of ships and aircraft, and so on.

These remains face a double threat which consists, on the one hand, of coastal erosion, human-made constructions and safety works; and, on the other, of looting from, in particular, detectorists who at best damage sites if they do not destroy them forever (Poudret-Barré 2013: 127-36).

Background

Ships that have beached during storms or because of technical failure often find themselves entombed within the sands of these intertidal zones. Surveys are, of course, carried out at such locations but they are perfunctory and rare for want of suitable tools for the job, and the capacity to document artefacts in between tidal movements. DRASSM, a department of France's Ministry of Culture and Communication, is responsible for underwater archaeological research at the national level. It has, for many years, been looking at technological solutions and, above all, methods that can ensure surveys and studies of sites lying in intertidal zones. For this reason, in 2014 DRASSM launched in France, on the coast of Brittany, a long-term programme to try out a range of methods and solutions on a series of wreck sites.

The first of these wrecks is known as the Erquy les Hôpitaux. Discovered along the Breton coast in 2002, the wreck underwent an initial survey the same year but had to wait another twelve years before a further study of the site was conducted. Since 2014 the site has become a testing ground for excavation methods that could be suitable for intertidal zones, in particular the deployment of three-dimensional recording techniques. Excavations on the wreck were directed by Olivia Hulot and Marine Jaouen (DRASSM) in collaboration with Eric Rieth (CNRS).

Constraints

The wreck is 9 m long, 3 m wide and completely buried in the sand in an area of the beach which gets submerged at high tide. After stripping the surface mechanically, the remains were uncovered by hand.

The excavation strategies deployed were adapted to the constraints of the surrounding environment, in particular a nearby river-the Islet-which flooded the site on numerous occasions. To get around this, we installed drainage channels and a soakaway which, in conjunction with a pump, displaced the excess water from the centre of the hull to a point further down the beach. Between each tide window we protected the structures from the pounding waves and the encroaching sediment by covering them entirely with plastic tarpaulins which were held in place by sandbags.

In addition to the usual manual recording techniques, we tested two digital methods:

Laser Process

First, data was acquired in the field through the use of a FARO Focus 3D-X-330 scanner. It was operated by a technician from the Conservatoire Numérique du Patrimoine Archéologique de l'Ouest, which is based in Rennes, Brittany. The technician completed seven measuring sessions, each lasting ten minutes. In total, we devoted two hours to laser surveying. We processed the data using SCENE software which allowed us to generate a point cloud, while the rendering was done with Unity 3D.

Where this laser method encountered problems was in the mirror effect (Figure 1) created by the ground water which, during the measuring sessions, rose progressively and could not be discharged. This constraint interfered with data acquisition and, unsurprisingly, had significant repercussions for the accuracy and detail of the resulting point cloud. In view of the length of time required for the acquisition, this method had to be rejected.

Photogrammetry Process

The second digitisation method under consideration involved taking a series of 3D photographs-done by a professional photographer-of the remains, in compliance with certain criteria, the principal ones being:

• Photos must overlap with at least four other photos by sixty to seventy percent of their surface and be spaced approximately fifteen degrees apart; • Photos must contain metre sticks to allow for determining scale and calculating errors; and • Photos must be free of shadows.

One of the 3D images comprised 885 separate photographs, which, once processed with the software Agisoft PhotoScan, produced a dense point cloud for the wreck. The mesh resulting from the dense point cloud gives a 3D model or, in the case of the Erquy wreck, a 2.5D model because the underside of the hull was not digitised. Texture can be added to the point cloud, it being calculated from the photomosaic of the model. Errors are calculated from standardised reference points placed on the ground, and this enables the model to be adjusted to scale.

The 3D model is georeferenced with help from topographical points that are determined during the excavation and used to generate a digital model of the terrain [START_REF] Hulot | Study of a wreck in foreshore context[END_REF]. A vertical, azimuthal projection of the 3D model provides an orthophotograph of the remains. The planimetrics of the remains were obtained by vectorising the orthophotograph of the site (Figure 2).

Contributions from GIS

Importing this post-excavation process into a GIS project, in this case QGIS, saves a considerable amount of time when working in the field during low-water windows.

The planimetric view of the site, obtained from the orthophotograph and imported into a GIS project, allows the archaeologist to make a spatial analysis of the data. Thus, all the components of the hull, including treenails and metal nails, can be vectorised piece by piece and joined to an attribute table. This table allows the user to generate planimetrics that highlight the conserved items according to their type (keel, floor, plank, and so on) or depending on their material (Figure 3).

Importing the digital model of the terrain into a GIS project also allows the user to generate crosssections and longitudinal sections (Figure 4). When we compared the measurements of the sections obtained from the post-excavation work with those obtained in the field, the results were very satisfactory. That said, interpreting the results of the manually obtained sections is clearly easier because the archaeologist can analyse the features in greater detail.

Archaeological data

When we analysed the design of the wrecked ship we discovered some very unusual architectural features, which make it a unique archaeological find. For example, all but one of the floor timbers were not fastened to the keel. In addition, except for two, none of the futtock timbers was fastened to a floor timber. These unusual design features of the Erquy wreck cannot be described in terms of 'primitive frames' or even 'primitive planking'. For the moment, without another archaeological example of this type of vessel, it is difficult for us to determine whether we have stumbled on an isolated case or are dealing with architectural features that are characteristic of some regional shipbuilding tradition.

The dimensions of the vessel have been estimated as follows: she was 9m long, had a 3.38m beam, and an approximate tonnage of nine. This suggests she was a small coaster involved in regional trade. Sailing in waters bordered with numerous sandy shores, the Erquy vessel was built for beaching both empty and laden. Wheat, firewood, slate tiles and lime were typical products of the Breton hinterland and undoubtedly constituted the main cargoes for the coaster as she worked to and from the great shipping hub in the region that was Saint-Malo (Le Bouëdec 2008: 9-37).

Wood Analysis

For the wooden components of the wreck Catherine Lavier of Paris 6 University identified the species used and dated them by dendrochronology. Elm was predominant in the ship's frames and planking, although some of the frames were made of oak. Only the keel was made of beech. The construction timbers were felled in the autumn of 1628 and were probably put to use the following year. Environmental periods were determined for the felling of the timber used in the frames and the barrels.

Restitution of the hull shape and sails

By analysing and interpreting the data from the 3D reconstruction, we were able to reconstruct the vessel's lines. We chose Rhinoceros 3D software, which has already been used by many international teams, to aid us in our subsequent study of the hull (Figure 5). Three-dimentional drawings of the architectural elements are essential if the archaeologist is to analyse distortions piece by piece. In the case of the Erquy wreck, we made twelve cross-sections at characteristic points. These uncorrected sections were then placed on the keel in order to reproduce the shape of the hull as it was at the time of excavation. This allowed us to visualise the lines of the ship's bottom.

By modelling the stem we were able to appreciate the rake of the bow. The initial rake of the sternpost was reconstructed thanks to the skeg of the keel and the curve of the existing part of the sternpost. The waterlines were then established. These lines, which define the 3D shape of the hull, allowed us to reconstruct the lines plan for the ship after much refining.

Finalising the wreck's lines plan helped us formulate hypotheses concerning the ship's mast and sail plan. Despite our efforts being hindered by the absence of any keelson or mast step, the fact that one of the floor timbers had a heavier section than the others allowed us to be confident of the mast's position.

We were also able to compare this data with a few rare written sources, as Pâris ( 1882) Forfaix (1788), Morineau (1763) or more precisely an anonymous album (1679) of drawings, which shows various inshore and offshore vessels that worked the Atlantic coasts. This source is both contemporary with the wreck and geographically coherent with our study. It depicts only one type of rigging: square sail. From this premise, two hypotheses are suggested: the vessel could have been single-masted or two-masted, the latter comprising a foremast and a mainmast (Figure 6).

Conclusion

Somewhat more accurate, the laser system could provide a significant amount of measurements and a higher density of points for the point cloud than the photogrammetric technique. However, these advantages are to be weighed against the practical issues that arise from the laser technique. For this, the following should to be taken into account:

• The cost of the equipment;

• The deployment time required in the field-more than twice the time is needed for an equivalent coverage via the photogrammetric way; and • The relative complexity of the whole process from the acquisition to the restitution.

It is most likely this last point that led us to prioritise the photogrammetric method, since it can be used singlehandedly by any archaeologist and photographer skilled enough to follow simple rules of acquisition, and does not require the enlisting of any technical specialist specifically trained to be deployed on fieldwork during the acquisition phases. The delivery products can also be managed by archaeologists provided they learn to use software such as Agisoft PhotoScan, which are more commonly being used in archaeology.

Three-dimensional digitisation from photographs is a useful tool for both documenting and analysing-postexcavation-wrecks found on the foreshore. It also helps communicate the archaeologist's work to the wider world. In addition, we were able to present one of the models on the platform Immersia at Rennes 1 University in Brittany. Measuring 10 x 3 x 3m, Immersia is one of the largest virtual reality rooms in the world.

Since the trials on the coaster of Erquy-les-Hôpitaux, other studies have been carried out on wrecks situated on the foreshore of France's coasts and these confirm, if confirmation were necessary, the essential contribution 3D digitisation techniques make to the study of archaeological remains in intertidal zones.

Figure 2 .

 2 Figure 2. Orthophotography of the wreck (Service 3D).

Figure 1 .

 1 Figure 1. Mirror effect with laser process (Y. Bernard/Université Rennes 1).

Figure 3 .

 3 Figure 3. Vectorisation of the hull (MCC/DRASSM).

Figure 4 .

 4 Figure 4. Comparison of manual and automatic cross-sections (MCC/DRASSM).

Figure 5 .

 5 Figure 5. Reconstruction of the vessel lines (S. Bertoliatti).

Acknowledgements

We would like to thank the town of Erquy and town hall officials, all the field team, Denis Degez the DRASSM geomatician and of course Yves Meslin who discovered the site and closely followed every step of this work.


Alexandre Cherrueau

Marie Delavergne

Adrien Lebre

Ronan-Alexandre Cherrueau

Geo-Distribute Cloud Applications at the Edge

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

The deployment of multiple micro and nano Data Centers (DCs) at the edge of the network is taking off. Unfortunately, our community is lacking tools to make applications benefit from the geo-distribution while dealing with high latency and frequent disconnections inherent to wide-area networks (WAN) [START_REF] Satyanarayanan | The emergence of edge computing[END_REF].

The current accepted research direction for developing geo-distributed applications consists in using globally distributed data stores [START_REF] Abadi | Consistency tradeoffs in modern distributed database system design: CAP is only part of the story[END_REF]. Roughly, distributed data stores emulate a shared memory space among DCs to make the development of geo-distributed application easier [START_REF] Shapiro | Just-right consistency: reconciling availability and safety[END_REF]. This approach however implies to entangle the geo-distribution concern in the business logic of the application. This contradicts the software engineering principle of externalizing concerns. A principle widely adopted in the cloud computing where a strict separation between development and operational (abbreviated as DevOps) teams exists [START_REF] Herbst | Elasticity in cloud computing: What it is, and what it is not[END_REF][START_REF] Li | Service mesh: Challenges, state of the art, and future research opportunities[END_REF]: Programmers focus on the development and support of the business logic of the application (i.e., the services), whereas DevOps are in charge of the execution of the application on the infrastructure (e.g., deployment, monitoring, scaling).

The lack of separation between the business logic and the geo-distribution concern is not the only problem when using distributed data stores. Data stores distribute resources across DCs in a pervasive manner. In most cases, resources are distributed across the infrastructure identically. However, all resources do not have the same scope in a geo-distributed context. Some are useful in one DC, whereas others need to be shared across multiple locations to control the latency, scalability and availability [START_REF] Cherrueau | Edge computing resource management system: a critical building block! initiating the debate via openstack[END_REF][START_REF] Corbett | Spanner: Google's globally-distributed database[END_REF]. And scopes may change as time passes. It is therefore tedious for programmers to envision all scenarios in advance, and a fine-grained control per resource is mandatory.

Based on these two observations, we propose to deal with the geo-distribution as an independent concern using the service mesh concept widely adopted in the cloud. A service mesh is a layer over microservices that intercepts requests in order to decouple concerns such as monitoring or auto-scaling [START_REF] Li | Service mesh: Challenges, state of the art, and future research opportunities[END_REF]. The code of the Netflix Zuul1 load balancer for example is independent of the domain and generic to any modular application by only considering their requests. In this paper, we explore the same idea for the geo-distribution concern. By default, one instance of the cloud application is deployed on each DC, and a dedicated service mesh forwards requests between the different DCs. The forwarding operation is programmed using a domain specific language (DSL) that enables two kinds of collaboration. First, the access of resources available at another DC. Second, the replication of resources on a set of DCs. The DSL reifies the resource location. This makes it clear how far a resource is and where its replicas are. It gives a glimpse of requests probable latencies and a control on resources availability.

The contributions of this paper are as follows:

-We state what it means to geo-distribute an application and illustrate why using a distributed data store is problematic, discussing a real use-case: OpenStack 2 for the edge. OpenStack is the defacto application for managing cloud infrastructures. (Section 2) -We present our DSL to program the forwarding of requests and our service mesh that interprets expressions of our language to implement the geodistribution of a cloud application. Our DSL lets clients specify for each request, in which DC a resource should be manipulated. Our service mesh relies on guarantees provided by modularity to be independent of the domain of the application and generic to any microservices application. (Section 3) -We present a proof of concept of our service mesh to geo-distribute Open-Stack. 3 With its 13 million of lines of code and more than 150 services, OpenStack is a complex cloud application, making it the perfect candidate to validate the independent geo-distribution mechanism that we advocate for. Thanks to our proposal, DevOps can make multiple independent instances of OpenStack collaborative to use a geo-distributed infrastructure. (Section 4)

We finally conclude and discuss about limitations and future work to push the collaboration between application instances further (Section 5).

Geo-distributing applications

In this section, we state what it means to geo-distribute a cloud application and illustrate the invasive aspect of using a distributed data store when geodistributing the OpenStack application.

Geo-distribution principles

We consider an edge infrastructure composed of several geo-distributed micro DCs, up to thousands. Each DC is in charge of delivering cloud capabilities to an edge location (i.e., an airport, a large city, a region . . . ) and is composed of up to one hundred servers, nearly two racks. The expected latency between DCs can range from 10 to 300ms round trip time according to the radius of the edge infrastructure (metropolitan, national . . . ) with throughput constraints (i.e., LAN vs WAN links). Finally, disconnections between DCs are the norm rather than the exception, which leads to network split-brain situations [START_REF] Markopoulou | Characterization of failures in an operational ip backbone network[END_REF]. We underline we do not consider network constraints within a DC (i.e., an edge location) since the edge objective is to bring resources as close as possible to its end use.

Cloud applications of the edge have to reckon with these edge specifics in addition to the geo-distribution of the resources themselves [START_REF] Tato | Split and migrate: Resource-driven placement and discovery of microservices at the edge[END_REF]. We hence suggest that geo-distributing an application implies adhering to the following principles:

Local-first. A geo-distributed cloud application should minimize communications between DCs and be able to deal with network partitioning issues by continuing to serve local requests at least (i.e., requests delivered by users in the vicinity of the isolated DC). Collaborative-then. A geo-distributed cloud application should be able to share resources across DCs on demand and replicate them to minimize user perceived latency and increase robustness when needed.

Unfortunately, implementing these principles produces a code that is hard to reason about and thus rarely addressed [START_REF] Alvaro | Consistency analysis in bloom: a CALM and collected approach[END_REF]. This is for instance the case of OpenStack that we discuss in the following section.

The issue of geo-distributing with a distributed data store

OpenStack is a resource management application to operate one DC. It is responsible for booting Virtual Machines (VMs), assigning VMs in networks, storing operating system images, administrating users, or any operation related to the management of a DC.

A complex but modular application. Similarly to other cloud applications such as Netflix or Uber, OpenStack follows a modular design with many services. The compute service for example manages the compute resources of a DC to boot VMs. The image service controls operating system BLOBs like Debian. The compute service handles the request and contacts the image service to get the Debian BLOB in return (Step 2). Finally, the compute does a bunch of internal calls -schedules VM, setups the network, mounts the drive with the BLOB -before booting the new VM on one of its compute nodes (Step 3). 4Geo-distributing Openstack. Following the local-first and collaborative-then principles implies two important considerations for OpenStack. First, each DC should behave like a usual cloud infrastructure where DevOps can make requests and use resources belonging to one site without any external communication to other sites. This minimizes the latency and satisfies the robustness criteria for local requests. Second, DevOps should be able to manipulate resources between DCs if needed [START_REF] Cherrueau | Edge computing resource management system: a critical building block! initiating the debate via openstack[END_REF]. For instance, Fig. 11 illustrates an hypothetical sharing with the "boot of a VM at one DC using the Debian image available in a second one". Fig. 2: Boot of a VM using a remote BLOB To provide this resource sharing between DC 1 and DC 2, the image service has to implement an additional dedicated means (Step 2b). Moreover, it should be configurable as it might be relevant to replicate the resource if the sharing is supposed to be done multiple times over a WAN link. Implementing such a mechanism is a tedious task for programmers of the application, who prefer to rely on a distributed data store [START_REF] Corbett | Spanner: Google's globally-distributed database[END_REF].

Distributed data store tangles the geo-distribution concern. The OpenStack image service team currently studies several solutions to implement the "booting a VM at DC 1 that benefits from images in DC 2" scenario. All are based on a distributed data store that provides resource sharing between multiple image services: a pull mode where DC 1 instance gets BLOBs from DC 2 using a message passing middleware, a system that replicates BLOBs around instances using a shared database, etc. 5 The bottom line is that they all require to tangle the geo-distribution concern with the logic of the application. This can be illustrated by the code that retrieves a BLOB when a request is issued on the image service (code at Step 2 from Fig. 11). Code 1.1 gives a coarse-grained description of that code. It first queries the data store to find the path of the BLOB (l. [START_REF] Cherrueau | Edge computing resource management system: a critical building block! initiating the debate via openstack[END_REF][START_REF] Corbett | Spanner: Google's globally-distributed database[END_REF]. It then retrieves that BLOB in the image_collection and returns it to the caller using the get method (l. 6-8). Particularly, this method resolves the protocol of the path and calls the proper library to get the image. Most of the time, that path refers to a BLOB on the local disk (e.g., file:///path/debian.qcow). In such a case, the method image_collection.get relies on the local open python function to get the BLOB. Fig. 3: Booting a VM at DC 1 with a BLOB in DC 2 using a distributed data store (does not work)

The code executes properly as long as only one OpenStack is involved. But things go wrong when multiple are unified through a data store. If Code 1.1 remains unchanged, then the sole difference in the workflow of "booting a VM at DC 1 using an image in DC 2" is the distributed data store that federates all image paths (including those in DC 2-see Fig. 3). Unfortunately, because DC 2 hosts the Debian image, the file path of that image returned at Step 2b is local to DC 2 and does not exist on the disk of DC 1. An error results in the image_collection.get (2c).

The execution of the method image_collection.get takes place in a specific environment called its execution context. This context contains explicit data such as the method parameters. In our case, the image path found from the data store. It also contains implicit assumptions made by the programmer: "A path with the file: prototype must refer to an image stored on the local disk". Alas, such kind of assumptions are wrong with a distributed data store. They have to be fixed. For this scenario of "booting a VM at DC 1 using an image in DC 2", it means changing the image_collection.get method in order to allow the access of the disk of DC 2 from DC 1. More generally, a distributed data store constrains programmers to take the distribution into account (and the struggle to achieved it) in the application. And besides this entanglement, a distributed data store also strongly limits the collaborative-then principle. In the code above, there is no way to specify whether a particular BLOB should be replicated or not.

Our position is that the geo-distribution must be handled outside of the logic and in a fine-grained manner due to its complexity.

Geo-distribute applications with a service mesh

In this section, we first introduce the foundations and notations for our proposal. We then build upon this to present our service mesh that decouples the geodistribution concern from the business logic of an application.

Microservices and service mesh basics

An application that follows a microservices architecture combines several services [START_REF] Jamshidi | Microservices: The journey so far and challenges ahead[END_REF]. Each service defines endpoints (operations) for managing one or various resources [START_REF] Fielding | Architectural Styles and the Design of Network-based Software Architectures[END_REF]. The combination of several services endpoints forms a series of calls called a workflow. Deploying all services of an application constitutes one application instance (often shortened instance in the rest). The services running in an application instance are called service instances. Each application instance achieves the application intent by exposing all of its workflows which enables the manipulation of resource values.

Fig. 4 illustrates such an architecture. Fig. 4a depicts an application App made of two services s, t that expose endpoints e, f , g, h and one example of a workflow s.e → t.h. App could be for example the OpenStack application. In this context, service s is the compute service that manages VMs. Its endpoint e creates VMs and f lists them. Service t is the image service that controls operating system BLOBs. Its endpoint g stores an image and h downloads one. The composition s.e → t.h models the boot workflow (as seen in Fig. 1). Fig. 4b shows two application instances of App and their corresponding service instances: s 1 and t 1 for App 1 ; s 2 and t 2 for App 2 . A client (•) triggers the execution of the workflow s.e → t.h on App 2 . It addresses a request to the endpoint e of s 2 which handles it and, in turn, contacts the endpoint h of t 2 .

Microservices architectures are the keystone of DevOps practices. They promote a modular decomposition of services for their independent instantiation and maintenance. A service mesh takes benefit of this situation to implement communication related features such as authentication, message routing or load balancing outside of the application. In its most basic form, a service mesh consists in a set of reverse proxies around service instances that encapsulate a specific code to control communications between services [START_REF] Li | Service mesh: Challenges, state of the art, and future research opportunities[END_REF]. This encapsulation Fig. 5 illustrates the service mesh approach with the monitoring of requests to have insight about the application. It shows reverse proxies mon s and mon t that collect metrics on requests toward service instances s i and t i during the execution of the workflow s.e → t.h on App i . The encapsulated code in mon s and mon t collects for example requests latency and success/error rates. It may send metrics to a time series database as InfluxDB and could be changed by anything else without touching the App code.

A tour of scope-lang and the service mesh for geo-distributing

Interestingly, running one instance of a microservices application automatically honors the local-first principle. In Fig. 4b, the two instances are independent. They can be deployed on two different DCs. This obviously cancels communications between DCs as well as the impact of DCs' disconnections. However for the global system, it results in plenty of concurrent values of the same resource distributed but isolated among all instances (e.g., s1 and s2 manage the same kind of resources but their values differ as time passes). Manipulating any concurrent value of any instance requires now on to code the collaboration in the service mesh and let clients to specify it at will.

In that regard, we developed a domain specific language called scope-lang. A scope-lang expression (referred to as the scope or σ in Fig. 6a) contains location information that defines, for each service involved in a workflow, in which instance the execution takes place. The scope "s : App 1 , t : App 2 " intuitively tells

• σ R[[σ[s]]] σ R[[σ[t]]]
(b) Scope σ interpreted by the geo-distribution service mesh geo during the execution of the s.e σ -→ t.h workflow in Appi. Reverse proxies perform requests forwarding based on the scope and the R function.

Fig. 6: A service mesh to geo-distribute a cloud application to use the service s from App 1 and t from App 2 . The scope "t : App 1 &App 2 " specifies to use the service t from App 1 and App 2 .

Clients set the scope of a request to specify the collaboration between instances they want for a specific execution. The scope is then interpreted by our service mesh during the execution of the workflow to fulfill that collaboration. The main operation it performs is request forwarding. Broadly speaking, reverse proxies in front of service instances (geo s and geo t in Fig. 6b) intercept the request and interpret its scope to forward the request somewhere. "Where" exactly depends on locations in the scope. However, the interpretation of the scope always occurs in the following stages:

1. A request is addressed to the endpoint of a service of one application instance. The request piggybacks a scope, typically as an HTTP header in a RESTful application. For example in Fig. 6b:

• s:Appi ,t:Appi ---------→ s.e. 2.
The reverse proxy in front of the service instance intercepts the request and reads the scope. In Fig. 6b: geo s intercepts the request and reads σ which is equal to s : App i , t : App i . 3. The reverse proxy extracts the location assigned to its service from the scope.

In Fig. 6b: geo s extracts the location assigned to s from σ. This operation, notated σ[s], returns App i . 4. The reverse proxy uses a specific function R (see Fig. 6a) to resolve the service instance at the assigned location. R uses an internal registry. Building the registry is a common pattern in service mesh using a service discovery [START_REF] Li | Service mesh: Challenges, state of the art, and future research opportunities[END_REF] and therefore is not presented here. In Fig. 6b: R[[s : σ[s]]] reduces to R[[s : App i ]] and is resolved to service instance s i . 5. The reverse proxy forwards the request to the endpoint of the resolved service instance. In Fig. 6b: geo s forwards the request to s i .e.

In this example of executing the workflow s.e σ -→ t.h, the endpoint s i .e has in turn to contact the endpoint h of service t. The reverse proxy geo s propagates the scope on the outgoing request towards the service t. The request then goes through stages 2 to 5 on behalf of the reverse proxy geo t . It results in a forwarding to the endpoint R[[t : σ

[t]]].h that is resolved to t i .h.
Here, the scope only refers to one location (i.e., App i ). Thus the execution of the workflow remains local to that location. The next sections detail the use of forwarding in order to perform collaboration between instances.

Forwarding for resource sharing

A modular decomposition of the code is popular for programmers of microservices architecture. It divides the functionality of the application into independent and interchangeable services [START_REF] Liskov | A design methodology for reliable software systems[END_REF]. This brings well-known benefits including ease of reasoning. More importantly, modularity also gives the ability to change a service with a certain API by any service exposing the same API and logic [START_REF] Parnas | On the criteria to be used in decomposing systems into modules[END_REF]. A load balancer, such a Netflix Zuul mentioned in the introduction, makes a good use of this property to distribute the load between multiple instances of the same modular service [START_REF] Fielding | Architectural Styles and the Design of Network-based Software Architectures[END_REF]. Fig. 7 shows this process with the reverse proxy for load balancing lb t of the service t. lb t intercepts and balances incoming requests within two service instances t i and t i during the execution of the workflow s.e → t.h in App i . From one execution to another, the endpoint s i .e gets result from t i .h or t i .h in a safe and transparent manner thanks to modularity.

We generalize this mechanism to share resources. In contrast to the load balancer that changes the composition between multiple instances of the same service inside a single application instance. Here, we change the composition between multiple instances of the same service across application instances. As a consequence, the different service instances share their resources during the execution of a workflow.

Fig. 8 depicts this cross dynamic composition mechanism during the execution of the workflow s.e s:App1 ,t:App2 ---------→ t.h. The service instance s 1 of App 1 is dynamically composed thanks to the forwarding operation of the service mesh with the service instance t 2 of App 2 . This forwarding is safe relying on the guaranty provided by modularity. (If t is modular, then we can swap t 1 by t 2 since they obviously have the same API and logic.) As a result, the endpoint s 1 .e benefits from resource values of t 2 .h instead of its usual t 1 .h.

• σ R[[s : App1]] σ R[[t : App2]]
Fig. 8: Resource sharing by forwarding across instances

Forwarding for resource replication

Replication is the ability to create and maintain identical resources on different DCs: an operation on one replica should be propagated to the other ones according to a certain consistency policy. In our context, it is used to deal with latency and availability. In terms of implementation, microservices often follow a RESTful HTTP API and so generate an identifier for each resource. This identifier is later used to retrieve, update or delete resources. Since each application instance is independent, our service mesh requires a meta-identifier to manipulate replicas across the different DCs as a unique resource.

For example, the image service t exposes an endpoint g to create an image. When using a scope for replication, such as t : App 1 &App 2 , the service mesh generates a meta-identifier and maps it {metaId : [App 1 : localID t1 , App 2 : localID t2 ]}. In Fig. 9, if t 1 creates a replica with the identifier 42 and t 2 6, and our meta-identifier was generated as 72, the mapping is: {72 : [App 1 : 42, App 2 : 6]}. Mappings are stored in an independent database alongside each application instance.

The replication process is as follows:

1. A request for replication is addressed to the endpoint of a service of one application instance. For example in Fig. 9:

• t:App1&App2 --------→ t.g. 2.
Similarly to the sharing, the R function is used to resolve the endpoints that will store replicas. R[[s

: Loc&Loc ]] = R[[s : Loc]] and R[[s : Loc ]]. In Fig. 9: R[[t : App 1 &App 2 ]] is equivalent to R[[t : App 1 ]] and R[[t : App 2 ]].
Consequently, t 1 and t 2 . 3. The meta-identifier is generated along with the mapping and added in the database. In Fig. 9: { 72 : [App 1 : none, App 2 : none]}. 4. Each request is forwarded to the corresponding endpoints on involved DCs and a copy of the mapping is stored in those DCs' database simultaneously. In Fig. 9: geo t forwards the request to t 1 .g and t 2 .g and stores the mapping {72 : [App 1 : none, App 2 : none]} in App 1 and App 2 databases. 5. Each contacted service instance executes the request and returns the results (including the local identifier) to the service mesh. In Fig. 9: t 1 and t 2 returns respectively the local identifier 42 and 6. 6. The service mesh completes the mapping and populates the involved DCs' databases. In Fig. 9: the mapping now is {72 : [App 1 : 6, App 2 : 42]} and added to databases of App 1 and App 2 . 7. By default, the meta identifier is returned as the final response. If a response other than an identifier is expected, the first received response is transferred (since others are replicas with similar values).

This process ensures that only interactions with the involved DCs occur, avoiding wasteful communications. Each operation that would later modify or

App1 App2 geot g h t1 g h geot g h t2 g h σ = t : App1 &App2 • σ R[[t : App1]] R[[t : App2]]
Fig. 9: Replication by forwarding on multiple instances delete one of the replicas will be applied to every others using the mapping available on each site. To prevent any direct manipulation, local identifiers of replicas are hidden. This replication control perfectly suits our collaborative-then principle. It allows a client to choose "when" and "where" to replicate. Regarding the "how", our current process for forwarding replica requests, maintaining mappings and ensuring that operations done on one replica are applied on others, is naive. Implementing advanced strategies is left as future work. However, we underline that it does not change the foundations of our proposal. Ultimately, choosing the strategy should be made possible at the scope-lang level (e.g., weak, eventual or strong consistency).

Towards a generalized control system

Scope-lang has been initially designed for resources sharing and replication. It is however a great place to implement additional operators that would give more control during the manipulation of resources. The code of the service mesh is independent of cloud applications and thus can be easily extended with new features in it. In this section, we show the process of adding a new operator to stress the generality of our approach.

The new otherwise operator ("Loc 1 ; Loc 2 " in Fig. 10a) intuitively tells to use the first location or fallback on the second one if there is a problem. This operator comes in handy when a client wants to deal with DCs' disconnections. Adding it to the service mesh implies to implement in the reverse proxy what to do when it interprets a scope with a (;). The implementation is straightforward: Make the reverse proxy forward the request to the first location and proceed if it succeeds, or forward the request to the second location otherwise.

Ultimately, we can built new operators upon existing ones. This is the case of the around function that considers all locations reachable in a certain amount of time, e.g., around(App 1 , 10ms). To achieve this, the function combines the available locations with the otherwise operator (;), as shown in Fig. 10b. Thus it does not require to change the code of the interpreter in the service mesh.

Validation on OpenStack

We demonstrate the feasibility of our approach with a prototype for OpenStack. 6 In this proof of concept, we set up an HAProxy 7 in front of OpenStack services. HAProxy is a reverse proxy that intercepts HTTP requests and forwards them to specific backends. In particular, HAProxy enables to dynamically choose a backend using a dedicated Lua code that reads information from HTTP headers. In our proof of concept, we have developed a specific Lua code that extracts the scope from HTTP headers and interprets it as described in Section 3.2 to 3.4.

In a normal OpenStack, the booting of a VM with a Debian image (as presented in Fig. 1) is done by issuing the following command:

$ openstack server create my -vm --image debian (Cmd. 1)

We have extended the OpenStack command-line client with an extra argument called --scope. This argument takes the scope and adds it as a specific header on the HTTP request. Thus it can latter be interpreted by our HAProxy. With it, a DevOps can execute the previous Cmd. 1 in a specific location by adding a --scope that, for instance, specifies to use the compute and image service instance of DC 1:

$ openstack server create my -vm --image debian \ --scope { compute : DC 1 , image : DC 1 } (Cmd. 2)
In the case where the DevOps is in DC 1, the request is entirely local and thus will be satisfied even during network disconnections between DCs. Actually, all locally scoped requests are always satisfied because each DC executes one particular instance of OpenStack. This ensures the local-first principle.

The next command mixes locations to do a resource sharing as explained in Section 3.3. It should be read as "boot a VM with a Debian image using the compute service instance of DC 1 and the image service instance of DC 2": $ openstack server create my -vm --image debian \ --scope { compute : DC 1 , image : DC 2 } (Cmd. 3) Fig. 11 shows the execution. Dotted red squares represent HAProxy instances.

Step 1b and 2b correspond to the forwarding of the request according to the Mixing locations in the scope makes it explicit to the DevOps that the request may be impacted by the latency. If DC 1 and DC 2 are far for from each other, then it should be clear to the DevOps that Cmd. 3 is going to last a certain amount of time. To mitigate this, the DevOps may choose to replicate the image:

$ openstack image create debian --file ./ debian . qcow2 \ --scope { image : DC 1 & DC 2 } (Cmd. 4)
This command creates an identical image on both DC 1 and DC 2 using the protocol seen in Section 3.4. It reduces the cost of fetching the image each time it is needed (as in Cmd. 3). Moreover, in case of partitioning, it is still possible to create a VM from this image on each site where the replicas are located. This fine-grained control ensures to replicate (and pay the cost of replication) only when and where it is needed to provide a collaborative-then application. Our proof of concept has been presented twice at the OpenStack Summit and is mentioned as an interesting approach to geo-distribute OpenStack in the second white paper published by the Edge Computing Working Group of the OpenStack foundation in 2020. 8

Conclusion

We propose a new approach to geo-distribute microservices applications without meddling in their code. By default, one instance of the application is deployed on each edge location and collaborations between the different instances are achieved through a generic service mesh. A DSL, called scope-lang, allows the configuration of the service mesh on demand and on a per request basis, enabling the manipulation of resources at different levels: locally to one instance (by default) and across distinct instances (sharing) and multiple (replication). Expliciting the location of services in each request makes the user aware of the number of sites that are involved (to control the scalability), the distance to these sites (to control the network latency/disconnections), and the number of replicas to maintain (to control the availability/network partitions).

We demonstrated the relevance of our approach with a proof of concept that enables to geo-distribute OpenStack, the defacto cloud manager. Thanks to it, DevOps can make multiple independent instances of OpenStack collaborative. This proof of concept is among the first concrete solutions to manage a geo-distributed edge infrastructure as an usual IaaS platform. Interestingly, by using our proposal over our proof of concept, DevOps can now envision to geo-distribute cloud applications.

This separation between the business logic and the geo-distribution concern is a major change with respect to the state of the art. It is important however to underline that our proposal is built on the modularity property of cloud applications. In other words, an application that does not respect this property cannot benefit from our service mesh. Regarding our future work, we have already identified additional collaboration mechanisms that can be relevant. For instance, we are investigating how a resource that has been created on one instance can be extended or even reassigned to another one.

We believe that a generic and non invasive approach for geo-distributing cloud applications, such as the one we propose, is an interesting direction that should be investigated by our community.

Fig. 1

 1 depicts this modular design in the context of a boot of a VM. The DevOp starts by addressing a boot request to the compute service of the DC (Step 1).

Fig. 1 :

 1 Fig. 1: Boot of a Debian VM in OpenStack

Code 1 . 1 :

 11 Retrieval of a BLOB in the image service @app . get ( '/ image /{ name }/ file ') def get_image ( name : String ) -> BLOB : # Lookup the image path in the data store : `path = proto :// path / debian . qcow path = ds . query (f '' ' SELECT path FROM images WHERE id IS "{ name }"; ' ' ') # Read path to get the image BLOB image_blob = i m a g e _ c o l l e c ti o n . get ( path ) return image_blob

  Application App made of two services s and t and four endpoints e, f, g, h. The s.e → t.h represents an example of a workflow. Two independent instances App1 and App2 of the App application. The • represents a client that executes the s.e → t.h workflow in App2.

Fig. 4 :

 4 Fig. 4: Microservices architecture of a cloud application

Fig. 5 :

 5 Fig. 5: Service mesh mon for the monitoring of requests

  Appi, Appj ::= application instance s, t ::= service si, tj::= service instance Loc ::= Appi single location | Loc&Loc multiple locations σ ::= s : Loc, σ scope | s : Loc R[[s : Appi]] = si R[[s : Loc&Loc ]] = R[[s : Loc]] and R[[s : Loc ]](a) scope-lang expressions σ and the function that resolves service instance from elements of the scope R. s : Appi , t : Appi

Fig. 7 :

 7 Fig. 7: Load balancing principle

  s : App1 , t : App2

  Loc ::= . . . see Fig. 6a | Loc; Loc otherwise location R[[s : Loc1; Loc2]] = R[[s : Loc1]] otherwise R[[s : Loc2]](a) The otherwise (;) operator. It requires to update the code of the interpreter in the service mesh. def around ( loc : Loc , radius : timedelta ) -> Loc : # Find all Locs in the `radius `of `loc # > locs = [ App1 , App2 , ... , Appn ] locs = _find_locs ( loc , radius ) # Combine all `locs `with `; # > App1 ; App2 ;...; Appn return foldr (; , locs , loc ) (b) The around operator build upon (;). It does not need to update the code of the interpreter in the service mesh.

Fig. 10 :

 10 Fig. 10: New operators for scope-lang

Fig. 11 :

 11 Fig. 11: Boot of a VM at DC 1 with a BLOB in DC 2 using scope-lang

https://netflixtechblog.com/open-sourcing-zuul-2-82ea476cb2b3. Accessed

2021-02-15. Zuul defines itself as an API gateway. In this paper, we do not make any difference with a service mesh. They both intercept and control requests on top of microservices. 2 https://www.openstack.org/software. Accessed 2021-02-15 3 https://github.com/BeyondTheClouds/openstackoid/tree/stable/rocky. Accessed 2021-02-15

For clarity, this paper simplifies the boot workflow. In a real OpenStack, the boot also requires at least the network and identity service. Many other service may also be involved. See https://www.openstack.org/software/project-navigator/ openstack-components. Accessed 2021-02-15

https://wiki.openstack.org/wiki/Image handling in edge environment. Accessed 2021-02-15

https://www.openstack.org/use-cases/edge-computing/ edge-computing-next-steps-in-architecture-design-and-testing Accessed 2021-02-18