Annabelle Gillet

Éric Leclercq
email: eric.leclercq@u-bourgogne.fr

Nadine Cullot
email: nadine.cullot@u-bourgogne.fr

Lambda+, the Renewal of the Lambda Architecture: Category Theory to the Rescue

Keywords: Architecture pattern, Category theory, Lambda Architecture

Designing software architectures for Big Data is a complex task that has to take into consideration multiple parameters, such as the expected functionalities, the properties that are untradeable, or the suitable technologies. Patterns are abstractions that guide the design of architectures to reach the requirements. One of the famous patterns is the Lambda Architecture, which proposes real-time computations with correctness and fault-tolerance guarantees. But the Lambda has also been highly criticized, mostly because of its complexity and because the realtime and correctness properties are each effective in a different layer but not in the overall architecture. Furthermore, its use cases are limited, whereas Big Data need an adaptive and flexible environment to fully reveal the value of data. Nevertheless, it proposes some interesting mechanisms. We present a renewal of the Lambda Architecture: the Lambda+ Architecture, supporting both exploratory and real-time analyzes on data. We propose to study the conservation of properties in composition of components in an architecture using the category theory. We relate a real implementation of our approach to architecture a social network observatory platform.

Introduction and motivations

All information systems have a common point: they need an architectural design before being developed and deployed. The architecture must guarantee some properties and guide the consistency of the overall structure of the information system. In this context, architectural styles and patterns are used to build a system having the expected characteristics for each of its part as well as for its entirety, and to state the requirements of the technologies and programming techniques needed to achieve the goal sought. Thus, global requirements such as scalability, performance, reliability must be clearly identified to select the style of architecture, the different components and the interactions among them [START_REF] Lampson | Hints for computer system design[END_REF], and then choose technologies with properties (such as ACID for databases or micro batch capabilities for stream processing) that fit all of the previous choices. The absence of coherence in a definition of an architecture can lead to the dreaded Big Ball of Mud [START_REF] Foote | Big ball of mud[END_REF], that reduces greatly the maintenance and evolutivity capabilities of the system. To help avoiding this situation, there are two major elements among architectural design artifacts: styles and patterns.

Styles are coarse grained specifications of the organisation of the architecture, that guide the interactions among components [START_REF] Abowd | Formalizing style to understand descriptions of software architecture[END_REF]. Each style brings naturally some architectural characteristics, while also imposing trade-offs on others. So, there is not a better style than the others, but solely situations where a style would be more suited to fulfill the expected characteristics. Some examples of architecture styles are the layered architecture [START_REF] Marmsoler | A model of layered architectures[END_REF], the microservices architecture [START_REF] Namiot | On micro-services architecture[END_REF] and the event-driven architecture [START_REF] Clark | Event driven architecture modelling and simulation[END_REF].

An architecture pattern is a specific abstraction of a fixed architecture style for a particular set of essential characteristics [START_REF] Harrison | Leveraging architecture patterns to satisfy quality attributes[END_REF]. It helps to identify within a style which combination of components will be more suited for a given context, but still provides enough freedom to adapt the implementation of the pattern to specificities of each situation. The level of detail can vary, as well as the restrictions of the application of the pattern. The Blackboard pattern [START_REF] Craig | Blackboard systems[END_REF], the Model-View-Controller [START_REF] Deacon | Model-view-controller (MVC) architecture[END_REF] and the Lambda Architecture [START_REF] Marz | How to beat the cap theorem[END_REF] are examples of architecture patterns. The Lambda Architecture is well-known in a Big Data context. It was introduced at the beginning of stream processing systems, and thus is oriented to compensate for the flaws of an emerging technology rather that taking advantage of the capabilities of such a technology.

Recent researches in software architecture try to formally define styles and patterns, to anticipate effects of the composition of components, and thus knowing beforehand the result of the evolution of a part of the architecture [START_REF] Broy | Can practitioners neglect theory and theoreticians neglect practice?[END_REF][START_REF] Johnson | Where's the theory for software engineering[END_REF]. When architectures evolve and grow, they can combine several smaller parts of architectures developed separately. When building a large scale, complex and distributed architecture, its parts can embed architecture styles on their own. These different cases can result in compositions of smaller architecture parts with their proper styles and patterns, so formalization should be able to express and control these compositions. Category theory [START_REF] Eilenberg | General theory of natural equivalences[END_REF] is a promising approach for formalization, due to its ease to represent compositions as it considers morphims and functors as first class citizens, and to its already existing proximity to the engineering software world, particularly with functional programming. Moreover, its graphical representation is a visual help to understand the formalization, and leads to a better comprehension of the system [START_REF] Spivak | Category theory for the sciences[END_REF].

In this article we propose the Lambda+ Architecture pattern, an update of the Lambda Architecture, and a formalization to study the conservation of properties in compositions of components using the category theory. The rest of the article is organized as follows: section 2 describes the original Lambda Architecture pattern, as well as its uses and its flaws ; Section 3 uses the category theory to prove the loss of properties in the Lambda Architecture ; Section 4 shows our improved Lambda+ Architecture pattern ; Section 5 describes a real applied example of our pattern and ; Section 6 concludes this article.

Lambda Architecture and related work

The properties of correctness, low latency and fault-tolerance have always been a major concern when designing architectures. In [START_REF] Lampson | Hints for computer system design[END_REF], Lampson sketches some suggestions that are still relevant today, and that can be found, among others, in the Lambda Architecture, introduced by Marz in 2011 [START_REF] Marz | How to beat the cap theorem[END_REF][START_REF] Marz | Big Data: Principles and best practices of scalable real-time data systems[END_REF]. The objective is simple: to compute predetermined queries with a very low latency and to ensure the correctness of the processing (figure 1). To do so, the Lambda is composed of three layers: the batch layer, the serving layer, and the speed layer. The batch layer takes care of storing raw data in the master dataset and of executing the computations on the batch of data. The correctness of the results is the main concern of this layer. With the master dataset, it is possible to recompute the whole set of data if the batch processing appears to be erroneous, or if the needs have evolved. As the batch processing is only computed once in a while, the speed layer has to compensate this downside by a low latency capability. The computations are the same as in the batch layer, but the processing has to be incremental. However, this layer lacks the correctness property. The batch processing puts to disposal its results in the serving layer, to facilitate their access by users. The serving layer has often a role of indexing and presenting data, to enable a fast access to the views created by the batch layer. To keep results up-to-date, the serving also appends the speed views to the batch views.

With these specifications, the advantages of the Lambda Architecture are a strong fault-tolerance for machine and human faults, a guarantee of a correct result with the batch layer and a low latency with the speed layer. It has inspired many Big Data architectures, such as RADStack [START_REF] Yang | The RAD-Stack: Open source lambda architecture for interactive analytics[END_REF], which is an open-source plateform used to produce interactive analyzes. They developed Druid to use it as the serving layer, that supports real-time and batch data ingestion. It is also used to allow guided exploratory analyzes concerning pre-determined insights. In [START_REF] Munshi | Data lake lambda architecture for smart grids big data analytics[END_REF], the authors apply the Lambda Architecture to process data of smart grids. They add a querying and an analytics layers, in order to propose more flexibility in the querying capabilities that overcomes the fixed precomputed views of the Lambda. The system developed in [START_REF] Lee | Implementation of Lambda Architecture: A Restaurant Recommender System over Apache Mesos[END_REF] uses the Lambda Architecture to develop a recommendation system for restaurants. It alters the original pattern of the Lambda, by having different computations in the batch and in the speed layers. The batch layer is responsible for executing heavy machine learning algorithms, while the speed layer exploits those results to propose recommendations to users. LinkedIn develops Pinot [START_REF] Im | Pinot: Realtime olap for 530 million users[END_REF], a real-time distributed datastore designed for processing OLAP queries with low latency. They use the Lambda Architecture pattern to provide LinkedIn users with near real-time analytics functionalities, such as who viewed their profile. They compute the speed views in real-time, and when the views are complete (i.e., when they represent a day or a hour depending on the granularity) they become the batch views. The incoming data are then computed in a new speed view.

However, the Lambda has also been criticized a lot, due to its complexity to maintain and to evolve both the speed and the batch layers, that have to perform the same computation, but with different paradigms. It also lacks in flexibility, as its goal is to answer only predetermined queries. Thus, as saw in the examples of implementation of the Lambda Architecture, alternative use cases such as exploratory analyzes require to modify the pattern. Furthermore, by delegating the correctness property only to the batch layer and not to the speed layer, the low latency and the correctness properties cannot be obtained simultaneously. To clarify this statement, the Lambda has to be replaced in the context of its creation. At this time, streaming systems were only at their early stages, and thus did not have all the capabilities that they have today.

The evolution of stream processing systems

Streaming systems [START_REF] Akidau | Streaming Systems: The What, Where, When, and how of Large-scale Data Processing[END_REF] had begun to emerge approximately at the same time as the conception of the Lambda Architecture. Marz had proposed Apache Storm in 2011 [START_REF] Toshniwal | Storm@ twitter[END_REF], the first stream processing system to encounter success in the industry field. It was born from the ascertainment that to produce a system with a realtime component, it took more time to create workers and queues and to ensure that their interactions are as expected than to develop domain logic. In 2015, Twitter had proposed Apache Heron [START_REF] Kulkarni | Twitter heron: Stream processing at scale[END_REF]. They used previously Apache Storm, but needed a system more suited to their needs, with better performance, scaling capabilities and easier to manage. Spark Streaming [START_REF] Zaharia | Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters[END_REF] in 2012 took a different approach and proposes a micro-batch streaming system, relying on the Spark engine dedicated to the batch processing. MillWheel [START_REF] Akidau | Millwheel: fault-tolerant stream processing at internet scale[END_REF] and Apache Flink [START_REF] Carbone | Apache flink: Stream and batch processing in a single engine[END_REF] also joined the world of streaming systems during those years.

The efficiency of streaming systems, on top of the low latency, comes with the guarantee of the processing [START_REF] Akidau | Streaming Systems: The What, Where, When, and how of Large-scale Data Processing[END_REF]. There are three main levels of guarantee: 1) at-most-once, where elements are never processed more than once, but can be never processed, 2) at-least-once, where elements are never processed less than once, but can be processed multiple times, and 3) exactly-once, that surpasses the at-most-once and at-least-once guarantees, and thus each element is processed once and only once, allowing to compute correct results. However, the exactlyonce guarantee is utopian, and in practice it is closer to effectively-once, where elements can be processed several times, but the effect on the state of the stream is only counted once. This adds a strong constraint: the processing must not have side effects that are not idempotent. It means that in case of the reprocessing of a message, the global result must not be altered. When the Lambda Architecture came out, stream processing systems had mostly only the at-most-once or atleast-once guarantee, and could not offer more. So, the Lambda can be seen as a mean to compensate flaws of an emerging technology, rather than a pattern that fully exploits it.

Toward the end of the Lambda Architecture

The Kappa Architecture [START_REF] Kreps | Questioning the Lambda Architecture[END_REF] proposes to get rid of the complexity, by keeping only the speed layer, arguing that it is enough to reach the goal of the Lambda. While this is a correct statement regarding the evolution of stream processing systems, it also discards the master dataset and the fault-tolerance property, one of the strengths of the Lambda. Another flaw in the Lambda Architecture is that it is loosely defined. Several interpretations of the serving layer can be found, that include or not the speed views. The aggregation of the speed and the batch views is not clearly defined, especially for unordered streams of data. The lack of precise definition extends to the style of the architecture. Each part is called layer, whereas layered architectures are a stacking of multiple components, where each component can interact with the components directly above or below it [START_REF] Richards | Fundamentals of Software Architecture[END_REF]. The event-driven architecture would be a more suited style. This leads to a need of a stricter definition of the architecture, as well as an adaptation of the role of its components, updated following the gain in maturity of the stream processing systems.

Using the category theory to study conservation of properties

In the research field of software engineering for architecture design, the need for proper theory and formalization has raised importance in the last decade [START_REF] Broy | Can practitioners neglect theory and theoreticians neglect practice?[END_REF][START_REF] Johnson | Where's the theory for software engineering[END_REF]. Designing, specifying and implementing software architectures are complex tasks, that require careful specifications to link and preserve characteristics through all the steps of creation. The development of theory in this field requests both practical and theoretical skills, in order to propose a model suited to the expectations, that takes into consideration the imperfections of the real-world of engineering.

ADLs [START_REF] Clements | A survey of architecture description languages[END_REF] (Architectural Description Language) have an important role to formalize architectures. Boxes and lines ADL as well as ADL based on UML [START_REF] Broy | Uml formal semantics: lessons learned[END_REF] cannot easily verify properties, due to their weak formalization. We focus on those ADL having well-established theoretical foundations. In [START_REF] Abowd | Formalizing style to understand descriptions of software architecture[END_REF], Abowd et al. provide a formal framework in Z to achieve a description of architectural styles. They argue that diagrams are not sufficient to impose only one meaning to represent an architecture, and that they can lead to misunderstanding. Malkis and Marmsoler in [START_REF] Malkis | A model of service-oriented architectures[END_REF][START_REF] Marmsoler | A model of layered architectures[END_REF] work on a formalization of architecture styles. They use the theory of sets and first order logic to build a model with ports and services to represent interactions among components. They apply their proposal on two styles: the layered and the service-oriented architecture. In [START_REF] Métayer | Describing software architecture styles using graph grammars[END_REF], Le Métayer uses the graph theory to propose a formalization of architectures. Nodes are entities of an architecture (client, server, or object entity depending on the level of abstraction), and links are communications between those entities. Mabrok in [START_REF] Mabrok | Category theory as a formal mathematical foundation for model-based systems engineering[END_REF] tries a different approach, and uses the category theory to formalize the requirements and the attributes of an architecture. Ologs, a particular application of category theory thought to represent the study of the ontology of a subject, is used to organize the architecture.

Existing ADLs are based on set theory, graph theory and use first or high order logic to check properties or consistency of architectures. However, they do not study the conservation of those properties in compositions of components. To fill this need, category theory [START_REF] Eilenberg | General theory of natural equivalences[END_REF] is a promising approach: it allows to switch from a model to another or to navigate among abstraction levels, and thus to express various problems from different science fields, such as mathematics, physics or computer science [START_REF] Spivak | Category theory for the sciences[END_REF]. By focusing on relations (the morphisms) and compositions, it proposes powerful mechanisms that can be applied to architectures. In this paper, we focus on studying the conservation or the discarding of properties in compositions of components, by relying on the behaviour of functors combined to preorders. This section only introduces some notions of category theory useful to understand this formalization, and cannot relate all the subtleties and the depth of it. We refer the reader to [START_REF] Spivak | Category theory for the sciences[END_REF] for a more complete explanation of the category theory, and to the supplementary material available1 .

A category C is composed of four basic elements: 1) Ob(C), a collection of objects ; 2) for each pair x, y ∈ Ob(C), a set Hom C (x, y) representing morphisms from x to y, namely a mean to get an object y (the codomain) from an object x (the domain). A morphism f from x to y is noted f : x → y ; 3) for each x ∈ Ob(C), a particular morphism id x known as the identity morphism on x ; 4) for each triplet x, y, z ∈ Ob(C), a composition • : Hom C (y, z) × Hom C (x, y) → Hom C (x, z). For two morphisms f : x → y and g : y → z, the composition is noted g • f : x → z. And of two laws: 1) for a morphism f : x → y with x, y ∈ Ob(C), we have f • id x = f and id y •f = f ; 2) for f : w → x, g : x → y and h : y → z with w, x, y, z ∈ Ob(C), we have (h

• g) • f = h • (g • f ) ∈ Hom C (w, z).
A product of two categories C1 and C2 produces a new category which objects are all the possible pairs (x, y) with x ∈ Ob(C1) and y ∈ Ob(C2) and morphisms (x, y) → (x , y ) are pairs (f, g) where f : x → x ∈ Hom C1 (x, x ) and g : y → y ∈ Hom C2 (y, y ).

Two particular cases of categories are of special interest to formalize architectures. The preorders, in which between each pair x, y ∈ Ob(C), there exists a unique morphism f : x → y. If there exist f : x → y and g : x → y, then f = g. The power sets, that are sets which contain all the subsets of a given set. In a category, power sets can be organized as preorder, where morphisms link two subsets if the first subset is integrally included in the second.

To formalize architectures, we define three core categories: 1) the Componentscategory, in which objects are all the components of the architecture, without any morphisms ; 2) the Architecture-category which contains all the components and with morphisms representing the interactions between components, f : x → y means that the component x sends data to the component y ; 3) the ComponentsP S-category containing all the objects of the power set of the components, that will be used to connect the components to properties. To link those categories, we use functors.

A functor F maps a category C to a category C . It is noted F : C → C , and affects both objects (F : Ob(C) → Ob(C )) and morphisms (for each pair x, y ∈ Ob(C), we have F : Hom C (x, y) → Hom C (F (x), F (y))). To be valid, a functor must observe two laws: 1) the preservation of identities: ∀x ∈ Ob(C), F (id x ) = id F (x) ; 2) the preservation of composition: for any triplet x, y, z ∈ Ob(C) with morphisms g : x → y, h : y → z, we have

F (h • g) = F (h) • F (g).
To link the categories we have previously defined, we use functors: 1) CA : Components → Architecture to integrate components in the architecture ; and 2) CCP S : Components → ComponentsP S to study the behaviour of components in a set of different components.

Properties are represented with preorders, and each value of a property is an object. Morphisms go from the most satisfying value of the property to the least satisfying value. The symbol is used as the top value to neutralize a component when it is not concerned by the property. Properties can be simple (only with true and f alse objects), multivalued, or more complex, and resulting of the composition of several other properties: in this case, properties are associated with a product of categories, and this product is linked to the next property with a functor that maps each combination to its signification in the next level property. The category ComponentsP S is connected to every property of level one (those than are not the result of a product of categories).

This formalization is applied on the Lambda Architecture, to formally prove its weaknesses. A composition exists with the batch, the serving and the speed layers because the serving layer merges the batch and the speed views to provide users with results. Using the category theory, we can extract some high-level knowledge from the known facts, given below (figure 2). The morphisms inside categories ComponentsP S and Correctness are defined as follows (with the notation B = Batch, Se = Serving and Sp = Speed):

CP S morphisms with ∅ as domain are omitted

C -t : → T rue (ComponentsP S) B -BSe : B → BSe (Correctness) t -f : T rue → F alse Se -BSe : Se → BSe id : → BSe -BSeSp : BSe → BSeSp idt : T rue → T rue Sp -BSeSp : Sp → BSeSp id f : F alse → F alse
The morphisms inside the category Real -time are the same as those of the Correctness-category. The effect on the objects of functors that link the ComponentsP S-category to the categories of the properties are given below:

CP S -C B → T rue (Correctness) Se → Sp → F alse CP S -RT B → F alse (Real -time) Se → Sp → T rue
From these given facts, we want to deduce the value taken by the first composition of components {Batch, Serving} for the correctness property. For this, we have to resolve the effect of the functor CP S -C on the morphisms: where the underlined elements are known. To establish the value of CP S -C(BSe), we have to find two morphisms in the category C that would have the same codomain: one with the domain T rue (CP S -C(B)), the other with the domain (CP S-C(Se)). Only the pair ( -t, id t ) satisfies the requirements. As the codomain is T rue, it allows us to deduce that CP S -C : BSe → T rue. Thus, the composition {Batch, Serving} yields T rue for the correctness property.

We use the same mechanism to deduce the value taken by the overall architecture for the same property: This time, the pair of morphisms that meets the requirements is (t -f , id f ). As the common codomain is F alse, it allows us to deduce that the overall architecture yields F alse for the correctness property. The same reasoning can be applied for the real-time property, that yields F alse for the overall architecture. With the category theory, we proved that the real-time and correctness properties are effective in a different layer, but that they do not hold in the whole Lambda Architecture. We can conclude with a fact about compositions of components in architectures: if an individual component does not support a property, it will cause its loss in a composition of components.

The Lambda+ Architecture pattern

To improve the Lambda Architecture, the correctness property should hold for all the components. Furthermore, the fault-tolerance should be kept, but as the reprocessing of data in a batch fashion is often incompatible with the real-time property, it should be integrated as an alternative running composition, activated only in case of a technical failure or to satisfy new needs. Use cases should also gain in flexibility, and the complexity induced by the development of the same process with different paradigms in the speed and in the batch layers should be avoided.

The Lambda+ Architecture is meant to be a renewal of the Lambda Architecture, by improving the support of the correctness property and by leveraging two main functionalities: 1) storing data for allowing flexible and exploratory data analyzes, and 2) computing in real-time predefined queries on data streams in order to have insights on well-known and identified needs. The duality between exploratory analyzes and predefined queries is of primary importance in a Big Data context, where the combination of volume and variety of data overcomes the capability of finding all the insights hidden in data. The fault-tolerance mechanism of the Lambda is kept, but is only activated when needed.

The adopted layer model of the Lambda Architecture, as stated in section 2, does not match the style of the architecture. Instead, the Lambda+ is composed of a set of components interacting together asynchronously with messages. This pattern borrows its principles from the Event-Driven Architecture style, which is well-suited for achieving performance, scalability and evolutivity. The trade-offs of this architecture style is a lack of simplicity and the difficulty of testing the whole architecture, due to the dynamic nature of the messaging workflow and the chaining of various processing components. Figure 3 shows an overview of the Lambda+ Architecture which includes five main components. The data traffic controller component is the entry point of the Lambda+ pattern. Data sources can have very different natures, such as an extraction from an existing store or a connection to an endpoint API that provides data. This diversity requires a structuring of the stream of data, that can be achieved in different ways depending of the functional needs behind the architecture. The data traffic controller organizes data into streams. The filter and pipe architecture style is a suitable fit for this component: only lightweight processing are applied, such as removing duplicates or adding a timestamp, before sending data in a communication system that allows other components to have access to them. This mechanism yields a great independence among components, and is the basement for the fault-tolerance characteristic.

The batch layer has no reason to be anymore, and it can be replaced by a component more suited to the situation: the streaming ETL component. ETL processes have been part of analytics since a long time, mainly used to populate a data warehouse. However, these last years the need for more freshness in data has become a major preoccupation, and the batch behaviour of ETL -often run once per day at night -cannot fulfill this need. So, several works focus on executing ETL in real-time in order to perform low latency analysis [START_REF] Vassiliadis | Near real time ETL[END_REF][START_REF] Fernandez | Liquid: Unifying Nearline and Offline Big Data Integration[END_REF]. Streaming ETL transforms data continuously, rather than periodically as it is the case in classic batch ETLs. Stream processing systems are often used to achieve this goal. The role of the streaming ETL component is to populate the storage component, and to transform data if needed for doing so. It works in real-time when data arrive from the data traffic controller, and also in deferred time when data arrive from the master dataset. Data are extracted from the master dataset for example when a schema modification occurs in the storage component.

The aim of the master dataset component is the same as in the Lambda Architecture. It stores all the raw data, in case of an evolution of the streaming ETL component or of a failure that requires the re-processing of the data. This component contributes to the fault-tolerance property. To avoid the complexity blamed on the Lambda Architecture with the duplication of processing in the batch and speed layers, data are not directly processed in this component, but rather only extracted from the master dataset, to then be sent to the streaming ETL component. Thus, to maintain or evolve the architecture, only processes in the streaming ETL component have to be modified, avoiding at the same time one of the most criticized pitfall of the Lambda Architecture.

The real-time insights component is dedicated to compute predetermined queries or algorithms directly on the stream of data, in real-time, essentially using stream processing systems. The latency and the correctness are critical, but the technical advances of stream processing systems can handle these requirements. Computations can be simple, such as an aggregation (count or sum), or more complex, such as anomaly detection in time series [START_REF] Ahmad | Unsupervised real-time anomaly detection for streaming data[END_REF]. Operations in this component are well identified and defined, and procure useful insights about data being processed. The storage component can be used to explore data and to find the value they can offer. This knowledge can then be exploited to automate the extraction of value in the real-time insights component. The result of these processing can be stored in the storage component, but due to the be-haviour of stream processing systems detailed in section 2, this storage step has to be idempotent.

The storage component can be of different nature following the needs. It can be a standard data warehouse, a polystore or a data lake. This storage system is fed with data from the streaming ETL component and eventually by the realtime insights component. It puts to disposal processed data in a more suitable format, used for offline and exploratory analyzes. Data warehouses are a mature technology, that have been around before the era of Big Data [START_REF] Inmon | Building the data warehouse[END_REF]. They often gather data from different sources among an organisation with the help of ETL processes, in order to format and clean data for a business intelligence use. Data warehouses are built to help business analysts by structuring data according to a static schema for a given subject. By doing so, the analyst must only know the schema of his subject to extract value from data. However, this structuring induces a lack of flexibility. A polystore refers to a system that integrates heterogeneous database engines, storage systems and multiple data manipulation or programming languages using different paradigms [START_REF] Gadepally | The BigDAWG Polystore System and Architecture[END_REF]. The use of polystore brings several advantages: it allows to organize data according to particular use cases (e.g. graph DBMS well support linked data and graph traversal or path queries) and it enables parallel processing among several datastores according to the specificities of each kind of system in the polystore [START_REF] Alotaibi | Towards Scalable Hybrid Stores: Constraint-Based Rewriting to the Rescue[END_REF]. Data lakes are less well-defined than data warehouses or polystores [START_REF] Sawadogo | On data lake architectures and metadata management[END_REF]. They are often a solution when all data available are harvested, but their use from an analytical point of view is not yet defined. They have emerged to compensate the lack of flexibility of data warehouses. In these lakes, all data are gathered without a common schema, often in a unstructured or semi-structured form. Yet, with this freedom comes a high heterogeneity among data of the lake (in their source, their format, their content, their veracity, etc.), which can turn the data lake into a data swamp if it is not correctly organized. The application of the formalization on the Lambda+ Architecture pattern (figure 4) is straightforward and follows the same mechanism that the one explained in section 3. A more detailed proof is given in the supplementary mate-rial, including the one for products of categories. Two running compositions of components are possible: with or without the master dataset, as it is only linked to the streaming ETL when data have to be reprocessed. There is no component that invalidates the correctness property, so it cannot be lost in a composition, and, contrary to the Lambda Architecture, this property holds for the overall architecture. Concerning the real-time property, only the master dataset induces its loss, but in exchange the Lambda+ activates the fault-tolerance property. It is an acceptable trade-off, as the master dataset is only used in emergency cases, to reprocess data after a failure or when the needs change. We have applied the Lambda+ pattern in an interdisciplinary research project (Cocktail) which aims at studying the discourses in the domains of health and food, as well as identifying weak signals in real time using social network data. We needed an architecture able of supporting a continuous gathering of data, while allowing real-time and flexible off-line analytics. It should also have a strong fault-tolerance property, as the use of data could change depending on the results of our researches. It resulted in the Hydre architecture (figure 5), that has been in production since April 2019, to harvest data from Twitter, compute real-time insights and store data for exploratory analyzes. It uses a 20 nodes Hadoop cluster, a 5 nodes Kafka cluster and 4 other servers to host a polystore and applications for exploratory analyzes, including Jupyter notebooks and Spark/Scala analytics processes. The major components of the Lambda+ Architecture are implemented as follows.

Data traffic controller

The data traffic controller is composed of harvesting sub-components, that use Twitter Search and Stream API with specific criteria (hashtags, ac-counts, etc.) in order to gather tweets in JSON format about a chosen subject. We have only one datasource in this case. The harvesting sub-components add only lightweight information such as the timestamp of the harvesting. They are developed following the actor model with Akka, and send data to Kafka topics, the backbone of the architecture, that allows loosely coupled communications with the other components. The master dataset is implemented with Hadoop HDFS. Raw tweets are stored as lines of files, and the re-proccessing of data can be done by reading these files and sending each line as is, in another Kafka topic. The streaming ETL uses Kafka consumers to insert data in micro batch. The streaming ETL applies some transformations, and then stores tweets in the storage component, a polystore in our case. The polystore includes a relational, a graph, and a time series DBMSs. These databases are used for exploratory analyzes, mainly performed with Jupyter notebooks. Alongside, the real-time insights component extracts and aggregates several information about the harvesting, such as popular hashtags or users, using Kafka Streams. It stores the results in the time series database of the polystore. Although this insertion is a side-effect of the stream processing, it is an idempotent action, because the count of the elements will always yield the same result with an effectively once guarantee, and this result is stored for each element, replacing the old value if it already exists.

We have three main harvests continuously running, two global on food and health, and one more specific on COVID-19 vaccines. With the architecture active for 20 months in production, we have gathered 8.3To of raw data. We have already leveraged the fault-tolerance property of the master dataset several times, to apply a new processing as the needs had changed. It was done without impacting the real-time insertions of the streaming ETL. We have also realized some maintenance operations on the streaming ETL without having to stop the harvesting thanks to the message retention of Kafka, and by resuming the processing where it was paused once the maintenance was over. From a user point of view, the polystore is used to cover different needs: 1) to help social science researchers to find new keywords by displaying some query results and macroscopique indicators computed by the real-time insights component on an application server and on Jupyter notebooks ; 2) during research meetings, exploratory analyzes are done in live to guide the formulation of a social hypothesis based on available data ; and 3) once the hypothesis is well-defined, to extract a specific corpus fixed in time to perform more deeper analyzes.

Conclusion

We showed the obsolescence of the Lambda Architecture, mainly due to its limited use cases and to the evolution of stream processing systems. We proposed the Lambda+ Architecture pattern, the successor of the Lambda Architecture, that gets rid of its flaws. The Lambda+ defines a more flexible architecture, capable of handling both exploratory and real-time analyzes, and that fits more various use cases than the Lambda Architecture. We uses the category theory to study the conservation of properties in compositions of components, and applied it on the Lambda and on the Lambda+ Architecture.

For future work, we plan to develop our formalization to study more various aspects of architectures: 1) to navigate among abstraction levels (i.e., the level of detail of the representation of the architecture) ; 2) to verify if an architecture follows a given style or pattern by using full functors (i.e., surjective functors) ; and 3) to extend the property description, including numerical values (e.g., to measure the execution time and deduce if it can be considered as real-time).

Fig. 1 :

 1 Fig. 1: Overview of the Lambda Architecture

  CP S -C : Hom CP S (B, BSe)→ Hom C (CP S -C(B), CP S -C(BSe)) CP S -C : Hom CP S (Se, BSe)→ Hom C (CP S -C(Se), CP S -C(BSe))

  CP S -C : Hom CP S (Sp, BSeSp)→ Hom C (CP S -C(Sp), CP S -C(BSeSp)) CP S -C : Hom CP S (BSe, BSeSp)→ Hom C (CP S -C(BSe), CP S -C(BSeSp))

Fig. 2 :

 2 Fig. 2: Study of the preservation of properties in the Lambda Architecture 2

Fig. 3 :

 3 Fig. 3: Overview of the Lambda+ Architecture

Fig. 4 :

 4 Fig. 4: The formalization of the Lambda+ Architecture pattern

Fig. 5 :

 5 Fig. 5: The Hydre architecture

https://github.com/AnnabelleGillet/CategoryTheoryForArchitectures

To simplify the schema, the ComponentsP S-category has only individual components, as well as possible compositions of the architecture, and only the links from components to properties that does not lead to are represented.

Acknowledgment: This work is supported by ISITE-BFC (ANR-15-IDEX-0003) coordinated by G. Brachotte, CIMEOS Laboratory (EA 4177), University of Burgundy.