Jean Raphael Richa

Hamza Noueihed
email: hamza.noueihed@etud.univ-pau.fr

Richard Chbeir
email: richard.chbeir@univ-pau.fr

Data Obsolescence Detection Within Connected Environments

Keywords: Connected environments, Data obsolescence detection, Data quality metrics, Dimensional architecture

Connected environments collect data from millions of devices every second presenting valuable information to the community. However, data integrity faces critical research gaps especially when it comes to data obsolescence and its detection. Thinking of data obsolescence as an independent terminology, we define it as the state wherein data is no longer significant or effective with respect to the device, parent zone and/or the environment. Accordingly, this paper provides connected environments with a dimensional architecture that : (1) assess predefined quality metrics of each sensed data, (2) discovers interrelation between deployed devices using clustering techniques, and (3) identifies data obsolescence through three main layers; namely, the device, hosting zone, and/or environment level.

I. INTRODUCTION

"Smart and more connected" is the current trend within the emerging metaverse. Smart cities deployed to collect millions and millions of data are helping in better understanding human activities, improving quality of life, and easing paths for sustainability. For instance, cities may boost their energy efficiency by 30% if their data is well utilized [START_REF] Moustaka | A systematic review for smart city data analytics[END_REF]. It is also the case in other sectors such as healthcare, finance, transportation, and retail to mention a few. However, this is possible when having a good information and communication technology (ICT) infrastructure [START_REF] Al Nuaimi | Applications of big data to smart cities[END_REF]. Connected environments represent one important sector reinforcing data collection, analysis, and processing [START_REF] Chbeir | Opencems: An open solution for easy data management in connected environments[END_REF]. Quantifying real data from a connected environment can aid in advanced advisory and warning information across various maneuvers (such as traffic interactions and driving decisions which showed no evidence of negative impact [START_REF] Ali | The impact of the connected environment on driving behavior and safety: A driving simulator study[END_REF]). This pushes, for instance, regional agglomeration's collective intelligence forward, supporting key urban flows; namely, forecast and management [START_REF] Novotnỳ | Smart city concept, applications and services[END_REF]. Accordingly, this allconnected paradigm is producing more and more valuable data for country, society, community, or even individual.

While connected environments are providing us with valuable data, it is notable that not all data is of benefit for the topic of interest. While many efforts have focused on data speed and security, less emphasis has been dedicated towards data quality and more particularly data significance. "Data Obsolescence", a key pillar of data integrity, is the representative behind this research gap. Knowing to which extent a data is useful and relevant to the question in hand will increase productivity and efficiency while reducing the technological lags that various sectors face. It is to be noted that the obsolescence term was initially introduced as electronic part obsolescence when military specifications were required by some stringent governmental products; however, these specifications were hard to attain causing the unfinished product to become insignificant [START_REF] Singh | Obsolescence driven design refresh planning for sustainment-dominated systems[END_REF]. Similarly, a data is thought of as a product with a level of importance impacting the efficiency of the whole data set. On the other hand, a data's significance towards the data set it belongs to experiences minimal research especially in the information technology sector. Connected environments are one of these sectors, especially that they are purposed to collect, store and analyze huge amount of data. In most existing approaches, obsolescence is represented globally as another terminology such as contradiction or relevance. Although these terminologies are key factors in data quality measurements, they should act as indicators helping in data obsolescence and not represent the latter itself. For this purpose, in our previous work [START_REF] Richa | Data quality computation for obsolescence detection within connected environments[END_REF], we aimed in officially defining data obsolescence as an independent terminology providing it with quality metrics essential for its computation and detection. Accordingly, in this paper, based on the metrics we provided in [START_REF] Richa | Data quality computation for obsolescence detection within connected environments[END_REF], we propose an efficient dimensional architecture for detecting obsolete data within connected environment infrastructures. Detecting data obsolescence is only the first step inside a bigger approach. Data processing is our future goal as it addresses the sources that constantly generate such obsolescence. Being able to detect such issues will help in providing processing solutions and a possibility to return the data's usefulness or even finding a new fitting scenario for the data. Take note, solving the obsolescence source does not necessarily bring improvements at the level of data storage solely; other benefits might originate in terms of location, deployment, and data querying of the device. For instance, why would a device stay in its location or zone if one knows that it is not producing useful data (such as a monitoring camera that is partly or fully blocked by a wall or any other object)? Detecting obsolescence can uncover such mishaps improving device deployment for a more comprehensive collection. The latter is one of many scenarios on how understanding data is a direct way of understanding productivity methodologies.

The remainder of the paper is organized as follows: Section II illustrates a motivating scenario. Section III presents the state of the art. Section IV describes our proposed data obsolescence detection architecture. Finally, Section V summarizes the paper and provides future research directions.

II. MOTIVATING SCENARIO

Here we consider a scenario of a smart connected conference venue to clearly highlight and describe connected environments and data obsolescence motivation and concerns. It is important to note that we are presenting the setup and needs of connected environments in a simplified manner, and it does not necessarily fully incorporate all data obsolescence issues. Figure 1 depicts this connected environment which includes five zones (i.e., geographical areas) whereby four zones designate conferences rooms sharing a common hallway zone. The zones host different categories of devices: (1) static devices (e.g, temperature (T1, T2, T3, T4, and T5), fixed camera (C4)), and (2) mobile devices (e.g., rotating camera (C1, and C2), mobile phone (P1, and P2)). Each device has a well defined location and senses different environmental parameters (e.g., temperature, presence, motion) within particular coverage areas (ca) represented by hashed shapes (i.e., hashed rectangle, circle, or cone). A device's coverage area is not necessarily limited by the respective hosting zone. In other words, a device can measure data from adjacent zones depending on the data and border nature (e.g., since zone2 and zone3 are separated by a glass wall, C2 located in zone2 might capture data from zone3).

One can interact with the environment through queries, by requesting data from any device. In Figure 1 queries are illustrated by dotted shapes. There are two types of queries:

(1) Projection: requesting all data collected by a device, (2) Selection: querying certain data gathered by a device through specifying particular conditions. Each query is assigned to one or multiple devices, and each device keeps record of the requests it receives. For instance, q1, q2 and q3 are assigned to device C1 while q4 and q5 are assigned to C4, both devices keeping track of when data was queried.

This scenario highlights the dynamicity of a connected environment as it exposes its evolution over time. This dy-namic nature is due to the zones capability of merging or splitting (e.g., zone4 and zone5 are combined into zone4bis resulting in a larger conference room), and mobile devices' capability of moving. For instance, mobile devices can change location within the same zone (e.g., device P1 moved within zone1), change zones (e.g., P2 moved from zone1 to zone2), and/or rotate to sense data from different angles (e.g., rotating cameras C1, C2, and C3).

One or more sources might be behind the generation of data obsolescence. Here are some illustrative cases:

• Case 1: A 360 • rotating camera (C1) covering the entire zone every minute while only data every hour are required (i.e., temporal source). • Case 2: A 360 • rotating camera (C1) periodically collecting data from the whole zone when only data belonging within 120 • portion are needed (i.e., spatial source). • Case 3: A mobile device (P1) capturing several types of data (e.g., temperature, presence) while only temperature data are required (i.e., contextual source). Now that the general view was described, the next step revolves around listing the challenges we will be addressing:

• Challenge 1: How to assess data as obsolete ?

• Challenge 2: How to evaluate obsolete data with respect to the connected environments and their dynamicity ? • Challenge 3: What dimensions to consider for obsolescence source interpretation ?

III. STATE OF THE ART

In 2015, McKinsey Group estimated by 2025 a potential loss of $3.9-11.1 trillion per year for the IoT sector; i.e., worth 11% of the world's economy [START_REF] Cremer | The integrity challenge of the internet-of-things (iot): on understanding its dark side[END_REF]. Lack of data processing being one of the latter's main causes, increasing data cohesiveness is deemed necessary. Heterogeneous solutions are required with the evolving smart cities concept supporting higher urbanism qualities [START_REF] Dameri | Searching for smart city definition: a comprehensive proposal[END_REF]. In [START_REF] Chamara | Ag-iot for crop and environment monitoring: Past, present, and future[END_REF], a generic IoT architecture is built of: (1) a perception layer: devices and sensors, (2) a network layer: data transmission, and (3) an application layer: user view point. Here, the authors explain how the perception layer continuously faces major challenges including storage supervision and device durability. Similar cases for the network layer, interference and communication range are in crucial

✓ ✓ ✓ [9] ✓ ✓ ✓ [10] ✓ ✓ ✓ ✓ [11] ✓ ✓ ✓ ✓ [12] ✓ ✓ ✓ ✓ ✓ [13] ✓ ✓ ✓ [14] ✓ ✓ [15] ✓ ✓ ✓ [16] ✓ ✓ ✓ Our contribution ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
need for solutions. These obstacles still need processing due to the aforementioned issue; lack of emphasis on the data itself and how significant it is. This reflects the impact that data obsolescence will have once it is detected and dealt with. Connected environments, using this generic IoT architecture, are in need of a detection methodology required by this domain's research breach.

The aim of our study is to provide a solution to detect data obsolescence within a connected environment. Although little (to no) research is done in this field, yet there exists some studies related to obsolescence in general. Life cycle curve forecasting is the main obsolescence detect method used in [START_REF] Sandborn | A data mining based approach to electronic part obsolescence forecasting[END_REF]. The authors depend on data mining; however, it is worth mentioning that this obsolescence is specific to physical electronic parts (e.g., integrated circuits). In other words, the authors deal with real vendor parts while connected environments generally focus on cloud data provided by the embedded devices (e.g., real and virtual sensors). A similar topic, which is the product life cycle, is the focus in [START_REF] Schulze | Obsolescence as a future key challenge for data centers[END_REF], especially when it comes to data centers and infrastructure. The authors suggest: (1) reactive approaches: solving obsolescence after it occurs, and (2) proactive approaches: preventive measures and obsolescence predictability, or the formers combined. While they appeal representative of data obsolescence, they are described theoretically without an actual detection and computation strategy; the gap this paper aims to fill in. In [START_REF] Romero Rojo | Obsolescence management for defence and aerospace systems: state of the art and future trends[END_REF], Rojo et al. extend on electronic part obsolescence agreeing how a component's life cycle is usually shorter than that of the system; i.e., insignificance at the technology level. Logic wise, the aim is to prove that a similar process occurs at the data level of evolving technologies. At one point, a piece of data is very important to a system or user while at another point, it might become no longer up to date and not representative of the real world. Similar to how electronic parts have detection methods, this paper's mission revolves around filling in this gap for data obsolescence via providing a layered detection infrastructure.

Storage data obsolescence, specifically temporal obsolescence, is the main topics [START_REF] Finger | Temporal data obsolescence: modelling problems[END_REF] discusses. Data repositories aim to save the world's dynamic states; however, if they change faster than our capability to save, they become obsolete. In other words, it's temporal obsolescence representing data going out of date whereby each data has a credibility parameter which changes over time. For that purpose, the authors agree that their analytical methodology is a difficult approach as the decay in credibility requires experts in the domain at hand. Similarly in [START_REF] Alieksieiev | Big data aggregation algorithm for storing obsolete data[END_REF], old data is considered obsolete with respect to time solely; i.e., the authors mainly focus on the timeliness factor for classifying obsolescence reiterating the fact how it is regularly confused as obsolescence itself. In this paper, we aim to reveal that data obsolescence does not necessarily have to be solely a temporal phenomena as it can also be spatial and/or contextual (i.e., providing an architecture capable of detecting and identifying the type of obsolescence). In [START_REF] Giralte | Managing data obsolescence in relational databases[END_REF], obsolescence has been also addressed in databases, namely relational databases. Each table undergoing obsolescence cycles goes through at least three checks each with a specified time limit. A data record is then considered obsolete when not accessed after a certain number of cycles 'M'. In this context, the provided approach focuses more on data popularity whereby the number of access is the primary pillar. This refers back to the idea of confusing data obsolescence with another existing quality indicator. Accordingly, popularity is a crucial parameter that works alongside other indicators towards a comprehensive obsolescence detection.

Few authors (e.g., authors of [START_REF] Sandborn | Software obsolescence-complicating the part and technology obsolescence management problem[END_REF]- [START_REF] Déméné | L'obsolescence des produits électroniques: des responsabilités partagées[END_REF]) categorize obsolescence as either: (1) technological obsolescence: new technologies replaces older ones, (2) functional obsolescence: reduction of a system's performance, (3) logistical obsolescence: shortages in manufacturing and materials, (4) ecological obsolescence: high environmental negative impacts, or (5) economic obsolescence: discarding a product due to high maintenance. In [START_REF] Trabelsi | Prediction of obsolescence degree as a function of time: A mathematical formulation[END_REF], the authors provide mathematical detection/prediction methods for obsolescence based on the category of interest. Gaussian distribution is used for data obsolescence at the endof-product level. For general technological obsolescence, the authors suggest logistic regression, binary assessment such as in or out of production, in order to predict the obsolescence. In [START_REF] Trabelsi | Prediction of obsolescence degree as a function of time: A mathematical formulation[END_REF], other insights are put forward tending to lean towards electronic part obsolescence and product life cycles. Accordingly, this reinforces the lack of detection and prediction methods at the level of data. In other words, technological obsolescence is not necessary that of a physical electronic part, rather a much wider scope. For that purpose, our study here extends our previous research [START_REF] Richa | Data quality computation for obsolescence detection within connected environments[END_REF] where we provided several definitions (about how the connected environment is built with all its zones and sensors, sensor measurements called individual data, the sensor's data set called data item, the various parameters provided by the sensor's documentation and environment, etc.). Specific indicators are computed at the level of individuals data (e.g., timeliness) and data item (e.g., accuracy) to provide a comprehensives detection pipeline. Table 1 re-emphasises, in a comparative manner, how existing obsolescence detection approaches fail in covering the following three criteria that address the challenges previously listed in Section II:

• Data quality metrics: Stating the data quality metrics taken into account by the approach (cf. Challenge 1). • Connected environment and dynamicity: Denoting whether the approach is compatible with the connected environments and its changing nature (cf. Challenge 2). • Obsolescence grounds: Specifying the obsolescence types supported by the approach; namely, temporal, spatial, and/or contextual (cf. Challenge 3). While few related works (cf. Table I) manage to discuss some quality indicators (such as popularity and relevance), none entirely fulfils all our pre-proposed criteria (i.e., accessibility ac, popularity pl, relevancy rl, timeliness tl, accuracy a, availability av, credibility cr, contradiction ct, correctness cn, format consistency f c). Worth mentioning, these approaches delve into the indicators from a general-scope perspective without necessarily putting forward a calculation methodology that can be applied. As noticed in Table I, most papers cover data quality from the time perspective with less focus on other obsolescence possibilities such as spatial and contextual aspects. Opposite these approaches, our contribution is comprehensive using various data quality metrics especially that it considers the dynamic nature of the connected environments. Focusing on connected environments and with the pre-identified algorithms, we discuss in the next section how quality metrics, being the key pillars for analysis, contribute to our layered architecture.

IV. PROPOSAL

Our previous work [START_REF] Richa | Data quality computation for obsolescence detection within connected environments[END_REF] selects data quality metrics necessary for data obsolescence, categorizes them into individual data and data item quality metrics, and adapts their definitions to deal with connected environments. In this context, we provide data obsolescence with the following definition: "Data obsolescence in connected environments describes the state wherein data is no longer significant or effective with respect to the device, parent zone and/or the environment". In this section, based on the provided definition, we propose an approach for detecting data obsolescence (cf. Fig. 2). This approach consists of three main modules: A) Quality Assessment and Storage: Pre-detection process responsible for collecting data and enriching it with efficient metadata. B) Device Clustering: Clustering process allowing to detect dynamically the device neighborhood. C) Data Obsolescence Detection: Detection process employed for identifying data obsolescence with respect to the device, parent zone and environment.

The modules (A) and (B) process differently than the module (C). More specifically, the first two modules are always running and listening to the environment (i.e., Batch modules), while module (C) only runs on requests (i.e., On-Demand module). Figure 2 represents the workflow followed by the modules. As we can see, one invokes the module (C) whenever he needs to detect the obsolescence of specific devices during a given time. During this detection process, the module (C) interacts with the batch modules (A) and (B) in order to fetch real-time information about data quality and device neighborhood respectively. This module calculates and saves the quality metrics necessary for data obsolescence detection pipeline (cf. Fig. 3). It maintains and provides the most up-to-date data quality metrics. The metrics computation is triggered for every new sensed data and/or every data interacting with any query. In other words, whenever d i gets collected via a sensor or becomes the point of interest of a query, its quality metrics (i.e. m d , m D ) gets created or modified respectively. Subsequently, two actors can execute this process: sensors that generate the data and the queries that request the existing data. For example, each time a query retrieves d i , its ac increases with respect to the incremented access number. In essence, whether a data is getting gathered or engaged by a query, it goes through the three following phases pursing its storage along with its corresponding quality metrics:

1) Individual Data Quality Assessment: computes the quality metrics of the new individual data d i represented by m d :≺ ac, pl, rl, tl, sa ≻. Precisely, it is composed of accessibility ac, popularity pl, relevancy rl, timeliness tl, and spatial accuracy sa. The storage of d i and m d is illustrated in Fig. 3.

When it comes to popularity (pl), it is based on the number of requests an individual data d i receives out of the overall requests to the respective data item set D i . In other words, every time d i is accessed, its popularity increases; however, all other individual's popularity would also require update to accommodate the new total number of requests towards D i . The latter is exhaustive; continuously updating all individuals' popularity percentage per access is inefficient. Consequently, we save pl as the number of access; instead of saving the popularity percentage, saving the number of access is sufficient whereby pl is computed when required. This approach leads to updating only the individual requested and incrementing the total access of the data set; a less bulky process providing the same outcome.

2) Data Item Quality Assessment: assesses the data item's new quality metrics computed after d i gets collected. It is designated by m D :≺ a, av, cr, ct, cn, f c ≻, and consists of accuracy a, availability av, credibility cr, contradiction ct, correctness cn, and format consistency f c. It is saved in the tuple of d i (cf. Fig. 3), allowing to track how the quality of the data item is getting altered over time as more and more data is generated by the sensor. Subsequently, it enables the preservation of m D history. It is worth mentioning that the current data item quality metrics are saved in the tuple of the last collected data d i .

3 This module is responsible for dynamically computing and storing device neighborhood; an essential requirement for data obsolescence detection pipeline (cf. Fig. 4). This is achieved by identifying clusters of neighbors; group of devices that are thought to impact on another. There are two types of neighbors: (1) devices sensing same physical properties, sharing similar spatial characteristics (i.e., located nearby), and having compatible coverage areas (i.e., intersects, contains, equals), and (2) devices receiving queries asking for same physical characteristics, and are compatible (i.e., intersects, contains, or equals).

To pinpoint the aforementioned neighbors, the clustering module goes through multiple steps (cf. Fig. 4):

1) Compatible Zones Fetching: The initial step consists of retrieving and grouping all compatible zones; zones that are directly adjacent and have physical barriers that do not block sensing. Namely, the latter are selected according to the adaptability of the physical barriers, type of detection and devices' coverage areas. Accordingly, this step helps determine which zones should be considered when locating devices' neighbors. Identifying neighbors from several zones is crucial since devices are capable of measuring data outside their hosting zone (i.e., device c2 cf. Fig. 1).

2) Device Clustering: The aim of this step is to identify all device's neighbors within compatible zones. To achieve this, two clustering algorithms are applied in parallel for every group of compatible zones. 3) Cluster Unionization: Each clustering algorithm, previously mentioned, results in several groups of devices. In this step, these two results are merged into a single one.

As a result, this model identifies all device's neighbors. However, as aforementioned in Section II, the connected environment's dynamicity leads to unstable variations in a device's neighborhood. For this purpose, the clusters are updated whenever the environment evolves (e.g., new devices appear, devices change zones).

C. Data Obsolescence Detection

This module is implemented for detecting obsolescence of data collected by particular devices. The detection is applied on a selection of devices based on the system's default choice, expert's recommendation, or user's customization (e.g., obsolescence detection every day at 3:00 for the data generated/accessed within the last 24 hours by certain devices). Consequently, two main inputs are taken into consideration:

(1) a queue of devices to check obsolescence for, and (2) a specific time span [t 0 , t 1 ] of interest that the data was generated within. As an end result, the framework identifies the obsolescence result and type for each data individual.

Data generated by a single device is concerned relative to this device, the related zone, and the environment itself. In other words, to be able to check data obsolescence, it is mandatory to check it with respect to all entities that might be in need of it. For that purpose, in order to have a cohesive detection pipeline, our architecture is based on three main levels: (1) Device level, (2) Parent zone level, and (3) Environment level, each to be later on delved into.

1) Device Level: As depicted in Figure 5, this level examines sequentially a queue of devices through retrieving, analyzing, and verifying each device's data individuals and qualities. The latter produces a group of the obsolete individuals per device. Each device then undergoes a further evaluation, once all its data have been assessed, to determine its overall quality. In essence, for every device concerned, this level evaluates each piece of data from two different perspectives: (a) Individual data layers: per data for a more comprehensive detection, and (b) Data item layers: against all the data collected by the same device. The first step is retrieving the quality metrics (i.e., m d , m D , and m avg ) of d i (cf. Fig. 3). Following this, the algorithm "Individual Data Quality Check" (algorithm 1) is applied for detecting whether the individual data d i is obsolete or not. It computes two main quality scores:

• m d score : reflecting a weighted average of the quality metrics of d i (cf. line 6 in algorithm 1). • m D score(di) : representing the variation of the data source's quality metrics resulting from the collection of d i (cf. line 7-8 in algorithm 1). Based on the thresholds assigned to each of the two scores, d i is acknowledged as either of low quality or not (cf. line 9-10 in algorithm 1). Subsequently, each d i is labeled as potentially obsolete or not whereby the obsolete ones get appended to an obsolescence data pool. Once all the data of a single device have been evaluated, the respective pool undergoes an overall check at the following data item layer. b) Data item layer: This layer takes as input the time settings [t 0 , t 1 ] and a set of devices {d 1 , d 2 ... d n } as explained earlier. In this context, for each device, we first retrieve the targeted set of data [d j , d i ] from the storage; sensor's data collected between t 0 and t 1 inclusive. Next, all d in [d j , d i ] goes iteratively through the data individual level which results in an obsolete data pool. This pool is necessary for applying the algorithm "Data Item Overall Check" (algorithm 2). It analyzes the device's overall obsolescence; a step that aims to confirm whether only the individual data is of low quality, or the data item it belongs to (cf. line 2-4 in algorithm 2). 2) Parent Zone Level: If the device level shows that d i is of low quality, this is not sufficient to approve its obsolescence. Accordingly, this level checks the validity of the obsolescence data pool, generated at the device level, with respect to the device's hosting zone. To do so, each d i belonging to the obsolete data pool gets checked against its devices' neighborhood within the same zone (i.e., explicit neighbors). It is necessary to perform the latter due to the possibility of an individual being obsolete to the data item it belongs to, but rather b) Data pool generation: In this step, a data pool is created and filled with the data individuals of every explicit neighbor of D i . Namely, this pool represents a collection of explicit data sensed within the [t 0 , t 1 ] time-frame. In the next step, each obsolete individual data collected by D i is analyzed against the pool to approve or disapprove its obsolescence.

c) Obsolescence classification: The parent zone architecture is given a queue of low quality data to check against the data pool created earlier in part b. At the device level, d i was proved to be obsolete; however, its mandatory to check if it might be comprehensive with the neighboring devices. For instance, consider cameras C2 and C3 of zone2 (cf. Fig. 1). These two cameras might help complement one another to get a comprehensive view of the whole room specs. In other words, while C2 might miss some data due to a blind spot, C3 can come in handy if it is collecting data from such areas. This reiterates how it is possible for one piece of data to be obsolete at the device level, but not with respect to other devices. Accordingly, to validate if an obsolete data is obsolete at the parent zone level, the individual undergoes classification machine learning algorithms with respect to the data pool. This results in either: (1) Processing Data: data proves to be not obsolete and sending it to to check what might have caused its obsolescence at the device level, or (2) Going up the hierarchy (i.e., environment level): data proves to be obsolete and is sent for obsolescence check with respect to other zones.

3) Environment Level: If both device and zone levels prove d i to be obsolete, the environment layer (cf. Fig. 6b) gets executed as the confirmation coating. Correspondingly, in this level, the obsolescence of d i is checked with respect to the neighboring zones of the hosting zone. In other words, each d i gets evaluated against its device's neighbors located across different but compatible zones (i.e., implicit neighbors). That is the case as when a device is deployed, the coverage area is based on, but not limited to, the zone's physical barrier. In this dynamic connected environment approach, a device can sense values from another zone if they are separated by what doesn't interrupt the sensing procedure (e.g., glass or virtual wall). Going back to zone2's C2 and C3 cameras (cf. Fig. 1), we notice camera C3 is located in zone 3 but is gathering some information from zone 2. Consequently, C3 might be collecting information proving C2's potentially obsolete data as significant or confirming its obsolescence. In essence, the same process applied at the parent level (neighbor fetching, pool generation, and obsolescence classification) is applied at the environment level. However, at the environment level, the neighbor fetching is concerned with implicit devices; devices from the different zones belonging to D i 's cluster group.

All the aforementioned levels work hand in hand providing one complete detection architecture depicted in Figure 7. In essence, each data sensed passes through device, parent zone, and/or environment level to check its obsolescence. Furthermore, it undergoes an additional service (i.e., explainable data obsolescence) that is essential before its final treatment.

D. Explainable Data Obsolescence

The detection architecture presented earlier classifies data into obsolete or not. However, relying solely on this categorization is insufficient for adjusting obsolescence efficiently; additional strategies are required to address this sophisticated issue appropriately. In this context, this section introduces a solution that revolves around pinpointing the source responsible for producing obsolescence; "explainable data obsolescence". This verification process plays a crucial role in properly reducing concerns related to obsolescence. Obsolescence can fall into one or a combination of the following categories:

• Temporal obsolescence: poor quality is related to the time at which the individual data was collected. • Spatial Obsolescence: poor quality is related to the location from which the individual data was collected. • Contextual Obsolescence: poor quality is related to the content of the individual data.

Similar to the detection architecture, the explainable data obsolescence is processed at three dimensions (i.e., device, zone, environment) forming three split parts: (1) Part 1: type check with respect to the device it belongs to, (2) Part 2: with respect to the explicit devices within the same zone, and (3) Part 3: with respect to the implicit devices from neighboring zones. This module is established at the end of each level. The latter requires as input the classified data with their corresponding metadata (i.e., collection time, spatiality, attribute) and outputs the obsolescence grounds. The aim of this module is to check what's in common between the data, it inspects the data of every level and checks whether the obsolescence is occurring during a periodic range of time, and/or within a particular common spatial location, and/or for one specific attribute. With each shift to a higher level (e.g., device to zone level), the number of metadata increments, resulting in a higher number of features, which subsequently leads into more powerful and semantic results. For instance, after implementing this process on the camera C1, illustrated in the motivating scenario (cf. Fig. 1, Section II case 2), obsolete data was noticed to share common spatial properties; namely, they originate from the 120-degree section of the camera's rotation. This helps infer how this spatial section is the potential source of obsolescence accordingly marked for inspection and treatment.

V. CONCLUSION Data integrity faces a critical research gap especially when it comes to data obsolescence -when data becomes insignificant towards its original purpose -and its detection. Thinking of data obsolescence as an independent terminology, this paper provides connected environments with a dimensional architecture to detect obsolescence using predefined data quality metrics. Namely, sensed data initially passes through a quality assessment phase after which it is checked against device, zone, and environment layers for obsolescence. With the detection architecture examined, the next step involves providing the experimental results on real deployed devices. The detection architecture this paper proposes relies on two main settings: (1) time span [t0, t 1 ], and (2) a queue of sensors to check obsolescence for. Accordingly, as future work, it is required to provide a methodology through which the inputs are automatically chosen and triggered to schedule obsolescence detection. In other words, the current hierarchy needs to be topped with a recommendation system for a more productive selection, detection, and treatment. Speaking of treatment, this layer is to be provided with its approaches of handling and solving obsolete data. The current architecture productively measures obsolescence within data and requires two other systems to provide its input and handle its output of obsolescence.

Fig. 1 :

 1 Fig. 1: Motivating scenario

Fig. 2 : 3 :

 23 Overall obsolescence detection architecture In what follows, we highlight and detail separately each module. A. Quality Assessment and Storage Fig. Data quality assessment and storage

  ) Additional Quality Computation: aggregates the m d of all individual data into averages. It is represented by m avg :≺ ac avg , an avg , rl avg , tl avg , sa avg ≻. Specifically, five main additional metrics relative to all the individuals' m d are computed; namely,: • ac avg : average of all individuals' ac • an sum : total accesses/requests to D • rl avg : average of all individuals' rl • tl avg : average of all individuals' tl • sa avg : average of all individuals' sa B. Device Clustering Fig. 4: Device clustering

  a) Characteristics Clustering of Devices: The first clustering is related to the physical characteristics of the devices themselves. Devices get grouped based on what their sensors are capable of measuring (e.g., temperature, humidity, wind) alongside their location, and accordingly coverage area. b) Query Clustering of Devices: The second clustering is according to the queries of the devices. Devices get grouped according to the intersection of queries they receive for the same physical characteristics.

Fig. 5 :

 5 Fig. 5: Obsolescence detection: Device level

Algorithm 1 :

 1 Individual Data Quality Check Inputs : di, // current individual data d i-1 , // individual data at a previous time step th 1 , th 2 , // thresholds k 1 , k 2 , k 3 , k 4 // weights Output : is obsolete // indicates whether the d i is obsolete or not Variables: m d score, m D score(di), m D list, x // Variables initialisation 1 m d score = 0; // score of d i quality metrics 2 m D score(di) = 0; // score of the impact of d i on D 3 m D list = [a, av, cr, ct, cn, f c]; // list of the m D metrics 4 is obsolete = F alse; /* Boolean variable that specify if data is obsolete or not */ // Calculate the popularity 5 di.m d .pl = d i .an ansum /* m d _score is calculated using weighted average */ 6 m d score = k1 × di.m d .ac + k2 × di.m d .pl + k3 × di.m d .r + k4 × di.m d .t; 7 for x in m D list do // Score of m D evolution between two consecutive individuals 8 m D score(di)+ = d i .m D .x-d i-1 .m D .x d i-1 .m D .x × 100; // Checking if scores are exceeding the thresholds 9 if m d score < th1 Or m D score(di) < th2 then 10 is obsolete = T rue; 11 return is obsolete;

Algorithm 2 : 4 is

 24 Data Item Overall Check Input : D, // data item OD, // obsolete data pool th // threshold Output : is overall obsolete// indicates whether the overall data item is obsolete or not Variables: percentage obsolete// % of obsolete data contained in D // Variables initialisation 1 percentage obsolete = 0; 2 is overall obsolete = F alse; // Calculating the percentage of obsolete data in the data item percentage obsolete = size(OD) size(D) × 100; 3 if percentage obsolete > th then /* Checking if the percentage of data obsolescence in the data item exceeds the threshold */ overall obsolete = T rue; 5 return is overall obsolete;

Fig. 6 :

 6 Fig.6: Zone and environment detection levels complementary and reasonable to the devices' neighborhood. This is accomplished via the parent zone dimension that passes every device containing data of low quality through several phases (cf. Fig.6a) as detailed in the following.a) Explicit neighbors fetching: As aforementioned, an obsolete individual data is to be checked with other related devices within the same hosting zone to verify the validity of the previously processed obsolescence check. Accordingly, the first step within the parent layer is to retrieve the explicit neighbors (i.e., interrelated devices located in the same zone) of the device D i undergoing the obsolescence check. Based on the device clustering results, devices from the same zone belonging to D i 's cluster group are fetched.b) Data pool generation: In this step, a data pool is created and filled with the data individuals of every explicit neighbor of D i . Namely, this pool represents a collection of explicit data sensed within the [t 0 , t 1 ] time-frame. In the next step, each obsolete individual data collected by D i is analyzed against the pool to approve or disapprove its obsolescence.c) Obsolescence classification: The parent zone architecture is given a queue of low quality data to check against the data pool created earlier in part b. At the device level, d i was proved to be obsolete; however, its mandatory to check if it might be comprehensive with the neighboring devices. For instance, consider cameras C2 and C3 of zone2 (cf. Fig.1). These two cameras might help complement one another to get a comprehensive view of the whole room specs. In other words, while C2 might miss some data due to a blind spot, C3 can come in handy if it is collecting data from such areas. This reiterates how it is possible for one piece of data to be obsolete at the device level, but not with respect to other devices. Accordingly, to validate if an obsolete data is obsolete at the parent zone level, the individual undergoes classification machine learning algorithms with respect to the data pool. This results in either: (1) Processing Data: data proves to be not obsolete and sending it to to check what might have caused its obsolescence at the device level, or (2) Going up the hierarchy (i.e., environment level): data proves to be obsolete and is sent for obsolescence check with respect to other zones.

Fig. 7 :

 7 Fig. 7: Obsolescence detection architecture

TABLE I :

 I Data Obsolescence Detection Comparison

	Approaches	ac	pl	rl	Data Quality Metrics tl a av cr	ct	cn	fc	CE and Dynamicity	Obsolescence Grounds Temporal Spatial Contextual
	[8]


Michèle Atié

Céline Drozd

Toinon Vigier

Daniel Siret

Method for Evaluating Fidelity of Restitution of Subjective Impressions in Immersive Virtual Reality in Daylit Interior Scenes

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Future Work: Future studies will examine all aspects of impressions across all buildings as well as effect of TMOs on the evaluation of these impressions.

Evaluation of Impressions in Real Buildings

z

  Method for Evaluating Fidelity of Restitution of Subjective Impressions in Immersive Virtual Reality in Daylit Interior Scenes Michèle Atié, Céline Drozd, Toinon Vigier, Daniel Siret Nantes Université, ENSA Nantes, Ecole Centrale Nantes, CNRS, AAU-CRENAU, UMR 1563 Nantes, France Selection of Buildings and Questionnaire Real Experiment VR Experiment Conclusions and Limitations : Findings indicate that HMD offer a high level of fidelity in restituting one aspect of subjective impressions for a single real light condition in one location. Differences in dazzling and changing light observed due to the limited contrast ratio of the screen and varying weather conditions in real location, respectively.

Acknowledgments

This project is founded by the "Agence Nationale de la Recherche-ANR-19-CE38-0010"


Cennet Oguz

Pascal Denis

Emmanuel Vincent
email: emmanuel.vincent@inria.fr

Simon Ostermann

Josef Van Genabith
email: josef.van_genabith@dfki.depascal.denis

Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization

come   L'archive ouverte pluridisciplinaire

Introduction

Deep neural networks have achieved enormous success in various language and computer vision tasks, such as multimodal understanding using video-text and image-text data [START_REF] Malmaud | What's cookin'? interpreting cooking videos using text, speech and vision[END_REF][START_REF] Alayrac | Unsupervised learning from narrated instruction videos[END_REF]Zhou et al., 2018b;[START_REF] Miech | Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[END_REF][START_REF] Miech | Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[END_REF]. However, many current systems require a large number of accurate annotations, including image-level labels, location-level labels (bounding boxes and key points), and pixellevel labels.

A specific type of video data with naturally occurring semi-aligned texts are narrated instructional videos. Such videos are available in large quantities (e.g. on YouTube) and often chosen for learning Figure 1: Examples of visual and linguistic ambiguities. Figure 1) represents the visual ambiguity related to which specific pan (in Figure 1a) is referenced with the phrase the pan because many pans occur on the stove.

Figure 2) shows the linguistic ambiguity with the use of the pronoun them (in Figure 2b).

joint text-video embeddings in multimodal understanding (Zhou et al., 2018b;[START_REF] Miech | Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[END_REF]. They often contain a narration explaining the visual content of the corresponding time frames in the video [START_REF] Malmaud | What's cookin'? interpreting cooking videos using text, speech and vision[END_REF][START_REF] Alayrac | Unsupervised learning from narrated instruction videos[END_REF]Zhou et al., 2018b;[START_REF] Miech | Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[END_REF][START_REF] Miech | Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[END_REF]. Instructional videos often contain visual and linguistic ambiguities that can be easily resolved by humans but pose two unique key challenges for automatic text-image processing systems. The first challenge is visual ambiguity, occurring when it is necessary to ground a referring expression in an image with ambiguous visual referents. In Figure 1 (1b), it is not clear from the picture which pan is referred to with the noun phrase the pan without a correct bounding box annotation. The second key challenge is linguistic ambiguity, instantiated for example by the use of anaphoric pronouns (Figure 1,(2b) or null pronouns (Figure 2).

In this work, we focus on modeling cases where both ambiguities are intertwined, a phenomenon we refer to as visual-linguistic ambiguity. 1 provides a motivating example: To find which pan in (1b) is denoted by the textual span the pan, we need to find its antecedent in the preceding text. The visual object localization of the textual antecedent the hot pan in (1a) then includes supplementary visual information about the correct pan: The referent is the hot pan with beef in it. In Figure 1, To find the correct location of the object referred to as them in (2b), we first need to find the textual antecedent to understand what them refers to. When identifying pierogi as antecedent, we can use the pierogi for the visual object localization of them.

An even more complex case is null pronouns (Figure 2): To do object localization and anaphora resolution, the first requirement is the detection of the null pronoun, which then needs to be resolved and located in the image. A located visual object of a null pronoun then assists in finding the textual antecedent, and a resolved null pronoun helps to apply object localization.

Our guiding idea in this work is that anaphora resolution, the task of connecting linguistic expressions such as the anaphor (i.e., the repeated reference) and its antecedent (i.e., the previous mention in the document). [START_REF] Poesio | Anaphora resolution with the ARRAU corpus[END_REF][START_REF] Fang | What does it take to bake a cake? the RecipeRef corpus and anaphora resolution in procedural text[END_REF][START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF], and object localization, the task of identifying the location of one or more objects in an image and drawing bounding boxes around their visual space [START_REF] Tompson | Efficient object localization using convolutional networks[END_REF]Zhou et al., 2016;[START_REF] Choe | Evaluating weakly supervised object localization methods right[END_REF], can jointly help to resolve visual-linguistic ambiguities. To test this we propose a multitask learning neural model for jointly resolving visual-linguistic ambiguity.

Our contributions are two-fold: First, we present a new dataset1 (Section 4), Find2Find, for the joint evaluation of anaphora resolution and object localization based on an extension of Chop&Change [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF]. Our new data set contains 500 recipes with annotated anaphora and associated object localization. Together with the new data set, we propose the new task of multimodal resolution of Visual-Linguistic Ambiguities. The task provides a unique opportunity for models to fuse text and vision information for solving anaphora resolution and object localization at the same time. Second, we present a new multitask learning system 1 for modeling the two tasks of anaphora resolution and object localization jointly, using a fusion of visual and textual data. Our experiments show that information from each of the tasks mutually benefits performance on the other task. Our idea is based on the fact that in both tasks, the goal is to extract mentions from a given text: what connects object localization and anaphora resolution is that in visual object localization, a corresponding language expression needs to be found, and in anaphora resolution, accurate spans that resolve the anaphoric relations between the anaphor and the antecedents need to be identified.

Related Work

Anaphora Resolution

Anaphora resolution [START_REF] Poesio | Anaphora resolution with the ARRAU corpus[END_REF][START_REF] Fang | What does it take to bake a cake? the RecipeRef corpus and anaphora resolution in procedural text[END_REF][START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF] is the process of resolving the relations between an anaphor (i.e., a reference expression) and its antecedent (i.e., the previous mention of the same entity). The relations between anaphor and antecedent mostly appear in two different forms: coreference and bridging. Coreference resolution (Clark and Manning, 2016a;[START_REF] Lee | End-to-end neural coreference resolution[END_REF] is the task of finding the linguistic expressions that refer to the same real-world entities in a document, whereas bridging resolution [START_REF] Yu | Multitask learning-based neural bridging reference resolution[END_REF][START_REF] Kobayashi | Constrained multi-task learning for bridging resolution[END_REF] fo-cuses on entities with an associative relation that does not express the same entity but relates to it (e.g., a car and its engine). Most previous work and datasets [START_REF] Yu | The CODI-CRAC 2022 shared task on anaphora, bridging, and discourse deixis in dialogue[END_REF] tackle coreference resolution and bridging resolution separately. An exception is [START_REF] Fang | ChEMU-ref: A corpus for modeling anaphora resolution in the chemical domain[END_REF], [START_REF] Fang | What does it take to bake a cake? the RecipeRef corpus and anaphora resolution in procedural text[END_REF][START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF], which focus on documents with rich anaphoric relations for anaphora annotation and resolution of coreference and bridging with end-to-end neural networks [START_REF] Lee | End-to-end neural coreference resolution[END_REF].

Anaphora resolution is composed of two subtasks: mention detection and antecedent selection. A typical neural-based method for anaphora resolution starts initially with neural-based mentionranking modeling (Clark and Manning, 2016a), using distributional features of entities (Clark and Manning, 2016b) with predefined mentions. [START_REF] Lee | End-to-end neural coreference resolution[END_REF] combine mention detection and antecedent selection in an end-to-end neural learning system. [START_REF] Yu | Multitask learning-based neural bridging reference resolution[END_REF] propose a multi-task learning system for coreference and bridging resolution with an end-to-end learning approach [START_REF] Lee | End-to-end neural coreference resolution[END_REF]. Here, coreference and bridging resolution models learn the mention detection with the objective of coreference/bridging resolution: if the span is resolved then it is a mention.

Various feature sets are used for anaphora resolution along with contextualized language features [START_REF] Joshi | BERT for coreference resolution: Baselines and analysis[END_REF][START_REF] Joshi | Span-BERT: Improving pre-training by representing and predicting spans[END_REF], e.g., the token length of spans (Clark and Manning, 2016b), contextdependent boundary representations with a headfinding attention mechanism over the span [START_REF] Lee | End-to-end neural coreference resolution[END_REF], distance features of the anaphor and the antecedent based on word distance (Clark and Manning, 2016a) and sentence distance [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF] where additionally visual features are used for anaphora resolution in recipes for cooking videos. Our work is similar in spirit: however, unlike [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF] we combine anaphora resolution with object localization tasks from computer vision in a joint multitask learning model to benefit both leveraging visual features of entities for anaphora resolution.

Object Localization

Object localization is the task of identifying the location of one or more objects in an image, e.g. Figure 3 (a,b,c). Object localization has been studied in computer vision for a long time with various learning methods [START_REF] Gokberk Cinbis | Multi-fold mil training for weakly supervised object localization[END_REF]Zhou et al., 2018a;[START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF]. Multiple Instance Learning (MIL). MIL [START_REF] Thomas G Dietterich | Solving the multiple instance problem with axis-parallel rectangles[END_REF]) is a learning strategy that adresses the essence of the incomplete annotation problem, where only coarse-grained labels are available for learning. MIL has been used effectively for weakly-supervised learning in several computer vision works including object tracking [START_REF] Babenko | Robust object tracking with online multiple instance learning[END_REF], object localization [START_REF] Gokberk Cinbis | Multi-fold mil training for weakly supervised object localization[END_REF], image classification [START_REF] Wu | Deep multiple instance learning for image classification and auto-annotation[END_REF]. Huang et al. ( 2018) extend MIL reference awareness for visual grounding of instructional videos. We extend MIL for object localization with anaphoric information to avoid the issue of ambiguous language references such as it, them (Zhou et al., 2018a;[START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF].

Weakly Supervised Object Localization The most common way to do object localization is supervised learning, which uses object-level categories with bounding boxes, e.g., Figure 3 a. However, for a data-greedy neural learning system, object-level bounding box annotation is timeconsuming and expensive. Weakly supervised object localization approaches avoid this problem and focus on learning object localization with imagelevel object labels (e.g., Figure 3 b) under the MIL paradigm [START_REF] Deselaers | Weakly supervised localization and learning with generic knowledge[END_REF][START_REF] Prest | Learning object class detectors from weakly annotated video[END_REF][START_REF] Gokberk Cinbis | Multi-fold mil training for weakly supervised object localization[END_REF][START_REF] Oquab | Is object localization for free?-weaklysupervised learning with convolutional neural networks[END_REF].

Object localization with the MIL approach aims to match object labels with object bounding boxes (e.g., Figure 3 b). Studies on object localization with image descriptions rather than object labels take weak supervision further [START_REF] Karpathy | Deep visualsemantic alignments for generating image descriptions[END_REF]Zhou et al., 2018a;[START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF], as in Figure 3 To date, object localization studies have not explored learning from to learn of mention extraction from image descriptions, instead extracting the mentions by using parsing methods [START_REF] Kiddon | Mise en place: Unsupervised interpretation of instructional recipes[END_REF][START_REF] De-An | Unsupervised visual-linguistic reference resolution in instructional videos[END_REF][START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF] or using the predefined mentions list (Zhou et al., 2018a) before the learning process. Thus, we claim object localization and anaphora resolution share a subtask of entity extraction like the mention detection process in anaphora resolution as explained in Section 2.1

Task

A recipe consists of instructions I where a cooking instruction I i (e.g. add chopped onions and carrots to the pan) consists of n nominal or null spans and one verbal predicate where n ≥ 1. A span x i of I i might be an incorrect consecutive fragment add chopped or a gold span e, e.g., a noun phrase chopped onions, a pronoun it, or a null pronoun [ϕ] as in mix [ϕ]. Null pronouns are extremely common in recipe instructions [START_REF] Kiddon | Mise en place: Unsupervised interpretation of instructional recipes[END_REF][START_REF] De-An | Unsupervised visual-linguistic reference resolution in instructional videos[END_REF]. In our approach, we also have a video clip that contains the visual content of the instruction I i with the action and the entities included in the process. Following Zhou et al. (2018a); Huang et al. ( 2018), we evenly divide each video clip into three equal parts and randomly sample one image (one frame) V i from each of the three sub-clips to capture the temporal changes of entities.

Anaphora Resolution

The task of anaphora resolution is to assign each gold span (anaphor) e i where e i ∈ {x i,1 , . . . , x i,1 } of each instruction I i to one or more gold spans (antecedent) y i ∈ {ϵ, I 1 , . . . , I i-1 , e 1,1 , . . . , e i-1,n }, a dummy antecedent ϵ, all preceding instructions I 1 , . . . , I i-1 and all preceding gold spans e 1,1 , . . . , e i-1,n from the previous instructions I 1 , . . . , I i-1 . For a nominal span in Figure 1 1b, the anaphor span the pan refers to the antecedent a hot pan in a previous instruction. For a null pronouns example in Figure 2, the null pronouns ϕ refers to two previous instructions as the antecedents because the null pronoun does not point to any entity and it is also not a new entity for the recipe, it is instead produced by the previous instructions.

The selection of dummy ϵ as antecedent indicates that the anaphor is an incorrect sequence of consecutive words or a singular entity without an antecedent.

Object Localization

An image V ∈ R H×W is represented as a bag of ten regions (e.g., the visualization of region proposals of the given positive frame in Figure 4), v h×w 1 , .., v h×w 10 with suitable side lengths of window patches h and w where h < H and w < W . Given a span x i and ten region proposals v h×w 1 , .., v h×w 10 , the task of object localization is to identify whether or not the region proposal belongs to the object of interest, i.e., the text x i when x i is a gold span (e.g., null pronoun [ϕ] or nominal spans) of the corresponding instruction I i . Therefore, the task is to link the text span x i to a corresponding visual region proposal v i where x i is a gold span {e i,1 , . . . , e i,n } of the instruction. We use the dot-product attention between the span x i and the region v i for ranking the visual-semantic matching. 2022) use the YouCookII dataset to propose a multimodal anaphora resolution dataset, Chop&Change, with a novel annotation schema to address the state change of entities in cooking instructions of recipe videos. While the Chop&Change annotation schema captures three anaphoric relations (coreference, near-identity, and bridging), in our work here we concentrate on anaphora resolution ignoring relations and focusing on finding the antecedent. Table 1 shows that the Chop&Change dataset includes 264 training recipe documents and 89 test documents in total. For our work here, we increase the number of annotated recipes to 400 for train and 100 for test recipes by using the Chop&Change annotation schema. All annotated recipes are associated with respective videos. The structure of annotation is explained with an example in Appendix A.

Object Localization Data. To construct our test set we examine Huang et al. ( 2018) who present a study on reference-aware visual grounding and provide an object localization dataset, FindIt, of 62 YouCookII videos for the given entities in the pro-vided textual descriptions (i.e., instructions). After a deep examination of the FindIt dataset, we find that 30 videos of FindIt could be used with our annotated recipes. Since only 30 videos produce an insufficient test set for the evaluation of object localization, we annotated 70 more videos from the recipes we annotated for anaphora resolution. We extract the frames of the instruction clips by using the temporal boundaries. We obtain 3 video subsets of consecutive frames of the instruction to acquire the state changes of entities in time. We pick one frame with a clear visual content of each entity from the 3 subsets. Thus, each entity is represented in a maximum of three frames for state changes. For evaluation, we annotate the visual object with a bounding box on the selected frames for each entity. In total, we have 5,688 images for 100 recipes annotated for our object localization test set. Note that we do not annotate bounding boxes for the training data as we train our models in a weakly supervised setting.

Methodology

Model

In this section, we explain the details of our anaphora resolution and object localization models. Additionally, we formulate the multitask learning approach of joint anaphora resolution and object localization (see Figure 4). In order to analyze a given text, it is important to resolve referring expressions. Thus, text-based anaphora resolution is an important method of identifying the antecedent of anaphora. However, here we also have object localization, providing a bounding box for the visual content of language references. The tasks of object localization and anaphora resolution share information via the mention extraction and representation part of the model. Therefore, our mention representations are trained with anaphora resolution and object localization over the mention extraction and representation. The implementation details can be seen in Appendix B.

As implied in the explanation of both tasks, before resolving references in the previous context or on the image, referring expressions and language references in the text need to be identified, a task known as Mention Extraction. In the next Section, we first describe our approach to representing mentions.

Mention Extraction and Representation

We consider all continuous token sequences with up to L words as a potential mention span and compute the corresponding span score. We use SpanBERT [START_REF] Joshi | Span-BERT: Improving pre-training by representing and predicting spans[END_REF] as a state-of-art representation for coreference resolution. We capture the linguistic dependencies between anaphor and antecedent in a recipe document by exploiting selfattention: we define SPANBERT(w 1 , . . . , w T ) to be the SpanBERT representation of a recipe, where w 1 is the first token and w T refers to the last token of the recipe. SpanBERT captures the long-range dependencies between the antecedent and anaphor in the recipes. A span x i consists of zero or more tokens of instruction I i . We use the verb as a pointer for null pronouns. For example, mix is the token of the null pronoun ϕ in the instruction mix [ϕ] (Figure 2). The vector representation g i of a given span x i is obtained by concatenating the contextualized SpanBERT word vectors of its boundary tokens and its width feature:

g i = [x * START(i) , x * END(i) , ϕ(i)] ϕ(i) = WIDTH(END(i) -START(i)).
START(i) and END(i) represent the starting and ending token indexes for g i , respectively. ϕ(i) is the width feature of the span where WIDTH(.) is the embedding function of the predefined bins of [1,2,3,4,8,16] as defined by Clark and Manning (2016b).

Weakly Supervised Object Localization

Following prior work [START_REF] Karpathy | Deep visualsemantic alignments for generating image descriptions[END_REF][START_REF] De-An | Unsupervised visual-linguistic reference resolution in instructional videos[END_REF][START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF], we observe that sentence descriptions of pictures make frequent references to objects in the pictures and their attributes. However, the references are not always clearly defined. Particularly in a video, the use of pronouns and ellipses are extremely common [START_REF] Kiddon | Mise en place: Unsupervised interpretation of instructional recipes[END_REF][START_REF] De-An | Unsupervised visual-linguistic reference resolution in instructional videos[END_REF]. Our object localization model follows a Weakly Supervised Object Localization (WSOL) strategy [START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF]Zhou et al., 2018a;[START_REF] Choe | Evaluating weakly supervised object localization methods right[END_REF]: Only full image descriptions (Figure 3 (c)) are used for localization instead of a specific object-level label (as in bounding box and specific label pairs for each object in a picture like in Figure 3 (a)).

Following the object region and text ranking approach, we formulate the task of WSOL as mapping the frame region v i to the text span x i . We use positive and negative frames where positive frames come from the video clip V i of instruction I i whereas negative frames are drawn from other videos without shared entities. We define pos i to be a region vector set that includes the region proposals v h×w 1 , .., v h×w 10 from the positive image V i , and neg i to contain the region proposals v h×w 1 , .., v h×w 10 from the negative image, e.g., the negative and positive frames in Figure 4.

The aim of WSOL is to produce a scoring function to maximize the joint probability of positive frame regions v i ∈ pos i and minimize the joint probability of negative regions r i ∈ neg i with the text span x i . We concatenate the mention representation vector g i of span x i and the visual object region vector r i of the region v i to obtain the WSOL input, [g i , r i ], effectively fusing textual and visual information in one input. We then prepare positive (FFNN(g i , r i ) = 1) and negative (FFNN(g i , r i ) = 0) samples to train a model with attention-based deep MIL [START_REF] Ilse | Attention-based deep multiple instance learning[END_REF]. Positive examples depict exactly the action in question, whereas negative examples correspond to one of four special cases as described below:

FFNN(gi, ri) =          0 xi = ϵ, ∀ri 0 xi / ∈ {ei,1, . . . , ei,n}, ∀ri 0 xi ∈ {ei,1, . . . , ei,n}, ri ∈ neg i 1 xi ∈ {ei,1, . . . , ei,n}, ri ∈ pos i
Our localization model uses the span representation g i that is extracted by the mention extraction and ten positive, i.e., pos i , and ten negatives, i.e., neg i , region representation vectors r i to learn the best region from pos i for the given span g i . Thus, our mention detection model learns the span vector g i also based on the object localization objective (see also Figure 4). We define the label of FFNN(g i , r i ) as 1 when the span x i is a gold span {e i,1 , . . . , e i,n } and the region vector r i represent the positive regions pos i .

Anaphora Resolution

Following [START_REF] Lee | End-to-end neural coreference resolution[END_REF] and [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF], we implement our anaphora resolution system as an end-to-end system with mention detection but now extended by object localization. For anaphora resolution, the representation of a span pair g ij is obtained by concatenating the two span embeddings [g i , g j ] and their element-wise multiplication, g i • g j , among others:

g ij = [g i , g j , g i • g j , ϕ dist (i, j)]
where the feature vector ϕ dist (i, j) is the distance DISTANCE(START(j) -START(i)) between the index of the instruction span i and span j. DIS-TANCE(•) is an embedding function of the predefined bins of [1, 2, 3.., 30] as in [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF]. We use softmax(FFNN(g ij )) to score the resolution for anaphor g i and antecedent g j pairs.

Evaluation

Following [START_REF] Hou | Unrestricted bridging resolution[END_REF] and [START_REF] Yu | Multitask learning-based neural bridging reference resolution[END_REF], we assess the performance of our end-toend anaphora resolution with the F1-score where precision is the result of dividing the number of correctly predicted pairs by the total number of predicted pairs and recall is computed by dividing the number of correctly predicted pairs by the total number of gold pairs.

To evaluate object localization, we follow prior work [START_REF] Fukui | Multimodal compact bilinear pooling for visual question answering and visual grounding[END_REF][START_REF] Rohrbach | Grounding of textual phrases in images by reconstruction[END_REF][START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF] and compute accuracy as the ratio of phrases for which the predicted bounding box overlaps with the ground-truth by more than 0.5 Intersection-over-Union (IoU).

6 Experimental Setup 6.1 Input 6.1.1 Cooking Instructions.

To encode the recipes we use spanBERT [START_REF] Joshi | Span-BERT: Improving pre-training by representing and predicting spans[END_REF], a transformer model designed to better represent and predict spans of text. We use the concatenation of the boundary tokens to represent each span (Clark and Manning, 2016a,b;[START_REF] Lee | End-to-end neural coreference resolution[END_REF][START_REF] Kobayashi | Constrained multi-task learning for bridging resolution[END_REF].

Frame Regions of Instructions

Image regions can be proposed by either off-theshelf object localizers, e.g., region proposal networks [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], or dense sliding windows (e.g., random regions). Region proposal networks and dense sliding window methods neglect language semantics. Therefore, we leverage the state-of-art method RegionCLIP [START_REF] Zhong | Regionclip: Region-based language-image pretraining[END_REF] which uses a CLIP model [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] to match image regions with template captions to align these region-text pairs in the feature space. We select the first 20 region proposals of each entity with the highest objectness scores.

Experiments

Anaphora Resolution

Candidate and Gold Spans. Without any pruning, we consider all continuous token sequences (Clark and Manning, 2016b;[START_REF] Lee | End-to-end neural coreference resolution[END_REF] as potential spans for anaphor/antecedent candidates for the training and testing phases. Additionally, we consider gold spans for the training and testing phases in order to investigate the performance of anaphora resolution models without mention detection noise.

With and without Object Localization. To understand the effect of object localization on anaphora resolution we examine the anaphora resolution results with and without object localization. Here, we remove the object localization model and data from the training and testing phases of our anaphora resolution model.

Object Localization

With and without Anaphora Resolution. We investigate the object localization performance with and without anaphora resolution to understand the impact of anaphora resolution. We perform four different experiments for object localization. [START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF]: Since we use the object region proposal with the highest objectness scores with the given ten region proposals, we first examine random selection with the highest objectness score to show the complexity of object localization.

Random

Deep Visual-Semantic Alignment (DVSA) [START_REF] Karpathy | Deep visualsemantic alignments for generating image descriptions[END_REF]: This weakly supervised visual grounding method is based on image-based regions and given gold and candidate mentions without anaphoric information, which uses multiple-instance learning. Thus, we consider DVSA as a baseline for weakly supervised object localization to our method without anaphora resolution. We examine the results of DVSA with gold and candidate mentions.

AR-Cand. Mentions

We train and test the object localization model with all continuous tokens (Clark and Manning, 2016b;[START_REF] Lee | End-to-end neural coreference resolution[END_REF] as potential spans for anaphor/antecedent candidates. Candidate spans also increase the noise in the object label set because we consider all possible spans as labels of the objects.

AR-Gold Mentions

We show the results of object localization when we use gold mentions and anaphor information for training and testing.

7 Results and Discussion

Overview

We investigate the anaphora resolution and object localization results of gold and candidate spans comparing the F1-scores and Top-1 scores with multitask learning using both tasks and single tasks. Overall, our results in Table 3 demonstrate that replacing single-task learning with our multitask joint learning approach improves anaphora reso-lution and object localization for both candidate and gold spans. The difference between the results of candidate and gold spans demonstrates that the mention extraction model propagates errors to anaphora resolution and object localization (the sequential structure of model see Figure 4). For example, we have the candidate spans such as all n-grams words (bigrams words such as cook the, the bacon, bacon fat) of an instruction cook the bacon fat with the same visual features with bacon and fat in a pan. Thus, an incorrect mention of mention detection directly causes an error in object localization and anaphora resolution.

Weakly Supervised Object Localization

Overall, we observe an improvement in the performance of object localization with multitask learning shown in Table 2. Table 2 shows that our joint multitask learning method outperforms DVSA even with candidate mention. Note that DVSA is not the component that is responsible for extracting the mentions for object localization. Thus, we do not analyze object localization for null pronouns and candidate mentions because DVSA ranks the language expression and region pairs and does not apply mention extraction. However, we attached the DVSA method for ranking the gold nominal mentions with the given region proposals.

The results of object localization with null pronouns clearly demonstrate the benefits of anaphora resolution for object localization. For example, our model localizes the singular mention a jar in Figure 5 (a), and the anaphor another piece of bread in Figure 5 (b) better than other methods. Thanks to the mention detection and representation of our multitask learning approach, our object localization model is capable of localization of null pronouns, e.g., Figure 5 (c,d) as our model learns to represent the null pronouns in the mention detection process as a mention. Thus, the results of null pronouns clearly evidence the contribution of anaphora resolution for object localization.

Anaphora Resolution

Multitask learning of anaphora resolution and object localization increases the performance of anaphora resolution. Table 3 shows > 2% improvements for anaphora resolution with gold mentions and more than 1% for candidate mentions with combined training and testing. When nominal and zero anaphora are investigated separately, the results of nominal anaphora demonstrate an improvement on the results of gold mentions when object localization is included. Additionally, the results of zero anaphora show significant improvements with multitask learning for gold and candidate mentions. Thus, we observe a big part of the improvement for the combined experiments with object localization comes from the zero anaphora resolution samples.

The benefit of object localization on anaphora resolution is also seen in candidate nominal mentions; it is however not as significant as in gold nominal mentions, as the errors of mention detection for candidate spans directly cause localization as well as anaphora resolution errors.

The most common error in anaphora resolution is to find the closest antecedent in the entity chain. For example, for tomato-→ it -→ tomato, the first tomato is the antecedent of it, and it is the antecedent of the second tomato. However, the model fails to select the first tomato as the antecedent for the second one.

Conclusion and Future Work

In this work, we study the problem of visuallinguistic ambiguity in multimodal data and propose the novel task of joint anaphora resolution and object localization. We create the Find2Find dataset for object localization and anaphora resolution with cooking recipes to improve the performance of both tasks by exploiting their commonalities. We implement a model for the joint learning of anaphora resolution and object localization, fusing visual and textual information, and show empirically that a multitask learning paradigm mutually improves both tasks. Especially, the results of zero anaphora resolution indicate that object localization helps to avoid linguistic ambiguity of null pronouns. Our joint multitask learning approach does not apply the temporal features of evolving visual entities even though the temporal features of textual recipes [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF] are included due to instruction order. In future work, we claim the temporal encoding of visual objects can improve the results of joint anaphora resolution and object localization. shape the mixture into small patty fry patties in a pan with some oil 1.

2.

3.

4.

shred the potatoes and sweet potatoes to a bowl add garlic salt , black pepper and chilli powder 5.

6.

7.

8.

9.

10.

cook the onions in it

A Anaphora Annotation

Anaphora resolution is a challenging task for instructional languages because of temporally evolving entities [START_REF] Oguz | Chop and change: Anaphora resolution in instructional cooking videos[END_REF]. For example, Figure 6 demonstrates how the onions of the sixth instruction step refers to the onions of the fourth instruction and the onions of the fourth instruction refers to the onions of the third instruction: the onions -→ the onion -→ the onions. Most of the errors of anaphora resolution for recipes occur when the model predicts the onions of the third instruction as the antecedent of the anaphor the onions of the sixth instruction. Additionally, Figure 6 shows the null pronouns in the eighth instruction and the antecedents such as the fifth, sixth, and seventh instructions.

B Implementation Details

We use Adam (Kingma and Ba, 2014) for optimization and a learning rate of 0.001. We clip gradients element-wise at 5 and use 0.3 dropouts for regularization. We use Negative log-likelihood as a loss function for the both task of anaphora resolution and object localization.

  add red wine to the pan add the beef into a hot pan boil pierogi in boiled water fry them with onion and butter

Figure

  add butter and garlic to the pan ...

Figure 2 :

 2 Figure 2: An example to display how visual-linguistic ambiguity occurs with a zero anaphor. The zero anaphor [ϕ] refers to two previous instructions as shown. The entities are aligned to the object with the arrows and the color codes.

Figure 3 :

 3 Figure 3: Different examples of object localization. The dashed lines in (b) and (c) represent the occurrence of annotation for only the test data whereas the straight lines (a) indicate that annotation is present for train and test.

Figure 4 :

 4 Figure 4: The architecture of the multitask learning framework of anaphora resolution and object localization.

Figure 5 :

 5 Figure 5: The examples of object localization results of random with the green bounding box, DVSA with the blue bounding box, and our multitask learning method with the red bounding box. The task is the object localization of the gold mention in bold font in the instructions.

Figure 6 :

 6 Figure 6: An example of the annotation of anaphora resolution for cooking instructions. The grey boxes represent the singular mentions without any anaphoric relations. The beginning of the arrows shows the anaphora whereas the ends point to the antecedent.

  c. For example,[START_REF] De-An | Finding "it": Weakly-supervised, reference-aware visual grounding in instructional videos[END_REF] propose to extend object localization to localization based on context-dependent referring expressions, and[START_REF] Kuo | Findit: Generalized localization with natural language queries[END_REF] offers a general-purpose model for object localization by using a wide range of referring expressions, localization or detection queries for zero, one, or multiple objects. Another challenge for object localization with image descriptions is the automatic extraction of object labels from image descriptions. Zhou et al. (2018a) extract object labels manually for training and testing object localization models whereas Huang et al.

(2018)

apply the pre-trained Stanford CoreNLP parser

[START_REF] Christopher D Manning | The stanford corenlp natural language processing toolkit[END_REF]

for entity detection.

Table 1 :

 1 Annotated Data Statistics

		Train	Test
	Entity	9,316 2,842
	Null Pronoun 1,002	282
	Pronoun	314	129
	Noun Phrases 8,000 2,431
	Instruction 4,633 1,422
	Recipe	400	100

Table 2 :

 2 The Top-1 results of object localization with gold and candidate mentions. The column group named Nominal shows the results of object localization for nominal phrases, Null depicts the results of null pronouns, and All refers to the full dataset.

Table 3 :

 3 Results of the anaphora resolution with and without object localization for gold and candidate mentions. We show the results for the full test datasets in Anaphora Res. columns, for only null pronouns in Zero Anaphora Res., and the resolution results of anaphoric mentions for all nominal phrases in Nominal Anaphora Res. part.

  on Computer Vision and Pattern Recognition, pages 16793-16803. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Luowei Zhou, Nathan Louis, and Jason J. Corso. 2018a. Weakly-supervised video object grounding from text by loss weighting and object interaction. In BMVC. Luowei Zhou, Chenliang Xu, and Jason J. Corso. 2018b. Towards automatic learning of procedures from web instructional videos. In AAAI. Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019. Cross-task weakly supervised learning from instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3537-3545.

https://github.com/OguzCennet/odar

Acknowledgements

We would like to thank Gokul Srinivasagan and David Meier for helping with the annotation of the object localization, Alina Leippert for helping with the annotation of the anaphora resolution. This research was funded by the joint IMPRESS (01|S20076) project between the French National Institute for Research in Digital Science and Technology (Inria) and the German Research Center for Artificial Intelligence (DFKI).

Limitations

The first and most important limitation is the pre-trained region proposal network (RegionCLIP [START_REF] Zhong | Regionclip: Region-based language-image pretraining[END_REF]). Our object localization is highly dependent on the quality of the region proposals. Therefore, a better region proposal network delivers more improvements for our multitask learning.


The Design and Implementation of an Abstract Interpreter for OCaml Programs: A Preliminary Report on the Salto Analyser BENOÎT MONTAGU, Inria, France

We report on a work in progress that aims at defining an effective static analyser for OCaml programs, by leveraging abstract interpretation techniques. The goal of the Salto static analyser is to detect precisely which exceptions an OCaml program might raise, and to report problematic cases, where a program execution might rely on elements of the OCaml semantics that are deemed under-specified or undefined. The Salto analyser exploits a novel abstract domain to represent inductively defined sets of trees, that draws inspiration from the theory of recursive types, from tree automata, and from the abstract domain of Type Graphs. The analyser itself is defined using a dynamic fixpoint solver, i.e., a generic library that implements an iteration strategy that finds a post-fixpoint. The solver automatically inserts widening points to ensure the convergence of the iteration process, and aims at limiting the unnecessary computations that may be asked by the iteration strategy.

ACM Reference Format: Benoît Montagu. 2023. The Design and Implementation of an Abstract Interpreter for OCaml Programs: A Preliminary Report on the Salto Analyser. In . ACM, New York, NY, USA, 4 pages.

WELL TYPED PROGRAMS CANNOT GO WRONG

As a member of the ML family, the OCaml programming language [START_REF] Leroy | The OCaml System[END_REF]] benefits from the increased safety that is induced by strong typing [START_REF] Milner | A Theory of Type Polymorphism in Programming[END_REF]]: "well typed programs cannot go wrong". This means that values are always used in a consistent manner by well typed programs, so that abrupt crashes, e.g., memory-related errors such as segmentation faults, are ruled out by the type safety theorem.

The ML type system does not, however, protect against other kind of programming mistakes, such as forgetting to catch some exceptions. Extensions of ML based on type and effect systems have been developed [START_REF] Leroy | Type-Based Analysis of Uncaught Exceptions[END_REF] to detect such programming mistakes.

Uncaught exception in OCaml programs may occur for a variety of reasons, such as:

• Explicit raises of exceptions,

• Non-exhaustive pattern matching,

• Out of bound array accesses (and similarly for strings and packed byte arrays),

• Divisions by zero,

• Assertions explicitly written by a programmer,

• Initialisations of recursive modules,

• Polymorphic equality tests and comparisons that involve functions.

Other parts of the OCaml language have deliberately under-specified or unspecified behaviours, and would benefit from early detection. This includes:

• Arithmetic overflows and underflows,

• Lazy patterns, whose evaluation may produce side effects,

• Conditions in patterns (when clauses), that may produce side effects, Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

ML Family Workshop, 2023, Seattle

© 2023 Copyright held by the owner/author(s).

• Calls to unsafe functions (unchecked array accesses, unmarshalling, etc.).

In this talk, we present an early prototype of a static analyser for OCaml programs, that is based on the theory of abstract interpretation. The Salto analyser1 is a whole program analyser that infers, for each sub-expression of a program that might be reachable, which values it may reduce to, and which exceptions its evaluation may raise.

The exceptions that result from memory or stack exhaustion, as well as the recently introduced features of OCaml 5.0-e.g., parallelism, algebraic effects-are out of the scope of this project for the moment.

A VALUE ANALYSIS FOR HIGHER-ORDER PROGRAMS

It is common knowledge that the control flow of higher-order programs cannot always be statically known, and that it makes the static analysis of such programs difficult. The dynamic nature of the control-flow graph is even exacerbated by the fact that exceptions are first-class values in OCaml (they can, for instance, be stored in data structures, or be passed as arguments to functions).

Many earlier work have been devoted to control flow analysis (CFA) [START_REF] Neil | Flow Analysis of Lambda Expressions (Preliminary Version)[END_REF][START_REF] Shivers | The Semantics of Scheme Control-Flow Analysis[END_REF][START_REF] Wright | Polymorphic Splitting: An Effective Polyvariant Flow Analysis[END_REF]] for the λ-calculus. The Salto analyser is based on recent work [START_REF] Montagu | Trace-Based Control-Flow Analysis[END_REF]] that defines a CFA as a value analysis, by abstracting the control flow traces that are produced by program executions. These control flow traces record which calls are performed, and which results are returned, and keep track of the contexts in which such control flow events are produced. For every reachable program point, the analyser infers an abstract value, that denotes an inductively-defined set of values that may be produced by the sub-expression at this program point.

The Salto analyser employs a novel abstract domain, whose salient feature is the presence of a fixpoint constructor 𝜇𝛼 .𝑎 to represent inductive sets of values. This idea is drawn from the theory of equi-recursive types [Pierce 2002, Chapter 4], and can be understood as a term-based representation of the Type Graphs that were used to analyse Prolog programs [P. [START_REF] Van Hentenryck | Type analysis of Prolog using type graphs[END_REF][START_REF] Van Hentenryck | Type analysis of Prolog using type graphs[END_REF].

Similarly to the methodology described in the "Abstracting Definitional Interpreters" article [START_REF] Darais | Abstracting definitional interpreters (functional pearl)[END_REF], the Salto analyser is defined as a function in open recursive style-i.e., a functional of type (𝜏 1 → 𝜏 2 ) → (𝜏 1 → 𝜏 2 )-that is passed as argument to a dynamic fixpoint solver, and produces a function of type 𝜏 1 → 𝜏 2 that computes on demand a solution of type 𝜏 2 when it is provided with an argument of type 𝜏 1 . The solver we have implemented is inspired by existing solvers from the literature [START_REF] Baudouin | A Universal Top-Down Fixpoint Algorithm[END_REF][START_REF] Schulze Frielinghaus | Enforcing Termination of Interprocedural Analysis[END_REF][START_REF] Seidl | Three Improvements to the Top-Down Solver[END_REF], that can handle abstract domains of unbounded heights, by automatically inserting calls to a widening operator. Widening operators are an essential ingredient of abstract interpretation, that are responsible for finding valuable generalisations of invariants, and ensure the convergence of the iteration process. A similar approach based on dynamic fixpoint solvers has been chosen to implement the Goblint static analyser for C programs [START_REF] Vojdani | Static Race Detection for Device Drivers: The Goblint Approach[END_REF]].

CURRENT STATE OF THE IMPLEMENTATION AND FUTURE CHALLENGES

The prototype implementation of the Salto analyser currently supports the purely functional subset of the OCaml core language only. This includes higher-order functions, recursion, algebraic datatypes, and exceptions. The Salto prototype has not been released yet. So far, it is able to analyse programs of the size of a few hundreds lines of code.

The analyser takes as input the typed abstract syntax tree (AST) of OCaml programs that is produced by the typing phase of the compiler, and transforms it into a simplified AST, that has less constructs, so as to alleviate the implementation effort of the actual analysis. For example, a single construct is responsible for pattern matching in this simplified AST.

The analysis is designed to handle untyped programs, and is so far independent from the typing discipline enforced by the OCaml typechecker. This leaves open the future possibility of exploiting type information during the analysis to infer more precise results, or to guide the creations or merges of disjuncts in abstract values.

Our short-to mid-term goals is to expand the set of features supported by the analyser, such as:

• Mutable state,

• Laziness,

• Precise support for strings, floats, finite sets and maps,

• Modules, functors, and first-class modules.

On the longer term, we would also like to support:

• Infinite (coinductive) data-structures,

• Recursive modules and their initialisations,

• Objects and classes,

• System-related features, such as primitives for file-system operations, process management, and signals (System and Unix modules), • OCaml 5.0 specific features, e.g., algebraic effects, parallelism, • Features related to the implementation of the OCaml runtime, e.g., that depend on how data are represented in memory, or that are related to the garbage collector. On a broader scope, we are also interested in exploring the following research questions:

• Can we detect when the behaviour of a program is sensitive to the order of evaluation of function arguments, of tuple members, etc.? • Can we detect when the behaviour of a program is sensitive to the initialisation order of its compilation units? • Can more modular analyses be developed for OCaml programs, that would produce precise function summaries, such as relational analyses [START_REF] Oana | Inferring Frame Conditions with Static Correlation Analysis[END_REF][START_REF] Montagu | Stable Relations and Abstract Interpretation of Higher-Order Programs[END_REF] that infer relations between inputs and outputs of programs?

https://salto.gitlabpages.inria.fr/

ACKNOWLEDGMENTS

This work is funded by a two-year grant from Nomadic Labs and Inria.

We would like to thank Pierre Lermusiaux, Thomas Genet and Thomas Jensen, from Inria and Université de Rennes, for their great contributions to the theory and implementation of the Salto analyser, as well as Mehdi Bouaziz and Vincent Botbol, from Nomadic Labs, for their essential support and feedback.