Mohammed Guermal
email: mohammed.guermal@inria.fr

Rui Dai
email: rui.dai@inria.fr

Franc ¸ois Brémond
email: francois.bremond@inria.fr

THORN: Temporal Human-Object Relation Network for Action Recognition

Most action recognition models treat human activities as unitary events. However, human activities often follow a certain hierarchy. In fact, many human activities are compositional. Also, these actions are mostly human-object interactions.

In this paper we propose to recognize human action by leveraging the set of interactions that define an action. In this work, we present an end-to-end network: THORN, that can leverage important human-object and object-object interactions to predict actions. This model is built on top of a 3D backbone network. The key components of our model are: 1) An object representation filter for modeling object. 2) An object relation reasoning module to capture object relations. 3) A classification layer to predict the action labels. To show the robustness of THORN, we evaluate it on EPIC-Kitchen55 and EGTEA Gaze+, two of the largest and most challenging first-person and human-object interaction datasets. THORN achieves state-of-the-art performance on both datasets.

I. INTRODUCTION

Human activity recognition in video is a fundamental problem in computer vision, due to its large field of applications, such as human-computer interaction [START_REF] Jiang | Action recognition scheme based on skeleton representation with ds-lstm network[END_REF] or video surveillance [START_REF] Nguyen | Dual-layer kernel extreme learning machine for action recognition[END_REF]. Machine learning and computer vision models have achieved interesting results in this field. Unfortunately, most of the State-of-the-art methods focus on simple activities such as walking or drinking, while the recognition of longeterm, complex, and composite activities such as assembling furniture or food preparation has been rarely addressed. These methods make use of end-to-end models that produce a video level label, and do not explicitly decompose the action into a hierarchical set of sub-actions or interactions. Moreover, neuroscience [START_REF] Barker | One boy's day; a specimen record of behavior[END_REF], [START_REF]Midwest and its children: The psychological ecology of an american town[END_REF] has shown that the human perception of action is actually based on decomposing an action into different groups of interactions which enables him/her to understand other human behaviors. In this paper we decide to visit this composite actions, that we refer to as actions of Human-Object Interaction (HOI). Not only that we also focus on first-person view HOI action recognition.

first-person action recognition also comes with its challenges, one of which is the narrow field of view that makes actions sometimes happen outside the video viewing range. Also, the huge ego-motion caused by the sharp movements of the camera can make it harder to recognize actions. Finally, in ego-vision, the field of view usually covers the human hands and an ensemble of objects. In this case, actions are generally involving interactions between the human and objects. Hence the challenge is also to recognize which of these objects are relevant to the action and which are distractors. In practice, the model captures all the objects detected. For simplicity, here we highlight only the relevant objects to wash plate.

A HOI action can be seen as combination of verbs and nouns, for instance the action cutting bread with knife is the combination of the verb cut and the nouns knife and bread. Hence recognizing an action of HOI, is a class of visual relationship detection, where the task is to not only recognize the objects (the noun), but also to infer the relation and motion (the verb) between different objects and the human. Fig. 1 represents an example of an object-based action: wash plate. Such action requires highlighting objects like the hand, the plate and the tap while giving less attention to other objects that are not important to the action.

Previous works such as two-stream CNNs [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF], [START_REF] Feichtenhofer | Spatiotemporal multiplier networks for video action recognition[END_REF], [START_REF] Feichtenhofer | Convolutional two-stream network fusion for video action recognition[END_REF] or 3D CNNs [START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF] [9] [START_REF] Wang | Non-local neural networks in[END_REF] have achieved very good results on third view and video level label datasets [11] [12] [13] [START_REF] Soomro | Ucf101: A dataset of 101 human actions classes from videos in the wild[END_REF]. However, when it comes to HOI actions they still lack in performance. That is due mainly to the fact that CNNs capture shareable local features in the image/videos, and they can not handle complex or fine-grained actions. Another major challenge is the fact that such activities can often be performed in a wide variety, making it harder for CNNs to learn significant patterns.

Thus, our intuition is to build a model that can, extract detailed and object specific semantics in the videos, as well as explore the cross-object relation at different time-steps. By doing so we can firstly, improve object recognition in actions of HOI (the noun). Moreover, we can refine the motion recognition (the verb) by having a clearer idea about the interaction of these objects and their roles in the action. Finally, by encoding the scenes into a graph of objects interactions, we make it easier to learn patterns for actions even if they have many variations, since the interactions are usually the same.

To step-up to the aforementioned challenges, we propose a new module built on top of 3D-CNNs, this module is divided into two sub parts. Firstly, we design an Object Representation filter. This first sub-module acts as a filter that retrieves specific and object-related semantics from the overall and mixed representation (extracted from the 3D-CNN). Secondly, we add an Object Relation Reasoning module that uses the detailed representations to explore cross-objects relations (interactions). Finally, we obtain an object-centric model that can predict actions of HOI by exploring human-object and object-object interactions.

To summarize, our main contributions are: 1. A model that can find and extract detailed semantics of specific objects; 2-A graph-based module capable of exploring interactions between different objects.

II. RELATED WORK

Human-object interaction action recognition became the focus of many research subjects lately, especially with the development of important datasets such as [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF], [START_REF] Alahari | Actor and observer: Joint modeling of first and thirdperson videos[END_REF], [START_REF] Fathi | Learning to recognize daily actions using gaze[END_REF]. Several approaches have been proposed to tackle this problematic. In the following, we review some of these approaches.

A. 3D-CNNs

3D-CNNs methods focus on getting the overall appearance of the videos without considering the objects interactions. Since these methods cannot capture specific or detailed semantics, they are still limited in case of actions of HOI. Making this architectures more adequate to video level labels. We cite as an example I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. Although it achieves good results on many action recognition datasets, its performance is still poor on actions of HOI. To improve the performances on these 3D-CNNs, Long Features Bank [START_REF] Wu | Long-term feature banks for detailed video understanding[END_REF] for instance, tries to capture HOI actions by extracting and fusing features from local clips as well as globally from the whole video. This method uses object detection and ROI-Align to capture detected object features. And though they successfully capture richer features and more temporal information, they fail to do any object interaction modeling. Hence, they cannot improve much on HOI actions. In the same direction, Temporal Binding Networks (TBN) [START_REF] Kazakos | Epic-fusion: Audio-visual temporal binding for egocentric action recognition[END_REF] proposes to capture local clip features from different clips and fuse them for later prediction. In addition to that, TBN uses multi-modalities as they capture audio-visual features using audio, RGB, and optical flow. However, we believe that this multi-modality will not always bring much information about the objects. sounds can be very noisy and very similar which can confuse the prediction. Moreover, fusing multi-modalities can be hard and requires lot of efforts the may not lead to significant improvements.

Finally, other works such as [START_REF] Wang | Self-supervising action recognition by statistical moment and subspace descriptors[END_REF] use also multi-modality reasoning. However, we argue that HOI actions recognition requires more focus on objects and their interactions.

B. Graph's Convolutions

Recently, graphs have also been considered a way for solving action recognition [START_REF] Wang | Videos as space-time region graphs[END_REF], [START_REF] Shi | Two-stream adaptive graph convolutional networks for skeleton-based action recognition[END_REF], [START_REF] Yan | Spatial temporal graph convolutional networks for skeleton-based action recognition[END_REF], [START_REF] Dai | CTRN: Class Temporal Relational Network For Action Detection[END_REF].

As for human-object interaction, videos as a space-time region graph [START_REF] Wang | Videos as space-time region graphs[END_REF] propose to model the interaction between objects and humans in two steps as they build two different graphs. This allows to correlate objects across space-time. Similarly, in [START_REF] Ghosh | Stacked spatio-temporal graph convolutional networks for action segmentation[END_REF], the authors construct the nodes of the graph with consideration to the node class. For instance, the node for the scene is computed using the aforementioned I3D. While for objects, they use the Faster-RCNN network [START_REF] Chen | Domain adaptive faster r-cnn for object detection in the wild[END_REF] trained on MS COCO. All these methods mentioned above try to define their nodes by using ROI-Align. However, this is not optimal as,in most cases, multiple objects are present at the scene and some of them are too close to each other. In this case, the projected coordinates of different objects tend to be in the same set of pixels. Therefore, extracting an object's specific feature from a feature map with low resolution becomes difficult. Not only that these methods rely on pretrained object detetctors, hence they can not leverage only objects relevant to the action. Whereas in our work, we learn to filter only relevant objects and learn specific representation to different object-classes in an end-to-end way.

In the domain of semantic modelling, Class Temporal Relational Network (CTRN) [START_REF] Dai | CTRN: Class Temporal Relational Network For Action Detection[END_REF] is proposed for the action detection tasks. However, CTRN is a two steps method, which is built on top of pre-extracted flattened 1-dimensional features. The dissociation between the visual encoder and temporal module makes the model overlook the appearance and spatial information in the video, while such information is critical to the HOI action recognition. In this work, we propose a one-step method THORN for HOI action recognition. Different from CTRN, our method leverages the object detector to extract the object semantics directly from the spatio-temporal features. After that, graph reasoning is applied to refine the object representation and to jointly model interobject relations. This design allows the model to capture the latent relations among the objects in the videos, which results in higher accuracy in HOI action recognition.

III. PROPOSED METHOD

In this section, we detail each sub-part of the proposed model, THORN. The main components in this model are: a 3D Visual Encoder which encodes the video into a spatiotemporal embedding. Then, the previously extracted embeddings are passed to the Object Representation Filter (ORF). This filter extracts class-specific features. Finally, the Object Relation Reasoning module computes the relation between the different objects to predict the action. Fig. 2 provides an overview of the model.

A. Visual Encoder

We start by using a visual encoder to extract an embedding that serves as a full understanding of the scene, and carries the global information of the input frames. We choose X3D [START_REF] Feichtenhofer | X3d: Expanding architectures for efficient video recognition[END_REF] as our visual encoder. X3D has many advantages as it does not do any temporal pooling and keeps the full temporal information, providing richer temporal information. Moreover, X3D is a lighter model compared to other architectures such as I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. The input to the 3D encoder is a set of video-clip frames. The output is a spatio-temporal representation F of shape (T × H ′ × W ′ × D 1 ), where: H ′ = W ′ = 7, D 1 = 432, while T is the same as the input. This embedding carries both spatial and temporal information. The spatial information is important, as it provides object related information, such as its appearance, shape and position (e.g. drawers usually appear at the bottom of the image). That is why instead of using the X3D final output of shape (T × 2048) to construct our nodes, we use a finer spatial representation of shape (T × 7 × 7 × 432), making nodes of our graph contain more and finer information about the objects. We provide more details on this in the ablation study, by comparing both settings. Finally, as X3D is a light-weighted model it is easier to train the Visual Encoder jointly with the following modules.

B. Object Representation Filter

Our main objective through this work is to have objectbased reasoning. Hence the first step is to obtain object in scene representations. Therefore, we developed the Object Representation Filter module, capable of extracting semantic representation specific to each object class from the previous overall representation. This module serves as a filter to obtain the object-specific representation from the output of visual encoder. In practice, firstly, we reshape the representation F from the visual encoder to shape (T × H ′ W ′ D 1 ). After that, we duplicate the reshaped features F ′ for C o times, where C o indicates the number of object classes in the dataset. For each class, we use a channel-mixer MLP (i.e., linear transformation layer), followed by non-linear activation and dropout. In Fig. 3, we show an overview of the ORF module. We argue that each MLP layer learns to filter features specific to a certain object class. The equations in this module can be formulated as:

F ′ i = ReLU (M LP (F )) (1)
F ′ = DropOut([F ′ 1 , F ′ 2 , F ′ 3 ...., F ′ Co ]) (2)
With F ′ ∈ R T ×Co×D2 . Where D 2 is smaller than D 1 to shallow the channel size. Finally, we add another MLP layer on top of F ′ that would represent the object classifier in Fig. 2.

F ′′ = ReLU (M LP (F ′ )) (3)
Here F ′′ ∈ R T ×Co×1 . To ensure the object-specific representation, we add a frame-level object classifier on F ′′ . As the frame-level object label is not provided by the dataset, the object classifier is trained with the pseudo label provided by an object detector (i.e. Fast-RCNN [START_REF] Chen | Domain adaptive faster r-cnn for object detection in the wild[END_REF]). In the video, multiple objects can appear in a frame, thus, we train the object classifier with binary cross-entropy loss: L clip-objects . Finally the ORF module outputs a representation for each object-class. However, we still need to correlate and refine these object representations to explore their interactions and model the actions. To do so, we introduce the next module of our pipeline in the next section.

C. Object Relation Reasoning Module

To correlate between the aforementioned representations in the previous section, we introduce the Object Relation Reasoning Module. In order to extract the relations between the filtered object classes, we propose to make use of graph convolutions. In the previous section, we transform the clip representation into a class-specific representation. Then, we map it to a graphlike structure, where each vertex of the graph represents an object class at a time step; the vertex would be the previously extracted embedding of a certain class. In total, the graph consists of C o × T nodes whose topology is defined by its vertex and an adjacency matrix A ′ Co . The adjacency matrix represents the connectivity or relation between the different nodes (objects) and its weights represent how strong their relationship is at different time steps. Fig. 4 represents an overview of this module.

1) Graph reasoning: The graph reasoning aims to do cross-class reasoning on the previously constructed graph. The objects relations are video dependent, and so multiple GCN blocks are stacked to learn multiple levels of semantics. Moreover, the adjacency matrix is also parameterized so that it can be learned and optimized with the pipeline during the training phase. Moreover, it can learn to adapt to the data itself. We also make use of self-attention mechanisms. Consequently, our adjacency matrix learns better to differentiate class relations owing to different videos. Fig. 4 represents a block of the graph convolution reasoning.

As the object relations are complex, it is hard to predefine the inter-object relations for each video. Therefore, by leveraging the self-attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF], [START_REF] Shi | Two-stream adaptive graph convolutional networks for skeleton-based action recognition[END_REF], our graph adjacency matrix is learnable and can vary with the videos. In practice, the adjacency matrix A Co is initialized with a fully connected matrix. Finally, the full topology of our graph is A Co ∈ R Co×Co and the vertexes representation G in ∈ R D2×T ×Co . First, we embed the input G in using bottleneck convolutional layer (i.e. 1 × 1), then the output feature maps are rearranged into R D2×T ×Co and R Co×D2×T followed by a matrix multiplication. The value of the resultant matrix is then normalized by a softmax activation. Now, the superimposed adjacency matrix A ′ Co can be formulated as:

A ′ Co = A Co + sof tmax(W T 1 G T in W 2 G in ) (4)
Where W 1 and W 2 are learnable weights of the bottleneck convolutions, and G in being F ′ the stacked class representations in section B. G out , the output of the graph layer is passed to the next graph layer and follows the same equations. In this work, we use 5 blocks of graph convolutions. As for the A ′ Co , each value represents an edge between two nodes (objects). We learn a graph that is shared across different time-steps but depends on each layer and for each video, as we said earlier we learn different semantics at each level. After bottleneck convolutions, we do the graph convolution operation with the formulation in [START_REF] Kipf | Semi-supervised classification with graph convolutional networks[END_REF]:

G out = A ′ Co G in W 3 (5)
Fig. 4. Overview of one layer of the Object Relation Reasoning module, using a graph architecture [START_REF] Shi | Two-stream adaptive graph convolutional networks for skeleton-based action recognition[END_REF]. As we can see, the input is a graph representation between different classes and the output is an updated representation of the graph. The × N block stands for the number of blocks used in total, while the × 3 at the bottom in blue stands for the number of used multi-head attentions.

W 3 is a learnable parameter where W 3 ∈ R D2×D2 . The equation 5 represents the message passing and node feature updating, and finally G out is rearranged to R D2×T ×Co .

From equation 5, we can understand how graph convolutions work. The graph convolutional layer represents each node as an aggregate of its neighborhood, hence each node gathers information from its neighborhood and adapts itself accordingly. In other words, at each graph block, each object collects information about other objects and finally finds to which it is most correlated, and thus whether there is an interaction or not. That is why we judge that the use of graphs is a promising idea in this domain.

2) TCN: stands for Temporal Convolution Network. The graph reasoning is capable of extracting the relation between objects. However, in our study, we aim at modelling the spatiotemporal interaction in a large time span. To do so, we add a 1D convolution layer on top of the previous output of the graph reasoning (i.e., G out ). As shown in Fig. 4, each Object Relation Reasoning Module contains a TCN. This 1Dconvolution layer is used to aggregate the information across time. While stacking multiple object relation reasoning blocks, each block is used to model the object relation in a specific temporal scale. Finally, the output of the Object Relation Reasoning Module is:

G out = Conv1D(G out ) + G in (6)
As mentioned earlier, the output of each block G out is the input G in to the next block.

D. Predictions

Predictions are based on the learned nodes and adjacency matrix. However, since in our case our actions are composed of verbs and nouns, we show that using the adjacency matrix for predicting the verb and the object feature representation for noun prediction is more effective. This makes sense since the adjacency carries more information about how different objects interact with each others, while the nodes carry a refined object representations, after been processed through the different graph convolutions blocks. Our final layers are two fully-connected layers one projecting G out from R D2×Co to R 1×Co , and the other fully-connected layer projecting

A ′ Co from R Co×Co into R 1×Cv
, where C o and C v stand for the number of object classes and verb classes respectively. Since we have 3 outputs, our loss is a sum of three losses and can be formulated as :

L = L verbs + L nouns + L clip-objects (7)
Where L verbs and L nouns are the negative log-likelihood losses (since each action is composed of one verb and one noun). As described earlier, the L clip-objects is the loss to ensure the semantic of the object representation.

IV. EXPERIMENTS Dataset. We have evaluated our model on two of the largest and challenging datasets for first-view and human-object interaction action recognition. Epic-Kitchen55 [START_REF] Damen | Scaling egocentric vision: The epic-kitchens dataset[END_REF] contains 55 hours of recording of 32 different kitchens in 4 cities. This dataset has a total of 125 verbs and 352 nouns. EGTEA Gaze+ [START_REF] Li | In the eye of beholder: Joint learning of gaze and actions in first person video[END_REF] contains 28 hours of cooking activities from 86 unique sessions of 32 subjects, with over 10k video clips of 106 fine-grained egocentric activities. In both datasets, each action is a combination of a verb and a noun. Actions are relevant to different steps of preparing food (e.g. cleaning the kitchen, cutting vegetables, preparing table). Implementation. We implement our method using X3D as the visual encoder where D 1 = 432, H ′ = W ′ = 7 and D 2 is 128. We input a clip of 16 RGB frames for Epic-Kitchen and 25 frames for EGTEA Gaze+. We use a dropout probability of 0.3. For the object relation reasoning module, N Block is 5 blocks.

For the temporal convolution network, we run our model with different values of the kernel size. As there was no impact on the results, we kept a kernel size of 9. In training phase, we utilized Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] to optimize the model with an initial learning rate of 0.00005. We scaled the learning rate by a factor of 0.1 with the patience of 5 epochs. The network was trained on a 4-GPU machine for 30 epochs. We evaluated our model using top1 and top5 accuracy on verbs and nouns for Epic-Kitchen, while for EGTEA Gaze+ we evaluated directly on actions using top 1 accuracy.

A. Ablation Study

In this section, we validate our model design for the modules in the THORN. The evaluation is conducted on the EPIC-Kitchen dataset. We propose different settings, and see how each setting can improve the performance. In table I, we can notice different results:

Firstly, we compare our baseline model X3D with THORN. Note that, in THORN, the graph nodes can be constructed either using the output of the last layer of X3D (temporal nodes) or using its intermediate layer (spatio-temporal nodes). Here, we first compared X3D with THORN (temporal nodes), i.e., we construct the nodes by the features in shape T × 2048. In this setting, nodes would serve to predict both verbs and nouns. In this scenario, we improve nouns prediction by +5.6%, while, the verbs accuracy increased by +9.3% . Proving the importance of the cross-object reasoning, compared to only capturing visual information from 3D-CNNs. Secondly, we study the importance of the adjacency matrix for predicting the verbs. To do so, we use the adjacency matrix (ADJ-matrix) to predict verbs, while keeping the nodes to predict the nouns. In this setting, the verb prediction improves by +4.5% compared to the previous setting and by +13.8% to the baseline X3D. This is because the adjacency matrix captures the object interaction, hence, it is more suitable for verb prediction. Thirdly, we study the effect of changing the temporal nodes with the spatio-temporal nodes. Spatio-temporal nodes are the nodes constructed by the middle layer of X3D which contains the spatial information (T ×7×7×432). With spatio-temporal nodes, THORN improves +1.8% on nouns. This is because, with spatial dimensions, the ORF can better capture the object relative locations and the size of the object, then embed them in the node representation. As a result, the noun accuracy improves. This setting also brings +0.7% improvement on verbs. Our overall architecture obtains +13.8% more accuracy on verbs and +8.6% on nouns w.r.t. vanilla X3D. This reflects the importance of our proposed modules in THORN and how an object-centric method can improve results on human-object interaction actions.

We then study the components for predicting the nouns in our model. In table II, we show that fusing scores of object detection and the scores obtained by the THORN nodes representation works better than using only one of them. We also find that predictions using only our model are better than the object detector itself. This shows that our model can refine the objects represented by the other objects (nodes) using our graph-based module. We then compare our proposed method with the state-of-theart methods on EPIC-Kitchen and EGTEA Gaze+ in table III and IV.

In Table III, we compare our results with the state-of-the-art methods. Among these methods, Long Features Bank (LFB) [START_REF] Wu | Long-term feature banks for detailed video understanding[END_REF] proposes to use global as well as local features for action recognition. To do so, they extract features on both clip and video levels, and combine them to have a better understanding of the scene. Nevertheless, this method still lacks accuracy for the objects. Moreover, LFB is a two step method which trains separately an object and verb recognizer modules. For our THORN, we train a single model for predicting both entities. As a result, we have a +8.5% improvement on top 1 nouns and a +4.9% w.r.t. LFB on action recognition.

Our method achieves the overall best performance. We claim that AssembleNet++ utilizes additional modality such as optical flow in both training and inference time. Even though, we still have the lead in top 1 accuracy for the verbs, nouns and actions, which proves again that having an object-centric and specific reasoning on object interactions is a key solution for having a better action recognition on HOI datasets. Finally, our results prove that using only RGB with an object-centric model achieves better or similar results compared to methods relying on heavy multi-modality reasoning.

In table IV, we compare our method with the state-of-the-art on EGTEA Gaze+ dataset. We have the best accuracy w.r.t. the others methods, which shows the generalization and robustness of our model on actions of HOI.

To sum up, compared to other methods, ours is lightly weighted as we use X3D, while other methods rely on heavy 3D-CNNs such as I3D. THORN is trained jointly on nouns and verbs as opposed to other methods such as LFB [START_REF] Wu | Long-term feature banks for detailed video understanding[END_REF], and we only need RGB frames and object classes per-frame.

C. Qualitative Study

In this section, we conduct a qualitative study of THORN. In Fig. 5, we show the impact on some classes after adding our proposed module w.r.t. vanilla X3D. In EPIC-Kitchen, we significantly improve accuracy on 28 verb classes. Only the accuracy of 3 out of 125 verbs decreases, while the decrease is negligible. This improvement on verbs shows that understanding the inter-relation of different objects is important for HOI.

For noun recognition, it is interesting to find that THORN can now predict some classes such as water and wall. These classes are barely detected with the object detector. This is a result of the reasoning process on cross object classes, which refines the nodes and can finally predict overlooked object classes.

We provide more interesting qualitative studies in the supplementary materials.

V. CONCLUSION First-view action recognition relies on capturing the visual relationships between different objects and the human. In this work, we propose an object-centric model, which projects the standard CNN features into object class-specific features. After that, we compute the inter-object relations in graph reasoning, where each node corresponds to an object class and each edge represents the relation between two different objects. We evaluate our model on two large and challenging datasets. THORN achieves state-of-the-art performance on both datasets, which shows the effectiveness and robustness of our method. As our method relies on object detection precision, our future work aims at developing an architecture that can combine object detection and action recognition tasks. We also want to extend our model for first-view action detection for untrimmed video.

Fig. 1 .

 1 Fig. 1. An example of the Human-Object Interactions of wash plate in an first-view video. Green arrows represent interactions at the same time step (i.e., spatial relation) while black arrows represent interactions across time.In practice, the model captures all the objects detected. For simplicity, here we highlight only the relevant objects to wash plate.

Fig. 2 .

 2 Fig. 2. THORN architecture contains three main components: (1) a Visual encoder (i.e., X3D) encodes the input RGB clip into a primary spatiotemporal representation. (2) The obtained representation is fed to the Object Representation Filter, which maps the previous representation into object-class representation. To ensure a discriminative object representation, an object classifier is added on top of the object-class representation. This classifier is trained with the pseudo-object ground truth provided by an object detector. (3) The object-class representation is also sent to the Object Relation Reasoning module to model the temporal-object relation in a dissociated manner. Finally, two classifiers are used to predict the verbs and nouns relevant to the action.

Fig. 3 .

 3 Fig. 3. Representation of our Object Representation Filter (ORF). The input is the feature map from the 3D encoder reshaped to T × H ′ W ′ D and the duplicated Co times, where Co is the number of classes. Finally, we have a representation specific to each object class.

Fig. 5 .

 5 Fig. 5. Accuracy improvement on nouns (right) and verbs (left) w.r.t X3D.

TABLE I ABLATION

 I STUDY ON DIFFERENT SETTINGS. THIS EVALUATION IS ON EPIC-KITCHEN DATASET. TEMPORAL NODES MEANS USING THE FINAL OUTPUT OF X3D OF SIZE T × 2048 TO CREATE NODES, WHILE SPATIO-TEMPORAL NODES MEANS USING A MID LAYER OF SIZE T × 7 × 7 × 432 WITH MORE SPATIAL INFORMATION. FINALLY ADJ-MATRIX STANDS FOR USING THE ADJACENCY MATRIX FOR PREDICTING THE VERBS INSTEAD OF USING ONLY NODES FOR NOUNS AND VERBS.

		verbs	nouns	actions
		top1	top5	top1	top5	top1	top5
	X3D	46.5	79.8	34.3	65.3	21.0	38.7
	THORN/temporal nodes	55.8	82.86	39.9	66.37	26.8	44.0
	THORN/temporal nodes + ADJ-matrix	60.3	86.0	41.1	66.9	30.1	47.3
	THORN/spatio-temporal nodes + ADJ-matrix	61.0	85.9	42.9	67.9	30.5	47.5

TABLE II ABLATION

 II STUDY ON FUSING THE SCORES OF THORN WITH THE SCORES FROM THE OBJECT DETECTOR (FASTER RCNN). THIS EVALUATION IS ON EPIC-KITCHEN DATASET. FUSING BOTH SCORES BRINGS SIGNIFICANT IMPROVEMENT ON TOP-1 ACCURACY. FOR THE OBJECT DETECTOR, WE USE AN AVERAGE POOLING ON ALL THE VIDEO CLIP FRAMES OBJECT DETECTION SCORES AND ADD A THRESH-HOLD OF 0.3

	Faster-RCNN scores THORN Nouns
	×	31.5
	×	32.8
		42.9

B. Comparison with the State-of-the-Art

TABLE III COMPARING

 III THORN MODEL WITH OTHER STATE-OF-THE-ART METHODS ON THE VALIDATION SET. EVEN THOUGH SOME OF THESE COMPARISONS ARE NOT FAIR SINCE THESE MODELS ARE USING MULTI-MODALITIES, WE STILL HOLD THE OVERALL BEST ACCURACY, WHICH SHOWS THE STRENGTH OF OUR MODEL

	Model	Obj	RGB Flow	Audio Verbs Nouns Actions
					top1	top1	top1
	Baradel[33]	×	×		40.9	-	-
	3D-CNN	×	×	×	49.8	26.1	19.0
	STO[18]		×	×	51.0	26.6	19.5
	LFB[18]		×	×	52.6	31.5	22.8
	AssembleNET++ ODF+SDF[20]			×	60.0	37.1	25.2
	THORN		×	×	61.0	42.9	30.5

TABLE IV COMPARING

 IV THORN MODEL WITH OTHER STATE-OF-THE-ART METHODS ON EGTEA GAZE+ SPLIT1. WE HOLD THE BEST ACCURACY ON ACTIONS

		Two-stream I3D [9] TSN [34]	ego-rnn [35] LSTA [36]	SAP [37] THORN
	ACC %	43.8	54.2	58.0	62.1	62.0	64.1	67.5


Outline

Outlooks

  Geophysical airborn surveys dedicated to mining prospection are outdated (conducted from the end of 60's to 80's)  >1990's surveys were dedicated to natural risks diagnosis, deep water research or prospection of construction material More recent methods with higher resolution exist to explore and refine underground mining potential  See following presentation by Elodie Willard (Orano Mining) 1973 → Government inquiry into the supply of raw materials for French industry The Ministry of Industry assigned the BRGM, as part of its public service mission, to realize an "Inventory of Mineral Resources in the National Territory" (1975-1991 metropolitan 1995 in some overseas territories as French Guiana): • Assess the French raw material potential • Promote mineral exploration and production  Strategic prospecting campaigns  "Stream sediments" (~285,000 samples over 102,000 km 2 )  "Alluvial" complementary surveys (97,000 samples) .mineralinfo.fr/page/lactivite-miniere-en-france and camino.beta.gouv.fr (data consulted on 15/06/2021) Identified reserves of primary resources and location of deposits  World production (mining and metallurgy : producers and processors with their strengths and weaknesses)  Risk factors (industrial, environmental, geopolitical and societal)  Analysis of demand:  Global consumption (uses and user industries)  Mapping industrial channels and positioning of the players (partnerships, competition, monopolies, etc.)  Assessment of a companies' exposure to risks (fluctuation of raw materials' prices, regulations, government policies, etc.)  Recent market developments and impacts on users (substitutions, technological changes, new markets, recyclability…) ://www.mineralinfo.fr Critical Raw Materials are both of high economic importance for the considered state/sector/industry and have a high risk of supply Uses: hard metals/cemented carbides in steels, superalloys and wear-resistant (drilling and cutting tools, automotive components and aerospace/defense)  Many known resources in FR (but almost no prospection since 1990) Fumade W skarn deposit in Tarn department: ore grades vs. tonnage and comparison with other skarn deposits (Audion, 2013 ; BRGM/RP-62718-FR) deposit types:  Quartz-Stibnite veins  By-product of Pb-Zn stratiform deposits  Disseminated in Carbonates  2 distinct minerals:  Stibnite (Sb 2 S 3 )  Berthierite (FeSb 2 S 4 )  Past production: 125 203 t Sb (42 000 t Sb for La Lucette) France was the 1 st leading World producer from 1890 to 1908  Sb deposits in the Variscan Armorican belt are spatially correlated with high-density and magnetic zones, reflecting probably the presence of mafic/ultramafic bodies at depth (Pochon et al.  2016, Terra Nova 28) a key for Sb prospecting in deformation belts  Uses : flame retardant materials, batteries Pb-Sb, insulating sheathing, plasturgy additive… et al. 2021 (BRGM+MTE) in prep. Tin (Sn) in France IMERYS CERAMICS FRANCE produces few tonnes of Sn-Ta concentrates (cassiterite-tantalite) as by-products of kaolins in the quarry of Echassières Stockwerk of Qtz-Uses: Li-ion batteries, ceramics and glasses, metallurgy…  Main deposit types: Top of the altered Beauvoir granite (kaolin quarry) © BRGM, Eric Gloaguen Predictive Li potential assessment in hard-rocks using BRGM's CBA method FR  EU (Gourcerol et al., 2019, Ore Geology Reviews Vol.109) IMERYS CERAMICS FRANCE explores Li in rare metal granites (Beauvoir deposit)  Hard-rock deposits (with several bearing minerals)  Deep geothermal brine deposits EIT Raw Materials EuGeLi project coordinated by ERAMET (Millot et al., AGU 2020) + following presentation by Clio Bosia for Lithium de France) Li contents in Eu geothermal brine deposits Source : © Eramet http://www.eramet.com/en/activities/innovate-design/eugeli-project The Soultz-sous-Forêts site in Bas-Rhin, France © BRGM -Albert Genter Gourcerol et al. 2021 in prep. Evolving interest in metals regarding technological innovation and end-uses driving consumption  Relatively old aerial geophysical surveys over metropolitan France for mining exploration/deep potential  Limited historical data on CRMs in past mineral resource inventories  No systematic multi-elemental geochemical analyses during exploration campaigns  Evolution of geological knowledge and ore genetic models  Importance of maintaining research and exploration in France  Reprocessing mining wastes and tailings from old historical mining sites with more efficient technologies  Supply risk disruption evolves, especially with the pandemic context (pressure on market, variability of prices and demand, new protection policies, transport restrictions, increase of operational costs…)  Evolution of the state's strategy according to economic importance and industrial dependency  relocation of primary production and value chains for some very critical commodities ? Reappraisal of the French metropolitan potential in selected critical metals is essential and promising:  W, Sb, Sn in Variscan orogenic belts (Armorican and Central Massives)  Li in hard rocks and geothermal brines  Legal mining framework policy is currently reassessed by the French Government (2021)


Benoit Combemale
email: benoit.combemale@enseeiht.fr

Xavier Crégut
email: xavier.cregut@enseeiht.fr

Alain Caplain
email: caplain@univ-tlse2.fr

Bernard Coulette Grimm-Isycom

Antonio Machado

TOWARDS A RIGOROUS PROCESS MODELING WITH SPEM

Keywords: Process engineering, Model engineering, SPEM, Semantics, OCL

Modeling software process is a good way to improve development and thus quality of resulting applications. The OMG proposes the SPEM metamodel to describe software processes. Its concepts are described through class diagrams. Unfortunately, it lacks a formal description of its semantics that makes it hard to use. So, we propose a specialization of SPEM that clarifies it and we use OCL to formally express constraints on it.

INTRODUCTION

A software application is a complex product which must be elaborated using a specific method : the software development process. The main aim of such a process is to guarantee the development of reliable software, conforming to their specification within expected deadlines and costs. Mastering the software appears to be a necessity because of the growing complexity and critical aspects of software. This must be done with the aim of obtaining better quality and improving reuse. The industrialization of software development began with methods like OMT and grew at the end of the 90's with unification proposals such as UML and RUP. Today, the last step is the use of Model Driven Engineering (MDE) for working on the complete lifecycle of software artifacts.

MDE [START_REF] Bézivin | In search of a basic principle for model driven engineering[END_REF] appeared with OMG proposals such as MOF (omg, 2002) and MDA [START_REF] Miller | Model Driven Architecture 1.0.1 Guide[END_REF]. It tries to unify processes detailing with the crucial aim of reusing them. SPEM (Software Process Engineering Metamodel) (omg, 2005) is one of the OMG proposals. It is a metamodel for software development process specification that gives a rigorous syntax but only partly formalizes the semantics (thus implying some ambiguities), and does not offer any help on how to build a process model. It is why we have defined a specialization for its rigorous exploitation [START_REF] Combemale | Spécification et vérification de modèles de procédés de développement[END_REF]. We first present the main concepts of SPEM (section 2) and then our specialization of SPEM (section 3). Before the conclusion, we present related works on section 4.

PRESENTATION OF SPEM

SPEM is a metamodel used for the specification of concrete software development process. It offers an object oriented approach using the UML notation. As an OMG proposal, SPEM is integrated in the pyramidal architecture of MDA organization as a MOF metamodel and as a UML profile [START_REF] Breton | Process-centered model engineering[END_REF]. SPEM takes up a great deal of UML diagrams (packages, use cases, classes, activities, sequences, statechart), excludes some elements (node, component, etc.) (omg, 2005, §11.1) and adds several stereotypes that are presented hereafter (omg, 2005, §11).

SPEM is based on the idea (fig. 1) that a software development process is a collaboration between active abstract entities called roles which perform operations called activities on concrete and real entities called In fact, the SPEM metamodel is more complex. The extract given on fig. 2 proves it. SPEM defines the concept of WorkDefinition which can be decomposed reflexively. Beside the activities there are other specializations which are not drawn on fig. 2 : Lifecycle which is a sequence of Phases and Iterations to define a complex work. An Activity can be divided into Steps. A step is a specialization of ActionState (fig. 2) which implies a partial order (omg, 2004, §4.12).

Each WorkDefinition is under the responsibility of a unique role (ProcessPerformer). In the case of an Activity, a set of other roles (ProcessRole) can assist the main role for the realization.

SPEM also defines two important concepts : Process and Discipline. A Process corresponds to the root of a process model from which a tool can do the transitive closing of a complete process. A Discipline allows, within the process, to partition activities according to a common "theme". The output products of each of the activities of a discipline must be categorized under this same theme.

ADDING RIGOR TO SPEM

Using SPEM is difficult because the OMG proposal is very generalist and provides no directives on how to use it. Furthermore, its semantics is essentially ex-pressed in natural language that leads to the construction of inconsistent process models because of the lack of a formal definition of concepts. For example, the ProcessPerformer is a concept known as being ambiguous [START_REF] Bendraou | Uml4spm : A uml2.0-based metamodel for software process modelling[END_REF].

So, we have decided to define a specialization of the SPEM metamodel whose purpose is to clearly define concepts and formally express their semantics with OCL (omg, 2003). Being more directive, our proposal brings more assistance in the industrial construction of a process and more facility in the use of SPEM. Thanks to the restrictions put on SPEM, it is possible to ensure the coherence of its models. Our proposal being a restriction of SPEM, our models conforms to SPEM.

SPEM metamodel specialization

Our metamodel (fig. 3) divides process models according to two main views. The structural view shows the process hierarchy : a process is associated to a lifecycle cut out in phases, themselves made up of activities (and iterations). The descriptive view details work definitions. It makes possible to classify them according to process roles and thus disciplines. Work definitions are specified through preconditions, goals and products used or realized.

As in SPEM, WorkDefinition metaclass is a concrete class. It allows to instantiate WorkDefinitions which will not be typed yet and thus not yet semantically defined in the process (e.g. analysis phase).

Because ProcessPerformer (fig. 2) has no clear semantics in the initial SPEM metamodel, we have merged it with ProcessRole. This merge avoids confusion In addition, most of the relations of our proposal are taken again from the original metamodel, either as is or by specialization. Thus, associations between Lifecycle, Phase, Iteration and Activity metaclasses fit to the redefinition of reflexive relation of the WorkDefinition metaclass of SPEM (fig. 2). We have constrained the multiplicity of the source of "0..*" by "1" (thus giving the composition semantics).

The relations which are not directly taken from SPEM metamodel are inherited or deduced from UML 1.4 (from the Core package, (omg, 2004, p.22)). For example, the reflexive relation on WorkProduct is a deduction of the relations that link Classifier metaclasses in UML. The "use" and "realize" relations between WorkDefinition and WorkProduct are also deduced from UML as shown on fig. 2. They allow to make explicit the relations described in the SPEM conceptual model (fig. 1).

Semantic details with OCL

Because syntactic added is not suffisant to semantically define the metamodel, we have used OCL (as recommended by the OMG) to add formal constraints. These constraints limit the possible instanciations and thus the valid process models. Here is an example of such a constraint : "A role must be responsible for all the products carried out by activities of which he is in charge, and reciprocally". The static checks that we made are not sufficient and there should be extra dynamic checks. For example, it is impossible to ensure before enactment that an activity will respect its time limit but it can be checked during enactment. Furthermore, this kind of check is meaningful only during real project enactment and not during process model simulation. Let us note that a process model can not generally be completely defined before the project starts. If the great stages of the development are generally known, the detail of their realization depends on the difficulties encountered at the time of their realization. It could then be necessary to reinforce a team, to consider other solutions or another approach.

The process modelling domain has been very active during the 90's. There was a strong separation between Process Description Languages (PDL) and process enactment engines. PDL could be classified into different categories : some of them were based on programming languages (ProcessWise/PML [START_REF] Greenwood | The use of a persistent language in the implementation of a process support system[END_REF] or RHODES/PBOOL [START_REF] Crégut | PBOOL : an objectoriented language for definition and reuse of enactable processes[END_REF]), others on rules (MARVEL/MSL [START_REF] Kaiser | Intelligent Assistant for Software Development and Maintenance[END_REF], ADELE/TEMPO [START_REF] Warboys | Proc. of EWSPT'94[END_REF] or EPOS/SPELL [START_REF] Warboys | Proc. of EWSPT'94[END_REF]), others on petri nets (SPADE/SLANG [START_REF] Bandinelli | Modeling and improving an industrial software process[END_REF]), and the others are hybrid solutions. Each of these languages had its own specific tools.

The actual tendancy is to unify PDL. Let us quote for example SPEM metamodel suggested by the OMG and XML-based languages like XPDL (XML Process Description Language) (WfMC, 2005) proposed by the WfMC(Workflow Management Coalition) or BPML (Business Process Management Language) [START_REF] Arkin | Business Process Modeling Language[END_REF] proposed by the BPMI (Business Process Management Initiative). All these approaches define LDP's concepts [START_REF] Breton | Contribution à la représentation de processus par des techniques de méta-modélisation[END_REF] by proposing a syntax in the form of a metamodel for SPEM or a XML schema for XPDL and BPML. Semantics is only described informally (in natural language). Furthermore, process enactment is not formally defined even if there are specific engines for specific targets (e.g. BPEL4WS & BPEL-J).

CONCLUSION

In this article, we have presented our work on the modelling of software processes. Because the SPEM metamodel lacks rigorous and formal definition, we have proposed a restriction of SPEM that remains compatible with the standard and puts the focus on hierachical decomposition of workdefinitions (the structural view) and the categorization of process components (roles, products and workdefinitions) according to disciplines (the descriptive view). Semantics that are not graphicaly captured are expressed using OCL constraints either at the metamodel level or at the process level. Our SPEM specialization has been used to model a UML based method called MACAO [START_REF] Combemale | Modélisation rigoureuse en SPEM de procédé de développement[END_REF].

Unfortunately, OCL can only capture structural constraints. Our future work is to define an operational semantic for SPEM in order to enact a process model described in SPEM. So we are investigating several approaches including the ones that describe operational semantics for metamodel such as Kermeta, Xion and xOCL.

FIG. 1 :

 1 FIG. 1: SPEM conceptual model

FIG. 3 :

 3 FIG. 3: Extract of our specialization of SPEM metamodel

context

  ProcessRole inv : let productsActivities : Set{WorkProduct} = -Definition of the WorkProduct set made by the activities of which 'self' is responsible self.work→ select(a :WorkDefinition | a.oclIsT ypeOf (Activity)). oclAsT ype(Activity).output→ asSet() in self.workProduct = productsActivities Another OCL constraint formalizes the merge of ProcessPerformer and ProcessRole (sec. 3.1) in prohibiting the instanciation of the ProcessPerformer : context ProcessPerformer inv : self.allInstances() → size() = 0