Camille Maurice

Francisco Madrigal

André Monin

Frédéric Lerasle
email: lerasle@laas.fr

A new Bayesian modeling for 3D human-object action recognition

Intelligent surveillance systems in human-centered environments require people behavioral monitoring. In this paper, we propose a new Bayesian framework to recognize actions on RGB-D videos by two different observations: the human pose and objects in its vicinity. We design a model for each action that integrates these observations and a probabilistic sequencing of actions performed during activities. We validate our approach on two public video datasets: CAD-120 and Watch-n-Patch. We show a performance gain of 4% in action detection on the fly on CAD-120 videos. Our approach is competitive to 2D image features and skeleton-based methods, as we present an improvement of 16% on Watch-n-Patch. Action recognition performance is clearly improved by our Bayesian and joint human-object perception.

Introduction

Activity recognition is an important task in the development of many practical applications such as home health monitoring, human-robot interaction, among others. An activity can be seen as a temporal sequence of actions e.g.: the activity prepare coffee involves actions such as pour water, add ground coffee and turn on machine. Activities involved at home or in an industrial environment may differ but their underlying actions may be similar as they cover object displacement, object grasping, object interaction... This is still a challenging topic where actions could be misinterpreted due to background clutter, partial occlusions of the body, presence of numerous objects in the human vicinity and viewpoint changes. Some of the problems can be handled by the use of additional sources, i.e., wearable devices such as accelerometers [START_REF] Chen | Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor[END_REF], but the possibility to use invasive equipment depends on the context or sometimes is not applicable.

Depth cue offers non-invasive information with the opportunity to segment the background and to create 3D models invariant to camera viewpoints. From these insights, we focus on non-physically invasive sensors and decide to only rely on a single low-cost RGB-D sensor (e.g. Kinect, XTion).

Convolutional Neural Network (CNN) based methods have shown good performance in the area of multiple objects detection and human pose estimation over color images. The fusion of both objects and human pose perception can be used to understand complex activities, especially in the case when objects are in interaction with the environment and the human pose trajectory is not highly discriminant alone. Thus, building upon the recent advances of CNNs we propose to use human pose and objects detection as low-level input for our Bayesian framework which allows labelling actions on the fly. Some actions do not differ much in the human pose involved but rather in the object configuration during action execution. Given the inferred class of each object, we can provide knowledge on how it is manipulated i.e. the socalled affordance [START_REF] Gibson | The theory of affordances. the ecological approach to visual perception[END_REF]. Multiple affordance labels can be associated with an object class e.g. an apple is eatable and peelable not readable. Those affordance labels may imply specific object configuration, in the case of the label peelable there must be a peeler or a knife nearby.

However human pose perception is relevant to achieve action recognition as some actions can be discriminated only by the pose. Grasping action type has a typical movement of the hand going away from the body to reach an object. Using the pose information, especially when all objects are static we can find which one is in interaction. Also, human pose information can help to reduce erroneous object movement due to flickering values in the depth image.

Our contributions in this work involve (1) the integration of contextual cues through human pose and object detection by open-source deep learning libraries into a Bayesian framework for action detection.(2) A model of actions reusable into a different context. [START_REF] Chen | Utd-mhad: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor[END_REF] The use of a hyperparameter optimization tool for tuning.

The rest of the paper is organized as follows. We briefly review the related work in Section 2. We then describe our joint human-object Bayesian modeling and details its implementation in Section 3. We present our evaluation protocol and highlight our gains over two public datasets in Section 4. We draw a conclusion of the presented work in Section 5.

Related Work

Early approaches [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF][START_REF] Li | Action recognition based on a bag of 3d points[END_REF] and datasets [START_REF] Laptev | Recognizing human actions: a local svm approach[END_REF][START_REF] Li | Action recognition based on a bag of 3d points[END_REF] focus solely on human perception and its pose trajectories in video sequences for action classification. Each video in those datasets contains a single human action without any context: no backgrounds nor surrounding objects. Still today, most of the largest RGB-D datasets focus on skeleton-based action detection with few to no objects to interact with, e.g.: MSRAction [START_REF] Li | Action recognition based on a bag of 3d points[END_REF], BerkeleyMHAD [START_REF] Ofli | Berkeley mhad: A comprehensive multimodal human action database[END_REF], Human3.6M [START_REF] Ionescu | Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments[END_REF], NTU RGB+D [START_REF] Shahroudy | Ntu rgb+ d: A large scale dataset for 3d human activity analysis[END_REF]. Even some datasets offer more realistic scenarios where a single person performs, in the same video sequence, different actions that may involve interaction with his/her environment, e.g. object manipulation, object-toobject interaction. This kind of scenario with contextual information can be found in CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] and Watch-n-Patch [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF] datasets.

Human action recognition approaches can be roughly categorized into data-driven or model-driven paradigms. Data-driven approaches such as Convolutional Neural Networks (CNN) for action recognition [START_REF] Baccouche | Sequential deep learning for human action recognition[END_REF][START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF][START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF], pose and object detection [START_REF] Cao | Realtime multiperson 2d pose estimation using part affinity fields[END_REF][START_REF] Liu | Ssd: Single shot multibox detector[END_REF] are state-of-the-art on 2D videos or single image datasets. These deep learning networks learn a representation of a model based on pre-recorded and labeled data, the so-called supervised learning.

Recent advances in 2D deep learning object (resp. human pose) detection with frameworks such as Single Shot Detector [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] (resp. OpenPose [START_REF] Cao | Realtime multiperson 2d pose estimation using part affinity fields[END_REF]) are also linked to the emergence of challenges. An example is the MSCOCO challenge [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] which provides images with 1.7 million of labeled keypoints for human pose detection and also images with 500k annotated objects. Proposals have demonstrated their robustness to a variety of situations and camera viewpoints by means of a heavy training over large annotated datasets.

Since the release of NTU-RGB+D [START_REF] Shahroudy | Ntu rgb+ d: A large scale dataset for 3d human activity analysis[END_REF], the largest RGB-D dataset for human action recognition, most of the 3D deep learning approaches are solely based on 3D human pose through the analysis of skeleton trajectories [START_REF] Liu | Spatio-temporal lstm with trust gates for 3d human action recognition[END_REF]. However, those techniques are not applicable on contextual action recognition for two reasons: (1) Sometimes skeleton motion alone cannot discriminate between two different actions, i.e., eating an apple and drinking water imply similar motion of the hand towards the head. (2) They are not straightly applicable due to the limited size of 3D datasets such as CAD-120 and Watch-n-Patch.

To handle the limited size of the aforementioned datasets, H. Kataoka et al. [START_REF] Kataoka | Recognition of transitional action for short-term action prediction using discriminative temporal cnn feature[END_REF] employ color and differential images as input of a CNN pre-trained on ImageNet (over 1 million annotations) to perform action detection on Watch-n-Patch. M. S. Ryoo et al. [START_REF] Ryoo | Pooled motion features for first-person videos[END_REF] combines 2D CNN features pre-trained on ImageNet and conventional motion features such as Histograms of Optical Flow (HOF). Even though the dataset provides depth information, both approaches [START_REF] Kataoka | Recognition of transitional action for short-term action prediction using discriminative temporal cnn feature[END_REF] [24] rely only on 2D image and motion features. A. Jain et al. [START_REF] Jain | Structuralrnn: Deep learning on spatio-temporal graphs[END_REF] use a Structural Recurrent Neural Network (S-RNN) with pre-defined spatio-temporal graph structures where nodes and edges are represented as Long Short-Term Memory units (LSTM). They reason on 3D skeletons and their interaction with objects.

While achieving undisputed performances when training data is sufficiently diverse and large, end-to-end deeplearning action recognition frameworks suffer from a lack of interpretability of their features [START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. When a CNN does not recognize an action, it is difficult to know whether the issue is related to the implicit inference of the object detection and/or related to human pose. Their dependency on a large volume of training data implies high computational costs and a lack of explanatory capacities.

For context-based action recognition, we are interested in the spatial configuration of the objects and their interaction with other objects of the scene and with the human. RGB-D videos propose a 3D segmentation of the scene based on the involved distances (sensor vs. scene, human vs. objects, object vs. object). The 3D perception of the scene allows us to derive distances in real units (e.g. meters) and is less subject to misinterpretation of the scene due to perspective effects. Thereby, irrelevant objects in the background that may lead to a mistake in the action recognition are easily removed. Furthermore, action motions are more discriminant with 3D information especially when the motion is not fronto-parallel.

Whereas model-based algorithms are inherently interpretable, they also require less training data. Furthermore, model-based approaches may include custom modeling of skeleton trajectories, affordances and spatial configuration of the scene. H. Koppula and A. Saxena [START_REF] Koppula | Learning spatio-temporal structure from rgb-d videos for human activity detection and anticipation[END_REF] propose a Conditional Random Field (CRF) that models the scene and the spatial-temporal relations through object affordances.

To recognize actions over contextual RGB-D datasets, we design Bayesian models to infer the action labels on video streams. The scene reveals different cues about the nature of objects, their spatial configuration in the human vicinity, and his/her posture. The previous action also gives information about the likelihood of the following actions.

Our framework considers all the aforementioned cues. We take the opportunity of the recent development of 2D deeplearning for human pose and object detection to use them as lower input for a higher-level inference: the current action on the video. No further data labeling or training is required as pre-trained deep-learning models are widely available.

Our framework is tuned from a small set of annotated data, using a Bayesian optimization tool: SMAC [START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF]. Thus our approach presents adaptability characteristics that ease the addition of new action classes or the use of another dataset.

To the best of our knowledge, such a framework i.e. sequencing deep-learning (data driven) and Bayesian (model driven) concepts has not been designed before in the literature. Fig. 1 illustrates our action recognition framework and we present the probabilistic action models in the following section.

Our Approach

In this section, we first define the probabilistic model, which is the core of our action detection process, then we provide details regarding its implementation.

Probabilistic Formulation

Here we consider activities performed by a single human with several objects known in a priori. An activity is defined by a sequence of actions as proposed by Moeslundet al. [START_REF] Moeslund | A survey of advances in vision-based human motion capture and analysis[END_REF]. We recognize the actions based on the joint observation of the human pose and the objects in the scene. We describe the human pose by a skeleton composed by a set of 3D points representing joints and some pairs of joints represent limbs.

We consider there are two types of actions to infer: (1) actions before a hand grasps an object and (2) actions while a hand is in contact with an object. In both cases, the whole surrounding objects and skeleton pose affect the probabilities of action classes. Object affordances are considered in the human-object interactions as well as in object-object interactions. Once an object is grasped it can further interact with: a part of the body (e.g. to eat fruit with the mouth, to make a phone call with the ear) or with another object (e.g. pour a bottle of milk into a cup, place a book on the table). No further interaction is also an alternative, (e.g. move an object, use a smartphone, turn on a light.) Our approach is designed to encompass all these different types of actions in a unified Bayesian framework based on the inferred observations of the spatial configuration of the objects and the human pose.

Each action a is associated with a model. Let A = {a 1 , a 2 , ..., a n } be the set of n actions. Let O t = {s t , Ω t } represent, at time t, the joint observation of the human skeleton s t and the set of objects detected in the human vicinity: Ω t = {ω 1 , ω 2 , ..., ω Card(Ω) } with Card(Ω) being the number of objects in the scene. We model the a posteriori probability of the actions given the observations as follows:

p(a 0:T |O 0:T ) ∝ T t=0 p(O t |a t ) T t=1 p(a t |a t-1 ). (1)
In equation ( 1), p(O t |a t ) is the likelihood of the observation given the action a t . The term p(a t |a t-1 ) denotes the transition probability between two successive actions. Thus, our approach aims to achieve online action classification, without the need of an action segmentation preprocessing step.

The classification is inferred through information aggregation over T frames. Therefore, the inference is near to realtime as T typically ranges between 2 and 10 frames.The observation likelihood in equation ( 1) is decomposed as:

p(O t |a t ) ∝ p(s t |a t , Ω t )p(Ω t |a t ). (2)
Here in equation ( 2), p(s t |a t , Ω t ) models the probability of the observed skeleton (s t ) to be a representation of the estimated skeleton model associated with action a t , in interaction with one object of the set Ω. The skeleton pose s is in our case is reduced to the 3D positions of the upper body extremities i.e. hands and head. This is represented as a 9dimensional vector of Cartesian joint positions in the world coordinate frame as shown in equation ( 3):

s = (s l.hand , s head , s r.hand ). (3)
The equation ( 2) is the product of two likelihoods, the first one can be expressed as:

p(s t |a t , Ω t ) ∝ N (d p (s, s n ); µ 1 , σ 1 )N (d o (s, Ω); µ 2 , σ 2 ), (4)
where s n is the skeleton model associated with action a t , for example if a t = eat, one of the hand should be close to the head. For clarity reasons we omit the term t. In The second likelihood of equation ( 2), i.e. p(Ω t |a t ), represents the spatial configuration of all objects detected on the scene and it is modeled as follows:

p(Ω t |a t ) = Card(Ωt) k=1 p(ω k t |a t ). (5)
In equation ( 5), the likelihood is also expressed as a normal distribution.

In Fig. 2 we present the RGB-D frames input, the 3D spatial configuration of objects as well as associated Gaussian probability distributions. We can see during the action pour that static objects are in interaction with the table, and the milk is in interaction with the bowl while being in the hands. Details regarding human pose and object detection, as well as the parameter optimization of our modeling, are in the following section.

Implementation

As mentioned in Section 3.1 our formulation requires the skeleton s as a representation of the human pose and the objects ω observed. They are both detected using 2D images through two deep learning neural network frame-works and projected onto the 3D space using the given calibration parameters of the Kinect camera. The 2D human pose is inferred by OpenPose [START_REF] Cao | Realtime multiperson 2d pose estimation using part affinity fields[END_REF], a popular open source library with up to 900 citations, that outperforms state-ofthe-art results while maintaining real-time execution. The authors also released pre-trained models on MSCOCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] keypoints challenge data. This framework takes as input an RGB image to infer the human posture, i.e. the localization in the 2D image of 25 skeleton joints and their corresponding confidence scores. This bottom-up approach does not rely first on the detection of the person, but rather the different body parts. It has proven to perform well even under self-occlusions situations and to achieve real-time performance.

Objects in the scene are detected by Single Shot Multi-Box Detector (SSD) [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] pre-trained on the MSCOCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] featuring 80 objects categories. Categories include everyday-life objects e.g.: cup, bottle, microwave, TV, desk, smartphone. This is a famous object detector with up to 3400 citations to this day.

As seen in Section 3.1, we have to set values for the action transition matrix and the standard deviations for each Gaussian probability density. In object-object interaction, they depend on distances between pairs of objects, e.g. pour milk in a cup. In the affordance model, they may depend on the distance between head and object, e.g. drink water, or also depending on velocity ranges. The transition matrix between actions needs also to be tuned and its size is n × n, n being the number of actions. Manual tuning of a great number of parameters is difficult, and due to the dimensionality of the configuration space, grid search methods are not suitable. Instead, we use a hyper-parameter optimization method: SMAC [START_REF] Hutter | Sequential model-based optimization for general algorithm configuration[END_REF], which has up to 900 citations. It has proven already to increase the performances of computer vision algorithms by tuning hyper-parameters [START_REF] Domhan | Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves[END_REF][START_REF] Maurice | Hyper-optimization tools comparison for parameter tuning applications[END_REF]. In order to find the set of parameters values, it requires a measure of the algorithm's performance. SMAC optimizes parameters with respect to a cost function. In our case, we want to find the set of parameters leading to the best results in terms of action classification. Therefore, we compute the performance metric F 1 -score, f a i of each action in A, with n being the number of actions to consider. Thus, the cost function C is defined by equation ( 6):

C = n - n i=1 f a i . ( 6
)
Upon each iteration, SMAC builds a model of the cost function, which allows it to find the optimal set of parameters with few evaluations. SMAC takes into account previous evaluation outcomes so that it spends less time on the evaluation of irrelevant parameters values, in contrast to random or grid search methods. We compare the inferred action labels to the ground truth to compute F 1 -scores and so the

Experiments

In this section, we present the public datasets and metrics used to evaluate our approach, and then we present and discuss the results in comparison to the literature.

Datasets and metrics

We evaluate our approach on two public datasets in the literature: CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] and Watch-n-Patch [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF]. Thus we can make a comparison of our approach with state-of-theart methods.

CAD-120 dataset [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] consists of 120 RGB-D videos that are performed by 4 subjects. There are 10 activities (making cereal, taking medicine, stacking objects, unstacking objects, microwaving food, picking objects, cleaning objects, taking food, arranging objects, having meal). In this dataset, each video sequence represents an activity. As defined in Section 3.1, an activity corresponds to the execution of successive actions. For example, the activity making cereals is made of the action classes reach, move, pour and place. All the actions of the dataset and their unbalanced distribution, expressed in percentage of corresponding frames, are shown on the bar chart of Fig. 4(a). This dataset offers front and side views, see Figs. Watch-n-Patch dataset [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF] is split into two environments: kitchen and office. The environment office consists of 196 videos recorded in 8 different offices with 10 different actions (reading, walking, leave-office, fetch-book, put-back-book, put-down-item, take-item, play-computer, turnon-monitor, turn-off-monitor). Distribution in the dataset of the 10 actions is on the bar chart of Fig. 4(b). Each video in the dataset contains 2-7 actions where the actor interacts with different objects in a cluttered background. Also, it offers a variety of camera viewpoints: front and side views are illustrated in Figs. 3(c) 3(d). Figure 4. Action class distribution in datasets CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] and Watch-n-Patch [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF]. Corresponding percentage of frames in the dataset to each action label on the y-axis.

In the literature, the approaches are evaluated and compared using two main metrics: F 1 -score and accuracy as micro precision and recall. The F 1 -score measures accuracy as the harmonic mean of precision and recall. As our approach only outputs one label per frame, in this case the accuracy represents the ratio of correctly labeled frames, also called frame-wise accuracy.

Then, approaches in comparison often provide details about macro precision and macro recall. The difference in the computation between micro or macro depends on the consideration of whether classes in the dataset are balanced or not. Those metrics are computed with or without the temporal action segmentation depending on the nature of the approach.

For a fair evaluation on CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF], we use a 4-fold Table 1. Precision (P), Recall (R) and F1-score (F1) averages over all the classes on CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] dataset using ground-truth segmentation.

Approach P R F1 GPNN [START_REF] Qi | Learning human-object interactions by graph graphparsing neural networks[END_REF] 0.88 0.86 0.87 HELK [START_REF] Hu | Learning latent structure for activity recognition[END_REF] 0.83 0.82 0.82 S-RNN [START_REF] Jain | Structuralrnn: Deep learning on spatio-temporal graphs[END_REF] N/A N/A 0.83 QHWZ [START_REF] Qi | Predicting human activities using stochastic grammar[END_REF] 0.71 0.68 0.69 Ours 0.84 0.80 0.82 cross-validation with a new human subject in each fold, in a similar way as [START_REF] Hu | Learning latent structure for activity recognition[END_REF][START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF][START_REF] Qi | Predicting human activities using stochastic grammar[END_REF][START_REF] Qi | Learning human-object interactions by graph graphparsing neural networks[END_REF]. We use the SMAC framework to tune the hyperparameters in the training folds, leaving one fold for testing.

Evaluations and discussion

Hereafter, we compare our action detection proposal against state-of-the-art approaches. Our method is originally developed to deal with online action detection and action transitions. We evaluate ourselves in two different settings: with and without ground-truth action segmentation. First on CAD-120 dataset, then on Watch-n-Patch dataset.

As our approach outputs one prediction per frame and not over a sequence, we slightly adapt it as follows. As time bounds of the different actions in the sequence are known, we use a weighted-vote to find the predicted class over this time segment. Average Precision, Recall and F 1 -score are derived from the action class distribution in Fig. 4(a) and confusion matrices found in the original publications when available. Results with ground-truth action segmentation on CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] are summarized in Table 1. Methods [START_REF] Qi | Learning human-object interactions by graph graphparsing neural networks[END_REF][START_REF] Jain | Structuralrnn: Deep learning on spatio-temporal graphs[END_REF] are data-driven and rely on neural networks for training and inference. Whereas [START_REF] Hu | Learning latent structure for activity recognition[END_REF] relies on the computation of maximum likelihood estimation over graphs and [START_REF] Qi | Predicting human activities using stochastic grammar[END_REF] employs stochastic grammars. We show similar results to S-RNN approach [START_REF] Jain | Structuralrnn: Deep learning on spatio-temporal graphs[END_REF] with a difference of 0.01 in the F 1 -score, without the need of a large-scale dataset with annotations for training. In comparison to HELK [START_REF] Hu | Learning latent structure for activity recognition[END_REF], we obtain similar average results and they show better performances in the detection of action reach and move. However, their weakest performances in terms of precision concern actions eat and clean (< 0.3), which are among the least frequent in the dataset as shown in Fig. 4(a). Whereas we achieve to maintain good performances over those two actions, as we can see on the next evaluation configuration. Now we consider a more challenging situation where the ground-truth action segmentation is not available. In this case we compute the most likely action of each frame and the results are shown in Table 2. Macro Precision and Recall are derived from confusion matrices of the original publications when available. Accuracy is computed as mi- 4(a). In the CAD-120 dataset, actions reach and move represent more than half of the instances. Therefore, we present precision and recall for a more precise analysis of the performances despite class imbalance. We show an overall improvement of at least 4% in accuracy. This improvement is especially significant regarding actions place (resp. clean) with +0.16 (resp. +0.3) in the F 1 -score. We have enhanced the original object detections given by the dataset through SSD detections (e.g. table or desk). As objects are often placed on the table, to know its position helps to design a more accurate action model associated with the action place.

In the confusion matrix of Fig. 5, we observe a strong diagonal and note that one of the major source of error is the high false positive rate of the action move. The action move is the most frequent one as it precedes most of the other actions (pour, eat-drink, place, clean). The false positive rate is due to transitions between move and the following action, being difficult to define precisely.

We now compare our results with the literature using the dataset Watch-n-Patch. This shows the great adaptability property of our framework because the two datasets are quite different in terms of points of view and actions involved. In Table 3, we report the ratio of correctly labeled frames (Accuracy) as in [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF][START_REF] Wu | Watch-bot: Unsupervised learning for reminding humans of forgotten actions[END_REF]. We observe a strong improvement compared to approaches of the literature, i.e. + ∼ 16% in the accuracy. Approach [START_REF] Ryoo | Pooled motion features for first-person videos[END_REF] in comparison, infer action labels solely based 2D image features and optical flow. We show that the additional information about objects configuration and their nature enhance action models, thus action recognition performances. The approach KMHIS [START_REF] Kataoka | Recognition of transitional action for short-term action prediction using discriminative temporal cnn feature[END_REF] relies on motion pattern recognition in 2D through differential images. The significant improvement is also due to our 3D modeling of the scene being more robust to changes in camera viewpoints present in this dataset. In Fig. 6, we observe good performances over the most frequent actions present in the dataset. The class distribution in the dataset is unbalanced: actions read and playcomputer represent 50% of the labeled frames. While the human pose is similar: the person is sitting in front of a desk with motion in hands, in both cases. They strongly differ in the nature of the object being in interaction: a book or a keyboard. We also achieve a good performance on less frequent actions such as turn-on-off-monitor, which represents only 4.27% of the frames. However, in our approach, actions put-back-book and put-down-item are prone to recognition errors because they involve similar pose and the book or the item is often place on the table. We should modify our models to be more specific towards put-down-item.

In Fig. 7 we observe our action detection behavior along the frames of a sequence of CAD-120 [START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF]. We can also note that actions differ also in execution time length. Some are typically longer such as move and open in contrast to Table 3. Accuracy as the percentage of frames correctly labelled of state-of-the-art methods without ground-truth action segmentation on the office environment of Watch-n-Patch [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF].

Approach

Accuracy CaTM [START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF] 38.5 WBTM [START_REF] Wu | Watch-bot: Unsupervised learning for reminding humans of forgotten actions[END_REF] 41.2 PoT [START_REF] Ryoo | Pooled motion features for first-person videos[END_REF] 49.93 KMHIS [START_REF] Kataoka | Recognition of transitional action for short-term action prediction using discriminative temporal cnn feature[END_REF] 59.75 or reach. We have errors on the action transition, on this example we tend to detect the following action too early compared to the ground truth. At the end of action, the probability of the model decreases and the transition matrix favors the probability of the next actions that are likely to occur. This is especially true in CAD-120 where some actions determine the following one e.g.: pour and drink are always followed by move.

Conclusion

This paper proposes an original framework for online action recognition based on the joint modeling of the human and observed objects in the 3D world. Models are described through Gaussian probability densities which standard deviation parameters are learned using a hyperparameter optimization tool i.e. SMAC.

Our joint perception mechanism aims to disambiguate action labeling, e.g. the displacement of an object will be found in many different actions class where the nature of the object and its human affordance will strengthen the right action labeling. We don't need a large number of labeled videos to recognize a similar action in another dataset as it might be the case especially for supervised learning techniques, especially CNN ones. Our framework is shown to be more robust to variations in camera viewpoints.

Our evaluations on two challenging public datasets highlight improvements, especially in online action detection. In the future, we will enhance the skeleton observation with the full upper-body, investigate skeleton trajectories, and trajectories predictions based on motion planning. We will also extend the proposal to online activity recognition through plan execution verification.

Figure 1 .

 1 Figure 1. Action detection framework.

Figure 2 .

 2 Figure 2. From the Kinect camera RGB-D input of 2(a) and 2(b) we model the 3D object configuration of the scene as shown on Figs. 2(c) and their associated Gaussian distribution as shown on 2(d).

Figure 3 .

 3 Figure 3. Example views of the dataset CAD-120 [13] on 3(a)-3(b) and Watch-n-Patch [28] on 3(c)-3(d) with associated ground-truth objects in red boxes, OpenPose upper-body skeleton and action labels are in blue.

  3(a) 3(b).

  Watch-n-Patch[START_REF] Wu | Watch-n-patch: Unsupervised understanding of actions and relations[END_REF]

Figure 5 .

 5 Figure 5. Confusion matrix without ground truth segmentation on CAD-120 [13].

Figure 6 .

 6 Figure 6. Confusion matrix without ground truth segmentation on Watch-n-Patch [28].

Figure 7 .

 7 Figure7. Qualitative result on the sequence microwaving food on CAD-120. Frames are on the x-axis. First row is our approach and second row is the labelled ground-truth.

Table 2 .

 2 Macro Precision (R), macro Recall (R), and accuracy per action on CAD-120[START_REF] Koppula | Learning human activities and object affordances from rgb-d videos[END_REF] dataset without ground-truth action segmentation.

	Approach	KGS [13]	KS [12]	Ours
	Action	P	R	P	R	P	R
	reach	N/A N/A 0.63 0.65 0.67 0.81
	move	N/A N/A 0.30 0.86 0.47 0.68
	pour	N/A N/A 0.93 0.59 0.8 0.73
	eat&drink	N/A N/A 0.92 0.52 0.90 0.79
	open&close N/A N/A 0.84 0.63 0.84 0.75
	place	N/A N/A 0.66 0.61 0.84 0.76
	clean	N/A N/A 0.46 0.58 0.88 0.83
	mean	0.71 0.62 0.75 0.63 0.80 0.76
	accuracy	0.68		0.70		0.74
	cro precision and recall, taking into account the balance of
	actions in the dataset presented in Fig.

Acknowledgement

This work has been partially supported by Bpifrance within the French Project LinTO and funded by the French government under the Investments for the Future Programme.


Theo Delemazure
email: theo.delemazure@ens.psl.fr

Prof Pierre Senellart

A Knowledge Base of Mathematical Results

The basic unit of information of use by researchers in theoretical fields are the mathematical results. We aim to build a knowledge base of these results, using information extraction techniques on scholarly documents. We present an algorithm which extracts mathematical results and references to mathematical results from scientific papers, using their PDF or L A T E X sources. We analyse the results of our algorithm on the whole arXiv database of scientific papers and explore the resulting graph of mathematical results, which contains more than 6 million results and 4.5 million edges. We present attempts to link theorems of different papers using a TFIDF vectorizer or an autoencoder.

Introduction

In the context of my IASD Master's degree, I did a 5-month internship with the VALDA Team of the École Normale Supérieure of Paris on the Theo-remKB project.

TheoremKB is a project started by my internship advisor Pierre Senellart. The goal of this project is to transform the way scientists navigate through scientific literature. Nowadays, access to scientific literature mainly relies on search engines and digital libraries like Google Scholar, DBLP, arXiv, or SemanticScholar. On these platforms, one can search for articles by keywords, authors, title, dates, and other metadata. For some of them, it is also possible to navigate through the graph of citations. However, the basic unit of information is always the paper itself, and the PDF associated with it.

But the actual unit of information of use by scientists in mathematical fields (mathematics, theoretical computer science) are the mathematical results (theorems, lemmas, claims, propositions, etc.) contained inside the paper, and how they rely on each other. The goal of TheoremKB is to turn the scientific literature from a collection of papers to a knowledge base of mathematical results.

Such a knowledge base could be very helpful to understand how different results rely on each other, or to explore classes of results (for instance, get all the known NP-Complete Problems). It can also help authors to understand the dependencies between the results contained in their own papers.

This internship and the one of Lucas Pluvinage, another IASD student, come at the beginning of the project. During this internship, our goal was to get a sense of how dense the knowledge base mentioned above could be, and to create a first connected graph of mathematical results. While I was more involved in finding a way to detect dependencies between mathematical results, Lucas' task was to extract the results and their proofs from the paper using information extraction techniques.

The remainder of this document is organized as follows. Section 2 introduces the existing literature that can be linked to the project, and the most advanced tools and platforms currently available to browse the graph of scientific publications. In Section 3, I describe the dataset of papers I used for my experiments during the first 4 months of the internship to test my scripts. My attempt at linking theorems from different papers based on the statement associated with them is described in Section 4. The main part of my work is detailed in Sections 5 and 6, respectively discussing the algorithm we built to find and link mathematical results together, and the analysis of how this algorithm performed on the whole arXiv database.

Finally, Section 7 presents some of the ideas that have been started during my internship, but remain unfinished and need to be pursued.

Related work

In this section, I will present the existing literature and tools on some topics related to the TheoremKB project and more precisely to my internship.

Information extraction from scientific publications

There as been a lot of research around the extraction of information from scholarly documents, mainly focused on two main areas of the document:

The header (title, authors, abstract, etc.) and citation data. A few other works have looked at the extraction of other parts of the document, such as figures [START_REF] Ray Choudhury | A figure search engine architecture for a chemistry digital library[END_REF] and tables [START_REF] Liu | Tableseer: Automatic table metadata extraction and searching in digital libraries[END_REF].

grobid [2,[START_REF] Lopez | Combining automatic bibliographic data recognition and term extraction for scholarship publications[END_REF], created by Patrick Lopez is a free, open-source tool implementing these techniques. It takes as input the PDF of a scientific article and returns an XML file of the paper parsed into different sections, with figures, tables and equations extracted. It relies on PDFAlto [START_REF]Pdfalto[END_REF], another useful tool created by the same team, which can read a PDF and return the position of every single word of the PDF.

This tool proved to be very powerful and useful for our task, even if it contains some bugs, like mathematical results often detected as figures. These bugs are probably due to the fact that grobid is trained on a very small corpus of paper, all of them being manually annotated.

Scientific digital libraries and open archives

An important part of the project is to create a knowledge base storing all relevant information about mathematical results and their proof, and more generally about scientific papers as a whole. It would then be essential to have a public web platform on which researchers could easily navigate, and query the graph of scientific papers and mathematical results. I will present in this section some existing digital libraries and open archives platform that enables to browse the graph of scholarly documents.

Digital libraries Nowadays, a large number of platforms give access to the scientific literature. Most of them are digital libraries. Often these services are directly provided by commercial publishers, such as Scopus (by Elsevier ), Web of Science (by Clarivate), or the ACM digital library. Some others are general search engines for scientific publication, the best known being Google Scholar and Microsoft Academic [START_REF] Shen | A web-scale system for scientific knowledge exploration[END_REF]. Some platforms are more independent digital libraries gathering data from various publishers on one platform (MathSciNet and EuDML for mathematics, DBLP for computer science, PubMed for biology and medicine, etc.).

All these platforms propose the PDF file of the publication, often metadata (authors, abstract, venue, etc.) and sometimes extract figures and tables from the paper, but they never allow to query directly the mathematical statements from the research article. Dissem.in is another digital library, gathering data from various platforms and showing precisely which papers are open access and which are not. It also proposes an API that enables to find the doi of a paper given its title and authors.

There also exist illegal digital libraries, such as Sci-Hub and Libgen, which give access to a lot of publication without any paywall. We only consider here sources with no legal issues.

Open archives. Another kind of platforms is open archives, on which authors can upload preprints and publications, together with the sources of these publications. For instance, the project HAL originated by the CNRS, counts more than 700, 000 hosted publications. The best known is arXiv, created by Cornell University, and it is the one we will use in this paper. It contains more than 1.7 million papers published on various scientific topics (mostly mathematics, computer science and physics) since 1991. We can also mix these open archives with a social network style and obtain something like ResearchGate, on which researchers can share and comment articles, as we comment posts on Facebook or LinkedIn.

The S2ORC dataset. SemanticScholar is one of the most interesting of these digital libraries. As a lot of them, it shows the metadata of the article, the references and the citations. They also automatically extract figures, tables and discussed topics in the paper, and they detect related papers. A very interesting feature of this platform is that it sorts references into three categories and automatically detect important references. I will talk about this in more detail in Section 2.3. The website counts more than 80 millions papers and propose a very useful API to easily get a JSON file of any article.

Even more interestingly, the Allen Institute for AI, at the origin of Se-manticScholar, released a huge dataset called s2orc [START_REF] Lo | 2orc: The semantic scholar open research corpus[END_REF]. This contextual citation graph contains 81.1 million academic publications and links between them, but it also contain the parsed full text of 8.1 million open access papers. For each publication, they gathered the different versions of the pa-Figure 1: An example of the webpage of one paper on a digital library, here SemanticScholar.

per and picked the most representative one. Then they used grobid [2,[START_REF] Lopez | Combining automatic bibliographic data recognition and term extraction for scholarship publications[END_REF] presented above to split the paper in different subsections and automatically extract figures, table, formulas from the text. As I explained before, grobid also extract references to these elements, and to bibliographical references. Finally, they simply linked papers in the bibliography of the PDF files to existing papers in their dataset. Figure 2 (and its caption), taken from [START_REF] Lo | 2orc: The semantic scholar open research corpus[END_REF], sums up the content of one document in the s2orc dataset. Other citation graphs. This is not the only contextual citation graph publicly available. In 2009 was published the ACL Anthology Network [START_REF] Radev | The acl anthology network corpus[END_REF], a graph of scientific publications derived from the ACL Anthology Corpus manually augmented in order to link references in the bibliography to papers in the corpus (it contains around 25K papers). More recently, Saier and Färber introduced a citation graph derived from arXiv L A T E X sources in [START_REF] Saier | Bibliometric-enhanced arxiv: A data set for paper-based and citation-based tasks[END_REF].

References extraction and analysis

Digital libraries and contextual citation graphs mentioned in the previous section are directly useful to researchers, but they are also useful indirectly by the mean of various research tasks. We can cite for instance the citation recommendation task for authors writing a paper, either at a paper level [START_REF] Bhagavatula | Content-based citation recommendation[END_REF][START_REF] Liu | Context-based collaborative filtering for citation recommendation[END_REF] or an in-line level, using the context [START_REF] Jeong | A context-aware citation recommendation model with bert and graph convolutional networks[END_REF]. It can also be used to detect similar papers in order to build a recommendation system (for instance, Microsoft Academic built is own recommendation system for scientific papers [START_REF] Kanakia | A scalable hybrid research paper recommender system for microsoft academic[END_REF]). There is also work done to generate citations in a realistic and natural sentence, given two papers (the first one citing the second one) [START_REF] Luu | Citation text generation[END_REF].

An interesting task for us is citation extraction and disambiguation. Cohan et al. proposed in [START_REF] Cohan | Structural scaffolds for citation intent classification in scientific publications[END_REF] a classifier trained to detect the intent of a citation, given the sentence in which the citation appears. They trained it to detect three main intents: Background (for instance, citations in this Section of this report are mainly background citation), Result (these citations are the one that we aim to extract with the algorithm presented in Section 5), and Method. To train their classifier, they used a bidirectional Long Short-Term Memory (LSTM) with an attention layer, followed by three independent Multi-Layer Perceptron (MLP) with three different tasks: one with the main task (guessing the citation intent) and the two other with scaffolds (e.g., guessing the title of the section from which the citation is taken).

Knowledge bases of theorems

Another aspect of this project is that we want a structured knowledge base of mathematical results. There has been work done on knowledge bases of Theorems in the context of automated theorem provers. Some of these tools include libraries which contain a lot of theorems and their proof. The model MBASE [START_REF] Kohlhase | Representing knowledge and context for the integration of mathematical software systems[END_REF] proposed in 2001 aimed to create a knowledge base with formal representations of mathematical results. However, the development of this model seems to have stopped in 2003, without any system demonstration.

Logipedia is one of the most complete library of mathematical results, put as formal statements and useable for a theorem prover. However, this is not as human-readable as TheoremKB aims to be.

Finally, Ganesalingam and Gowers proposed in [START_REF] Ganesalingam | A fully automatic problem solver with human-style output[END_REF] a very interesting automated model which aims to write proofs in a human style by trying to mimic researchers' reasoning.

3 The CS-CC database Dataset creation. For our experiments, we needed a dataset of scientific papers dense enough so that this is likely that there exist links between results from different papers. The dataset must not be too big so that our scripts will not take too much time. It is possible to download bulks of papers from arXiv (and other websites), but publications are sorted by months and paper published the same month are less likely to quote each other. Moreover, arXiv enables to download both the pdf and the source of papers published on it (when available).

Instead, I crawled the export version of the arXiv open archives, and downloaded all PDFs and sources of papers published between 2010 and 2020 under the cs-cc category. cs-cc stands for "Computer Science -Computational Complexity". We chose this category because computational complexity is a domain with a lot of mathematical results and close enough so that papers will very probably quote each other.

The dataset I gathered contains 6.000 scientific papers. The second step was to get the links between the different articles of the dataset. The first methods I used to obtain them were not very efficient, but it enabled me to familiarize myself with some useful tools.

Extracting citations. Indeed, I first tried to use the dissem.in API. If one sends the title of an article, the publication year and the names of the authors to the dissem.in API, the API returns the DOI of the paper. The idea was to find the DOI of each paper in the dataset, and then the DOI of articles in the bibliography of all papers in the dataset. Once this is done, we just have to find matching DOI. I tried to extract the information required by the API from the L A T E X source code, but, as it is shown in Table 1, more than one fifth of all papers in the dataset do not have any latex file in their sources (or no sources at all). Only 62% have a file for the bibliography, often a .bbl file, which is not ideal because its format can vary from one paper to another. Only 3% of them have a .bib file for the bibliography.

.bib .bbl .tex 3% 62% 78% Table 1: Proportion of papers in the cs-cc dataset having a .bib, .bbl or .tex file for the bibliography.

Instead of directly using the source files, I decided to extract information directly from the PDF using grobid [2,[START_REF] Lopez | Combining automatic bibliographic data recognition and term extraction for scholarship publications[END_REF], a tool based on machine learning and able to convert a PDF of a scientific paper into an XML file. The returned XML file contains structured information on the paper, such as header data (title, authors name, universities, etc.), body data (sections, figures, equations, references, etc.) and most importantly bibliographic data (title, authors name and publication year of each quoted article). However, the results were still not convincing since grobid and dissem.in have their own bugs.

Fortunately, the digital library SemanticScholar for scientific literature offers an API which only needs the arXiv id of an article and returns a detailed JSON file, containing the arXiv id of every arXiv paper referenced in the bibliography. Thanks to that, I was able to obtain a dense graph of articles and inter-articles dependencies.

Graph analysis. 2.666 of the 6.000 papers are cited by at least one other paper from the corpus, and 3.190 of them are citing another paper of the corpus. Figure 3 shows the distribution of the number of citations intracorpus per article. This highlights the fact that despite its small size, this dataset is dense enough for our experiments.

Theorem Matching

This section is dedicated to my first attempt at linking results from different papers. The idea was to detect pairs of theorems which are similar. For example, it would detect if a theorem in some paper is a restatement of a theorem in one of the papers referenced in the bibliography.

The theorem dataset

For each paper in the cs-cc dataset, I extracted the set of mathematical results and their statement from the L A T E X source files. To do so, I simply searched for each "\begin{theorem}" and "\end{theorem}" using the TexSoup library in Python. For each result, a regexp detects if some paper is cited inside the result and we check if the cited paper is in the cs-cc dataset. If a citation to another paper is detected, we use a classifier to find which particular results of the cited paper is associated to this one. The different classifiers we tried are described in the next sections. Note that we only consider citations which are inside the statement and not the ones which are in the proofs. Indeed, in the first case, it is more likely that the result is a restatement of a result in the cited paper, and in the second case, the result probably relies on a result of the cited paper.

A total of 41, 167 results were successfully extracted from L A T E X files and 308 theorems are referencing another paper from the corpus (there might be duplicates). To see the efficiency of the different methods, I selected a subset of 39 theorems in which the precise result of the cited paper we want to find is already mentioned in the header (e.g "Theorem 3 from [4]").

The remainder of this section is dedicated to the different methods I tried to detect the similarities between different theorems. The accuracy obtained with each method is reported in Table 2.

Method

Found Accuracy TF-IDF 29/39 75% Autoencoder 21/39 54% Table 2: Efficiency of the different methods for theorem matching.

TF-IDF Vectorizer

The first model I used is based on sentence vectorization using the frequencies of words in the sentence. I tried several vectorizers, but the most efficient one was the TDF-IDF Vectorizer. TF-IDF stands for Term Frequency Inverse Document Frequency. Given a set of documents D, we will associate to each pair of document and term (d i , t j ) a weight tfidf i,j . This weight is the product of two numbers:

1. Term frequency : tf i,j is equal to the number of occurrences of the term j in the document i.

Inverse Document Frequency :

The idf j of the term j is a function of the inverse of the proportion of documents containing this precise term. For instance, it can be the log of it.

idf j = log |D| |{d i | t j ∈ d i }|
Finally, we can compute the tfidf weight for the pair (d i , t j ) :

tfidf i,j = tf i,j × idf j
In our case, documents are theorems taken in a bunch of papers, containing the citing paper and the cited paper. We denote the set of theorems T . Terms are simply words appearing in these theorems. We can associate to each result a vector v(t i ) = (tfidf i,1 , • • • , tfdif 1,m ). I used the TfidfVectorizer function from sklearn library, with english stopwords, which directly returns the vector of every theorem given the set of theorems.

If we want to find the result in a paper P 2 ⊂ T which is the most similar to a result t 1 in a paper P 1 ⊂ T , we simply take

t max = arg max t 2 ∈P 2 v(t 1 ) t v(t 2 )
This method works well when the two theorems are almost identical. However, it gets complicated when there is a lot of theorems in the paper P 2 , since it is very likely that theorems of the same paper contain almost the same terms. I also tried to use N-grams instead of unique words, but the results were worse.

Autoencoder

I also tried to create my own theorem vectorizer using a variational autoencoder, inspired by [START_REF] Wei | A variational autoencoding approach for inducing cross-lingual word embeddings[END_REF][START_REF] Bowman | Generating sentences from a continuous space[END_REF] and more generally by autoencoders seen in class and in [START_REF] Rao | Natural Language Processing with Py-Torch: Build Intelligent Language Applications Using Deep Learning[END_REF].

First, I needed to choose embeddings for words. Given that all the theorems share a scientific vocabulary, I tried to train embeddings using the CBOW method [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]. However, this did not work well, maybe because the dataset of theorems was too small. I decided to use pretrained Glove embeddings [START_REF] Pennington | Global vectors for word representation[END_REF] with 50 dimensions and 6 billion tokens instead.

In our autoencoder, the input of the decoder is a fraction of the statement and the output of the encoder and it must guess the masked words of the sentence. The fraction of words which is masked is called the word dropout. For instance, when the word dropout is equal to 0.3 we obtain the loss shown in Figure 5. Both encoder and decoder use a GRU recurrent neural network. The best results I obtained using this method are not really convincing (see Table 2), and much worse than the results obtained using TF-IDF.

I did not investigate more on this topic, because the dataset was too small and it was hard to train a classifier in an unsupervised fashion.

The graph extraction algorithm

In this section, I will present the main algorithm of this internship, which enables us to create a first knowledge base of mathematical results. Our goal is to extract results, as well as references to external and internal mathematical results from PDF files of scholarly documents, and gather everything to form a knowledge base of theorems and proofs. By internal reference I mean a reference to another result of the same paper. On the contrary, an external reference is a reference to a result in another paper. This extraction is done in three steps:

1. First, we want to convert the PDF into an XML file containing the positions of the mathematical results and their proofs. One of Lucas Pluvinage's goals during this internship was to build a strong classifier to automatically extract results and proofs. He uses Conditional Random Fields (CRF) for that, for more information please see his report.

2. Now that we have boxes and we know they contain either results or proofs, we can extract information that interest us. This includes the name/number of each result (or name of the result proven if it is a proof) and more importantly, references to internal and external results. This part is harder than it looks and it was an important part of my work.

3. Finally, we need to associate external references to arXiv papers using their tags. By tag, I mean the number or word between brackets, associated to a paper in the bibliography (e.g, the tag of [START_REF] Lopez | Combining automatic bibliographic data recognition and term extraction for scholarship publications[END_REF] is "14").

However, since the CRF was not yet ready for the first step, we extracted results and their proofs directly from the L A T E X sources. I explain how we did it in more detail in Section 5.1. Figure 6 sums up the extraction algorithm.

Figure 6: Flow of the algorithm. We convert a PDF file into an XML file using a CRF, or we use source files for this purpose, then we extract results and references. Finally, we associate references with papers in the corpus to build a graph of mathematical results.

From PDF to XML

The first step is to detect the bounding box surrounding mathematical results and proofs. Lucas was working on a CRF, which would automatically detect these boxes from the PDF file. However, the classifier was not ready and we had L A T E X sources of a big proportion of the downloaded papers, so we thought of another solution. Indeed, we use a L A T E X extension script to set a hyperlink around results and proofs during compilation. For more information on how this is done, please see Lucas' thesis.

Consequently, we need to compile the L A T E X source code ourselves. Once compiled, we obtain a new PDF with bounding box around results and proofs, which are actually false hyperlinks. Using PDFAlto created by Patrice Lopez [START_REF]Pdfalto[END_REF] on the new PDF, we were able to extract the position of every hyperlink box, together with the address they are pointing to. Therefore, with a simple regexp, we can easily extract the full statement of the mathematical results and proofs of the document.

Extracting results

Reading the boxes. At this point, we know the position of hyperlink boxes. We can divide this boxes into two groups: Results and Links. Result boxes contain either a mathematical result or a proof, and Link boxes are external links or internal hyperlinks to other elements of the article (e.g., "Section 6", "Figure 12" or "[2]").

For each word of the document, we will detect if it is inside a Result box and inside a Link box (it can also be in both, none, and even nested Result boxes). Checking it naively for each word and each box would take a lot of time (number of words × number of boxes). KD-Tree is a really good data structure for this problem.

KD-Trees

In a KD-Tree [START_REF] Bentley | Multidimensional binary search trees used for associative searching[END_REF], the space is partitioned to efficiently store the points and find them afterwards. Thanks to this optimization, the running time of this second step was divided by a factor 10.

However, KD-Trees store points, not boxes. My algorithm memorizes one KD-Tree for each page of the PDF and each kind of box (Link and Result), containing the center of every box of the page. Then, if we want to know if a word is in a box, we search for the nearest boxes of one word in the selected KD-Tree and check if the word is inside one of these neighbouring boxes.

Extract results. Now that we have extracted the full text of results and proof, we would like to be able to enumerate results and references of the document. First of all, we use the regexp 1 to detect the name of a mathematical result, which is any word followed by some kind of numbering (e.g., "Theorem 1", "Lemma 2.54.2", "Problem A.1", "Conjecture E.7.1.3.1", etc.). However, this script will fail if the result is not numbered or is preceded by some character in the text.

(\w+)\s+([a-z]\.)?[\d]+(\.\d+)*

(1)

We also need to associate each proof to one result. For that, there are two possibilities:

1. It is specified at the beginning of the proof which result is being proven, and a simple regexp is sufficient to find it.

2. There is no indication. In that case, we associate the proof to the most recent mathematical result seen in the document.

However, I believe that this part can be improved. For instance, there is sometimes a Remark between a Theorem and its Proof. In that case, my algorithm will associate the proof to the Remark if the authors did not specify anything in the proof.

Extract reference. Now, we want to find references to other mathematical results. For this we are searching for particular words followed by some kind of numbering in the text. To this list I added some abbreviations, like "thm.", "lem." and "prop.". However, a lot of abbreviations are still not detected, as explained in Section 6.3.

Once we have detected a reference to a result, we need to determine whether this is an internal or an external reference:

• If a hyperlink is detected on the result name or numbering, and this hyperlink points to a result inside the paper, then it is obviously an internal reference to this result.

• If there is no such hyperlink, we look at the context of the reference. By context, I mean the 5 words before and after the reference to the mathematical result. If a link to the bibliography is detected inside the context, we know it is a reference to an external result. Otherwise, we say by default that it is a reference to an internal result.

For external references, we memorize the tag detected in the context (e.g. " [START_REF] Bhagavatula | Content-based citation recommendation[END_REF]" or "[MoI17]") so we can find if the cited paper is in the arXiv corpus later (see more details in Section 5.3).

There are some other optimizations in this small part of the algorithm:

• We skip the reference if this is a reference to the result itself (for instance, "this concludes the proof of Theorem 3").

• If there are several links to the bibliography in the context, we take the closest one.

• We detect when several results are referenced at the same time, and separate them into several references. For instance, "Theorems 1, 5 from [2]" becomes "Theorem 1 from [2]" and "Theorem 5 from [2]".

• Sometimes, references to results or sections of the paper (e.g., "See Section 5.3") are falsely detected as links to the bibliography. We automatically remove these links by detecting if they contain particular words, like those in Table 3.

Associate tags and papers

The last step is to associate the tags found in the previous steps to other arXiv papers. For that, we want to know which tag is associated to which paper, and it requires a bit of work.

Using S2ORC. I downloaded the latest version of the s2orc dataset [START_REF] Lo | 2orc: The semantic scholar open research corpus[END_REF] from SemanticScholar, that I already presented in Section 2. The interesting feature of this dataset for us here is that SemanticScholar already found the arXiv id of every reference in the bibliography, when such an id exists (i.e. the paper must exists on arXiv). Moreover, since the s2orc also uses grobid to parse the text of the document, in-line bibliographical references in the text are already extracted, and so are tags associated with these references. Consequently, using only this dataset, I was able to obtain the tag associated with each arXiv reference in the bibliography of every document of my dataset. This gives us a first dataset of links between arXiv papers. Let's call it dataset A.

However, I was frequently faced with a particular issue: The version of the paper I had in the cs-cc dataset (See Section 3) or in the full arXiv dataset (see Section 6) was not the same as the version used by SemanticScholar in their s2orc dataset. This has unwanted consequences: sometimes tags in the bibliography are simply different, for instance tags are numbers in one version ([1], [2], etc.) and names in another one ([VK16], [Hi15], etc.), but sometimes the order of the references is changed. By that I mean that the tag [1] might not represent the same reference in our paper and in the paper used in the s2orc dataset. As one can see on Figure 12 of Section 6, this kind of problem is frequent enough to require fixing.

Using GROBID. My solution is pretty simple and can be explained as follows. Using grobid, we can extract the list of bibliographical references (mainly, the title of each reference) and associate each one to a tag in the article. This creates a second dataset, let's call it dataset B. Unlike the first dataset, this dataset contains the title of the references, but not their arXiv id. Now, we want to merge the two datasets and delete rows of the dataset A which referred to an older version of the paper. For that, we can treat each paper separately.

For each paper, we look at bibliographical references that have the same title in both dataset and compare their tags in each dataset. If the dataset A and B agree on at least 75% of the tags, then we keep references extracted with s2orc. Otherwise, we remove the references extracted with s2orc and simply keep the arXiv ids and associate them to references extracted with grobid which have the same title.

Results on the CS-CC dataset

Now that all the steps of the algorithm presented in Figure 6 are ready, we can combine them together and check the results of the algorithm on our datasets. It is interesting to see that every step can be parallelized since we work on every paper independently at each step of the algorithm.

The CS-CC dataset. We first tested our algorithm on the cs-cc dataset presented in Section 3. As a reminder, there are 6.000 papers on this dataset, published between 2010 and 2020 under the category "Computational Complexity" of arXiv. A bit less than 4.500 papers passed the first step (L A T E X compilation and box extraction) without errors. 3.976 papers contain at least one result and 3.338 of them contain references (either internal or external).

With the fully optimized algorithm, we obtained 82.637 results and 46.578 references. Among these references, only 3.960 are references to external results. Only 829 of the references results are found, scattered between 515 papers.

The extended dataset. Before using the algorithm on the whole arXiv dataset (see Section 6), I wanted to see what happened if we iteratively extend the dataset by adding referenced papers to it. I started with the results of the cs-cc 6.000 paper dataset and filtered all papers which were neither at the source nor the target of any external reference. Then, I add every referenced paper which was not already in the dataset, run the algorithm again, and repeat the operation. After only 5 iterations, there were no new papers to add to the dataset (see Figure 7). From the 6.000 papers of our dataset, I only selected the ones involved in an external reference (either as a source or a target). Even after this pruning step, some of these papers are lost due to parsing error when running the script. This gives us less than 700 papers before the first iteration.

The dataset obtained at the end of the fifth iteration contains 829 papers, around 21.000 results and 14.600 references, with 2.103 external references and 884 of them with an arXiv paper as a target.

I was a bit disappointed that so few papers were added to the dataset. The problem probably comes from the fact that we can only add older and older papers, because they need to be in the bibliography of some document in the dataset.

Qualitative analysis. Finally, I did a qualitative analysis of 20 papers randomly selected in the cs-cc dataset. This helped me find some bugs and update the algorithm. It improved the quality of the results by ∼ 10% (for instance, the number of references found for the extended dataset went from 884 to 962).

The arXiv Database

Now that we were confident in the theorem and reference extraction script, we want to test it on the largest possible database. We chose to run the script on the whole arXiv database. This section describes the specific aspects of the dataset used, as well as the quantitative and qualitative analysis of the results of the script.

The dataset

We downloaded the whole arXiv database (pdf and sources). This dataset contains almost 1.7 million papers and weighs around 2.8 TB. However, it would have taken too long to run our script on so many papers and a lot of them do not even contain any mathematical result. Consequently, we removed every paper which do not contains either a Theorem, a Lemma or a Proposition. I did that using pdf2text and a simple regular expression.

After this pruning step, we obtain around 500.000 papers and we are pretty convinced they all contain mathematical results. Figure 8 highlights that the number of papers published each year is increasing exponentially. We can also see that the proportion of scientific papers containing mathematical results is increasing until 2015. My hypothesis is that it is due to the rise of deep learning, with few theoretical papers. I ran the extraction scripts on these 500.000 papers on the RIOC cluster of the Inria (which is not easy, given that all the software installed on the cluster is outdated). It took around one week to complete. The next subsections detail the quantitative and qualitative analysis of the results obtained.

Quantitative Analysis

Results and Reference. We obtained a total of 6.218.044 mathematical results, which corresponds to more than 12 results per papers. Moreover, the algorithms counted 4.511.850 references to mathematical results, including 848.124 external references. The number of results and references for each month logically increases with the number of papers (see Figure 9).

An external reference is external to the paper in which it appears, not external to the arXiv dataset. Citations and Tags. In parallel, the script to detect bibliographical references and to associate tags with them returned 2.215.465 citations to other arXiv papers (∼ 5 per paper), and only 1.808.775 were successfully associated to some tag (∼ 81%). Again, we can see on Figure 11 that these numbers increase with the number of papers.

Moreover, I measured the confidence indicator for each month. It corresponds to the proportion of papers for which the dataset A, extracted from the s2orc dataset of SemanticScholar, and the dataset B, extracted using grobid, agree on at least 50% of the bibliography. We say that they agree on a reference in the bibliography if both datasets associate the same tag to this reference. This indicator swings between 85% and 95% of confidence (see Figure 12). Link results together. If we try to find the papers referenced in the external references (the green line in Figure 8) using the association between tags and bibliographical references described above (Figure 11), only 176.675 of these papers were successfully retrieved.

This corresponds to 21% of the total. In 23% of the cases, there are no arXiv papers detected in the bibliographical references of the paper. In the remaining 56% of the case, the tag of the reference is not associated to any arXiv paper. Among these 176.675 results referenced and for which we found the supposed paper from which they are taken from, only 89.538 of these results were found inside the paper (nearly half of them). For the other half which are not found, some results come from papers which were not successfully parsed (see below about errors during extraction) and some It is interesting to see that there is not a mega-component with more than 80% of the nodes, but actually a lot of micro-components with less than 10 papers, which is rarely the case in large graphs like this Figure 13: Total number of external references, the proportion of them having a bibliography in our citation dataset, the proportion having the reference tag inside this dataset, the proportion for which the paper containing the referenced paper is found and finally, the proportion for which the exact referenced result is found. These proportions are shown for each month between 1991 and 2020.

one. An explanation can be that we are missing a large amount of papers that are not on arXiv.

The longest path (when cycles are removed) in this graph is of length 13. However, since it is the aggregated version of the graph, we cannot be sure that the results used as the target of the arriving edge is the same than the results used as the source of the leaving edge. A representation of the biggest connected component with a Gaussian noise is shown in Figure 14.

I also looked at which results are cited in the biggest number of papers and I found that Proposition 5.3 of "Stability Conditions On Triangulated Categories" by Tom Bridgeland and Lemma 5.2 of "Introduction to the non-asymptotic analysis of random matrices" by Roman Vershynin are ex-aequo with 18 citations each. These superstars of mathematical results are shown in Figure 15.

The non-aggregated version of the graph also includes the internal references (i.e., links between the results of the same paper). It contains 2.889.376 nodes and 3.001.424 edges. 97% of these edges are internal references.

However, there is nothing very interesting to say about this graph, because it is very big but mostly very sparse. Indeed, most results only cite results from the same paper. For instance, the biggest connected component contains less than 2% of all the nodes. Moreover, the longest paths Errors and bugs. We also dived into the different kinds of errors raised by the first two steps of the script (The L A T E X compilation and the box extraction using PDFAlto). Figure 16 highlights that the script failed a lot on old papers and the success rate of the script stabilizes at 80% for paper published after 2010.

As one can see on Figure 17, most errors from old papers are due to the script not finding the main file, probably because the version of L A T E X used in the source is not compatible with the version of L A T E X used in our script. Errors on more recent papers are mostly due to PDFAlto (during the box extraction), or because of too many errors during the L A T E X compilation. Table 5 sums up the sources of errors of these first two parts of the script. Please note that this part of the script has been improved since then and the success rate might be higher with the new version.

Success

Qualitative Analysis

The previous subsection gave us an insight of the number of results and references found by the algorithm described in Section 5, and it would be interesting to see the actual quality of these results. To get a sense of it, I randomly selected 25 papers and compared the output of the algorithm and the desired output, that I gathered manually. To do so, I read these papers, and I noted down every mathematical results and references to mathematical results for each paper. This is the same process that I used in Section 5 to detect which part of the script could be optimized. Table 6 sums up the conclusion of this analysis for references.

Mathematical results. There was a total of 604 mathematical results expected (around 24 per paper on average) and 90.6% of them were successfully retrieved. 6.1% were found but their name was not detected. In most cases, we can change the script so that it can detect the name. For instance, if a result has no number (e.g "Remark" instead of "Remark 2.3"), the name is not detected. Finally, the remaining 3.3% are simply not detected, probably because the authors used a specific command for these results, which is not detected by our L A T E X extraction script. The only false positives for mathematical results are "Case X" and "Step X" and whether or not these are false positives is up to debate. However, the errors for mathematical results are not really interesting to me because they come from parts of the script I did not work on. Let us now discuss the quality of the extracted references.

References. The analysis of references does not take account 2 of the 25 papers, because I could not reproduce on my own computer the results obtained on Inria clusters. I obtained 345 internal references (15 per paper on average) and 49 external references (around 2 per paper on average).

The success rate for internal (resp. external) references is of 81% (resp. 65%). A lot of errors cannot be solved by my script and are either caused by the structure of the paper (e.g., nested proofs) or by errors during L A T E X compilation. For instance, Proofs are sometimes not detected. Some errors are direct consequences of errors on results. For instance, if the name of a result is not detected, we cannot associate a source name to the reference. Another frequent error is that the authors used an abbreviation or misspelled a word (for instance, "Therem 5" or "Cor. 2.1"). I believe that these cases could be solved with some work.

Finally, there are some errors that are unique to a paper and can be solved easily. The following errors are gathered as "Other errors":

• Some kinds of results are missing on Table 3, namely Algorithms and Assumptions.

• Sometimes, the hyperlink associated to a reference to an internal result does not point precisely into the box containing the result, causing a bug.

• The algorithm can detect "Lemmas 2 and 3" and change it as "Lemma 2 and Lemma 3" but cannot deal with "Lemmas 2, 3 and 4" (it does not detect Lemma 4 ).

I also obtained false positives. Sometimes, internal references are detected as external ones, or parts of the text are wrongfully detected as proof or results. This caused us to have 29 (resp. 9) false positives for internal references (resp. external references). The precision is specified in Table 6

Future work

This last section presents some of the other works I started during my internship. Some of these ideas are directly related to the extraction algorithm presented in Section 5, and others are independent of it.

Improve the extraction algorithm

The qualitative analysis detailed in Section 6.3 gave me a lot of ideas to improve the quality of the results. First of all, there is the correction of all the bugs mentioned above, but also the optimization of the references detection algorithm. Lucas is also working on improving his part of the script. Moreover, we will need to find a new way to detect references when we will use CRF to extract results directly from the PDF instead of the source files. For instance, a lot of papers do not actually use the hyperref L A T E X library. Therefore, since the detection of external references is based on hyperlinks to the bibliography, we will not be able to detect them anymore. However, this can be solved by simply detecting tags and brackets instead of links. For instance, we can train a classifier to help detecting references to the bibliography (an hybrid between grobid and our current code).

The algorithm is currently able to detect references which are in the result statement and the text of the proof, but not in the rest of the document. This represents a big loss of information, and it might be useful to find a way to extract them too, but also to be able to associate them to some result in the paper, using information extraction techniques.

Crowdsourcing

During this internship, I also created some crowdsourcing tools in order to label various datasets. In this section, I will quickly present the three tools I built and their goal. Each crowdsourcing platform is divided in two parts: the part in which the user answer the questions, and another part displaying the PDF of the paper, such that the user can directly check inside in case of doubt.

Contextual citations. The goal of the first tool was to gather enough data to train a classifier to detect the nature of a reference to a mathematical result, given its context. The idea was to do a work similar to that in [START_REF] Cohan | Structural scaffolds for citation intent classification in scientific publications[END_REF], but for mathematical results instead of article citations. I divided the possible nature of the reference into 3 categories:

• Statement : When the result is defined. "Theorem 1 : If a = 6 and b = 7..."

• Use : When the result is used, for instance in a proof. "...then by Lemma 5 we can conclude..."

• Discuss : When the result is not used but only mentioned, for instance as a comparison. "...the difference with Proposition 3 is that..."

However, I stopped working on reference disambiguation, and I did not have time to use the collected data to build a classifier.

External or Internal? Another classifier I wanted to train was a classifier able to detect, given a sentence containing a reference to a mathematical result and a tag, whether the reference is associated to the tag or not. Then, I could use it in the algorithm presented in Section 5 to disambiguate internal and external references.

For instance, in the sentence "We use Theorem 3 from [START_REF] Bentley | Multidimensional binary search trees used for associative searching[END_REF] and Lemma 1.5 to...", "Theorem 3" should be associated to " [START_REF] Bentley | Multidimensional binary search trees used for associative searching[END_REF]", but not "Lemma 1.5", since it is an internal reference. The tool I created shows a sentence with the reference and the tag highlighted, and asks if it is indeed an external reference.

After I went through several hundred sentences myself with this tool, I obtained a recall of 91%. In the end, I did not use it to train a classifier, but it helped me a lot to understand how to eliminate some false positives directly by modifying the code.

Parsing of definitions. One of the goals of TheoremKB is to create a structured knowledge base. Consequently, there is some work to be done on the results of the algorithm presented in Section 6 before we can obtain the right data structure. Ideally, we want a structure close to what already exists for other knowledge bases, like Wikidata or DBpedia, such that it is easy to navigate inside the graph of results, but also user-friendly, such that information are easily accessible and understandable to anyone.

For instance, it would be great to be able to extract what is defined in a Definition, so we can detect further references to it. The goal of this last tool is actually to manually gather a dataset with data and labels, and build an NLP classifier able to automatically detect what is defined in a Definition.

The user simply has to highlight which part of the definition he thinks correspond to the defined term. I still think it can be useful, but the information on what is defined is often unclear and insufficient: there can be several terms defined in one definition, it can also be a notation and not a definition.

Conclusion

Finally, after these 6 months of this internship, we successfully extracted a connected graph of mathematical results from the whole arXiv corpus of scientific publications. I optimized the extraction algorithm such that it takes the less possible time (for instance, by using KD-Trees) and return results with the best precision and recall (by dealing with the various particular cases). During this project, I also implemented some neural networks and other NLP methods for Theorem matching.

Thanks to this project, I also familiarized myself with a large family of tools for information extraction and retrieval in scholarly documents: GRO-BID,PDFAlto, the dissem.in API, the SemanticScholar API and a lot of python libraries (TexSoup, KDTree, etc.). Some of the work done during this internship can be found on our Github repository [1]. It mainly contains the code of the algorithm presented in Section 5.

This internship is one of the first steps of the much bigger and promising TheoremKB project. Hopefully, it will be helpful to have a first idea of the density of the graph of mathematical results, even with simple methods such that those we are using for the moment.

Figure 2 : [ 14 ]

 214 Figure 2: [14] For each paper, in-line citations are linked to the corresponding bibliography entry within that paper. The bibliography entries are then linked to one of 81.1M candidate papers. References to figures and tables are also extracted from the full text and linked to figure and table objects within the paper.

Figure 3 :

 3 Figure 3: Distributions of the number of citations from (left panel) and to (right panel) other papers of the corpus.

Figure 4

 4 shows a representation of this component obtained using the Net-workX library on Python.

Figure 4 :

 4 Figure 4: Representation of the graph of dependencies between papers of the corpus. The x-coordinate of a point represents its publication date (between 2009 and 2020) and the y-coordinate represents its number of citations from papers in the corpus.

Figure 5 :

 5 Figure 5: Evolution of the training loss and validation loss of the autoencoder with a word dropout of 0.3.

Figure 7 :

 7 Figure 7: Evolution of the number of external references to other arXiv papers after each iteration.

Figure 8 :

 8 Figure 8: Evolution of the number of papers on arXiv published each month since 1991 (left panel ) and the proportion of them containing mathematical results (right panel ).

Figure 9 :

 9 Figure 9: Evolution of the number of results and references to mathematical results in arXiv papers for each month since 1991.

Figure 10

 10 Figure 10 shows the proportion of each kind of mathematical results for each month since 1991. Without surprise, Theorem, Lemma and Proposition come first. More surprisingly, Examples are less present. There is no notable change of the proportion of each kind of result with time.

Figure 10 :

 10 Figure 10: Evolution of the proportion of each kind of results.

Figure 11 :

 11 Figure 11: Evolution of the number of arXiv papers found in the bibliography of papers published each month.

Figure 12 :

 12 Figure 12: Confidence indicator for bibliographical results since 1991.

Figure 14 :

 14 Figure 14: Representation of the biggest connected component of the aggregated graph using Networkx spectral layout.

Figure 15 :

 15 Figure 15: Mathematical results cited by the highest number of paper in our database.

Figure 16 :

 16 Figure 16: Success rate of the first step of the script from 1991 to 2020.

Figure 17 :

 17 Figure 17: Proportion of the kinds of errors for the first step of the script from 1991 to 2020.

Figure 18 :

 18 Figure 18: Screenshots of my crowdsourcing projects. Links : Contextual citations, External or Internal ?, Parsing of definitions.

Table 3 :

 3 The words of interest are the following: List of all the possible mathematical result names used to detect references.

	Theorem	Claim	Conjecture Corollary	Definition
	Lemma Proposition Observation Property Construction
	Example	Exercise	Note	Problem	Case
	Question	Solution	Remark	Fact	Hypothesis

Table 4 and

 4 Figure 13 sum up all these results.

	Internal references	3.663.726
	External references	848.124 100%
	No bibliography	202.872 23%
	Tag not in bibliography	468.554 56%
	Papers referenced not parsed	51.481	6%
	Result not found	35.656	4%
	Result found	89.538 11%
	References (Total)	4.511.850

Table 4 :

 4 Summary of the number of references extractedThe graph of results. With the 89.538 references between two mathematical results, I was able to build a dependency graph of mathematical results, and a more simple version of the graph by aggregating results of the same paper together.The aggregated version of the graph contains 89.128 paper nodes, and 85.222 reference edges. The graph contains 17.598 connected components and the biggest one has size 35.594, which represents around 40% of the papers nodes.

Table 5 :

 5 Summary of the errors obtained during the first step of the script.

	Finally, among the 346, 040 successfully parsed papers, 315, 434 (91%) of
	them contain mathematical results and 258, 235 (75%) contain references.

Table 6 :

 6 . Qualitative analysis of internal and external references.

		Internal ref. External ref.
	Recall	81.4%	65.3%
	Not named	2.0%	8.2%
	Not detected	16.6%	26.5%
	L A T E X failed	6.7%	6.1%
	Bad structure	3.2%	4.1%
	Abbreviation	0.2%	14.3%
	Other errors	6.5%	2.0%
	Precision	92.2%	84.5%

Aknowledgments

I learnt a lot during this internship, and I want to thank my advisor, Prof. Pierre Senellart, for guiding me during these five months. I also want to thank Lucas Pluvinage, which always had great ideas when I was stuck on the project. Finally, thanks to my friends and my advisor for helping me proofreading this report.-