Victoria Callet
email: victoria.callet@math.unistra.fr

Persistent Homology on Musical Bars

Keywords: Musical bars, Filtered complex, Persistent homology, Barcodes, Musical analysis

This article presents a new way of building a filtered simplicial complex from a music piece and applying persistent homology in the context of musical analysis. Our approach consists of considering any musical score as the set of its musical bars, which we see as subsets of R 3 . With this definition, we may consider the Hausdorff distance between two musical bars, which gives us a point cloud from any score, and that allows us to build the associated Vietoris-Rips complex. We will then use barcodes to visualize persistent homology and give an illustration of our construction on a famous movie music piece.

Introduction

Persistent Homology

A filtration of a simplicial complex K is a nested sequence of sub-complexes ∅ = K -1 ⊂ K 0 ⊂ . . . ⊂ K N = K of K: we call K a filtered complex. A simple filtration is presented for instance in figure 1. Starting from a filtered complex, we can compute its simplicial homology (over F 2 ) at each time of the filtration, and persistent homology gives information about inclusions between the various complexes, as explained in [START_REF] Zomorodian | Computing persistent homology[END_REF]. The associated homology groups H * (K s ) are characterized by their dimensions, which are called the Betti numbers. We can visualize persistent homology on a figure called a barcode, where the horizontal axis represents progress in the filtration and a bar that starts at time s and ends at time t corresponds to a generator of H * (K s ) that is still one for H * (K t-1 ) but not anymore at time t (see [START_REF] Ghrist | Barcodes: the persistent topology of data[END_REF]). Barcodes allow us to immediately identify classes that persist during the filtration. For instance, barcodes associated to the filtered complex in figure 1 are presented in figure 2, in degrees 0 and 1. Persistent homology is commonly used in a recent field known as Topological Data Analysis (TDA), where the main idea is to use topological structures to study and sometimes "recognize" some objects. The general process is summarized in figure 3, where in our context the starting object is a musical score.

Starting object

Point cloud Filtered complex Barcodes Shape recognition Fig. 3: Topological Data Analysis process.

Context and Problematic

TDA via persistent homology was already used in the context of musical analysis and especially in automatic classification of musical style, as we can see in [START_REF] Sethares | Topology of musical data[END_REF], [START_REF] Bergomi | Towards a topological fingerprint of music[END_REF], [START_REF] Bigo | Filtration of pitch-class sets complexes[END_REF] or [START_REF] Bergomi | Homological persistence in time series: an application to music classification[END_REF]. The main and common issue is always the following problematic: how should we associate a filtered complex with a given musical piece?

The papers we cited above have different approaches to try to find a consistent answer to this question, using pitch-class sets complexes, time series or the Tonnetz. In our work, we chose to consider that a music piece can be represented by a set of distinct musical bars, which can be thought of as subsets of R 3 , as defined in section 2.2. Our starting object is a musical score, from which we will extract a point cloud of R 3 by considering its musical bars and the associated Hausdorff distance, as presented in section 2.3. To build the filtration, we will use the Vietoris-Rips method, which is described in section 2.1. Once we obtain a filtered complex, we can compute persistent homology, i.e. barcodes. In section 3, we will illustrate our approach by giving an analysis of the French movie musical piece: Comptine d'un autre été: l'Après-midi, by composer Yann Tiersen. In order to allow musical interpretation, we will be interested in dimension 0 and 1 initially, and we will show that these barcodes, especially in degree 0, can capture structural information about the piece.

Persistent Homology on Musical Bars

Filtration: the Vietoris-Rips Method

The basic object of a Vietoris-Rips complex is a point cloud, that is, a set of data points with a metric over it. Definition 1. Let X = {x 1 , . . . , x n } be a point cloud and ≥ 0 be a parameter. The Vietoris-Rips complex R (X) is the simplicial complex with X as its set of vertices and σ = {x 1 , . . . , x k } is a k-simplex if and only if its vertices are pairwise close, that is, when d(x i , x j ) ≤ for all pairs x i , x j of σ.

Figure 4 shows the classical construction of a Vietoris-Rips complex starting from a given point cloud X and a parameter . For two given parameters and such as < , there is the obvious inclusion R (X) → R (X) and so by simply increasing the parameter , we will get a natural sequence of nested simplicial complexes, that is, a filtered complex. For instance, R 0 (X) is the 0-dimensional simplicial complex with n connected components while R (X) will have only one connected component for large enough.

• • • • • • • • • • • • • • • • • • • • • • • •
Fig. 4: The Vietoris-Rips method.

Remark 1. For all ≥ 0, we have a simplicial complex, but in practice we will discretize time by choosing a finite number of values for . In this paper, we choose to work with all values of the form = tρ, where t ∈ {0, 1, . . . , 100} and ρ is a fixed constant, so all our filtrations and also barcodes will be at the same scale, as we will see in section 2.3.

Musical Bars of a Score

The main idea of our construction is to consider that a musical score S is simply a set of its musical bars: S = {B 1 , B 1 , . . . , B n }, where the indexing corresponds to the musical flow. In this context, our definition of a bar is the following one: Definition 2. A musical bar is a finite subset B of R 3 where an element of B is called a note characterized by three coordinates:

the position, which refers to its place in the bar the duration, expressed in beats the pitch, which is the value of the note in term of its fundamental frequency Example 1. Here is an example of a musical bar coding. The first two coordinates are determined by using the meter and the third is coded in midicent, which is related to the position of the note in a piano. Notice that if there are some rests in the bar, we can ignore them because the information is already encoded in the position of the next note. In our case, each musical bar of a score S = {B 1 , . . . , B n } is a subset of R 3 so we may naturally consider the Hausdorff distance between B i and B j for any i, j. Following up on remark 1, we will consider the maximal distance d max of all distances for a given score S = {B 1 , . . . , B n } and define ρ = dmax 100 as the precision we want to work with. We then consider for each t ∈ {0, 1, . . . , 100} the associated Vietoris-Rips complex R tρ (S). Furthermore, instead of speaking about "time t of the filtration", we will now say that we look at the filtration with an error margin of t%. Indeed, we think of the presence of an edge between two musical bars as an indication that they are "similar", and the parameter t controls the way in which we choose to make this rigorous; for a small value of t, there are few edges which means that bars are finely distinguished, while for t large enough we allow coarser identifications.

q 4 4 E § c ¨ § B = {(0, 1/2,
We now have a point cloud from any musical score, so we have defined a new way to associate a filtered complex with a score by considering the Vietoris-Rips complex of section 2.1. Thus, we are now able to compute persistent homology (barcodes) and the next section shows an example of a musical analysis using this approach. Recall the filtration and barcodes from figures 1 and 2 respectively; they correspond to the method we just defined applied to the little score with 4 distinct bars of figure 6, which was built arbitrarily to test it. q R 4 4

B1 ¨ § § § B2 § ¦ ¦ § V § § V § @ § B3 ¨ § § § B4 § Ô Ô § ¢ § § ¢ § E §
Fig. 6: The score from which we build filtration and barcodes of figures 1 and 2.

Application: Analysis of a Musical Piece

The idea is now to test our approach on a music piece to see in which way it is useful in the context of musical analysis. As we have already mentioned in the introduction, we are interested in the barcodes in degree 0 and 1. For computing homologies, Betti numbers and barcodes, we used our own algorithms and programs written with the help of the SageMath system (based on Python).

The musical piece we choose to analyse is taken from the soundtrack of the French movie Le Fabuleux Destin d'Amélie Poulain, directed by Jean-Pierre Jeunet (2001). The music is the famous Comptine d'un autre été: l'Après-midi for piano, composed and played by the minimalist composer Yann Tiersen (2001). The version we took for our analysis is extracted from [START_REF] Tiersen | Comptine d'un autre été : L'après midi[END_REF].

The score has 53 muscial bars, some of which are repeated so it only contains 39 distinct bars, and it is split in two parts: actually, the music has 3 different themes in the first part that are played again one octave higher in the second one. These 3 themes and their respective structures are presented in figure 7. Notice that there are some repetitions of these themes in the original score, but here we suppress them in order to work with distinct musical bars only. Moreover, all the melody is constructed over 4 musical bars that are repeated and which constitute the musical accompaniment from figure 8. Let us look at the barcode in degree 0 from figure 9: there are several levels of analysis depending of the error margin we choose to take, and the main idea of persistent homology is to focus on the largest bars (those which persist), while the smallest ones can be considered as noise. In our case, there are 2 bars that stand out when we take an error margin larger than 21%, that means that the corresponding complex has only 2 connected components. One of them corresponds to the last musical bar B 39 of the score, which only consists of the final chord Em played with whole notes, and the other is a large dimensional complex where all the musical bars are connected together. This first analysis shows that the barcode in degree 0 separates the end from the rest of the piece, which is a start. For t% with t ≤ 8, there are only small bars so we ignore them as noise. Between 8% and 21%, there are 5, 6 or 7 classes that seem to last and more precisely, we found that with an error margin of 14%, the associated complex in the filtration looks like in figure 10, which is really remarkable: actually, there are 6 connected components and if we look at the vertices, we see that each one corresponds to a theme of the song, except for B 8 and B 28 which have a slight different structure than the rest of the first theme. We may then conclude that, with a well-chosen error margin, the barcode in degree 0 captures the structure of this piece by separating its different themes in the associated complex.

q R 4 4 First theme c § Î Î Î Î § § § § ¨c § § § § § ¨c § Î Î Î Î § § § ¡ ¡ § ¨c § Î Î Î Î § qR 4 4 Second theme § E § ¢ § ¢ F § d § E § ¢ § ¢ F § d § E § ¢ § ¢ F § d § E § ¢ § ¢ A § d q R 4 4 Third theme § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § §
4 4 Accompaniment § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § § §
On the other hand, barcode in degree 1 displays 3 different one-dimensional cycles that are presented in figure 11. Note that some edges of these cycles linked musical bars of one given theme to the same one octave higher, but not systematically and for now we are not able to interpret these cycles musically.

Conclusion and prospect

This paper has presented a new method to provide a filtered complex associated with a musical piece. Moreover, our construction reveals interesting results on the piece we chose to study. Actually, it shows that there is a way to capture the global structure of the piece by using barcodes in degree 0. In contrast, barcodes in degree 1 did not display an obvious musical interpretation, and we plan to focus on this dimension in our future work. In fact, one-dimensional cycles could be related to repeating patterns or musical loops in the score, and we are working on highlighting this interpretation from our construction. We also plan to apply our approach to a more general and diverse corpus of music data to see in which way it can capture the global structure of a music piece.

Fig. 1 :

 1 Fig.1: A filtered complex with 6 times of filtration.

Fig. 2 :

 2 Fig. 2: Barcodes for filtration of figure 1 in degree 0 (left) and degree 1 (right).

dFig. 5 :

 5 Fig. 5: Calculation of the Hausdorff distance between two metric spaces X (the ellipse) and Y (the square).

Definition 3 .

 3 Let B i and B j be two musical bars. The Hausdorff distance d H between B i and B j is defined by d H (B i , B j ) = max max ni∈Bi min nj ∈Bj d 1 (n i , n j ) ; max nj ∈Bj min ni∈Bi d 1 (n i , n j ) where d 1 (x, y) = x -y 1 = i |x i -y j |.

Fig. 7 :

 7 Fig. 7: Part of each theme of Comptine d'un autre été: l'Après-midi. The first one goes from B 5 to B 8 and is repeated one octave higher from B 22 to B 25 . The second goes from B 9 to B 12 , then is repeated with extra notes from B 13 to B 16 and the whole 8 bars are played one octave higher from B 26 to B 33 . The third one goes from B 17 to B 21 and is repeated one octave higher from B 34 to B 38 , with a slight change in B 38 to bring us to the end of the piece.

  sR

Fig. 8 :

 8 Fig. 8: The musical accompaniment of Comptine d'un autre été: l'Après-midi. These 4 bars consists of 4 arpeggiated chords Em -G -B -D and are played once at the beginning of the score without any melody, from B 1 to B 4 .

Fig. 10 :

 10 Fig. 10: The associated complex of Comptine d'un autre été: l'Après-midi with an error margin of 14%: each component characterizes a theme of the piece.

Fig. 11 :

 11 Fig.11: One-dimensional cycles from Comptine d'un autre été : L'Après-midi that appear with an error margin of 8% (left) and 26% (middle and right).

  The score as a Point Cloud If X and Y are two non-empty subsets of a metric space (M, d), it is possible to define a metric d H called the Hausdorff distance between X and Y by :

	2.3
	71), (1, 2, 69), (3, 1, 72)}


S Bernonville
email: stephanie.bernonville@univ-valenciennes.fr

C Kolski

M C Beuscart-Zéphir

Towards an Assistance for Selecting Methods and Models for Interactive Software Design or Re-engineering within Complex Organisation: Application Case of a CPOE Software

The hospital activities display all of the characteristics of a very complex sociotechnic organisation. Consequently, the design of Computerized Physician Order Entry (CPOE) softwares which support this type of activity is very difficult. The Study of the procedure of these projects shows communication information problems between project partners. This article proposes the principles of an approach, based on Software Engineering (SE) and Human-Computer Interaction (HCI) methods and models, allowing improving the transmission of information during design or re-engineering projects within complex organisation.

Introduction

The hospital activities display all of the characteristics of a very complex sociotechnic organisation [START_REF] Vicente | Cognitive Work Analysis[END_REF]: security issue, communication between actors, information management... CPOE software which supports ordering-dispensingadministration medication process appears more and more. These softwares attempt to improve the work situations and to reduce medication errors (ex: to avoid recopying errors or to carry out automatic checking of drug interactions) [START_REF] Kaushal | Information technology and medication safety: what is the benefit?[END_REF]. However, implementing such softwares is very difficult. Design or re-engineering projects of CPOE softwares involve the participation of different partners from specifics domains such as final users, organisation specialists, ergonomists, managers, designers, developers, graphic designers… We observed procedures used for of designing and re-engineering CPOE softwares and more particularly the exchanges between ergonomists and designers / developers. Table 1 shows different problems identified with theirs consequences. These problems often come back in design or re-engineering projects, more particularly for taking into account Humans Factors in design or re-engineering [START_REF] Beuscart-Zéphir | The Human Factors Engineering approach to biomedical informatics projects : state of the art, results, benefits and challenges[END_REF] or for using specific methods and models by each partners [START_REF] Nayak | Analyzing and communicating usability data: Now that you have the data what do you do?[END_REF]. To attempt to solve these problems, we propose an approach based on SE and HCI methods and models to help software design or re-engineering within complex organisation. The purpose of this approach is to create models which can be used as common work supports between project partners. It proposes modelling solutions using existing methods and models, adapting them and creating new methods if necessary [START_REF] Bernonville | Contribution and limits of UML models for task modelling in a complex organizational context: case studyin the healthcare domain[END_REF]. The purpose of the obtained supports is to improve the transmission of relevant information within project. Figure 1 shows common work supports goals for each partner.

In this article, the approach bases will be explained in a first part that is the needs definition and the concepts presentation. Then, an example applied to a CPOE software case will be presented in a second part, that is the creation of models with our approach and the results obtained.

The approach bases

Needs definition

In this part, needs which we plan to take into account in our approach, are defined. These needs attempt to answer to problems concerning information transmissions within design or reengineering projects within complex organisation. For problem 1 (see table 1), the defined need is to propose modelling solution for difficult notions to be represented by ergonomists. For problem 2 (see table 1), the defined need is to have a common representation between all projects partners. For problem 3 (see table 1), the defined need is to propose modelling solution for missing notions to be represented by ergonomists and designers / developers. For problem 4 (see table 1), the defined need is to show relevant results from activity analysis and software analysis (i.e. organisation changes from new software integration…). We need to adopt an ecological approach [START_REF] Gibson | The Ecological Approach to Visual Perception[END_REF] allowing ergonomists and designers / developers to achieve naturally the expected results. This ecological approach must correspond to ergonomists and designers / developers habits. Finally, we need to ensure the project documentation in order to store acquired knowledge and to reuse common work supports in other projects. Needs being defined, we are going to see now the approach organisation.

Proposal

Needs for SE and HCI methods and models to obtain common work supports.

To obtain common work support between project partners, SE and HCI methods and models (with some specific methods and models from ergonomics) have been utilized. Designers / developers use usually these techniques to construct the software units and to model organisational or technical aspects. There are a lot of SE and HCI methods and models taking into account specifics elements according to different points of view: cartesian methods allowing hierarchical decomposition of process (i.e. : SADT) [START_REF] Ross | Structured Analysis: A Language for Communicating Ideas[END_REF]; systemic methods allowing to exchange and utilize information modelling within organisation (i.e. : MERISE) [START_REF] Tardieu | La méthode Merise[END_REF]; object-oriented approaches using UML (www.omg.org); Petri Nets allowing to model dynamic aspects of system [START_REF] Helander | Handbook of Human-Computer Interaction[END_REF]. Several tools supporting these techniques have been created as Computer Aided Software Engineering (CASE). They propose modelling tools supporting specific languages and methods (i.e.: ArgoUML is a CASE supporting UML language). Other CASE combine modelling tools and development tools. SE and HCI methods and models are known to have a good potential concerning organisation analysis [12]. They can help with some adaptations, in filling in methods and models limits and lacks usually used by ergonomists to translate data issued from activity analysis within complex organisation and software analysis [13]. We used SE and HCI methods and models according two ways: (1) Usual methods and models to keep partners project customary procedures and (2) Adapted and combined methods and models to complex organisation analysis and software analysis. The second use completes the first to offer ergonomists new modelling solutions. The following part presents in details the approach concepts allowing partners project to obtain common work supports.

The approach concepts.

We defined six concepts to construct our approach: Profiles: For the moment, we distinguish the "Human-Computer Interaction specialist" profile, the « designer / developers » profile and the « organisation specialist» profile because their modelling objectives are very different. Modelling solutions: We distinguish several modelling solutions: usual modelling solutions for each profile and adapted and combined modelling solutions issued from existing methods and models (for the moment, these modelling solutions concern only the "HCI" profile, particularly ergonomists)). These solutions are made up of SE and HCI methods and models (with some very specific to ergonomics). We added some methods and models well known in the SE, HCI and ergonomics domain as UML, SADT, Petri Nets, MAD [START_REF] Scapin | Towards a method for task description: MAD[END_REF]… Finally these methods and models have been classified in three categories: (1) HCI methods and models, (2) generic methods, models and languages and (3) organisational methods and models. Each category is made up of methods and models allowing taking into account specific modelling needs (with some adaptations if necessary). Modelling choices: we distinguish two categories (modelling choices corresponding to fields of each profile and adapted choices corresponding to « ergonomists » profile. These modelling choices have been defined by us, in order to identify the project partners modelling needs. For the first category, the terms issue from the literature. They correspond to the objectives of each usual modelling solution (i.e.: for the UML activity diagram, the modelling choice is « operation structure in actions »). For the second category, the terms have been adapted in order to be more explicit for the project partners such as ergonomists which they are not used to SE and HCI methods and models in their activity (i.e.: for the adapted UML activity diagram, the modelling choice is "tasks or actions chain for an actor"). Modelling context identification: we distinguish two categories: modelling context identification corresponding to the different steps of Human Factors lifecycle (i.e.: « work situation analysis », « evaluation »…) and modelling context identification corresponding to the different steps of interactive system development lifecycle (i.e.: "requirements analysis", "analysis", "design stage" stages). Each project partner belonging to a profile can identify the model that he/she wants to create. Graphics elements taken into account by each modelling solutions: a set of graphics elements is assigned to each modelling solutions (i.e.: for the UML activity diagram, the graphic elements can be swimlanes, actions, flow, UML objects…). These graphics elements issue from the literature. They correspond to the modelling possibilities proposed by each methods and models of SE and HCI (with some very specific to ergonomics).

The possibilities for each category, previously mentioned have been drawn up in the form of lists. The following part describes the connection between these different concepts.

The approach dynamic structure

The figure 2 shows the connection between the concepts to come to common work supports. Referring to this figure, one can see that initial state is starting data corresponding to collected data to be modelled, issued from activity analysis, software analysis, design stage… The first step is the profile identification ("designers / developers", "organisation specialist" or "Human-Computer Interaction specialist"). Then, the step two is the identification of the modelling context which is different according to the profile. The user identifies their modelling choices. One or several modelling solutions are proposed. The data entry form with the graphics elements corresponding to the selected modelling solution displays. Finally, the user realises the model. A complement step can be used, the user can analyse the created model (The step principle is not described in this article). The final step is created model can be used as the common work support between project partners.

The concepts and the dynamic structure of the approach have been explained. Now we are going to see an application example of our approach for a CPOE software case.

Application example: case of a CPOE software

Context

We interested in a software which was the subject of a (re)design project. This software is a web application which was designed to replace an existing software. Its purposes are the ordering and the planning of medication orders in hospital. During this project, designers have been entrusted mocks-up to HCI specialists, which is more particularly an ergonomists team, in order to be analysed. The ergonomists general purpose was to analyse the potential integration of the future software within an existing organisation. We interested particularly in the acquisition information of the physician during the ordering activity. Firstly, we are going to see the ergonomists collected data issued from an activity analysis performed by ergonomists. Then, we will see how to obtain a common work support from the collected data, showing the results analysis.

Collected data

Thanks to the activity analysis, ergonomists collected necessary information for the physicians to update their representation of the patient case. Ergonomists distinguished two categories of information among collected data: necessary information to sum up the patient case and detailed information to obtain more precisions about the patient. Ergonomists established a table (figure 3) with all collected information categorised by types (i.e. : patient information, patient history…) used by the physician and classified according the two categories previously mentioned (i.e. : the patient constants can be used in the both cases, the physician needs last constants to update his/her representation of the patient case (necessary information to sum up the patient case) and he/she needs to see all constants of the patient stay because he/she wants to have precisions to make a decision (detailed information)). Concerning the software analysis, ergonomists studied mocks-up and textual specifications supplied by designers.

MODELLING CONTEXT IDENTIFICATION

MODELLING CONTEXT IDENTIFICATION

•Business process stage

Approach application to obtain common work support

The ergonomists needed to show clearly the location of information used to the acquisition information of the physician during the ordering activity, planned by the future software, and to show the interactions planned by the future software, to reach these information. The final purpose was to see if the location and the interactions planned by the future software were compatible with physician activity observed. We applied our approach to answer to the ergonomists modelling needs. Referring to the figure 2, the principle is as follows. Starting data are the collected data previously defined in the part « data collected ». The identified profile is "Human-Computer Interaction" profile. The identified modelling context is "evaluation" and particularly "software in the course of design". The modelling choices are "information location used by the actor" and "browsing between screen pages". The modelling solution is the adapted UML State transition diagram. Indeed, we used the "state" element to represent each application screen page or window and to obtain an application view, and the "event" element to represent planned user actions to browse between each screen page. We modified the usual state description. The usual description is a rectangle with the state name, the entry events and exit events which allow to activate the state or to exit the state. The modified state description is a rectangle with the screen page name and the information contained in the screen page. We added also the condition element from The UML activity diagram to represent conditional event. Finally the last step is the display of the graphic elements corresponding to the modelling solution (adapted UML State transition diagram) is displayed to create the models (i.e.: initial state, final state, event/action, screen page, window, condition, data, and acronym).

Figure. 3. Extract of table, defined by ergonomists and illustrated collected data

We obtained the model represented in figure 4. The screen pages are represented by a rectangle and the windows by an oval form. The events (or actions) are represented by oriented arrows and show the interaction between screen page and windows. The conditions are represented by a little rhombus. Finally the information location is represented by text located in rectangle and by acronyms corresponding to each categories of information defined by the ergonomists (i.e.: information patient is characterised by the acronym « IP »). To locate the information, one took each information used by the physician and one looked for if this information is represented in a screen page of the future software. If one found it, we wrote the information in the corresponding rectangle and one added the corresponding acronym. One added also the title of each screen page planned by the analysed software in order to the model is more explicit.

Created model analysis

The created model shows that information used by the physician to update his / her representation of patient case is broken up within the software in the course of design at this moment. The physician must browse between the different screen pages to find necessary and detailed information. For example, to obtain the current patient treatment, the physician must go to the screen page « ordering ». Then if he / she would like to see the last patient constants, he / she must go to another screen page "patient folder" and on a link to see the constants. While the activity analysis shows that the physician needs firstly for the last information on the patient (i.e.: last constants, last exams planned…). He/she consults in details other information if he/she needs more precisions (i.e.: all constants of the patient stay, medical history…).

The recommendation proposed by the ergonomists was to display a rundown of the patient case and to display links leading to other screen pages with detailed information; it had been represented with our approach by a complete analysis illustrating the possible problem and the recommendation proposed (for lack of place, it is not presented in this article).

Conclusion and perspectives

In this article, we explained the principles of an approach aiming to help the design or reengineering of software within complex organisation. The purpose of this approach is to create models which can be used as common work supports between projects partners. It proposes usual modelling solutions (based on existing SE and HCI methods and models and a few very specific methods from ergonomics) to keep designers / developers and ergonomists customary procedures and adapted modelling solutions (based on combinations and adaptations of existing methods and models) to fill in methods and models limits and lacks usually used by ergonomists.

Our perspectives consist in assembling in the same assistance tool modelling solutions necessary to designers / developers and ergonomists, to facilitate information transmission between projects partners and to improve the human factors integration to help software design or reengineering within complex organisation. This tool is in the course of design. Finally, in this tool, we plan also to propose a means to analyse created models proposing analysis criteria for each modelling solution used by ergonomists.

Acknowledgment

This research was supported financially by the FEDER, the NPdC Region (Miaou & Eucue projects), the Univ. Hospital of Lille, the French Ministry of Research (Rnts & Presc'Info projects).

Figure. 1 .

 1 Figure. 1. Goals of common work supports

3 -

 3 Information transmitted by the designers / developers to the ergonomists can be inadequate to understand the analysis context, mocks-up or prototypes proposed -Information transmitted by the ergonomists to the designers / developers can be inadequate to understand recommendations proposed -Software evaluation is difficult to the ergonomists -The software design or modifications do not correspond to the expectations 4 Human factors are not taken into account in software design or re-engineering The designed software does not correspond to the final user activity.

Figure. 4 .

 4 Figure. 4. Example of common work support.

Table 1 .

 1 The different problems identified with theirs consequences

n° Problems identified Consequences

	1	The existing methods and models used by ergonomists are limited to translate data from activity analysis and software (existing or in the course of design) analysis within complex organisation	The ergonomists have difficulties to transmit relevant information helping software design or re-engineering
	2	The ergonomists and the designers / developers come from specific domains. They use specific methods and models	The representations of analysed work situations are different according to ergonomists and designers / developers

number bed -PI sex -PI married name, firstname, family name -PI age PI surname and firstname Patient information card -PI surname and firstname -PI family situation -PI address -PI telephon number Physological data -risk factors card -PI patient usual weight General history card -HM medical -HM chirurgical -HM family Allergy card -HM médical -HM manifestation

			PATIENT FOLDER
		Necessary	Detailed
	Information categories	information to sum up the	information to obtain
		patient case	more
	Patient Information
			Hospitalization
	First name, surname		subfolder (patient stay)
	Room number		Follow-up card
	Sex		temperature, blood pressure, pulse,
	Age
	Usual Weight
	Address and phone number
	Family situation
	Profession
	…	…	…
	Patient History
	Medical history
	Chirurgical history
	Family history
	…	…	…

-PI -C temperature -C

blood pressure -C pulse -C bowel movement -C weight Patient folder tree

	TASKS GLOBAL PLANNING	session] [first « patient folder connexion in the same	(login + password) identification
		[« patient folder » connexion in the same
		session]
			Click on the card link
			Click on the
			card link
			Click on the
			card link
			Click on the card link

PI patients surnames, patients firstname, sex, number bed, number room (each task is represented by pictogram) EAA appointment, CT unusual drug taken (oral voice), unusual one shot taken, perfusion ( horizontal green line trait = perfusion lenght)

	Selection of a patient and click on the « patient folder»
	ORDERING

-PI

patient number bed -PI patient sex -PI married name, firstname, family name -PI age -CT sorting (by 24 h, current…) -EAA act name -EAA text icon (ordering condition, search object) -CT drug name -CT currentposology -CT old posology (if modifications) -CT posology crossed out (if stopping) -CT text icon (ordering condition, search object) « Text » lab -EAA and CT

	EXAMS FOLLOW-UP LIST
	Selection of a patient and click		[first « patient folder	identification
	on the « ordering » button		connexion in the same	(login + password)
			session]
	[« patient folder »	connexion in the	same session]
	Click on « patients list »
	button and selection of
	« patients list » and selection of « planning and selection of « time planning »			Click on the tab	Click on the tab	complement text ( condition and/or search object) « Events » tab -EAA and CT orders (date, time, modified state, Click on the tab
					creation, asked)
					Click on the tab

-EAA planning (date, time, state) -realised the (date, time, state) -PI

patients usual name, number bed, sex -EAA object, ordering, asking appointment with specialists APPOINTMENT FOLLOW-UP LIST -IP patients usual name, number bed, sex -EAA object, orders, appointment date, state

	Click on the « planning » button and
	selection of «time planning » or
	« weekly planning » or « drugs
	planning »
	Click on the « follow-up list»
	and selection of « exams follow-up » and se of « exams follow-up »	Selection of a patient and click on the « patient folder » button
	Click on the « follow-up
	list » button and selection of
	« exams follow-up » and
	selection of «appointment of
	the day » or « appointment
	to come » or « appointment
	realised »
	Click on the « planning » button and selection of « time
	planning » or « weekly planning » or «drugs planning »
	Tasks to be realised CT Drug name, concentration ou dosage, realised Click on the pictogram Click on the « come back » date and time of tasks to be button	Click on the « ordering » link
	Click on « patients list »
	button and selection of
	« patients list »
		Click on « patient folder » link
		Click on « ordering » button
		Click on « patient folder » tab
	« Follow-up » tab

-EAB and CT

périod/date (asking appointment planning, planned date and time, treatment lenght, périodicité) -EAB and CT physician who orders -CT step (to be transmit, current, signed) Legend :

	PI Patient information
	HM Hospitalisation grounds
	C Constants
	Ev Events, disease evolution
	EAB Exams, Analyses, Assessments
	CT Current treatment


Keywords: Evaluation, Human-Computer Interaction (HCI), Intelligent Transport System (ITS), Personalized Information System (PIS)

Thanks to the emergence of the Personalized Information System (PIS), it becomes possible to supply the user only with the pertinent information that directly interests him/her and suits his/her preferences. However, we need beforehand to evaluate these systems in real situations. In this paper, we point out the insufficiencies in the evaluation of the PIS. Then, we propose a new evaluation method for PIS. Finally, we describe the evaluation results of a demonstrator developed during a project called MouverPerso. This demonstrator was tested among a group of subjects in the University of Valenciennes.

I. INTRODUCTION

Nowadays, the information systems, addressed to the users and mainly in the field of transport, tend to be more and more personalized. In the midst of a universal project labelled Intelligent Transport System (ITS), researchers' primary preoccupation is to provide personalized information for the public transport users. In fact, the PIS is a system which has the capacity to be adapted to the user taking in consideration his/her preferences [START_REF] Barrett | How to personalize the Web[END_REF]- [START_REF] Cingil | A broader approach to personalization[END_REF]- [START_REF] Maglio | Intermediaries personalize information streams[END_REF]. For example, in transport field, the traveler hopes to have at his/her disposal only some information, just what he/she is directly interested in [START_REF] Basu | Recommendation as Classification: Using Social and Content-Based Information in Recommendation[END_REF]- [START_REF] Hoyer | Approach to personalised information services to public transport[END_REF]. In addition, the traveler may have access to a reliable, multimodal and personalized information using various supports (PC, PDA, mobile phone, etc.) [START_REF] Abowd | Towards a Better Understanding of Context and Context-Awareness[END_REF]. Though, there are different methods and approaches to conceive PIS systems, to our knowledge and at the present point of research, we notice a lack of methods to evaluate the personalization quality of PIS. This article is made of three principal parts: at first we focus on the insufficiencies concerning the evaluation of PIS. The second part is devoted to describe the basic principles of a proposed method that permits the evaluation of SIP. The third and last part is meant to describe the evaluation results of the demonstrator developed during a project called MouverPerso.

II. INSUFFICIENCY IN THE PIS EVALUATION

The evaluation of the interactive system has been a recurrent problem since the last three decades. To ameliorate the quality of human machine interaction many studies were oriented towards the evaluation of the interactive system from different angles and view points. Concerning this subject we should mention that several papers and works had defined the basic principals of the evaluation as well as the methods used in this evaluation [START_REF] Wilson | Evaluation of human work, a practical ergonomics methodology[END_REF] [START_REF] Soui | Criteria devoted to evaluate personalized interactive systems[END_REF]. We notice that the focal point in these works was on the utility and usability while the dimension of personalization was neglected. Nowadays, the PIS users are facing many difficulties to interact with the badly studied or evaluated personalized systems that do not always answer their needs. Due to the complexity of personalized system interface, new criteria and methods are needed to evaluate the human machine interaction. In this context, we propose a method which is based on the evaluation criterion of PIS detailed in [START_REF] Soui | Criteria devoted to evaluate personalized interactive systems[END_REF] and on the explicit intervention of the user who fills up questionnaires.

-[8]-[9]-[10]-[11]-[12]-[13]-[14]-

III. PROPOSITION OF A METHOD FOR EVALUATING PIS

The process of this evaluation is illustrated in the fig. 1. To give a clear structure to this proposition we use the SADT formalism; a well known in software engineering and in human machine interaction. This method is made up of three phases. According to the SADT formalism, we find in the box A0 (the box in top of diagram) the general objective that consists in the evaluation of PIS. To reach this target we have decomposed it into three sub-targets presented in the boxes A1, A2 and A3 ( preparation, evaluation and analysis).

-Phase A1: it represents the preparation stage wherein the evaluator chooses the representative tasks on which the evaluation will be based. The evaluator also prepares two types of documents necessary to the evaluation. The first is a general questionnaire including general information about the users and the second is an index-card that includes the definition of every criterion and the parts which the user fills up during the experimentation. We distinguish seven global criteria, detailed in [START_REF] Soui | Criteria devoted to evaluate personalized interactive systems[END_REF]. -Phase A2: it represents the stage of testing the model; it is made of two sub-phases (A21 and A22): -Sub-phase A21: it represents the pre-experimental stage; the evaluator presents to the users the tasks already chosen in addition to the evaluation criteria and their definition. Then the evaluator asks them to select the most important criteria according to their needs and/or preferences. The users must attribute to every selected criterion a weight representing the importance accorded to this criterion.

M.Soui 1 , C.Kolski 1 , M.Abed 1 , G. Uster 2
-Sub-phase A22: it represents the experimental stage, in fact the users have to evaluate the system progressively while executing the tasks in terms of the already defined criteria. Then he/she allocates to every criterion a mark according to his/her level of satisfaction and according to a well predefined scale.

-Phase A3: it represents the analysis phase. In this level, the evaluator calculates to every user the level of satisfaction N s that translates the level of personalisation N p of the system applying the following formula:

N s = N p =        n k K K n K K K K I W N I W 1 1
With n: number of criteria the user is concerned with. W K : the weight of interest representing the importance the user gives to this criterion. I K : the criterion index of activation.

1 if the user U is concerned with the criterion k 0 if not N K : the mark attributed by the user to the criterion k 0 if the user is not satisfied at all 0.25 if the user is a bit satisfied 0.5 if the user is fairly satisfied 0.75 if the user is satisfied 1 if the user is very satisfied

The result is a value restricted between 0 and 1.0 so that the more the satisfaction level is close to 1 the more the system is adapted to the user. To judge the system according to its degree of personalization, we calculate the average of the satisfaction level N s of all the subject participated in the evaluation.

IV. CASE STUDY: IN ITS DOMAIN • Context: Our work is a part of the project « MOUVER.PERSO» achieved with the collaboration of National Institute of Research on Transport and its Security (INRETS). This system aims to incite the usage of collective transport by ensuring the complementarity between different modes of transport and ameliorating the quality as well as the availability of the personalized information.

• Protocol test: The objective of this experience was to evaluate the demonstrator MouverPerso taking as basis the proposed approach and the criteria we have just defined for the evaluation of SIP.

-The participants: Twenty three people have participated in this evaluation, two expert evaluators, twenty subjects (twenty students in computer science) and a technician.

-The evaluated task: We focused on a representative task of the application, a research of itinerary after adding an appointment. This task needs the consultation of at least three interfaces of the application (an interface which permits to add an appointment, an interface which permits to consult the itinerary details and an interface which permits to consult the appointment list). The dynamic task is represented by Statechart diagram from UML (see fig. 2).

-Tools and techniques: In this experimentation, several tools were used namely: ▪ The general questionnaire: the evaluator invites the user to fill in this questionnaire which includes some general information such as last name, first name, email, age, gender. This questionnaire is distributed among the subjects in the preexperimental step.

▪ The criterion index-card: the user makes use of his/her indexcard to judge the system for example s/he could attribute weights and marks, mention problems and/or draw the attention to them and may propose ideas to improve the system during the experience.

▪ A PDA (Portable Digital Assistant): is a personal assistant taking the form of a mobile digital appliance. It consists of a computer equipped with a tactile screen and a stylet (fig. 6). Fig. 2. The dynamic task of adding an appointment.

V. RESULTS AND ANALYSIS

This part represents the principal results issued from the test and it is concerned with the dimensions related to the content and container personalization.

A. Content Personalization

Three criteria were considered in the personalization of content: the preferences, point of focus and user experience as we see in fig. 3.: 47% of the users are very satisfied, 41% are satisfied, 6% are less satisfied and finally 6% of them are not satisfied at all (see fig. 3). In order to give more details about the students' opinion concerning the personalization content, we have compared the average of their level of satisfaction according to the three criteria related to the quality of content personalization. For preferences and experiences, the averages of satisfaction in connection with these criteria exceed 0.6. For the interests the average is less satisfying (see fig. 4).

B. Container Personalization

The personalization related to the container groups four criteria: adaptation to interactive platform, adaptation to environment, adaptation to user's behaviour and finally adaptation to users' physical capacities (accessibility). The rate of satisfaction is 33% for the very satisfied subject, 17% for the satisfied and 50% for the bit satisfied ones (see fig. 5). In order to give more details about the students' opinion concerning the personalization related to the container, we have compared the average of their level of satisfaction according to the four criteria related to these dimensions. After having tested the application in using two different interactive supports (PC, PDA) (see fig. 6.) the subject gives his/her opinion about the adaptation of the system to the interactive platform. Concerning the adaptation criterion of the users behaviour, the students' opinion are centred on the neutral response (the average = 5), while as showed in fig. 7, the student have a disapproving opinion about the adaptation to the users' physical capacity and to the environment (the average <4).

VI. PROBLEMS MENTIONED BY THE SUBJECTS

The fig. 8 shows that the majority of the subjects have mentioned some problems related to experience, accessibility and users behaviour. About accessibility a problem is cited several times which is the smallness of the characters that causes a bad legibility. Concerning the behaviour, the subjects notice the lack of warning signals during the validation task. The subjects had mentioned few problems related to the preferences, to interests and the interactive platform. This could be explained by the importance the designers of this system had given to the dimensions related to the personalization of the content and to the adaptation to the interactive platform when compared with the others criteria.

VII. CONCLUSION AND PERSPECTIVES

In this article we have mentioned the insufficiencies in the evaluation of PIS and proposed a method to evaluate such systems. This method was tested among a group of subjects using the demonstrator MouverPerso. The information we have collected allowed us to know the subjects' level of satisfaction, the problems they have encountered and the ideas they proposed to ameliorate this system. This method is based on the explicit user intervention and on the existence of a real system (model, prototype or final system). Despite that it provides concise results, since they are explicitly obtained from users' opinions, the evaluator and user spend much time to achieve PIS evaluation. It also disturbs the user in his/her main activity. Besides, this evaluation needs several iterations to find maximum of problems. These reasons make us think about another evaluation method that does not need the direct intervention of the user. The principle of this proposed method which makes the object of our future research will be based on the usage traces of the system.

Fig. 1 .

 1 Fig. 1. Evaluation phases of the proposed method.

Fig. 3 .

 3 Fig. 3. The subjects' opinion about the personalization related to the content.

Fig. 4 .

 4 Fig.4. Average of the student's level of satisfaction per personalized criterion related to the content.

Fig. 5 .

 5 Fig. 5. The subjects' opinion about the personalization related to the container.

Fig. 6 .

 6 Fig.6. Test with PDA

Fig. 7 .

 7 Fig.7. Average of the student's level of satisfaction per personalized orientation related to the container.

Fig. 8 .

 8 Fig.8. Problems repartition according the evaluation criteria evaluation.

LAMIH-UMR CNRS 8530, University of Valenciennes and Hainaut-Cambrésis, Le Mont Houy, 59313 Valenciennes cedex 9, France {Makram.Soui, Mourad.Abed, Christophe.Kolski, emmanuelle.grislin}@univ-valenciennes.fr

The French national institute of research for transport and security, INRETS, Villeneuve d'Ascq, France guillaume.uster@inrets.fr

ACKNOWLEDGEMENTS

The present research work has been partially supported by the "Ministère de l'Education Nationale, de la Recherche et de la Technologie», the «Région Nord Pas-de-Calais» and the FEDER (projects MIAOU, EUCUE, SART), l'ANR ADEME (Viatic.Mobilité), la PREDIM (MouverPerso). The authors gratefully acknowledge the support of these institutions.


Xuan Son Nguyen
email: xuan-son.nguyen@ensea.fr

GeomNet: A Neural Network Based on Riemannian Geometries of SPD Matrix Space and Cholesky Space for 3D Skeleton-Based Interaction Recognition

In this paper, we propose a novel method for representation and classification of two-person interactions from 3D skeleton sequences. The key idea of our approach is to use Gaussian distributions to capture statistics on R n and those on the space of symmetric positive definite (SPD) matrices. The main challenge is how to parametrize those distributions. Towards this end, we develop methods for embedding Gaussian distributions in matrix groups based on the theory of Lie groups and Riemannian symmetric spaces. Our method relies on the Riemannian geometry of the underlying manifolds and has the advantage of encoding highorder statistics from 3D joint positions. We show that the proposed method achieves competitive results in two-person interaction recognition on three benchmarks for 3D human activity understanding.

Introduction

3D skeleton-based action recognition has been an active research topic in recent years with many potential applications. In this work, we focus on 3D skeleton-based twoperson interaction recognition (3DTPIR). Compared to a large number of general-purpose methods for 3D skeletonbased action recognition, methods for 3DTPIR are much less studied in the literature. Recent works [START_REF] Perez | Interaction Relational Network for Mutual Action Recognition[END_REF][START_REF] Yang | Pairwise Adjacency Matrix on Spatial Temporal Graph Convolution Network for Skeleton-Based Two-Person Interaction Recognition[END_REF] have shown that state-of-the-art action recognition methods do not always perform well on human interaction recognition. This is because they lack an effective mechanism for capturing intra-person and inter-person joint relationships [START_REF] Perez | Interaction Relational Network for Mutual Action Recognition[END_REF].

In two-person interactions, arm and leg movements are highly correlated. However, these correlations are simply encoded by distances between joints in most existing works for 3DTPIR [17,[START_REF] Ji | Interactive Body Part Contrast Mining for Human Interaction Recognition[END_REF][START_REF] Ouyed | Group-of-features Relevance in Multinomial Kernel Logistic Regression and Application to Human Interaction Recognition[END_REF][START_REF] Yun | Two-person Interaction Detection Using Bodypose Features And Multiple Instance Learning[END_REF]. This motivates us to use highorder statistics, i.e. covariance matrices to better capture these interactions. It has been known that n × n covariance matrices lie on a special type of Riemannian manifolds, i.e. SPD manifolds (denoted by Sym + n ). A large body of works has been developed for classification of SPD-valued data.

Recently, SPD neural networks have demonstrated impressive results [START_REF] Huang | A Riemannian Network for SPD Matrix Learning[END_REF]. One of the core issues that remains open is the finding of effective and efficient methods for modeling probability distributions on Sym + n . Since Gaussian distributions (abbreviated as Gaussians) on R n are the most popular probability distributions used in statistics, existing works mainly focused on generalizing them to Sym + n . Such a generalization was first given in [START_REF] Pennec | Probabilities and Statistics on Riemannian Manifolds : A Geometric approach[END_REF] in a more general context of Riemannian manifolds. However, the asymptotic formulae of Riemannian Gaussian distributions (abbreviated as Riemannian Gaussians) proposed in this work make them hard to evaluate and apply in practice. Some works aim to address this shortcoming by introducing notions of Riemannian Gaussians in symmetric spaces [START_REF] Said | Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices[END_REF][START_REF] Said | Gaussian Distributions on Riemannian Symmetric Spaces: Statistical Learning With Structured Covariance Matrices[END_REF] and homogeneous spaces [6]. These have been successfully applied to classification problems. In this work, we also interested in Riemannian Gaussians for classification. However, differently from the above works, we seek methods for embedding Riemannian Gaussians in matrix groups. This allows us to perform classification of Riemannian Gaussians without having to resort to an exact expression of their probability density function as in [6, [START_REF] Said | Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices[END_REF][START_REF] Said | Gaussian Distributions on Riemannian Symmetric Spaces: Statistical Learning With Structured Covariance Matrices[END_REF].

In summary, the main contributions of this work are:

• We propose an embedding method for Gaussians by mapping them diffeomorphically to Riemannian symmetric spaces.

• We consider representing a 3D skeleton sequence by a set of SPD matrices that leads us to the study of statistics on Sym + n . We show that the product space of mean and covariance on Sym + n can be viewed as a Lie group with an appropriate group product. Moreover, we point out a connection between this space and the group of lower triangular matrices with positive diagonal entries.

• Based on the theory described above, we introduce a neural network for learning a geometric representation from a 3D skeleton sequence.

• Experiments on three benchmarks for 3D human activity understanding demonstrate the competitiveness of our method with state-of-the-art methods.

Related Works

We will briefly discuss representative works for 3DTPIR (Section 2.1), embeddings of Gaussians (Section 2.2), and probability distributions on Sym + n (Section 2.3).

Two-person Interaction Recognition from 3D Skeleton Sequences

A variety of approaches has been proposed for 3D skeleton-based action recognition. These are based on hand-crafted features [START_REF] Evangelidis | Skeletal Quads: Human Action Recognition Using Joint Quadruples[END_REF][START_REF] Luo | Group Sparsity and Geometry Constrained Dictionary Learning for Action Recognition from Depth Maps[END_REF][START_REF] Smedt | Skeleton-Based Dynamic Hand Gesture Recognition[END_REF]58,[START_REF] Yang | EigenJoints-based Action Recognition Using Naive-Bayes-Nearest-Neighbor[END_REF]67] and deep learning [9, [START_REF] Ke | A New Representation of Skeleton Sequences for 3D Action Recognition[END_REF][START_REF] Du | Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition[END_REF][START_REF] Liu | Spatio-Temporal LSTM with Trust Gates for 3D Human Action Recognition[END_REF][START_REF] Liu | Global Context-Aware Attention LSTM Networks for 3D Action Recognition[END_REF]32,33,40,[START_REF] Shahroudy | NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[END_REF][START_REF] Wang | Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks[END_REF][START_REF] Weng | Deformable Pose Traversal Convolution for 3D Action and Gesture Recognition[END_REF]68]. Recent works focus on neural networks on manifolds [START_REF] Huang | A Riemannian Network for SPD Matrix Learning[END_REF][START_REF] Huang | Deep Learning on Lie Groups for Skeleton-Based Action Recognition[END_REF][START_REF] Huang | Building Deep Networks on Grassmann Manifolds[END_REF]41] and on graphs [START_REF] Cheng | Skeleton-Based Action Recognition With Shift Graph Convolutional Network[END_REF][START_REF] Li | Spatio-Temporal Graph Convolution for Skeleton Based Action Recognition[END_REF][START_REF] Barbaresco | Jean-Louis Koszul and the Elementary Structures of Information Geometry[END_REF][START_REF] Si | An Attention Enhanced Graph Convolutional LSTM Network for Skeleton-Based Action Recognition[END_REF]62]. Due to space limit, we refer the interested reader to [48] for a more comprehensive survey. Below we focus our discussion on 3DTPIR.

Approaches for 3DTPIR are much less studied. Handcrafted feature based methods mainly rely on distances [17,[START_REF] Ji | Interactive Body Part Contrast Mining for Human Interaction Recognition[END_REF][START_REF] Ouyed | Group-of-features Relevance in Multinomial Kernel Logistic Regression and Application to Human Interaction Recognition[END_REF][START_REF] Yun | Two-person Interaction Detection Using Bodypose Features And Multiple Instance Learning[END_REF] or moving similarity [27] between joints of two persons. Li and Leung [START_REF] Li | Multiview Skeletal Interaction Recognition Using Active Joint Interaction Graph[END_REF] applied a multiple kernel learning method to an interaction graph constructed from the relative variance of joint relative distances. Two-stream RNNs are proposed in [START_REF] Men | A Two-Stream Recurrent Network for Skeleton-based Human Interaction Recognition[END_REF][START_REF] Wang | Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks[END_REF] where interactions between two persons are modeled by concatenating the 3D coordinates of their corresponding joints, or by augmenting the input sequence with distances between their joints. In [START_REF] Perez | Interaction Relational Network for Mutual Action Recognition[END_REF], Relational Network [START_REF] Santoro | A Simple Neural Network Module for Relational Reasoning[END_REF] is extended to automatically infer intra-person and inter-person joint relationships. The recent work [START_REF] Yang | Pairwise Adjacency Matrix on Spatial Temporal Graph Convolution Network for Skeleton-Based Two-Person Interaction Recognition[END_REF] deals with graph construction in graph convolutional networks for 3DTPIR.

Embedding of Gaussians

Methods for embedding Gaussians are widely used in statistics, e.g. for measuring the distance between probability distributions. The work of [START_REF] Rao | Information and the Accuracy Attainable in the Estimation of Statistical Parameters[END_REF] first proposed a distance function based on the Fisher information as a Riemannian metric. However, in the general case of multivariate Gaussians, an exact formula for the distance function is difficult to obtain. In computer vision, one of the most widely used embedding is derived from [START_REF] Lovrić | Multivariate Normal Distributions Parametrized As a Riemannian Symmetric Space[END_REF]. The key idea is to identify Gaussians with SPD matrices by parametrizing the space of Gaussians as a Riemannian symmetric space. The work of [START_REF] Calvo | A Distance between Multivariate Normal Distributions Based in an Embedding into the Siegel Group[END_REF] shares a similar idea of identifying Gaussians with SPD matrices. However, it is based on embedding Gaussians into the Siegel group. In [START_REF] Gong | Shape of Gaussians as Feature Descriptors[END_REF], a connection is established between Gaussians and a subspace of affine matrices. The method of [START_REF] Li | Local Log-Euclidean Multivariate Gaussian Descriptor and Its Application to Image Classification[END_REF] relies on the Log-Euclidean metrics [START_REF] Arsigny | Fast and Simple Computations on Tensors with Log-Euclidean Metrics[END_REF] for embedding Gaussians in linear spaces.

Probability Distributions on Sym + n

Existing works mainly focused on generalizing Gaussians to Sym + n due to their popularity in statistics. Generalizations of Gaussians are proposed in Riemannian manifolds [START_REF] Pennec | Probabilities and Statistics on Riemannian Manifolds : A Geometric approach[END_REF][START_REF] Zhang | Probabilistic Principal Geodesic Analysis[END_REF], symmetric spaces [START_REF] Said | Riemannian Gaussian Distributions on the Space of Symmetric Positive Definite Matrices[END_REF][START_REF] Said | Gaussian Distributions on Riemannian Symmetric Spaces: Statistical Learning With Structured Covariance Matrices[END_REF], and homogeneous spaces [6]. In [2,[START_REF] Brooks | Riemannian Batch Normalization for SPD Neural Networks[END_REF], Riemannian Gaussians are derived from the definition of maximum entropy on exponential families. Family of Alpha-Divergences and other related divergences have also been extensively studied [8].

Background Theory

Statistics on Riemannian Manifolds

The theory presented here is based on [START_REF] Pennec | Probabilities and Statistics on Riemannian Manifolds : A Geometric approach[END_REF]. In this framework, the structure of a manifold M is specified by a Riemannian metric. Let x be a point of M as a local reference and T x M be the tangent space at x. A Riemannian metric is a continuous collection of dot products < .|. > x on T x M. The distance between two points of a connected Riemannian manifold is the minimum length among the smooth curves joining them. The curves realizing this minimum for any two points are called geodesics. Let v ∈ T x M be a tangent vector at x. We define the exponential map at x as the function that maps v to the point y ∈ M that is reached after a unit time by the geodesic starting at x with this tangent vector. This map is defined in the whole tangent space T x M but it is generally one-to-one only locally around 0 in the tangent space (i.e. around x in the manifold). Let -→ xy = log x (y) be the inverse of the exponential map that is the smallest vector as measured by the Riemannian metric such that y = exp x ( -→ xy). The exponential chart at x can be seen as the development of M in the tangent space at a given point along the geodesics.

The definitions of mean and covariance on a Riemannian manifold are given below.

Definition 1 Let x be a random point of probability density function p x . Denote by dist R (y, x) the distance between x, y ∈ M induced by the Riemannian metric of M. The set of expected or mean values is:

E[x] = arg min y∈M M dist R (y, z) 2 .p x (z).dM(z) , (1)
where dM(z) is the volume measure induced by the Riemannian metric of M, and we assume that the integral is finite for all points y ∈ M (which is true for a density with a compact support).

Definition 2 Let x be a random point, x be a mean value that we assume to be unique. The covariance is defined as:

Cov x(x) = E[ -→ xx. -→ xx T ] = D(x) ( -→ xx).( -→ xx) T .p x (x).dM(x), (2)
where D(x) is the maximal definition domain for the exponential chart at x.

Proposed Approach

In what follows, we are interested in matrices over the field of real numbers, unless otherwise stated.

Embedding of Gaussians

Let N (n) be the space of n-variate Gaussians, Sym +,1 n be the space of SPD n × n matrices with determinant 1. Lemma 1 gives an embedding of Gaussians in N (n).

Lemma 1 We can identify the Gaussian (Σ Σ Σ, µ µ µ) ∈ N (n) with the following matrix in Sym +,1 n+k :

(det Σ Σ Σ) -1 n+k Σ Σ Σ + kµ µ µµ µ µ T µ µ µ(k) µ µ µ(k) T I k , (3)
where µ µ µ and Σ Σ Σ are the mean and covariance of the Gaussian, I k is the k × k identity matrix, µ µ µ(k) is a matrix with k identical column vectors µ µ µ.

Proof. See supplementary material. When k = 1, the embedding in (3) becomes the one introduced in [START_REF] Lovrić | Multivariate Normal Distributions Parametrized As a Riemannian Symmetric Space[END_REF]. The natural symmetric Riemannian metric resulting from the above embedding is given in Lemma 2.

Lemma 2 The Riemannian metric is given by:

< A 1 , A 2 > P =T r(A 1 P -1 A 2 P -1 )- - 1 n + k T r(A 1 P -1 )T r(A 2 P -1 ), (4)
where A 1 and A 2 are two tangent vectors at P. Proof. See supplementary material. It turns out that the Riemannian metric given in (4) belongs to the family of affine-invariant metrics proposed in [START_REF] Pennec | Statistical Computing on Manifolds for Computational Anatomy[END_REF]. Consequently, the exponential map at a point can be obtained [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF] as:

exp P (A) = P 1 2 exp P -1 2 AP -1 2 P 1 2 , ( 5
)
where P is a SPD matrix, A is a tangent vector at P, and exp(.) is the matrix exponential. By inverting the exponential map, we obtain the logarithmic map:

log P (Q) = P 1 2 log P -1 2 QP -1 2 P 1 2 , (6)
where P and Q are two SPD matrices, and log(.) is the matrix logarithm.

Statistics on Sym +

n Suppose that we are given a set of matrices P 1 , P 2 , . . . , P L ∈ Sym + n . From (1), one can define the empirical or discrete mean value of P 1 , P 2 , . . . , P L as:

P m = arg min y∈Sym + n 1 L L i=1 dist R (y, P i ) 2 . ( 7
)
The mean can be computed by an iterative procedure consisting in: (1) projecting the SPD matrices in the tangent space at the current mean; (2) estimating the arithmetic mean in that space; (3) projecting the mean back in Sym + n . These steps are iterated until convergence [39].

Similarly, the empirical covariance of a set of L SPD matrices of mean P m is defined using the discrete version of the expectation operator in (2):

P c = 1 L -1 L i=1 ---→ P m P i ⊗ ---→ P m P i , (8)
where ⊗ denotes the tensor product.

We propose to learn a transformation of ---→ P m P i from T P m Sym + n to another tangent space so that the covariance computed in this space is more discriminative for classification. The transformation is performed by parallel transport (PT). We need Lemma 3 for our transformation.

Lemma 3 Let P, Q ∈ Sym + n .
Let the Riemannian metric be the one given in [START_REF] Brooks | Riemannian Batch Normalization for SPD Neural Networks[END_REF]. The PT from Q to P along geodesics connecting Q and P of a tangent vector A ∈ T Q Sym + n is given by:

T Q,P (A) (PQ -1 ) 1 2 A (PQ -1 ) 1 2 T . (9)
Proof. See supplementary material. The formula of PT in (9) is the same as those in [11, [START_REF] Sra | Conic Geometric Optimization on the Manifold of Positive Definite Matrices[END_REF][START_REF] Yair | Parallel Transport on the Cone Manifold of SPD Matrices for Domain Adaptation[END_REF] which are all based on the Riemannian metric of the following form:

< A 1 , A 2 > P = T r(A 1 P -1 A 2 P -1 ), (10)
where A 1 and A 2 are two tangent vectors at P. In [START_REF] Brooks | Riemannian Batch Normalization for SPD Neural Networks[END_REF], the authors also use PT for designing Riemannian batch normalization (RBN) layers. Our method differs from theirs in three main aspects. First, their method learns the parameters of RBN layers from the statistics of minibatches, while our method deals with the statistics within each sequence. Note that a RBN layer can also be designed in our framework and can potentially improve the accuracy of our network. Second, their formulation of Riemannian Gaussians involves only a Riemannian mean without notion of variance. Third, their method does not aim to leverage the second-order statistics (covariance) on SPD manifolds.

Now suppose that

---→ P m P i , i = 1, . . . , L are transported to another tangent space at P (the target point of PT), then the covariance can be estimated as:

P c = 1 L -1 L i=1 T P m , P( ---→ P m P i ) ⊗ T P m , P( ---→ P m P i ). (11)
Let f v (.) be a mapping that vectorizes a symmetric matrix by taking its lower triangular part and applying a √ 2 coefficient on its off-diagonal entries in order to preserve the norm [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF]. Then, the covariance is given by:

P c = 1 L -1 L i=1 f v T P m , P( ---→ P m P i ) f v T P m , P( ---→ P m P i ) T .
(12) If P c is a matrix of size n × n , then we use a point (P m , P c ) that lies on the product manifold Sym + n ×Sym + n to parametrize the distribution of the given set of SPD matrices. Next, we propose an embedding of this point based on the Lie group theory.

Embedding of Riemannian Gaussians

We first define an appropriate group product on the product manifold Sym + n × Sym + n .

Definition 3 Let M(n, n ) be the product manifold

Sym + n × Sym + n . Let (P m i , P c i ) ∈ M(n, n ), i = 1, 2 where P m i ∈ Sym + n , P c i ∈ Sym + n
, and P c i = L i L T i be the Cholesky decomposition of P c i . Denote by ϕ : Sym + n → M k ×n a smooth bijective mapping with a smooth inverse where M k ×n is a subset of the set of k × n matrices. The group product between two elements of M(n, n ) is defined as:

:M(n, n ) × M(n, n ) → M(n, n ) (P m 1 , P c 1 ) (P m 2 , P c 2 ) = (ϕ -1 (ϕ(P m 1 )L 2 + ϕ(P m 2 )), (L 1 L 2 )(L 1 L 2 ) T ). (13)
Theorem 1 shows that M(n, n ) forms a Lie group.

Theorem 1 M(n, n ) is a Lie group under product .

Proof. See supplementary material.

Based on Theorem 1, we can establish a Lie group isomorphism between M(n, n ) and a subgroup of a group of lower triangular matrices with positive diagonal entries.

Theorem 2 Denote by LT + (n ) the group of lower triangular n ×n matrices with positive diagonal entries, 0 n ×k the n × k matrix with all elements equal to zero. Let

K + (n + k ) = K P m ,H H 0 n ×k ϕ(P m ) I k , (14)
where H ∈ LT + (n ), and

φ : K + (n + k ) → M(n, n ), φ(K P m ,L ) = (P m , P c ), (15)
where P c = LL T , L ∈ LT + (n ). Then φ is a Lie group isomorphism.

Proof. See supplementary material. We now can give the embedding matrix of a point (P m , P c ) ∈ M(n, n ) where P c = LL T as follows:

(P m , P c ) → L 0 n ×k ϕ(P m ) I k . ( 16
)
The embedding matrix in (16) depends on the choice of function ϕ(.). In this work, we set ϕ

= (f v • f lm )(k ) T where (f v • f lm )(k ) T is the transpose of (f v • f lm )(k ), (f v • f lm )(k ) is a matrix with k identical column vectors obtained from f v • f lm ,
and f lm (.) is given by:

f lm (P) = log(P) = U log(Z)U T , (17)
where P = UZU T is the eigenvalue decomposition of P, and log(Z) is the diagonal matrix of eigenvalue logarithms.

A Neural Network for 3DTPIR

We are now ready to introduce a neural network (Ge-omNet) for 3DTPIR based on the theory developed in the previous sections. Let N j and N f be the number of joints and that of frames in a given sequence, respectively, Let x in t,i ∈ R 3 , t = 1, . . . , N f , i = 1, . . . , N j be the feature vector (3D coordinates) of joint i at frame t. Two joints i and j are neighbors if they are connected by a bone. Denote by S i the set of neighbors of joint i. Let i 1,r , i 2,r be the two joints selected as the roots of the first and second skeleton, respectively (see Fig. 1). For any two joints i and i r ∈ {i 1,r , i 2,r } that belong to the same skeleton, the distance dist J (i, i r ) between them is defined as the number of bones connecting them (see Fig. 1). The first layer of Ge-omNet is a convolutional layer written as:

x out t,i = t+1 t =t-1 i ∈Si Wt ,i x in t ,i , (18)
where x out t,i ∈ R d is the output feature vector of joint i at frame t, and Wt ,i is defined as:

Wt ,i =      W t +2-t,1 , if dist J (i , i r ) = dist J (i, i r ) -1 W t +2-t,2 , if i = i W t +2-t,3 , if dist J (i , i r ) = dist J (i, i r ) + 1 (19)
Here, the set of weights {W u,v }, u, v = 1, 2, 3 completely defines the convolution filters in Eq. [START_REF] Ji | Interactive Body Part Contrast Mining for Human Interaction Recognition[END_REF]. Let N j,1 and N j,2 be the numbers of joints belonging to the arms and legs of two skeletons, respectively (see Fig. 1). Let X out,1 and X out,2 respectively of size N j,1 × N f × d and N j,2 × N f × d be the data associated with the arms and legs of two persons. The motivation behind this partition is that the interaction between two persons often involve those among their arms and those among their legs. ) whose parameters can be estimated as:

µ µ µ b l = 1 i b l i b l j=1 y b l,j , (20)
Σ Σ Σ b l = 1 i b l -1 i b l j=1 (y b l,j -µ µ µ b l )(y b l,j -µ µ µ b l ) T . (21)
Based on the theory developed in Section 4.1, the Gaussian (Σ Σ Σ b l , µ µ µ b l ) can be identified with the following matrix:

P b l = (det Σ Σ Σ b l ) -1 n+k Σ Σ Σ b l + kµ µ µ b l (µ µ µ b l ) T µ µ µ b l (k) (µ µ µ b l (k)) T I k . (22)
The above computations can be performed by a layer as:

{P b l } b=1,2 l=1,...,L = f gaussemb ({X out,b } b=1,2 ). ( 23
)
The next layer is designed to compute statistics on SPD manifolds and can be written by: {P b,m , P b,c } b=1,2 = f spdstats {P b l , W b pt } b=1,2 l=1,...,L , (24) where W b pt , b = 1, 2 are the parameters corresponding to the target points of PT (see Section 4.2). Specifically, P b,m is the mean of P b l , l = 1, . . . , L, and P b,c is given by:

P b,c = 1 L -1 L i=1 f v T P b,m ,W b pt ( -----→ P b,m P i ) × f v T P b,m ,W b pt ( -----→ P b,m P i ) T . ( 25
)
The next layer computes the embeddings of statistics P b,m , P b,c , b = 1, 2 and can be written as:

{B b } b=1,2 = f spdstatsemb {P b,m , P b,c } b=1,2 , (26)
where B b is the embedding matrix of (P b,m , P b,c ) given in the right-hand side of ( 16). The next layer transforms B b , b = 1, 2 to some matrices in LT + (n + k ) as:

{D b } b=1,2 = f trilmap {B b , W b lw } b=1,2 , (27)
where

D b = B b W b lw , W b lw , b = 1
, 2 are the parameters that are required to be in LT + (n + k ) so that the outputs D b are also in LT + (n + k ). The network then performs a projection:

{E b } b=1,2 = f triltoeud ({D b } b=1,2 ), (28)
where

E b = f lm (D b (D b ) T ), b = 1, 2.
Finally, a fullyconnected (FC) layer and a softmax layer are used to obtain class probabilities:

C out = f prob concat(f v (E 1 ), f v (E 2 )), W f c , (29)
where W f c are the parameters of the FC layer, the operator concat(V 1 , V 2 ) concatenates the two column vectors V 1 and V 2 vertically, and C out are the output class probabilities. We use the cross-entropy loss for training GeomNet.

Geometry Aware Constrained Optimization

Some layers of GeomNet rely on the eigenvalue decomposition. To derive the backpropagation updates for these layers, we follow the framework of [16] for computation of the involved partial derivatives. The optimization procedure for the parameters W b pt , W b lw , b = 1, 2 is based on the Adam algorithm in Riemannian manifolds [START_REF] Bécigneul | Riemannian Adaptive Optimization Methods[END_REF]. The Riemannian Adam update rule is given by:

x t+1 = exp xt -α mt √ vt + , (30)
where x t and x t+1 are respectively the parameters updated at timesteps t and t + 1, mt = m t /(1 -

β t 1 ), vt = v t /(1 -β t 2 ), m t = β 1 τ t-1 + (1 -β 1 )g t is a momentum term, v t = β 2 v t-1 + (1 -β 2 ) g t 2
xt is an adaptivity term, g t is the gradient evaluated at timestep t, α, , β 1 , β 2 are constant values. The squared Riemannian norm g t 2 xt =< g t |g t > xt corresponds to the squared gradient value in Riemannian settings. Here, < .|. > xt is the dot product for the Riemannian metric of the manifold in consideration, as discussed in Section 3.1. After updating x t+1 in Eq. ( 30), we update τ t as the PT of m t along geodesics connecting x t and x t+1 , i.e. τ t = T xt,xt+1 (m t ).

The update rule in Eq. ( 30) requires the computation of the exponential map and the PT. For SPD manifolds, these operations are given in Eqs. ( 5) and ( 9). It remains to define these operations for the update of the parameters W b lw , b = 1, 2. To this aim, we rely on the Riemannian geometry of LT + (n) studied in the recent work [26]. By considering the following metric:

< U, V > K = i>j U ij V ij + n j=1 U jj V jj K -2 jj , (31)
where

K ∈ LT + (n), U, V ∈ T K LT + (n),
U ij is the element on the i th row and j th column of U, Lin has shown [26] that the space LT + (n) (referred to as Cholesky space) equipped with the above metric forms a Riemannian manifold. On this manifold, the exponential map at a point can be computed as:

exp K U = K + U +D(K) exp(D(U)D(K) -1 ), ( 32
)
where

K ∈ LT + (n), U ∈ T K LT + (n), K is a matrix of the same size as K whose (i, j) element is K ij if i > j
and is zero otherwise, D(K) is a diagonal matrix whose (i, i) element is K ii . Also, the PT of a tangent vector U ∈ T K LT + (n) to a tangent vector at H ∈ LT + (n) is given by:

T K,H (U) = U + D(H)D(K) -1 D(U), (33)
where K, H ∈ LT + (n).

Experiments

Our network was implemented with Tensorflow deep learning framework and the experiments were conducted using two NVIDIA GeForce GTX 1080 GPUs. We used GeomStats library [START_REF] Miolane | geomstats: a Python Package for Riemannian Geometry in Machine Learning[END_REF] for geometric computations. The dimension d of output vectors at the convolutional layer, the number of clusters L, and the learning rate were set to 9, 180, and 10 -2 , respectively. The batch sizes were set respectively to 30 and 256 for the experiments on SBU Interaction dataset and those on NTU datasets. The values of the pair (k, k ) (see [START_REF] Bécigneul | Riemannian Adaptive Optimization Methods[END_REF] and ( 16)) were set to (2, 3) and (2, 1) for the experiments on SBU Interaction and NTU datasets, respectively. The values of α, , β 1 , and β 2 in the Riemannian Adam algorithm1 were set to 10 -3 , 10 -8 , 0.9, and 0.999, respectively [20]. In our experiments, GeomNet converged well after 600 epochs. For more details on our experiments, we refer the interested reader to the supplementary material.

Datasets and Experimental Settings

SBU Interaction dataset. This dataset [START_REF] Yun | Two-person Interaction Detection Using Bodypose Features And Multiple Instance Learning[END_REF] contains 282 sequences in 8 action classes created from 7 subjects. Each action is performed by two subjects where each subject has 15 joints. The joints 4,21,1,5,6,7,9,10,11,13,14,15,17,18,19 in Fig. 1 correspond respectively to the joints 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 of the first skeleton of SBU Interaction dataset. We followed the experimental protocol based on 5-fold cross validation with the provided training/testing splits [START_REF] Yun | Two-person Interaction Detection Using Bodypose Features And Multiple Instance Learning[END_REF].

NTU RGB+D 60 dataset. This dataset [START_REF] Shahroudy | NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[END_REF] contains 56,880 sequences created from 40 subjects with three cameras views and categorized into 60 classes. We followed the two experimental protocols cross-subject (X-subject) and cross-view (X-view) [START_REF] Shahroudy | NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[END_REF].

NTU RGB+D 120 dataset. This dataset [START_REF] Cichocki | Families of Alpha-Beta-and Gamma-Divergences: Flexible and Robust Measures of Similarities[END_REF] contains 114,480 sequences in 120 action classes, captured by 106 subjects with three cameras views. We followed the two experimental protocols cross-subject (X-subject) and crosssetup (X-setup) [START_REF] Cichocki | Families of Alpha-Beta-and Gamma-Divergences: Flexible and Robust Measures of Similarities[END_REF].

Ablation Study

In this section, we study the impact of different components of GeomNet on its accuracy 2 on SBU Interaction and NTU RGB+D 60 datasets.

Embedding dimensions. Here we investigate the impact of the parameters k and k (see [START_REF] Bécigneul | Riemannian Adaptive Optimization Methods[END_REF] and ( 16)). Fig. 2 shows the accuracies of GeomNet on SBU Interaction dataset with different settings of (k, k ), i.e. k = 0, 1, 2 and k = 0, . . . , 10. Note that when k = 0, the layer f gaussemb relies only on the covariance information. Also, when k = 0, the outputs B b , b = 1, 2 of the layer f spdstatsemb are simply obtained by the Cholesky decomposition of P b,c , i.e. B b (B b ) T = P b,c . It is interesting to note that Geom-Net achieves the best accuracy with (k, k ) = (2, 3), i.e. none of k and k is equal to 1. This is opposed to previous works [START_REF] Gong | Shape of Gaussians as Feature Descriptors[END_REF][START_REF] Li | Is Second-order Information Helpful for Large-scale Visual Recognition?[END_REF]41,[START_REF] Wang | G2DeNet: Global Gaussian Distribution Embedding Network and Its Application to Visual Recognition[END_REF], where n-variate Gaussians are always identified with SPD (n + 1) × (n + 1) matrices. To the best of our knowledge, this is the first work that shows the benefits of identifying n-variate Gaussians with SPD (n + k) × (n + k) matrices where k > 1. The results also reveal that the setting of (k, k ) has a non-negligible impact on the accuracy of GeomNet. Namely, the performance gap between two settings (k, k ) = (1, 1) (94.54%) 0 1 2 3 [START_REF] Brooks | Riemannian Batch Normalization for SPD Neural Networks[END_REF] and (k, k ) = (2, 3) (96.33%) is 1.79%. We can also notice that when k is fixed, GeomNet always performs best with k > 1. This shows the effectiveness of our parameterization of Riemannian Gaussians in (16).

To investigate the effectiveness of our proposed embedding of Gaussians outside of our framework, we used it to improve the state-of-the-art neural network on SPD manifolds SPDNet [START_REF] Huang | A Riemannian Network for SPD Matrix Learning[END_REF]. In [START_REF] Huang | A Riemannian Network for SPD Matrix Learning[END_REF], the authors performed action recognition experiments by representing each sequence by a joint covariance descriptor. The covariance descriptor is computed from the second order statistics of the 3D coordinates of all body joints in each frame. For SBU Interaction dataset, the size of the covariance matrix is 90×90 (30 body joints in each frame). In our experiment, we combined the covariance matrix and the mean vector using the proposed embedding of Gaussians to represent each sequence. Each sequence is then represented by a SPD (90 + k) × (90 + k) matrix. We used the code of SPDNet3 published by the authors. Fig. 3 shows the accuracies of SPDNet4 on SBU Interaction dataset with different settings of k. As can be observed, SPDNet gives the best accuracy with the setting k = 10. The performance gap between two settings k = 1 (90.5%) and k = 10 (92.38%) is 1.88%. The accuracy of SPDNet when using only the covariance (k = 0) is 79.48%, which is significantly worse than its accuracy with the setting k = 10. The results confirm that our proposed embedding of Gaussians is effective in the framework of SPDNet and that it is advantageous over the one of [START_REF] Lovrić | Multivariate Normal Distributions Parametrized As a Riemannian Symmetric Space[END_REF]. This suggests that our method could also be beneficial to previous works that rely on Gaussians to capture local feature distribution, e.g. [START_REF] Gong | Shape of Gaussians as Feature Descriptors[END_REF][START_REF] Li | Local Log-Euclidean Multivariate Gaussian Descriptor and Its Application to Image Classification[END_REF][START_REF] Li | Is Second-order Information Helpful for Large-scale Visual Recognition?[END_REF]36,41,[START_REF] Wang | G2DeNet: Global Gaussian Distribution Embedding Network and Its Application to Visual Recognition[END_REF]. Parallel transport. Tab. 1 gives the accuracies of Geom-Net without using PT on SBU Interaction and NTU RGB+D 60 datasets. The accuracies of GeomNet are also shown for comparison purposes. When PT is not used, the covariance in Eq. ( 12) is computed as:

P c = 1 L -1 L i=1 f v ---→ P m P i f v ---→ P m P i T . (34)
It can be seen that the use of PT is crucial for obtaining high accuracy. Specifically, on NTU RGB+D 60 dataset, computing the covariance without PT results in a loss of 31.44% on X-Subject protocol and a loss of 29.49% on X-View protocol. On SBU Interaction dataset, a significant reduction in accuracy (24.82%) can also be observed when PT is not used. These results highlight the importance of learning the parameters W b pt , b = 1, 2 in GeomNet. Lower triangular matrix learning. Tab. 2 gives the accuracies of GeomNet without using the layer f trilmap on SBU Interaction and NTU RGB+D 60 datasets. Again, the accuracies of GeomNet are also shown for comparison purposes. We can note that the introduction of the layer f trilmap brings performance improvement, i.e. 1.43% on SBU Interaction dataset, and 1.32% on X-Subject protocol and 1.27% on X-View protocol on NTU RGB+D 60 dataset.

Results on SBU Interaction Dataset

Results of GeomNet and state-of-the-art methods on SBU Interaction dataset are given in Tab. 3. For SPDNet, we report its best accuracy using the embedding in (3) with k = 10. We can remark that the accuracies of most of the hand-crafted feature based methods [START_REF] Ji | Interactive Body Part Contrast Mining for Human Interaction Recognition[END_REF]56] the second worst accuracy among the competing methods. GeomNet achieves the best accuracy of 96.33%, which is 16.85% better than that of SPDNet.

Results on NTU RGB+D 60 Dataset

Tab. 4 shows the results of GeomNet and state-of-theart methods on NTU RGB+D 60 dataset. For ST-GCN and AS-GCN, we used the codes 5,6 published by the authors. For SPDNet, we report its best accuracy using the embedding in (3) with k = 3. We can observe that Geom-Net gives the best results on this dataset. Since ST-GCN is based on fixed skeleton graphs which might miss implicit joint correlations, AS-GCN improves it by learning actional links to capture the latent dependencies between joints. AS-GCN also extends the skeleton graphs to represent structural links. However, AS-GCN does not achieve significant improvements over ST-GCN. This indicates that actional and structural links in AS-GCN are still not able to cope with complex patterns in 3DTPIR. As can be seen, GeomNet outperforms ST-GCN and AS-GCN by large margins. We can also note a large performance gap between GeomNet and SPDNet. This can probably be explained by the fact that:

(1) GeomNet aims to learn inter-person joint relationships;

(2) GeomNet leverages the covariance information on SPD manifolds.

Results on NTU RGB+D 120 Dataset

Results of GeomNet and state-of-the-art methods on NTU RGB+D 120 dataset are given in Tab. 5. For SPDNet, we report its best accuracy using the embedding in (3) with k = 3. As can be observed, GeomNet performs best on this dataset. Note that LSTM-IRN performs significantly worse than GeomNet on this most challenging dataset. By adapting the graph structure in ST-GCN to involve connections between two skeletons, ST-GCN-PAM achieves significant improvements. However, ST-GCN-PAM is still outperformed by GeomNet by 3.21% on X-Subject protocol 7 . The results indicate that: (1) without any prior knowledge, automatic inference of intra-person and inter-person joint relationships is difficult; (2) even with prior knowledge, the state-of-the-art ST-GCN performs worse than GeomNet. Compared to the results on NTU RGB+D 60 dataset, the performance gap between GeomNet and SPDNet is more pronounced on this dataset. Notice that our method is based only on the assumption that the joints of the arms of two persons and those of their legs are highly correlated during their interaction. Therefore, no explicit assumption in pairwise joint connections is required for interaction recognition.

Conclusion

We have presented GeomNet, a neural network based on embeddings of Gaussians and Riemannian Gaussians for 3DTPIR. To improve the accuracy of GeomNet, we have proposed the use of PT and a layer that learns lower triangular matrices with positive diagonal entries. Finally, we have provided experimental results on three benchmarks showing the effectiveness of GeomNet.

Figure 1 :

 1 Figure 1: Illustration of body joints' positions (only the first skeleton is shown). The joint at the hip is selected as the root of the skeleton. The distance between joints 1 and 6 is 4. The joints 22,23,24,25 are not used in the convolution operation. The arms then contain the following joints: 5, 6, 7, 8, 9, 10, 11, 12. The legs contain the following joints: 13, 14, 15, 16, 17, 18, 19, 20 (figure reproduced from [52]).

  For each b ∈ {1, 2}, the set of N j,b N f d-dim feature vectors from X out,b is partitioned into L subsets using K-means clustering. Let y b l,1 , . . . , y b l,i b l be the feature vectors in the l th subset. We assume that y b l,1 , . . . , y b l,i b l are i.i.d. samples from a Gaussian (Σ Σ Σ b l , µ µ µ b l

Figure 3 :

 3 Figure 3: Accuracy of SPDNet on SBU Interaction dataset with different settings of k.

Table 1 :

 1 Effectiveness of PT on SBU Interaction and NTU RGB+D 60 datasets.

	Accuracy (%)	92 93 94 95 96		k=0 k=1 k=2
				5 6 7 8 9 10 k'
	Figure 2: Accuracy of GeomNet on SBU Interaction
	dataset with different settings of (k, k ).
		Dataset	SBU Interaction	NTU RGB+D 60 Dataset X-Subject X-View
		Without PT	71.51	62.18	66.83
		PT	96.33	93.62	96.32

Table 2 :

 2 Effectiveness of lower triangular matrix learning (LTML) on SBU Interaction and NTU RGB+D 60 datasets.

	Dataset	SBU Interaction	NTU RGB+D 60 Dataset X-Subject X-View
	Without LTML	94.90	92.30	95.05
	LTML	96.33	93.62	96.32

Table 3 :

 3 are lower than 90%. The state-of-the-art method [9] for skeleton-based action recognition only gives a modest accuracy of 80.35%, Recognition accuracy (%) of GeomNet and stateof-the-art methods on SBU Interaction dataset.

	Method		Accuracy
	Lie Group [56]		47.92
	Constrast Mining [18]	86.90
	Interaction Graph [23]	92.56
	Trust Gate LSTM [29]	93.30
	Hierarchical RNN [9]		80.35
	Deep LSTM+Co-occurence [68]	90.41
	SPDNet [13]		92.38
	GeomNet		96.33
	Method	X-Subject X-View
	ST-LSTM [30]	83.0	87.3
	ST-GCN [62]	86.75	91.17
	AS-GCN [22]	87.08	92.04
	LSTM-IRN [46]	90.5	93.5
	SPDNet [13]	74.85	76.07
	GeomNet	93.62	96.32

Table 4 :

 4 Recognition accuracy (%) of GeomNet and stateof-the-art methods on NTU RGB+D 60 dataset.

Table 5 :

 5 Recognition accuracy (%) of GeomNet and stateof-the-art methods on NTU RGB+D 120 dataset.

	Method	X-Subject X-Setup
	ST-LSTM [30]	63.0	66.6
	ST-GCN [62]	78.60	79.92
	AS-GCN [22]	77.83	79.30
	LSTM-IRN [46]	77.7	79.6
	ST-GCN-PAM [63]	83.28
	SPDNet [13]	60.72	62.08
	GeomNet	86.49	87.58

Our code deals with constrained and unconstrained parameters.

The results of GeomNet are averaged over

runs.

https://github.com/zhiwu-huang/SPDNet

The results are averaged over 10 runs.

https://github.com/yysijie/st-gcn

https://github.com/limaosen0/AS-GCN

Acknowledgments. We thank the authors of NTU RGB+D datasets for providing access to their datasets.