Di Yang
email: di.yang@inria.fr

Yaohui Wang
email: yaohui.wang@inria.fr

Antitza Dantcheva
email: antitza.dantcheva@inria.fr

Quan Kong

Lorenzo Garattoni

Gianpiero Francesca
email: gianpiero.francesca@toyota-europe.comquan.kong@woven-planet.global

Franc ¸ois Brémond
email: francois.bremond@inria.fr

LAC -Latent Action Composition for Skeleton-based Action Segmentation

Keywords:

Skeleton-based action segmentation requires recognizing composable actions in untrimmed videos. Current approaches decouple this problem by first extracting local visual features from skeleton sequences and then processing them by a temporal model to classify frame-wise actions. However, their performances remain limited as the visual features cannot sufficiently express composable actions. In this context, we propose Latent Action Composition (LAC) 1 , a novel self-supervised framework aiming at learning from synthesized composable motions for skeleton-based action segmentation. LAC is composed of a novel generation module towards synthesizing new sequences. Specifically, we design a linear latent space in the generator to represent primitive motion. New composed motions can be synthesized by simply performing arithmetic operations on latent representations of multiple input skeleton sequences. LAC leverages such synthesized sequences, which have large diversity and complexity, for learning visual representations of skeletons in both sequence and frame spaces via contrastive learning. The resulting visual encoder has a high expressive power and can be effectively transferred onto action segmentation tasks by end-to-end fine-tuning without the need for additional temporal models. We conduct a study focusing on transfer-learning and we show that representations learned from pre-trained LAC outperform the state-of-the-art by a large margin on TSU, Charades, PKU-MMD datasets.

Introduction

Human-centric activity recognition is a crucial task in realworld video understanding. In this context, skeleton data that can be represented by 2D or 3D human keypoints plays an important role, as it is complementary to other modalities such as RGB [START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Hara | Learning spatio-temporal features with 3D residual networks for actio recognition[END_REF][START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF][START_REF] Feichtenhofer | X3d: Expanding architectures for efficient video recognition[END_REF][START_REF] Ryoo | Assemblenet++: Assembling modality representations via attention connections[END_REF][START_REF] Li | Ct-net: Channel tensorization network for video classification[END_REF][START_REF] Wang | Tdn: Temporal difference networks for efficient action recognition[END_REF][START_REF] Arnab | Vivit: A video vision transformer[END_REF] and optical flow [START_REF] Karen | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Feichtenhofer | Convolutional two-stream network fusion for video action recognition[END_REF]. As the human skeleton modality has witnessed a tremendous boost in robustness w.r.t. content changes related to camera viewpoints and subject appearances, the study of recognizing activities directly from 2D/3D skeletons has gained increasing attention [START_REF] Du | Hierarchical recurrent neural network for skeleton based action recognition[END_REF][START_REF] Ding | Stfc: Spatio-temporal feature chain for skeleton-based human action recognition[END_REF][START_REF] Caetano | Skeleton image representation for 3D action recognition based on tree structure and reference joints[END_REF][START_REF] Yan | Spatial temporal graph convolutional networks for skeleton-based action recognition[END_REF][START_REF] Shi | Twostream adaptive graph convolutional networks for skeletonbased action recognition[END_REF][START_REF] Chen | Multi-scale spatial temporal graph convolutional network for skeleton-based action recognition[END_REF][START_REF] Song | Stronger, faster and more explainable: A graph convolutional baseline for skeleton-based action recognition[END_REF][START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF][START_REF] Chen | Channel-wise topology refinement graph convolution for skeleton-based action recognition[END_REF][START_REF] Li | Skeleton graph scattering networks for 3d skeleton-based human motion prediction[END_REF][START_REF] Duan | Revisiting skeleton-based action recognition[END_REF][START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF]. While aforementioned approaches have achieved remarkable success, such approaches often focus on trimmed videos containing single actions, which constitutes a highly simplified scenario. Deviating from this, in this work, we tackle the challenging setting of action segmentation in untrimmed videos based on skeleton sequences.

In untrimmed videos, activities are composable i.e., motion performed by a person generally comprises multiple actions (co-occurrence), each with the duration of a few seconds. Towards modeling long-term dependency among different actions, expressive skeleton features are required. Current approaches [33,[START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Piergiovanni | Learning latent superevents to detect multiple activities in videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] obtain such features through visual encoder such as AGCNs [START_REF] Shi | Twostream adaptive graph convolutional networks for skeletonbased action recognition[END_REF] pre-trained on trimmed datasets. However, due to the limited motion information in the trimmed samples, the performance of such features in classifying complex actions is far from satisfactory. Towards addressing this issue, we propose to construct synthesized composable skeleton data for training a more effective visual encoder, endowed with strong representability of subtle action details for action segmentation.

In this paper, we propose Latent Action Composition (LAC), a novel framework aiming at leveraging synthesized composable motion data for self-supervised action representation learning. As illustrated in Fig. 1 (left), as opposed to current self-supervised approaches [33,[START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Piergiovanni | Learning latent superevents to detect multiple activities in videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF], LAC learns action representations in two steps: a first action composition step is then followed by a contrastive learning step.

Action composition is a novel initialization step to train a generative module that can generate new skeleton sequences by combining multiple videos. As high-level motions are Figure 1. General pipeline of LAC. Firstly, in the representation learning stage (left), we propose (i) a novel action generation module to combine skeletons of multiple videos (e.g., 'Walking' and 'Drinking' shown in the top and bottom respectively). We then adopt a (ii) contrastive module to pre-train a visual encoder by learning data augmentation invariant representations of the generated skeletons in both video space and frame space. Secondly (right), the pre-trained visual encoder is evaluated by transferring to action segmentation tasks. difficult to combine directly by the joint coordinates (e.g., 'drink' and 'sitdown'), LAC incorporates a novel Linear Action Decomposition (LAD) mechanism within an autoencoder. LAD seeks to learn an action dictionary to express subtle motion distribution in a discrete manner. Such action dictionary incorporates an orthogonal basis in the latent encoding space, containing two sets of directions. The first set named 'Static' includes directions representing static information of the skeleton sequence, e.g., viewpoints and body size. The other set named 'Motion' includes directions representing temporal information of the skeleton sequence, e.g., the primitive dynamics of the action performed by the subject. The new skeleton sequence is generated via a linear combination of the learned 'Static' and 'Motion' directions. We adopt motion retargeting to train the autoencoder and the dictionary using skeleton sequences with 'Static' and 'Motion' information built from 3D synthetic data [30]. Once the action dictionary is constructed, in the following contrastive learning step, 'Static'/'Motion' information and action labels are not required and composable motions can be generated from any multiple input skeleton sequences by combining their latent 'Motion' sets.

The contrastive learning step aims at training a skeleton visual encoder such as UNIK [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] in a self-supervised manner, without the need for action labels (see Fig. 1

(middle)).

It is designed for the resulting visual encoder to be able to maximize the similarity of different skeleton sequences, that are obtained via data augmentation from the same original sequence, across large-scale datasets. Unlike current methods [START_REF] Ding | Motion-aware contrastive video representation learning via foreground-background merging[END_REF][START_REF] Huang | Self-supervised video representation learning by context and motion decoupling[END_REF][START_REF] Ranasinghe | Self-supervised video transformer[END_REF][START_REF] Huang | Self-supervised video representation learning by context and motion decoupling[END_REF]57,[START_REF] Li | 3d human action representation learning via cross-view consistency pursuit[END_REF][START_REF] Mao | Cmd: Self-supervised 3d action representation learning with cross-modal mutual distillation[END_REF][START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF] that perform contrastive learning for the video-level representations, we perform contrastive learning additionally on the frame space to finely maximize the per-frame similarities between the positive samples. Subsequently, the so-trained frame-level skeleton visual encoder is transferred and retrained on action segmentation datasets [START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF][START_REF] Gunnar | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF].

To assess the performance of LAC, we train the skeleton visual encoder on the large-scale dataset Posetics [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] and we evaluate the quality of the learned skeleton representations (see Fig. 1 (right)) by fine-tuning onto unseen action segmentation datasets (e.g., TSU [START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF], Charades [START_REF] Gunnar | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF], PKU-MMD [START_REF] Chunhui | Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding[END_REF]). Experimental analyses confirm that action composition and contrastive learning can significantly increase the expressive power of the visual encoder. The fine-tuning results outperform state-of-the-art accuracy.

In summary, the contributions of this paper include the following. (i) We introduce LAC, a novel generative and contrastive framework, streamlined to synthesize complex motions and improve the skeleton action representation capability. (ii) In the generative step, we introduce a novel Linear Action Decomposition (LAD) mechanism to represent high-level motion features thanks to an orthogonal basis. The motions for multiple skeleton sequences can thus be linearly combined by latent space manipulation. (iii) In the contrastive learning step, we propose to learn the skeleton representations in both, video and frame space to improve generalization onto frame-wise action segmentation tasks. (iv) We conduct experimental analysis and show that pretraining LAC on Posetics and transferring it onto an unseen target untrimmed video dataset represents a generic and effective methodology for action segmentation.

Related Work

Temporal Action Segmentation focuses on per-frame activity classification in untrimmed videos. The main challenge has to do with how to model long-term relationships among various activities at different time steps. Current methods mostly focus on directly using untrimmed RGB videos. Since untrimmed videos usually contain thousands of frames, training a single deep neural network directly on such videos is quite expensive. Hence, to solve this problem efficiently, previous works proposed to use a two-step method. In the first step, a pre-trained feature extractor (e.g., I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]) is applied on short sequences to extract corresponding visual features. In the second step, action segmentation is modeled as a sequence-to-sequence (seq2seq) task to trans- late extracted visual features into per-frame action labels. Temporal Convolution Networks (TCNs) [33,[START_REF] Dai | Pdan: Pyramid dilated attention network for action detection[END_REF][START_REF] Zhang | Temporal query networks for fine-grained video understanding[END_REF] and Transformers [START_REF] Dai | MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection[END_REF] are generally applied in the second step due to their ability to capture long-term dependencies.

Source: 𝒑 . ! ,0 ! 𝐫 !,# 𝐫 ! ! ,# ! 𝒅 ! " , 𝒅 ! # , … , 𝒅 ! $ , 𝒅 # " , 𝒅 # # , … , 𝒅 # % 𝑨 ! , 𝑨 " = 𝑎 !" , … , 𝑎 !# , 𝑎 "" , … , 𝑎 "$ = 𝒓 !," × 𝑫 $ 𝑨 ′ ! , 𝑨 ′ " = 𝑎 ′ !" , … , 𝑎 ′ !# , 𝑎 ′ "" , … , 𝑎 ′ "$ = 𝒓 ! ! ," ! × 𝑫 $ 𝐫 !! ! ," = 1 2 , %&' ( 𝑎 !% + 𝑎 ′ !% 𝐝 !% + , %&' ) 𝑎 "% 𝐝 "% 𝐫 !," ! = ,
Recently, few methods [START_REF] Dai | Learning an augmented rgb representation with cross-modal knowledge distillation for action detection[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] started to explore using skeletons in this task, in order to benefit from multi-modality information. In such methods, a pre-trained Graph Convolutional Network (GCN) such as AGCN [START_REF] Shi | Twostream adaptive graph convolutional networks for skeletonbased action recognition[END_REF] is used as a visual encoder to obtain skeleton features in the first step. However, unlike in pre-trained I3D which has strong generalizability across domains, pre-trained AGCN is not able to provide high-quality features due to its laboratory-based pre-trained dataset NTU-RGB+D [START_REF] Shahroudy | Ntu rgb+d: A large scale dataset for 3D human activity analysis[END_REF]. We found that the performance significantly decreases when the pre-trained model is applied to more challenging real-world untrimmed skeleton videos datasets such as TSU [START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] and Charades [START_REF] Gunnar | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF]. The main issue is that the pre-trained visual encoder does not have a sufficient expressive power to extract the complex action features especially for composable actions that often occur in real-world videos.

LAC differs from previous two-step methods. We propose a motion generative module to synthesize complex composable actions and to leverage such synthetic data to train a more general skeleton visual encoder [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] which is sensitive to composable action. Unlike previous approaches, the pre-trained visual encoder in LAC has stronger representation capability for skeleton sequences compared to previous two-step methods [START_REF] Dai | Learning an augmented rgb representation with cross-modal knowledge distillation for action detection[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] using pre-trained AGCN. In such strategy, the model can be end-to-end refined on the action segmentation tasks without need for the second stage.

Motion Retargeting aims to transfer motion from sequence of target subject onto source subject, where the main challenge lies in developing effective mechanisms to disentangle motion and appearance. As one of the most important applications of video generation [START_REF] Tulyakov | MoCoGAN: Decomposing motion and content for video generation[END_REF][START_REF] Wang | G3AN: Disentangling appearance and motion for video generation[END_REF][START_REF] Yaohui | ImaGINator: Conditional spatio-temporal gan for video generation[END_REF][START_REF] Yu | Generating videos with dynamics-aware implicit generative adversarial networks[END_REF][START_REF] Skorokhodov | Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2[END_REF][START_REF] Wang | Inmodegan: Interpretable motion decomposition generative adversarial network for video generation[END_REF], previous image-based motion retargeting approaches explore to leverage structure representations such as 2D human keypoints [START_REF] Villegas | Neural kinematic networks for unsupervised motion retargetting[END_REF]3,[START_REF] Chan | Everybody dance now[END_REF][START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF] and 3D human meshes [START_REF] Liu | Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis[END_REF][START_REF] Tuanfeng | Dance in the wild: Monocular human animation with neural dynamic appearance synthesis[END_REF] as motion guidance. Recently, self-supervised methods [START_REF] Siarohin | First order motion model for image animation[END_REF][START_REF] Siarohin | Motion representations for articulated animation[END_REF]68] showed remarkable results on human bodies and faces by only relying on data without extract information.

Skeleton-based methods [START_REF] Kfir Aberman | Skeletonaware networks for deep motion retargeting[END_REF][START_REF] Villegas | Contact-aware retargeting of skinned motion[END_REF]3,[START_REF] Kfir Aberman | Unpaired motion style transfer from video to animation[END_REF] focus on transferring motion across skeletons of different shapes. Previous method [3] showed that transferring motion across characters enforces the disentanglement of static and dynamic information in a skeleton sequence. While they have achieved good performance, such method is unable to compose different actions for creating novel actions. Our method is different, we seek to learn an orthogonal basis in the feature space to represent the action distribution in a linear and discrete manner. In such a novel strategy, both static and dynamic features can be learned from a single encoder and skeleton sequences with complex motions are able to be synthesized by simply modifying the magnitudes along the basis.

Self-supervised Skeleton Action Representation learning involves extracting spatio-temporal features from numerous unlabeled data. Current methods [START_REF] Yang | Selfsupervised video pose representation learning for occlusionrobust action recognition[END_REF][START_REF] Li | 3d human action representation learning via cross-view consistency pursuit[END_REF][START_REF] Tianyu | Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition[END_REF][START_REF] Mao | Cmd: Self-supervised 3d action representation learning with cross-modal mutual distillation[END_REF][START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF] adopt contrastive learning [START_REF] Tian | Contrastive multiview coding[END_REF][START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF][START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] as the pretext task to learn skeleton representations invariant to data augmentation. However, recent techniques [START_REF] Su | Self-supervised 3d skeleton action representation learning with motion consistency and continuity[END_REF][START_REF] Yang | Skeleton cloud colorization for unsupervised 3d action representation learning[END_REF][START_REF] Yang | Selfsupervised video pose representation learning for occlusionrobust action recognition[END_REF][START_REF] Li | 3d human action representation learning via cross-view consistency pursuit[END_REF][START_REF] Tianyu | Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition[END_REF][START_REF] Mao | Cmd: Self-supervised 3d action representation learning with cross-modal mutual distillation[END_REF][START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF] merge the temporal features by average pooling and conduct contrastive learning on top of the global temporal features for the skeleton sequences. Thus they may lose important information of complex actions particularly in the case of co-occurring actions [START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF][START_REF] Gunnar | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF]. In our work, we extend the visual encoder and the contrastive module to finely extract per-frame features. We use contrastive loss for both sequence and frame, to make sure that the skeleton sequences are discriminative in both spaces. The skeleton visual encoder can have a strong representation ability for the sequence and also for each frame to better generalize to frame-wise action segmentation tasks.

Proposed Approach

LAC is composed of two modules (see Fig. 1), a skeleton sequence generation module to synthesize the co-occurring actions and a self-supervised contrastive module to learn skeleton visual representations using the synthetic data. Subsequently, the skeleton visual encoder trained by the contrastive module can be transferred to downstream finegrained action segmentation tasks. In this section, we introduce the full architecture and training strategy of LAC.

Composable Action Generation

In this work, we denote the static information of a skeleton sequence (i.e., 'viewpoint', 'subject body size', etc.) as 'Static', while the temporal information (i.e., the dynamics of the 'action' performed by the subject) as 'Motion'. As shown in Fig. 2, the generative module is an autoencoder, consisting of an encoder and a decoder for skeleton sequences. To disentangle 'Motion' features from 'Static' in a linear latent space, we introduce a Linear Action Decomposition mechanism to learn an action dictionary where each direction represents a basic high-level action for the skeleton encoding. We apply motion retargeting for training the autoencoder (i.e., transferring the motion of a driving skeleton sequence to the source skeleton sequence maintaining the source skeletons invariant in viewpoint and body size). In the inference stage, the extracted 'Motion' features from multiple skeleton sequences can be combined linearly and composable skeletons can be generated by the decoder. The input skeletons can be in 3D or 2D.

Skeleton Sequence Autoencoder: The input skeleton sequence with 'Static' c and 'Motion' m is modeled by a spatio-temporal matrix, noted as p m,c ∈ R T ×V ×Cin . T , V , and C in respectively represent the length of the video, the number of body joints in each frame, and the input channels (C in = 2 for 2D data, or C in = 3 if we use 3D skeletons). As shown in Fig. 2 (i), LAC adopts an encoder E LAC to embed a pair of input skeleton sequences p m,c /p m ′ ,c ′ into r m,c /r m ′ ,c ′ ∈ R T ′ ×Cout . T ′ is the size of temporal dimension after convolutions and C out is the output channel size. To generate skeleton sequences, a skeleton sequence decoder D LAC (see Fig. 2 a.(iii)) is used to generate new skeleton sequences from the representation space. The autoencoder is designed by multiple 1D temporal convolutions and upsampling to respectively encode and decode the skeleton sequence. We provide in Tab. 1 and Supplementary Material (Appendix) building details of E LAC and D LAC .

Linear Action Decomposition: The goal of Linear Action Decomposition (LAD) is to obtain the 'Motion' features on top of the encoded latent code of a skeleton sequence (see Fig. 2 a.(ii)). Our insight is that the high-level action of a skeleton sequence can be considered as a combination of multiple basic and independent 'Motion' and 'Static' transformations (e.g., raising hand, bending over) with their amplitude from a fixed reference pose (i.e., standing in the front view, see Fig. 4). Hence, we explicitly model the basic 'Static' and 'Motion' transformations using a unified action dictionary for the encoded latent skeleton features. Specifically, we first pre-define a learnable orthogonal basis, noted as

D v = {d m1 , d m2 , ..., d mJ , d c1 , d c2 , ..., d cK } with J ∈ [1, C out ) and K = C out -J,
where each vector indicates a basic 'Motion'/'Static' transformation from the reference pose. Due to D v entailing an orthogonal basis, any two directions d i , d j follow the constraint:

< d i , d j >= 0 i ̸ = j 1 i = j. (1)
We implement D v ∈ R Cout×Cout as a learnable matrix and we apply the Gram-Schmidt algorithm during each forward pass in order to satisfy the orthogonality. Then, we consider the 'Motion' features of p m,c , denoted as r m , as a linear combination between motion orthogonal directions in D v , and associated magnitudes (amplitude) A m = {a m1 , a m2 , ..., a mJ }. Similarly, the 'Static' features r c are the linear combination between 'Static' orthogonal directions in D v , and associated magnitudes A c = {a c1 , a c2 , ..., a cK }. For p m ′ ,c ′ , we can obtain its decomposed components r m ′ , r c ′ in the same way:

rm = J i=1 amidm i , rc = K i=1 acidc i , r m ′ = J i=1 a ′ mi dm i , r c ′ = K i=1 a ′ ci dc i . (2)
For the skeleton encoding r m,c /r m ′ ,c ′ , the set of magnitudes A m /A ′ m and A c /A ′ c can be computed as the projections of r m,c /r m ′ ,c ′ onto D v , as Eq. 3:

ami = < rm,c • dm i > ∥dm i ∥ 2 , aci = < rm,c • dc i > ∥dc i ∥ 2 , a ′ mi = < r m ′ ,c ′ • dm i > ∥dm i ∥ 2 , a ′ ci = < r m ′ ,c ′ • dc i > ∥dc i ∥ 2 .
(3)

As r m,c has the temporal dimension of size T ′ , for each 'Motion' feature in the temporal dimension, we can obtain T ′ × sets of motion magnitudes A m to represent the temporal dynamics of r m . For r c , as static information, we firstly merge the temporal dimension of r m,c by average pooling and then

Stages .

E LAC D LAC E V Input 2D sequence Rep. 2D sequence [T, 2V ] [T ′ , 160] [T × V,
L self = E[∥DLAC(rm + rc) -pm,c∥ 2 ], Ltarget = E[∥DLAC(rm + r c ′ ) -p m,c ′ ∥ 2 ]. (4)
Moreover, at each iteration, the decoder is also encouraged to re-compose new combinations. As the generative module is trained on a synthetic dataset [30] including the cross-character motion retargeting ground-truth skeleton sequences, we can explicitly apply the cross reconstruction loss L target (see Eq. 4) through the generation. The same reconstruction losses are also computed for p m ′ ,c ′ .

Inference (Motion Composition):

As the trained LAD represents high-level motions in a linear space by the action dictionary, we can generate at the inference stage (see Fig. 2 (iv)) composable motions by the linear addition of 'Motion' features encoded from multiple skeleton sequences. We use the average latent 'Motion' features for the decoder to generate composable motions. We note that even if in some cases the combined motions may not be realistic, it can still help to increase the expressive power of the representation, which is important to express subtle details. Taking the motion combination of the two sequences p m,c and p m ′ ,c ′ as an example, the skeleton sequences p mm ′ ,c and p mm ′ ,c ′ with the combined motions m and m ′ are generated as follows:

p mm ′ ,c = DLAC 1 2 (rm + r m ′ ) + rc , p mm ′ ,c ′ = DLAC 1 2 (rm + r m ′ ) + r c ′ . (5)
As skeleton sequences p mm ′ ,c and p mm ′ ,c ′ have the same composed motion but different 'Static' (e.g., viewpoints), they can form a positive pair for self-supervised contrastive learning to train a transferable skeleton visual encoder for fine-grained action segmentation tasks in Sec. 3.2.

Self-supervised Skeleton Contrastive Learning

In this section, we provide details of the self-supervised contrastive module of LAC. We re-denote the generated composable skeleton sequence p mm ′ ,c (in Sec. 3.1) as a query clip q and multiple positive keys (e.g., the sequence p mm ′ ,c ′ ), denoted as k + 1 , ..., k + P , can be generated by only modifying its 'Static' magnitudes A c in the latent space. We follow the general contrastive learning method [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] based on the momentum encoder, to maximize the mutual information of positive pairs (i.e., the generated composable skeleton sequences with the same motion but different Statics), while pushing negative pairs (i.e., other skeleton sequences with different Motions) apart. Deviating from [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF], the queue (memory) [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] stores the features of each frame for skeleton sequences and we propose to additionally enhance the perframe representation similarity of positive pairs. The visual encoder can extract skeleton features that are globally invariant and also finely invariant to data augmentation and can generalize better to frame-wise action segmentation tasks.

Skeleton Visual Encoder:

To have a strong capability to extract skeleton spatio-temporal features, we adopt the recent topology-free skeleton backbone network UNIK [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] as the skeleton visual encoder E V (see Tab. 1 and Appendix for details). To obtain the global sequence space, we adopt temporal average pooling layer to merge the temporal dimension of the visual representations, denoted as E Vs (q), E Vs (k + 1 ), ..., E Vs (k + P ) ∈ R Cout×1 (see Tab. 1). Per-frame features can be obtained by E V before the temporal average pooling layer (see Tab. 1) and denoted as E Vf (q, τ ), E Vf (k + 1 , τ ), ..., E Vf (k + P , τ ) ∈ R Cout×T . Contrastive Loss: We apply general contrastive InfoNCE loss [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF] to train our visual encoder E V to encourage similarities between both sequence-level and frame-level representations of positive pairs, and discourage similarities between negative representations, denoted as E Vs (k - 1 ), ..., E Vs (k - N ) in sequence space and E Vf (k - 1 , τ ), ..., E Vf (k - N , τ ) in frame space. The InfoNCE [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF] objective is defined as: L q = L q-s + L q-f , where

Lq-s = -E log P p=1 e Sim E Vs (q),E Vs (k + p ) N n=1 e Sim E Vs (q),E Vs (k - n ) , (6)
L q-f = -E log P p=1 e T τ =1 Sim E Vf (q,τ ),E Vf (k + p ,τ ) N n=1 e T τ =1 Sim E Vf (q,τ ),E Vf (k - n ,τ ) , (7)
where τ represents the frame index in the temporal dimension of frame-level representations, P represents the number of positive keys, N denotes the number of negative keys (we use P = 4 and N = 65, 536 for experiments), and the similarity is computed as:

Sim(x, y) = ϕ(x) • ϕ(y) ∥ϕ(x)∥ • ∥ϕ(y)∥ • 1 T emp , (8)
where T emp refers to the temperature hyper-parameter [START_REF] Wu | Unsupervised feature learning via non-parametric instance discrimination[END_REF], and ϕ is a learnable mapping function (e.g., a MLP projection head [START_REF] Feichtenhofer | A large-scale study on unsupervised spatiotemporal representation learning[END_REF]) that can substantially improve the learned representations.

Transfer-Learning for Action Segmentation: For transferring the visual encoder on downstream tasks, we attach E Vf to a fully-connected layer followed by a Softmax Layer to predict per-frame actions. The output size of each fullyconnected layer depends on the number of action classes (see Tab. 1). Then, we re-train the visual encoder E V with action labels. For processing long sequences, we adopt a sliding window to extract features for a temporal segment and use Binary Cross Entropy loss to optimize the visual encoder step by step. In this way, E V can be re-trained endto-end instead of pre-extracting features for all frames. In the inference stage, we combine the predictions of all the temporal sliding windows in an online manner [START_REF] Li | Online human action detection using joint classification-regression recurrent neural networks[END_REF].

Experiments and Analysis

In this section, we conduct extensive experiments to evaluate LAC on both generation and action segmentation tasks. Firstly, we study the generalization ability of LAC by quantifying the performance improvement obtained by transfer-learning on target action segmentation datasets (i.e., Toyota Smarthome Untrimmed, Charades and PKU-MMD) after pre-training on the large-scale dataset Posetics. Secondly, we evaluate the quality of the skeleton sequences generated by LAC using the synthetic dataset Mixamo. Finally, we provide an exhaustive ablation study. See Appendix for implementation details, limitation discussion and additional studies, e.g., computational cost analysis.

Datasets and Evaluation Protocols

Posetics [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] contains 142,000 real-world trimmed video clips from Kinetics-400 [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF] with corresponding 2D and 3D skeletons. We use Posetics to pre-train the contrastive model of LAC with skeleton data and we study the transfer-learning on skeleton-based action segmentation.

Toyota Smarthome Untrimmed (TSU) [START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] is a large-scale real-world dataset for daily living action segmentation. It contains densely annotated long-term composite activities where up to 5 actions can happen at the same time in a given frame. We only use the provided 2D skeleton data [START_REF] Yang | Selective spatio-temporal aggregation based pose refinement system: Towards understanding human activities in real-world videos[END_REF] for the experiments. For evaluation, we report per-frame mAP (mean Average Precision) as [START_REF] Dai | Pdan: Pyramid dilated attention network for action detection[END_REF][START_REF] Dai | MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection[END_REF] following the crosssubject (CS) and cross-view (CV) evaluation protocols. Charades [START_REF] Gunnar | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF] is a real-world dataset containing finegrained activities similar to TSU. It provides only raw video clips without skeleton data. In work, we use the 2D skeleton data (2D coordinates) estimated by the toolbox [START_REF] Yang | Selective spatio-temporal aggregation based pose refinement system: Towards understanding human activities in real-world videos[END_REF].

Methods

We report per-frame mAP on the localization setting of the dataset. For sake of reproducibility, we will release the estimated skeleton data on Charades. [START_REF] Chunhui | Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding[END_REF] is a basic untrimmed video dataset recorded in the laboratory setting. We use only the official 3D skeleton data. As this dataset is not densely labeled, we report the event-based mAP for fair comparisons by applying a post-processing [START_REF] Luo | Graph distillation for action detection with privileged modalities[END_REF] on the frame-level predictions to get the action boundaries.

PKU-MMD

Mixamo [30] is a 3D animation collection, which contains elementary actions and various dancing moves. We use such a synthetic dataset for training and evaluating the generation module in LAC prior to contrastive learning on Posetics.

Evaluation on Temporal Action Segmentation

In this section, we evaluate the transfer ability of LAC by both linear evaluation (i.e., by training only the fullyconnected layer while keeping frozen the backbone) and fine-tuning evaluation (i.e., by refining the whole network) on three action segmentation datasets TSU, PKU-MMD and Charades with self-supervised pre-training on Posetics. We also report the results with supervised pre-training for reference (i.e., we use the generated composable skeletons and the combined action labels for pre-training).

Linear Evaluation: Tab. 5 (top) shows the linear results on the three datasets. This evaluates the effectiveness of transfer-learning with fewer parameters (only the classifier is trained) compared to training directly on the target datasets from scratch (random initialization). The results suggest that the weights of the model can be well pre-trained without action labels, providing a strong transfer ability (e.g., +10.4% on TSU CS and +6.6% on Charades) and the pre-trained visual encoder is generic enough to extract meaningful action features from skeleton sequences.

Fine-tuning: Tab. 5 (bottom) shows the fine-tuning results, where the whole network is re-trained. The self-supervised pre-trained model also performs competitively compared to supervised pre-trained models. From these results we conclude that collecting a large-scale trimmed skeleton dataset, without the need of action annotation, can be beneficial to downstream fine-grained tasks for untrimmed videos (e.g., +5.9% on TSU CS and +11.8% on CV).

Training with fewer labels: In many real-world applications, labeled data may be lacking, which makes it challenging to train models with good performance. To evaluate LAC in such cases, we transfer the visual encoder pre-trained on Posetics onto all the tested datasets by fine-tuning with only 5% and 10% of the labeled data. As shown in Tab. 4, without pre-training, the accuracy of the visual encoder [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] significantly decreases. In contrast, LAC with prior action representation learning achieves good performance on all three datasets in such setting.

Comparison with SoTA: We compare our fine-tuning results to other SoTA skeleton-based approaches [START_REF] Graves | Framewise phoneme classification with bidirectional lstm and other neural network architectures[END_REF][START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF][START_REF] Li | Online human action detection using joint classification-regression recurrent neural networks[END_REF][START_REF] Li | Skeleton boxes: Solving skeleton based action detection with a single deep convolutional neural network[END_REF][START_REF] Chunhui | Pku-mmd: A large scale benchmark for continuous multi-modal human action understanding[END_REF][START_REF] Chen | Hierarchically self-supervised transformer for human skeleton representation learning[END_REF][START_REF] Li | Joint distance maps based action recognition with convolutional neural networks[END_REF] on the real-world datasets TSU and Charades (see Tab. 2) and also laboratory dataset PKU-MMD (see Tab. 3). As previous approaches are based on supervised pre-training on large-scale datasets [START_REF] Liu | Ntu rgb+d 120: A large-scale benchmark for 3D human activity understanding[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF], we also report our supervised results. The results in Tab. 2 show that LAC, even with self-supervised pre-training, outperforms all previous skeleton-based approaches [START_REF] Graves | Framewise phoneme classification with bidirectional lstm and other neural network architectures[END_REF][START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] with supervised pre-training on our main target real-world datasets in a large margin (e.g., +7.4% on TSU CS and +12.5% on Charades). It suggests that composable motions are important to increase the expressive power of the visual representation and the end-to-end fine-tuning can benefit downstream tasks. Even if PKU-MMD does not contain composable actions, the performance is still slightly improved by learning a fine-grained skeleton representation. The results using RGB data are also reported for reference. The TSU and Charades datasets contain many object-oriented actions that are difficult to identify using skeleton data only. However, even in the absence of the object information, LAC surprisingly achieves better accuracy compared to all SoTA RGB-based methods [START_REF] Dai | Pdan: Pyramid dilated attention network for action detection[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF][START_REF] Dai | MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection[END_REF][START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Dai | Learning an augmented rgb representation with cross-modal knowledge distillation for action detection[END_REF]. We deduce that training the visual encoder end-to-end is more effective compared to using two-step processing. Moreover, skeletons can always be combined with RGB data by multi-modal fusion networks [START_REF] Dai | Learning an augmented rgb representation with cross-modal knowledge distillation for action detection[END_REF][START_REF] Das | Vpn++: Rethinking video-pose embeddings for understanding activities of daily living[END_REF] to further improve the performance.

𝒅 𝑚8 a 𝑚8 + - 𝒅 𝑚32 a 𝑚32 + - 𝒅 𝑚1 a 𝑚1 + - 𝒅 𝑚2 a 𝑚2 + - 𝒅 𝑚4 a 𝑚4 + - 𝒅 𝑚64 a 𝑚64 + -

Evaluation on Action Generation.

As the generative model with LAD represents our main novelty for addressing the action segmentation challenges, we evaluate here the generation quality of LAC. sition ability before the contrastive learning. We compare the motion retargeting accuracy on this dataset. Specifically, we randomly split training and test sets on this dataset and follow the same setting and protocol described in [3,[START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF]. We firstly explore how many directions (i.e., the values of J and K) are required in the proposed action dictionary D v . We empirically test four different values for J from 16 to 144. From results reported in Tab. 6, we observe that when using 128 directions (out of all dim=160 directions) for 'Motion', the model achieves the best reconstruction accuracy and outperforms SoTA methods

Ablation Study

To understand the contribution of the two individual components of LAC, we conduct ablation experiments on our main target fine-grained dataset TSU, with self-supervised pre-training and fine-tuning protocol.

Impact of Action Composition:

We start from a baseline model [START_REF] Yang | Unik: A unified framework for real-world skeleton-based action recognition[END_REF] that is pre-trained on the trimmed dataset (i.e., Posetics) in a general contrastive learning strategy [START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF] without using composable motions and frame-level contrast for action segmentation. The results in Tab. 7 (see L0) suggest that the visual encoder has a weak capability to learn features on top of an untrimmed skeleton sequence without learning a composable action representation. We then perform the self-supervised training on Posetics (in only the video space) with composable motions from different number of motions. As daily living videos contain in average two co-occurring actions [START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF], combining motions from two skeleton sequences in the pre-training stage can significantly improve the representation ability of the visual encoder and generalize better to real-world untrimmed action segmentation tasks (see Tab. 7 L1). Such number can simply be changed to adapt to different target datasets. Impact of Frame-wise Contrast: To validate that framewise contrastive learning can further improve the finegrained action segmentation tasks, we additionally maximize the per-frame similarity between the positive samples. We also select different uniform temporal sampling rates to reduce the redundant computational cost instead of using all the frames. The results in Tab. 7 L2 suggest that frame-wise contrast with uniformly sampling every 4 frames is the most effective to improve the action segmentation accuracy.

Further Discussion

Transfer Learning vs. Self Pre-training: Our target is to train a generic skeleton encoder that can fit different downstream tasks. Hence, similar to current RGB-based methods using large-scale dataset such as Kinetics [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Carreira | A short note on the kinetics-700 human action dataset[END_REF] LAC with SoTA methods [START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF] in Tab. 8 with self pretraining, i.e., solely self-supervised pre-training the encoder on the tested dataset (on TSU, PKU-MMD CS-IoU@0.1 and Charades) using the proposed contrastive module without additional data and without action labels. The results show that, without extra training data, LAC can still outperform previous models [START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF], as in the second stage, LAC adopts end-to-end fine-tuning to refine the visual encoder, which is more effective than using temporal modeling on the preextracted features [START_REF] Piergiovanni | Temporal gaussian mixture layer for videos[END_REF][START_REF] Dai | Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection[END_REF]. Moreover, current untrimmed datasets are not large enough, the generated actions have less diversity, so the representation ability of the skeleton encoder is less impressive than pre-training on Posetics.

Conclusion

In this work, we present LAC, a novel self-supervised action representation learning framework for the setting of skeleton action segmentation. We show that high-level motions of skeleton sequences can be learned and linearly combined using an orthogonal basis in the latent space. Moreover, we augment a contrastive learning module to better extract frame-level features, in addition to the generated composable skeleton sequences. Our experimental analysis confirms that a skeleton visual encoder that extracts such skeleton representation is able to boost downstream action segmentation tasks. Future work will extend our generative approach to RGB videos, in order to improve the capturing of the object information, which can be crucial and complementary to the skeleton-based model.

Figure 2 .

 2 Figure 2. Overview of the Composable Action Generation model in LAC. The model consists of a visual encoder ELAC and a decoder DLAC. In the latent space, we apply Linear Action Decomposition (LAD) by learning a visual action dictionary Dv, which is an orthogonal basis where each vector represents a basic 'Motion'/'Static' transformation. Given a pair of skeleton sequences pm,c and p m ′ ,c ′ , (i) their latent codes rm,c and r m ′ ,c ′ are embedded by ELAC. (ii) Their projections Am, Ac and A m ′ , A c ′ along Dv can be computed. The linear combination of Am/A m ′ with corresponding directions in Dv constitutes the 'Motion' features and similarly the 'Static' features can also be obtained. (iii) In the training stage, we leverage motion retargeting for learning the whole framework by swapping their 'Motion' features and generating transferred motions. (iv) In the inference stage, we adopt linear combination of rm and r m ′ to obtain the composable motion features r mm ′ and the composable skeleton sequences can be generated.

  conduct the projection process to obtain a unified A c . With such trained LAD, the decoder D LAC can generate different skeleton sequences by taking an arbitrary combination of magnitudes A m and A c along their corresponding directions as input. The high-level action can thus be controlled by the manipulations in the latent space. Training (Motion Retargeting): We apply a general motion retargeting [3] to train the generative autoencoder and ensure that 'Motion' directions in LAD orthogonal basis D v are 'Static'-disentangled (see Fig. 2 (iii)). The main training loss function is the reconstruction loss: L gen =L rec .Reconstruction loss aims at guiding the network towards a high generation quality. The new retargeted (motion swapped) skeleton sequence with 'Motion' m, and 'Static' c ′ , noted as p m,c ′ is generated from the recombined features, r m + r c ′ . Similarly, p m ′ ,c can also be generated by swapping the pair of sequences. The skeleton sequence generation can be formulated as p m,c ′ = D LAC (r m + r c ′ ) and p m ′ ,c = D LAC (r m ′ + r c ). The reconstruction loss consists of two components: L rec = L self + L target . Specifically, at every training iteration, the decoder network D LAC is firstly used to reconstruct each of the original input samples p m,c using its representation r m + r c . This component of the loss is denoted as L self and formulated as a standard autoencoder reconstruction loss (see Eq. 4)

Figure 3 .

 3 Figure 3. Motion composition visualization. The input pair of videos and corresponding skeleton sequences (left) have simple motions. The generated skeleton sequences (right) are composed by both motions while keeping their respective viewpoint and body size ('Static') invariant.

Figure 4 .

 4 Figure 4. Linear manipulation of six 'Motion' directions in Dv on a skeleton sequence. Results indicate that each direction represents a meaningful motion transformation from a 'reference pose' marked in red (e.g., dm 8 for squat, dm 32 for bending over).

  Quantitative Comparison: The generation model of LAC is trained on the Mixamo dataset to have an action compo-LAC w/ D v (Ours) size J = 16, K = 144 1.23 size J = 32, K = 128 1.02 size J = 64, K = 96 0.88 size J = 128, K = 32 0.82 size J = 144, K = 16 0.85

  [START_REF] Villegas | Neural kinematic networks for unsupervised motion retargetting[END_REF] 3,[START_REF] Yang | Via: Viewinvariant skeleton action representation learning via motion retargeting[END_REF]. Hence, we set J=128 and K=32 for all other experiments.Motion Direction Interpretation and Visualization: We visualize an example of motion composition inference of two videos. Fig.3demonstrates that 'Static' and 'Motion' are well disentangled and the high-level motions can be effectively composed by decoding the linear combination of both latent 'Motion' components learned by the proposed LAD.To further understand what each direction in D v represents, we proceed to visualize d mi . We generate skeletons for a single input skeleton sequence using its disentangled 'Static' features r c combined by different r m respectively obtained by a linearly grown a mi on its corresponding 'Motion' directions d mi (see Fig.4for visualization of six directions), where other magnitudes on directions except d mi are set to 0. We find that each direction represents a basic high-level motion transformation (e.g., d m32 represents bending over) and the corresponding magnitude represents the range of the motion. All motion transformations start from a fixed 'reference pose', regardless of original motions of the input skeleton sequences. Such a 'reference pose' can be considered as a normalized form of the given skeleton sequence. In such a learning strategy, complex motions can be combined and the motion diversity can be controlled in an interpretive way by latent space manipulation. More real-world examples with different viewpoints are provided in the Appendix.

Table 1 .

 1 Main building blocks of the autoencoder ELAC, DLAC and the skeleton visual encoder EV in LAC. We take the 2D sequence as example. The dimensions of kernels are denoted by t × s, c (2D kernels) and t, c (1D kernels) for temporal, spatial, channel sizes. S/T-GAP, FC denotes temporal/spatial global average pooling, and fully-connected layer respectively. Rep. indicates the learned representation.

	2]

Table 2 .

 2 Frame-level mAP on TSU and Charades for comparison with SoTA action segmentation methods. RGB-based results (top) are shown for reference. Mod.: Modality.

	Methods	Mod.	TSU CS(%) CV(%) mAP(%) Charades
	TGM [46]	RGB	26.7	-	13.4
	PDAN [15]	RGB	32.7	-	23.7
	SD-TCN [16]	RGB	29.2	18.3	21.6
	MS-TCT [14]	RGB	33.7	-	25.4
	Bi-LSTM [26]	Skeleton 17.0	14.8	8.2
	TGM [46]	Skeleton 26.7	13.4	9.0
	SD-TCN [16]	Skeleton 26.2	22.4	9.8
	LAC-unsup (Ours) Skeleton 34.1	22.8	22.3
	LAC-sup (Ours)	Skeleton 36.8	23.1	25.6
	Methods	Mod.	PKU-MMD mAP@IoU 0.1(%) 0.3(%) 0.5(%)
	GRU-GD [42]	RGB	82.4	81.3	74.3
	SSTCN-GD [13]	RGB	83.7	82.1	76.5
	Augmented-RGB [13]	RGB	86.3	84.5	81.1
	JCRRNN [39]	Skeleton 45.2	-	32.5
	Convolution Skeleton [12] Skeleton 49.3	31.8	12.1
	Skeleton boxes [34]	Skeleton 61.3	-	54.8
	Hi-TRS [10]	Skeleton	-		-	67.3
	Window proposal [35]	Skeleton 92.2	-	90.4
	LAC-unsup (Ours)	Skeleton 91.8	90.2	88.5
	LAC-sup (Ours)	Skeleton 92.6	91.4	90.6

Table 3

 3

. Event-level mAP on PKU-MMD CS at IoU thresholds of 0.1, 0.3 and 0.5 for comparison with SoTA methods. RGB-based results (top) are shown for reference. Mod.: Modality.

Table 4 .

 4 Transfer learning results by fine-tuning on all benchmarks of Toyota Smarthome Untrimmed, PKU-MMD and Charades with randomly selected 5% (top) and 10% (bottom) of labeled training data.

		Pre-training	Training data	Toyota Smarthome Untrimmed PKU-MMD (IoU=0.1) Charades CS(%) CV(%) CS(%) CV(%) mAP(%)
	Random init. [73]	Scratch	5%		8.5	6.8	57.4		59.5	8.8
	Self-supervised	Posetics w/o labels	5%		25.2	15.6	73.9		75.4	12.6
	Random init. [73]	Scratch	10%		12.9	9.5	66.4		68.1	9.3
	Self-supervised	Posetics w/o labels	10%		29.0	17.9	79.8		81.1	17.4
	Methods	Pre-training	Toyota Smarthome Untrimmed #Params CS(%) CV(%)	PKU-MMD (IoU=0.1) #Params CS(%) CV(%) #Params mAP(%) Charades
	Random init.	Scratch	13.1K	8.1	6.9	13.3K	11.8	12.4	40.2K	6.1
	Supervised	Posetics w/ labels	13.1K	20.8	18.3	13.3K	61.8	62.4	40.2K	14.3
	Self-supervised Posetics w/o labels	13.1K	18.5	16.6	13.3K	55.2	58.8	40.2K	12.7
	Random init.	Scratch	3.45M	28.2	11.0	3.45M	86.5	92.9	3.45M	18.6
	Supervised	Posetics w/ labels	3.45M	36.8	23.1	3.45M	92.6	94.6	3.45M	25.6
	Self-supervised Posetics w/o labels	3.45M	34.1	22.8	3.45M	91.8	93.9	3.45M	22.3

Table 5 .

 5 Transfer-learning results by linear evaluation (top) and fine-tuning (bottom) on Toyota Smarthome Untrimmed, PKU-MMD and Charades with self-supervised pre-training on Posetics. Results with supervised pre-training are also reported for reference.

Table 6 .

 6 Quantitative comparisons of LAC to other SoTA motion retargeting methods on the Mixamo dataset.

Table 8 .

 8 for pretraining, our model is pre-trained on the large-scale Posetics dataset to learn a generic skeleton representation. Such representation can be transferred onto different downstream tasks without the need for individual pre-training. This is a very effective practice for action segmentation models. To demonstrate the advantage of transfer-learning and to further compare LAC with SoTA methods, we here compare Fine-tuning results (i.e., Frame-level mAP on TSU andCharades and Event-level mAP on PKU-MMD) with individual pre-training only on the target action segmentation datasets for further comparison with SoTA methods.

	Dataset	TGM [46] SD-TCN [16] LAC (Ours)
	TSU-CS(%)	25.6	24.4	33.2
	TSU-CV(%)	13.9	20.8	21.7
	Charades(%)	9.1	8.7	21.4
	PKU-MMD(%)	87.3	87.5	91.0

Acknowledgements: This work was supported by Toyota Motor Europe (TME) and the French government, through the 3IA Cote d'Azur Investments In the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002. This work was performed using HPC resources from GENCI-IDRIS (Grant AD011012798R1).


Thanh-Trung Van
email: van@emse.fr

Michel Beigbeder

Web Co-citation: Discovering Relatedness Between Scientific Papers

In this paper we review two well-known citation methods to find relatedness between scientific papers: co-citation and bibliographic coupling. We propose a practical method to estimate the co-citation relatedness using the Google search engine. We call this method Web co-citation. We conducted experiments on a collection of scientific papers to compare the performances of different methods. The experimental results show that our approach, despite its simplicity, is efficient in discovering the relatedness between scientific papers.

Introduction

For a long time, citation-based methods have been used to find relatedness between scientific papers beside content-based methods. In 1963 Kessler [START_REF] Kessler | Bibliographic coupling between scientific papers[END_REF] proposed the bibliographic coupling method. In this method, the similarity between two pap e r si sb a s e do nt h en u m b e ro ft h e i rco-references. He supposed that if two papers have common references in their bibliographies, they may focus (entirely or partially) on the same topic. In 1973 Marshakova [START_REF] Marshakova | System of document connections based on references[END_REF] and Small [START_REF] Small | Co-citation in the scientific literature: A new measure of the relationship between two documents[END_REF] independently proposed another method called "co-citation". In this method, the relatedness between two papers is based on their co-citation frequency. The co-citation frequency is the frequency that two papers are co-cited. Two papers are said to be co-cited if they appear together in the bibliography section of a third paper.

The two methods bibliographic coupling and co-citation have been used widely since about 40 years for different purposes. The digital library CiteSeer 1 uses these methods to find related papers. In [START_REF] Lai | Using the patent co-citation approach to establish a new patent classification system[END_REF] the co-citation method is used to create a patent classification system for conducting patent analysis and management. Recently, these methods are used in hyperlinked environments to find the relatedness between Web pages [START_REF] Pitkow | Life, death, and lawfulness on the electronic frontier[END_REF][START_REF] Dean | Finding related pages in the world wide web[END_REF] because of the similarity between the notion of "citations between scientific papers" and "links between Web pages". However, both of these methods have their limits. In the bibliographic coupling method, the relatedness between two papers is fixed since their publication date because they are based on the number of their co-references which is unchanged.

1 http://citeseer.ist.psu.edu/ 1

In the co-citation methods, with the time two related papers may receive more and more citations and their co-citation frequency can increase. However if we want to know this citation information, we have to extract from the citation graph of the actual library or read from a citation database2 which are usually limited; i.e. we can only know citing papers of a given paper if these citing papers exist within the same digital library or citation database. That is why in this paper we propose an approach to compute co-citation relatedness between scientific papers which can overcome this limit.

The rest of this paper is organized as follows. In Sec. 2 we describe two approaches to compute co-citation relatedness between scientific papers: traditional approach using the Web of Science citation database and our new approach using the Google search engine. Sec. 3 presents our experiments: simulation of personalized searching using different citation methods. The paper concludes in Sec. 4.

Methodology

Using Web of Science as Citation Database

Actually, there are many citation databases like Web of Science3 , Scopus4 and digital libraries like CiteSeer, ACM Digital Library which provide citation information about scientific papers. After regarding in detail these sources, we decided to choose Web of Science (WoS) as a citation database in our experiments. The Web of Science of Thomson ISI is an important citation database which is used widely for citation studies [START_REF] Jacso | As we may search : Comparison of major features of the web of science, scopus, and google scholar citation-based and citation-enhanced databases[END_REF]. Besides, it also provides an API which facilitates the access to its database without using an Web browser. Another important reason for using WoS is that it contains most of journals used in our experiments (see Sec. 3.)

In WoS, an article is represented by a primary key called UT. Its API supports many operations on its database. Thanks to the search service of ISI, if we know some information about a paper (like title, year of publication, journal etc.) we can use these information to find the UT primary key of this paper in WoS database by calling the searchRetrieve function. Then using this UT primary key we can find all papers that cite this paper with the citingArticles function. From these information we can know the number of times that a paper is cited or the frequency that two papers are co-cited in WoS database. More documentation about ISI search service could be found in its support site5 .

Web Co-citation Method

With the explosion of the World Wide Web, Web search engines have to be more and more complete in order to satisfy information needs of users and their databases become bigger with the time. With their huge databases, Web search engines could be a good source for many data mining tasks.

Recently, a new method for citation analysis called Web citation analysis begins attracting the research community. Web citation analysis finds citations to a scientific paper on the Web by sending the query containing the title of this paper (as phrase search using quotation marks) to a Web search engine and analyze returned pages [START_REF] Vaughan | Bibliographic and web citations: what is the difference?[END_REF]. Because a Web search engine can index many kinds of documents in many different formats, the notion of "citation" used here is a "relaxation" in comparison with traditional definition

In our Web co-citation method, we compute the co-citation similarity of two scientific papers by the frequency that they are "co-cited" on the Web; i.e. the frequency that they are mentioned by a Web page. The notion of "co-citation" used here is also a "relaxation" in comparison with the traditional definition. If the Web document that mentions two scientific papers is another scientific paper then these two papers are normally co-cited. However, if this is a table of content of a conference proceeding, we could also say that these two papers are co-cited and have a relation because a conference normally has a common general theme. If these two papers appear in the same conference, they may have the same general theme. Similarly, if two papers are in the reading list for a course, they may focus on the same topic of this course. In summary, if two papers appear in the same Web document, we can assume that they have a (strong or weak) relation. The search engine used in our experiment is the Google search engine. To find the number of times that two papers are "co-cited", we send the titles of these two papers (as phrase search and in the same query) to Google and note the number of hits returned. In our experiments, we use a script to automatically query Google instead of manually using a Web browser.

Experiments

As stated above, in this work we conduct experiments for evaluating performance of two methods: bibliographic coupling and co-citation with Web of Science and Google. The experiments described here are simulations of personalized searching in a digital library using user profiles. Users of information retrieval systems generally use short queries to describe their information need. Because of the polysemy and synonym problems of natural language, these short queries become ambiguous and lead to wrong answers. However if the system knows about user, it can use these information to improve searching performance. The information about each user is called user profile. Generally, a user profile is a set of information that represent interests and preferences of a user.

Test Collection and Evaluation Procedure

The test collection that we use in our experiments is the collection used in INEX 2005 (version 1.9 6 ). In the first step we remove all elements that are not scientific papers. After this process, the collection contains 14237 documents. Then we extract all necessary information for our experiments from these documents (title, journal, publication year, bibliography etc.). There are also many topics with relevance assessments distributed with the collection. Each topic represents an information need and the relevance assessments were done by INEX participants. In our experiments we use only CO topics which do not contain structure of documents to create user queries. INEX uses a two-dimensional, multi-valued scale for relevance assessments of each topic. However in our experiments we use precision/recall metrics with binary scale relevant/non-relevant). Therefore we did a transformation on the relevance assessments of INEX: if a document has at least one element which is judged relevant (entirely or partially), this document will be considered as relevant; otherwise it will be considered as non-relevant. There are 29 original CO topics but only 20 topics that have more than 30 relevant documents will be used for experiments.

As mentioned above, our experiments are simulations of personalized searching using user profiles. In this case, 20 topics represent different information needs of 20 different people. For each topic, we choose some relevant papers as "pseudo user profile" of this person (5 in average in our experiments). (Please note that our goal is not to learn user profiles but to evaluate citation-based methods). The selected papers are chosen among the highly relevant papers to the correspondence topic and those that receive many citations from other documents. The papers which are included in these profiles are removed from the collection to avoid effect on the experimental results.

After the preparation step, we use the zettair7 search engine to index the INEX collection (the default model is Dirichlet-smoothed), then we send 20 queries (which are formed from above topics) to zettair;w i t he a c hq u e r yw e take the first 300 documents for re-ranking using "user profiles"of correspondence topic. The similarity between a document d and a user profile p is computed as:

similarity(p, d)= d ′ ∈p similarity(d ′ ,d)( 1 )
In Eq. 1, similarity(d ′ ,d) is the similarity (bibliographic coupling and cocitation) between a document d ′ in profile p and document d. The co-citation similarity between two papers is defined as:

cocitation similarity(d ′ ,d)=ln( cocitation(d ′ , d ) 2 citation(d ′ ) • citation(d) ) ( 2 )
In Eq. 2, cocitation(d ′ ,d ′ ) is the number of times that these two papers are co-cited, citation(d ′ )andcitation(d) are respectively the citation frequency that papers d ′ and d received. The bibliographic coupling similarity is computed by a similar formula. The final score of a document is obtained by combining its original score computed by zettair and the similarity document-profile. In our experiments we tried two combining functions: a linear function and a product function. However, in our experiments the product combination seems to be better than linear combination, thus it is used in final results which are presented in the next part.

Results and Discussion

The experimental results are presented in Fig. 1 (precision/recall) and Tab. 1 (precision at 5, 10, 20, 30 documents). The trec eval 8 program is used for evaluation. From the experimental results, we can see that the co-citation method using the WoS database does not bring any improvement, it even causes a slight performance decrease. The bibliographic coupling method performs better but not very clearly. The co-citation method using Google is the best, it brings 15.06% improvement for the precision at top 30 documents. Now we will analyze the experimental data to explain these results. To compute the similarity between documents and "profiles" for re-ranking, we have to compute the co-citation (or co-reference) frequency of 25497 pairs of documents (each pair consists of a document to be re-ranked and a document in a "user profile"). In the co-citation methods using Web of Science database, only 213 pairs are co-cited with the average co-citation frequency of each pair is 1.94. This small number of co-cited pairs is the reason why it could not bring any improvement and even becomes a noisy source which causes bad effect on the final result. In the bibliographic coupling method, there are 1126 pairs of documents which have co-references with the average number of co-references of each pair is 1.69. This is a little better than the first case and it is able to make some improvement. In the co-citation method using Google, there are 4845 pairs of documents which are "co-cited"with the average co-citation frequency of each pair is 4.84. This is much better than the first two cases. That is why it gains the best performance.

Conclusions and Future Work

In this paper we consider two famous citation-based methods: bibliographic coupling and co-citation. We propose new approach to compute co-citation relatedness between scientific papers using the Google search engine. Experimental results show that such approach could be more efficient than the traditional approach. We believe that this new approach could be successfully applied to other applications like classification, clustering of scientific papers, finding related papers etc. Another approach we are considering is to combine multiple different citation databases that could lead to better performance of co-citation method.

Fig. 1 .

 1 Fig. 1. Experimental results: re-ranking search results of zettair with different citationbased methods

Table 1 .

 1 Precision at 5, 10, 20, 30 documents

	Original Bibliographic Co-citation Co-citation
	Result coupling	using WoS using Google
	At 5 docs 0.6600	0.7300	0.6300	0.7100
	At 10 docs 0.6150	0.6050	0.5900	0.6800
	At 20 docs 0.5375	0.5600	0.5150	0.6025
	At 30 docs 0.4867	0.4883	0.4567	0.5600

A citation database is a system that can provide bibliographic/citation information of papers.

http://portal.isiknowledge.com

http://www.scopus.com/scopus/home.url

http://scientific.thomson.com/support/faq/webservices/

http://inex.is.informatik.uni-duisburg.de/2005/

http://www.seg.rmit.edu.au/zettair

http://trec.nist.gov/trec eval/