Philipp Müller
email: philipp.mueller@dfki.de

François Brémond
email: francois.bremond@inria.fr

Michal Balazia
email: michal.balazia@inria.fr

Bodily Behaviors in Social Interaction: Novel Annotations and State-of-the-Art Evaluation

Keywords: Computing methodologies → Image and video acquisition, Activity recognition and understanding, • Human-centered computing → Collaborative and social computing, • Applied computing → Psychology dataset, body pose, gesture, social signals, behavior detection

Body language is an eye-catching social signal and its automatic analysis can significantly advance artificial intelligence systems to understand and actively participate in social interactions. While computer vision has made impressive progress in low-level tasks like head and body pose estimation, the detection of more subtle behaviors such as gesturing, grooming, or fumbling is not well explored. In this paper we present BBSI, the first set of annotations of complex Bodily Behaviors embedded in continuous Social Interactions in a group setting. Based on previous work in psychology, we manually annotated 26 hours of spontaneous human behavior in the MPIIGroupInteraction dataset with 15 distinct body language classes. We present comprehensive descriptive statistics on the resulting dataset as well as results of annotation quality evaluations. For automatic detection of these behaviors, we adapt the Pyramid Dilated Attention Network (PDAN), a state-of-the-art approach for * Authors contributed equally.

INTRODUCTION

Bodily movements and poses are a key aspect of human behavior in social interaction [START_REF] Vinciarelli | Social signal processing: Survey of an emerging domain[END_REF] and are indicative of a large variety of personal and interpersonal information [START_REF] Oneri | A Review of Communication, Body Language and Communication Conflict[END_REF]. For example, leaning of the torso was found to be related to liking of the addressee [START_REF] Mehrabian | Relationship of attitude to seated posture, orientation, and distance[END_REF], and behaviors like fumbling, grooming or face touching are related to the regulation of stress [START_REF] Bardi | Behavioral and physiological correlates of stress related to examination performance in college chemistry students[END_REF]. Furthermore, body language was shown to have distinct effects both on its perceivers as well as on its producers. Dominant, open bodily displays can be perceived as attractive [START_REF] Vacharkulksemsuk | Dominant, open nonverbal displays are attractive at zero-acquaintance[END_REF] and gesturing was shown to lighten the cognitive load [START_REF] Goldin-Meadow | Explaining math: Gesturing lightens the load[END_REF][START_REF] Ping | Gesturing saves cognitive resources when talking about nonpresent objects[END_REF] and improve memory [START_REF] Cook | Gesturing makes memories that last[END_REF]. As a result, machines that are supposed to understand and participate in social interactions need to be able to accurately sense and interpret body language.

Over recent decades, huge advances were made in human bodyand hand pose estimation [START_REF] Andriluka | People-tracking-bydetection and people-detection-by-tracking[END_REF][START_REF] Cao | OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields[END_REF][START_REF] Insafutdinov | Deepercut: A deeper, stronger, and faster multi-person pose estimation model[END_REF][START_REF] Ng | Body2hands: Learning to infer 3d hands from conversational gesture body dynamics[END_REF]. At the same time, a large number of works investigated the prediction of high-level attributes based on bodily behavior [START_REF] Agrawal | Multimodal Personality Recognition using Cross-attention Transformer and Behaviour Encoding[END_REF][START_REF] Beyan | Moving as a leader: Detecting emergent leadership in small groups using body pose[END_REF][START_REF] Müller | Emotion recognition from embedded bodily expressions and speech during dyadic interactions[END_REF]. For example, body movements were utilized for detection of emergent leadership [START_REF] Beyan | Moving as a leader: Detecting emergent leadership in small groups using body pose[END_REF] and recognition of emotions [START_REF] Müller | Emotion recognition from embedded bodily expressions and speech during dyadic interactions[END_REF] or personality types [START_REF] Agrawal | Multimodal Personality Recognition using Cross-attention Transformer and Behaviour Encoding[END_REF]. These approaches typically use generic feature sets extracted from pose estimates or rely on CNN-based visual representations. While such approaches have the advantage of being relatively task-agnostic, they run the danger of missing subtle differences in behavior, such as between scratching and fumbling, that can only be exploited with fine-grained annotation. They further suffer from subjective or ambiguous annotation and from the lack of interpretability associated with a psychologically-motivated mid-level representation of behavior [START_REF] Girard | Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses[END_REF][START_REF] Lya | Study of application and validation of the Ethological Coding System for Interviews (ECSI)[END_REF][START_REF] Scherer | Automatic audiovisual behavior descriptors for psychological disorder analysis[END_REF][START_REF] Troisi | Ethological research in clinical psychiatry: the study of nonverbal behavior during interviews[END_REF], which is especially important if a behavior analysis is supposed to be accepted by practitioners like clinical or organizational psychologists.

Despite the advantages of a mid-level representation of bodily behavior in human interactions, automatic approaches for the detection of such behaviors are scarce [START_REF] Beyan | Analysis of face-touching behavior in large scale social interaction dataset[END_REF][START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF]. The main reason for this is the lack of suitable datasets for training and evaluation. The few existing datasets either only cover a single behavior like touching the face with the hands [START_REF] Beyan | Analysis of face-touching behavior in large scale social interaction dataset[END_REF], or focus on single people only and are at present not publicly available [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF]. To overcome this limitation, we present the first publicly available annotations of a comprehensive set of body language classes embedded in continuous group conversations. Our choice of behavior classes is motivated by previous work in psychology [START_REF] Troisi | Ethological research in clinical psychiatry: the study of nonverbal behavior during interviews[END_REF]. As a basis for annotation, we make use of a naturalistic multi-view group interaction dataset [START_REF] Müller | Multi-Mediate: Multi-modal Group Behaviour Analysis for Artificial Mediation[END_REF][START_REF] Müller | Detecting low rapport during natural interactions in small groups from non-verbal behaviour[END_REF] which will enable future research to study body language in the context of high-level social phenomena such as leadership, rapport, or liking.

Our specific contributions are threefold: First, we introduce Bodily Behaviors in Social Interaction (BBSI), a set of novel annotations for 15 bodily behavior classes on the MPIIGroupInteraction dataset [START_REF] Müller | Detecting low rapport during natural interactions in small groups from non-verbal behaviour[END_REF]. BBSI comprises 2.87 million frames of annotated behavior classes from 26 hours of human behavior embedded in continuous group interactions. Second, we provide detailed descriptive analyses on the collected annotations as well as the results of a dedicated experiment quantifying annotator agreement. Third, we evaluate several state-of-the-art action detection approaches on BBSI, reaching 61.3% True Positive Rate with the Pyramid Dilated Attention Network [START_REF] Dai | PDAN: Pyramid Dilated Attention Network for Action Detection[END_REF] and Swin Transformer [START_REF] Liu | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows[END_REF] features. 1 1 Data and code are available at https://git.opendfki.de/body_language/acm_mm22.

RELATED WORK

Our work is related to the function of body language in social interactions, to approaches for the recognition of actions and body language, as well as to existing human body language datasets.

Body Language in Social Interaction

Body language has been actively researched by psychologists for decades [START_REF] Goldin-Meadow | Explaining math: Gesturing lightens the load[END_REF][START_REF] Mehrabian | Relationship of attitude to seated posture, orientation, and distance[END_REF][START_REF] Vacharkulksemsuk | Dominant, open nonverbal displays are attractive at zero-acquaintance[END_REF]. Early work by Mehrabian [START_REF] Mehrabian | Relationship of attitude to seated posture, orientation, and distance[END_REF][START_REF] Mehrabian | Encoding of attitude by a seated communicator via posture and position cues[END_REF] found that, among other signals, backward leaning of the torso is indicative of liking. Dominant and open nonverbal displays, as opposed to folded arms and crossed legs, are perceived as attractive when meeting with strangers [START_REF] Vacharkulksemsuk | Dominant, open nonverbal displays are attractive at zero-acquaintance[END_REF]. In a meta-analysis, [START_REF] Hall | Nonverbal Behavior and the Vertical Dimension of Social Relations: A Meta-Analysis[END_REF] found significant correlations between perceived social verticality and for example self-touching and gesturing. A further study by [START_REF] Dana | 2005-06-01. Beliefs about the nonverbal expression of social power[END_REF] indicated that people believe power is expressed with nonverbal cues like open posture (i.e. no arms crossed or legs crossed), more gesturing, and less self-touching (both hands and face). Furthermore, leaning towards the interlocutor was shown to be associated with rapport [START_REF] Sharpley | When does counsellor forward lean influence client-perceived rapport[END_REF], and crossed arms were shown to be associated with emotion expressions [START_REF] Harald | Bodily expression of emotion[END_REF]. Displacement behaviors such as grooming, face touching or fumbling are related to anxiety and stress regulation [START_REF] Bardi | Behavioral and physiological correlates of stress related to examination performance in college chemistry students[END_REF][START_REF] Mohiyeddini | Displacement behaviour is associated with reduced stress levels among men but not women[END_REF][START_REF] Mohiyeddini | Neuroticism and stress: The role of displacement behavior[END_REF].

As a consequence of these manifold connections of body language with important personal and social attributes, body language analysis has been a focus of automatic approaches attempting to infer high-level attributes such as emotion [START_REF] Gavrilescu | Recognizing emotions from videos by studying facial expressions, body postures and hand gestures[END_REF][START_REF] Müller | Emotion recognition from embedded bodily expressions and speech during dyadic interactions[END_REF][START_REF] Hiranmayi Ranganathan | Multimodal emotion recognition using deep learning architectures[END_REF], leadership role [START_REF] Beyan | Moving as a leader: Detecting emergent leadership in small groups using body pose[END_REF][START_REF] Müller | Emergent leadership detection across datasets[END_REF], or personality type [START_REF] Agrawal | Multimodal Personality Recognition using Cross-attention Transformer and Behaviour Encoding[END_REF][START_REF] Romeo | Predicting apparent personality from body language: benchmarking deep learning architectures for adaptive social human-robot interaction[END_REF]. In contrast to the human science studies discussed above, these automatic approaches commonly lack an explicit intermediate representation of functional bodily behavior categories. Instead, they rely on a generic feature representation encoding body postures and movements [START_REF] Beyan | Moving as a leader: Detecting emergent leadership in small groups using body pose[END_REF][START_REF] Müller | Emotion recognition from embedded bodily expressions and speech during dyadic interactions[END_REF][START_REF] Müller | Emergent leadership detection across datasets[END_REF] or on deep learning approaches [START_REF] Hiranmayi Ranganathan | Multimodal emotion recognition using deep learning architectures[END_REF][START_REF] Romeo | Predicting apparent personality from body language: benchmarking deep learning architectures for adaptive social human-robot interaction[END_REF] without easily interpretable internal structure. While such representations can be effective in prediction scenarios, they often lack interpretability and may miss subtle but meaningful differences, e.g. between fumbling and scratching. In this work, we draw upon the ethological rating scheme of functional body language categories described in [START_REF] Troisi | Ethological research in clinical psychiatry: the study of nonverbal behavior during interviews[END_REF] to derive a set of bodily behaviors that are intuitively interpretable and allow to train models for fine-grained behavior distinctions.

Recognition of Actions and Body Language

RGB-based human action recognition has often been addressed by three main approaches. Two-stream 2D Convolutional Neural Networks [START_REF] Karpathy | Large-Scale Video Classification with Convolutional Neural Networks[END_REF][START_REF] Simonyan | Two-Stream Convolutional Networks for Action Recognition in Videos[END_REF][START_REF] Zong | Motion saliency based multi-stream multiplier ResNets for action recognition[END_REF] generally contain two 2D CNN branches taking different input features extracted from the RGB videos for action recognition. Recurrent Neural Networks (RNN) [START_REF] Donahue | Long-Term Recurrent Convolutional Networks for Visual Recognition and Description[END_REF][START_REF] Liu | Spatiotemporal Saliency-Based Multi-Stream Networks with Attention-Aware LSTM for Action Recognition[END_REF][START_REF] Ng | Beyond Short Snippets: Deep Networks for Video Classification[END_REF] usually employ 2D CNNs as feature extractors for an LSTM model. 3D CNNbased methods [START_REF] Fayyaz | 3D CNNs with Adaptive Temporal Feature Resolutions[END_REF][START_REF] Stroud | D3D: Distilled 3D Networks for Video Action Recognition[END_REF][START_REF] Tran | A Closer Look at Spatiotemporal Convolutions for Action Recognition[END_REF] extend 2D CNNs to 3D structures, to simultaneously model the spatial and temporal context information in videos that is crucial for action recognition.

Among the many available human action recognition methods we choose the following three for our evaluations: A well-cited two-stream 2D CNN architecture by Wang et al. [START_REF] Wang | Temporal Segment Networks: Towards Good Practices for Deep Action Recognition[END_REF] which divides each video into three segments and processes each segment with a two-stream network, fusing the individual classification scores by an average pooling method to produce the video-level prediction. A revolutionary method by Carreira and Zisserman [START_REF] Carreira | Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset[END_REF] which Table 1: Datasets annotated with body language classes described in the literature. Behaviors indicates the number of annotated body language classes, Participants the number of human individuals, Length the length of annotated behavior, Views the number of synchronized camera views on each participant, Group Size the number of participants that were synchronously annotated, Spontaneous whether behavior was shown spontaneously, and Public whether the dataset is publicly available. NTU is in italic, as only a subset of its classes are body language. introduces the two-stream Inflated 3D CNN inflating the convolutional and pooling kernels of a 2D CNN with an additional temporal dimension. And the best performance method tested by [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF] on body language recognition by Lin et al. [START_REF] Lin | TSM: Temporal Shift Module for Efficient Video Understanding[END_REF] of a parameter-free Temporal Shift Module, which shifts a part of the channels along the temporal dimension to perform temporal interaction between the features from adjacent frames. We also experiment with the transformer method by Liu et al. [START_REF] Liu | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows[END_REF] that was designed for natural language processing but its application has been recently extended to computer vision tasks [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF][START_REF] Khan | Transformers in Vision: A Survey[END_REF].

Name

In contrast to action recognition, which typically considers freely moving people [START_REF] Das | Toyota Smarthome: Real-World Activities of Daily Living[END_REF][START_REF] Kay | The Kinetics Human Action Video Dataset[END_REF][START_REF] Gunnar | Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding[END_REF], the much thinner work on body language recognition addresses more constrained social interaction scenarios. For example, Yang et al. [START_REF] Yang | Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation[END_REF] generate sequences of body language predictions from estimated human poses and feed them to an RNN for emotion interpretation and psychiatric symptom prediction. Kratimenos et al. [START_REF] Kratimenos | Independent Sign Language Recognition with 3d Body, Hands, and Face Reconstruction[END_REF] extract a holistic 3D body shape, including hands and face, from a single image and feed them also to an RNN for sign language recognition. Singh et al. [START_REF] Singh | EDBL -algorithm for detection and analysis of emotion using body language[END_REF] use handcrafted features to analyze body language for estimating a person's emotions and state of mind. Santhoshkumar et al. [START_REF] Santhoshkumar | Deep Learning Approach for Emotion Recognition from Human Body Movements with Feedforward Deep Convolution Neural Networks[END_REF] use Feedforward Deep CNNs for detecting emotions from full body motions. We observe that the common denominator of body language analysis methods are the employment of a general action recognition method and the lack of a benchmark body language dataset.

Human Body Language Datasets

In contrast to datasets with annotations of high-level attributes like emotions [START_REF] Müller | Emotion recognition from embedded bodily expressions and speech during dyadic interactions[END_REF][START_REF] Hiranmayi Ranganathan | Multimodal emotion recognition using deep learning architectures[END_REF], leadership [START_REF] Beyan | Moving as a leader: Detecting emergent leadership in small groups using body pose[END_REF][START_REF] Müller | Detecting low rapport during natural interactions in small groups from non-verbal behaviour[END_REF], or personality [START_REF] Palmero | Context-aware personality inference in dyadic scenarios: Introducing the udiva dataset[END_REF], datasets annotated with concrete classes of bodily behavior are sparse. Table 1 summarizes four relevant datasets with manual body language annotations. Research that extracted body language automatically but did not provide human annotations is not included [START_REF] Douglas-Cowie | The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data[END_REF][START_REF] Gavrilescu | Recognizing emotions from videos by studying facial expressions, body postures and hand gestures[END_REF][START_REF] Hiranmayi Ranganathan | Multimodal emotion recognition using deep learning architectures[END_REF].

In EMILYA [START_REF] Fourati | Emilya: Emotional body expression in daily actions database[END_REF], actors were asked to express different emotions while performing daily actions such as walking, sitting down, or moving objects. Two datasets NTU RGB+D 60/120 [START_REF] Liu | NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding[END_REF][START_REF] Shahroudy | NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[END_REF] contain a large number of general action classes that also include a number of body language classes. However, these datasets do not consist of spontaneous human behavior. The two most relevant datasets for our work are the PAVIS Face-Touching dataset [START_REF] Beyan | Analysis of face-touching behavior in large scale social interaction dataset[END_REF] and iMiGUE [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF]. PAVIS Face-Touching is similar to BBSI as it also consists of recordings of group discussions. In contrast to the 15 behavior classes annotated in BBSI, PAVIS Face-Touching only has binary annotations of whether a participant touches her face or not. Furthermore it only has a single frontal view on each participant. The recently introduced iMiGUE dataset [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF] consists of annotations of 32 behaviors classes of speakers at sports press conferences. Annotations are only provided for a single person (i.e. no annotations of discussion partners), and only a single view on the target person is provided. At the time of submission, the iMiGUE videos are not publicly accessible due to privacy issues2 . We hereby present the first publicly available annotations of body language on a multi-view dataset of three to four people engaged in spontaneous group discussions.

DATASET

BBSI builds upon the MPIIGroupInteraction dataset [START_REF] Müller | Detecting low rapport during natural interactions in small groups from non-verbal behaviour[END_REF]. This dataset comprises of 22 three-to four-person group discussion on controversial topics, each lasting for 20 minutes. In total, it consists of 78 participants and 26 hours of behavior recordings. Every interaction was recorded by 8 frame-synchronized cameras as well as with 4 microphones. After the discussions, participants rated their perceived leadership, competence, dominance and liking of all other members, as well as their feelings of rapport towards each other. In addition to rapport and emergent leadership prediction [START_REF] Müller | Emergent leadership detection across datasets[END_REF][START_REF] Müller | Detecting low rapport during natural interactions in small groups from non-verbal behaviour[END_REF], the dataset was further annotated and used for eye contact detection [START_REF] Fu | Using Motion Histories for Eye Contact Detection in Multiperson Group Conversations[END_REF][START_REF] Müller | Robust eye contact detection in natural multi-person interactions using gaze and speaking behaviour[END_REF] and for next speaker prediction [START_REF] Birmingham | Group-Level Focus of Visual Attention for Improved Next Speaker Prediction[END_REF][START_REF] Müller | Multi-Mediate: Multi-modal Group Behaviour Analysis for Artificial Mediation[END_REF]. This wealth of already existing annotations makes the MPIIGroupInteraction dataset a perfect choice for the collection of body language labels as it will allow future research on the connections and the utility of body language information with key group phenomena.

Body Language Annotation

We densely annotated the full MPIIGroupInteraction dataset with 15 body language classes (see Figure 1 and Table 2). Our set of behavior classes is based on the the Ethological Coding System for Interviews (ECSI) [START_REF] Troisi | Ethological research in clinical psychiatry: the study of nonverbal behavior during interviews[END_REF]. This coding system includes many bodily behaviors that were shown to be connected to different social phenomena, as described in Section 2.1. We selected all ECSI behaviors involving the limbs and torso and excluded behavior classes based on facial behavior, gaze, and head pose as these are not the focus of this work and highly accurate methods to analyze such behaviors To achieve high-quality annotations while keeping costs manageable, we designed the following annotation procedure. First, we trained three annotators on the task by providing examples and discussing edge cases jointly. In this way, we made sure that the annotators arrived at a common understanding of the body language classes. Each of the 78 participants of the MPIIGroupInteraction dataset was fully annotated by one of the annotators. Subsequently, each of the resulting annotations was checked by another annotator to further improve quality. This procedure of annotation followed by checking proved to be much more economical than collecting several separate annotations of the same video. We used a separate experiment to quantify annotation quality (see Section 3.2).

Analysis of Annotations

3.2.1 Descriptive Statistics. In total, 2.87 million frames of body language were annotated across all classes for the full 26 hours of video. Each annotation instance is defined by a specific behavior label and a start time and an end time between which the behavior appears continuously on all frames. Table 2 shows that the annotation across the 15 behavior classes has highly uneven number of annotated frames and instances. The most frequently annotated class, Legs Crossed was annotated for 1397k frames, while Stretching was only annotated for 4k frames. As a complementary view on the quantity of annotations, Legs Crossed has the highest number of annotated frames, but only over 77 annotation instances, meaning that participants remained for a crossed-leg position for extended periods of time. On the other hand, Gesture is annotated on less frames (373k), but consists of many more distinct instances (2607).

Another important aspect of BBSI is its multi-label characteristic, that is, several body language classes can occur simultaneously. Figure 2 shows the co-occurrence patterns of body language classes. Strong co-occurrences can be observed between the lower body classes (Legs Crossed, Leg Movement) and upper-body behaviors. Co-occurrences between upper-body behaviors do exist, but are more sparse. As a result, BBSI creates a challenging multi-label classification problem.

Annotation

Quality. To obtain a numerical estimate of annotation quality, we performed a dedicated experiment based on the collected annotations. We sampled 800 4-second clips from the full dataset that were classified into body language classes separately by all three annotators. These samples were drawn randomly from the whole dataset with the following constraints: First, we considered a 4-second window to be a sample of a body language class if either the class is annotated for at least 2 seconds of this window, or if the 4-second window completely encompasses the corresponding annotation instance. Second, we drew 50 samples of each behavior class. Estimated with the rate of class co-occurrences, the precise number of instances for each class in the 800 samples may be larger than 50. For comparability, we used the same metric as [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF] which computes the agreement of two annotators by dividing twice the number of annotated behaviors for which they agree by the total number of behaviors annotated by both.

Table 2 shows the resulting agreements for each class separately. Very high agreements above 0.8 are reached for frequent classes such as Legs Crossed, Gesture, or Fold Arms. All other classes are in the range of 0.5 to 0.8 with the only exception of Lean Towards which was proven very challenging to annotate with only 0.13 agreement. Liu et al. [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF] do not provide class-specific annotator agreement but only a global measure in which frequent classes contribute more than less frequent classes, i.e. micro average. To get an estimate how our annotator agreement relates to the agreement of 0.81 reported in [START_REF] Liu | iMiGUE: An identity-free video dataset for micro-gesture understanding and emotion analysis[END_REF], we weight our class-specific agreements by the frame-wise label distribution on BBSI, reaching an agreement of 0.78. Note however, that these numbers are not directly comparable due to different behavior classes and annotation protocols.

METHOD

For detecting the behaviors in the long input videos, we propose a baseline method based on the Pyramid Dilated Attention Network [START_REF] Dai | PDAN: Pyramid Dilated Attention Network for Action Detection[END_REF] for action detection. The model is fed with features extracted by four types of action recognition architectures.

Feature Extraction Networks

We examine the following four established algorithms that are designed for general action recognition tasks.

Temporal Segment

Networks. An obvious problem of the two-stream CNNs is their inability to model long-range temporal structure due to their access to only a limited stack of frames. Temporal Segment Networks (TSN) [START_REF] Wang | Temporal Segment Networks: Towards Good Practices for Deep Action Recognition[END_REF] operate on a sequence of short video clips sparsely sampled from the entire video. Each clip in this sequence will produce its own preliminary prediction of the action classes. Prediction over the full video is then derived from a consensus among the partial clip predictions. In the learning process, the loss values of video-level predictions, other than those of clip-level predictions which were used in two-stream CNNs, are optimized by iteratively updating the model parameters.

Temporal Shift Module.

Traditional 3D convolution uses a 3D convolution kernel to perform convolution operations between adjacent multiple frames at the same time, which can extract the spatio-temporal feature information in the video at the cost of an increase in calculation. Temporal Shift Module (TSM) [START_REF] Lin | TSM: Temporal Shift Module for Efficient Video Understanding[END_REF] uses a simple data preprocessing method to convert the invisible temporal information in a single frame into extractable spatial feature information. Several consecutive frames are stacked to form the original tensor and the channels are moved forward and backward in the temporal dimension to perform a simple feature fusion between the consecutive frames. The fusion makes an independent single frame contain certain temporal information, and simple 2D convolution can be used to achieve spatiotemporal feature extraction.

Swin Transformers.

Adapting the network architectures in natural language processing to the domain of computer vision suffers from large variations in the scale of visual entities and the high resolution of pixels in images compared to words in text. Building upon the Transformer designed for sequence modeling and translation tasks, the Swin Transformer (Swin) [START_REF] Liu | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows[END_REF] is a hierarchical Transformer whose representation is computed with Shifted windows. The shifted windowing scheme brings greater efficiency by limiting self-attention computation to non-overlapping local windows while also allowing for cross-window connection.

Training Feature Extractors

As human body language ground truth contains temporally overlapping labeled segments as well as unlabeled sections, we investigate two training settings for the feature extraction networks: Single-Label training and Multi-Label training. In Multi-Label training, a label is assigned to a clip if it overlaps an annotated segment in at least half of its duration. On BBSI, 25% of the samples have no label assigned, 49% samples have one label and the remaining 26% on an overlap have between two and five labels. For the Single-Label setting, we selected the samples from the Multi-Label setting with at least one label. Each sample was assigned precisely one label and those originally with multiple labels are copied multiple times, each time with a single and unique label. The loss function we used was the cross entropy loss followed by the softmax activation function. For each sample, each feature extractor returns a confidence score for each behavior class. Prior to the output layer, each feature extractor produces a 2048-dimensional feature vector that is used as input to the behavior detection method.

Behavior Detection

Detection of behaviors from a long video is done by feeding the extracted feature vectors into an action detection architecture. The Pyramid Dilated Attention Network (PDAN) [START_REF] Dai | PDAN: Pyramid Dilated Attention Network for Action Detection[END_REF] uses a selfattention mechanism to capture temporal relations. The layers that make up the network are called Dilated Attention Layers (DAL). A DAL takes each segment as a center segment and concatenates its feature representation with the feature representation of segments being 𝐷-far in both directions from the center segment, where 𝐷 is the dilation rate. At this point, it applies self-attention on the extracted segment-feature representations. PDAN is based on a pyramid of DALs with same kernel sizes and dilation rates that exponentially increase their temporal receptive field. The output of PDAN consists of a list of predicted behaviors with their beginnings and endings tied to the segmentation cuts, and their confidences.

Implementation Details

The feature extraction methods operate on fixed inputs of length 16 frames and size 224×224 pixels. Consequently, we resize the videos appropriately and cut the long dataset videos into 16-frame video clips. These clips are assigned with the corresponding behavior class label and treated as independent samples for training and evaluation. This splitting of videos does not disconnect the flow of the actions as the annotated behaviors are mostly non-transitional, that is, the actions described by these behaviors do not change people's body poses from one to another. For instance, a 64-framelong behavior Hand-mouth can be split into four 16-frame-long clips in which a person keeps touching their mouth. Advantages are that the number of samples increases significantly and that the fixed-length clips can be input to all methods with an equal FPS.

All action recognition models are pre-trained on ImageNet and Kinetics-400 and the action detection model is used without any pretraining. Fine-tuning on BBSI is performed on both levels, action recognition and action detection. For comparability, all models were trained for 15 epochs. Learning rates are set to: I3D 10 -2 , TSN 10 -3 , TSM 7.5 • 10 -4 , Swin 10 -3 with AdamW optimizer, and PDAN 10 -1 . Our implementation uses the open-source toolbox MMaction2 [START_REF]OpenMMLab's Next Generation Video Understanding Toolbox and Benchmark[END_REF] built on top of PyCharm.

EVALUATION

We provide evaluation results of our baseline method at two levels: quality of the extracted features and quality of the final detection. As feature extractors are trained as classifiers, they are evaluated with standard classification metrics, and the final detector is evaluated on standard detection metrics. In all experiments, we use the training/validation split of MPIIGroupInteraction reported in [START_REF] Müller | Multi-Mediate: Multi-modal Group Behaviour Analysis for Artificial Mediation[END_REF]: training recordings 07, 10-25; validation recordings 08, 09, 26-28.

Classification Evaluation

Table 3 shows classification performance of the feature extraction networks, both for the Multi-Label as well as for the Single-Label training setting. We also include the random classifier as a baseline.

Results are reported in terms of Mean Average Precision (MAP)

Table 3: Five action recognition architectures, four proposed methods and one random, in both labeling settings. Each value is MAP score computed using micro/macro averaging.

Method

Single-Label Multi-Label random 0.258 / 0.067 0.377 / 0.106 I3D [START_REF] Carreira | Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset[END_REF] 0.445 / 0.212 0.624 / 0.284 TSN [START_REF] Wang | Temporal Segment Networks: Towards Good Practices for Deep Action Recognition[END_REF] 0.520 / 0.232 0.661 / 0.308 TSM [START_REF] Lin | TSM: Temporal Shift Module for Efficient Video Understanding[END_REF] Evaluation of all methods can be visualized by aggregating all confidence vectors into a confidence matrix. Rows of this 15×15 square matrix are ground truth classes and columns are prediction confidences. The matrix is constructed by adding all confidence vectors into the corresponding ground truth rows and then dividing each row by the number of its summands. In the ideal case, this matrix would coincide with the co-occurrence matrix presented in Figure 2. See Figure 3 for the confidence matrices of behavior recognition by the Swin Transformer for both Single-Label and Multi-Label training. Compared to Single-Label training, the Multi-Label network is able to more accurately model class co-occurrences, especially with Legs Crossed. We report further confidence matrices in the supplementary material.

We performed additional ablation experiments with the TSM model. First, as the behaviors Legs Crossed and Fold Arms make forms of static positional body pose rather than dynamic motion actions, they can be recognized on a frame level and with an eventual aid of a generally non-temporal skeleton estimation technique. We evaluated the training and evaluation scenario with only 13 classes, excluding these two static classes. And second, as the dataset has considerably imbalanced class frequencies (ranging from 4k annotated frames to more than a million, see Table 2), overrepresented behaviors have a too high impact on training compared to underrepresented ones. Therefore, we evaluated the influence of class balancing by randomly selecting 20k samples from each class to counteract weight of overrepresented classes while keeping all samples of the underrepresented classes. Table 4 shows the effects of static class exclusion and class balancing. We observe systematic advantage of excluding static classes in the Single-Label setting and of including static classes in the Multi-Label setting, and of no class balancing overall.

Detection Evaluation

In addition to MAP, the evaluation metrics are calculated from the detection confidence vector on the frame level: True Positives (TP) as the sum of confidences in true classes, False Positives (FP) as the sum of confidences in false classes, and False Negatives (FN) as the sum of 1 minus confidences in true classes. From TP, TN and FN on the frame level, we calculate the F1 score globally using both micro and macro averaging. As expected from behavior recognition, Swin achieves the best results with F1 0.728/0.544 and MAP 0.742/0.415 on the Single-Label setting, and F1 0.726/0.511 and MAP 0.691/0.367 on Multi-Label, using micro/macro averaging respectively. Figure 4 illustrates examples of true positive, false positive and false negative predictions. As in behavior detection without class balancing, there is a high inter-class performance variance. The most frequent class Legs Crossed reaches the highest performance among all classes.

DISCUSSION 6.1 Annotations

We presented the first publicly available annotations of 15 body language classes on a multi-view group discussion dataset. Annotating human bodily behavior is challenging due to the subtle and often subjective nature of body language. To evaluate the agreement of our annotators, we conducted a dedicated experiment. Our classbased analysis of annotator agreement revealed clear differences between the agreement for different behavior classes, which should be taken into account by potential users of the dataset. Restricting to a subset of the annotated classes to those with a proper relevance to the particular application and a high inter-annotator agreement can be a good practice for any body language analysis system. On the other hand, if body language annotations are used to train a feature representation that is used in a downstream task, even low agreement classes can still be useful. On the other hand, if body language annotations are used to train a feature representation that is used in a downstream task, even classes with low annotator agreement can still be useful. We include classes with low agreement scores for full transparency and leave it to the users to decide which classes to use depending on their preferences. Supplementary material provides class-specific evaluation results to facilitate comparison with researchers who choose a subset of classes.

Achieved Performance

In line with the recent trend on computer vision tasks [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF][START_REF] Khan | Transformers in Vision: A Survey[END_REF], the effectiveness of transformers is also reflected on BBSI. Even the Tiny version of Swin Transformers has outperformed all other Table 5: Evaluation of PDAN [START_REF] Dai | PDAN: Pyramid Dilated Attention Network for Action Detection[END_REF] with four types of features trained in both labeling settings in terms of F1 and MAP using micro/macro averaging.

Features

Single-Label Multi-Label F1 MAP F1 MAP random 0.348 / 0.114 0.312 / 0.092 0.348 / 0.115 0.312 / 0.092 I3D [START_REF] Carreira | Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset[END_REF] 0.542 / 0.340 0.502 / 0.276 0.581 / 0.339 0.557 / 0.271 TSN [START_REF] Wang | Temporal Segment Networks: Towards Good Practices for Deep Action Recognition[END_REF] 0.550 / 0.309 0.624 / 0.291 0.531 / 0.343 0.658 / 0.311 TSM [START_REF] Lin | TSM: Temporal Shift Module for Efficient Video Understanding[END_REF] 0.660 / 0.419 0.600 / 0.311 0.627 / 0.347 0.626 / 0.293 Swin [START_REF] Liu | Swin Transformer: Hierarchical Vision Transformer using Shifted Windows[END_REF] 0.729 / 0.545 0.742 / 0.415 0.726 / 0.511 0.691 / 0.367 CNN-based architectures in every setup where it was applied. This is usually followed by TSM and TSN, although I3D has a higher potential due to its 10-times larger number of parameters.

Class balancing degrades the performance in any setup. Although it was introduced to counteract the dominance of static classes, the MAP drop is the highest in those setups where the static classes are included. Our assumption is that equally balancing the dataset is not adequate in this case as the distribution of instance numbers per classes are exponential. Despite giving equal weight to the classes of very few instances increases their performance on the training set, they are not possible to achieve good recognition on unseen data. Not only it does not improve testing inference, the metrics of other classes fall as well.

Applying the Single-Label setting on a detection task inherently produces incorrect predictions. As in most of the cases there is at least one static class involved in concurrent actions, excluding static classes results in a classification problem of a significantly reduced rate of multiple labels. Thus, the difference between the Single-Label and Multi-Label experiments when the static classes are excluded is almost negligible compared to the case when all the classes are included, which is in the range between 0.005-0.088 MAP if there is no class balancing applied.

Applications

The primary intended application for BBSI annotations is to train and evaluate algorithms that predict body language classes. However, our annotations can also be useful in a pre-training step or for auxiliary training of approaches that address high-level behavior interpretation tasks such as leadership detection [START_REF] Beyan | Moving as a leader: Detecting emergent leadership in small groups using body pose[END_REF][START_REF] Müller | Emergent leadership detection across datasets[END_REF] or personality prediction [START_REF] Agrawal | Multimodal Personality Recognition using Cross-attention Transformer and Behaviour Encoding[END_REF][START_REF] Palmero | Context-aware personality inference in dyadic scenarios: Introducing the udiva dataset[END_REF] for which only limited amount of training data is available. Furthermore, it can be of interest for behavioral scientists to use our annotations for research on the expression of nonverbal behavior in group interactions and how it relates to aspects like leadership, rapport, or interpersonal synchrony.

As BBSI is based on a rating scheme developed in the context of psychiatric interactions [START_REF] Troisi | Ethological research in clinical psychiatry: the study of nonverbal behavior during interviews[END_REF], we expect our body language predictions to be highly useful in clinical tasks, e.g. for depression detection [START_REF] Shi Yin | A multi-modal hierarchical recurrent neural network for depression detection[END_REF] or to estimate the quality of the therapist-patient relationship [START_REF] Adam | The role of the therapeutic alliance in psychotherapy[END_REF]. Using a set of psychologically motivated behaviors as an intermediate representation instead of generic pose-based features or deep learning representations will allow for better interpretability and build trust with clinicians and patients alike. Our presented prediction methods can also be integrated into existing conversation analysis tools [START_REF] Penzkofer | ConAn: A Usable Tool for Multimodal Conversation Analysis[END_REF], which at present do not have the ability to detect fine-grained body language.

Limitations and Future Work

Our novel annotations and state-of-the-art evaluations represent an important step towards automatic analysis of body language in social interaction. At the same time, several challenges remain that need to be addressed in future work. While the BBSI set of behavior classes is motivated by previous work linking those classes to social attributes like leadership, rapport, or emotions, this link needs to be solidified by investigating the predictive power of bodily behaviors for such downstream tasks. Furthermore, as the MPI-IGroupInteraction dataset consists of participants recruited at a German university, future work should collect comparable datasets with more diverse cultural backgrounds. A key challenge on BBSI is the large class imbalance that makes it difficult to train accurate models for classes that occur seldomly in natural behavior. Future work could investigate generation of synthesized training examples or advanced data augmentation techniques. The detection and classification approaches presented in this paper learn a single model that is applied to all participants. While this is a meaningful first step to approach the task, the expression of body language is highly individual. Future work should investigate personalization and test-time adaptation [START_REF] Amin | Test-time adaptation for 3d human pose estimation[END_REF][START_REF] Charles | Personalizing human video pose estimation[END_REF] to model personal idiosyncrasies adequately. Another possible improvement is to use multi-channel inputs, e.g. by exploiting all three views on a person, or adding pose information [START_REF] Cao | OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields[END_REF].

CONCLUSION

In this work, we presented BBSI, the first publicly available set of annotations of subtle bodily behaviors in group interactions. The novel annotations consist of 15 body language classes that were densely annotated for 26 hours of human behavior recorded from 78 participants on the publicly available MPIIGroupInteraction dataset. We provided results of descriptive analyses of the annotations as well as a dedicated experiment on annotation quality, as they were done manually by our human annotators. Furthermore, we presented the results of state-of-the-art action recognition approaches evaluated on the MPIIGroupInteraction dataset with the BBSI annotations. As such, our work is a key contribution to advance in-depth analyses of subtle body language cues in human interactions.

  A d ju s t in g C lo t h in g F o ld A r m s F u m b le G e s t u r e G r o o m H a n dfa c e H a n dm o u t h L e a n T o w a r d s L e g M o v e m e n t L e g s C r o s s e d S c r a t c h S e t t le S h r u g S m e a r in g H a n d s S t r e t c h in g

Figure 2 :

 2 Figure 2: Co-occurrences of body language classes. Each row shows the percentage with which other classes are annotated at the same time as the class designated on the y-axis.

4. 1 . 1

 11 Two-Stream Inflated 3D CNN. Extent of a pre-training boost depends on the ability of a model architecture to adapt to a given pretraining dataset. As the 3D image classification backbone, the Two-Stream Inflated 3D CNN (I3D) [11] uses the ImageNet-pretrained Inception V1 with batch normalization. Filters and pooling kernels of very deep 2D image classification CNNs are inflated into 3D to learn spatio-temporal feature extractors from video.

  0.508 / 0.228 0.721 / 0.313 Swin [37] 0.601 / 0.305 0.745 / 0.374 A d ju st in g C lo th in g F o ld A rm s F u m b le G e st u re G ro o m H a n d -f a c e H a n d -m o u th L e a n T o w a rd s L e g M o v e m e n t L e g s C ro ss e d S c ra tc h S e tt le S h ru g S m e a ri n g H a n d s S tr e tc h in g Single-Label Swin [37] A d ju st in g C lo th in g F o ld A rm s F u m b le G e st u re G ro o m H a n d -f a c e H a n d -m o u th L e a n T o w a rd s L e g M o v e m e n t L e g s C ro ss e d S c ra tc h S e tt le S h ru g S m e a ri n g H a n d s S tr e tc h in g

Figure 3 :

 3 Figure 3: Confidence matrices of behavior recognition by the Swin Transformer trained in both labeling settings.

Figure 4 :

 4 Figure 4: Illustrative examples of true positive, false positive and false negative predictions.

Table 2 :

 2 Behavior classes in the dataset, including descriptions, number of annotated frames, annotation instances, and annotator agreement.

	Behavior	Description

already exist

[START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF][START_REF] Sinha | FLAME: Facial Landmark Heatmap Activated Multimodal Gaze Estimation[END_REF]

. We also excluded the two classes Crouch and Relax, as they were only very rarely annotated (Crouch: 411 frames, Relax: 2k frames), rendering estimation of classification performance meaningless. In addition to the bodily behaviors included in ECSI, we scanned the MPIIGroupInteraction dataset for additional behaviors that occur frequently and carry potential meaning in a social situation. As a result, we included the five additional classes: Adjusting Clothing, Leg Movement, Legs Crossed, Smearing Hands, Stretching.

According to a note dating from September 2021 on the official github page of iMiGUE (https://github.com/linuxsino/iMiGUE), the file containing the links to the videos used in the dataset has been removed for privacy protection.

ACKNOWLEDGMENTS

This work was supported by the French National Research Agency under the UCA JEDI Investments into the Future, project number ANR-15-IDEX-01, and by the German Ministry for Education and Research, grant number 01IS20075.