Anna Kukleva
email: akukleva@mpi-inf.mpg.de

Makarand Tapaswi

Ivan Laptev
email: ivan.laptev@inria.fr

Mark Vijay

Paulie Juno

Learning Interactions and Relationships between Movie Characters

Let's talk about how we're

Introduction

A salient aspect of being human is our ability to interact with other people and develop various relationships over the period of our lives. While some relationships drive the typical interactions experienced by a pair of people in a top-down manner (e.g. parents customarily love and nurture their children); almost all social (non-family) relationships are driven through bottom-up interactions (e.g. strangers become friends over a good chat or a shared drink) [START_REF] Hinde | Interactions, Relationships and Social Structure[END_REF]. For an intelligent agent to truly be a part of our lives, we will need it to assimilate this complex interplay and learn to behave appropriately in different social situations.

We hypothesize that a first step in this direction involves learning how people interact and what their relationships might be. However, training machines with live, real world, experience-based data is an extremely complicated proposition. Instead, we rely on movies that provide snapshots into key moments of our lives, portraying human behavior at its best and worst in various social situations [START_REF] Vicol | Moviegraphs: Towards understanding human-centric situations from videos[END_REF].

Interactions and relationships have been addressed separately in literature. Interactions are often modeled as simple actions [START_REF] Gu | Ava: A video dataset of spatio-temporally localized atomic visual actions[END_REF][START_REF] Patron | High five: Recognising human interactions in tv shows[END_REF], and relationships are primarily studied in still images [START_REF] Li | Dual-glance model for deciphering social relationships[END_REF][START_REF] Sun | A domain based approach to social relation recognition[END_REF] and recently in videos [START_REF] Liu | Social relation recognition from videos via multi-scale spatial-temporal reasoning[END_REF]. However, we believe that a complete understanding of social situations can only be achieved by modeling them jointly. For example, consider the evolution of interactions and relationships between a pair of individuals in a romantic movie. We see that the characters first meet and talk with each other and gradually fall in love, changing their relationship from strangers to friends to lovers. This often leads to them getting married, followed subsequently by arguments or infidelity (a strong bias in movies) and a falling out, which is then reconciled by one of their friends.

The goal of our work is to attempt an understanding of these rich moments of peoples' lives. Given short clips from a movie, we wish to predict the interactions and relationships, and localize the characters that experience them throughout the movie. Note that our goals necessitate the combination of visual as well as language cues; some interactions are best expressed visually (e.g. runs with), while others are driven through dialog (e.g. confesses) -see Fig. 1. As our objectives are quite challenging, we make one simplifying assumption -we use trimmed (temporally localized) clips in which the interactions are known to occur. We are interested in studying two important questions: (i) can learning to jointly predict relationships and interactions help improve the performance of both? and (ii) can we use interaction and relationship labels at the clip or movie level and learn to identify/localize the pair of characters involved? We refer to this as weak track prediction. A solution for the first question is attempted using a multi-task formulation operating on several clips spanning the common pair of characters, while the second uses a combination of max-margin losses with multiple instance learning (see Sec. 3).

Contributions. We conduct our study on 51 movies from the recently released MovieGraphs [START_REF] Vicol | Moviegraphs: Towards understanding human-centric situations from videos[END_REF] dataset (see Sec. 4.2). The dataset annotations are based on free-text labels and have long tails for over 300 interaction classes and about 100 relationships. To the best of our knowledge, ours is the first work that attempts to predict interactions and longterm relationships between characters in movies based on visual and language cues. We also show that we can learn to localize characters in the video clips while predicting interactions and relationships using weak clip/movie level labels without a significant reduction in performance.

Related Work

We present related work in understanding actions/interactions in videos, studying social relationships, and analyzing movies or TV shows for other related tasks.

Actions and interactions in videos. Understanding actions performed by people can be approached in many different ways. Among them, action classification involves predicting the dominant activity in a short trimmed video clip [START_REF] Kuehne | HMDB: A Large Video Human Motion Recognition[END_REF][START_REF] Soomro | UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild[END_REF], while action localization involves predicting the activity as well as temporal extent [START_REF] Girdhar | Video action transformer network[END_REF][START_REF] Shou | Cdc: Convolutional-deconvolutional networks for precise temporal action localization in untrimmed videos[END_REF][START_REF] Xu | R-c3d: Region convolutional 3d network for temporal activity detection[END_REF]]. An emerging area involves discovering actions in an unsupervised manner by clustering temporal segments across all videos corresponding to the same action class [START_REF] Alayrac | Unsupervised learning from narrated instruction videos[END_REF][START_REF] Kukleva | Unsupervised learning of action classes with continuous temporal embedding[END_REF][START_REF] Sener | Unsupervised learning and segmentation of complex activities from video[END_REF].

Recently, there has been an interest in creating largescale datasets (millions of clips, several hundred classes) for learning actions [START_REF] Abu-El-Haija | Youtube-8m: A large-scale video classification benchmark[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Diba | Holistic large scale video understanding[END_REF][START_REF] Goyal | The "something something" video database for learning and evaluating visual common sense[END_REF][START_REF] Monfort | Moments in time dataset: one million videos for event understanding[END_REF] but none of them reflect person-to-person (p2p) multimodal interactions where several complex actions may occur simultaneously. The AVA challenge and dataset [START_REF] Gu | Ava: A video dataset of spatio-temporally localized atomic visual actions[END_REF] is composed of 15 minute video clips from old movies with atomic actions such as pose, person-object interactions, and person-person interactions (e.g. talk to, hand wave). However, all labels are based on a short (3 second) temporal window, p2p actions are not annotated between multiple people, and relationship labels are not available. Perhaps closest to our work on studying interactions, Alonso et al. [START_REF] Patron | High five: Recognising human interactions in tv shows[END_REF] predict interactions between two people using person-centered descriptors with tracks. However, the TV-Human Interactions dataset [START_REF] Patron | High five: Recognising human interactions in tv shows[END_REF] is limited to 4 visual classes in contrast to 101 multimodal categories in our work. As we are interested in studying intricate multimodal p2p interactions and long-range relationships, we demonstrate our methods on the MovieGraphs dataset [START_REF] Vicol | Moviegraphs: Towards understanding human-centric situations from videos[END_REF].

Recognizing actions in videos requires aggregation of spatio-temporal information. Early approaches include hand-crafted features such as interest points [START_REF] Laptev | Space-time Interest Points[END_REF] and Improved Dense Trajectories [START_REF] Wang | Action Recognition with Improved Trajectories[END_REF]. With end-to-end deep learning, spatio-temporal 3D Convolutional Neural Networks (e.g. I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]) are used to learn video representations resulting in state-of-the-art results on video understanding tasks. For modeling long-videos, learning aggregation functions [START_REF] Girdhar | ActionVLAD: Learning spatio-temporal aggregation for action classification[END_REF][START_REF] Miech | Learnable pooling with context gating for video classification[END_REF], subsampling frames [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF], or accumulating information from a feature bank [START_REF] Chao-Yuan | Long-term feature banks for detailed video understanding[END_REF] are popular options. Relationships in still images. Most studies on predicting social relationships are based on images [START_REF] Andrew | Understanding images of groups of people[END_REF][START_REF] Goel | An end-to-end network for generating social relationship graphs[END_REF][START_REF] Li | Dual-glance model for deciphering social relationships[END_REF][START_REF] Socher | Reasoning with neural tensor networks for knowledge base completion[END_REF][START_REF] Sun | A domain based approach to social relation recognition[END_REF]. For example, the People in Photo Albums (PIPA) [START_REF] Zhang | Beyond Frontal Faces: Improving Person Recognition Using Multiple Cues[END_REF] and the People in Social Context (PISC) datasets [START_REF] Li | Dual-glance model for deciphering social relationships[END_REF] are popular among social relationship recognition. The latter contains 5 relationship types (3 personal, 2 professional), and [START_REF] Li | Dual-glance model for deciphering social relationships[END_REF] employs an attention-based model that looks at the entire scene as well as person detections to predict relationships. Alternately, a domain based approach is presented by Sun et al. [START_REF] Sun | A domain based approach to social relation recognition[END_REF] that extends the PIPA dataset and groups 16 social relationships into 5 categories based on Burgental's theory [START_REF] Burgental | Acquisition of the Algorithms of Social Life: A Domain-Based Approach[END_REF]. Semantic attributes are used to build interpretable models for predicting relationships [START_REF] Sun | A domain based approach to social relation recognition[END_REF].

We believe that modeling relationships requires looking at long-term temporal interactions between pairs of people, something that still image works do not allow. Thus, our work is fundamentally different from above literature. Social understanding in videos. Understanding people in videos goes beyond studying actions. Related topics include clustering faces in videos [START_REF] Jin | End-to-end Face Detection and Cast Grouping in Movies using ErdsRnyi Clustering[END_REF][START_REF] Tapaswi | Video Face Clustering with Unknown Number of Clusters[END_REF], naming tracks based on multimodal information [START_REF] Nagrani | From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Script[END_REF][START_REF] Tapaswi | Knock! Knock! Who is it?" Probabilistic Person Identification in TV series[END_REF], studying where people look while interacting [START_REF] Fan | Understanding human gaze communication by spatio-temporal graph reasoning[END_REF][START_REF] Manuel J Marin-Jimenez | Laeo-net: revisiting people looking at each other in videos[END_REF], predicting character emotions [START_REF] Dhall | Emotion recognition in the wild challenge 2013[END_REF][START_REF] Vicol | Moviegraphs: Towards understanding human-centric situations from videos[END_REF], modeling spatial relations between objects and characters [START_REF] Laxton | Leveraging temporal, contextual and ordering constraints for recognizing complex activities in video[END_REF][START_REF] Sun | Actor-centric relation network[END_REF][START_REF] Zhang | A Structured Model For Action Detection[END_REF], recognizing actions performed in groups [START_REF] Bagautdinov | Social Scene Understanding: End-To-End Multi-Person Action Localization and Collective Activity Recognition[END_REF][START_REF] Deng | Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition[END_REF], predicting effects for characters [START_REF] Zhou | Now You Shake Me: Towards Automatic 4D Cinema[END_REF], producing captions for what people are doing [START_REF] Rohrbach | Movie description[END_REF][START_REF] Damen | Scaling Egocentric Vision: The EPIC-KITCHENS Dataset[END_REF], answering questions about events, activities, and character motivations [START_REF] Lei | TVQA: Localized, Compositional Video Question Answering[END_REF][START_REF] Tapaswi | Movieqa: Understanding stories in movies through question-answering[END_REF][START_REF] Kim | PororoQA: A Cartoon Video Series Dataset for Story Understanding[END_REF], reasoning about social scenes and events [START_REF] Zadeh | Social-IQ: A Question Answering Benchmark for Artificial Social Intelligence[END_REF][START_REF] Zellers | From Recognition to Cognition: Visual Commonsense Reasoning[END_REF], understanding social relationships [START_REF] Liu | Social relation recognition from videos via multi-scale spatial-temporal reasoning[END_REF][START_REF] Vicol | Moviegraphs: Towards understanding human-centric situations from videos[END_REF], and meta-data prediction using multiple modalities [START_REF] Cascante-Bonilla | Moviescope: Large-scale analysis of movies using multiple modalities[END_REF].

Perhaps most related to our work on predicting relationships are [START_REF] Liu | Social relation recognition from videos via multi-scale spatial-temporal reasoning[END_REF][START_REF] Lv | Multi-stream Fusion Model for Social Relation Recognition from Videos[END_REF]. Lv et al. [START_REF] Lv | Multi-stream Fusion Model for Social Relation Recognition from Videos[END_REF] present the first dataset for modeling relationships in video clips, and propose a multistream model to classify 16 relationships. More recently, Liu et al. [START_REF] Liu | Social relation recognition from videos via multi-scale spatial-temporal reasoning[END_REF] propose a graph network to capture long-term and short-term temporal cues in the video. Different from above works, we address predicting relationships between pairs of characters in an entire movie. We propose a joint model for interactions and relationships as they may influence each other, and also localize the characters in the video.

Model

In this section, we present our approach towards predicting the interactions and relationships between pairs of characters (Sec. 3.1), and localizing characters in the video as tracks (Sec. 3.2).

Notation. We define A as the set of all interaction labels, both visual and spoken (e.g. runs with, consoles); and R as the set of all relationship labels between people (e.g. parentchild, friends). We process complete movies, where each movie M consists of three sets of information:

1. Characters C M = {c 1 , . . . , c P }, each c j representing a cluster of all face/person tracks for that character.

Trimmed video clips annotated with interactions

T M = {(v 1 , a * 1 , c 1j , c 1k ), . . . , (v N , a * N , c N j , c N k )}
, where v i corresponds to a multimodal video clip, a * i ∈ A is a directed interaction label, and c ij is used to denote the tracks for character c j in the clip v i .

Directed relationships between all pairs of characters

R M = {r i jk = relationship(v i , c j , c k )} for all clips i ∈ [1, N ].
For simplicity of notation, we assign a relationship label r i jk to each clip. However, note that relationships typically span more than a clip, and often the entire movie (e.g. parent-child).

For each clip v i , our goal is to predict the primary interaction a i , the characters c ij and c ik that perform this interaction, and their relationship r i jk . In practice, we process several clips belonging to the same pair of characters as predicting relationships with a single short clip can be quite challenging, and using multiple clips helps improve performance.

We denote the correct pair of characters in a tuple

(v i , a * i , c ij , c ik ) from T as p * i = (c ij , c ik )
, and the set of all character pairs as P M = {(c j , c k )∀j, k, j = k}.

Note that the interaction tuples in T may be temporally overlapping with each other. For example, Jack may look at Jill while she talks to him. We deal with such interaction labels from overlapping clips in our learning approach by masking them out in the loss function.

Interactions and Relationships in a Clip

Fig. 2 shows example correlations between a few selected interactions and all 15 relationships in our dataset. We observe that interactions such as obeys go together with workermanager relationships, while an enemy may shoot, or pull (a weapon), or commit a crime. Motivated by these correlations, we wish to learn interactions and relationships jointly.

When the pair of characters that interact is known, we predict their interactions and relationships using a multi-modal clip representation Φ(v i , p * i ) ∈ R D . As depicted in Fig. 3, Φ combines features from multiple sources such as visual and dialog cues from the video, and character representations by modeling their spatio-temporal extents (via tracking). Interactions. We use a two-layer MLP with a classification layer W I2 ∈ R |A|×D , b I2 ∈ R |A| to predict interactions between characters. The score for an interaction a in clip v is computed as

s I (v, a) = σ I (w I2 a • ReLU(W I1 Φ(v, p * ) + b I1 ) + b I2 a ), (1)
where σ(•) represents the sigmoid operator. We learn the clip representation parameters along with the MLP by minimizing the max-margin loss function for each sample

L I (v) = ā∈A\O I (v) ā =a * [m I -s I (v, a * ) + s I (v, ā)] + , (2)
where [•] + is a ReLU operator, m I is the margin, and O I (v) is the set of interaction labels from clips temporally over- lapping with v. The loss encourages our model to associate the correct interaction a * with the clip v, while pushing other non-overlapping interaction labels ā away. During inference, we predict the interaction for a clip v as â = arg max a s I (v, a).

Relationships. While interactions are often short in duration (few seconds to a minute), relationships in a movie may last for several minutes to the entire movie. To obtain robust predictions for relationships, we train a model that observes several trimmed video clips that portray the same pair of characters. Let us denote V jk ⊂ {v 1 , . . . , v N } as one such subset of clips that focus on characters c j and c k . In the following, we drop the subscripts jk for brevity. Similar to predicting interactions, we represent individual clips of V using Φ, apply a pooling function g(•) (e.g. avg, max) to combine the individual clip representations as Φ(V, p * ) = g(Φ(v, p * )) and adopt a linear classifier

W R ∈ R |R|×D , b R ∈ R |R| to predict relationships. The scoring function s R (V, r) = σ r w R r Φ(V, p * ) + b R r (3)
computes the likelihood of character pair p * from the clips V having relationship r. We train model parameters using a similar max-margin loss function

L R (V ) = r∈R r =r * [m R -s R (V, r * ) + s R (V, r)] + , (4)
that attempts to score the correct relationship r * higher than others r. Unlike L I , we assume that a single label applies to all clips in V . If a pair of characters change relationships (e.g. from strangers to friends), we select the set of clips V during which a single relationship is present. At test time, we predict the relationship as r = arg max r s R (r, V ).

Joint prediction of interactions and relationships is performed using a multi-task formulation. We consider multiple clips V and train our model to predict the relationship as well as all interactions of the individual clips jointly. We introduce a dependency between the two tasks by concatenating the clip representations for interactions Φ I (v, p * ) and relationships Φ R (V, p * ). Fig. 3 visualizes the architecture used for this task. We predict interactions as follows:

s I (v, V, a) = σ(w I2 a •ReLU(W I1 [Φ I (v, p * ); Φ R (V, p * )])).
(5) Linear layers include biases, but are omitted for brevity. The loss function L I (v) now uses s I (v, V, a), but remains unchanged otherwise. The combined loss function is

L IR (V ) = L R (V ) + λ |V | v∈V L I (v) , (6)
where λ balances the two losses.

Who is interacting?

Up until now, we assumed that a clip v portrays two known characters that performed interaction a. However, movies (and the real world) are often more complex, and we observe that several characters may be interacting simultaneously. To obtain a better understanding of videos, we present an approach to predict the characters along with the interactions they perform, and their relationship.

While the interaction or relationship may be readily available as a clip-level label, localizing the pair of characters in the video can be a tedious task as it requires annotating tracks in the video. We present an approach that can work with such weak (clip-level) labels, and estimate the pair of characters that may be interacting. Predicting interactions and characters. As a first step, we look at jointly predicting interactions and the pair of charac-ters. Recall that p * i denotes the correct pair of characters in a clip tuple consisting of v i , and P M is the set of all character pairs in the movie. We update the scoring function (Eq. 1) to depend on the chosen pair of characters p ∈ P M as

s IC (v, a, p) = σ(w I2 a • ReLU(W I1 Φ(v, p))) , (7)
where Φ(v, p) now encodes the clip representation for any character pair p (we use zeros for unseen characters). We train our model to predict interactions and the character pair by minimizing the following loss

LIC (v) = ā∈A\O I (v) p∈P M (ā, p) =(a * ,p * ) [mIC -sIC (v, a * , p * ) + sIC (v, ā, p)] + . (8)
If we consider the scoring function s IC (v, a, p) as a matrix of dimensions |P M |×|A|, the negative samples are taken from everywhere except columns that have an overlapping interaction label O I (v), and the element where (ā = a * , p = p * ). At test time, we compute the character pair prediction accuracy given ground-truth (GT) interaction, interaction accuracy given GT character pair, and joint accuracy by picking the maximum score along both dimensions. Training with weak labels. When the GT character pair p * is not known during training, we modify the loss from Eq. 8 by first choosing the pair p * that scores highest for the current parameters and a * , that is known during training.

p * = arg max p s IC (v, a * , p) , (9)
L weak IC (v) = ā∈A\O I (v), ā =a * p∈P M [m IC -s IC (v, a * , p * ) + s IC (v, ā, p)] + . (10)
In contrast to the case when we know GT p * , we discard negatives from the entire column (a = a * ) to prevent minor changes in choosing p * from suppressing other character pairs. In practice, we treat s IC (v, a * , p) as a multinomial distribution and sample p * from it to prevent the model from getting stuck at only one pair. Inference is performed in a similar way as above.

Hard negatives. Training a model with max-margin loss can affect performance if the loss is satisfied (= 0) for most negative samples. As demonstrated in [START_REF] Faghri | VSE++: Improving Visual-Semantic Embeddings with Hard Negatives[END_REF], choosing hard negatives by using max instead of can help improve performance. We adopt a similar strategy for selecting hard negatives, and compute the loss over all character pairs with their best interaction, i.e. p∈P M max ā(•) in Eq. 8 and 10. Prediction of interactions, relationships, and characters. We present the loss function used to learn a model that jointly estimates which characters are performing what interactions and what are their relationships. Similar to Eq. 7, we first modify the relationship score to depend on p:

s RC (V, r, p) = σ(w R r g(Φ(V, p)) + b R r ) . (11)
This is used in a weak label loss function similar to Eq. 10.

p * = arg max

p sRC (V, r * , p) + sIC (v, a * , p) , (12)
L weak RC (V ) = r∈R, r =r * p∈P M [mRC -sRC (V, r * , p * ) + sRC (V, r, p)] + , (13)
L weak IRC (V ) = L weak RC (V ) + λ |V | v∈V L weak IC (v) . (14)
During inference, we combine the scoring functions s IC and s RC to produce a 3D tensor in |P M |×|A|×|R|. As before, we compute character pair accuracy given GT a * and r * , interaction accuracy given GT p * and r * , and relationship accuracy given GT p * and a * . We are also able to make joint predictions on all three by picking the element that maximizes the tensor over all three dimensions.

Experiments

We start by describing implementation details (Sec. 4.1), followed by a brief analysis of the dataset and the challenging nature of the task (Sec. 4.2). In Sec. 4.3 we examine interaction and relationship prediction performance as separate and joint tasks. Sec. 4.4 starts with learning interactions and estimating the pair of characters simultaneously. Finally, we also discuss predicting relationships jointly with interactions and localizing character pairs. We present both quantitative and qualitative evaluation throughout this section.

Implementation Details

Visual features. We extract visual features for all clips using a ResNeXt-101 model [START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF] pre-trained on the Kinetics-400 dataset. A batch of 16 consecutive frames is encoded, and feature maps are global average-pooled for the clip representation, and average pooled over a region of interest (ROIPool) to represent characters. Given a trimmed clip v i , we max pool above extracted features over the temporal span of the clip to pick the most important segments. Dialog features. To obtain a text representation, all dialogues are first parsed into sentences. A complete sentence may be as short as a single word (e.g. "Hi.") or consist of several subtitle lines. Multiple lines are also joined if they end with "...". Then, each sentence is processed using pretrained BERT-base model with a masked sentence from the next person if it exists. We supply a mask for every second sentence as done in the sentence pair classification task (for more details, c.f . [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF]). We max pool over all sentences uttered in a trimmed clip to obtain a final representation.

Note that every clip always has a visual representation. In the absence of dialog or tracks, we set the representations for missing modalities to 0. Clip representation Φ. We process the feature vector corresponding to each modality obtained after max pooling over the temporal extent with a two-layer MLP. Dropout (with p = 0.3), ReLU and tanh(•) non-linearities are used in the MLP. The final clip representation is a concatenation of all modalities (see Fig. 3

left).

Multi-label masking. As multiple interactions may occur at the same time or have overlapping temporal extents with other clips, we use masking to exclude negative contributions to the loss function by such labels. O I (v), the labels corresponding to the set of clips overlapping with v, are created by checking for an overlap (IoU) greater than 0.2.

Learning. We train our models with a batch size of 64, and use the Adam optimizer with a learning rate of 3e-5.

Dataset

We evaluate our approach on the MovieGraphs dataset [START_REF] Vicol | Moviegraphs: Towards understanding human-centric situations from videos[END_REF]. The dataset provides detailed graph-based annotations of social situations for 7600 scenes in 51 movies. Two main types of interactions are present-detailed interactions (e.g. laughs at) last for a few seconds and are often a part of an overarching summary interaction (e.g. entertains) that may span up to a minute. We ignore this distinction for this work and treat all interactions in a similar manner. These hierarchical annotations are a common source of multiple labels being associated with the same timespan in the video.

The total number of interactions is different from the number of p2p instances as some interactions involve multiple people. For example, in an interaction where a couple (c j and c k ) listens to their therapist (c l ), two p2p instances are created: c j → listens to → c l and c k → listens to → c l .

The dataset is partitioned into train (35 movies), validation (7 movies) and test (9 movies) splits. The train set consists of 15,516 interactions (and 20,426 p2p instances) and 2,676 pairs of people with annotated relationships. Validation and test sets have 3,992 and 5,380 p2p instances respectively, and about 600 relationship pairs each.

Missing labels. A relationship label is available for 64% of the interactions in which at least two people participate. For a pair of people associated with an interaction, both have track features for 76% of the dataset. In other cases one or no characters appear due to failure in tracking or being out of the scene. For evaluation, we only consider samples that have a relationship, or when a pair of characters appear.

Merging interaction and relationship labels. We reduce the number of interaction labels from 324 to 101, and relationships from 106 to 15 to mitigate severe problems of long tail with only 1-3 samples per class. However, the merging does not adversely affect the diversity of classes, e.g. reassures, wishes, informs, ignores are different interactions in our label set related to communication.

We adopt a hierarchical approach to merge interactions. Firstly, all classes are divided into 4 categories: (i) informative or guiding (e.g. explains, proposes, assists, guide) (ii) involving movement (e.g. hits, plays, embraces, catches);

(iii) neutral valence (e.g. avoids, pretends, reads, searches); and (iv) negative valence (e.g. scolds, mocks, steals, complains). Within each of the subclasses we merge interactions based on how similar their meanings are in common usagethis process is verified by multiple people.

We also reduce the number of relationships to 15 major classes: stranger, friend, colleague, lover, enemy, acquaintance, ex-lover, boss, worker, manager, customer, knows-byreputation, parent, child and sibling. Directed interactions and relationships are used between one person to another. For example when a parent → informs → child, the opposite directed interaction from the child to their parent can be listens to or ignores. Additionally, interactions and relationships can also be bidirectional, both people act with the same intention e.g. lovers kiss each other.

Predicting Interactions and Relationships

We first present results for predicting interactions and relationships separately, followed by our joint model. Interaction classification. We analyze the influence of each modality for interaction classification separately in Table 1. Dialogs have a stronger impact on model performance as compared to visual features owing to the prevalence of conversation based interactions in movies. However, both modalities are complementary and when taken together provide a 2.6% increase in accuracy. As expected, combining all modalities including tracks for each participating character provide the highest prediction accuracy at 26.1%.

Apart from accuracy, we report soft accuracy, a metric that treats a prediction as correct when it matches any of the interactions overlapping with the clip, i.e. â ∈ a * ∪ O I (v). When using all modalities, we achieve 32.6% accuracy.

In Fig. 4 we see two example interactions that are challenging to predict based on visual cues alone. In the top example, we see that the ground-truth label reads is emphasized, possibly due to the dialog mentioning letters, and is chosen with highest score upon examining the visual tracks. The bottom example is an interesting case where no dialog (all 0 vector) helps predictions. In this case, our model seems to have learned that leaving corresponds to walking without any dialog. Again, by including information about tracks, our model is able to predict the correct label.

We also investigate the influence of different temporal feature aggregation methods in forms both average and sum as it allows to form the clip-level representations including the most influential segments.

Relationship classification. Relationships are often consistent for long durations in a movie. For example, strangers do not become friends in one moment, and parents always stay parents. We hypothesize that it is challenging to predict a relationship by watching one interaction, and show the impact of varying the number of clips (size of V ) in Fig. 5. We see that the relationship accuracy improves steadily as we increase the number of clips. The drop at 6 clips is within variance. We choose 18 clips as a trade-off between performance and speed. During training, we randomly sample up to 18 clips for the same pair of people having the same relationship. At test time, the clips are fixed and uniformly distributed along all occurrences of pairs of character.

Joint prediction for interactions and relationships. We set the loss trade-off parameter λ = 1.5 and jointly optimize the network to predict interactions and relationships. We evaluate different options on how the two tasks are modeled jointly in Table 3. Overall, concatenating relationship feature for predicting interactions performs best (Rel. → Int.). Table 2 shows that the relationship accuracy improves by 1.3%, while interactions gain a meagre 0.2%. On further study, we observe that some interactions achieve large improvements, while others see a drop in performance. For example, interactions such as hugs (+17%), introduces (+14%), and runs (+12%), are associated with specific relationships: siblings, child, lover with hugs; enemy, lover with runs. On the other hand, a few other interactions such as talks to, accuses, greets, informs, yells see a drop in performance from 1-8%, perhaps as they have the same top-3 relationships: friend, colleague, stranger.

Relationships show a similar trend. Sibling, acquaintance, lover correspond to specific interactions such as hugs, greets, kisses and improve by 11%, 8%, and 7% respectively. While boss and manager have rather generic interactions asks, orders, explains and reduce by 5-7%.

We observe that joint learning does helps improve performance. However, interactions performed by people with common relationships, or relationships that exhibit common interactions are harder for our joint model to identify leading to small overall improvement. We believe this is made harder due to the long tail classes.

Localizing Characters

We present an evaluation of character localization and interaction prediction in Table 5. We report interaction accuracy given the correct character pair; character pair prediction accuracy given the correct interaction; and the overall accuracy as joint.

Training with full supervision. In the case when the pair of characters are known (ground-truth pair p * is given), we achieve 25.5% accuracy for interactions. This is comparable to the setting where we only predict interactions (at 26.1%). We believe that the difference is due to our goal to maximize the score for the correct interaction and character pair over the entire matrix |P M |×|A|. The joint accuracy is 14.2%, significantly higher than random at 0.15%.

Training with weak supervision. Here, interaction labels are applicable at the clip-level, and we are unaware of which characters participate in the interaction even during training. Table 5 shows that sampling a character pair is better than arg max in Eq. 9 (4.6% vs. 7.8% joint accuracy) as it prevents the model from getting stuck at a particular selection. Furthermore, switching training from sum over all negatives to hard negatives (sum-max) after a burn-in period of 20 epochs further improves accuracy to 8.2%. increases to 25.8% as compared against 25.5% when training without relationships (Table 5).

Joint character localization, interaction and relationship prediction is presented in

When learning with weak labels, we see similar trends as before. Both multinomial sampling and switching from all (sum) to hard (sum-max) negatives improves the joint accuracy to a respectable 2.14% as compared to 2.71% with full supervision. Fig. 15 shows an example prediction from our dataset. We present joint prediction when no information is provided in part d in contrast to parts a, b, c where two of three pieces of information are given.

Test set. Table 7 compiles results of all our models on the test set. We see similar trends, apart from a drop in relationship accuracy due to different val and test distributions.

Overall, we observe that learning interactions and relationships jointly helps improve performance, especially for classes that have unique correspondences, but needs further work on other categories. Additionally, character localization is achievable and we can train models with weak labels without significant drop in performance.

Conclusion

We presented new tasks and models to study the interplay of interactions and relationships between pairs of characters in movies. Our neural architecture efficiently encoded multimodal information in the form of visual clips, dialog, and character pairs that were demonstrated to be complementary for predicting interactions. Joint prediction of interactions and relationships was found to be particularly beneficial for some classes. We also presented an approach to localize character pairs given their interaction/relationship labels at a clip-level, i.e. without character-level supervision during training. We will share modifications made to the MovieGraphs dataset to promote future work in this exciting area of improving understanding of human social situations.

We provide additional analysis of our task and models including confusion matrices, prediction examples for all our models, skewed distribution of number of samples for our classes, and diagrams depicting how we grouped the interaction and relationship classes.

A. Impact of Modalities

We analyze the impact of modalities by presenting qualitative examples where using multiple modalities help predict the correct interactions. Qualitative results presented here, refer to the quantitative performance indicated in Table 1. Fig. 7 shows that using dialog can help to improve predictions, Fig. 8 demonstrates the necessity of visual clip information and highlights that the two modalities are complementary. Finally, Fig. 9 shows that focusing on tracks (visual representations in which the two characters appear) provides further improvements to our model. Furthermore, Fig. 10 shows top-5 interaction classes that benefit most from using additional modalities.

Analyzing modalities. We also analyze the two models trained on only visual or only dialog cues (first two rows of Table 1). Some interactions can be recognized only with visual (v) features: rides 63% (v) / 0% (d), walks 29% (v) / 0% (d), runs 26% (v) / 0% (d); while others only with dialog (d) cues: apologizes 0% (v) / 66% (d), compliments 0% (v) / 26% (d), agrees 0% (v) / 25% (d).

Interactions that achieve non-zero accuracy with both modalities are: hits 64% (v) / 5% (d), greets 12% (v) / 57% (d), explains 25% (v) / 51% (d).

Additionally, the top-5 predicted classes for visual cues are asks 77%, hits 64%, rides 63%, watches 49%, talks on phone 41%; and dialog cues are asks 75%, apologizes 66%, greets 57%, explains 51%, watches 30%. As asks is the most common class, and watches is the second most common, these interactions work well with both modalities.

B. Joint Interaction and Relationships

Confusion matrices. Fig. 11 shows the confusion matrix in the top-15 most commonly occurring interactions on the validation and test sets. We see that multiple dialog based interactions (e.g. talks to, informs, and explains) are often confused. We also present confusion matrices for relationships in Fig. 12. A large part of the confusion is due to lack of sufficient data to model the tail of relationship classes.

Qualitative examples. Related to Table 2, Fig. 13 shows some examples where interaction predictions improve by jointly learning to model both interactions and relationships. Similarly, Fig. 14 shows how relationship classification benefits from our multi-task training setup.

C. Examples for Who is Interacting

Empirical evaluation shows that the knowledge about the relationship is important for localizing the pair of characters (Table 6). In Fig. 15, we illustrate an example where the dad walks into a room, sees his daughter with someone, and asks questions (see figure caption for details).

Finally, in Fig. 16, we show an example where the model is able to correctly predict all components (interaction class, relationship type and the pair of tracks) in a complex situation with more than 2 people appearing in the clip.

D. Dataset Analysis

Fig. 17 and Fig. 18 show normalized distributions for the number of samples in each class for train, validation and test sets of interactions and relationships respectively. As can be seen the most common classes appear many more times than the others. Data from a complete movie belongs to one of the three train/val/test sets to avoid model bias on the plot and characters behaviour. Notably, this means that the relative ratios between number of samples per class are also not necessarily consistent making the dataset and task even more challenging.

In the main paper, we described our approach to group over 300 interactions into 101 classes, and over 100 relationships into 15. We use radial tree diagrams to depict the groupings for interaction and relationship labels, visualized in Fig. 19 and 20 respectively. -What a bruiser.

-The Cardinals just refuse to go quietly into the desert night. I'm just trying to be a little poetic.

Figure 2 :

 2 Figure 2: Normalized correlation map between (selected) interactions and relationships. Darker regions indicate higher scores.

Figure 3 :

 3 Figure3: Model architecture. Left: Our input is a trimmed video clip for one interaction, and consists of visual frames and all dialogues within its duration. Each interaction is associated with two characters, and they are represented visually by extracting features from cropped bounding boxes. Modalities are processed using fixed pre-trained models (BERT for textual, I3D for visual) to extract clip representations denoted by Φ(v). Right: In the second panel, we show the architecture of our joint interaction and relationship prediction model. In particular, multiple clips are used to compute relationships, and we fuse these features while computing interaction labels.

Figure 4 :

 4 Figure 4: Influence of different modalities on interaction prediction performance. In each example, we show the top 5 interactions predicted by the visual cues (left), the visual + dialog cues (center), and visual + dialog + track information (right). The correct label is marked with a green bounding rectangle. Discussion in Sec. 4.3.

Figure 5 :

 5 Figure 5: Relationship accuracy increases as we analyze more clips. We choose 18 clips as a trade-off between performance and speed.

-Figure 6 :

 6 Figure 6: Example for joint prediction of interaction (Int), relationship (Rel), and character pair (Char) by our model. The visual clip, dialog, and possible track pairs are presented on the left. Given 2 pieces of information, we are able to answer the third: Who? Int + Rel → Char; Doing what? Char + Rel → Int; and What relationship? Char + Int → Rel. We can also jointly predict all three components by maximizing scores along all dimensions of the 3D tensor. Best seen on screen with zoom.

Visual-Figure 7 :

 7 Figure 7: Improvement in prediction of interactions by including textual modality in addition to visual. The model learns to recognize subtle differences between interactions based on dialog. The example is from Meet the Parents (2000).

  -A tough hit on Tidwell. I'd have made a touchdown. That's his sixth catch tonight. A first down at the 19 yard line. -Another savage hit on Tidwell. They're really working Tidwell.

Figure 8 :Figure 9 :

 89 Figure 8: Improvement in prediction of interactions by including visual modality in addition to textual. The top-5 predicted interactions reflect the impact of visual input rather than relying only on the dialog. The example is from Jerry Maguire (1996).

Figure 10 :

 10 Figure 10: Each plot shows 5 interaction classes that have the most number of improved instances by including an additional modality. Specifically, the x-axis denotes the number of samples in which interaction prediction performance improves. Left: From only visual clip representation to visual and textual. As expected, using dialogues in addition to video frames boosts performance for classes that rely on dialog e.g. explains, informs. Middle: From only textual clip representation to visual and textual. Visual clip representations influence classes as kisses, runs during which people usually do not talk (dialog modality filled with zeros). Right: Finally, including all three modalities visual, textual, tracks improves performance over using visual and textual. Track pair localization improves recognition of classes typically used in group activities.

Figure 11 :s t r a n g e r f r ie n d lo v e r c o ll e a g u e s ib li n g a c q u a in t a n c e p a r e n t w o r k e r c h il d b o s s k b r c u s t o m e r m a n a g eFigure 12 :

 1112 Figure 11: Confusion matrices for top-15 most common interactions for validation set (left) and test set (right). Model corresponds to the "Int. only" performance of 26.1% shown in Table 2. Numbers on the right axis indicate number of samples for each class. s t r a n g e r f r ie n d lo v e r c o ll e a g u e s ib li n g a c q u a in t a n c e p a r e n t w o r k e r c h il d b o s s k b r c u s t o m e r m a n a g e r e x -lo v e r e n e m y

Figure 17 :s t r a n g e r c o l l e a g u e f r i e n d l o v e r b o s s e n e m y w o r k e r a c q u a i n t a n c e p a r e n t k b r m a n a g e r c h i l d c u s t o m e r s i b l i n g e

 17 Figure 17: Distribution of interaction labels in train/val/test sets. Sorted by descending order based on train set.

Figure 18 :

 18 Figure 18: Distribution of relationship labels in train/val/test sets. Sorted by descending order based on train set.

  ad m it s ad m its ad m it s to co n fe ss e s to a n n o u n c e s a n n o u n c e s b re a k s n e w s to d e c la re s lo v e to re c e iv e s n e w s fr o m re c e iv e s n e w s fr o m re c e iv e s n e w s a b o u t w is h e s w is h e s h o p e s fo r s a v e s s a v e s d e fe n d s c h e c k s c h e c k s c h e c k s o n t a k e s c a r e o f t a k e s c a r e o f p e t s t r e a t s a s k s a s k s q u e s t io n s a s k s a b o u t a s k s f o r h e lp a s k s a b o u t b e h a v io r a s k s a b o u t s it u a t io n a s k s p e r m is s io n a s k s f o r a d v ic e

  rea ds rea ds rea ds to loo ks for loo ks for se ar ch es di sc us se s di sc us se s ne go tia te s di sc us se s pr ob le m no ti ce s no tic es re co gn iz es un de rs ta nd s sp ot s re m e m b e rs re m e m b e rs k n o w s a b o u g r e e s w it h p r o t e s t s c r it ic iz e s in te rr u p ts d is li k e s d is b e li e v e s d is a p p ro v e s o f c o m p la in s c o m p la in s c o m p la in s a b o u t re g re ts a c c u se s a c c u s e s b la m e s re p ri m a n d s re p ro a ch e s in su lt s in su lt s off en ds sc off s at di sr es pe ct s

Figure 19 :

 19 Figure 19: Diagram depicting how we group 324 interaction classes (outer circle) into 101 (inner circle). Best seen on the screen.

Figure 20 :

 20 Figure 20: Diagram depicting how we group 107 relationship classes (outer circle) into 15 (inner circle).

Table 1 :

 1 Interaction prediction accuracy for different modalities.

	Modalities		Interaction Accuracy
	Visual Dialog Tracks Top-1 Top-1 Soft Top-5
	-	-	18.7	24.6	45.8
	-	-	22.4	30.1	50.6
		-	25.0	31.9	54.8
			26.1	32.6	57.3

Table 4 .

 4 Max-pooling outper-

	Clip Visual	Clip Textual	Tracks
	asks 0.77	explains 0.7	reads 0.66
	informs 0.72	apologizes 0.67	explains 0.62
	explains 0.69	reads 0.65	sits 0.6
	suggests 0.65	reassures 0.62	eats with 0.58
	compliments 0.63	informs 0.62	greets 0.58
	Clip Visual	Clip Textual	Tracks
		-Hello. Long time no see!
		-Hello. Long time no see!
	walks 0.72	watches 0.76	leaves 0.75
	explains 0.68	leaves 0.73	walks 0.74
	watches 0.66	walks 0.72	runs 0.7
	stays 0.65	runs 0.71	watches 0.62
	accuses 0.65	follows 0.69	follows 0.69

-«Dear Sleepless and Son, I have never written a letter like this in my life.» -That's what everyone writes at the beginning of letters to strangers.

Table 2 :

 2 Top-1 accuracy for the joint prediction of Int. and Rel.

	Task	Random Int. only Rel. only Joint
	Interaction		0.99	26.1	-	26.3
	Relationship		6.67	-	26.8	28.1
	Method	Int.	Rel.

Rel. ↔ Int. 25.3 26.8 Rel. ← Int. 26.3 25.9 Rel. → Int. 26.3 28.1

Table 3 :

 3 Different architectures for joint modeling of interactions and relationships.

	Method	Int.
	avg	24.2
	sum	25.4
	max	26.1

Table 4 :

 4 Impact of temporal aggregation methods on interaction accuracy.

	Supervision Negatives	Multinom. Sampling Int. Character Joint Accuracy
	Random	-	-	0.99	15.42	0.15
	Full	sum	-	25.5	55.0	14.2
	Weak	sum	-	18.9	20.0	4.6
	Weak	sum		25.1	25.0	7.8
	Weak	sum-max		23.0	32.3	8.2

Table 5 :

 5

Joint prediction of interactions and character pairs for fully and weakly supervised settings. See Sec. 4.4 for a discussion.

Table 6 :

 6

	Supervision Negatives	Multinom. Sampling Int. Rel. Char. Joint Accuracy
	Random	-	-	0.99 6.67 15.42 0.01
	Full	sum	-	25.8 16.6 88.3 2.71
	Weak	sum		25.8 12.0 42.0 0.86
	Weak	sum-max		20.8 21.8 33.9 2.14

Joint interaction, relationship, and character pair prediction accuracy. Other labels are provided when predicting columns: Int., Rel., and Char. See Sec. 4.4 for a discussion.

Table 6

 6

	All methods	Supervision	Accuracy Int. Rel. Char. Joint
	Int only	-	20.7	-	-	-
	Rel only	-	-	22.4	-	-
	Int + Rel	-	20.7 20.5	-	-
	Int + Char	Full	19.7	-	52.8 11.1
	Int + Char	Weak	17.9	-	29.7 6.34
	Int + Rel + Char	Full	20.0 18.6 88.8 2.29
	Int + Rel + Char	Weak	15.6 29.6 21.6 1.50

. In the case of learning with GT character pairs (fully supervised), including learning of relationships boosts accuracy for predicting character pairs to an impressive 88.3%. The interaction accuracy also

Table 7 :

 7 Evaluation on the test set. The columns Int., Rel, and Char refer to interaction, relationship, and character pair prediction accuracy. During joint learning with full/weak supervision, individual accuracies are reported when other labels are given.

Table 2 .

 2 Numbers on the right axis indicate number of samples for each class.

	Normalized #samples	train val test
	asks watches explains informs talks to suggests orders greets hears talks about answers leaves compliments stays reassures walks kisses yells argues agrees hugs talks on the phone disagrees with gives shows plays eats with helps runs hits sits congratulates meets teases rides admits follows apologizes introduces assures works with accuses jokes laughs refuses complains reminds convinces calls ignores runs from encourages approaches insults holds pushes shoots leads discusses threatens dances with takes reveals comes home smiles at looks for restrains wakes up announces brings notices reads lies thinks fishes tries commits crime wishes saves remembers visits photographs takes care of shushes walks away from receives news from opens hides from states gossips passes by points at throws checks puts pulls sings to sends away repeats has sex obeys

Acknowledgements. This project was partially supported by the Louis Vuitton -ENS Chair on Artificial Intelligence and the French government under Agence Nationale de la Recherche as part of "Investissements d'avenir" program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute).

-If anybody else wants to come with me, this is a chance for something real, and fun, and inspiring in this godforsaken business,and we will do it together. Who's coming with me? Who's coming with me? Who's coming with me besides Flipper here? This is embarrassing. where a father (Jack) walks into a room while his daughter (Pam) and the guy (Greg) are kissing. Our goal is to predict the two characters when the interaction and relationship labels are provided. In this particular example, we see that Dad asks Pam a question (What are you two doing in here?). Note that their relationship is encoded as (Pam → child → Jack), or equivalently, (Jack → parent → Pam). When searching for the pair of characters with a given interaction asks and relationship as parent, our model is able to focus on the question at the clip level as it is asked by Jack in the interaction, and correctly predict (Jack, Pam) as the ordered character pair. Note that our model not only considers all possible directed track pairs (e.g. (Greg, Pam) and (Pam, Greg)) between characters, but also singleton tracks (e.g. (Jack, None)) to deal with situations when a person is absent due to failure in tracking or does not appear in the scene.

Ints Prediction

-Oh, hi. Hi, Jess. The model is able to correctly predict the two characters, their order, interaction and relationship. In this case, Emily introduces David to Jess. Jess is also her hired babysitter, and thus their relationship is -Emily is boss of Jess.