Yoann Baveye
email: yoann.baveye@univ-nantes.fr

Christel Chamaret
email: christel.chamaret@technicolor.com

Emmanuel Dellandr Éa
email: emmanuel.dellandrea@ec-lyon.fr

Liming Chen
email: liming.chen@ec-lyon.fr

E Dellandréa

Affective Video Content Analysis: A Multidisciplinary Insight

Keywords: Multidisciplinary review, Video content analysis, Cognitive models, Affective computing, State of the art. !

In our present society, the cinema has become one of the major forms of entertainment providing unlimited contexts of emotion elicitation for the emotional needs of human beings. Since emotions are universal and shape all aspects of our interpersonal and intellectual experience, they have proved to be a highly multidisciplinary research field, ranging from psychology, sociology, neuroscience, etc., to computer science. However, affective multimedia content analysis work from the computer science community benefits but little from the progress achieved in other research fields. In this paper, a multidisciplinary state-of-the-art for affective movie content analysis is given, in order to promote and encourage exchanges between researchers from a very wide range of fields. In contrast to other state-of-the-art papers on affective video content analysis, this work confronts the ideas and models of psychology, sociology, neuroscience, and computer science. The concepts of aesthetic emotions and emotion induction, as well as the different representations of emotions are introduced, based on psychological and sociological theories. Previous global and continuous affective video content analysis work, including video emotion recognition and violence detection, are also presented in order to point out the limitations of affective video content analysis work.

INTRODUCTION

A CCORDING to legend, when "L'arrivée d'un train en gare de La Ciotat" directed and produced by Auguste and Louis Lumière was first screened in 1896, the audience was so terrified at the sight of the oncoming locomotive that people screamed and tried to hide under their seats. Nowadays, film-lovers are more hardened but still enjoy the almost unlimited potential of movies for experiencing emotions. And with the massive movie collection available online through popular on-demand streaming media websites such as Netflix 1 or M-GO 2 , increasing day after day, film spectators can feel everywhere a large variety of emotions.

Under these circumstances, knowing in advance the emotions that a movie is likely to elicit from its viewers is highly beneficial; not only to improve accuracy for video indexing, and summarization (e.g. [START_REF] Arifin | Affective level video segmentation by utilizing the pleasure-arousal-dominance information[END_REF], [START_REF] Zhao | Video indexing and recommendation based on affective analysis of viewers[END_REF]), but also for mood-based personalized content delivery [START_REF] Hanjalic | Extracting moods from pictures and sounds: Towards truly personalized tv[END_REF]. While major progress has been achieved in computer vision for visual object detection, scene understanding and high-level concept recognition, a natural further step is the modeling and recognition of affective concepts. This explains why the affective video content analysis research topic has emerged and attracted increasingly more attention in recent years.

1. https://www.netflix.com/ 2. http://www.mgo.com/ Affective video content analysis aims at automatic recognition of emotions elicited by videos. There are three perspectives for affective movie content analysis work, each related to specific emotion detection: intended, induced and expected emotion [START_REF] Hanjalic | Extracting moods from pictures and sounds: Towards truly personalized tv[END_REF]. The intended emotion is the emotion that the film maker wants to induce in the viewers. The induced emotion is the emotion that a viewer feels in response to the movie. The expected emotion is the emotion that the majority of the audience feels in response to the same content. In other words, the expected emotion is the expected value of experienced (i.e. induced) emotion in a population. While the induced emotion is subjective and context dependent, the expected emotion can be considered objective, as it reflects the more-or-less unanimous response of a general audience to a given stimulus [START_REF] Hanjalic | Extracting moods from pictures and sounds: Towards truly personalized tv[END_REF]. Intended and expected emotions do not always match [START_REF] Philippot | Inducing and assessing differentiated emotionfeeling states in the laboratory[END_REF], [START_REF] Gross | Emotion elicitation using films[END_REF]. It is a fact that a movie can be unsuccessful in conveying the intended emotion (e.g., viewers laughing while watching horror movies intended to elicit fear). The degree of effectiveness with which a movie creates the desired emotion in the viewer can be a basic criterion for assessing movie quality [START_REF] Malandrakis | A supervised approach to movie emotion tracking[END_REF].

A large number of work has been proposed in the literature to tackle this highly challenging task. However, a review of the state of the art shows that affective video content analysis work from the computer science community benefits very little from the progress made in other research fields, including psychology, sociology, and neuroscience. Indeed, it appears from previous works that the main approach for analyzing the affective content of videos is to use machine learning to build, using a dataset, models such as Hidden Markov Models (HMMs), Support Vector Machines for Regression (SVRs), and more recently Convolutional Neural Networks (CNNs).

The main contribution of this paper is to propose a multidisciplinary state of the art for affective movie content analysis, in order to promote and stimulate exchanges between researchers from a wide variety of fields. In contrast to previous state of the art studies [START_REF] Calvo | Affect detection: An interdisciplinary review of models, methods, and their applications[END_REF], [START_REF] Wang | Video affective content analysis: A survey of state-of-the-art methods[END_REF], this work thus confronts the ideas and models from psychology, sociology, neuroscience, and computer science with a focus on the emotions elicited by movies. General psychological emotional theories are described alongside models for aesthetic emotions, focusing on emotions in response to a work of art. With these theoretical models in mind, previous works on continuous and global affective video content analysis are presented, including video emotion recognition and violence detection, which are related to the expected affect estimation from audiovisual features. Studies, known as implicit video affective content analysis, have also been conducted using the spontaneous response of users watching the videos (e.g., facial recordings, physiological responses). However, this topic has already been investigated in [START_REF] Calvo | Affect detection: An interdisciplinary review of models, methods, and their applications[END_REF]. This paper focuses rather on direct video affective content analysis studies, using audiovisual features extracted from the videos to infer a given type of emotion.

The paper is organized as follows. Section 2 describes main psychological emotional theories, gives an overview of neuroscience work, and defines the concepts of aesthetic emotions and emotion induction, as well as different representations of emotions. Section 3 presents previous global and continuous affective video content analysis workand lists current existing affective video databases.Limitations are discussed in Section 3.5, while the paper ends in Section 5 with conclusions.

EMOTIONS

More than one hundred different scientific definitions were inventoried [START_REF] Kleinginna | A categorized list of emotion definitions, with suggestions for a consensual definition[END_REF]. Kleinginna and Kleinginna proposed a consensus by suggesting the following definition considered as one of the most comprehensive definitions of emotions [START_REF] Kleinginna | A categorized list of emotion definitions, with suggestions for a consensual definition[END_REF]:

"Emotion is a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can (a) give rise to affective experiences such as feelings of arousal, pleasure/displeasure; (b) generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labeling processes; (c) activate widespread physiological adjustments to the arousing conditions; and (d) lead to behavior that is often, but not always, expressive, goal-directed, and adaptive." (pp. 355)

Main psychological approaches

Decades of research driven by psychologists interested in the comprehension of emotions led to three major theoretical approaches: basic, appraisal, and psychological constructionist [START_REF] Gendron | Reconstructing the past: A century of ideas about emotion in psychology[END_REF].

Basic emotions: facial expression and emotion

Basic emotion theorists were originally inspired by the ideas expressed by Darwin. In The Expression of the Emotions in Man and Animals [START_REF] Darwin | The expression of the emotions in man and animals[END_REF], Darwin deduced the universal nature of facial expressions. Thus, basic emotion models assume that several emotions are automatically triggered by objects and situations in the same way everywhere in the world. This idea of universality was reused by Ekman and Friesen to develop the Emotion Facial Action Coding System (EM-FACS) [START_REF] Ekman | Em-facs coding manual[END_REF], which is a simplified although less common version of FACS [START_REF] Ekman | Facial expression and emotion[END_REF] designed explicitly for the coding of facial behavior related only to emotion. Ekman considers emotions to be discrete states that are associated with facial expressions. Consequently, he postulates that there are a fixed number of emotions. Plutchik agreed with Ekman's biologically driven perspective and theorized that basic emotions are biologically primitive and have evolved in order to increase the reproductive fitness of the animal [START_REF] Plutchik | A general psychoevolutionary theory of emotion[END_REF].

The term basic is used in the literature to describe three perspectives [START_REF] Ekman | Basic emotions[END_REF]: More specifically, the basic emotion theory identifies several criteria for an emotion to be considered basic [START_REF] Ekman | Basic emotions[END_REF], such as: distinctive universal signals, distinctive physiology, distinctive universals in antecedent events (i.e., some stimuli render a universal response), presence in other primates, quick onset, brief duration. Ekman used these criteria to define six basic emotions: fear, anger, sadness, disgust, joy, and surprise [START_REF]An argument for basic emotions[END_REF]. In later works, Ekman and Heider [START_REF] Ekman | The universality of a contempt expression: A replication[END_REF] argued for the inclusion of additional basic emotions, such as contempt. Many other lists of basic emotions have been proposed in the literature [START_REF] Ortony | What's basic about basic emotions?[END_REF]. For example, Plutchik identified eight basic emotions, which he grouped into four pairs of polar opposites (joy-sadness, anger-fear, trust-distrust, surprise-anticipation) [START_REF] Plutchik | A general psychoevolutionary theory of emotion[END_REF], while Izard et al. [START_REF] Izard | The differential emotions scale: A method of measuring the subjective experience of discrete emotions[END_REF] postulated 10 fundamental emotions, universally discernible in the human facial expression.

Appraisal theories

Appraisal models assume that emotions are triggered by the interpretation of stimulus events and thus can be seen as relevance detectors [START_REF] Frijda | The emotions, ser. Studies in Emotion and Social Interaction[END_REF]. Appraisal theories study why individuals react differently when confronted with the same stimulus [START_REF] Scherer | Appraisal considered as a process of multi-level sequential checking[END_REF]. Arnold made early contributions to the appraisal theory, suggesting that an initial appraisal triggers the emotional sequence by arousing both the appropriate physiological reactions and the emotional experience itself [START_REF] Arnold | Emotion and personality[END_REF]. Lazarus specified two types of appraisal: primary and secondary appraisals [START_REF] Lazarus | Psychological stress and the coping process[END_REF]. Lazarus suggested that primary appraisal is focused on the conception of the significance or meaning of the stimulus to the organism, while secondary appraisal aims at assessing the ability of the organism to cope with the consequences of the stimulus. Another influential appraisal theory of emotion is the Component Process Model (CPM), Scherer's major contribution to the study of appraisal processes, described in the remainder of this section.

The CPM postulates that the emotion process is a psychological construct driven by subjective appraisal and is the consequence of synchronized changes in five components corresponding to five distinctive functions [START_REF] Scherer | The component process model: Architecture for a comprehensive computational model of emergent emotion[END_REF]: cognitive appraisal (evaluation of objects and events), physiological activation (system regulation), motivational tendencies (preparation and direction of action), motor expression (communication of reaction and behavioral intention), and subjective feeling state (monitoring of internal state and external environment). Figure 1 shows the general architecture of the CPM organized into three modules: appraisal, response patterning, and integration/categorization.

The CPM considers the experience of emotions as the result of the recursive multilevel sequential evaluation checking of an emotional stimulus or event. It is a response to the evaluation of a stimulus relevant to major concerns of the organism. The first module, i.e., the appraisal module, is the most important element in the CPM. It determines whether an emotion episode is elicited and its characteristics are based on four major appraisal objectives [START_REF] Scherer | Appraisal considered as a process of multi-level sequential checking[END_REF]: Relevance Detection (scan for salient events requiring attention), Implication Assessment (estimation of implication, consequences), Coping Potential Determination (assessment of coping potential, i.e., actions) and Normative Significance Evaluation (compatibility with social norms and self-concept). These objectives are evaluated based on stimulus evaluation checks (SECs) defined as novelty, intrinsic pleasantness, relevance to goals and needs, cause, probable outcomes, failure to meet expectations, conduciveness to goals and need, urgency, control, power, adjustment, internal standards, and external standards. The results of the SECs are highly subjective and are biased by individual differences, moods, cultural values, group pressures, or other context factors [START_REF] Scherer | Culture-specific appraisal biases contribute to emotion dispositions[END_REF]. This is why a stimulus may evoke different emotions for different people at different places: it is the evaluation of the events, rather than the events themselves, which determines the characteristics of an emotional episode.

The result of the appraisal updates the motivational state that existed before the occurrence of the emotional stimulus or event. Both the appraisal results and motivational changes affect the automatic nervous system and the somatic nervous system. These components are all continuously updated as events and appraisals change. Scherer claims that, even if most of these components are examined unconsciously through parallel processing, some components may be evaluated consciously to allow more controlled regulation processes [START_REF] Scherer | The component process model: Architecture for a comprehensive computational model of emergent emotion[END_REF].

A psychological constructionist model: the Core Affect

Psychological constructionist models presume that emotions can be broken down into primitives that are also involved in other mental states [START_REF] Gendron | Reconstructing the past: A century of ideas about emotion in psychology[END_REF]. While appraisal models assume that it is the evaluation of the stimulus that determines the characteristics of an emotional episode, psychological constructionist models assume that an emotion emerges when one's internal state is consciously understood in relation to an event.

Russell describes emotions as a part of a general component process called the Core Affect [START_REF] Russell | Core affect and the psychological construction of emotion[END_REF]. The core affect is a primitive, universal and simple neurophysiological state that controls moods, when core affect is experienced as freefloating, and emotions, when core affect can be attributed to some cause, whether based on reality or fiction. In the core affect model, raw feelings (the conscious experience) are a blend of two dimensions: pleasure-displeasure (called pleasure or valence) and activation-deactivation (called arousal or energy).

Psychological constructionist models suggest that an emotional episode is an event that counts as a member of an emotion category. The prototypical emotional episode begins with an antecedent event perceived in terms of its affective quality. The core affect is altered and attributed to the antecedent to enable perceptual cognitive processing of the antecedent with various manifestations: instrumental action, physiological and expressive changes, subjective conscious experiences, and emotional meta-experience. The emotional meta-experience is a self-perception: the emotional episode is noticed by the person who categorizes the emotional episode. Emotion categories are subjective since they are mental representation of emotions. For Russell, emotions respond to the continuous flow of events (whether based on reality or fiction) and are influenced at the same time by the background environment (weather, odors, noise) and the social environment [START_REF] Russell | Emotion and the environment[END_REF].

Summary of the main psychological approaches

The three main psychological approaches introduced in the previous sections, namely the basic, appraisal, and psychological constructionist models, share common ideas.

All three approaches emphasize that emotions are constructed from universal basic biological or psychological parts. However, psychological constructionist and appraisal models differ from basic models insofar as they assume that the social context of the situation and/or cultural differences have an effect on the experience of emotions [START_REF] Gendron | Reconstructing the past: A century of ideas about emotion in psychology[END_REF].

Psychological constructionist models and appraisal models both consider emotion as an act of making meaning. However, the meaning analysis is different for each approach. Psychological constructionist models, including Russell's Core Affect [START_REF] Russell | Core affect and the psychological construction of emotion[END_REF], assume that an emotion arises when a person's internal state is understood. For appraisal models, emotions are intentional states created by the evaluation of an original stimulus, and not by the internal state of the body. The internal state is only affected by this meaning analysis.

Neuroscience

Related to the marked improvement in techniques, the neuroscience field has also contributed to the understanding of emotion mechanisms over the last ten years. The findings of specific brain structures and neural networks can be credited to new methods and experimental paradigms in brain imaging techniques. As an example, while electrophysiological methods, such as EEG or ERP, provided high temporal sampling rates (roughly at a millisecond), functional brain imaging methods are able to provide spatial details at a high resolution. Both have allowed a better understanding of when and where emotions occur [START_REF] Deak | Brain and emotion: Cognitive neuroscience of emotions[END_REF].

By essence, the neuroscience field and its findings oppose experimental psychology by proving that emotional brain mechanisms are typically unconscious. These involuntary processes entail behavioral tendencies thus generating conscious feelings that interact back with emotion [START_REF] Franks | The neuroscience of emotions[END_REF]. Even if emotional mechanisms originate in the brain, they are expressed and transmitted through the subject body as electrical and/or hormonal activities.

The first theories on brain modeling emerged at the end of the 19 th century. As a precursor in the field, James assumed that emotion is created by means of the body's reaction to an external stimulus [START_REF] James | What is an emotion?[END_REF]. Despite being controversial, this theory inspired a number of scientists whose grail was to locate the centrum for emotional perception. Several experiments during the 20 th century validated the assumptions that the thalamus is the centrum for emotional perception, while the hypothalamus conciliates emotional expressions [START_REF] Morris | Functional neuroanatomy of human emotion[END_REF]. Alternatively, Papez [START_REF] Papez | A proposed mechanism of emotion[END_REF] demonstrated a more complex structure with a neural circuit in several brain areas. In addition to the thalamus and hypothalamus, the mamillary bodies, the cingular gyrus and the hippocampus were also concerned by emotional mechanisms. Nowadays, it is universally admitted that the limbic system, enclosing the thalamus, hippocampus, amygdala, hypothalamus and anterior cingulate cortex, is the major brain feature dealing with emotion.

Based on such experiments and deep analysis of brain functions, some computational models were designed in the neuroscience community [START_REF] Fellous | Emotional circuits and computational neuroscience[END_REF]. Such models reflect the way in which emotional mechanisms are interpreted by authors. Thus, Grossberg and Schmajuk [START_REF] Grossberg | Neural dynamics of attentionally modulated pavlovian conditioning: Conditioned reinforcement, inhibition, and opponent processing[END_REF] designed a model of conditioned affective states based on antagonistic neural processes, for example fear and relief. They considered the previous state of the model to evaluate the new value of the current emotional event. They simulated the neural circuits involved in emotion elicitation through the different brain areas (e.g., amygdala), which they identified as being stakeholders in emotion expression. Emotional activity proved to be difficult to isolate from other functions. Also, recent research specifically associated emotion with memory [START_REF] Hamann | Cognitive and neural mechanisms of emotional memory[END_REF], attention [START_REF] Brosch | The flexibility of emotional attention: Accessible social identities guide rapid attentional orienting[END_REF] and decision-making [START_REF] Brosch | Appraising value: The role of universal core values and emotions in decision-making[END_REF]. These other functions are potentially a means of refining the existing computational models and of creating more collaboration between neuroscience and image processing communities.

Aesthetic emotions

The theories introduced in the previous sections model emotions regardless of the significant stimulus that provoked the emotions. This stimulus could be either a natural phenomenon, the behavior of other people or animals, or even one's own behavior [START_REF] Scherer | What are emotions? and how can they be measured?[END_REF]. However, this paper is concerned with the emotions that are provoked by movies. In this section we focus on a particular type of emotions, namely aesthetic emotions, for the reasons explained below.

Definition and models

An aesthetic emotion is an emotional response to a work of art (paintings, pictures, but also songs, movies, etc.) which can be described by three characteristics [START_REF] Marković | Components of aesthetic experience: aesthetic fascination, aesthetic appraisal, and aesthetic emotion[END_REF]:

• Persons are involved, in the state of intense attention engagement, and are strongly focused on a particular object.

•

The viewer appraises the aesthetic objects as parts of virtual reality and, finally, has a strong feeling of unity with the object of aesthetic appraisal.

• Aesthetic emotions are not oriented towards the satisfaction of bodily needs.

In the literature, aesthetic information processing models share a common architecture: the emotional experience starts with a stimulus input, continues with the appraisal of the stimulus (from low level characteristics to deeper memorial instances) and ends with the evaluative judgment of the stimulus and the emotional response.

Linking aesthetics and emotion, Leder et al. [START_REF] Leder | A model of aesthetic appreciation and aesthetic judgments[END_REF] proposed a model of aesthetic and emotional experience of modern art in an artistic context. This model is composed of five main stages resulting in two distinct outputs: an aesthetic judgement which is the result of the evaluation of the cognitive mastering stage and an aesthetic emotion based on the affective state of satisfaction. Like Scherer, the model proposed by Leder postulates both automatic and deliberate processing.

Each of the five stages is concerned with different cognitive analyses. Processing during the first two stages is automatic and implicit. Features such as complexity, contrast, symmetry, order, and grouping are perceptually analyzed on the perceptual analysis level. In the second stage, an unconscious recall based on past experience begins. This implicit memory integration stage is also influenced by features such as familiarity, prototypicality, and the peak-shift principle. Then, during the last three stages, processing takes place consciously and forms a feedback-loop. The explicit classification and cognitive mastering, based on analysis of the style, content, and interpretation of the work of art, are also influenced by previous evaluations that have not been subjectively experienced as successful. Finally, the evaluation stage guides the aesthetic processing by measuring its success and enters into social interaction discourses which will be the input of another artistic evaluation.

However, the Leder's model is too simplified and only considers aesthetic experience as affectively positive and self-rewarding (successful processing). It does not take into account the negative aspect, the antinomy, and the variety of art experience [START_REF] Hsu | Visible and expression. study on the intersubjective relation between visual perception, aesthetic feeling, and pictorial form[END_REF]. Several works extended Leder's model to offer a more precise model of aesthetic processing. For example, Marković created a model in which the aesthetic experience is closer to arousal, i.e., the interest for the work of art, than to other dimensions of subjective experience [START_REF] Marković | Components of aesthetic experience: aesthetic fascination, aesthetic appraisal, and aesthetic emotion[END_REF].

In this multimodal model, the narrative content (e.g., story, symbolism) and the composition form (e.g., colors, gestures, sounds) both influence the aesthetic emotion, films being more focused on the narrative content.

A sociological perspective

Aesthetic emotions, and, more specifically, emotions provoked by movies, can also be analyzed from a sociological perspective. Hochschild was one of the first to study the sociology of emotions and how shared norms can influence the way we want to try to feel emotions in given social relations [START_REF] Hochschild | Emotion work, feeling rules, and social structure[END_REF].

Wiley discussed the differences between the emotions people experience watching a movie, called "movie emotions" and those occurring in our everyday life [START_REF] Wiley | Emotion and film theory[END_REF]. He explained that viewers split attention between the movie and the physical environment, which helps regulate distance from the movie and creates an aura of safety and that the effects of movie emotions tend to end with the movie or at least decrease. Furthermore, the viewer is not the subject of the movie emotions, they happen because an identification is built with the character. For Wiley, movie emotions come with clear labels since narratives are written with precise emotional scripts. They are included in the movie with dialogues, clearly structured situations, transparent tendencies and musical cues. Movie emotions can be anticipated since films tend to follow the usual stability-instability-stability rule and because the music is geared to place the viewer in the appropriate emotional channel. These arguments show that it makes sense to investigate computational models based on multimodal features to predict the emotions provoked by movies.

For Wiley, movie emotions are desired: in watching a movie, the viewer wants to feel frequent, dense and almost wall-to-wall emotions [START_REF] Wiley | Emotion and film theory[END_REF]. This may result, for example, from a need to escape boredom or to forget the day's upsets. Movie emotions are also quite intense: viewers deal with more emotions than in a comparable period of time of a typical day. Movie emotions can be increased with social aspects (co-viewers' laughs). It is also easier to admit and talk about movie emotions because they are just fantasy so nobody can be blamed for having them.

Types of emotional processes in response to multimedia

Three types of emotional processes in response to multimedia are described in previous work: emotion induction, emotion contagion, and empathic sympathy [START_REF] Wirth | Media and emotions[END_REF].

Induced emotions are the emotions that viewers feel in response to a multimedia content with respect to their goals and values. For example, the animated movie "Big Buck Bunny"3 features an evil squirrel tearing up a butterfly. The situation is likely to elicit negative emotions from viewers (disgust, anger, etc.), although the imaginary squirrel is enjoying the situation. The negative response from viewers, i.e. the induced emotion, is due to their perception of the context according to their goals and values biased by the identification built with one or more characters of the movie.

With the emotional contagion process, the viewer is affected by the expressed emotion from a multimedia content without understanding in detail how the emotional expression of the multimedia content may have been developed. This process has to be distinguished from emotion perception, which refers to the perception of emotions expressed by the multimedia content without evoking affective responses in viewers. Last but not least, empathic sympathy occurs when viewers are not affected by the situation or event directly, but follow the appraisal steps leading to the emotion experienced by the characters in the multimedia content.

Finally, emotions, as defined above, have to be distinguished from other affective phenomena such as moods. Moods are diffuse affect states generally of low intensity, may last for hours or even days, and are often not clearly linked to an event or specific appraisals [START_REF] Scherer | What are emotions? and how can they be measured?[END_REF].

Representations of emotions

Diverse representations for emotions have been proposed in the literature. They are derived from the theories introduced in Section 2.1.

Categorical

The categorical emotions approach, also commonly referred to as the discrete emotions approach, is very natural since it goes back to the origin of language and the emergence of words and expressions representing clearly separable states. Many discrete categorizations of emotions have been proposed. The basic emotions introduced in Section 2.1.1, such as the six basic universal emotions proposed by Ekman [START_REF] Ekman | Basic emotions[END_REF], or the eight primary emotions defined by Plutchik [START_REF] Plutchik | A general psychoevolutionary theory of emotion[END_REF], are often used in the literature to represent emotions categorically.

This categorical representation is used in many affective movie content analysis work but faces a granularity issue since the number of emotion classes is too small in comparison with the diversity of emotions perceived by film viewers. For example, the list of the six basic emotions introduced by Ekman is criticized because it contains only one positive emotion [START_REF] Fredrickson | What good are positive emotions?[END_REF]. If the number of classes is increased, ambiguities due to language difficulties or personal interpretation appear.

Dimensional

Dimensional approaches have also been proposed to model emotions as points in a continuous n-dimensional space. The most famous one is the valence-arousal-dominance space, also known as the pleasure-arousal-dominance (PAD) space, introduced by Russell and Mehrabian [START_REF] Russell | Evidence for a three-factor theory of emotions[END_REF] and extensively used in research dealing with affective understanding. In this space, each subjective feeling can be described by its position in a three-dimensional space formed by the dimensions of valence, arousal, and dominance. Valence ranges from negative (e.g., sad, disappointed) to positive (e.g., joyous, elated), whereas arousal can range from inactive (e.g., tired, pensive) to active (e.g., alarmed, angry), and dominance ranges from dominated (e.g., bored, sad) to in control (e.g., excited, delighted). Given the difficulty of consistently identifying a third dimension (such as dominance, tension or potency) which differs from arousal, many studies, including this work, limit themselves to the valence and arousal (VA) dimensions. Indeed, especially when dealing with emotions elicited by videos, valence and arousal account for most of the independent variance [START_REF] Greenwald | Affective judgment and psychophysiological response: Dimensional covariation in the evaluation of pictorial stimuli[END_REF], [START_REF] Lang | International affective picture system (IAPS): Technical manual and affective ratings[END_REF]. Moreover, psychophysiological experiments have revealed that only certain areas of this two-dimensional space are relevant [START_REF] Dietz | Aefective agents: Effects of agent affect on arousal, attention, liking and learning[END_REF], [START_REF] Kron | Are valence and arousal separable in emotional experience?[END_REF] and that emotions induced by media can be mapped onto a parabolic space created by the arousal and valence axes.

However, this common two-dimensional space is questioned. Fontaine et al. demonstrated that two dimensions are not sufficient to represent emotions satisfactorily [START_REF] Fontaine | The world of emotions is not two-dimensional[END_REF]. They showed that use of at least four dimensions is more appropriate to represent the diversity of emotions (valence, arousal, dominance, and predictability) but that the optimal number of dimensions to be included in a model depends on the purpose of the model. More recently, Joffily and Coricelli [START_REF] Joffily | Emotional valence and the free-energy principle[END_REF] proposed a formal definition of emotional valence in terms of the negative rate of change of freeenergy over time. They also investigated the relationship between the dynamics of free-energy and some basic forms of emotion such as happiness, unhappiness, hope, fear, relief, disappointment, and surprise. Thus, they claim that freeenergy should be considered in order to create a biologically plausible computational model of emotional valence

COMPUTATIONAL MODELS AND DATASETS FOR AFFECTIVE VIDEO CONTENT ANALYSIS

Affective video content analysis aims at automatically predicting the emotions elicited by videos. Work on affective video analysis can be categorized into two subgroups: continuous 4 affective video content analysis, which estimates a time-dependent affective score for consecutive portions (e.g. each frame or group of frames) of a video, and global 5 affective video content analysis, which assigns an affective score to an entire video. Some work represents emotions in the 2D valence-arousal space or in the 3D valence-arousaldominance space, while other work represents emotions using discrete categories. Furthermore, the models are sometimes dedicated to specific video categories, i.e. music videos or a particular movie genre.

As stated in the introduction, studies on emotion assessment using physiological signals beyond audiovisual (AV) features, i.e., implicit video affective content analysis, are not presented in this paper so as to focus on affective video content analysis work.

Affective video content analysis: the perspectives

The video investigated with respect to its affective content can be either the stimulus eliciting emotions (known as "affective movie content analysis") or a tool for investigating the emotions expressed by agents (i.e., "video emotion recognition"). While the focus of this paper is affective movie content analysis, research in both affective video content 4. The term continuous is used in this paper to refer to time-dependent annotations and should not be confused with the dimensional annotations referring to the dimensional representation of emotions.

5. Global works are also commonly referred to as discrete affective movie content analysis. However, the term global was preferred in this paper because the term discrete also refers to the categorical representation of emotions. analysis fields will be presented in this section as they are closely related, with emphasis on previous affective movie content analysis work.

Video emotion recognition work aims at automatically estimating the emotion expressed by an agent from a video recording, in relation to an emotion induced by a stimulus. Sometimes, these emotions are not real but played by an actor [START_REF] Busso | Analysis of emotion recognition using facial expressions, speech and multimodal information[END_REF]. The main goal of video emotion recognition models is to enable affective interaction between human beings and computers. Such models are inherently different from affective movie content analysis models. However, temporal modeling of the emotions of such models may be useful for designing new continuous affective movie content analysis frameworks.

Affective movie content analysis work focuses on the video, which is the stimulus of the emotion to be investigated. The emotion to be investigated, related to the intended, induced and expected emotion defined in the introduction, is defined by the perspective of the models and their applications.

Due to the exciting new possibilities offered by such affective computing techniques, they can be naturally applied to help standard multimedia systems [START_REF] Picard | Affective computing: From laughter to ieee[END_REF]. Thus, affective movie content analysis work has a large number of applications, including mood based personalized content recommendation [START_REF] Canini | Affective recommendation of movies based on selected connotative features[END_REF] or video indexing [START_REF] Zhang | Affective visualization and retrieval for music video[END_REF], and efficient movie visualization and browsing [START_REF] Zhao | Flexible presentation of videos based on affective content analysis[END_REF]. Beyond the analysis of existing video material, affective computing techniques can also be used to generate new content, e.g., movie summarization [START_REF] Katti | Affective video summarization and story board generation using pupillary dilation and eye gaze[END_REF], or personalized soundtrack recommendation to make user-generated videos more attractive [START_REF] Shah | Advisor: Personalized video soundtrack recommendation by late fusion with heuristic rankings[END_REF]. Affective techniques have even been used to enhance user engagement with advertising content by optimizing the way ads are inserted inside videos [START_REF] Yadati | Cavva: Computational affective video-in-video advertising[END_REF].

Continuous affective video content analysis

This section presents previous work on continuous affective video content analysis, including movie content analysis and video emotion recognition. To predict or classify emotions, previous work either directly combines linearly audiovisual (AV) features extracted from the data, or uses machine learning models. Temporal information can be included in the machine learning models, for example using Long Short-Term Memory neural networks, or by simply applying a temporal smoothing to the predicted values. A summary is given in Table 1.

Continuous valence and arousal movie content analysis

Hanjalic and Xu pioneered in [START_REF] Hanjalic | Affective video content representation and modeling[END_REF] analysis of affective movie content by directly mapping video features onto the valencearousal space to create continuous representations. Based on film theorists' work, they selected low-level features that are known to be related to arousal or valence, such as motion intensity, shot lengths, and audio features (loudness, speech rate, rhythm, etc.). They manually designed the functions modeling arousal and valence for consecutive frames based on the selected features and used a Kaiser window to temporally smooth the resulting curves. However, they only offered a qualitative evaluation of their model. [START_REF] Soleymani | A bayesian framework for video affective representation[END_REF] using audiovisual features and textual features extracted from subtitles but also taking into account contextual information (e.g. user's personal profile, gender or age). However, as their ground truth was annotated by a single participant only, they did not study the relevance of such contextual information and assumed the model to be personalized for this participant. Arousal estimation, obtained for each shot by means of a Relevance Vector Machine (RVM), is then used as an arousal indicator feature and merged with other content-based features for global scene affective classification thanks to a Bayesian framework. Thus, their framework is a trade-off between continuous and global affective video content analysis. The Bayesian framework relies on two priors: the movie genre prior and the temporal dimension prior consisting of the probability transition between emotions in consecutive scenes. However, movie scenes are categorized into three emotional classes, which is too restrictive. Furthermore, they only provided a qualitative evaluation of the continuous arousal estimation but achieved an accuracy of 63.9% for classification of movie scenes among three emotional classes. [START_REF] Malandrakis | A supervised approach to movie emotion tracking[END_REF] a continuous affective video content analysis relying on audiovisual features extracted on each video frame, combined at an early stage and used by two Hidden Markov Models (HMMs). These two classifiers are trained independently to model simultaneously the intended arousal and valence. However, HMMs predict discrete labels. Arousal and valence are thus discretized into seven categories and the model only allows transitions between adjacent categories. Finally, they output time series composed of seven categories interpolated into a continuous-valued curve via spline interpolation. However, the continuous curves are thus approximations and cannot recover the precision lost by discretizing the affective space. Their discrete and continuous curves are compared using the leave-one-movie-out approach to the ground truth collected on 30-min video clips from 12 movies. The smoothed predicted curves achieved an average correlation of 0.54 for arousal and 0.23 for valence.

Malandrakis et al. also proposed in

Video emotion recognition

As stated in Section 3.1, video emotion recognition models are inherently different from affective movie content analysis models. While affective movie content analysis models aim at estimating the intended, expected or induced emotion from the intrinsic properties of a movie, video emotion recognition models aim at automatically estimating the emotion expressed by an agent from a video recording, in relation to an emotion induced by a stimulus. Emotion recognition models thus typically rely on facial characteristics such as action units (AU) [START_REF] Cowie | Emotion recognition in humancomputer interaction[END_REF]. Since felt and expressed emotions are linked, the temporal modeling of emotions of emotion recognition models may be of interest for designing new continuous affective movie content analysis frameworks. This is especially true since, as mentioned in Section 2, psychologists suggest that the evaluation of an emotion is a recursive and continuous process.

Nicolaou et al. introduced in [63] a framework for continuous prediction of spontaneous affect in the VA space based on facial expression, shoulder gestures, and audio cues. They compared the performance of the bidirectional Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) and Support Vector Machines for Regression (SVR) for continuous spontaneous affect prediction and proposed an output-associative prediction framework. This framework is a modified bidirectional LSTM-RNN taking into account the correlation between the predicted valence and arousal dimensions: it depends on the entire sequence of intermediate output predictions of both dimensions to perform the prediction. They showed that the bidirectional LSTM-RNN outperforms the SVR and that the output-associative prediction framework significantly improves prediction performance. However, to train and evaluate their models, Nicolaou et al. used recordings made in a lab setting, using a uniform background and constant lighting conditions. Their model is thus highly sensitive to recording conditions, such as illumination and occlusions.

More recently, Kahou et al. [START_REF] Kahou | Combining modality specific deep neural networks for emotion recognition in video[END_REF] designed a framework to assign one of seven acted-out emotions to very short video clips (1 to 2 seconds long) extracted from Hollywood movies. It was the winning submission in the 2013 Emotion Recognition in the Wild Challenge [START_REF] Dhall | Emotion recognition in the wild challenge (emotiw) challenge and workshop summary[END_REF]. Unlike [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF], videos provided for the challenge depict acted-out emotions under realistic conditions (large degree of variation in attributes such as pose and illumination). Their framework combines a Convolutional Neural Network (CNN) focusing on capturing visual information in detected faces, a Deep Belief Network for representation of the audio stream, a K-Means based model for extracting visual features around the mouth region, and a relational autoencoder to take into account the spatiotemporal aspects of videos. To efficiently train the CNN, they downloaded two large alternative image databases of facial expressions for seven emotion categories. The final result is a concatenation of models based on a single modality. However, due to the characteristics of the data provided for the challenge, emotional relationships between consecutive video segments are not investigated.

Global affective movie content analysis

This section focuses on previous work on global affective video content analysis, including valence and arousal estimation but also violence detection. Global work assigns an affective score to an entire video. A summary is given in Table 2.

Global valence and arousal movie content analysis

Global affective video content analysis has been more frequently investigated than continuous affective video content analysis over the last decade.

Kang [START_REF] Kang | Affective content detection using HMMs[END_REF] was the first to propose a model where classifiers are adopted for affective analysis. He suggested detecting affective states in movies including "sadness", "joy" and "fear" from low-level features using HMMs. HMM topology is designed such that the only possible transitions between consecutive affective states are those between the neutral state and the other affective states. However, this topology is very restrictive and is unrealistic.

Wang and Cheong introduced features inspired from psychology and film-making rules [START_REF] Wang | Affective understanding in film[END_REF]. One Support Vector Machine (SVM) is especially dedicated to audio cues to obtain high-level audio information at scene level. Each video segment is then classified with a second SVM to obtain probabilistic membership vectors for seven discrete emotional states. Their training data are made up of 36 full-length popular Hollywood movies divided into 2,040 scenes labeled with one or two emotional states by only three annotators. Due to the limited number of annotators, ambiguities arise and make it necessary to assign two labels to some videos. Furthermore, they do not take into account the relations between consecutive scenes.

In the work of Sun and Yu [START_REF] Sun | Video affective content representation and recognition using video affective tree and hidden markov models[END_REF], movie units are first represented in different granularities using an excitement curve based on the arousal curve introduced in [START_REF] Hanjalic | Affective video content representation and modeling[END_REF]. Then, four HMMs are trained independently using features extracted on these granularities to recognize one of the four emotional states among "joy", "anger", "sadness" and "fear". Each HMM has the same topology and is composed of two states: a neutral state, and a state representing the emotion assigned to the HMM. Thus, each HMM only computes for a given observation sequence, the state transition probabilities between the neutral state and one of the four emotional states. As in [START_REF] Kang | Affective content detection using HMMs[END_REF], this topology is restrictive and unrealistic. Their ground truth consists of 10 movies labeled at different levels by 30 annotators. Xu et al. used a similar approach in [START_REF] Xu | Hierarchical movie affective content analysis based on arousal and valence features[END_REF] sharing the same disadvantages. However, to compute the emotion intensity level they used fuzzy clustering instead of linearly combining audiovisual features. It is a fact that fuzzy clustering is closer to human perception. Five HMMs were then trained using emotion intensity and low-level features to model five emotional classes with different levels of valence. They evaluated the efficiency of their method for several movie genres, where the highest accuracy was obtained for action movies.

Soleymani et al. [START_REF] Soleymani | Affective characterization of movie scenes based on multimedia content analysis and user's physiological emotional responses[END_REF] compared in the VA space the values obtained automatically from either physiological responses or from audiovisual features. They showed significant correlations between multimedia features, physiological features and spectators' self-assessments for both valence and arousal. Affective scores are estimated by a linear combination of content-based features and compared to the estimation of affective scores using a linear combination of physiological features only. They showed that the performance of both models is the same and thus that none has a significant advantage over the other. A dataset composed of 64 movie scenes extracted from 8 Hollywood movies was created, from which 42 movie scenes were randomly selected for the training set, while the remaining ones were used in the test set. Movie scenes extracted from the same movie can thus be part of the training set and also of the test set, questioning the reliability of the results.

Zhang et al. developed in [START_REF] Zhang | Affective visualization and retrieval for music video[END_REF] a personalized affective analysis for music videos composed of SVR-based arousal and valence models using both multimedia features and user profiles. In fact, two nonlinear SVRs are learned for each user, taking into account the underlying relationships between user's affective descriptions and the extracted features. However, SVRs are not able to model the temporal transition characteristics of emotions. Their dataset of 552 music videos is used to train and update the models based on user feedback.

Irie et al. [START_REF] Irie | Affective audio-visual words and latent topic driving model for realizing movie affective scene classification[END_REF] proposed an approach based on Latent Dirichlet Allocation (LDA) considering the temporal transition characteristics of emotions. Emotion-category-specific audiovisual features are extracted and transformed into affective audio-visual words using k-mean clustering. These higher level features are then used to classify movie scenes using a latent topic driving model. The model considers the probability of emotional changes between consecutive scenes based on the Plutchik's wheel [START_REF] Plutchik | A general psychoevolutionary theory of emotion[END_REF]. However, these probabilities have not been estimated from a real dataset but empirically defined by the authors. The rate of agreement of their model is equal to 85.5%. The good results obtained by their framework may be due to the evaluation protocol. Their data, composed of 206 scenes from 24 movie titles available as DVDs, were randomly selected to form the training and test sets. Consequently, most films appear both in the training and in the test sets as in [START_REF] Soleymani | Affective characterization of movie scenes based on multimedia content analysis and user's physiological emotional responses[END_REF], which biases the results.

More recently, Acar et al. proposed the use of CNNs in order to learn mid-level representations for affective classification of music video clips [START_REF] Acar | Understanding affective content of music videos through learned representations[END_REF]. Two CNNs are learned to output mid-level representation of 5-second music video clips: one uses as input audio features (MFCC), while the other uses as input one color channel, i.e., red, green or blue color channel, of the resized frame in the middle of the video segment. The mid-level representations are then each used in a dedicated SVM, and their predictions fed to a final multi-class audiovisual SVM to output the category of the video clip (one of the four quadrants of the valence-arousal space). Their framework achieves an accuracy of 52.63% on the DEAP dataset [START_REF] Koelstra | DEAP: a database for emotion analysis using physiological signals[END_REF].

Violence detection

Emotion detection is closely related to violence detection. Both works presented in this section aim at detecting whether a video shot contains violence, defined as a physical action that results in human injury or pain. Penet et al. introduced a framework using both multimodal and temporal information for violence detection [START_REF] Penet | Multimodal information fusion and temporal integration for violence detection in movies[END_REF]. For temporal integration, their model uses the features extracted from the investigated video excerpts but also contextual features extracted from the five previous and next video shots. However, it is not clear why they preferred fixedlength time windows computing features for 10 consecutive video shots. The audio features (audio energy, audio zero crossing rate, etc.) , and video features (shot duration, number of flashes, etc.) are separately used in two independent Bayesian networks. The two probabilities given by the networks are finally fused using a late fusion approach.

One year later, to classify video excerpts as violent or non-violent, Eyben et al. fused by simple score averaging the predictions made by acoustic and visual linear kernel SVMs [START_REF] Eyben | Affective video retrieval: Violence detection in hollywood movies by large-scale segmental feature extraction[END_REF]. To capture temporal dynamics, numerous statistics are computed for the features over fixed size windows. The authors also provided a detailed analysis of the features extracted from the audio and video modalities. They showed that some features are particularly relevant for violence detection: color and optical flow for the video modality, and spectral distribution descriptors and peak-based functional extraction for the audio channel.

While using short fixed-length time windows may be relevant for violence detection, estimating induced emotions requires consideration of longer-term dependencies [START_REF] Öllmer | Abandoning emotion classes -towards continuous emotion recognition with modelling of long-range dependencies[END_REF].

Affective video datasets

Main affective video datasets and their limitations have already been reviewed in recent papers [START_REF] Soleymani | Corpus development for affective video indexing[END_REF], [START_REF] Baveye | LIRIS-ACCEDE: A video database for affective content analysis[END_REF]. They outlined that main public affective video datasets, namely the HUMAINE [START_REF] Douglas-Cowie | The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data[END_REF], FilmStim [START_REF] Schaefer | Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers[END_REF], DEAP [START_REF] Koelstra | DEAP: a database for emotion analysis using physiological signals[END_REF], MAHNOB-HCI [START_REF] Soleymani | A multimodal database for affect recognition and implicit tagging[END_REF], EMDB [START_REF] Carvalho | The emotional movie database (EMDB): a self-report and psychophysiological study[END_REF], and VSD [START_REF] Demarty | A benchmarking campaign for the multimodal detection of violent scenes in movies[END_REF] datasets, suffer from limitations preventing their use in training and evaluating computational models in the field of affective content analysis. Indeed, these datasets are either of limited size and content diversity, suffer from copyright issues, or have different emotional labels that are sometimes not representative of the whole range of emotions in movies. However, some attempts are starting to emerge to solve these limitations.

Recently, the LIRIS-ACCEDE dataset has been released publiclyin order to overcome the limitations of the existing affective video datasets and to foster research in affective video content analysis [START_REF] Baveye | LIRIS-ACCEDE: A video database for affective content analysis[END_REF]. This dataset has been used in particular in the MediaEval benchmarking initiative to compare computational models estimating the emotions elicited by movies [START_REF] Sj Öberg | The mediaeval 2015 affective impact of movies task[END_REF]. Based on the FilmStim dataset [START_REF] Schaefer | Assessing the effectiveness of a large database of emotion-eliciting films: A new tool for emotion researchers[END_REF], Guntuku et al. proposed the CP-QAE-I dataset to explore the effect of quality of experience, personality and culture on the felt emotions [START_REF] Guntuku | The cpqae-i: A video dataset for exploring the effect of personality and culture on perceived quality and affect in multimedia[END_REF]. This dataset is composed of 144 videos, with various bit-rate, frame dimension, and frame rate, derived from 12 original video excerpts. Although limited in size and content diversity, the interest of this dataset lies in the affective self-assessments collected by participants with different cultural backgrounds. It is also interesting to mention the Audio/Visual Emotion Challenge and Workshop (AVEC) [START_REF] Valstar | Avec 2014: 3d dimensional affect and depression recognition challenge[END_REF], and the Emotion Recognition In The Wild Challenge (EmotiW) workshop [START_REF] Dhall | Emotion recognition in the wild challenge (emotiw) challenge and workshop summary[END_REF], both consisting of recognition of the emotions expressed by actors in laboratory environments or in movies using large and public datasets. Such benchmarking campaigns are the key to building more reliable computational models in the future.

Issues

Due to the constraints on databases presented in Section 3.4, most state of the art work about affective movie content analysis uses a private dataset of a very limited size and content diversity, designed according to their goals and needs (see Tables 1 and2). Thus, it makes fair comparisons and results reproducibility impossible, preventing achievement of major strides in the field. For example, some work represents emotions in the 2D VA space or in the 3D valence-arousaldominance space [START_REF] Arifin | Affective level video segmentation by utilizing the pleasure-arousal-dominance information[END_REF], [START_REF] Zhang | Affective visualization and retrieval for music video[END_REF], while other work represents emotions using discrete categories [START_REF] Wang | Affective understanding in film[END_REF]. Furthermore, the models are sometimes dedicated to specific video categories, i.e., music videos [START_REF] Zhang | Affective visualization and retrieval for music video[END_REF], [START_REF] Acar | Understanding affective content of music videos through learned representations[END_REF] or to a particular movie genre [START_REF] Xu | Affective content analysis in comedy and horror videos by audio emotional event detection[END_REF]. It is a fact that choice of representation of emotions is highly dependent on the model goal. For example, if a model targets end-users, natural categorical representation of emotions may be preferred over dimensional representation since the former is simpler to comprehend. On the contrary, continuous representation of emotions may be preferred, for example, for work modeling the transitions between consecutive emotions. Benchmarking previous work is also limited because affective movie content analysis work can be identified either as continuous or global approaches. The choice of one approach over the other is also biased by the model goal. For example, the global approach may be chosen for models dealing with video indexing considering a movie as a whole, while the continuous approach may be preferred for models requiring a fine-grained temporal resolution (e.g., efficient movie visualization and browsing).

The difficulty in collecting reliable affective ground truth is the last issue discussed in this section. The term "ground truth" should be taken with a pinch of salt in affective video analysis work. Indeed, the affective self-assessments collected in previous work dealing with emotions induced by video content are intrinsically biased. This is because they represent the interpretation of the emotional experience that the annotator is currently feeling, which may be different from the emotion felt by the annotator (i.e., the induced emotion). Recording the facial expression of the annotators and collecting physiological signals, such as skin conductance, heart rate, or even electroencephalogram signals, could help to improve the reliability of affective self-assessments. However, the correlation between such modalities and felt emotions is still ongoing [START_REF] Soleymani | Analysis of eeg signals and facial expressions for continuous emotion detection[END_REF] and there is no direct evidence that any particular facial movement provides an unambiguous expression of a specific emotion [START_REF] Parkinson | Do facial movements express emotions or communicate motives?[END_REF]. The continuous ground truth used in previous work (such as [START_REF] Malandrakis | A supervised approach to movie emotion tracking[END_REF], [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF]) is also biased because of the post-processing steps for taking into account the annotator-specific delays amongst the annotations, and because it is the result of the aggregation of the multiple annotators' self-assessments [START_REF] Metallinou | Annotation and processing of continuous emotional attributes: Challenges and opportunities[END_REF], [START_REF] Mariooryad | Analysis and compensation of the reaction lag of evaluators in continuous emotional annotations[END_REF], [START_REF] Nicolaou | Dynamic probabilistic cca for analysis of affective behavior and fusion of continuous annotations[END_REF]. To tackle these issues, future work should start investigating individual emotional differences in order to design personalized models, as presented in the next section. To summarize, the "ground truth" used in most affective video analysis work, does not represent the emotions that an annotator has felt during an experiment, but rather represents the emotions that most annotators say they have experienced while watching movies during an experiment. Thus, these values correspond to the interpretation of the expected emotions. Such feedback, requiring a true introspection, have to be carefully considered [START_REF] Nisbett | Telling more than we can know: Verbal reports on mental processes[END_REF].

BRINGING PSYCHOLOGICAL THEORIES IN COM-PUTATIONAL MODELS

The issues listed in the previous section are closely related to the machine learning-based architectures used to build the computational models estimating the emotions expected or induced by movies. However, psychological theories and computational models rely very little on each other. A Human-Computer Interaction (HCI) system would require implementation of psychological models while considering a large variety of interactions. However, computational models estimating the emotions elicited by movies form a special case where the stimulus is a video scene. Thus, only specific sections of the psychological models could be first investigated in order to maximize the performance of computational models. In this section, we share some insights for integrating ideas expressed in Section 2 by stating that future affective video content analysis work should:

• Integrate the recursive aspect of emotions.

• Emulate bottom-up and top-down processes.

• Predict intermediate dimensions.

•

Personalize prediction.

We hope these insights will help build models closer to human perception and thus allow emotions to be modeled with greater reliability and fidelity. Furthermore, some insights are general ideas and are thus also relevant for computational models over and beyond affective movie content analysis.

First, psychologists suggest that evaluation of an emotion is an iterative process. For example, Russell defines the Core Affect as [START_REF] Russell | Core affect and the psychological construction of emotion[END_REF]:

"Emotional life consists of the continuous fluctuations in core affect, in pervasive perception of affective qualities, and in the frequent attribution of core affect to a single Object, all interacting with perceptual, cognitive, and behavior processes." This process of recursive and continuous evaluations is also the core of the appraisal evaluation [START_REF] Scherer | The component process model: Architecture for a comprehensive computational model of emergent emotion[END_REF] and of aesthetic emotions [START_REF] Leder | A model of aesthetic appreciation and aesthetic judgments[END_REF]. This is particularly important when dealing with movies because the emotions induced by such types of stimulus are closely linked to the narrative content, and thus to the temporal sequence of events within movies [START_REF] Marković | Components of aesthetic experience: aesthetic fascination, aesthetic appraisal, and aesthetic emotion[END_REF]. Furthermore, as stated in Section 2.3.2, the narrative content tends to follow typical rules, which could be learned by a recursive model taking into account the sequence of events. Thus, this key aspect of emotions has to be considered in affective computational models. However, most computational models do not take into account the recursivity of the emotional episode. This is not the case for HMM-based [START_REF] Kang | Affective content detection using HMMs[END_REF], [START_REF] Sun | Video affective content representation and recognition using video affective tree and hidden markov models[END_REF], [START_REF] Xu | Hierarchical movie affective content analysis based on arousal and valence features[END_REF] and RNN-based [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF] affective frameworks. Indeed, HMMs are statistical models of sequential data, inherently able to take into consideration consecutive emotional changes through hidden state transitions. However, they are composed of a specific number of discrete states and thus cannot be used to directly infer dimensional scores. Malandrakis et al. converted discrete affective curves obtained with HMMs into continuous curves using a Savitzky-Golay filter [START_REF] Malandrakis | A supervised approach to movie emotion tracking[END_REF]. However, the continuous curves are thus approximations and cannot recover the precision lost by discretizing the affective space. RNN-based affective frameworks are also able to take into account temporal transitions between consecutive emotions. Combined with LSTM cells, they can efficiently learn longterm dependencies, as shown by previous work on video emotion recognition [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF], [START_REF] Baveye | Automatic prediction of emotions induced by movies[END_REF].

Beyond the recursive aspect of the models, their structures themselves can be inspired from neuroscience and psychological work. Indeed, psychological theories describe emotions as generated through the interaction of bottom-up and top-down processes (but with emphasis on the bottomup process) [START_REF] Scherer | The component process model: Architecture for a comprehensive computational model of emergent emotion[END_REF], [START_REF] Leder | A model of aesthetic appreciation and aesthetic judgments[END_REF]. This is confirmed by neuroscience work demonstrating that both types of response activate the amygdala (bottom-up responses activating the amygdala more strongly) [START_REF] Ochsner | Bottom-up and top-down processes in emotion generation common and distinct neural mechanisms[END_REF]. Bottom-up processes correspond to the elicitation of emotion in response to inherently emotional perceptual properties of the stimulus. On the other hand, top-down processes are cognitive evaluations occurring, for example, during the relevance to goals and needs SEC defined in appraisal models [START_REF] Scherer | Appraisal considered as a process of multi-level sequential checking[END_REF]. Bottom-up emotions help the subject to respond quickly and accurately to emotion-relevant aspects of the environment, whereas top-down emotions help achieve greater flexibility in producing these emotional responses [START_REF] Mcrae | Bottom-up and top-down emotion generation: implications for emotion regulation[END_REF]. Such descriptions of the interaction between bottom-up and top-down responses can help future affective movie content analysis work to create computational models with appropriate structures as in other fields of study [START_REF] Itti | Models of bottom-up and top-down visual attention[END_REF]. For example, saliency models predicting where people are looking in pictures have greatly benefited from the integration of such structures by considering neuroanatomical and psychological work [START_REF] Borji | Boosting bottom-up and top-down visual features for saliency estimation[END_REF].

Psychological theories have also shown that several intermediate dimensions are important characteristics of the emotional experience. For example, Scherer has postulated that dimensions such as novelty (including suddenness, familiarity and predictability), as well as pleasantness, are important factors in the appraisal module for determining the characteristics of the emotional episode and are linked to attention and memory allocation [START_REF] Scherer | Emotions are emergent processes: they require a dynamic computational architecture[END_REF]. Thus, predicting intermediate dimensions, such as predictability or novelty, may be of interest to help estimate valence and arousal elicited by movies. Recently, first attempts have been proposed in order to predict the memorability of images [START_REF] Isola | What makes a photograph memorable?[END_REF]. Other high-level intermediate factors can be inspired by Wiley [START_REF] Wiley | Emotion and film theory[END_REF] using a sociological perspective, introduced in Section 2.3.2. In particular, information obtained from movie scripts, such as relations among movie characters [START_REF] Ding | Learning relations among movie characters: A social network perspective[END_REF] as well as their personalities [START_REF] Bamman | Learning latent personas of film characters[END_REF], could improve future affective movie content analysis work.

Predicting the emotions that most people feel while watching movies (i.e., expected emotions) is extremely difficult to solve with a universal solution. This is because emotional experience is a highly subjective process. Conse-quently, personalized solutions should also be considered to predict induced emotions (i.e., emotions that a specific viewer feels while watching a movie). This will be possible by collecting self-assessments across highly diverse viewer groups with different cultural and socio-demographical backgrounds. Personalized video affective models would be useful for applications such as personalized content delivery, while general video affective models predicting intended or expected emotions would be applied for video indexing or as a support to help create new video content. This implies the need for different types of ground truth depending on the model goal: annotations from video creators are necessary for evaluating predictions of intended emotion, annotations from individual viewers are necessary for evaluating predictions of induced emotion, and annotations from multiple viewers are necessary for evaluating predictions of expected emotion. Nowadays, most work only considers expected emotions and computes the ground truth as the mean of the individual affective self-assessments. Subjective factors such as previous experience, interest and personal taste play an important role in the experience of an emotion [START_REF] Leder | A model of aesthetic appreciation and aesthetic judgments[END_REF]. Future personalized solutions aiming to predict the induced emotions of a given user could thus use basic characteristics as additional features (e.g., age, gender, nationality), but also the results of personality tests previously filled in by users (e.g., measurements of the Big Five personality traits), or the list of the last played movies to take into account previous experience and interest. The environment (i.e., the contextual information) is also a key element for reliably predicting induced emotions. Both the background environment and the social environment influence emotions [START_REF] Russell | Emotion and the environment[END_REF], including for example co-viewers' laughs [START_REF] Wiley | Emotion and film theory[END_REF] or group pressures [START_REF] Scherer | Culture-specific appraisal biases contribute to emotion dispositions[END_REF]. Despite the importance of such information in predicting induced emotions, currently no affective movie content analysis work analyzes the influence of personalized features.

Previous attempts have been proposed to personalize computational models. As mentioned in Section 3.2.1, Soleymani et al. introduced a personalized framework for video affective representation [START_REF] Soleymani | A bayesian framework for video affective representation[END_REF] taking into account contextual and personal information. Nevertheless, reliability of the personalization priors was not evaluated due to the difficulty of collecting large samples of individual affective self-assessments. More recently, affective datasets mentioned in Section 3.4, aiming at exploring the effects of personality and culture , are starting to emerge, but are still of limited size and content diversity. However, it is of great interest to start analyzing the effect of personality and culture on the perception of video content quality and affect. First results confirm that personality and culture play an important role and have to be taken into account in future computational models [START_REF] Guntuku | Modelling the influence of personality and culture on affect and enjoyment in multimedia[END_REF], [START_REF] Scott | Modelling human factors in perceptual multimedia quality: On the role of personality and culture[END_REF]. Bigger affective datasets, composed of self-assessments across highly diverse viewer groups with different cultural and socio-demographical backgrounds, are needed to generalize and confirm these findings. These would pave the way for the next generation of personalized computational models of induced emotions.

CONCLUSION

In this paper, we gave an overview of emotion theories and emotion representations, as well as computational models for affective video content analysis.

A review of previous work developing computational models to infer emotions showed that they rely on a predefined set of highly problem-dependent handcrafted features requiring strong domain knowledge and that they use private datasets to evaluate their performance, thus making fair comparisons and result reproducibility impossible, and preventing achievement of major strides in the field. Computational models trained using private datasets are often not generalizable and fail when applied to new videos due to the limited size and content diversity of the datasets used for training. Bigger datasets and experimental evaluations on new contents are thus needed to create more robust and generalizable computational models.

Furthermore, previous affective multimedia content analysis work should be designed to be closer to psychological models. In particular, they should model emotions recursively using intermediate dimensions such as predictability or novelty, structured using insights from the neuroscience and vision science fields. Finally, to reliably predict induced emotions, future computational models will need to be personalized in order to better model the individual differences among viewers and to take into account the environment, which plays a major role during emotional processes. Without such personalized solutions, future computational models will fail to predict induced emotions reliably.

This concluding discussion shows that future tasks in this research field are challenging and need to be addressed by close collaboration of experts in psychology, sociology, neuroscience, vision science and image processing.

Fig. 1 .

 1 Fig. 1. Architecture of Scherer's Component Process Model of emotion (originally published in [24])

TABLE 1

 1 Summary of previous work on continuous affective movie content analysis and video emotion recognition

	Authors Method	Output			Ground truth	Annotators Result
	Hanjalic and	No classification: AV fea-	Continuous	induced	Unknown number of	None	Qualitative evalua-
	Xu [61]	tures are linearly combined	arousal	and	valence	movie scenes and soccer		tion only
			functions at a frame level	television broadcasts
	Soleymani	Bayesian framework using	Videos categorized into	21 full-length popular	1	Accuracy: 64%, F1
	et al. [62]	AV and textual features, re-	three induced emotional	movies		measurement: 63%
		lying on temporal priors	classes
	Malandrakis	Two HMMs using AV fea-	Time series of 7 categories	30-min video clips from 12	7	Correlation	for
	et al. [6]	tures at frame level	interpolated into continu-	movies		arousal: 0.54, and
			ous intended VA curves			valence: 0.23
	Nicolaou et	LSTM-RNN-based frame-	Continuous recognition of	10 hours of footage from	4	Correlation	for
	al. [63]	work using facial expres-	emotions expressed by ac-	the SAL database capturing		arousal: 0.642,
		sion, shoulder gesture, and	tors in terms of arousal and	the interaction between a		valence: 0.796
		audio cues	valence			human and an operator
	Kahou et	Combination of multiple	Emotions expressed by the	Short video clips extracted	2	Accuracy: 41.03%
	al. [64]	deep neural networks for	main actor in a video	from movies, provided for
		different data modalities	among 7 emotional classes	EmotiW 2013
	Soleymani et al. introduced a Bayesian framework for
	video affective representation

TABLE 2

 2 Summary of previous work on global affective movie content analysis and violence detection

	Authors Method	Output	Ground truth	Annotators Result
	Kang [67] HMMs relying on visual	Discrete labels among 3	Scenes extracted from six	10	Accuracy: ≈79%
		features only	classes (fear, sadness, joy)	30min videos
	Wang and	Two SVMs using audio	Video scenes classified us-	2,040 scenes from 36 full-	3	Accuracy: 74.69%
	Cheong [68]	cues or AV features	ing 7 emotional classes	length popular movies
	Sun and	4 HMMs using AV features Labels among 4 classes	10 popular movies labeled	30	Precision: ≈68%,
	Yu [69]		(anger, fear, sadness, joy)	at different levels		Recall: ≈79%
	Xu et al. [70] 5 HMMs using AV cues	5 affective discrete classes	Videos from 24 movies	?	Accuracy: 80.7%
	Soleymani et	Personalized RVM using	1 global arousal and va-	64 movie scenes from 8	8	MSEs for each par-
	al. [71]	AV or physiological cues	lence score per video	popular movies		ticipant
	Zhang et	2 SVRs using both VA fea-	One valence and arousal	552 representative music	10 & 27	Performances for
	al. [56]	tures and user profile	score per music video	videos		two applications
	Irie et al. [72] Latent topic driving model	Probabilities for 9 emotion	206 scenes from 24 movie	16	Agreement: 85.5%
		using VA features	categories for each scene	titles available as DVDs
	Acar et	Two CNNs and one SVM	1 of the 4 quadrants of the	Music videos from the	32	Accuracy: 52.63%
	al. [73]	using AV features	VA space for each video	DEAP dataset
	Penet et	2 Bayesian networks using	Violence probability for	MediaEval 2011 Affect	7	False alarms and
	al. [74]	temporal AV features	each video shot	Task corpus		missed curves
	Eyben et	SVMs using temporal AV	Confidence score to classify	MediaEval 2012 Affect	7	MAP@100: 0.398
	al. [75]	features	a video shot as violent	Task corpus

Big Buck Bunny, licensed under the Creative Commons Attribution 3.0 license, (c) copyright 2008, Blender Foundation, www.bigbuckbunny.org

ACKNOWLEDGMENTS

This work was supported in part by the French Research Agency ANR through the VideoSense project under the grant 2009 CORD 026 02 and the Visen project within the CHIST-ERA program under the grant ANR-12-CHRI-0002-04.