Alexandre Bruckert
email: alexandre.bruckert@irisa.fr

Marc Christie

Olivier Le Meur

Where to look at the movies : Analyzing visual attention to understand movie editing

In the process of making a movie, directors constantly care about where the spectator will look on the screen. Shot composition, framing, camera movements or editing are tools commonly used to direct attention. In order to provide a quantitative analysis of the relationship between those tools and gaze patterns, we propose a new eye-tracking database, containing gaze pattern information on movie sequences, as well as editing annotations, and we show how state-of-the-art computational saliency techniques behave on this dataset. In this work, we expose strong links between movie editing and spectators scanpaths, and open several leads on how the knowledge of editing information could improve human visual attention modeling for cinematic content.

Introduction

In order to deal with the incredibly large amount of data coming from our visual environment, human beings have developed a biological mechanism called overt visual attention. While watching a scene, the eye makes sudden noncontinuous movements called saccades, only stopping during events called fixations. Eye fixations occur so that regions of visual interest are centered on the densest zone in photoreceptors of the retina, called the fovea. This area is heavily packed with cone cells, which allows maximum visual acuity, even if it only represents approximately one degree of the visual field. Several studies have shown that eye fixations and visual attention are closely associated [START_REF] Findlay | Saccade target selection during visual search[END_REF]. Therefore, studying gaze patterns is of great interest in a wide range of fields [START_REF] Duchowski | A breadth-first survey of eyetracking applications[END_REF][START_REF] Zhang | Human gaze assisted artificial intelligence: A review[END_REF]. For instance, [START_REF] Karessli | Gaze embeddings for zero-shot image classification[END_REF] showed that human gaze is class discriminative, and thus can help improve classification models. In image and video processing, visual attention and saliency have been widely used in compression algorithms [START_REF] Yu | Image compression based on visual saliency at individual scales[END_REF][START_REF] Zünd | Content-aware compression using saliencydriven image retargeting[END_REF][START_REF] Hadizadeh | Saliency-aware video compression[END_REF]. In the medical field, eye-tracking devices are used to help the communication in cases of locked-in syndrom [START_REF] Majaranta | Twenty years of eye typing: Systems and design issues[END_REF] or for diagnosis purposes [START_REF] Harezlak | Application of eye tracking for diagnosis and therapy of children with brain disabilities[END_REF]; for more applications in medicine, see for instance [START_REF] Harezlak | Application of eye tracking in medicine: A survey, research issues and challenges[END_REF].

The factors explaining where people look in a video are usually divided into two categories: bottom-up and topdown factors. Top-down characteristics refer to observer dependant properties, such as the age of the observers, their cultural background, or the task at hand. These factors have been shown to be the cause of sometimes extreme discrepancies in gaze patterns; see for instance Le [START_REF] Meur | Visual attention saccadic models learn to emulate gaze patterns from childhood to adulthood[END_REF] for an exploration of the age factor, or [START_REF] Chua | Cultural variation in eye movements during scene perception[END_REF] and [START_REF] Rayner | Eye movements when looking at unusual/weird scenes: Are there cultural differences?[END_REF] for the cultural parameter. Bottom-up factors refer to stimuli characteristics, such as the spatial properties of the visual scene, or the temporal characteristics of a video. It also includes the implicit properties of the stimuli, such as the presence of faces [START_REF] Cerf | Predicting human gaze using low-level saliency combined with face detection[END_REF] or text in the scene. Most of the visual attention models are purely bottom-up models, meaning that they only extract information from the stimulus. Indeed, bottom-up visual saliency has proven to be a reliable predictor of fixations location in images (Borji et al., 2013).

Over the last century, filmmakers have developed an instinctive knowledge of how to guide the gaze of the audience, manipulating bottom-up characteristics, such as visual cuts, camera movements, shot composition and sizing, and so on. This empirical knowledge contributes to building a set of cinematographic rules and conventions, designed to accommodate the artistic intention of the director with the perception of the audience. However, formalizing these visual tools is not an easy task, and several frameworks and languages have been proposed [START_REF] Ronfard | The prose storyboard language : A tool for annotating and directing movies[END_REF][START_REF] Wu | Analyzing elements of style in annotated film clips[END_REF][START_REF] Wu | Thinking like a director: Film editing patterns for virtual cinematographic storytelling[END_REF]. Such languages help quantifying common cinematographic rules, and allow automated models to be used in the movie production process.

As a consequence, studying the quantitative perceptual effects of the visual tools available to filmmakers is of great interest, both for understanding the way humans perceive movies, but also for filmmakers themselves, who could get quantitative feedback on the effects of their work and techniques. Understanding the mechanisms underlying the visual attention on movies can also be of help for computational models related to movie production, such as automated camera placement, automated editing or 3D animated scenes design.

In this paper, we extend the work of [START_REF] Breeden | Gaze data for the analysis of attention in feature films[END_REF] by proposing a new eye-tracking database on 20 different movie clips, of duration 2 to 7 minutes each. For each clip, we provide cinematographic features annotations drawn from [START_REF] Wu | Analyzing elements of style in annotated film clips[END_REF], such as the camera movement and angle, the framing size, and the temporal location of cuts and edits. Alongside with a comprehensive analysis of the collected data, we expose several strong correlations between high-level cinematographic features and gaze patterns, that can be easily used to improve visual attention modeling. We also perform a benchmark of visual attention models on this database, and we show that state-of-theart models often struggle to grasp and use these high-level cinematographic characteristics. Finally, we discuss several leads on how that information could be included in human visual attention models, in order to improve their performances on cinematic content.

Related work

In this section, we provide a quick overview of the recent works in visual attention modeling, and especially bottomup approaches, such as visual saliency modeling. We then give a very brief review of the field of visual attention in the context of cinematography, and of the databases available to conduct such studies.

Modeling visual attention

As mentioned earlier, eye movements rely on two kinds of attention mechanisms: top down (or endogenous) influences, which are shaped by high-level cognitive processes, such as the task at hand, the cultural background of the observer, or its medical condition, and bottom-up (or exogenous) movements, which are driven by the features of the stimulus itself. The most common way of representing attention, whether it is endogenous or exogenous, is through a representation called saliency map, which is a distribution predicting the likelihood of an eye fixation to occur at a given location. In this work, we will mostly focus on this representation, even if it is not the only one, nor does it captures the full range of human visual attention mechanisms [START_REF] Foulsham | What can saliency models predict about eye movements? spatial and sequential aspects of fixations during encoding and recognition[END_REF][START_REF] Koehler | What do saliency models predict[END_REF].

There has been only a few studies dedicated to model top-down visual attention in scenes. For instance, [START_REF] Kanan | Sun: Top-down saliency using natural statistics[END_REF] proposed a top-down saliency detector, based on object appearance in a Bayesian framework. Other attempts of such models, by [START_REF] Jodogne | Closed-loop learning of visual control policies[END_REF] or [START_REF] Borji | Costsensitive learning of top-down modulation for attentional control[END_REF] for example, yield decent predictive results, considering that the internal cognitive state of an observer is extremely hard to predict, and can lead to less coordination and congruency among gaze patterns of observers [START_REF] Mital | Clustering of gaze during dynamic scene viewing is predicted by motion[END_REF][START_REF] Bruckert | Deep learning for inter-observer congruency prediction[END_REF].

On the other hand, many attention models are dealing with bottom-up features (see for instance Borji and Itti (2013); [START_REF] Borji | Saliency prediction in the deep learning era: Successes and limitations[END_REF]; [START_REF] Wang | Revisiting video saliency: A large-scale benchmark and a new model[END_REF] for extensive reviews). Early models focused on static images, using linear filtering to extract meaningful feature vectors, which are then used to predict a saliency map [START_REF] Itti | A model of saliency-based visual attention for rapid scene analysis[END_REF]Bruce and Tsotsos, 2005;[START_REF] Meur | A coherent computational approach to model bottom-up visual attention[END_REF][START_REF] Harel | Graph-based visual saliency[END_REF][START_REF] Gao | Discriminant saliency, the detection of suspicious coincidences, and applications to visual recognition[END_REF]. Those meaningful visual features include contrast, orientation, edges, or colors, for instance. In the case of dynamic scene viewing, the early investigations underlined the importance of temporal features, such as optical flow or flicker [START_REF] Guo | A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression[END_REF][START_REF] Mahadevan | Spatiotemporal saliency in dynamic scenes[END_REF][START_REF] Mital | Clustering of gaze during dynamic scene viewing is predicted by motion[END_REF][START_REF] Rudoy | Learning video saliency from human gaze using candidate selection[END_REF]. Most of the early dynamic saliency models are however extensions of existing static models, and are limited by the representation power of the chosen hand-crafted features, therefore not grasping the full amount of information delivered by ground-truth saliency.

Recently, deep learning approaches managed to significantly improve performances of attention models. The first attempt of using automatically extracted features was conducted by [START_REF] Vig | Large-scale optimization of hierarchical features for saliency prediction in natural images[END_REF], and managed to outperform most models of the state of the art at the time. Later on, several deep learning models using transfer-learning were proposed, where features learned on large-scale classification datasets were used, like DeepFix [START_REF] Kruthiventi | Deepfix: A fully convolutional neural network for predicting human eye fixations[END_REF], SALICON [START_REF] Huang | Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks[END_REF], DeepNet [START_REF] Pan | Shallow and deep convolutional networks for saliency prediction[END_REF] or Deep Gaze II [START_REF] Kümmerer | Understanding low-and high-level contributions to fixation prediction[END_REF]. More recently, the emergence of large-scale fixation datasets allowed for end-to-end approaches, in order to learn features more specific to visual saliency. These new models, like SalGan [START_REF] Pan | Salgan: Visual saliency prediction with generative adversarial networks[END_REF], SAM-VGG and SAM-Resnet [START_REF] Cornia | Predicting human eye fixations via an lstm-based saliency attentive model[END_REF], or MSI-Net [START_REF] Kroner | Contextual encoder-decoder network for visual saliency prediction[END_REF], exhibit great predictive behaviors, and constitute a very strong baseline for modeling human visual attention. Dynamic models followed the same path towards deep learning, with models such as DeepVS [START_REF] Jiang | Deepvs: A deep learning based video saliency prediction approach[END_REF], ACLNet [START_REF] Wang | Revisiting video saliency: A large-scale benchmark and a new model[END_REF], [START_REF] Bak | Spatiotemporal saliency networks for dynamic saliency prediction[END_REF] or [START_REF] Gorji | Going from image to video saliency: Augmenting image salience with dynamic attentional push[END_REF]. Simi-larly to the static case, they exhibit significantly better predictive performances than earlier approaches (see for instance [START_REF] Wang | Revisiting video saliency prediction in the deep learning era[END_REF] for a more detailed review).

Visual attention and movies

Studying film perception and comprehension is still an emerging field, relying on broader studies on scene perception [START_REF] Smith | A window on reality: Perceiving edited moving images[END_REF][START_REF] Smith | Watchin you watch movies : Using eye tracking to inform cognitive film theory[END_REF]. While the effects of low-level features have been studied in great detail, in part thanks to the progress of saliency models, the effects of higher-level film characteristics are far less well understood. [START_REF] Loschky | What would jaws do? the tyranny of film and the relationship between gaze and higher-level comprehension processes for narrative film[END_REF] showed that the context of a sequence is particularly relevant to understand the way humans are viewing a particular shot, thus underlying the need for a better comprehension of the high-level features. Valuch and Ansorge (2015) studied the influence of colors during editorial cuts, showing that continuity editing techniques result in faster re-orientations of gaze after a cut, and that color contributes to directing attention during edits. Other studies showed strong relationships between eye movement patterns and the number and the size of faces in a scene [START_REF] Rahman | Influence of number, location and size of faces on gaze in video[END_REF][START_REF] Cutting | Facial expression, size, and clutter: Inferences from movie structure to emotion judgments and back[END_REF].

A few studies focused on gaze congruency, or attentional synchrony. [START_REF] Goldstein | Where people look when watching movies: do all viewers look at the same place?[END_REF] showed that observers tend to exhibit very similar gaze patterns while watching films, and that the inter-observer agreement would be sufficient for effective attention based applications, like magnification around the most important points of the scene. [START_REF] Mital | Clustering of gaze during dynamic scene viewing is predicted by motion[END_REF]; [START_REF] Smith | Attentional synchrony and the influence of viewing task on gaze behavior in static and dynamic scenes[END_REF] later showed that attentional synchrony was positively correlated with low-level features, like contrast, motion and flicker. Breathnach Breathnach (2016) also studied the effect of repetitive viewing on gaze agreement, showing a diminution of the inter-observer congruency when movie clips were watched several times. More generally, it appears that understanding human visual attention while watching movies ultimately requires a framework combining both low-and high-level features. From a cognitive point of view, [START_REF] Loschky | The scene perception & event comprehension theory (spect) applied to visual narratives[END_REF] recently proposed a perception and comprehension theory, distinguishing between the front-end processes, occurring during a single fixation, and back-end processes, occurring across multiple fixations and allowing a global understanding of the scene. From a computational and modeling point of view, no model combining low-and high-level film characteristics has yet been proposed. Alongside with [START_REF] Breeden | Gaze data for the analysis of attention in feature films[END_REF], this paper aims to facilitate the development of such a model.

Movies eye-tracking datasets

In the field of visual attention modeling for videos, a majority of the large-scale databases used to train various models contain mostly non-cinematographic stimuli. As we show in Section 6, this leads to consistent errors when saliency models are used on film sequences. Moreover, most studies involving visual attention and movies use their own collected eye-tracking data, as the experimental setups are often very specific to the characteristics studied. However, there exists a few available eye-tracking databases on movie scenes, that can be general enough for modeling purposes.

Hollywood-2 (Mathe and Sminchisescu, 2015) includes 1707 movie clips, from 69 Hollywood movies, as well as fixation data on those clips from 19 observers. Observers were split into three groups, each with a different task (3 observers free-viewing, 12 observers with an action recognition task, and 4 observers with a context recognition task). Each group being relatively small, the common way to use this data for visual attention modeling is by merging those groups, thus introducing potential biases. The large scale of this dataset (around 20 hours of video) is well fit for training deep saliency models, however few conclusions regarding gaze patterns on movies can be drawn from the data itself, since it mainly focuses on task-driven viewing mode, and that each clip is only around 15 seconds long.

SAVAM [START_REF] Gitman | Semiautomatic Visual-Attention modeling and its application to video compression[END_REF] includes 41 highdefinition videos, 28 of which are movie sequences (or use movie-like realisation, like commercials for instance). Eye fixations are recorded from 50 observers, in a free viewing situation. As for Hollywood-2, the each clip is quite short, only 20 seconds on average. [START_REF] Breeden | Gaze data for the analysis of attention in feature films[END_REF] proposed eye-tracking data from 21 observers, on 15 clips from 13 films, for a total of 38 minutes of content. Each clip is between 1 and 4 minutes. Alongside this data, they also provide high-level feature annotations, such as the camera movements in shots, the temporal location and types of edits, the presence or absence of faces on screen, and whether or not the characters are speaking. However, the main limitations of this dataset are the relatively low precision of the eye-tracking device used, and the duration of the total content of the base itself.

It follows that the saliency modeling community, as well as cinematographic studies, would greatly benefit from an extension of Breeden and Hanrahan's work, i.e. a relatively large-scale eye-tracking database on movies sequences, including a large diversity of editing styles, genres and epochs, alongside with high-level features annotations, related to different film-making parameters. In this work, we propose such a database, and the conclusions that we can draw from it.

3 Dataset overview

Films and clips selection

In [START_REF] Wu | Analyzing elements of style in annotated film clips[END_REF], the authors proposed a language called Film Editing Patterns (FEP) to annotate the production and edition style of a film sequence. Alongside this formalization of cinematographic rules, they present an open database of annotations on several film sequences, for pattern analysis purposes. In order to simplify the annotation process of our dataset, we decided to use the same clips.

We selected 20 clips, extracted from 17 different movies. The movies span different times (from 1966 to 2012) and genres, and are from different directors and editors, in order to eliminate bias coming from individual style. Table 1 gives an overview of the selected clips. The sequences were selected as they were the most memorable or famous sequences from each movie, based on scenes that users uploaded to YouTube, indicating popularity and interest to the general public.

Here we give a small description of each scene, and its most remarkable characteristics:

• American History X: Flashback scene, dialogue between characters seated at a table. Mostly static shots on the faces of the characters. This scene is in black and white.

• Armageddon: Action scene, high frequency of edits.

The shot size varies a lot, from extreme closeups to large establishing shots. A lot of camera movements.

• Benjamin Button: Flashback scene. A lot of camera movements tracking the characters. A narrator comments the whole sequence. Some of the shots are replicated, with variations, in order to indicate alternative possibilities in the unfolding of the narrated story.

• Big Fish: Crowd scene, with two main characters walking through the crowd. A few shots take place in a whole different location, with only the two characters conversing.

• The Constant Gardener: Dramatic scene, the camera is handheld, and follows a single character throughout the sequence.

• Departures : Closing scene, alternation of static camera shots. Three characters are present, but no dialogue.

• Forrest Gump: Flashback scene, narrated by a character. Camera movements are used to reveal actors in the scene.

• Gattaca (1): Dialogue scene between two characters. A lot of play on camera angles, since one of the characters is in a wheelchair, and the other one is standing.

• Gattaca (2): Dialogue scene between three characters.

• The Godfather : Dramatic sequence, where the edits alternate back and forth from one central quiet scene to several simultaneous dramatic situations.

• The Good, The Bad and The Ugly: Mexican standoff scene, with three characters, where the frequency of the edits accelerate and the shot sizes go from larger to closer as the tension builds up.

• The Hunger Games: Dramatic scene, alternating a lot of different camera movements, angles and shot sizes.

A crowd is present, but several tricks (colored clothing, focus) are used to distinguish the main characters.

• Invictus: Contemplative scene, starting in a cell and ending in outdoors. Characters appear and disappear as ghosts. A narrator reads a poem.

• Lord of The Rings: Dialogue scene between two characters, alternating with flashbacks, mostly of action scenes. Different camera movements, angles and shot sizes.

• Pulp Fiction: Dialogue scene between two characters seated face to face. The exact same camera angle is used throughout the scene.

• Shawshank Redemption (1): Dialogue between several characters, various camera movements, angles and shot sizes.

• Shawshank Redemption (2): Flashback scene, following a single character, explaining a prison escape.

A narrator comments a part of the sequence. Various camera movements, angles and shot sizes.

• The Shining: Dialogue scene between two characters. Very low frequency of edits, and abundant presence of the color red in the scene.

• The Help (1): Flashback scene, dialogue between two characters.

• The Help (2): Flashback scene, in between a dialogue scene between two characters. A lot of faces and colored clothing.

The length of the clips varies from 1 minute 30 to 7minutes. This length is voluntarily higher than in the other datasets presented in Section 2.3, in order to allow the observer to feel immersed in the sequence, and thus exhibiting more natural gaze patterns. In total, the dataset contains

High-level features annotations

Films typically contain many high-level features aiming to attract or to divert the observers' visual attention [START_REF] Smith | A window on reality: Perceiving edited moving images[END_REF]. These features can be of different sorts : the presence of faces or text, the framing properties, the scene composition, or the camera motion and angle, for instance. The timing of the shots, the selection of the shots from rushes by the editor and the narrative it creates are also high-level features specific to films. Audio cues, like the presence of music or dialogue can also be considered as a form of high-level movie features, and have been increasingly studied as a way to improve visual attention models [START_REF] Tavakoli | Dave: A deep audio-visual embedding for dynamic saliency prediction[END_REF]. However, all of those features can prove very challenging to extract automatically, which can explain why saliency models seem to only learn nontemporal image characteristics, at the scale of the frame, like contrast-or texture-like information. We then used the database of Film Editing Patterns described in [START_REF] Wu | Thinking like a director: Film editing patterns for virtual cinematographic storytelling[END_REF] to select a hand-crafted set of high-level annotations that can help in the study of visual attention and gaze patterns on films. More particularly, such annotations enable us to conduct quantitative analysis on the influence of these cinematographic features over visual attention.

Camera motion

Camera motion is an efficient tool used on set by the filmmaker to direct attention. For each shot of the database, we differentiate several possible camera motions:

• Static: The camera is mounted on a stand and does not move.

•

Camera angle

In order to convey the emotional states of the characters, or power relationships, filmmakers often use camera angles [START_REF] Thompson | Grammar of the Shot Valuch C, Ansorge U (2015) The influence of color during continuity cuts in edited movies: an eye-tracking study[END_REF]. For instance, a rolled plan will often indicates that the characters are lost, or in an unstable state of mind, while filming actors with a low angle will give them an impression of power over the other characters, as they tower over the scene. We relied on six different degrees of camera angles [START_REF] Wu | Analyzing elements of style in annotated film clips[END_REF]):

• Eye: The camera is at the same level as the eyes of the actors

• Low: The camera is lower than the eyes of the actors, pointing up

• High: The camera is higher than the eyes of the actors, pointing down

• Worm: The camera is on the ground, or very low, pointing up with a sharp angle

• Bird: The camera is very high, pointing down with a sharp angle

• Top: The camera is at the vertical of the actors, pointing straight down

Shot size

The size of a shot represents how close to the camera, for a given lens, the main characters or objects are, and thus how much of their body area is displayed on the screen. Shot size is a way for filmmakers to convey meaning about the importance of a character, for instance, or the tension in a scene. Very large shots can also be used to establish the environment in which the characters will progress. To annotate the shot sizes, we use the 9-size scale defined by [START_REF] Thompson | Grammar of the Shot Valuch C, Ansorge U (2015) The influence of color during continuity cuts in edited movies: an eye-tracking study[END_REF]. Figure 2 shows the differences between those shot sizes.

Faces

As explained by [START_REF] Cerf | Predicting human gaze using low-level saliency combined with face detection[END_REF], the presence of faces in images is a very important high-level information to take into account when studying visual attention. We then provide bounding boxes delimiting each face on each frame.

Recent state of the art face detection models show that deep learning models extract this information very well. It is then probable that deep visual attention models are also great at extracting faces features, making this hand-crafted feature redundant. However, we include it as it permits an easier analysis of the editing style: for instance, continuity edits will often display faces on the same area of the image, while shot/reverse shots often display faces on opposite sides of the image.

4 Eye-tracking data collection

Participants and experimental conduct

We have collected eye-tracking data from 24 volunteers (11 female and 13 male), aged 19 to 56 (average 28.8). Participants were split into two groups, each group watching half of the videos. A few observers were part of both groups, and viewed the whole dataset. In total, we acquired exploitable eye fixation data for 14 participants for each video. Details of the sequences viewed by each group can be found in supplementary material.

Viewers were ask to fill an explicit consent form, and to perform a pre-test form. The objective of the pre-test form was to detect any kind of visual impairment that could interfere with the conduct of the experiment (colourblindess, or strabism, for instance), as well as ensuring that they could understand English language well enough, as sequences were extracted from the English version of the movies. Participants were informed that they could end the experiment at any moment.

During a session, subjects viewed the 10 movie sequences assigned to their group, in a random order. Sound was delivered by a headset, and volume was set before the first sequence. They could also adjust the volume at will during the experiment. After each sequence, a 15 seconds dark gray screen was displayed. After a series of five clips (around 15 to 20 minutes of video), participants were asked to make a break, as long as they needed, and fill a form, recording whether or not they could recall the scenes they saw, whether or not they had seen the movies previously, or if they recognized any actors in the scenes. After the second series of five clips, at the end of the experiment, they were asked to fill the same form. The total duration of the experiment for a participant was between fifty minutes and one hour.

Recording environment and calibration

Eye movements were recorded using a Tobii X3-120 eye tracker, sampling at 120 Hz. The device was placed at the bottom of a 24,1" screen with a display resolution of 1920 × 1200 pixels. All stimuli had the same resolution (96 dpi), and were displayed respecting the original aspect ratio, using letterboxing. The participants were asked to sit at a distance of 65cm from the screen. They were asked to sit as comfortably as possible, in order to minimize head movements. In order to replicate natural viewing conditions, we did not use chin rests. Stereo sound, with a sampling frequency of 44100Hz, was delivered to the participant, using a headset. Calibration was performed using the 9-points Tobii calibration process. In the case of errors of more than one degree, the participant was asked to reposition, and recalibrate. After the break, before viewing the five last clips, participants were asked to validate the previous calibration, and to recalibrate if necessary.

After recording the data for all participants, we used the following cleaning procedure. First, we ensured that every participant had a gaze sampling rate of more than 90% (i.e. more than 90% of the sampled points were considered as valid). We then kept only points that were flagged as fixations, eliminating tracking errors due to blinks or other factors, as well as points recorded during saccades. This choice was motivated by the relatively low frequency rate of the eye-tracker, making the analysis of saccadic data impossible. Then, we discarded all points that fell in the letterboxing or outside the screen. Finally, we used the position of the remaining raw points to construct binary fixation maps : for each frame, we create an image the same size of the frame, where we give the value 1 to each pixel where a fixation point was flagged during the time the frame was on screen (i.e. 1/24th of a second), and 0 to each pixel where no fixation occurred.

Exploring the effects of film making patterns on gaze

In this section, we explore several characteristics throughout our database, and analyze underlying relationships between editing patterns and eye fixations patterns. In the following, we will often refer to fixation maps and saliency maps. For each frame, the fixation map is the binary matrix where each pixel value is 1 if a fixation occurred at the pixel location during the frame, and 0 if not, as described previously. Saliency maps are obtained by convolving the fixation maps with a 2-D Gaussian kernel, which variance is set to one degree of visual angle (in our case, one degree of visual angle equals to roughly 45 pixels), in order to approximate the size of the fovea.

Editing-induced visual biases

Studying the average of the saliency maps usually reveals strong attentional biases. For instance, on static images, [START_REF] Tatler | The central fixation bias in scene viewing: Selecting an optimal viewing position independently of motor biases and image feature distributions[END_REF] showed that humans tend to look at the center of the frame. That center bias is also commonly used as a lower baseline for saliency models. In order to avoid recording this bias too much, we did not take into account for our analysis the first 10 frames of each clip, as people tend to look in the middle of the screen before each stimulus. This center bias is also strong on video stimuli: for instance, Fig. 4 (a) and (b) shows the average saliency map on our dataset and on the DHF1K dataset [START_REF] Wang | Revisiting video saliency prediction in the deep learning era[END_REF] respectively. However, the latter is composed of Youtube videos, with a great diversity in the content, and no cinematographic scenes, which might cause a different viewing bias. Fig. 4 (a) shows a peak density slightly above the center of the frame, which would indicate that filmmakers use a different composition rule. Fig. 4 (c) shows a centered Gaussian map, often used as a baseline for centered bias. Correlation between the average saliency map on our dataset and this centered Gaussian is 0.81, whereas the correlation between the average map on DHF1K and the centered Gaussian is 0.84, which highlights this position discrepancy between the two average saliency maps. This is consistent with the findings of [START_REF] Breeden | Gaze data for the analysis of attention in feature films[END_REF], and is most likely due to the rule of thirds [START_REF] Brown | Cinematography: theory and practice: image making for cinematographers and directors Bruce NDB[END_REF] stating that in cinematography, important elements of the scene should be placed on thirds lines, i.e. lines dividing the frame in thirds horizontally and vertically.

We also observe disparities in this bias depending on the size of the shot: the wider the shot, the more diffuse that bias is, indicating that directors tend to use a bigger part of the screen area when shooting long shots, while using mostly the center of the frames for important elements dur- ing closeups and medium shots (Fig. 5,(a,b,c). We also observe a leftward (resp. rightward) bias during pans and dolly shots, where the camera moves towards the left (resp. right), as exposed in Fig. 5 (d,e). This confirms that camera movements are an important tool for filmmakers to guide the attention of the spectators.

Inter-observer visual congruency

Inter-observer congruency (IOC) is a measure of the dispersion of gaze patterns between several observers watching the same stimulus. In other words, it measures how well gaze patterns from a subset of the observers is predictive of the whole set of observers. Thus, it has been used in saliency modeling as an upper baseline. This characteristic is very similar to attentional synchrony [START_REF] Smith | Attentional synchrony in static and dynamic scenes[END_REF], and many methods have been proposed to measure it. For instance, [START_REF] Goldstein | Where people look when watching movies: do all viewers look at the same place?[END_REF] measure how well gaze points are fitted by a single ellipsoid, while [START_REF] Mital | Clustering of gaze during dynamic scene viewing is predicted by motion[END_REF] use Gaussian mixture models, associating low cluster covariance to high attentional synchrony.

In their work, [START_REF] Breeden | Gaze data for the analysis of attention in feature films[END_REF] use the area of the convex hull of the fixation points, for each frame of their dataset. This allows to take into account all the fixation points, and requires no prior hypothesis about the shape of the regions of interest. However, as they mention, this approach can only be viewed as an upper bound on IOC, as it is very sensitive to outliers. Using it on each frame also does not take into account the temporal aspect of movie viewing: if several observers watch the same two or three points of interest, chances are, if the points are spatially distant from one another, that the convex hull area will be high, even though all the observers exhibited similar gaze patterns in a different order, in terms of fixation locations.

In order to remedy this issue, we used a leave-one-out approach, over a temporal sliding window. Assuming that there is N observers, we gather the locations of all the fixation points of (N -1) observers during a window of n frames, as well as the locations of the fixation points of the left out observer. We then can build a saliency map using the fixation points of the N -1 observers, by convolving the fixation map with a 2-D Gaussian kernel, and use any saliency metric to compare it to the fixation map (or saliency map) of the left out observer. The process is then iterated and averaged over all observers. To compare the saliency map of the N -1 observers to the left-out one, we used the Normalized Scanpath Saliency metric (NSS); more details about can be found in Le [START_REF] Meur | Methods for comparing scanpaths and saliency maps: strengths and weaknesses[END_REF]. A high value of this score will mean that people tend to look in the same region, and a low value will indicate a higher dispersion in the fixations. The main drawback of this way of computing IOC, especially for large-scale datasets, is its computational cost, as the process is iterated over every observers, and every n-frame window of the dataset.

The size n of the sliding window can be adjusted, depending on the number of observers, and the studied characteristics. In this work, we chose two window sizes: 5 frames and 20 frames (roughly 200 and 800ms). 5 frames corresponds roughly to the average fixation time, and 20 frames allows for a wider point of view, with less noise. While a shorter window allows for more noise, especially with a relatively short number of observers, it can also underline short-timed patterns. For instance, using the shorter window size, we noticed on every stimulus a significant drop of inter-observer congruency during the five frames consecu-tive to a cut (see Fig. 6). This would tend to indicate that a short adjustment phase is taking place, as the observers search for the new regions of interest. The amplitude of this drop can be a meaningful perceptual characteristic of a cut, as some editors would voluntarily try to increase that drop, in order to confuse the viewer, or decrease it, in order to create a certain sense of continuity through the cut. Longer window sizes would be better suited for analyzing more general trends, at the scale of a shot, for instance. In the following, we will use the values computed using a 20frames window, since our editing annotations are at the level of the shot.

As we suspected, the IOC values are relatively high (average of 4.1 over the whole database), especially compared to IOC values in the static case; see for instance [START_REF] Bruckert | Deep learning for inter-observer congruency prediction[END_REF] for IOC distributions on static datasets. This coroborates the findings of [START_REF] Goldstein | Where people look when watching movies: do all viewers look at the same place?[END_REF] and [START_REF] Breeden | Gaze data for the analysis of attention in feature films[END_REF], that viewers only attend to a small portion of the screen area. We also observe a disparity in IOC scores between the movies : the scene with the highest score on average, is the clip from The Shining (5.76), and the lowest score on average is the clip from Armageddon (3.41). This would tend to indicate that inter-observer congruency reflects certain features in terms of editing style; for instance, Figure 7 shows a correlation (0.35) between the average IOC score of a sequence and the average length of the shots of this sequence. However, due to the low number of samples, this correlation is not significant (p = 0.19).

Inter-observer congruency and cinematographic features

We then considered the effects of directors' choices and editing characteristics on inter-observer congruency. Fig. 8 shows the distributions of IOC scores depending on the high-level annotations described in 3.2. We performed a one-way ANOVA for each annotation group (camera movements, camera angles and size of the shot), and confirmed that camera movements, angles and shot sizes have a significant influence on IOC scores (p << 10 -5 in the three cases). Post-hoc pairwise t-tests within the annotation groups show significant differences (p << 10 -5 ) between all the characteristics, except between static and dolly shots (p = 0.026), extreme closeups and establishing shots (p = 0.177), and all pairs among medium closeups, medium shots, medium-long shots and long shots (p > 10 -2 in all cases). This might be due to categories sometimes not very well defined, as it can be hard distinguishing between a medium shot and a medium-long shot, for instance.

Visual attention modeling

In this section, we evaluate several visual saliency models on our database, and highlight certain limitations of current dynamic saliency models. We also discuss how editing patterns can explain some of the failure cases of the models.

Performance results

In Table 3, we show the performances of state-of-the-art static and dynamic saliency models. In order to evaluate the models, we used the following six classic saliency metrics, • Pearson's correlation coefficient (CC ∈ [-1, 1]) evaluates the degree of linear correlation between the predicted saliency map and the ground truth map.

• SIM (SIM ∈ [0, 1]) evaluates the similarity between two saliency maps through the intersection between their histograms.

• AUC (AUC-J, AUC-B ∈ [0, 1]) is the area under the Receiver Operator Curve (ROC). Differences between AUC-J and AUC-B relies on the way true and false positive are computed (see Le [START_REF] Meur | Methods for comparing scanpaths and saliency maps: strengths and weaknesses[END_REF] for more details).

• Normalised Scanpath Saliency (NSS ∈ [0, +∞[) is computed between the predicted saliency map and the ground truth fixation map by measuring the saliency values at the locations of the fixations.

• Kullback-Lieber Divergence (KLD ∈ [0, +∞[) between the two probability distributions represented by the saliency maps.

In general, those results are quite low, compared to performances on non-cinematic video datasets (see for instance [START_REF] Wang | Revisiting video saliency prediction in the deep learning era[END_REF]).

This would indicate, in the case of deep-learning models, that either the training sets do not contain enough of videos with features specific to cinematic stimuli, or the deep neural networks cannot grasp the information from some of those features. Even though the best performing model is a dynamic one [START_REF] Zhang | Video saliency prediction based on spatial-temporal two-stream network[END_REF], we observe that static models (DeepGaze II and MSINet) performances are quite close to those of dynamic models. This might support the latter hypothesis, that dynamic models fail to extract important temporal features.

Recent work from [START_REF] Tangemann | Measuring the importance of temporal features in video saliency[END_REF] on the failure cases of saliency models in the context of dynamic stimuli also highlight this point, listing cases like appearing objects, movements or interactions between objects as some of the temporal causes of failure. Figure 9 shows an example from our database of such a failure case. It should be noted that all the deep learning models are trained on non-cinematic databases, with the exception of ACLNet, which include the Hollywood 2 dataset in its training base. However, this base is not well-fit to learn meaningful cinematographic features, as explained in Section 2.3

In order to confirm this hypothesis, we retrained the ACLNet model using the same training procedure described in [START_REF] Wang | Revisiting video saliency: A large-scale benchmark and a new model[END_REF]. For the static batches, we used the same dataset (SALICON [START_REF] Huang | Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks[END_REF]), and for the dynamic batches, we created a set composed of half of our videos, randomly selected, leaving the other videos out for testing (roughly 490000 frames for training, and 450000 frames for test). We only obtained marginally better results on some of the metrics (0.550 instead of 0.544 on the correlation coefficient metric, 2.592 instead of 2.54 on the NSS metric), and did not outperform the original model settings on the other. All of this would tend to indicate that some features, specific to cinematographic images, could not be extracted by the model.

Edition annotation and model performances

We also studied how the two best dynamic models, [START_REF] Zhang | Human gaze assisted artificial intelligence: A review[END_REF] and ACLNet [START_REF] Wang | Revisiting video saliency prediction in the deep learning era[END_REF], performed on our database, depending on shot, camera motion and camera angle characteristics. Table 4 shows the average results of the models depending on the annotation characteristics. Similarly to Subsection 5.2, we performed one-way ANOVAs to ensure that results within each table would yield significant differences. In all cases, p-values were under 10 -5 .

As shown in Table 4 (a), it appears that saliency models perform relatively well on static scenes, or when the camera movement tracks an actor, or an object on screen. Performances are also quite good on shots including rack focuses, which was expected, as this is a very strong tool for the filmmaker to use to direct attention, and deep feature extractors distinguish very well blurry background from clear objects. However, when a more complex camera motion appears, like pans or tilts, models seem to fail more often; this might indicate that saliency models are unable to anticipate that an object is likely to appear in the direction of the motion, which humans usually do.

With Table 4 (b), we observe that camera angles show little variations in the performances of the models. However, it seems that scenes with high amplitude angles (Bird or Worm) are easier for a model to predict. This is probably due to the fact that those camera angles are often used when filming characters and faces, in order to convey a dominant or a submissive feeling from the characters [START_REF] Thompson | Grammar of the Shot Valuch C, Ansorge U (2015) The influence of color during continuity cuts in edited movies: an eye-tracking study[END_REF]; since deep learning models are very efficient at recognizing faces, and faces tend to attract gaze, saliency models naturally perform better on those shots.

Finally, looking at Table 4 (c), saliency models seem to exhibit great performances on closeups scenes, which could be, again, because closeup scenes often display faces. Medium to long shots are however harder to predict, maybe because a wider shot allows the director to add more objects or actors on screen, and as shown by [START_REF] Tangemann | Measuring the importance of temporal features in video saliency[END_REF], interactions between objects is often a failure case for deep saliency models. Closeup shots also display one of the lowest mean IOC, which could also explain why they are easier to predict.

Conclusion and future work

In this work, we introduced a new eye-tracking dataset dedicated to study visual attention deployment and eye fixations patterns on cinematographic stimuli. Alongside with the gaze points and saliency data, we provide annotations on several film-specific characteristics, such as camera motion, camera angles or shot size. These annotations allow us to explain a part of the causes of discrepancies between shots in terms of inter-observer visual congruency, and in terms of performances of saliency models.

In particular, we highlight the conclusions of [START_REF] Tangemann | Measuring the importance of temporal features in video saliency[END_REF] regarding failure cases of state-of-the-art visual attention models. Video stimuli sometimes contain a lot of non-static information, that, in some cases, is more important for directing attention than image-related spatial cues. As directors and editors includes consciously a lot of meaning with their choices of cinematographic parameters (camera motion, choice of the shots within a sequence, shot sizes, etc.), we would advocate researchers in the field of dynamic saliency to take a closer look at movie sequences, in order to develop different sets of features to explain visual attention.

Looking forward, we can investigate whether or not the high-level cinematic features that we provided would be of help to predict visual deployment, by building a model that includes this kind of metadata at the shot level. Another crucial point that we did not pursue is the context of the shot : the order of the shots within the sequence has been proven to influence gaze patterns [START_REF] Loschky | What would jaws do? the tyranny of film and the relationship between gaze and higher-level comprehension processes for narrative film[END_REF][START_REF] Loschky | The scene perception & event comprehension theory (spect) applied to visual narratives[END_REF]. As these questions have been tackled from a psychological or cognitive point of view, they remain to be studied by the computer science part of the field, and to be included in visual attention models. This would greatly benefit multiple areas in the image processing field, like video compression for streaming, or automated video description.

Furthermore, we hope that this data would help cinema scholars to quantify potential perceptual reasons to filmmaking conventions, assess continuity editing on sequences and hopefully improve models of automated edition [START_REF] Galvane | Cameraon-rails: Automated computation of constrained camera paths[END_REF].

Finally, developing automated tools to extract similar high-level cinematic information could be particularly of interest, both for the design of such tools, as it would give cues on the way to design better visual attention models on cinematographic data, but also with its outcome, as it would allow the provision of large-scale annotated cinematic databases, which would give a new -quantitativedimension to research on movie contents by cinema scholars.

  Track: The camera moves in order to keep an object or a character in the same region of the image • Zoom: The camera operator is zooming in or out • Pan: The camera rotates on the horizontal plan • Tilt: The camera rotates on the vertical plan • Dolly: The camera is being moved using a dolly • Crane: Complex camera motion, where both the camera base and the mount are in motion • Handheld: The camera operator holds the camera by hand, creating a jerky motion • Rack focus: The focus of the lens shifts from one point of the scene to an other Those features are binary for each shot, and a single shot can include different camera motions.

Figure 1 :

 1 Figure 1: Examples of different camera angles, extracted from The Hunger Games (from Wu et al. (2017)).

Figure 2 :

 2 Figure 2: The nine framing sizes, all appearing in The Good, The Bad & The Ugly (from Wu et al. (2017)).

Figure 3 :

 3 Figure 3: Examples of saliency heatmaps created from the collected fixation points

Figure 4 :

 4 Figure 4: Average saliency map of our dataset (a) compared to DHF1K (Wang et al., 2019) dataset (b) and to a centered gaussian map (c). Both average maps exclude the first 10 frames of each clip.

Figure 5 :

 5 Figure 5: Average saliency maps for closeup shots (XCU-BCU-CU) (a), medium shots (MCU-MS-MLS) (b) and long shots (LS-VLS-EST) (b). Subfigure (d) is the average saliency map during pans and dolly shots moving to the left, and (e) is the average saliency map during pans and dolly shots moving to the right.

Figure 6 :

 6 Figure 6: Example of the evolution of inter-observer congruency over a sequence of Benjamin Button. Blue is the IOC value computed with a 5-frames time window (n = 5), yellow is using a 20-frames time window (n = 20). Red lines show the locations of the edits; notice the characteristic drop in the IOC value after each cut (for the 5-frames window).

Figure 7 :

 7 Figure 7: IOC scores depending on the average length of the shots for each sequence (excluding The Shining, as it is an outlier in terms of length of the shots)

Figure 8 :

 8 Figure 8: IOC scores depending on camera movement features (a), camera angles (b) and shot size (c)

Figure 9 :

 9 Figure 9: An example of failure case in ShawshankRedemption. Here, the camera pans from the face of the prison director to the poster on the wall. While observers quickly shift their attention towards the poster, as suggested by the camera movement, even though it is not yet on screen, models tend to predict areas of interest on the faces.

Table 1 :

 1 Overview of the selected clips

	Title	Director	Genre (IMDb)	Nb. Frames Aspect ratio Year
	American History X	Tony Kaye	Drama	5702	1.85
	Armageddon	Michael Bay	Action, Adventure, Sci-Fi	4598	2.39
	The Curious Case of Benjamin Button	David Fincher	Drama, Fantasy, Romance	4666	2.40
	Big Fish	Tim Burton	Adventure, Drama, Fantasy	3166	1.37
	The Constant Gardener	Fernando Meirelles	Drama, Mystery, Romance	5417	1.85
	Departures	Yôjirô Takita	Drama, Music	10117	1.85
	Forrest Gump	Robert Zemekis	Drama, Romance	2689	2.39
	Gattaca (1)	Andrew Niccol	Drama, Sci-Fi, Thriller	3086	2.39
	Gattaca (2)	Andrew Niccol	Drama, Sci-Fi, Thriller	3068	2.39
	The Godfather	Francis Ford Coppola	Crime, Drama	1918	1.37
	The Good, The Bad & The Ugly	Sergio Leone	Western	9101	2.35
	The Hunger Games	Gary Ross	Action, Adventure, Sci-Fi	5771	2.35
	Invictus	Clint Eastwood	Biography, Drama, History	2203	2.39
	LOTR : The Fellowship of the Ring	Peter Jackson	Action, Adventure, Drama	5109	2.40
	Pulp Fiction	Quentin Tarantino	Crime, Drama	3211	2.39
	The Shawshank Redemption (1)	Frank Darabont	Drama	5374	1.85
	The Shawshank Redemption (2)	Frank Darabont	Drama	4821	1.85
	The Shining	Stanley Kubrick	Drama, Horror	4781	1.33
	The Help (1)	Tate Taylor	Drama	4151	1.85
	The Help (2)	Tate Taylor	Drama	5244	1.85

roughly one hour of content. Table

2

show the lengths of the average shots for each sequence. The high diversity in terms of shot lengths underlines the diversity in terms of editing styles.

Table 2 :

 2 Lengths of the sequences, and of the longest, shortest and average shots of each sequence.

	Armageddon	191.8	12.1	0.0	1.6
	The Hunger Games	240.8	16.7	0.6	2.4
	The Curious Case of Benjamin Button	194.7	11.8	0.3	2.5
	The Godfather	80.0	6.8	0.5	2.7
	Big Fish	132.1	7.6	0.7	2.8
	The Constant Gardener	226.0	13.8	0.4	3.5
	LOTR : The Fellowship of the Ring	213.1	8.4	0.5	3.6
	The Good, The Bad & The Ugly	379,7	36.5	0.2	3.8
	The Help (2)	218.8	14.0	1.0	4.0
	Invictus	91.9	8.6	1.8	4.2
	American History X	237.9	14.7	1.0	4.2
	Pulp Fiction	134.0	12.2	1.4	4.6
	The Shawshank Redemption (1)	224.2	19.2	0.8	4.7
	The Help (1)	173.2	17.7	1.8	6.0
	Gattaca (1)	128.7	23.7	0.2	6.1
	Departures	422.0	21.8	1.8	6.6
	Forrest Gump	112.2	16.6	1.8	6.7
	Gattaca (2)	128.0	17.1	1.8	6.7
	The Shawshank Redemption (2)	201.1	18.0	1.8	7.7
	The Shining	199.5	107.1	8.8	39.9

Sequence

Sequence Length (s) Longest shot (s) Shortest shot (s) Average shot (s)

Table 3 :

 3 Scores of several saliency models on the database. Non-deep models are marked with *. Best performances are bolded. †Note that the testing dataset for the retrained ACLNet model is not exactly the same as the other models, as it is a subset of half of our dataset.

		Model	CC ↑ SIM ↑ AUC-J ↑ AUC-B ↑ NSS ↑ KLD ↓
	Baseline	Center Prior*	0.398 0.302	0.859	0.771	1.762	2.490
		PQFT* (Guo and Zhang, 2010)	0.146 0.189	0.702	0.621	0.783	2.948
		Two-stream (Bak et al., 2018)	0.404 0.329	0.873	0.830	1.738	1.410
	Dynamic models	DeepVS (Jiang et al., 2018) ACLNet (Wang et al., 2019)	0.457 0.361 0.544 0.429	0.880 0.892	0.829 0.858	2.270 2.54	1.245 1.387
		ACLNet (retrained) †	0.550 0.423	0.890	0.858	2.592	1.408
		Zhang and Chen (2019)	0.608 0.454	0.903	0.881	2.847	1.154
		Itti* (Itti et al., 1998)	0.208 0.195	0.756	0.640	1.005	2.573
		SalGAN (Pan et al., 2017)	0.533 0.390	0.897	0.781	2.622	1.372
	Static models	DeepGaze II (Kümmerer et al., 2017) 0.584 0.362	0.846	0.774	3.188	2.307
		MSINet (Kroner et al., 2020)	0.597 0.417	0.901	0.893	2.893	1.226

Table 4 :

 4 Scores of two saliency models on the database, depending on hand-crafted editing features. Highest score for each metric and each model is bolded, lowest score is italicized. .532 0.586 0.549 0.497 0.510 0.473 0.520 0.512 NSS 2.596 2.271 2.689 2.677 2.497 2.481 2.255 2.478 2.543 Zhang et al. CC 0.656 0.607 0.663 0.645 0.580 0.615 0.567 0.628 0.636 NSS 3.320 2.679 3.099 3.186 2.889 3.027 2.733 3.089 3.221 (c) Scores depending on shot size

	Model		Metric Static Track Zoom	Pan	Tilt	Dolly Rack Focus
	ACLNet	CC NSS	0.561 0.545 0.538 0.466 0.488 0.517 2.631 2.610 2.523 2.138 2.269 2.481	0.545 2.610
	Zhang et al.	CC NSS	0.637 0.608 0.643 0.556 0.584 0.615 3.014 2.908 3.118 2.615 2.797 3.022	0.675 3.338
				(a) Scores depending on camera motion
		Model	Metric	Eye	High	Low		Bird Worm	Top
	ACLNet	CC NSS	0.552 0.500 0.525 0.544 0.532 0.540 2.602 2.343 2.465 2.699 2.679 2.628
	Zhang et al.	CC NSS	0.621 0.582 0.605 0.648 0.679 0.672 2.932 2.777 2.918 3.286 3.513 3.375
				(b) Scores depending on camera angles
	Model	Metric XCU BCU	CU	MCU	MS	MLS	LS	VLS	EST
	ACLNet	CC	0.526 0

The dataset generated and analysed during the current study is available at https://github. com/abruckert/eye_tracking_filmmaking

Conflict of interest

The authors declare that they have no conflict of interest.

Open practices statement

The study was not preregisered; the dataset generated and analysed during the current study is available at https://github.com/abruckert/eye_ tracking_filmmaking.