Ondrej Miksik

Juan-Manuel Pérez-Rúa

Philip H S Torr

Patrick Pérez

ROAM: a Rich Object Appearance Model with Application to Rotoscoping

: ROAM for video object segmentation. Designed to help rotoscoping, the proposed object appearance model allows the automatic delineation of a complex object in a shot, starting from an initial outline provided by the user.

Introduction

Modern high-end visual effects (vfx) and post-production rely on complex workflows whereby each shot undergoes a succession of artistic operations. Among those, rotoscoping is probably the most ubiquitous and demanding one [START_REF] Bratt | Rotoscoping: Techniques and tools for the Aspiring Artist[END_REF][START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF]. Rotoscoping amounts to outlining accurately one or several scene elements in each frame of a shot. This is a key operation for compositing [START_REF] Wright | Digital compositing for film and video[END_REF] (insertion of a different background, whether natural or synthetic), where it serves as an input to subsequent operations such as matting and motion blur removal. 1 Rotoscoping is also a pre-requisite for other important operations, such as object colour grading, rig removal and new view synthesis, with large amounts of elements to be handled in the latter case.

Creating such binary masks is a painstaking task accomplished by trained artists. It can take up to several days of work for a complex shot of only a few seconds, using dedicated tools within video editing softwares like Silhouettefx, Adobe After Effect, Autodesk Flame or The Foundry Nuke. As discussed in [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF], professional roto artists use mostly tools based on roto-curves, i.e., parametric closed curves that can be easily defined, moved and edited throughout shots. By contrast, these artists hardly use brush-based tools, even if empowered by local appearance modelling, graphbased regularization and optic flow-based tracking as After Effect's ROTOBRUSH.

Due to its massive prevalence in professional workflows, we address here rotoscoping in its closed contour form, which we aim to facilitate. Roto-curves being interactively placed in selected keyframes, automation can be sought either at the key-frame level (reducing the number of user's inputs) or at the tracking level (reducing the number of required key-frames). In their recent work, Li et al. [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF] offers with ROTO++, a tool that helps on both fronts, thanks to an elegant shape modelling.

In the present work, we explore a complementary route that focuses on the automatic tracking from a given keyframe.

In essence, we propose to equip the roto-curve with a rich, adaptive modelling of the appearance of the enclosed object. This model, coined ROAM for Rich Online Appearance Model, combines in a flexible way various appearance modelling ingredients: (i) Local foreground/background colour modelling, in the spirit of VIDEO SNAPCUT [START_REF] Bai | Video Snapcut: Robust video object cutout using localized classifiers[END_REF] but attached here to the roto-curve; (ii) Fragment-based modelling to handle large displacements and deformations and (iii) Global appearance modelling, which has proved very powerful in binary segmentation with graph cuts, e.g. in [START_REF] Boykov | Interactive graph cuts for optimal boundary and region segmentation of objects in nd images[END_REF].

We would like to emphasize that our model is the first that combines local appearance models along the closed contour with global appearance model of the enclosed object using discrete Green theorem, and pictorial structure to capture locally rigid deformations, in a principled structured prediction framework. As demonstrated on recent benchmarks, ROAM outperforms state-of-art approaches when a single initial roto-curve is provided. It is in particular less prone to spurious changes of topology that lead to eventual losses than After Effect ROTOBRUSH, and more robust than ROTO++ [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF] in the absence of additional user inputs. This robustness makes it appealing to facilitate rotoscoping, either as a standalone tool, or combined with existing curve-based tools such as ROTO++.

Related work and motivation

Rotoscoping is a form of interactive "video object"2 segmentation. As such, the relevant literature is vast. For sake of brevity, we focus mostly our discussion on works that explicitly target rotoscoping or very similar scenarios.

Rotoscoping and curve-based approaches

Li et al. [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF] recently released a very detailed study of professional rotoscoping workflow. They first establish that trained artists mostly use parametric curves such as Bezier splines to delineate objects of interest in key-frames, "track" them from one frame to the next, edit them at any stage of the pipeline and, last but not least, pass them in a compact and manipulable format to the next stage of the vfx pipeline, e.g., to the compo-artists. Professional rotoscoping tools such as Silhouettefx, Blender, Nuke or Flame are thus based on parametric curves, which can be either interpolated between key-frames or tracked with a homographic "planar tracker" when suitable. Sticking to this ubiquitous workflow, the authors propose ROTO++ to speed it up. Bezier roto-curves defined by the artist in the selected key-frames allow the realtime learning of a non-linear low-dimensional shape space based on a Gaussian process latent variable model. Shape tracking between key-frames, as well as subsequent edits, are then constrained within this smooth manifold (up to planar transforms), with substantial gains in work time. Our work is fully complementary to ROTO++: while ROAM does not use a strong shape prior in its current form, it allows to capture the dynamic appearance of the video object, something that ROTO++ does not address.

In their seminal rotoscoping work, Agarwala et al. [START_REF] Agarwala | Keyframe-based tracking for rotoscoping and animation[END_REF] proposed a complete interactive system to track and edit Bezier roto-curves. It relies on the popular active contour framework [START_REF] Blake | Active contours[END_REF][START_REF] Kass | Snakes: Active contour models[END_REF]: a curve, parametrized by control points, finely discretized and equipped with a second-order smoothness prior is encouraged to evolve smoothly and snap to strong image edges. Their energy-based approach also uses local optical flow along each side of the shape's border. In contrast to this work, our approach offers a richer local appearance modelling along the roto-shape as well as additional intraobject appearance modelling.

Similarly to [START_REF] Agarwala | Keyframe-based tracking for rotoscoping and animation[END_REF], Lu et al. [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF] recently introduced an interactive object segmentation system called "coherence parametric contours" (CPC), which combines planar tracking with active contours. Our system includes similar ingredients, with the difference that the planar tracker is subsumed by a fragment-based tracker and that the appearance of the object and of its close surrounding is also captured and modeled. We demonstrate the benefits of these additional features on the evaluation dataset introduced by Lu et al. [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF].

Masks and region-based approaches

Other notable approaches to interactive video segmentation address directly the problem of extracting binary masks, i.e. labelling pixels of non-keyframes as foreground or background. As discussed in [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF][START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF], a region-based approach is less compatible with professional rotoscoping, yet provides powerful tools. Bai et al. [START_REF] Bai | Video Snapcut: Robust video object cutout using localized classifiers[END_REF] introduced VIDEO SNAPCUT, which lies at the heart of After Effect's ROTOBRUSH. Interaction in VIDEO SNAPCUT is based on foreground/background brushes, following the popular scribble paradigm of Boykov and Jolly [START_REF] Boykov | Interactive graph cuts for optimal boundary and region segmentation of objects in nd images[END_REF]. The mask available in a given frame is tracked to the next frame through the propagation of local windows that straddle its border. Each window is equipped with a local foreground/background colour model and a local shape template, both updated through time. After propagation along an object-centric optical-flow, these windows provide suitable pixel-wise unaries that are fed to a classic graph-cut. This approach provides a powerful way to capture on-the-fly local colour models and combine them adaptively with some shape persistence. However, being based on graph-cut (pixel-wise labelling), ROTOBRUSH can be penalized by its extreme topology flexibility: as will be showed in the experiments, rapid movements of the object, for instance, can cause large spurious deformations of the mask that can eventually lead to complete losses in the absence of user intervention. In ROAM, we take inspiration from the local colour modelling at the object's border and revisit it in a curve-based segmentation framework that allows tighter shape control and easier subsequent interaction.

More recently, Fan et al. introduced JUMPCUT [START_REF] Fan | JumpCut: Non-successive mask transfer and interpolation for video cutout[END_REF], another mask-based approach where frame-to-frame propagation is replaced by mask transfer from the key-frame(s) to distant frames. This long-range transfer leverages dense patch correspondences computed over the inside and outside of the known mask, respectively. The transfered mask is subsequently refined using a standard level set segmentation (region encoded via a spatial map). A salient edge classifier is trained online to locate likely fragments of object's new silhouette and drive the level set accordingly. They reported impressive results with complex deformable objects going through rapid changes in scene foreground. However, similarly to ROTOBRUSH, this agility might also become a drawback in real rotoscoping scenarios, as is the lack of shape parametrization. Also, the underlying figure/ground assumption (the object is moving distinctly in front of a background) is not met in many cases, e.g. rotoscoping of a static scene element or of an object in a dynamic surrounding.

Introducing ROAM

Our model consists of a graphical model with the following components: (i) a closed curve that defines an object and a collection of local foreground/background3 appearance models along it; (ii) a global appearance model of the enclosed object; and (iii) a set of distinctive object's landmarks. While the global appearance model captures image statistics as in graph-cut approaches [START_REF] Boykov | Interactive graph cuts for optimal boundary and region segmentation of objects in nd images[END_REF][START_REF] Rother | Grabcut: Interactive foreground extraction using iterated graph cuts[END_REF], it is the set of local fg/bg appearance models placed along the boundary that enables accurate object delineation. The distinctive object's landmarks organized in a star-shaped model (Fig. 3, left) help to prevent the contour from sliding along itself and to control the level of non-rigid deformations. The landmarks are also used to robustly estimate a rigid transformation between the frames to "pre-warp" the contour, which significantly speeds-up the inference. In addition, the control points of the roto-curve, as well as the local fg/bg models and the landmarks are maintained through time, which provides us

I Y y 0 y 1 y m y M X x 1 x n x N Figure 2: Graphical model of ROAM.
In joint model defined by energy E(X , Y; I) in ( 1), contour node variables (white squares) form a closed 1-st order chain conditioned on image data (grey box) and landmark variables (white circles), the latter variables forming a shallow tree conditioned on all others.

with different types of temporal correspondences. Given a colour image I = {I p } p∈P , a conditional graphical model (Fig. 2) is defined through the energy function

E(X , Y; I) := E C (X ; I) + E L (Y; I) + E J (X , Y), (1)
where E C and E L depend only on the roto-curve configuration X and the landmarks configuration Y respectively, and E J links the two together (independently of the image). In the following, we describe these three energy terms in detail.

Curve-based modelling: E C

While Bezier splines are a popular representation for rotoscoping [START_REF] Agarwala | Keyframe-based tracking for rotoscoping and animation[END_REF][START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF], we simply consider polygonal shapes here: roto-curve X is a polyline with N vertices x 1 . . . x N ∈ Z 2 and N non-intersecting edges e n = (x n , x n+1 ), where x N +1 stands for x 1 , i.e. the curve is closed. Given an orientation convention (e.g. clockwise), the interior of this curve defines a connected subset R(X ) ⊂ P of the image pixel grid (Fig. 3, left), which will be denoted R in short when allowed by the context.

Energy E C is composed of two types of edge potentials ψ loc n and ψ glob n that relate to local and global appearance respectively:

E C (X ; I) := N n=1 [ψ loc n (e n ) + ψ glob n (e n )]. (2)
As with classic active contours [START_REF] Kass | Snakes: Active contour models[END_REF], the first type of potential will encapsulate both a simple 2 -regularizer that penalizes stretching and acts as a curve prior (we are not using secondorder smoothing in the current model), and a data term that encourages the shape to snap to strong edges. It will in addition capture colour likelihood of pixels on each side of each edge via local appearance models. The second set of potentials results from the transformation of object-wise colour statistics (discrete surface integral) into edge-based costs (discrete line integrals). Note that, since we do not impose any constraint on the various potentials, the one specified below could be replaced by more sophisticated ones, e.g. using semantic edges [START_REF] Dollar | Sketch tokens: A learned mid-level representation for contour and object detection[END_REF] instead of intensity gradients, or using statistics of convolutional features [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] rather than colour for local and global appearance modelling.

Local appearance model. Each edge e n is equipped with a local appearance model p n = (p f n , p b n ) composed of a fg/bg colour distribution and of a rectangular support R n , with the edge as medial axis and a fixed width in the perpendicular direction (Fig. 3, right). Denoting R in n and R out n the two equal-sized parts of R n that are respectively inside and outside R, we construct a simple edge-based energy term (the smaller, the better) that rewards edge-configurations such that colours in R in n (resp. R out n ) are well explained by model p f n (resp. p g n ) and edge e n is short and goes through high intensity gradients:

ψ loc n (e n ) := - p∈R in n ln p f n (I p ) - p∈R out n ln p b n (I p ) (3) + µ x n+1 -x n 2 - p∈en λ ∇I(p)) 2 ,
with µ and λ two positive parameters.

Global appearance

This discrete region integral can be turned into a discrete contour integral using one form of discrete Green theorem [START_REF] Tang | A discrete version of Green's theorem[END_REF].

Using horizontal line integrals for instance, we get

p∈R ln p b 0 (I p ) p f 0 (I p ) = N n=1 p∈en α n (p)Q(p) :=ψ glob n (en) , (5)
where Q(p) = "q p ln(p b 0 (I p )/p f 0 (I p )) is the discrete line integral over pixels to the left of p on the same row, and α n (p) ∈ {-1, +1} depends on the direction and orientation, relative to curve's interior, of the oriented edge e n In (5), the second sum in r.h.s. is taken over the pixel chain resulting from the discretization of the line segment [x n , x n+1 ] with the final vertex excluded to avoid double-counting. P R(X )

X x n x n+1 e n R in n R out n R n R(X ) y m y m+1

Landmark-based modelling: E L

Our model also makes use of a set Y of M distinctive landmarks y 1 . . . y M ∈ R(X ) detected inside the object of interest. Similarly to pictorial structures [START_REF] Felzenszwalb | Object detection with discriminatively trained partbased models[END_REF], these landmarks form the leaves of a star-shaped graphical model 4with a virtual root-node y 0 . This part of the model is defined by leaf potentials φ m (y m ) and leaf to root potentials ϕ m (y 0 , y m ):

E L (Y; I) := M m=1 φ m (y m ) + M m=1 ϕ m (y 0 , y m ). (6)
Each landmark is associated with a model, e.g. a template or a filter, that allows the computation of a matching cost at any location in the image. The leave potential φ m (y m ) corresponds to the negative matching cost for m-th landmark. The pairwise potentials ϕ m penalize the difference in 2 -norm between the current configuration and the one, Ŷ, estimated in previous frame:

ϕ m (y 0 , y m ) = 1 2 y m -y 0 -ŷm + ŷ0 2 . (7)

Curve-landmarks interaction: E J

The joint energy E J (X , Y) captures correlation between object's outline and object's landmarks. Based on proximity, shape vertices and landmarks can be associated. Let denote n ∼ m the pairing of vertex x n with landmark y m . Energy term E J decomposes over all such pairs as:

E J (X , Y) = n∼m ξ nm (x n , y m ). (8)
For each pair n ∼ m, the interaction potential is defined as:

ξ mn (x n , y m ) = 1 2 x n -y m -µ mn 2 , (9)
where µ mn is the landmark-to-vertex shift vector in the first image.

Using ROAM

Sequential alternating inference. Using ROAM to outline the object of interest in a new image amounts to solving the discrete optimization problem:

min X ,Y E(X , Y; I), (10)
where E is defined by (1) and depends on previous curve/landmarks configuration ( X , Ŷ) through several of its components. Despite this problem could be formulated as an integer linear program, we opt for simpler alternating optimization with exact minimization at each step which converges within a few iterations.

In the first step, we fix the roto-curve X and find the best configuration of landmarks Y using dynamic programming. Exact solution for such a problem can be obtained in two passes, solving exactly

min y0 min y 1:M M m=1 φ m (y m ) + ϕ m (y 0 , y m ) + n∼m ξ mn (x n , y m ) . (11)
Default implementation leads to complexity O(M S 2 ), with S the size of individual landmark state-spaces, i.e. the number of possible pixel positions allowed for each. However, the quadratic form of the pairwise terms allows making it linear in the number of pixels, i.e. O(M S), by resorting to generalized distance transform [START_REF] Felzenszwalb | Object detection with discriminatively trained partbased models[END_REF].

In the second step, we fix the landmarks Y and find the best configuration of contour X . This is a classic first-order active contour problem. Allowing continuous values for nodes coordinates, a gradient descent can be conducted with all nodes being moved simultaneously at each iteration. We prefer the discrete approach, whereby only integral positions are allowed and dynamic programming can be used [START_REF] Amini | Using dynamic programming for solving variational problems in vision[END_REF]. In that formulation, exact global inference is theoretically possible, but with a prohibitive complexity of O(N P 3 ), where P = card(P) is the number of pixels in images. We follow the classic iterative approach that considers only D possible moves ∆x for each node around its current position. For each of the D positions of first node x 1 , Viterbi algorithm provides the best moves of all others in two passes and with complexity O(N D 2 ). Final complexity is thus O(N D 3 ) for each iteration of optimal update of previous contour, solving:

Note that sacrifying optimality of each update, the complexity could even been reduced as much as O(N D) [START_REF] Williams | A fast algorithm for active contours and curvature estimation[END_REF]. Given some initialization for (X , Y), we thus alternate between two exact block-wise inference procedures. This guaranties convergence toward a local minima of joint energy E(X , Y; I). Also, the complexity of each iteration is linear in the number of vertices and landmarks, linear in the number of pixels, and cubic in the small number of allowed moves for a curve's vertex.

Online learning of appearance models. Local fg/bg colour models p n s and global colour model p 0 are GMMs. Given the roto-curve in the initial frame, these GMMs are first learned over region pairs (R in n , R out n )s and (R, P \ R) respectively and subsequently adapted through time using Stauffer and Grimson's classic technique [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF].

Selection and adaption of landmarks. A pool of distinctive landmarks is maintained at each instant. They can be any type of classic interest points. In order to handle texture-less objects, we use maximally stable extremal regions (MSERs) [START_REF] Matas | Robust widebaseline stereo from maximally stable extremal regions. Image and vision computing[END_REF]. Each landmark is associated with a correlation filter whose response over a given image area can be computed very efficiently [START_REF] Henriques | Kernelized correlation filters[END_REF]. At any time, landmarks whose filter response is too ambiguous are deemed insufficiently discriminative and removed from the current pool in the same way tracker loss is monitored in [START_REF] Henriques | Kernelized correlation filters[END_REF]. The collection is re-populated through new detections. Note that correlation filters can be computed over arbitrary features and kernelized [START_REF] Henriques | Kernelized correlation filters[END_REF]; for simplicity, we use just grayscale features without kernel function.

Allowing topology changes. Using a closed curve is crucial to comply with rotoscoping workflows and allows the definition of a rich appearance model. Also, it prevents abrupt changes of topology. While this behavior is overall beneficial (See §5), segmenting a complete articulated 3D object as in Fig. 1 might turn difficult. Roto-artists naturally handle this by using multiple independent roto-curves, one per meaningful part of the object. As an alternative for less professional, more automatic scenarios, we propose to make ROAM benefit from the best of both worlds: standard graph-cut based segmentation [START_REF] Boykov | Interactive graph cuts for optimal boundary and region segmentation of objects in nd images[END_REF], with its superior agility, is used to propose drastic changes to current curve, if relevant. Space-dependent unaries are derived in ad-hoc way from both global and local colour models and combined with classic contrast-dependent spatial regularization. 5 The exact minimizer of this instrumental cost function is obtained through graph-cut (or its dynamic variant for efficiency [START_REF] Kohli | Dynamic graph cuts for efficient inference in Markov random fields[END_REF]) and compared to the binary segmentation associated to the current shape X . At places where the two differ significantly, a modification of current configuration (displacement, removal or addition of vertices) is proposed and accepted if it reduces the energy E(X , Y; I).

Results

We report experimental comparisons that focus on the minimum input scenario: an initial object selection (curve or mask, depending on the tool) is provided to the system and automatic object segmentation is produced in the rest of the shot. 6 We do not consider additional user interactions.

Datasets. We evaluate our approach on the recent CPC rotoscoping dataset [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF]. It contains 9 videos consisting of 60 to 128 frames which represent typical length of shots for rotoscoping. These sequences were annotated by professional artists using standard post-production tools. We also provide qualitative results on shots from the ROTO++ [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF] dataset for which the authors have not released the ground-truth yet, as well as from the VIDEO SNAPCUT dataset [START_REF] Bai | Video Snapcut: Robust video object cutout using localized classifiers[END_REF].

In addition to that, we use the DAVIS video segmentation dataset [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] which comprises 50 challenging sequences with a wide range of difficulties: large occlusions, long-range displacements, non-rigid deformations, camouflaging effects and complex multi-part objects. Let us note that this dataset is intended to benchmark pixel-based video segmentation methods, not rotoscoping tools based on roto-curves.

Evaluation Metrics. We use standard video segmentation evaluation metrics and report the average accuracy, i.e., the proportion of ground-truth pixels that are correctly identified, and the more demanding average intersection-overunion (IoU), i.e., the area of the intersection of ground-truth and extracted objects over the area of their union. We also report runtimes and evolution of IoU as sequences proceed.

Baselines. We compare with several state-of-the-art methods. Our main comparison is with recent approaches that rely on a closed-curve, i.e., CPC [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF] and ROTO++ [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF]. We initialize all methods with the same object and measure their performance over the rest of each sequence. Since ROTO++ requires at least two key-frames to benefit from its online shape model, we report scores with letting the method access the ground-truth of the last frame as well. We also run it with the initial keyframe only, a configuration in which ROTO++ boils down to the Blender planar tracker.

In addition to that, we also compare with two approaches based on pixel-wise labelling: JUMPCUT [START_REF] Fan | JumpCut: Non-successive mask transfer and interpolation for video cutout[END_REF] and VIDEO SNAPCUT [START_REF] Bai | Video Snapcut: Robust video object cutout using localized classifiers[END_REF] as implemented in After Effect ROTOBRUSH and three recent video-segmentation approaches [9, [START_REF] Märki | Bilateral space video segmentation[END_REF][START_REF] Tsai | Video segmentation via object flow[END_REF]. As a naive baseline, we use a combination of a bounding-box tracker [START_REF] Henriques | Kernelized correlation filters[END_REF] and GRABCUT [START_REF] Rother | Grabcut: Interactive foreground extraction using iterated graph cuts[END_REF]. Ablation study. To evaluate the importance of each part of our model, we consider 4 different configurations:

• Baseline: negative gradient with 2 -regularizer;

• Lean: baseline + local appearance model;

• Medium: lean + landmarks;

• Full: medium + automatic re-parametrization and global appearance model; 6 Video results are available at https://youtu.be/ UvO7IacS9pQ Avg. Time (s) / frame Method Accuracy IoU min max avg GCUT [START_REF] Rother | Grabcut: Interactive foreground extraction using iterated graph cuts[END_REF] + KCF [START_REF] Henriques | Kernelized correlation filters[END_REF] .891 .572 0.394 0.875 0.455 AE ROTOBRUSH [START_REF] Bai | Video Snapcut: Robust video object cutout using localized classifiers[END_REF] .992 .895 ---ROTO++(1 keyframe) [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF] .969 .642 --0.108 ROTO++(2 keyframes) [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF] .974

.691 --0.156 CPC [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF] .998 * .975 * ---NLCV [9] .896 * .194 * ---BSVS [START_REF] Märki | Bilateral space video segmentation[END_REF] .991 .872 ---OBJECTFLOW [START_REF] Tsai | Video segmentation via object flow[END_REF] . For all configurations, we used cross-validation (maximizing the mean IoU) on the training fold of the DAVIS dataset to set the parameters and kept them fixed for all experiments.

Quantitative results. The quantitative results for the CPC dataset are summarized in Tab. 1. While average accuracy is quite similar and saturated for all methods, all configurations of ROAM outperform all baselines. In terms of IoU, all versions of ROAM outperform significantly all others with the full configuration being the best. The reason why landmarks ("medium conf.") do not add much to ROAM is that the CPC dataset does not exhibit many large displacements. The CPC method [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF] was evaluated only on the first ten frames of each sequence since their authors have released results only on these frames and not yet their code. Hence, the scores reported in Tab. 1 for CPC are based on partial videos, as opposed to the scores for all the other methods (including ours). When similarly restricted to the first 10 frames, ROAM performs on par with CPC for all the sequences except "drop" sequence. This sequence shows a water drop falling down -a transparent object, making color models (both local and global) useless if not harmful, and exhibits a very smooth round shape. For this sequence, the CPC method [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF] performs better since it uses Bézier curves and relies solely on the strength of the image gradients.

Results for the DAVIS dataset are reported in Tab. 2. While our method is on par with JUMPCUT (pixel-wise labelling), we again significantly outperform ROTO++ by almost 25 IoU points (note that using ROTO++ with only two keyframes is not a typical scenario, however, this shows how complementary our approaches are). Despite [START_REF] Märki | Bilateral space video segmentation[END_REF] is better by 100 and [START_REF] Tsai | Video segmentation via object flow[END_REF] by 17 IoU points on DAVIS, our model outperforms [START_REF] Märki | Bilateral space video segmentation[END_REF] by 80 and [START_REF] Tsai | Video segmentation via object flow[END_REF] by 450 points on the CPC. In other words, our approach should in the worst case be considered on par. However, we would like to stress that [9, [START_REF] Märki | Bilateral space video segmentation[END_REF][START_REF] Tsai | Video segmentation via object flow[END_REF] are not our (main) competitors since all are based on pixel-wise labelling and as such cannot provide the same flexibility for rotoscoping as the closed contour counter- part [START_REF] Li | Roto++: Accelerating professional rotoscoping using shape manifolds[END_REF]. Note, that we could not provide more quantitative comparisons since results/implementations of other methods were not available from the authors. In particular, comparison with the CPC method [START_REF] Lu | Coherent parametric contours for interactive video object segmentation[END_REF] would be interesting since the DAVIS dataset [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] exhibits many large displacements and major topology changes.

Comparing the different configurations of ROAM -local appearance models add 3 points, landmarks 15 and global model with re-parametrization another 5 points -demonstrates the importance of all components of our framework. To examine behaviour of each method in detail, we report IoU score for each frame in Fig. 7, with in addition the effect of varying the size of the displacement space in ROAM (from windows of 3 × 3 to 13 × 13 pixels) represented with a blue shadow. It can be seen that ROAM is more robust in time, not experiencing sudden performance drops as others.

Importance of landmarks and warping. Using alternating optimization has one more benefit. We can use the predicted position of landmarks in the next frame to estimate the transformation between the two and "warp" the contour to the next frame. This allows us to reduce the number of D possible moves of nodes which i) significantly speeds-up the algorithm, ii) allows us to handle large displacement and iii) allows to better control non-rigid deformations.

We have experimented with three settings for warping of contour: using a smoothed optical flow masked as in [START_REF] Pérez-Rúa | Object-guided motion estimation[END_REF], moving each node by averaging the motion of all landmarks connected to given node and robustly estimating similarity transformation with RANSAC from position of landmarks. Table 3 and Fig. 8 show the effect of using robustly estimated similarity transformation from position of landmarks. Global colour models and reparametrization. We investigated the effects of adding reparametrization and global colour models to our framework. The numeric benefits of these elements can be seen in Tab. 2 and qualitative results on the surfer sequence from VIDEO SNAPCUT dataset are provided in Figs. 5 and10. Observe that the local colour models are a powerful way to capture local appearance complexities of an object through a video sequence. However, self-occlusions and articulated motion can cause these models to fail (right arm crossing the right leg of the surfer). Our contour reparametrization allows the efficient handling of this situation. Furthermore, the beneficial effect of the global colour models can be observed in Fig. 10, where the right foot of the surfer is successfully tracked along the whole video. Qualitative results. Result samples on several sequences from DAVIS dataset in Fig. 4 demonstrate the superior robustness of ROAM compared to other approaches when rotoscoping of the first image only is provided as input (and last image as well for ROTO++). Additional results are provided in the supplementary material.

Timing breakdown. Table 4 provides detailed timing breakdown for our algorithm. These timings were obtained on an Intel Xeon 32@3.1GHz CPU machine with 8GB RAM and Nvidia GeForce Titan X GPU. Note that only part of the approach (evaluation of various potentials and dynamic programming) was run on the GPU. In particular, the reparametrization steps could also be easily run on the graphics card, yielding real-time performance.

Convergence. Fig. 9 demonstrates that the alternating optimization described in §4 converges quickly within a few iterations.

Conclusion

We have introduced ROAM, a model to capture the appearance of the object defined by a closed curve. This model is well suited to conduct rotoscoping in video shots, a difficult task of considerable importance in modern production pipelines. We have demonstrated its merit on various competitive benchmarks. Beside its use within a full rotoscoping pipeline, ROAM could also be useful for various forms of object editing that require both accurate enough segmentation of arbitrary objects in videos and tracking through time of part correspondences, e.g. [START_REF] Khan | Image-based material editing[END_REF][START_REF] Rav-Acha | Unwrap mosaics: A new representation for video editing[END_REF]. Due to its flexibility, ROAM can be easily extended; in particular, with the recent ROTO++ and its powerful low-dimensional shape model.

  model. A global appearance model captures image statistics over the object's interior. As such, it also helps pushing the roto-curve closer to the object's boundary, especially when local boundary terms are not able to explain foreground and background reliably. Defining p 0 = (p f 0 , p b 0 ) the global fg/bg colour distribution, the bagof-pixel assumption allows us to define region energy term p∈R ln p b 0 (I p ) p f 0 (I p ).

Figure 3 :

 3 Figure 3: Structure and notations of proposed model. (Left) A simple closed curve X outlines the object region R(X ) in the image plane P. Several landmarks, forming a star-shaped graphical model, are defined in this region. (Right) Each edge e n of the closed polyline defines a region R n that staddles R(X ); each node x n of the polyline is possibly connected to one or several landmarks.

  n + ∆e n ) + ψ glob n (e n + ∆e n ) + m∼n ξ mn (x n + ∆e n , y m ) .

Figure 4 :

 4 Figure 4: Qualitative results on the DAVIS dataset: Comparisons on blackswan and car-roundabout sequences, between (from top to bottom for each sequence): JUMPCUT, ROTOBRUSH, ROTO++ and ROAM.Global colour models and reparametrization. We investigated the effects of adding reparametrization and global colour models to our framework. The numeric benefits of these elements can be seen in Tab. 2 and qualitative results on the surfer sequence from VIDEO SNAPCUT dataset are provided in Figs.5 and 10. Observe that the local colour models are a powerful way to capture local appearance complexities of an object through a video sequence. However, self-occlusions and articulated motion can cause these models to fail (right arm crossing the right leg of the surfer). Our contour reparametrization allows the efficient handling of this situation. Furthermore, the beneficial effect of the global colour models can be observed in Fig.10, where the right foot of the surfer is successfully tracked along the whole

Figure 5 :

 5 Figure 5: Using proposals based on graph-cut: Proposals (in pink) obtained through graph-cut minimization of an instrumental labeling energy using current colour models allows ROAM to monitor and accommodate drastic changes of object's outline (Bottom). Without this mechanism, parts of surrounding water get absorbed in surfer's region, between the leg and the moving arm (Top).

Figure 6 :

 6 Figure 6: Assessing first part of the model. (a) Edge strength only; (b) Global colour model; (c) Edge strength combined with global colour model; (d) With full cost function E C , including local colour modeling, on frame 13 from surfer sequence.

Figure 7 :Figure 8 :

 78 Figure 7: Evolution of IoU for different sequences of the DAVIS dataset. For our method, the blue shadow indicates influence of varying the label space size for each node (set of possible moves in dynamic programming inference).

Figure 9 :

 9 Figure 9: Energy vs. number of iterations on three sequences from the experimental datasets.

Table 1 :

 1 Quantitative evaluation on CPC dataset ( * : partial evaluation only, see text)

	Avg.

Table 2 :

 2 Quantitative comparisons on DAVIS dataset

		Average Accuracy			Average IoU		Time / frame (s)
	Method	Validation set Training set Full set Validation set Training set Full set min max avg
	GRABCUT [25] + Tracker [13]	0.896	0.914	0.907	0.277	0.296	0.289 0.405 0.675 0.461
	JUMPCUT [10]	0.952	0.957	0.956	0.570	0.632	0.616	-	-	-
	AE ROTOBRUSH [3]	0.951	0.942	0.946	0.533	0.479	0.500	-	-	-
	ROTO++ (single keyframe)[18]	0.910	0.922	0.917	0.248	0.310	0.284	-	-0.118
	ROTO++ (two keyframes) [18]	0.925	0.933	0.930	0.335	0.394	0.358	-	-0.312
	NLCV [9]	0.948	0.963	0.957	0.551	0.701	0.641	-	-	-
	BSVS [20]	0.966	0.974	0.971	0.683	0.709	0.665	-	-	-
	OBJECTFLOW [28]	-	-	-	0.600	0.732	0.711	-	-	-
	ROAM: Baseline Conf.	0.930	0.937	0.932	0.358	0.385	0.377 0.017 0.113 0.049
	ROAM: Lean Conf.	0.935	0.937	0.936	0.409	0.417	0.412 0.187 0.641 0.342
	ROAM: Medium Conf.	0.942	0.952	0.948	0.532	0.591	0.564 0.302 1.785 0.746
	ROAM: Full Conf.	0.952	0.956	0.953	0.583	0.624	0.615 0.546 7.952 3.058

Table 3 :

 3 Different types of contour warping for handling long displacements on a subset of sequences of the DAVIS dataset.

		Average Average
	Warping method	Accuracy IoU
	Optical flow	0.878	0.312
	Node projection from landmark tracking 0.906	0.480
	Robust rigid transf. from landmarks	0.934	0.581

Table 4 :

 4 Timing details for full configuration of ROAM.

	Step	Min. Max. Avg.
	DP Contour	0.018 0.113 0.084
	DP Landmarks	0.003 0.072 0.052
	Local models edge terms 0.342 0.671 0.581
	Other terms	0.012 0.015 0.013
	Reparametrization	0.032 7.403 2.226

The use of blue or green screens on set can ease compositing but remains a contrived set-up. Even if accessible, such screens lead to chromakeying and de-spilling operations that are not trivial and are not suited to all foreground elements, thus rotoscoping remains crucial.

Throughout, "video object", or simply "object", is a generic term to designate a scene element of interest and the associated image region in the video.

"Foreground/background" terminology, "fg/bg" in short, merely refers here to inside and outside of the roto-curve; it does not imply that the object stands at the forefront of the 3D scene with a background behind it.

The star shape is used for its simplicity but could be replaced by another tree-shaped structure.

Note that this instrumental energy is too poor to compete on its own with proposed model, but is a good enough proxy for the purpose of proposing possibly interesting new shapes at certain instants. It is also very different from the one in the final graph-cut of VIDEO SNAPCUT where unaries are based on the already computed soft segmentation to turn it into a hard segmentation. Also, graph-cut segmentation is the final output in VIDEO SNAPCUT, unless further interaction is used, while we only use it to explore alternative topologies under the control of our joint energy model.

This work was supported by Technicolor, EPSRC, ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant Seebibyte EP/M013774/1, EPSRC/-MURI grant EP/N019474/1, ANRT and Cifre convention No. 2014/1041.


Paul Holmes

Claire Calmels

Mental practice: neuroscientific support for a new approach

Introduction

Mental practice techniques continue to be popular for performers in sport, exercise, dance and clinical settings. Typically, the techniques form part of a psychological skills training programme that supports physical or rehabilitative practice, the goal normally being to enhance or maintain performance of a skill or task. Mental practice techniques can take many forms and can focus on different aspects ofanindividual'smentalprofile.Forexample, whileone athlete may feel the need to reduce pre-competition worry, others may need to improve attention and concentration skills or experience greater confidence prior to an important event. Applied sport psychology literature has always supported the inclusionofsome form of mental practice regimeina performer's training programme.

One technique that persists in applied sport and exercise psychology is imagery (e.g. [START_REF] Morris | Imagery in sport[END_REF][START_REF] Murphy | Imagery in sport, exercise and dance[END_REF]. Imagery has received considerable academic attention and, according to Short et al (2002), remains one of the most popular intervention techniques delivered by sport psychologists. It has many reported uses: visual rehearsal of a skill; skill learning; strategy development; enhancing confidence; and improving recovery from injury are just a few. Therefore, given the comprehensive research, the prevalence of its use and its perceived importance within the mental skills 'toolkit', it should be expected that there would be detailed and agreed procedures for the delivery of effective imagery interventions. Unfortunately, this does not seem to be the case. [START_REF] Callow | A critical analysis of applied imagery research[END_REF] have recently provided a critical analysis of applied imagery research. In this, they report what is known, what might be known and what is not known about effective imagery interventions. The authors raise a number of important research questions from the review that suggest that there is still a considerable way to go before imagery can live up to its claim to be the central pillar of sport psychology interventions.

While many individuals attribute their performance to their structured mental practice regimes, a detailed understanding of what practice comprises during imagery remains very much an illusion. By its very nature, mental practice is covert and indefinite; the most detailed, prescriptive imagery programmes cannot predict what performers actually generate in their brains. It is not disputed that imagery and associated mental practice techniques offer benefit to performers through a number of psychological and/or psychophysiological mechanisms. Theories that have received support include: [START_REF] Lang | Imagery in therapy: an information processing analysis of fear[END_REF] bioinformational theory, [START_REF] Ahsen | ISM: the triplecode model for imagery and psychophysiology[END_REF] triple code theory, [START_REF] Schmidt | Motor control and learning: A behavioral emphasis[END_REF] attention-arousal set theory, Bandura's (1977a, b) selfefficacy/self-confidence theories, [START_REF] Paivio | Cognitive and motivational functions of imagery in human performance[END_REF] motivational theory and, more recently, [START_REF] Jeannerod | The representing brain. Neural correlates of motor intention and imagery[END_REF] simulation theories. As with many theoretical debates, there are similarities across these theories and, as you would expect, there is evidence that imagery advocates. We will introduce neuroscientific research to provide an evidence base to support the development of our observation model.

Analysis of imagery for mental practice interventions

Defining imagery and observation would support all the ideas. However, 'none of the theories... have sufficient research to support them as definitive theories of imagery functioning' (Morris et al 2005, p. 55). As a consequence of these ongoing debates, some authors have suggested that imagery cannot be explained easily by one theory alone and that progress will only be made through the integration of approaches. This idea has also been advocated by [START_REF] Murphy | Imagery in sport, exercise and dance[END_REF]. They suggest that imagery investigations should be guided by a new comprehensive model. We would certainly support such ventures but, given the well documented methodological and operational concerns that will always remain with covert techniques, maybe the time has come to use what we know, develop what we might know and explore what we don't know with a different, more overt approach to mental training.

It is our opinion that imagery and associated mental practice techniques will continue to be useful interventions for performers but that their effectiveness will always be compromised by the brain's ability to (re)create an experience that mimics a real experience. Further, the moderators and mediators of imagery effectiveness, highlighted by [START_REF] Callow | A critical analysis of applied imagery research[END_REF], remain important aspects of research in this area. They include imagery perspective, movement agency, task type, practice conditions and timing. In addition, imager skill expertise, imaging ability, skill confidence and arousal state should also be considered (the reader is referred to [START_REF] Callow | A critical analysis of applied imagery research[END_REF] for further consideration of some of these variables). Without a greater understanding and control of these and other variables, predictions and applications from the imagery research should continue to be made cautiously.

In this regard, we propose that the best elements of imagery research should be retained and integrated into a new working model of observation for mental practice. We will highlight some of our fundamental concerns with imagery as a sport psychology intervention and suggest that observation-based techniques may be better positioned to meet the goals that In order to develop the concept of observation as a valid technique that may operate alongside, or in some cases replace, conventional imagery-type interventions, it is important to define clearly what we mean by the two processes; this is not easy. [START_REF] Morris | Imagery in sport[END_REF] have highlighted the problems with trying to define the imagery process in a sporting context; they suggest that there is a lack of consistency in the features that constitute the process, stating 'the focus of each definition seems to vary depending on the purpose for which the imagery description is used' (Morris et al 2005, p. 14). Similar problems are evident in the more sparse definitions for observation (see McCullagh & Weiss 2001 p. 221-222). To reflect the neuroscientific focus of our paper and the underlying assumption in some theories that imagery and observation share at least some brain activity with the corresponding physical execution, it would seem appropriate to include reference to this within our working definitions.

Imagery, in the context of performance, may be considered as the neural activation of a brain representation/neural network involving primarily top-down sensorial, perceptual and affective characteristics that are normally under the conscious control of the imager and which may occur in the absence of perceptual afference functionally equivalent to the actual physical experience.

Holmes & Calmels 2008, p. 433 We contrast the definition of imagery with that of observation.

Observation, in the context of performance, may be considered as the neural activation of a brain representation/neural network involving primarily bottom-up sensorial, perceptual and affective characteristics, that are primarily under the subconscious control of the observer and which normally occur in the presence of perceptual afference functionally equivalent to the actual physical experience.

Imagery is, therefore, a top-down, knowledge-driven process whereas observation is a bottom up, perceptdriven process. What is of interest here is how much these processes truly reflect the neural activity associated with overt behaviours, their ability to influence future behaviour, and how they can be used effectively in performance contexts.

Much of what follows in our discussions will relate to the terminology used in these definitions. We will provide evidence to support a distinction between the processes of imagery and observation in relation to the control afforded to the practitioner and the performer over some of the moderating variables identified above. In this regard, we will also propose that observation offers greater efficacy in terms of access to functional cortical and subcortical neural networks shared with overt behaviour. We do not propose that imagery and observation are in direct competition as interventions, as they too share, in part, similar brain sites. Common sense dictates that there will be times when one technique is more appropriate than another; for example, immediately prior to a performance, imagery may offer far greater flexibility of use than observation simply because the process does not require equipment (i.e. camera, screen). In contrast, during training sessions, observation delivered through portable digital media may be more effective. Therefore, we see their roles as separate although occasionally complementary if there is consideration of some important procedural concerns. Imagery and observation both have the potential to support permanent structural changes in the brain but it is the extent of the neuronal modulation offered by each approach that will be considered here.

Our investigation of brain substrates for observation will, at this stage, join the pre-reflective simulation camp and focus on shared circuits; that is, those neural networks that are common to both physical execution of an action and also the observation of the same action. These shared circuits are involved in our intuitive actions, sensations and emotions and in the perception of these behaviours in others. The concept of shared circuits may be a better term than functional equivalence for understanding the neuroscience of imagery and observation, since there is increasing evidence that neural overlap is inexact and incomplete. Given the variability in methods of delivery of the interventions, this 'sharedness' will also be variable. Functional equivalence implies a more comprehensive neuronal matching that cannot, at present, be substantiated theoretically or empirically.

The term should be restricted to the matching of behaviours and environments that attempt to achieve shared circuits.

Some concerns with imagery

If observation is to provide a valid alternative to imagery for performance training, the theoretical justification to explain its effectiveness must be strong and address the methodological concerns observed for performance imagery. In a recent paper, summarized in Figure 16.1, we suggest that there are a number of procedural problems when considering imagery as an intervention. These include: • image generation, maintenance and transformation ability (i.e. can the performer actually 'create' an image and, if so, for how long, and how well are they able to manipulate the content?); • control of visual perspective and viewing angle (i.e. whether the imager uses a first person, third person, or combination of visual perspectives and what angle the imager takes for viewing the behaviour); • behaviour agency (i.e. is the imaged behaviour associated with the self or another?); and • image modalities (i.e. which senses are represented within the image, is it purely visual? are there concurrent sounds? does the imager 'feel' movements either concurrently with the visual image or independent of it?). Furthermore, for many performers, practitioners and coaches, it is generally accepted that the practice of imagery is performed with the eyes closed while inhibiting overt movement. These actions directly compromise the shared circuit with physical performance; eye closing significantly increases alpha-band synchronization over the posterior cortex as visual information is reduced. Similarly, as a consequence of the motor inhibition associated with imagery, activity is seen in the posterior cerebellum. These neural profiles are not observed for the physical action. We have already tried to address some of these issues through the use of a mnemonic for effective mental practice usage -PETTLEP [START_REF] Holmes | The PETTLEP approach to motor imagery: a functional equivalence model for sport psychologists[END_REF].

However imagery is approached there will be some inherent problems, many of which cannot be addressed. The performer may be provided with extensive and detailed scripts in an attempt to direct the exact content of the image but the resultant image will always be unknown, even with comprehensive debriefs. Given this lack of control over image content, practitioners should look to more manageable interventions. Therefore, we aim to provide a neuroscientific approach to developing a working model for observation as a more valid intervention. With the modern technology available to performers and support teams, we propose that many of the concerns highlighted above can be addressed. In addition, we have also found some associated procedural benefits for observation interventions in comparison to imagery by adopting this approach; these include improved adherence to observation interventions, greater mental skills training time, improved imagery ability, and direct involvement in managing and progressing the content of the intervention.

Observation as a mental practice intervention for performance A mechanism to support the observation process

The mechanisms that underlie the process of imagery are not fully understood; they are complicated further by the lack of control over the covert behaviours. However, the recent discovery of 'mirror neurons' has provided some evidence to begin to explain observation-based behaviours. Early work suggested that mirror neurons (MNs) may be capable of encoding not just the goal of the movement but also the characteristics and meaning behind the movement; specifically action and intention understanding. MNs have also been suggested to have an important role in imitation. While there are limits to how much these neurons can explain, and with a number of functions still limited to the primate brain, their discovery does provide an insight into a mechanism by which humans may communicate their actions, desires and feelings.

MNs were first discovered in the ventral premotor cortex of the macaque monkey with single neuron recordings (e.g. [START_REF] Pellegrino | Understanding motor events: a neurophysiological study[END_REF][START_REF] Rizzolatti | Functional organization of inferior area 6 in the macaque monkey: II. Area F5 and the control of distal movements[END_REF]. These visuomotor neurons showed special characteristics. They fired when the monkey executed a goal-directed hand movement and also when it observed the same action executed by another monkey or by a human. Some evidence for the existence of similar neurons in humans has been provided by neurophysiological (i.e. electroencephalographic and magnetoencephalographic techniques) and brain imaging studies, which suggests that MNs may contribute to a shared circuit or observationexecution matching system (Calmels et al 2006a, b). This system, also known as the motor resonance system or mirror neuron system (MNS), supports the proposal that perception of an action activates a brain representation similar but not identical to that used to perform the action, in effect, a 'shared' neural circuit [START_REF] Grèzes | Activations related to 'mirror' and 'canonical' neurones in the human brain: an fMRI study[END_REF].

Characteristics of the MNS in humans

In humans, the MNS shows differential activation depending upon the forms of observed motor behaviours. For example, viewing a grasping action performed by another human activates the MNS more than viewing the same action performed by a non-biological, robotic model. Further, the MNS shows less activity during observation of objectless actions, where movements are mimicked, compared to the identical movement that includes the object of interest. Therefore, the MNS would be expected to be more active while viewing a thumb and finger grasp of a golf ball than the replica movement without the golf ball present; an important practical consideration for demonstrators. In a similar way, observation of non-object-related actions, so-called intransitive actions, also activates the MNS.

The MNS is only minimally active when the observed action is impossible for the observer to perform and, similarly, only limited activation is seen in the MNS when the observed movements do not belong to the observer's motor repertoire. For example, strong activation of premotor cortex, parietal cortex and superior temporal sulcus (a shallow cleft at the top of the temporal area) has been recorded in the brains of experienced dancers when observing dance movements they possess the motor capabilities to perform. This brain activity contrasts with that recorded in the same dancers observing dance movements not previously performed before, even though the movements are visually familiar to them [START_REF] Calvo-Merino | Seeing or doing? Influence of visual and motor familiarity in action observation[END_REF]. The moderating influences of motor familiarity and task expertise interaction on the involvement of the MNS are of obvious importance to the development of the applied use of observation. They will be discussed in more detail throughout the chapter.

Functions of the MNS in humans

The mirror system has been linked to four main functional roles in humans: action and intention understanding, imitation and empathy [START_REF] Rizzolatti | The mirror neuron system and its function in humans[END_REF]. Simulation theorists would suggest that the involvement of the motor system is a necessary requirement to understand fully an observed action; the perception of an action without the involvement of the motor system only providing a superficial description of the action and not allowing for its thorough comprehension. Intention understanding (i.e. why an individual is performing a particular action) has also been linked to activation of the MNS. The intention of a movement should not be confused with the movement goal. For example, one could observe a tennis player who makes a movement to pick up a ball from the court; the movement goal. However, the intention of the movement is not known at this stage; it may be to use that specific ball for the next serve, reject it or put it in a pocket for a possible second serve. Observing actions in different contexts will allow the observer to infer intentions for these actions, a concept not dissimilar to variable practice. We will discuss observation context in later sections.

MNS activation is also implied during motor imitation. Imitation of an action that belongs to the motor repertoire of the observer includes activity in the neural circuitry of superior temporal sulcus and frontal and parietal mirror areas [START_REF] Iacoboni | Neural mechanisms of imitation[END_REF]. In contrast, imitation of a novel action activates the same neural circuitry and, additionally, Brodmann's area 46, an area suggested to be associated with the selection of appropriate motor acts [START_REF] Buccino | Neural circuits underlying imitation learning of hand actions: an event-related fMRI study[END_REF].

Finally, the MNS seems to be closely associated with human empathy, the capacity to feel and understand the same emotional states experienced by another. It is well known that observing a person laughing or crying can generate a similar emotional state in the observer. In such observation situations, neural activity is present in the insula and limbic system of the observer in a similar way to actually experiencing the real emotional behaviour. Since human performance is frequently associated with emotive behaviours and environments, for example, winning and losing in front of large crowds of spectators, understanding how observation interventions can access and share neural circuitry linked to these emotive contexts to generate empathy more optimally would seem beneficial.

Factors that impact on observation

The following section will consider some of the task and observation factors and individual characteristics that can be identified from the neuroscience research that impact directly on observation modulation.

Task type (form-based tasks and perceptually driven tasks) and general observation factors

In closed skill, form-based tasks, movement aesthetics and quality are important for successful performance (e.g. dance, gymnastics and diving). Performing these kinds of task requires a precise body segment organization. For example, gymnasts who do not reproduce an exact form of their routine have penalties imposed on their score by the judging panel. This form of task contrasts with more open, perceptually driven tasks (e.g. tennis, soccer or wrestling). For example, a tennis player will be judged on her ability to win points against an opponent and not on the form or aesthetic quality of her shot in doing so. Therefore, while it is useful to have a model for form-based tasks, and it is easy to see how observation-based interventions can provide the relevant information, perhaps it would also be useful to observe a model for perceptually driven tasks.

The imagery literature on this issue has identified that performance of closed skills may benefit more from imagery because of the predictability of the movement patterns without direct, less controllable opposition. The imagery research also proposes that task type may direct the choice of visual perspective. Adopting a third-person visual perspective, as if seeing oneself on television (the optimal viewing angle for this perspective has not been specified, and is also likely to be task-specific) has been shown to enhance performance of form-based tasks in novice athletes, probably because of the additional relative visual cues available. This contrasts with more perceptual tasks, where a first-person visual perspective (as if looking through one's own eyes) seems to be more beneficial, possibly because of the kinesthesis associated with this visual perspective for these tasks.

Research is ongoing to explore further these taskperspective distinctions. For example, there seems to be a task-visual perspective/modality-motor capability interaction for imagery effectiveness. Once motoric ability for a closed-skill, form-based task has been acquired, performers are able to use thirdperson perspective visual imagery and kinesthesis to enhance future performance.

These findings are important and should direct the development of observation interventions. For form-based, closed skills the observed movements will, without any manipulation, provide a thirdperson visual perspective (e.g. viewing oneself or another performing via DVD). To extend the perspective issue, the observation intervention should provide multiple angles of the behaviour of interest. Where the observer has motoric capability, the instructions for observation should include reference to kinesthetic experiences during the viewing. For tasks that rely on perceptual information the process is more complex and requires us to expand on some of the terminology used in this field.

Three separate image characteristics have tended to be compounded: image perspective, image agency and image modality. There is now considerable evidence that these characteristics show different and separate patterns of cortical and subcortical activity both within and across variables. For example, in imagery conditions, employing a different visual

C H A P T E R 1 6
perspective involves different parts of the cortex: the right inferior parietal, precuneus (posteromedial portion of the parietal lobe) and somatosensory cortex. These areas have been found to distinguish selfproduced actions from those generated by others. The language used in the sport psychology literature does not help the issue: 'internal' imagery, for example, is typically a combination of a first-person visual perspective and a kinesthetic modality. It is also used synonymously for just first-person visual imagery, kinesthetic imagery and motor imagery in some literature. The ambiguity raises obvious concerns for the user and, in the case of the former definition, assumes imagery ability in two modalities. However, if the performer is less skilled in kinesthetic imagery and is instructed to use this 'internal' approach, the content of the imagery will only be visual and, therefore, hypothetically, sharing a limited circuitry with motor areas. While perspective, agency and modality factors may be related and share some neural properties, they are not the same. Their conflation can complicate the delivery of imagery for the practitioner and confuse the recipient about what s/he is required to image. Observation, therefore, may be better positioned to support perceptual tasks. Control of visual perspective, behaviour agency and modality should be effective, since observation interventions overcome the language ambiguity that currently exists in imagery. The ability to (re)create and control variable practice environments through edited filming can also address the need to provide multiple training environments for performers.

Internally generated movement versus externally triggered movements

In the neuroscience literature, it has been shown that internally generated movements (i.e. movements that can be initiated by a performer) do not involve the same cortical pathways as externally triggered movements (i.e. movements that can be imposed on a performer by some external agent). The findings of van Donkelaar et al (1999) suggest that the former engages the ganglio-thalamocortical pathway whereas the second engages the cerebellothalamocortical pathway.

This point is of interest because in performance both movement types exist. For example, reacting to a gun to start a race or performing to a tightly choreographed piece of music contrast with self-selected movements. In imagery interventions, however, it could be argued that both types of movement become internally generated through the more reflective and conscious process. Again, observation may offer a more ecologically valid environment to address these tasks. However, the observed actions need to access the shared neuronal circuitry for the task for the intervention to be valid. Often, in anticipation-coincidence tasks, coaches and educators use internally generated movements to teach athletes externally generated movements; this is true for the physical execution condition as well as the observation condition. For example, in baseball practice, a novice performer may execute and observe internally generated movement towards a motionless ball and also movements without the ball before progressing to perform dissimilar, externally generated movements linked to a moving ball. Pedagogically, this progression may seem to be full of common sense and practical for the instructor. However, the cortical activation associated with the procedure does not concur. Although movements made without the inclusion of the object of interest seem identical, they actually require the involvement of quite different neuronal populations. Therefore, the transfer of learning from one movement style to another may not be systematic and may explain some of the performance decrements at transition phases. (This same point is well made, albeit from a different perspective, in Ch. 14.)

Specular imitation versus anatomical imitation

In the context of performance, imitation refers to 'a copying of the feature of the bodily movement of a model by an observer' (Heyes 2001, p. 254). This activity represents a fundamental part of human behaviour, is used to acquire new skills and is part of social interaction with other individuals. It has also been identified as one of the key roles of the MNS. Imitation can take different forms and two are particularly relevant for this chapter: specular imitation and anatomical imitation. Specular imitation describes a situation where the observational behaviour occurs as if looking into a mirror. For example, when the model moves the left hand, the observer moves the right hand. In contrast, anatomical imitation describes a state where the imitated movement is reproduced in the same hand as the observed model: when the observed model moves the left hand, the observer moves the left hand. Both could be adopted using a third-or first-person visual perspective. Specular imitation may be considered a more natural behaviour than anatomical imitation and there is evidence that specular responses predominate over non-specular responses until about 10 years of age (Wapner & Cirillo 1968).

The contralateral action associated with specular observation may predominate to reduce the image rotation ability required to anatomically match the observed action. MN research has also revealed that specular imitation activates the mirror neuron system more than anatomical imitation [START_REF] Koski | Modulation of cortical activity during different imitative behaviors[END_REF] and this could explain the greater frequency with which individuals imitate others' behaviours and actions in a specular way.

In applied observation settings, coaches and educators should be aware of these imitation processes. For example, when artistic gymnasts are involved in a choreographic session, often the gymnasts will be facing a large mirror displaying not only their movements but also most of the environment in which they are performing. Instructors can either face the observers or face the mirror; thereby providing both specular and anatomical perspectives. Simultaneously, gymnasts can observe themselves in the mirror with a specular configuration.

MNS activation has also been shown to be related to the physical position of the person being observed [START_REF] Kilner | Modulation of the mirror system by social relevance[END_REF]. When the observed person faces towards the observer, the MNS is actively involved. The authors observed, in the parietal area of the brain, a reduction in the alpha band activity during action observation. This reduction was also seen during the execution of the same action. In contrast, however, when the observed person faced away from the observer, no electroencephalographic attenuation was detected. While some of this activity may be a consequence of the facial interaction, it may be that specular observation is more effective at eliciting similar activity in the MNS.

Live observation versus video-based observation

In this section, we will discuss the use of live and video-based observations. We will also include discussion relating to observation perspectives and agency, since we believe the concepts are linked. A question that could legitimately be posed is whether the observation of videotapes or live performance benefits the performer more. The answer is not easy, as performance depends on many factors. However, what is known is that stronger primary motor cortex activation has been detected during live hand movements compared to viewing of video and that this higher activation may be due to the greater ecological validity and contextual meaning of live movements [START_REF] Jarvelainen | Stronger reactivity of the human primary motor cortex during observation of live rather than video motor acts[END_REF].

In applied settings, coaches and educators frequently invite their performers to observe teammates performing live motor skills, normally to facilitate the learning or execution of motor performance. Similarly, performers spontaneously observe their teammates during training sessions, mainly to improve their own performance. We have found that elite young female gymnasts observe peers mainly to increase engagement and activation and to improve performance of technical execution by detecting and correcting technical errors performed by others.

Video sequences taken during training sessions or competitions can also be presented to individuals. Performers can observe their own behaviours and actions or teammates' performances. Just as for imagery, these records of performance can be used for many purposes: instant review, pre-performance preview and stimulated recall are just a few. Therefore, coaches and educators may find the use of video sequences more beneficial and practical than live performance. Video permits an unlimited number of viewings of edited, successful performances and allows for a decreased physical training load. The visual content can also be controlled by the coach, in terms of perspective, agency and other modalities, to a far greater extent than for imagery interventions.

Observation of live actions is frequently about actions performed by others. Observation agency through video, however, can include actions performed by oneself or by others. Agency can be differentiated by different neural correlates. [START_REF] Knoblich | Predicting the effects of actions: interactions of perception and action[END_REF] have found behavioural differences between the observation of one's own performance and the observation of others' performance. They presented participants with videos of dart-throwing actions that they had previously performed. They were also shown the same action performed by others. Unsurprisingly, the results indicated that prediction of accuracy of the observed actions was greater when participants observed their own actions. According to the shared circuit theory, observation of self-generated actions may be more informative because of the functional similarity of the neural activity to that during motor execution.

As with agency, different observational perspectives also show different neurological profiles. Using a technique known as transcranial magnetic C H A P T E R 1 6 stimulation to modulate neural activity, researchers have shown that action observation enhances corticospinal excitability; the motor evoked potential observed in a muscle associated with the observed movement is of greater amplitude. Observation of a hand movement increased functional motor output and the degree of motor evoked potential modulation was maximal when the observed action was presented from a first-person perspective. It is also known that the extrastriate body area, an area that responds to the visual appearance of the human body, distinguishes an egocentric view of the self and other people from an allocentric, third-person visual perspective; extrastriate body area activity has been found to increase significantly for allocentric relative to egocentric views in the right hemisphere.

Still and temporally altered motion

As video sequences progress, the images can be slowed, sped up or paused to allow observation of slow or temporally altered actions. While temporally incongruent observation challenges the shared circuit theory, which has both spatial and temporal characteristics, observation of still action shots should not be ruled out completely. There is strong evidence that observation of photographs and still video frames that imply motion can also activate cortical MNS areas involved in the analysis of movement sequences [START_REF] Kourtzi | Activation in human MT/MST by static images with implied motion[END_REF]. While we are aware that slow-motion video is frequently used in performance contexts, especially for technique analysis, its neural benefit remains untested. Similarly, increasing the speed of the viewed image has been proposed as a possible observation intervention to improve performance. With the technology now available to support such interventions, well designed studies would be welcome in this area.

Instructions provided prior to the observation process

The observational neural profile is also sensitive to the instructions that are provided prior to the observation process. Participants can be invited to observe a movement with the purpose of later imitation, of recognizing it or with no specific goal. Each of these instructional sets has been shown to significantly change the observational neural profile. For example, [START_REF] Decety | Brain activity during observation of actions: influence of action content and subject's strategy[END_REF] showed that the dorsolateral prefrontal cortex and the presupplementary motor area were activated when participants were provided with instructions to observe a movement with a later requirement to imitate it. In contrast, the right parahippocampal gyrus was activated in a situation where there was a requirement to recognize the movement after its observation. Coaches and educators should be aware that clear, unambiguous instructions relating to the observation content may be important to optimize the theoretical neural equivalence with the executed profile. When we work with performers, weareveryspecificinaskingthemtoobservetounderstand the actions, the goals of the actions, the intentions of the model and, where appropriate, to imitate the action or to empathize with the behaviour.

The observation context

Neuroscience is not purely cognitive. Where human performers are acting as agents of behaviour, the social impact will, inevitably, lead to affective reactions. Therefore, the MNS, in contextually relevant situations, is likely to also include cognitive and emotional neural systems.

An action, performed in different psychosocial contexts, can have very different meanings and intentions to the same observer. Similarly, two observers may have quite different neural responses to the same observed action.

MN areas in the inferior frontal cortex are activated more strongly when individuals observe an action embedded in a particular context compared to observation of the same action in a non-contextual environment. Neuroscience studies are beginning to explore the influence of context in observation. For example, grasping hand actions, seen in different contexts, allow an observer to infer different intentions for the actions. A grasping hand action associated with a cup embedded in a 'before tea' context and an 'after tea' context provides the observer with a different intentional understanding: to drink the contents of the cup or tidy away the cup from the table [START_REF] Iacoboni | Grasping the intentions of others with one's own mirror neuron system[END_REF]. Therefore, it will be important to develop observation interventions that have contextual meaning for the observer and actions may need to be embedded within performance contexts to increase shared circuit activity.

Contextual congruence may also need to be considered alongside the observation viewing environment; recent evidence from the imagery research suggests that this may be true. While the viewed action offers contextual relevance, it is possible that the neural response will be influenced by the context in which the observation is performed. For example, [START_REF] Chatterton | Observation of meaningful activities: a case study of a personalized intervention on poststroke functional state[END_REF] have found evidence for greater action and intention understanding, empathy and sensorimotor engagement to observation performed in training environments compared to home environments. While interesting, these findings are still at the case study stage and will require more detailed empirical research to develop the observation context debate. To be effective, observation interventions cannot just provide action viewing; they must activate social, cognitive, affective neural systems. Understanding and developing performance context will be essential to the effectiveness of the intervention.

Observer motor expertise

Using models whose motor skill expertise is similar to that of the observer has been recommended to promote more optimal motor representation access. More specifically, [START_REF] Calvo-Merino | Action observation and acquired motor skills: an fMRI study with expert dancers[END_REF] have shown stronger activation of the MNS for expert classical ballet dancers compared to novice dancers when observing professional classical dance movements. They also recorded stronger activation of the MNS when the expert ballet dancers viewed movements for which they had been trained compared to movements they had not (e.g. classical ballet vs capoiera). The modulation of the MNS as a result of motor competence may, therefore, be closely associated with visual and motor familiarity of the action. The same research group studied male and female expert classical dancers who had trained together for the same period of time, suggesting that they possessed a similar visual familiarity with the ballet moves. However, some of the moves were specific to one sex and had, therefore, not been physically trained by the other gender group. Activity in the MNS was increased only for observation of gender-specific ballet moves in contrast to familiar, but untrained ballet moves. To be optimally effective therefore, an observation session requires the observer to possess a visual and motor familiarity with the observed action; just because a coach shows a video of a double back somersault to a novice gymnast several times does not mean that the gymnast will then be able to use the (audio) visual information concerning the movement's goals and intentions to be able to perform it! The gymnast will require some motor capabilities to integrate with the visual percepts from the video. Observation should, therefore, be considered carefully alongside the maturational progression of the performer and introduced as their motor capabilities develop. Just as with some of the imagery research, there is evidence that only skilled performers are able to use the third-person visual perspective information to support concurrent kinesthetic proprioception. The sensorimotor integration of the MNS requires physical capability acquired through physical experiences for observation to be most effective in accessing a more elaborate shared circuitry. Conducting some form of observational skill analysis prior to employing observation as an intervention is certainly recommended. Imagery ability questionnaires are regularly used to assess imaging skill; the same should be done for observation.

Observer gender

Until recently, observer gender was not considered as a moderating variable of the neural profile. Recently, however, it has been shown that the MNS is sensitive to sex differences [START_REF] Cheng | Gender differences in the human mirror system: a magnetoencephalography study[END_REF]. There is evidence that female observers show a stronger MNS activation in the right primary motor cortex than their male counterparts when viewing sensorimotor scenes. This finding has been interpreted as a marker of greater empathy by females. The research has not been conducted in sporting or performance contexts and may, therefore, be of limited applied value at this stage.

Applied perspectives

Coaches or educators planning to use observation as a mental practice intervention should be aware that a number of factors can have an impact on observation effectiveness. Figure 16.2 displays a list of primary components to help coaches and educators when devising observation sessions. Support staff should make clear choices in the delivery of the observation factors. The choices will be dictated by the observed task characteristics, the individual observer characteristics and the functions of the observation intervention. Components of tasks and individual characteristics, except motivation, will not be developed here since they have been discussed previously.

Coaches or educators should consider the motivation of their athletes to adhere to observation interventions; athletes can observe and remember a modelled skill, they can have the physical abilities to perform that skill but, if they are not motivated, observation sessions will not be effective.

Understanding the functions of observation is also important when devising observation sessions since it helps the coaches or psychologist define appropriate (2007) showed that the content of observation is linked clearly to its function. For example, observation of one portion of a complex gymnastics movement and only one part of the body resulted in the performers detecting and correcting technical errors. Whereas, when the same gymnasts observed the full movement but again for only one body part, they used the observation to check their self-assessment against the analysis of the coach. The same kind of relationship has also been highlighted by [START_REF] Fournier | Content, characteristics and function of mental images[END_REF] for mental imagery. Therefore, coaches should consider the observation function when using the technique. For example, in a form-based task, such as a floor routine in gymnastics, the observation function could be to increase performance or to increase engagement in the task. The observation content, the pre-observation instructions and the (temporally altered) motion are, therefore, likely to be different. If the observation function is to increase performance, coaches could choose a third-person visual perspective, invite the performer to observe their own behaviour at a slow rate with paused images followed by a normal rate and then ask the performer to focus their attention on a particular point of the movement. If the observation function is to increase engagement, the choices may be different. The visual perspective could still be third-person; however, the observed model may be a respected other and the video could be observed at a normal rate. Coaches should use Figure 16.2 as a reminder and individualize the observation sessions for each performer just as they would for imagery interventions.

If mental practice techniques propose to offer, at best, degraded versions of physical practice, there will always be reduced neural afference to the sensorimotor network. Therefore, one could legitimately propose that training time may be spent more effectively in physically training. In an ideal performance world, the answer is probably 'Yes, it could'. However, the world is not ideal and the mental practice techniques described here allow the performer to continue to reinforce shared circuits outside of physical practice conditions. A number of advantages are proposed. First, using observation within a learning framework to support more traditional physical training sessions could be a useful addition to the holistic training environment. Training supported in this way can remain positive and reduce physical training loads, fatigue and, potentially, injury. Second, observing third-person perspective parts of team plays, such as offensive and defensive strategies or exchange of actions between two opponents, allows athletes to anticipate the actions of others (i.e. opponents or partners) more effectively than just first-person physical training for the skill. If the MNS is concerned fundamentally with movement prediction, then where and how others move has obvious relevance for sporting interactions. Understanding others' actions in terms of movement kinematics allows performers to make predictions about their behaviour goals; performers can infer the intentions behind movements and judge whether movements are intended or not. In these cases, the mirror neuron system could be depicted as the neural substrate of our capacity to understand an action, an intention or the state of mind of others. Finally, in sports rehabilitation, athletes may recover more quickly after viewing diverse and repeated sport video sequences. Observation of relevant performance sequences could allow cortical structural changes, reorganization and reinforcement in the motor architecture to support more physically based therapies. These ideas remain to be tested empirically in sporting contexts.

Conclusion

It isouropinionthatobservationmayofferamorevalid mental practice technique than more traditional covert interventions. More specifically, the modifiers of the neural activity during imagery are better controlled through an observation process than through imagery. The MNS offers a theoretical mechanism, and the reviewed research provides some evidence, to support theapplication of observation-based interventions in sport and performance environments. We propose that observation affords both practitioners and researchers greater control over some of the fundamental characteristics to optimize the abstract concept of shared neural circuitry with overt behaviour. Imagery also has the potential to achieve this outcome, but with less assurance that it will be conducted in the prescribed way by the athlete.

Generating an image is no longer a problem since the visual percepts are provided by the information displayed, typically through individualized DVDs, and observed with the eyes open. Observation of still photographic action shots can also generate functional neural activity, since the percepts imply motion. Video observation controls for the imagery ability factors of clarity, vividness and image management (maintenance and rate of exposure) because they are manipulated through the filming and editing processes. Image transformations and rotations are also more effectively managed through dynamic use of camera angles. Observation, provided through digital video, is able to offer the viewer every conceivable viewing angle from either first-or thirdperson visual perspective and avoids the need for the performer to transform or rotate a transient image. The major benefit, however, is the 'shared' observational image that really differentiates observation from imagery; the researcher or practitioner no longer has to accept the debriefed account of the process from the imager.

There are likely to be further factors that influence neural activity during observation and imagery. Because of the limited or equivocal neuroscientific research findings, we have not considered individual differences in: age; amount and duration of observation dosage; intervention adherence strategies; motivation to undertake the intervention; outcomes goals; external encouragement; and many other psychosocial factors. As brain imaging techniques become more readily available, cognitive and social neuroscientists will provide answers to many of these issues; the future for sport and performance psychology looks exciting!

  Fig. 16.1 • A schematic to represent the factors influencing the potential effectiveness of imagery and observation when used in the context of performance. From Holmes & Calmels 2008; Journal of Motor Behaviour 40(5): 441. Reprinted with permission of the Helen Dwight Reid Educational Foundation. Published by Heldref Publications, 1319 18th Street NW, Washington, DC 20036-1802, USA. Copyright ã 2008.

  These include: • task type (form-based tasks and perceptually driven tasks) and general observation factors • internally generated and externally triggered tasks • specular and anatomical imitation • live and video observation • still and temporally altered motion • pre-observation instructions • observation context • observer motor expertise; and • observer gender.

  Fig. 16.2 • Schematic representation of the factors impacting on observation effectiveness. Factors identified by asterisks are not included within the text as they cannot be supported by the neuroscientific research at present.

Holmes & Calmels 2008, p. 433

Short, S.E., Bruggeman, J.M., Engel, S.G., et al., 2002