Gunnar A Sigurdsson

Abhinav Gupta

Cordelia Schmid

Karteek Alahari

Beyond the Camera: Neural Networks in World Coordinates

Eye movement and strategic placement of the visual field onto the retina, gives animals increased resolution of the scene and suppresses distracting information. This fundamental system has been missing from video understanding with deep networks, typically limited to 224 by 224 pixel content locked to the camera frame. We propose a simple idea, WorldFeatures, where each feature at every layer has a spatial transformation, and the feature map is only transformed as needed. We show that a network built with these WorldFeatures, can be used to model eye movements, such as saccades, fixation, and smooth pursuit, even in a batch setting on pre-recorded video. That is, the network can for example use all 224 by 224 pixels to look at a small detail one moment, and the whole scene the next. We show that typical building blocks, such as convolutions and pooling, can be adapted to support WorldFeatures using available tools. Experiments are presented on the Charades, Olympic Sports, and Caltech-UCSD Birds-200-2011 datasets, exploring action recognition, fine-grained recognition, and video stabilization.

Introduction

The success of recent vision systems is in a way surprising, since the input is typically a 224×224 pixel image, or in the case of videos, a temporal stack of such images [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF]. This is both low resolution, and does not give much flexibility to investigate important signals in the image. The system is constrained to what the camera recorded, and has typically no active role in collecting the data. In particular, video recognition architectures have been shown to be vulnerable to camera motion, subject size, and temporal scale [START_REF] Sigurdsson | What actions are needed for understanding human actions in videos?[END_REF]. In comparison, the human visual system is not a passive receiver-our eyes are constantly moving to increase the effective resolution of the scene and suppress irrelevant signals [START_REF] Findlay | Saccadic eye movement programming: Sensory and attentional factors[END_REF].

These eye movements exist in humans and animals with foveal vision, and can be categorized as: Stabilization, the vestibulo-ocular reflex describes a control system the human visual system uses to stabilize images on the retina given proprioceptive information, like head rotation [START_REF] Crawford | Axes of eye rotation and listing's law during rotations of the head[END_REF]. Smooth Pursuit, where the Fig. 1. In this paper we explore how to model the variety of eye motions, even on pre-recorded data. We generalize the concept of feature maps, to WorldFeatures, that include a relative location in space and time, which can encode any type of "eye motion" we want apply to the recorded data, or undo. Orange to pink is used to denote features at time t-1, t, and t+1. We show a video from a first-person view, along with the Camera Motion in the reconstructed world frame. The original video follows a particular sequence of viewpoints, however, crucial information about the activities in the video requires different views-Stabilization transforms the views to the world frame, Pursuit transforms the views to follow the subject of the video, and Saccades & Fixation scans the scene to extract fine-grained information from the video.

gaze is voluntarily shifted to track a moving object, effectively stabilizing the object of interest on the retina [START_REF] Krauzlis | Temporal properties of visual motion signals for the initiation of smooth pursuit eye movements in monkeys[END_REF]. Fixation, where gaze is fixed towards a single location, to enhance resolution of that area [START_REF] Rucci | Control and functions of fixational eye movements[END_REF]. Saccades, where the eyes quickly jerk between phases of fixation to explore the scene [START_REF] Javal | Essai sur la physiologie de la lecture[END_REF]. In Fig. 1 we illustrate how a video is constrained to what the camera recorded, but that also the video may be refocused. How can we build a vision system that has this flexibility to freely explore the data, and move beyond a fixed 224×224 window?

We propose a simple and effective idea-each feature, has a location in realworld coordinates. This combination of (feature, transformation) pairs is referred to as a WorldFeature. An example transformation might be image coordinates to real-world coordinates (camera matrix). It turns out preprocessing the video data to, for example, stabilize it, introduces new problems, as demonstrated in Fig. 2. Instead, we use these WorldFeatures in all layers in the network and index the data according to the transformation or transform it as needed. That is, implicit transformation instead of explicit. With such network, we can utilize any type of "eye motion" we want apply to the recorded data, or undo. Video Fig. 2. Illustration of WorldFeatures, attaching transformations to features. Top left is the original video shown as two frame featuremap. We show the temporal average of an aligned feature map, with three different transformations T1 (identity), T2 (pursuit), T3 (stabilization). Observe how different transforms highlight different aspects of the data. We show a potential coordinate frame, that could be used to fit the data into a network, highlighting a problem with naively stabilizing the data, for example, when 64 frames in a row need to be aligned. WorldFeatures get around this by keeping the transformation and only using it as needed.

To illustrate the idea, in Fig. 2 we present a simple two frame featuremap with three different transformations T 1 , T 2 , T 3 to showcase how different transformations can highlight different signals in the data. For example, in the third it is easy to understand the global layout of the scene, and in the second it is easier to understand the difference in the pose of the person. In Sec. 3 we show more details on WorldFeatures, and how to keep track of the transformations after every layer to build networks with these operations.

Contributions.

In this paper we propose the idea of WorldFeatures, where each feature has a transformation, on multiple vision tasks: Fine-grained Image Recognition (Sec. 4.1), Video Stabilization (Sec. 4.2), and Video Activity Recognition (Sec. 4.3). We formalize WorldFeatures, and show how different eye movements can be modeled in this framework. Further, we propose a simple implementation that can adapt optimized neural networks tools to operate on WorldFeatures. This implementation can extend many state-of-the-art building blocks that operate over space and time, such as 3D Convolutions, and MaxPooling. The output of each building block, is also WorldFeatures, such that it can be processed hierarchically (e.g. 3D-ResNet [START_REF] Wang | Non-local neural networks[END_REF]).

Related Work

Many image and video understanding architectures have been proposed in the last few decades and we refer the reader to [START_REF] Poppe | A survey on vision-based human action recognition[END_REF][START_REF] Weinland | A survey of vision-based methods for action representation, segmentation and recognition[END_REF] for a detailed survey, and focus here on recent video architectures related to our approach.

Video Architectures. Recently, networks have become more evolved for separating content and motion [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF][START_REF] Ryoo | First-person activity recognition: What are they doing to me?[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF]. This direction is analogous to the different specialization of the ventral and dorsal streams in the mammalian brain. Similarly, separating camera motion and subject motion has shown great promise in video segmentation [START_REF] Vijayanarasimhan | Sfm-net: Learning of structure and motion from video[END_REF]. As the field moves towards utilizing longer temporal context [START_REF] Wu | Longterm feature banks for detailed video understanding[END_REF], equipping these networks with the capability to separate camera motion, object permanence, scene dynamics, and subject motion can greatly improve those efforts. This paper aims to provide the building blocks necessary to construct these systems.

Video Stabilization. Related to our work on understanding eye motion, is the task of video stabilization that has been pursued directly [START_REF] Grundmann | Auto-directed video stabilization with robust l1 optimal camera paths[END_REF][START_REF] Wang | Deep online video stabilization[END_REF][START_REF] Liu | Steadyflow: Spatially smooth optical flow for video stabilization[END_REF], or with additional supervision [START_REF] Liu | Video stabilization with a depth camera[END_REF]. Of particular relevance is recent work that has incorporated stabilization and tracking of points over time to improve accuracy of video recognition [START_REF] Wang | Action recognition by dense trajectories[END_REF][START_REF] Wang | Action recognition with trajectory-pooled deepconvolutional descriptors[END_REF]. Inspired by these successes, we go a step further, and build a framework that can operate given an arbitrary transformation signal, and use this to improve the vision system.

Improving Visual Input. On the other hand, recent work has also explored how to use the input data more effectively. This has been done through spatially or temporally modifiable connections [START_REF] Dai | Deformable convolutional networks[END_REF], non-local connections where feature similarity guides the connections [START_REF] Wang | Non-local neural networks[END_REF], or transformations of the input [START_REF] Jaderberg | Spatial transformer networks[END_REF]. Furthermore, [START_REF] Recasens | Learning to zoom: a saliency-based sampling layer for neural networks[END_REF] investigated the idea that not all input features are equally useful, and explored how to learn how to highlight and "zoom in" on the important content in each image. As we will see, eye movement is primarily useful to allow for efficient allocation of resources, and since our system can handle arbitrary transformations, we can explicitly utilize various "zoom in" of the data. Furthermore, our WorldFeatures is a system that allows a network to generalize beyond the camera, and is complementary to frameworks that learn attention to emphasize parts of the data.

WorldFeatures: Feature Maps with Camera Movements

We start by introducing the ideas behind WorldFeatures, that is, (feature, transformation) pairs. Then we demonstrate how different eye movements can be modeled in this framework. A video is a specific window into the world at the time it was recorded, and a system analyzing this video might have a completely different objective for watching the scene than the camera operator. To undo the camera bias or highlight signals in the video, we allow a transformations that are analogous to eye movements, after the data has already been recorded.

To undo camera transformations, could we pre-process the data? For example, applying affine transformations to each frame to undo camera movement, for stabilization. In fact, even this simple case has problems: How to fit this transformed data into the model? At what scale should we process the data? How to use imperfect transforms? Consider eyes scanning a scene, fixating on a variety of points in the scene at multiple resolutions. Creating data that follows a similar pattern without an understanding how they relate to each other creates a noisy and disjointed view of the scene. Instead, by keeping track of the relative transformations between features, we can crop, scale, and zoom in on the data as required, while still maintaining correct relationships between features.

WorldFeatures Definition

The key to our idea is the concept of WorldFeatures x T . That is, a feature map x of size T ×H×W (time, height, width) and an arbitrary transform T which can encode any type of "eye motion" we want to apply to the recorded data-or undo. We refer to a single H×W timepoint as a frame, and note that x=x I , where I is the identity transform. The feature of x T at (t,h,w) is written x (T (t,h,w)). Therefore, extending a convolution operation to work on x T :

y T (t, h, w) = i j k x(T (i, j, k)) • f (t-i, h-j, w-k) , (1)
where y T is also a WorldFeature, that can then be passed to the next convolution operation and so on. Here f is a filter, and we omit batch and channel dimensions

For example, the affine transform, T (t,h,w) = (t, a 0 t h+a 1 t w+a 2 t , a 3 t h+a 4 t w+a 5 t ), where a n t are the parameters at frame t.

for clarity. Compared to regular convolutions, the transformation T is necessary to transform the features as needed. For example, the valid values of each feature map are only defined for t∈ [0, T ), h∈ [0, H), and w∈ [0, W ), and transforming the video before processing would move content out of the frame, etc. This convolution layer could use a Spatial Transformer Network [START_REF] Jaderberg | Spatial transformer networks[END_REF] independently transforming the receptive field for each output frame, at every layer, exponentially increasing computation. This implementation of WorldFeatures is illustrated in Fig. 3. Fortunately, in Sec. 4 we outline an efficient implementation of this for CNN building blocks, which allows us to explore this type of networks.

Transformation Types

This framework can be used to model eye movements: Stabilization, Smooth Pursuit, Fixation and Saccades. Fig. 1 contains illustrative examples for each that we refer to for each of these transformations.

Stabilization. The first and most intuitive transformation T we encounter is the stabilizing transform. We obtain the camera matrix for every frame, and invert it to obtain the stabilizing transform. That is, the transformed feature map x (T (t,h,w)), will contain the same point in the world at location (h,w) for any t. The Stabilization example in Fig. 1 visualizes how the frames of the given video are arranged in the world frame.

Smooth Pursuit. Since the transform T is arbitrary we can also choose it to stabilize with respect to a different point of reference. That is, instead of (h,w) for any t pointing to the same point in the world, we can make it point to the same point on a moving object of interest, such as a person. The Pursuit example in Fig. 1 visualizes how the frames of the given video can be arranged, and combined with attention to the hands of the person.

Fixation. To model fixation, where the high-resolution part of the retina (the fovea) is used, we use a transform that enhances part of the frame. That is, if x T (t,h,w) is the original WorldFeature, then we add a "fixation transform" F to get the WorldFeature f T •F where f = x (F(t,h,w)). This fixation transform "zooms in" on a part of the input, increasing the resolution.

Saccades / Visual Search. To model saccades, we proceed similarly to fixation, and define a transform S that gazes at a particular part of the frame, allowing the model to attend to different parts of the image. The similarity with the implementation of fixation is because saccades are in fact defined as the jump between fixations. In the Saccades & Fixation example in Fig. 1 we demonstrate how the transformations can be chosen to pay close attention to important aspects of the scene, such as the hands, objects and context.

In conclusion, there are multiple eye movements possible, allowing a variety of augmentations without sacrificing relative location between features. In the experiments we use these definitions to provide the model with a transformation for each video, but learned transformations could be used as well. 4 Experiments.

Here we explore WorldFeatures for diverse applications. We use a 3DResNet-style architecture for videos built from WorldFeatures. This can be used to incorporate arbitrary eye movements in videos and images, where images are just treated as an uneventful video (replicating images in time). The network can then utilize the transformations to "zoom in" on parts of the image. First, we demonstrate that a 3D network, can outperform 2D networks on fine-grained recognition of nature categories (image task). Second, we look at video stabilization, and show how implicit transformations can bypass the problems with explicitly stabilizing the input data with preprocessing. Third, we utilize enhancing transformations, smooth pursuit and fixation, to improve video recognition accuracy.

Implementation. Our implementation extends a 3DResNet50 model, following [START_REF] Wang | Non-local neural networks[END_REF] (similar to I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]). All convolution, max pool, and average pool layers are converted to WorldFeature layers, which implicitly operate on a transformed video. All models start with parameters from a regular 3DResNet50 pretrained on the Kinetics dataset [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. In Fig. 4 we illustrate a simple method to convert a layer, such as a conv layer, to use WorldFeatures, without having to engineer special low-level GPU programs for each. More details are provided in the Appendix. We use the grid sampler from Spatial Transformer Networks [START_REF] Jaderberg | Spatial transformer networks[END_REF], with arbitrary transformation grids. After fitting the transformer, we use the grid sampler in the nearest (pixel) setting. This avoids blurring of the feature maps as they are repeatedly transformed. To address missing values after transforming, we add a channel denoting missing data at a point, similar to [START_REF] Felzenszwalb | Object detection with discriminatively trained part-based models[END_REF]. For comparison, all methods use the same batch-size of 2 unless otherwise noted, and follow the same learning rate schedule tuned for the baseline. The models are implemented using the PyVideoResearch [26] framework in PyTorch, and will be available at github.com/gsig/world-features. † For k=3 the first transformer aligns frames {0,1,2} (the receptive field for output frame 1), and frames {3,4,5} (receptive field for output frame 4), etc. The second transformer aligns {1,2,3} (receptive field for output frame 2), etc.

Top

Saccades -Fine-Grained Recognition

Motivated by human gaze studies in image recognition, we explore using 3D video networks for fine-grained recognition on the Caltech-UCSD Birds-200-2011 dataset [START_REF] Wah | The Caltech-UCSD Birds-200-2011 Dataset[END_REF]. To provide a gaze trajectory for the network, we use saliency [START_REF] Hou | Saliency detection: A spectral residual approach[END_REF] and objectness [START_REF] Cheng | Bing: Binarized normed gradients for objectness estimation at 300fps[END_REF] to generate 64 bounding boxes. We replicate each 2D image 64 times in time to form a 3D input. The 64 bounding boxes form the basis for the fixation in each frame, and are ordered to maximize the overlap between consecutive boxes, allowing the network to learn filters that combine information between various fixations and scales. See Fig. 5 for an example. We compare with regular 3DResNet50, and ResNet50. We start with a 3DResNet50 and fine-tune with the saccades/fixation setup, and combine with ResNet50.

Results. The results are presented in Table 1. With a input size of 224 pixels, our method outperforms architectures specialized for fine-grained recognition, even with recent methods such as [START_REF] Recasens | Learning to zoom: a saliency-based sampling layer for neural networks[END_REF] that learn the function of what to zoom to, whereas we use off-the-shelf saliency. Combining our framework and learning saliency is likely to yield additional gains. The method from [START_REF] Li | Dynamic computational time for visual attention[END_REF] only outperforms other methods when provided with 448×448 image requiring a special network beyond the scope of this work. These results demonstrate an exciting new avenue for video architectures applied to the conventionally 2D task of fine-grained recognition. In Fig. 5 we visualize the points of interests and scales suggested by the saliency algorithm, and how that is translated into pixel data that the network sees (along with the transformations between those). We also show example of featuremaps from 3 layers in the network, where we can see how the network combines information across the different saccades, and builds detailed fine-grained understanding of the image.

Stabilization -Video Activity Recognition

Stabilization is a fundamental vision task, and has been explored in various contexts. Here we specifically explore stabilization for improving video recognition.

To evaluate the stabilization performance, we start with videos with synthetic camera motion. We use 50% of the Charades [START_REF] Sigurdsson | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF] videos with the least motion ,

We used variance of optical flow, yielding 4396 videos of average 30 sec. and add synthetic camera movement by linearly moving between 3 locations at randomly chosen scale levels (30%-300%, Synthetic). Next we consider Olympic-Sports [START_REF] Niebles | Modeling temporal structure of decomposable motion segments for activity classification[END_REF], that was used to showcase the advantage of stabilization in the seminal IDT work [START_REF] Wang | Action recognition by dense trajectories[END_REF]. Charades is evaluated with a video-level mean average precision over 157 classes, and Olympic-Sports uses a top1 accuracy for each video over 20 classes. Our baselines are 3DResNet50 [START_REF] Wang | Non-local neural networks[END_REF], and 3DResNet50 with all frames are stabilized to the center frame (3DResNet50++). All models use a 16 temporal frame stack as input to make stabilization feasible for the baseline, and are trained with the same learning rate schedule and hyperparameters tuned to the baseline. For each video we construct the camera motion from the video using direct optimization of affine transformations parameters between consecutive frames. The details are provided in the Appendix.

Results. In Table 2 we see that using the stabilization transformation allows our method to improve over the baselines. That is, modelling features and transformations together instead of preprocessing allows our method to avoid compounding errors do to stabilization over many frames, and loss of resolution. Fig. 6. A single frame from 4 feature maps. First feature map is original video, second is the output of a temporal difference filter applied to an ideal (stable) video, third is output of the filter applied to the actual unstable video, and finally the output of our method given the estimated stabilizing transform.

Analysis. We also looked at an egocentric video dataset, Charades-Ego [START_REF] Sigurdsson | Actor and observer: Joint modeling of first and third-person videos[END_REF].

With 64 frame inputs, preprocessing (3DResNet50++) actually reduces performance in the baseline (23.6 → 22.8 mAP), and with 16 frame inputs (only 0.67 sec video clip), preprocessing only slightly improves performance (17.5 → 18.5 mAP). This demonstrates the challenge of stabilizing a video before passing it to the network, and interestingly, stabilization does not seem to help activity recognition much. Perhaps, since what the camera operator is looking at is highly informative in first-person video. In any case, WorldFeatures have performance at par with the better method in each case (23.4 and 18.4 mAP).

In Fig. 6 we demonstrate how camera motion makes learning from video challenging. We show point in time from 4 feature maps. We observe that World-Features can utilize a transformation to suppress irrelevant signals. Although a deep network might, with sufficient data, be able to learn all possible variations, this adds complexity the model. We will see in the next section how we can go even further and use the transformations to focus on particular parts of the data.

Smooth Pursuit & Fixation -Video Activity Recognition

To demonstrate how WorldFeatures can utilize any transformations, we explored Smooth Pursuit, where we use a transformation to stabilize the content with respect to a human bounding box (Pursuit). We use an RCNN person detector [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF], and construct a the transform such that the person is always centered and fills the input to the network. Going further, we explored how to utilize the transformations to increase the effective resolution of particular parts of the scene or the subject in a video. Concretely, we used simple video saliency (temporal difference), and fit a bounding box around 80% of the variance in the saliency, and the fixation transforms the input such that the box fills the input (Fixation).

Our experiments are on the Charades dataset [START_REF] Sigurdsson | Hollywood in homes: Crowdsourcing data collection for activity understanding[END_REF], evaluated with a video-level mean average precision over 157 classes. We start with a standard pre-trained baseline network from the literature (3DResNet50 ), fine-tune with the new framework, and combine with the old network.

Results. Our results are presented in Table 3. Interestingly, adding smooth pursuit on its own does not significantly improve over the baseline, which stabilizes the video with respect to the person. However, this is expected since many of the videos in Charades have camera motion that already follows the video subject. Fortunately, we do see that using smooth pursuit followed by fixation (FixationPursuit) we can improve performance. This suggests that whereas stabilization on its own is not particularly helpful to current neural networks, when it is used to locate an area to increase the resolution it can offer significant benefits, since 224×224 pixels is not provide much detail, and learn filters that combine information across multiple locations and resolutions. In Fig. 7 we visualize frames from a Conv1 feature map applied to a video on a person sitting at their desk, since each feature has a transformation associated with it, even though some of the feature maps are at very small scales, the network can move them into a common coordinate frame as needed.

Discussion & Conclusion

We presented a framework that can use transformations to better use available data, moving beyond the implicit biases of the camera that recorded a video.

Video Stabilization. One of the hypotheses that we explore is stabilization in terms of a world frame, and in terms of a target (smooth pursuit). We discovered that stabilization on its own is not very helpful for current vision systems. However, when combined with fixation and saccades that highlight useful aspects of the input, it has significant advantages.

Fixation in Animals. Similarly, eye movement and fixation in animals is hypothesized to be primarily in order to fully utilize the center of the retina, the fovea, that has high-resolution and only covers about 1-2 degrees of vision [START_REF] Provis | Adaptation of the central retina for high acuity vision: cones, the fovea and the avascular zone[END_REF]. This is, it is about efficient usage of resources. Furthermore, average fixation duration in humans is only 330 ms (8 frames at 24 fps) [START_REF] Henderson | Human gaze control during real-world scene perception[END_REF], suggesting long-term stabilization might be unnecessary for some visual systems. Thus eye movement primarily plays a role in allowing efficient visual search of the scene.

Conv1 feature maps

Conv1 transformed (composite) Stabilization in First-Person Videos. Stabilization in the first-person vision case is complicated by the fact that the camera is commonly doing smooth pursuit of the objects/hands of interest already, and zooming in is unnecessary because the field of view is already narrow. In the Charades third-person videos the camera often follows the person, that is, smooth pursuit is already present.

Conclusion.

We addressed several challenges that come with moving beyond the camera, and highlighted problems that have to be considered. Such as, variability due to scale and aspect ratio changes, how to include data augmentation, and working with pretrained networks that only consider particular types of input data. We hope this work opens the door for systems that learn policies for visual search and efficient allocation of visual resources.

Appendix

Implementation Details. Layers using WorldFeatures could be implemented with efficiency similar to their regular counterparts, requiring dedicated engineering. To iterate on the idea, we implemented WorldFeatures in high-level PyTorch code. We observe that current architectures have filter extent of 3 frames [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Wang | Non-local neural networks[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. Thus, we simply transform each chunk of consecutive frames into common coordinates as needed for each filter computation. This is illustrated in Fig. 8, where 3 frames are temporarily transformed into a common coordinate frame (the coordinates of frame t), and used to compute the convolution output. Concretely, mirroring Eq. ( 1) above, suppose we need to calculate y T (t, h, w) for some particular frame: t=t 0 , h ∈ {0,1, . . . ,H-1}, and w ∈ {0,1, . . . ,W -1}. Then we can rewrite as follows:

x t (t ,h ,w ) = x(T (t ,h ,w )) (2)
y T (t, h, w) = i j k x t (i,j,k) • f (t-i, h-j, w-k) (3)
which describes convolution on x . Here x is an explicitly transformed version of x. Since f only has non-zero values at t = {-1, 0, +1} (if filter extent is 3) x only needs to be computed for t-1, t, and t+1. Thus if we need to compute this for all values T of t, we need to create T such versions of x , which corresponds to 3 versions of the size of the input (T ×H×W ). Finally, we apply the regular convolution to these 3 copies and merge the results to create y T (t, h, w). This has numerous advantages: 1) Since 3 or less frames are being transformed at a time, both cumulative errors in transformation and missing value problems are minimal. 2) Transforming across long timescales is only done in higher layers, Fig. 8. Illustration of our implementation of WorldFeatures. We start with a world feature map x T and and in order to calculate the output of a convolution applied to the feature map at time t, that is y T (t, h, w), we first transform the needed features into the coordinate system at time t, and then apply the convolution as normal. Black, gray and white, are used to denote t-1, t, and t+1, respectively. where spatial resolution is lower, meaning less accuracy is needed. 3) Any layer that operates on the time dimension, such as maxpool, average pool, and 3d convolution, can be adapted in the same way using this method.

In Table 4 we illustrate 3DResNet50 extended with WorldFeatures, and how the size of the transformer changes to correspond to each layer. In our setup, all consecutive layers receive a (potentially resampled) transformer that contains the transformations to align consecutive timepoints of the feature map.

Computing Stabilization. Before stabilization takes place, we need to compute the transformations between consecutive frames. Traditionally, optical flow or feature matching have been utilized for this purpose, but with increase in computation, recent work has moved towards direct methods for estimating the transformation. This optimizes the transformation directly on the image intensities, which enables utilizing all of the information in the image, making the estimation higher accuracy, and particularly robust to frames with few keypoints [START_REF] Engel | Lsd-slam: Large-scale direct monocular slam[END_REF]. In particular, we directly optimize the parameters of a transformation (affine or homography), but in a batch fashion on each pair of consecutive frames in each mini batch. The objective from LSD-SLAM [START_REF] Engel | Lsd-slam: Large-scale direct monocular slam[END_REF] operates directly on images intensities (Huber norm) and downweighs large values. We simplify it to: min θ h,w min x(t,h,w) -x(T θ (t+1,h,w)) 2 , δ ,

where we align the frame at t and t+1. We scale the source (t + 1) and target (t) image equally such that target image has unit variance, and δ = 0.01. We then use gradient-based optimization with adaptive learning rate to minimize this objective with respect to the transformation parameters θ.

Fig. 3 .

 3 Fig.3. Demonstration of a model using WorldFeatures at each layer. Here we look at an illustration of the outputs of conv1 and maxpool layers, when applied to an "eye movement" task. The input and output of each layer are WorldFeatures. The receptive field of each output featuremap is aligned, explicitly or implicitly depending on implementation, before computing the output. Since each output feature map has a different alignment, each output has a different transformation, and we repeat the process at every layer. This shows that WorldFeatures allow the network to combine information across multiple locations and scales in the image.

Fig. 5 .

 5 Fig.[START_REF] Engel | Lsd-slam: Large-scale direct monocular slam[END_REF]. In the left column we visualize the path (the saccades) that the model chooses, given static saliency and objectness, where dark to light denotes frames 1 through 64, and the large circles denote the scale of the fixaton in that frame. In the the middle column, we show the pixel data that the model uses (the model also receives the transformations). On the right, we show featuremaps from Conv1, MaxPool1, and MaxPool2, high values denote high variance over time at that pixel.

Fig. 7 .

 7 Fig. 7. Composite of 32 Conv1 feature maps from the fixation model. Different color indicates different frames.

  Our simple implementation of WorldFeatures "wraps" a layer (e.g. conv layer) and applies a spatial transformer to the data to align the receptive field of each output, k times for a layer with temporal extent of k (e.g. 3 in ResNet), and combines the outputs. † More details are provided in the Appendix.

	Spatial Transformer
	Conv Layer
	Conv Layer	Resampling
	Fig. 4.

Table 1 .

 1 Fine-grained Recognition on the Caltech-UCSD Birds-200-2011 dataset.

		-1 (%)	Resolution (px.)
	3DResNet50	79.8	224
	DT-RAM [16]	82.8	227
	ResNet50	83.4	224
	Learning to Zoom [22]	84.5	227
	3DResNet50 w/Saccades	85.3	224
	DT-RAM [16]	86.0	448

Table 2 .

 2 Results on stabilization on two video activity recognition datasets. Stabilizing the video before inputting it to the network (3DResNet50++) helps in some cases, but has drawbacks, whereas implicit stabilization can better utilize the input.

		3DResNet50	3DResNet50++	Pursuit
	Synthetic (Video mAP)	13.1	13.5	14.1
	Olympic-Sports (% Top-1)	96.4	97.2	97.2

Table 3 .

 3 Activity Recognition on the Charades dataset using smooth pursuit and fixation transformations.

		3DResNet50 Pursuit Fixation FixationPursuit
	Charades [29] (Video mAP)	31.3	31.5	32.3	32.6

Table 4 .

 4 Our 3DResNet50 model for video. The dimensions of 3D output maps and filter kernels are in T×H×W, with the number of channels following. The input is 64×224×224, and the pre-computed transformer corresponding to the input is of size 64. Residual blocks are shown in brackets. The table is adapted from[START_REF] Wang | Non-local neural networks[END_REF].

		layer		output size	transformer output size
	conv1	7×7, 64, stride 2, 2, 2	32×112×112	32
	pool1 3×3×3 max, stride 2, 2, 2	16×56×56	16
			1×1, 64	
	res2		3×3, 64	 ×3	16×56×56	16
			1×1, 256
	pool2 3×1×1 max, stride 2, 1, 1	8×56×56	8
			1×1, 128	
	res3		3×3, 128	 ×4	8×28×28	8
			1×1, 512
			1×1, 256	
	res4		3×3, 256	 ×6	8×14×14	8
		1×1, 1024
			1×1, 512	
	res5		3×3, 512	 ×3	8×7×7	8
		1×1, 2048
		global average pool, fc	1×1×1	1

Acknowledgements. This work was supported by Intel via the Intel Science and Technology Center for Visual Cloud Systems, Sloan Fellowship to AG, the Inria associate team GAYA, the ERC advanced grant ALLEGRO, gifts from Amazon and Intel, and the Indo-French project EVEREST (no. 5302-1) funded by CEFIPRA. The authors would like to thank Kenneth Marino for feedback on the manuscript.


Mariam Bouzid
email: mariam.bouzid@utt.fr

Oussama Masmoudi
email: oussama.masmoudi@utt.fr

Alice Yalaoui
email: alice.yalaoui@utt.fr

Single machine Order Acceptance Scheduling with periodic energy consumption limits

Keywords: Order Acceptance Scheduling, Energy consumption, Timeindexed

This work presents an Order Acceptance Scheduling (OAS) problem with the introduction of energy consumption limits at each period of the planning horizon. The objective is to maximize total profit of accepted orders while considering energy usage over time, in accordance with imposed deadlines. A time-indexed formulation is presented to tackle the investigated problem.

Introduction

Resource depletion has become a major concern these past decades for both economic and environmental reasons. Standards, taxes and limits on resource use, are ones of many tools implemented by authorities to curb the current trend. Although, as stated in [START_REF] O'brien | Resource efficiency in european industry 12[END_REF], new business opportunities are opened by these challenges and structural changes are required in the supply chain in the context of sustainability. This lead industrials to reconsider their rate of production at the operational level. During production, many levers can be taken into account, such as machine speed, shutdown mechanisms, among others. Monitoring energy consumption and costs, or carbon emissions during the production processes are useful indicators to mitigate impacts on the environment and to make substantial savings. Meanwhile, accepting or rejecting orders occurs in a plethora of manufacturing sectors when a company is limited by its facilities, client deadlines or in general by resources availability. Optimizing the production schedule is therefore essential to make a good balance between available production capacity and sales volume.

In this vein, this paper focuses on a capacity-constrained system by addressing an Order Acceptance Scheduling (OAS) problem with periodic energy consumption limits with the objective to maximize total profit. Section 2 presents a short literature review on resource-efficiency and energy-efficiency in scheduling problems with a focus on OAS problems. Section 3 gives a description of the proposed problem. Section 4 provides the solving approach and section 5 gives the preliminary results on a small benchmark. The final section features conclusions and perspectives to this work.

Related literature

Resource-efficiency and more globally resource management appeared at early stage of scheduling problem researches [START_REF] Yalaoui | Scheduling under resource constraints: Deterministic models[END_REF]. The characterization of resources in terms of availability and divisibility allow to cover a broad range of situations. Time-dependent resources, i.e. when the amount of resources varies over time, is for instance studied for an open shop and a parallel machine system in [START_REF] Tautenhahn | Unit-time scheduling problems with time dependent resources[END_REF].

Resource management typically occurs during a project with dependent activities such as construction engineering, or manufacturing. In order to carry out a project at a lower cost while satisfying timing, labor, material, or equipment constraints, the Resource-Constrained Project Scheduling problem (RCPSP) [START_REF] Brucker | Resource-constrained project scheduling: Notation, classification, models, and methods[END_REF] (and its variants) addresses this problem particularly well. On another side, financial scheduling problem insists on other interesting aspects of resources, in particular when they are assimilated to a stock (e.g. wages, raw materials) that can be replenished at different time of the horizon. Most of the studies on this topic provide solid complexity analysis and approximation schemes [START_REF] Györgyi | Minimizing total weighted completion time on a single machine subject to non-renewable resource constraints[END_REF][START_REF] Györgyi | New complexity and approximability results for minimizing the total weighted completion time on a single machine subject to non-renewable resource constraints[END_REF]. Gafarov et al. [START_REF] Gafarov | Single machine scheduling problems with financial resource constraints: Some complexity results and properties[END_REF] proved the NP-hardness of the single machine problem with non-renewable resources with the objective to minimize the makespan. They provide the optimal sequence to a particular polynomial case of the problem with the objective to minimize total tardiness. When it comes to energy, usually, efficiency is conceptualized by incorporating a specific criteria within the objective [START_REF] Gao | A review of energy-efficient scheduling in intelligent production systems[END_REF]. Therefore, total energy cost or consumption is minimized in numerous works including single machine and more complex systems. However, few researches address energy-efficiency by introducing dedicated constraints that differ from conventional resource constraints. For parallel machines, [START_REF] Módos | Algorithms for robust production scheduling with energy consumption limits[END_REF], minimized makespan with interval limits on energy consumption. They used two exact solving approaches, which are Branch and Bound and Benders decomposition, and a Tabu search to tackle this problem. In the meantime, Liao et al. [START_REF] Liao | Multi-objective optimization of single machine scheduling with energy consumption constraints[END_REF] minimized both weighted completion times and weighted tardiness on a single machine with interval limits on energy consumption. They proposed an evolutionary algorithm for this multi-objective problem.

Introducing resource constraints is no exception in OAS, since it can be expected that limited resources force manufacturers to reconsider how much they can produce, avoiding losses as much as possible. For instance, [START_REF] Garcia | Resource-constrained scheduling with hard due windows and rejection penalties[END_REF] are the first to consider an OAS on a single machine with resource-constraints, rejecting orders if they cannot be completed within a predetermined time-window. The work of Kong et al. [START_REF] Kong | Green manufacturing: Order acceptance and scheduling subject to the budgets of energy consumption and machine launch[END_REF] presents an order acceptance scheduling problem with a global budget on energy use in a parallel machines system.

To the best of our knowledge, the OAS problem on a single machine with periodic resource consumption limits has not yet been tackled. This problem is interesting and appears when a resource consumed by production (such as energy) cannot be stored and is different from a period to an other. This typically appears in period of high energy consumption.

Problem description

This paper investigates an Order Acceptance scheduling problem on a single non-preemptive machine with energy consumption limits at each period. The objective is to maximize total profit while satisfying periodic energy consumption limits. This problem is NP-hard since the OAS problem on a single machine, maximizing the total profit is NP-hard in its basic form [12].

As in [START_REF] Tautenhahn | Unit-time scheduling problems with time dependent resources[END_REF], in this problem, the resource is renewable, that is, a energy unit available at time t shall be consumed at time t but can not be saved for later. The availability of the resource at time t + 1 does not depend on the amount consumed at or before time t, unlike financial scheduling problem. Energy is thus available in time-dependent quantities, i.e. their availability changes over time. Each order is defined by its processing time p j , its deadline dj , its unitary power consumption e j (kW) and its revenue f j .

The following assumptions are formulated:

-The energy consumed by an order remains constant during the totality of its production. -Preemption is not allowed.

-The manufacturer earns the totality of the revenue of an order j if it can be scheduled completely before its deadline.

The horizon is split into unitary slots t = 1, . . . , T each characterized by an energy limit E t . The length of the horizon T is defined by the maximum of the deadlines. In fig. 1, the bar chart represents the amount of available energy E t at each period t = 1, . . . , T . The sequence is represented by a Gantt diagram below. An order j is represented by two components : its energy consumption e j in height and its processing time p j in width.

Solving approach

A time-indexed On/Off formulation is used to model the proposed problem. In this formulation, the binary decision variables x jt = 1 refer to the production of a unit of order j at time t (or not x jt = 0). Moreover, the binary decision variables a j equals 1 if the order j is accepted, 0 otherwise. The problem is formulated as a Mixed Integer Linear Program as follows.

maximize n j=1 a j f j (1) subject to n j=1 x jt ≤ 1 t = 1, . . . , T (2)
dj t=1 x jt = p j a j j = 1, . . . , n (3)
t-pj t =1

x jt + dj t =t+pj

x jt ≤ p j (1 -x jt ) j = 1, . . . , n, t = p j + 1, . . . , dj (4)

n j=1 e j x jt ≤ E t t = 1, . . . , T (5)
T t= dj +1

x jt = 0 j = 1, . . . , n (6)
The objective (1) is the maximization of the sum of the revenue of the accepted orders. Constraints (2) ensure that the machine can either produce or do nothing. Constraints (3) state that an accepted order must be processed p j time units within its time-window r j , . . . , dj . Constraints (4) guarantee nonpreemption by forcing the continuity of the decision variables x jt . If at time period t order j is produced (x jt = 1), the constraints exclude production p j units before and after t, i.e in periods 1, . . . , t -p j and t + p j , . . . , dj . Constraints (5) are the periodic energy consumption limits, imposing that a unit of production of an order at time t does not exceeds E t , the available energy at time t. Constraints (6) prevent any order j to be produced after its deadline dj .

Preliminary results

The tested instances have been randomly generated with diverse number of orders n = 10, 15, 25, 50, 100, 5 for each value of n. Processing times and revenues are generated from the uniform distribution p j , f j ∼ U(1, 20). The unitary power consumption is computed as follows : e j ∼ U(1, f j ) × 1 2 . The values of the deadlines are determined by dj = U(p j , p T ) with p T = n j=1 p j . The energy consumption limits over time are randomly generated with E t ∼ U(1, 10) for all t = 1, . . . , T . The value of the energy consumption limits are the same across the benchmark. The tests have been performed on a PC with Intel i5 2GHz CPU processor and 4GB RAM. The provided model is written in the IBM CPLEX Solver v 12.9. Solving time is limited to 3600 seconds. 1 presents the results obtained by the solver. Each line represents 5 instances with varying number of orders. The average solving time (in seconds) and the average gap (CPLEX relative Gap in %) is reported for each batch of instances. Finally, the average proportion of accepted orders is indicated (in %). The model performs well for small to medium instance size with less than a second to optimally solve the instances. As can be seen, from n = 50, the average cpu time sharply increases. With n = 100 orders, the model cannot find some feasible solutions (3/5 are feasibles) with poor solution quality.

Conclusion and perspectives

In this paper, the OAS problem on a single machine with periodic energy consumption limits is studied. A time-indexed model with O(nT ) constraints and O(nT ) decision variables is used to solve this problem. Since the objective is the maximization of total profit with time-dependent resource constraints, the time-indexed model is a purposeful choice as it exploits to its benefit the time dependency of the investigated problem. A comparison between the proposed time-indexed model and a classical formulation using completion times shall be done. The development of dedicated exact methods is planned in the future work due to solver limitations. A special focus will be made on finding dedicated cuts, bounds and properties of the problem. Moreover, the formalisation of diverse type of constraints that make sense in the context of energy-efficient scheduling must be performed. Global, cumulative and rolling-window constraints must be studied with a careful examination of mathematical properties and a complexity analysis for each of the problem variants. Preliminaries results will be presented on this occasion.

For

  illustration purpose, let's take an example of n = 5 orders where p = [8, 3, 4, 4, 3], d = [25, 16, 9, 21, 9], e = [3.5, 4, 3.5, 3.5, 2] and f = [15, 10, 10, 5, 10]. Assume that orders 2 and 4 are rejected.

Fig. 1 .

 1 Fig. 1. An example with n = 5 orders.

Table 1 .

 1 Computational results.

	n cpu	gap	% acc
	10 0.03	0	0.22
	15 0.07	0	0.24
	25 0.55	0	0.28
	50 675	0	0.38
	100 3600	53	0.32
	Table


Havelková Petra
email: havelkova.petra@gmail.com

Villotte Sébastien
email: s.villotte@anthropologie.u-bordeaux1.fr

ENTHESOPATHIES: TEST OF THE REPRODUCIBILITY OF THE NEW SCORING SYSTEM BASED ON CURRENT MEDICAL DATA

Keywords: enthesopathies, reproducibility of method, insertion site

Many anthropological studies have focused on enthesopathies to find the correlation between their occurrence on muscle insertion sites and physical activities. However, these works are rarely aimed at methodology of the scoring system. A new method (Villotte 2006) is based on current anatomical and histological data. The aim of this work was to propose criteria for recording or not a stage with this method and test its reproducibility (interobserver error). We used a sample of twenty adult male skeletons from the medieval settlement agglomeration -Mikulčice. According to the Kappa coefficient, reproducibility of the method was substantial, and almost perfect for fibrocartilaginous insertions.

Introduction:

The first research attempting to find more information about musculoskeletal stress markers (term used by [START_REF] Hawkey | Activity-induces musculoskeletal stress markers (MSM) and subsistence strategy changes among ancient Hudson Bay eskimos[END_REF] on the skeleton appeared in 16 th century, especially in medical literature focusing on trade and military diseases [START_REF] Kennedy | Skeletal markers of occupational stress[END_REF]). However, only in 19 th century scientists became aware that some irregularities in the region of enthesis, i.e. the insertion site of ligament, tendon and joint capsule [START_REF] Benjamin | The skeletal attachment of tendons-tendon "entheses[END_REF], could be related to life habits [START_REF] Cunha | What can bones tell about labour and occupation: the analysis of skeletal markers of occupational stress in the Identified Skeletal Collection of the Anthropological Museum of the University of Coimbra[END_REF]. Recently, many anthropological works have focused on the study of the relationship between pathological changes of enthesis and occupational and physical stress [START_REF] Dutour | Enthesopathies (Lesions of Muscular Insertions) as Indicators of the Activities of Neolithic Saharan Populations[END_REF][START_REF] Dutour | Activités physique et squelette humain: Le difficile passage de l'actuel au fassile[END_REF][START_REF] Knüsel | On the Biomechanical and Osteoarthritic Differeces Between Hunter-Gatherers and Agriculturalists[END_REF][START_REF] Cunha | What can bones tell about labour and occupation: the analysis of skeletal markers of occupational stress in the Identified Skeletal Collection of the Anthropological Museum of the University of Coimbra[END_REF][START_REF] Hawkey | Activity-induces musculoskeletal stress markers (MSM) and subsistence strategy changes among ancient Hudson Bay eskimos[END_REF][START_REF] Crubézy | Épidémiologie de l'arthrose et des enthésopathies dans une population européenne d'il y a 7700 ans[END_REF], Cappaso et al. 2004). The manifestations of physical activities at enthesis, associated with the development of cortical defects (enthesophytes, lesions, foramina, cysts) are termed enthesopathies (Niepel, Siťaj 1966[START_REF] Lagier | Enthèses normales et enthésopathies[END_REF]. A number of methods for evaluation of enthesopathies were proposed in the last twenty years [START_REF] Crubézy | Interactions entre facteurs bio-culturels, pathologie et caractères discrets, Exemple d'une population médiévale : Canac (Aveyron)[END_REF][START_REF] Hawkey | Use of upper extremety enthesopathies to indicate habitual activity patterns[END_REF][START_REF] Mariotti | Enthesopathies -Proposal of a standardized scoring method and applications[END_REF][START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF]. All the mentioned methods are visual and use the scale system of evaluation. In contrast to the other, Villotte's method is based on current anatomical and histological studies of entheses (insertion site, attachment site) (e.g. [START_REF] Resnick | Entheses and enthesopathy[END_REF], Benjamin, Ralph 1998[START_REF] Benjamin | The skeletal attachment of tendons-tendon "entheses[END_REF]. In accordance with medical knowledge, it is possible to distinguish two types of insertions by the character of the tissue at the bone-tendon interface [START_REF] Benjamin | The histology of tendon attachments to bone in man[END_REF], Benjamin, Ralph 1998[START_REF] Benjamin | The skeletal attachment of tendons-tendon "entheses[END_REF]. The first type of entheses is termed fibrous and it occurs in all regions of the appendicular skeleton with a thick layer of cortical bone (especially mid-shaft of long bones). The second type, the fibrocartilaginous entheses, is associated mainly with the region of the epiphyses and apophyses. These two types of insertions differ in their mechanical properties and appearance and the methods of evaluation of enthesopathies should take into account the character of the entheses. The categorisation of insertion sites according to the character of entheses is thus the most important advantage of the newly proposed scoring system for evaluating enthesopathies [START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF]. The aim of this study was to conduct a test of reproducibility based on inter-observer error on this method. Good result of test reproducibility is the main presumption for the next application of this method.

Materials:

The material contains 20 skeletons from the Mikulčice -Valy (Czech Republic) housed in National Museum in Prague. Mikulčice is the medieval (Middle "Hillfort" Period) settlement agglomeration, dating from the 9 th -10 th century and is considered one of the possible centres of Great Moravia. Since this work was focused on the evaluation of enthesopathies related to physical activity, only male adult skeletons were selected. The upper age limit was set at 50-60 years. We also excluded individuals with DISH or spondyloarthropathies for the same reason. We evaluated the insertions on all long bones, (except the fibula), the pelvis, the patella, the calcaneus and the vertebrae (C2-S1).

Methods:

This study uses a new scoring system proposed by S. Villotte [START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF] for the evaluation of enthesopathies which was applied to 36 insertion sites (18 on each side) of appendicular skeleton and 46 insertions (proximal and distal part) of yellow ligament at the spinal column (Table 1).

G1 G2 G4
Based on a study of differences between normal (non-pathological) and pathological aspects of individual entheses and comparison with medical descriptions, Villotte differentiates four scoring systems (four muscle groups). The first three groups include fibrocartilaginous entheses (including vertebrae), while the fourth group includes fibrous entheses. Enthesopathies are classified using a three-stage scale (A, B, C). In the case of fibrocartilaginous insertions, especially those in the first group, the contour and surface of the insertion must be evaluated separately.

1 st group (G1): All types of remodelling (enthesophytes, lesions, foramina or cysts) can be found on the surface of the insertion. The contour is most commonly remodelled in the form of enthesophytes. The contour and surface is evaluated separately by partial stage (0-2). The final stage (A, B, C) classification is based on the sum of these individual evaluations. 2 nd group (G2): The most common form of remodelling of the insertion contour is enthesophytes. Stages are defined metrically, by the size of enthesophytes. The surface is rarely affected (erosive zones).

3 rd group (G3). The method used for evaluation of enthesopathies in 3 rd group was made according to [START_REF] Crubézy | Interactions entre facteurs bio-culturels, pathologie et caractères discrets, Exemple d'une population médiévale : Canac (Aveyron)[END_REF]. This group includes the most of insertions of yellow ligaments in the medial part of vertebrae (C2-S1). The proximal and distal part of the vertebral arc in the medial section is classified separately. Stages are also defined metrically. 4 th group (G4). This is the most problematic group regarding to the methodology. The lack of anatomical and pathological descriptions does not allow define the stage precisely. This group includes fibrous entheses whose remodelling is expressed as an increasing irregularity of the surface or, rarely, as a cortical lacuna.

The second part of this work, beside the scoring system, is focused on the evaluation of taphonomy, i.e. if it is possible to evaluate the insertion due to damage (A, B or C) or not (NR -non-recordable). The evaluation of enthesopathies was realized by two observes. One of these is the author of the methodology. The reproducibility test was focused on two aspects. First, we monitored the reproducibility of the scoring system itself, i.e. the number of disagreements between the two observers determining the stage for a concrete insertion. The second part of the test involved the evaluation of taphonomy, i.e. the number of disagreements when one of the observers considered the insertion to be non-recordable (NR) while the second one assigned it a certain stage (A, B,C).

Only those insertions that were completely missing, not those that were damaged (NR), were taken into consideration as an absent (Abs). The number of missing insertions was identical for both observers. NR were considered when less than 50% of the inner or the outer part of an enthesis were observable for G1 and G2 and when less than 50% of the insertion were observable for G3 and G4. This method is based on a study of V. Mariotti [START_REF] Mariotti | Enthesopathies -Proposal of a standardized scoring method and applications[END_REF]), according to which, it is only possible to evaluate insertions if more than 50% of the surface has been preserved. The disadvantage of Mariotti proposal is that it does not distinguish between the contour and surface of insertion The concordance rate between the observers was expressed as a percentage value as well as with the aid of the Kappa's coefficient that also assesses the quality of agreement between the observers (Kappa value was interpreted according to [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF].

Results:

The study of reproducibility of scoring system and taphonomy involved 1210 insertions (430 insertions were absent). The reproducibility test results for both systems (scoring -A, B, C and taphonomy -NR) are summarized in Table 2 (Part I). The overall agreement between the two observers for all the groups and all the variants of evaluation (A, B, C, NR) was 88% (kappa 0, 81). It is important to note that this result is based on partial results that are very different. For groups G1 and G2, the overall concordance rate when evaluating taphonomy and developmental stages was greater than 92%, while that for groups G3 and G4 was less than 85%.

Results of the reproducibility test for each scoring system (G1-4) and overall result (percentage value of direct agreement among observers (Part I-III), the Kappa's coefficients and the interpretation according to these coefficients, ABC-scoring system, NR -non-recordable).

Part III ABC

Kappa

Kappa N of insertions

Part I NR+ABC

Part II NR

The results of test for taphonomy (NR) are contained in Table 2 (Part II).When evaluating taphonomy of the insertions we observed less than 5% of disagreements between both observers for all the groups as well as for each group alone. If we focus only on groups G1 and G2, we recorded more disagreements when evaluating taphonomy than when evaluating the developmental stage itself. However, these differences were not statistically significant. If we rule out all situations that include one of the variants of evaluation, NR/NR or NR/developmental stage, we can conduct the statistical test of reproducibility only for the stages of the scoring system (A, B, C; 997 insertions). The frequency of these is detailed in Table 2 (Part III). The consequent concordance rate for the scoring system was 89% (kappa koeficient = 0,8). This overall value is similar to the result achieved in previous work [START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF], Table 3). Inter-observer agreement in groups G1 and G2 is almost perfect (kappa koeficient = 0, 93 for both groups). Conversely, groups G4 (kappa koeficient = 0, 74) and G3 (kappa koeficient = 0, 72) appear to be the most problematic. This difference is supposed to be due to the different methodology of evaluation of enthesopathies.

Discussion:

The aim of this work was to test the reproducibility of a newly proposed method for the evaluation of enthesopathies in the region of muscle or ligament insertions. Two observers evaluated 20 adult male skeletons from the Early Medieval burial ground Mikulčice -Valy. Percentage value and the Kappa's coefficient were used for expression of the concordance rate between the observers. The Kappa's coefficient is used to assess inter-observer reliability and reproducibility of method when observing or otherwise coding qualitative or categorical variables. Kappa is considered an improvement over using % agreement to evaluate this type of reliability. The Kappa value, indicates how much better the observers are compared to a throw of the dice, and therefore gives the real credit to the agreement, which was found [START_REF] Svanholm | Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic[END_REF]). The Kappa coefficient is statistical method for observers using nominal and ordinal categories (e.g. [START_REF] Svanholm | Reproducibility of histomorphologic diagnoses with special reference to the kappa statistic[END_REF][START_REF] Sidor | The Neer classification system for proximal humeral fractures: An assessment of interobserver reliability and intraobserver reproducibility[END_REF][START_REF] Cummings | Interobserver Reliability and Intraobserver Reproducibility of the System of King et al. for the Classification of Adolescent Idiopathic Scoliosis[END_REF][START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF]). The concordance rate achieved for the evaluation of taphonomy and developmental stages was 88%. We may thus consider the test of reproducibility almost perfect. Compared to other works that focused on the evaluation of enthesopathies, this result is by no means significantly different. While the reproducibility of the method presented by E. Crubézy [START_REF] Crubézy | Épidémiologie de l'arthrose et des enthésopathies dans une population européenne d'il y a 7700 ans[END_REF] was lower (the inter-observer error is cited as 20%), V. Mariotti [START_REF] Mariotti | Enthesopathies -Proposal of a standardized scoring method and applications[END_REF]) presents a degree of concordance higher than 95%, as well as D. Hawkey [START_REF] Hawkey | Activity-induces musculoskeletal stress markers (MSM) and subsistence strategy changes among ancient Hudson Bay eskimos[END_REF]. The lower concordance rate in this case is mainly due to the low partial resultes of interobserver error for groups G3 (vertebral insertions) and G4 (fibrous insertions). Concordance rate for these two groups was less than 85%. The reason for such different results is given by the different methodology of evaluation in the individual groups for both taphonomy and the scoring system. Concerning just a taphonomy evaluation, G1 and G2 are the most problematic groups. Attributing NR depends both on conservation of inner and outer part for these groups. This may probably be the reason for a greater number of disagreements between the observers. Despite this, we observed less than 5% of errors. On the other hand, if developmental stage only (A, B nebo C) was evaluated, the best results were achieved in these groups, G1 and G2, including fibrocartilaginous insertions. For group G1, we recorded concordance rate 98%, for G2 97% (Table 2, Part III). These results show that the newly proposed methodology, from the aspect of these types of insertions, is reproducible very well. A similarly high degree of agreement was also achieved in the previous study [START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF], Table 3). The concordance rate for these two groups is better than in other comparative studies [START_REF] Crubézy | Épidémiologie de l'arthrose et des enthésopathies dans une population européenne d'il y a 7700 ans[END_REF][START_REF] Hawkey | Use of upper extremety enthesopathies to indicate habitual activity patterns[END_REF][START_REF] Mariotti | Enthesopathies -Proposal of a standardized scoring method and applications[END_REF]). The differences in inter-observer error between these studies may also be due to the different methodology of evaluating the incidence of enthesopathies. Frequently used methodology is method of E. Crubézy [START_REF] Crubézy | Interactions entre facteurs bio-culturels, pathologie et caractères discrets, Exemple d'une population médiévale : Canac (Aveyron)[END_REF][START_REF] Crubézy | Épidémiologie de l'arthrose et des enthésopathies dans une population européenne d'il y a 7700 ans[END_REF]. This methodology focuses mainly on the evaluation of enthesophytes. It does not evaluate lesions. Besides scopic evaluation, this methodology also uses metric assessment in the case of vertebral insertions. The methodology according to D. Hawkey (Hawkey 1998) is more precisely elaborated. It differentiates between the robusticity and two variants of a defect (enthesophytes and cortical lesions). It is specified for concrete muscle insertions. The methodology of V. Mariotti [START_REF] Mariotti | Enthesopathies -Proposal of a standardized scoring method and applications[END_REF]) is based on the previous two. It also evaluates separately robusticity and defects (enthesopathies). Concerning to enthesopathies, similarly as in the case of D. Hawkey, it differentiates between osteophytic (enthesophytes) and osteolytic (lesions, foramina, cysts) formations. It uses metric evaluation for both enthesopyhtes and lesions. None of these methodologies takes into account the anatomical and histological structure of the insertions. When calculating inter-observer error, scopic evaluation is always problematic. This evaluation is subjective to a various degree and impossible to verify. In methodology of S. Villotte, sub-groups were used for group G1 to render the final stages more precise. This form of evaluation may be one of the reasons for the greater degree of agreement between observers for group G1. In the group G2, the individual stages are defined metrically, thus the concordance rate is relatively high too. This does not hold for the remaining two groups -G3 (vertebral insertions) and G4 (fibrous insertions). The different results for group G3 in this study and that of S. [START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF]Table 3.) are probably due the incorrect understanding of the methodology in the first case. An error also may occur during measuring, when the gauge is placed differently or when enthesophytes are mistaken for osteophytes near the articular facets. The concordance rate for group G4, i.e. fibrous insertions, is the worst in both previously mentioned studies (Table 3.). Although the methodology of evaluation is detailed for each insertion (especially stage C) evaluation of fibrous insertions appears to be the most difficult and the resulting kappa coefficient is only good.

Based on these conclusions (and the results of inter-observer error in the study [START_REF] Villotte | Connaissances médicales actuelles, cotation des enthésopathies: nouvelle méthode[END_REF], this method was applied to another sample of 250 skeletons from Mikulčice (by the first author) and recent, documented human skeletal collections from Greece (Havelková, unpublished report) and from Portugal, Great Britain and Italy (Villotte, in prep.).

Conclusion:

Compared with currently used methodologies for evaluating muscle insertions and enthesopathies, we may conclude that the test of reproducibility of the new method is generally good, and for evaluating fibrocartilaginous insertions, it is excellent. This method may be applied on the archaeological material as well as on the recent bone collections.

Table 1 .

 1

	Group Muscle(s)

Evaluated insertion sites, their scoring groups (G1-G4) and the type of enthesis (FC -fibrocartilaginous, F -fibrous)

Table 2 .

 2

	Gr.	N -insertions	% of concordance	Kappa	% of concordance				% of concordance
	G1	342	92,7	0,89 almost perfect	95,0	0,88 almost perfect	228	96,4	0,93 almost perfect
	G2	97	92,8	0,90 almost perfect	95,9	0,90 almost perfect	65	95,5	0,93 almost perfect
	G3	544	84,9	0,71 substantial	97,4	0,72	good	511	86,7	0,72 substantial
	G4	227	83,7	0,76 substantial	96,9	0,87 almost perfect	193	84,5	0,74 substantial
	total	1210	87,5	0,81 almost perfect	96,5	0,87 almost perfect	997	89,0	0,80 almost perfect

Table 3 .

 3

		% of			% of
		concordance			concordance
	G1	85	0,75	substantial	96,4	0,93 almost perfect
	G2	100	1	perfect	95,5	0,93 almost perfect
	G3	92	0,82 almost perfect	86,7	0,72	substantial
	G4	89	0,62	substantial	84,5	0,74	substantial
	total	90	0,83 almost perfect	89,0	0,80 almost perfect

The comparison of reproducibility test results for the scoring system (ABC) with the study of Villotte (2006)

	Villotte, 2006	Havelková, Villotte
	ABC
	Kappa	Kappa

Acknowledgements:

The authors thank Prof. Karel Zvára for the assistance in the statistical analysis. This project was financed by grants from the GAČR 206/03/0699, Eco-Net No. 16368PB, the Czech Ministry of Culture VZ PM MK00002327201 and Research project MSM 0021620843.