Ronan Riochet
email: ronan.riochet@inria.fr

Mario Ynocente Castro

Mathieu Bernard

Adam Lerer

Rob Fergus

Véronique Izard

Emmanuel Dupoux

Mario Ynocente

Castro Mathieu Bernard

IntPhys 2019: A Benchmark for Visual Intuitive Physics Understanding

come

L'archive ouverte

Introduction

Despite impressive progress in machine vision on many tasks (face recognition [START_REF] Wright | Robust face recognition via sparse representation[END_REF], object recognition [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF], object segmentation [START_REF] Pinheiro | Learning to segment object candidates[END_REF], etc.), artificial systems are still far from human performance when it comes to common sense reasoning about objects in the world or understanding of complex visual scenes. Indeed, even very young children have the ability to represent macroscopic objects and track their interactions through time and space. Just a few days after birth, infants can parse their visual inputs into solid objects [START_REF] Valenza | Perceptual completion in newborn human infants[END_REF]. At 2-4 months, they understand object permanence, and recognize that objects should follow spatio-temporally continuous trajectories [START_REF] Kellman | Perception of partly occluded objects in infancy[END_REF][START_REF] Spelke | Spatiotemporal continuity, smoothness of motion and object identity in infancy[END_REF]. At 6 months, they understand the notion of stability, support and causality [START_REF] Saxe | The perception of causality in infancy[END_REF][START_REF] Baillargeon | The development of young infants' intuitions about support[END_REF][START_REF] Baillargeon | Is the top object adequately supported by the bottom object? young infants' understanding of support relations[END_REF]. Between 8 and 10 months, they grasp the notions of gravity, inertia, and conservation of momentum in collision; between 10 and 12 months, shape constancy [START_REF] Xu | Infants' metaphysics: The case of numerical identity[END_REF], and so on. Reverse engineering the capacity to autonomously learn and exploit intuitive physical knowledge would help building more robust and adaptable real life applications (self-driving cars, workplace or household robots).

Although very diverse vision tasks could benefit from some understanding of the physical world (see Figure 1), model of intuitive physics has been mostly developed through some form of future prediction task []. Being presented with inputs that can be pictures, video clips or actions to be performed in the case of a robot, the task is to predict future states of these input variables. Future prediction objectives have a lot of appeal because there is no need for human annotations, and abundant data can be collected easily. The flip side is that it is difficult to find the right metric to evaluate these systems. Even though pixel-wise prediction error can be a good loss function, it is not particularly interpretable, depends on the scale and resolution of the sensors making cross datasets comparison difficult, may not even rank the systems in a useful way: a good physics model could predict well the position of objects, but fail to reconstruct the color or texture of objects. In addition, Figure 1: Popular end-to-end applications involving scene understanding and proposed evaluation method based on physical plausibility judgments. 'Visual' tasks aim at recovering high level structure from low level (pixel) information: for instance, recovering 3D structure from static or dynamic images (e.g., [START_REF] Chang | Shapenet: An information-rich 3d model repository[END_REF][START_REF] Choy | 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction[END_REF]) or tracking objects (e.g., [START_REF] Kristan | A novel performance evaluation methodology for single-target trackers[END_REF][START_REF] Bertinetto | Fully-convolutional siamese networks for object tracking[END_REF]). 'Motor' tasks aim at predicting the visual outcome of particular actions (e.g., [START_REF] Finn | Unsupervised learning for physical interaction through video prediction[END_REF]) or to plan an action in order to reach a given outcome (e.g. [START_REF] Oh | Action-conditional video prediction using deep networks in atari games[END_REF]). 'Language tasks' requires the artificial system to translate input pixels into a verbal description, either through captioning [START_REF] Farhadi | Every picture tells a story: Generating sentences from images[END_REF] or visual question answering (VQA [START_REF] Zitnick | Bringing semantics into focus using visual abstraction[END_REF]). All of these tasks involve indirectly some notion of intuitive physics. Our proposed test directly measures physical understanding in a task-and model-agnostic way.

even though the laws of macroscopic physics are deterministic, in practice many outcomes are stochastic (this is why people play dice). In other words, the outcome of any interaction between object is a distribution of object positions, making the evaluation problem even harder.

Here, we propose to use an evaluation method which escapes these problems by using the prediction error not directly as a metric, but indirectly as informing a forced choice between two categories of events: possible versus impossible events. The intuition is the following. If a model has learned the laws of physics, it should be able to predict relatively accurately the future in video clips that show possible events, even if these videos are entirely novel. However, the model should give large prediction errors when some unlikely or impossible event happens. In other words, impossible events have a zero probability in the real world, so a more trained only with possible events should be able to generalize to other possible events, while rejecting impossible ones. This is directly inspired by the "violation of expectation" (VOE) paradigm in cognitive psychology, whereby infants or animals are presented with real or virtual animated 3D scenes which may contain a physical impossibility. The "surprise" reaction to the physical impossibility is measured through looking time or other physiological measures, and is taken to reflect a violation of it's internal predictions [START_REF] Baillargeon | Object permanence in five-month-old infants[END_REF]. Similarly, our evaluation requires systems to output a scalar variable upon the presentation of a video clip, which we will call a 'plausibility score' (it could be a log probability, an inverse reconstruction error, etc). We expect the plausibility score to be lower for clips containing the violation of a physical principle than for matched clips with no violation. By varying the nature of the physical violation, one can probe different types of physical laws (conservation of objects properties, objects movement, etc.). 1As in infant's experiments, our tests are constructed in well matched sets of clips, i.e., the possible versus impossible clips differ minimally, in order to minimize the possibility of dataset biases, but are quite varied, to maximize the difficulty of solving the test through simple heuristics. Three additional advantages of this method are that (1) they provide directly interpretable results (as opposed to a prediction error, or a composite score reflecting an entire pipeline), (2) they enable to probe generalization for difficult cases outside of the training distribution, which is useful for systems that are intended to work in the real world, and (3) they enable for rigorous human-machine comparison, which is important in order to quantify how far are artificial system in matching human intuitive physical understanding.

Our tests have also limits, which are the flip side of their advantage: They measure intuitive physics as looked through the prediction errors of a system, but do not measure how well a system might be able to use this kind of understanding. For instance, an end-to-end VQA system may have superb physical understanding (as measured by VOE) but fail miserably in connecting it with language. In this sense, VOE should be viewed as a diagnostic tool, a kind if unit testing for physics that needs to be combined with other measures to fully evaluate end-to-end systems. Similarly these tests do not exhaustively probe for all aspects of intuitive physics, but rather break it down into a small set of basic concepts tested one at a time. Here again, unit testing does not guarantee that an entire system will work correctly, but it helps to understand what happens when it does not.

This paper is structured as follows. In Section 2, we present the IntPhys 2019 Benchmark, which tests for 3 basic concepts of intuitive physics in a VOE paradigm. In Section 3, we describe two baseline systems which are trained with a self-supervised frame prediction objective on the training set, and in Section 4 we analyse their performance compared to that of humans participants. In Section 5 we present related work and conclude in Section 6 by discussing the next steps in extending this approach to more intuitive physics concepts and how they could be augmented to incorporate testing of decision and planning.

Structure of the IntPhys 2019 benchmark

IntPhys 2019 is a benchmark designed to address the evaluation challenges for intuitive physics in vision systems. It can be run on any of machine vision system (captioning and VQA systems, systems performing 3D reconstruction, tracking, planning, etc), be they engineered by hand or trained using statistical learning, the only requirement being that the tested system should output a scalar for each test video clip reflecting the plausibility of the clip as a whole. Such a score can be derived from prediction errors, or posterior probabilities, depending on the system.

In this release, which will be the first evaluation of the DARPA Machine Common Sense project, we have implemented tests for three basic concepts of the physics of macroscopic solid objects: object permanence, shape constancy, spatio-temporal continuity. Each of these concepts are tested in a series of controlled possible and impossible clips, which are presented without labels, and for which models have to return a plausibility score. The evaluation is done upon submission of these scores in CodaLab, and the results are automatically presented in a leaderboard. This benchmark also contains a training set of videos with random object interactions, in a similar environment as for the test set. This can be used either to train predictive systems or to conduct domain adaptation for systems trained on other datasets (live videos, virtual environments, robots). Obviously, the training set only contains physically possible events.

Three basic concepts of intuitive physics

Behavioral work on intuitive physics in infants and animal define a number of core conceptual components which can be tested experimentally using VEO [START_REF] Baillargeon | Core cognition and beyond[END_REF]. Figure 2 shows a number of different landmarks in infants. Here, we have selected three of the most basic components and turned them into three test blocks (see 1), each one corresponding to a core principle of intuitive physics, and each raising its particular machine vision challenge. The first two blocks are related to the conservation through time of intrinsic properties of objects. Object permanence (O1), corresponds to the fact that objects continuously exist through time and do not pop in or out of existence. This turns into the computational challenge of tracking objects through occlusion. The second block, shape constancy (O2) describes the tendency of rigid objects to preserve their shape through time. This principle is more challenging than the preceding one, because even rigid objects undergo a change in appearance due to other factors (illumination, distance, viewpoint, partial occlusion, etc.). The final block (O3) relate to object's trajectories, and posit that each object's motion has to be continuous through space and tile (an object cannot teleport from one place to another). This principle is distinct from object permanence and requires a to incorporate smoothness constraints on the tracking of objects (even if they are not visible). Future releases of the Benchmark will continue adding progressively more complex scenarios inpired by Figure 2, including object interactions and agent motion.

Pixels matched quadruplets

An important design principle of our evaluation framework relates to the organization of the possible and impossible movies in extremely well matched sets to minimize the existence of low level biases. This is illustrated in Figure 3 for object permanence. We constructed matched sets comprising four movies, which contain an initial scene at time t 1 (either one or two objects), and a final scene at time t 2 (either one or two objects), separated by a potential occlusion by a screen which is raised and then lowered for a variable amount of time. At its maximal height, the screen completely occludes the objects so that it is impossible to know, in 3: Illustration of the minimal sets design with object permanence. Schematic description of a static condition with one vs. two objects and one occluder. In the two possible movies (green arrows), the number of objects remains constant despite the occlusion. In the two impossible movies (red arrows), the number of objects changes (goes from 1 to 2 or from 2 to 1).

this frame, how many objects are behind the occluder.

The four movies are constructed by combining the two possible beginnings with the two possible endings, giving rise to two possible (1→1 and 2→2) and two impossible (1→2 and 2→1) movies. Importantly, across these 4 movies, the possible and impossible ones are made of frames with the exact same pixels, the only factor distinguishing them being the temporal coherence of these frames. Such a design is intended to make it difficult for algorithms to use cheap tricks to distinguish possible from impossible movies by focusing on low level details, but rather requires models to focus on higher level temporal dependencies between frames.

Parametric task complexity

Our second design principle is that in each block, we vary the stimulus complexity in a parametric fashion. In the case of the object permanence block, for instance, stimulus complexity can vary according to three dimensions. The first dimension is whether the change in number of objects occurs in plain view (visible) or hidden behind an occluder (occluded ). A change in plain view is evidently easier to detect whereas a hidden change requires an element of short term memory in order to keep a trace of the object's through time. The second dimension is the complexity of the object's motion. Tracking an immobile object is easier than if the object has a complicated motion; we introduce three levels of motion complexity (static, dynamic 1, and dynamic 2). The third dimension is the number of objects involved in the scene. This tests for the attentional capacity of the system as defined by the number of objects it can track simultaneously. Manipulating stimulus complexity is important to establish the limit of what a vision system can do, and where it will fail. For instance, humans are well known to fail when the number of objects to track simultaneously is greater than four [START_REF] Pylyshyn | Tracking multiple independent targets: Evidence for a parallel tracking mechanism[END_REF]. In total, a given block contains 2 by 3 by 3, ie, 18 different scenarios varying in difficulty (see Table x).

Procedurally generated variability

Our final design principle is that each scenario within each block is procedurally generated in 200 examplars with random variations in objects shapes and textures, distances, trajectories, occluder motion and position of the camera. This is to minimize the possibility of focusing on only certain frames or parts of the screen to solve the task. Note that the dynamic 2 condition contains two violations instead of one. These violations are inverses of one another, such that the first and last segment of the impossible video clips are compatible with with the absence of any violation in the central part of the video (for instance, the initial and final number of objects is the same, but varies in the middle of the clip). This insures that physical violations occur in unpredictable moments in a video clip.

Figure 4: Illustration of the 'dynamic 2' condition. In the two possible movies (green arrows), the number of objects remains constant despite the occlusion. In the two impossible movies (red arrows), the number of objects changes temporarily (goes from 0 to 1 to 0 or from 1 to 0 to 1).

The possible versus impossible discrimination metric

Our evaluation metrics depend on the system's ability to compute a plausibility score P (x) given a movie x. Because the test movies are structured in N matched k-uplets (in Figure 3, k = 4) of positive and negative movies S i=1..N = {P os 1 i ..P os k i , Imp 1 i ..Imp k i }, we derive two different metrics. The relative error rate L R computes a score within each set. It requires only that within a set, the positive movies are more plausible than the negative movies.

L R = 1 N i 1 j P (P os j i )< j P (Imp j i ) (1)
The absolute error rate L A requires that globally, the score of the positive movies is greater than the score of the negative movies. It is computed as:

L A = 1 -AU C({i, j; P (P os j i )}, {i, j; P (Imp j i )}) (2)
Where AU C is the Area Under the ROC Curve, which plots the true positive rate against the false positive rate at various threshold settings.

Implementation

The video clips in IntPhys 2019 are constructed with Unreal Engine 4.0 ((UnrealEnginePython 4.19; See Figure 5 for some examples). They are accessible in https://intphys.com/.

The training set

The training set contains a large variety of objects interacting one with another, occluders, textures, etc. It is composed of 15K videos of possible events (around 7 seconds each at 15fps), totalling 21 hours of videos. There are no video of impossible events, but the training set contains the objects and occluders presented in the test set. Each video is delivered as stacks of raw image (288 x 288 pixels), totalling 157Gb of uncompressed data. We also release the source code for data generation, allowing users to generate a larger training set if desired.

The dev and test sets

As described above, each of the three blocks contain 18 different scenario. In the dev set, each scenario is instantiated by 20 different renderings resulting in 360 movies per block (30 min, 3.7Gb). In the test set, a scenario has 200 different renderings of these scenarios, resulting in a total of 3600 movies per block (5h,37Gb). All of the objects and textures of the dev and test sets are present in the training set.

The purpose of the dev set released in IntPhys 2019 V1.0 is to help in the selection of an appropriate plausibility score, and in the comparison of various architectures (hyper-parameters), but it should not serve to train the model's parameters (this should be done only with the training set). This is why the dev set is kept intentionally small. The test set has more statistical power and enables a fine grained evaluation of the results across the different movie subtypes. Video examples of each blocks are available on the project page www.intphys.com/.

Metadata

Even though the spirit of IntPhys 2019 is the unsupervised learning of intuitive physics, we do provide in the test set additional information which may help the learner. The first one is the depth field for each image. This is not unreasonable, given that in infants, stereo vision and motion cues could provide an approximation of this information. The second one is object instance segmentation masks, which are helpful to recover abstract object positions but only provide local low-level information. Importantly, these masks are not linked to a specific object ID, and are randomly shuffled at each time frame. Linking instance segmentation masks to unique object IDs through time is indeed part of the object permanence problem that systems are supposed to solve. Similarly, if an object is partly occluded and appears as two pieces of object the two pieces will receive a different instance mask.

In the train set, we do provide additional metadata about the ground truth 3D position of each object, the position of the camera, and the link between object IDs and instance masks. These metadata are not present in the dev or test sets.

Submission procedure

For each movie in the dev or test set, the model should issue a scalar plausibility score. This number together with the movie ID is then fed to the evaluation software which outputs two tables of results, one for the absolute score and the other for the relative score.

The evaluation software is provided for the dev set, but not the test set. For evaluating on the test set, participants are invited to submit their system and results on CodaLab (see www.intphys.com) and their results will be registered and time-stamped on the website leaderboard.

Two baseline learning models

In this section, we present two learning systems which attempt to learn intuitive physics in an unsupervised/self-supervised observational setting. One can imagine an agent who only sees physical interactions between objects seen from a first-person perspective, but cannot move nor interact with them. Arguably, this is a much more empoverished learning situation than that faced by infants, who can explore and interact with their environment, even with the limited motor abilities of their first year of life. It is however interesting to establish how far one can get with such simplified inputs, which are easy to gather in abundant amounts in the real world with video cameras. In addition, this enables an easier comparison between models, because they all get the same training data.

In a setup like this, a rich source of learning information resides in the temporal dependencies between successive frames. Based on the literature on next frame prediction, we propose two neural network models, trained on a future frame objective. Our first model has a CNN encoder-decoder structure and the second is a conditional Generative Adversarial Network (GAN, [START_REF] Goodfellow | Generative adversarial nets[END_REF]), with a similar structure as DCGAN [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF]. For both model architectures, we investigate two different training procedures: in the first, we train models to predict short-future images with a prediction span of 5 frames; in the second, we predict long-future images with a prediction span of 35 frames.

Preliminary work with predictions at the pixel level revealed that our models failed at predicting convincing object motions, especially for small objects on a rich background. For this reason, we switched to computing predictions at a higher level, using object masks. We use the metadata provided in the benchmark training (see section ??) set to train a semantic mask Deep Neural Network (DNN). This DNN uses a resnet-18 pretrained on Imagenet to extract features from the image, from which a deconvolution network is trained to predict the semantic mask (distinguishing three types of entities: background, occluders and objects). We then use this mask as input to a prediction component which predicts future masks based on past ones.

To evaluate these models on our benchmark, our system needs to output a plausibility score for each movie. For this, we compute the prediction loss along the movie. Given past frames, a plausibility score for the frame f t can be derived by comparing f t with the prediction ft . Like in [START_REF] Fragkiadaki | Learning visual predictive models of physics for playing billiards[END_REF], we use the analogy with an agent running an internal simulation ("visual imagination"); here we assimilate a greater distance between prediction and observation with a lower plausibility. In subsection 3.2 we detail how we aggregate the scores of all frames into a plausibility score for the whole video.

Models

Through out the movie, our models take as input two frames (f i1 , f i2 ) and predict a future frame f target . The prediction span is independent from the model's architecture and depends only on the triplets (f i1 , f i2 , f target ) provided during the training phase. Our two architectures are trained either on a short term prediction task (5 frames in the future), or a long term prediction task (35 frames). Intuitively, short-term prediction will be more robust, but long-term prediction will allow the model to grasp long-term dependencies and deal with long occlusions.

CNN encoder-decoder

We use a resnet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] pretrained on Imagenet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] to extract features from input frames (f i1 , f i2 ). A deconvolution network is trained to predict the semantic mask of future frame f target conditioned to these features, using a L2 loss. See details in

Generative Adversarial Network

As a second model, we propose a conditional generative adversarial network (GAN, [START_REF] Mirza | Conditional generative adversarial nets[END_REF]) that takes as input predicted semantic masks from frames (f i1 , f i2 ), and predicts the semantic mask of future frame f target . In this setup, the discriminator has to distinguish between a mask predicted from f target directly (real ), and a mask predicted from past frames (f i1 , f i2 ). Like in [START_REF] Denton | Semi-supervised learning with context-conditional generative adversarial networks[END_REF], our model combines a conditional approach with a similar structure as of DCGAN [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF]. At test time, we derive a plausibility score by computing the conditioned discriminator's score for every conditioned frame. This is a novel approach based on the observation that the optimal discriminator D computes a score for x of

D(x) = P data (x) P G (x) + P data (x) (3)
For non-physical events x, P data (x) = 0; therefore, as long as P G (x) > 0, D(x) should be 0 for non-physical events, and D(x) > 0 for physical events x. Note that this is a strong assumption, as there is no guarantee that the generator will ever have support at the part of the distribution corresponding to impossible videos. The generator and discriminator are detailed in Table 3 and 4, respectively.

Training Procedure

We separate 10% of the training dataset to control the overfitting of our forward predictions. All our models are trained using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. For the CNN encoderdecoder we use Adam's default parameters and stop the training after one epoch. For the GAN, we use the same hyper-parameters as in [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF]: we set the generator's learning rate to 8e -4 and discriminator's learning rate to 2e -4. On the short-term prediction task, we train the GAN for 1 epoch; on the long-term prediction task we train it for 5 epochs. Learning rate decays are set to 0 and beta1 is set to 0.5 for both generator and discriminator.

The code for all our experiments is available on https://github.com/rronan/IntPhys-Baselines.

Video Plausibility Score

From forward models presented above, we can compute a plausibility score for every frame f target , conditioned to previous frames (f i1 , f i2 ). However, because the temporal positions of impossible events are not given, we must decide of a score for a video, given the scores of all its conditioned frames. An impossible event can be characterized by the presence of one or more impossible frame(s), conditioned to previous frames. Hence, a natural approach to compute a video plausibility score is to take the minimum of all conditioned frames' scores:

Plaus(v) = min (fi 1 ,fi 2 ,ftarget)∈v Plaus(f target |f i1 , f i2 ) (4)
where v is the video, and (f i1 , f i2 , f target ) are all the frame triplets in v, as given in the training phase.

Results

Block O1

Short-term prediction The first training procedure is a short-term prediction task; it takes as input frames f t-2 , f t and predicts f t+5 , which we note (f t-2 , f t ) → f t+5 in the following. We train the two architectures presented above on short-term prediction task and evaluate them on the test set. For the relative classification task, CNN encoder-decoder has an error rate of 0.09 when impossible events are visible and 0.49 when they are occluded. The GAN has an error rate of 0.15 when visible and 0.48 when occluded. For the absolute classification task, CNN encoder-decoder has a L A (see eq. 2) of 0.33 when impossible events are visible and 0.50 when they are occluded. The GAN has a L A of 0.38 when visible and 0.50 when occluded. Results are detailed in Supplementary Materials (Tables 1,2,3,4). We observe that our short-term prediction models show good performances when the impossible events are visible, especially on the relative classifications task. However they perform poorly when the impossible events are occluded. This is easily explained by the fact that they have a prediction span of 5 frames, which is usually lower than the occlusion time. Hence, these models don't have enough "memory" to catch occluded impossible events.

Long-term prediction

The second training procedure consists in a long-term prediction task: (f t-5 , f t ) → f t+35 . For the relative classification task, CNN encoder-decoder has an error rate of 0.07 when impossible events are visible and 0.52 when they are occluded. The GAN has an error rate of 0.17 when visible and 0.48 when occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.37 when impossible events are visible and 0.50 when they are occluded. The GAN has a L A of 0.40 when visible and 0.50 when occluded. Results are detailed in Supplementary Materials (Tables 5,6, 7, 8). As expected, long-term models perform better than short-term models on occluded impossible events. Moreover, results on absolute classification task confirm that it is way more challenging than the relative classification task. Because some movies are more complex than others, the average score of each quadruplet of movies may vary a lot. It results in cases where one model returns a higher plausibility score to an impossible movie M {imp, easy} from an easy quadruplet than to a possible movie M {pos, complex} from a complex quadruplet. Aggregated model On the relative classification task, the aggregated CNN encoder-decoder has an error rate of 0.07 when impossible events are visible and 0.52 when they are occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.37 when impossible events are visible and 0.50 when they are occluded. Results are detailed in Figures 6,7, 8 and Supplementary Materials (Tables 9,10).

Block O2

Short-term prediction For the first training procedure (f t-2 , f t ) → f t+5 : CNN encoder-decoder has an relactive classification error rate of 0.16 when impossible events are visible and 0.49 when they are occluded. The GAN has an error rate of 0.30 when visible and 0.52 when occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.40 when impossible events are visible and 0.50 when they are occluded. The GAN has a L A of 0.43 when visible and 0.50 when occluded. Results are detailed in Supplementary Materials (Tables 11,12, 13, 14).

Long-term prediction For the second training procedure (f t-5 , f t ) → f t+35 : the CNN encoder-decoder has an error rate of 0.11 when impossible events are visible and 0.52 when they are occluded. The GAN has an error rate of 0.31 when visible and 0.50 when occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.43 when impossible events are visible and 0.50 when they are occluded. The GAN has a L A of 0.33 when visible and 0.50 when occluded. Results are detailed in Supplementary Materials (Tables 15, [START_REF] Finn | Unsupervised learning for physical interaction through video prediction[END_REF][START_REF] Oh | Action-conditional video prediction using deep networks in atari games[END_REF][START_REF] Farhadi | Every picture tells a story: Generating sentences from images[END_REF]. Aggregated model On the relative classification task, the aggregated CNN encoder-decoder has an error rate of 0.11 when impossible events are visible and 0.52 when they are occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.43 when impossible events are visible and 0.50 when they are occluded. Results are detailed in Figures 6,7,8 and Supplementary Materials (Tables 19,20).

Block O3

Short-term prediction For the first training procedure (f t-2 , f t ) → f t+5 : CNN encoder-decoder has an relactive classification error rate of 0.28 when impossible events are visible and 0.49 when they are occluded. The GAN has an error rate of 0.26 when visible and 0.48 when occluded. For the absolute classification task, CNN encoder-decoder has a L A (see eq. 2) of 0.40 when impossible events are visible and 0.50 when they are occluded. The GAN has a L A of 0.42 when visible and 0.50 when occluded. Results are detailed in Supplementary Materials (Tables 21,22 1) for the relative performance and L A (see Equation 2) for the absolute performance. 1) for the relative performance and L A (see Equation 2) for the absolute performance.

Long-term prediction For the second training procedure (f t-5 , f t ) → f t+35 : the CNN encoder-decoder has an error rate of 0.32 when impossible events are visible and 0.51 when they are occluded. The GAN has an error rate of 0.34 when visible and 0.52 when occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.46 when impossible events are visible and 0.50 when they are occluded. The GAN has a L A of 0.44 when visible and 0.50 when occluded. Results are detailed in Supplementary Materials (Tables 25,26, 27, 28).

Aggregated model On the relative classification task, the aggregated CNN encoder-decoder has an error rate of 0.32 when impossible events are visible and 0.51 when they are occluded. For the absolute classification task, CNN encoder-decoder has a L A of 0.46 when impossible events are visible and 0.50 when they are occluded. Results are detailed in Figures 6,7,8 and Supplementary Materials (Tables 29,30).

As expected, we observe that models' performance decrease when impossible events are occluded. This enlightens the difficulty to perform long-term predictions in videos. We also observe that their performances vary with the types of impossible events tested. Results are the highest when testing presence / absence of object, and the lowest when testing the temporal continuity of trajectories. 1) for the relative performance and L A (see Equation 2) for the absolute performance.

Human Judgements Experiment

To give a second reference to evaluate physical understanding in models, and provide a good description of human performance on this benchmark, we presented the 3600 videos from each block to human participants using Amazon Mechanical Turk. Participants were first presented 8 examples of possible scenes from the training set, some simple, some more complex. They were told that some of the test movies were incorrect or corrupted, in that they showed events that could not possibly take place in the real world (without specifying how). Participants were each presented with 40 randomly selected videos, and were asked to score them from 1 (most implausible) to 6 (most plausible). They completed the task in about 7 minutes, and were paid $1. A total of 540 persons participated, such that every video tested was seen by 2 different participants. A mock sample of the AMT test is available on http: //129.199.81.135/naive_physics_experiment/.

The average error rates were computed across condition, number of objects and visibility and are shown in Tables 5,6, 7. In general, observers missed violations more often when the scene was occluded; we observe error rates going from 18% (visible) to 30% (occluded) for block O1, from 22% (visible) to 30% (occluded) for block O2, from 28% (visible) to 47% (occluded) for block O3. An interesting result is that the score of humans on block O3 is close to chance when objects are visible. This shows that humans have trouble to detect changes in velocity of objects, when these changes occur when the object is occluded. We also observe an increase in error going from static to dynamic 1 (one occlusion) and from dynamic 1 to dynamic 2 (two occlu-sions), but this pattern was only consistently observed in the occluded condition. For visible scenario, the dynamic 1 appeared more difficult than the dynamic 2. This was probably due to the fact that when objects are visible, the dynamic 2 impossible scenarios contain two local discontinuities and are therefore easier to spot than when one discontinuity only is present. When the discontinuities occurred behind the occluder, the pattern of difficulties was reversed, presumably because participants started using heuristics, such as checking that the number of objects at the beginning is the same as at the end, and therefore missed the intermediate disappearance of an object.

These results suggest that human participants are not responding according to the gold standard laws of physics due to limitations in attentional capacity -and this, even though the number of objects to track is below the theoretical limit of 4 objects. The performance of human observers can thus serve as a reference besides ground truth, especially for systems intended to model human perception.

Interestingly, we observe similar patterns of performance between models and humans (see Figures 6,7,8), with increasing error rates from blocks O1 to O3. As expected, both humans and models show higher error rates when the considered impossible event is occluded.

Related work

The modeling of intuitive physics has been addressed mostly through systems trained with some form of future prediction as a training objective. Some studies have investigated models for predicting the stability and forward modeling the dynamics of towers of blocks

). [START_REF] Battaglia | Simulation as an engine of physical scene understanding[END_REF] proposes a model based on an intuitive physics engine, [START_REF] Lerer | Learning physical intuition of block towers by example[END_REF] and [START_REF] Li | To fall or not to fall: A visual approach to physical stability prediction[END_REF] follow a supervised approach using Convolutional Neural Networks (CNNs), [START_REF] Zhang | A comparative evaluation of approximate probabilistic simulation and deep neural networks as accounts of human physical scene understanding[END_REF] makes a comparison between simulationbased models and CNN-based models, [START_REF] Mirza | Generalizable features from unsupervised learning[END_REF] improves the predictions of a CNN model by providing it with a prediction of a generative model. In [START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF], authors propose different feature learning strategies (multi-scale architecture, adversarial training method, image gradient difference loss function) to predict future frames in raw videos.

Other models use more structured representation of objects to derive longer-term predictions. In [START_REF] Battaglia | Interaction networks for learning about objects, relations and physics[END_REF] and [START_REF] Chang | A compositional object-based approach to learning physical dynamics[END_REF], authors learn objects dynamics by modelling their pairwise interactions and predicting the resulting objects states representation (e.g. position / velocity / object intrinsic properties) . In [START_REF] Watters | Visual Interaction Networks[END_REF], [START_REF] Marco Fraccaro | A disentangled recognition and nonlinear dynamics model for unsupervised learning[END_REF] and [START_REF] Ehrhardt | Learning a physical long-term predictor[END_REF] authors combine factored latent object representations, object centric dynamic models and visual encoders. Each frame is parsed into a set of object state representations, which are used as input of a dynamic model. In [START_REF] Marco Fraccaro | A disentangled recognition and nonlinear dynamics model for unsupervised learning[END_REF] and [START_REF] Ehrhardt | Learning a physical long-term predictor[END_REF], authors use a visual decoder to reconstruct the future frames, allowing the model to learn from raw (though synthetic) videos.

Regarding evaluation and benchmarks, apart from frame prediction datasets, which are not strictly speaking about intuitive physics, one can distinguish the Visual Newtonian Dynamics (VIND) dataset which includes more than 6000 videos with bounding boxes on key objects across frames, and annotated with a 3D plane which would most closely fit the object trajectory [START_REF] Mottaghi | learning to predict the effect of forces in images[END_REF]. There is also recent dataset proposed by a DeepMind team [START_REF] Piloto | Probing physics knowledge using tools from developmental psychology[END_REF]. This last dataset seems very similar to ours. It is also inspired by the developmental literature and based on the violation of expectation principles and is structured around 3 blocks similar to our first 3 blocks (object permanence, shape constancy, continuity) and two other ones (solidity and containment). The number and characteristics of this dataset is not known at present. From the sample videos, two differences emerged: our dataset is better matched in terms of quadruplets of clips controlled at the level of the pixels, and our dataset has a factorial manipulation of scene and movement complexity. It would be interesting to explore the possibility to merge these two datasets, as well as add more blocks in order to increase the diversity and coverage of the physical phenomena.

Discussion

We presented IntPhys 2019, a benchmark for measuring intuitive physics in artificial vision systems inspired by research on conceptual development in infants. To pass the benchmark, a system is asked to return a plausibility score for each video clip. The system's performance is assessed by measuring its ability to discriminate possible from impossible videos illustrating several types of physical principles. Naive humans were tested on the same dataset, to give an idea of what performance could be expected by a good model. These results show error rates increasing with the presence of occlusion, but not with number of objects. This is congruent with data showing that humans can track up to three objects simultaneously. We presented two unsupervised learning models based on semantic masks, which learn from a training set only composed of physically plausible clips, and are tested on the same block as the humans. The computational system generally performed poorly compared to humans but obtained above chance performance using a mask prediction task, with a very strong effect of the presence of occlusion. The relative success of the semantic mask prediction system compared to what we originally found with pixel-based systems indicates that operating at a more abstract level is a worthwhile pursuing strategy when it comes to modeling intuitive physics. Future work will explore alternative ways of constructing this abstract representation in particular instance masks and object detection bounding boxes. In addition, enriching the training through embedding the learner in an interactive version of the environment could add more information for the learning of the physics of macroscopic objects.

In brief, the systematic way of constructing the Int-Phys 2019 Benchmark shows that it is possible to adapt developmental paradigm in a machine learning setting, and that the resulting benchmark is a relatively challenging one. The three blocks that we present here could be extended to cover more aspects of object perception, including more difficult ones like interactions between objects, or prediction of trajectories of animated agents. As we discussed in the introduction, this benchmark only provides unit tests regarding the computation of prediction probabilities of object positions based on past frames. Further work will be needed to construct benchmarks testing how theses probabilities can be used by a system to make decision or plan trajectories.

Figure 2 :

 2 Figure 2: Landmark of intuitive physics acquisition in infants. Each box is an experiment showing a particular ability at a given age.

Figure 5 :

 5 Figure 5: Examples of frames from the training set.

  , 23, 24).

Figure 6 :

 6 Figure 6: Results of our baselines on block O1, in cases where the impossible event occurs in the open (visible) or behind an occluder (occluded ). Y-axis represents the losses L R (see Equation1) for the relative performance and L A (see Equation2) for the absolute performance.

Figure 7 :

 7 Figure 7: Results of our baselines on block O2, in cases where the impossible event occurs in the open (visible) or behind an occluder (occluded ). Y-axis represents the losses L R (see Equation1) for the relative performance and L A (see Equation2) for the absolute performance.

Figure 8 :

 8 Figure 8: Results of our baselines on block O3, in cases where the impossible event occurs in the open (visible) or behind an occluder (occluded ). Y-axis represents the losses L R (see Equation1) for the relative performance and L A (see Equation2) for the absolute performance.

Table 1 :

 1 List of the conceptual blocks of the Intuitive Physics Framework. Spatio-temporal continuity Trajectories of objects are continuous Tracking/predicting object trajectories

	Block Name	Physical principles	Computational challenge
	O1. Object permanence	Objects don't pop in and out of existence Occlusion-resistant object tracking
	O2. Shape constancy	Objects keep their shapes	Appearance-robust object tracking
	O3. Figure

Table 2 Table 2 :

 22 CNN for forward prediction (13941315 parameters). BN stands for batch-normalization.

	Input frames
	2 x 3 x 64 x 64
	7 first layers of resnet-18 (pretrained, frozen weights)
	applied to each frame
	Reshape 1 x 16384
	FC 16384 → 512
	FC 512 → 8192
	Reshape 128 x 8 x 8
	UpSamplingNearest(2), 3 x 3 Conv. 128 -1 str., BN, ReLU
	UpSamplingNearest(2), 3 x 3 Conv. 64 -1 str., BN, ReLU
	UpSamplingNearest(2), 3 x 3 Conv. 3 -1 str., BN, ReLU
	3 sigmoid
	Target mask

Table 3 :

 3 Generator G (14729347 parameters). SFConv stands for spatial full convolution and BN stands for batch-normalization.

	Input masks
	2 x 3 x 64 x 64
	4 x 4 conv 64 -2 str., BN, ReLU
	4 x 4 conv 128 -2 str., BN, ReLU
	4 x 4 conv 256 -2 str., BN, ReLU	Noise ∈ R 100
	4 x 4 conv 512 -2 str., BN, ReLU	∼ Unif(-1, 1)
	4 x 4 conv 512, BN, ReLU
	stack input and noise
	4 x 4 SFConv. 512 -2 str., BN, ReLU
	4 x 4 SFConv. 256 -2 str., BN, ReLU
	4 x 4 SFConv. 128 -2 str., BN, ReLU
	4 x 4 SFConv. 64 -2 str., BN, ReLU
	4 x 4 SFConv. 3 -2 str., BN, ReLU
	3 sigmoid
	Target mask

Table 4 :

 4 Discriminator D (7629698 parameters). BN stands for batch-normalization.

	history	input
	2 x 3 x 64 x 64 3 x 64 x 64
	Reshape 3 x 3 x 64 x 64
	4 x 4 convolution 512 -2 strides, BN, LeakyReLU
	4 x 4 convolution 254 -2 strides, BN, LeakyReLU
	4 x 4 convolution 128 -2 strides, BN, LeakyReLU
	4 x 4 convolution 64 -2 strides, BN, LeakyReLU
	4 x 4 convolution 5 -2 strides, BN, LeakyReLU
	fully-connected layer
	1 sigmoid

Table 5 :

 5 Average error rate on plausibility judgments collected in humans using MTurk for the IntPhys 2019(Block O1) test set.

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.13	0.14	0.09	0.12	0.32	0.34	0.28	0.31
	Dynamic (1 violation)	0.15	0.29	0.27	0.24	0.24	0.30	0.33	0.29
	Dynamic (2 violations)	0.14	0.20	0.23	0.19	0.28	0.26	0.36	0.30
	Total	0.14	0.21	0.20	0.18	0.28	0.30	0.32	0.30

Table 6 :

 6 Average error rate on plausibility judgments collected in humans using MTurk for the IntPhys 2019(Block O2) test set.

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.13	0.18	0.15	0.16	0.22	0.33	0.28	0.28
	Dynamic (1 violation)	0.29	0.24	0.27	0.27	0.29	0.35	0.29	0.31
	Dynamic (2 violations)	0.21	0.27	0.26	0.24	0.32	0.32	0.29	0.31
	Total	0.21	0.23	0.23	0.22	0.28	0.33	0.29	0.30
	( [31, 32,

Table 7 :

 7 Average error rate on plausibility judgments collected in humans using MTurk for the IntPhys 2019(Block O3) test set.

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.29	0.32	0.27	0.29	0.36	0.36	0.45	0.39
	Dynamic (1 violation)	0.28	0.33	0.30	0.30	0.49	0.55	0.49	0.51
	Dynamic (2 violations)	0.23	0.23	0.26	0.24	0.47	0.53	0.55	0.52
	Total	0.27	0.29	0.28	0.28	0.44	0.48	0.50	0.47

Table 8 :

 8 Block O1 | Model: CNN (short-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.49	0.52	0.41	0.47
	Dynamic (1 violation)	0.00	0.22	0.27	0.17	0.51	0.47	0.49	0.49
	Dynamic (2 violations)	0.00	0.13	0.20	0.11	0.50	0.50	0.49	0.50
	Total	0.00	0.12	0.16	0.09	0.50	0.50	0.46	0.49

Table 9 :

 9 Block O1 | Model: CNN (short-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.15	0.17	0.19	0.17	0.50	0.50	0.49	0.50
	Dynamic (1 violation)	0.32	0.44	0.47	0.41	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.33	0.43	0.47	0.41	0.50	0.50	0.50	0.50
	Total	0.26	0.35	0.38	0.33	0.50	0.50	0.50	0.50

Table 10 :

 10 Block O1 | Model: GAN (short-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.44	0.45	0.53	0.48
	Dynamic (1 violation)	0.00	0.35	0.39	0.25	0.44	0.50	0.47	0.47
	Dynamic (2 violations)	0.00	0.21	0.39	0.20	0.51	0.50	0.49	0.50
	Total	0.00	0.18	0.26	0.15	0.46	0.48	0.50	0.48

Table 11 :

 11 Block O1 | Model: GAN (short-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.23	0.31	0.32	0.28	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.33	0.47	0.50	0.43	0.49	0.49	0.50	0.49
	Dynamic (2 violations)	0.34	0.44	0.46	0.41	0.50	0.50	0.50	0.50
	Total	0.30	0.41	0.43	0.38	0.49	0.50	0.50	0.50
	Table 12: Block O1 | Model: CNN (long-term prediction task) | Relative classification (L R )
			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.52	0.55	0.51	0.53
	Dynamic (1 violation)	0.00	0.13	0.22	0.12	0.49	0.53	0.48	0.50
	Dynamic (2 violations)	0.00	0.06	0.20	0.09	0.53	0.48	0.60	0.54
	Total	0.00	0.06	0.14	0.07	0.51	0.52	0.53	0.52

Table 13 :

 13 Block O1 | Model: CNN (long-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.30	0.34	0.36	0.33	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.30	0.43	0.44	0.39	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.32	0.40	0.43	0.39	0.50	0.50	0.50	0.50
	Total	0.31	0.39	0.41	0.37	0.50	0.50	0.50	0.50

Table 14 :

 14 Block O1 | Model: GAN (long-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.01	0.00	0.00	0.41	0.58	0.57	0.52
	Dynamic (1 violation)	0.00	0.28	0.45	0.24	0.39	0.56	0.54	0.50
	Dynamic (2 violations)	0.01	0.29	0.46	0.25	0.43	0.46	0.40	0.43
	Total	0.00	0.19	0.30	0.17	0.41	0.54	0.50	0.48

Table 15 :

 15 Block O1 | Model: GAN (long-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.26	0.33	0.37	0.32	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.36	0.46	0.49	0.44	0.49	0.50	0.50	0.50
	Dynamic (2 violations)	0.35	0.47	0.48	0.43	0.50	0.50	0.50	0.50
	Total	0.32	0.42	0.45	0.40	0.50	0.50	0.50	0.50

Table 16 :

 16 Block O1 | Model: CNN aggregated | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.52	0.55	0.51	0.53
	Dynamic (1 violation)	0.00	0.13	0.22	0.12	0.49	0.53	0.48	0.50
	Dynamic (2 violations)	0.00	0.06	0.20	0.09	0.53	0.48	0.60	0.54
	Total	0.00	0.06	0.14	0.07	0.51	0.52	0.53	0.52
	Table 17: Block O1 | Model: CNN aggregated | Absolute classification (L A )
			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.30	0.34	0.36	0.33	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.30	0.43	0.44	0.39	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.32	0.40	0.43	0.39	0.50	0.50	0.50	0.50
	Total	0.31	0.39	0.41	0.37	0.50	0.50	0.50	0.50

Table 18 :

 18 Block O2 | Model: CNN (short-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.48	0.49	0.47	0.48
	Dynamic (1 violation)	0.18	0.26	0.40	0.28	0.50	0.49	0.50	0.50
	Dynamic (2 violations)	0.12	0.16	0.32	0.20	0.50	0.50	0.50	0.50
	Total	0.10	0.14	0.24	0.16	0.49	0.49	0.49	0.49

Table 19 :

 19 Block O2 | Model: CNN (short-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.22	0.29	0.28	0.26	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.46	0.48	0.48	0.48	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.46	0.47	0.48	0.47	0.50	0.50	0.50	0.50
	Total	0.38	0.42	0.42	0.40	0.50	0.50	0.50	0.50

Table 20 :

 20 Block O2 | Model: GAN (short-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.54	0.55	0.47	0.52
	Dynamic (1 violation)	0.44	0.38	0.47	0.43	0.56	0.52	0.54	0.54
	Dynamic (2 violations)	0.38	0.52	0.51	0.47	0.50	0.50	0.48	0.49
	Total	0.27	0.30	0.33	0.30	0.53	0.52	0.50	0.52

Table 21 :

 21 Block O2 | Model: GAN (short-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.29	0.30	0.32	0.30	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.49	0.49	0.49	0.49	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.48	0.49	0.50	0.49	0.50	0.50	0.50	0.50
	Total	0.42	0.43	0.44	0.43	0.50	0.50	0.50	0.50
	Table 22: Block O2 | Model: CNN (long-term prediction task) | Relative classification (L R )
			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.01	0.00	0.50	0.47	0.52	0.50
	Dynamic (1 violation)	0.13	0.22	0.25	0.20	0.51	0.50	0.55	0.52
	Dynamic (2 violations)	0.11	0.10	0.17	0.13	0.56	0.49	0.53	0.53
	Total	0.08	0.11	0.14	0.11	0.52	0.49	0.54	0.52

Table 23 :

 23 Block O2 | Model: CNN (long-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.34	0.41	0.40	0.38	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.43	0.45	0.46	0.45	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.43	0.44	0.46	0.45	0.50	0.50	0.50	0.50
	Total	0.40	0.43	0.44	0.43	0.50	0.50	0.50	0.50

Table 24 :

 24 Block O2 | Model: GAN (long-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.02	0.02	0.00	0.01	0.49	0.55	0.53	0.52
	Dynamic (1 violation)	0.35	0.42	0.54	0.44	0.50	0.40	0.45	0.45
	Dynamic (2 violations)	0.44	0.51	0.53	0.50	0.56	0.44	0.53	0.51
	Total	0.27	0.32	0.36	0.31	0.52	0.46	0.51	0.50

Table 25 :

 25 Block O2 | Model: GAN (long-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.40	0.39	0.38	0.39	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.47	0.49	0.50	0.49	0.50	0.49	0.50	0.50
	Dynamic (2 violations)	0.47	0.50	0.50	0.49	0.50	0.50	0.50	0.50
	Total	0.45	0.46	0.46	0.46	0.50	0.50	0.50	0.50

Table 26 :

 26 Block O2 | Model: CNN aggregated | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.01	0.00	0.50	0.47	0.52	0.50
	Dynamic (1 violation)	0.13	0.22	0.25	0.20	0.51	0.50	0.55	0.52
	Dynamic (2 violations)	0.11	0.10	0.17	0.13	0.56	0.49	0.53	0.53
	Total	0.08	0.11	0.14	0.11	0.52	0.49	0.54	0.52

Table 27 :

 27 Block O2 | Model: CNN aggregated | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.34	0.41	0.40	0.38	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.43	0.45	0.46	0.45	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.43	0.44	0.46	0.45	0.50	0.50	0.50	0.50
	Total	0.40	0.43	0.44	0.43	0.50	0.50	0.50	0.50

Table 28 :

 28 Block O3 | Model: CNN (short-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.48	0.43	0.46	0.46
	Dynamic (1 violation)	0.34	0.38	0.46	0.39	0.47	0.48	0.50	0.49
	Dynamic (2 violations)	0.47	0.45	0.45	0.45	0.52	0.51	0.53	0.52
	Total	0.27	0.27	0.30	0.28	0.49	0.47	0.49	0.49

Table 29 :

 29 Block O3 | Model: CNN (short-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.22	0.21	0.23	0.22	0.50	0.49	0.50	0.50
	Dynamic (1 violation)	0.49	0.49	0.49	0.49	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.49	0.49	0.50	0.49	0.50	0.50	0.50	0.50
	Total	0.40	0.40	0.41	0.40	0.50	0.50	0.50	0.50

Table 30 :

 30 Block O3 | Model: GAN (short-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.00	0.00	0.00	0.47	0.43	0.37	0.43
	Dynamic (1 violation)	0.31	0.45	0.43	0.40	0.50	0.47	0.54	0.50
	Dynamic (2 violations)	0.34	0.42	0.43	0.40	0.48	0.52	0.54	0.51
	Total	0.22	0.29	0.29	0.26	0.48	0.48	0.48	0.48

Table 31 :

 31 Block O3 | Model: GAN (short-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.29	0.33	0.30	0.31	0.50	0.49	0.50	0.50
	Dynamic (1 violation)	0.46	0.49	0.49	0.48	0.50	0.50	0.51	0.50
	Dynamic (2 violations)	0.44	0.47	0.47	0.46	0.50	0.50	0.50	0.50
	Total	0.40	0.43	0.42	0.42	0.50	0.50	0.50	0.50

Table 32 :

 32 Block O3 | Model: CNN (long-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.02	0.00	0.00	0.01	0.48	0.49	0.47	0.48
	Dynamic (1 violation)	0.45	0.52	0.43	0.47	0.54	0.48	0.53	0.52
	Dynamic (2 violations)	0.56	0.45	0.44	0.48	0.51	0.59	0.52	0.54
	Total	0.35	0.32	0.29	0.32	0.51	0.52	0.51	0.51

Table 33 :

 33 Block O3 | Model: CNN (long-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.35	0.36	0.40	0.37	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.50	0.50	0.50	0.50	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.51	0.50	0.50	0.50	0.50	0.50	0.50	0.50
	Total	0.45	0.45	0.47	0.46	0.50	0.50	0.50	0.50

Table 34 :

 34 Block O3 | Model: GAN (long-term prediction task) | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.01	0.00	0.00	0.00	0.53	0.53	0.59	0.55
	Dynamic (1 violation)	0.53	0.50	0.60	0.54	0.55	0.55	0.48	0.53
	Dynamic (2 violations)	0.42	0.51	0.54	0.49	0.43	0.52	0.52	0.49
	Total	0.32	0.34	0.38	0.34	0.50	0.53	0.53	0.52

Table 35 :

 35 Block O3 | Model: GAN (long-term prediction task) | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.29	0.33	0.35	0.32	0.50	0.50	0.51	0.50
	Dynamic (1 violation)	0.50	0.49	0.52	0.50	0.50	0.51	0.50	0.50
	Dynamic (2 violations)	0.50	0.49	0.50	0.50	0.50	0.50	0.50	0.50
	Total	0.43	0.44	0.46	0.44	0.50	0.50	0.50	0.50

Table 36 :

 36 Block O3 | Model: CNN aggregated | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.02	0.00	0.00	0.01	0.48	0.49	0.47	0.48
	Dynamic (1 violation)	0.45	0.52	0.43	0.47	0.54	0.48	0.53	0.52
	Dynamic (2 violations)	0.56	0.45	0.44	0.48	0.51	0.59	0.52	0.54
	Total	0.35	0.32	0.29	0.32	0.51	0.52	0.51	0.51

Table 37 :

 37 Block O3 | Model: CNN aggregated | Absolute classification (L A )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.35	0.36	0.40	0.37	0.50	0.50	0.50	0.50
	Dynamic (1 violation)	0.50	0.50	0.50	0.50	0.50	0.50	0.50	0.50
	Dynamic (2 violations)	0.51	0.50	0.50	0.50	0.50	0.50	0.50	0.50
	Total	0.45	0.45	0.47	0.46	0.50	0.50	0.50	0.50

Table 38 :

 38 Block O1 | Human evaluation | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.01	0.06	0.00	0.02	0.12	0.22	0.20	0.18
	Dynamic (1 violation)	0.04	0.19	0.18	0.14	0.06	0.12	0.17	0.12
	Dynamic (2 violations)	0.04	0.25	0.09	0.13	0.26	0.10	0.13	0.16
	Total	0.03	0.17	0.09	0.10	0.15	0.15	0.17	0.15

Table 39 :

 39 Block O2 | Human evaluation | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.00	0.03	0.02	0.02	0.14	0.18	0.17	0.16
	Dynamic (1 violation)	0.16	0.04	0.22	0.14	0.12	0.23	0.09	0.15
	Dynamic (2 violations)	0.17	0.25	0.33	0.25	0.20	0.23	0.18	0.20
	Total	0.11	0.11	0.19	0.14	0.15	0.21	0.15	0.17

Table 40 :

 40 Block O3 | Human evaluation | Relative classification (L R )

			Visible				Occluded
	Type of scene	1 obj. 2 obj. 3 obj. Total	1 obj. 2 obj. 3 obj. Total
	Static	0.23	0.10	0.24	0.19	0.32	0.17	0.40	0.30
	Dynamic (1 violation)	0.24	0.29	0.32	0.28	0.44	0.60	0.50	0.51
	Dynamic (2 violations)	0.06	0.21	0.20	0.16	0.38	0.57	0.44	0.46
	Total	0.18	0.20	0.25	0.21	0.38	0.45	0.45	0.42

This has a direct parallel in 'black box' evaluation of language models in NLP. Language models are typically trained with a future prediction objective (predicting future characters or words conditioned on past ones). However, instead of evaluating theses models directly on the loss function or derivatives like perplexity, an emerging research direction is to the models on artificially constructed sentences that violate certain grammatical rules (like number agreement) measure the ability of the system to detect these violations[START_REF] Linzen | Assessing the ability of lstms to learn syntax-sensitive dependencies[END_REF].


Erik Olin Wright (1947Wright ( -2019)): classes and utopia TARRIT Fabien * All past history, with the exception of its primitive stages, was the history of class struggle [START_REF] Engels | Socialism: utopian and scientific[END_REF].

This claim has sometimes been used in a simplistic way, as if class relations could exclusively explain all social issues, omitting the interaction between a vast number of other explanatory items. Besides, world-wide economic and social transformations since the end of the Second World War have notably revolved around social classes, and new questions appeared on that issue. We display three groups of authors according to their use of that notion 1 .

Marxists authors, like Edward P. [START_REF] Thompson | The Making of the English working class, Vintage[END_REF], view classes as a historical process which cannot be analysed in a formal way. Class experience is determined by relations of production, so that a social class is defined by people who live their own history and then class consciousness is the cultural achievement of their own experience.

Non Marxist authors including Barry Hindess and Paul Hirst (1978) articulate classes around categories of distribution with no reference to domination. Since social groups are articulated around distributional categories, relations of domination only exist as incidents.

They admit the existence of social classes, but they claim that there is no necessary correspondence between classes and politics, and that political struggles are not a fight between classes. Such an approach rests upon a Weberian background, and it leads to a system of stratification, which distributes the population in continuous levels, in the middle of which the "middle class". It leads to empirical descriptions and to statistical data that are autonomous from the historical record, and then the analysis of social differentiation gets separated from the analysis of conflict.

Others non Marxist authors articulate classes around categories of domination with no reference to distribution. Ralf [START_REF] Dahrendorf | Class and Class conflict in Industrial Society[END_REF] defines classes in exclusive terms of power. He criticises the consequences of Weber's theory on Marx's classes without completely rejecting it: his approach could be characterised as an objective pluralism, namely that many groups are generated by objective relations. He criticises Marx's theory as a historicism, which would ignore continuous minor changes, and for that reason, would not relate to historical record.

Those contradictory issues lead us to analytical Marxism, with Erik O. Wright's contributionhe defends an approach in terms of contradictory class locations -and its interaction with John Roemer's, complemented by Gerald A. Cohen's. Analytical Marxism's insight is an attempt to reconstruct Marx's theoretical corpus with methods that are traditionally considered as contradictory with it 2 . This paper is specifically aimed to study its contribution to the understanding of the concept of social class within the current capitalism. Class transformation will be dealt as an aspect of the inner dynamics of capitalism.

The paper is articulated as follows. First, we display how Wright reconstructs the Marxian theory of classes with the concept of contradictory class locations (1). Then, we analyse how such an approach corresponds to an alliance between Marxism and non Marxist social science (2). Next, we show how Roemer relates class theory with exploitation through game theory (3). Finally, we describe how Cohen makes the issue of social classes a normative issue rather than a structural one, and we propose that dialectics is necessary to avoid it (4). We will attempt to consider the necessity to integrate the issue of social classes within the historical process of social transformation.

Middle classes and contradictory class locations

Until recently it has been widely claimed that the transformations of capitalism after the Second World War allowed the development of a new social class, known as the middle class, which is neither proletarian nor bourgeois, and this would blur the traditional class distinctions. Wright dealt with such an evolution with a view to preserve the Marxian concept of social classes. We discuss how he considers the issue of the middle class (1.1), and then we study how he answers to that challenge with the concept of the contradictory class locations (1.2).

Classes in changing social structure

Social classes must be understood in the context of a noticeable change in the social structure appreciably since the Fifties. Wright intends to save Marx's social classes (1.1.1), which obliges to grasp the distinction between profession and class (1.1.2). He introduces a new approach on the middle class (1.1.3).

Marx's classes in danger

It is often claimed that Marx did not anticipate the consequences of capitalist development on the evolution of class structure. The evolution of the pattern of social locations in the production system would separate, on the one hand, the proletariat as a group of persons that are excluded from the means of production from, on the other hand, the proletariat as a set of productive workers. So, it seems that there are indeterminate social relations in terms of class.

Such an evolution stimulated Wright's research program. Social classes have been his main research subject3 . He reformulates the explanatory content of the Marxian concept of social class. He questions its central role through the structural dynamics of capital, i.e. labour relationship, with a view to better taking into account the differentiated character of the contemporary capitalist class structure, without dropping the idea of class polarization. He aims to reassert the specificity of the Marxian class concept in enlarging the definition of class beyond the production process.

A fuzzy boarder between professional structure and class structure

He examines the interactions between class and profession, with four concepts -class formation, class struggle, class structure and professional structure -and he presents four sets of claims:

1. Class structure sets limits on class formation, on class consciousness and on class struggle.

2. Class structure is the main social frontier in social change.

3. The concept of class deals with social relations.

4. Social relations are antagonistic rather than symmetrical, and exploitation would be the objective foundation of those relations.

I am not arguing that class structures define a unique path of social development. Rather the claim is that class structures constitute the lines of demarcation in trajectories of social change. There is no teleological implication that there is a 'final destination' towards which all social change inexorably moves. (Wright 1985a: 32) He examines the distinction between "white collar workers" and "blue collar workers", wondering if it appears within the working class or between classes. In the latter case, it would be a foundation for an additional level in the class structure, and he concludes that, for allowing an understanding of class struggle and of class formation, class structure cannot be analysed for itself.

Not only should class structures be viewed as setting the basic limits of possibility on class formation, class consciousness and class struggle, but they also constitute the most fundamental social determinant of limits of possibility for others aspects of social structure. (Wright 1985a: 31)

Wright's original approach on middle classes

He claims that the Marxists have never given a satisfying answer to the issue of the "middle class". He displays four sets of answers, within Marxian academics:

1. The middle class is an ideological illusion. This approach would be the traditional Marxist one, and Wright considers it as deficient since it avoids the problem.

2. The middle class is a part of another class [START_REF] Poulantzas | Political Powerr and Social Classes[END_REF].

3. The middle class is a new class [START_REF] Gouldner | The Future of Intellectuals and the Rise of the New Class[END_REF].

4. The middle class stands in various classes, which would correspond to the existence of contradictory class locations (Wright 1985a).

For Wright, two kinds of non polarized positions appear: the traditional petite-bourgeoisie, which is neither exploited nor exploiter, and cross relations of exploitation, which result into being both exploiter and exploited. Besides, extra-classes identifications would stimulate internal divisions -like racial, ethnic, linguistic, national, religious ones -within the working class and they would blur class distinctions4 .

The conceptualization of contradictory class locations

Wright's answer to the middle class is the concept of contradictory class locations. We will first give a brief account of this concept (1.2.1), before stating that it is a peculiar interpretation of Marx's theory (1.2.2).

1.2.1.The birth of a concept

For Wright, the capitalists do not implement all the functions of capital, and the non polarized categories described above are contradictory rather than intermediate, both because they are in two classes at the same time and because they share interests of the two classes, hence their ambiguous role in the class struggle.

Instead of regarding all positions as located uniquely within particular classes and thus as having a coherent class character in their own right, we should see some positions as possibly having a multiple class character; they may be in more than one class simultaneously. (Wright, 1985a: 43) In order to specify the way relations of production are the mechanism of exploitation, he displays three essential and interrelated dimensions: the control on money capital (investment), the control on physical capital (means of production) and the control on labour (production process). The pure capitalist exploitation would then rest upon the exclusion of the workers, by the capitalists, from all those relations.

He claims that the pure capitalist mode of production is not straightforwardly operational, since the exclusion from the three dimensions above does not necessarily coincide, so that there would be "contradictory locations within class relations" (Idem), mainly the managers, the employers in small or medium-sized business and the semi-autonomous wage-earners.

The managers control labour and a part of the physical capital, but not money capital; they are socially situated between the working class and the capitalist class. The employers in small or medium-sized business control labour and physical capital, but not money capital; they are socially situated between the petite-bourgeoisie and the capitalist class. The semiautonomous wage-earners, like engineers, only control some parts of labour; they are socially situated between the working class and the petite-bourgeoisie.

BASIC TYPOLOGY

Bourgeoisie

Small and medium employers Management

Petite-bourgeoisie

Semi-autonomous employees Proletariat

An arrangement of Marx's theory of classes

Wright presents his theory as "Neo-Marxist" (1985b: 231) at variance with the "orthodoxy", and his method as an anti-empirical one: the concepts are constrained by the theories in which they work, so that the priority must be given to logical consistency. He claims that the existence of contradictory class locations obscure exploitation and class relations, and it corresponds to the existence of non capitalist forms of exploitation. Namely, various forms of exploitation -in terms of qualification, of organisation, of status…-would exist beyond the traditional Marxist one, so that it would be possible to be both exploiter and exploited.

He does not pretend to expound the Marxist statement on classes, but just a possible statement in the Marxist tradition. He preserved what he judges as the fundamental criteria, namely that the difference between class structures is based on the mechanisms of appropriation of the surplus. The nature of a social structure is established by the nature of the relations of exploitation which substantially generate a set of contradictory interests5 .

Wright's aim is to specify the concept of class in the Marxist theory, in order to make it operative. He builds his own theoretical posture between, on the one hand, the economism and essentialism that he attributes to the Marxian "orthodoxy" and, on the other hand, the relativism and subjectivism of some critics of Marxism. He asserts that the essentialist idea of a genuinely socialist revolutionary proletariat which is determined by its class location clashes with the observation that most of the working class departs from this, if it ever occupied it. He claims that defining classes on the exclusive basis of the social relations of production is not a firm foundation for specifying particular classes in any social organisation of production, since it would tend to economism and to class reductionism. Here he took a further distance from the traditional Marxian conception, according to which any form of domination can be ultimately explained in terms of relations of production, and at the same time he got closer to a Weberian-type conception.

Beyond a Marxist definition of classes

The theoretical background of Wright's approach is not exactly the Marxian one. More precisely, his theory is between Marx's and Weber's (2.1), and it can be seen as a static approach (2.2).

Classes and professions: between Marx and Weber

Wright separates professions, defined as positions within technical relations of production, from classes, defined as positions within social relations of production, and capitalist organisations are articulated with both social relations and technical relations. Yet he departs from a Parkin-type approach 6 , for which the class structure is essentially determined by the professional structure. He distinguishes the class structure from a typology of professional categories. Then, class and profession fill two fundamentally distinct theoretical spaces, and Wright aims to work out a conceptual relation between class and profession, with the concept of organisation, through the analysis of their empirical relation. Socio-professional categories contain heterogeneous class elements, and professional distributions vary between classes.

Such a Weberian-type typological sketch reproduces the pre-Marxist dualism between subject and object, and Wright's Marxism is a sociologised Marxism, as the expression of a neo-Weberian logic of social stratification. Basically the difference between Marx and Weber rests upon the explanatory primacy. Marx defends a primacy of the economic structure, whereas Weber advocates a causal pluralism which weakens the intelligibility of class conflicts.

Wright states three levels of abstraction: the mode of production defines a class structure, the social formation defines class alliances and the circumstances define the organisation of production. Here he departs from Weber, who did not distinguish a mode of production and a social formation, since he never considered historical development as a pattern of class structures. As a result, Wright claims that the consequences of classes on society are not limited to class struggle and, rather than claiming that classes are crucial for social change, he simply judges them as important for social change, alongwith the "new social movements" (feminism, environmentalism...). Then "traditional forms of class struggles -the direct confrontation of organised workers against the capitalist class -are less central to the politics of Western democracies in the 1990s than they were in the 1930s or 1960s" (1997: 120). Therefore, in order to be effective, a structural analysis must be complemented with an autonomous analysis of culture and of ideology7 .

A static approach

It might be the case that one of the strongest charges against Wright's theory is that it is non dynamic. This might be due to a structuralist influence (2.2.1); we will propose a dynamic approach instead (2.2.2).

A structuralist influence

It seems that such an approach has been inspired by Nicos Poulantzas, who describes classes exclusively by relations of exploitation and of domination, and who restricts the working class to manual work without responsibility. For Poulantzas (1974), like for Martin [START_REF] Nicolaus | Proletariat and middle class in Marx: Hegelian choreography and the capitalist dialectic[END_REF] or for John [START_REF] Urry | Towards a structural theory of the middle class[END_REF], the definition of classes is exclusively structural, and class interests can be objectified. Yet, for Wright, the form of class struggle is defined by economic, ideological and political relations in a given historical situation. Such a differentiation leads him to set up taxonomical categories in order to work out a terminology for a distinguishing class locations.

For a dynamic approach

For Marxists like Edward P. Thompson, a class is not a thing with an independent existence, it cannot be defined in mathematical terms and it cannot be attributed "the classconsciousness which 'it' ought to have " (1963: 10). Instead, a social class is a relation, so that a precise definition would lead to confusion rather than to clarification. Therefore, status and stratifications are considered as dogmas of the bourgeois sociology, and they lead the sociologists not to deeply reflect on their own subject.

The finest-meshed sociological net cannot give us a pure specimen of class, any more than it can give us one of deference or love. The relationship must always be embodied in real people and in a real context. (Idem: 13) Social classes are not given before the history of concrete struggles, and social reality does not appear directly to us. Classes are constituted during struggles that are articulated by objective -i.e. economic, political and ideological -conditions, which in return shape movements that organise the workers into classes.

Social classes, rational choice and game theory

Wright explicitly relies upon Roemer's work, mainly his General Theory of Exploitation and Class. We will see first how Roemer founds classes on rational choice (3.1), and then how he articulates classes and exploitation with game theory in a historical materialist background (3.2).

Class relations and rational choice

With Rational Choice Theory, Roemer's peculiar reading of Marx seems contradictory with Marx's own insights. His approach corresponds to a counter-intuitive reading of Marx (3.1.1), which influenced analytical Marxism, and Wright in particular, on the ground of class collaboration (3.1.2).

Roemer's counter-intuitive reading of Marx's

Like Morishima and Steedman, Roemer reconstructs the Marxian theory of exploitation independently from the labour theory of value. He considers exploitation as a transfer of the value of any commodity, not necessarily labour, so that any commodity can be exploited.

There is a decided superiority of our formulation of capitalist exploitation in the gametheoretic manner to the Marxian formulation in terms of surplus-value: the gametheoretic formulation is independent of the labor theory of value. (Roemer, 1982: 20) He further confuses the distinction between Marx and Weber, in denying any structural content to social classes and to exploitation. "Exploitation… does not appear to be of direct interest from either the positive or the normative viewpoint" (Roemer 1988: 89). His definition of classes is formulated in terms of relations of property, with the Class-Exploitation Correspondence Principle: Any agent is exploited if he cannot afford as many goods as his amount of labour contains, and he is exploiter if he can afford more goods than his amount of labour contains. Thus, all the producers who optimize in selling their labour force are exploited, and all the producers who optimize in hiring labour force are exploiters. At the equilibrium there are three groups: exploiters, exploited, and an "obscure" area, which corresponds to Wright's contradictory class locations. Then, the institutional reasons for exploitation would not be the appropriation of labour, but the existence of competitive markets and the differential ownership of the means of production.

A goal of this book has been to demonstrate that the central Marxist concepts of exploitation and class do not require a special logic for their construction. They can be studied in a model of private property system, using standard tools of microeconomic analysis. (Idem: 172)

Roemer explains classes exclusively in terms of individual behaviours, and he misses the theoretical link between individual positions and social phenomena. His microsocial logic only allows a social analysis in aggregating individual unities. Such an approach of the relationship between structure and consciousness seems to be static and deterministic, close to the behavioural sociology, and it achieves an unlikely union between individualism and structuralism.

Analytical Marxism and class collaboration

In such a rational choice framework, Adam Przeworski, an analytical Marxist as well, claims that if the transition to socialism leads to a worsening, even a temporary one, of the workers' welfare and if the workers are allowed to improve their material situation in cooperating with the capitalists, then the socialist perspective cannot result from the material interests of the workers.

A vigorously developing capitalism in which workers can reasonably expect to benefit from past exploitation is the second best for workers as well as for capitalists. The struggle for improvement of material conditions under capitalism is precisely that. It is not a struggle for socialism. (Przeworski 1980 : 146) Insofar as these authors claim that class polarization corresponds to capitalism and socialism, the development of contradictory class locations obscures this distinction, as well as the distinction between exploiters and exploited.

The practical objective of class struggle is not smashing the capitalist class, but creating a viable compromise with capital…

To forge a class compromise workers need to be able to deliver to the bourgeoisie a well co-ordinated and disciplined labour force. (Wright 1997a: 114-115) It would then be possible to abolish capitalist exploitation within a market economy; this may explain the interest of the analytical Marxists in market socialism (see Roemer, Wright 1996).

A completely non-market socialism is not a stable and suitable form of economy and, in any event, is unlikely to be compatible with democratic institutions. (Wright 1997a, 103, stress in the original)

It is noteworthy that, in Roemer's and Wright's theoretical background, with new class divisions, Philippe Van Parijs (1987), an analytical Marxist too, has been led to assume that class struggle now occurs between steady employees on the one hand, and unemployed and precarious employees on the other hand.

Exploitation, historical materialism and game theory

Now, we will see how the analytical Marxists expound classes in an historical materialist framework, with the help of game theory. Roemer's approach is based on Cohen's historical materialism (3.2.1). We defend the claim that game theory is not a suitable basis for historical analysis (3.2.2).

Exploitation and historical materialism

Roemer analyses the former so-called socialist countries as an abnormality in the Marxian framework, and he aims to explain what appeared to be class relations in those countries. He uses the concept of exploitation as a specific mode of explanation of inequalities, as a causal relationship between individuals' incomes. He develops this analysis in terms of game theory: each individual is exploited if there is a potential alternative in which he would be better situated.

I shall construct the model so that, as before, classes are endogenously determined by individual optimization. (Roemer 1982: 110) He defines four historical situations of exploitation according to the rules of retreat and to the main good which is unequally distributed.

 The feudal exploitation is founded on an unequal ownership of the labour force.

 The capitalist exploitation is founded on an unequal ownership of the means of production.

 The status exploitation or organization exploitation, corresponding to statism, or to bureaucratic state socialism, is founded on an unequal control of the planning process.

 The socialist exploitation is founded on an unequal ownership of the qualifications.

"Each revolutionary transition has the historical task of eliminating its characteristic associated form of exploitation" (Roemer 1982: 21). It must be noted that such a stagist approach rests upon Cohen's historical materialism (2000), namely that the development of productive forces determines the historical pattern of the modes of production. Wright admits "an extended engagement with the theoretical work of the economist John Roemer" (Wright 1985b: 250), which would set a rich foundation for empirical study and for theoretical elaboration of class structure, and which would rigorously allow the establishment of a contradictory class structure. On such a basis, he constitutes a typology of class structures, which roughly corresponds to the analytical Marxist theory of history, of exploitation and of classes: Different classes have an objective interest in corresponding social structures: The capitalist class in capitalism, the middle classes in statism, and the working class in socialism. Marx's theory is then challenged on three points: Socialism is not necessarily the future of humanity, the proletariat is not the only holder of the revolution, and socialism may lead to exploitation.

The limits of game theory

It might be the case that game theory is useful for explaining some individual behaviours. Yet, it is not able to explain the general laws of society The conceptual and methodological peculiarity of Marxism is swamped within the abstractions of game theory, that are antagonistic with historical abstractions. Classes are replaced with individuals, the collective interest in social change is replaced with individual consciousness, the relations of production are replaced with professional categories.

A class is a group of people who by virtue of what they possess are compelled to engage in the same activities if they want to make the best use of their endowments. (Elster, 1986: 147, stress in the original)

For Roemer, as for Elster, Marx chose labour force as a reference for political reasons -"clearly, then, historical materialism directs us to construct a theory which views people as exploited, and not corn, as the most efficacious research program" (Roemer 1982: 284) -and for normative reasons -"if the exploitation of the worker is an important concept, it is so for normative reasons -because it is indicative of some injustice and not because the exploitability of labor power is the unique source of profits" (Roemer 1988: 54) -but not for scientific reasons.

From structure to norm: the need for dialectics

For Roemer, a normative approach is necessary for analysing social classes. Cohen develops further that point, and he relates it to historical materialism (4.1). We propose the claim that such a turn results from the initial structural and non dialectical proposition on classes (4.2).

Class heterogeneity and normative approach

For Cohen, the central explanatory foundation for the retreat of critical thinking is the transformation of the class structure rather than the collapse of the Soviet world. The current class structure of capitalist societies would not allow a peculiar social group to have both a capacity and an interest for social change.

That class, traditionally conceived, had four characteristics which, taken together, ensured that it would produce a socialist revolution. First, it was the section of society on whose productive activity everyone depended. Second, it constituted the great majority of society. Third, it was composed of the exploited people in society. And, finally, its members were in serious need; they experienced a substantial lack of the essentials of a fulfilling life... In practical terms, what are the implications of the fact that the working class is not what it used to be, or what we once thought it was? (Cohen, 2001) The social transformation would not rest upon the resolution of structural contradictions, but on issues in social justice. For Cohen, normative answers are necessary, because the change in the class structure of the capitalist societies since the early eighties does not allow any longer the specification of a social group which can hold the social transformation. The tendency to more diversity and less polarization -less workers strictly speaking, a larger petite-bourgeoisie, a change in the structure of qualifications, a weaker labour movement… -should lead to ask ethical questions and to give normative answers to this new background.

The questions about base and superstructure and forces and relations of production… occupied me for some twenty years, before my interests shifted decisively in the direction of moral and political philosophy. I think, instead, that our shift of attention is explained by profound changes in the class structure of Western capitalist societies, changes which raise normative problems which did not exist before, or, rather, which previously had a little political significance. Those normative problems have great political significance now. (Cohen 1990 : 364)

The limits of a structural definition -class in itself and class for itself

We state that classes are not given before the history of class struggles, and social reality is not directly understandable. "The term 'working class', properly understood, never precisely delineated a specified body of people, but was rather an expression for an ongoing social process" (Braverman 1974: 17). People get conscious of social relations through ideology. Therefore, political actions result from the process of conviction and organization by ideological and political forces. Marxism is not only a theory of class structure, but a theory of class formation and of social change as well. Class structures are then part of the explanation of structural constraints.

The propertied class and the class of the proletariat present the same human selfestrangement. But the former class feels at ease and strengthened in this selfestrangement, it recognizes estrangement as its own power and has in it the semblance of a human existence. The class of the proletariat feels annihilated in estrangement; it sees in it its own powerlessness and the reality of an inhuman existence (Marx, 1845: 36).

For Marx, human features are powers, that are capacities to produce. In the capitalist mode of production, property is what is produced, so that having some property corresponds to creating ownership. The worker creates ownership, in an alienated way, and then he has properties, in a distorted way. The capitalist, as an ownership owner, has no properties. He does not possess his ownership, in that possessing something corresponds to be intimately connected with it. The capitalist is then an affected owner -he is not the creator then he is not the real owner -and the worker is a defective creator. Therefore, any human being is dominated by capital: the machine for the workers, money for the capitalists.

Wright and Roemer do not consider social classes as an organized social power. There is no peculiar interest to an economic structure and the distinction between class in itself and class for itself is rejected. A class in itself refers to structural positions within social relations of production -classes in organisations -and a class for itself refers to classes as organized social forces -organisations in classes. Cohen defines classes in a structural way, as an objective reality. "A person's class is established by nothing but his objective place in the network of ownership relations" (Cohen 2000: 73). Paradoxically, he implicitly admits the distinction between class in itself and class for itself, in acknowledging the existence of subjective expressions in a political or cultural form, as class identity, but he assumes that political struggles, consciousness and culture do not make up class structures, but are just its expressions, its effects.

Conclusion

As an answer to a central challenge of contemporary class societies, namely a visible weakening of the polarization between social classes, Wright attempts to preserve the Marxian conception of social classes. His approach, with an alliance between a Marxian background and the Weberian sociology, is founded on and complemented with Roemer's work, that establishes classes within a methodological individualist framework. Cohen joins that posture in stating that classes are not the vector of social transformation anymore.

Our point is that the analytical Marxists insufficiently integrate their approach within the historical process, and they have linked two conflicting approaches within the field of social sciences, namely on the one hand, the Althusserian-type structuralist materialism, as originally advocated by Wright, and on the other hand, methodological individualism, to which Wright got gradually closer. In separating the analysis of social structure and the theory of history, as it was defended and then revised by Cohen, the analytical Marxists present a sociologised Marxism. It has been outlined that its effectiveness in terms of analysis of the social and structural transformations and in terms of political action is limited.

See Wright, 1979, 1985a, 1997b.

Such divisions might well be a conscious strategy of the capitalist class for splitting the working class.

It might be noted that Wright has paradoxically presented his model not only as a structural one but as well as a dialectical one, since class struggle may transform the conditions of its own determination.

See Parkin 1979.

Our point is that they face obstacles that are closely with the class structure, and that are inspired by the capitalist class power.