Abdullah-Haroon H Rasheed
email: abdullah-haroon.rasheed@inria.fr

Victor Romero

Florence Bertails-Descoubes

Stefanie Wuhrer

Jean-Sébastien Franco

Arnaud Lazarus

Abdullah Haroon Rasheed

Jean-Sebastien Franco

J S Franco

A Visual Approach to Measure

Keywords: Friction Estimation, Cloth Simulation, Deep Learning, Material Estimation, Inverse Problem. !

teaching and research institutions in France or abroad, or from public or private research centers.

INTRODUCTION

C OMPUTER vision techniques are powerful in analysing the shape and motion of physical objects from mere images. Yet, accurately inferring physical properties from visual data is still a challenging problem [START_REF] Wu | Physics 101: Learning physical object properties from unlabeled videos[END_REF]. Our work aims at estimating physical parameters from the rich dynamic effects that can be observed in cloth motion. Applications range from non-invasive estimation of cloth properties in textile engineering to realistic clothing synthesis for virtual and augmented reality. So far, most existing methods for cloth acquisition have focused on estimating cloth material parameters, that is, its stiffness and mass [START_REF] Bhat | Estimating cloth simulation parameters from video[END_REF], [START_REF] Davis | Visual vibrometry: Estimating material properties from small motions in video[END_REF], [START_REF] Yang | Learning-based cloth material recovery from video[END_REF]. Friction at contact is a much less studied problem due to the difficulty of modelling, measuring, and simulating dry frictional contact accurately. However, friction has a high impact on the overall cloth dynamics, as illustrated in Figure 1.

Measuring a friction coefficient accurately in cloth is a longstanding problem, mainly studied in textile engineering [START_REF] Moorthy | Surface friction characteristics of woven fabrics with nonconventional fibers and their blends[END_REF], [START_REF] Thorndike | Measurement of the coefficient of friction between samples of the same cloth[END_REF]. Better accuracy often comes at the expense of tedious and generally invasive protocols with specialised mechanical sensors and hardware. Yet, the visual variability induced by friction shown in Figure 1 hints at the possibility of a purely visionbased protocol, discriminating the different motions and folding patterns observed under friction variation. This general idea has recently been leveraged by a few studies in physics to infer friction coefficients from contacting slender structures like hair strands [START_REF] Chevalier | Hair-on-hair static friction coefficient can be determined by tying a knot[END_REF] or stiff ribbons [START_REF] Sano | Slip morphology of elastic strips on frictional rigid substrates[END_REF]. While such methods effectively exploit the connection between observed geometry and friction, they are only Fig. 1. Three cloth motion sequences simulated with the same material but different friction coefficients at contact (µ). Top: µ = 0.0, Centre: µ = 0.5, Bottom: µ = 0.6. Differences are significant between µ = 0.0 and µ = 0.5, but more subtle between µ = 0.5 and µ = 0.6.

valid for very specific classes of materials and conditions -such as setups where gravity plays a negligible role. A major challenge for cloth is that, larger friction coefficient differences are easily distinguished, but smaller differences yield increasingly subtle visual differences (Fig. 1, centre and bottom rows).

In this work, we propose a first step towards a generally applicable vision-based method to estimate the dry friction coefficient between cloth and a contacting surface, and between two cloth samples.

To this goal, we use a Long-term Recurrent Convolutional Network (LRCN) [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF] to regress from an input video showing a cloth motion under contact to both material parameters and a dry friction coefficient. Our network is trained on sequences of simulated cloth deformations. We leverage the recent simulator ARGUS [START_REF] Li | An implicit frictional contact solver for adaptive cloth simulation[END_REF] for capturing dry frictional contact in cloth, and search parameter settings that lead to sufficiently accurate results for our needs. Our results indicate that the network, trained exclusively on simulated data, does generalise to real videos showing similar cloth motions; and this in spite of the differences between the renderings of the simulations and the real captures.

In scenarios where capturing significant amounts of real data is prohibitive, many works have explored the idea of training on simulated data with the aim to generalise inference on real data. These include approaches for cross domain transfer learning [START_REF] James | Sim-to-real via sim-tosim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks[END_REF], [START_REF] Tobin | Domain randomization for transferring deep neural networks from simulation to the real world[END_REF] and interpretable low dimensional representation learning [START_REF] Gondal | On the transfer of inductive bias from simulation to the real world:a new disentanglement dataset[END_REF]. Our approach can be considered orthogonal to such efforts as we present a protocol to calibrate simulated data generation by experimental verification, before learning a model from this data. This paper is an extension to our previously presented work [START_REF] Rasheed | Learning to measure the static friction coefficient in cloth contact[END_REF], which trains a model on synthetic data, generated from a verified simulator, and tests it on real captured data to estimate material and friction parameters between a cloth and a substrate. In addition to [START_REF] Rasheed | Learning to measure the static friction coefficient in cloth contact[END_REF], we show here that calibrating the simulator properly before generating the training data is a mandatory step for achieving good estimates on real data. Furthermore, we extend our framework [START_REF] Rasheed | Learning to measure the static friction coefficient in cloth contact[END_REF] to estimate friction in case of cloth on cloth contact, a harder measurement problem which benefits from the visual nature of our measurement protocol. Finally, we show examples of applications of our measurement protocol by creating simulations of cloth materials and substrates from our real captured test data.

Altogether, the combined contributions of our previously presented work [START_REF] Rasheed | Learning to measure the static friction coefficient in cloth contact[END_REF] and that of our above extensions, merged in this paper, are the following:

• We present a deep learning based algorithm to solve the inverse problem of parameter estimation for cloth simulation. Our algorithm measures the material and friction properties of cloth given a video sequence.

RELATED WORKS

Inferring material properties of an object from geometric or visual data has been studied in various communities including computer vision, computer graphics, textile engineering and physics. We first introduce the general model of friction we consider and existing techniques to measure it, before focusing on slender elastic structures such as cloth. Dry friction models: Dry friction is a force that opposes the relative motion of two solid objects. The way in which the interface of two compliant objects in contact evolves when subjected to load is complex, and finding the precise law that describes this phenomenon remains an open problem in physics and mechanics.

Our work is based on Amontons-Coulomb's law [START_REF] Amontons | De la résistance causée dans les machines[END_REF] for friction, which is a commonly used model that successfully approximates this complex scenario at the macroscopic scale. In this model, surfaces in contact interact throughout normal and shear forces, and sliding occurs when the ratio between the shear and normal force reaches a threshold value, called the static friction coefficient, which is independent of the area of contact and depends only on the roughness of the interacting surfaces. While cloth simulators usually rely on an isotropic Coulomb friction model [START_REF] Bridson | Robust treatment of collisions, contact and friction for cloth animation[END_REF], [START_REF] Selle | Robust high-resolution cloth using parallelism, history-based collisions, and accurate friction[END_REF], some recent works have explored anisotropic variations of the Coulomb model when simulating interacting rigid bodies and cloth [START_REF] Chen | Modeling friction and air effects between cloth and deformable bodies[END_REF], [START_REF] Erleben | The matchstick model for anisotropic friction cones[END_REF], [START_REF] Pabst | Anisotropic friction for deformable surfaces and solids[END_REF]. Interestingly, Chen et al. [START_REF] Chen | Modeling friction and air effects between cloth and deformable bodies[END_REF] perform extensive cloth-solid experiments and report a few frictional measurements that exhibit either some anisotropic behaviour, or a non-constant friction coefficient (i.e., a nonlinear relationship between the tangential and normal contact forces). Furthermore, in our clothcloth experiments, we noted that in a few cases, departures from the Coulomb model would arise from cohesive effects due to interlocking fibres. However, despite such discrepancies, overall we observed that the isotropic Coulomb model could already capture most of the cloth-solid and cloth-cloth interactions faithfully. Moreover, although simple in appearance, the isotropic Coulomb model actually proves difficult to be simulated numerically, as it is a non-smooth model characterised by a non-constant threshold depending (linearly) on the normal contact force. One strength of the ARGUS simulator precisely lies in its capability to capture this threshold accurately, without relying on any regularisation. For these reasons, we stick to the isotropic Coulomb model in our work, and show the validity of this model through our results. In the future, it would be interesting to refine our study by considering non-constant friction coefficients, anisotropy, and cohesion.

Friction measurement: Estimating friction and material parameters jointly from visual data has recently become a topic of study in computer vision. Miguel et al. [START_REF] Miguel | Modeling and estimation of internal friction in cloth[END_REF] propose a reparameterization of Dahl's friction model [START_REF] Dahl | A solid friction model[END_REF] for estimating internal friction in cloth from geometric information. Internal friction is however different from static friction at contact since it models internal dissipation within the cloth, hence this technique does not apply to our case. Wu et al. [START_REF] Wu | Galileo: Perceiving physical object properties by integrating a physics engine with deep learning[END_REF] propose to use a generative model to estimate friction and material parameters. Unlike our work, this method is targeted at rigid objects. Zhang et al. [START_REF] Zhang | Friction from reflectance: Deep reflectance codes for predicting physical surface properties from oneshot in-field reflectance[END_REF] analyse visible reflections, while Yuan et al. [START_REF] Yuan | Connecting look and feel: Associating the visual and tactile properties of physical materials[END_REF] combine visual and haptic data to estimate friction information. Both works present a static joint estimation of material and friction based on visual attributes of the material, whereas we focus on the dynamic behaviour of cloth under frictional contact and wish to estimate the friction coefficient directly.

In textile engineering, friction estimation has been studied using invasive techniques [START_REF] Dreby | A friction meter for determining the coefficient of kinetic friction of fabrics[END_REF], [START_REF] Mao | Smoothness and roughness: Characteristics of fabric-to-fabric self-friction properties[END_REF], [START_REF] Moorthy | Surface friction characteristics of woven fabrics with nonconventional fibers and their blends[END_REF], [START_REF] Thorndike | Measurement of the coefficient of friction between samples of the same cloth[END_REF]. Some studies in other fields connect the perception of friction (i.e. feeling) to actual measurements [START_REF] Candelier | The role of exploratory conditions in bio-inspired tactile sensing of single topogical features[END_REF], [START_REF] Liu | Quantifying touch-feel perception: Tribological aspects[END_REF] and visual features [START_REF] Brando | Friction from vision: A study of algorithmic and human performance with consequences for robot perception and teleoperation[END_REF]. These works show that visual features correlate to friction information using perceptual studies.

In physics, a few isolated studies consider inferring dry static friction coefficients from contacting elastic slender structures. In particular, the friction of a hair fibre can be inferred by the geometric configuration of a relaxed knot formed by the hair [START_REF] Chevalier | Hair-on-hair static friction coefficient can be determined by tying a knot[END_REF], and the friction of a relatively stiff isotropic ribbon can be inferred by the geometric configuration of the ribbon's bending and slipping behaviour when pushed onto a substrate [START_REF] Sano | Slip morphology of elastic strips on frictional rigid substrates[END_REF]. While these studies only hold in specific scenarios with strict boundary conditions, they demonstrate the relationship between geometric configurations and friction for elastic slender structures. Our work of estimating cloth friction based on videos is inspired by such intriguing studies. Note that in order to obtain reference friction measurements for evaluation, we use a classical inclined-plane protocol [START_REF] Courtel | A brief history of friction[END_REF], [START_REF] Thorndike | Measurement of the coefficient of friction between samples of the same cloth[END_REF]. Material parameter estimation: The problem of estimating material parameters from geometric or visual data has been studied in different scenarios and for various types of materials, including soft deformable materials [START_REF] Syllebranque | Estimation of mechanical parameters of deformable solids from videos[END_REF], [START_REF] Wang | Deformation capture and modeling of soft objects[END_REF], [START_REF] Yang | Materialcloning: Acquiring elasticity parameters from images for medical applications[END_REF]. To our knowledge, recovering material information from cloth has always been formulated as a fitting problem, consisting of three steps: first, considering a material model as a function of deformation of cloth, then capturing deformations, and finally fitting the function to captured data. One approach is to control boundary conditions by setting up a physical apparatus for yielding deformations [START_REF] Clyde | Modeling and data-driven parameter estimation for woven fabrics[END_REF], [START_REF] Miguel | Data-driven estimation of cloth simulation models[END_REF], [START_REF] Wang | Data-driven elastic models for cloth: modeling and measurement[END_REF]. To avoid capturing deformations under controlled conditions, some works have explored extracting material parameters from casually captured videos of cloth [START_REF] Bhat | Estimating cloth simulation parameters from video[END_REF], [START_REF] Kunitomo | Optimization of cloth simulation parameters by considering static and dynamic features[END_REF]. Bouman et al. [START_REF] Bouman | Estimating the material properties of fabric from video[END_REF] capture stiffness and density of fabrics from video using handcrafted discriminatory features. Davis et al. [START_REF] Davis | Visual vibrometry: Estimating material properties from small motions in video[END_REF] estimate material parameters based on videos that show small vibrating motions, and apply this method to fabrics. Recently, Liang et al. [START_REF] Liang | Differentiable cloth simulation for inverse problems[END_REF] propose a computationally efficient way of estimating material parameters on synthetic data with a differentiable cloth simulator.

The closest methods to ours identify cloth material parameters from videos [START_REF] Davis | Visual vibrometry: Estimating material properties from small motions in video[END_REF], [START_REF] Liang | Differentiable cloth simulation for inverse problems[END_REF], [START_REF] Yang | Learning-based cloth material recovery from video[END_REF]. Yang et al. [START_REF] Yang | Learning-based cloth material recovery from video[END_REF] train a neural network on simulated cloth deformations and use the resulting architecture to infer bending and stiffness parameters. This approach follows a recent trend in computer vision and machine learning, where deep neural networks are trained with simulated input data and tested on real data. Training with simulated input offers the advantage of yielding sufficiently large training datasets, hence we follow the same approach here. However, unlike previous methods, we carefully validate and finely tune our simulator for physical realism, through an experimental protocol described in Sec. 4.2.1.

Note that none of the works discussed in this section consider estimation of friction at contact.

BACKGROUND

In this section we describe the elastic model used for our material properties, and the Amontons-Coulomb's law for friction. We chose these models as they are known to be realistic in the range of macroscopic deformations.

Material Parameter Space

To obtain physically accurate material parameter settings in the simulator, we leverage the work of Wang et al. [START_REF] Wang | Data-driven elastic models for cloth: modeling and measurement[END_REF] that encodes the material properties of ten representative classes of cloth ranging from very soft to stiffer materials. Material parameters are encoded as three parameters of the cloth simulator ARCSIM [START_REF] Narain | Adaptive anisotropic remeshing for cloth simulation[END_REF]: a linear mass density, the coefficients of the strain-stress matrix and the coefficients of bending stiffness. This model has been previously used by Yang et al. [START_REF] Yang | Learning-based cloth material recovery from video[END_REF] for recovering cloth material parameters.

Amontons-Coulomb's Law for Friction

Our work is based on Amontons-Coulomb's law for dry friction as this model successfully approximates the macroscopic behaviour of two solid surfaces at contact. In a simplified version that we shall use here (no distinction between static and dynamic friction coefficient), this law defines the friction coefficient µ as a threshold value for the stick to slip transition for two contacting surfaces, and as the coefficient relating normal and tangential forces during sliding. More specifically, in our scenario, let R denote the reaction force on the surface in contact with a piece of cloth. We can divide R into two components: the force component normal to the surface and the one tangential to the surface, denoted by P and Q, respectively. The force P keeps the two contacting surfaces from interpenetrating, and Q opposes relative displacements between the two surfaces in contact; more precisely, the two surfaces stick if Q ≤ µ P and they slip if Q = µ P (the case Q > µ P is not admissible). Fig. 3-right illustrates these forces for a strip of material that is pushed onto a substrate.

DATA GENERATION

Our first contribution consists in generating a dataset of closely matching captured videos and simulations with, for each video, corresponding material classes and friction coefficients. We choose a simple motion that can easily be replicated with a real piece of cloth while containing representative material classes and friction parameters. In particular, we consider a drop and drag motion, in which a square of cloth of side length 20 cm, suspended by its corners, is dropped vertically on a substrate floor and then dragged back and forth as shown in Fig. 4.

Real Data Capture

We now present our experimental data capture setup. Since we aim at predicting material and frictional properties from videos, we constrain the setup of generating real data to a controlled environment in order to remove sources of variation other than material properties and friction. Cloth materials We use 9 materials in bright colours, out of which 8 are close in composition and density to material classes defined in [START_REF] Wang | Data-driven elastic models for cloth: modeling and measurement[END_REF], and one (silk) is not covered by these material classes. Details on the material features can be found in Table S1 of the supplemental material for this paper. From each material we laser cut 20x20 cm pieces with 1x1cm holders at two corners of one edge for clamping purposes. Substrates We choose 7 substrates that, combined with our material samples, allow to cover a wide range of friction behaviours. In particular, chosen substrate materials are aluminium, aluminium-PET, ceramic, rough glass, smooth glass, polyester-mirror and stainless steel.

Frictional behaviour of a single substrate can be very complicated and difficult to characterise, and through our experiments we have learnt that intuition is not reliable enough to describe the frictional behaviour of a given cloth-substrate pair. For this reason, we test several materials that can be used as a substrate, and select 7 of them which give us the broadest range of frictional coefficients with the cloth materials we have chosen. Experimental setup Using two translational stages (Thorlabs LTS-300M) in front of a black background, we control the drop and drag movement of the cloth for repeatability. The total movement is characterised as follows. First, the material is held 1 cm above the substrate and dropped by 6 cm in a movement, which accelerates from rest at 10 mm/s 2 until it reaches 10 mm/s. Subsequently, the system decelerates to reach 6 cm of total displacement. Immediately, the second motorised stage, for the horizontal displacement, starts the horizontal drag by accelerating at 10 mm/s 2 until it reaches 10 mm/s. This velocity is kept constant until the system starts decelerating to achieve a total displacement of 30 cm. We repeat this process forward and backward twice per experiment to allow the observation of a sufficient number of drag and wrinkling patterns. Video acquisition With a calibrated camera, we record the cloth motion from the viewpoint shown in Fig. 4, which allows to observe wrinkling patterns. The whole video contains around 300 frames, corresponding to 2.4 frames per second. For each material / substrate pair, we repeat the experiment 5 times, leading to 5 synchronised videos. Reference friction measurements A quantitative evaluation of the predicted friction coefficients requires measurements of the friction coefficient µ for each material / substrate pair. Accurately measuring µ is an involved process in physics and mechanics which is often performed through invasive protocols. For simplicity, we capture merely a reference measurement for µ using the non-invasive inclined plane technique [START_REF] Courtel | A brief history of friction[END_REF], [START_REF] Thorndike | Measurement of the coefficient of friction between samples of the same cloth[END_REF]. We believe that the friction in the inclined plane scenario is close to the one in the drag situation, and hence its quantification provides a good reference estimation for our purposes. The inclined plane protocol measures friction by placing an object on an inclined plane, and by increasing the slope of the plane until the object starts slipping. The friction coefficient is then computed based on the slope of the plane at the point where the object slips.

To ensure that the reference measurement is robust, we test for each material / substrate pair different locations and orientations of the cloth on the substrate and do not find a noticeable difference in the slippage angle. Our physical setup of the inclined plane gives rise to a measurement error due to the discrete motion of the plane. More details are provided in Section S1.2 of the supplemental material. Fig. 2 shows a histogram of the values of µ that were measured for all material / substrate pairs. Note that our dataset covers a wide range of friction coefficients.

Simulated Data Generation

For training and testing, we simulate a physically accurate dataset that closely resembles the videos captured using the experimental protocol. Implementing Amontons-Coulomb's law poses difficulties in practice because the force response is nonsmooth. This is further complicated by the requirement for discrete representations in both space and time. We use the ARGUS implementation [START_REF] Li | An implicit frictional contact solver for adaptive cloth simulation[END_REF] to simulate cloth deformation subject to frictional contact, for two reasons. First, this state-of-the-art simulator uses an efficient nonsmooth solver to model the dry frictional behaviour for mesh-based Fig. 2. Histogram of µ for all material / substrate pairs. Fig. 3. Physical validation of ARGUS [START_REF] Li | An implicit frictional contact solver for adaptive cloth simulation[END_REF] under a constrained setting that is well understood in physics [START_REF] Sano | Slip morphology of elastic strips on frictional rigid substrates[END_REF]. After proper calibration of the simulator, we observe simulations (dotted curve) that are in very good agreement with the theory (black curve).

systems. Second, a free implementation of ARGUS is available on github.

Physical Validation of the ARGUS Simulator

To our knowledge, most frictional contact solvers for cloth, including ARGUS, have never been validated against real experiments. To verify that ARGUS produces physically accurate simulations, we compare simulations produced by ARGUS against physical experiments. This comparison is performed in a constrained setting, in which the friction behaviour can be derived analytically. The reason for restraining the experimental setting is that a verification against the ground truth physical behaviour is not possible in more complex scenarios, as measuring static friction accurately remains a challenging problem in physics and mechanics.

The constrained experiment we use has recently been proposed by Sano et al. [START_REF] Sano | Slip morphology of elastic strips on frictional rigid substrates[END_REF] in the physics community, and is depicted in Fig. 3 and further illustrated in the supplemental material for this paper. The experiment considers the deflection of a strip, clamped at its top, that is vertically pushed against a substrate with a vertical strain y , as illustrated in Fig. 3-right. Because of friction, this strip remains pinned at its bottom for small y , which is shown as the red region in Fig. 3-left. The more the strip is pushed down, the higher the frictional force holding the strip pinned. The strip is geometrically constrained, hence depending on the value of the friction coefficient, this system has two different outcomes. First, the elastic forces can overcome friction, and the strip abruptly slips. This regime is called slipped, and illustrated by the green region in Fig. 3-left. Second, for rougher surfaces, the frictional forces are high enough to prevent the strip from slipping and it deforms until there is a real contact between the strip and the substrate. This regime is called partially slipped, and illustrated in blue in Fig. 3-left. Sano et al. show analytically that in the case where gravity is negligible, these outcomes are purely regulated by the friction coefficient, and independent of the strip's material parameters. That is, the deformed shape of the strip only depends on the vertical strain y . The analytical solution allows to calculate at what vertical strain the strip will slip (and thus will move from the pinned to the slipped region), which is depicted as a black curve in Fig. 3-left.

When simulating the experiment of Sano et al., a physically accurate simulator should simulate strips that slip when the vertical strain lies on the black curve. We succeeded in matching this theoretical result with the output of ARGUS, but this required carefully setting different numerical parameters, as explained in the following. First, the temporal resolution of the simulation can be controlled by a timestep parameter between two adjacent frames. Second, the spatial resolution can be controlled by either forcing a static discretisation to be used (whose resolution needs to be provided), or by allowing an adaptive remeshing of the geometry, in which case a rate of refinement needs to be provided. Finally, ARGUS also provides contact solver tolerance values that can be controlled by a residual value and a maximum iteration count of the solver, and damping forces are allowed to stabilise the simulations. In our experiments, we found that all of these numerical parameters heavily influence the geometry of the resulting simulations, and thus should be chosen carefully. Our rationale for finding an adequate set of numerical parameters was, first, to set a fixed and high resolution for the cloth mesh, avoiding small energy perturbations due to remeshing; then, mimic a quasistatic experiment by increasing damping forces, thus removing spurious dynamical vibrations of the cloth; finally, find the right level of accuracy by sufficiently decreasing the timestep of the simulation, as well as the solver tolerance, to the point where further refinement would lead to indistinguishable results at our observation scale. In our comparisons, we use a timestep of 0.5 milliseconds, a mesh resolution of 149 vertices for a 20 cm long ribbon, a maximum number of iterations of 50000 and a damping value set to 0.002. We repeat the simulation using 2 materials having different Young's moduli. Our simulations yield the aqua curve shown in Fig. 3-left, which matches the black analytic curve almost perfectly. Fig. 3-right shows an overlay of the physical strip with our simulated result shown as a red curve, and they are also in agreement. Animated illustrations of the same is provided in the supplemental material for this paper.

These results certify that under correct numerical calibration, ARGUS generates simulations that match macroscopic observations in a constrained setting, under negligible effect of gravity. Note that while generating our training dataset, we work under the hypothesis that the same numerical calibration will guarantee highly realistic results while simulating cloth in a more general setting, i.e. when gravity is no more negligible, and where both friction and material parameters influence the geometry of the cloth. Our good prediction results presented in Sec. 6.2 confirm that this hypothesis is reasonable. As the calibrated parameter settings make the simulation run-time significantly slower, conversely, one could argue that our calibration process imposes an overly strong constraint for our solver, hence an unnecessarily large training cost. By generating training datasets with varying accuracy, we check in the following that our high accuracy calibration is actually mandatory to obtain satisfying prediction results from real data. This comparison is presented in Sec. 6.2.

Dataset Generation

To generate the cloth simulation dataset, we implement the exact path of the motorised stage used for the physical data captures, which is analytically accessible, leading to a high temporal synchronisation between the real data and the simulations. Furthermore, the calibrated camera parameters of the real data captures are used to render a similar view for our simulations. This results in simulations that can be considered physically valid to train a discriminatory model. In order to ascertain the usefulness of our validation experiment, described in the previous section, we generate 3 datasets with varying simulator parameters. For all datasets we use a high resolution mesh. However, we degrade the accuracy of the generated data by relaxing the solver tolerance and the simulation timestep. The solver tolerance is relaxed by keeping the residual value constant at 1e -15 and by lowering down the number of solver iterations (which results in a increase of the actual residual at each timestep). The parameters used for each dataset are described in Table 1. Out of the 3 datasets, High-accurate-solversmall-timestep dataset corresponds to calibrated settings obtained from the validation experiment. In each dataset the drop and drag motion is simulated for the 10 material classes measured by Wang et al. [START_REF] Wang | Data-driven elastic models for cloth: modeling and measurement[END_REF]. For each material, 16 friction coefficients evenly distributed between 0.0 and 1.5 are explored, which represents a reasonable range for fabrics according to values tabulated in [START_REF] Dreby | A friction meter for determining the coefficient of kinetic friction of fabrics[END_REF] and is in agreement with our reference measurements. The resulting simulated 3D sequences contain 300 frames each. To generate a dataset of 2D videos, each simulated sequence is rendered using 8 different texture maps and from 3 different viewpoints and varying lighting conditions, using the free Blender software. The addition of texture variation increases generalisation of learning as shown in Sec. 6.3. One of the rendered viewpoints is calibrated based on the real data to replicate the experimental settings in our simulated data. Two additional viewpoints, are selected to increase visual variation. Furthermore, to match the experimental setup and reduce variability due to environmental factors, we render a dark background and a substrate floor in all of our renderings. The viewpoints for simulated data, alongside real data, are depicted in Fig. 4. Further illustrations of dataset are provided in the supplemental material.

PREDICTION MODEL ARCHITECTURE

We now specify our pipeline to estimate friction of cloth from a 2D input video. As the input to our model is a sequence of images, we use a Long-term Recurrent Convolutional Network (LRCN) model similar to those used for action classification [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF]. We are inspired by a previous model that recovers a material class label of cloth from an input image sequence [START_REF] Yang | Learning-based cloth material recovery from video[END_REF]. A variant of this model, called baseline model in the following, is explained in Sec. 5.1.

While the baseline model is effective at predicting a material class, it performs significantly worse when trained to predict friction coefficients (see Sec. 6.3 for details). We therefore propose a novel pipeline for this task that conditions friction on material classes, as outlined in Sec. 5.2.

Baseline Model

In the baseline model by Yang et al. [START_REF] Yang | Learning-based cloth material recovery from video[END_REF], convolution and pooling layers are used to extract image features from each frame. Their architecture of this feature extraction block is a modified version of AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] of VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] with ReLU activations as shown in Table 2. The weights are shared between all feature extraction blocks. Let

f i = CN N V GG (I i ) (1)
denote the image features extracted from frame I i , where CN N V GG is the simplified VGG16 of Table 2.

A sequence of learned image features f 1 , f 2 , . . . , f k is then passed to long short term memory (LSTM) layers, which extract temporal information. The output of these layers is finally passed to fully connected layers to learn a function from the extracted spatial and temporal features to the data labels. This can be written as ĥ

= F C(LST M (f 1 , f 2 , . . . , f k )), (2)
where ĥ is the final likelihood computed for each material class label, LST M is a set of two LSTM layers and F C is a set of two fully connected layers. The architecture is depicted in Fig. 5. This architecture is trained with a standard categorical cross-entropy classification loss, and for prediction, the class label with the highest likelihood is reported.

Conditional Friction Model

The baseline model is significantly worse at predicting friction than at predicting material class. The reason is that different materials combined with different friction values can yield visually similar features. For a fixed material, however, different friction values typically lead to visually distinctive behaviours. Motivated by this observation, we estimate the friction coefficient using a model that is conditioned on material classes. Model The model is shown in Fig. 5. In addition to a sequence of 2D video frames I i , i = 1, . . . , k, the model takes as input a material class label. The material class label is represented by a one-hot vector m which is passed to a fully connected layer with softmax activation. For this architecture, the video frames are processed using the same convolution and pooling layers as for the baseline model. The material information is then cloned for each input frame and concatenated with the feature vectors of each frame, before being passed to the LSTM layers and on to the fully connected layers. That is, the vector ŷ containing the likelihood for each friction class label is computed as

ŷ = F C(LST M (F Cs(m) f1, . . . , F Cs(m) f k )), (3)
where F C s denotes a fully connected layer with softmax activation and is the concatenation operator.

Training The training loss can be written as

L = -logP (y|I 1 , I 2 , . . . , I k , m), ( 4
)
where y is the friction label provided for the training examples. This loss is implemented as the categorical cross-entropy loss function. In both models, we use dropout layers between fully connected layers for regularisation. We train the baseline model to predict the material class. We use true material class labels to train the conditional friction model. Prediction At test time, we provide the true material class label to the conditional friction model in order to observe the effect of lower to higher accuracy friction information present in our datasets. For real captured data, we provide the model with closest matching material class label that is present in our synthetic dataset. For a class label that is not present in our synthetic training datasets i.e. silk, we provide the conditional model with a material class label predicted by our baseline model.

RESULTS AND DISCUSSION

In this section we present and analyse the results of testing our model on real data, before discussing limitations. We also perform ablation studies on synthetic data, in order to perform a comparison between our conditional friction estimator and baseline models, and to evaluate the model's generalisation capability.

Implementation and Evaluation Details

Our implementation uses Keras [10] and Tensorflow [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. While we experimented with various optimisers, we empirically observed Adadelta [START_REF] Zeiler | Adadelta: an adaptive learning rate method[END_REF] to converge faster. We use a learning rate of 1.0 and a decay factor of 0.95. The training time of our model and the baseline is between 4 to 6 hours on all datasets with a single NVIDIA TitanX GPU and the training converges in circa 30 epochs.

Evaluation protocol As our goal is to build a friction measurement protocol for cloth, we evaluate our friction prediction by considering the absolute difference between the predicted value and the reference value. As the reference measurement r for real data is only known up to a measurement error e, we consider any value within the interval r ± e as having no error, and report the absolute difference of our prediction to this interval. This provides an optimistic estimate of the error as the error is calculated from the range and not from an absolute value. For material parameter estimation we report the top-1 and top-2 accuracy.

Results on Real Test Data

We present results on real data captured through our experimental setup presented in Sec. 4.1. Our datasets contains 5 videos for each material / substrate pair, leading to 5 test datapoints. We take the median prediction value out of the five test datapoints to be the final prediction for any material-substrate pair. We evaluate our test data using models trained on 3 different datasets generated using the ARGUS simulator as mentioned in Sec. 4.2.2. Impact of the simulator's accuracy As can be observed in Fig. 6, the model trained on the dataset High-accurate-small-timestep outperforms the models trained on the two other datasets Lowaccurate-big-timestep and Low-accurate-small-timestep (error of < 0.1 on 65.07% of test data compared to 25.3% and 34.23%, respectively). Recall that the former dataset is generated using our calibrated parameters settings based on the validation experiment described in Sec. 4.2.1. Furthermore, it can be observed that while the < 0.1 error is not significantly different between models trained on datasets Low-accurate-big-timestep and Low-accurate-small-timestep, the model trained on the latter makes significantly less catastrophic errors, thus achieving an error of < 0.2 on 52.3% of test data as compared to 33.3% of the model trained on the former.

Overall, this study shows that degrading the accuracy of the simulator causes a downgrade in prediction of our learning-based method, hence it confirms the need for a realistic enough simulator to generate input data. This justifies the need for validating and calibrating the ARGUS simulator properly before learning from it. Impact of the reflectance Fig. 7 shows a detailed breakdown of results for the model trained on the dataset High-accuratesmall-timestep. It is noteworthy that the model accuracy reached on real experiments when tested on all substrates (red curve, error of < 0.1 on "only" 65.07% of data) is significantly lower than the model accuracy obtained on synthetic test data (dark blue curve, see details in next subsection). This decay in model accuracy can be mostly attributed to the presence of reflecting substrates in the real data, a factor that is not modelled in our training data. The light blue curve shows that the model accuracy is significantly worse on substrates which are reflective. Removing these substrates from the evaluation yields an error of < 0.1 for 86.1% of data, as shown in the black curve. This shows that the model has indeed learnt the friction behaviour of cloth which is transferable from simulated to real data, barring the confounding factors. One way to mitigate this degeneration on reflective substrates is to vary the reflectance of the substrate floor during rendering of our simulated training data. After adding reflective substrate renderings, we obtain an error of < 0.1 for 93.6% of real test data on all substrates, as shown in the green curve. Note that adding the variation to the training data improves the results overall, yielding better results than training and testing without reflective materials (black curve).

Results on Simulated Data

We perform ablation studies on simulated data in order to assess our trained model's ability to generalise for our trained model. In the following experiments, we use a model trained on our calibrated simulated dataset (High-accurate-small-timestep).

Utility of the Conditional Friction Model

To compare the performance of our conditional model with the baseline model, we train our model on synthetic data while leaving out 10% of our dataset for testing. The baseline model performs vastly better at predicting material parameters than at predicting friction coefficients. The conditional estimation model performs better at predicting friction coefficients than the baseline model which indicates that the material and friction parameters are not decoupled in the global behaviour of the cloth, and that adding material information as an input parameter reduces the search space. Furthermore, the error distribution for test data indicates that our model learns coherently.

Generalisation to Unseen Textures

We render our training data with different textures to make the model agnostic to appearance variations. To test this generalisation ability, we render our simulations with a texture that has not been seen by the model during training. We test our model on 160 sequences, and our model achieves an error of < 0.1 on 87.2%, and an error of < 0.2 on 95% of the samples. This shows that the model's predictions do not degrade dramatically if an unseen texture is encountered, implying that the model is capturing more nuanced spatio-temporal phenomena to classify friction behaviour.

Generalisation to Unseen Viewpoints

We demonstrate the model's generalisation ability to unseen viewpoints. Starting with the camera position from one of our simulated viewpoints, we rotate the camera origin, in both azimuthal and transverse directions by ±5 degree intervals on either side to generate 6 viewpoints which are 5, 10 and 15 degrees apart on either side. Afterwards we randomly select 20 material, friction and texture combinations for each viewpoint and render them as our test data. We report the accuracy for each viewpoint in Table 4. While the accuracy degenerates with unseen viewpoints, the decay happens progressively based on the difference in viewpoint from the original one. The asymmetry of results in different directions of the azimuthal movement can be attributed to the fact that in one direction less of the cloth contacting the substrate is visible as compared with the other one. Furthermore, during horizontal movement, frontal viewpoints display better accuracy as opposed to side viewpoints.

Horizontal

-15 • -10

Generalisation to Unseen Rendering Conditions

As our training data contains a variation of rendering conditions, we demonstrate the model's generalisation ability by arbitrarily changing the lighting conditions and reflectance of the substrate. We train our model without adding additional reflectance variance mentioned in Sec. 6.2. We test our model on 160 sequences, and our model achieves an error of < 0.1 on 72.7%, and an error of < 0.2 on 84.1% of the samples. This degradation is in line with the results observed on real captured data. The rendering condition variability is thus necessary to increase the generalisation of the model.

EXTENDING TO CLOTH ON CLOTH FRICTION

We extend our proposed non-invasive parameter estimation model to cloth on cloth motion data captured by a physical experimental setup. Estimating cloth on cloth friction has been a relatively more difficult task due to the woven nature of the materials. In the following sections, we describe the modalities for generating our experimental data, extending our synthetic dataset and the results from applying our method in cloth on cloth friction regime.

Cloth on Cloth Friction Test Data

Using the same setup as detailed in Sec. 4.1 we generate a new test dataset, using our 9 cloth samples from the previous experimental dataset. We now use each cloth both as a substrate and as the suspended cloth for the 'drop and drag' motion, thus having 81 cloth-cloth pairs in total. As a substrate, the cloth is stretched out on to the base of the experiment, such that the suspended cloth can drag on it. The motion is further explained in Sec. 4.1. In total we obtain 5 videos of 300 frames each of 81 cloth-cloth pairs. An extract from this dataset is shown in Fig. 8. Note that our setup is not symmetric, as the roles played by the suspended cloth and the cloth substrate are not equivalent due to the stretching of the cloth substrate. This asymmetry will be revealed by our friction measurements below. In addition to our main setup, we record a baseline measurement of friction of each cloth-cloth pair based on the inclined plane method described in Sec. 4.1, using a similar stretching for the cloth serving as a substrate. As our substrate is a non-smooth surface, some cloth pairs display a thread interlocking behaviour such that a stable measurement of a baseline value is not possible. Thread interlocking is a phenomenon that lies outside the scope of our friction model. Therefore, we omit these pairs from our evaluation, considering only 5 cloth substrates, and as a result, only 45 cloth-cloth pairs out of the 81 initial ones. The distribution of measured baseline values for this reduced dataset is provided in Fig. 9 Out of the measured baseline values, certain cloth pairs display a friction measurement higher than 1.5. Such values, shown as orange bars in Fig. 9, clearly lie outside the purview of our synthetic training dataset. We thus expect our prediction method to fail on such data. Finally, as anticipated before, it is noticeable on the distribution of friction coefficients that symmetry is not preserved between a cloth pair in terms of friction values. This is understandable given the nature of the setup, the cloth used as substrate being stretched out while the cloth at the top sliding across it.

Extending the Synthetic Dataset

As our synthetic dataset does not include coloured substrates, in order to improve our model's capacity to generalise to cloth on cloth friction test data we extend our synthetic dataset as follows. We use the dataset High-accuracy-small-timestep and out of the simulation sequence of each friction coefficient (0.0 to 1.5) we take each material and render it with a random colour chosen for both cloth and substrate out of the colours picked from our real cloth dataset. Images rendered from simulation sequences of the same friction coefficient are presented in Fig.

Evaluation and Discussion

In order to estimate friction on the cloth on cloth data, we use a model pre-trained on the synthetic dataset High-accuracysmall-timestep. We further fine tune this model by training it on additional synthetic renderings mentioned in the previous section. We present prediction error plot from both before and after finetuning our model in Fig. 11 In Fig. 11, we show results on test data that is within the scope of our training dataset. Before fine-tuning the model predicts an error of < 0.2 on 72.7% of the data and with fine-tuning the model improves to an error of < 0.2 on 87.8% of the data and an error of < 0.1 on 72.7% of test data. This is comparable with the results achieved for our previous experimental test dataset discussed in Sec. 6.2. If we include friction values that are beyond the scope of our training data i.e. > 1.5, we achieve an error of < 0.2 on 66.6% and < 0.1 on 53.4% of test data using the model fine tuned on the extended synthetic dataset. Given the fact that estimating cloth on cloth friction is a relatively more difficult task, our method provides reasonably accurate results on the test dataset while being trained purely on synthetic data.

APPLICATION TO CLOTH SIMULATION

Our measurement method can be applied successfully to the simulation of cloth with realistic materials. Instead of testing and trying various simulation parameters independently to obtain a desired effect, as often done in Computer Graphics, here the user can directly pick real materials for both clothes and external bodies, and simply inject the corresponding measured parameters in the simulator. Resulting simulations then capture cloth dynamics as well as cloth-cloth and cloth-body interactions in a consistent fashion, since all the parameters only stem from the chosen materials. To illustrate such a benefit, we perform two kinds simulations, shown in Fig. 12 and13.

First, we simulate two square pieces of cloth of different material such that each one is dropped on a rotating sphere made of a particular substrate, and dragged on the floor. One scenario was performed by choosing Sparkle Sweat (M02) for cloth and Aluminium for the sphere, and another one by choosing Denim (M03) combined with Ceramic. These scenarios take as friction coefficients 0.3 and 0.6, respectively, which exactly correspond to our measurements (see Table S6 in supplemental material). As shown in Fig. 12, the wrinkle patterns in both experiments differ significantly given the choice of material and friction coefficients.

Secondly, we demonstrate a realistic interaction between real life cloth materials on an animated character. We simulate two sequences of cloth on cloth friction using our test materials Sparkle Sweat (M02) and Tango Red (M06) with the ARGUS simulator. In each sequence a shawl is dropped on a clothed animated character.

As shown in Fig. 13, the behaviour of cloth to cloth contact is dictated by the specified friction coefficient. The figure shows corresponding frames from two simulated sequences. The materials corresponding to M02 and M06 have been used for both garments in the top and bottom rows respectively. The friction coefficients used are the ones predicted by our measurement protocol on the test data, i.e. 0.3 and 0.7 (see Table S9 in supplemental material).

CONCLUSION

We propose the first protocol for vision-based measurement of dry friction in cloth coming into contact with a substrate and with another cloth. This protocol relied on a CNN-based model trained on data generated by a physics based simulator which has been verified using a physical experiment. In this extended paper, we conclusively show the effect of performing such verification on the simulator before using it to generate training data. We also demonstrate the measurement of cloth on cloth friction using the same protocol and release a new dataset for the community. These contributions open interesting future directions. First, our method paves the way towards estimation of friction in-the-wild by progressively relaxing the video-acquisition protocol, with interesting applications for non-invasive physics measurements, finer-grain capture of real surfaces, and physically accurate re-simulations of pre-observed surfaces. Second, the successful use of calibrated synthetic simulator-based training could be transposed to other inverse parameter estimation problems.

Fig. 4 .

 4 Fig. 4. Dataset Examples: First and second row show corresponding frames from real and synthetic data respectively. Third row shows 3 viewpoints rendered in the simulated dataset.

Fig. 5 .

 5 Fig. 5. Proposed architecture to estimate friction conditioned on material parameters. The coloured inlay shows the baseline model, which is augmented with material class information to form the conditional friction model.

Fig. 6 .

 6 Fig. 6. Cumulative error plots for all datasets

Fig. 7 .

 7 Fig. 7. Cumulative error plots for dataset High-accurate-small-timestep (red, black, light blue, green) and simulated test data (dark blue).

Fig. 8 .

 8 Fig. 8. The figure shows an excerpt from our new Cloth to Cloth friction evaluation dataset. This dataset includes a cloth stretched out as a substrate and another cloth sliding over it using a motorised clamp.

Fig. 9 .

 9 Fig. 9. Distribution of baseline friction values for cloth to cloth dataset. The orange bars are outside the range of our synthetic training data.

10

 10

Fig. 10 .

 10 Fig. 10. Extension of our synthetic dataset, with renderings more similar to the new cloth on cloth dataset

Fig. 11 .

 11 Fig. 11. Prediction errors on Cloth to Cloth friction test dataset

Fig. 12 .

 12 Fig. 12. Corresponding frames showing the cloth coming into contact with a rotating sphere substrate. Top Row: M02 (Polyester 95 %, Spandex 5 %) with µ = 0.3, substrate: aluminium, Bottom Row M03 (Cotton 100 %) with µ = 0.6, substrate: ceramic

  We demonstrate the usefulness of physically validating the simulator before learning data from it; to achieve this we generate 3 datasets with a variable level of physical accuracy, and show that only the dataset produced by the calibrated simulator leads to satisfactory prediction results of our learning-based method.• We release 2 real captured test datasets of 315 and 225 real videos in controlled conditions closely matching the simulation setup, and make baseline estimates of friction values within a measurement error range.

• To generate physically correct training data, we validate the ARGUS simulator physically and calibrate its accuracy against a constrained, measurable real-world physics experiment. • • We demonstrate the validity of our learned model by achieving an error < 0.1 from the baseline range on 93.6% of real test data capturing friction between a cloth and substrate. • We extend our full method to cloth on cloth contact. We present friction parameter estimation results on this test dataset and achieve an error of < 0.1 from the baseline range on 81.1% of the test dataset.

  . We replace this architecture by a simplified version

	Name	Description
	Input	Image 224, 224, 3
	conv1a	Conv 3 x 3, 64, ReLU
	conv1b	Conv 3 x 3, 64, ReLU
	maxpool1 3 x 3, stride 2 x 2
	conv2a	Conv 3 x 3, 128, ReLU
	conv2b	Conv 3 x 3, 128, ReLU
	maxpool2 3 x 3, stride 2 x 2
	conv3a	Conv 3 x 3, 256, ReLU
	maxpool3 3 x 3, stride 2 x 2
	conv4a	Conv 3 x 3, 512, ReLU

TABLE 2

 2 Architecture details for a feature extractor block.

  Data representation A data point for training or testing consists of 30 frames sampled at regular intervals from a video sequence and corresponding material class and friction coefficient as label. We select an input size of 30 frames as the maximum number of frames from a single video sequence that we can fit during the training cycle on a single GPU, without running into memory constraints. We use a split of 80, 10, 10 percent for training, crossvalidation and testing, respectively, which results in training on 92160 images from 3072 video sequences and testing on 11520 images from 384 video sequences for each dataset.Model initialisationWe observed experimentally that model initialisation is important for training convergence. Intuitively, this might be due to the fact that certain viewpoints provide better discriminatory information than others. To obtain training convergence in practice, we therefore train our models progressively by adding one rendered viewpoint from the dataset in each training cycle to our training data. After training simultaneously on all viewpoints, we finely tune our model on the viewpoint which is calibrated based on our experimental setup.

TABLE 3

 3

	Results on Simulated Test Data

TABLE 4

 4

	Results on unseen viewpoints

ACKNOWLEDGMENTS

This work was supported in part by the ERC grant GEM (StG-2014-639139). A special thanks to Mickaël Ly and Laurence Boissieux for their help with final animations and renderings.