J.-M Chomaz
email: <chomaz@ladhyx.polytechnique.fr>.

SCIENCE AND ART OF SCULPTURING FLUIDS

science and art & science.

Fundamental research aims at building an understanding of natural phenomena by constructing a representation made of elementary mechanisms. A scientific proof is then the design of a specific experiment to perform measurement of one of these mechanisms as much as possible in isolation, in order to exclude the complexity and the artefacts coming from the interaction of different processes acting in the dynamics or in the measures implemented. For example, in my research, I questioned the idea that large vortex structures observed in planetary atmospheres are the result of a fusion, whereby smaller structures transfer energy from small to large scales. Such a so-called inverse cascade was assumed to be due to the shallowness of the troposphere, 10km deep, compared to the horizontal scale of the vortices, around 1000km. To reproduce a similar free shallow flow in the laboratory, Y. Couder, my PhD adviser, had the idea to use soap films stretched flat on a frame or made hemispherical by a pressure difference. Then by displacing objects in or close to the membrane, such as a rack of cylinders or disks in differential rotation, vortices were generated on their surface. Tracking the evolution of the Another example is my research on the ocean or the atmosphere at mesoscale horizontal scales between approximately 10m and 100km, where motion cannot be assumed to be uniform across the vertical but is organized in layers as the density of the fluid increases with depth. Such a fluid is called stratified, and by a series of experiments, where vertically uniform vortices were generated by a set of vertical flaps that rotated around their vertical axis, we isolated a new mechanism responsible for the layering of the ocean that we named Zigzag instability. From this observation we proposed a new scaling law that invalidated the turbulence models used in atmosphere and ocean numerical simulation [START_REF] Billant | Asymptotic selfsimilarity of strongly stratified flows[END_REF].

The concept of scaling laws is essential in science, since it states that the phenomena observed at one particular scale is representative of a broader class of flows, if non dimensional numbers, to be defined, have the same values. Turbulence in the ocean is controlled by the balance between buoyancy due to the stratification and the momentum of fluid blobs defining the Froude number. Motion in a one-meter high tank in the laboratory will be similar to a much larger scale ocean flow if the Froude number is the same in both cases. Since the size is thousands of times smaller, the velocity has to be smaller in the lab and the stratification higher, with up to 20% variation of the density over half a meter, compared to 1% over kilometres in the ocean. In our experiment [START_REF] Williamson | Gallery of Fluid Motion, Instabilities of Stratified Vortex Pairs[END_REF], tall columnar vortices, never encountered in the ocean, were created. Visualizing flow with dye (Fig. 1), we have shown that tall vortices were breaking apart evidencing broken symmetries or lost invariance. The proofs, experimental images, data, numerical simulations or equations from a theory intrinsically carry the aesthetic of this questioning and are designed and selected by the researcher to make the statement stronger by their subjective aesthetic aspect even though the proof itself is objective. In science, aesthetics then comes into play at least at three different levels, in the quest of elementary processes, in the proof and in the images, sketches or equations that convey the proof. In this last case, aesthetic serves the proof in a subjective way since it intends to convince our peers.

Science is an intrinsically aesthetic search for elementary processes that may touch even a robot mind but it has to be shared among humans and a more sensitive aesthetic delivers proof.

Aesthetics in the Mediation of Science

When mediation of scientific ideas is targeting a nonscientific public, the aesthetic may be pushed by adding complexity to the demonstration in order to trigger questioning and not only to give easy answers. For "The Tornado Tamed" demonstrator (Fig. 2), by adding hot colour lights and irregular injections of smoke, the tornado seems to form instantly when smoke is emitted and illuminates the core of the vortex, and to vertically under the action of the zigzag instability. In neither the atmosphere nor the ocean can this instability be observed because it is so strong that it dismantles any nascent attempt of tall vortices to appear. But as an effect of the pushed disorganize when smoke reaches the periphery, giving the illusion of an aesthetic life cycle that puzzles observers vortices allows us to demonstrate and quantify the inverse cascade of energy from small to large scales [START_REF] Couder | On the hydrodynamics of soap films[END_REF]. back instability, energy cascades at the mesoscale from large scale to small.

The Aesthetics of Scientific Questioning and of Proof

As illustrated by these two examples, fundamental research aims at defining and at the same time answering simple questions through a highly aesthetic process of disentanglement of the complexity of the reality often and engages them in a scientific search to decompose this complex scenery into elementary processes.

Art using Science

If science is aesthetics, and if aesthetics may serve scientific proof and the mediation of science, then science may serve art first by extending the field of possibilities, scientific effect or results to become material for artists to play with and construct their own proposal. I have practiced this art using science with the sculptor Manolis Maridakis to produce three works. The first one consisted of a two-meter water disk for which the scientific challenge was to reduce the turbulence of the flow, to keep the disk smooth-a problem that appears only for a large-scale disk since at a small scale the stiffness of the water surface is sufficient to smooth the flow.

Art & Science, Sensitive Rather Than Sensible Side of Science

Working with Laurent Karst and François Eudes Chanfrault forming the artist group Labofactory, or with the duo HeHe, I investigated another path between art and science that mixes both types of imagination. Formulating questions in science is pure imagination and intuition; this does not involve only the rational side of the brain but the sensitive side, which is able to understand faster and anticipate. With scientific intuition one suddenly sees far ahead in a direction hidden to us before. Then it will take years for the rational side to move and pave the way toward that invented frontier, validating or invalidating what the sensitive side has foreseen. Instead of showing scientific language, proof or technique, it is possible with art & science to directly attempt to share the sensitive side, to build art pieces spanning both artistic and scientific imaginations.

The Labofactory "infraespace" installation is made of 12 computer controlled cubic drums, each with a different and artificial specialized sound. Each percussion of each drum produces a 20cm vortex ring visualized by smoke that shoots up through the air. The ring travels between a light inside the drum and a light 4 meters above, giving an intrinsic rhythm to the installation. Without percussion each drum appears as a black 50cm high cube with a 20cm hole filled with bright smoke looking like a volcano. This large installation can be played live, and the public is then immersed in a ballet of ephemeral rings evoking jellyfishes in an abyssal world.

The Labofactory installation "Wave" proposes a different universe, where electronic music is played live and composed to include infrasound that, since it is not audible, is rendered on the surface of a shallow water tank installed flush at the top of three 3m long black steles. Shallow water waves behave like sound and unfold time into space. A light projector underneath the water immerses the public in a shadow graphic effect where bright and dark fringes magnify the water's wave.

The work with the duo HeHe is different and more committed to political issues, but if it proposes an unusual angle of view it never imposes an answer or a point of view. Their motto is the present impossibility to discriminate the atmosphere from man-made emissions, their "man made clouds". One of their approaches is to use toy models in an odd situation to displace the viewpoint of the watcher. In Fleur de Lys a toy model power plant is set inside a water tank in a scene looking like our lab, and fluorescent plumes are emitted sporadically from the cooling tower in an ever going, slowly evolving nuclear accident, producing a fascinating green toxic cloud. In "Domestic Disaster 3: Planet Laboratory" we mix the HeHe "man made clouds" with my intention to stress the fragility of the atmosphere in our only spaceship, which lacks a windshield. In a laboratory-like staging, a toy globe with a realistic satellite printing is rotating with the music of a Theremin. The water is already greenish and dimly lit. Suddenly a green cloud of dye is injected close to the pole. It slowly moves in a thin boundary layer close to the globe surface, experiencing instability and exhibiting spiralling patterns. Eventually the dye escapes at the equator to the open spaces and mixes.

My final example will be my photo series "time traces" where I produce spatio-temporal cuts through classical movies. Science is not dealing with forms but with dynamics, and the form cannot be distinguished from its creation process. This brings a kind of equivalence between time and space, and equations ruling different processes involve in a similar way time and space variables, the only difference being the causality principle that applies to time. Spatio-temporal images where variations of a quantity are plotted versus time and space are then the tools commonly used to unravel complex dynamics. A movie is also a space-time journey since the director has ordered a time series of scenes with particular visual rhythm and dominant colours and prescribed in each scene the space-time motions of the actors and the camera. The tool science uses to analyse space-time processes can be transposed to movies by producing a single image by the concatenation of the same vertical line cut out of the thousand images of the film. Within this spacetime cut, each scene will be separated by a vertical cut, the whole image being a direct immersion into the director's space-time imagination revealing unforeseen symmetries and constructions. Figure 3 presents the "femme en noir" image extracted from the spatio-temporal cut of the movie "à bout de souffle" by Jean-Luc Godard. In that "nouvelle vague" movement, the camera was carried on the shoulder, enabling the story to be told at the human level since the spectator, staged by the camera, became a character in the scenario. The "time trace" images issuing from this movie have the very human wavy texture given by the uncontrolled motion of the shoulderborne camera. Being an image in time, it carries dynamic information, a kind of imbalance that sets the image in motion.

Fig. 1 .

 1 Fig. 1. Top view of the zigzag instability destructuring a vertical vortex dipole [2].

Fig

  Fig. 2. "Tornade apprivoisée" a 3m high open tornado simulator.

Fig. 3 .

 3 Fig. 3. "Time trace" "Femme en noir" extracted from the spatio-temporal cut of the movie "à bout de souffle" by J.L. Godard.


Jean-Marc Chomaz

The Flow

Science views the universe as a dynamic system, from the cosmic to human to subatomic scale; as a trajectory rushing through the ages; as a gigantic flow which shapes all structures and textures, visible or invisible to the naked eye. Following Arnold, we will use the term the Flow to refer to Anosov's flow, which is defined as the bundle of all trajectories of all the restless particles in our uni-verse, and of all the degrees of freedom in the emptiness of space. The Flow is the effective form science is bound to invent; a form composed of motion, matter, and trajectories. Our current perspective, that of an expanded reality made more tangible by technology, is only a snapshot: an image of the Flow at a particular instant that we perceive of as our age. Our observations of the depth of the sky do not capture an ancient point in the age of the universe, but rather intercept it on its race to the future: particles, photons, and neutrinos, emitted during an ancient transformation from one energy form to another, propagating in continuous inter-action with the rest of the universe. In what we perceive of as the present, these particles and processes are transformed by our extended senses into images that the electrons in our brains can make sense of.

The Flow is tremendously complex. Even if each individual particle obeys elementary deterministic rules, collective forms emerge at all scales in time and space transcending the rules individual particles are submitted to. Our senses interpret these collective structures as mental images or projections: n-dimensional objects or concepts that are a pale perception of the complexity of forms which shape the spatio-temporal Flow. However, our minds unconsciously detect these complex morphodynamics when they occur at our scale of time and space -for example, the mystery of clouds arises from their apparently still yet ever-changing form. Evelina Domnitch and Dmitry Gelfand's Camera Lucida installation has succeeded in manifesting the Flow in a technological setting. It consists of a spherical tank inside which sonoluminescence is generated via the interaction of three transducers to produce intense ultrasonic beams. In doing so, the installation creates evanescent boreal auroras and evokes a universe that seems to emanate from our own thoughts.

Science continuously proposes partial representations of the Flow, at cosmic, geologic, or subatomic scales of time and space, which are aimed at describing morphodynamics compatible with present observations. These models project a form where matter, space, and time are in motion, allowing our imagination to screen and insinuate scenarios with the feeling of embracing a larger view of the Flow than our minds may ever be able to grasp.

In an objective sense, science gives physical existence to the Flow by formulating simple rules of symmetry, conservation, and causality. Mathematics can therefore be regarded as a language that represents the Flow in a mental form, revealing unforeseen principles and phrases that, when formulated, allow for definitive new tests to challenge accepted measurements. Invariably, newly formulated observations will question the Flow model, and scientists will be compelled to discard their old constructions and build a new language: a novel mathematical grammar with new physical verbs that will generate a new mental projection of the Flow to explain both past and present data. All of the hypotheses -however poetical they may be -that the reinvented language may generate should be patiently written, and their effects carefully tested against the observations in the expectation to discover an inconsistence, a dissonance in the real that will bring us once again roll the stone of knowledge from the base to the top of the mountain, free and happy to reformulate the world.

Mixing and segregation are two processes that shape the Flow at all scales of time and space, and explain the textures of the instantaneous images our minds project. In the present model of the cosmic Flow, the initial uniformly composed dense cloud of particles expand into space, and then concentrate under the action of gravity from vanishing fluctuations. The induced segregation of mass and mo-mentum creates primitive, massive stars that transform primordial matter into heavier atoms, which rapidly blow outward into the present dusty universe; where, from gigantic dark clouds, galaxies arise. This ancient period of the Flow is presently tested with computers, which implement the language of mathematics to automatically create new generations of phrases called results, and to compare predictions of the texture of the 3 Kelvin Background Radiation with present observations. This proto-universe model is certainly different from the real Flow, but confronting the model with the present should enable the validation of assumptions about these remote ages and the explanation of the current distribution of galactic clusters. It is particularly conjectural and fascinating, because the remnants of these primordial epochs are hidden from us, behind the blossoms of particles emitted during explosions of these primordial-stars, explosions nearly synchronized since occurring at approximately the same instant after the Big start of the Flow.

Inside actual galaxies, stars are constantly being generated by the high energy interaction of molecular clouds, which create shockwaves that in turn produce fluctuations and gravitational collapse. These galactic clouds are made up of dust and molecules; they diffuse the light of the stars behind or within them, creating intricate patterns which incite our imagination to see familiar shapes or animals. They resemble atmospheric clouds, since they obey a similar law of mixing and segregation to preserve rotational invariance resulting in the conservation of angular momentum. Coherent structures appear to physically manifest these laws, while at the same time, mixing occurs; cloud plumes are chaotically transported, folded, and stretched at spatial and temporal scales compared to which humanity's existence is but a mere blink. The structures eventually condense into proto-stars wherein the laws of conservation of angular momentum and energy of agitation are balanced by gravitational collapse. Proto-stars transfer their momentum to dusty accretion disks from which, by a process still hidden from us, planets emerge -like a giant whirlpool concentrating the surrounding clouds into proto-planets. Again matter segregates, under the simultaneous action of gravity and rotation which defines planetary composition. Volatile elements then combine to form ethereal, yet superficial layers called oceans or atmospheres, the compositions of which vary amazingly when we consider our immediate surroundings: the carbon dioxide-and nitrogen-dominated Venus or the hydrogenand helium-rich Jupiter. Thermal forcing of the atmosphere by the Sun or by internal heat sources induce convective motion that forms vortices to conserve angular momentum. These vortices structure the atmosphere, transporting heat and chemicals around but in isolation from the planetary core, while at the same time promoting the mixing of surrounding fluids via stirring and folding at their periphery. Minor constituents undergo phase changes amid liquids, solids, or gases that render both the folds and core visible: this creates clouds which can appear white when made of water (on Venus, Earth, and Jupiter, for instance) and multicolored when made of sulphuric acid, ammonia ice, or ammonium hydrosulfide). Among all of these cloudy maelstroms, the Great Red Spot of Jupiter, first been observed by Galileo, is clearly the most recognizable. The mechanisms responsible for its existence and color are still debated, but what is known is that The Great Red Spot is a vortex preserved from dispersion by the conservation of its angular momentum, which creates a barrier against mixing with the external fluid and secures its colorful segregation. Our project, Luminiferous Drift, in collaboration with Evelina Domnitch and Dmitry Gelfand, aims to examine a recently discovered stable coherent structure at the south pole of Saturn that strongly resembles a white hexagon. It is believed to have been created by destabilization of the shear present at the edge of the circumpolar vortex, shear that acts as a barrier against mixing shapping the edges of the hexagone. Our installation will mechanically reproduce the vortex and the shear, and via the segregation of bioluminescent proto-cells, will also explore the potential future of Saturn's vortex from the perspective of climate change immanence in the universe.

On Earth, cloud dynamics are better understood, and also involve a similar mechanism of mixture and segregation. The processes from which they originate determine their texture. As we all know, Earth's white clouds are composed of water droplets or water ice crystals that condensate when an air mass is cooled down, generally by upward movement due its own buoyancy, topographical features such as mountains, or colder air masses flowing beneath it. It may also cool down due to heat loss or mixture with colder air. A summertime thermal plume, rising above a cotton field, produces a clear-sky cumulus cloud, and soil moisture condenses out of the rising hot streams of thin air. Stormy weather is generated by a similar but far more dramatic process. When hot, humid, saturated air, often of tropical origin, is pushed upward by cold northern air, a potentially catastrophic sequence of events occurs. The moisture condenses when the depressurized air, moving upward and releasing its latent heat, warms the air and accelerates its ascent; at the same time, rain, rain is coming down, all over the intruding lower layer of fresh air cooling it more. This process progresses even faster, consequently destabilizing the entire region. A mountain standing against the wind will also force air to rise, which can then condense into layers of lenticular clouds when the humidity and vertical displacement are large enough. Amazingly, these topography-based clouds appear stationary, even though they originate from the fast displacement of air as it continuously enters and exits the clouds. Often, the air will oscillate up and down as it moves away from the mountain, producing a spatial echo of its shape. On occasion, clouds can become trapped or twisted into vortices because of mountains or other topographic features, producing the kinds of vortex streets that impressed the first cosmonauts by the arabesques they traced over our paper planet.

These topographic effects, when associated with the trade winds, explain rainfall on the windward sides of tropical islands and many other microclimates, as well as the drier and sunnier weather on the west cost of Leeward islands. Although topographic clouds are periodic in terms of space, other types of cloud patterns are periodic in terms of time. Sea breezes, for example, are diurnally evolving cloud patterns that commence when cold air from the sea moves inland during the day, upwardly displacing warmer air from the land, and generating a band of clouds that hug the shoreline until nightfall, when the wind reverses direction as the land cools off faster than the sea. Mountain breezes are similarly heated by the Sun on the adret (or sun-receiving) side of a mountain, only to then be cooled in the shade of the ubac (or sun-deprived) side as they spill down into the valley.

All clouds have distinctive shapes that arise from the very process of their generation. Plumes and cumulus borders are rounded by horizontal vortices that mix the cloudy air with the drier surrounding air, evaporating the droplets. High altitude stratus clouds are torn apart by chaotic motions induced by large-scale horizontal vortices. Lenticular clouds or caps may either appear extraordinarily smooth like flying saucers, or more paste-like when the wind is stronger or the stratification becomes weaker.

Clouds can also form large-scale patterns resembling planetary circulation, like in the case of a tropical convection band that is surrounded by a clear, dry subsidence zone. Then, at mid latitude, the scenic band of rainy depression, coming from the Rossby wave undulating on the northern potential vorticity, appears as a distinguishable vortex in the sky. Again, these patterns are made visible by the transportation of moisture, either vertically or horizontally, via vortices at all scales of space and time. Because of the conservation of angular momentum, the gyroscopic effect prevents fluid in the vortex from mixing, in contrast to the strongly mixing outside vortices, both of which are impelled by the same ever-present stretching and folding of space. The formation of clouds, however, is not so simple, since without hydrophilic dust -called condensation nuclei -no cloud could form, even if the air was oversaturated. The energy needed to form even the first infinitesimal drop is much too large to be incited by random encounters of agitated water molecules. Water does not condense into droplets spontaneously, but rather into small hydrophilic particles which allow them to take on a finite size, larger than that at which they would evanesce. But when the rain falls it washes out these particles and clears up the sky -and indeed, it looks different, deeper, after the rain. The condensation nuclei must then be regenerated. Amazingly, these nuclei mainly come from the ocean, as salt particles from dried sea spray or as biological particles from the marine water, both of which are formed directly at the singular cusp of wave breaking against the wind, as well as through an explosion of micro air droplets formerly trapped beneath the broken waves. A similar chaotic process that moves masses of moisture around the planet, transports and spreads sea-born nuclei throughout the atmosphere and across the surface of the Earth. The nature, density, and size of the condensation nuclei influence the albedo, or optical density and reflectivity of the clouds' texture. Amazingly the atmosphere is permanently seeded and inhabited by plankton, which deepens the relationship between clouds and the ocean. In our project, Luminiferous Drift, we use synthetic biology to produce luciferine protein encapsulated in vesicles to reproduce the luminescence of some of the Earth's Phytoplankton. These vesicles, in the scenario of the project's installation, are strongly related to life at its origin, to the air encompassing oceans, and to extraterrestrial life. The barely luminous hexagon that starts vacillating in the installation is a reference and probe to the curiosity of Saturn's current vortices.

When humans started burning their fossil past in order to power their industrial future, they not only released greenhouse gasses but also anthropogenic con-densation nuclei such as sulfate, nitrate, and ammonium. The first types warmed the atmosphere, while the second types increased the density of the clouds and, by reflecting solar radiation back into space, cooled the atmosphere. Simply put, manmade clouds did somewhat compensate for global warming by hiding the greenhouse effect. However, because man-made clouds did produce acid rain, the United Nations succeeded in regulating their emission while unraveling the imminent greenhouse gas catastrophe. This catastrophe is in motion within the Flow; its dynamics are hard to predict, as they are leading to an unprecedented balance of the biosphere, atmosphere, ocean and homo-economicus-sphere.

Climate engineering, another of Pandora's temptations, would be an energy industry-led attempt to control the climate by creating planetsize clouds whose purpose is to introduce acidic nuclei into the high atmosphere. Is it reasonable to control such a large part of the Flow, so gigantic when compared to human scale? Is it ethical? Or shall humans just burn their wax wings, imagining themselves as rivals to the Sun, not leaving enough hope to flow with their divine ambitions?

Fig. 1 :

 1 Fig. 1: Luminiferous Drift Installation, Evelina Domnitch, Dmitry Gelfand, and Jean-Marc Chomaz as created for the exhibition Wetware, curated by Jens Hauser and David Familian.

Fig. 2 :

 2 Fig. 2: Exoplanet, Salon des Réalités Nouvelles, Paris 2015. The installation Exoplanet is an intimate inverted cosmos inhabited by the bioluminescence of living phytoplankton Pyrocistis Noctiluca. Pale blue pulses draw a vortex, an evanescence of spirals and stripes. Sparks are produced by the oxidation of the luciferin, a protein photosynthesized during the day by living cells.

Vladimir Igorevitch, Arnold: Chapitres supplémentaires de la théorie des équations différentielles ordinaires (Translated from French by Djilali Embarek), Editions Mir Moscou 1978; in English: Vladimir Igorevitch, Arnold: Lectures on partial differential equations, Springer


Abhijit Das

Hao Lu

Hu Han

Antitza Dantcheva

Shiguang Shan

Xilin Chen

BVPNet: Video-to-BVP Signal Prediction for Remote Heart Rate Estimation

In this paper, we propose a new method for remote photoplethysmography (rPPG) based heart rate (HR) estimation. In particular, our proposed method BVPNet is streamlined to predict the blood volume pulse (BVP) signals from face videos. Towards this, we firstly define ROIs based on facial landmarks and then extract the raw temporal signal from each ROI. Then the extracted signals are pre-processed via first-order difference and Butterworth filter and combined to form a Spatial-Temporal map (STMap). We then propose to revise U-Net, in order to predict BVP signals from the STMap. BVPNet takes into account both temporal and frequency domain losses in order to learn better than conventional models. Our experimental results suggest that our BVPNet outperforms the state-of-the-art methods on two publicly available datasets (MMSE-HR and VIPL-HR).

I. INTRODUCTION

Traditional heart rate (HR) measurement methods such as electrocardiograph (ECG) and contact photoplethysmography (PPG) are not instrumental in portable scenarios. Therefore, non-contact HR measurement-based on rPPG has received increasing attention in recent years [START_REF] Allen | Photoplethysmography and its application in clinical physiological measurement[END_REF], [START_REF] Wang | Robust heart rate from fitness videos[END_REF], [START_REF] Niu | Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation[END_REF]. Applications of rPPG include gauging HR variability, monitoring respiration and emotion state [START_REF] Sun | Photoplethysmography revisited: from contact to noncontact, from point to imaging[END_REF], [START_REF] Sikdar | Computer-vision-guided human pulse rate estimation: a review[END_REF]. The principle behind remote HR measurement using rPPG is based on the variation in optical absorption by the skin owing to the periodic change in blood volume. Thus, the frequency of the periodic skin color changes can indicate the HR.

While significant progress has been made in rPPG based HR measurement [START_REF] Poh | Non-contact, automated cardiac pulse measurements using video imaging and blind source separation[END_REF], [START_REF] Poh | Advancements in noncontact multiparameter physiological measurements using a webcam[END_REF], [START_REF] Li | Remote heart rate measurement from face videos under realistic situations[END_REF], [START_REF] Wang | Exploiting spatial redundancy of image sensor for motion robust rppg[END_REF], the studies were mainly limited to controlled settings. Less-constrained settings including large variations of lighting, expression, movement, occlusion remain challenging. Currently, machine learning methods, and specifically deep learning methods based on Convolutional Neural Networks (CNNs), have shown outstanding modelling power and achieved remarkable success in many research problems such as object detection [START_REF] Girshick | Fast r-cnn[END_REF], image classification [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], as well as face recognition [START_REF] Wang | Cosface: Large margin cosine loss for deep face recognition[END_REF]. Similarly, several works have employed the strong modelling ability of CNNs in remote HR estimation [START_REF] Gee-Sern | Deep learning with time-frequency representation for pulse estimation[END_REF], [START_REF] Niu | VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video[END_REF], [START_REF] Yu | Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement[END_REF], [START_REF] Špetlík | Visual heart rate estimation with convolutional neural network[END_REF], [START_REF] Chen | Deepphys: Video-based physiological measurement using convolutional attention networks[END_REF].

However, as shown in Fig. 1, we note that existing methods are not robust in extracting the BVP signal from video, and are being particularly challenged in unconstrained scenarios including variations of illumination, and pose. In order to tackle such challenges, we propose a new method to estimate BVP signals (BVPMap) from raw temporal signals with noises using constraints from both, temporal and frequency domains. Such a method is more effective in dealing with noise incorporated in temporal signals under unconstrained conditions.

In addition, the inclusion of non-skin areas of the face such as hair and other objects has been found to introduce large noise that is able to impact model robustness. A mechanism that can help to remove this noise is ROI selection. Incidentally, previous works considered the whole face region while extracting BVP, whereas only the exposed skin area is naturally the most suitable ROI. Hence, we propose a ROI selection method based on facial landmarks.

Unstable camera sampling frequency has been an additional unresolved challenge in rPPG based HR estimation. Towards tackling this, we here propose a novel preprocessing technique (including first-order difference, cubic spline interpolation and band-pass filter), in order to obtain normalized physiological signals. Next, we utilize this signal as input of U-Net aimed at producing a refined STMap.

The main contributions of this work for efficient HR estimation can be summarized as follows.

• We propose a technique, which directly computes HR based on raw temporal signals from videos, and then employs different filters to improve the PSNR. We leverage the strong modeling capacity and robustness of U-Net with temporal and frequency constraints to predict accurate BVP signals from raw temporal signals.

• We propose an ROI selection method based on facial landmarks to reduce the influence of non-skin face areas to improve the robustness. • We propose a pre-processing method based on cubic spline interpolation across frames to solve the problem of unstable camera sampling frequency and improve the HR estimation accuracy. The rest of the paper is organized as follows. In Section II, we summarize the conventional and deep learning-based methods for remote heart rate estimation. In Section III, we provide details pertained to BVPNet. We provide experimen- tal evaluations in Section IV and summarize this work in Section V.

II. RELATED WORK

In this section, we briefly review scientific works on remote HR estimation, which can be generally classified into traditional approaches and deep learning based approaches.

A. Traditional HR Estimation

Verkruysse et al. verified the possibility of performing rPPG analysis from videos with commercial cameras [START_REF] Verkruysse | Remote plethysmographic imaging using ambient light[END_REF]. What followed were a number of traditional signal analysis based algorithms aimed at estimating HR from face videos. For example, Balakrishnan et al. proposed a heart rate estimation algorithm based on principal component analysis (PCA) [START_REF] Balakrishnan | Detecting pulse from head motions in video[END_REF]. However, this method was based on the assumption that heartbeat can cause minor head shakes. In reality, maintaining a still head position is challenging; instead, various rigid head movements are common. This hinders accurate HR estimation based on the named method. Most following studies have attempted to estimate HR based on changes in facial brightness. For example, independent component analysis (ICA) [START_REF] Tsouri | Constrained independent component analysis approach to nonobtrusive pulse rate measurements[END_REF] was employed as a blind signal separation (BSS) method, to separate time-domain signals from three color channels into independent signal sources to retrieve the pulse [START_REF] De Haan | Robust pulse rate from chrominancebased rppg[END_REF]. The chrominance signals were linearly combined to reduce the noise in the time series [START_REF] De Haan | Robust pulse rate from chrominancebased rppg[END_REF]. Later, pixel-wise chrominance was proposed to improve the accuracy of heart rate estimation [START_REF] Wang | Exploiting spatial redundancy of image sensor for motion robust rppg[END_REF]. [START_REF] De Haan | Improved motion robustness of remote-ppg by using the blood volume pulse signature[END_REF] studied the characteristics of rPPG signals at different wavelengths and then distinguished between noise and motion brightness changes caused by heartbeat.

B. Deep Leaning based HR Estimation

Deep learning (DL) entails powerful nonlinear fitting capabilities and has been widely used in various applications. In recent years, DL based methods have been studied for remote heart rate estimation [START_REF] Niu | Synrhythm: Learning a deep heart rate estimator from general to specific[END_REF], [START_REF] Chen | Deepphys: Video-based physiological measurement using convolutional attention networks[END_REF], [START_REF] Špetlík | Visual heart rate estimation with convolutional neural network[END_REF], [START_REF] Bian | An accurate lstm based video heart rate estimation method[END_REF], [START_REF] Wang | Vision-based heart rate estimation via a two-stream cnn[END_REF], [START_REF] Yu | Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement[END_REF], [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF], [START_REF] Gee-Sern | Deep learning with time-frequency representation for pulse estimation[END_REF], [START_REF] Niu | Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation[END_REF]. Many DL methods are proposed for extracting BVP signals, in order to obtain HR values. Specifically, the BVP signal is learned from a continuous video sequence without prior knowledge. DeepPhys was proposed to extract physiological signals by two consecutive video frames, and then obtain the HR from the signal's spectrum in [START_REF] Chen | Deepphys: Video-based physiological measurement using convolutional attention networks[END_REF]. Vspetlik et al. used two successive CNNs to extract BVP signals with spectrum-based loss, and estimated the heart rate from BVP [START_REF] Špetlík | Visual heart rate estimation with convolutional neural network[END_REF]. LSTM was used to predict BVP signals from face video in [START_REF] Bian | An accurate lstm based video heart rate estimation method[END_REF]. A two-stream method was proposed in [START_REF] Wang | Vision-based heart rate estimation via a two-stream cnn[END_REF] to estimate HR, by using a low-rank constraint loss to derive reliable features. The 3D convolutional network (named Spatial-Temporal Net) was used to obtain the BVP signal directly from video [START_REF] Špetlík | Visual heart rate estimation with convolutional neural network[END_REF]. Yu et al. attempted to extract physiological signals from highly compressed images [START_REF] Yu | Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement[END_REF]. These methods have achieved good performance attributed to the strong modelling capacity of CNNs.

HR estimation from face videos does not constitute naive learning, as datasets for training are relatively small. Hence, DL based HR estimation methods have predominantly investigated following techniques to improve network convergence.

1) Transfer learning and data augmentation. There are various sources of video noise, and the amount of data containing the HR and BVP signals labels is small, whereas the data for training the depth model is insufficient. Hence, using ImageNet or synthetic noise signals for pre-training is common, in order to migrate drawbacks [START_REF] Niu | Synrhythm: Learning a deep heart rate estimator from general to specific[END_REF], [START_REF] Bian | An accurate lstm based video heart rate estimation method[END_REF]. In [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF] Niu et al. proposed an upsampling and downsampling strategy for augmenting the dataset, which improved accuracy for more extensive and smaller valued HR estimation.

2) New feature representation methods. In DL, the input format of the network is essential. It is essential that it contains sufficient information about the heart rhythm, as well as possess a concise form, with removed background information. HR-CNN used aligned face images to predict HR [START_REF] Špetlík | Visual heart rate estimation with convolutional neural network[END_REF]. Time-frequency representation was proposed as the input of CNN and had exhibited state-of-the-art results for HR estimation on the constrained MAHNOB-HCI database [START_REF] Gee-Sern | Deep learning with time-frequency representation for pulse estimation[END_REF]. Spatial-temporal representation was designed as the input of ResNet in [START_REF] Niu | Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation[END_REF], which led to improved performance over state-of-the-art methods on MAHNOB-HCI, MMSE-HR [START_REF] Tulyakov | Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions[END_REF] and VIPL-HR databases [START_REF] Niu | VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video[END_REF].

III. PROPOSED APPROACH

As shown in Fig. 2

A. ROI Extraction

Towards reducing the environmental impact and placing attention to the change of skin brightness in the face, we propose an ROI extraction method. In the first step of the ROI extraction, we employ Seetaface 1 for landmark detection. There are following two issues faced with landmark detection.

1. Face landmarks fluctuate randomly, hence the face positions between frames may become inconsistent, so we use an average filter with a window size of 5 to smooth the landmark detection process across adjacent frames [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF].

2. A facial landmark may be located outside the face area, so we move all landmarks toward the tip of the nose in a particular proportion.

Hence, we refine the landmark coordinates as follows:

P i (x, y) = P i (x, y) + k(P n (x, y) -P i (x, y)) P n (x, y) = P n (x, y) i ∈ { 1, 2, ...81 | i = n } ,
where P i (x, y) denotes the i-th detected landmark that has been processed on average. n represents the index of the landmark of the nose. P n (x, y) is the landmark on the tip of the nose, which is kept unchanged during the landmark refinement. P i (x, y) denotes the i-th refined landmark. k denotes the proportionality coefficient, which is fixed as 0.05 in our experiments. Based on the refined landmarks, we define 15 ROIs as shown in Fig. 3.

B. Spatial-Temporal Map Representation

Raw Signal Preprocessing. The raw signals/ROIs extracted from the video have the following three characteristics that go through the corresponding proposed processing.

1) The rigid movement of the head and the change in light source can cause low-frequency components with large amplitude [START_REF] Niu | Automatic engagement prediction with gap feature[END_REF], whereas the brightness change of skin owing to the volume and oxygen saturation of the blood is of low amplitude. So, the first-order difference is used to segregate them, solving this problem.

2) The high-frequency noise (such as the non-rigid motion of the face and the shake of landmarks) renders the extracted original signal noisy. The normal HR ranges within [40, 160] 1 https://github.com/seetafaceengine/SeetaFace2 beats/min, or [0.67, 2.67] Hz. Hence towards normalization, the Butterworth filter is used to extract the frequency band of interest.

3) The sampling frequency of the ordinary camera is unstable, but the acquisition frequency of BVP signal in video is highly stable. Therefore, the signal obtained from the video sequence cannot correspond to the ground-truth BVP signal. So, we use cubic spline interpolation to solve the above problem in this paper.

Moreover, the information extracted directly in the RGB color space is not necessarily the best choice [START_REF] Tsouri | On the benefits of alternative color spaces for noncontact heart rate measurements using standard red-green-blue cameras[END_REF], hence the ROIs in the RGB space domain is converted to the YUV color space.

In summary, our preprocessing technique comprises of the following steps: 1) ROIs is converted from RGB to YUV, 2) The first-order difference is used to reduce lowfrequency components, 3) The sampling frequency of signals is normalized by cubic spline interpolation to 30Hz. 4) Butterworth filter with parameter [0.4,10] Hz is used as a band-pass filter to retain the frequency band of interest, related to HR.

To convert the Signal to Spatial-temporal Map, after preprocessing, R × C signals of length T can be extracted from C colour channels of R ROIs. In order to use DL to predict BVP signals, we design a spatial-temporal map and BVPMap. We construct the STMap [START_REF] Niu | Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation[END_REF], as shown in Fig. 4. To obtain the ground-truth BVPMap for training the model, we copy and extend the ground-truth BVP signals into different rows to establish 2D-BVPMap. Fig. 5 shows the BVPMap construction process in detail.

C. BVP Prediction via Revised U-Net

Based on the properties of the STMap, we propose a revised U-Net to predict the BVP signal. The structure of U-Net allows feature extraction on different scales of the signal, which is equivalent to different filtering and enhancement methods for different frequencies of the signal. Besides, cascading them at the same scale endows U-Net the ability to retain features at different scales. In order to customize U-Net for our task, we change the pooling technique. In the classic U-Net, which is designed for segmentation, max pooling was used, which we replace by average pooling in our scenario [START_REF] Chen | Deepphys: Video-based physiological measurement using convolutional attention networks[END_REF]. We note that max pooling ignores the sampling theorem, small input translations or shifts of signals can cause drastic changes in the output. In contrast, average pooling can be normalized owing to its mean property. The architecture and parameters of the network are listed in Table I.

It is challenging to accurately restore the BVPMap using traditional reconstruction loss, e.g., L1 and MSE loss. The reason behind this is that the input signal and ground truth BVP signal have a random phase difference. Hence, we define a loss based on the Pearson correlation coefficient [START_REF] Yu | Remote heart rate measurement from highly compressed facial videos: an end-to-end deep learning solution with video enhancement[END_REF]:

L p = D d=1 C j=1 (1 -corr(S(d j ), S g (d))) D × C ,
SP (d j , f i ) = SP g (d j , f i ) fm fn SP (d, f i ) , SP g (d, f i ) = SP g (d, f i ) fm fn SP g (d, f i )
,

where f n and f m represent the maximum and minimum values of the HR range, with the value of 180 and 21 beats/min, respectively. Both high-frequency and low-frequency components are removed. While the energy of the low-frequency component is often very high, it is usually the DC component that is irrelevant to HR. At the same time, the energy of highfrequency components is often very low and meaningless. Hence, the final loss in the frequency domain is as follows

L sp = D d=1 C j=1 m i=n |SP (d j , f i ) -SP g (d, f i )| D × C ,
where L sp denotes the difference between predicted signals and ground truth in the frequency domain, ranging between 0 and 2. Hence, the final loss constitutes the sum of L p and L sp .

D. From Spectrum to HR

The fast Fourier transform (FFT) transforms of each row of the predicted BVPMap is computed to obtain a spectrum. The frequency with the largest amplitude in this spectrum is the HR value. Then, we can obtain D number of HRs. We use a trimmed mean to compute the final estimated HR. Specifically, the three largest and smallest HRs among the D number of HRs are dropped, and the remaining ones are averaged to obtain the final HR.

IV. EXPERIMENTS

In this section, we evaluate the proposed approach on two public-domain datasets (VIPL-HR [START_REF] Niu | VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video[END_REF] and MMSE-HR [START_REF] Tulyakov | Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions[END_REF] ), and provide comparisons with state-of-the-art methods. We also provide an ablation study to validate the effectiveness of individual components in our approach.

A. Datasets and Experimental Settings

Datasets: We tested our algorithm on the following two databases:

(1) VIPL-HR [START_REF] Niu | VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video[END_REF] database is a challenging database for remote HR estimation, containing 2,378 RGB videos and 752 near-infrared videos of 107 subjects captured under nine different conditions. It is a dataset in a real environment, as shown in Fig. 6. Large head movements, different lighting conditions, as well as different acquisition equipment have brought to the fore great challenges for HR estimation. In addition, a considerable challenge is that the sampling frequency of the camera is unstable.

(2) MMSE-HR [START_REF] Tulyakov | Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions[END_REF] database is widely used for remote HR estimation, which contains 101 video segments and provides real-time blood pressure signals for each video. In this dataset, head movements are weak, whereas the background is consistent for all videos.

Data augmentation: In order to improve the generalization ability of our algorithm, we propose the following three data augmentation strategies.

(1) We use a 2% probability that the positions of the two random rows of the STMap are randomly exchanged, i.e., to simulate the vibration of the ROIs across the aligned face images;

(2) We refer to [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF], and perform temporal down-sampling and up-sampling to the original face video with sampling rates from 0.67 to 1.5, to enrich the HR distribution of the training dataset.

(3) For every epoch of the training, we use a window of D × T to crop the STMap, which reduces the effect of the phase difference of input videos.

Parameter settings: Our BVPNet is implemented in PyTorch, and the FFT algorithm is implemented by Numpy. We use the Adam solver as an optimizer with an initial learning rate of 0.001, and the batch size is 64. On both the MMSE-HR database and VIPL-HR database, we use a 5-fold cross-validation protocol on the shuffled database; HR ground truth is obtained from the ground truth BVP signal collected by FDA approved sensor. Specifically, the frequency of the amplitude peak of the spectrum of BVP signal is the HR value; hyperparameters D, T and C are empirically set to 15, 256 and 3 respectively.

Performance Metrics: We follow existing methods [START_REF] Li | Remote heart rate measurement from face videos under realistic situations[END_REF], [START_REF] Niu | Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation[END_REF], and use several different measures to report the performance, i.e., the mean error (ME), standard deviation (Std) and root mean squared (RMSE), mean absolute HR error (MAE), mean of error rate percentage (MER) and Pearson's correlation coefficients (ρ).

B. Pre-processing Details and Result Discussion

As mentioned previously, in order to better predict the BVP signal, we pre-process the raw temporal signals obtained from the video to make the model easier to converge by using first-order difference and Butterworth filter. We show the effect pre-processing step and similarity with the ground truth BVP signal in Fig. 7 using one video form MMSE-HR.

In Fig. 7 (a), we observe that the original signal has a DC trend component and the high-frequency noise, which is very different from the ground truth BVP signal. The DC component may be caused by a rigid movement of the head or the change in ambient lighting, while the high-frequency component may be caused by the non-rigid movement of the face or face landmarks shake. The first-order difference effectively suppresses the tendency of the original signal as shown in Fig. 7 (b), and the Butterworth filter removes highfrequency noise as shown in Fig. 7 (c). After the preprocessing, the signal becomes more similar to the ground-truth BVP signal. This suggests that the proposed preprocessing step is instrumental in the final HR estimation task.

There is an irreversible gap between the timestamp of the real BVP signal and the timestamp of our pre-processed signal. The final HR value is derived from the spectrum of the signal. Therefore, we propose a loss in the frequency domain. Towards better defining the frequency domain loss, we analyze the spectral characteristics of the BVP signal. As shown in Fig. 8, the blue line is the spectrum of the BVP signal. Along the X-axis and Y-axis are the HR in beats per minute and the normalized signal amplitude, respectively.

As shown in Fig. 8, the spectrum has three characteristics: (i) The energy of the low-frequency component is very high, which is often a DC trend component irrelevant to HR, which should be removed. (ii) There are often higher harmonics than HR frequency in the spectrum that are multiples of the HR frequency. (iii) The high-frequency components after the triple HR frequency harmonics are very small. Therefore, we only select the frequency band of interest to define the loss. The physical meaning of the frequency of the spectrum peak refers to the HR value. Therefore, performing a square operation on the obtained spectrum renders peaks more pronounced and helps to estimate HR accurately.

C. Results on VIPL-HR

VIPL-HR has a large number of RGB videos for remote HR estimation, which contains nine complex environments and large head movements. In order to test the effectiveness of our proposed method, we compare it with stateof-the-art methods including deep learning based methods, namely I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF], DeepPhy [START_REF] Chen | Deepphys: Video-based physiological measurement using convolutional attention networks[END_REF] and RhythmNet [START_REF] Niu | Synrhythm: Learning a deep heart rate estimator from general to specific[END_REF] and RhythmNet+Attention [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF] and several hand-crafted methods (Tulyakov [27], POS [START_REF] Wang | Algorithmic principles of remote ppg[END_REF] and Haan2013 [START_REF] De Haan | Robust pulse rate from chrominancebased rppg[END_REF]. The deep learning methods are tested on VIPL-HR datasets using a 5fold cross-validation [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF]. A comparison of these algorithms is shown in Table II.

From Table II, we can see that our BVPNet performs the best with respect to Std, MAE, RMSE, and MER. Rhythm-Net [START_REF] Niu | VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video[END_REF] and RhythmNet+DA+Attention [START_REF] Niu | Robust remote heart rate estimation from face utilizing spatial-temporal attention[END_REF], as end-toend deep learning-based algorithms, perform very well for the indicator ME and ρ. However, for other indicators, our algorithm has achieved better performance. At the same time, our method also outperforms methods reconstructing signals, i.e., Haan2013 [START_REF] De Haan | Robust pulse rate from chrominancebased rppg[END_REF], Tulyakov2016 [START_REF] Tulyakov | Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions[END_REF], POS [START_REF] Wang | Algorithmic principles of remote ppg[END_REF] and DeepPhy [START_REF] Chen | Deepphys: Video-based physiological measurement using convolutional attention networks[END_REF], by a large margin.

D. Results on MMSE-HR

In order to test the generality of our algorithm, we also perform experiments on the MMSE-HR dataset. MMSE-HR contains 101 videos, out of which we remove one video, as the video length is too short. Three of the state-of-theart methods were chosen as a comparison of our method, including Li2014 [START_REF] Li | Remote heart rate measurement from face videos under realistic situations[END_REF], Haan2013 [START_REF] De Haan | Robust pulse rate from chrominancebased rppg[END_REF], Tulyakov2016 [START_REF] Tulyakov | Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions[END_REF]. The evaluation results of these algorithms are shown in Table III.

From Table III, we can see that the results of our BVPNet are the best in terms of all measures, proving the superiority of our approach. Compared to the best of the baseline methods, i.e., the method of Tulyakov etal. [START_REF] Tulyakov | Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions[END_REF], our method reduces the HR estimation errors by a large margin, e.g., 2.39 for Std, 2.63 for RMSE, 0.39 for MER, and increases the ρ by 0.0567. Despite the MMSE-HR dataset being small, our algorithm still performs very well.

E. Ablation Study

To discuss the effect of each part of our algorithm, we perform ablation experiments. Time-domain loss(L p ), frequency-domain loss(L sp ) and ROI extraction (RE) are the pertinent parts of our algorithm, and are tested on the VIPL-HR separately. The test results are shown in Table IV.

We discuss the results in Table IV from two aspects: Time-domain and frequency-domain loss: When either of the two losses is removed, the algorithm evaluation index will become worse. In other words, it is effective to use both time-domain and frequency-domain loss. Time-domain loss can make the shape of the predicted BVP signal similar to the ground truth BVP signal. The frequency-domain loss guarantees that each frequency component of the predicted BVP signal is the same as the real BVP signal. Nevertheless, using frequency-domain loss alone can ensure that the spectrum of the predicted signal is similar to the true value as a whole, but it does not guarantee the peak position of the spectrum perfectly. This suggests that it is effective to use both time and frequency domain loss.

ROI extraction: When only the original STMap method [START_REF] Niu | VIPL-HR: A multi-modal database for pulse estimation from less-constrained face video[END_REF] is used without the ROI extraction method, the accuracy of the algorithm decreases significantly. The ROI region extracted according to landmarks can effectively reduce the motion impact, which is the main reason for improving the robustness of the algorithm.

F. Visualization and Discussion

To discuss the effects of the revised U-Net, we visualize the spectrum of the physiological signal. We select three videos of the test dataset of the VIPL-HR dataset. The first column of each row in the BVPMap is predicted. The first row in each predicted BVPMap is selected for visualization, i.e., the BVP signal predicted from the first ROI. Its timefrequency plot is drawn, and the sliding window is 128, and its step is 64. In order to prove the effectiveness of the revised U-Net, we also plot the frequency spectrum of the pre-processed signal and the ground truth BVP signal (GT) accordingly. As shown in Fig. 9, the first column denotes the input -the video sequence(Fig. 9 v1, v2, v3), the second to the fourth columns are the frequency spectrum of the preprocessed signals(Fig. 9 a1, b1, c1), our prediction signals (Fig. 9 a2, b2, c2) and the real ground truth BVP signals (Fig. 9 a3, b3, c3). In the time-frequency diagram, the x-and y-coordinates denote the time and frequency, respectively. Different colors represent different values.

As shown in Fig. 9, all time-frequency diagrams in the lowfrequency component appear in yellow to indicate a very high value, and the high-frequency component appears blue to indicate a low amplitude. Except for the low-frequency DC component, the frequency with the highest amplitude in the spectrum is the HR value. We observe that the preprocessed signal spectrum (Fig. 9 a1, b1, c1) does not have apparent peaks, and the predicted signal spectrum (Fig. 9 a2, b2, c2) and GT spectrum (Fig. 9 a3, b3, c3) have obvious peaks. This proves that the revised U-Net can learn effective functions in the frequency domain. In addition, we see that there are apparent peaks near the second and third octave of the HR frequency. This phenomenon can be further studied to improve the HR estimation accuracy.

V. CONCLUSIONS AND FUTURE WORK

Non-contact heart rate estimation is a challenging task due to variations in lighting, head movements, expression changes, occlusion, and frequency domain information variation. In this paper, we propose BVPNet streamlined to predict BVPMap from face video to overcome the challenge of frequency domain information variation. In addition, we propose an ROI selection method aimed at overcoming occlusion and discarding non-skin face areas. In BVPNet, we propose raw signal preprocessing (includes color space conversion, first-order difference, cubic spline interpolation, and Butterworth filtering) on the data according to prior knowledge. Then a spatial-temporal map is created in a revised U-Net, designed to predict BVP signals from the spatial-temporal map using both, time and spectral domain supervision information. Our algorithm achieves better performance than several state-of-the-art methods. In future work, we will improve rPPG based heart rate estimation by (1) further analyzing the frequency domain characteristics of BVP signals, as well as the noise signals, in order to improve the stability of the algorithm; (2) more robust modeling of head movements, facial expressions, changes in ambient lighting, etc.; (3) designing and testing with more efficient deep learning modules for BVP signal prediction from STMap; (4) reducing the amount of depth model parameters for real-time application.

Fig. 1 .

 1 Fig. 1. Comparison of traditional rPPG based methods and the proposed method (BVPNet). (a) Traditional methods usually extract the periodical temporal signal from a face video (filtering may also be used to improve PSNR) and then perform FFT to compute the HR. (b) Our BVPNet aims to predict a more accurate BVP signal via U-Net, and then perform FFT to get the HR. Moreover, we also introduce a region of interest (ROI) selection to improve the robustness.

  , we here propose BVPNet, consisting of four main components: (a) facial landmarks detection and ROIs determination; (b) the raw time signals extracted from ROIs are then preprocessed and then combined into a spatiotemporal map (STMap); (c) a revised U-Net is designed to predict the BVP signals from STMap; (d) computation of the final HR estimate via trimmed mean.

Fig. 2 .Fig. 3 .

 23 Fig. 2. The framework of BVPNet: (a) Facial landmark detection and ROIs extraction based on detected landmarks; (b) Raw temporal signal extraction from each ROI followed by preprocessing via first-order difference, cubic spline interpolation and band-pass filtering. The preprocessed temporal signals are combined into a D × T × C spatial-temporal map (STMap); (c) A revised U-Net is designed to predict more accurate BVP signals from STMap; (d) The three largest and smallest HRs computed from all the spectral distributions are removed, and the remaining ones are averaged to obtain the final HR estimate.

2 ×

 2 [48 × 2] D × T × 48 Upsample DualConv outc D × T × 48 D × T × 1 Conv 1. D and T represent the number of ROIs and the length of the signal, respectively. 2. DualConv includes Conv2d, BatchNorm2d, ReLU, Conv2d, BatchNorm2d, and ReLU in this order. 3. The input of 'up' includes the output of the previous layer and the same-scale output of 'down'. Hence, the feature map size has doubled.

Fig. 4 .

 4 Fig.4. The spatial-temporal map (STMap) calculation process: Firstly, face video is converted from RGB to YUV. Then, we extract three signals (for each of R, G, B respectively) of length T from each ROI (after first-order difference and Butterworth filter). The signals from all n ROIs are further combined into a D × T × 3 matrix, which can be visualized as a 3-channel color image. We call such a D × T × 3 matrix as a spatial-temporal map containing the heart rhythm signal of the input face video, which is expected to retain most of the information for the rhythm signal, while suppressing the background information irrelevant to HR estimation.

Fig. 5 .Fig. 6 .

 56 Fig. 5. The ground-truth BVP signal is copied and extended into a 2D BVPMap, with every row the same as the ground-truth BVP signal.

Fig. 7 .

 7 Fig. 7. Visualization of the signal pre-processing process: (a) raw temporal signal extracted from ROI, (b) temporal signal after first-order difference filtering, (c) temporal signal after both first-order difference and Butterworth filtering, and (d) corresponding ground-truth BVP signal.

Fig. 8 .

 8 Fig. 8. The frequency spectrum computed from the predicted BVP signal by our BVPNet.

Fig. 9 .

 9 Fig. 9. Examples of three face videos (v1, v2, v3) with associated preprocessed temporal signals extracted from ROIs (a1, b1, c1), the predicted BVP signals by revised U-Net (a2, b2, c2), and the ground-truth BVP signals (a3, b3, c3).

TABLE I ARCHITECTURE

 I

OF THE REVISED U-NET FOR BVP PREDICTION FROM RAW TEMPORAL SIGNALS.

TABLE II THE

 II HR ESTIMATION RESULTS PERTAINED TO OUR METHOD AND STATE-OF-THE-ART METHODS ON THE VIPL-HR DATABASE.

	Method	ME	Std	MAE RMSE	MER		ρ
	SAMC [27]	10.8	18.0	15.9	21.0	26.7%	0.11
	POS [31]	7.87	15.3	11.5	17.2	18.5%	0.30
	Haan2013 [21]	7.63	15.1	11.4	16.9	17.8%	0.28
	I3D [30]	1.37	15.9	12.0	15.9	15.6%	0.07
	DeepPhy [17]	-2.60 13.6	11.0	13.8	13.6%	0.11
	RhythmNet [14]	1.02	8.88	5.79	8.94	7.38%	0.73
	RhythmNet+DA +Attention [26]	-0.16	7.99	5.40	7.99	6.70%	0.66
	BVPNet	-1.15	7.75	5.34	7.85	6.62% 0.70
			TABLE III
	HR ESTIMATION RESULTS PERTAININT TO OUR METHOD AND
	STATE-OF-THE-ART METHODS ON THE MMSE-HR DATABASE.
	Method	ME	Std	RMSE MER	ρ
	Li2014 [8]	11.56 20.02 19.95	14.64% 0.38
	Haan2013 [21]	9.41	14.08 13.97	12.22% 0.55
	Tulyakov2016 [27] 7.61	12.24 11.37	10.84% 0.71
	BVPNet	1.57	7.27	7.47	4.96%	0.79

TABLE IV THE

 IV ABLATION STUDY RESULTS OF BVPNET ON VIPL-HR.

	Method	ME	Std	MAE RMSE	MER	ρ
	BVPNet w/o Lp	1.32	8.51	6.38	10.05	7.85% 0.63
	BVPNet w/o Lsp	1.55	8.12	5.98	9.91	7.45% 0.65
	BVPNet w/o RE	2.01	7.92	5.65	8.43	7.03% 0.69
	BVPNet	-1.15 7.75	5.34	7.85	6.62% 0.70

* Equal contribution. This research was supported in part by the National Key R&D Program of China (grant 2018AAA0102501), Natural Science Foundation of China (grant 62176249), and Youth Innovation Promotion Association CAS (grant 2018135).