Member, IEEE Xianye Ben

Yi Ren
email: yiren@mail.sdu.edu.cn

Member, IEEE Junping Zhang
email: jpzhang@fudan.edu.cn

Senior Member, IEEE Su-Jing J Wang
email: wangsujing@psych.ac.cn.

Kidiyo Kpalma
email: kidiyo.kpalma@insa-rennes.fr.

Senior Member, IEEE Weixiao Meng
email: wxmeng@hit.edu.cn.

Senior Member, IEEE Yong-Jin J Liu
email: liuyongjin@tsinghua.edu.cn

• X Ben

Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms

Keywords: Micro-expression analysis, survey, spotting, recognition, facial features, datasets !

Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro-and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset -called micro-and-macro expression warehouse (MMEW)containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME) 2 for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.

INTRODUCTION

E MOTIONS are an inherent part of human life, and appear voluntarily or involuntarily through facial expressions when people communicate with each other face-to-face. As a typical form of nonverbal communication, facial expressions play an important role in the analysis of human emotion [START_REF] Scherer | What are emotions? and how can they be measured?[END_REF], and have thus been widely studied in various domains (e.g., [START_REF] Freitas-Magalhães | The psychology of emotions: The allure of human face[END_REF], [START_REF] Corneanu | Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications[END_REF]).

Broadly speaking, there are two classes of facial expressions: macro-and micro-expressions (Figure 1). The major difference between these two classes lies in both their duration and intensity. Macro-expressions are voluntary, usually last for between 0.5 and 4 seconds [START_REF] Ekman | Emotions revealed: Recognizing faces and feelings to improve communication and emotional life[END_REF], and are Fig. 1: Six macro-expressions and six micro-expressions, sampled from the same person, in the MMEW dataset (see Section 3). While macro-expressions can be analyzed based on a single image, micro-expressions need to be analyzed across an image sequence due to their low intensity. For micro-expressions, the subtle changes outlined in red boxes are explained in the supplemental material. made using underlying facial movements that cover a large facial area [START_REF] Corneanu | Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications[END_REF]; thus, they can be clearly distinguished from noise such as eye blinks. By contrast, micro-expressions are involuntary, rapid and local expressions [START_REF] Ekman | Nonverbal leakage and clues to deception[END_REF], [START_REF] Bhushan | Study of facial micro-expressions in psychology[END_REF], the typical duration of which is between 0.065 and 0.5 seconds [START_REF] Yan | How fast are the leaked facial expressions: The duration of micro-expressions[END_REF]. Although people may intentionally conceal or restrain their real emotions by disguising their macro-expressions, micro-expressions are uncontrollable and can thus reveal the genuine emotions that humans try to conceal [START_REF] Ekman | Nonverbal leakage and clues to deception[END_REF], [START_REF] Bhushan | Study of facial micro-expressions in psychology[END_REF], [START_REF] Ekman | Telling lies: Clues to deceit in the marketplace, politics, and marriage[END_REF], [START_REF] Porter | Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions[END_REF].

Due to their inherent properties (short duration, involuntariness and low intensity), micro-expressions are very difficult to identify with the naked eye; only experts who have been extensively trained can distinguish microexpressions. Nevertheless, even after intense training, humans can recognize only 47% of micro-expressions on average [START_REF] Frank | I see how you feel: Training laypeople and professionals to recognize fleeting emotions[END_REF]. Moreover, human analysis of micro-expressions is time-consuming, expensive and error-prone. Therefore, it is highly desirable to develop automatic systems for microexpressions analysis based on computer vision and pattern analysis techniques [START_REF] Nag | Facial microexpression spotting and recognition using time contrasted feature with visual memory[END_REF].

Note that, in this paper, the concept of micro-expression analysis involves two aspects: namely, spotting and recognition. Micro-expression spotting involves identifying whether a given video clip contains a micro-expression, and if such an expression is found, identifying the onset (starting time), apex (the time with the highest intensity of expression) and offset (ending time) frames. Micro-expression recognition involves classifying a micro-expression into a set of predefined emotion types, e.g., happiness, surprise, sadness, disgust or anger, etc.

Challenges and differences from conventional techniques

There have been a large number of various techniques proposed for conventional macro-expression analysis (see e.g. [START_REF] Corneanu | Survey on RGB, 3D, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications[END_REF] and reference therein). However, it is non-trivial to adapt these existing techniques to micro-expression analysis. Below, we list the three main technical challenges when compared to conventional macro-expression analysis.

(1) Challenges in collecting datasets. It is very difficult to elicit proper micro-expressions from participants in a controlled environment. Moreover, it is also difficult to correctly label these elicited micro-expressions as the ground truth, even for experts. Therefore, useful micro-expression datasets are scarce. In the few datasets available, as summarized in Section 3, the number of samples is much smaller than the case for conventional macro-expression datasets, which presents a key challenge for designing or training micro-expression analysis algorithms.

(2) Challenges in designing spotting algorithms. Macroexpression can usually be accomplished using a single face image. However, due to their low intensity, it is almost impossible to detect micro-expressions in a single image; instead, an image sequence is required. Furthermore, different from macro-expression detection, spotting is a novel problem in micro-expression analysis. The techniques of micro-expression spotting are summarized in Section 5.

(3) Challenges in designing recognition algorithms. Like micro-expression spotting, micro-expression recognition requires an image sequence. Usually, the image/video features used in macro-expression analysis can also be used in micro-expression analysis: however, special treatment is required for the latter. Due to their short duration and low intensity, the signals of these features are very weak, while indiscriminately amplifying these signals will amplify noises from head movement, illumination, optical flow estimation, etc. Section 4 summarizes facial features for microexpression and Section 6 summarizes the techniques of micro-expression recognition.

Contributions

Over the past decade, research into micro-expression analysis has been blossoming in terms of datasets, spotting and recognition techniques. However, the large part of microexpression research remains scattered, and very few systematic surveys exist. One recent survey work [START_REF] Takalkar | A survey: facial micro-expression recognition[END_REF] focuses on the pipeline of a micro-expression recognition system from an engineering perspective. A second survey [START_REF] Oh | A survey of automatic facial micro-expression analysis: Databases, methods, and challenges[END_REF] provides a collection of results from classic methods, but it lacks a consistent experimental analysis. Different from [START_REF] Takalkar | A survey: facial micro-expression recognition[END_REF] and [START_REF] Oh | A survey of automatic facial micro-expression analysis: Databases, methods, and challenges[END_REF], this paper presents a comprehensive survey that makes the following contributions:

• We provide a comprehensive overview focusing on computing methodologies related to microexpression spotting and recognition, as well as the common image and video features that can be used to build an appropriate automatic system. This survey also includes a detailed summary of up-to-date micro-expression datasets; based on this summary, we perform a unified comparison of representative micro-expression spotting and recognition methods.

• Based on our summary of the limitations of existing micro-expression datasets, we present and release a new dataset, called micro-and-macro expression warehouse (MMEW), which contains both macroand micro-expressions sampled from the same subjects. This new characteristic can inspire promising future research to explore the relationship between macro-and micro-expressions of the same subject.

The remainder of this paper is organized as follows. Section 2 highlights the neuropsychological difference between macro-and micro-expressions. Section 3 summarizes existing datasets. The image and video features useful for micro-expression analysis are collected and categorized in Section 4. Representative algorithms are summarized in Sections 5 (for micro-expression spotting) and 6 (for microexpression recognition). Potential applications of microexpression analysis are summarized in Section 7. Section 8 presents a detailed comparison and recommendation of different methods. Some remaining challenges and future research directions are summarized in Section 9. Finally Section 10 presents our concluding remarks.

DIFFERENCES BETWEEN MACRO-AND MICRO-

EXPRESSIONS

Facial expressions are the results of the movement of facial skin and connective tissue. Facial muscles, which control these movements, are activated by facial nerve nuclei, which are in turn controlled by cortical and subcortical upper motor neuron circuits. One neuropsychological study of facial expression [START_REF] Rinn | The neuropsychology of facial expression: A review of the neurological and psychological mechanisms for producing facial expressions[END_REF] showed that there are two distinct neural pathways (located in different brain areas) for mediating facial behavior. The cortical circuit is located in the cortical motor strip and is primarily responsible for posed facial expressions (i.e., voluntary facial actions). Moreover, the subcortical circuit, which is located in the subcortical areas of the brain, is primarily responsible for spontaneous facial expressions (i.e., involuntary emotion). When people attempt to conceal or restrain their expressions in an intensely emotional situation, both systems are likely to be activated, resulting in the fleeting leakage of genuine emotions in the form of micro-expressions [START_REF] Matsumoto | Evidence for training the ability to read microexpressions of emotion[END_REF] (Figure 2). Throughout this paper, we will focus on spontaneous micro-expressions. Micro-expressions are localized facial deformations caused by the involuntary contraction of facial muscles [START_REF] Ekman | The repertoire of nonverbal behavior: Categories, origins, usage, and coding[END_REF]. In comparison, macro-expressions involve more muscle in a larger facial area and the intensity of muscle motion is also relatively stronger. Compared to macro-expressions, the intrinsic differences in terms of neuroanatomical mechanism is that micro-expressions have very short duration, slight variation and fewer action areas on the external facial features [START_REF] Ekman | What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS)[END_REF], [START_REF] Wu | Micro-expression and its applications[END_REF]. This difference can also be concluded from the mechanism of concealment [START_REF] Ekman | Nonverbal leakage and clues to deception[END_REF]: when people try to conceal their feelings, their true emotion can quickly 'leak out' and may be manifested as micro-expressions.

Other studies have also shown that the motion of muscles in the upper face (forehead and upper eyelid) during fake expressions are controlled by the precentral gyrus, which is dominated by the bilateral nerve, while the motion of muscle in the lower face is controlled by the contralateral nerve [START_REF] Bhushan | Study of facial micro-expressions in psychology[END_REF], [START_REF] Porter | Reading between the lies: Identifying concealed and falsified emotions in universal facial expressions[END_REF], [START_REF] Rothwell | Silent talker: A new computerbased system for the analysis of facial cues to deception[END_REF]. The spontaneous expressions are controlled by the subcortical structure, the muscular movement of which is controlled by the bilateral fibre. This also provides evidence for the restraint hypothesis [START_REF] Darwin | The expression of the emotions in man and animals[END_REF], which states that random expressions controlled by the pyramidal tract neural pathway undergo a back-and-forth battle with the spontaneous expressions controlled by the extrapyramidal tract; during this process, the extrapyramidal tract is dominated by bilateral fibers. Furthermore, if activity in the extrapyramidal tract is reflected on the face, it controls the muscles of the upper face, leading to spontaneous expressions [START_REF] Bhushan | Study of facial micro-expressions in psychology[END_REF]. This also leads to the conclusion that when microexpressions occur, they can happen on different parts of the face [START_REF] Wojciechowski | Emotional intelligence and mismatching expressive and verbal messages: A contribution to detection of deception[END_REF].

The hypothesis that feedback from the upper and lower facial regions has different effects on micro-expression recognition was proposed in [START_REF] Zeng | A false trail to follow: differential effects of the facial feedback signals from the upper and lower face on the recognition of micro-expressions[END_REF]. Three supporting studies were subsequently presented to highlight the three roles of facial feedback in judging the subtle movements of micro- expressions. First, when feedback in the upper face is enhanced by a restricting gel, micro-expressions with a duration of 450 ms can be more easily detected. Second, when lower face feedback is enhanced, the accuracy of sensing micro-expressions (duration conditions of 50, 150, 333, and 450 ms) is reduced. Third, blocking lower face feedback improves the recognition accuracy of micro-expressions.

Characteristics of micro-expressions. Micro-expressions can express seven universal emotions: disgust, anger, fear, sadness, happiness, surprise and contempt [START_REF] Polikovsky | Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[END_REF]. Ekman suggests that there are certain facial muscles that cannot be consciously controlled, which he refers to as reliable muscles; these muscles serve as sound indicators of the occurrence of related emotions [START_REF] Ekman | From flawed self-assessment to blatant whoppers: the utility of voluntary and involuntary behavior in detecting deception[END_REF]. Studies have also shown that micro-expressions may contain all or only part of the muscle movements that make up common expressions [START_REF] Ekman | From flawed self-assessment to blatant whoppers: the utility of voluntary and involuntary behavior in detecting deception[END_REF], [START_REF] Ekman | Darwin, deception, and facial expression[END_REF], [START_REF]Lie catching and microexpressions[END_REF]. Therefore, compared to macro-expressions, microexpressions are facial expressions with short duration and characterized by greater facial muscle movement inhibition [START_REF] Ekman | From flawed self-assessment to blatant whoppers: the utility of voluntary and involuntary behavior in detecting deception[END_REF], [START_REF] Ekman | Darwin, deception, and facial expression[END_REF], which can reflect a person's true emotions and are more difficult to control [START_REF] Hurley | Executing facial control during deception situations[END_REF]. Table 1 summarizes the major differences between macro-and micro-expressions.

Short duration. The short duration is considered to be the most important characteristic of micro-expressions. Most psychologists now agree that micro-expressions do not last for more than half a second. Yan et al. [START_REF] Yan | How fast are the leaked facial expressions: The duration of micro-expressions[END_REF] performed an elegant analysis that employed distribution curves of total duration and onset duration for the purposes of microexpression analysis. These authors revealed that in a microexpression, the start-up time (time from the onset frame to the apex frame) is usually within 0.26 seconds. Furthermore, the intensity of muscle motion in the micro-expression is very weak and the expression itself is uncontrollable due to the psychological inhibition of human instinctive response.

Dynamic features. Neuropsychology shows that left and right sides of the face differ from each other in terms of expressing emotions [START_REF] Bhushan | Study of facial micro-expressions in psychology[END_REF]: the left side expresses emotions that are more intense. Studies on facial asymmetry have also found that the right side expresses social context cues more conspicuously, while the left side expresses more personal feelings [START_REF] Bhushan | Study of facial micro-expressions in psychology[END_REF]. These studies provide further evidence for the distinction between fake expressions and natural expressions, and thus indirectly explain the dynamic features of micro-expressions.

DATASETS OF MICRO-EXPRESSIONS

The development of micro-expression analysis techniques has been largely dependent on well-established datasets with correctly labelled ground truth. Due to their intrinsic characteristics, such as involuntariness, short duration and slight variation, eliciting micro-expressions in a controlled environment is very difficult. Thus far, a few microexpression datasets have been developed. However, as summarized in this section, most of these still have various deficiencies as regards their elicitation paradigms, labelling methods or small data size. Therefore, at the end of this section, we present and release a new dataset for microexpression recognition.

Some early studies elicited micro-expressions by constructing high-stakes situations; for example, asking people to lie by concealing negative affect aroused by an unpleasant film and simulating pleasant feelings [START_REF] Ekman | Detecting deception from the body or face[END_REF], or telling lies about a mock theft in a crime scenario [START_REF] Frank | The ability to detect deceit generalizes across different types of high-stake lies[END_REF]. However, microexpressions elicited in this way are often contaminated by other non-emotional facial movements, such as conversational behavior.

Since 2011, nine representative micro-expression datasets have been developed: namely, USF-HD [START_REF] Shreve | Macro-and micro-expression spotting in long videos using spatio-temporal strain[END_REF], Polikovsky's dataset [START_REF] Polikovsky | Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[END_REF], York DDT [START_REF] Warren | Detecting deception from emotional and unemotional cues[END_REF], MEVIEW [START_REF] Petr Husak | Spotting facial micro-expressions "in the wild[END_REF], SMIC [START_REF] Li | A spontaneous microexpression database: Inducement, collection and baseline[END_REF], CASME [START_REF] Yan | CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces[END_REF], CASME II [START_REF] Yan | CASME II: An improved spontaneous micro-expression database and the baseline evaluation[END_REF], SAMM [START_REF] Davison | SAMM: A spontaneous micro-facial movement dataset[END_REF] and CAS(ME) 2 [START_REF] Qu | CAS (ME) 2 : a database for spontaneous macro-expression and microexpression spotting and recognition[END_REF]. Both USF-HD and Polikovsky's dataset contain posed micro-expressions:

• in USF-HD, participants were asked to perform both macro-and micro-expressions, and • in Polikovsky's dataset, participants were asked to simulate micro-expression motion.

It should be noted that these posed micro-expressions are different from spontaneous ones. Moreover, York DDT is made up of spontaneous micro-expressions with high ecological validity. Nevertheless, similar to lie detection [START_REF] Ekman | Detecting deception from the body or face[END_REF], [START_REF] Frank | The ability to detect deceit generalizes across different types of high-stake lies[END_REF], the data in York DDT was mixed with non-emotional facial movement resulting from talking. Furthermore, all of three datasets (USF-HD, Polikovsky's dataset and York DDT) are not publicly available.

The paradigm of elicitation by telling lies has two major drawbacks:

• micro-expressions are often contaminated with irrelevant (i.e., talking) facial movements, and • the types of elicited micro-expressions are seriously restricted (e.g., the happiness type can never be elicited).

By contrast, it is well recognized that watching an emotional video while maintaining a neutral expression (i.e. suppressing emotions) is an effective method for eliciting microexpressions.

Five datasets -SMIC, CASME, CASME II, SAMM and CAS(ME) 2 -used this elicitation paradigm. MEVIEW used another quite different elicitation paradigm: namely, constructing a high-stakes situation by making use of poker games or TV interviews with difficult questions.

In the below, we first review MEVIEW, summarizing its advantages and disadvantages, and then move on to focus on the mainstream datasets (SMIC, CASME, CASME II, SAMM and CAS(ME) 2 ). All six of these datasets, which are publicly available, are summarized in Table 2; see also Figure 3 for some snapshots. For comparison, Table 2 also includes our newly released dataset MMEW, which is presented in more detail at the end of this section.

MEVIEW [START_REF] Petr Husak | Spotting facial micro-expressions "in the wild[END_REF] consists of realistic video clips (i.e., shots in a non-lab controlled environment) from two scenarios: (1) some key moments in poker games, and (2) a person being asked a difficult question in a TV interview. Both scenarios are notable for their high stress factor. For example, in poker games, players try to conceal or fake their true emotions, and the key moments in the videos show the detail of a player's face while the cards are being uncovered: at these moments, micro-expressions are most likely to appear. The MEVIEW dataset contains 40 micro-expression video clips at 25 fps with a resolution of 1280 × 720. The average length of the video clips in the dataset is 3 seconds and the camera shot is often switched. The emotion types in MEVIEW are divided into seven classes: happiness, contempt, disgust, surprise, fear, anger and unclear emotions.

The advantage of the MEVIEW dataset is that its scenarios are real, which will benefit the training or testing of micro-expression analysis algorithms. The disadvantages include the following: (1) in these real-scene videos, the participants were seldom shot from the frontal pose, meaning that the number of valid samples is quite small, and (2) the number of participants in the dataset is only 16, which is small.

We will now review SMIC, SAMM, CASME, CASME II and CAS(ME) 2 . All the image sequences in these datasets are shots in controlled laboratory environments.

The SMIC dataset [START_REF] Li | A spontaneous microexpression database: Inducement, collection and baseline[END_REF] provides three data subsets with different types of recording cameras: HS (standing for highspeed camera), VIS (for normal visual camera) and NIR (for near-infrared camera). Since micro-expressions have a short duration and low intensity, a higher spatial and temporal resolution may help to capture more details. Therefore, the HS subset was collected with a high-speed camera with a frame rate of 100 fps. HS can be used to study the characteristics of the rapid changes of the micro-expressions. Moreover, the VIS and NIR subsets are used to add diversity of the dataset, meaning that different algorithms and comparisons that address more modes of micro-expressions can possibly be developed. Both the VIS and NIR subsets were collected with 25 fps and 640 × 480 resolution. In contrast to a down-sampled version of the data with 100 fps, VIS can be used to study the normal behavior of micro-expressions, e.g. when motion blurs in web camera footage appear. NIR images were photographed using a near-infrared camera and can be used to eliminate the effect of lighting illumination on micro-expressions. The drawbacks of the SMIC dataset pertain to its labelling methods: (1) only three emotion labels (positive, negative and surprise) are provided, (2) the emotion labeling was only based on participants' selfreporting, and given that different participants may rank the same emotion differently, this overall self-reporting may not be precise, and (3) the labels of action units (AUs) are not provided in SMIC.

AUs are fundamental actions of individual muscles or groups of muscles, which are defined according to the Facial Action Coding System (FACS) [START_REF] Ekman | Facial action coding system (FACS): a technique for the measurement of facial actions[END_REF], a system used to categorize human facial movements by their appearance on the face. AUs are widely used to characterize the physical expression of emotions. Although FACS has successfully TABLE 2: Seven publicly released micro-expression datasets.

Here, samples * refers to the samples of micro-expressions. Note that CAS(ME) 2 and MMEW also contain samples of macro-expressions. Not all participants produce micro-expressions; here, Participants * refers to the number of participants who generate micro-expressions. The parentheses enclose the number of emotional video clips in that category. Fig. 3: Snapshots of six micro-expression datasets. From left to right and from top to bottom, these are as follows: MEVIEW [START_REF] Petr Husak | Spotting facial micro-expressions "in the wild[END_REF], SAMM [START_REF] Davison | SAMM: A spontaneous micro-facial movement dataset[END_REF], SMIC [START_REF] Li | A spontaneous microexpression database: Inducement, collection and baseline[END_REF], CASME [START_REF] Yan | CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces[END_REF], CASME II [START_REF] Yan | CASME II: An improved spontaneous micro-expression database and the baseline evaluation[END_REF], and CAS(ME) 2 [START_REF] Qu | CAS (ME) 2 : a database for spontaneous macro-expression and microexpression spotting and recognition[END_REF]. been used to code emotion-related facial action in macroexpressions (Table 3), the AU labels in different emotion categories of micro-expressions are diverse and consequently deserving of study in their own right. Different from SMIC, AU labels were provided in CASME, CASME II, CAS(ME) 2 and SAMM. CASME, CASME II and CAS(ME) 2 were developed by the same group and utilized the same experimental protocol. In a well-controlled laboratory environment, four lamps were chosen to provide steady and high-intensity illumination (this approach can also effectively avoid the flickering of lights caused by alternative current). To elicit micro-expressions, participants were instructed to maintain a neutral facial expression when watching video episodes with high emotional valence. CASME II is an extended version of CASME, with the major differences between them being as follows:

• CASME II used a high-speed camera with a sampling rate of 200 fps, while the sampling rate in CASME is only 60 fps;

• CASME II has a larger face size (of 280 × 340 pixels) in image sequences, while the face size in the CASME samples is only 150 × 190 pixels;

• CASME and CASME II have 195 and 247 microexpression samples respectively. CASME II has a more uniformly balanced number of samples across each emotion class, i.e., Happiness (33 samples), Repression [START_REF] Hurley | Executing facial control during deception situations[END_REF], Surprise [START_REF] Ekman | Darwin, deception, and facial expression[END_REF], Disgust [START_REF] Zhao | Visual speaker identification with spatiotemporal directional features[END_REF] and Others [START_REF] Asthana | Robust discriminative response map fitting with constrained local models[END_REF]; by contrast, the CASME samples are much more poorly distributed, with some classes having very few samples (e.g., only 2 samples in fear and 3 samples in contempt; see Table 2).

Successfully eliciting micro-expressions by maintaining neutral faces is a difficult task in itself; it is even more difficult to elicit both macro-and micro-expressions. CAS(ME) 2 collected this kind of data, in which macro-and microexpressions are distinguished by duration time; i.e., whether or not it is smaller than 0.5 seconds. Figure 4 The SAMM dataset [START_REF] Davison | SAMM: A spontaneous micro-facial movement dataset[END_REF] contains 159 samples (i.e., image sequences containing spontaneous micro-expressions) recorded by a high-speed camera with 200 fps and a resolution of 2040 × 1088. Similar to the series of CASMEs, these samples were recorded in a well controlled laboratory environment with carefully designed lighting conditions, such that light fluctuations (which resulted in flickering on the recorded images) can be avoided. A total of 32 participants (16 males, 16 females, mean age 33.24) with a diversity of ethnicities (including 13 races) were recruited for the experiment. To assign emotion labels, in addition to self-reporting, each participant was required to fill in a questionnaire before starting the experiment, after which each emotional stimuli video was specially tailored for different participants to elicit the desired emotions in micro-expressions. Seven emotion categories were labelled in SAMM: contempt, disgust, fear, anger, sadness, happiness and surprise.

So far, CAS(ME) 2 is the most appropriate for microexpression spotting, since it is close to the real scenarios in that it contains both macro-and micro-expressions. However, it is not suitable for micro-expression recognition, since the number of micro-expression samples is small (only 57). Among SMIC, CASME, CASME II and SAMM, SAMM is the best for micro-expression recognition, but also has the limitation of small sample numbers. Accordingly, to both provide a suitable dataset for micro-expression recognition and inspire new research directions, we introduce a new dataset micro-and-macro expression warehouse (MMEW).

MMEW follows the same elicitation paradigm used in SMIC, CASME, CASME II, SAMM and CAS(ME) 2 , i.e., watching emotional video episodes while attempting to maintain a neutral expression. The full details of the MMEW dataset construction process are presented in the supplemental material. All samples in MMEW were carefully calibrated by experts with onset, apex and offset frames, and the AU annotation from FACS was used to describe the facial muscle movement area (Table 4). Compared with the state-of-the-art CASME II, the major advantages of MMEW are as follows:

•

The samples in MMEW have a larger image resolution (1920×1080 pixels), while the resolution of samples in CASME II is 640 × 480 pixels. Furthermore, MMEW has a larger face size in image sequences of 400 × 400 pixels, while the face size in the CASME II samples is only 280 × 340 pixels.

• MMEW and CASME II contain 300 and 247 microexpression samples, respectively. MMEW has more elaborate emotion classes, i.e. Happiness [START_REF] Shreve | Macro-and micro-expression spotting in long videos using spatio-temporal strain[END_REF], Anger [START_REF] Ekman | Telling lies: Clues to deceit in the marketplace, politics, and marriage[END_REF], Surprise [START_REF] Bartlett | Insights on spontaneous facial expressions from automatic expression measurement[END_REF], Disgust [START_REF] Sun | A quantitative analysis of current practices in optical flow estimation and the principles behind them[END_REF], Fear [START_REF] Ekman | The repertoire of nonverbal behavior: Categories, origins, usage, and coding[END_REF], Sadness [START_REF] Oh | A survey of automatic facial micro-expression analysis: Databases, methods, and challenges[END_REF] and Others [START_REF] Zhang | Micro-expression recognition system[END_REF]; by contrast, the emotion classes of Anger, Fear and Sadness are not included in CASME II. Furthermore, the class of Others in CASME II contains 102 samples (41.3% of all samples).

• 900 macro-expression samples with the same class category (Happiness, Anger, Surprise, Disgust, Fear, Sadness), acted out by the same group of participants, are also provided in MMEW (see Figure 1 for an example). These may be helpful for further cross-modal research (e.g., from macro-to microexpressions).

Compared with the state-of-the-art SAMM, MMEW contains more samples (300 vs. 159; see Table 2). Moreover, MMEW contains sufficient samples of both macro-and micro-expressions from the same subjects (Figure 1). This new feature also opens up a promising new avenue of pretraining with macro-expression data from the own dataset instead of looking for other datasets, which we summarize in Section 8.3. MMEW is publicly available 1 .

In Section 8, we use CAS(ME) 2 to evaluate microexpression spotting methods, and further use SAMM and MMEW to evaluate micro-expression recognition, which can serve as baselines for the comparison of new methods developed in the future.

MICRO-EXPRESSION FEATURES

Since a micro-expression is a subtle and indiscernible motion of human facial muscles, the effectiveness of microexpression spotting and recognition relies heavily on discriminative features that can be extracted from image sequences. In this section, we review existing features using a hierarchical taxonomy. Note that state-of-the-art deep learning techniques have been applied to micro-expression analysis in an end-to-end manner, so that hand-crafted features are less necessary. The related deep learning techniques will be reviewed in Section 6.2.

Throughout this section, all summarized features are computed on a preprocessing and normalized image sequence. The preprocessing involves detecting the facial region, identifying facial feature points, aligning the facial regions in each frame to remove head movements, normalizing or interpolating additional images into the original sequence, etc. The reader is referred to Sections 3.1 and 3.2 in [START_REF] Takalkar | A survey: facial micro-expression recognition[END_REF] for more details on preprocessing.

The classical voxel representation depicts a given image sequence in a spatiotemporal space R 3 , with three dimensions x, y and t: here, x and y are pixel locations in one frame and t is the frame time. At each voxel v(x, y, t), gray or color values are assigned. Subsequently, the local, tiny transient changes of facial regions within an image sequence can be effectively captured by local patterns of gray or color information at each voxel, which we refer to as a kind of dynamic texture (DT). DT has been widely studied in the computer vision field and is traditionally defined as image sequences of moving objects/scenes that exhibit certain stationarity properties in time (see e.g. [START_REF] Doretto | Dynamic textures[END_REF]). Here, we treat DT as an extension of texture -local patterns of color at each image pixel -from the spatial domain to the spatiotemporal domain in an image sequence. There are three ways to make use of this local intensity pattern information:

• DT features in the original spatiotemporal domain (x, y, t) ∈ R 3 (Section 4.1);

• DT features in the frequency domain: by applying Fourier or wavelet transforms to a signal in R 3 , the information in the spatiotemporal domain can also be dealt with in the frequency domain (Section 4.2);

• DT features in a transformed domain by tensor decomposition: by representing a color microexpression image sequence as a 4th-order tensor, all the information in (x, y, t) ∈ R 3 can be interpreted via tensor decomposition: namely, the facial spatial information is mode-1 and mode-2 of the tensor, temporal information is mode-3 and color information is mode-4 (Section 4.3); We examine these four feature classes in more detail in the following four subsections and all presented features are summarized in Table S2 in the supplementary material.

DT features in the spatiotemporal domain

LBP-based features

Many DT features in the spatiotemporal domain are related to local binary patterns from three orthogonal planes (LBP-TOP) [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF]. The local binary pattern (LBP) [START_REF] Ojala | Multiresolution grayscale and rotation invariant texture classification with local binary patterns[END_REF] on each plane is represented as

LBP P,r = P -1 p=0 s(g p -g c )2 p , ( 1
)
where c is a pixel at frame t, referred to the center, g c is the gray value of c, g p denotes the gray value of the pth pixel, P is the number of neighboring points located inside the circle of radius r centered at c, and s(x) is an indicator function

s(x) = 1 x ≥ 0 0 x < 0 (2)
By concatenating the LBPs' co-occurrence statistics in three orthogonal planes, Zhao et al. [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF] proposed the novel LBP-TOP descriptor, which concatenates LBP histograms from three planes. LBP-TOP has been successfully applied to the recognition of both macro-and micro-expressions [START_REF] Li | A spontaneous microexpression database: Inducement, collection and baseline[END_REF], [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF]. Rotation-invariant features, such as LBP-TOP, do not fully consider all directional information. Ben et al. [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF] proposed hot wheel patterns (HWP), HWP-TOP, and dual-cross patterns from three orthogonal planes (DCP-TOP) based on DCP [START_REF] Ding | Multi-directional multilevel dual-cross patterns for robust face recognition[END_REF]. The experiments in [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF] show that these features, when enhanced by directional information description, can further improve the micro-expression recognition accuracy.

To resolve the issue that LBP-TOP may be coded repeatedly, Wang et al. [START_REF] Wang | LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition[END_REF] proposed an LBP with six intersection points (LBP-SIP), which can delete the six intersections of repetitive coding. In doing so, it reduces redundancy and the histogram length, and also improves the speed. Moreover, in order to preserve the essential patterns so as to improve recognition accuracy and reduce redundancy, Wang et al. [START_REF]Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition[END_REF] proposed the super-compact LBP-three mean orthogonal planes (MOP) for micro-expression recognition. However, the accuracy of this approach is slightly worse when dealing with short video. To make the features more compact and robust to changes in light intensity, Huang et al. [START_REF] Huang | Texture description with completed local quantized patterns[END_REF] proposed a completed local quantization patterns (CLQP) approach that decomposes the local dominant pattern of the central pixel with the surrounding pixels into sign, magnitude and orientation respectively, then transforms it into a binary code. They also extended CLQP into 3D space, referred to as spatiotemporal CLQP (STCLQP) [START_REF] Huang | Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[END_REF].

Second-order features

This class of features make use of second-order statistics to characterize new DT representations of micro-expressions [START_REF] Niu | Discriminative video representation with temporal order for micro-expression recognition[END_REF]. Hong et al. [START_REF] Hong | Capturing correlations of local features for image representation[END_REF] proposed a second-order standardized moment average pooling (called 2Standmap) that calculates low-level features (such as RGB) and intermediate local descriptors (such as Histogram of Gradient Orientation; HIGO) for each pixel by means of second-order average and max operations.

John et al. [START_REF] Ruiz-Hernandez | Encoding local binary patterns using the re-parametrization of the second order gaussian jet[END_REF] proposed the re-parametrization of the second-order Gaussian jet for encoding LBP. This approach obtains a set of three-dimensional blocks; based on these, more robust and reliable histograms can be generated, which are suitable for different facial analysis tasks. Kamarol et al. [START_REF] Kamarol | Spatiotemporal feature extraction for facial expression recognition[END_REF] proposed a spatiotemporal texture map (called STTM) to convolve an input video sequence with a 3D Gaussian kernel function in a linear space representation. By calculating the second-order moment matrix, the spatiotemporal texture and the histogram of the microexpression sequence are obtained. This algorithm captures subtle spatial and temporal variance in facial expressions with lower computational complexity, and is also robust to illumination variations.

Integral projection

Integral projection -which is a one-dimensional curved shape pattern -have also been investigated in the microexpression analysis context. Integral projection can be represented by a vector in which each entry corresponds to a 1D position (obtained by projecting the image along a given direction) and the entry value is the sum of the gray values of the projected image pixels at this position. Huang et al. [START_REF] Huang | Facial microexpression recognition using spatiotemporal local binary pattern with integral projection[END_REF] proposed a spatiotemporal LBP with Integral Projection (STLBP-IP); according to this approach, LBP is evaluated on the integral projections of image pixels obtained in the horizontal and vertical directions.

The same research group [START_REF] Huang | Facial microexpression recognition using spatiotemporal local binary pattern with integral projection[END_REF] also developed a revisited integral projection algorithm to maintain the shape property of micro-expressions, followed by LBP operators to further describe the appearance and motion changes from horizontal and vertical integral projections. They then proposed a discriminative spatiotemporal LBP with revisited integral projection (DiSTLBP-RIP) for micro-expression recognition [START_REF] Huang | Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition[END_REF]. Finally, they used a new feature selection method that extracted discriminative information based on the Laplacian method.

Other miscellaneous features

In addition to the aforementioned features, Polikovsky et al. [START_REF] Polikovsky | Facial micro-expression detection in hi-speed video based on facial action coding system (FACS)[END_REF] proposed a 3D gradient descriptor that employs AAM [START_REF] Cootes | Active appearance models[END_REF] to divide a face into 12 regions. By calculating and quantizing the gradient in all directions of each pixel, then constructing a 3D gradient histogram in each region, this descriptor extends the plane gradient histogram to capture the correlation between frames. Another widely used feature is the histogram of oriented gradients (HOG) [START_REF] Davison | Micro-facial movement detection using individualised baselines and histogrambased descriptors[END_REF] incorporating the convolution operation and weighted voting. These authors also proposed a histogram of image gradient orientation (HIGO) [START_REF] Davison | Micro-facial movement detection using individualised baselines and histogrambased descriptors[END_REF], which uses a simple vote rather than a weighted vote. HIGO can maintain good invariance of the geometric and optical deformation of the image. Moreover, the gradient or edge direction density distribution can better describe the local image appearance and shape, while small head motions can be ignored without affecting the recognition accuracy. HIGO is particularly suitable for occasions where the lighting conditions vary widely.

To maintain the invariance to both geometric and optical deformations of images, Chen et al. [START_REF] Chen | Emotion recognition using fixed length micro-expressions sequence and weighting method[END_REF] employed weighted features and weighted fuzzy classification to enhance the valid information contained in micro-expression sequences; however, their recognition process is still expensive in terms of time cost. Lu et al. [START_REF] Lu | A delaunaybased temporal coding model for micro-expression recognition[END_REF] proposed a Delaunay-based temporal coding model. This model divides the facial region into smaller triangular regions based on the detected feature points, then calculates the accumulated value of the difference between the pixel values of each triangular region in the adjacent frame by means of local temporal variations (LTVs). This approach encodes texture variations corresponding to facial muscle activities, meaning that any influence of personal appearance that is irrelevant to microexpressions can be suppressed.

Wang et al. [START_REF] Wang | Microexpression recognition using robust principal component analysis and local spatiotemporal directional features[END_REF] used robust PCA (RPCA) to decompose a micro-expression sequence into dynamic micro-expressions with subtle motion information. More specifically, they utilized an improved local spatiotemporal directional feature (LSTD) [START_REF] Zhao | Visual speaker identification with spatiotemporal directional features[END_REF] in order to obtain a set of directional codes for all six directions: i.e., XY(YX), XT(TX) and YT(TX). Subsequently, the decorrelated LSTD (DLSTD) was obtained by singular value decomposition (SVD) in order to remove the irrelevant information and thus allow the important micro-motion information to be emphasized. Zong et al. [START_REF] Zong | Learning from hierarchical spatiotemporal descriptors for microexpression recognition[END_REF] designed a hierarchical spatial division scheme for spatiotemporal descriptor extraction, and further proposed a kernelized group sparse learning (KGSL) model to process hierarchical scheme-based spatiotemporal descriptors. This approach can effectively choose a good division grid for different micro-expression samples, and is thus more effective for micro-expression recognition tasks. Zheng et al. [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF] developed a novel multi-task mid-level feature learning algorithm that boosts the discrimination ability of low-level features extracted by learning a set of class-specific feature mappings.

Frequency domain features

The micro-expression sequence can be transformed into the frequency domain by means of either Fourier or wavelet transforms. This makes relevant frequency features, such as amplitude and phase information, available for subsequent tasks. For example, the local geometric features (such as the corners of facial contours and the facial lines, which are easily overlooked by some feature description algorithms) can be easily identified by high-frequency information.

Oh et al. [START_REF] Oh | Monogenic riesz wavelet representation for microexpression recognition[END_REF] extracted the magnitude, phase and orientation of the transformed image by means of Riesz wavelet transform. To discover the intrinsic two-dimensional local structures (i2D) of micro-expressions, Oh et al. [START_REF] Oh | Intrinsic two-dimensional local structures for micro-expression recognition[END_REF] also performed a Fourier transform to restore the phase and orientation of the i2D via a high-order Riesz transformation using a Laplacian of Poisson (LOP) band-pass filter; this is followed by extracting LBP-TOP features and the corresponding feature histogram after quantification. An advantage of this approach is that the i2D can extract some easyto-lose local structures, such as complex facial contours. Similar to i2D, the i1D [START_REF] Fleischmann | 2D signal analysis by generalized Hilbert transforms[END_REF] is extracted through the use of a first-order Riesz transformation. To obtain directional statistical structures from micro-expressions, Zhang et al. [START_REF] Zhang | Micro-expression recognition system[END_REF] used the Gabor filter to obtain the texture images of important frequencies and suppress other texture images.

Different from the aforementioned research, Li et al. [START_REF] Li | Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods[END_REF] utilized Eulerian video magnification (EVM) to magnify the subtle motion in a video. In more detail, the representation of the frequency domain of the micro-expression sequence was obtained via Laplace transform, and some frequency bands were band-fed to enhance the corresponding scale of the movement and achieve directional amplification of the micro-expressions. Refining the EVM approach, Oh et al. [START_REF] Le Ngo | Eulerian emotion magnification for subtle expression recognition[END_REF] proposed the Eulerian motion magnification (EMM) method, which consists of amplitude-based EMM (A-EMM) and phase-based EMM (P-EMM). LBP-TOP was then used to extract the features of the micro-expression sequence, such that their algorithm enables the micro-movements to be magnified.

DT features in tensor-decomposition spaces

Treating the micro-expression sequence as a tensor enables rich structural spatiotemporal information to be extracted from it. Due to their high dimensionality, many tensorbased dimension reduction algorithms -which keep the inter-class distances as large as possible and the intra-class distances as small as possible -can be used to obtain a more effective discriminant subspace.

By viewing a gray-valued micro-expression sequence as a three-dimensional spatiotemporal tensor, Wang et al. [START_REF] Wang | Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine[END_REF] proposed a discriminant tensor subspace analysis (DTSA) that preserves some useful spatial structure information. More specifically, this method projects the micro-expression tensor to a low-dimensional tensor space in which the inter-class distance is maximized and intra-class distance is minimized.

Ben et al. [START_REF] Ben | Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[END_REF] proposed a maximum margin projection with tensor representation (MMPTR) approach. This method also views a micro-expression sequence as a thirdorder tensor, and can directly extract discriminative and geometry-preserving features by maximizing the inter-class Laplacian scatter and minimizing the intra-class Laplacian scatter.

To obtain better discriminant performance, Wang et al. [START_REF] Wang | Micro-expression recognition using color spaces[END_REF] proposed tensor independent color space (TICS). Representing a micro-expression sequence in three RGB channels, this approach extracted features from the four-order tensor space by utilizing LBP-TOP to estimate four projection matrices, each representing one side of the tensor data.

Optical flow features

Optical flow is a motion pattern of moving objects/scenes in an image sequence, which can be detected by the intensity change of pixels between two image frames over time. Many elegant optical flow algorithms have been proposed that are suitable for diverse application scenarios [START_REF] Sun | A quantitative analysis of current practices in optical flow estimation and the principles behind them[END_REF]. In microexpression research, optical flow has been investigated as an important feature.

Patel et al. proposed the spatiotemporal integration of optical flow vectors (STIOF) [START_REF] Patel | Spatiotemporal integration of optical flow vectors for micro-expression detection[END_REF], which computes optical flow vectors inside small local spatial regions and then integrates these vectors into the local spatiotemporal volumes. Liong et al. proposed an optical strain weighted features (OSWF) algorithm [START_REF] Liong | Subtle expression recognition using optical strain weighted features[END_REF], which extracts the optical strain magnitude for each pixel and uses the feature extractor to form the final feature histogram. Xu et al. [START_REF] Xu | Microexpression identification and categorization using a facial dynamics map[END_REF] proposed facial dynamics map (FDM) to capture small facial motions based on optical flow estimation [START_REF] Sun | A quantitative analysis of current practices in optical flow estimation and the principles behind them[END_REF].

Wang et al. [START_REF] Wang | A main directional maximal difference analysis for spotting facial movements from long-term videos[END_REF] proposed a main directional maximal difference (MDMD) algorithm to characterize the magnitude of maximal difference in the main direction of optical flow features. More recently, Liu et al. proposed the Mean Directional Mean Optical Flow (MDMO) feature, which integrates the magnitude and direction of the main optical flow vectors from a total of 36 non-overlapping regions of interest (ROIs) in a human face [START_REF] Liu | A main directional mean optical flow feature for spontaneous micro-expression recognition[END_REF]. While MDMO is a simple and effective feature, the average MDMO operation often loses the underlying manifold structure inherent in the feature space. To address this, the same research group further proposed a sparse MDMO [START_REF] Liu | Sparse MDMO: Learning a discriminative feature for spontaneous micro-expression recognition[END_REF] by constructing a dictionary containing all the atomic optical flow features in the entire video, as well as applying a time pool to achieve the sparse representation. MDMO and sparse MDMO are similar to the histogram of oriented optical flow (HOOF) feature [START_REF] Chaudhry | Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions[END_REF]; however, the key difference is that MDMO and sparse MDMO use HOOF features in a local way (i.e., in each ROI region) and are thus more discriminative.

Deep features

In addition to the aforementioned hand-crafted features, deep learning methods that can automatically extract optimal deep features have also been applied recently to microexpression analysis; we summarize this class of methods in Section 6.2. In these deep network models, the feature maps preceding the full connected layers (particularly the last full connected layer) can usually be regarded as deep features. However, these deep models are often designed as black boxes, and the interpretability or explainability of these deep features is frequently poor [START_REF] Ras | Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges[END_REF].

Feature interpretability

Compared with deep features which are implicitly learned through deep network optimization based on large training data, conventional hand-crafted features are usually designed based on either human experiences or statistical properties, which have better explainability. For examples, LBP-based features explicitly reflect a statistical distribution of local binary patterns. Integral projection method based on difference images can preserve the shape attributes of facial images. Optical-flow-feature-based methods use a kind of normalized statistic features that consider both local statistic motion information and spatial locations.

SPOTTING ALGORITHMS

In the literature, micro-expression detection and spotting are two related but often confused terminologies. In our study, we define micro-expression detection as the process of identifying whether or not a given video clip (or an image sequence) contains a micro-expression. Moreover, we put emphasis on micro-expression spotting, which goes beyond detection: in addition to detecting the existence of microexpressions, spotting also identifies three time spots, i.e. onset, apex and offset frames, in the whole image sequence:

• the onset is the first frame at which a microexpression starts (i.e., changing from the baseline, which is usually the neutral facial expression);

• the apex is the frame at which the highest intensity of the facial expression is reached;

• the offset is the last frame at which a microexpression ends (i.e., returning back to the neutral facial expression).

In the early stages of micro-expression research, manual spotting based on the FACS coding system was used [START_REF] Ekman | Facial action coding system (FACS): a technique for the measurement of facial actions[END_REF]. However, manual coding is laborious and time-consuming; e.g., recording a one-minute micro-expression video sample takes two hours on average [START_REF] Bartlett | Insights on spontaneous facial expressions from automatic expression measurement[END_REF]. Moreover, manual coding is subjective due to differences in the cognitive ability and living background of the participants [START_REF] Hess | Differentiating emotion elicited and deliberate emotional facial expressions[END_REF]; it is therefore highly desirable to develop automatic algorithms for spotting micro-expressions.

There are three major challenges associated with developing accurate and efficient micro-expression spotting algorithms. First, detecting micro-expressions usually relies on setting the optimal upper and lower thresholds for any given feature (see Section 4:) the upper threshold aims at distinguishing micro-expressions from macroexpressions, while the lower threshold defines the minimal motion amplitude of micro-expressions. Second, different people may perform different extra facial actions. For example, some people blink habitually, while other people sniff more frequently, which can cause movement in the facial area. The impact of these facial motion areas on expression spotting should thus be taken into consideration. Third, when recording videos, many comprehensive factors (including head movement, physical activity, recording environment, and lighting) may significantly influence the micro-expression spotting.

Existing automatic micro-expression spotting algorithms can be broadly divided into two classes: namely, opticalflow-based and feature-descriptor-based methods. For ease of reading, we highlight the pros and cons of these two classes in Table 5.

Optical-flow-based spotting

Optical flow algorithms can be used to measure the intensity change of image pixels over the time. Shreve et al. [START_REF] Shreve | Macro-and micro-expression spotting in long videos using spatio-temporal strain[END_REF], [START_REF] Shreve | Towards macro-and micro-expression spotting in video using strain patterns[END_REF] proposed a strain pattern that was used as a measure of motion intensity to detect micro-expressions. However, this method relies on manually selecting thresholds to distinguish between macro-expression and micro-expression. Furthermore, this method was designed to detect posed micro-expressions, but not spontaneous micro-expressions (note that posed and spontaneous micro-expressions vary widely in terms of facial movement intensity, muscle movement, and time intervals). Shreve et al. [START_REF] Shreve | Automatic expression spotting in videos[END_REF] used optical flow to exploit non-rigid facial motion by capturing the optical strains. This method can achieve an 80% true positive rate with a 0.3% false positive rate in spotting micro-expressions. Moreover, this method can also plot the strains and visualize a micro-expression as it occurs over the time.

Patel et al. [START_REF] Patel | Spatiotemporal integration of optical flow vectors for micro-expression detection[END_REF] used a discriminative response map fitting (DRMF) model [START_REF] Asthana | Robust discriminative response map fitting with constrained local models[END_REF] to locate key points of the face based on the FACS system. This method then groups the motion vectors of key points (indicated by the optical flow) and computes a cumulative value of the motion amplitude, shifted over time, to detect the onset, apex and offset frames of a micro-expression. However, this method also requires a manually specified threshold. Wang et al. [START_REF] Wang | A main directional maximal difference analysis for spotting facial movements from long-term videos[END_REF] used the magnitude maximal difference in the main direction of the optical flow features to spot micro-expressions. Guo et al. [START_REF] Guo | A magnitude and angle combined optical flow fea-ture for micro-expression spotting[END_REF] proposed a magnitude and angle combined optical flow feature for micro-expression spotting, and obtained more accurate results than the method proposed in [START_REF] Wang | A main directional maximal difference analysis for spotting facial movements from long-term videos[END_REF].

Feature descriptor based spotting

Polikovsky et al. [START_REF] Polikovsky | Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[END_REF], [START_REF] Polikovsky | Detection and measurement of facial micro-expression characteristics for psychological analysis[END_REF] proposed to use the gradient histogram descriptor and K-Means algorithm to locate the onset, apex and offset frames of posed micro-expressions. While their method is simple, it only works for posed microexpressions (not for spontaneous micro-expressions). Moilanen et al. [START_REF] Moilanen | Spotting rapid facial movements from videos using appearance-based feature difference analysis[END_REF] divided the face into 36 regions, calculated the LBP histogram of each region, then used the Chi-square distance between the current frame and the average to determine the degree of change in the video. While this method is novel, the design concept is somewhat complicated and the parameters in this method also need to be set manually.

Davison et al. [START_REF] Davison | Micro-facial movement detection using individualised baselines and histogrambased descriptors[END_REF] aligned and cropped faces for each video. This method involves splitting these faces into blocks and calculating the HOG for each frame. Afterwards, this method used Chi-Squared distance to calculate the dissimilarity between frames at a set interval in order to spot microexpressions. Xia et al. [START_REF] Xia | Spontaneous micro-expression spotting via geometric deformation modeling[END_REF] modeled the geometric deformation using an active shape model with SIFT descriptor [START_REF] Milborrow | Active shape models with SIFT descriptors and mars[END_REF] to locate the key points. Their method then performed the Procrustes transform between each frame and the first frame to remove the bias caused by the head motion. Subsequently, this method evaluated an absolute feature and its relative feature and combined them to estimate the transition probability. Finally, the micro-expression was detected according to a threshold. This method can minimize the error caused by head movement, but an optimal threshold is difficult to determine.

Li et al. [START_REF] Li | Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods[END_REF] made use of the Kanade-Lucas-Tomasi algorithm [START_REF] Tomasi | Detection and tracking of point features[END_REF] to track three specific points of every frame. This method extracted LBP and HOOF features from each [START_REF] Patel | Selective deep features for micro-expression recognition[END_REF] Avoids irrelevant deep features The fitness function is subjective to overfitting DTSCNN [START_REF] Peng | Dual temporal scale convolutional neural network for micro-expression recognition[END_REF] Avoids the overfitting problem Highly dependent on hardware DWLD +DBN [START_REF] Hao | Deep belief network based on double weber local descriptor in micro-expression recognition[END_REF] Reduces the unnecessary learning for the redundant features Insufficient samples TLCNN [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF] Solves the problem of limited micro-expression samples Complicated Deep learning ELRCN [START_REF] Khor | Enriched longterm recurrent convolutional network for facial micro-expression recognition[END_REF] Solves the problem of limited micro-expression samples There are shortcomings in preprocessing techniques and data augmentation DR [START_REF] Zong | Domain regeneration for cross-database micro-expression recognition[END_REF] Handles the unsupervised cross-dataset micro-expression recognition problem

The consistent feature distribution between the training and testing samples is seriously broken Source domain targetized [START_REF] Zhu | Coupled source domain targetized with updating tag vectors for micro-expression recognition[END_REF] Utilizes efficient speech data to enhance micro-expression recognition accuracy Large difference of data distribution SVD [START_REF] Jia | Macro-tomicro transformation model for micro-expression recognition[END_REF] Coupled metric learning [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF] Solves the limited samples problem Based on an assumption that a linear transformation exists between macro-and micro-expressions Transfer learning from other domains Auxiliary set selection model + a transductive transfer regression model [START_REF] Zong | Toward bridging microexpressions from different domains[END_REF] Selects a small number of representative samples from the target domain data, rather than choosing all of them Difficult to select the best sample number area and obtained the differential value of every frame, then detected the onset, apex and offset frames based on a threshold. This method successfully fuses two different features to get more discriminative information; however, again, the threshold is difficult to determine. Yan et al. [START_REF] Yan | Quantifying micro-expressions with constraint local model and local binary pattern[END_REF] and Han et al. [START_REF] Han | CFD: A collaborative feature difference method for spontaneous micro-expression spotting[END_REF] proposed to using feature difference to locate the apex frame of a microexpression. This method [START_REF] Yan | Quantifying micro-expressions with constraint local model and local binary pattern[END_REF] used a constrained local model (CLM) [START_REF] Cristinacce | Feature detection and tracking with constrained local models[END_REF] to locate 66 key points in a face, and then divided the face into subareas relative to these key points. Subsequently, they calculated the correlation between the LBP histogram feature vectors of every frame and first frame. Finally, the frame with the highest correlation value was regarded as the apex frame of a micro-expression. Different from the above methods, Liong et al. [START_REF] Liong | Automatic apex frame spotting in micro-expression database[END_REF] spotted the apex frame by utilizing a binary search strategy and restricting the region of interest (ROI) to a predefined facial sub-region; here, the ROI selection was based on the landmark coordinates of the face. Meanwhile, three distinct feature descriptors, namely, CLM, LBP and Optical Strain (OS), were adopted to further confirm the reliability of the proposed method. Although the methods in [START_REF] Yan | Quantifying micro-expressions with constraint local model and local binary pattern[END_REF] and [START_REF] Liong | Automatic apex frame spotting in micro-expression database[END_REF] did not require the manual setting of parameters and thresholds, they are only able to spot the apex frame. Some disadvantages of the above-mentioned feature-descriptorbased methods include their high computational cost and the instability caused by noise and illumination variation.

RECOGNITION ALGORITHMS

Micro-expression recognition usually comprises two steps: feature extraction and feature classification. Traditional classification methods rely on artificially designed features (e.g., those summarized in Section 4). State-of-the-art deep learning methods can automatically infer an optimal feature representation and offer an end-to-end classification solution. However, deep learning requires a large number of training samples, while all existing micro-expression datasets are small; therefore, transfer learning that makes use of the knowledge from a related domain (in which large datasets are available) has also been considered in microexpression recognition. In this section, we summarize these three classes of recognition methods: traditional methods, deep learning and transfer learning methods. For ease of reading, we highlight the pros and cons of these methods in Table 6.

Traditional classification algorithms

Pattern classification/recognition has a long history [START_REF] Duda | Pattern Classification (2nd Edition)[END_REF], and many well-developed classifiers have already been proposed; the reader is referred to [START_REF] Fernández-Delgado | Do we need hundreds of classifiers to solve real world classification problems?[END_REF] for more details. In the special application of micro-expression recognition, many classic classifiers have been applied, including the support vector machine (SVM) [START_REF] Wang | Micro-expression recognition using color spaces[END_REF], [START_REF] Xu | Microexpression identification and categorization using a facial dynamics map[END_REF], [START_REF] Liu | A main directional mean optical flow feature for spontaneous micro-expression recognition[END_REF], extreme learning machine (ELM) [START_REF] Guo | Micro-expression recognition based on CBP-TOP feature with ELM[END_REF], [START_REF] Wang | Microexpression recognition using robust principal component analysis and local spatiotemporal directional features[END_REF], [START_REF] Wang | Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine[END_REF] and K nearest neighbor (KNN) classifier [START_REF] Chaudhry | Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions[END_REF], [START_REF] Zhang | Micro-expression recognition by aggregating local spatio-temporal patterns[END_REF], to name only a few.

Deep learning

In recent years, deep learning approaches that integrate automatic feature extraction and classification in an endto-end manner have been great success. These deep models can obtain state-of-the-art predictive performance in many applications including micro-expression recognition [START_REF] Van Quang | Capsulenet for microexpression recognition[END_REF], [START_REF] Verma | Learnet: Dynamic imaging network for micro expression recognition[END_REF], [START_REF] Xia | Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions[END_REF].

Hao et al. [START_REF] Hao | Deep belief network based on double weber local descriptor in micro-expression recognition[END_REF] proposed an efficient deep network for micro-expression recognition. This network involves a twostage strategy: the first stage use a double Weber local descriptor for extracting initial local texture features, and the second stage uses a deep belief net (DWLD+DBN) for global feature extraction. Wang et al. [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF] proposed a transferring long-term convolutional neural network (TLCNN), which used Deep CNN to extract features from each frame of micro-expression video clips. Khor et al. [START_REF] Khor | Enriched longterm recurrent convolutional network for facial micro-expression recognition[END_REF] proposed an Enriched Long-term Recurrent Convolutional Network (ELRCN) that operates by encoding each micro-expression frame into a feature vector via CNN. It then predicts the micro-expression by passing the feature vector through a Long Short-term Memory (LSTM) module. Through the enrichment of the spatial dimension (via input channel stacking) and the temporal dimension (via deep feature stacking), TLCNN and LSTM can effectively deal with the problem of small sample size.

To tackle the overfitting problem, Patel et al. [START_REF] Patel | Selective deep features for micro-expression recognition[END_REF] developed a selective deep model that can remove irrelevant deep information from micro-expression recognition. Their model has a good generalization ability. Peng et al. [START_REF] Peng | Dual temporal scale convolutional neural network for micro-expression recognition[END_REF] further proposed a dual temporal scale convolutional neural network (DTSCNN) for micro-expressions recognition. DTSCNN used different streams to adapt to different frame rates of micro-expression video clips, each of which included an independent shallow network to prevent overfitting. These shallow networks were fed with opticalflow sequences to ensure that higher-level features could be extracted.

Since the deep network models have a huge number of parameters and weights, sufficiently well-labelled microexpression samples are urgently needed in these models to improve the recognition performance.

Transfer learning from other domains

Due to the small data size of existing micro-expression datasets, transfer learning -which transfers knowledge from a source domain (in which a large sample size is available) to a target domain -has been considered in micro-expression recognition. Usually, the source and target domains are different but related (e.g., macro-and microexpressions) so that they share some common knowledge (e.g., macro-and micro-expressions have similar AUs and DT information when expressing emotions) and benefitting from transferring this knowledge the performance in the target domain can be improved.

Zong et al. [START_REF] Zong | Domain regeneration for cross-database micro-expression recognition[END_REF] proposed an effective framework, called domain regeneration (DR), for cross-dataset microexpression recognition. The training and testing samples are derived from different micro-expression datasets. The DR framework is able to learn a regenerator that regenerates samples with similar feature distributions in source and target micro-expression datasets.

Another research group developed a series of transfer learning works [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF], [START_REF] Zhu | Coupled source domain targetized with updating tag vectors for micro-expression recognition[END_REF], [START_REF] Jia | Macro-tomicro transformation model for micro-expression recognition[END_REF] on micro-expression recognition. In [START_REF] Jia | Macro-tomicro transformation model for micro-expression recognition[END_REF], they proposed a macro-to-micro transformation model using singular value decomposition (SVD). This model takes advantage of sufficiently labelled macroexpression samples to increase the number of training samples. In [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF], the authors extracted several local binary operators (that jointly characterize macro-and microexpressions) and transferred these operators into a common subspace shared by source and target domains for learning purposes. In [START_REF] Zhu | Coupled source domain targetized with updating tag vectors for micro-expression recognition[END_REF], they proposed to transfer the knowledge of speech samples in the source domain to the microexpression samples in the target domain, in such a way that the transferred samples would have a similar feature distribution in the target domain. All these three transfer learning methods have been shown to achieve better recognition accuracy than some previous works.

Moreover, rather than transferring learning from macroexpressions or speech samples, a recent work [START_REF] Zong | Toward bridging microexpressions from different domains[END_REF] proposed an auxiliary set selection model (ASSM) and a transductive transfer regression model (TTRM) to bridge the gap between the source and target micro-expression domains. This method outperforms many state-of-the-art approaches, as the feature distribution difference between the source and target domains is small.

APPLICATIONS

Micro-expression analysis has a wide range of potential applications in the fields of criminal justice, business negotiation and psychological consultation, etc. In the below, we present the details of lie detection, which serves as a common and typical application scenario for micro-expression analysis.

One automatic lie detection instrument that is widely used today is the polygraph. This instrument usually collects multi-track physiology information such as respiration, pulse, blood pressure, pupil, skin electricity, and brain waves. If the test subject tells a lie, this instrument will record a fluctuation in some of the above physiological signals. However, whether the polygraph will work properly depends on many factors, such as the external environment during the test, the intelligence level of the tested person, the physical and mental condition of the tested person, and even the skill level of the polygraph operator. Sometimes, due to fear experienced as a result of the unfamiliar polygraph testing environment, some innocent people might show a state of panic and anxiety during the test, leading to false positives in the test results. On the other hand, some well-trained professionals may be able to pass a polygraph even while lying. In such a situation, micro-expression analysis can provide another method of lie detection.

Ekman [START_REF] Ekman | Telling lies: Clues to deceit in the marketplace, politics, and marriage[END_REF] pointed out that as an effective clue for use when identifying lies, micro-expressions have broad applications in lie detection. By detecting the occurrence of micro-expressions and understanding the emotion type behind them, it will be possible to accurately identify the true intentions of the participant and improve the rate of success in lie detection. For example, when the participant reveals a micro-expression of happiness, this may indicate hidden delight at having successfully passed the test [START_REF] Ekman | Emotions revealed: Recognizing faces and feelings to improve communication and emotional life[END_REF]. However, when the participant shows the micro-expression of surprise, this may indicate that the participant has never considered or understood the relevant questions. Because the micro-expression is difficult to hide and is usually the same as a person's true state of mind when lying, psychologists believe that the micro-expression is an important clue for lie detection. Pérez-Rosas et al. [START_REF] Pérez-Rosas | Deception detection using real-life trial data[END_REF] discovered that the five micro-expressions most related to falsehood are: frowning, eyebrow raising, lip corners turning up, lips protruded and head turning to the side.

COMPARISON

The research on micro-expression features, spotting and recognition algorithms, such as that summarized in this paper, are scattered throughout the literature. A great challenge is that their performance results are reported across different experimental settings, making it difficult to conduct a fair comparison based on a common framework. In this section, we present a study that compares a selected set of representative spotting and recognition algorithms. As pointed out in Section 3, thus far, the CAS(ME)2 dataset is most appropriate for spotting evaluation (Section 8.1), where the SAMM and MMEW datasets are most suitable for recognition evaluation (Section 8.2). By taking the advantage of the fact that it contains both macro-and microexpressions of the same subjects, we also compared the experimental results by using different datasets for pretraining, and tested the performance on MMEW and SAMM (Section 8.3). By ensuring that all the factors (including data sample size and pre-processing) are the same, the analysis provided in this section can serve as a baseline for the evaluation of new algorithms in future work.

ACCEPTED MANUSCRIPT

Evaluation of spotting algorithms

First, we study the influence of the feature extraction methods on the performance of micro-expression spotting. We use the MDMD [START_REF] Wang | A main directional maximal difference analysis for spotting facial movements from long-term videos[END_REF], HOG [START_REF] Davison | Micro-facial movement detection using individualised baselines and histogrambased descriptors[END_REF] and LBP [START_REF] Moilanen | Spotting rapid facial movements from videos using appearance-based feature difference analysis[END_REF] methods for comparison in our spotting experiments on CAS(ME) 2 . For MDMD and LBP, the micro-expression samples are divided into 5×5 and 6×6 blocks, respectively. For HOG, the number of blocks is 6×6, the signed gradient direction binning is set to 2π, and the number of direction bins is set to 8. Figure 5 plots the ROC curves of these three methods on CAS(ME) 2 ; the results show that MDMD performs the best in spotting micro-expressions. The possible reason is that MDMD applies the magnitude maximal difference in the main direction of optical flow features to detect/spot microexpressions, leading to more notable differences or discriminant ability than those from the HOG or LBP features.

Evaluation of recognition algorithms

Compared to spotting, micro-expression recognition strongly depends on the amplification of subtle features, and then usually requires an additional preprocessing step. In Section 8. 234 samples from 6 classes (i.e., happiness, surprise, anger, disgust, fear, sadness) were used 2 ; in the SAMM dataset, 72 samples from 5 classes (i.e., happiness, surprise, anger, disgust, fear) were used 3 . In both datasets, all samples were randomly split into five subsets according to "subject independent" approach, and the number of subjects in each subset is equal; i.e., our random split scheme ensures there is no overlapped subject between the test and the training sets at the same time. Five-fold cross-validation was performed, after which the average recognition results were reported.

Evaluation of different preprocessing methods

Image sequence alignment and interpolation are two common preprocessing methods utilized in micro-expression recognition. We analyze the influences of these two methods by keeping the other modules unchanged. The details are summarized below.

Alignment algorithms. In the micro-expression recognition task, it is necessary for the micro-expression sequence to normalize the size of faces and align the face shapes across all of the different video samples. After all background parts are removed, only the facial areas in each video are preserved. More specifically, each image is normalized to a size of 231 × 231 pixels. Since the largest number of frames among all the image sequences is 108, the micro-expression image sequences are interpolated to the maximal value of 110 frames. Here, we evaluate three alignment algorithms, namely ASM+LWM [START_REF] Yan | CASME II: An improved spontaneous micro-expression database and the baseline evaluation[END_REF], DRMF+Optical flow alignment (OFA) [START_REF] Liu | A main directional mean optical flow feature for spontaneous micro-expression recognition[END_REF] and joint cascade face detection and alignment (JCFDA) [START_REF] Chen | Joint cascade face detection and alignment[END_REF], for face alignment on the MMEW dataset. The LBP-TOP is used as a baseline to evaluate these three alignment algorithms.

In the settings of LBP-TOP, the radii R x , R y of axes X and Y vary from 1 to 4. In order to avoid having too many parameter combinations, R x is set to be equal to R y , and the variation range of the radius R t of the T axis is also ranged from 1 to 4. Thus, the numbers of neighborhood points of the XY , XT and Y T planes are all set to be 8. The recognition rates of these three alignment algorithms are obtained based on the uniform pattern under different parameters and different radius configurations. Finally, an SVM classifier with RBF kernel [START_REF] Chang | Libsvm: a library for support vector machines[END_REF] is applied. The experimental results are presented in Figures 6,7 and S3 in supplemental material. From the figures, we can draw the following conclusions. First, compared with the other two methods, JCFDA [START_REF] Chen | Joint cascade face detection and alignment[END_REF] achieves the best recognition rate of 38.9%. The reason is that JCFDA [START_REF] Chen | Joint cascade face detection and alignment[END_REF] combines face detection with alignment and learns at the same time under a cascade framework; this joint learning greatly improves the alignment and real-time performance. In the following experiments, we thus choose JCFDA [START_REF] Chen | Joint cascade face detection and alignment[END_REF] to preprocess the micro-expression image sequences. Interpolation algorithms. Two interpolation algorithms, namely Newton interpolation [START_REF] Ben | Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[END_REF] and TIM interpolation [START_REF] Pfister | Recognising spontaneous facial micro-expressions[END_REF], are used to interpolate micro-expression sequences into different numbers of frames, including [START_REF] Frank | I see how you feel: Training laypeople and professionals to recognize fleeting emotions[END_REF][START_REF] Darwin | The expression of the emotions in man and animals[END_REF][START_REF] Li | A spontaneous microexpression database: Inducement, collection and baseline[END_REF][START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF][START_REF] Ruiz-Hernandez | Encoding local binary patterns using the re-parametrization of the second order gaussian jet[END_REF][START_REF] Zhao | Visual speaker identification with spatiotemporal directional features[END_REF][START_REF] Ben | Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[END_REF][START_REF] Xia | Spontaneous micro-expression spotting via geometric deformation modeling[END_REF][START_REF] Hess | Differentiating emotion elicited and deliberate emotional facial expressions[END_REF][START_REF] Jia | Macro-tomicro transformation model for micro-expression recognition[END_REF][START_REF] Verma | Learnet: Dynamic imaging network for micro expression recognition[END_REF] frames. The highest recognition rates of these two algorithms with different interpolated frame numbers are provided in Figure S4 in supplemental material. We can observe that as the number of frames increases, the recognition rates of the two interpolation algorithms increase initially and then decrease towards the end. Newton interpolation can obtain its highest recognition rate of 33.3% under the interpolated frame number of 60. Compared with Newton interpolation, TIM interpolation has relatively higher recognition rate, with the highest recognition rate reaching 38.9% when the micro-expression sequence is respectively interpolated to 30 or 60 frames. Accordingly, in the following experiments, TIM is used to interpolate each micro-expression sequence to 60 frames.

Comparisons of traditional methods

In this section, we evaluate the recognition performances of representative traditional methods that use hand-crafted features, specifically LBP-TOP [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF], DCP-TOP [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF], LHWP-TOP [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF], RHWP-TOP [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF], LBP-SIP [START_REF] Wang | LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition[END_REF], LBP-MOP [START_REF]Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition[END_REF], STLBP-IP [START_REF] Huang | Facial microexpression recognition using spatiotemporal local binary pattern with integral projection[END_REF], DiSTLBP-RIP [START_REF] Huang | Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition[END_REF], FDM [START_REF] Xu | Microexpression identification and categorization using a facial dynamics map[END_REF], MDMO [START_REF] Liu | A main directional mean optical flow feature for spontaneous micro-expression recognition[END_REF] and Sparse MDMO [START_REF] Liu | Sparse MDMO: Learning a discriminative feature for spontaneous micro-expression recognition[END_REF]. Meanwhile, we select KNN, SVM (with RBF kernel) and ELM [START_REF] Guo | Micro-expression recognition based on CBP-TOP feature with ELM[END_REF] as three representative classifiers, due to the fact that these have been selected by most previous micro-expression recognition works. All of the above-mentioned methods follow their original settings outlined in the respective publications, except for STLBP-IP and DiSTLBP-RIP. Based on [START_REF] Huang | Facial microexpression recognition using spatiotemporal local binary pattern with integral projection[END_REF], we divide each frame into 5 × 5 blocks before feature extraction using STLBP-IP. In DiSTLBP-RIP, we produce the difference images of micro-expression image sequence, but we do not use robust principal component analysis (RPCA) [START_REF] Wright | Robust principal component analysis: Exact recovery of corrupted lowrank matrices via convex optimization[END_REF] as in [START_REF] Huang | Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition[END_REF].

Table 7 lists the best recognition rates of these handcrafted features combined with different classifiers on MMEW and SAMM. Except for LBP-TOP and sparse MDMO, the chosen parameters of each compared method Moreover, for DiSTLBP-RIP, each frame is also divided into 5 × 5 blocks, and the dimension of the discriminative groupbased feature using feature selection is 95. For FDM, each frame is divided into 6×6 blocks. Sparse MDMO achieves its best performance with the following parameters on MMEW (and SAMM): the sparsity balance weight is 0.4 (and 0.4), the dictionary size is 256 (and 256), and the pooling parameter is 1 (and 0.1).

From Table 7, it can be observed that generally speaking, better recognition accuracies are obtained by either SVM or ELM, but the performance of ELM is relatively unstable. MDMO and Sparse MDMO have better recognition rates on both datasets. MDMO (65.7%, 50%) and sparse MDMO (60%, 52.9%) are the top two methods on both MMEW and SAMM. The reasons are as follows: MDMO and sparse MDMO are based on optical flow features extracted from ROI. Both methods utilize local statistical motion and spatial location information, and further apply the robust optical flow calculation method to capture the texture part of the image, followed by affine transformation. Therefore, the obtained optical flow field is insensitive to illumination conditions and head movement, which is important to have a good performance on SAMM.

Comparisons of state-of-the-art methods

We next evaluate the recognition performance of state-ofthe-art methods (including deep learning methods, multitask mid-level feature learning [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF] and KGSL [START_REF] Zong | Learning from hierarchical spatiotemporal descriptors for microexpression recognition[END_REF]) on MMEW and SAMM. Table 8 summarizes the comparison results. The handcrafted features in Table 7 are also kept in Table 8 for ease of comparison. We note that the ranking of these methods is almost consistent on both datasets.

For the multi-task mid-level feature learning [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF], LBP-TOP, LBP-MOP and an extension of LBP-MOP are used as the low-level features. When compared with the performance of the original low-level features (such as LBP- [START_REF] Hu | Multi-task micro-expression recognition combining deep and handcrafted features[END_REF] 36.6 47.1 LBP-TOP [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF] 38.9 37.0 Selective deep features [START_REF] Patel | Selective deep features for micro-expression recognition[END_REF] 39.0 42.9 ELRCN [START_REF] Khor | Enriched longterm recurrent convolutional network for facial micro-expression recognition[END_REF] 41.5 46.2 DCP-TOP [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF] 42.5 36.8 ESCSTF [START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF] 42.7 46.9 LHWP-TOP [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF] 43.2 41.7 LBP-MOP [START_REF]Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition[END_REF] 43.9 35.3 LBP-SIP [START_REF] Wang | LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition[END_REF] 43.9 37.4 DiSTLBP-RIP [START_REF] Huang | Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition[END_REF] 44.0 46.2 RHWP-TOP [START_REF] Ben | Learning effective binary descriptors for micro-expression recognition transferred by macro-information[END_REF] 45.9 38.1 STLBP-IP [START_REF] Huang | Facial microexpression recognition using spatiotemporal local binary pattern with integral projection[END_REF] 46.6 42.9 ApexME [START_REF] Li | Can micro-expression be recognized based on single apex frame?[END_REF] 48.8 50.0 Transfer Learning [START_REF] Peng | From macro to micro expression recognition: deep learning on small datasets using transfer learning[END_REF] 52.4 55.9 Multi-task mid-level feature learning [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF] 54.2 55.0 KGSL [START_REF] Zong | Learning from hierarchical spatiotemporal descriptors for microexpression recognition[END_REF] 56.9 48.6 Sparse MDMO [START_REF] Liu | Sparse MDMO: Learning a discriminative feature for spontaneous micro-expression recognition[END_REF] 60.0 52.9 MDMO [START_REF] Liu | A main directional mean optical flow feature for spontaneous micro-expression recognition[END_REF] 65.7 50.0 DTSCNN [START_REF] Peng | Dual temporal scale convolutional neural network for micro-expression recognition[END_REF] 65.9 69.2 TLCNN [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF] 69.4 73.5

TOP [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF] and LBP-MOP [START_REF]Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition[END_REF]), multi-task mid-level feature learning [START_REF] He | Multi-task mid-level feature learning for micro-expression recognition[END_REF] can enhance the discrimination ability of the original features [START_REF] Zhao | Dynamic texture recognition using local binary patterns with an application to facial expressions[END_REF], [START_REF]Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition[END_REF]. In KGSL [START_REF] Zong | Learning from hierarchical spatiotemporal descriptors for microexpression recognition[END_REF], hierarchical STLBP-IP is used as the spatio-temporal descriptor, which benefits from a hierarchical spatial division scheme, so that KGSL [START_REF] Zong | Learning from hierarchical spatiotemporal descriptors for microexpression recognition[END_REF] obtains the third best results among the non-deeplearning methods (inferior to MDMO and Sparse MDMO). Moreover, it is noticeable that all deep learning methods perform better than those utilizing handcrafted features. up deep learning methods includes parameters, architectures, convolution layers, size of filters, batch normalization, etc.; the details are presented below.

Setting 1. The learning rate of ResNet10 [START_REF] He | Deep residual learning for image recognition[END_REF] is 0.0001. The batch size is set to 8. ResNet10 includes 3 blocks and 2 fully connected layers; each block consists of two 3D convolution layers with convolution kernel of 3 × 3 × 3 and one down-sampling layer with convolution kernel of 1 × 1 × 1. The output sizes of the two fully connected layers are 128 and 6 respectively.

Setting 2. For handcrafted features + deep features [START_REF] Hu | Multi-task micro-expression recognition combining deep and handcrafted features[END_REF], we employ two scales (10, 20 pixels) and three orientations (0, 60 and 120 degrees), resulting in 6 different Gabor filters, and set the sizes of radii on the three orthogonal planes of LBP-TOP to (1, 1, 2). The deep CNN contains 5 convolutional layers and 3 fully connected layers. The convolution kernel sizes are 11 × 11, 5 × 5, 3 × 3, 3 × 3 and 3 × 3, respectively. The output sizes of the 3 fully connected layers are 9216, 4096 and 1000, respectively.

Setting 3. ELRCN [START_REF] Khor | Enriched longterm recurrent convolutional network for facial micro-expression recognition[END_REF] is based on VGG-16, which contains 13 convolutional layers (3 × 3 conv) and 3 fully connected layers. The output sizes of 3 fully connected layers are 4096, 4096 and 2622, respectively. The output size of LSTM is 3000.

Setting 4. ApexME [START_REF] Li | Can micro-expression be recognized based on single apex frame?[END_REF] is also based on VGG-16, with the same number of layers and convolution kernel size as ELRCN [START_REF] Khor | Enriched longterm recurrent convolutional network for facial micro-expression recognition[END_REF]. The output sizes of the 3 fully connected layers are 256, 64 and 6, respectively. The batch size is set to 128. During the fine-tuning, the drop-out rate is set to 0.8 in order to avoid over-fitting.

Setting 5. In TLCNN [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF], the CNN contains 5 convolutional layers (11 × 11, 5 × 5, 3 × 3, 3 × 3, 3 × 3 conv) and 3 fully-connected layers. Moreover, there is a dropout layer after the first and second fully-connected layer. The learning rates for all training layers are all set to 0.001. The output sizes of the 3 fully connected layers are 256, 64 and 6, respectively. Setting 6. In DTSCNN [START_REF] Peng | Dual temporal scale convolutional neural network for micro-expression recognition[END_REF], the number of samples is extended to 90 for each class. All sequences are interpolated into different numbers of frames, i.e. 129 and 65. Their optical flow features are input into DTSCNN, which is a two-stream network containing 3D convolution and pooling units. The first network contains 4 convolutional layers (3 × 3 × 8, 3 × 3 × 3, 3 × 3 × 3, 3 × 3 × 4 conv) and 1 fully connected layer, the output size of which is 6. Different from the first network, the convolution kernel sizes are 3 × 3 × 16, 3 × 3 × 3, 3 × 3 × 3, and 3 × 3 × 4.

Setting 7. For selective deep features [START_REF] Patel | Selective deep features for micro-expression recognition[END_REF], the CNN contains 10 convolutional layers (3 × 3 × 3 conv) and 2 fully-connected layers. After every two convolutional layers, there is a down-sampling layer (1 × 1 × 1 conv). In order to remove the irrelevant deep features trained by ImageNet and the facial expression dataset, evolutionary search is used to generalize the micro-expression data. The values for mutation probability, iteration number and population size are set to 0.02, 25 and 60, respectively.

Setting 8. The same ResNet10, LSTM and fully connected layers are used in transfer learning [START_REF] Peng | From macro to micro expression recognition: deep learning on small datasets using transfer learning[END_REF] and ESCSTF [START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF]. ResNet10 contains 3 blocks and 2 fully connected layers; moreover, each block consists of two 3D convolution layers (3 × 3 × 3 conv) and one down-sampling layer (1 × 1 × 1 conv). The output sizes are [START_REF] Ekman | Telling lies: Clues to deceit in the marketplace, politics, and marriage[END_REF][START_REF] Ekman | Emotions revealed: Recognizing faces and feelings to improve communication and emotional life[END_REF]256,[START_REF] Petr Husak | Spotting facial micro-expressions "in the wild[END_REF][START_REF] Petr Husak | Spotting facial micro-expressions "in the wild[END_REF] respectively. The output of ResNet10 is fed to LSTM (3 × 3 × 3 conv). Finally, the output sizes of 2 fully connected layers are 1024 and 6. Although the two networks have the same structure, the training methods are different.

ResNet10 is pre-trained on ImageNet, then fine-tuned on macro-and micro-expression samples (16 interpolated frames for each sample) through transfer learning [START_REF] Peng | From macro to micro expression recognition: deep learning on small datasets using transfer learning[END_REF]. Moreover, in ESCSTF [START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF], 5 key frames are used to train ResNet10 for each micro-expression sample; namely, the onset, onset to apex transition, apex, apex to offset transition and offset frames.

ResNet10 [START_REF] He | Deep residual learning for image recognition[END_REF], TLCNN [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF], transfer learning [START_REF] Peng | From macro to micro expression recognition: deep learning on small datasets using transfer learning[END_REF] and ESCSTF [START_REF] Kim | Micro-expression recognition with expression-state constrained spatio-temporal feature representations[END_REF] use batch-normalization, which is implemented before the activation function (for example, ReLU). The purpose is to normalize the data to a mean of 0 and a variance of 1.

In Table 8, it can be seen that TLCNN [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF] achieves the best recognition performance (69.4% on MMEW and 73.5% on SAMM). This is due to (1) the use of the macroexpression samples 4 for pre-training, and (2) the use of

FUTURE DIRECTIONS

Despite the significant progress made in micro-expression analysis over the last decade, several outstanding issues and new avenues exist for future development. Below, we propose some potential research directions.

Privacy-protection micro-expression analysis. Akin to macro-expressions, micro-expressions also constitute a kind of private facial information. Micro-expression spotting and recognition conducted by utilizing federated learning from decentralized data distributed across private user devices deserves study in its own right [START_REF] Yang | Federated machine learning: Concept and applications[END_REF].

Utilizability of macro-expressions for micro-expression analysis. Both macro-and micro-expressions can be characterized according to the emotional facial action coding system. Considering the benefits offered by the new MMEW dataset, it would be interesting to explore the mutual effect of macroand micro-expressions, particularly those from the same subject. Furthermore, as previous research has indicated that many deceptive behaviors are dependent on individual differences [START_REF] Warren | Detecting deception from emotional and unemotional cues[END_REF], we conjecture that micro-expression behavior is likely subject-dependent and MMEW dataset provides a new platform for both subject-dependent and subject-independent research in the future.

Standardized datasets. There are two major problems in the current micro-expression datasets. First, the existing number of micro-expression samples is too limited to facilitate proper training, since inducing micro-expressions is quite difficult. Researchers often require participants to watch emotional videos to elicit their emotions, and even to disguise their expressions. However, some participants may not exhibit micro-expressions under these circumstances, or may only exhibit them rarely. In addition, the encoding/labeling of micro-expressions is time-consuming and laborious, since it requires the viewer to (1) watch the video at a slow speed and (2) select the onset, apex and offset frames of the facial motion, then calculate their duration. Consequently, there is no uniform standard available to annotate the emotion of micro-expressions. Second, owing to the poor quality of the videos in many existing micro-expression datasets, the full details of the lowintensity micro-expressions cannot be fully captured. Therefore, videos with higher temporal and spatial resolutions are needed for future algorithm design. Data augmentation. Some data augmentation tricks used by the deep learning community could be helpful to increase the amount of available micro-expression data. For example, image rotation, translating, cropping and similar techniques will not change the labels of micro-expression but will increase the amount of available micro-expression data, leading to a potential performance improvement.

GAN-based sample generation. Deep learning methods achieve superior performance in facial expression recognition tasks. For example, Deng et al. [START_REF] Li | Deep facial expression recognition: A survey[END_REF] present an extensive and detailed review of state-of-the-art deep learning techniques for facial expression recognition. However, it is difficult to apply them to micro-expression recognition, since they will suffer due to the lack of sufficient training samples. In addition to manipulating images for data augmentation, it would be feasible to address this issue through utilizing generative adversarial networks (GANs) in order to generate a large number of pseudo-micro-expression samples, provided that we can define some criteria to ensure that the generated samples are indeed micro-expressions.

Multi-task learning. The extraction of micro-expression heavily depends on the ability to detect facial feature points, the semantic location of which has been predefined. The reason is that the motion amplitude of micro-expression is quite subtle, and facial feature points detection can reduce the effect caused by head movement during data preprocessing. It would therefore be useful in the future to design an end-to-end model capable of both learning the motion amplitude of micro-expression and detecting facial feature points. In this way, we could potentially alleviate the cost of micro-expression annotations.

Explainable micro-expression analysis. Although deep learning methods have received increasing attention and achieved good performance in micro-expression analysis, these deep models are usually treated as black boxes and have poor interpretability and explainability. In the critical applications such as lie detection and criminal justice, explainability is very important to help human understand the reasons behind predictions [START_REF] Rudin | Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead[END_REF].

CONCLUSION

Micro-expression analysis has a wide range of potential real-world applications; for example, enabling people to detect micro-expressions in daily life and develop a good interpretation/understanding of what lies behind microexpressions. To make micro-expression analysis useful in practice, we need to develop robust algorithms with valid and reliable samples in order to make the spotting and recognition of micro-expressions applicable to real situations. Accordingly, in this survey, we review the current research on spontaneous facial micro-expression analysis (including datasets, features and algorithms) and propose a new dataset, MMEW, for micro-expression recognition. We further compare the performance of existing state-ofthe-art methods, analyze the potential, and highlight the outstanding issues for future research on micro-expression analysis. Micro-expression analysis has recently become an active research area; accordingly, we hope this survey can help researchers, as a starting point, to review the developments in the state-of-the-art and identify possible directions for their future research.

Fig. 2 :

 2 Fig.2: Two distinct neural pathways exist for conveying facial behavior: namely, the pyramidal and extrapyramidal tract neural pathways. The former is responsible for macroexpressions with voluntary facial actions, while the latter is responsible for spontaneous facial expressions. In highrisk scenarios, such as lying, both pathways are activated and engage in a back-and-forth battle, resulting in the fleeting leakage of genuine emotions in the form of microexpressions.

Fig. 4 :

 4 Fig. 4: Micro and macro-expression for the same person on the CAS(ME) 2 dataset. The first row is the original images, while the second row corresponds to the zoomed local regions of the micro-expression apex and the macroexpression of the same expression.

•

  Optical flow features, which indicate the patterns of motion of objects (facial regions in our case) and are often computed by the changing intensity of pixels between two successive image frames over time, based on partial derivatives of the image signals (Section 4.4).

Fig. 5 :

 5 Fig. 5: ROC results of three methods -MDMD, HOG and LBP -on CAS(ME) 2 in terms of micro-expression spotting.

Fig. 6 :

 6 Fig. 6: Comparisons of three alignment algorithms. Due to the space limitations, we only show four frames here. From left to right: (a) an original image sequence; (b) results yielded by ASM+LWM [32]; (c) results yielded by DRMF+OFA [85]; (d) results yielded by JCFDA [113].

Fig. 7 :

 7 Fig. 7: The highest recognition rates of LBP-TOP with three alignment and two interpolation algorithms on MMEW.

TABLE 1 :

 1 Main differences between macro-and microexpressions.

	Difference	Micro-expression	Macro-expression
	Noticeability	Easy to ignore	Easily noticed
	Time interval	Short duration (0.065-0.5 seconds) (0.5-4 seconds) Long duration
	Motion intensity Slight variation	Large variation
	Subjectivity	Involuntary (uncontrollable)	Voluntary (under control)
	Action areas	Fewer	Almost all areas

TABLE 3 :

 3 Emotional facial action coding system for macroexpressions

	Emotion	Action units
	Happiness 6+12
	Sadness	1+4+15
	Surprise	1+2+5B+26
	Fear	1+2+4+5+7+20+26
	Anger	4+5+7+23
	Disgust	9+15+16
	Contempt	R12A+R14A

TABLE 4 :

 4 The micro-expression categories and AU labeling

	Emotion categories	Main AU labeling type	Quantity
	Happiness AU6/AU12/AU6+AU12	36
	Surprise	AU5/AU26/AU1+AU2/AU18	89
	Anger	AU17+AU13/AU7/AU9	8
	Disgust	AU9/AU10/AU4+7/AU4+9	72
	Fear	AU4+5/AU20/AU4+11/AU14	16
	Sadness	AU14/AU17/AU14+17/AU1+2	13
	Others	AU4/AU18/AU4+14/AU38/AU4+7 66

macro-expression samples and 57 micro-expression samples. The emotion classes includes positive, negative, surprise and others. All samples in CASME, CASME II and CAS(ME) 2 were coded into onset, apex and offset frames, and labeled based on a combination of AUs, the emotion types of emotion-evoking videos and self-reported emotion.

TABLE 5 :

 5 Comparison of representative micro-expression spotting algorithms

			References Detection/spotting results	Advantages	Disadvantages
	Optical flow	based methods	[36] [73] [74] [75] [76] [77]	Distinguishing macro-expression from micro-expression Micro-expression detection/spotting Onset, apex and offset frames detection Spotting facial movements from long-term videos	Able to capture tiny expression variations Able to obtain more accurate features of movement	A threshold is manually set, when the training data is small. Time-consuming
	Feature-descriptor-based	methods	[23] [78] [79] [56] [80] [67] [81] [82]	Onset, apex and offset frames detection Micro-expression spotting Micro-expression spotting Onset, apex and offset frames detection Apex frame detection	Simple Satisfactory results Error due to head motion is minimized Combination of two different features is more powerful Parameters and thresholds are set automatically	Only suitable for posed micro-expression, not spontaneous micro-expression Very complicated; parameters and thresholds are manually set A threshold is not easy to determine Only able to spot the apex frame

TABLE 6 :

 6 Micro-expression recognition algorithms

	Traditional	classifiers	Algorithm SVM [71] [84] [85] ELM [91] [59] [69] KNN [87] [92]	Advantages Most commonly used Obtains an optimal solution Simple to use	Disadvantages Restricted by the limited training samples Restricted by the limited training samples Classification performance needs to be further improved
			Selective CNN

  2.1, we first evaluate the effects of different preprocessing methods on the MMEW dataset. Then we perform a unified comparison on the MMEW and SAMM datasets to evaluate the recognition accuracies of traditional methods (using hand crafted features) in Section 8.2.2 and state-of-the-art methods (including deep learning methods) in Section 8.2.3, respectively.Throughout this section, all recognition results were obtained under the following settings. In the MMEW dataset,

TABLE 7 :

 7 Recognition rates (%) of micro-expressions using hand-crafted features with different classifiers on MMEW and SAMM. The parameter for LBP-TOP is (1, 1, 4) on both MMEW and SAMM. For STLBP-IP, the frame number of each video clip is set to 45, and the size of the linear mask of the one-dimensional local binary pattern (1DLBP) is set to 9.

	Hand-crafted features	MMEW KNN SVM ELM KNN SVM ELM SAMM
	LBP-TOP [40]	34.5	38.9	32.4	31.9	37.0	30.5
	DCP-TOP [42]	37.6	42.5	36.2	36.1	36.8	32.8
	LHWP-TOP [42]	39.4	43.2	36.2	40.5	41.7	36.1
	RHWP-TOP [42]	37.6	45.9	37.8	36.1	38.1	43.8
	LBP-SIP [44]	35.3	43.9	36.6	36.5	37.4	35.8
	LBP-MOP [45]	43.9	41.5	34.6	32.4	35.3	34.7
	STLBP-IP [52]	36.6	46.3	46.6	35.7	42.9	39.3
	DiSTLBP-RIP [53]	39.0	44.0	41.5	42.9	46.2	42.9
	FDM [84]	30.8	34.6	31.4	33.3	34.1	32.4
	MDMO [85]	53.4	60.6	65.7	44.1	50.0	50.0
	Sparse MDMO [86]	42.9	51.0	60.0	44.7	52.9	52.9
	are the same on MMEW and SAMM. The sizes of radii
	R x , R y and R t on the three orthogonal planes (YT, XT and
	XY) of DCP-TOP, LHWP-TOP, RHWP-TOP, LBP-SIP, LBP-
	MOP are (2, 2, 4), (4, 8, 4), (5, 8, 4), (1, 1, 2), and (1, 1, 2),
	respectively.

TABLE 8 :

 8 Recognition rates (%) of micro-expressions using the state-of-the-art methods on MMEW and SAMM. The ranking of these methods are almost consistent on both datasets. (The best record of each dataset is marked in bold.)

	Methods	Recognition rate (%) MMEW SAMM
	FDM [84]	34.6	34.1
	ResNet10 [117]	36.6	39.3
	Handcrafted features + deep features

TABLE 9 :

 9 Data sources and recognition rates (%) of the pre-training, fine-tuning and testing phases.

	Experiments	Pre-training Data source Rec. rate	Fine-tuning Data source Rec. rate	Data source	Testing	Rec. rate
	CK+ → MMEW (Micro)	CK+	99.9%	MMEW (Micro)	99.8%	MMEW (Micro)	65.6%
	MMEW (Macro) → MMEW (Micro)	MMEW (Macro)	92.0%	MMEW (Micro)	96.6%	MMEW (Micro)	69.4%
	CK+ → SAMM	CK+	99.0%	SAMM	99.7%	SAMM		73.5%

http://fu.psych. ac.cn/CASME/ casme2-en.php http://fu.psych. ac.cn/CASME/ cas(me)2-en.php http://www2. docm.mmu.ac. uk/STAFF/M. Yap/dataset.php http://www. dpailab.com/ database.html

SUBMITTED TO IEEE T-PAMI

Because transfer learning methods were included in our comparison, we exclude 66 samples in the "Others" category.

3. We exclude 3 samples from the "Sadness" category (due to its small sample size) and 84 samples from the "Others" category due to the inclusion of transfer learning in our comparison.

MMEW contains macro-expressions. For SAMM, the CK+ dataset[START_REF] Lucey | The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression[END_REF] are used as macro-expression samples.

ACKNOWLEDGEMENT

This work was partially supported by the Natural Science Foundation of China (61725204). The authors would like to thank the associate editor and three anonymous reviewers for their valuable comments, which greatly improve the quality of this paper.

the micro-expression samples fine-tuning, which solves the problem of the insufficient number of micro-expression samples. Moreover, LSTM is used to extract the discriminative dynamic characteristics from micro-expression sample sequences. We also present confusion matrices in Figure 8, from which we observe that in MMEW, all the "disgust" and "surprise" samples can be completely recognized on MMEW, instead the "fear" and "sadness" samples are difficult to train. Because about four fifths of fear ( 16) and sadness (13) samples of MMEW were used for fine-tuning (where the number in brackets is the total sample number of each class) and the samples for fine-tuning are too few.

A similar situation occurs in SAMM where the total sample number of fear and sadness are 7 and 3, respectively. The classes for instance "fear" and "sadness" are more likely to be inconsistent in the classification results (Figure 8b). Among deep learning methods, the baseline is ResNet10 [START_REF] He | Deep residual learning for image recognition[END_REF], which can only obtain a recognition rate of 36.6% on MMEW and 39.3% on SAMM, due to over-fitting and the limited number of samples. The samples from the ImageNet dataset differ greatly from micro-expression samples, and a limited fine-tuning effect spoils the recognition performance; therefore, transfer learning [START_REF] Peng | From macro to micro expression recognition: deep learning on small datasets using transfer learning[END_REF] achieves recognition accuracies of 52.4% on MMEW and 55.9% on SAMM.

Encoding micro-from macro-expressions

The success of TLCNN [START_REF] Wang | Micro-expression recognition with small sample size by transferring long-term convolutional neural network[END_REF] demonstrates that the macroexpressions are relevant for pre-training the network; this effectively alleviates the famous 3S (small sample size) problem inherent in micro-expression datasets. We further performed three group experiments: (1) CK+ was used for pre-training while MMEW (Micro) was applied for finetuning and testing, noted as "CK+ → MMEW (Micro)"; (2) MMEW (Macro) was used for pre-training while MMEW (Micro) was applied for fine-tuning and testing, noted as "MMEW (Macro) → MMEW (Micro)"; (3) CK+ was used for pre-training while SAMM was applied for fine-tuning and testing, noted as "CK+ → SAMM". The training and testing sets of MMEW were set as above-mentioned. Table 9 lists the data source and accuracy of the pre-training, finetuning and testing phases for each experiment. The results, i.e., MMEW (Macro) → MMEW (Micro) > CK+ → MMEW (Micro) (with descending effect), indicate that the knowledge of macro-expressions is useful for micro-expression recognition, and using macro-and micro-expressions from the same dataset (collected in the same laboratory environment) to pre-trained and fine-tune has better performance than using those from different datasets. This also verifies our motivation to establish MMEW, i.e. dataset containing both macro-and micro-expression of the same subject is useful for transferring macro-expression knowledge to assist micro-expression recognition. The advantage of the new released MMEW dataset brings up an interesting problem:

"Do the macro-expressions of the same person help with recognizing his/her own micro-expressions?" Since MMEW contains the macro-and micro-expressions of the same participants, we also study this problem in a subject-dependent way. Note that although CAS(ME) 2 also contains the macroand micro-expressions of the same participants, the number of micro-expression samples is small (only 57), so that CAS(ME) 2 is not suitable for studying this problem.

Accordingly, by utilizing MMEW, we conducted a preliminary experiment to evaluate different combinations of macro-and micro-expressions. (1) Subject-independent evaluation: The final average recognition rate was 69.4%. (2) Subject-dependent evaluation: All the micro-expression samples in MMEW were randomly divided into five folds, and for each fold, all macro-expressions in MMEW were used for pre-training. Then five-fold cross validation was also used for performance evaluation. The final average recognition rate was increased to 87.2%. These results demonstrate that macro-expressions of the same person are more relevant than those of different persons for pre-training the network. Therefore, MMEW can be used to explore new research directions, including subjectindependent and subject-dependent encoding from macroto micro-expressions.