Keywords: humanoid robot, entertainment robot, assistive robotics, emotion recognition I

In this paper, a system to determine the emotion of a group of people via facial expression analysis is proposed for the Waseda Entertainment Robots.

General models and standard methods for emotion definition and recognition are briefly described, as well as strategies for computing the group global emotion, knowing the individual emotions of group members. This work is based on Ekman's extended "Big Six" emotional model, popular in Computer Science and Affective Computing. Emotion recognition via facial expression analysis is performed with a cloud-computing based solution, using Microsoft Azure Cognitive services. First, the performances of both the Face API to detect faces, and Emotion API, to compute emotion via face expression analysis, are tested. After that, a solution to compute the emotion of a group of people has been implemented and its performances compared to human perceptions. This work presents concepts and strategies which can be generalized for applications within the scope of assistive robotics and, more broadly, affective computing, wherever it will be necessary to determine the emotion of a group of people.

interaction [START_REF] Nass | The media equation: How people treat computers, televisions, and new media as real people and places[END_REF]. In fact, modern media devices are treated as social agents, and the rules which people apply to everyday social interaction with other people apply equally well to their interactions with these devices. For this reason, as the next generations of robots are expected to navigate and perform tasks in human-centered environments, participating and supporting humans in daily routines, they are expected to be social robots capable of interacting with humans in a natural way. Thus, social assistive robots should learn not only how to communicate rationally, using a predefined, structured language, but also emotionally, recognizing and responding to subtle and sometimes ambiguous emotional signals.

Most importantly, socially assistive robots will often have to deal with more than one user at once, so they should be able to identify the general emotional context in which multiple interactions are performed, and, if needed, to assign a priority level to each single interaction and to first choose and respond to higher priority ones. However, even if emotional analysis has been a long-studied problem in affective computing for its importance in human-robot interaction, research focused on inferring the emotional state of a single subject only. Analyzing the general emotion of a group of people is an unexplored problem that has many implications, especially for assistive robots [START_REF] Dhall | Finding Happiest Moments in a Social Context[END_REF].

The objective of this research is to verify the feasibility of a system able to recognize the emotions of a group of people and compute the group general emotion, to enable an assistive robot to generate in response tailored emotional feedback.

Group emotion recognition strategies for entertainment robots

Sarah Cosentino 1 § , Estelle I. S. Randria 2 § , Jia-Yeu Lin 1 , Thomas Pellegrini 3 , Salvatore Sessa 1 , Atsuo Takanishi 1,4,5 , Member, IEEE More specifically, in this work: 1) a system to track and analyze simultaneously the emotions of many people has been developed and tested 2) automatic strategies to evaluate the general emotion of a group of people, provided the individual emotions of most people in the group are estimated correctly, have been implemented and compared with human strategies. This paper is organized as follow: in Section II the tools and methods used for the detection and interpretation of emotions are presented, together with the issue of establishing criteria to model the emotion of a group of people; experiments and results are presented in Section III, followed by a discussion in Section IV, and conclusions and future research directions in the same field in Section V.

WAS-4 KOBIAN-RIV

II. MATERIALS AND METHODS

A. Waseda social robot platforms

Entertainment robotics is a subgroup of socially assistive robotics. Entertainment robots are not made for utilitarian use, but for the pleasure and well-being of the user. They can be useful in a wide range of applications, from the educational to the healthcare field [START_REF] Wainer | The role of physical embodiment in human-robot interaction[END_REF], [START_REF] Kory Westlund | Flat vs. Expressive Storytelling: Young Children's Learning and Retention of a Social Robot's Narrative[END_REF]. Entertainment robots must be able to recognize, adapt to and possibly positively influence the emotional state of the user(s). The Waseda Anthropomorphic Saxophonist (WAS) and KOBIAN humanoid robots are both an example of entertainment robotics (Figure 1). WAS-4 is a musical humanoid robot, its dimensions are 30% larger than an average human [START_REF]Anthropometry and biomechanics[END_REF]. KOBIAN-RIV is a purely social humanoid, its dimensions are comparable to an average Japanese female [START_REF] Endo | Development of Whole-Body Emotional Expression Humanoid Robot for ADL-Assistive RT Services[END_REF]. It is currently used as a comedian robot to explore the effects of amusement on the user long-term emotional state [START_REF] Kishi | Development of a Humorous Humanoid Robot Capable of Quick-and-Wide Arm Motion[END_REF]. Both robots must be able to execute and interpret gaze, body movements, and facial expressions to communicate emotionally with their audience.

Most importantly, they should be able to analyze simultaneously the emotions of a group of people, compute the group emotion, and learn how to respond appropriately, with the objective of generating pleasure and increase the well-being of their audience. To be able to do so, the robots must rely on a framework to define and identify emotions and develop psychological strategies for appropriate emotional feedback.

In this section, the definition of emotions on which this research is based, as well as the chosen emotion recognition method are presented, explained, and validated.

B. Emotion definition

As Scherer writes, there is still no generally accepted definition of emotion [START_REF] Scherer | What are emotions? And how can they be measured?[END_REF]. However, it is accepted by now that emotions are states of interrelated, synchronized physiologic and behavioral changes in response to an external or internal stimulus [START_REF] Scherer | Toward a dynamic theory of emotion[END_REF]- [START_REF] Philosophia | Johannalie Susanna Knoetze[END_REF]. Emotions are short-term mental states provoked by a particular stimulus, whilst a pervasive and sustained emotional state is called mood [START_REF] Philosophia | Johannalie Susanna Knoetze[END_REF]. Psychologists have extensively studied emotions and emotional states, and several different models have been devised. These models try to define which emotions can be considered basic, as innate, unique, distinct psychophysiological states, and which ones can instead be considered a mix of two or more basic emotions. Determining basic emotions is a complex task, as deciding the criteria for defining an emotion as basic are difficult to settle [START_REF] Ekman | Are there basic emotions?[END_REF], [START_REF] Ortony | What's basic about basic emotions?[END_REF].

To cite the most important models:

Plutchik considered eight basic emotions: acceptance, anger, anticipation, disgust, joy, fear, sadness, surprise. He elaborated the well-known Plutchik's wheel of emotions [START_REF] Plutchik | Emotion: theory, research, and experience[END_REF];

Ekman first considered six basic emotions, the "Big Six": happiness, sadness, fear, surprise, anger, and disgust. He then expanded his model to seven basic emotions, adding contempt, and eventually to fifteen, with amusement, embarrassment, excitement, guilt, pride in achievement, relief, satisfaction, sensory pleasure and shame [START_REF] Ekman | Basic Emotions[END_REF].

Weiner and Graham considered as basic emotions only sadness and happiness [START_REF] Weiner | An attributional approach to emotional development[END_REF].

James considered as basic emotions fear, grief, love and rage [START_REF] James | What is an emotion?[END_REF].

Psychological frameworks also conceptualize emotion along two dimensions, valence and arousal. Valence describes the extent of pleasure or displeasure, and arousal describes the extent of physiological excitement [START_REF] Plutchik | Emotion: theory, research, and experience[END_REF], [START_REF] Russell | A circumplex model of affect[END_REF]- [START_REF] Markov | Music Genre and Emotion Recognition Using Gaussian Processes[END_REF]. Of the basic seven, including contempt, happiness only has a positive valence (pleasurable state), surprise has none (neutral), while all the other have negative valence (Table I).

C. Emotions recognition system

As emotions are linked to specific psychophysiological states, there are different methods to monitor and measure physiological parameters to estimate the linked emotion. However, the most ecological methods rely on audio and visual information. As WAS-4 and KOBIAN-RIV and in general most of the existing assistive robots are equipped with front cameras to detect and identify users, the proposed system relies on emotion recognition from visual cues. In particular, facial expressions are the most direct and clear visual display of emotions [START_REF] Hjortsjö | Man's face and mimic language[END_REF]. Following this approach, Ekman and Friesen developed the Facial Action Coding System (FACS) [START_REF] Ekman | Facial Action Coding System[END_REF], a system to taxonomize human facial movements, and linked each emotion in Ekman's model to a specific set of facial movements. Most of the successful works on automatic emotion recognition are based on facial expressions analysis and on Ekman's basic or extended emotions models, and so is the proposed system.

Moreover, as the system will be used for recognition of the global emotion of the robot audience, it is expected to analyze simultaneously facial features of several people, and this will involve a high computational load. To solve this problem and easily scale computational power and expressions database, cloud computing has been used. In particular, the system is based on a cloud computing solution using Microsoft Azure, as this cloud computing service already contains two APIs for face detection and emotion detection: Microsoft Cognitive Services -Face API and Emotion API [START_REF]Microsoft Cognitive Services[END_REF]. Face API analyzes an image and returns the number and the position of the faces in the image. Emotion API analyzes a face image and returns coefficients in the range [0-1] for all the seven basic emotions of Ekman's model, plus neutral (no emotion). The higher the coefficient value, the stronger is the corresponding emotional component in the facial expression. The displayed emotion can be estimated using these coefficients, and in the simplest case, the estimated emotion equals the emotion with the highest coefficient.

D. Strategies for defining group emotion and mood

Defining the emotion or the mood of an audience composed by many people is not a simple task and several strategies can be used.

For example, the general emotion of the audience can be defined as the main emotion perceived: the emotion of each single person in the audience can be estimated, and the emotion with the highest occurrence selected. This strategy can lead to ambiguity problems when dealing with large audience where different emotions are present in equal number.

Another possibility, instead, is to define an emotion hierarchy. By arbitrarily prioritizing some emotions over others, when one person displays a higher priority emotion, this is chosen as the audience general emotion. This approach is convenient when addressing specific emotional states, and also to automatically resolve possible ambiguity problems (Table I).

A third possibility is to take into consideration not the emotions per se, but the overall emotional valence, the mood, of the estimated emotional states. This is a simpler strategy that reduces the problem to a three-state choice. As happiness is the only emotion with positive valence in Ekman's basic model, this is also a useful simplified strategy if the ultimate objective is to identify happiness, or a positive mood.

An important point is to check how these strategies compare with human perception of a group emotion. It is reasonable to expect that, to appear as natural as possible, a social assistive robot should respond to a group in in the same way in which a human would do in the same context. However, human strategies to determine group emotion, given individual emotions in the group are identified, are unknown, and little to no literature is available. To have an idea on how to model, and possibly reproduce, human strategies to determine group emotion, human perception of group emotion must be tested and analyzed.

III. EXPERIMENTS AND RESULTS

A. Face detection

The first step to examine the feasibility of our system is to test the performances of the system in tracking and analyzing simultaneously the emotions of many people. The performances of Face API and Emotion API have been tested separately. The performances in face detection of Face API are strongly influenced by the orientation of the face compared to camera. Orientation of an object in front of the camera is expressed by tilt (angle around the horizontal axis) roll (angle around the frontal axis) and pan (angle around the vertical axis) as shown in Figure 2. As the audience might move during the robot performance, pan might vary accordingly, and being the cameras fixed on the robot, tilt will result when the height of camera and the person face are mismatched. To test performances depending on tilt and pan variations an existing facial recognition database has been used, the Head Pose Image database [START_REF] Gourier | Estimating face orientation from robust detection of salient facial structures[END_REF]. This database is a benchmark of 2790 face images of 15 persons with variations of pan and tilt angles from -90 to +90 degrees. Specifically, variations are in the ranges:

• Tilt = {-90, -60, -30, -15, 0, +15, +30, +60, +90}

• Pan = {-90, -75, -60, -45, -30, -15, 0, +15, +30, +45, +60, +75, +90}

Results showed that face detection is over 90% within the range [-30°, +30°] of both tilt and pan (Figure 3). It can be observed that tilt is more critical for recognition than pan, so the camera should have adjustable height for a more reliable recognition. In fact, if there is no tilt, face detection rate is a steady 100% in the pan range [-45°, +45°].

The API has also been tested with a sample of 50 pictures of frontal faces covered with various accessories, manually excerpted from the web. It has been found that accessories that cover part of the face like hat, sunglasses, or scarfs, strongly influence face detection, with extremely variable results and a recognition rate under 30%. Accessories that cover the lower part of the face are critical and do not allow the face detection with this API.

B. Emotion recognition

The performances of the Emotion API have been tested with the Karolinska Directed Emotional Faces (KDEF) database [START_REF] Lundqvist | The Karolinska directed emotional faces (KDEF)[END_REF]. This database consists of 4900 pictures of human facial expressions of emotion with variations of pan angles. The database contains two sets of images of 70 actors, displaying the 6 basic emotional expressions, plus neutral -contempt is excluded-photographed from 5 different angles: • Pan = {-90, -45, 0, +45, +90}

• Emotions = {anger,disgust,fear,happiness,neutral,sadness,surprise} Results show that pan angles have also an influence on recognition rate of emotions, decreasing the correct recognition rate of a few percentage points (Figure 4).

In Figure 5 the emotional recognition confusion matrix for all frontal images from KDEF database is shown. It is clear that the Emotion API is finely tuned to recognized neutral, surprise and happiness expressions, thus neutral and positive emotional valence, but weak in differentiating basic emotions with a negative valence, especially fear, which is either misclassified as surprise, sadness, or even neutral. This should be corrected, as fear is not an uncommon emotion evoked by human-sized humanoid robots, and robots must be able to recognize and appropriately respond to it.

C. Group emotion evaluation

To simulate multiple emotion recognition from facial expression situation, 6 frontal pictures extracted randomly from the KDEF have been combined (Figure 6). 25 images containing 6 different emotion faces have then been created with this method and used as sample to test the performances of the automatic emotion recognition system against human perception.

An online survey has been created and distributed to reach as many people as possible and to ensure diversity in the subjects' pool. In total, 25 people of age in the range of 20-40 y.o., 45.5% female-to-male ratio, from several countries (mostly Asian and European), and of different education backgrounds and levels, participated in the survey. Results of the survey can be seen in Figure 7, compared with the results of automatic group emotion estimation performed by the system on the same images. As could be expected, very few questions led to a unanimous 100% response on a single emotion.

Using Emotion API there are two methods of computing the global group emotion:

1) for each face, the main emotion can be computed from the highest emotional coefficient, so that 6 emotions can be determined first, then the group emotion is equaled to the emotion with the highest occurrence in the group

2) the relative emotional coefficients for all faces can be summed up for each emotion, obtaining a set of global emotional coefficients for the group, and the group image main emotion can be defined as the emotion with highest global coefficient PAN TILT .

- Method 1) can easily lead to ambiguity problems when different emotions are present in equal occurrence, whilst method 2) is more reliable and can provide also a set of global emotional coefficients to use for more advanced recognition techniques, so this is the computational method used in the proposed system. Results are shown in Figure 7: human responses for each group image are shown in percentages, whilst for automatic recognition using Emotion API, for each group image the computed global coefficients are shown. To be noted, as contempt is not represented in KDEF, survey answers did not include contempt, but Emotion API computes the contempt coefficients anyway, obtaining as expected very low values. In the table, the computed contempt coefficients are in light grey. Highest responses in both situations are shaded in dark grey, second higher values in light grey.

Matching responses in automatic and human perception are circled. The automatic system recognition matches the majority of human responses for each group on 17 cases out of 25, with a majority matching rate of 68%.

IV. DISCUSSION

Observing the results, several considerations can be done.

In the real-time multiple faces detection stage, the main limitation is to track faces with tilt and pan in a certain range. Detection errors could be reduced placing the camera in a position allowing filming the most people possible regardless of their height and position with respect to the robot. For a tall robot this could be implemented placing multiple cameras at different heights, having the robot move around often to check faces in its surroundings, or with a movable camera. To have information about tilt and pan during recognition would also provide useful, as this could be passed on to the second stage of emotional recognition and provide information on possible emotion recognition errors. Emotional coefficient could be then weighted differently to adjust for tilt and pan induced errors.

The performances of the emotional recognition stage should be improved, especially toward the recognition rate of emotions with negative valence, anger, disgust and fear. The automatic system failed to recognize disgust and anger in nearly all the cases, except for group 18, composed by 1 happy, 3 angry, and 2 disgusted expressions. Coefficient normalization could be implemented to reduce the bias of the Emotion API towards happiness and neutral.

An important point of this research is that, as it is shown in Figure 7, a simple computational approach to determine the global emotion of a group cannot be used. In the presence of different emotional cues from different people, some subjects are more sensitive to positive emotions; some subjects are more sensitive to negative emotions. Moreover, depending on the actual situation context, it is probably appropriate to establish an emotional hierarchy, as theorized in Section II, to respond quickly to some contextually more important emotional cues. For example, in the case of the robot in a public setting, it could be appropriate to respond to anger or fear cues by receding and rolling away, to preserve the robot integrity or to avoid distress to the interacting human subject. Also, if the social assistive robot is expected to react to a group in a similar way a human would, in the same context, the robot should be able to approximate the human global group emotion perception, with a certain degree of uncertainty, and to be able to respond appropriately to mixed emotional states. Emotional responses should be tailored on both the group mood, positive or negative, and individual emotional cues, to avoid degrading the group mood and at the same time improving individual emotional states.

V. CONCLUSION

This work is a preliminary study on recognition of group emotions for assistive social robots. Basic theoretical concepts and design choices have been explained and analyzed, and an automatic computational approach has been compared with human perception. In the future, more human perception data should be acquired to train a machine learning system, to develop a group emotion assessment paradigm similar to the one humans use, and teach the robot to respond appropriately to mixed group emotions, enhancing the decision power of the robot and its ability to entertain and engage its users. However, before implementing a magic black box emotional response tool, it is important to analyze in deep the mechanisms of human group emotional interaction, and to identify and contextualize the key strategies humans use during emotional interactions, to devise a method for qualitative and quantitative assessment of the automatic recognition system and identify and correct its possible biases.

Figure 1

 1 Figure 1 Waseda Humanoid Entertainment Robots

Figure 3 Figure 4

 34 Figure 3 Face recognition rate for tilt and pan combinations [%]

Figure 5

 5 Figure 5 Emotion recognition confusion matrix [%]Figure6KDEF six emotional faces image sample (survey question 1)

  Figure 7 six emotional faces perception

TABLE I .

 I VALENCE, EMOTIONS, AND AN EXAMPLE OF ARBITRARILY DEFINED EMOTION HIERARCHY (1: HIGH)

	Valence	Emotion	Priority
	Negative	Sadness	1
	Negative	Anger	2
	Negative	Fear	3
	Negative	Disgust	4
	Negative	Contempt	5
	Positive	Happiness	6
	None	Surprise	7
	None	Neutral	8

Figure 2 Tilt, Roll and Pan angles

ACKNOWLEDGMENT

The authors would like to thank the Tokyo Women Medical University and Waseda University Joint Institution for Advanced Biomedical Sciences (TWIns) and the International Center for Science and Engineering Program (ICSEP) of Waseda University.

*Research supported by the Waseda University 2016 Grant-in-Aid for particular research subjects [2016S-081] and by the JSPS Grant-in-Aid for Young Scientists (Wakate B) [17K18178]; by ST Microelectronics, and Microsoft.