Jing Han
email: jing.han@uni-passau.de

Zixing Zhang

Fabien Ringeval

Björn Schuller

PREDICTION-BASED LEARNING FOR CONTINUOUS EMOTION RECOGNITION IN SPEECH

Keywords: Affective computing, hierarchical regression models, support vector regression, long short-term memory

In this paper, a prediction-based learning framework is proposed for a continuous prediction task of emotion recognition from speech, which is one of the key components of affective computing in multimedia. The main goal of this framework is to utmost exploit the individual advantages of different regression models cooperatively. To this end, we take two widely used regression models for example, i. e., support vector regression and bidirectional long short-term memory recurrent neural network. We concatenate the two models in a tandem structure by different ways, forming a united cascaded framework. The outputs predicted by the former model are combined together with the original features as the input of the following model for final predictions. The experimental results on a time-and value-continuous spontaneous emotion database (RECOLA) show that, the prediction-based learning framework significantly outperforms the individual models for both arousal and valence dimensions, and provides significantly better results in comparison to other state-of-the-art methodologies on this corpus.

INTRODUCTION

In recent years, increasing efforts in the affective computing community have been paid on the automatic and continuous emotion prediction of humans' spontaneous behaviour [START_REF] Gunes | Automatic, dimensional and continuous emotion recognition[END_REF][START_REF] Petridis | Prediction-based audiovisual fusion for classification of non-linguistic vocalisations[END_REF][START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF]. For such a regression problem, a variety of regression models have been proposed and investigated, such as Support Vector Regression (SVR) [START_REF] Drucker | Support vector regression machines[END_REF], Relevance Vector Regression (RVR) [START_REF] Tipping | Sparse bayesian learning and the relevance vector machine[END_REF], Feedforward Neural Networks (FNNs) [START_REF] Kwok | Constructive algorithms for structure learning in feedforward neural networks for regression problems[END_REF], Deep Belief Networks (DBNs) [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF], and Recurrent Neural Networks (RNNs) [START_REF] Williams | A learning algorithm for continually running fully recurrent neural networks[END_REF]. To better choose the regressors, distinct advantages of each model have been compared in the literature. For instance, the work in [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF] has compared the performance of SVR and Bidirectional Long Short-Term RNNs (BLSTM-RNNs) for the continuous prediction of arousal and valence on the Sensitive Artificial Listener database, and the results indicate that the latter performed better when using 15 acoustic Low-Level-Descriptors (LLDs) as the feature set. Whereas, an opposite conclusion has been drawn in the work of [START_REF] Tian | Emotion recognition in spontaneous and acted dialogues[END_REF], where SVR was superior to the BLSTM-RNN on the same database when using a statistical feature set that applied functionals over a large ensemble of LLDs. Other results in the literature confirm this inconsistent performance observation between SVR and diverse neural networks like (B)LSTM-RNNs and Feedforward Neural Networks (FNNs) [START_REF] Ringeval | AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data[END_REF]. A reasonable explanation behind this could be that each prediction model has its pros and cons.

For example, BLSTM-RNNs are highly sensitive to overfitting, but SVR cannot explicitly model contextual dependencies.

In order to integrate the advantages of different regression models, some ensemble-based approaches have been introduced to further improve the continuous emotion prediction. The research in [START_REF] Wei | Multimodal continuous affect recognition based on LSTM and multiple kernel learning[END_REF] has presented a stacked system, where LSTM-RNNs are first employed to predict the emotions in multi-dimensional and multimodality spaces. Then, a Multiple Kernel SVR (MK-SVR) is used to correlate the generated multiple predictions to make a final decision. This method takes the benefits of LSTM-RNNs for modelling contextual dependencies, and MK-SVR for modelling the non-linear correlations between inputs and outputs. Furthermore, another similar structure has been introduced in [START_REF] Qiu | Ensemble deep learning for regression and time series forecasting[END_REF], where predictions obtained by 20 variants of DBN topology structures are then aggregated by a SVR model. However, due to the similarity of characteristics of different DBNs, the predictions can not provide many variations that could be mutually complemented and improve the system performance. Besides, 20 DBNs are rather complex to be applied in the real world.

To further efficiently aggregate the advantages of different models, we put forward a simply constructed method. Different from the methods used in [START_REF] Qiu | Ensemble deep learning for regression and time series forecasting[END_REF] and [START_REF] Wei | Multimodal continuous affect recognition based on LSTM and multiple kernel learning[END_REF], we feed the predictions of the model in the first stage together with the original raw features into the model in the second stage. In doing this, only one raw feature set and two recognition models are required. The underlying idea is also in line with the expectation in the human community that one's suggestion can often ameliorate the others' judgement. Moreover, the idea is also inspired by a tandem structure for Automatic Speech Recognition (ASR) [START_REF] Wöllmer | Noise robust ASR in reverberated multisource environments applying convolutive NMF and Long Short-Term Memory[END_REF], where the phoneme predicted by neural networks is considered as an additional attribute for a Gaussian Mixture Model (GMM). In the present paper, we develop an prediction-based learning framework for the regression task of emotion recognition in speech.

In the remainder of this paper, we first briefly introduce the related work in Section 2. We then introduce the predictionbased learning method to exploit the predictive ability of SVR and BLSTM-RNN in Section 3. Afterwards, in Section 4, we implement and evaluate the system on the RECOLA dataset. Finally, we conclude our work and give an outlook in Section 5.

RELATED WORK

Our prediction-based method is related to the Output Associative Relevance Vector Machine (OA-RVM) regression framework originally proposed in [START_REF] Nicolaou | Output-associative rvm regression for dimensional and continuous emotion prediction[END_REF]. The OA-RVM framework attempts to incorporate the contextual relationships that exist within and between dif-ferent affective dimensions and various multimodal feature spaces, by training a secondary RVM with an initial set of multi-dimensional output predictions (learnt using any prediction scheme) concatenated with the original input features spaces. Additionally, the RVM framework also attempts to capture the temporal dynamics by employing a sliding window that incorporates both past and future initial outputs into the new feature space. Results presented in [START_REF] Huang | An investigation of annotation delay compensation and output-associative fusion for multimodal continuous emotion prediction[END_REF] also indicate that the OA-RVM framework, is well suited to emotion recognition problem.

The OA-RVM systems, like our proposed method, take input features and output predictions into consideration to train a subsequent regression model to perform the final affective predictions. The strength of the OA-RVM framework is that it is underpinned by the RVM. However, the results in [START_REF] Huang | An investigation of annotation delay compensation and output-associative fusion for multimodal continuous emotion prediction[END_REF] indicate that the framework is not as successful as it is when using either an SVR or a simple linear regressor as the secondary model. Further, the OA-RVM is non-casual and requires careful tuning to find a suitable window size so as to efficiently combine the initial outputs. This takes considerable time and effort. The proposed prediction-based learning framework, however, is designed to work with any combinations of learning paradigms, which aims to use the initial set of predictions to help improve the accuracy of any subsequent model. Furthermore, our method is simple constructed. It combines the original input features and the initial predictions frame by frame, which holds a strong advantage over the OA-RVM especially in the real-time applications.

PREDICTION-BASED LEARNING

Overview

The prediction-based learning framework for emotion recognition is depicted in Fig. 1. Specifically, in the training phase, the first regression model (M odel1) provides an initial prediction ŷt based on the original acoustic feature vector xt. Then, the initial prediction ŷt is concatenated with original feature vector xt as the input [xt, ŷt] of the second regression model (M odel2) for a final emotion prediction. An alternative strategy to train the second model is to employ a 'pseudo' prediction ỹt of model1, which is simulated by applying noise to the true label. In this case, the combined feature set [xt, ỹt] is considered as the input of M odel2. In the evaluation phase, nevertheless, only the true initial prediction ŷt is jointed with original feature vector xt, i. e., [xt, ŷt], is employed as the input of M odel2 for a final emotion prediction. The motivation of using simulated prediction is that there is no need for us to know the type of model1 beforehand. Therefore, it offers us a benefit that the model2 can be simply and directly integrated with any pre-existing systems no matter what regression models they used (i. e., a plug and play). In this case, the prediction from the unknown model can be fused with the original feature set as the input of M odel2 for a final prediction in order to further improve the system performance.

In this preliminary study, we only take two widely used regression models, i.e., SVR and BLSTM-RNN, into consideration to evaluate the effectiveness of the proposed framework. In the rest of this section, the selected two models and the aforementioned two training schemes will be described in detail.

Regression Models

SVR is an extended version Support Vector Machine (SVM) to solve regression problems. It was first introduced in [START_REF] Drucker | Support vector regression machines[END_REF] and has become one of the most dominant methods in the context of machine learning, particularly for continuous emotion recognition [START_REF] Ringeval | AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data[END_REF].

Applying SVR for a regression task, the target is to optimise the generalisation bounds for regression by a loss function which is used to measure the cost of the errors of the prediction. Moreover, a predefined parameter C is set accordingly for different cases to balance emphasis on the errors and the generalisation performance. Usually, a (non-)linear kernel function is learned by the model to map the raw features into a higher mapped feature space. In our study, we use the SVR with a linear kernel function, as the features in the original feature space perform well for emotion prediction in our case. One of the most important advantages of SVR is that the global optimal solution is guaranteed owing to the characteristics of the convex optimisation function. Besides, SVR is learned by minimising an upper bound on the expected risk, as opposed to the neural networks trained by minimising the errors on all training data, which equips SVM a superior ability to generalise [START_REF] Gunn | Support vector machines for classification and regression[END_REF]. For more details about the SVR model, please refer to [START_REF] Drucker | Support vector regression machines[END_REF].

Another model utilised in our study is BLSTM-RNN which has been successfully applied to continuous emotion prediction [START_REF] Ringeval | AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data[END_REF] as well as for other regression tasks, such as non-linguistic vocalisations classification [START_REF] Petridis | Prediction-based audiovisual fusion for classification of non-linguistic vocalisations[END_REF]. In general, it is composed of one input layer, one or multiple hidden layers, and one output layer [START_REF] Hochreiter | Long short-term memory[END_REF]. The bidirectional hidden layers separately scan the input sequences in a forward and a backward order separately and connect to the same output layer. Compared with traditional RNNs, it introduces recurrently connected memory blocks to replace the network neurons in the hidden layers. Each block consists of a self-connected memory cell and three gate units, namely input, output, and forget gate. These three gates allow the network to learn when to write, read, or reset the value in the memory cell. Such a structure grants BLSTM-RNN to learn past and future context in both short and long range. For a more in-depth explanation of BLSTM-RNNs the reader is referred to [START_REF] Hochreiter | Long short-term memory[END_REF].

Hierarchical System

In this study, we chose the models of SVR and BLSTM-RNN, the advantages and drawbacks of which are as follows

• the SVR model is more likely to achieve the globally optimal solution, but it is not context-sensitive [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF]; • the BLSTM-RNN model is easily trapped in a local minimum which can be hardly avoided and has a risk of over-fitting [START_REF] Graves | Framewise phoneme classification with bidirectional lstm and other neural network architectures[END_REF], while it is good at capturing the correlation between the past and the future context [START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF]. Thus, by integrating the predictions from each of these two models, it is expected to overcome the limitations of each model whilst preserving the advantages of each. In other words, it aims to effectively make use of the advantages of the first model to complement the disadvantages of the second model.

As shown in Fig. 1, M odel1 and M odel2 in the framework could be either a SVR model or a BLSTM-RNN model, resulting in four possible permutations, i.e., SVR-SVR, SVR-RNN, RNN-SVR, RNN-RNN. Particularly, it has to be noticed that the structure of RNN-RNN is regarded as a deep variation of neural networks. In addition, the structure of SVR-SVR is not considered since the training is done by solving a large margin separator, which will be likely the same if only adding its own prediction as one more attribute.

To train such a prediction-based learning system, the two models could be trained either dependently or independently, resulting in two training schemes as described below.

Training with True Predictions

When two models are trained dependently, M odel2 takes the predictive ability of M odel1 into account for training. The detailed procedure is given as follows:

-First, train M odel1 with the original feature set xt to make an initial prediction ŷt.

-Then, train M odel2 with the combined feature set [xt, ŷt] to learn the expected prediction yt.

Training with Pseudo Predictions

M odel1 and M odel2 can also be trained independently, which means that training process of model2 does not includes any information of model1. In doing this, we use a pseudo prediction that is synthesised by adding noise to disturb the truth label yt (gold standard or ground truth) to server as the additional attribute (denoted as ỹt). Details about the training process are as follows:

-First, apply noise to the gold standard yt to generate pseudo prediction ỹt.

-Second, train M odel2 with the combined feature set [xt, ỹt] to learn the expected prediction yt.

EXPERIMENTS

Data and Features

We evaluated the proposed method on the RECOLA corpus [START_REF] Ringeval | Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions[END_REF], which contains fully spontaneous and natural affective behaviours. It includes 46 multimodal (audio, video, and physiological data) recordings of French speaking participants involved in a dyadic collaborative task. Affective behaviour of the participants was evaluated by six different annotators (3 females), for the first five minutes of each recording, i.e., all annotators consistently annotated all recordings. Annotation was performed for arousal and valence separately. The obtained labels were then resampled to a constant 40 ms frame rate, and averaged over all raters by considering interevaluator agreement, to provide a 'gold standard' [START_REF] Ringeval | Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data[END_REF]. In order to ensure speaker-independence in the experiments, the corpus was split into three partitions, by balancing the gender and the age of the subjects (cf. Table 1): training (16 subjects), validation (15 subjects), and testing (15 subjects). It is worth mentioning that, we used exactly the same partitions as in [START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? Endto-end speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF] for comparison.

To extract acoustic features from the speech recordings, we used the openSMILE toolkit [START_REF] Eyben | openSMILE -the Munich versatile and fast open-source audio feature extractor[END_REF] to extract 13 LLDs, i.e., 12 Mel-Frequency Cepstral Coefficients, and 1 log energy with a frame window size of 25 ms at a step size of 10 ms. Then, the arithmetic mean and the coefficient of variance were computed over the sequential LLDs within an analysed window of 8 s, resulting in 26 statistical features per window. The analysed window moves forward at the

Implementation and Evaluation

For the SVR models, we chose a linear kernel function with optimised C in [.00001, .00002, .00005, .0001, . . . , .2, .5, 1]. For the BLSTM-RNN regression models, we adopted two hidden layers. In our experiments, the BLSTM-RNN model consists of 20 nodes per hidden layers with a learning rate of 10 -5 and a momentum of 0.90. Moreover, zero mean Gaussian noise with a standard deviation of 0.2 was added to the input activations in the training phase to improve generalisation. All weights were randomly initialised in the range from -0.1 to 0.1. The parameters of each model were optimised on the validation set with an early stopping strategy for its corresponding task. To implement the SVR and BLSTM-RNN models, we selected LIBLINEAR [START_REF] Fan | LIBLINEAR: A library for large linear classification[END_REF] and CURRENNT [START_REF] Weninger | Introducing CUR-RENNT: The munich open-source cuda recurrent neural network toolkit[END_REF] toolkits, respectively, for the sake of reproducibility.

For the baseline, we performed the emotion evaluation on the SVR model and the BLSTM-RNN model independently. Further, to better compare performance between the prediction-based RNN-RNN framework and the traditional deeper RNN framework, we further provided another baseline BLSTM-RNN model with four hidden layers. Whereas, for the proposed methods, two models were successively combined in order to enhance the capability of prediction as demonstrated in Fig. 1. Moreover, as described in Section 3.3, the M odel2 in the proposed framework can be trained either with true or pseudo predictions. To simulate pseudo predictions, white Gaussian noise was added to the gold standard with different signalto-noise ratios, i.e., 0, 3, 6, 9, and 12 dB.

It should be noted that, all feature sets in the training set were standardised to zero mean and unit variance before training M odel2, except the initial true predictions or the pseudo predictions. We carried out an on-line standardisation over the validation and the test set, using the means and variances from the corresponding training sets.

Additionally, annotation delay compensation was also performed to compensate for the temporal delay between the observable cues, as shown by the participants, and the corresponding emotion reported by the annotators [START_REF] Mariooryad | Correcting time-continuous emotional labels by modeling the reaction lag of evaluators[END_REF]. As in [START_REF] Huang | An investigation of annotation delay compensation and output-associative fusion for multimodal continuous emotion prediction[END_REF] we identified this delay to be four seconds which was duly compensated, by shifting the gold standard back in time with respect to the features for both arousal and valence in all of our experiments.

To evaluate the agreement level between the predictions of emotion and the gold standard, the standard metric is the Concordance Correlation Coefficient (CCC) [START_REF] Ringeval | AV+EC 2015: The first affect recognition challenge bridging across audio, video, and physiological data[END_REF], which is calculated by , where x and y are two series. In contrast to the largely used Pearson's correlation coefficient, CCC takes also the bias, i.e., (µx -µy) 2 , into account. Hence, the value of CCC is in range of [-1, 1], where ±1 represents perfect concordance and discordance while 0 means no correlation.

ρc = 2σ 2 xy σ 2 x + σ 2 y + (µx -µy) 2 , (1)
To further access the significance level of performance improvement, a statistical evaluation was carried out over the whole predictions between the proposed and the baseline methods by means of Fisher's r-to-z transformation [START_REF] Cohen | Applied multiple regression/correlation analysis for the behavioral sciences[END_REF]. Unless stated otherwise, a p value less than .05 indicates significance.

Results and Discussion

Table 2 demonstrates the performance of our proposed methods by applying three different hierarchical structures (i.e., RNN-SVR, SVR-RNN, and RNN-RNN) and two training schemes as mentioned in Section 3.3, evaluating on the validation and the test sets for both the arousal and valence dimensions in terms of CCC. The parameters of the complexity value of SVR and the SNR of the simulated predictions were optimised on the validation set, which were then applied for the test set.

Generally speaking, the proposed methods outperform the corresponding baselines in most cases on the validation set for both arousal and valence. Regarding the test set, the proposed methods performed better than the baseline in some cases for arousal and in all cases for valence. Moreover, we could observe that the improvement on valence is much more than that on arousal. This may be due to the fact that, the models have already fit well for the arousal with the original feature vector.

Furthermore, comparing RNN-SVR and SVR-RNN, one may observe that, the performance of the latter is on average higher than the former. This is possibly because SVR could help the following RNN to avoid the local minima by using its prediction. Whereas, the local minimum problem of the RNN in the RNN-SVR framework could not be eased by the SVR model followed it. This might imply that the order of the models in the hierarchical structure matters and the order should be chosen with care. Additionally, when comparing the two different training strategies, the pseudo predictions performs competitive to or even slightly better than the true predictions, whilst it does not require any prior knowledge of the first model.

Comparing the cascaded RNN-RNN structure and the deeper RNN with four layers, one may observe that, simply adding more hidden layers to make the RNN deeper, does not lead to the same improvement as the cascaded RNN-RNN in which the output predictions of the first shallow RNN model jointed with other features pairwise are fed into a second shallow RNN model. A rationale behind this is that the gradient of the neural network is supposed to be vanished when simply increasing the hidden layers. The error information can not efficiently back-propagate to the low layers near the visual input layer. Meanwhile, the data information at the low layers can also not be accessed well by the high layers that are far from the visual layer [START_REF] Hochreiter | Long short-term memory[END_REF]. By our method, however, the data information can be explicitly transferred to the hidden layers in the later RNNs by means of the predictions from the former RNNs.

Particularly, the best performance on the test set is obtained at .744 of CCC for arousal when using RNN-RNN trained with pseudo predictions, and .393 of CCC for valence by SVR-RNN trained with the original SVR predictions. These results are also much better than the ones achieved by using a novel discriminative training approach based on CCC [START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF], and the ones by using novel features extracted from the raw waveform by performing convolutional neural networks [START_REF] Trigeorgis | Adieu features? Endto-end speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF], respectively.

CONCLUSION

Various regression models have been successfully applied independently or associatively for different tasks, particularly in continuous affective computing. In this paper, we presented a prediction-based learning framework which associates different regression models to leverage the performance of continuous prediction of emotion. The proposed method takes advantages of multiple models, such as the context-sensitive capability of memory-enhanced neural networks, and the global-optimisation capability of Support Vector Regression. By implementing the experiments on a spontaneous database, significant improvement of performance indicates the effectiveness of our methods.

Although the proposed framework aims to exploit the advantages of various models, it is sometimes inevitable to take the disadvantages, which may results in a negative effect to the framework. In future, more efforts will be paid to address this problem. Further, the proposed framework will be evaluated by other applications (e. g., text and video) and other regression models.

Fig. 1 .

 1 Fig. 1. Prediction-based learning framework

Table 1 .

 1 Three partitions of the RECOLA database. ms such that it can match the granularity of the annotation. The total numbers of the segments in the train, development and test partitions were 120 000, 112 500, and 112 500, respectively.

	#	train	development	test
	female	10	9	8
	male	6	6	7
	French	11	11	11
	Italian	3	2	3
	German	2	1	1
	Portuguese	0	1	0
	age µ (σ)	22.3 (3.4)	21.6 (2.1)	21.2 (2.0)
	rate of 40

Table 2 .

 2 Concordance Correlation Coefficient (CCC) of the three different hierarchical structures (i.e., RNN-SVR, SVR-RNN, or RNN-RNN) on the validation (val.) and test sets for both arousal and valence regression. For training the second model, either the true predictions from the first model or the simulated pseudo predictions were used. Baselines and other state-of-the-art results on RECOLA are also presented. The best achieved CCCs are highlighted. The symbol of * indicates the significance of the performance improvement.

	CCC	arousal	valence
		val.	test	val.	test
	Baseline
	SVR	0.744	0.726	0.403	0.300
	RNN (2 layers)	0.752	0.738	0.329	0.278
	RNN (4 layers)	0.747	0.708	0.401	0.305
	Models trained with true predictions
	RNN-SVR	0.744	0.726	0.422 * 0.387 *
	SVR-RNN	0.769 * 0.730	0.440 * 0.393 *
	RNN-RNN	0.757 * 0.726	0.418 * 0.369 *
	Models trained with pseudo predictions
	RNN-SVR	0.746	0.729	0.407	0.301
	SVR-RNN	0.774 * 0.743 * 0.420 * 0.373 *
	RNN-RNN	0.774 * 0.744 * 0.412 * 0.377 *
	State-of-the-art
	CCC-objected [3] 0.412	0.350	0.242	0.199
	End-to-End [22]	0.741	0.686	0.325	0.261
	with µx = E(x), µy = E(y), σ 2 x = var(x), σ 2 y = var(y) and σ 2 xy = cov(xy)

ACKNOWLEDGEMENTS

This work was partially supported by the EC's 7th Framework Programme through the ERC Starting Grant No. 338164 (iHEARu), the EU's Horizon 2020 Programme through the Innovative Action No. 645094 (SEWA) and the Research Innovative Action No. 645378 (ARIA-VALUSPA), and by the German Federal Ministry of Education, Science, Research and Technology (BMBF) under grant agreement No. 16SV7213 (EmotAsS).


Jing Han
email: jing.han@uni-passau.de

Zixing Zhang

Fabien Ringeval

Björn Schuller

RECONSTRUCTION-ERROR-BASED LEARNING FOR CONTINUOUS EMOTION RECOGNITION IN SPEECH

Keywords: Continuous emotion recognition, reconstruction error, bidirectional long short-term memory

To advance the performance of continuous emotion recognition from speech, we introduce a reconstruction-error-based (RE-based) learning framework with memory-enhanced Recurrent Neural Networks (RNN). In the framework, two successive RNN models are adopted, where the first model is used as an autoencoder for reconstructing the original features, and the second is employed to perform emotion prediction. The RE of the original features is used as a complementary descriptor, which is merged with the original features and fed to the second model. The assumption of this framework is that the system has the ability to learn its 'drawback' which is expressed by the RE. Experimental results on the RECOLA database show that the proposed framework significantly outperforms the baseline systems without any RE information in terms of Concordance Correlation Coefficient (.729 vs .710 for arousal, .360 vs .237 for valence), and also significantly overcomes other state-of-the-art methods.

INTRODUCTION

Automatic continuous emotion recognition in speech is one of the most active research areas in the affective computing community in recent years [START_REF] Ringeval | AVEC 2015: The 5th international audio/visual emotion challenge and workshop[END_REF][START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF]. Many efforts have been reported in order to advance the system performance and further facilitate its applications in the real world [START_REF] Gunes | Automatic, dimensional and continuous emotion recognition[END_REF][START_REF] Gunes | Categorical and dimensional affect analysis in continuous input: Current trends and future directions[END_REF][START_REF] Petridis | Prediction-based audiovisual fusion for classification of non-linguistic vocalisations[END_REF][START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF]. Among these efforts, a memory-enhanced Recurrent Neural Network (RNN), namely Long Short-Term Memory (LSTM) RNN, has attracted considerable attention to address the raised task in the past few years [START_REF] Wöllmer | Abandoning emotion classes-towards continuous emotion recognition with modelling of long-range dependencies[END_REF][START_REF] Wöllmer | LSTM-Modeling of continuous emotions in an audiovisual affect recognition framework[END_REF][START_REF] Chao | Long short term memory recurrent neural network based multimodal dimensional emotion recognition[END_REF][START_REF] Zhang | Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks[END_REF][START_REF] Nicolaou | Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space[END_REF][START_REF] He | Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks[END_REF]. The successful implementation of LSTM-RNN is mainly due to its powerful capability of learning long-range contextual information for sequential patterns [START_REF] Hochreiter | Long short-term memory[END_REF][START_REF] Graves | Supervised sequence labelling with recurrent neural networks[END_REF].

More recently, advanced research has begun to explore the benefits of LSTM in the context of continuous emotion recognition. In [START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF], Weninger et al. proposed a novel discriminative learning method, which exploits the Concordance Correlation Coefficient (CCC), rather than the traditional Root Mean Square Error (RMSE), as a differentiable objective function to train a LSTM-RNN model. In [START_REF] Trigeorgis | Adieu features? End-toend speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF], Trigeorgis et al. utilised data-learnt highlevel representations, instead of the traditional hand-crafted features like Mel-Frequency Cepstral Coefficients (MFCC), as the inputs of LSTM-RNN. The high-level representations are derived from the raw speech signals with the aid of Convolutional Neural Networks (CNN), which are jointly combined with the LSTM-based recognition model to constitute an end-to-end framework.

In this paper, we propose another novel framework based on the Reconstruction Error (RE) for continuous emotion recognition in speech. The underlying assumption is that, the LSTM-RNNs could learn their 'drawbacks' if we provide them a way to know when the inputs are more challenging to process. Specifically, we utilise the RE from one LSTM-RNN model, used as an Auto Encoder (AE), to express the drawbacks of another LSTM-RNN model used for emotion regression. The RE is thus used as a complementary descriptor of the original features, that describes the ability of the LSTM-RNN model to encode the emotion related information.

The idea of the proposed method is partially inspired by the promising attention modelling in the field of natural language processing [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF], where the decoder pays different attention to the regions of the input based on their relevant degree with the output. As to our method, the continuous variation of the RE provides different levels of disturbance when training a model with time-continuous features. In this contribution, we demonstrate by empirical analysis that this approach can help heighten the attention of a model on the error-sensitivity regions, so as to ameliorate the model performance for emotion recognition from speech.

RELATED WORK

RE has been originally used as an objective function of AE for extracting high-level representations. Recently, however, a trend in the machine learning community has emerged towards exploiting directly the RE as descriptors for other tasks, e. g. , it was considered as a novelty measure for novelty detection [START_REF] Marchi | A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional LSTM neural networks[END_REF]. An AE is trained on normal samples beforehand to serve as a novel event detector. When a new sample is fed into the AE, its RE is compared with a predefined threshold to decide whether the current sample is abnormal. When an unknown sample passes through the AEs that are trained class-specifically, the corresponding RE indicates the likelihood to belong to a class.However, to the best of our knowledge, none of the works takes the RE into consideration for augmenting the learning capability of another similar model for recognition.

This work is also relevant to the tandem architectures in the speech recognition domain, which use the state or phoneme posterior probabilities generated by a neural network as features observed by a Hidden Markov Model (HMM). Thanks to the development of neural networks, the topologies have been shifted from the early Multilayer Perceptron (MLP) [START_REF] Hermansky | Tandem connectionist feature extraction for conventional HMM systems[END_REF] to the recent LSTM-RNN [START_REF] Wöllmer | Bidirectional LSTM networks for context-sensitive keyword detection in a cognitive virtual agent framework[END_REF]. For emotion recognition, and to the best of our knowledge, only the tandem structure of LSTM-DBN has been exploited in [START_REF] Wöllmer | Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening[END_REF] so far.

In this work, the advantages of disparate models, i. e., the contextsensitivity capabilities of LSTM, and the generalisation competence of HMM or DBN, have been explored.

Nevertheless, these architectures only take the prediction or the prediction probability as another dimensional representation, which are merely designed for classification tasks rather than regression problems. In this paper, we leverage for the first time the RE information of an LSTM-based AE to perform a time-continuous regression task (emotion) from speech.

RECONSTRUCTION-ERROR-BASED LEARNING

Overview

The framework of our proposed RE-based learning is depicted in Fig. 1 One key question of this framework is how to guarantee that the RE information, that is derived from the first model, can be successfully exploited by the second model. To this end, we assume that the two models have a similar structure such that the RE extracted with the first model can well reflect the drawbacks of the second model. In this paper, we employ Bidirectional LSTM-RNN (BLSTM-RNN) as the basic model because of its great success in continuous recognition of emotion, cf. Section 1.

In general, the BLSTM-RNN structure is composed of one input layer, one or multiple hidden layers, and one output layer [START_REF] Hochreiter | Long short-term memory[END_REF]. The bidirectional hidden layers separately process the input sequences in a forward and a backward order and connect them to the same output layer. Compared with conventional RNNs, it adopts LSTM blocks to replace the neurons in the hidden layers. Each block consists of a self-connected memory cell and three gate units, namely input, output, and forget gate. These three gates allow the network to learn when to write, read, or reset the value in the memory cell. Such a structure grants BLSTM-RNN to learn past and future context in both short and long range. For a more in-depth explanation of BLSTM-RNNs the reader is referred to [START_REF] Hochreiter | Long short-term memory[END_REF].

Extraction Stage

To extract the RE information, the BLSTM-RNN model is trained in a completely unsupervised way. That is, the inputs and the targets are exactly the same. If the model would be sufficiently powerful, one would expect that all inputs should be recovered. However, many empirical experiments have shown that the results are far from this expectation [START_REF] Zhang | Facing realism in spontaneous emotion recognition from speech: Feature enhancement by autoencoder with LSTM neural networks[END_REF], which somewhat implies that BLSTM-RNN has its own drawbacks, just like any other machine learning technique.

Specifically, given a time sequence as input, the BLSTM-RNN is trained to minimise the cost function as

J (θ) = T t=1 (xt -xt) 2 , ( 1
)
where xt is a sample at time t from an input sequence lasting a period of time T , and xt denotes the reconstructed sample of the corresponding input xt.

Once the first BLSTM-RNN model is trained, the RE t can be obtained by computing the sum of the Euclidean distance between the input xt and its corresponding reconstruction xt over all L dimensions, as expressed in Eq. ( 2). The result, t, is therefore a scalar.

t = L l=1 |x t,l -x t,l |, t ∈ T.
(2)

Exploitation Stage

In this stage, it is expected that the features with their respective RE values will attract different attention from the regression model. After the RE information t is generated, it is simply concatenated with the original features xt to form a longer feature vector [xt, t], which thus incorporates information regarding reconstruction issues obtained with the first model. The conventional machine learning paradigms can be then applied sequentially.

It is worth noting that the features xt used for training M odel1 in the extraction stage could be either frame-based Low-Level Descriptors (LLD), or the segment-based statistical features -after applying functionals over the LLD -, which may result in a difference in the final feature dimensions used for the exploitation stage. More details on these settings are explained in Section 4.2.

EXPERIMENTS AND RESULTS

In this section, we present a set of experiments on a time-and valuecontinuous dimensional emotion (arousal and valence) prediction task. The objective of the experiments is to empirically demonstrate the benefits of adding RE information to the conventional features for the proposed task.

Selected Database and Features

We evaluate our RE-based learning method on the RECOLA [START_REF] Ringeval | Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions[END_REF] database, which has been adopted for the AudioVisual Emotion recognition Challenges (AVEC) in 2015 [START_REF] Ringeval | AVEC 2015: The 5th international audio/visual emotion challenge and workshop[END_REF] and 2016 [START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF]. This database contains spontaneous and natural interactions between remote participants that solved a task in dyads. The whole dataset contains recordings from 46 different subjects. The dataset is further divided into three disjoint partitions while balancing the gender, age, and mother tongue of the participants (cf. Table 1). It is worth mentioning that, we used exactly the same partitions as in [START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? End-toend speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF].

To annotate the corpus, time-and value-continuous dimensional affect ratings in terms of arousal and valence were performed by six French-speaking raters (three females) for the first five minutes of all recorded sequences. The obtained labels were then resampled at a constant frame rate of 40 ms, and averaged over all raters by considering inter-evaluator agreement, to provide a 'gold standard' [START_REF] Ringeval | Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions[END_REF]. Measures of inter-rater agreement show the reliability of the annotation data and the used post-processing techniques [START_REF] Ringeval | AVEC 2015: The 5th international audio/visual emotion challenge and workshop[END_REF][START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF]. To extract acoustic features from the speech recordings, we used our open-source openSMILE toolkit [START_REF] Eyben | openSMILE -the Munich versatile and fast open-source audio feature extractor[END_REF] to extract 13 LLDs, i. e., MFCC 0-12 and logarithmic energy, with a frame window size of 25 ms and a step of 10 ms. The arithmetic mean and the coefficient of variance were then computed over the sequential LLDs at a rate of 40 ms -to match the granularity of the annotation -using overlapping windows of 8 s length, resulting in 26 statistical features per analysis window. The total numbers of segments in the train, development, and test partitions are 120 000, 112 500, and 112 500, respectively.

Implementation and Evaluation

When training the first BLSTM-RNN model as an AE, there are two methods for choosing the training data as mentioned in Section 3.3 -either by the 13 LLDs (lld-based strategy), or by the 26 statistical features (functional-based strategy). This choice defines then the number of nodes of the input and output layers of the model, i. e., either 13 or 26, respectively. In both cases, only one dimensional RE t is produced per input by Eq. ( 2), giving rise to 14 and 27 dimensional features, respectively. However, for the LLDs, functionals have to be applied before feeding them into the second BLSTM-RNN model. In doing this, there are 28 statistical features obtained in total for the lld-based strategy, since the means and variations are further calculated over the 13 LLDs plus the generated RE t. Therefore, when training the second model to perform emotion prediction, the number of nodes of its input layer is either 28 for the lld-based strategy, or 27 for the functional-based strategy, respectively. We adopted two hidden layers for both BLSTM-RNN models. In our experiments, the first model consists of 30 nodes per hidden layer with a learning rate of 10 -6 , whereas 26 nodes are used for the second model, with a learning rate of 10 -5 . Zero mean Gaussian noise with standard deviation 0.2 was added to the input activations in the training phase of both models to improve generalisation. The parameters of each model were optimised on the validation set with an early stopping strategy. Note that, an online standardisation was carried out on the features for both validation and test partitions, i. e., the means and variances of the features were calculated on the training partition and used on the two other partitions for standardisation. Additionally, annotation delay compensation was performed to compensate for the temporal delay between the observable cues, as seen in the recordings, and the corresponding emotion reported by the annotators [START_REF] Mariooryad | Correcting time-continuous emotional labels by modeling the reaction lag of evaluators[END_REF]. As in [START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF][START_REF] Huang | An investigation of annotation delay compensation and output-associative fusion for multimodal continuous emotion prediction[END_REF], we identified this delay to be four seconds which was duly compensated, by shifting the gold standard back in time with respect to the features for both arousal and valence.

To evaluate the performance of our methods, we used the CCC, which is a standard evaluation metric for time-and value-continuous predictions of emotion [START_REF] Ringeval | AVEC 2015: The 5th international audio/visual emotion challenge and workshop[END_REF]; it measures here the agreement between the gold standard and the prediction provided by the second BLSTM-RNN model. Given two time sequences x and y, their CCC

ρc = 2ρσxσy σ 2 x + σ 2 y + (µx -µy) 2 , ( 3
)
where ρ is the Pearson's Correlation Coefficient (PCC) between two time series (e.g., prediction and gold-standard); µx and µy are the means of each time series; and σ 2 x and σ 2 y are the corresponding variances. In contrast to the PCC, CCC takes not only the linear correlation, but also the bias between the two temporal series, i.e., (µx -µy) 2 , into account. Hence, the value of CCC is within the range of [-1, 1], where ±1 represents perfect concordance and discordance while 0 means no correlation.

To further assess the significance level of performance improvement, a statistical evaluation was carried out over all predictions obtained with the proposed RE-based learning and with a baseline method (i.e., original features without RE), by means of the Fisher's r-to-z transformation [START_REF] Cohen | Applied multiple regression/correlation analysis for the behavioral sciences[END_REF]. Unless stated otherwise, a p value lower than .05 indicates statistical significance.

Results and Discussion

The performance of the proposed RE-based learning with two RE extraction strategies (i.e., lld-based or functional-based) on the development and the test sets, for both arousal and valence regression tasks, is shown in Table 2. One can notice that in almost all cases, the RE-based learning methods significantly outperform the baseline systems that do not include the RE information. Even though a slightly decrease of performance can be observed on the development set for arousal and the functional based approach, the performance on the test set still benefits from the use of RE as additional information.

These results are highly encouraging since we only added a summary of the ability of the LSTM-RNN model to encode emotion related features into the original feature set, which yet contributes to a significant improvement of performance. In addition, we tentatively combined the RE features from another unmatched model rather than BLSTM-RNN (e. g., Support Vector Regression), no similar performance improvement was observed from the second BLSTM-RNN model. This might imply that the RE extracted from a similar structure better represent its drawbacks and is more helpful for the exploitation.

Further, following the AVEC's post-processing procedure of predictions [START_REF] Ringeval | AVEC 2015: The 5th international audio/visual emotion challenge and workshop[END_REF][START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF], that was successfully replicated in other studies [START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? End-toend speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF], we applied the same chain of post-processing on the obtained predictions; smoothing, centring, scaling, and time-shifting. All the modification parameters are optimised on the development set.

The resulting performance for both RE-based learning methods and the baseline system is presented in Table 3. As expected, all Table 3. Performance comparison (CCC) between the proposed reconstruction-error-based learning methods (lld-or functionalbased), the baseline system, and two other approaches (LSTM with CCC as cost function, and end-to-end learning), on the dev(elopment) and test partitions for both arousal and valence regression, after post-processing the predictions. The symbol of * indicates the significance of the performance improvement over the baseline. [START_REF] Weninger | Discriminatively trained recurrent neural networks for continuous dimensional emotion recognition from audio[END_REF][START_REF] Trigeorgis | Adieu features? End-toend speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF] and outlined in Section 1.

arousal
To further investigate whether the proposed features contribute to the observed performance improvement, we calculate the PCC between the RE-based features and the performance improvement, and the PCC between the RE-based features and the gold standard. Specifically, the performance improvement is defined as δ = |p b -gs| -|p -gs|, given the golden standard (gs) and the predictions of the input by using the baseline method (p b ) or the RE-based methods (p ). Furthermore, for the lld-based strategy, the RE-based feature is calculated by = 0.5 * std( t) + 0.5 * var( t); while for the functional-based strategy, = t.

The corresponding values of the PCC are presented in Table 4.

From the table, one can see that the RE features have a relationship with the performance improvement, particularly for valence regression. This somewhat implies that the second BLSTM-RNN model pays more attention to the input regions that have high RE values, and yields better performance in these regions. This also means that the model holds the capability of learning from its drawbacks, as we hypothesised at the beginning of the paper. Furthermore, the PCC between the RE features and the performance improvement is higher for valence than the values for arousal, which confirms the observation in Table 2 and 3 that more improvement was obtained for valence regression than for arousal when integrating the RE-based features. In addition, one can also notice that the RE-based features have some correlation with the gold standard for both arousal and valence regressions. This indicates that the RE-based features take some pattern information which might be complementary for the original features.

To highlight the performance of the RE-based learning methods, Fig. 2 illustrates the automatic predictions of arousal and valence obtained with the best settings among the two proposed strategies (lld-or functional-based) for a single test subject. In general, the predictions generated by the proposed method are closer to the gold standard, which consequently contributes to better results in terms of CCC.

CONCLUSIONS

In this paper, we proposed a novel Reconstruction-Error-based (REbased) learning framework for continuous emotion recognition in speech. It extracts the RE information from one learning model, then regards this information as additional features for another similar learning model for regression. Experiments were performed on the spontaneous emotional database -RECOLA -with BLSTM-RNN. Two variants were investigated, as RE information could be extracted from either the frame-based Low-Level Descriptors (LLD) or the segment-based statistical features after applying functionals on the LLD. When combining the RE information with the original features, the performance of the continuous emotion recognition systems can be significantly improved, and perform even better than the most recent approaches. Moreover, the high correlation between the RE and the performance improvement indicates that the RE information has a positive impact on the model. One may further note that, the proposed methods could also be applied to other regression problems, or even for classification tasks. For future work, we will investigate into more details the reasons that cause the BLSTM-RNN model to fail in the reconstruction, and will exploit other popular regression models, like Support Vector Regression.

Fig. 1 .

 1 Fig. 1. Framework of reconstruction error based learning

Fig. 2 .

 2 Fig.2. Illustration of arousal and valence predictions obtained by the reconstruction-error-based learning method and the baseline method for the 4th subject from the test partition.

  , which consists of two stages: (i) the extraction stage, and (ii) the exploitation stage. In the first stage, the RE information is collected by applying a first model to rebuild each frame of the input as an AE. Given the inputs xt of the model at time t, and its output counterparts xt, the RE t can be calculated by xtxt . In the second stage, the RE information combined with original features, i. e.,[xt, t], are used as new inputs to build a regression model that performs emotion prediction.

	xt	M odel1	xt	-	xt -xt	M odel2	yt
		extraction stage			exploitation stage

Table 1 .

 1 Three partitions of the RECOLA database.

	#	train	development	test
	female/male	10/6	9/6	8/7
	French	11	11	11
	Italian	3	2	3
	German	2	1	1
	Portuguese	0	1	0
	age µ (σ)	22.3 (3.4)	21.6 (2.1)	21.2 (2.0)

Table 2 .

 2 Performance comparison (CCC) between the baseline and the proposed reconstruction-error-based learning methods (lld-or functional-based) on the dev(elopment) and test partitions for both arousal and valence regression. The symbol of * indicates the significance of the performance improvement over the baseline.

		arousal	valence
	CCC	dev	test	dev	test
	baseline	.712	.703	.333	.238
	lld-based	.762 * .723 * .360 * .263 *
	functional-based .710	.714

* .368 * .298 * is defined as follows:

  .729 * .364 * .309 * functional-based .754 .720 * .378 * .360 * of the post-processed results performed better than the results without post-processing of the predictions. In this scenario, one can still observe that the RE-based learning methods persistently outperform the baseline by a significant margin. Particularly, the best results on the test set are achieved at .729 of CCC for arousal with the lldbased strategy, and at .360 of CCC for valence for the functionalbased strategy. Additionally, the RE-based learning methods remarkably outperforms the state-of-the-art methods, which are investigated in

				valence
	CCC	dev	test	dev	test
	baseline	.776	.710	.333	.237
	LSTM [6]	.412	.350	.242	.199
	End-to-End [15] .741	.686	.325	.261
	lld-based	.785

*

Table 4 .

 4 Linear correlations (PCC, ρ) between the reconstruction error ( ) and the prediction improvement (δ), or between the reconstruction error ( ) and the golden standard (gs), on the test partition for both arousal and valence regression.

				arousal	valence
		PCC	ρ( , δ) ρ( , gs) ρ( , δ) ρ( , gs)
		lld-based	.086	.372	.561	.208
		functional-based .019	.417	.467	.200
		0.4
	arousal	-0.6 -0.2 0 0.2 -0.4			gold standard baseline model proposed model
		50	100	150	200	250	300
				time in seconds
		0.6
	valence	0 0.2 0.4
		-0.2			gold standard baseline model proposed model
		50	100	150	200	250	300
				time in seconds

ACKNOWLEDGEMENTS

This work was partially supported by the EC's 7th Framework Programme through the ERC Starting Grant No. 338164 (iHEARu), the EU's Horizon 2020 Programme through the Innovative Action No. 645094 (SEWA) and the Research Innovative Action No. 645378 (ARIA-VALUSPA), and by the German Federal Ministry of Education, Science, Research and Technology (BMBF) under grant agreement No. 16SV7213 (EmotAsS).


Jouni Pohjalainen

Fabien Ringeval

Zixing Zhang

Björn Schuller

Spectral and Cepstral Audio Noise Reduction Techniques in Speech Emotion Recognition

Keywords: noise reduction, denoising, speech emotion recognition

Signal noise reduction can improve the performance of machine learning systems dealing with time signals such as audio. Real-life applicability of these recognition technologies requires the system to uphold its performance level in variable, challenging conditions such as noisy environments. In this contribution, we investigate audio signal denoising methods in cepstral and log-spectral domains and compare them with common implementations of standard techniques. The di↵erent approaches are first compared generally using averaged acoustic distance metrics. They are then applied to automatic recognition of spontaneous and natural emotions under simulated smartphone-recorded noisy conditions. Emotion recognition is implemented as support vector regression for continuous-valued prediction of arousal and valence on a realistic multimodal database. In the experiments, the proposed methods are found to generally outperform standard noise reduction algorithms.

INTRODUCTION

Audio signals are commonly a↵ected by factors other than the signal of interest. Additive noise arises due to other sound sources competing with the target signal and due to the combined e↵ects of the recording equipment and the transmission channel. Convolutive variation appears due to the channel and acoustic reverberation. The performance of machine learning systems is generally negatively a↵ected by these e↵ects, because they may mask target signals, such as speech, and cause mismatch in feature statistics between the training and recognition conditions.

Additive noise is a ubiquitous problem that a↵ects the performance of machine-learning speech systems in many common environments. In the field of automatic speech recognition (ASR), this is addressed on di↵erent stages of Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

MM '16, October 15-19, 2016, Amsterdam, Netherlands c 2016 ACM. ISBN 978-1-4503-3603-1/16/10. . . $15.00 DOI: http://dx.doi.org/10.1145/2964284.2967306 the system, including feature extraction, feature enhancement, machine learning models and their training [START_REF] Huang | Spoken Language Processing[END_REF][START_REF] Schuller | Recognition of noisy speech: A comparative survey of robust model architecture and feature enhancement[END_REF] and also by using better hardware or multiple microphones.

Noise reduction for speech audio is often evaluated from the viewpoint of listeners in terms of enhancing either the subjective quality or speech intelligibility [START_REF] Loizou | Speech Enhancement: Theory and Practice[END_REF]. Of these, intelligibility, whose improvement is the more di cult problem [START_REF] Loizou | Speech Enhancement: Theory and Practice[END_REF], is potentially relevant for ASR. Computational paralinguistic analysis of speech, whose general aim is to uncover various external attributes of speech not related to its linguistic message (the concern of ASR), has emerged as an active field of research within the past decade [START_REF] Schuller | Paralinguistics in speech and languagestate-of-the-art and the challenge[END_REF]. Two of its most central topics are speaker recognition (identification and verification) and, more recently, automatic recognition of emotion in speech. These applications, which are important for multimedia content analysis, retrieval and processing, do not have an obvious connection with the traditional objectives of speech enhancement. Nevertheless, in speaker recognition, traditional single-channel noise reduction systems are frequently found helpful as preprocessing in improving system performance under noisy or mismatched conditions [START_REF] Godin | Impact of noise reduction and spectrum estimation on noise robust speaker identification[END_REF][START_REF] Ortega-Garcia | Overview of speech enhancement techniques for automatic speaker recognition[END_REF][START_REF] Sadjadi | Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions[END_REF][START_REF] Saeidi | Temporally weighted linear prediction features for tackling additive noise in speaker verification[END_REF]. General-purpose signalbased denoising methods, such as power spectral subtraction and minimum-mean-square-error estimation of log-spectral amplitude, have been found to improve speaker recognition performance quite reliably across di↵erent back-end machine learning systems, test conditions (type and level of degradation) and training conditions (noise-matched or mismatched), even more so than auditory-based methods designed to improve subjective quality [START_REF] Godin | Impact of noise reduction and spectrum estimation on noise robust speaker identification[END_REF]. In general, the relative performance of di↵erent denoising methods is quite strongly a↵ected by the aforementioned factors [START_REF] Godin | Impact of noise reduction and spectrum estimation on noise robust speaker identification[END_REF][START_REF] Sadjadi | Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions[END_REF].

In this paper, following on the results of the studies focusing on signal denoising in text-independent speaker recognition [START_REF] Godin | Impact of noise reduction and spectrum estimation on noise robust speaker identification[END_REF][START_REF] Sadjadi | Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions[END_REF], a machine learning task that is commonly based on identifying the distribution of the short-time magnitude spectrum, we compare denoising techniques from the viewpoint of audio and multimedia content analysis in speech emotion recognition. It is another central paralinguistic problem with other applications ranging from ASR systems and call centers to intelligent user interfaces [START_REF] Schuller | Paralinguistics in speech and languagestate-of-the-art and the challenge[END_REF]. Work on noise robustness in speech emotion recognition has also been started [START_REF] Schuller | Emotion recognition in the noise applying large acoustic feature sets[END_REF][START_REF] Zhang | Facing realism in spontaneous emotion recognition from speech: Feature enhancement by LSTM neural autoencoders[END_REF], but to our knowledge, signal denoising has not been explicitly studied for this purpose. As emotional or stressed speech often coincides with acoustically noisy environments, it is worthwhile to investigate the performance of general-purpose noise reduction solutions in improving short-time spectral representation of the two pri-mary emotion dimensions -activation (arousal), i. e., the intensity of emotional expression, and valence, i. e., the positive/negative a↵ective dimension [START_REF] Goudbeek | Beyond arousal: Valence and potency/control cues in the vocal expression of emotion[END_REF].

We propose a family of general-purpose, adaptive signal denoising techniques that can be simply configured with respect to spectral smoothing, temporal smoothing and adaptation rate of the noise model in order to tackle di↵erent nonstationary noise conditions and application requirements. They operate in a log-spectral or cepstral domain, which di↵er in terms of spectral smoothing facilitated by the cepstral representation. These methods are compared against published implementations of baseline methods suggested by the previous studies, firstly by using general, objective signal enhancement quality measures. The most promising variants are then compared as preprocessing for an emotion recognition system performing continuous-valued prediction of arousal and valence. In this study, we focus on relatively similar noise conditions in the training and testing phase, so that denoising is used to recover important information in both training and testing.

NOISE REDUCTION

Previous Work

Classical single-channel noise reduction methods include variants of spectral subtraction [START_REF] Berouti | Enhancement of speech corrupted by acoustic noise[END_REF][START_REF] Boll | Suppression of acoustic noise in speech using spectral subtraction[END_REF], where an averaged noise (magnitude or power) spectrum is subtracted from the noisy signal spectrum while keeping the resultant spectral magnitudes positive, and Wiener filtering [START_REF] Loizou | Speech Enhancement: Theory and Practice[END_REF], often implemented in practice using an iterative approach [START_REF] Hansen | Constrained iterative speech enhancement with application to speech recognition[END_REF].

The performance of the minimum mean square error (MMSE) [START_REF] Ephraim | Speech enhancement using a minimum mean-square error short-time spectral amplitude estimator[END_REF] and log-spectral amplitude MMSE (Log-MMSE) estimators [START_REF] Ephraim | Speech enhancement using a minimum mean-square error log-spectral amplitude estimator[END_REF] still remains among the best of the published methods [START_REF] Paliwal | Speech enhancement using a minimum mean-square error short-time spectral modulation magnitude estimator[END_REF]. In part, this can be attributed to their decision-directed estimation approach, which bases the spectral estimate of each frame partially on the estimates from previous frames via the a priori SNR estimate updated by using a memory coe cient [START_REF] Cappé | Elimination of the musical noise phenomenon with the Ephraim and Malah noise suppressor[END_REF]. Spectral subtraction [START_REF] Paliwal | Single-channel speech enhancement using spectral subtraction in the short-time modulation domain[END_REF] and the decision-directed MMSE [START_REF] Paliwal | Speech enhancement using a minimum mean-square error short-time spectral modulation magnitude estimator[END_REF] methods have also been applied in the spectral modulation domain in order to better handle nonstationary noise.

In recent comprehensive evaluations looking at noise reduction preprocessing techniques to combat mismatch in speaker recognition, MMSE, Log-MMSE and spectral subtraction have performed well [START_REF] Godin | Impact of noise reduction and spectrum estimation on noise robust speaker identification[END_REF][START_REF] Sadjadi | Assessment of single-channel speech enhancement techniques for speaker identification under mismatched conditions[END_REF]. Power spectral subtraction also resulted in large performance gains in [START_REF] Saeidi | Temporally weighted linear prediction features for tackling additive noise in speaker verification[END_REF]. Therefore, as the baseline noise reduction methods in this study, we choose two published implementations of spectral subtraction, one working in the power spectral domain (SS-P [START_REF] Loizou | Speech Enhancement: Theory and Practice[END_REF]) and one in the magnitude spectral domain (SS-M [START_REF] Brookes | VOICEBOX: Speech processing toolbox for MATLAB[END_REF]), and two published implementations of Log-MMSE (MMSE-1 [START_REF] Loizou | Speech Enhancement: Theory and Practice[END_REF]; MMSE-2 [START_REF] Brookes | VOICEBOX: Speech processing toolbox for MATLAB[END_REF]).

Methods Under Study

The methods proposed in this study are based on obtaining the noise model by averaging over low-energy frames in the representation domain -log-spectral or cepstral -and, optionally, by adapting the noise model constantly based on the (Gaussian) similarity of the short-term moving average of the representation to the initial noise model. A general overview is presented in Fig. 1.

The audio signal is processed in Hann-windowed frames of 30 ms extracted every 10 ms. The discrete Fourier transform (DFT) of each audio frame is first transformed into either a logarithmic magnitude spectrum or a truncated cepstrum [START_REF] Deller | Discrete-Time Processing of Speech Signals[END_REF][START_REF] Rabiner | Digital Processing of Speech Signals[END_REF]. The real cepstrum is a frequency transform (specifically, the inverse Fourier transform) of the logarithmic spectrum. A truncated cepstrum, where only the lower-order coecients are retained, thus represents a smoothed version of the original logarithmic spectrum [START_REF] Rabiner | Digital Processing of Speech Signals[END_REF]. With the sampling rate of 44100 Hz used in this study, we choose to retain the first J = 50 cepstral coe cients in accordance to a rule of thumb of choosing the length of the truncated cepstrum to be less than an expected pitch period [START_REF] Rabiner | Digital Processing of Speech Signals[END_REF].

In the lower branch of Fig. 1, low-energy frames within the analysis block of six seconds are automatically found by clustering the frame logarithmic energies into two clusters by using k-means and selecting the frames assigned to the cluster with the lower mean value. The initial noise model D = {µj, 2 j }, consisting of a mean and variance for each (spectral or cepstral) coe cient, is then obtained by averaging over these frames. In the upper branch, temporal smoothing is applied to the complete sequence of the noisy vectors {Y n} by using the equation

Zn = Zn 1 + (1 )Y n, ( 1
)
where n is the frame index, = (w 1)/w and w is an "equivalent rectangular window length" of the moving average integrator in the sense that the contribution of each new observation is weighted by 1/w. Setting w = 1 causes no temporal integration to be performed. In the next stage, the similarity of the noisy representation Zn = (Zn,1, . . . , Zn,J ) with the initial noise model is evaluated as

n,j = exp( (Zn,j µj) 2 / 2 j ), 1  j  J, (2)
yielding scores in the range (0, 1). The components of the current noise model at the nth frame µn,j (initially, µ0,j = µj) are then adapted towards the direction of the current smoothed noisy representation Zn, in proportion ↵ to the similarity n,j of the latter to the initial noise model:

µn,j = µn 1,j + ↵ n,j (Zn,j µn 1,j ). (3)
In both log-spectral and cepstral domains, spectral-domain division corresponds to subtraction. Therefore, the DFTdomain noise extraction filter Gn(k) is represented by µn,j Zn,j in the analysis domain, and can be obtained by transforming this quantity back to the DFT domain. It is then used to create the noise suppression filter, with the lower limit of suppression at zero: Hn(k) = max(1 Gn(k), 0). The modified frames are combined using an overlap-add method to produce the enhanced signal.

One motivation for processing the signal in one of these domains is to be able to apply cepstral smoothing in generating the noise and noisy signal models by temporal averaging and integration. The log-spectral representation is both an intermediate stage in cepstrum computation and a special case in the sense that it is equivalent to a non-truncated cepstral representation. Considering distance metrics in the cepstral and log-spectral domains, the cepstral Euclidean distance can be shown to be a lower bound for the rootmean-square (RMS) log-spectral distance; with the inclusion of more cepstral coe cients in the truncated cepstrum, more spectral fine structure information is preserved and the cepstral Euclidean distance approaches the RMS log-spectral distance from below [START_REF] Gray | Distance measures for speech processing[END_REF]. The logarithmic spectrum representation is frequently used in audio signal analysis due to the large dynamic range of hearing in the amplitude dimension [START_REF] Zwicker | Psychoacoustics, Facts and Models[END_REF], and is also used in denoising by the Log-MMSE method [START_REF] Ephraim | Speech enhancement using a minimum mean-square error log-spectral amplitude estimator[END_REF]. In this paper, methods using the two representations are distinguished by referring to them as cepstral noise reduction (CNR) and log-spectral noise reduction (LNR). The variants of these main types are further distinguished from each other based on the temporal smoothing parameter w and the adaptation rate ↵ (Eqs. 1-3), which control the focus of the noise suppression filter on di↵erent modulation frequencies of the nonstationary noise and the target signal.

EXPERIMENTS

Overview and Material

We evaluate the denoising methods on the RECOLA multimodal a↵ective interaction corpus [START_REF] Ringeval | Introducing the RECOLA multimodal corpus of remote collaborative and a↵ective interactions[END_REF]. It includes 46 multimodal (audio, video and physiological data) recordings of French-speaking participants involved in a dyadic collaborative task. A↵ective dimensions expressed by the participants were evaluated by six annotators for the first five minutes of each recording. This was done for arousal and valence separately. Obtained labels were then resampled to a constant 40 ms frame rate and averaged over all raters by considering inter-evaluator agreement, to provide a 'gold standard' [START_REF] Ringeval | Prediction of asynchronous dimensional emotion ratings from audiovisual and physiological data[END_REF]. In order to ensure speaker-independence in the experiments, the corpus was split into three partitions, by balancing the gender and the age of the subjects: training (16 subjects), validation (15 subjects) and testing (15 subjects).

In adding noise to the audio material, the recordings were convolved with the impulse response of a smartphone [START_REF] Mauch | The audio degradation toolbox and its application to robustness evaluation[END_REF]. This step was performed to simulate talking over a smartphone in di↵erent places. Di↵erent types of additive noise were then added, using the CHiME-2013 database [START_REF] Barker | The PASCAL CHiME speech separation and recognition challenge[END_REF] for simulating a living room environment (CHiME ), and data collected from the freesound platform 1 to simulate public transport (trains) environments. We collected, in total, 230 minutes of noise, to match the duration of the RECOLA database. In order to provide realistic conditions, we concatenated all the recordings of noise into three independent partitions and added them to the smartphone-simulated recordings of the RECOLA database, with two di↵erent signal-to-noise ratios (SNRs): 0 and 6 dB.

In the first part dealing with acoustic distance measures (Section 3.2), we use the first six recordings of the RECOLA training set (30 minutes in total) and study the quality of noise reduction using these distance measures. The original audio is used as reference signal.

An emotion recognition system is then equipped with different types of noise reduction preprocessing, in both the 1 https://www.freesound.org/ training and evaluation phase, using the most promising variants of the proposed methods and the relevant baseline methods. The di↵erent systems are trained on the training set of RECOLA to predict continuous-valued arousal and valence using support vector regression (SVR) on the mean and variance of 13 mel-frequency cepstral coe cients (MFCCs). In the training phase, we adjust the windowing for mean and variance computation to be optimal according to the validation set, as in [START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF]. For SVR, we use a linear kernel and also tune the complexity on the validation partition [START_REF] Zhang | Facing realism in spontaneous emotion recognition from speech: Feature enhancement by LSTM neural autoencoders[END_REF]. Post-processing of the predictions is applied using the same methodology as described in [START_REF] Trigeorgis | Adieu features? End-to-end speech emotion recognition using a Deep Convolutional Recurrent Network[END_REF]. Performance in emotion recognition is evaluated with the concordance correlation coe cient (CCC) [START_REF] Lin | A concordance correlation coe cient to evaluate reproducibility[END_REF][START_REF] Valstar | AVEC 2016 -depression, mood, and emotion recognition workshop and challenge[END_REF].

Acoustic Quality Measures

Several quality measures exist for automatically evaluating denoising and enhancement results for speech signals. These include simple energy-based measures such as SNR and segmental SNR, perceptual measures such as PESQ and distance measures in di↵erent representation domains of the magnitude spectrum [START_REF] Loizou | Speech Enhancement: Theory and Practice[END_REF].

The short-time magnitude spectrum is the basis of most audio feature extraction techniques. Audio pattern recognition and machine learning systems generally aim to distinguish sound classes based on the distributions of the shape of the short-time magnitude spectrum, and typically are not concerned with the true sound level or its psychoacoustical counterpart, loudness [START_REF] Zwicker | Psychoacoustics, Facts and Models[END_REF]. Therefore, in this study, we choose not to use measures that depend on the signal level in any manner, and whose results would depend on proper gain adjustment applied to the enhanced signal. These include the SNR measures as well as the RMS log-spectral distance [START_REF] Gray | Distance measures for speech processing[END_REF]. We also do not use perceptual measures that explicitly aim to predict subjective quality or intelligibility. Instead, we focus on the preservation of the information about the shape of the short-time magnitude spectrum. These considerations resulted in the choice of two measures. Firstly, we apply the Itakura distance [START_REF] Deller | Discrete-Time Processing of Speech Signals[END_REF][START_REF] Itakura | Minimum prediction residual principle applied to speech recognition[END_REF], given by dITA = log(aR0a 0 /a0R0a 0 0 ), where a=(1, a1, . . . , ap) is the all-pole model under test for its similarity to the reference model a0 = (1, a0,1, . . . , a0,p), estimated using the Toeplitz autocorrelation matrix R0 [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF]. The numerator represents the total squared prediction error of the reference signal using the test model and the denominator represents the error using the reference model. In this study, R0 and the reference model a0 are obtained from the original, i. e., not noisified signal. This is a purely signal-based, non-perceptual distance measure that focuses on accuracy in all-pole modeling, which in turn is known to focus on spectral peaks and formants [START_REF] Makhoul | Linear prediction: A tutorial review[END_REF][START_REF] Rabiner | Digital Processing of Speech Signals[END_REF]. As another measure, we apply the Euclidean distance between two mel-frequency cepstral vectors, the noisy test vector and the reference vector obtained from the original signal, consisting of 12 MFCCs while excluding the zeroth coe cient [START_REF] Huang | Spoken Language Processing[END_REF]; we denote this measure by dMFCC. This measure has the perceptual aspect of an auditory-based warped frequency scale and it is also connected to the feature extraction procedure of many machine learning systems, including the one used in the present study.

Fig. 2 shows the Itakura and MFCC distances, averaged over a large number of speech frames, computed such that denoised frames are compared against corresponding reference frames in the original signal. Three conditions are con-sidered: smartphone speech, and smartphone with CHiME and trains noise, each added at 6 dB SNR.

Emotion Recognition

Table 1 shows the emotion recognition results in various noise conditions with approximately matched training.

When training is performed with the original audio material, the system is evaluated with the original and smartphoneprocessed material. For CHiME and trains noise, the training is performed with 6 dB SNR and the system is evaluated with 0 dB and 6 dB SNR of the same type of noise corruption. This setup requires the noise reduction preprocessing methods to preserve emotion-related information to a sucient precision in both the training and recognition phases.

The two leftmost results columns indicate that denoising clean speech generally degrades recognition performance. A similar observation has been made in a recent study on feature enhancement with the same data [START_REF] Zhang | Facing realism in spontaneous emotion recognition from speech: Feature enhancement by LSTM neural autoencoders[END_REF]. An exception here, however, is the cepstral approach with moderate adaptation and no temporal smoothing (CNR/1,0.5), which actually improves prediction of the arousal dimension. None of the evaluated methods improve the prediction of arousal under the nonstationary CHiME noise; this might be explained by highly variable residual noise corrupting spectral energy trajectories. Otherwise, the proposed methods (CNR, LNR) outperform both the not denoised approach and the standard baseline noise reduction methods.

CONCLUSIONS

Noise reduction methods were evaluated with a focus on paralinguistic machine learning applications (two emotion recognition tasks). Baseline methods were chosen based on earlier, related studies and compared with a proposed new approach, which can be configured according to the specific learning task and the noise conditions using a small number of parameters. The experiments involved objective signal degradation measures and recognition of emotions on a multimodal interaction corpus. In the former evaluation, the proposed approach is shown to perform competitively to the standard noise reduction methods (spectral subtraction and MMSE, recommended as robust baseline methods in the field of speaker recognition [START_REF] Godin | Impact of noise reduction and spectrum estimation on noise robust speaker identification[END_REF]) over most of the range of the parameter values. By increasing the temporal smoothing and adaptation rate in noise modeling, the log-spectral approach clearly outperforms the standard techniques.

In emotion recognition evaluations, arousal and valence are noticed to place somewhat di↵erent requirements on the denoising scheme. Denoising improves the results in a majority of the evaluated training/test setups and in these cases, the proposed methods outperform the standard baselines. The cepstral approach without temporal smoothing of the noise leads to noticeable improvement in the high-SNR conditions. In noisy training and test conditions, temporal smoothing combined with adaptation outperforms all other evaluated approaches. Therefore, it is noted that the proposed methods show promise in audio noise reduction and audio content analysis. In future work, the relationships of di↵erent learning tasks, noise conditions and noise reduction approaches can be further investigated.

Figure 1 :

 1 Figure 1: The noise reduction framework.

Figure 2 :

 2 Figure 2: Signal degradation (left: Itakura distance, right: MFCC distance) evaluated for denoised signals against original, frame by frame, averaged over three noise conditions and the 25 % frames with most target signal energy. The performance of the proposed methods is shown with respect to the temporal smoothing parameter w and the noise model adaptation rate ↵. Corresponding averages for unprocessed noisy signals are shown in parentheses.

Table 1 :

 1 Correlation (CCC) between predicted and ground-truth arousal and valence over the test set for closely matched training while using a given denoising method in both training and testing. The cases in which denoising improves upon the not denoised case are indicated in bold. .308 0.180 0.127 0.243 0.156 CNR (20,1.0) 0.207 0.179 0.261 0.227 0.252 0.198 LNR (1,0.5) 0.359 0.288 0.174 0.124 0.147 0.107 LNR (20,1.0) 0.278 0.250 0.137 0.109 0.241 0.218

	Training	Original	CHiME 6 dB trains 6 dB
	Noise			Test
	reduction	Orig. phone	CHiME	trains
	method (w,↵)	-	-	6 dB 0 dB 6 dB 0 dB
				AROUSAL
	none	0.735 0.728 0.670 0.557 0.483 0.379
	SS-P	0.704 0.697 0.634 0.553 0.538 0.441
	SS-M	0.708 0.701 0.637 0.535 0.564 0.488
	MMSE-1	0.672 0.680 0.601 0.490 0.566 0.488
	MMSE-2	0.679 0.698 0.620 0.510 0.514 0.430
	CNR (1,0.5) 0.753 0.755 0.599 0.502 0.535 0.438
	CNR (20,1.0) 0.676 0.650 0.642 0.539 0.547 0.440
	LNR (1,0.5)	0.706 0.664 0.635 0.538 0.532 0.435
	LNR (20,1.0) 0.676 0.650 0.641 0.535 0.701 0.637
				VALENCE
	none	0.400 0.342 0.173 0.120 0.154 0.108
	SS-P	0.328 0.262 0.112 0.107 0.195 0.228
	SS-M	0.386 0.303 0.128 0.114 0.190 0.177
	MMSE-1	0.318 0.294 0.098 0.066 0.218 0.190
	MMSE-2	0.340 0.292 0.087 0.071 0.219 0.160
	CNR (1,0.5)	0.308 0

ACKNOWLEDGMENTS

This work has been supported by the European Community's Seventh Framework Programme through the ERC Starting Grant No. 338164 (iHEARu).


P Kerfriden
email: pierre.kerfriden@gmail.com

S Claus

A stable cut finite element method for multiple unilateral contact

Keywords: cut finite elements, contact, multiple materials, LaTIn

This paper presents a novel CutFEM-LaTIn algorithm to solve multiple unilateral contact problems over geometries that do not conform with the finite element mesh. We show that our method is (i) stable, independently of the interface locations (ii) optimally convergent with mesh refinement and (iii) efficient from an algorithmic point of view.

Introduction

Since the pioneer work presented in [START_REF] Dolbow | An extended finite element method for modeling crack growth with frictional contact[END_REF], several research teams have successfully developed stable XFEM schemes for unilateral contact (see e.g. [START_REF] Liu | A contact algorithm for frictional crack propagation with the extended finite element method[END_REF][START_REF] Gravouil | Stabilized global-local x-fem for 3d non-planar frictional crack using relevant meshes[END_REF][START_REF] Annavarapu | A Nitsche stabilized finite element method for frictional sliding on embedded interfaces. Part I: single interface[END_REF]). However, it is extremely challenging to make sure that these methods are optimally convergent with mesh refinement. As far as we are aware, this has only been achieved recently in [START_REF] Chouly | An overview of recent results on Nitsche's method for contact problems[END_REF][START_REF] Burman | The penalty free Nitsche method and nonconforming finite elements for the Signorini problem[END_REF]. In this paper, we propose an alternative to the latter contributions that is based on an extension of the CutFEM [START_REF] Hansbo | A finite element method for the simulation of strong and weak discontinuities in solid mechanics[END_REF][START_REF] Burman | CutFEM: Discretizing geometry and partial differential equations[END_REF] technology for embedded interface problems, coupled with the LaTIn mixed formulation of contact [START_REF] Ladevèze | On a multiscale computational strategy with time and space homogenization for structural mechanics[END_REF][START_REF] Dolbow | An extended finite element method for modeling crack growth with frictional contact[END_REF][START_REF] Kerfriden | A three-scale domain decomposition method for the 3D analysis of debonding in laminates[END_REF][START_REF] Gravouil | Stabilized global-local x-fem for 3d non-planar frictional crack using relevant meshes[END_REF]. We show that the formulation is always stable, and that our high-performance computing implementation allows us to perform 3D computations involving arbitrarily complex intersections of solids, potentially leading to field singularities. Moreover, we show optimal convergence with mesh refinement of the background grid. Incidentally, our work is also a stabilisation of the LaTIn hybrid-mixed formulation. In this respect, our method is an alternative to the two previous non-locking LaTIn formulations based on (i) a local coarsening of trace meshes (see e.g [START_REF] Ladevèze | On a multiscale computational strategy with time and space homogenization for structural mechanics[END_REF][START_REF] Kerfriden | A three-scale domain decomposition method for the 3D analysis of debonding in laminates[END_REF]) and (ii) a relaxation of interface kinematic conditions together with the addition of penalty terms, as presented in [START_REF] Gravouil | Stabilized global-local x-fem for 3d non-planar frictional crack using relevant meshes[END_REF]. Our approach is algorithmically quite advanced. We therefore choose, in this short contribution, to focus on the most novel aspects of it, which is the discrete non-conforming and non-locking mixed LaTIn formulation of contact, enriched and further stabilised with tools of the CutFEM technology. This scheme is developed in section 2, whilst section 3 contains some representative examples and short conclusions.

Finite Element Formulation

Domain Discretisation

We consider a domain Ω divided into n d non-overlapping subdomains Ω i . We partition the boundary of domain Ω, ∂Ω, into a Dirichlet ∂Ω D (body is clamped) and a Neumann part ∂Ω N (tractions are imposed). Furthermore, each subdomain Ω i is considered to be covered by a linear elastic body (i). We assume that if two elastic bodies share an interface Γ i, j = ∂Ω i ∩ ∂Ω j these two bodies are expected to come into contact. For example, in Figure 1a, we employ contact condtions on the interfaces Γ 1,2 , Γ 1,3 and Γ 2,3 . We introduce a non-conforming finite element discretisation of the domains as follows. Firstly, let T h denote a tessellation of domain Ω independent of the interface locations Γ i, j . Secondly, we introduce the following mesh subsets of our background mesh T h . For each subdomain Ω i , we introduce a so-called fictitious domain Ω * ,i := {K ∈ T h : K ∩ Ω i ∅}. In a similar fashion, we define the set of elements intersected by an interface

Γ i = (k,l ) ∈I Γ , (k=i or l=i) Γ k,l as G i := {K ∈ T h : K ∩ Γ i
∅}. We also define the following two subsets of faces between two elements F (K, K ) = K ∩ K : the set of element faces associated with G i , namely

F i G := {F ∈ F (K, K ) : K ∈ G i or K ∈ G i },
which we call ghost penalty faces, and the set of intersected faces 1b).

F i I := {F ∈ F (K, K ) : K ∈ G i and K ∈ G i } (see Figure
Ω 3 Ω 1 Ω 2 n 2,1 Γ 1,2 Γ 1,3 Γ 2,3 ∂Ω (a) Meshed Example Domain.
Ω * ,3

Γ 1,3 Γ 2,3 G 3 F 3 G Ω * ,3 Γ 1,3 Γ 2,3 G 3 F 3 I (b) Schematics of mesh subsets for Ω 3

Discrete hybrid-mixed formulation of the contact problem

Let U i h denote the vector valued space of continuous piecewise linear polynomials defined on the fictitious domain Ω * ,i . Our weak formulation of linear elasticity reads: For all Ω * ,i , find

u i h ∈ U i h such that for all δu i h ∈ U i h a(u i h , δu i h ) + a b (u i h , δu i h ) + j (u i h , δu i h ) = l (δu i h ) + l b (δu i h ). (1)
The terms of a(u i h , δu i h ) and l (δu i h ) are given by

a(u i h , δu i h ) = Ω i h σ(u i h ) : (δu i h ) dΩ - Γ i, j h (σ(u i h ) • n i, j ) δu i h dΓ, l (δu i h ) = Ω i h f • δu i dΩ, (2)
where σ(u) = λ i tr( (u)) I I I + 2 µ i (u) is the Cauchy-stress tensor, (u) = 1 2 ∇u + ∇u T is the strain tensor, n i, j is the normal on Γ i, j h pointing from Ω i h to Ω j h , λ i and µ i are the two Lamé coefficients, i.e.

λ i = E i ν (1+ν)(1-2ν) , µ i = E i 2(1+ν)
, with E i the Young's modulus and ν = 0.3 the Poisson's ratio. Ω i h and Γ i, j h denote piecewise polynomial approximations of exact geometrical sets. The terms a b (u i h , δu i h ) and l b (δu i h ) weakly enforce Dirichlet conditions using Nitsche's method and Neumann boundary conditions

a b (u i h , δu i h ) = - ∂Ω D (σ(u i h ) • n i, j ) δu i h dΓ - ∂Ω D (σ(δu i h ) • n i, j ) u i h dΓ + ∂Ω D αE i h u i h • δu i h dΓ, l b (δu i h )) = - ∂Ω D (σ(δu i h ) • n i, j ) g dΓ + ∂Ω D αE i h g • δu i h + ∂Ω N T δu i h dΓ dΓ, (3)
where T is a prescribed surface load, g is a prescribed displacement and α > 0 is the Nitsche penalty parameter. The term j (u i h , δu i h ) regularises the solution in the interface region and is given by

j (u i h , δu i h ) = F ∈ F i G F γ g h E i σ(u i h ) • n F • σ(δu i h ) • n F ds. (4)
We call j (u i h , δu i h ) ghost penalty stabilisation ( [START_REF] Burman | CutFEM: Discretizing geometry and partial differential equations[END_REF]). Here, x • n F denotes the normal jump of the quantity x over the face, F, defined as x • n F = x| K n F -x| K n F , where n F denotes a unit normal to the facet F with fixed but arbitrary orientation and γ g > 0 is the ghost penalty parameter. At contact interfaces Γ i, j h , we introduce interface force F i h and interface displacement W i h satisfying

Γ i, j h F i h • δu i h dΓ = Γ i, j h σ(u i h ) • n i, j • δu i h dΓ , and W i h = u i h at Γ i, j h . (5) Now, let Q h (G i ) := v h ∈ C 0 (G i ) : v h | K ∈ P 1 (K ) d
, ∀K ∈ G i denote the space of continuous piecewise linear polynomials on the band of intersected elements. Then, we determine the interface fields as follows. Find

F i h ∈ Q h (G i ) and W i h ∈ Q h (G i ), such that for all δF i h ∈ Q h (G i ) and for all δW i h ∈ Q h (G i ) Γ i, j h (F i h -F i ♥ ) δF i h dΓ + j F (F i h , δF i h ) = 0 , Γ i, j h (W i h -W i ♥ ) δW i h dΓ + j F (W i h , δW i h ) = 0 , (6)
where

j F (F i h , δF i h ) = F ∈ F i h γ Π h 2 F ∇F i h • n F • ∇δF i h , •n F ds (7)
regularises the interface fields with a penalty parameter γ Π > 0. Heart quantities F i ♥ , W i ♥ are required to fulfill contact at a set of quadrature points along the interface

Γ i, j h W j ♥ -W i ♥ • n i, j ≥ 0 ; F i ♥ • n i, j ≤ 0 ; (W j ♥ -W i ♥ ) • n i, j • F i ♥ • n i, j = 0 . (8)
and the system is closed by requiring that the fluctuation of the heart quantities around F i h and W i h satisfy a Robin condition, which reads as

F i h -F i ♥ -β W i h -W i ♥ = 0 , ( 9
)
where β is an algorithmic parameter that is homogeneous to a stiffness (we set it equal to the LaTIn search direction parameter for reasons not detailed here). We can finally solve the nonlinear coupled system of equations [START_REF] Annavarapu | A Nitsche stabilized finite element method for frictional sliding on embedded interfaces. Part I: single interface[END_REF][START_REF] Dolbow | An extended finite element method for modeling crack growth with frictional contact[END_REF][START_REF] Gravouil | Stabilized global-local x-fem for 3d non-planar frictional crack using relevant meshes[END_REF][START_REF] Kerfriden | A three-scale domain decomposition method for the 3D analysis of debonding in laminates[END_REF] for u i h , F i h , W i h , F i ♥ , W i ♥ , using an extension of the LaTIn iterative algorithm [START_REF] Ladevèze | On a multiscale computational strategy with time and space homogenization for structural mechanics[END_REF] whereby the local stage is "non-localised" by two-scale continuity condition (6). The structure illustrated in Figure 2 is made of two interacting inclusions within a square matrix. The square matrix is meshed using a regular grid of triangles (Figure 1a). The matrix is compressed in the vertical direction. We plot the vertical compressive component of the stress. We also show the convergence rate together with the element size h. The convergence rate is optimum in both H 1 and energy semi-norm. Figure 3 shows the versatility of the method and the stability of the implementation in 3D.

Results

Conclusions

We have presented a novel non-conforming finite element formulation for multiple linear elastic bodies in contact. We have demonstrated that our discretisation scheme is stable independent of how the contact interfaces intersect our fixed regular background mesh and that the contact problem solution converges optimally with mesh refinement. The next stage of our investigations is to extend the approach to the context of large displacements, whereby the contact region is an unknown of the problem.

Figure 1 :

 1 Figure 1: Example of rectangular domain Ω with two circular inclusions Ω 2 and Ω 3 and the definition of important mesh subsets for domain Ω 3 .

1 E

 1 (b) Convergence rates.

Figure 2 :

 2 Figure 2: Convergence rates and energy norm error for two inclusions with E 1 = E 2 = E 3 = 1.

  (a) Displacement magnitude. (b) von Mises stress on pig.

Figure 3 :

 3 Figure 3: Angry bird in contact with a pig. The bird is being pushed into the pig, which results in the development of a stress concentration around the contact zone.

Acknowledgements

We acknowledge the support provided by the Welsh Government and Higher Education Funding Council for Wales through the Sêr Cymru National Research Network in Advanced Engineering and Materials.