Diederick Vermetten

Sander Van Rijn

Thomas Bäck

Carola Doerr

Online Selection of CMA-ES Variants

Keywords: Theory of computation → Evolutionary algorithms, Bioinspired optimization, Online algorithms, Algorithm design techniques Evolution Strategies, Algorithm Configuration, Algorithm Selection

In the field of evolutionary computation, one of the most challenging topics is algorithm selection. Knowing which heuristics to use for which optimization problem is key to obtaining high-quality solutions. We aim to extend this research topic by taking a first step towards a selection method for adaptive CMA-ES algorithms. We build upon the theoretical work done by van Rijn et al. [PPSN'18], in which the potential of switching between different CMA-ES variants was quantified in the context of a modular CMA-ES framework.

We demonstrate in this work that their proposed approach is not very reliable, in that implementing the suggested adaptive configurations does not yield the predicted performance gains. We propose a revised approach, which results in a more robust fit between predicted and actual performance. The adaptive CMA-ES approach obtains performance gains on 18 out of 24 tested functions of the BBOB benchmark, with stable advantages of up to 23%. An analysis of module activation indicates which modules are most crucial for the different phases of optimizing each of the 24 benchmark problems. The module activation also suggests that additional gains are possible when including the (B)IPOP modules, which we have excluded for this present work.

INTRODUCTION

The creation of optimization algorithms has long been a topic of study in Mathematics and Computer Science. As the complexity of optimization problems increased, the need for fast heuristic algorithms became clear. Techniques like evolution strategies have been around for quite some time, and much research has been done into improving them, such as the popular Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [START_REF] Hansen | Completely Derandomized Self-Adaptation in Evolution Strategies[END_REF].

For CMA-ES, many adaptations have been developed over the last decade, all with the goal of creating a better optimization algorithm for certain types of problems. However, with all these different options to choose from, it can be difficult to know which version to use. The comparison of different CMA-ES variants consistently shows that there is no single best variant [START_REF] Bäck | Contemporary Evolution Strategies[END_REF][START_REF] Hansen | Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009[END_REF], giving rise to the question how to best choose which of the variants to use for which kind of optimization problems. In addition, all CMA-ES variants have several hyper-parameters, whose values can have considerable impact on their performance. This means that besides the algorithm selection problem we also face an algorithm configuration problem. These two problems are highly interlinked. The choice of algorithm and corresponding parameter configuration for a given problem at hand thereby forms itself a new optimization problem, which has gained increasing attention in the last decades. With the increased accuracy of performance prediction models, state-of-the-art research in evolutionary computation focuses on automating the decision of which algorithm and which parameter configuration to select. Among the most widely used prediction models are Gaussian processes [START_REF] Bartz-Beielstein | Experimental methods for the analysis of optimization algorithms[END_REF][START_REF] Bartz-Beielstein | The sequential parameter optimization toolbox[END_REF], but also techniques from the machine learning community, such as random forests [START_REF] Hutter | Sequential modelbased optimization for general algorithm configuration[END_REF][START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning[END_REF]. Surveys on algorithm selection are available in [START_REF] Holger | Automated Algorithm Selection and Configuration (Dagstuhl Seminar 16412)[END_REF][START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF][START_REF] Muñoz | Algorithm selection for black-box continuous optimization problems: a survey on methods and challenges[END_REF].

While algorithm selection generally considers static choices of algorithms, we aim to extend existing work to allow for adaptive changes of the algorithm configuration. Note here that on-the-fly algorithm selection and configurations are also studied under the umbrella term hyper-heuristics [START_REF] Burke | Hyper-heuristics: a survey of the state of the art[END_REF][START_REF] Burke | A graph-based hyper-heuristic for educational timetabling problems[END_REF]. To date, however, research on hyper-heuristics mostly focuses on discrete optimization and much less on continuous optimization. In particular, we are not aware of any previous approaches to address a dynamic selection of CMA-ES module variants.

We base our work on the modular CMA-ES framework introduced in [START_REF] Sander Van Rijn | Evolving the structure of Evolution Strategies[END_REF]. Within this framework, we can create many variants of CMA-ES by turning on or off certain modules, e.g. elitism, orthogonal sampling, and different weighting schemes (cf. Table 1). Each module combination gives a different configuration of the modular CMA-ES.

Using this modular CMA-ES framework, a theoretical analysis of the potential benefits of adaptive CMA-ES configurations has been provided in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF]. In this work, we will extend their study to all 24 BBOB functions [START_REF] Hansen | COCO: Performance Assessment[END_REF], quantifying the actual performance gains, identify a need for a more reliable prediction, and propose an alternative method, which shows a much better fit between predicted performances and actual achieved running times.

We focus on the case in which the CMA-ES variant can be changed only once. Our main results show that even such single switches allow for performance gains on 18 out of the 24 studied BBOB functions. In 11 cases the relative gain against the best static configurations are larger than 5%, 8 functions show a relative gain ≥ 10% and the largest observed improvement (for F21) is 51%. However, the 51% is achieved for one particular configuration swap only, and when averaging the 10 best adaptive configuration pairs, the relative gain over the best static variant vanishes. We nevertheless see other functions with stable relative gains. The largest stable gains of around 20% are obtained for F5 and F6.

All numbers reported above are with respect to the default parameter-choices. That is, we focus in this work exclusively on the algorithms selection aspect, thereby leaving the question of configuring the default hyper-parameters for future work. Additional performance gains can be expected from such parameter tuning, as, for example, the results presented in [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF] indicate.

Investigating the best-performing modules per each stage of the optimization process, we identify which configurations are most efficient for which parts of the optimization process.

Structure of the paper: Section 2 summarizes the modular framework, in which we build our adaptive CMA-ES. We introduce our implementation of the approach used in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF] in Section 3, followed by a discussion of our additional techniques for configuration selection. Section 4 shows that the original approach is not very stable. In Section 5 we discuss the results from our experiment, with a focus on the configurations and switching points at which the modules are changed. We conclude our contribution in Section 6 by looking at possible directions for future work.

THE MODULAR CMA-ES FRAMEWORK

The adaptive CMA-ES configurations we use were implemented in the modular CMA-ES framework introduced in [START_REF] Sander Van Rijn | Evolving the structure of Evolution Strategies[END_REF], which is freely available at [START_REF] Sander Van Rijn | Modular CMA-ES framework from[END_REF].

This framework implements 11 different modules. Of these 11 modules, 9 are binary and 2 are ternary, allowing for a combined total of 4,608 different configurations. The full list of available modules is shown in Table 1.

To clarify the terminology, note that we follow the example given in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF] and consider each combination of the 11 modules a "configuration". As mentioned in the introduction, we do not, in this work, change or optimize the hyper-parameters of the CMA-ES variants (such as the population size, the speed of the step size adaptation, etc.). We use the default values for these parameters instead.

For our benchmarking, we use the BBOB-suite of benchmark functions [START_REF] Hansen | Real-Parameter Black-Box Optimization Benchmarking[END_REF]. We used all noiseless, five-dimensional functions. Each BBOB "function" is a set of functions (commonly referred to as "instances") with similar fitness landscapes. In accordance with most comparative studies within the BBOB framework, we study the first five instances. For each of these instances, we run every configuration five times. This results in 2,649,600 total runs. The budget for these runs are set at 10 4 • D.

While this gives us data for a total of 4,608 different configurations, we ended up only using only one third of the configuration space in our adaptive experiments. This is due to the fact that

ADAPTIVE CONFIGURATIONS

Our main research question in this work is the following. For each of the 24 BBOB functions f we aim to estimate from the detailed runtime data of the 1,536 static configurations the triples (C 1 , C 2 , τ ) that exhibit the best performance for optimizing f . Each triple is interpreted as follows. The first configuration, C 1 , is run until hitting target value ("splitpoint") τ . Immediately after the first iteration in which a solution of fitness ≤ τ is sampled, we switch to configuration C 2 and run this configuration until the target 10 -8 is reached or the budget exhausted.

The long term vision of our research is to extend our study to a landscape-aware configuration. To this end, we will combine our data with explanatory landscape analysis, and to build an automated tool to select CMA-ES configurations on the fly. Put differently, we aim at extending the per instance algorithm configuration approach analyzed in [START_REF] Belkhir | Per instance algorithm configuration of CMA-ES with limited budget[END_REF] to the modular CMA-ES and towards a non-static selection. Rather than absolute performance gains, our main interests is therefore in identifying the best performing configurations per each function and each phase of the optimization process.

Performance Measures

The hitting time for a certain target is defined as follows: Definition 3.1 (Hitting Time). The hitting time of a target ϕ, a configuration c and a run i on a function f (written as HT (f , c, ϕ, i)) is defined as the first function evaluation during run i of c in which a function value with a difference of at most ϕ to the global optimum is observed.

In this paper, we use two different performance measures. The first is the average hitting time (AHT), which is based on the previously defined hitting time as follows: Definition 3.2 (AHT). The average hitting time of a configuration c for a target ϕ on function f is defined as:

AHT (f , c, ϕ) = ∞ if ∃i : HT (f , c, ϕ, i) = ∞ i HT (f ,c ,ϕ,i) i 1 otherwise
This method places an infinite penalty on non-finished runs. This can be beneficial when dealing with relatively few configurations to avoid selecting those which are unreliable. However, when dealing with more difficult functions and more runs, we want to deal with these non-finished runs differently. This can be done using the expected running time (ERT), which is defined as follows: Definition 3.3 (ERT). The expected running time (ERT) of a configuration c and target ϕ on a function f is calculated as follows:

ERT (f , c, ϕ) = i (F (f , c, ϕ, i) • b + HT (f , c, ϕ, i)) i (1 -F (f , c, ϕ, i))
, where b is the budget of each run (i.e., the maximum number of evaluations) and

F (f , c, ϕ, i) = 1 if run i on function f failed to reach the target ϕ, while F (f , c, ϕ, i) = 0 otherwise. Note that if each run of a configuration c on function f found a solution x with f (x) -f opt ≤ ϕ, then ERT (f , c, ϕ) = AHT (f , c, ϕ).

Selecting Adaptive Configurations

We gather the AHTs for targets

Φ = {10 2-(0.2•i) | i ∈ {0 . . . 50}}.
Based on these AHT values, the adaptive configurations suggested in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF] are chosen as follows:

• For each configuration c, each of the 24 BBOB functions f , and each of the 51 target values ϕ, we calculate the AHT over all 25 runs (5 runs for each of the first five instances). • From this data, we determine the best target value ϕ min for which there exists at least one configuration whose 25 runs all reached this target. • For every target value ϕ ∈ Φ satisfying ϕ > ϕ min we calculate the best configuration before this target, i.e., we select the configuration c for which AHT (c, ϕ) is minimized. We denote this configuration C 1 . We then compute the best configuration c from this target until ϕ min , which we denote as C 2 , i.e., C 2 is the configuration for which AHT (f , c, ϕ min ) -AHT (f , c, ϕ min ) is minimized. In [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF], the theoretical performance (TH for 'theoretical hitting time') is then calculated as

T H (f , C 1 , C 2 , ϕ) = AHT (f , C 1 , ϕ) -AHT (f , C 2 , ϕ) + AHT (f , C 2 , ϕ min ).
• From this data we compute the target value τ for which the overall performance

T H (f , C 1 , C 2 , ϕ) is minimized. This gives us the adaptive configuration (C 1 , C 2 , τ ).
We refer to τ as the 'splitpoint' of the adaptive configuration. We illustrate this approach in Fig. 1. In this figure we can clearly see that following C 1 until the splitpoint and then switching to C 2 gives an adaptive configuration which reaches the target with less evaluations than the best static configuration.

Mitigating Uncertainty in the Selection

Initial experiments showed a poor match between the predicted and achieved performance of the adaptive configurations as selected by the previously described procedure. A relatively large variance among the performance profiles of the original runs was identified as cause for this. To select good adaptive configurations despite this variance in the original data, we use the following two techniques:

Sliding window: Instead of considering only the performance at a single target and choosing that as the splitpoint to use, we consider a sliding window around the target, and take the average of the performances at all these targets. This should deal with variance by smoothing out the hitting times. Using this method and the AHT, the value we assign to a splitpoint and configuration pair is defined as follows: Definition 3.4 (Sliding window value). For a function f , adaptive configuration (C 1 , C 2 , ϕ i ) and using AHT as a performance measure, we define the sliding window value SW V as follows:

SW V (f , C 1 , C 2 , ϕ i ) = ∞ if i < w or (50 -i) < w i+w j=i-w HT (f , ϕ j , C 1 , C 2 ) otherwise
Here, w determines the radius of the window, in our case we use w = 2 for a window of size 2w + 1 = 5.

Worst Case: Instead of considering the mean performance, we take a version of the worst-case performance. To still deal with the different instances, we take the worst-case performance for every instance and average these to get a performance value. This should reduce the impact of the variance, since even when we assume to get bad runs, this configuration performs the best out of all possible adaptive configurations. This way, the variance might improve the performance of the adaptive configuration.

We run our tests with 3 different settings: sliding window using means, using worstcase, and the original method.

Two-Stage Configuration Selection

In addition to these two techniques, we introduce an entirely new procedure to more robustly select which adaptive configurations to run. This is based on the finding that the static configurations are not quite stable enough to use as a baseline. The first step in this process consists of selecting some static configurations for which we should gather more data. The configurations we will consider are made up of two parts. The first part consists of the 50 best performing static configurations. 1 We then extend this set by looking at the configurations which have been selected to be a part of the 50 theoretically best adaptive configurations. Since this might not be a diverse set of configurations, as one configuration might be chosen as C 1 50 times, we decide to limit the amount of times a certain configuration can be selected as C 1 and as C 2 to three times each ('limited selection method'). This should give us a more diverse set of configurations which might contribute to good adaptive configurations. We then rerun these configurations using 50 runs on each of the 5 instances, for a total of 250 runs each. In total, this set of configurations will contain anywhere from 50 to 150 configurations to rerun. We can then compare the ERT of the reruns to the original data. This is done for F10 in Fig. 2, from which we can see that this fit is indeed not very good.

Based on the new, more robust data set, we can now determine the best adaptive configurations to run. This is done using the means and a sliding window method. We still use a sliding window, but instead of taking the worst-case runs for every instance, we revert back to using the AHT. This is done because we have a lot more data available, and the mean should be more stable. We still use the same limited selection method as described previously.

INITIAL EXPERIMENTS

As previously mentioned, our first aim was to extend the work presented in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF]. We recall that in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF] only 4 selected functions have been considered. We extend this to all 24 BBOB-functions. We then run the adaptive configurations suggested by this approach, and observe that the fit between the theoretical performance prediction and the actual running times is not very stable.

Using the approach mentioned in the Section 3.2, we can calculate the theoretical improvements when using adaptive configurations. These results are not shown here, but are available in [START_REF] Vermetten | Online Selection of CMA-ES Variants[END_REF]. Note that our results differ slightly from those found in the original paper [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF]. The reason for this is twofold: First, the configuration space we consider is more restricted since we do not use the (B)IPOP module. Second, we added another restriction on which configurations to use in the adaptive configurations. We require both C 1 and C 2 to reach the target ϕ min . This is done to avoid situations where a C 1 is selected which very quickly converges to a local optimum and gets stuck there. In practice, switching to C 2 would then still result in being stuck in this local optimum, which would not be a benefit to finding the global optimum.

Overall, we found expected improvements to range from 0 to 15%. The 0% improvements are caused by the fact that for some functions, only one or two configurations manage to reach the selected target value. So in these cases, we have C 1 = C 2 , leading to the same performance for the adaptive and static configuration. This is present in 4 out of the 24 functions, while all others do have some level of expected improvement.

When running the adaptive configurations,2 we decided on running them on the same 5 instances as the original data, but instead of the 5 runs per instance which we used to gather the data for the static configurations, we run them 50 times per instance. This should give us more robust results to work with.

The resulting ERTs are compared to the ones we expected in Fig. 3. This comparison shows a clear mismatch between the actually achieved and the expected ERT values. This indicates that the original approach suggested in [START_REF] Van Rijn | Towards an Adaptive CMA-ES Configurator[END_REF] and summarized in Section 3.2 is not very reliable.

RESULTS

We next show how the ideas presented in Sections 3.3 and 3.4 yield more robust results.

In a first step, we ran the adaptive configurations determined by the additional methods described in Section 3.3. The results from running these adaptive configurations showed that these ideas alone are not sufficient to obtain reliable performance predictions. More detailed information about these experiments can be found in [START_REF] Vermetten | Online Selection of CMA-ES Variants[END_REF]. We summarize the main findings.

In total, we manage to see improvements (relative to the rerun static configuration) on 10, 8 and 13 functions for the original, sliding window with AHT, and sliding window using worst-case methods, respectively. That is, the sliding window using the worstcase approach performs slightly better than the other two methods we tested. We expect this to be caused by the fact that this method is less prone to major decreases in performance as a result of variance.

Even though we do see improvements for several functions, all methods have significant outliers, both in positive and negative sense. This seems to be caused by the fact that the ERT of the static configurations is not robust. As shown in Fig. 2, the ERT on the 5×50 runs can deviate significantly from the original ERT calculated based on 5 × 5 runs. And since the static configurations do not have robust ERTs, the adaptive configurations which are based on these values are not necessarily reliable. This lead us to implementing the two-stage approach, as described in Section 3.4.

Comparison of different methods

We next compare the resulting ERT using three different methods: the original method from Section 3.2, the worst-case method using sliding window from Section 3.3 and the two-stage method from Section 3.4. Since the two-stage approach selects and evaluates 50 configurations instead of just one, we show both the ERT of the configuration with the best predicted performance, as well as the one which performed best out of all 50 selected configurations. This is shown in Fig. 4. From this figure, we can see that for most functions, an improvement relative to the static configuration is achieved for at least some configuration selected in the two-stage method. However, this does not always correspond to the configuration which was predicted to perform best. In the remainder of this paper, we focus on the two-stage approach to see how these performances can be achieved.

Performance Comparison

The results of the two-stage method are shown in more detail in Fig. 6 for F10. From this figure, we can see that the fit between theory and practice is quite good, and many of the adaptive configurations manage to outperform the best static configuration by around 10%. Some outliers are present, but the general trend is positive. In this figure, we also note the ERT of the best "common" CMA-ES variant. The set of these configurations is taken from [START_REF] Sander Van Rijn | Evolving the structure of Evolution Strategies[END_REF] and shown in Table 2. Table 3 summarizes the results of the two-stage method for all 24 benchmark functions. The first notable result from this table is the fact that the best "common" static configuration can outperform the general best static. This is caused by the fact that these common configurations can have (B)IPOP enabled, which is not the case for the best static. In these cases, we assume that this (B)IPOP module is important to finding the optimum, and an adaptive configuration without this module will not be able to perform very well.

Next, we consider the functions for which the best static ERT is lower than that of the common variants. For these functions, we manage to improve upon this best static configuration when using an adaptive configuration. More specifically, we can see that when the best static configuration from the entire configuration space does not have (B)IPOP enabled, we can reliably achieve an improvement when using adaptive configurations.

We also note that when the best static configuration with (B)IPOP significantly outperforms the best rerun configuration, we do not manage to get the same improvements. If we would consider the best static configurations to include those with (B)IPOP and compare the performance of the adaptive configurations to those, no improvement is made at all.

In total, we find performance gains on 18 out of 24 functions of the BBOB benchmark, with stable advantages of up to 23%.

Module Activation Plots

We will now study two functions in more detail. Plots and values for other functions are available in [START_REF] Vermetten | Online Selection of CMA-ES Variants[END_REF]. The functions we will analyze are F10, for which we see a decent improvement for most adaptive configurations, and F24, for which we see very negative results.

First, we look at which static configurations have been selected, and how they are used within the adaptive configurations. To do this, we introduce what we call combined module activation plots. These plots consist of two parts, corresponding to C 1 and C 2 respectively. In each of these subplots, every line indicates a configuration. The lowest line corresponds with the theoretically best adaptive configuration, increasing from there.

In Fig. 5 and7 we see these combined module activation plots for the selected adaptive configurations for F10 and F24 respectively. These figures clearly show that for F10 there is a pattern present among the adaptive configurations: the modules TPA and threshold start activated and in almost all cases get turned off after the splitpoint. Such patterns are not present in the adaptive configurations for F24. This seems to indicate that for F24 the splits are mostly chosen because of small variances between the different configurations, instead of actual inherent properties of the configurations to perform well at certain points of the search.

For the other BBOB-functions, we gathered the same data. In Fig. 8 we plot a heatmap of the module activity in the 50 best adaptive configurations for all functions we considered. For every function and all binary modules, we indicate in how many (out of 50) configurations they are selected, while differentiating between C 1 (left) and C 2 (right). From this figure, we can see that for some functions, there are clear differences between the two parts. These functions correspond to those in which the adaptive configurations improved upon the static ones. For the other functions, the differences between the two parts are not as pronounced.

This conclusion is supported by Fig. 10, where we show the relation between module difference and the fraction of adaptive configurations which do improve upon the best static configuration. The module difference is calculated as follows:

M = 9 max j=1 50 i=1 |C 1 (i, j) -C 2 (i, j)| 50
Here, we have C 1 (i, j) indicating the value of module j of the configuration chosen as C 1 in adaptive configuration number i, and the same holds for C 2 (i, j). We only use the binary modules in this calculation, so the sampler is ignored. Based on this figure, we can see that there seems to be a correlation between the module difference percentage and the potential of the adaptive configurations. This supports our claim that the functions where few modules are consistently changed do not exploit any inherent properties of the functions, and instead rely on small differences in search behaviour to predict improvements. While for the functions which do see larger module differences between the first and second part, the improvements are more likely to be based on inherent properties of the function.

Summary of Results

From our experiment, we found large differences in the potential of our approach between functions. For some functions, such as F10, our approach seems quite stable, resulting in improvements of over 10% for several adaptive configurations, as can be seen in Fig. 6. However, this is not representative of all functions. We can see from Table 3, in the column "Amount of impr (% of configs)", that there are many functions for which few if any adaptive configurations manage to outperform the static configurations. The total improvements gained over the static configurations can be seen in Fig. 9. In this figure, we can see the relative differences between the best static and best adaptive configurations. We can see that, for the functions where the non-IPOP common static configuration performs similarly to the general best common static ones, the adaptive configuration manages to outperform the static one. This is in line with our previous comments, indicating that the lack of (B)IPOP modules in our configuration space is a large part of the reason why some functions do not see the amount of improvement we would have hoped for.

To verify that the negative improvements are indeed not caused by our approach itself, we we compare the best expected and achieved ERT of the adaptive configurations for each function. This is shown in Fig. 11. We can see that the fit between achieved and predicted ERT is quite good, with some negative outliers for the functions with relatively high predicted ERT. This seems to confirm that our methodology is solid, and that the reason we are not seeing improvement for some functions is most likely caused by the limited configuration space and evaluation budgets.

CONCLUSION AND FUTURE WORK

In this work, we have shown the potential for configuration adaptation in CMA-ES to outperform static configurations. We have determined that for more difficult functions (B)IPOP might be required to be able to achieve improvements, but for the easier functions this module is not required.

Since this work breaks the black-box assumption of optimization by including knowledge of the value of the global optimum, which we need to determine when the target τ is hit to switch configurations, it is not directly applicable to real-world problems.

A next step would be to implement some form of landscape analysis and base the splitpoint and configurations on features of the function. We would expect the improvement on real-world problems to be larger than those we achieved on the BBOB-suite of functions, since we expect that the transition in the local fitness landscape as seen by the algorithms is more drastic than in the "sterile" BBOB functions. A cross-validation of our approach on problems found in practice forms therefore another step that we plan to work on in the near future.

Another approach would be to extend the framework we used with more modules, and allow the (B)IPOP module to be used in adaptive configurations. This would greatly increase the configuration space, and might necessitate a different approach to determining which configurations to consider.

Finally, we plan on extending our work by tuning the hyperparameters of the most successful CMA-ES configurations. 3: Overview of the results from the experiment described in Section 3.4. The "Common config" and "Common config (non IPOP)" are selected from the configurations in Table 2. The column "Best static (B)IPOP" and its ERT ("Best static ERT") are selected from the full configuration space using the 5 × 5 run data. The remaining ERTs are from the 5 × 50 runs, as described in Section 3.4. The columns "Best adaptive" and "Predicted adaptive ERT" do not necessarily relate to the same configuration. When this is not the case, it is indicated by the italic values in the predicted column. The column "Relative improvement" shows the relative improvement in ERT of the best adaptive configuration vs the best rerun static configuration. The column "Avg 10 splits ERT" is calculated as the average of the 10 best achieved ERTs of the adaptive configurations. The column "Fraction of impr. configs (%)" indicates how many of the adaptive configurations performed better than the "Best rerun" configuration.

Figure 1 :

 1 Figure 1: Example of a theoretical adaptive configuration performing better than a static configuration (on F5).

Figure 2 :

 2 Figure 2: ERT of the selected static configurations for F10. The predicted values are based on the original 5 × 5 runs, while the achieved ones are on 5 × 50 runs.

Figure 3 :

 3 Figure 3: Comparison of the expected ERTs (logscale) for the adaptive configurations against the achieved values. Percentage differences are shown as well.

Figure 4 :F10: Combined module activation for C 1 a c t iv e e li t is t m ir r o r e d o r t h o g o n a l s e q u e n t ia l t h r e s h o ld t p a s e le c t io n w e ig h t s s a m p leFigure 5 :

 415 Figure 4: Relative improvement in ERTs of the adaptive configurations over the best static configuration.

Figure 6 :

 6 Figure 6: F10: ERT of adaptive configurations compared to the best static and "common" static configurations.

Figure 7 :

 7 Figure 7: Combined module activation plot for the 50 best adaptive configurations for F24.

Figure 8 :

 8 Figure 8: Heatmap of the module activations in the 50 best adaptive configuration for all our functions. For each function, activations for C 1 are on the left, while those for C 2 are on the right. The black lines separate the functions from each other.

Figure 10 :

 10 Figure 10: Relation between the maximum module difference % and the fraction of configurations which manage to improve relative to the best static configuration.

Figure 11 :

 11 Figure 11: Relation between the best predicted and best achieved ERT for adaptive configurations. Blue dots represent functions for which the average of the 10 best achieved ERT is lower than the best static ERT, while yellow dots indicate the other functions.

Table 1 :

 1 Overview of the CMA-ES modules available in the used framework. The entries in row 9 specify the formula for calculating each weight w i .

	#	Module name	0		1	2
	1	Active Update [20]	off		on	-
	2	Elitism	(µ, λ)		(µ+λ) -
	3	Mirrored Sampling [9]	off		on	-
	4	Orthogonal Sampling [29]	off		on	-
	5	Sequential Selection [9]	off		on	-
	6	Threshold Convergence [24] off		on	-
	7	TPA [12]	off		on	-
	8	Pairwise Selection [1]	off		on	-
	9	Recombination Weights [3]	log(µ+ 1 2 )-	log(i ) j w j	1 µ	-
	10 Quasi-Gaussian Sampling	off		Sobol Halton
	11 Increasing Population [2, 13] off		IPOP	BIPOP
	switching between configurations with a different setting for in-
	creasing population ((B)IPOP) would possibly create an information
	deficit, in that we do not know what population size we should
	have at the moment of switching. To eliminate this uncertainty,
	we decided not to consider (B)IPOP in our adaptive experiments,
	i.e., we dropped two options out of three. This leaves us with 1,536
	configurations to consider.

Table 2 :

 2 Common CMA-ES Variants. A selection of ten common CMA-ES variants is listed here, taken from[START_REF] Sander Van Rijn | Evolving the structure of Evolution Strategies[END_REF].

	Variant	Representation Short name
	CMA-ES	00000000000	C M 0
	Active CMA-ES	10000000000	C M 1
	Elitist CMA-ES	01000000000	C M 2
	Mirrored-pairwise CMA-ES	00100001000	C M 3
	IPOP-CMA-ES	00000000001	C M 4
	Active IPOP-CMA-ES	10000000001	C M 5
	Elitist Active IPOP-CMA-ES	11000000001	C M 6
	BIPOP-CMA-ES	00000000002	C M 7
	Active BIPOP-CMA-ES	10000000002	C M 8
	Elitist Active BIPOP-CMA-ES 11000000002	C M 9

  Comparison of the best achieved split ERT relative to the ERTs of the common statics (with and without IPOP; 5 × 5 runs) and the best non-IPOP static on 5 × 50 runs. Improvements are cut of at 60% and -50%, respectively. The precise values of the improvements are shown above the x-axis for the improvement relative to the best static (top) and relative to the best common (bottom) configuration. The precise ERT values can be found in Table3

		50															vs common
	Improvement %	-50 -25 0 25	1 1% 44%	2 15% 47%	3 1% 49%	4 6% -43%	5 24% 35%	6 21% 37%	7 14% 9%	8 1% 16%	9 -0% 23%	10 12% 40%	11 9% 41%	12 6% 28%	13 -9% 2%	14 3% 46%	15 -4% -35% vs non-IPOP common 16 17 -4% -65% 4% -338% vs best static	18 -19% 100%	19 13% 100%	20 10% 3%	21 51% 100%	22 21% -179%	23 100% 5%
													Function ID
	Figure 9:

The best static configurations are determined by their AHT at the final reached target. If fewer than 50 configurations reached this target for a function, we extend these configurations by the ones that have the lowest AHT for the previous target. We repeat this process until we have selected 50 configurations.

Implementation detail: For all our experiments, when switching between different configurations, we reset all static parameters that depend on the configuration. The dynamic parameters such as population, covariance matrix, etc. are kept from the first configurations.

ACKNOWLEDGMENTS

This work was supported by the Paris Ile-de-France Region. Our experiments were run on the Leiden DSLAB and DAS-4 [5] servers.


Mathieu Chollet
email: mchollet@usc.ict.edu

Talie Massachi
email: tmassachi@usc.ict.edu

Stefan Scherer
email: scherer@usc.ict.edu

Investigating the Physiological Responses to Virtual Audience Behavioral Changes A Stress-Aware Audience for Public Speaking Training

Keywords: Virtual Audience, Physiological Signals, Challenge and Threat

, however research on the topic of the effect of virtual audience behaviors has been scarce. In particular, it is unclear how variations in audience behavior can make the user's stress levels vary while they are presenting. In this paper, we present a study where we intend to investigate the relationship between virtual audience behaviors and physiological measurements of stress. We use the Cicero virtual audience framework which allows for precise manipulation of its perceived level of arousal and valence by incremental changes in individual audience members behaviors. Additionally, we introduce a concept of a stress-aware virtual audience for public speaking training, which uses physiological assessments and virtual audience stimuli to maintain the user in a challenging, non-threatening state.

Introduction

Public speaking can be a very stressful task, to the point where some individuals report it as their number one fear [START_REF] Bodie | A racing heart, rattling knees, and ruminative thoughts: Defining, explaining, and treating public speaking anxiety[END_REF][START_REF] Furmark | Social phobia subtypes in the general population revealed by cluster analysis[END_REF]. However it is also a very common task that we are often required to perform in our lives, and good public speaking skills can help an individual in their professional career as well as in their personal lives. As such, tools that allow users to improve their public speaking skills, or to reduce the amount of stress they experience while performing public speaking, are very desirable. Virtual audiences were first studied for treating public speaking anxiety in the early 2000's [START_REF] Harris | Brief virtual reality therapy for public speaking anxiety[END_REF][START_REF] North | Virtual reality therapy: An effective treatment for the fear of public speaking[END_REF][START_REF] Pertaub | An experiment on public speaking anxiety in response to three different types of virtual audience[END_REF]. Since then, a number of experiments have been realized that have shown that exposure to virtual audiences can reduce public speaking anxiety [START_REF] Bissonnette | Evolution of music performance anxiety and quality of performance during virtual reality exposure training[END_REF][START_REF] Safir | Virtual reality cognitive-behavior therapy for public speaking anxiety: one-year follow-up[END_REF][START_REF] Wallergård | A virtual reality version of the trier social stress test: a pilot study[END_REF][START_REF] Wiederhold | Social anxiety disorder: Efficacy and virtual humans[END_REF].

However, the scenarios used in psychotherapy often do not use the full potential of virtual humans, as the virtual audiences used are often scripted or controlled by an experimenter. In particular, multimodal interaction technologies now allow for the real-time perception of user behaviors and physiological states, and recent advances in the field of virtual humans enable a high realism level and precise behavioral control. Pertaub et al. studied the influence of virtual audience valence (i.e. negative, neutral or positive) on the stress level of participants, and shown that the type of valence influenced the stress response of participants. However, the valence of the audience did not change through the experiment and it is unclear how changes in audience behavior affect user's physiological states.

In previous work, we proposed to use virtual audiences for training public speaking skills. We introduced the Cicero public speaking training system [START_REF] Chollet | Exploring Feedback Strategies to Improve Public Speaking: An Interactive Virtual Audience Framework[END_REF], which uses a virtual audience to produce feedback in the form of non-verbal behaviors from the audience. Recently, we investigated the perception of virtual audience behaviors and how virtual audience comprising members of disparate states (e.g. two positive, four negative) are perceived, and found that the overall perception of the audience could be precisely manipulated by varying the behaviors of individual audience members [START_REF] Chollet | Manipulating the perception of virtual audiences using crowdsourced behaviors[END_REF]. In this paper, we propose to use the Cicero virtual audience framework to investigate more systematically how different types of virtual audiences behaviors can affect users' physiological states, and how varying these behaviors in real-time can be used to influence their states during a training or exposure session. In the next section, we begin by presenting related works on public speaking training using multimodal technologies, and studies on the effect of virtual audiences for reducing public speaking anxiety. We then present the Cicero virtual audience framework, which can be used for finely controlling the stimuli perceived by a user when they are practicing their public speaking. In section 4, we propose an experimental framework for studying the effect of varying virtual audience behavioral stimuli on users. Finally, we present in section 5 a concept for a stress-aware interpersonal skills training system which uses virtual humans stimuli and physiological assessment to maintain users in a challenging, nonthreatening state.

Related work

Multimodal interaction systems for training public speaking skills are becoming a popular research topic numerous systems have been introduced in the recent years which offer various approaches to it. The Rhema [START_REF] Tanveer | Rhema: A real-time in-situ intelligent interface to help people with public speaking[END_REF] and Logue [START_REF] Damian | Augmenting social interactions: Realtime behavioural feedback using social signal processing techniques[END_REF] systems use Google Glass to provide feedback to public speakers while they speak. Schneider et al. introduced the Presentation Trainer system [START_REF] Schneider | Presentation trainer, your public speaking multimodal coach[END_REF] which uses visual and haptic feedback (using a vibrating wristband) to provide feedback to trainees on mistakes (e.g. crossing arms, inappropriate volume) they make while presenting. Barmaki and Hugues presented a system for training teachers to adopt better body postures, using a virtual classroom populated with manually controlled virtual students augmented with a feedback screen displaying red or green stimuli depending on the user's body posture [START_REF] Barmaki | Providing real-time feedback for student teachers in a virtual rehearsal environment[END_REF]. Tanveer et al. [START_REF] Tanveer | Automanner: An automated interface for making public speakers aware of their mannerisms[END_REF] introduced Automanner, a training system which focuses on detecting and providing feedback on presenters' mannerisms, i.e. body movements frequently exhibited by a speaker, often unbeknownst to them.

A particular paradigm for interfaces for public speaking training is the virtual audience. Such a system aims at reproducing a public speaking situation with high fidelity, using an environment that is typical of public speaking situations and populating it with virtual characters acting as listening to the user. Virtual audiences have first been investigated to treat public speaking anxiety. North et al. found in a series of studies that virtual audiences were effective in reducing public speaking anxiety [START_REF] Harris | Brief virtual reality therapy for public speaking anxiety[END_REF][START_REF] North | Virtual reality therapy: An effective treatment for the fear of public speaking[END_REF]. Researchers also investigated the effect of three different types of virtual audiences, namely a neutral, a positive, and a negative audience, consisting of eight virtual characters [START_REF] Pertaub | An experiment on public speaking anxiety in response to three different types of virtual audience[END_REF]. They showed that the three settings had an influence on participants, generating anxiety even in participants who were not particularly afraid of public speaking situations, underlining the immersive characteristic of such virtual audiences. A virtual reality version of the Trier Social Stress Test (TSST) was investigated in [START_REF] Wallergård | A virtual reality version of the trier social stress test: a pilot study[END_REF], with three virtual characters being used instead of three human actors, and the authors found that participants' heart rate patterns were similar to those of participants enrolled in real instances of the TSST. A randomized clinical trial was realized by Safir et al. [START_REF] Safir | Virtual reality cognitive-behavior therapy for public speaking anxiety: one-year follow-up[END_REF]. 88 participants were randomly assigned between 3 conditions: a waiting list, cogntive-behavior therapy (CBT) with imagination (participants had to imagine a public speaking situation), and CBT with virtual reality exposure including virtual audiences. The experiment included 12 standardized sessions of one hour. Using self-rating anxiety questionnaires, the authors found a statistically significant reduction of anxiety in both CBT groups. A follow-up one year after the treatment found that the reduction in anxiety was maintained for both groups, which is very promising. While there was no difference in anxiety reduction between the two CBT groups, the group that underwent CBT with virtual reality suffered much lower attrition rates (15 dropouts in the imagination group, 6 in the VR group). Another group realized a large scale clinical trial comparing a group undergoing CBT with virtual reality exposure with virtual audiences and a group undergoing group exposure therapy [START_REF] Price | Outcome expectancy as a predictor of treatment response in cognitive behavioral therapy for public speaking fears within social anxiety disorder[END_REF] and found similar results: both groups benefited from a significant reduction in public speaking anxiety, maintained a year after the treatment. No difference was found between the virtual audience condition and the traditional group therapy condition. In a related domain, Bissonnette et al. studied the use of virtual audiences for reducing musicians' performance anxiety (MPA), successfully reducing it after 6 sessions of using the system over 3 weeks [START_REF] Bissonnette | Evolution of music performance anxiety and quality of performance during virtual reality exposure training[END_REF]. A review of the studies on the use of virtual audiences and virtual reality for treating social anxiety disorders can be found in [START_REF] Wiederhold | Social anxiety disorder: Efficacy and virtual humans[END_REF].

While all these studies show strong evidence that virtual audiences are a valuable tool for mitigating public speaking anxiety, all the audiences used suffered from limitations. They were either static in the type of feedback they produced, or manually controlled by an experimenter. The effect of the behaviors of a vir-tual audience on the physiological state of users were not systematically studied.

In this paper we propose to investigate this aspect of virtual audience effects. In the next section, we introduce the virtual audience framework we intend to use for this study.

The Cicero Virtual Audience Framework

In previous work, we introduced the Cicero virtual audience framework [START_REF] Chollet | Exploring Feedback Strategies to Improve Public Speaking: An Interactive Virtual Audience Framework[END_REF]. The architecture of the system is presented in Figure 2. Our primary goal for this system was to study public speaking training. In a preliminary experiment to validate the potential of virtual audiences, we designed a behavioral reinforcement training scenario where the user would train one specific behavior at a time, and the virtual audience would produce feedback on this particular behavior. For example, one training scenario would be designed to help participants avoid pause fillers (i.e. hesitation words such as hmm or err ): when the participants produced a pause filler, some audience members would start to shake their head in disagreement. Conversely, if the participants did not utter any pause filler during a certain duration, the audience members would nod. The main idea behind this training paradigm would be that positive feedback would reinforce the users' tendencies to produce appropriate behaviors, whereas negative behaviors would inhibit them from producing them unwanted behaviors. The experiment produced positive results as we found the virtual audience system not only improved the participants' performance as evaluated by public speaking experts, but also fostered high engagement by participants, which presumably improves the likelihood of participants to sustain training with the system for prolonged periods of time. More recently, we studied the behaviors of the virtual audience more systematically, in order to find out which are the most relevant stimuli for public speaking training [START_REF] Chollet | Manipulating the perception of virtual audiences using crowdsourced behaviors[END_REF]. We realized a first experiment using a crowdsourcing methodology [START_REF] Ochs | A crowdsourcing toolbox for a userperception based design of social virtual actors[END_REF]: participants had to choose behaviors (i.e. postures, gaze patterns, head movements, facial expressions) for an individual audience member in order for it to express a particular level of arousal and valence, out of 5 possible levels (i.e. very low, low, medium, high, very high). We chose arousal and valence as previous studies showed that those are the primary dimensions that participants could identify in virtual audiences [START_REF] Kang | The design of virtual audiences: Noticeable and recognizable behavioral styles[END_REF]. Using the data we collected by crowdsourcing, we built a model for choosing audience members' behaviors based on their individual state. We then realized a second study in order to validate the overall perception of virtual audiences using our model. We found that we could incrementally change how participants perceived an audience by modifying the state of individual audience members: this means we can precisely control the stimuli received by a user while they are training their public speaking with our system. In this paper, we propose to use this result to investigate the effect of audience behavior change on physiological states.

Proposed experiment

The main idea behind the experiment is to expose participants to various successive changes in virtual audience behaviors while they are presenting. The system used in the experiment is illustrated in Figure 2. The architecture of the system that will be used in the experiment. The dotted feedback line represents an envisaged addition to the our public speaking training framework where physiological data would be looped back to alter the virtual audience states. We discuss this concept in Section 5.

Participants are recorded through a variety of sensors: a webcam (centered on their face for monitoring facial expressions), a Microsoft Kinect and a mi-crophone. Additionally, an Empatica E4 wristband1 is used in order to capture physiological measurements. One advantage of using a wristband is that it does not hinder participants' movements at all, leaving them to behave as they would do in a pubclis speaking situation. During the participants' presentations, the virtual audience's behaviors are manipulated. While the participants are told that the system chooses to adjust the virtual audience behaviors depending on their performance, the virtual audience behaviors are in fact fixed in order to follow trajectories defined by the condition they are in. For instance, in the first moments of a presentation, the virtual audience will change from neutral to positive, then back to neutral, and will finish negative. The research questions, protocol, virtual audience behaviors and measurements are described in more detail in the following subsections.

Research questions

In this experiment, we want to assess whether participants' experiences are affected by virtual audience behaviors. We will look at this question using three perspectives: a physiological perspective, a self-efficacy perspective, and a behavioral perspective. Specifically, we will look at the following questions:

R1 Are participants' stress levels affected by a change in virtual audience stimuli? R2 Are participants' perception of self modified by a change in virtual audience stimuli? R3 Do participants exhibit different behaviors after a change in virtual audience stimuli?

For R1, we will utilize physiological measures to assess participants' stress levels after different trajectories of virtual audience behaviors. For R2, we will a self-reported measures from participants. Finally, for R3, we will the recorded use audio-visual data.

Virtual audience behaviors

In order to study the effect of virtual audience behavioral changes on users' physiological states, we designed a software component, the Audience Controller, that transitions the state of the overall audience from a starting state S i to a target state S t over a period of time of duration d. We validated in [START_REF] Chollet | Manipulating the perception of virtual audiences using crowdsourced behaviors[END_REF] that we can adjust the perceived overall state by incrementally modifying the state of its individual members; therefore the Audience Controller adjusts individual audience member states over time in order to reach the target audience state. We designed this component so that a transition instance from S i to S t over time d can be saved and replayed: this allows us to expose different participants to the exact same stimuli.

Using this component, we recorded different audience behavior sequences, that we call trajectories. For reducing the amount of variables in the experiment, we fixed the arousal level at a medium value, and only investigated valence changes. We also fixed the duration of transitions at 45 seconds. We defined 6 trajectories: LN (low to neutral), LH (low to high), N L (neutral to low), N H (neutral to high), HL (high to low) and HN (high to neutral).

Measurements

Our primary mode of investigation will by the physiological data recorded by the Empatica E4 wristband, which consists of measurements for temperature, electrodermal activity, heart rate, and acceleration. Additionally, we will be recording every interaction using a webcam for facial expressions, a Microsoft Kinect for full body behavior, and a microphone for vocal behavior. These 3 channels of recording will be synchronized together and with the E4. We intend to use these additional modes of recording to perform multimodal analyses.

Questionnaires will also be used. At the beginning of a session, every participant will fill in a demographics questionnaire, the Personal Report of Confidence as a Speaker [START_REF] Paul | Insight vs. Desensitization in Psychotherapy: An Experiment in Anxiety Reduction[END_REF], which allows use to get a self-rated measure of public speaking anxiety, and the 10-item Big Five Inventory [START_REF] Rammstedt | Measuring personality in one minute or less: A 10-item short version of the big five inventory in english and german[END_REF] for assessing participants' personalities. After each presentation, they will fill in the Positive and Negative Affect Schedule [START_REF] Crawford | The positive and negative affect schedule (panas): Construct validity, measurement properties and normative data in a large nonclinical sample[END_REF]. At the end of the session, participants will also fill in the immersive experience questionnaire [START_REF] Jennett | Measuring and defining the experience of immersion in games[END_REF].

Finally, during the public speaking presentations, participants will be asked regularly to rate how they feel they are doing. For this to be minimally invasive, we will ask the participants to simply do a thumbs-up or thumbs-down gesture with one or both hands (no hands means neither positive nor negative).

Protocol

The experimental protocol is the following. When arriving for the experiment, they participants are fitted with the E4 Empatica wristband and are asked to sit and rest for five minutes as the experimenter prepares the software. This allows us to get baseline measurements for the participant. The participants then fills in a number of questionnaires. After this, the participants realizes four impromptu presentations: given a list of topics, the participants has to give his opinion in a speech after being given 5 minutes for preparation. During each of those presentations, the virtual audience is configured to behave following a fixed set of four trajectories. The set of trajectories constitutes the experimental condition, and is randomly chosen. Each of the trajectories last for 45 seconds. Between each trajectory, a popup appears on the screen, asking the participant to give their impression of their performance in a quick thumbs up/down gesture. After the first 3 minutes, the audience then picks trajectories randomly, and continues behaving until the participant is finished. Once the four presentations are finished, the participant fills in the last questionnaires, and is then debriefed.

Expected Outcomes

This experiment will allow us to study the three research questions outlined in subsection 4.1. We here describe a number of hypotheses that we will study in the data analysis that will follow the experiment.

Heart Rate Variability (HRV) refers to the variability between each successive inter-beat intervals, and prior research indicates that HRV may reflect emotion regulation [START_REF] Quigley | Emotional learning and mechanisms of intentional psychological change[END_REF]. Therefore, we hypothesize that participants with higher resting HRV will be less affected by negative audience trajectories compared to participants with lower resting HRV, as they will be better able to compensate for the negative stimuli:

H1a High resting HRV will be less correlated with higher heart rate after negative audience trajectories due to individuals' increased emotion regulation abilities.

Electrodermal activity, on the other hand, reflects current levels of stress [START_REF] Visnovcova | Heart rate variability and electrodermal activity as noninvasive indices of sympathovagal balance in response to stress[END_REF]. Therefore, we write our second hypothesis: EDA will rise when the audience changes to a more negative state, and lower after a trajectory where it is getting more positive.

H1b EDA will be lower after a increase in audience valence, and higher after a decrease in audience valence.

In terms of self-efficacy, the only measure that we will obtain is the gesture performed by participants between audience trajectories. We simply assume that the virtual audience affects the participants' perception of self, regardless of how they are actually performing: H2 Participants will display more thumbs-up gestures after a positive trajectory, and more thumbs-down after negative trajectories.

One other hypothesis we will study is that the type of audience stimuli will affect the participants' behaviors: if the audience is very positive, the public speaking performance of the speaker will improve. For instance, they make make less hesitations and display more positive facial expressions. As a first step, we will have annotators watch the videos and rate how they thought the participants did.

H3a Ratings of participants' performance will be higher for segments after a positive trajectory compared to segments following a negative trajectory.

Finally, recent advances in multimodal have allowed for the automatic assessment of public speaking. Using existing approaches such as [START_REF] Ramanarayanan | Evaluating speech, face, emotion and body movement time-series features for automated multimodal presentation scoring[END_REF], we will compute a performance score automatically for all segments of participants' presentations, and will study the following hypothesis:

H3b Predicted scores of participants' performance will be higher for segments after a positive trajectory compared to segments following a negative trajectory.

In the next section, we present a concept for a new type of public speaking training framework, which will utilize the results of the study presented above.

A Stress-Aware Virtual Audience for Public Speaking Training

In this section, we present a concept for a stress-aware public speaking skills training system. Using physiological measurements, the idea is to assess the stress level of the participant in real-time, and to adjust the stimuli produced by the virtual audience to maintain an appropriately challenging but not threatening training experience.

The theoretical framework behind this concept is Blascovich's challenge and threat framework [START_REF] Blascovich | Challenge and threat appraisals: The role of affective cues[END_REF]. The main idea behind this framework is that performance situations will be seen by an individual as challenging or threatening depending on cognitive and affective appraisals on the demands required by the situation and the resources that the individual has at their disposal to meet these demands. By performance situations, we mean situations where an individual is evaluated, either by others or oneself, and where this evaluation determines the outcome of a relevant goal for the individual, for instance career goals. Public speaking is often a good example of such a performance situation. The appraisals of a situation's demands refer to the perception of the situation's risks (e.g. losing one's job over a poor presentation), required effort or uncertainty, while the appraisal of resources refers to assessing what skills are deemed available in this situation (e.g. the participants' perception of their knowledge about the presentation topic). Challenge occurs when individuals believe they hold sufficient resources to meet the demands, and conversely threat occurs when those resources are seen as insufficient. Challenge and threat states affect patterns of cardiovascular activity, and as such can be assessed through physiological measures. Assuming that the hypotheses we formulated in section 4.5 are at least partly verified (i.e. virtual audience behavior changes affect the participants' states), we can frame the effect of the virtual audience stimuli as part of the challenge and threat framework. Our view is that changes in virtual audience stimuli affect the demands appraisals of the speaker; if the audience produces more positive feedback, then the user will see the situation as less emotionally demanding, and vice-versa. Using physiological measurements of the participants' heart rate, we can assess in real-time their psychological state, and adjust the virtual audience stimuli in order to keep them in a nonthreatening, challenging state. The concept of such a system is illustrated in figure 3. Such a concept should be applicable in any social situation that fits the requirements of the challenge and threat theory, that is, any evaluative performance situation which goals are important to the user. Therefore, a similar architecture should be usable for other applications such as job interview skills training or negotiations training.

Challenge/Threat Model

Conclusion

Virtual audiences have been shown to constitute an effective tool for helping treating public speaking anxiety and social anxiety disorders in general. However, the virtual audiences used in earlier experiments did not use the full capabilities of virtual humans, in particular they did not systematically investigate the role of audiences behaviors. In this paper we presented an experiment that is designed at evaluating the influence of virtual audiences behavioral changes on physiological states. We hope this research will help improve the understanding of how virtual audiences can be used for mitigating public speaking anxiety and training public speaking, and ultimately lead to better alternatives for patients and users. Additionally, we presented a concept for a public speaking training system that relies on the virtual audience's behaviors to manipulate the user's stress level, in order to keep them in a challenging, nonthreatening state.

We are currently running the study, and expect we may be able to show preliminary results at the workshop. In future work, we will process and analyze the data collect within the experiments, in order to study the hypotheses presented in section 4.5. Additionally, were those hypotheses to be verified, we will implement our concept for a stress-aware virtual audience and evaluate its potential for public speaking training. We believe this concept should be applicable to other interpersonal skills training scenarios, such as job interview training, and will also look at expanding our domain to those other settings.

Fig. 1 :

 1 Fig. 1: Screenshot of the Cicero virtual audience.

  Fig.2:The architecture of the system that will be used in the experiment. The dotted feedback line represents an envisaged addition to the our public speaking training framework where physiological data would be looped back to alter the virtual audience states. We discuss this concept in Section 5.

Fig. 3 :

 3 Fig. 3: Our concept of a stress-aware virtual audience.

https://www.empatica.com/e4-wristband

Acknowledgments This material is based upon work supported by the National Science Foundation under Grants No. IIS-1421330 and U.S. Army Research Laboratory (ARL) under contract number W911NF-14-D-0005. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or the Government, and no official endorsement should be inferred.