Alexey Brodoline

Nicolas Verrier

Ludovic Depreater

Didier Felbacq

Michel Gross

Quantitative study of open channels by digital holography

Quantitative Study of open channels by digital holography

Alexey Brodoline 1 , Nicolas Verrier 2 , Ludovic Depreater 1 , Didier Felbacq 1 Open channels are peculiar incidents modes of the optical field that are transmitted by a highly diffusing medium with high efficiency (100 % in the ideal case). The existence of these modes has been predicted by Dorokhov et al. [START_REF] Dorokhov | On the coexistence of localized and extended electronic states in the metallic phase[END_REF] more than 25 years ago. Although many experiments were made to prepare these modes [START_REF] Yu | Measuring large optical transmission matrices of disordered media[END_REF][START_REF] Popoff | Coherent control of total transmission of light through disorder[END_REF], the maxima of transmission obtained in experiments, are ever much lower than 100 %. To test the theory, we have considered here another Dorokhov's prediction: the average transmission of the modes that sustain the transmitted field must be TT=66%, where TT is defined by N2 TT= N1 T (where T is the averaged transmission of energy, and N1 and N2 the number of incident and transmitted modes i.e. the size of the incident and transmitted mode basis). To verify this prediction, we have measured N2 on a slab sample of ZNO powder (transverse size L<=70 m, thickness 20 m) [START_REF] Verrier | Measuring enhanced optical correlations induced by transmission open channels in a slab geometry[END_REF], by assuming that N1 is given by the sample geometry. The experimental setup is seen in Fig. 1(a). The incident fields are random (there are scrambled by a diffusing liquid DL) and digital holography is used to measure the complex transmitted field E(x,y) for the two polarisations. N2 is measured by analysing the residual correlations of the transmitted field measured for uncorrelated incident fields.

and Michel Gross

Since the average transmission is T=1/25, we expect N2=N1/16. To get this optimal result, the transverse size of sample L must be infinite and the defocusing SQ' of the microscope objective MO2 must be zero (where S is the sample outgoing surface and Q' the plane imaged by MO2). Figure 1 (b) shows the residual correlation (which is proportional to 1/N2) as a function these two parameters. We get N2=N1/12 without extrapolation, and N2=N1/16 with extrapolation to L infinite. Here, the open channel effect is huge (factor 16) and the aggreement with the Dorokhov's prediction is excellent.

We acknowledge Labex Numev (convention ANR-10-LABX-20) for funding.

Fig. 1

 1 Fig.1 (a) Holographic experimental setup. (b) Residual correlation of the transmitted field. The horizonal axis is SQ'. The sample transverse size is L=70 m red, 50 m green, 35 m blue.... The horizontal grey line is the residual correlation expected for N1=N2.

1 *

 1 1 Laboratoire Charles Coulomb -UMR 5221 CNRS-Université Montpellier Place Eugène Bataillon 34095 Montpellier (FRANCE)

	2 Laboratoire MIPS -EA 2332 IUT de Mulhouse (GEII) 61 rue Albert Camus 68200 Mulhouse (FRANCE)
	* michel . gross@umontpellie r. fr
	Citation: A. Brodoline, N. Verrier, L. Depreater, D. Felbacq, and M. Gross, "Quantitative Study of open channels by digital holography," in 2017 European Conference on Lasers and Electro-Optics and European Quantum Electronics Conference, (Optical Society of America, 2017), paper EG_2_5.
	European Quantum Electronics Conference 2017 : Munich Germany 25-29 June 2017 ISBN: 978-1-5090-6736-7 DOI : 10.1109/CLEOE-EQEC.2017.8087591
	From the session Complex Electromagnetic Fields (EG_2)


Rui Li
email: rui.li@ed.ac.uk

Chaoyun Zhang
email: chaoyun.zhang@ed.ac.uk

Paul Patras
email: paul.patras@ed.ac.uk

Razvan Stanica

Fabrice Valois
email: fabrice.valois@insa-lyon.fr

Learning Driven Mobility Control of Airborne Base Stations in Emergency Networks

Keywords: Emergency networks, mobility control, airborne base stations, deep reinforcement learning, AI in networks

Mobile base stations mounted on unmanned aerial vehicles (UAVs) provide viable wireless coverage solutions in challenging landscapes and conditions, where cellular/WiFi infrastructure is unavailable. Operating multiple such airborne base stations, to ensure reliable user connectivity, demands intelligent control of UAV movements, as poor signal strength and user outage can be catastrophic to mission critical scenarios. In this paper, we propose a deep reinforcement learning based solution to tackle the challenges of base stations mobility control. We design an Asynchronous Advantage Actor-Critic (A3C) algorithm that employs a custom reward function, which incorporates SINR and outage events information, and seeks to provide mobile user coverage with the highest possible signal quality. Preliminary results reveal that our solution converges after 4 × 10 5 steps of training, after which it outperforms a benchmark gradientbased alternative, as we attain 5dB higher median SINR during an entire test mission of 10,000 steps.

INTRODUCTION

Public safety and civilian operations intrinsically require stable wireless connectivity for both rescue services and postdisaster recovery. Contemporary military activities such as territorial search and emergency response also rely heavily on reliable data connections. Certain areas under extreme conditions, e.g. following earthquakes, floods, fire, and nuclear plant emergencies, are hardly accessible with legacy emergency cellular infrastructure carried on vans (i.e. cellson-wheels). Meanwhile, following recent hardware/software advances, commercially available unmanned aerial vehicles (UAVs) are increasingly used for various applications, including aerial imaging and asset inspection. As a result, regulatory bodies, such as the Federal Aviation Administration (FAA), defined rules to enforce the safe operation of commercial UAVs [START_REF]Drone test trials to shape FAA rules[END_REF]. The telecom industry also shows growing interest in deploying UAV-mounted base stations (BSs) for sporadic cellular services, with an emphasis on challenging use cases. For instance, following hurricane Marie's dev-astation of Puerto Rico, AT&T obtained FAA approval to fly UAVs for temporary cellular coverage [START_REF]FAA Approves Drone to Restore Puerto Rico Cell Service[END_REF]. Similarly, Altaeros rolled out SoftTower, an UAV carrying multi-sector LTE BS to provide connectivity to hard-to-reach areas [START_REF]Next generation global telecom infrastructure[END_REF].

Providing wireless connectivity in a large area to a sizeable group of users, including citizens and rescue teams (e.g. police force, medical personnel, or firefighters), often requires more than one airborne vehicle with networking capabilities. In contrast to cellular networks, where the BS deployment is carefully planed and conducted, BSs mounted on UAVs are mobile themselves. Coordination among flying BSs and movement control, to set up a core network, provide the needed coverage, and ensure sufficient and stable user data rates, is of paramount importance, whilst any signal failure can be catastrophic to critical missions. Stochastic wireless channels and user equipment (UE) movement uncertainty, however, render the BS mobility management task complex, involving an exponentially growing action space as the number of BSs increases. Traditional solutions, such as optimal control, require precise environment models, which are hardly obtainable in real-time and require strong assumptions that can compromise their usefulness. Heuristic alternatives only produce sub-optimal results.

In this paper, we tackle these challenges facing UAV mobility control with a deep reinforcement learning (DRL) approach. We devise a domain-specific reward function that encourages the UAV mobility control agent to provide high quality signal coverage to users, and we leverage an Asynchronous Advantage Actor-Critic (A3C) scheme to learn the optimal action policy via interaction with the wireless environment. Our design is motivated by the rapid convergence requirements specific to emergency settings. Simulation results demonstrate that our solution converges rapidly, and once trained, it makes accurate movement control decisions, outperforming a benchmark gradient-based scheme that has perfect knowledge of the stochastic channel. More precisely, we obtain a 5dB median SINR improvement, while only requiring current location and association information.

SYSTEM MODEL

We consider a fleet of B UAVs, each carrying one BS. The BSs run LTE protocol stacks with simplified functionality, e.g. disabled MME authentication, to simplify the overall architecture and prolong network lifetime. Each BS serves a number of UEs and is connected wirelessly (via satellite or µ-/mm-wave links) to a central controller. The controller hosts a DRL agent that learns to make optimal decisions about the BSs mobility control.

Wireless Channel: We consider BSs share the same frequency band, i.e. reuse factor 1. We focus on the downlink communication, assuming the SINR is directly related to the quality of service received by UEs. The transmit power employed by BS b to a user is P b and we denote G b,u the channel gain between BS b and user u, which is a linear combination of the free-space path-loss l b,u , shadow fading, and antenna gain Ga. The log-distance path-loss l b,u can be computed following the 3GPP model for urban cellular scenarios with standard coefficients α and β, i.e. l b,u = α + β log(D b,u ), where D b,u is the Euclidean distance between b and u. Given I b ⊂B\b the set of BSs that interfere with b and N0, the power of per-channel additive white noise, the SINR observed by UE u is:

SINR b,u = P b G b,u N0 + b ∈I b P b G b ,u (1)
UE Mobility and LTE Handovers: We assume a reference group mobility model, by which users cluster around group centres that move along random way points [START_REF] Hong | A group mobility model for ad hoc wireless networks[END_REF]. The motivation for employing this mobility model is that, in the envisioned emergency scenario, rescue and medical teams, or fire fighters, rush towards a target scene while the population may be moving away from that location. Further, as both UEs and BSs continuously change their position, we follow the standard LTE handover policy, i.e. employ hysteresis and time-to-trigger to avoid ping-pong effects. Specifically, when the received signal strength is below a certain threshold SIN R th for a duration of ttrigger, the user can be handed off to adjacent cells, if a new BS provides an SINR higher by σ than the value currently measured.

REINFORCEMENT LEARNING FOR AIRBORNE BS CONTROL

We address mobility control of airborne BSs in emergency settings, considering stochastic wireless channels and user mobility, as modelled in Sec. 2. While this is a challenging problem, deep reinforcement learning (DRL) has achieved promising results in similarly complex tasks, such as Atari game play [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] and adaptive video streaming [START_REF] Mao | Neural adaptive video streaming with pensieve[END_REF]. This motivates us to take a DRL approach, formulating our task as Markov Decision Process (MDP), and designing an A3C based solution tailored to our target networking scenario.

Markov Decision Process (MDP): The BSs mobility control can be modelled as a 5-tuple MDP, < S, A, P, R, γ >, where S is the state of the environment including BSs locations, LB(t), UEs locations, LU (t), and their associations, αB,U (t). A is the action taken by each agent, i.e. the movement direction of each BS, M b (t), and P is the state transition matrix. Precisely, the system moves from state s to s following action a according to P a ss = P[St+1 = s |St = s, At = a]. R is the reward function, which quantifies the system performance following an action, i.e. R a s = E[Rt+1| St = s, At = a]. In our problem, this will depend on the SINR experienced by the UEs, as we detail next. γ ∈ [0, 1] is the discount factor, which dictates the importance of future rewards.

A policy π is the probability distribution of taking an action a in a given state s, i.e. π(a|s) = P[At = a|St = s].

To attain an optimal policy π * , we employ a DRL method and give an overview of the learning procedure in Fig. 1. As shown in the figure, the agent updates a policy π. By this, at time step t, given state st including UEs and BSs locations and their associations, the agent takes an action at, according to which each drone b ∈ B moves from (x b , y b , z b ) to (x b , y b , z b ). The agent thereby receives a reward rt, and the system enters a new state st+1. This process is repeated until the episodic return, i.e. the sum of future discounted rewards, converges. This indicates that an optimal policy was found.

UAV Fleet

Proposed Solution: There exist two main approaches to solving control problems through reinforcement learning (RL): policy-gradient RL methods learn by 'trial-and-error' a policy, i.e. the probability distribution of actions to take given the state of the environment; value-based methods learn to estimate the value for each action. Combining these two approaches, an actor-critic RL agent employs an 'actor' that performs actions to improve the policy, while a 'critic' makes judgements on the actor's performance and learns to estimate the action values.

To tackle the UAV mobility control problem, we propose a custom asynchronous advantage actor-critic (A3C) algorithm. A3C is a state-of-the-art actor-critic method that exploits multi-threading to create several learning agents, each exploring the state space in their own environment and updating periodically a global neural network with learned knowledge [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. The key advantages of this approach are that it can be trained on a CPU, it de-correlates past experiences gained by each learning agent, and it converges rapidly. Such paralleled learning is a viable alternative to experience replay (i.e. Deep Q-Learning), as it removes large memory requirements.

In our A3C-based solution, we employ two neural networks that have the most simple deep learning architecture, i.e. the multi-layer perceptron (MLP). MLPs have fully connected layers and the neurons in the hidden layers implement non-linear activation functions of the output of the previous layer. The parameters (weights and biases) of these functions are obtained by training through backpropagation [START_REF] Lecun | Deep learning[END_REF]. In our case, one of the MLPs acts as the actor and the other as the critic. Both of them employ 2 hidden layers, each consisting of 200 neurons.

We design a reward function that captures the specifics of mobile users served by airborne BSs, aiming to ensure best connectivity, i.e. highest SINR and lowest outage likelihood. Hence, the proposed reward function is

R = θ * SINR - Nout NUE , ( 2
)
where SINR is the mean SINR computed across all the NUE users, θ denotes a normalising factor, and Nout is the number of UEs whose SINR is below a minimum service requirement.

In what follows we discuss how we train the proposed algorithm and present the results of the performance evaluation campaign conducted.

EVALUATION

To evaluate the proposed A3C algorithm in UAV mobility control scenarios, we examine its convergence properties and compare its performance with that of an SINR gradient based benchmark.

Simulation Setup

We consider a 100×100 grid area, with a grid cell width of 5m. Within this area, 40 users move in groups of 10 UEs, following the group reference model [START_REF] Hong | A group mobility model for ad hoc wireless networks[END_REF], and are provided with connectivity by BSs mounted on UAVs. For each user we compute the SINR of the link to the serving BS, using the model described in Sec. 2. A sample SINR heatmap and UE locations are illustrated on the left in Fig. 2. The histogram on the right shows the distribution of the SINR experienced by users in this instance. The UAVs move at a fixed altitude, i.e. 10m from the ground level. The movement of a BS at each time step is chosen between 4 candidate directions (i.e. N, S, W, E) towards adjacent points on the grid, or idling (no move).

We train the DRL model with 10 random seeds, each time with 1,000 training episodes. A single episode lasts 2,000 time steps and the locations the UAVs are reset to the same coordinates at beginning of each episode. At each discrete time step, the mobility control agent performs actions based on the current learning policy, and chooses from 5 4 = 625 possible actions to take (4 directions or movement plus idling, 4 UAVs). The simulation parameters used are summarised in Table 1. The model is trained and tested on a 8-core desktop with Intel Xeon W-2125 CPU clocked at 4.00GHz, and we employ Python libraries TensorFlow to implement the neural networks [START_REF] Abadi | TensorFlow: a system for largescale machine learning[END_REF].

Benchmark: To assess the performance of the proposed DRL solution, we devise a benchmark SINR gradient based method, and test it with the same network settings as with the proposed DRL approach. At each time step, this benchmark computes the average SINR at the associated UEs along each of the possible directions of movement. It then moves that UAV in the direction of the lowest average SINR. By this approach, the aim is to avoid outage while maintaining good signal quality for all the UEs served.

Simulation Results

Training Convergence: At the end of each episode, we compute the moving average of the episodic return R as:

R = 0.01 * rep + 0.99R, ( 3
)
where rep is the total reward of the episode. To examine the learning convergence, in Fig. 3 we plot the evolution of R. The training converges within 200 episodes (i.e. 4 * 10 5 steps), corresponding to approximately 2.5 hrs of training in real world. Such training can be performed once during pre-deployment stage, after which the agent can be used for multiple missions, given the wireless channel characteristics remain largely similar. Observe that during this phase the proposed A3C solution improves the average episodic return from around -50 to 100, with the minimum value being improved from -280 to 80. Once trained, the agent can be used directly to make decisions about BSs movement, according to the current conditions. Performance: Ultimately, we are interested in quantifying the performance gains our DRL approach can attain over other solutions, such as the SINR gradient-based benchmark considered. To this end, we test the trained neural network model over 10,000 steps, resetting the environment 4: CDF of the SINR attained by all users with the proposed DRL over 10,000 testing steps, after 100, 500, 1,000, and 2,000 training episodes (EPs), and respectively with the benchmark gradient method. every 2,000 steps, and examine the signal quality provided by both approaches. Specifically, in Fig. 4 we plot the cumulative distribution (CDF) of the SINR experienced by all users with the benchmark scheme and the proposed DRL algorithm, studying also the impact of the number of training episodes (i.e. 100, 500, 1,000, and 2,000 episodes) on the performance achieved.

The results confirm that our DRL solution achieves a 5dB improvement of the median SINR, over the gradient-based benchmark. If considering -5dB as the signal outage level, the proposed DRL scheme only experiences approximately 5% user outage, which is one forth of that experienced by the benchmark gradient method, i.e. 20%. The results further confirm that after convergence, our learning algorithm performs stably. Specifically, the distribution of the SINR after 2,000 training episodes achieves a median value that is only marginally better than that of an algorithm trained over 500 episodes. We conclude that the proposed DRL scheme for mobility control of airborne BSs attains more than 3× higher median SINR and 4× lower outage rate compared to the gradient-based benchmark. Furthermore, our solution performs stably after training. [START_REF] Orsino | Effects of Heterogeneous Mobility on D2D-and Drone-Assisted Mission-Critical MTC in 5G[END_REF]. Grossglauser and Tse study the per-user throughput in mobile ad-hoc networks and conclude that performance can be improved dramatically when BSs are mobile [START_REF] Grossglauser | Mobility increases the capacity of ad hoc wireless networks[END_REF].

RELATED WORK

Employing deep learning to solve complex networking problems is becoming a hot research area [START_REF] Zhang | Deep Learning in Mobile and Wireless Networking: A Survey[END_REF]. For instance, Li et al. leveraged supervised learning to optimise utility in backhaul networks [START_REF] Li | DELMU: A Deep Learning Approach to Maximising the Utility of Virtualised Millimetre-Wave Backhauls[END_REF] and Mao et al. employed a reinforcement learning technique for adaptive video streaming [START_REF] Mao | Neural adaptive video streaming with pensieve[END_REF]. Advances in deep reinforcement learning (DRL) such as the asynchronous actor-critic method (A3C) achieved remark-able performance in game-play applications, using only half the training time on a multi-core CPU [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. To our knowledge, our work is the first to employ DRL for airborne BS mobility control.

CONCLUSION

In this paper, we introduced a deep reinforcement learning solution for the mobility control of a fleet of UAV-mounted base stations, with the goal of providing reliable service coverage in scenarios where wireless infrastructure is unavailable. We took a A3C approach and designed a reward function that captures the specifics of such scenarios. By means of simulation experiments of our study reveals that the proposed DRL algorithm converges fast, and achieves 5dB higher median SINR and 4× lower outage range, as compared to a gradient-based benchmark solution.

Figure 2 :Figure 3 :

 23 Figure 2: Left: SINR heatmap and UEs location (grey dots). Right: The corresponding distribution of UEs' experienced SINR.

Figure

  Figure4: CDF of the SINR attained by all users with the proposed DRL over 10,000 testing steps, after 100, 500, 1,000, and 2,000 training episodes (EPs), and respectively with the benchmark gradient method.

Table 1 :

 1 Simulation Parameters

		Parameter	Value
	Wireless Channel	BS Transmit power Antenna gain Log normal shadowing Gaussian noise Handover time-to-trigger 3 20 dBm 2 dB N (0, 2) -121 dBm Handover threshold 1 dB Minimum SINR -5 dBm
	Learning	Learning rate Discount factor Number of A3C workers 4 0.0001 0.9 Global update step 10 Normalising factor θ 0.05

  Fotouhi et al. propose a mobility model to improve spectral efficiency in drone-based BS scenarios, though consider fixed user group coverage without UE handover [4]. In [13], Oueis et al. provide technical overview of LTE operation for public safety. Orsino et al. then highlight that drones carrying radio transceivers improve network coverage and bring higher data rates to challenging locations


Maria Elisa Ruizechartea

Isaure Chauvot De Beauchêne

David W Ritchie

Accelerating Protein Docking Calculations using the ATTRACT CoarseGrained Force Field and 3D Rotation Maps

Conclusions

We have found that many proteinprotein complexes have at least one pair of beads in nearoptimal contact. Thus, using pairs of surface beads to initialise a docking search is a valid heuristic for treating 3D translations.

The algorithm described here will allow us to perform an exhaustive recursive docking search of 3D rotation space. Thus, in principle, our approach will never miss any local energy minima, and is guaranteed to find the global minimum of the ATTRACT energy function for each pair of starting beads.

Our algorithm is still under development. Nonetheless, we believe it could be faster than the conventional gradient minimisation that is currently used in ATTRACT. It also promises to be more accurate if we can train a new scoring function to optimise the local minima among the sampled poses.

Eventually, we will test our approach on the Protein Docking Benchmark.

Introduction

Proteinprotein docking algorithms aim to predict how two proteins interact with each other to form a complex. Docking algorithms need to fulfil two main tasks: (1) sampling the possible relative positions of the two proteins and (2) computing the interaction energy at each position. Although the docking problem has been studied for over 25 years, doing this accurately and thoroughly remains a computationally expensive task. We are therefore developing a new algorithm to face the protein docking problem in a novel and efficient way.

We are interested in performing an exhaustive search of the rigidbody docking space, where the positioning of the two proteins is not driven by any prior knowledge (e.g. from homology), while still using an accurate force field interaction energy model. In order to reduce the O(N^2) cost of atomistic forcefield models, we use a coarsegrained (CG) bead model taken from ATTRACT [1]. However, a naive energy calculation for every trial orientation would still cost O(N^2) energy evaluations in the number of beads. We are therefore developing a method to detect the locations of all possible clashes before performing any energy calculations, thus allowing us to avoid calculating energies for many millions of useless trial orientations.

Based on a preliminary study of proteinprotein interfaces in complexes from the Protein Docking Benchmark [2], we found that a large number of interfaces contain at least one pair of CG beads at almost their optimal distance. Therefore, our idea is to perform a series of restricted docking searches in which one surface bead from each docking partner is placed in contact at the coordinate origin. This leaves a 3D rotational search, in which ligand may rotate around a fixed receptor.

Of course, a full docking search requires all possible pairs of surface beads to be placed together. However, within each rotational subproblem, we can exploit the fact that rotations do not change any distances from the origin. Thus, we can precalculate a "3D rotational map" of all of the rotations that will cause the beads to clash. We can then restrict the remaining rotational search and energy calculation to a small region near the forbidden rotations in the clash map.

Detecting Steric Clashes -3D Clash Maps

Maria Elisa Ruiz Echartea is supported by a doctoral bursary (CORDIS) from Inria Nancy Grand Est.

3D Rotation Sampling and ATTRACT Energy Calculation

For the translational part of a docking search, we consider all possible pairs of receptor and ligand surface beads in turn, and for each pair we translate the ligand and receptor so that the selected beads (here shown in yellow) are in perfect contact at the origin with their centres of mass are aligned on the zaxis. This absorbs 3 translational degrees of freedom, and leaves a pure 3D rotational search problem about the origin for each selected pair of beads, as illustrated on the left.

Before calculating ATTRACT bead energies, we use the bead coordinates to calculate a list of rotations that will cause steric clashes. These are rotations that should be avoided during the docking search. A 3D representation of this "clash map" is shown on the right. It can be seen that only a small region of the search space is not forbidden by steric clashes.

The Protein Docking Benchmark v5.0

We looked at all of the proteinprotein complexes in the Protein Docking Benchmark (v5) in order to check how often it is possible to find at least one pair of coarsegrained beads located at their optimal distance (shown in yellow on the right), namely, pairs of beads which have the lowest possible ATTRACT interaction energy. To do this, we computed the difference between the best beadbead distance in the complex and compared it to the optimal distance for the corresponding bead types (deltaD). The results are shown in the table below.

The Protein Docking Benchmark v5.0

Each 3D docking search is performed by recursively subdividing a 3D quaternion "ball" representation of the rotational space. Rotational samples generated in this way may be rapidly compared to rotations in the 3D clash map. Protein Science, 12, 12711282 (2003).

E = ε AB [ [R AB /r ij ] 8 -[R AB /r ij ] 6 ] + q i q j /ε(r ij )r ij 1. Zacharias, M.
2. Hwang, H., Vreven T., Janin, J., Weng Z. Proteins, 78, 31113114 (2010).

Each complex has between 3 and 80 nearoptimal bead pairs, and as this table shows 98% of the unbound complexes has at least one pair of beads within 0.2 Å of the optimal separation.

Each point in the -ball defines a rotation axis and angle. A rotational search is perfor med by recursively subdividing the -ball.

x may be used to calculate pairwise bead distances very efficiently from a quaternion rotation (details not shown). Thus, we can detect steric clashes and calculate interaction energies without ever needing to physically move any beads.

  Using each bead's distance from the origin (here, R a and R b ), the cosine rule The Protein Docking Benchmark v5.0 Interaction Energy (cal/mol) inter-beads distance (Å)

Acknowledgements


David Doukhan
email: ddoukhan@ina.fr

Jean Carrive
email: jcarrive@ina.fr

Félicien Vallet
email: fvallet@cnil.fr

Anthony Larcher

Sylvain Meignier

Sylvain Meignier An

AN OPEN-SOURCE SPEAKER GENDER DETECTION FRAMEWORK FOR MONITORING GENDER EQUALITY

Keywords: Digital Humanities, Speech Gender Detection, Convolutional Neural Network, i-vector

d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Gender equality in media is a concern which has been analyzed through various descriptors such as the proportion of male and female employees in media (journalists, top decision-making posts) [START_REF]Gender equality commission[END_REF], speaker roles (expert, interviewer), topics covered [START_REF] Reiser | L'image des femmes dans les médias[END_REF] or correlation between the proportion of male and female speakers and the amount of audience [START_REF]La représentation des femmes à la télévision et à la radio -rapport sur l'exercice[END_REF]. Among the descriptors of gender equality, male and female speaking-time duration has been used in Global Media Monitoring Project world-scale study [START_REF]Who Makes the News?: Global Media Monitoring Project[END_REF]. Due to the cost associated to the manual annotation of speech, this monitoring is restricted to the analysis of a single day every five years, and limited to news-related content. Such a limitation induces biases, such as the particular context in which these measures were made, which may influence the topics covered in the news, as well as the amount of male and female speakers. Moreover, it describes a limited set of programs that does not fully reflect audiovisual diversity. † The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the CNIL.

In the present work, we design an automatic speaker gender detection system for large scale monitoring of raw audiovisual streams in order to reduce the bias associated to a particular measure context. Quantitative gender speaking time estimates may help describing the evolution of gender equality across time and could be used to guide qualitative studies. They could also enable TV channels and radio stations to monitor gender equality in their programs. Long term goal of this research is based on the assumption that quantitative description of gender equality in media may increase consciousness on those issues and result in societal changes.

As it is easier than other automatic speech-related tasks, gender detection's difficulty is often under-estimated. Female speech is generally associated to higher pitch, vowel formants located in higher frequencies and considered to be more breathy. Language-dependent prosodic differences between male and female speakers were also reported, suggesting gender acoustic differences are not only due to gender physiological properties, but also to a given socio-cultural context [START_REF] Pépiot | Voice, speech and gender:. malefemale acoustic differences and cross-language variation in english and french speakers[END_REF]. Consequently, the robustness of speech gender detection systems is still challenged by speakers having marked accents (regional, foreign), extreme pitch ranges, or speaking using non-standard prosodic patterns.

This study addresses three major issues: Which methods result in the most accurate speech-time estimates? Does the robustness of automatic systems allows to perform reliable gender equality descriptions? What knowledge can be gathered from audiovisual streams using gender speech-time information?

RELATION TO PRIOR WORK

Speech gender detection systems have been used for a long time as a preprocessing step in Automatic Speech Recognition (ASR) engines, allowing the selection of gender dependent acoustic models [START_REF] Lori | A phone-based approach to non-linguistic speech feature identification[END_REF][START_REF] Bocklet | Age and gender recognition for telephone applications based on gmm supervectors and support vector machines[END_REF]. Gender detection was not necessarily considered as an end itself, but rather as a way of selecting models having the closest acoustic properties to the utterance being analyzed. More recent studies have used speaker gender detection as a preprocessing step for selecting gender-specific emotion recognition engines [START_REF] Xia | Modeling gender information for emotion recognition using denoising autoencoder[END_REF], as well as a way of enhancing human-machine interactions by defining genderspecific interaction strategies [START_REF] Shafey | Audio-visual gender recognition in uncontrolled environment using variability modeling techniques[END_REF].

Gaussian Mixture Models (GMM), trained with MFCCs have been used as a de facto standard for speech gender recognition since the 90's and are still used in most of modern systems [START_REF] Lori | A phone-based approach to non-linguistic speech feature identification[END_REF]. Approaches based on i-vectors, compact representations of speaker's utterances, have been successfully used over the last few years for speaker recognition tasks [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF]. They have recently shown better performances over GMMs for the detection of speaker gender, in a context where already segmented utterances were assumed to belong to a single speaker [START_REF] Shafey | Audio-visual gender recognition in uncontrolled environment using variability modeling techniques[END_REF]. Other approaches based on Convolutional Neural Networks (CNN) were proposed and evaluated on utterances assumed to belong to a single speaker [START_REF] Royo-Letelier | Detection and characterization of singing voice using deep neural netwo rks[END_REF]. However, those approaches were trained using a limited amount of data, that does not allow to take advantage of deep architectures.

The study presented in this paper aims at designing a speaker gender detection system for large-scale analysis of raw audio streams. For this purpose, we compare the performances of GMMs, i-vector and CNN based systems. Audiovisual streams contain speech recorded in various conditions (telephone of variable quality, studio, spontaneous and prepared speech, speech-over-music, speech-over-noise) which should be handled robustly. While GMM and CNN approaches may be directly used on speech streams, i-vector based systems require a preliminary speaker segmentation step, in charge of splitting speech excerpts into homogeneous segments assumed to belong to a single speaker. The reliability of this pre-segmentation stage may affect the discriminative power of the resulting i-vectors.

TRAINING DATASET: INA'S SPEAKER DICTIONARY

Gender detection models were trained using INA's Speaker Dictionary [START_REF] Franc | An effortless way to create large-scale datasets for famous speakers[END_REF][START_REF] Vallet | Speech trax: A bottom to the top approach for speaker tracking and indexing in an archiving context[END_REF], which is to our knowledge the largest manually annotated speaker database issued from broadcast material reported in the literature, providing a definite technological advantage regardless of the algorithmic strategy used (see [START_REF] Nagrani | Voxceleb: A large-scale speaker identification dataset[END_REF] for a recent review of speaker databases). This audiovisual corpus, constituted using a semi-automatic annotation protocol, contains about 32000 speech excerpts, corresponding to 1780 male (94 hours) and 494 female speakers (27h).

A first set of excerpts was collected from TV news broadcasted from 2007 to 2014. Speech streams were first segmented in speaker turns using LIUM SpkDiarization [START_REF] Meignier | Lium spkdiarization: an open source toolkit for diarization[END_REF]. An Optical Character Recognition system was used to match personality name appearing on screen to referenced people [START_REF] Poignant | From text detection in videos to person identification[END_REF]. The corresponding excerpts were thus presented to annotators in charge of validating speech excerpts. A second set of archives was obtained from INA's collections, for a period of time ranging from 1957 to 2012. Queries based on fa-mous personalities names were performed on Ina.fr website. Speaker diarization procedures were used on resulting archives and segments found in largest speaker clusters were presented to annotators. The last set of excerpts was obtained from 3 radio stations collected from 2012 to 2014. Extracted speech turns were associated to i-vectors, and matched to previously known speakers, before being manually validated.

GENDER DETECTION SYSTEMS

Gender detection processing pipeline is composed of several modules. A Speech/Music segmenter based on CNN is used to discard music and empty segments [START_REF] Doukhan | Investigating the Use of Semi-Supervised Convolutional Neural Network Models for Speech/Music Classification and Segmentation[END_REF]. Features corresponding to speech segments are then computed using a common extraction framework. A simple energy threshold is used to discard frames associated with low energy. GMM, i-vector and CNN systems are then used to classify the remaining speech segments into male and female excerpts.

Feature Extraction

Feature extraction was realized on audio excerpts sampled at 16KHz using SIDEKIT [START_REF] Larcher | An extensible speaker identification sidekit in python[END_REF]. 24 Mel-scaled filter-banks coefficients were computed on 25ms sliding windows with a 10ms shift. Those coefficients were fed directly to CNN models while a second set of acoustic features (MFCC) was obtained by applying Discrete Cosine Transform (DCT) to those filterbank coefficients to extract 19 Mel-Frequency Cepstrums coefficients (MFCC), which were augmented with log energy. A feature wrapping method was used to gaussianize MFCC distributions using windows of 3 seconds [START_REF] Pelecanos | Feature warping for robust speaker verification[END_REF]. Time derivatives (∆) and acceleration (∆∆) coefficients were concatenated to MFCC frames, resulting in 60-dimensional input features used by GMM and i-vector models.

Gaussian Mixture Models

GMMs are the most common approach used for modeling speaker gender, and were used as a baseline for evaluating more complex models. Female and male GMM's were trained using 1024 gaussians and diagonal covariance matrices. Each GMM was trained using 2000 frames for each speaker found in the training set. Resulting models were used in a 2 state Hidden Markov Model (HMM) to segment speech into male and female excerpts.

I-vector gender segmentation system

Our i-vector-based gender segmentation system is composed of three main modules. A speech diarization module is firstly used to segment speech into homogeneous speaker turns. An i-vector extractor is then used in a second stage to model each speaker turn as a compact vector, assumed to contain discriminative speaker information. Lastly, a classifier trained with i-vectors is used to predict the gender associated to a given speaker turn.

The diarization procedure was realized using S4D [START_REF] Meignier | S4d: Sidekit for speaker diarization[END_REF]. Speech segments were pre-segmented using Gaussian Divergence criterion, with a 2.5 second-window. Segments assumed to belong to the same speaker were merged through two clustering procedures based on Bayesian Information Criterion: a first procedure of Linear clustering, merging adjacent candidates, followed by a global Hierarchical Agglomerative Clustering procedure. Viterbi re-segmentation procedure was carried on the resulting clusters. Clustering algorithms and Viterbi re-segmentation were used with default thresholds (2, 3, -250). I-vector extraction was realized using SIDEKIT [START_REF] Larcher | An extensible speaker identification sidekit in python[END_REF]. A 1024 gaussians Universal Background Model (UBM) with diagonal covariance matrices was trained using 2500 frames per female and 700 frames per male speaker, in order to deal with the gender balance of the training dataset. The total variability matrix was defined with rank 200, and trained using 10 randomly selected excerpts per speaker. Training excerpts used for the estimation of the total variability matrix were used to extract i-vectors, which were fed into a linear SVM in charge of predicting speaker gender. Training examples were weighted according to the gender distribution found in the training examples.

Convolutional Neural Network system (CNN)

Figure 1 describes the deep CNN architecture which was implemented using Keras to discriminate between male and female speech [START_REF] Franc | Keras[END_REF]. It is inspired from speech recognition models [START_REF] Qian | Very deep convolutional neural networks for noise robust speech recognition[END_REF], and differs by using filter banks frames having lower dimensionality, associated to a larger time context of 68 frames (695 ms, 24 dimensions) 1 . Two 3x3 convolutional layers of 64 neurons are first used, followed by a 1x2 maxpooling layer, aimed at providing invariance in the frequency domain. Two additional 3x3 layers of 128 neurons are then followed by a 2x2 max-pooling layer and a last 3x3 convolutional layer of 256 neurons. A maximal temporal pooling layer, in charge of selecting the most discriminating pattern found in this relatively large time interval is finally used, and followed by 4 dense layers of 512 neurons, associated to increasing dropout rates, before the last softmax layer outputting 1 probability per gender. Each layer of the network is followed by batch normalizations and RELU activations.

A data selection strategy aimed at using all available training data with limited over-fitting for a particular gender or speaker was used, consisting in generating 1000 examples batches, composed of 500 randomly examples of distinct speakers of each gender. Epochs were defined as an amount of 1,000 batches.

Model selection consisted in splitting INA's speaker dictionary into training and validation subsets containing respec- 1 Additional experiments were realized using 40 dimension filter banks, as well as compressed log-spectrograms [START_REF] Royo-Letelier | Detection and characterization of singing voice using deep neural netwo rks[END_REF], resulting in lower performances. tively 80% and 20% of disjoint speakers. Models' performances were assessed after each epoch on the validation set, using speaker sampling strategies. F-measure was used as performance criterion, and training iterations were stopped after 5 epochs not resulting in performance improvement. Best models were obtained using less than 10 epochs. CNN's output was then used to feed a 2 states HMM.

EVALUATION

Evaluation Material

REPERE challenge corpus contains annotated TV streams gathered from two French channels from 2011 to 2013 [START_REF] Giraudel | The repere corpus: a multimodal corpus for person recognition[END_REF]. This includes street interviews, debates and news corresponding to 46 hours of speech attributed to 1129 male speakers, and 12 hours attributed to 557 female speakers 2 . 457 speakers found in REPERE corpus were also found in INA's speaker dictionary used for training, and 606 speakers were associated to a gender label, without identity annotation.

Frame-Level Evaluation

Table 1 presents gender recognition frame-level performances obtained on REPERE corpus. Since small pauses are not systematically annotated in ground-truth, this evaluation was carried on the subset of frames annotated as speech in REPERE and predicted as speech by the speech/music segmenter, resulting in a binary classification problem. Best results were obtained with the CNN model, which was associated to a Fmeasure of 96.52. All models were associated to better recall rates for male speakers, which may lead to systems underestimating female speaking time. This may partly be explained by the largest amount of male speakers found in the training database, despite the class balancing strategies deployed.

End-to-End Evaluation

This evaluation describes the ability of models to optimize Gender Speaking-Time Percentage (GSTP) estimation on variable length raw audiovisual excerpts. GSTP is the main descriptor retained in our ongoing gender equality studies, and was computed for each annotated stream found in REPERE challenge dataset using the following formula: GSTP = 100× female speech time male speech time + female speech time

Frame-level male and female gender detection errors may counter-balance and provide robust GSTP estimators for a reasonably long time-interval. Based on this assumption, REPERE shows were split into four categories according to their duration, resulting in 133 excepts of less than 2 minutes, 82 excerpts of duration varying between 2 and 10 minutes excerpts, 104 10-30 minutes excerpts and 25 30-60 minutes excerpts. Root Mean Square Error (RMSE) associated to GSTP estimation for each of these categories is shown in figure 2. As expected, longer excerpt durations were associated to lower GSTP estimation errors. Best results were obtained with the CNN model, associated to a RMSE of 0.68 for excerpts longer than 30 minutes together with GSTP estimation mean and worst-case error rates as low as 0.59% and 1.8%.

Maximal error-rates were found to be always lower for CNN than for i-vector model. This phenomenon may be explained by the architecture of the i-vector model, relying on a preliminary diarization step, which may miss small speaker turns, resulting in i-vectors corresponding to several speakers.

CONCLUSION AND FUTURE WORK

A deep CNN model has been realized to segment speech streams into male and female excerpts and compared to GMM and i-vector models on REPERE corpus, showing superior performances. The proposed CNN model is accessible from INA's GitHub repository 3 : http://github.com/ ina-foss. 3 Code and trained models will be released upon acceptance of this paper. This positive outcome allows considering the reliability of this gender detection system sufficient to perform large-scale gender equality descriptions, offering concrete perspectives for digital humanities. Detailed analyses realized using this framework goes beyond the scope of this paper and will be addressed in future works [START_REF] Doukhan | Describing gender equality in french audiovisual streams with a deep learning approach (submitted)[END_REF]. Early results obtained using 1 week of raw stream corresponding to 6 French TV channels and 8 radio stations highlighted several interesting tendencies: Speaking time was mainly attributed to male speaker, especially in radio material. The proportion of male speakers was higher in TV channels associated to intellectual content, and the day period associated to the highest female speaking time was between 6 and 9 AM. The proportion of male speakers in radio and TV programs was higher on Sunday than during other week days. Longer term studies may take advantage of INA's available meta-data (audience, type of program, speaker identities) to proposed more detailed analysis.

The impact of our evaluation is limited to the properties of REPERE corpus, containing only broadcast news archives. It does not include child speakers, nor very specific speech utterances such as found in movies, comedy sketches or speaker imitations, which should be addressed in future studies. Improvements of the proposed i-vector system could be investigated using more recent diarization methods [START_REF] Gaël Le Lan | A triplet ranking-based neural network for speaker diarization and linking[END_REF]. The use of recurrent neural network units in CNN's last layers, replacing the final temporal max-pooling layer proposed will also be investigated, together with training example balancing strategies based on pitch estimation and data augmentation.

Since speaking time is not sufficient to describe fully gender equality, longer-term research may include using ASR to describe gender roles and interactions.

Fig. 1 .

 1 Fig.1. CNN gender detection architecture, using input features of 68 concatenated 24-dimension filter banks frames.

Fig. 2 .

 2 Fig. 2. Root Mean Square Error associated to the estimation of gender speaking time percentage, obtained on variablelength shows found in REPERE corpus.

Table 1 .

 1 Gender Detection frame-level Performances obtained on REPERE challenge corpus

	Model	Male Recall Female Recall F-measure
	GMM	97.25	94.28	95.74
	I-vector	98.19	92.98	95.51
	CNN	98.04	95.05	96.52

92 speech segments corresponding to 22 minutes of speech were found to be associated to wrong gender annotations and were corrected manually.


K A Lee

V Hautamäki

T Kinnunen

A Larcher

C Zhang

A Nautsch

T Stafylakis

G Liu

M Rouvier

W Rao

F Alegre

J Ma

M W Mak

A K Sarkar

H Delgado

R Saeidi

H Aronowitz

A Sizov

H Sun

T H Nguyen

G Wang

B Ma

V Vestman

M Sahidullah

M Halonen

A Kanervisto

G Le Lan

F Bahmaninezhad

S Isadskiy

C Rathgeb

C Busch

G Tzimiropoulos

Q Qian

Z Wang

Q Zhao

T Wang

H Li

J Xue

S Zhu

R Jin

T Zhao

P.-M Bousquet

M Ajili

W B Kheder

D Matrouf

Z H Lim

C Xu

H Xu

X Xiao

E S Chng

B Fauve

K Sriskandaraja

V Sethu

W W Lin

D A L Thomsen

Z.-H Tan

M Todisco

N Evans

J H L Hansen

J.-F Bonastre

E Ambikairajah

The I4U Mega Fusion and Collaboration for NIST Speaker Recognition Evaluation 2016

Keywords: speaker recognition evaluation, fusion, benchmark, Call My Net

The 2016 speaker recognition evaluation (SRE'16) is the latest edition in the series of benchmarking events conducted by the National Institute of Standards and Technology (NIST). I4U is a joint entry to SRE'16 as the result from the collaboration and active exchange of information among researchers from sixteen Institutes and Universities across 4 continents. The joint submission and several of its 32 sub-systems were among topperforming systems. A lot of efforts have been devoted to two major challenges, namely, unlabeled training data and dataset shift from Switchboard-Mixer to the new Call My Net dataset. This paper summarizes the lessons learned, presents our shared view from the sixteen research groups on recent advances, major paradigm shift, and common tool chain used in speaker recognition as we have witnessed in SRE'16. More importantly, we look into the intriguing question of fusing a large ensemble of sub-systems and the potential benefit of large-scale collaboration.

Introduction

The speaker recognition evaluation (SRE) benchmark regularly conducted by the National Institute of Standards and Technology (NIST) has been a major driving force advancing speaker recognition technology. Since the first SRE'96 [START_REF] Martin | NIST speech processing evaluations: LVCSR, speaker recognition, language recognition[END_REF], the NIST evaluations have been focusing on speaker verification: given a segment of speech, decide whether a specified target speaker is speaking in that segment. Apart from the large datasets made available by the organizer, NIST SREs have been driving the research directions in speaker recognition by specifying the performance metrics and evaluation protocol -conversational versus interview speaking style in SRE'08 and SRE'10, multiple versus single session enrolment in SRE'12 [START_REF]The NIST Year 2012 Speaker Reecogition Evaluation[END_REF], and more recently fixed versus open training conditions in SRE'16 [START_REF]NIST 2016 Speaker Reecogition Evaluation[END_REF], just to name a few. SRE'16 is different from the prior SREs in certain key aspects. For the first time, non-English evaluation data collected outside North America was used in SRE' [START_REF] Snyder | Time delay deep neural network-based universal background models for speaker recognition[END_REF]. In particular, the evaluation data is in Cantonese and Tagalog while the development set is in Mandarin and Cebuano. One more difference from previous editions is that SRE'16 explores the use of unlabeled dataset, which might hold the key to cope with the language mismatch and dataset shift problem [START_REF] Aronowitz | Compensating inter-dataset variability in PLDA hyper-parameters for robust speaker recognition[END_REF].

Aside from a joint submission, the I4U consortium was formed with a common vision to promote and facilitate active exchange of information and experience toward SRE '16. Following the success of I4U'12 [START_REF] Saeidi | I4U submission to NIST SRE 2012: A large-scale collaborative effort for noiserobust speaker verification[END_REF], the I4U consortium for SRE'16 is a collaboration of 62 researchers from 16 research Institutes and Universities across 4 continents. The names of the organization and corresponding system identifiers are provided in Table 1. The collaboration started off with the first I4U meeting conducted via teleconference in early May, 2016. This was followed by regular bi-weekly and weekly meetings toward the end of SRE'16. An online group was also set up, providing a discussion platform across various issues surrounding NIST SRE'16. In particular, test segment variability, domain adaptation for language and channel shift, uncertainty propagation, score normalization, session compensation, and various issues concerning score calibration (quality measure, supervised versus unsupervised) have been actively discussed. Solutions were put in place as part of the I4U submission.

The submitted results were based on fusion of multiple classifiers from a pool of 32 sub-systems contributed by I4U members. The availability of such a large ensemble of sub-systems allows us to look into classifier selection and fusion strategies, and more importantly develop a strategy for megafusion1 as detailed in Section 4. Listed in Table 1 are the 17 best sub-systems selected from each site to form the I4U primary submission.

The paper is organized as follows. Section 2 presents an overview of SRE'16 datasets and our strategies to cope with new conditions. Section 3 highlights some important aspects of the submitted system and the component classifiers. Results of individual sub-systems and their fusions are presented in Section 4. Section 5 concludes the paper.

Train, development, and test sets

To build and evaluate a speaker recognition system, three disjoint datasets are required for training, development, and test, respectively. In the context of NIST SREs, the training and development sets are usually released at a much earlier date than the test set (a.k.a the evaluation set), which the participants have to process and submit their system outputs (in the form of loglikelihood scores) in a limited period of time. Similar to previous evaluations, the emphasis of SRE'16 has been on conversational telephone speech recorded from the public telephone system (e.g., cell phone, landline). Table 2 shows the list of corpora we used in I4U for the core task of SRE'16. In particular, the training set constitutes data used to the universal background model (UBM) [START_REF] Reynolds | Speaker verification using adapted gaussian mixture models[END_REF], deep neural network (DNN) to predict senone target [START_REF] Lei | A novel scheme for speaker recognition using a phonetically-aware deep neural network[END_REF][START_REF] Kenny | Deep neural networks for extracting baum-welch statistics for speaker recognition[END_REF][START_REF] Richardson | Deep neural network approaches to speaker and language recognition[END_REF], i-vector extractor [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF], linear discriminant analysis (LDA), probabilistic LDA (PLDA) [START_REF] Prince | Probabilistic linear discriminant analysis for inferences about identity[END_REF][START_REF] Kenny | Bayesian speaker verification with heavy-tailed priors[END_REF][START_REF] Lee | Multi-session PLDA scoring of i-vector for partially open-set speaker detection[END_REF] and other parametric models found in most state-of-the-art systems, as in those top performance systems with I4U. It is worth mentioning that Switchboard 1 (Swb 1) and the two Fisher corpora come with transcription which is required in training DNN [START_REF] Dahl | Context-dependent pretrained deep neural networks for large vocabulary speech recognition[END_REF][START_REF] Hinton | Deep neural networks for acoustic modeling in speech recognition[END_REF], or splice time delay DNN (TDNN) [START_REF] Snyder | Time delay deep neural network-based universal background models for speaker recognition[END_REF], to produce frame-level senone label and to extract deep bottleneck features [START_REF] Richardson | Deep neural network approaches to speaker and language recognition[END_REF][START_REF] Yu | Improved bottleneck features using pretrained deep neural networks[END_REF]. The Fisher corpora were also used for UBM training in some of I4U systems. Very specific to I4U was that we set aside SRE'10 and SRE'12 data for development and experiment purposes instead of including them in the training set. This setting allows us to validate the results across multiple validation and test sets.

Different from previous evaluations, the test data of SRE'16 consists of speech utterances spoken in Tagalog and Mandarin instead of English in previous evaluations. Also, the test segments have varying duration ranging from 10 to 60 seconds. These changes introduce two new challenges to SRE '16: i. Dataset shift between the training and test sets due to language mismatch and differences in data collection infrastructure (e.g., telephone network, front-end devices).

ii. Test duration variability where test duration is uniformly distributed from 10 to 60 seconds.

The SRE'16 test data is a subset of the on-going Call My Net speech collection by the Linguistic Data Consortium (LDC).

The changes from the Switchboard and Mixer datasets used in the previous SREs to the new Call My Net is cast as a dataset shift problem. In this regard, we found the inter dataset variability compensation (IDVC) [START_REF] Aronowitz | Compensating inter-dataset variability in PLDA hyper-parameters for robust speaker recognition[END_REF] to be extremely effective in dealing with this problem. On the other hand, the duration variability was accounted for with the use uncertainty propagation [START_REF] Kenny | PLDA for speaker verification with utterances of arbitrary duration[END_REF] and variance compensated length-norm [START_REF] Kenny | JFA-based front ends for speaker recognition[END_REF] for i-vector PLDA system. Most sub-systems developed in I4U were equipped with the dataset and duration compensation techniques mentioned above. Also shown in Table 2 is the development set consisting of speech utterances spoken in Cebuano and Cantonese drawn from the same Call My Net collection as the test set. A small unlabeled set in Tagalog, Mandarin, Cebuano, and Cantonese was made available with an intention to bridge the gap between the development and test sets. Numerous efforts and discussion among I4U members have focused on the use of the unlabeled set for score normalization and calibration.

Recognition systems

Table 3 shows the key features of 17 systems used to form the I4U'16 primary submission. At the core of all the sub-systems listed in Table 3 is the i-vector approach [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF], which represents the current mainstream technique in text-independent speaker recognition. At the i-vector extraction stage, we have subsystems that use either GMM or DNN posteriors [START_REF] Lei | A novel scheme for speaker recognition using a phonetically-aware deep neural network[END_REF][START_REF] Kenny | Deep neural networks for extracting baum-welch statistics for speaker recognition[END_REF] for frame alignment (i.e., the role of the UBM). Except for Sys2 that uses support vector machine (SVM), PLDA was used to handle session variability and as the scoring back-end in all other subsystems. In addition, a rich set of acoustic feature extraction including MFCC, PLP, PNCC (power normalized cepstral coefficients) [START_REF] Kim | Power-normalized cepstral coefficients (pncc) for robust speech recognition[END_REF], tandem feature [START_REF] Hermansky | Tandem connectionist feature extraction for conventional HMM systems[END_REF][START_REF] Grézl | Probabilistic and bottleneck features for LVCSR of meetings[END_REF], and the recently proposed ICMC (infinite impulse response constant Q mel-scaled cepstral coefficients) [START_REF] Delgado | Further optimisations of constant q cepstral processing for integrated utterance verification and textdependent speaker verification[END_REF] were used at the front-end. Among these, MFCC remains the most commonly used acoustic features. Also, a vast majority of our sub-systems use energy-based voice activity detector (VAD) in view of its simplicity and effectiveness. Other options for VAD that have been adopted are (i) VQ-VAD [START_REF] Kinnunen | A practical, self-adaptive voice activity detector for speaker verification with noisy telephone and microphone data[END_REF] in Sys1 and Sys14, (ii) speech/non-speech probabilities inferred from the DNN senone posterior in Sys9, and (iii) two-channel VAD [START_REF] Mak | A study of voice activity detection techniques for NIST speaker recognition evaluations[END_REF] in Sys12. Increased Complexity: We observed a general trend in using higher dimensional feature vector and a larger UBM. From Table 3, the input dimensionality of acoustic features ranges from 39 to 139 with majority of the systems settle at 60. The socalled tandem features used in Sys9 and Sys11 were formed by concatenating MFCC and deep bottleneck feature (DBF) lead-Table 3: List of 17 systems used to form the I4U primary submission to SRE'16, their key features at the front-end, the classifier, unlabeled data, and toolkits that have been used. ing to a dimensionality of around a hundred. The size of the UBM increases tremendously from the typical value of 512 in SRE'12 [START_REF] Saeidi | I4U submission to NIST SRE 2012: A large-scale collaborative effort for noiserobust speaker verification[END_REF] to 2048 being commonly used in SRE'16. Notably, those large UBMs used in Sys2, Sys9, Sys10, and Sys16 are based on DNN and trained in a supervised manner to predict senone targets. Compared to previous evaluations, the increased complexity was made possible by advances in deep learning and also the availability of increasingly powerful computers.

Embracing Deep Learning: An emerging trend that we have observed is the adoption of deep learning technique. Though we did not observe a remarkable performance boost in general, as in other speech applications [START_REF] Dahl | Context-dependent pretrained deep neural networks for large vocabulary speech recognition[END_REF][START_REF] Hinton | Deep neural networks for acoustic modeling in speech recognition[END_REF][START_REF] Lee | The 2015 NIST Language Recognition Evaluation: the Shared View of I2R, Fantastic4 and SingaMS[END_REF], there are a handful of our sub-systems (six out of seventeen in Table 3) that have successfully incorporated deep learning in one form or another: (i) Deep bottleneck feature (DBF) in Sys9, (ii) Stacked bottleneck feature in Sys11, (iii) DNN posterior in Sys2, Sys9, Sys10, Sys16, (iv) Splice time delay DNN (TDNN) [START_REF] Snyder | Time delay deep neural network-based universal background models for speaker recognition[END_REF] in Sys2, and (v) Denoising autoencoder in Sys14. For the bottleneck features in Sys9 we used a DNN with seven hidden layers each having 1024 hidden units except for the third layer with only 80 units. The DBF was extracted from this layer and concatenated with MFCC forming the tandem feature. The stacked bottleneck features used in Sys11 were obtained by feeding the DBF to another DNN [START_REF] Grezl | Adaptation of multilingual stacked bottle-neck neural network structure for new language[END_REF][START_REF] Xu | A comparative study of BNF and DNN multilingual training on cross-lingual low-resource speech recognition[END_REF]. Another use of DNN is to replace the GMM posterior for frame alignment as proposed in [START_REF] Lei | A novel scheme for speaker recognition using a phonetically-aware deep neural network[END_REF][START_REF] Kenny | Deep neural networks for extracting baum-welch statistics for speaker recognition[END_REF][START_REF] Richardson | Deep neural network approaches to speaker and language recognition[END_REF]. This was adopted in four sub-systems as shown in Table 3, where the entries to the UBM type are denoted as DNN. Finally, Sys14 explores the idea of transforming i-vectors of noisy short utterances to clean long utterances using denoising autoencoder [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF]. The implementation details can be found in I4U system description [START_REF] Lee | The I4U submission to the 2016 NIST speaker recognition evaluation[END_REF] and the references therein.

Dataset Shift: The DNNs used to generate DBF and frame posterior probabilities were trained in supervised manner, which required orthographic transcription. As shown in Table 2, transcribed data is only available in English which exhibits a slightly difference phonetic coverage than the development and test sets in Cebuano/Tagalog and Mandarin/Cantonese. Also, the PLDA trained on the Switchboard and Mixer datasets might have modeled a distribution different from that of the test ivectors. This could be cast as a dataset shift problem. One simple solution is to replace the mean vector of the PLDA with the global mean estimated from the unlabeled in-domain data (see Table 2), or equivalently, subtract the global mean (estimated from the unlabeled set) from the test i-vectors. A more elaborate solution is the IDVC (inter-dataset variability compensation) [START_REF] Aronowitz | Compensating inter-dataset variability in PLDA hyper-parameters for robust speaker recognition[END_REF] which removes the low-rank subspace corresponds to the local means estimated from various datasets. IDVC was implemented in Sys8, Sys9, Sys11, Sys13, and Sys17. Another effective way of using the unlabeled data is score normalization as in Sys1 and Sys12. In addition, support vector discriminant analysis (SVDA) [START_REF] Bahmaninezhad | i-vector/PLDA speaker recognition using support vectors with discriminant analysis[END_REF] performed well in Sys5 for compensating this mismatch by incorporating unlabeled in-domain data without using any pseudo labels.

Tools: Also listed in Table 3 are various open-source toolkits used in developing the I4U systems. More than half of the systems use the Kaldi toolkit. Apart from i-vector extraction, Kaldi was also used to implement and train DNN for extracting bottleneck features and senone posteriors. Readily available Kaldi scripts tailored for SRE tasks have made fast prototyping relatively easier than before and therefore promote wider adoption. Other popular tools utilized by I4U consortium members are SideKit [START_REF] Larcher | An extensible speaker identification sidekit in python[END_REF], Alize [START_REF] Larcher | ALIZE 3.0 -Open Source Toolkit for State-of-the-Art Speaker Recognition[END_REF], MSR toolkit [START_REF] Sadjadi | MSR identity toolbox v1. 0: A matlab toolbox for speaker-recognition research[END_REF], and BOSARIS [START_REF] Bümmer | The BOSARIS Toolkit User Guide: Theory, Algorithms and Code for Binary Classifier Score Processing[END_REF] which were developed more specifically for speaker recognition, and LibSVM [START_REF] Chang | Libsvm: a library for support vector machines[END_REF], Bob [START_REF] Anjos | Bob: a free signal processing and machine learning toolbox for researchers[END_REF], Theano [START_REF] Theano | Theano: A python framework for fast computation of mathematical expressions[END_REF] designed for general machine learning.

System and fusion performance

Considering that the I4U consortium had in total 32 base classifiers, an idea to use a classifier selection was clear from the beginning. To that end, we performed initially a 5-fold crossvalidation (CV) on the development set. Base classifier selection and whether pre-calibration of the scores was to be performed were decided using a 5-fold CV setting (recording average performance over folds). Pre-calibration showed systematically better results than without pre-calibration. In the selection setting, we experimented with many different ideas on how to fix the final subset of classifiers (noting that we started with 32 base classifiers). Final ensemble was selected using heuristic rule of using the best single systems (min Cprimary < 0.7) and one system from each site. This resulted in better crossvalidated results than using just the best systems. The base classifier results on Eval set are shown in Table 4. We show separately the performance on Tagalog (tgl) and Cantonese (yue), This is true for all base classifiers except for Sys1 and Sys14, which appear to be over-fit on the Dev set. Despite some base classifiers performing two times worse than the single best classifier, the fusion results consistently outperform the single best, which signifies the benefit of megafusion of a large ensemble of classifiers. The primary submission fusion was designed so that we first pre-calibrated all 17 base classifiers, via minimizing the C wlr cost, with ptar = 0.01. Development set was used to estimate the scale and bias for all the base classifiers and then applied to the eval-set scores. Then the linear fusion parameters were estimated on the pre-calibrated scores using the same settings. We notice in the Table 4 that this fusion strategy significantly improves in terms of EER compared to any single best base classifier.

The first contrastive system was based on the approach to focus on a smooth transition from minor to major languages. It consists of a trial-wise unsupervised calibration followed by a simple score averaging. To this end, we divided the unlabeled data (only the major language partition was used) into 5 subsets: 4 high-confidence language-gender subsets that account for about 60% of the data and a single subset containing the remaining utterances to be discarded. The purpose of our language-gender subsets was two-fold. Firstly, we used pairwise scores within each subset to train Gaussian classifiers for target and non-target distributions. Secondly, we used it to find subset-dependent log-likelihoods for each enrolment and test i-vectors separately. Summed log-likelihood scores for all ivectors within each trial was used to determine the subset label of a trial and apply the corresponding calibration parameters. The second contrastive system was based on the fusion of 32 base classifiers, where unwanted base classifiers were automatically removed. The scores were subjected to znormalization, where the z-norm parameters were learned from the dev set. We applied the OSCAR [START_REF] Petry | The OSCAR for generalized linear models[END_REF] sparsity promoting regularization to the C wlr objective. We notice in Table 4 that OSCAR based fusion system reaches the lowest EER on dev set, but on eval set the primary submission has a better performance.

Calibration performance of the primary and contrastive fusions are shown in Table 5. We contrast their performance in terms of minimum and actual DCF on the Eval set. We notice that only in Contrastive 1 fusion we achieved low calibration error (i.e., actual DCF is closer to the minimum DCF values) on Eval set which indicates a more effective use of unlabeled data in score calibration. We note that most of the base classifiers did not use unlabeled set in score normalization.

Conclusions

This paper presents an overview of the recognition systems and their fusion developed for NIST SRE'16 by I4U consortium. The collaboration of 62 researchers from 16 research teams benefited all members during the preparation of robust speaker recognition systems. I4U submissions to SRE'16 encompass 32 sub-systems, each one of them presenting a high-end system involving several weeks or months of careful parameter optimization and data engineering. Even if such massive megafusion may be challenging to apply in real use cases, shared evaluation resources and fusion methodology are important tools to enable large-scale collaboration; it facilitates attacking a common challenge engineering goal with shared resources.

NIST SRE'16 was positioned to tackle a more open-ended problem commonly encountered in practical deployment -language mismatch, lack of labeled data, fast adaptation using limited/unlabeled data. To this end, dataset shift could be accounted for with mean compensation. It is also clear that unlabeled dataset could be used for UBM training and score normalization, which does not required labels.

Table 1 :

 1 Component classifiers and their assigned system indexes used for I4U primary submission.

	Sys Feature and classifier	Site
	1	PNCC IV-PLDA + GMM-UBM	AAU
	2	MFCC IV-SVM	Alibaba Inc
	3	MFCC IV-PLDA	CRSS, UTD
	4	MFCC IV-PLDA	CRSS, UTD
	5	MFCC IV-SVDA-PLDA	CRSS, UTD
	6	ICMC IV-PLDA	EURECOM
	7	MFCC IV-PLDA + Quality Meas Func HDA
	8	MFCC IV-PLDA	I 2 R
	9	Tandem DNN IV-PLDA	I 2 R
	10	MFCC DNN I-vector	LIUM
	11	Tandem IV-PLDA	NTU
	12	MFCC IV-PLDA	HK Poly U
	13	MFCC IV-PLDA	UEF
	14	MFCC IV-PLDA	EET, UNSW
	15	PLP IV-PLDA	ValidSoft
	16	MFCC DNN IV-PLDA	Nottingham
	17	MFCC IV-PLDA	LIA

Table 2 :

 2 List of telephone speech copora partitioned into training, development, and test sets used in I4U for SRE'16.

	Partition	Corpus	Language
		SRE'04-05-06-08
		Swb-2 Phase II & III	English
		Swb-Cell Part 1 & 2
	Training	Fisher 1 & 2	English
		Swb-1 Release 2	(with transcript)
		CallMyNet-Unlabeled	Cebuano, Mandarin Tagalog, Cantonese
	Development	CallMyNet-Dev SRE'10, SRE'12	Cebuano, Mandarin English
	Test	CallMyNet-Test	Tagalog, Cantonese

Table 4 :

 4 Base classifier performance and their fusions in terms of equal-error-rate (EER%) on Eval and Dev sets, shown separately for Tagalog (tgl), Cantonese (yue), Cebuano (ceb) and Mandarin (cmn) trials. Majority of the base classifier obtain around 20% EER on Tagalog and around 10% EER on Cantonese. Minimum is 15.85% and 7.10% EER for Tagalog and Cantonese, respectively. Also shown in the table are the EERs on Dev set by which the hyper-parameters for individual sub-systems were optimized. The Eval set is generally easier than the Dev set, where the EERs on Tagalog (tgl) and Cantonese (yue) are lower compared to Cebuano (ceb) and Mandarin (cmn), respectively.

			Eval set	Dev set
	Sys.	EER (tgl) EER (yue) EER (ceb) EER (cmn)
	1	36.96	36.85	24.00	13.82
	2	16.96	9.13	23.33	12.34
	3	18.16	9.46	22.33	12.84
	4	17.28	8.46	21.78	9.75
	5	15.85	7.10	22.50	9.83
	6	21.13	11.05	26.80	14.47
	7	17.38	8.81	23.35	10.68
	8	16.67	7.83	20.66	9.50
	9	16.21	7.62	24.08	10.79
	10	20.68	11.98	27.63	13.79
	11	18.30	9.07	25.00	10.12
	12	19.78	13.23	24.83	13.13
	13	21.20	12.71	27.05	14.47
	14	36.80	32.13	23.66	13.73
	15	19.83	9.70	22.25	12.38
	16	19.78	11.06	27.82	13.33
	17	16.73	7.91	23.04	10.39
	Primary	12.94	5.03	17.70	6.70
	Cont. 1	13.54	6.22	17.19	6.83
	Cont. 2	13.93	5.60	15.53	5.90
	and notice immediately that Tagalog is remarkably harder set
	than Cantonese.

Table 5 :

 5 Performance of the primary and contrastive fusions on Eval set in terms of EER, Minimum and Actual Cprimary. Each entry represents the Equalized and Unequalized performance metrics[START_REF]NIST 2016 Speaker Reecogition Evaluation[END_REF].

			Evalset
	Equalized	EER (%) C primary (Min)	C primary (Act)
	Primary	8.59	0.6392	0.8779
	Cont. 1	9.58	0.7538	0.7615
	Cont. 2	9.28	0.7118	2.5538
	Unequalized EER (%) C primary (Min)	C primary (Act)
	Primary	8.80	0.6328	1.0617
	Cont. 1	10.00	0.7473	0.8410
	Cont. 2	9.60	0.7218	3.2037

Thanks to Doug Reynolds of MITLL who coined the term 'megafusion' during the SRE'16 workshop, which has inspired the title of this paper. † Guangsen Wang is now with Tencent AI Lab, Shenzhen, China.

Acknowledgements

This work was partially funded by the German BMBF within the CRISP center and the Hesse state (project no. 467-15/09, 518/16-30), and by the European Commission in its framework programme Horizon 2020 under grant agreement no. 706668 (Talking Heads) and OCTAVE Project (#647850). The views expressed in this paper are those of the authors and do not engage any official position of the funding agencies.