Lina Ochoa

Thomas Degueule
email: thomas.degueule@labri.fr

Jean-Rémy Falleri
email: jean-remy.falleri@labri.fr

BreakBot: Analyzing the Impact of Breaking Changes to Assist Library Evolution

If we make this change to our code, how will it impact our clients?" It is difficult for library maintainers to answer this simple-yet essential!-question when evolving their libraries. Library maintainers are constantly balancing between two opposing positions: make changes at the risk of breaking some of their clients, or avoid changes and maintain compatibility at the cost of immobility and growing technical debt. We argue that the lack of objective usage data and tool support leaves maintainers with their own subjective perception of their community to make these decisions.

We introduce BreakBot, a bot that analyses the pull requests of Java libraries on GitHub to identify the breaking changes they introduce and their impact on client projects. Through static analysis of libraries and clients, it extracts and summarizes objective data that enrich the code review process by providing maintainers with the appropriate information to decide whether-and how-changes should be accepted, directly in the pull requests.

INTRODUCTION

Leveraging the time-honored principles of modularity and reuse, modern software systems development typically entails the use of external software libraries. Rather than implementing new systems from scratch, developers incorporate libraries that provide functionalities of interest into their projects. Libraries expose their features through Application Programming Interfaces (APIs) which govern the interactions between client projects and libraries.

Libraries constantly evolve to incorporate new features, bug fixes, security patches, refactorings, and extra-functional improvements [START_REF] Bavota | The Evolution of Project Inter-dependencies in a Software Ecosystem: The Case of Apache[END_REF][START_REF] Brito | Why and how Java developers break APIs[END_REF]. It is critical for clients to stay up-to-date with the libraries they use to benefit from these improvements and to avoid security issues, technical lag, and technical debt [START_REF] Chinthanet | Lags in the release, adoption, and propagation of npm vulnerability fixes[END_REF][START_REF] Gonzalez-Barahona | Technical Lag in Software Compilations: Measuring How Outdated a Software Deployment Is[END_REF]. When a library evolves, however, it may break the contract previously established with its clients by introducing Breaking Changes (BCs) in its public API, resulting in compilation-time, link-time, or run-time errors. Seemingly innocuous changes such as altering the visibility or abstractness modifier of a type declaration or inserting a new abstract method can, under certain conditions, break client code. Errors triggered by these changes burden client developers, given the sudden urgency to fix issues out of their control without intrinsic motivation. As a result, clients sometimes hesitate to upgrade their dependencies, raising security concerns and making future upgrades even more difficult. Thus, it does not come as a surprise that the problem of helping clients respond to library evolution has garnered considerable interest in recent years [START_REF] Lawall | Coccinelle: 10 Years of Automated Evolution in the Linux Kernel[END_REF][START_REF] Meng | A historybased matching approach to identification of framework evolution[END_REF][START_REF] Hoan | A Graph-Based Approach to API Usage Adaptation[END_REF][START_REF] Wu | AURA: a hybrid approach to identify framework evolution[END_REF][START_REF] Xing | API-Evolution Support with Diff-CatchUp[END_REF].

Surprisingly, however, the problem of helping library maintainers anticipate the impact of their changes and plan accordingly-the other side of the coin-has received little attention. Libraries are generally accountable to their clients for providing stability [START_REF] Bogart | When and How to Make Breaking Changes: Policies and Practices in 18 Open Source Software Ecosystems[END_REF]. We claim that library maintainers currently lack the necessary information and tool support to live up to this responsibility. Indeed, library maintainers have limited means to foresee the consequences of their actions on their clients [START_REF] Hora | How do developers react to API evolution? A large-scale empirical study[END_REF][START_REF] Robbes | How Do Developers React to API Deprecation? The Case of a Smalltalk Ecosystem[END_REF] and would benefit from knowing precisely how their APIs are used in client projects.

The consequences of breaking an API declaration used by many clients, by a popular library, or by commercial clients may be dramatic: clients may lose trust and migrate to another library. As a result, some very cautious maintainers refrain from changing existing declarations, thinking it might be too impactful, leading to an accumulation of technical debt and an aging design that drive developers away [START_REF] Bogart | When and How to Make Breaking Changes: Policies and Practices in 18 Open Source Software Ecosystems[END_REF]. Some more adventurous maintainers will push any change, disregarding all clients using the affected declaration, at the risk of breaking too many clients and having to revert the change ultimately [START_REF] Shimagaki | Why are Commits Being Reverted?: A Comparative Study of Industrial and Open Source Projects[END_REF]. We believe that the sweet spot lies at the crossroads of innovation and stability. Unfortunately, library maintainers have limited means to acquire evidence-based knowledge about the use of their API, and they often rely on their own judgment and subjective knowledge of their community to figure out how to evolve it [START_REF] Zhang | Enabling Data-Driven API Design with Community Usage Data: A Need-Finding Study[END_REF].

In order to assist maintainers in the evolution of their libraries while avoiding breaking their clients unexpectedly, we propose the following vision: empowering library maintainers to make evidencebacked decisions regarding the evolution of their libraries by accurately and efficiently informing them about breaking changes and their impact on relevant clients. Concretely, this will enable maintainers to ask what-if questions and explore what consequences evolving their library would have on their clients: "What if we push this refactoring?", "What if we alter this method's signature?", etc.

In this paper, we introduce a proof-of-concept implementation of our vision: BreakBot, a GitHub bot that analyzes the content of Pull Requests (PRs) in Java repositories and employs static analysis to identify BCs and their impact on relevant clients. The resulting reports are fed back into the PRs, enabling maintainers to quickly review changes and their impact and decide whether they should be accepted. Because our tool employs static analysis, we focus on source-and binary-incompatible changes, which are an important class of BCs as they force clients to modify their code at upgrade time, and leave behavioral changes aside. We expect our vision and its implementation in BreakBot to fight immobility and stagnation by pushing maintainers to introduce BCs when it is safe and avoid pushing these changes when they are too impactful. Making the co-evolution of library and clients more harmonious should benefit library maintainers and client developers alike.

BACKGROUND & MOTIVATING EXAMPLE

Deciding whether and when to introduce BCs is no trifling matter. The decision depends on values and constraints established at the ecosystem and library levels. However, the desire to preserve stability and to prioritize the sense of community are generalized concerns among all projects in most ecosystems [START_REF] Bogart | When and How to Make Breaking Changes: Policies and Practices in 18 Open Source Software Ecosystems[END_REF]. To enact stability, libraries tend to assume that BCs are always harmful [START_REF] Abbas | Dependency Smells in JavaScript Projects[END_REF]. This reasoning has led to the misconception that stability is equivalent to change stagnation [START_REF] Bogart | When and How to Make Breaking Changes: Policies and Practices in 18 Open Source Software Ecosystems[END_REF][START_REF] Businge | How Stable Are Eclipse Application Framework Internal Interfaces?[END_REF][START_REF] Mcdonnell | An Empirical Study of API Stability and Adoption in the Android Ecosystem[END_REF]. However, we claim that stability is the capability to not break clients even under the presence of change. What truly affects stability are changes that break clients, not BCs themselves. This distinction is essential: it has been shown that, for instance in Maven Central, most BCs do not impact the clients whatsoever [START_REF] Ochoa | Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central[END_REF]. Another problem strikes when evidence about how changes are impacting clients is not communicated, increasing the mistrust in the community. Thus, some evolving libraries opt for communicating change via their API, versioning conventions such as semantic versioning, or official documentation such as release notes and changelogs. However, these artifacts might be seen as promises that, when broken, result in further mistrust. Furthermore, official documentation tends to be incomplete given that only notable changes are requested to be included [START_REF] Lacan | Keep a ChangeLog[END_REF].

To illustrate the issues that library maintainers run into when introducing BCs, we take an example brought to our attention by a core maintainer of Spoon-a source code analysis and transformation library for Java developed on GitHub [START_REF] Pawlak | Spoon: A Library for Implementing Analyses and Transformations of Java Source Code[END_REF]. Spoon developers are very concerned about stability and therefore avoid BCs. When they do want to introduce a BC, they use deprecation to warn clients before proceeding with the change. In pull request PR#2683 a new feature to pretty-print Java import statements was rolled in, making a previous implementation-the ImportScanner interface and corresponding ImportScannerImpl implementation-obsolete. Therefore, the author of the PR labeled both types as deprecated. Two months later, in PR#3184, Spoon developers removed a set of deprecated types and methods from the code, notably the two types mentioned above. The PR was merged, making its way into Spoon's main branch. However, three months later, Spoon's developers noticed that these changes broke two important clients (Astor [START_REF] Martinez | ASTOR: A Program Repair Library for Java[END_REF] and DSpot [START_REF] Danglot | Automatic Test Improvement with DSpot: A Study with Ten Mature Open-Source Projects[END_REF]) and realized that the ImportScanner interface and implementation were in fact used by clients and thus reconsidered the deprecation. They finally reintroduced them in PR#3266.

It is evident that having to revert changes after introducing them is not ideal. If other changes had been made based on the original changes, they would have had to be reverted too, triggering a displeasing ripple effect within the library's code. How could this have been avoided? To the best of our knowledge, there are three approaches enabling library maintainers to evaluate the BCs they introduce.

Regression testing. The traditional approach is to use a regression test suite. However, a regression test suite cannot cover every possible use of the library, and therefore unanticipated BCs may leak to the clients. In addition, when the test suite detects a regression, it is not able to evaluate its impact on the library's clients. Maintainers must resort to their subjective opinion to decide whether or not to include the change. In our example case, Spoon's test suite noticed the BCs introduced in PR#3184, and the ImportScannerTest class was deleted as part of the same PR. However, since the deleted types were deprecated, the maintainers did not reconsider the change.

Static analysis. Various tools can scan two versions of a library using static analysis to output the list of BCs between them [START_REF] Jezek | API Evolution and Compatibility: A Data Corpus and Tool Evaluation[END_REF]. This approach is more complete than regression testing (although false positives or negatives can arise), therefore it is unlikely that BCs will leak to clients. For instance, Guava relies on JDiff [START_REF] Google | API Differences between Guava 30.0-jre and Guava 30[END_REF] while the Apache foundation uses japicmp [START_REF] Gregory | How we handle binary compatibility at Apache Commons[END_REF] to systematically search for BCs. Nevertheless, these tools only list the introduced BCs without showing the actual impact on client projects. If Spoon's developers were using this type of tool, it would have noticed the changes the same way their test suite did.

Reverse dependency compatibility testing. The third approachthe only one able to evaluate the impact on clients-is Reverse Dependency Compatibility Testing (RDCT) [START_REF] Zimmermann | Challenges in the collaborative evolution of a proof language and its ecosystem[END_REF]. The idea of RDCT is to identify a set of relevant clients, retrieve their source code, inject the new version of the library in their dependencies, and finally build them and run their test suite. If any error is detected during the process, maintainers are notified and can analyze the issue by browsing through the build log files. Projects like Scala, Rust, or Spoon perform periodical RDCT on their clients. The Coq project goes a step further by directly integrating RDCT as part of its Continuous Integration (CI) process, so that any BC impacting a client is detected right away [32].

However, RDCT suffers from several drawbacks. First, selected clients must always be buildable and have a test suite that passes on the previous version of the library. Second, RDCT takes a considerable amount of time and computational resources to build and test every client. For instance, the Rust project reports requiring up to a week to check 74,234 clients, as of September 2019. Third, when BCs are introduced, the resulting log files are verbose, forcing the maintainers to filter irrelevant information to understand the root cause. Last, whenever a BC is merged, the impacted clients can no longer be used for subsequent RDCT until they are repaired. For this reason, Coq's maintainers resort to fixing the clients themselves to keep their CI process running, which requires considerable effort.

In our example case, Spoon's RDCT runs every night on 13 projects. Following PR#3184, it ran into errors for the Astor and DSpot clients. An analysis of the log file of Astor (Listing 1), containing 266 errors among 704 log statements, revealed that the deleted types raised missing symbol errors at compile-time. Thanks to RDCT, Spoon developers discovered the impact of the BC. However, Astor's RDCT Listing 1: Extract of Astor's build log file (704 lines in total). build was failing for three months before the revert and thus could not be used to check for other BCs in this period. In summary, while RDCT helps analyze the impact of BCs on clients, its drawbacks severely hinder its usability and adoption. In the next section, we present a novel and lightweight approach based on static analysis to evaluate the impact of source-and binaryincompatible BCs on clients and integrate it within the maintainers' workflow. Naturally, RDCT still goes beyond our approach regarding behavioral-incompatible changes, at the expense of building and running the client test suites.

BREAKBOT

BreakBot (https://github.com/alien-tools/breakbot) is a proof-ofconcept implementation of our vision. It analyzes libraries and clients written in Java and integrates with GitHub's pull-based development workflow. It is implemented as a GitHub bot that provides library maintainers with the right information whenever and wherever needed. Making BreakBot a bot allows us to fully automate the process and be as close possible to the maintainers' development workflow. Because BreakBot feeds helpful information directly in GitHub's PRs, we expect it to enhance the code review process, as was shown for other bots in previous work [START_REF] Wessel | Effects of Adopting Code Review Bots on Pull Requests to OSS Projects[END_REF].

Overview

Once maintainers install BreakBot on a library's repository and configure a list of clients of interest, BreakBot registers itself via the GitHub's Checks API [START_REF]GitHub Checks API[END_REF] to analyze each new or updated PR. Then, it reports: i) the list of BCs they introduce, and ii) the impact of those BCs on the configured list of clients. Under the hood, BreakBot leverages a novel static analysis tool, Maracas [START_REF]Maracas[END_REF], to analyze libraries and clients and to infer the list and impact of BCs [START_REF] Ochoa | Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central[END_REF]. Note that BreakBot's design enables it to be reused with little effort for other analysis frameworks similar to Maracas, targeting other programming languages. Figure 1 gives a high-level view of BreakBot. We detail below the main steps of the process.

➊ Breaking changes analysis. In a first step, BreakBot communicates to Maracas the GitHub PR to analyze. In every PR, the head pointer points to the branch containing the PR's code, while the base pointer points to the branch where the changes should be pulled into. The diff between head and base represents the changes introduced by the PR. Maracas first clones and builds both these versions to obtain binary JARs, uses japicmp to extract the list of BCs, and stores it in a dedicated Δ-model [START_REF] Ochoa | Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central[END_REF]. Each BC in a Δmodel specifies the affected declaration (a particular type, method, or field) and the kind of BC (e.g., method removed, field now final, class less accessible). A configuration file hosted on the library's repository allows maintainers to specify how to build their library and which parts of their API should be excluded from the analysis because they are exempt from compatibility guarantees (e.g., @Experimental declarations and * test * packages).

➋ Impact analysis. In a second step, for each configured client, Maracas builds an impact model that pinpoints how the changes in the Δ-model impact the project. While we originally designed Maracas to analyze the impact of BCs on bytecode [START_REF] Ochoa | Breaking Bad? Semantic Versioning and Impact of Breaking Changes in Maven Central[END_REF], we redesign it to statically analyze their impact on source code using the Spoon framework. Because this new implementation does not require clients to be built, it frees BreakBot from the limitations of state-of-the-practice RDCT regarding build times and the need to repair clients constantly. Each location in client code impacted by a BC is associated with: i) the impacted AST node in client code (e.g., a variable, a method invocation, an import); ii) the library declaration it uses that is impacted by a BC, and; iii) the way the client is using the declaration (e.g., through class instantiation or extension, as a type argument, through field access).

➌ Report builder. In a third step, the Δ-model and impact models are sent back to BreakBot, which builds a final report incorporated directly within the PR. This report displays the result of the analysis in a way that library maintainers can easily interpret. We design the report to highlight the essential information first: the list of BCs, how each of them individually impacts client code, and how many of the clients are impacted. Then, a more detailed report describes, for each client, every location in its source code impacted by the changes. To make it easier to understand the causes and consequences of BCs, every code reference in the report directly links to the corresponding source code on GitHub.

Revisiting the Spoon Example

We used BreakBot and Maracas to analyze PR#3184 and showcase how it alleviates the drawbacks of RDCT identified in Section 2. Figure 2 depicts the most important parts of the report.

First, because Maracas employs static analysis to infer the impact of BCs, it only needs to parse the clients to build their ASTs. It does not require clients to be built and thus avoids the need for a complex CI system that may suffer from additional failures: dependency resolution, flaky tests, etc. Besides, while Spoon's RDCT took 6 minutes to analyze Astor only, BreakBot could analyze all 13 clients in 2 minutes.

Second, RDCT suffers from the need to analyze verbose and cryptic log files for each client, separated from the PR's code, riddled with unrelated outputs and errors, and messages that do not explicitly map client errors to their root cause at the library level (Listing 1). In contrast, the report built by BreakBot centralizes and highlights the essential information: the PR introduces 13 BCs, causing 32 broken locations in client code, and 5 out of 13 clients are impacted by the changes (Figure 2c). The remainder of the report details, for each client, which particular locations are impacted to let the maintainers understand why the clients break and how they might improve the PR (Figure 2b for Astor). In this particular case, BreakBot accurately reports that the class removals impact Astor and DSpot, but also that the type CompilationUnit, deprecated in the same PR, is used by clients (Figure 2a). This helps maintainers reflect on whether deprecating this type is appropriate to avoid making the same mistake again. The removal of the MinimalImportScanner type, however, does not impact any client and is thus innocuous. The final report effectively answers the what-if questions stemming from the PR.

FUTURE PLANS

Our first implementation of the vision, BreakBot, is a mere stepping stone. We intend to implement the following research roadmap.

Configurability. Policies to deal with backward compatibility range from permissive (e.g., npm, favoring innovation) to restrictive (e.g., Eclipse, favoring compatibility) [START_REF] Bogart | When and How to Make Breaking Changes: Policies and Practices in 18 Open Source Software Ecosystems[END_REF]. Regardless of their posture, projects should be given the necessary information to decide whether to introduce a change. Maintainers, therefore, need highly configurable impact analysis tools to anticipate the impact of their changes. What are, then, the configuration properties that impact analysis tools should capture? For instance, defining the type and number of allowed BCs, the parts of the API subject to analysis (e.g., non-experimental interfaces), and the key set of relevant clients (e.g., commercially-related projects), is of foremost importance. Discovery of clients. Knowing how BCs impact relevant clients is key when deciding to apply or revert changes. Some projects, such as Coq and Spoon, have a precise list of relevant clients, but not all projects do. Even when such a list exists, it might be incomplete and hide important evidence. Automatically discovering a diverse and representative sample [START_REF] Nagappan | Diversity in software engineering research[END_REF] of clients should give a clearer picture of the library's usage. To do so, one must first pinpoint which versions of the library are affected by a given BC, and thus which clients may be affected. As it is likely that clients' usage of a library will overlap [START_REF] Harrand | API beauty is in the eye of the clients: 2.2 million Maven dependencies reveal the spectrum of client-API usages[END_REF], it is essential to avoid analyzing similar clients multiple times, e.g., by clustering clients around their usage of the library.

Reporting actionable insights. BreakBot identifies the list of library declarations introducing BCs, and the list of impacted locations in client code. Although it improves the state of practice, it may still be hard for developers to interpret this information. Regarding BCs, should they be represented as atomic modifications to the code, or should more meaningful operations that show the developer's intentions (e.g., refactorings) be listed instead? Then, what is the right level of granularity to represent impacted client code (e.g., at the declaration, line, or AST node level)? Finally, to decide whether to introduce a BC, how to measure its criticality? How to define proper thresholds for metrics to enact project-specific policies? Approach evaluation. To evaluate BreakBot, we plan to use both quantitative and qualitative evaluation methods. We will first investigate the accuracy of Maracas during the BCs and impact analysis phases using a proper benchmark such as the one introduced by Jezek and Dietrich [START_REF] Jezek | API Evolution and Compatibility: A Data Corpus and Tool Evaluation[END_REF]. Second, we aim at better understanding the current state of practice when dealing with change and the effects of BreakBot on these practices. Therefore, we plan to deploy BreakBot on the GitHub repositories of a few libraries (e.g., Spoon, ASM, JavaParser, Vallang, GumTree) and interview their maintainers. In particular, we plan to inquire whether BreakBot improves the analysis of code changes during code reviews. Does BreakBot "scare" developers more, or does it encourage them to make changes they were afraid of because they could not foresee their impact?

Figure 1 :

 1 Figure 1: BreakBot handles the communication between GitHub and Maracas to produce meaningful reports.

Figure 2 :

 2 Figure 2: Excerpts of the report generated by BreakBot for Spoon's PR#3184 on GitHub

1

  Started by timer . . . 211 [ERROR] The goal you specified requires a project to execute but there is no POM in this directory.

	. . .
	226 [ERROR] Entities.java:[63,29] unmappable character for UTF-8
	. . .
	295

[ERROR] TypeUtilsTest.java:[524,47] incompatible types: inferred type does not conform to upper bound(s) . . . 654 [ERROR] LibParser.

java:[23,29] cannot find symbol 655 [ERROR] symbol: class ImportScanner

ACKNOWLEDGMENTS

We thank Théo Zimmerman, Matias Martinez, and Jurgen Vinju for their feedback on earlier versions of this manuscript, and Léonard Rizzo for the initial implementation of BreakBot. This work was partially funded by the French National Research Agency through grant ANR ALIEN (ANR-21-CE25-0007).


Priyantha Wijayatunga
email: priyantha.wijayatunga@umu.se

Resolving the Lord's Paradox

Keywords: Effect, Predictive, Causal, Confounding

An explanation to Lord's paradox using ordinary least square regression models is given. It is not a paradox at all, if the regression parameters are interpreted as predictive or as causal with stricter conditions and be aware of laws of averages. We use derivation of a super-model from a given sub-model, when its residuals can be modelled with other potential predictors as a solution.

Introduction

In 1967 Frederic Lord posed following question (see [START_REF] Lord | A Paradox in the Interpretation of Group Comparisons[END_REF][START_REF] Pearl | Lord's Paradox Revisted -(Oh Lord Kumbaya!)[END_REF]) that became a paradox among applied statistical community. To see effects and if there is any sex difference of diet provided in a university weights of students at time of their arrival and those a year later are recorded. The data are independently examined by two statisticians. The first examines the mean weight of the girls at the beginning and at the end of the year, and finds that they are to be identical, i.e., frequency distribution of the weight for the girls is not changed, so is for the boys. The second statistician finds that the slope of the regression line of the final weight on the initial weight is essentially the same for both sexes but the regression coefficient of the variable sex to be statistically significant and concludes that the boys showed significantly more gain in weight than the girls when proper allowance is made for differences for initial weight. Conclusions of the two statisticians seem to contradict with each other; the first is predictive and the second is both predictive, and causal if the initial weight is the only confounder of causal relation between the sex and the final weight. The second has given causal effect of the sex on the final weight (weight gain) by a regression coefficient. In fact, to give it by comparing, two supports of the confounder of both sexes should coincide. But one can assume that the population initial weight ranges of boys and girls coincide even though sample counterparts differ (so, extrapolation is meaningful). Let the initial weight, final weight and sex are denoted by WI , WF , S respectively (S = 0, a girl and S = 1, a boy) and weight gain be D = WF -WI . If the effect of This paper was published as a part of the proceedings of the 32nd International Workshop on Statistical Modelling (IWSM), Johann Bernoulli Institute, Rijksuniversiteit Groningen, Netherlands, 3-7 July 2017. The copyright remains with the author(s). Permission to reproduce or extract any parts of this abstract should be requested from the author(s).

S on D is found by difference of conditional means, E{D|S = 1} -E{D|S = 0} then it is no effect. This can be found by running regression of D on S. Note that E{WF |S = 1} = E{WI |S = 1}, (say, µB) and E{WF |S = 0} = E{WI |S = 0}, (say, µG). If E{D|WI = i, S = 1} and E{D|WI = i, S = 0} are calculated simply by partitioning the data by taking WI to be discrete or as a functions of i, then the difference E{D|WI = i, S = 1} -E{D|WI = i, S = 0} may not be zero for each i, so may be difference of their weighted means, i E{D|WI = i, S = 1}p(WI = i)i E{D|WI = i, S = 0}p(WI = i). If the effect of S on WF is calculated by it then it is different from former value (paradoxical!). Now let us see why two types of differences of averages differ by simple algebra, that will say that they should have two different interpretations. First assume that we have a number of subgroups of boys and, for simplicity, the same is true for girls. Let D 1 ij be the weight gain of the j-th boy in the i-th subgroup of boys where sub-group size is ni and D 0 ij be that of the girls where sub-group size is mi and furthermore, let

f 1 i = ni/ k n k , f 0 i = mi/ k m k and fi = (ni + mi)/ k (n k + m k ) for j = 1, ..., ni and i = 1, ..., a.
And let A1 be difference of the average weight gain of the boys and the girls,

D1 i = j D 1 ij /ni and D0 i = j D 0 ij /mi for i = 1, ..., a. So, A1 = a i=1 n i j=1 D 1 ij a i=1 ni - a i=1 m i j=1 D 0 ij a i=1 mi = a i=1 D1 i f 1 i - a i=1 D0 i f 0 i = 1 2 a i=1 D1 i f 1 i + a i=1 D1 i f 0 i - a i=1 D0 i f 0 i - a i=1 D0 i f 1 i ; generally = 1 2 a i=1 ( D1 i -D0 i )(f 1 i + f 0 i ) = a i=1 ( D1 i -D0 i )fi = A2
where fi = αf 1 i + (1 -α)f 0 i for i = 1, ..., a such that α = a i=1 ni/ a i=1 (ni + mi) and A2 is the difference of weighted averages of the sub-group weight gain averages. So, the difference of group averages A1 (which is zero in our case) is different from the difference of pooled-weighted average of the sub-group averages A2. The second statistician compares the boys and the girls subgroup-wise and finds that it is a constant gain for the boys over the girls across the subgroups, i.e., D1

i -D0 i is constant for all initial weight i. Therefore he finds that the boys gain more weight than the girls in corresponding sub-groups. Note that for simplicity we have taken initial weights as discrete values. In fact,

A2 = i E{D|S = 1, WI = i}p(WI = i) -i E{D|S = 0, WI = i}p(WI = i) is the causal effect of S on D if WI
is the only confounder, under the linear assumption. It is different from A1 unless E{D|S, WI } = E{D|S}. The confounding effect (A2 -A1) depends on how different f 1 and f 0 are (can have a measure from them).

Regression Solution

Now we define interpretation of ordinary least square (OLS) estimates of the regression coefficients (parameters). The OLS estimation is based on the variation of the response variable Y for a given functional form of the values of explanatory factors. Regression coefficients are estimated so that sum of squared prediction errors for the data in the sample is the minimum. So, reverse regression is not generally obtainable from forward regression and may not be consistent with the latter. For simple linear regression one can easily establish that the reverse regression and the forward regression are consistent with each other if and only if one of the regressions have symmetric residuals about and uni-modal at conditional expectation of response, that implies other regression too. Now consider the OLS linear regression model Y = β0 + β1X1 + β2X2 + , then linear effect of X1 on Y when X2 is held unchanged is given by β1 if Y values are symmetric about and uni-modal at β0 + β1X1 + β2X2. It is clear that the supports of X2 for each value of X1 are the same (or extrapolation is meaningful if empirical supports differ). Symmetry and uni-modality of Y values for given values of X1 and X2 are observed if all other factors that affect or are associated, but are not taken into consideration are allowed to vary pure randomly. This is a fundamental assumption used in statistical modelling often implicitly. Let us do a regression of WF on the binary variable S. Then we get the model WF = µG +(µB -µG)S + 1. where the regression co-efficient of S is the predictive effect of S on WF provided that above requirement is fulfilled. The residuals of the model are just individual values of D, i.e., 1 = D for each subject and it is easy to see in Fig. 1 of [START_REF] Lord | A Paradox in the Interpretation of Group Comparisons[END_REF], that the residuals are predictive by WI for each sex category separately, 1 ⊥ WI |S. However, it may be that 1 ⊥ S. So, if the two clusters of values of WF for two sexes are symmetric about and uni-modal at the respective means then the effect of S on WF is the regression coefficient of S in the model. But it is uncontrolled confounders that are associated with WF , then it should be interpreted accordingly. That is, it is the predictive effect of sex differences and causal if there are no confounders such as WI . And we see that we get zero predictive effect from the meal change since the regression coefficient is the same as that when the girls and boys had previous meal type. Let we can write the distribution of residuals for each value s of S, say, f ( 1|s) as a mixture, f ( 1|s) = g( 1|x, s)π(x, s)dx for some random variable X, and for each value x of X the component distribution g( 1|x, s) may have non-zero mean such that E{ 1|s} = E{ 1|x, s}π(x, s)dx = 0 and then we have that V ar{ 1|x, s} ≤ V ar{ 1|s} where π(x, s) = h(x|s)p(s); here h(x|s) is the conditional probability density of X given S = s and p(s) is the marginal probability distribution of S. If X could be identified meaningfully, then model should include such feature variables too. In this case, X could be identified as the initial weight WI . Then one should accept the upgraded model that includes WI too. It has residuals that have a smaller conditional standard deviation given WI and S. Furthermore, if WI is the only confounding factor and when it is also included in the model the the coefficient of S is the causal effect of S on WF . Let the residual 1 corresponds to the context that WI = wI and S = s and then it can be written as 1 = µ w I ,s 1 + 2 where µ w I ,s 1 is the expectation of it. So, we have E{ 2|WI = wI , S = s} = 0 and also that V ar{ 2|WI = wI , S = s} ≤ V ar{ 1|S = s}. And furthermore, we can have that µ w I ,0 1 = a0 + b0wI for s = 0 and µ w I ,1 1 = a1 + b0wI for s = 1 where a0, b0 and a1 are constants. Now, given that WI = wI and S = s, for s = 0, 1, and I(A) = 1 when A is a true statement and I(A) = 0 otherwise, we have

WF = WF = µG + (µB -µG)s + µ w I ,s 1 + 2 = µG + (µB -µG)s + (a0 + b0wI )I(S = 0) + (a1 + b0wI )I(S = 1) + 2 = µG + (µB -µG)s + a0I(S = 0) + a1I(S = 1) + b0wI + 2 = µG + (µB -µG)s + a0(1 -s) + a1s + b0wI + 2 = µG + a0 + (µB -µG -a0 + a1)s + b0wI + 2
So we can obtain a super-model (regression) from a given regression model (it is a sub-model of the former) as long as its residuals are predictive (linearly in this case) with another explanatory variable. The predictive effect of S on WF when controlled for WI is µB -µG -a0 + a1 that is generally different from earlier value of µB -µG and for each individual model prediction is more accurate than that of the previous model, therefore new model is preferred to the previous one. If WI is only a confounder but not an intermediate variable between the causal pathway between S and WF , and has a common support for all values of S, then β1 is the average causal effect of S on WF in the linear case. In our example, sample supports of WI for S = 1 and S = 0 differ but we can assume that they are the same in the population (so, extrapolation is meaningful). Note that the above arguments can be generalised. For restrictions of space, we avoid presenting solution to the paradox, that is based on causal diagrams. We object recent solution by Pearl. Our explanations comply with Lord's initial comments.


Xuechen Liu

Md Sahidullah

Tomi Kinnunen

LEARNABLE NONLINEAR COMPRESSION FOR ROBUST SPEAKER VERIFICATION

Keywords: Speaker Verification, Nonlinear Compression, Multi-Regime Compression

In this study, we focus on nonlinear compression methods in spectral features for speaker verification based on deep neural network. We consider different kinds of channel-dependent (CD) nonlinear compression methods optimized in a data-driven manner. Our methods are based on power nonlinearities and dynamic range compression (DRC). We also propose multi-regime (MR) design on the nonlinearities, at improving robustness. Results on VoxCeleb1 and Vox-Movies data demonstrate improvements brought by proposed compression methods over both the commonly-used logarithm and their static counterparts, especially for ones based on power function. While CD generalization improves performance on VoxCeleb1, MR provides more robustness on VoxMovies, with a maximum relative equal error rate reduction of 21.6%.

INTRODUCTION

Automatic speaker verification (ASV) [START_REF] Hansen | Speaker recognition by machines and humans: A tutorial review[END_REF], [START_REF] Bai | Speaker recognition based on deep learning: An overview[END_REF] is the task of verifying a person's identity using his or her voice. Modern ASV systems consists of three main components: acoustic feature extractor, speaker embedding extractor, and back-end classifier. In recent years, substantial improvement has been achieved by using deep neural networks (DNNs) to implement, especially the last two components. Concerning speaker embedding extractor, statistical models such as i-vectors [START_REF] Dehak | Front-end factor analysis for speaker verification[END_REF] have been replaced by deep models such as x-vector with time-delayed neural network (TDNN) [START_REF] Snyder | X-vectors: Robust DNN embeddings for speaker recognition[END_REF]. As for the back-end, recent studies have replaced probabilistic linear discriminant analysis (PLDA) [START_REF] Ioffe | Probabilistic linear discriminant analysis[END_REF] with neural approaches [START_REF] Ramoji | NPLDA: A deep neural plda model for speaker verification[END_REF].

Concerning features, however, many ASV systems still use melfrequency cepstral coefficients (MFCCs) [START_REF] Davis | Comparison of parametric representations for monosyllabic word recognition in continuously spoken sentences[END_REF], which are not specialized for ASV and neglects information such as phase and temporal characteristics [START_REF] Loweimi | Robust source-filter separation of speech signal in the phase domain[END_REF]- [START_REF] Kim | Power-normalized cepstral coefficients (PNCC) for robust speech recognition[END_REF]. Meanwhile, spectrograms are also widely used [START_REF] Nagrani | Voxceleb: A large-scale speaker identification dataset[END_REF]- [START_REF] Xie | Utterance-level aggregation for speaker recognition in the wild[END_REF]. There are multiple types of spectrograms such as raw one where no filter is applied [START_REF] Nagrani | Voxceleb: A large-scale speaker identification dataset[END_REF], [START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF] and more widely-used spectral energies output with mel filters [START_REF] Garcia-Romero | MagNetO: X-vector magnitude estimation network plus offset for improved speaker recognition[END_REF], [START_REF] Desplanques | ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification[END_REF]. Even if the spectral representations are usually higher-dimensional (hence, more expressive) than MFCCs, problems of lacking specialization and missing information remain.

There are also attempts to replace hand-crafted features with neural networks [START_REF] Ravanelli | Speaker recognition from raw waveform with sincnet[END_REF]- [START_REF] Lin | Wav2Spk: A simple DNN architecture for learning speaker embeddings from waveforms[END_REF]. However, such design may be hard to interpret. Moreover, many state-of-the-art DNN extractors are based on convolutional kernels, whose modeling capability on variabilities across different frequency (channel or subband) components have been questioned [START_REF] Ravanelli | Speaker recognition from raw waveform with sincnet[END_REF], [START_REF] Loweimi | On learning interpretable CNNs with parametric modulated kernel-based filters[END_REF]. These potential shortcomings motivate the idea of optimizing signal processing modules of feature extractor, including spectrogram-based features. Such topic has been addressed recently for audio representation learning [START_REF] Won | Data-driven harmonic filters for audio representation learning[END_REF] and ASV [START_REF] Liu | Learnable MFCCs for speaker verification[END_REF], but expanding and optimizing nonlinear compression module has received less attention.

This study, motivated by the above, addresses channel-dependent (CD) nonlinear compression of spectrogram energies. This is realized, as presented in Section 2, by expanding the nonlinearity from a channel-independent to channel-dependent operation. Similar ideas on mel spectrogram have been effective in keyword spotting [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF], audio classification [START_REF] Ick | Sound event detection in urban audio with single and multi-rate pcen[END_REF], and far-field speaker verification [START_REF] Liu | Parameterized channel normalization for far-field deep speaker verification[END_REF]. However, to the best of our knowledge, it has not been applied to various nonlinear compression methods in the task we consider.

Our main contributions are summarized in two folds: 1) We leverage the power of such channel-dependent setting by revisiting two established nonlinear compression methods that have been efficient in previous works and generalized them to be channeldependent, namely power function and dynamic range compression; 2) In order to capture different level of variabilities and compromise instabilities during the joint optimization, we propose a multi-regime (MR) design based on CD.

NONLINEAR COMPRESSION IN ACOUSTIC FEATURE EXTRACTION

When using spectrogram to extract features, as illustrated in Fig. 1, we typically apply logarithmic compression to spectral energies. However, logarithm has a singularity at zero. This problem is often addressed by adding a small positive offset: log(x + offset). Even if it avoids the singularity, the ad-hoc design still lacks specificity to a given task and has unpredictable impacts for different kinds of input [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF]. We consider two alternative parameterized methods and further make them to be channel-dependent, which are described below.

Power Function

The concept of applying power nonlinearity to compress the signal amplitude is inspired by human-auditory processing [START_REF] Stern | Signal processing for robust speech recognition[END_REF], [START_REF] Kim | Power-normalized cepstral coefficients (PNCC) for robust speech recognition[END_REF]. By using X and Y to denote the input and output magnitude spectra respectively, power nonlinearity is expressed as:

Y [t, f ] = X[t, f ] 1/α , (1)
where α is known as temperature coefficient for the compression. t and f are the time and channel indices, respectively, for spectrogram energies. Experimentally, two particular values of α have been popular in speech front-ends. The first one is α = 3, known as cube-root [START_REF] Hermansky | RASTA processing of speech[END_REF]- [START_REF] Sadjadi | Mean hilbert envelope coefficients (MHEC) for robust speaker and language identification[END_REF]. The other one is α = 15, known as power-law [START_REF] Kim | Signal processing for robust speech recognition motivated by auditory processing[END_REF]. Setting higher values of α can provide better recognition performance in the presence of white noise, while lower values may be required for maintaining accuracy for cleaner speech [START_REF] Kim | Power-normalized cepstral coefficients (PNCC) for robust speech recognition[END_REF].

Fig. 1: Feature extraction with log and proposed compression methods, using power function as example.

Dynamic Range Compression

Power nonlinearity neither addresses foreground-background noise nor other variations. These problems can be addressed by applying dynamic range compression (DRC). It was proposed and applied originally to far-field keyword spotting as part of per-channel energy normalization (PCEN) [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF], [START_REF] Lostanlen | Per-channel energy normalization: Why and how[END_REF]. PCEN has also been applied recently in audio event detection [START_REF] Ick | Sound event detection in urban audio with single and multi-rate PCEN[END_REF]. Using the same notation as above, the DRC operation is defined by:

Y [t, f ] = (X[t, f ] + δ) r -δ r , (2)
where δ > 0 is a positive bias and r is the exponential offset. This method bears resemblance to spectral subtraction [START_REF] Porter | Optimal estimators for spectral restoration of noisy speech[END_REF] in speech denoising. In the context of PCEN, it is applied to the spectral energies processed with automatic gain control (AGC) [START_REF] Prez | Automatic Gain Control: Techniques and Architectures for RF Receivers[END_REF].. In this study, under the framework of DNN-based ASV, we investigate the efficiency of DRC by directly applying it to spectral energies.

PROPOSED METHOD

Channel-Dependent Design

While the control parameters in Eq. ( 1) and ( 2) can be set by hand, this may lead to suboptimal performance in a recognition task. Related prior studies on channel-dependent compression utilize information such as loudness and signal-to-noise ratio (SNR) [START_REF] Nasersharif | A framework for robust MFCC feature extraction using SNR-dependent compression of enhanced mel filter bank energies[END_REF], which motivates data-driven settings. From the equations, we can see that the parameters α, δ, and r are differentiable. Therefore, we propose to optimize them as part of the neural network by generalizing to their channel-dependent (CD) counterparts:

α = α(f ), δ = δ(f ), r = r(f )
, where f is the channel index. This design follows the proposal from [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF]. The generalized parameters are then jointly optimized with the neural network. Furthermore, we employ kernelized initialization where the parameters are initialized from their static counterparts [START_REF] Liu | Learnable MFCCs for speaker verification[END_REF].

Multi-Regime Design

Learnable parameters are tuned and selected by the training data during the learning process, thus might be suboptimal if domain mismatch between training and testing data is large, due to joint training which may let the parameters suffer from the overparameterized DNN models [START_REF] Liu | On the linearity of large non-linear models: When and why the tangent kernel is constant[END_REF]. CD generalization of the parameters with kernel initialization may scrutinize such problem by larger search space and proper starting point, but it still may fail to have a wide-enough coverage of different level of speech variabilities. This is especially the case when the DNN has large number of layers, which may cause the problem of vanishing gradient when being back-propagated to first early layers, then nonlinearities [START_REF] Loweimi | On the robustness and training dynamics of raw waveform models[END_REF]. Therefore, inspired by the design of multiple feature maps in image processing [START_REF] Gonzalez | Digital image processing[END_REF] and audio event detection [START_REF] Ick | Sound event detection in urban audio with single and multi-rate PCEN[END_REF], we use a multiregime (MR) design by passing the spectrum to multiple submodules, with shared compression algorithm, but different initialized parameters, as shown in Fig. 1. The output of each module is averaged to form the input for further operations. Using power function as an example:

Y [t, f ] = 1 N N i=1 (X[t, f ]) 1/α i (f ) (3)
We define the initial values by defining minimum and maximum to create N evenly spaced values: αi = αmin + (αmax -αmin) * i/(N -1), i = 1, ..., N , where αmax and αmin are maximum and minimum reference values and N denotes number of intermediate spectrograms generated. The setup for this work is shown in Table 1. For all cases in this work, N = 3. Further tuning of number of intermediates and parameter search is left as future work.

Method

CD MR-CD cube root [START_REF] Hermansky | RASTA processing of speech[END_REF] α = 3 αmax = 3, αmin = 1 power law [START_REF] Sadjadi | Mean hilbert envelope coefficients (MHEC) for robust speaker and language identification[END_REF] [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF] while for MR-CD it was hand-crafted based on [START_REF] Lostanlen | Per-channel energy normalization: Why and how[END_REF] and pilot experiments.

EXPERIMENTAL PROTOCOL

Data. For all experiments, we train the DNN speaker embedding extractor using the dev set of VoxCeleb2 [START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF], which consists of 5994 speakers. We report the performance of different methods on two evaluation sets: 1) The two test sets from VoxCeleb1 [START_REF] Nagrani | Voxceleb: A large-scale speaker identification dataset[END_REF] following [START_REF] Chung | VoxSRC 2019: The first voxceleb speaker recognition challenge[END_REF], known as VoxCeleb1-E and VoxCeleb1-H.

2) The recent Vox-Movies [START_REF] Brown | Playing a part: Speaker verification at the movies[END_REF], which overlaps with VoxCeleb1 in terms of speakers and contains various levels of mismatch between the enrollment and test utterances. It consists of five trial sets, denoted E-1 (easiest) through E-5 (hardest). Besides condition-specific results, we also report the pooled performance over the all five sets. Features. We use raw magnitude spectrogram obtained using short-time Fourier transform (STFT) as the time-frequency representation, to which different compression methods are applied, as illustrated in Fig. 1. The number of frequency bins NSTFT = 512 for all systems. The sampling rate is 16 kHz, and the STFT is computed using a [START_REF] Liu | Learnable MFCCs for speaker verification[END_REF] we include a system where the logarithm is factorized by an offset as part of the baseline, as mentioned in Section 2: log(x + offset). The offset is parameterized by an exponential function offset = exp(β), where β = β(f ) is CD and initialized with normal distribution. Speaker embeddings. We use x-vector with extended TDNN to generate speaker embeddings, following the design choice from [START_REF] Snyder | Speaker recognition for multi-speaker conversations using x-vectors[END_REF] with two main modifications: 1) We replace the statistics pooling layer with attentive statistics pooling [START_REF] Okabe | Attentive statistics pooling for deep speaker embedding[END_REF]; 2) Instead of multi-class cross-entropy, we use additive angular softmax [START_REF] Wang | Additive margin softmax for face verification[END_REF] as the training objective. We set the scaling factor s = 30 and the margin m = 0.2. We extract the embedding vectors from the first fullyconnected layer after the pooling layer. The extracted vectors are centered and projected via a 150-dimensional linear discriminant classifier (LDA).

Evaluation. For both VoxCeleb1 and VoxMovies, we train probabilistic LDA (PLDA) classifier using VoxCeleb1. We report ASV performance in terms of equal error rate (EER) and minimum detection cost function (minDCF). For minDCF, the target speaker prior is ptar = 0.01 and detection costs were Cfa = Cmiss = 1.0.

RESULTS

Speaker Verification

The results are presented in Table 2. Let us first focus on VoxCeleb1. The EERs for both of the two power functions are improved from both the logarithm baseline and their static counterparts (marked as '-' in the 'Design' column of the table) by the CD design. As part of the baseline, applying CD exponential offset on the logarithm compression degrades the performance for both test sets. The lowest EER on both test sets is obtained using cube-root with CD, outperforming the baseline logarithm by 14.3% and 13.3%, respectively. This indicates the usefulness of CD. Nevertheless, the same design does not work well with DRC, which contradicts the findings reported in [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF] with mel spectrogram (in a different task, though). This indicates that in ASV with spectrogram input, DRC may not combine well with CD compression. For all compression methods, the MR-CD design degrades performance and fails to show substantial improvement over the logarithm. One reason could be suboptimal parameter initialization as in [START_REF] Wang | Trainable frontend for robust and far-field keyword spotting[END_REF], where the DRC parameters are set for far-field keyword spotting.

On the other hand, the trend is different for VoxMovies with more severe mismatch. Both pooled and condition-specific indicate that improvements from CD generalization are modest, as opposed with the observations from VoxCeleb1. For cube-root, CD actually degrades the performance for pooled and individual trial sets apart from E-3, where 14.7% relative EER reduction is obtained over the logarithm. However, for power-law CD improves upon its static counterpart, with lowest EER on E-1 across all systems.

Generalization via both MR and CD brings substantial improvements on nonlinearities based on power function. Lowest EER of pooled, E-2 and E-4 is obtained using cube-root with MR and CD. Its pooled performance outperforms its static counterpart by relatively 21.6%. We notice the same for power-law, whose pooled performance with MR and CD outperform its static version by relatively 15.5%. This indicates usefulness of MR in enhancing the robustness.

Nevertheless, the behavior of DRC is different from the power function. While its static setting reduced the EER from the logarithm by relative 18.2%, applying CD results in only slight relative EER reduction (3.1%) on pooled results and does not lead to better performance for the individual trial sets apart from E-4. Generalizing it with MR degrades the performance, which agrees with our observations for VoxCeleb. However, its static setting reaches the best performance across all methods. This indicates the parameters for DRC (bias and offset) being not suitable to cope with CD and MR, at least not within the DRC framework itself. Further investigation is needed on its parameterization.

Representation Analysis

We illustrate the learnt temperature representation of the two power nonlinear functions from VoxCeleb2 dev set (as described in Section 4) in Fig. 2. Note that for power function, larger magnitude of temperature parameters will result in higher compressing effect.

As shown in the figure, applying only CD on power functions casts more compression on both low and high frequency regions (higher temperature values imply more aggressive compression, according to Eq.( 1)). Meanwhile, applying MR results in relatively less compression on some of the middle frequency components as well as low frequency regions, while relatively maintaining its pattern on high frequency components. Interestingly, we see that both power function methods result in similar numerical range, even if their initialized values are very different (Table 1).

CONCLUSION

In this work, we have investigated the alternative nonlinearities for spectrogram compression and their dynamic, channel-dependent variants. We have extended their representation via channel-wise manner and utilized a multi-regime design based on it. Initialization on relevant parameters has been based on the corresponding static known values. We have evaluated the performance of proposed extended dynamic compression methods for different degree of mismatch conditions. Results demonstrates the efficacy of the proposed methods on power nonlinearities, with a maximum of 21.6% pooled EER reduction on VoxMovies. Future work may focus on: 1) extending the framework with other types of spectrogram; 2) exploring more advanced design and appropriate initialization and tuning methods, especially for DRC.

ACKNOWLEDGEMENTS

This work was partially supported by Inria Nancy Grand Est.

  α = 15 αmax = 15, αmin = 1 DRC δ = 2.0 δmax = 2.0, δmin = 1.0 r = 0.5 rmax = 1.0, rmin = 0.0

Fig. 2 :

 2 Fig. 2: Learnt temperature values of nonlinear compression based on power function. The x-axis denotes frequency (channel) bin index and y-axis measures the values.

Table 1 :

 1 Parameter settings for kernel initialization. DRC values in CD are from

Table 2 :

 2 ms Hamming window every 10 ms. Additionally, Speaker verification results on VoxCeleb and VoxMovies. 'CD' means channel-dependent and 'MR' means multi-regime setups. Rows with slight grey shades are regarded as baselines. '-' indicates the system with static nonlinearity without learning involved. For VoxMovies all results are reported in EER(%).

			VoxCeleb1-E	VoxCeleb1-H			VoxMovies
	Method	Design	EER(%) minDCF EER(%) minDCF	E-1	E-2	E-3	E-4	E-5	pooled
	log	-	2.23	0.2676	4.43	0.5371	10.55 16.82 15.17 24.73 19.55	20.64
	log(x + offset)	-	2.41	0.2920	4.93	0.6219	13.3	16.49 14.41 25.44 18.38	19.00
	cube-root	-CD MR-CD	1.93 1.91 2.24	0.2377 0.2329 0.2876	4.02 3.84 4.52	0.5090 0.5071 0.5421	10.68 14.74 13.00 25.21 16.61 10.84 14.79 12.93 25.71 16.67 10.92 12.99 13.77 15.86 15.79	18.21 18.33 14.27
	power-law	-CD MR-CD	2.14 2.14 2.78	0.2542 0.2505 0.3141	4.38 4.29 5.31	0.5239 0.5186 0.5683	11.09 15.15 13.58 25.96 17.13 10.25 13.44 13.23 21.76 15.82 12.07 14.27 15.35 18.05 16.92	18.70 16.39 15.80
		-	2.25	0.2598	4.60	0.5629	11.09 14.07 14.14 22.20 15.16	16.89
	DRC	CD	2.67	0.2993	5.20	0.6408	11.95 14.07 14.45 19.78 16.24	16.38
		MR-CD	2.90	0.3526	5.81	0.6442	12.96 15.48 16.70 19.55 16.94	17.03