Pawel Drozdowski
email: pawel.drozdowski@h-da.de

Christian Rathgeb

Antitza Dantcheva

Naser Damer

Christoph Busch

Demographic Bias in Biometrics: A Survey on an Emerging Challenge

published or not. The documents may come

Demographic Bias in Biometrics:

A Survey on an Emerging Challenge Pawel Drozdowski , Christian Rathgeb , Antitza Dantcheva, Naser Damer , and Christoph Busch Abstract-Systems incorporating biometric technologies have become ubiquitous in personal, commercial, and governmental identity management applications. Both cooperative (e.g., access control) and noncooperative (e.g., surveillance and forensics) systems have benefited from biometrics. Such systems rely on the uniqueness of certain biological or behavioral characteristics of human beings, which enable for individuals to be reliably recognized using automated algorithms. Recently, however, there has been a wave of public and academic concerns regarding the existence of systemic bias in automated decision systems (including biometrics). Most prominently, face recognition algorithms have often been labeled as "racist" or "biased" by the media, nongovernmental organizations, and researchers alike. The main contributions of this article are: 1) an overview of the topic of algorithmic bias in the context of biometrics; 2) a comprehensive survey of the existing literature on biometric bias estimation and mitigation; 3) a discussion of the pertinent technical and social matters; and 4) an outline of the remaining challenges and future work items, both from technological and social points of view. Index Terms-Bias, bias estimation, bias mitigation, biometrics, demographics, fairness.

I. INTRODUCTION

A RTIFICIAL intelligence (AI) systems increasingly sup- port humans in complex decision-making tasks. Domains of interest include learning, problem solving, classifying, as well as making predictions and risk assessments. Automated algorithms have in many cases already outperformed humans and hence are used to support or replace human operators [START_REF] Pasquale | The Black Box Society: The Secret Algorithms That Control Money and Information[END_REF]. Those systems, referred to as "automated decision systems," can yield various benefits, e.g., increased efficiency and decreased monetary costs. At the same time, a number of ethical and legal concerns have been raised, specifically relating to transparency, accountability, explainability, and fairness of such systems [START_REF] Osoba | An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence[END_REF]. Automated algorithms can be utilized in diverse critical areas, such as criminal justice [START_REF] Washington | How to argue with an algorithm: Lessons from the COMPAS-ProPublica debate[END_REF], healthcare [START_REF] Yu | Framing the challenges of artificial intelligence in medicine[END_REF], creditworthiness [START_REF] Hurley | Credit scoring in the era of big data[END_REF], and others [START_REF] Castelluccia | Understanding algorithmic decision-making: Opportunities and challenges[END_REF], hence often sparking controversial discussions. This article focuses on algorithmic bias and fairness in biometric systems w.r.t. demographic attributes. In this context, an algorithm is considered to be biased if significant differences in its operation can be observed for different demographic groups of individuals (e.g., females or dark-skinned people), thereby privileging and disadvantaging certain groups of individuals.

A. Motivation

The interest and investment into biometric technologies are large and rapidly growing according to various market value studies [START_REF] Bhutani | Biometrics market size by application[END_REF]- [START_REF] Thakkar | Global Biometric Market Analysis: Trends and Future Prospects[END_REF]. Biometrics are utilized widely by governmental and commercial organizations around the world for purposes, such as border control, law enforcement and forensic investigations, voter registration for elections, as well as national identity management systems. Currently, the largest biometric system is operated by the Unique Identification Authority of India, whose national ID system (Aadhaar) accommodates almost the entire Indian population of 125 billion enrolled subjects at the time of this writing, see the online dashboard [START_REF]Unique Identification Authority of India. Aadhaar Dashboard[END_REF] for live data.

In recent years, reports of demographically unfair/biased biometric systems have emerged (see Section III), fueling a debate on the use, ethics, and limitations of related technologies between various stakeholders, such as the general population, consumer advocates, nongovernmental and governmental organizations, academic researchers, and commercial vendors. Such discussions are intense and have even raised demands and considerations that biometric applications should be discontinued in operation, until sufficient privacy protection and demographic bias mitigation can be achieved. 1,2,3,4 Algorithmic bias is considered to be one of the important open challenges in biometrics by Ross et al. [START_REF] Ross | Some research problems in biometrics: The future beckons[END_REF].

B. Article Contribution and Organization

In this article, an overview of the emerging challenge of algorithmic bias and fairness in the context of biometric systems is presented. Accordingly, the biometric algorithms which might be susceptible to bias are summarized; furthermore, the existing approaches of bias estimation and bias mitigation are surveyed. This article additionally discusses other pertinent matters, including the potential social impact of bias in biometric systems, as well as the remaining challenges and open issues in this area.

The remainder of this article is organized as follows. The relevant background information is provided in Section II. Section III contains a comprehensive survey of the scientific literature on bias estimation and mitigation in biometric systems. Other relevant matters are discussed in Section IV, while concluding remarks and a summary are presented in Section V.

II. BACKGROUND

The following sections provide relevant background information with respect to (w.r.t.) the topic of bias in automated decision systems in general (Section II-A) and the basics of biometric systems (Section II-B). Furthermore, due to the sensitive nature of the matter at hand, Section II-C outlines the choices made w.r.t. the nomenclature used throughout this article.

A. Bias in Automated Decision Systems

In recent years, numerous concerns have been raised regarding the accuracy and fairness of automated decision-making systems. For instance, many studies regarding the risk assessment and welfare distribution tools found a number of issues concerning systemic bias and discrimination of the systems' predictions (e.g., against dark-skinned people). The impact of such automated decisions on the lives of the affected individuals can be tremendous, e.g., being jailed, denied a bail, parole, or welfare payments [START_REF] Osoba | An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence[END_REF], [START_REF] Washington | How to argue with an algorithm: Lessons from the COMPAS-ProPublica debate[END_REF], [START_REF] Garvie | The Perpetual Line-Up: Unregulated Police Face Recognition in America[END_REF], [START_REF] Neil | Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy[END_REF]. Demographicsbased bias and discrimination are especially concerning in this context, even if they occur unintentionally. One would intuitively expect that certain decisions be impacted exclusively by hard facts and evidence, and not factors often associated with discrimination-such as sex or race, or other context-specific discriminatory factors. Nonetheless, biases in decision making are a common occurrence; along with the notions of fairness, this topic has been extensively studied from the point of view of various disciplines, such as psychology, sociology, statistics, and information theory [START_REF] Evans | Bias in Human Reasoning: Causes and Consequences[END_REF]- [START_REF] Friedman | The Elements of Statistical Learning[END_REF]. Recently, the field of bias and fairness in automated computer algorithms and machine learning has emerged [START_REF] Zweig | On chances and risks of security related algorithmic decision making systems[END_REF], [START_REF] Mehrabi | A survey on bias and fairness in machine learning[END_REF].

A good discussion of the topic of bias was provided by Danks and London [START_REF] Danks | Algorithmic bias in autonomous systems[END_REF], as well as Friedman and Nissenbaum [START_REF] Friedman | Bias in computer systems[END_REF], both of which explored various sources and types of bias in the context of computer systems. In many cases, the bias in the automated decision systems is directly related to the human designers or operators of a system. Semiautomatic decision systems are a good example of this. In such systems, a human decision maker can be aided by an algorithm (e.g., risk assessment). In such cases, errors in interpretation of the results of the system might occur; in other words, the human might misunderstand or misrepresent the outputs or general functioning principles of an algorithm [START_REF] Lansing | New York state COMPAS-probation risk and need assessment study: Examining the recidivism scale's effectiveness and predictive accuracy[END_REF]- [START_REF] Chouldechova | Fair prediction with disparate impact: A study of bias in recidivism prediction instruments[END_REF]. Furthermore, it has been shown that humans, in general, tend to over-rely on such automated systems, i.e., to overestimate the accuracy of their results [START_REF] Mosier | Automation bias: Decision making and performance in high-tech cockpits[END_REF]. While human cognitive biases are an important and actively researched topic, this article focuses exclusively on bias occurring in the context of automated algorithms themselves. Human cognitive biases have been analyzed, e.g., by Evans [START_REF] Evans | Bias in Human Reasoning: Causes and Consequences[END_REF], whereas bias in human interactions with the automated system was explored, e.g., by Parasuraman and Manzey [START_REF] Parasuraman | Complacency and bias in human use of automation: An attentional integration[END_REF].

In the context of automated decision algorithms themselves, numerous potential bias causes exist. Most prominently, the training data could be skewed, incomplete, outdated, disproportionate, or have embedded historical biases, all of which are detrimental to algorithm training and propagate the biases present in the data. Likewise, the implementation of an algorithm itself could be statistically biased or otherwise, flawed in some way, for example, due to moral or legal norms, poor design, or data processing steps, such as parameter regularization or smoothing. For more details on the topic of algorithmic bias in general, the reader is referred to, e.g., [START_REF] Castelluccia | Understanding algorithmic decision-making: Opportunities and challenges[END_REF], [START_REF] Danks | Algorithmic bias in autonomous systems[END_REF], and [START_REF] Friedman | Bias in computer systems[END_REF]. In the next sections, an introduction to biometric systems is provided, followed by a survey on algorithmic bias in such systems specifically.

B. Biometric Systems

Biometric systems aim at establishing or verifying the identity or demographic attributes of individuals. In the international standard ISO/IEC 2382-37 [START_REF]Information Technology-Vocabulary-Part 37: Biometrics[END_REF], "biometrics" is defined as: "automated recognition of individuals based on their biological and behavioral characteristics."

Humans possess, nearly universally, physiological characteristics which are highly distinctive and can, therefore, be used to distinguish between different individuals with a high degree of confidence. Example images of several prominent biometric characteristics are shown in Fig. 1.

Broadly speaking, an automated biometric system consists of: 1) a capture device (e.g., a camera), with which the biometric samples (e.g., images) are acquired; 2) a database which stores the biometric information and other personal data; 3) signal processing algorithms, which estimate the quality of the acquired sample, find the region of interest (e.g., a face) and extract the distinguishing features from it; and 4) comparison and decision algorithms, which enable ascertaining of similarity of two biometric samples by comparing the extracted feature vectors and establishing whether or not the two biometric samples belong to the same source.

In the past, biometric systems typically utilized handcrafted features and algorithms (i.e., texture descriptors, see Liu et al. [START_REF] Liu | From BoW to CNN: Two decades of texture representation for texture classification[END_REF]). Nowadays, the use of machine learning and deep learning has become increasingly popular and successful. Relevant-related works include [START_REF] Taigman | DeepFace: Closing the gap to human-level performance in face verification[END_REF]- [START_REF] Parkhi | Deep face recognition[END_REF], which achieved breakthrough biometric performances in facial recognition. Furthermore, promising results for deep learning-based fingerprint (see [START_REF] Tang | FingerNet: An unified deep network for fingerprint minutiae extraction[END_REF]) and iris (see [START_REF] Nguyen | Iris recognition with off-the-shelf CNN features: A deep learning perspective[END_REF]) recognition have also been achieved. For a review of deep learning techniques applied within biometrics, the reader is referred to as Sundararajan and Woodard [START_REF] Sundararajan | Deep learning for biometrics: A survey[END_REF]. For a highly comprehensive introduction to biometrics, the reader is referred to Li and Jain [START_REF] Li | Encyclopedia of Biometrics[END_REF] and the handbook series [START_REF] Jain | Handbook of Biometrics[END_REF]- [START_REF] Uhl | Handbook of Vascular Biometrics[END_REF].

C. Nomenclature

In this section, the nomenclature used throughout this article is explained. The authors note that demographic words, groups, and concepts, such as "gender," "sex," "race," and "ethnicity" can be extremely divisive and bear a heavy historical, cultural, social, political, or legislative load. The authors do not seek to define or redefine those terms; we merely report on the current state of the research. In the literature surveyed later on in this article, the following trends can be distinguished.

1) The terms gender and sex are often used in a binary and conflated manner. Readers interested in the possible consequences of this narrow approach are referred to [START_REF] Keyes | The misgendering machines: Trans/HCI implications of automatic gender recognition[END_REF]. 2) Similarly, very often no real distinction between the terms race and ethnicity is made; moreover, the typical categorization is very coarse, only allowing for a small and finite (less than ten) possible racial/ethnic categories. 3) In general, and especially in the case of facial biometrics, demographic factors seem to be considered on the phenotypic basis, i.e., concerning the observable traits of the subjects (e.g., the color of the skin or masculine appearance). Due to the demographic terms carrying a large amount of complexity and potential social divisiveness, the authors do not engage in those debates in this article, and merely reproduce and discuss the technical aspects of the current research. For the sake of consistency, certain decisions regarding the used nomenclature have to be made, especially since the surveyed literature does often seem to use the aforementioned demographic terms ambiguously or interchangeably.

Recently, in the context of biometrics, ISO/IEC has made the following separation [START_REF]Information Technology-Biometrics-Identifying and Mitigating the Differential Impact of Demographic Factors in Biometric Systems, ISO/IEC Standard WD[END_REF] 5 : while the term gender is defined as "the state of being male or female as it relates to social, cultural, or behavioral factors," the term sex is understood as "the state of being male or female as it relates to biological factors, such as DNA, anatomy, and physiology." The report also defines the term ethnicity as "the state of belonging to a group with a common origin, set of customs, or traditions," while the term race is not defined there. While the cultural and religious norms can certainly affect biometric operations, the surveyed literature mostly considers the appearance-based features and categorization-hence, the term race is used instead of ethnicity and the term sex is used instead of gender in accordance with ISO/IEC 22116 [START_REF]Information Technology-Biometrics-Identifying and Mitigating the Differential Impact of Demographic Factors in Biometric Systems, ISO/IEC Standard WD[END_REF]. In the context of biometrics in general, the standardized biometric vocabulary is used (see ISO/IEC 2382-37 [START_REF]Information Technology-Vocabulary-Part 37: Biometrics[END_REF]). Finally, it is noted that a large part of the surveyed biometric literature follows the notions and metrics regarding the evaluation of biometric algorithms irrespective of the chosen biometric characteristic defined in ISO/IEC 19795-1 [START_REF]ISO/IEC JTC1 SC37 Biometrics, Information Technology-Biometric Performance Testing and Reporting-Part 1: Principles and Framework, International Organization for Standardization and International Electrotechnical Committee[END_REF].

Those limitations and imprecisions of the nomenclature notwithstanding, due to the potential of real and disparate impacts [START_REF] Feldman | Certifying and removing disparate impact[END_REF] of automated decision systems including biometrics, it is imperative to study the bias and fairness of such algorithms w.r.t. the demographic attributes of the population, regardless of their precise definitions.

III. BIAS IN BIOMETRIC SYSTEMS

To facilitate discussions on algorithmic fairness in biometric systems, Howard et al. [START_REF] Howard | The effect of broad and specific demographic homogeneity on the imposter distributions and false match rates in face recognition algorithm performance[END_REF] introduced the following two terms.

1) Differential performance concerns the differences in (genuine and/or impostor) score distributions between the demographic groups. Those effects are closely related to the so-called "biometric menagerie" [START_REF] Doddington | Sheep, goats, lambs and wolves: A statistical analysis of speaker performance in the NIST 1998 speaker recognition evaluation[END_REF]- [START_REF] Daugman | Searching for Doppelgängers: Assessing the universality of the IrisCode impostors distribution[END_REF].

While the menagerie describes the score distributions being statistically different for specific individual subjects, the introduced term describes the analogous effect for different demographic groups of subjects. 2) Differential outcomes relate to the decision results of the biometric system, i.e., the differences in the false-match and false-nonmatch rates at a specific decision threshold. Given that these terms have been introduced relatively recently, the vast majority of surveyed literature have not (directly) used them, instead ad hoc methodologies based on existing metrics were used. However, Grother et al. [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF] presented a highly comprehensive study of the demographic effects in biometric recognition, conducting their benchmark utilizing the terms and notions above. A standardization effort in this area under the auspices of ISO/IEC is ongoing [START_REF]Information Technology-Biometrics-Identifying and Mitigating the Differential Impact of Demographic Factors in Biometric Systems, ISO/IEC Standard WD[END_REF].

Before surveying the literature on bias estimation and mitigation (Sections III-C and III-D, respectively), this section begins with an outline of biometric algorithms which might be affected by bias (Section III-A), as well as of covariates which might affect them (Section III-B).

A. Algorithms

Similar to other automated decision systems, human biases have been shown to exist in the context of biometrics. The socalled "other-race effect" has long been known to affect the human ability to recognize faces [START_REF] Furl | Face recognition algorithms and the other-race effect: Computational mechanisms for a developmental contact hypothesis[END_REF]. As previously stated, the cognitive biases of humans are out of scope for this article, as it focuses on the biases in the algorithms themselves. The processing pipeline of a biometric system can consist of various algorithms depending on the application scenario and the chosen biometric characteristic. Said algorithms might be subject to algorithmic bias w.r.t. certain covariates, which are described in Section III-B. Below, the most important algorithms used in the context of biometrics are described and visualized conceptually in Fig. 2.

One of the most prevalent uses of biometrics is recognition. Here, distinguishing features of biometric samples are compared to ascertain their similarity. Such systems typically seek to: 1) determine if an individual is who they claim to be (i.e., one-to-one comparison) or 2) to determine the identity of an individual by searching a database (i.e., one-to-many search). Accordingly, the following two scenarios might be used in biometric recognition.

1) Verification: Referring to the "process of confirming a biometric claim through biometric comparison" [START_REF]Information Technology-Vocabulary-Part 37: Biometrics[END_REF], [START_REF]ISO/IEC JTC1 SC37 Biometrics, Information Technology-Biometric Performance Testing and Reporting-Part 1: Principles and Framework, International Organization for Standardization and International Electrotechnical Committee[END_REF]. 2) Identification: Referring to the "process of searching against a biometric enrolment database to find and return the biometric reference identifier(s) attributable to a single individual" [START_REF]Information Technology-Vocabulary-Part 37: Biometrics[END_REF], [START_REF]ISO/IEC JTC1 SC37 Biometrics, Information Technology-Biometric Performance Testing and Reporting-Part 1: Principles and Framework, International Organization for Standardization and International Electrotechnical Committee[END_REF]. The biometric samples are a rich source of information beyond the mere identity of the data subject. Another use case of biometrics is the extraction of auxiliary information from a biometric sample, primarily using the following algorithms.

1) Classification and Estimation: Referring to the process of assigning demographic or other labels to biometric samples [START_REF] Dantcheva | What else does your biometric data reveal? A survey on soft biometrics[END_REF], [START_REF] Sun | Demographic analysis from biometric data: Achievements, challenges, and new frontiers[END_REF]. Prior to recognition or classification tasks, the system must acquire and preprocess the biometric sample(s). Here, most prominently, the following algorithms might be used. 2) Segmentation and Feature Extraction: Referring to the process of locating the region of interest and extracting a set of biometric features from a biometric sample [START_REF] Li | Encyclopedia of Biometrics[END_REF].

3) Quality Assessment: Referring to the process of quantifying the quality of an acquired biometric sample [START_REF]ISO/IEC JTC1 SC37 Biometrics, Information Technology-Biometric Sample Quality-Part 1: Framework, International Organization for Standardization and International Electrotechnical Committee[END_REF], [START_REF] Bharadwaj | Biometric quality: A review of fingerprint, iris, and face[END_REF]. 4) Presentation Attack Detection (PAD): Referring to the "automated determination of a presentation attack," i.e., detecting a "presentation to the biometric data capture subsystem with the goal of interfering with the operation of the biometric system" [START_REF]ISO/IEC JTC1 SC37 Biometrics, Information Technology-Biometric Presentation Attack Detection-Part 1: Framework, International Organization for Standardization and International Electrotechnical Committee[END_REF], [START_REF] Marcel | Handbook of Biometric Anti-spoofing: Presentation Attack Detection[END_REF].

B. Covariates

Broadly, three categories of covariates relevant to the effectiveness of the biometric algorithms can be distinguished.

1) Demographic: Referring to, e.g., the sex, age, and race of the data subject. 2) Subject Specific: Referring to the behavior of the subject (e.g., pose or expression, and use of accessories, such as eyewear or makeup), as well as their interaction with the capture device (e.g., distance from a camera or pressure applied to a touch-based sensor). 3) Environmental: Referring to the effects of the surroundings on the data acquisition process (e.g., illumination, occlusion, and resolution of the images captured by the sensor). Fig. 3 shows the example images of the aforementioned covariates using the facial biometric characteristic. While there do exist studies that investigate environmental and subjectspecific covariates (e.g., [START_REF] Kortylewski | Empirically analyzing the effect of dataset biases on deep face recognition systems[END_REF]), this article concentrates on the demographic covariates. as follows: the studies are divided by biometric characteristic and listed chronologically. The third column lists the algorithms (recall Section III-A) evaluated by the studies, while the covariates (recall Section III-B) considered in the studies are listed in the next column. Finally, the last column outlines the key finding(s) of the studies. Wherever possible, those were extracted directly from the abstract or summary sections of the respective studies.

C. Estimation

By surveying the existing literature, the following trends can be distinguished.

1) Most existing studies conducted the experiments using face-based biometrics. There are significantly fewer studies on other modalities (primarily fingerprint). 2) The majority of studies concentrated on biometric recognition algorithms (primarily verification), followed by quality assessment and classification algorithms. 3) Some scenarios have barely been investigated, e.g., PAD. 4) The existing studies predominantly considered the sex covariate; the race covariate is also often addressed (possibly due to the recent press coverage [START_REF] Orcutt | Are Face Recognition Systems Accurate? Depends on Your Race[END_REF], [135]). The age covariate is least often considered in the context of bias in the surveyed literature. The impact of aging on biometric recognition is an active field of research, but out of scope for this article. The interested reader is referred to, e.g., [START_REF] Best-Rowden | Longitudinal study of automatic face recognition[END_REF], [START_REF] Yoon | Longitudinal study of fingerprint recognition[END_REF], and [START_REF] Fairhurst | Age Factors in Biometric Processing[END_REF]- [START_REF] Arnold | Investigating performance and impacts on fingerprint recognition systems[END_REF]. 5) Many studies focused on general accuracy rather than distinguishing between false-positive and false-negative errors. Recent works [START_REF] Howard | The effect of broad and specific demographic homogeneity on the imposter distributions and false match rates in face recognition algorithm performance[END_REF], [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF] introduced and used the useful concepts of "false-positive differentials" and "false-negative differentials" in demographic bias benchmarks. 6) A significant number of studies (e.g., [START_REF] Howard | The effect of broad and specific demographic homogeneity on the imposter distributions and false match rates in face recognition algorithm performance[END_REF], [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF],

and [START_REF] Cook | Demographic effects in facial recognition and their dependence on image acquisition: An evaluation of eleven commercial systems[END_REF]) conducted evaluations on sequestered databases and/or commercial systems. Especially the results of Grother et al. [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF] in the context of an evaluation conducted by the National Institute of Standards and Technology (NIST) were valuable due to the realistic/operational nature of the data, the large scale of used databases, as well as the testing of state-of-the-art commercial and academic algorithms. However, reproducing or analyzing their results may be impossible due to the unattainability of data and/or tested systems. The following common findings for the evaluated biometric algorithms can be discerned.

1) Recognition: One result which appears to be mostly consistent across surveyed studies is that of worse biometric performance (both in terms of false positives and false negatives) for female subjects (see [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF], [START_REF] Klare | Face recognition performance: Role of demographic information[END_REF]). Furthermore, several studies associated race as a major factor influencing biometric performance. However, the results were not attributed to a specific race being inherently more challenging. Rather, the country of software development (and presumably the training data) appears to play a major role; in this context, the evidence of the other-race effect in facial recognition has been found [START_REF] Phillips | An other-race effect for face recognition algorithms[END_REF], e.g., algorithms developed in Asia were more easily recognizing Asian individuals and conversely algorithms developed in Europe were found to be more easily recognizing Caucasians. Finally, the age has been determined to be an important factor as well-especially the very young subjects were a challenge (with effects of aging also playing a major role). Grother et al. [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF] presented hitherto the largest and most comprehensive study of demographic bias in biometric recognition. Their benchmark showed that false-negative differentials usually vary by a factor of less than 3 across the benchmarked algorithms. On the other hand, the false-positive differentials were much more prevalent (albeit not universal) and often larger, i.e., varying by two to three orders of magnitude across the benchmarked algorithms. 6 Most existing studies considered biometric verification, with only a few addressing biometric identification. Estimating bias in biometric identification is nontrivial, due to the contents of the screening database being an additional variable factor susceptible to bias.

Specifically, in addition to potential biases in the biometric algorithms themselves, certain biases stemming from data acquisition might occur and be propagated (e.g., historical and societal biases having impact on the demographic composition of a criminal database). Consequently, demographic bias estimation in biometric identification is an interesting and important item for future research.

2) Classification and Estimation: Scientific literature predominantly studied face as the biometric characteristic, since the facial region contains rich information from which demographic attributes can be estimated. Several studies showed a substantial impact of sex and race on the accuracy of demographic attribute classification. In particular, numerous commercial algorithms exhibited significantly lower accuracy w.r.t. dark-skinned female subjects (see [START_REF] Buolamwini | Gender shades: Intersectional accuracy disparities in commercial gender classification[END_REF], [START_REF] Raji | Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products[END_REF]). Research on the classification of sex from iris and periocular images exists, but biases in those algorithms have not yet been studied. Additionally, it is not clear if such classifiers rely on actual anatomical properties or merely the application of mascara [START_REF] Kuehlkamp | Gender-from-Iris or gender-from-mascara?[END_REF].

3) Quality Assessment: Most existing studies conducted experiments using fingerprint-based biometrics. This could be partially caused by the standardization of reliable fingerprint quality assessment metrics [START_REF]ISO/IEC JTC1 SC37 Biometrics, Information Technology-Biometric Sample Quality-Part 4: Finger Image Data, International Organization for Standardization and International Electrotechnical Committee[END_REF], whereas this remains an open challenge for the face characteristic [START_REF] Galbally | Study on face identification technology for its implementation in the Schengen information system[END_REF]. The existing fingerprint quality assessment studies consistently indicated that the extreme ranges of the age distribution (infants and elderly) can pose a challenge for current systems [START_REF] Galbally | A study of age and ageing in fingerprint biometrics[END_REF].

Correlations between the quality metrics of facial images (obtained using state-of-the-art estimators) and demographic covariates were recently pointed out in a preliminary study [START_REF] Terhörst | Face quality estimation and its correlation to demographic and nondemographic bias in face recognition[END_REF]. Additional nonobvious, hidden biases can also occur. For example, the presence of eyeglasses [START_REF] Drozdowski | Detection of glasses in near-infrared ocular images[END_REF], [START_REF] Osorio-Roig | Iris recognition in visible wavelength: Impact and automated detection of glasses[END_REF] or contact lenses [START_REF] Baker | Degradation of iris recognition performance due to non-cosmetic prescription contact lenses[END_REF] lowers the sample quality and biometric performance under objective metrics in iris recognition systems. The demographics disproportionately afflicted with myopia (i.e., most likely to wear corrective eyewear) are those from "developed" countries and East Asia [START_REF] Dolgin | The myopia boom[END_REF]. Admittedly, the inability of the algorithms to compensate for the presence of corrective eyewear might be argued not to be a bias per se. This argument notwithstanding, specific demographic groups could clearly be disadvantaged in this case-either by increased error rates or the requirement for a more elaborate (especially for contact lenses) interaction with the acquisition device. Issues, such as this one push the boundaries of what might be considered biased or fair in the context of biometric systems and constitute an interesting area of future technical and philosophical research. In addition, it is necessary to point out potential issues in surveyed studies, such as the following.

1) Differences in experimental setups, used toolchains and datasets, training-testing data partitioning, imbalanced datasets, etc. 2) Statistical significance of the results due to relatively small size of the used datasets in most cases (except, e.g., [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF] and [START_REF] Galbally | Fingerprint quality: A lifetime story[END_REF]). 3) Lack of a single definition of bias/fairness (see also Section IV-A), as well as a standardized methodology and metrics for conducting evaluations. 4) Difficulty of sufficiently isolating the influence of demographic factors from other important covariates (e.g., pose and illumination). 5) Potential for bias propagation from the previous steps of the pipeline (e.g., data acquisition). Nevertheless, some results appear to be intuitive, e.g., worse accuracies for women. These could be due to numerous reasons, such as: larger intraclass variations due to makeup [START_REF] Rathgeb | Impact and detection of facial beautification in face recognition: An overview[END_REF], occlusion by hairstyle and accessories, or pose differences due to women being shorter than men and cameras being calibrated with the height of men. Likewise, lower sample quality of infant fingerprints makes sense due to anatomical constraints and the fact that the size of the fingerprint area is considered as a relevant factor for fingerprint sample quality. In order to acquire high-quality fingerprint samples from very young data subjects, specialized hardware may be necessary (see [START_REF] Koda | Development of 2,400 ppi fingerprint sensor for capturing neonate fingerprint within 24 hours after birth[END_REF]).

D. Mitigation

Table II summarizes the existing research in the area of bias mitigation in biometrics. Similar to the above, related [START_REF] Wang | Racial faces in-the-wild: Reducing racial bias by information maximization adaptation network[END_REF]) and synthetic data to enhance the training datasets (e.g., [START_REF] Kortylewski | Analyzing and reducing the damage of dataset bias to face recognition with synthetic data[END_REF]), as well as learning specialized loss or similarity functions (e.g., [START_REF] Terhörst | Comparison-level mitigation of ethnic bias in face recognition[END_REF]). A of balanced training datasets have been released to the research community, as shown in Table III. 2) Dynamic Selection: Deviating from preventing demographic bias, some methods attempted to employ a bias-aware approach. Examples in this category include the dynamic selection of the recognition algorithms (e.g., [START_REF] Guo | Human age estimation: What is the influence across race and gender?[END_REF]) or decision thresholds (e.g., [START_REF] Krishnapriya | Characterizing the variability in face recognition accuracy relative to race[END_REF]) based on demographic attributes of the individual subjects. In addition to the categories above, other approaches may be considered in the context of bias mitigation. For example, modeling of factors, such as fingerprint growth, can be used to improve the biometric recognition performance for children (see [START_REF] Preciozzi | Fingerprint biometrics from newborn to adult: A study from a national identity database system[END_REF]) and to mitigate the effects of aging (see [START_REF] Haraksim | Fingerprint growth model for mitigating the ageing effect on children's fingerprints matching[END_REF]). Other examples include de-identification and anonymization methods (see [START_REF] Ribarić | De-identification for privacy protection in biometrics[END_REF], [START_REF] Chhabra | Anonymizing k-facial attributes via adversarial perturbations[END_REF]), whose primary use case is privacy protection in biometrics. Such methods aim to remove, change, or obfuscate certain information (e.g., demographics) either from the image (e.g., [START_REF] Mirjalili | Semiadversarial networks: Convolutional autoencoders for imparting privacy to face images[END_REF]) or feature (e.g., [START_REF] Acien | Measuring the gender and ethnicity bias in deep models for face recognition[END_REF] and [START_REF] Morales | SensitiveNets: Learning agnostic representations with application to face recognition[END_REF]) domain, often through a form of adversarial learning. One could hypothesize that a system trained on such data might not exhibit biases w.r.t. the de-identified demographic covariates. However, the validity of such hypotheses has not yet been ascertained experimentally.

IV. DISCUSSION

In this section, several issues relevant to the topic of this article are discussed. Concretely, Section IV-A addresses the topic of algorithmic fairness in general, while Section IV-B does so in the context of biometrics specifically. Section IV-C illustrates the importance of further research on algorithmic bias and fairness in biometrics by describing the social impact of demographically biased systems.

A. Algorithmic Fairness in General

The challenge of fairness is common in machine learning and computer vision, i.e., it is by no means limited to biometrics. A survey focusing on issues and challenges associated with algorithmic fairness was conducted among industry practitioners by Holstein et al. [START_REF] Holstein | Improving fairness in machine learning systems: What do industry practitioners need?[END_REF]. For a comprehensive overview of bias in automated algorithms in general, the reader is referred to, e.g., [START_REF] Mehrabi | A survey on bias and fairness in machine learning[END_REF] and [START_REF] Du | Fairness in deep learning: A computational perspective[END_REF]. In addition to algorithmic fairness, algorithmic transparency, explainability, interpretability, and accountability (see [START_REF] Diakopoulos | Accountability in algorithmic decision making[END_REF]- [START_REF] New | How policymakers can foster algorithmic accountability[END_REF]) have also been heavily researched in recent years both from the technical and social perspective. The current research in the area of algorithmic fairness concentrates on the following topics.

1) Theoretical and formal definitions of bias and fairness (see [START_REF] Mehrabi | A survey on bias and fairness in machine learning[END_REF], [START_REF] Verma | Fairness definitions explained[END_REF], [START_REF] Hutchinson | 50 years of test (un)fairness: Lessons for machine learning[END_REF]). 2) Fairness metrics, software, and benchmarks (see [START_REF] Hardt | Equality of opportunity in supervised learning[END_REF]- [START_REF] Bellamy | AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias[END_REF]). 3) Societal, ethical, and legal aspects of algorithmic decision making and fairness therein (see [START_REF] Pasquale | The Black Box Society: The Secret Algorithms That Control Money and Information[END_REF], [START_REF] Tufekci | Algorithmic harms beyond Facebook and Google: Emergent challenges of computational agency[END_REF]- [START_REF] Fra Focus | Data quality and artificial intelligence-Mitigating bias and error to protect fundamental rights[END_REF]). 4) Estimation and mitigation of bias in algorithms and datasets (see [START_REF] Torralba | Unbiased look at dataset bias[END_REF]- [START_REF] Roy | Mitigating information leakage in image representations: A maximum entropy approach[END_REF]). Despite decades of research, there exists no single agreed coherent definition of algorithmic fairness. In fact, dozens of formal definitions (see [START_REF] Verma | Fairness definitions explained[END_REF], [START_REF] Hutchinson | 50 years of test (un)fairness: Lessons for machine learning[END_REF]) have been proposed to address different situations and possible criteria of fairness. 7Certain definitions, which are commonly used and advocated for, are even provably mutually exclusive [START_REF] Friedler | On the (im)possibility of fairness[END_REF]. Therefore, depending on the definition of fairness one chooses to adopt, a system can effectively always be shown to exhibit some form of bias. As such, the "correct" approach is essentially application dependent. This in turn necessitates a keen domain knowledge and awareness of those issues from the system operators and stakeholders, as they need to select the definitions and metrics of fairness relevant to their particular use case. Research in this area strongly suggests that the notion of fairness in machine learning is context sensitive [START_REF] Green | The myth in the methodology: Towards a recontextualization of fairness in machine learning[END_REF], [START_REF] Liu | Delayed impact of fair machine learning[END_REF]; this presumably also applies to the field of biometrics, especially for machine learning-based systems. In the next section, the notions of fairness and bias are discussed in the context of biometrics specifically based on the literature surveyed in Section III.

B. Algorithmic Fairness in Biometrics

Although the topic of demographic bias and fairness in biometrics has emerged relatively recently, it has quickly established itself as an important and popular research area. Several high-ranking conferences featured special sessions,8,9,10 NIST conducted large-scale evaluations [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF], while ISO/IEC is currently preparing a technical report on this subject [START_REF]Information Technology-Biometrics-Identifying and Mitigating the Differential Impact of Demographic Factors in Biometric Systems, ISO/IEC Standard WD[END_REF]. Likewise, a significant number of scientific publications has appeared on this topic (surveyed in Section III). The existing studies concentrated on face-based biometrics-more research is urgently needed for other biometric characteristics, e.g., fingerprints [START_REF] Marasco | Biases in fingerprint recognition systems: Where are we at?[END_REF]. The existing studies primarily address the following aspects.

1) Evaluations with the aim of quantitatively ascertaining the degree of demographic bias in various biometric algorithms. 2) Methods that seek to mitigate the effects of demographic bias in various biometric algorithms. The existing bias estimation studies have uncovered new trends w.r.t. algorithmic bias and fairness in biometric algorithms (recall Section III-C). However, it should be noted as described as follows.

1) In many cases, the biases were algorithm specific, i.e., while given the same benchmark dataset, some algorithms exhibited a bias (e.g., lower biometric performance for a certain demographic group), others did not. In aggregate, however, the existing studies did seem to agree on certain points, as described in Section III-C. 2) While a high relative increase in error rates for a certain demographic group may appear quite substantial, its importance in absolute terms could be negligible, especially for very accurate algorithms which hardly make any errors whatsoever [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF]. Those caveats notwithstanding, the commitment of the academic researchers and commercial vendors to researching algorithmic fairness is especially important for the public perception of biometric technologies. The field of algorithmic fairness in the context of biometrics is in its infancy and a large number of issues are yet to be comprehensively addressed (see Section IV-A).

1) Limited theoretical work has been conducted in this field specifically focusing on biometrics. Indeed, the majority of publications surveyed in Section III do not approach the notions of bias and fairness rigorously; rather, they tend to concentrate on an equivalent of some of the simpler statistical definitions, such as group fairness and error rate parity. Extending the existing estimation and mitigation works, for example, to consider other and more complex notions of fairness (see [START_REF] Terhörst | Post-comparison mitigation of demographic bias in face recognition using fair score normalization[END_REF]) could be seen as important future work in the field. Likewise, investigating tradeoffs between biometric performance, fairness, user experience, social perceptions, monetary costs, and other aspects of the biometric systems might be of interest. 2) In addition to empiric studies (especially in the case of bias mitigation, see Section III-D), stricter theoretical approaches need to be pursued in order to provably demonstrate the bias-mitigating properties of the proposed methods. 3) Isolating the effects of the demographic factors from other confounding factors (i.e., the environmental and subject-specific covariates, such as illumination and use of accessories) is a challenging task, which is not sufficiently addressed in many existing studies. An example of a study which partially addressed those issues in a systematic manner is the work of Grother et al. [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF]. 4) More complex analyses based on demographic attributes and combinations thereof (intersectionality) could be conducted for a more detailed and nuanced view of demographic biases in biometric systems. 5) Comprehensive independent benchmarks utilizing various algorithmic fairness measurement methodologies and metrics are, as of yet, lacking. Only recently, in [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF], first independent benchmarks of biometric recognition algorithms have been conducted. Similar and more extensive benchmarks for other biometric algorithms (recall Section III-A) are needed. 6) Large-scale datasets designed specifically for biasrelated research need to be collected. The existing datasets only pertain to face-based biometrics (see Table III). 7) Humans are known to exhibit a broad range of biases [START_REF] Evans | Bias in Human Reasoning: Causes and Consequences[END_REF], [START_REF] Parasuraman | Complacency and bias in human use of automation: An attentional integration[END_REF]. The influence of those factors on the biometric algorithm design, interactions with and use of biometric systems, as well as perceptions of biometric systems could be investigated. 8) Most of the surveyed studies did not explicitly provide information about ethics approval. Future works could improve on those practices, especially considering the sensitive nature of the research topic at hand. In the next section, the possible consequences of failing to appropriately address the issues of algorithmic fairness in biometrics are discussed.

C. Social Impact

Numerous studies described the potential of real harms as a consequence of biased algorithmic decision-making systems [START_REF] Tufekci | Algorithmic harms beyond Facebook and Google: Emergent challenges of computational agency[END_REF], [START_REF] Kirkpatrick | Battling algorithmic bias: How do we ensure algorithms treat us fairly?[END_REF] in general. Regarding biometric systems, in particular, facial recognition technologies have been the main focus of such discussions (see [START_REF] Hallowell | Ethical issues arising from the police use of live facial recognition technology[END_REF]). Concerning notions of bias and fairness, in addition to being context sensitive (recall Section IV-A), one might argue the impact assessments to also be purpose sensitive. Specifically, depending on the application scenario, the impact and importance of systemic biases might differ significantly. As an example, consider an application of biometrics in cooperative access control systems or personal devices. A demographic bias in such a system might cause a certain demographic group to be inconvenienced through additional authentication attempt(s) being necessary due to false-negative errors. On the other hand, the stakes are much higher in, e.g., a state surveillance scenario. There, demographic biases could directly cause substantial personal harms, e.g., higher (unjustified) arrest rates [START_REF] Garvie | The Perpetual Line-Up: Unregulated Police Face Recognition in America[END_REF], due to false-positive errors. At the same time, it is also clear that the biometric recognition technology can be highly accurate. Taking the recently contested facial recognition as an example, given prerequisites, such as a high-resolution camera, proper lighting and image quality controls, as well as high-quality comparison algorithms, the absolute error rates can become vanishingly small [START_REF] Grother | Ongoing face recognition vendor test (FRVT) part 3: Demographic effects[END_REF], thereby potentially rendering the relative imbalance of error rates across demographic groups insignificant.

It should be noted that there are no indications of the algorithmic biases in biometrics being deliberately put into the algorithms by design; rather, they are typically a result of the used training data and other factors. In any case, one should also be mindful, that as any technology, biometrics could be used in malicious or dystopian ways (e.g., privacy violations through mass surveillance [START_REF] Kindt | Privacy and Data Protection Issues of Biometric Applications[END_REF] or "crime prediction" [START_REF] Wu | Automated inference on criminality using face images[END_REF]). Consequently, a framework for human impact assessments [START_REF] Calvo | Advancing impact assessment for intelligent systems[END_REF] should be developed for biometrics as soon as possible. A proactive and cognizant approach could foster awareness among the citizens and policymakers, as well as contribute to minimizing the potential negative perception of the biometric technology and innovation by individuals and society as a whole.

In a broader context, algorithmic bias and fairness is one of the topics in the larger discourse on the ethical design in AI systems [START_REF] Bryson | Standardizing ethical design for artificial intelligence and autonomous systems[END_REF], most prominently encompassing: 1) transparency; 2) accountability; 3) explainability; 4) fairness.

Currently, the legal and societal scrutiny of the technologies utilizing automated decision systems seem to be insufficient. However, recent legislation in the European Union [START_REF] Parliament | Regulation (EU) 2016/679[END_REF], [START_REF] Goodman | European Union regulations on algorithmic decision-making and a right to explanation[END_REF] constitutes a step in that direction. Below, several social and technological provisions, which might be considered in this context, are listed.

1) Carefully selecting the data used to train the algorithms is the first and perhaps the most important step: inherent biases in training data should be avoided wherever possible. Furthermore, the size of the dataset matterssome systems have been reported to be trained on very small datasets (in the order of thousands of items), which are usually wholly insufficient to show that an approach generalizes well. 2) Higher degree of transparency and/or independent insight into data and algorithms, as well as validation of the results could be established to foster the public trust and acceptance of the systems. 3) Thresholds for acceptable accuracy (i.e., how much the systems can error) could be established legally (potentially in a system purpose-sensitive manner), as well as reviewed and validated periodically. 4) Special training of the systems' personnel could be established to make them aware of the potential issues and to establish proper protocols for dealing with them. 5) Due diligence could be legally expected from vendors of such systems, i.e., in reasonably ensuring some or all of the aforementioned matters and rectifying problems as they come up. Additionally, certain accountability provisions could be incorporated to further facilitate this. The issues of fairness (including algorithmic fairness) are complicated from the point of view of the legislation-a somewhat deep understanding of statistics, formal fairness definitions, and other concepts is essential for an informed discourse. Furthermore, the ethical and moral perceptions and decisions are not uniform across different population demographics and by geographical location (see [START_REF] Awad | The moral machine experiment[END_REF]). This reinforces an important dilemma regarding the regulation of automated decision systems-since many situations are morally and ethically ambiguous to humans, how should they be able to encode ethical decision making into laws? Once that issue is somehow surmounted, there also remains the issue of feasibility of technical solutions, as described in the previous two sections.

Currently, many laws and rules exist (international treaties, constitutions of many countries, and employment law) which aim to protect against generic discrimination on the basis of demographics [START_REF]European Union Agency for Fundamental Rights and Council of Europe, Handbook on European Non-Discrimination Law[END_REF]. However, historically, the enforcement of those has been fraught with difficulties and controversies. In this context, the algorithmic decision systems are merely one of the most recent and technologically advanced cases. The policymakers and other stakeholders will have to tackle it in the upcoming years in order to develop a legal framework similar to those already governing other areas and aspects of the society [START_REF] Nutman | Algorithm and blues[END_REF].

V. SUMMARY

This article has investigated the challenge of demographic bias in biometric systems. Following an overview of the topic and challenges associated therewith, a comprehensive survey of the literature on bias estimation and mitigation in biometric algorithms has been conducted. It has been found that demographic factors can have a large influence on various biometric algorithms and that current algorithms tend to exhibit some degree of bias w.r.t. certain demographic groups. Most effects are algorithm dependent, but some consistent trends do also appear (as discussed in Section III-C). Specifically, many studies point to a lower biometric performance for females and youngest subjects in biometric recognition systems, as well as lower classification accuracy for dark-skinned females in the classification of demographic attributes from facial images. It should be noted that many of the studies conducted their experiments using relatively small datasets, which emphasize the need for large-scale studies. In general, a broad spectrum of open technical (and other) challenges exists in this field (see Section IV).

Biased automated decision systems can be detrimental to their users, with issues ranging from simple inconveniences, through disadvantages, to lasting serious harms. This relevance notwithstanding, the topic of algorithmic fairness is still relatively new, with many unexplored areas and few legal and practical provisions in existence. Recently, a growing academic and media coverage has emerged, where the overwhelming consensus appears to be that such systems need to be properly assessed (e.g., through independent benchmarks), compelled to some degree of transparency, accountability, and explainability in addition to guaranteeing some fairness definitions. Furthermore, it appears that, in certain cases, legal provisions might need to be introduced to regulate these technologies.

Automatic decision systems (including biometrics) are experiencing a rapid technological progress, thus simultaneously holding a potential of beneficial and harmful applications, as well as unintentional discrimination. Zweig et al. [START_REF] Zweig | On chances and risks of security related algorithmic decision making systems[END_REF] even argued that the issues (including, but not limited to bias and fairness) concerning algorithmic decision systems are directly related to the so-called "quality of democracy" measure of countries. As such, developing proper frameworks and rules for such technologies is a large challenge which the policymakers and the society as a whole must face in the upcoming future [START_REF] Citron | Technological due process[END_REF], [START_REF] Citron | The scored society: Due process for automated predictions[END_REF].

Fig. 1 .

 1 Fig. 1. Examples of biometric characteristics (images from publicly available research databases [27]-[30]). (a) Face. (b) Iris. (c) Fingerprint. (d) Veins.

Fig. 2 .

 2 Fig. 2. Conceptual overview of algorithms used in biometric systems. (a) Verification. (b) Identification. (c) Classification and estimation. (d) Quality assessment. (e) Segmentation and feature extraction. (f) PAD.

Fig. 3 .

 3 Fig. 3. Example images of covariates which might influence a biometric system utilizing facial information (images from a publicly available research database [27]). Black rectangles were added in an effort to respect individual anonymity and privacy. (a) Demographic (different sex, age, and race). (b) Subject specific (different pose and expression, and use of makeup and accessories). (c) Environmental (different lighting conditions, sharpness, and resolution).

  Table I summarizes the existing research in the area of bias estimation in biometrics. The table is organized conceptually

TABLE I SUMMARY

 I OF STUDIES CONCERNING BIAS ESTIMATION IN BIOMETRIC SYSTEMS

TABLE II SUMMARY

 II OF STUDIES CONCERNING BIAS MITIGATION IN BIOMETRIC SYSTEMS

https://www.banfacialrecognition.com/

https://www.cnet.com/news/facial-recognition-could-be-temporarily-ban ned-for-law-enforcement-use/

https://www.theguardian.com/technology/2020/jan/17/eu-eyes-temporaryban-on-facial-recognition-in-public-places

[START_REF] Yu | Framing the challenges of artificial intelligence in medicine[END_REF] https://www.biometricupdate.com/202001/eu-no-longer-consideringfacial-recognition-ban-in-public-spaces This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/

Note that the document is currently in a draft stage.

Note that this is a very high-level summary to illustrate the general size of the demographic differentials. The experimental results are much more nuanced and complex, as well as dependent on a number of factors in the used data, experimental setup, and the algorithms themselves.

See also https://towardsdatascience.com/a-tutorial-on-fairness-in-machinelearning-3ff8ba1040cb and https://fairmlbook.org/ for visual tutorials on bias and fairness in machine learning.

https://sites.google.com/view/wacv2020demographics

https://sites.google.com/site/eccvbefa2018

https://dasec.h-da.de/wp-content/uploads/2020/01/EUSIPCO2020-ss_bias_in_biometrics.pdf

by Associate Editor RThis work was supported by the German Federal Ministry of Education and Research and the Hessen State Ministry for Higher Education, Research and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE. The work of Antitza Dantcheva was supported by the French Government National Research Agency (ANR) under Grant ANR-17-CE39-0002.


Keywords: Gravettian, Rediscovery, Old excavations, Immature Gravettien, Redécouverte, Anciennes fouilles, Immature

Unexpected Discovery of More Elements from the Prehistoric Immature Skeleton from Baousso da Torre (Bausu da Ture) (Liguria, Italy). Inventory, Age-at-Death Estimation, and Probable Sex Assessment of BT3 Découverte inattendue d'éléments supplémentaires du squelette préhistorique immature de Baousso da Torre (Bausu da Ture) (Ligurie, Italie). Inventaire, estimation de l'âge-au-décès et détermination probable du sexe de BT3 S. Villotte

Introduction

In 1873, Rivière discovered three single prehistoric burials at Baousso da Torre (Liguria, Italy). The Baousso da Torre cave (also called Bausu da Ture) is part of the Balzi Rossi (also called Grimaldi) complex of caves and rock shelters, one of the most important European Paleolithic site complexes on the Mediterranean coast, on the border between Italy and France [START_REF] Formicola | Tall guys and fat ladies: Grimaldi's Upper Paleolithic burials and figurines in an historical perspective[END_REF]. BT1, a male who died before 50 years of age, was found in February 1873, whereas the skeletons of BT2 (a male who died in his third decade) and BT3 (an immature individual) were discovered in June of the same year [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF][START_REF] Villotte | The rediscovery of two Upper Palaeolithic skeletons from Baousso da Torre cave (Liguria-Italy)[END_REF][START_REF] Villotte | The paleobiology of two adult skeletons from Baousso da Torre (Bausu da Ture) (Liguria, Italy): Implications for Gravettian lifestyle[END_REF]. BT1 and BT3 were subsequently lost and rediscovered in 2008 by Henry-Gambier in the collections of the Musée lorrain [START_REF] Villotte | The rediscovery of two Upper Palaeolithic skeletons from Baousso da Torre cave (Liguria-Italy)[END_REF]. Several attempts to obtain direct and indirect radiocarbon dates for these individuals failed [START_REF] Villotte | The rediscovery of two Upper Palaeolithic skeletons from Baousso da Torre cave (Liguria-Italy)[END_REF][START_REF] Villotte | The paleobiology of two adult skeletons from Baousso da Torre (Bausu da Ture) (Liguria, Italy): Implications for Gravettian lifestyle[END_REF], but the archaeological context indicates that an Upper Paleolithic date is secure and that a Gravettian date is probable [START_REF] Henry-Gambier | Comportement des populations d'Europe au Gravettien : Pratiques funéraires et interprétations[END_REF][START_REF] Henry-Gambier | La sépulture des enfants de Grimaldi (Grotte des Enfants site des Baoussé-Roussé, Italie). Anthropologie et palethnologie funéraire des populations de la fin du Paléolithique supérieur[END_REF][START_REF] Mussi | Earliest Italy: an overview of the Italian Paleolithic and Mesolithic[END_REF][START_REF] Mussi | Italian Palaeolithic and Mesolithic Burials[END_REF]. The comprehensive descriptive and comparative study of the two adults was published recently [START_REF] Villotte | The paleobiology of two adult skeletons from Baousso da Torre (Bausu da Ture) (Liguria, Italy): Implications for Gravettian lifestyle[END_REF]. This note focuses on the third individual, BT3.

BT3 skeleton as it was described by Rivière and rediscovered by Henry-Gambier BT3 was stratigraphically slightly above BT2, and buried in a ventral decubitus position (i.e. prone, a rare position for the Gravettian [START_REF] Henry-Gambier | Comportement des populations d'Europe au Gravettien : Pratiques funéraires et interprétations[END_REF]), without ochre or associated goods. Apparently, the BT3 skeleton was badly preserved at the time of the discovery : "C'est le 3 juin 1873 que nous avons commencé à découvrir, à 3m,90 de profondeur au-dessous de la surface du sol, le second squelette d'adulte de la sixième S. Villotte (*) CNRS, UMR 5199 PACEA, UMR 5199 PACEA, Université de Bordeaux -CNRS Bât B8, allée Geoffroy Saint Hilaire, CS 50023, 33615 Pessac CEDEX France e-mail : sebastien.villotte@u-bordeaux.fr caverne et, deux jours plus tard, quelques os brisés provenant d'un enfant dont nous parlerons plus loin." [2, p. 219]. Rivière was an archeologist but also a physician. His descriptions of burials and human remains from the Balzi Rossi are usually quite lengthy and seem accurate, but he was notably briefer in the case of BT3, likely due to the relatively poor preservation of the skeleton and, maybe, the age of the individual (which may have made it of less interest in terms of morphological comparisons). Nevertheless, Rivière specified the state of preservation of each bone and gave the maximum preserved length for most of them [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF]. The extremely good match between these data and those obtained from the immature human remains found in the Musée lorrain collection supports a positive identification [START_REF] Villotte | The rediscovery of two Upper Palaeolithic skeletons from Baousso da Torre cave (Liguria-Italy)[END_REF]. Four examples, which will be useful in the following part of this article, are provided to illustrate these similarities.

For the neurocranium, Rivière [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF] was extremely clear. He explicitly said that only one roughly quadrangular fragment (56 x 58 mm), likely from the right parietal, was preserved. Moreover, he specified that sutures were preserved on two borders, and that the fragment was thin. Only one fragment of immature neurocranium was indeed found in the Musée lorrain collection, and it has exactly these dimensions, morphology, and state of preservation.

For the right femur, Rivière wrote that only the complete distal unfused epiphysis and a very short part of the diaphysis (less than 6 centimeters), lacking the greater trochanter and the head, were found [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF]. Three fragments of a right immature femur were found in the Musée lorain collection, two of them fit this description perfectly, the third one being the unfused femoral head [START_REF] Villotte | The rediscovery of two Upper Palaeolithic skeletons from Baousso da Torre cave (Liguria-Italy)[END_REF].

Rivière described the left patella (well preserved with slight damage to the inferior part) but did not mention the right one [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF]. Only one left immature patella was found in the Musée lorrain collection.

For the right talus, Rivière explicitly mentioned that only the anterior part was preserved [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF], and indeed that is what was found in the Musée lorain.

Unexpected new discoveries

Except for a few minor discrepancies, the skeletal elements described by Rivière in 1887 as belonging to BT3 match those found in the Musée lorrain, and only a few more immature bones not described by Rivière were found in these collections. Thus, until July 2012, most of the history of the BT3 remains was apparently clear [START_REF] Villotte | The rediscovery of two Upper Palaeolithic skeletons from Baousso da Torre cave (Liguria-Italy)[END_REF]:

• -The remains were excavated by Rivière in June 1873, who published the description of all the bones in 1887.

• -The skeleton stayed in Rivière's collection until his death in 1922.

• -This collection was then auctioned at the ''Hotel Drouot'' in Paris, and a part of it, including the BT3 skeleton, was bought by Goury (1877-1959), a prehistorian, and curator of the Musée lorrain from 1920 to 1927.

• -Goury donated his collection to the museum in 1955 and the skeletal remains of BT3 remained there until their rediscovery by Henry-Gambier in 2008.

In 2012, on the advice of Henry-Gambier, the author had a brief look at some bones curated at the Institut de paléontologie humaine (IPH) in Paris. These bones are labeled in the IPH database as "Baoussé Roussé (entrée 1968-69)", and "Menton (entrée 1972)". As this database contains 79 entries with these labels, whereas 102 specimens were identified, the author subsequently created a new coding system and gave the IPH curators an excel sheet with their exact descriptions and the relationships between the labeling systems. A minimum number of six individuals was determined (based on the left humerus) but taking into account the varied preservation and morphologies (some bones are likely not Pleistocene) of the remains, a greater number of individuals is suspected. At least one of these bones belongs to the BT2 skeleton (a first thoracic vertebra [START_REF] Villotte | The paleobiology of two adult skeletons from Baousso da Torre (Bausu da Ture) (Liguria, Italy): Implications for Gravettian lifestyle[END_REF]), and some others may belong to BT1 or BT2. More importantly, two sets of immature bones were identified. The first one comprises 20 fragments of the neurocranium (most of them being unidentified precisely at that time) and the second set is composed of 19 infracranial remains (Table 1). There is no duplication in the infracranial assemblage (the same seems true for the neurocranium, but this needs to be carefully checked at a future date). All the infracranial bones have the same color (greyish), robusticity (or lack of robusticity) and stage of maturation, and most of the cranial and infracranial remains are partly covered by a very thin layer of concretion. As the author was studying the human material from Baousso da Torre at that time, the striking morphological similarities, taphonomic alterations and color between these bones and those from BT3 were obvious. In October 2013, this initial impression was confirmed by bringing four fragments of bone from the Musée lorrain collection to the IPH. The result is illustrated by the figure 1. The association of fragments from both collections is unquestionable, and it seems highly probable that, at least for the infracranial remains, all of the selected bones belong to BT3. For the cranial remains, some refits have been found (at least two fragments with the quadrangular fragment of parietal) but more time needs to be spent to carefully analyze these remains (Table 1, Figure 1).

Some remarks on the paleobiology of BT3

This discovery significantly increases the representation of the BT3 skeleton (Table 2). When data from both collections are brought together, they permit an estimate of the most likely age-at-death of the individual, following [START_REF] Schaefer | Juvenile osteology: a laboratory and field manual[END_REF][START_REF] Alqahtani | Brief communication: The London atlas of human tooth development and eruption[END_REF][START_REF] Passalacqua | Subadult Age-at-Death Estimation From the Human Calcaneus[END_REF], as being between 10 and 14 years (Table 3).

Fortunately, this age-range is the best for attempting to assess the sex of an immature individual based on the morphology of the ilium: it has been shown that iliac shape differences between the sexes increases during ontogeny [START_REF] Bilfeld | Ontogeny of Size and Shape Sexual Dimorphism in the Ilium: A Multislice Computed Tomography Study by Geometric Morphometry[END_REF][START_REF] Wilson | Shape, size, and maturity trajectories of the human ilium[END_REF], and that that shape becomes significantly sexually dimorphic after 11 years of age, although visible shape differences can be observed even earlier [START_REF] Bilfeld | Ontogeny of Size and Shape Sexual Dimorphism in the Ilium: A Multislice Computed Tomography Study by Geometric Morphometry[END_REF][START_REF] Wilson | On the reliability of a geometric morphometric approach to sex determination: A blind test of six criteria of the juvenile ilium[END_REF]. A sex assessment for BT3 individual was attempted using the complete right ilium (Fig. 2), even though (and taking into account) sex-assessment of juveniles has been considered as one of the most thorny problems in biological anthropology since the origin of the discipline. Pictures of the bone were taken following recommendations in [START_REF] Wilson | Morphometric Criteria for Sexing Juvenile Human Skeletons Using the Ilium[END_REF][START_REF] Luna | New Method for Sex Prediction Using the Human Non-Adult Auricular Surface of the Ilium in the Collection of Identified Skeletons of the University of Coimbra[END_REF], and analyzed with ImageJ. Observations were made according to [START_REF] Luna | New Method for Sex Prediction Using the Human Non-Adult Auricular Surface of the Ilium in the Collection of Identified Skeletons of the University of Coimbra[END_REF][START_REF] Schutkowski | Sex determination of infant and juvenile skeletons: I. Morphognostic features[END_REF][START_REF] Weaver | Sex differences in the Ilia of a known sex and age sample of fetal and infant Skeletons[END_REF]. Schutkowski's and Weaver's methods [START_REF] Schutkowski | Sex determination of infant and juvenile skeletons: I. Morphognostic features[END_REF][START_REF] Weaver | Sex differences in the Ilia of a known sex and age sample of fetal and infant Skeletons[END_REF] were originally created on neonates and young children, and their accuracy may be fairly low when applied to samples other than those used to create them (e.g. [START_REF] Majó | Test des fonctions discriminantes de Schutkowski impliquant l'ilium pour la détermination du sexe dans des séries d'enfants de sexe et d'âge connus[END_REF][START_REF] Hunt | Sex Determination in the Subadult Ilia: An Indirect Test of Weaver's Nonmetric Sexing Method[END_REF]). However when the results of the methods are displayed by age group, they show relatively good accuracy for individuals (especially males) who died between 10 and 14 years old [START_REF] Sutter | Nonmetric subadult skeletal sexing traits: I. A blind test of the accuracy of eight previously proposed methods using prehistoric known-sex mummies from northern Chile[END_REF][START_REF] Vlak | Greater sciatic notch as a sex indicator in juveniles[END_REF][START_REF] Mittler | Sex determination in subadults using auricular surface morphology: a forensic science perspective[END_REF]. Luna et al [START_REF] Luna | New Method for Sex Prediction Using the Human Non-Adult Auricular Surface of the Ilium in the Collection of Identified Skeletons of the University of Coimbra[END_REF] presented clear and well illustrated metrical and non-metrical, morphological methods for recording the juvenile auricular surface [START_REF] Luna | New Method for Sex Prediction Using the Human Non-Adult Auricular Surface of the Ilium in the Collection of Identified Skeletons of the University of Coimbra[END_REF], something that was previously lacking [24]. The visual recording part was applied to BT3, but unfortunately the measures were not recordable due to the extent of taphonomic alterations of the margin of the auricular surface. Table 4 displays the results obtained. All criteria point toward a male assessment, and the author considers it to be reasonably reliable.

Finally, it is worthwhile noting that at least one fragment of fibula (and maybe two other fragments, one of the frontal bone and of the right ulna) displays pathological changes. This fibula is broken through an abnormal enlargement of the diaphysis, which displays an active sub-periosteal reaction in this area.

Some concluding thoughts

At least 19 infracranial fragments and two cranial fragments belonging to BT3 were discovered at the IPH. It is likely that many other cranial fragments belong to this individual. The skeletal remains of BT3 have still to be studied in greater detail due to the difficulty of analyzing fragments curated in two different places. However, this discovery permits allocation of BT3 to the age class 10 to 14 years and a likely male sex assessment, as well as the identification of at least one pathological bone. Gravettian immature individuals are relatively rare [START_REF] Henry-Gambier | Comportement des populations d'Europe au Gravettien : Pratiques funéraires et interprétations[END_REF]. A detailed study of BT3 would thus provide new data on the paleobiology, including ontogenetic aspects, of folks from this time period.

Two "mysteries" still remain: why did Rivière not mention these remains, and how did they come to be in the IPH drawers? At the end of the description of the BT3 skeletal remains, Rivière clearly stated that he found no other skeletal elements, apart from a few fragments that could not be identified. It seems he was not referring to the bones curated at the IPH, so one can posit that Rivière did not actually see these remains. Rivière [START_REF] Rivière | Antiquité de l'homme dans les Alpes-Maritimes[END_REF] did not clearly state who found the burials, even though he gave the impression that he was there (in his monograph he generally used the term "we" for "I", and did use "we" for BT3 discovery as well). One hypothesis would be that Rivière was not actually present at the time of the discovery and that somebody else excavated the remains. For some unknown reason, some of the bones were removed and not seen by Rivière. This would explain the first mystery. To attempt to resolve the second one, one would have to access to the archives of the IPH and look for some information on the material. Table 4 The most probable sex for each criterion scored on the BT3 right ilium / Sexe le plus probable pour chaque critère observé sur l'ilium droit de BT3

Criterion (reference) Morphology of BT3 ilium Considered as indicator of:

Test for 10+ years old juvenile individuals

Greater sciatic notch angle [START_REF] Schutkowski | Sex determination of infant and juvenile skeletons: I. Morphognostic features[END_REF] 90.9°(small) Male Never great in males, sometimes small in females [START_REF] Sutter | Nonmetric subadult skeletal sexing traits: I. A blind test of the accuracy of eight previously proposed methods using prehistoric known-sex mummies from northern Chile[END_REF][START_REF] Vlak | Greater sciatic notch as a sex indicator in juveniles[END_REF] Greater sciatic notch depth [START_REF] Schutkowski | Sex determination of infant and juvenile skeletons: I. Morphognostic features[END_REF] Deep Male Never shallow in males, sometimes deep in females [START_REF] Sutter | Nonmetric subadult skeletal sexing traits: I. A blind test of the accuracy of eight previously proposed methods using prehistoric known-sex mummies from northern Chile[END_REF][START_REF] Vlak | Greater sciatic notch as a sex indicator in juveniles[END_REF] Auricular elevation [START_REF] Weaver | Sex differences in the Ilia of a known sex and age sample of fetal and infant Skeletons[END_REF] Not elevated Male Rarely elevated in males, rarely not in females [START_REF] Sutter | Nonmetric subadult skeletal sexing traits: I. A blind test of the accuracy of eight previously proposed methods using prehistoric known-sex mummies from northern Chile[END_REF][START_REF] Mittler | Sex determination in subadults using auricular surface morphology: a forensic science perspective[END_REF] Arch criteria [START_REF] Schutkowski | Sex determination of infant and juvenile skeletons: I. Morphognostic features[END_REF] Bordering auricular surface Male Variation across studies and observers, likely a poor indicator [21,25] Auricular surface overall morphology [START_REF] Luna | New Method for Sex Prediction Using the Human Non-Adult Auricular Surface of the Ilium in the Collection of Identified Skeletons of the University of Coimbra[END_REF] V shape, the inferior edge is slightly longer than the anterior edge, and obtuse angle

Male

Configuration frequently seen in males, rarely in females [START_REF] Luna | New Method for Sex Prediction Using the Human Non-Adult Auricular Surface of the Ilium in the Collection of Identified Skeletons of the University of Coimbra[END_REF] Apex morphology [

Fig. 1

 1 Fig. 1 Examples of refitting between fragments from the Musée lorrain (ML) and the IPH collections. Scales =2cm / Exemples de remontages entre des fragments du Musée lorrain (ML) et de l'IPH. Echelles = 2 cm

  24. Majó T (1996).Réflexions méthodologiques liées à la diagnose sexuelle des squelettes non-adultes. BMSAP 8(3-4): 481-90. 25. Cardoso HFV, Saunders SR (2008). Two arch criteria of the ilium for sex determination of immature skeletal remains: A test of their accuracy and an assessment of intra-and inter-observer error. Forensic Sci Int 178(1): 24-9.

Fig. 2

 2 Fig. 2 Medial and superior views of the right os coxae. Scale = 2cm / Coxal droit, vues médiale et supérieure. Echelle = 2 cm

Table 1

 1 Infra-cranial immature remains labeled and curated at the IPH and probably belonging to BT3 / Restes immatures infracrâniens conservés à l'IPH et appartenant probablement à BT3

	Villotte Labels	IPH labels	"Site"	Drawer	N°on	Bone	Preservation
		(database)			the bone
	BRIPH68-022	1968-9-25	Baoussé Roussé	23	1968-9-25	Thoracic vertebra Posterior part
	BRIPH68-023	1968-9-27	Baoussé Roussé	23	1968-9-27	Lumbar vertebra	Right lamina and
							articular process
	BRIPH68-024	1968-9	Baoussé Roussé	22	1968-9	Right humerus	Distal third
							of the diaphysis
	BRIPH68-025	1968-9-12	Baoussé Roussé	22	1968-9-19	Right radius	Fragment of diaphysis
	BRIPH68-026	1968-9-20	Baoussé Roussé	22	1968-9-20	Right ulna	Fragment of diaphysis
	BRIPH68-027	1968-9-15	Baoussé Roussé	23	1968-9-15	Right Ilium	Complete
	BRIPH68-028		Baoussé Roussé	23	1968-9-15	Right ischium	Fragmented
	BRIPH68-029		Baoussé Roussé	23	1968-9-15	Left ischium	Subcomplete
	BRIPH68-030	1968-9-21	Baoussé Roussé	23	1968-9-21	Right patella	Complete
	BRIPH68-031	1968-9-28	Baoussé Roussé	23	1968-9-28	Right (?) tibia	Medial part
							of the proximal epiphysis
	BRIPH68-032		Baoussé Roussé	22	1968-9	Right femur	Proximal third
							of the diaphysis
	BRIPH68-033	1968-9-17	Baoussé Roussé	22	1968-9-17	Left tibia	Proximal third
							of the diaphysis
	BRIPH68-034		Baoussé Roussé	23	1968-9-16	Left fibula	distal extremity
							of the diaphysis
	BRIPH68-035	1968-9-18	Baoussé Roussé	23	1968-9-18	Left (?) fibula	Fragment of diaphysis
	BRIPH68-036	1968-9-30	Baoussé Roussé	23	1968-9-30	Right fibula	Distal epiphysis
	BRIPH68-037	1968-9-24	Baoussé Roussé	23	1968-9-24	Right (?) Fibula	Fragment of diaphysis
	BRIPH68-038	1968-9-40	Baoussé Roussé	23	1968-9-40	Right talus	Postero lateral portion
	BRIPH68-039	1968-9-44	Baoussé Roussé	23	1968-9-44	Right cuboïd	complete
	BRIPH68-040	1968-9-47	Baoussé Roussé	23	1968-9-47	Right metatarsal IV Proximal end

Table 2

 2 The state of preservation of the BT3 remains, reported separately for bones from the right and the left sides, respectively. -: missing; i: intact; d: damaged; f: fragmented; ff: very fragmented / Etat de conservation des restes squelettiques de BT3, pour chaque os du coté droit et gauche respectivement.-: manquant; i: intact; d: endommagé; f: fragmenté; ff: très fragmenté.

	Musée lorrain	Musée lorrain + IPH
	Neurocranium (ff), face (ff), sacrum	Neurocranium (?), face
	(ff), humeri (d / -), ulnae (f / -), coxal	(ff), vertebrae t. (1f),
	bones (-/ ff), femora (f / d), patellae	vertebrae l. (1f),
	(-/ i), tibiae (-/ ff), tali (f / i),	sacrum (ff), humeri (d /
	calcanei (d / f), other tarsals (1f / 4i-	-), radii (d / -), ulnae (d
	d), metatarsals (2d-f / 3i-f),	/ -), coxal bones (d / f),
	phalanges (3i / 5i-f).	femora (d / d), patellae
		(i / i), tibiae (ff / d),
		fibulae (f / f), tali (d / i),
		calcanei (d / f), other
		tarsals (2i-f / 4i-d),
		metatarsals (3d-f / 3i-f),
		phalanges (3i / 5i-f).

Table 3

 3 Age range estimates from each of the observable indicators present in the BT3 skeletal remains / Estimations de classes d'âge pour chacun des indicateurs observables pour le squelette de BT3.

	Indicator of age-at-death

  16] Angular Male Rarely rounded in males, rarely angular in females [16] Curvature of the iliac crest [17] Marked S-shape Male Variable in males, Marked S-Shape rare in females [21]

Acknowledgments The author thanks D. Henry-Gambier (UMR 5199 PACEA, Université de Bordeaux -CNRS) for giving him the opportunity to study this fossil, and R. Dagorne and T. Dechezleprêtre (Musée lorrain), Stéphanie Renault (Muséum national d'Histoire naturelle) and Amélie Vialet (Institut de Paléontologie Humaine) who allowed access to the material. Thanks to C. Knüsel (UMR 5199 PACEA, Université de Bordeaux -CNRS), E. Trinkaus (Department of Anthropology, Washington University), the two editors and the anonymous reviewer for their useful comments on the manuscript. This research was partially funded by the CNRS, the ANR (ANR GUEROPE (ANR-06-CONF-0008); ANR GRAVETT'OS (ANR-15-CE33-0004)), and the British Academy (BA small grant).

Conflict of interst : To come