Till Speicher
email: tspeicher@mpi-sws.org

Muhammad Ali
email: muali@mpi-sws.org

Giridhari Venkatadri
email: venkatadri.g@husky.neu.edu

Filipe Ribeiro
email: filiperibeiro@dcc.ufmg.br

George Arvanitakis
email: george.arvanitakis@eurecom.fr

Fabrício Benevenuto
email: fabricio@dcc.ufmg.brufmg

Krishna P Gummadi
email: gummadi@mpi-sws.org

Patrick Loiseau
email: patrick.loiseau@univ-grenoble-alpes.fr

Alan Mislove
email: amislove@ccs.neu.edu

Filipe Nunes

Sorelle A Friedler

Christo Wilson

Potential for Discrimination in Online Targeted Advertising

Keywords: Discrimination, advertising, Facebook

la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Much recent work has focused on detecting instances of discrimination in online services ranging from discriminatory pricing on e-commerce and travel sites like Staples [START_REF] Mikians | Detecting price and search discrimination on the internet[END_REF] and Hotels.com [START_REF] Hannák | Measuring price discrimination and steering on e-commerce web sites[END_REF] to discriminatory prioritization of service requests and offerings from certain users over others in crowdsourcing and social networking sites like TaskRabbit [START_REF] Hannák | Bias in online freelance marketplaces: Evidence from TaskRabbit and Fiverr[END_REF]. In this paper, we focus on the potential for discrimination in online advertising, which underpins much of the Internet's economy. Specifically, we focus on targeted advertising, where ads are shown only to a subset of users that have attributes (features) selected c 2018 T. Speicher et al.

by the advertiser. Targeted ads stand in contrast to non-targeted ads, such as banner ads on websites, that are shown to all users of the sites, independent of their attributes.

The targeted advertising ecosystem comprises of (i) advertisers, who decide which users an ad should (not) be shown to; (ii) ad platforms, such as Google and Facebook, that aggregate data about their users and make it available to advertisers for targeting; and (iii) users of ad platforms that are consumers of the ads. The potential for discrimination in targeted advertising arises from the ability of an advertiser to use the extensive personal (demographic, behavioral, and interests) data that ad platforms gather about their users to target their ads. An intentionally malicious-or unintentionally ignorantadvertiser could leverage such data to preferentially target (i.e., include or exclude from targeting) users belonging to certain sensitive social groups (e.g., minority race, religion, or sexual orientation).

Recently, the Facebook ad platform was the target of intense media scrutiny [START_REF] Angwin | Facebook lets advertisers exclude users by race[END_REF] and a civil rights lawsuit for allowing advertisers to target ads with an attribute named "ethnic affinity." After clarifying that a user's "ethnic affinity" does not represent the user's ethnicity, but rather represents how interested the user is in content related to different ethnic communities, Facebook agreed to not allow ads related to housing, employment, and financial services be targeted using the attribute (Facebook, 2017) and renamed it to "multicultural affinity."1

In this paper, we conduct a systematic study of the potential for discriminatory advertising on the Facebook advertisement platform. We focus on Facebook because it is one of the largest online advertising platforms in terms of number of users reached by ads, the number of advertisers, and the amount of personal data gathered about the users that is made available to advertisers. Furthermore, Facebook is an innovator in introducing new methods for targeting users, such as cus-tom audience 2 and look-alike audience 3 targeting that are then subsequently adopted by other online social media and social networking platforms like Twitter, 4 Pinterest, 5 LinkedIn, 6 and YouTube. 7 Thus, many of our findings may also be applicable to these other online ad targeting platforms as well.

Our study here is driven by the following highlevel question: What are all the different ways in which a Facebook advertiser, out of malice or ignorance, can target users in a discriminatory manner (i.e., include or exclude users based on their sensitive attributes like race)?

To answer this question, we begin by proposing an intuitive measure to quantify discrimination in targeted ads. We then systematically investigate three different targeting methods (attributebased targeting, PII-based targeting, and lookalike audience targeting) offered by Facebook for their ability to enable discriminatory advertising. At a high-level, we find that all three methods enable advertisers to run highly discriminatory ads. Worse, we show that the existing solution approaches of banning the use of certain attributes like "ethnic affinity" in targeting is not only inadequate, but does not even apply in two out of the three ad targeting methods.

While our findings primarily serve to demonstrate the perniciousness of the problem of discriminatory advertising in today's ad platforms, it also lays the foundations for solving (i.e., detecting and mitigating) ad discrimination.

Quantifying Ad Discrimination

Next, we begin by outlining different ad targeting methods offered by Facebook. We then discuss the current approach to determining whether an ad is discriminatory and argue why it is inade- quate. Finally, we propose a new and intuitive approach to quantify discrimination.

Methods for targeted ads

Facebook gathers and infers several hundreds of attributes for all of its users, covering their demographical, behavioral, and interest features 8 [START_REF] Andreou | Investigating ad transparency mech-anisms in social media: A case study of Facebook's explanations[END_REF]. Some of those attributes, such as gender or race, are considered sensitive meaning targeting (i.e., including or excluding) people based on those attributes is restricted by law for certain types of advertisements (e.g., those announcing access to housing or employment or financial services [START_REF] Barocas | Big data's disparate impact[END_REF]).

Facebook allows advertisers to select their target audience in three ways:

1. Attribute-based targeting: Advertisers can select audiences that have (or do not have) a certain attribute (or a combination of attributes), e.g., select users who are "men", "aged 35", and are interested in "tennis."

2. PII-based (custom audience) targeting: Advertisers can directly specify who should be targeted by providing a list of personally identifiable information (PII) such as phone numbers or email addresses.

3. Look-alike audience targeting: Advertisers can ask Facebook to target users who are similar to (i.e., "look like") their existing set of customers, specified using their PII.

Quantification approaches

Next, we discuss three basic approaches to quantifying discrimination and their trade-offs.

1. Based on advertiser's intent: An intuitive (moralized) way to quantify discrimination would be to base it on the advertiser's intent. However, not only is such a measure challenging to operationalize (i.e., to measure from empirical observations), but it also overlooks the harmful effects of unintentionally discriminatory ads that may be placed by a well-meaning but ignorant or careless advertiser. In this paper, we do not consider such approaches.

2. Based on ad targeting process: Another approach to determine whether an ad is 8. https://www.facebook.com/business/learn/ facebook-ads-choose-audience discriminatory is based on the process used to target the ads. Any ads placed using the right process would be non-discriminatory (by definition), while those using a wrong process would be declared discriminatory (by definition). Existing approaches, such as those that determine whether an ad is discriminatory based on the use of sensitive attributes (e.g., "ethnic affinity") in targeting, fall under this category. As we show in this paper, attempting to quantify discrimination based on the process (means or methods) of targeting is quite difficult when there exist multiple different processes for targeting users. Instead, in this work, we advocate for a third approach.

Based on targeted audience (outcomes):

We propose to quantify discrimination based on the outcomes of the ad targeting process, i.e., the audience selected for targeting. Put differently, we do not take into account how users are being targeted but only who they are. Outcomebased approaches to quantifying discrimination have the advantage that they can be generally applied to all scenarios, independently of the employed method of targeting. We discuss one such method in the next section.

Outcome-based discrimination

To formalize our discrimination measure, we will assume that an ad platform like Facebook keeps track of a database D = (u i ) i=1,...,n of userrecords u i where each user is represented by a vector of boolean attributes, i.e., u i ∈ B m . We denote the sensitive attribute (e.g., race or gender) that we are interested in a particular situation by s ∈ {1, . . . , m} and its value for a user u by u s . The corresponding sensitive group S is the set of all users that have the sensitive attribute, i.e., S = {u ∈ D |u s = 1}. To measure outcome-(i.e., targeted audience-) based discrimination, we define a metric for how discriminatory an advertiser's targeting is. It is inspired by the disparate impact measure that is frequently used to detect discrimination in selecting candidates from a pool of applicants in recruiting and housing allotment scenarios [START_REF] Barocas | Big data's disparate impact[END_REF].

Our key observation is that ad targeting, like recruiting, involves selecting the target audience (TA) from a much larger pool of relevant audience (RA). The relevant audience of an ad is the set of all users in the database D who would find the ad useful and interesting and thus might interact with it. Intuitively, the discrimination measure should capture the extent to which the target audience selection is biased based on sensitive group membership of relevant users.

We define the representation ratio measure for sensitive attribute s to capture how much more likely a relevant user u is to be targeted when having the sensitive attribute compared to not having it. More specifically, it is the ratio between the fraction of relevant audience with attribute s that are selected for targeting and the fraction of relevant audience without attribute s that are selected, i.e.,

rep_ratio s (TA, RA) = | TA∩RAs|/| RAs| | TA∩RA¬s|/| RA¬s| , (1)
where

RA s = {u ∈ RA | u s = 1} and RA ¬s = {u ∈ RA | u s = 0}.
Based on the representation ratio we define a measure that we call disparity in targeting, defined for a sensitive attribute s as:

disparity s (TA, RA) = (2)
max rep_ratio s (TA, RA), 1 rep_ratio s (TA, RA) .

Note that it is important to compute disparity based on the relevant audience RA because RA may have a very different composition in terms of attribute s than the whole database D. For example, an ad for men's clothes may have a relevant audience RA with a gender-ratio highly skewed towards men. A random selection of users from RA would be non-disparate with respect to RA, but might be highly disparate with respect to D. Similarly, for the same targeted audience (including mostly males), some ads could be non-discriminatory (e.g., ads for men's clothes) while others could be highly discriminatory (e.g., ads for high-paying jobs), depending on the corresponding relevant audience. Throughout the paper, we implicitly assume that, for the sensitive attributes considered, the relevant audience has the same distribution as the global population; and we show that the advertiser can include or exclude certains groups based on the sensitive attribute-hence the ad targeting is discriminatory.

We propose to detect discriminatory targeting using our disparity measure as follows: we declare a targeting formula as discriminatory when its disparity for some sensitive attribute value group exceeds a certain threshold (i.e., the group is over-or under-represented). For instance, a reasonable threshold value may be 1.25, mimicking the popular "80%" disparate impact rule [START_REF] Biddle | Adverse Impact and Test Validation: A Practitioner's Guide to Valid and Defensible Employment Testing[END_REF], to declare a group over-or underrepresented.

In addition to disparity, we would be interested in the recall of an ad, which quantifies how many of the relevant users with the sensitive attribute the discriminatory ad targets or excludes. It can be defined as

recall(TA, RA ) = |TA ∩ RA | |RA | , (3)
where RA might be the restriction of RA to RA s or RA ¬s , depending on whether the discriminatory advertiser wants to target or exclude users with the sensitive attribute s.

PII-based Targeting

In this section, we show how the audience targeting mechanism based on personally identifiable information (PII) recently introduced by Facebook can be exploited by advertisers to covertly implement discriminatory advertising. We first briefly describe the PII-based audience targeting feature of Facebook; we then explain how this feature can be exploited to implement discriminatory advertising. Next, we explain how public data sources have data that advertisers can use to implement discriminatory advertising, and finally demonstrate the feasibility of such an attack by using information from public records to create audiences for advertising that are discriminatory.

PII-based audience selection: While Facebook traditionally allowed advertisers to select audiences to advertise to by specifying attributes of the audience (e.g., age, gender, etc.), Facebook recently introduced custom audiences. This feature allows advertisers to specify exactly which users they want to target by specifying personally identifying information (PII) that uniquely identifies those users. Facebook allows 15 different types of PII to be used, including phone numbers, email addresses, and combinations of name with other attributes (such as date of birth or ZIP code). The advertiser uploads a file containing a list of PII; Facebook then matches these PII to Facebook accounts to create a custom audience. Custom audiences can be viewed as implementing a linking function that allows advertisers to link the large amounts of external personal data available today with Facebook's user information. The linking function that custom audiences provide to advertisers is not a one-to-one function (i.e., advertisers cannot determine the exact Facebook account of a given person), but rather, it is an aggregate function that maps PII to a group of Facebook users. In the next section, we show that, despite this limitation, custom audiences can be abused to covertly implement discriminatory advertising by exploiting external data to create lists that selectively include only people with the sensitive attribute.

Potential for discrimination

To implement discriminatory advertising using custom audiences, an advertiser could simply create a list of PII corresponding selectively to people who have the sensitive attribute, uploading this list of PII to create a custom audience, and then advertising to that custom audience. Since the advertiser does not upload the sensitive attribute (instead uploading only a list of PII), and since the advertising platform itself may not have the sensitive user attribute, such targeting becomes difficult to detect.

Most advertisers already possess significant amounts of customer information (e.g., customer data, information from data brokers); however, even if they do not have such data, there are many other sources of data-including public records, data brokers, and web data-that can be accessed for free or at low cost. We next describe public sources of data from which one can get sensitive attributes for large sets of people; we then demonstrate how these data sources can be used in combination with custom audiences to implement discriminatory advertising.

Public data sources

An increasing amount of information about people is publicly available; we now briefly discuss how advertisers could obtain large amounts of external personal information.

Race, age, and gender Most U.S. states release voter records that contain the personal information of all registered voters (names, phone numbers, addresses, etc) along with other sensitive attributes such as race, age, and gender. For example, date of birth and gender are available in the records released by 38 and 34 states, respectively [START_REF] Minkus | The city privacy attack: Combining social media and public records for detailed profiles of adults and children[END_REF]; race information is available in the records of eight states (North Carolina, New Mexico, Louisiana, Tennessee, Alabama, Georgia, Florida, and South Carolina) [START_REF] Ansolabehere | Gender, race, age and voting: A research note[END_REF]. Even when the race or gender is not available they can often be predicted with reasonable precision from other attributes [START_REF] Mislove | Understanding the demographics of Twitter users[END_REF]. For example, Tang et al. [START_REF] Tang | What's in a name: A study of names, gender inference, and gender behavior in Facebook[END_REF] propose a technique to infer the gender of a person from their name with an accuracy of 96.3% while covering more than 95% of users. Other companies such as Catalist 9 aggregate voter records from states and infer missing values of gender (from the first name) and race (from the name and address); the resulting race attributes matched voters' selfreported race 91% of the time [START_REF] Ansolabehere | Gender, race, age and voting: A research note[END_REF].

Criminal history People with criminal records-even those who have completed their sentence-are often victims of discrimination. We quickly survey the U.S. and find that more than 40 states in the U.S. make criminal records available online, and that 18 states offer free access to their state-wide criminal record databases; these records often contain significant amounts of personal information such as name, race, gender, and date of birth, along with the specific criminal record. Thus, advertisers can easily create custom audiences consisting only of users in this vulnerable population.

Discriminatory audience creation

We briefly demonstrate how it is possible to create discriminatory custom audiences on today's advertising platforms. Note that we did not actually advertise to these users or affect them in any way. Rather, our goal here is simply to demonstrate that using only public sources of data, advertisers can target protected classes and vulnerable populations with little effort. 9. https://www.catalist.us Table 1: Results from experiment creating custom audiences using only users with certain attributes from the North Carolina voter records. For each sensitive attribute, we created and uploaded a custom audience of 10K random voters with that attribute. Shown is the total number of records per attribute, the number of Facebook users in the resulting Targetable custom audience, and the percentage of Targetable users who match the sensitive attribute as per Facebook's estimates. We downloaded the public voter records from North Carolina, 10 giving us 7.5M records. Using data from the voter records, we then created custom audiences on Facebook for each sensitive attribute, selecting a random subset of 10K users from the voter file with each attribute. For example, we created a custom audience of women by uploading a list of 10K voters listed as female; we created a custom audience of white users by uploading a list of 10K voters listed as white. We created these custom audiences by uploading records containing the following fields: last name, first name, city, state, zip code, phone number, and country.

We then examine how many of these records match to Facebook accounts that can be targeted with advertisements, and then evaluate whether the created audiences are indeed discriminatory. Whenever we target an audience (either based on attributes, or by specifying a custom audience), Facebook provides an estimate of the number of users in the audience who can be targeted with advertisements; this estimate is called the potential reach. 11 We first target only the custom audiences created, without any additional targeting attributes specified, and use the potential reach estimate to measure how many records in the audience are Targetable.

10. http://dl.ncsbe.gov/index.html?prefix=data/ 11. Facebook previously defined the potential reach as "the number of daily active people on Facebook that match the audience you defined through your audience targeting selections."

Finally, in order to validate that advertisements targeted to these custom audiences would indeed be discriminatory, we take each custom audience and then target users with the corresponding sensitive attribute (e.g., for the male voter records audience, we target the Male attribute); we then measure the potential reach, and use the potential reach to measure what percentage of the Targetable users in the audience actually have that sensitive attribute (according to Facebook). Ideally, the percentage of Targetable users with the sensitive attribute would be 100%; however, Facebook may not know the attributes of some users, may have errors in their matching algorithm, or there may be errors in the userprovided data, making this percentage smaller.

It is important to note that definitions in our data sources (the voter file and census data) do not always line up with the targeting options that Facebook presents. For race, Facebook does not provide race directly but instead provides "ethnic affinity"; this is the same targeting parameter by which Facebook was accused of allowing discriminatory advertising [START_REF] Angwin | Facebook lets advertisers exclude users by race[END_REF].

Results

The results of this experiment are shown in Table 1, and we make a number of interesting observations. First, the fraction of voter records that are Targetable (i.e., online on a daily basis) is both significantly high (over 65% for most audiences we create) and fairly consistent across custom audiences. The only notable outliers are the Age (55+) audience, with only 57% matching.

Second, we observe that the fraction of the Targetable audience that matches the sensitive attribute, although it varies fairly widely across the different sensitive attributes, is consistently much higher than the fraction of the general adult population that has those sensitive attributes (assuming the voter records to be representative of the general adult population). In particular, for many sensitive attributes including gender, most races, and all ages, the percentage of Targetable audience that matches the sensitive attribute is higher than 80%. We suspect that the reason this fraction is low for the Asian attribute is due to the fact that race is an attribute that users typically do not upload to Facebook directly; however, we leave determining the source of this inconsistency to future work. We also note that even for these cases, the fraction of the Targetable audience that matches the sensitive attribute is significantly higher than the fraction of the voter records with the sensitive attribute. Taken together, our results show that advertisers can exploit public records to easily target discriminatory advertisements to a large number of people.

Summary

We explored the inherent risks that custom audiences induce for end users by allowing the linking of external information with Facebook's user data. We demonstrated the ease with which malicious advertisers could leverage the custom audience feature now present on advertising platforms like Facebook to implement discriminatory advertising. In fact, the wide variety of sources of public data available today means that even if an advertiser does not possess customer records of its own, it can easily find data sources to feed into custom audience creation.

Attribute-based Targeting

In this section, we examine how Facebook's attribute-based targeting mechanism can be used to launch discriminatory ads. First, we briefly explain how attribute-based targeting works and then examine the potential for abusing it.

Attribute-based audience selection: In brief, attribute-based targeting refers to the pro-cess of selecting an ad audience by specifying that recipients need to have a certain attribute or a combination of attributes; this is the traditional way of targeting ads on Facebook. For each user in the US, Facebook tracks a list of over 1,100 binary attributes spanning demographic, behavioral and interest categories that we refer to as curated attributes. Additionally, Facebook tracks users' interests in entities such as websites, apps, and services as well as topics ranging from food preferences (e.g., pizza) to niche interests (e.g., space exploration). We refer to these as free-form attributes, as they number at least in hundreds of thousands. It is unclear how exactly Facebook infers these attributes, but from their own description 12 this information can be gathered in many different ways such as user activity on Facebook pages, apps and services, check-ins with Facebook, and accesses to external webpages that use Facebook ad technologies. Beyond specifying a target region, language, age and gender for their ad, advertisers can choose that an ad should be shown to people that have some of these curated or free-form attributes turned on or off.

Potential for discrimination

The potential for discrimination on the Facebook ad platform was first publicly highlighted when researchers discovered the ability to exclude people based on their "ethnic affinity" (a curated attribute) when targeting ads related to housing [START_REF] Angwin | Facebook lets advertisers exclude users by race[END_REF]. Facebook responded by banning the use of ethnic affinity attribute for certain types of ads (Facebook, 2017). More recently, researchers discovered the ability to target people interested in or holding anti-semitic viewpoints via free-form attributes like "jew haters" (Angwin et al., 2017b).

These findings raise several questions about the potential for discriminatory targeting using Facebook's curated and free-form attributes. First, given that ethnic affinity-based targeting was disallowed for its potential correlation with ethnicity (race) of users, are there other demographic, behavioral, or interest attributes that are similarly correlated, if not more? Second, given that there exist hundreds of thousands of free-form attributes, can malicious advertisers 12. https://www.facebook.com/ads/about/?entry_ product=ad_preferences find facially neutral free-form attributes that disproportionately target or exclude users of a sensitive group. For example, an advertiser seeking to create an audience excluding certain ethnic groups may choose to select her target audience from users interested in particular news media sites or magazines.

To answer these questions and understand how vulnerable the Facebook ad platform is to these kinds of indirect discrimination, we investigate how strongly curated attributes other than "ethnic affinity" correlate with ethnicity and whether free-form attributes that are facially unrelated to sensitive attributes can be used as proxies for sensitive attributes. We executed these experiments by automatically querying the Facebook ad interface for the number of people belonging (or not belonging) to sensitive groups that have a certain curated or free-form attribute.

Discriminatory audience creation

We now explore how both curated and free-form attributes are correlated with ethnicity.

Curated attributes:

We conduct our analysis in the way described in Section 3.3. We use the custom audience mechanism to create groups of people from the North Carolina voter records that only contain particular ethnicities (White, African-American, Asian, and Hispanic). We then create sub-audiences by choosing to only target users matching each curated attribute and observe the size estimates of these sub-audiences. The percentage of users from each audience for whom Facebook inferred a curated attribute reveals how prevalent the attribute is within the audiences of different ethnicities.

The top three inclusive and exclusive attributes per ethnicity are shown in Table 2. The results point out that ethnic affinity is by far not the only and-in many cases-not even the most disparate feature with respect to ethnicity. For example, when targeting Asians on Facebook, it is more effective to do so based on political leaning or eating habits. The tradeoffs between representation ratio and recall for members outside the sensitive group, which an advertiser has to consider when aiming to exclude sensitive group members, can also be gauged from the table. In particular, there are a number of curated attributes with low representation ratio (i.e., high disparity), some of which achieve high recall for members not belonging to the sensitive group.

Free-form attributes: We begin our investigation by gathering an extensive (though not exhaustive) list of free-form attributes that are supported by the Facebook marketing API. 13 The API provides two useful calls that we exploit: i) given a piece of text, the API provides a list of free-form attributes that match the given text; and ii) given an attribute, the API provides a Table 3: Free-form attributes that may be used for discriminatory targeting. We show the percentage of the attribute audience that are members of the sensitive group as well as the fraction of the U.S. Facebook population that are members of the sensitive group, as a reference. LGBT community(21M), Gay pride (13M), Same-sex marriage(4.2M) Vulnerable people Addicted (100K), REHAB (450K), AA (50K), Support group (610K) list of other related attribute suggestions. For instance, the list of related attributes for 'The New York Times' includes 'The Washington Post', 'The Wall Street Journal', and 'The Economist'.

Free-form

We start with a seed set of names of news outlets extracted from three different sources: Google News [START_REF] Leskovec | Meme-tracking and the dynamics of the news cycle[END_REF], List of Newspapers, 14 and the top 1,000 newspapers from Alexa. 15 We first identify around 3,000 free-form attributes that exactly match with the names of the news outlets. We then execute a snowball sampling on these attributes, using Facebook's related attribute suggestions recursively starting from them. This process resulted in retrieving nearly 240,000 free-form attributes.

We begin by trying to find attributes from the above set of 240,000 attributes that can be used to primarily target or exclude people belonging to sensitive groups. Table 3 shows example free-form attributes that could be exploited for discriminatory targeting. For example, the attribute 'Marie Claire' has an audience with 90% of women, a much larger fraction than the proportion of U.S. women in Facebook (54%). Similarly, the attribute 'myGayTrip.com' has an audience of 38.6% men interested in men, while only 0.38% of the U.S. population in Facebook consists of men interested in men. We also identi-14. http://www.listofnewspapers.com/ 15. https://www.alexa.com/topsites/category/Top/ News/Newspapers fied a number of attributes with very biased audiences in terms of racial affinities. For example, 'BlackNews.com' has an audience with 89% of the users with African American affinity (in contrast with 16% of African American affinity in the reference population), the audience of 'Hoa hoc Tro Magazine' is composed of 95% users with Asian American affinity, which corresponds to 28 times more in comparison with the reference population. Similarly, 'Nuestro Diario' has an audience with 98% of Hispanic affinity (16% on the reference population). These results suggest that a malicious advertiser could easily find free-form attributes to launch discriminatory ads based on gender, race, and sexual orientation.

More worryingly, some free-form attributes allow a malicious advertiser to target people based on their beliefs. Table 4 presents a few sensitive free-form attributes from our dataset along with their potential audience in the U.S. These attributes correspond to a large audience with specific religious beliefs, including 'islam' (5.7M), 'catholic church' (6.5M), and 'evangelicalism' (5.6M). Thus, although it is not possible to target religion using curated attributes, one can use free-form attribute targeting to narrow the audience to people who are interested in a specific religion. Finally, we note that it is possible to target or exclude gay and LGBT users (or people sympathetic to their causes) via attributes like 'LGBT community' (21M), 'Gay Table 5: Suggestions for the most conservative news outlets. The left column shows a set of freeform attributes for conservative news outlets and the right column shows the corresponding free-form attributes suggested by Facebook. The percentage of very conservative users in the audience of each of these free-form attributes is shown in parentheses. pride' (13M), 'Same-sex marriage' (4.2M), as well as groups of vulnerable people, including 'Addicted' (100K), 'REHAB' (450K), 'AA' (50K). While this last set of free-form attributes might be useful, for example, for an advertiser to prevent addicted people to receive ads about alcoholic beverages, a discriminatory advertiser could explicitly exclude them.

Using Facebook's attribute suggestions:

We first investigate the free-form attributes suggested by Facebook to better understand the criteria used to select these suggestions. Table 5 shows the suggestions returned by the Facebook Marketing API (right column) given a freeform attribute (left column). We selected attributes associated with news outlets biased towards conservative audience to check whether their respective suggestions are also similarly biased. For instance, almost 80% of the audience of Townhall.com 16 are "very conservative" Facebook users, whereas the average amount of very conservative U.S. users of Facebook is about 13.89%. We note that the audiences corresponding to most of the suggested attributes also exhibit a strong bias towards conservative audience. From all suggestions presented, only Harper's Magazine 17 and RedEye 18 have a less conservative audience in comparison with the U.S. distribution.

16. https://www.facebook.com/townhallcom/ 17. https://www.facebook.com/HarpersMagazine/ 18. https://www.facebook.com/TheRedEye/ A malicious advertiser could exploit free-form attribute suggestions from Facebook in two different ways. First, a malicious advertiser could exploit the Facebook's attribute suggestions to discover attributes that are facially neutral, but are similarly biased as a given free-form attribute. For example, one suggestion from Facebook for 'myGayTrip.com' is the free-form attribute 'Matt Dallas', who is a gay actor.19 19.4% of the audience for 'Matt Dallas' are men interested in men, which is 51 times more than the U.S. distribution (0.38%). Thus, a malicious advertiser may use 'Matt Dallas' as a facially neutral proxy for targeting or excluding gay users.

Second, an advertiser can use the suggestion mechanism to search for extremely biased free-form attributes. For example, suppose an advertiser is interested in conservative leaning audiences and the most biased free-form attribute they know is 'Fox', with 37% of conservative audience. The advertiser can start with 'Fox' and keep choosing more and more conservative attribute suggestions until she reaches attributes with extreme conservative audience bias. Below, we show a sequence of suggested attributes, starting from 'Fox', that leads to 'The Sean Hannity Show', a free-form attribute with 95% conservative audience.

Fox (37%) → Fox News Channel (67%) → Sean Hannity (88%) → Mark Levin (93%) → Rush Limbaugh (93%) → The Rush Limbaugh Show (94%) → The Sean Hannity Show (95%).

Summary

In this section, we demonstrated that many curated attributes (beyond ethnic affinity) exhibit correlations with sensitive attributes like race, which makes them potential vectors for discrimination. We also investigated whether the freeform attribute targeting mechanism allows advertisers to target or exclude sensitive groups of users in a discriminatory manner. Specifically, we showed that advertisers can circumvent existing limitations on targeting users based on their interests in sensitive topics like religion and sexual orientation. Furthermore, we show that malicious advertisers can exploit Facebook's suggestions to discover new facially neutral free-form attributes that allow extremely biased targeting.

Look-Alike Audience Targeting

In this section, we show how the recently introduced look-alike audience targeting mechanism can be exploited by advertisers to covertly implement discriminatory advertising. We first briefly describe the look-alike audience targeting feature of Facebook; we then explain how this feature can be exploited to implement discriminatory advertising.

Look-alike audience selection: Recently, Facebook introduced the look-alike audience targeting feature to help advertisers reach people that are similar to (i.e., look like) their existing set of customers. 20 Look-alike audiences are a particularly useful feature for advertisers who have limited data about their customers and want to grow their customer base. Advertisers can use it to outsource the job of marketing (i.e., identifying the attributes of their potential customers and finding them) to Facebook.

To select look-alike audiences, advertisers need to first provide Facebook with information about their existing (initial) set of customers called the source audience. An advertiser can choose source audience users in a variety of ways, including by 20. https://www.facebook.com/business/help/ 164749007013531

uploading their customers' PII (similar to creating a custom audience) or by specifying them to be the fans of their Facebook page.

After specifying a source audience, Facebook allows advertisers to specify a geographical region (either countries or groups of countries) from which the look-alike audience should be chosen. Facebook orders (ranks) all users in the geographical region based on their similarity to (i.e., how closely they look like) the source audience and allows advertisers to select look-alike audiences by specifying a percentile range (e.g., <2% or 2%-4%) over these ordered users from the geographical region's population. Thus, an advertiser can select X to Y percentile of closest matching users from any country's population to target. In practice, Facebook limits Y to 10%.

Potential for discrimination

Our concern is that a malicious advertiser seeking to place discriminatory advertisements could exploit look-alike audiences as follows: they could start by creating a highly biased (highly discriminatory) source audience and use the look-alike audience feature to find a larger set of users that is similarly biased, effectively scaling the bias to much larger populations. Put differently, our concern is that when the source audience is discriminatory, its look-alike audience would also be discriminatory. In the following sections, we first investigate whether biases in source audience selection propagate to look-alike audience selection. Later, we show how an advertiser seeking to selectively target people of a particular race could simply create a small (in the order of a few thousands) but highly biased source audience consisting primarily of people of a particular race (as described in Section 3.3) and use it to effectively target a large (in the order of tens of millions) yet similarly-or worse, exaggeratedly-biased lookalike audience.

Bias in look-alike audience selection

In this section, we construct several highly biased source audiences and check if and how the selection biases in source audience propagate to look-alike audiences.

Similar to what we did in Sections 3 and 4, we use the North Carolina voter database to construct several groups of 10,000 randomly selected people based on their ethnicity (Asian, Black, White, Hispanic), gender, political affiliation (registered Democrat or Republican) and age (18-24, 24-35, 35-54, 55+). We construct a source audience corresponding to each group and for each source audience, we ask Facebook to construct look-alike audiences from the US in five percentile ranges: closest matching 2%, 2-4%, 4-6%, 6-8%, and 8-10% of the US population. Note that the audiences in the different percentile ranges do not overlap with one another and each subsequent percentile range becomes less similar (i.e., less closely matching) to the source audience.

Each of the five look-alike audiences we create (for every source audience of 10,000 people) consist of approximately 4.2 million people, thus totaling to an approximate of 21.1 million unique people in the US. Thus, look-alike audiences allow expansion of the source audience by over three orders of magnitude. The key remaining question is whether the look-alike audience selection reflects the biases in source audience selection.

To capture the biases in our audience selection, we define a measure that we call representation bias of a target audience for every user at-tribute f maintained by Facebook. Simply put, representation bias captures how disproportionately an attribute is observed amongst the target audience (TA) compared to the people in the geographic location from where the look-alike audience is being selected (the geographic location is the US in our scenario and we refer to people in the US as the relevant audience, RA). More formally, the representation bias of an attribute f in an audience is defined as

rep_bias f (TA, RA) = | TA f | | TA | | RA | | RA f | , (4)
where similar to the representation ratio (Equation (1)), TA f and RA f are the subsets of people with attribute f in TA and RA respectively. We leave out attributes with very low prevalence in Facebook from our analysis (i.e., attributes for which

| RA f |/| RA | < 0.01).
Knowing the representation bias of each attribute allows us to construct a ranking of attributes from most to least biased; we refer to attributes at the top of the ranking as overrepresented and those at the bottom to be underrepresented in a target audience. Table 6 shows the top 5 over-represented and under-represented attributes for the source audience of African Americans and its 2% and 2-4% (the most similar two) look-alike audiences. The table shows that a majority of attributes that were found to be overrepresented in the source audience remain so for the look-alike audiences; similar behavior can be observed for the underrepresented attributes. These results-particularly the presence of multicultural affinity attributes-suggest that Facebook is using its extensive set of attributes to likely infer the biases that we introduced into our source audience. Moreover, it is propagating these biases to the selection of look-alike audiences, constantly over-representing African Americans and under-representing Hispanics and Asian Americans compared to their proportions in the national population.

To further validate our findings above, we take the top 10 over-represented and underrepresented attributes in the source audience and computed their average rank in the look-alike audiences. We performed these computations for differently biased source audiences (selected along the basis of gender, age, ethnicity and political affiliation). Figure 1 shows how the average rank changes across the look-alike audiences given by Facebook. We can see that attributes that were most over-and under-represented in source audience tend to stay, on average, amongst the most over-and under-represented in the look-alike audiences, respectively. These results strengthen our inference that the look-alike audience feature in Facebook is able to both capture the biases in a source audience and propagate the biases to the larger audiences it helps construct.

Discriminatory audience creation

Having observed that the look-alike audience selection mimics the biases of the source audience selection, we now check whether the bias propagation is sufficiently strong to lead to discriminatory audience creation. To answer this question, we compute the disparity of the sensitive attribute on which the source audience was biased, and observe how disparate that attribute remains in the look-alike audiences made by Facebook. Note that since we are observing look-alike audiences built from source audiences where the sensitive attribute was severely exaggerated, we expect the disparity measure to reflect the disparity in favor of the attribute.

Source 2%

2-4% 4-6% 6-8% 8-10%

Look-alike audience Figure 2 shows how source audiences that were disparate in favor of an ethnic group tend to produce look-alike audiences also disparate in favor of that ethnicity; although as the audiences become less similar, the disparity tapers off. Only one of these audiences, the 2% look-alike audience for White, has a disparity below 1.25, the threshold obtained from the 80% disparate impact rule [START_REF] Biddle | Adverse Impact and Test Validation: A Practitioner's Guide to Valid and Defensible Employment Testing[END_REF]. These results show that Source 2% 2-4% 4-6% 6-8% 8-10%

Look-alike audience look-alike audiences selected using highly biased source audiences can be highly discriminatory.

Summary

In this section, we investigated whether it is possible to start with a small discriminatory source audience and then leverage Facebook's look-alike audience feature to construct a considerably larger discriminatory audience. We show that in order to select a look-alike audience, Facebook tries to infer the attributes that distinguish the audience from the general population and propagates these biases in the selection of look-alike audiences. Such bias propagation can amplify the explicit (intentionally created) or implicit (unintentionally overlooked) biases in a source audience of a few thousand to a lookalike audience of tens of millions. As Facebook is actively involved in the selection of the look-alike audience, one might argue that Facebook needs to be more accountable for the selection of such a discriminatory audience.

Concluding Discussion

Recently, concerns have been raised about the potential abuse of online advertising platforms to target ads related to housing, employment, and financial services only to users of a particular race, in violation of anti-discrimination laws.

In this paper, we set out to investigate the following high-level question: can a malicious advertiser leverage the different targeting methods offered by platforms like Facebook to target users in a discriminatory manner? At a high-level, our study makes the following contributions:

(i) We argue that the determination of whether a targeted ad is discriminatory should not be made based on the use or non-use of specific user attributes by advertisers. Rather, inspired by the notion of disparate impact [START_REF] Feldman | Certifying and removing disparate impact[END_REF], we propose a simple outcome-based measure for discriminatory targeting that is computed independently of the user attributes used in targeting.

(ii) Next, using public voter record data in the US, we conduct an empirical study demonstrating that several user attributes in Facebook, beyond the much-criticized "ethnic affinity," show strong positive and negative correlations with users belonging to different races. Worse, Facebook's related attribute suggestions can be exploited by advertisers to discover facially-neutral attributes that can be used for highly discriminatory audience targeting. Thus, simply banning certain attributes is insufficient to solve the problem.

(iii) Finally, we explore the vulnerability of two previously overlooked methods of targeting supported by Facebook namely, PII-based (custom) audience targeting and look-alike audience targeting. We show that both these methods can be exploited by a malicious advertiser to include or exclude users with certain sensitive features at scale (i.e., in the order of tens of millions of users).

Future work -Towards detecting and mitigating ad discrimination: Our study here has largely focussed on understanding the problem of discriminatory advertising rather than proposing solutions for detecting or mitigating discriminatory targeting. However, in the process, we lay the foundations for the future solutions. First, the discrimination measure proposed here could be used when designing procedures to detect discrimination in the future. Second, we argue that the look-alike audience selection feature also presents a promising solution to the problem of mitigating discrimination in audience selection. Specifically, ad platform providers could expand the targeted audience to include look-alike (most similar) users that belong to under-represented groups (rather than select all look-alike audience). We plan to explore the effectiveness of this approach in mitigating discrimination in future work.

Figure 2 :

 2 Figure 2: Disparity (in favor) for each ethnicity when the look-alike audiences are created from an audience biased on that ethnicity.

Table 2 :

 2 Most inclusive and exclusive curated attributes for each race. In parentheses are the recall and representation ratio for a population from North Carolina. These were obtained by uploading voter records filtered to contain only a single race, and then measuring the size of the subaudience targeted by each attribute. Attributes present in less than 5% of the population are not considered.

	Race	Most inclusive	Most exclusive
		US Politics: Liberal (8%, 2.76)	US Politics: Very Conservative (14%, 0.30)
	Asian	Frequent travelers (15%, 2.70)	African American affinity (17%, 0.41)
		Interest: Vegetarianism (7%, 2.23)	Interest: Country music (20%, 0.48)
		African American affinity (17%, 7.06)	US Politics: Very Conservative (14%, 0.18)
	Black	US Politics: Very Liberal (12%, 6.44)	US Politics: Conservative (17%, 0.22)
		Interest: Online games (9%, 4.91)	Interest: Mountain biking (6%, 0.35)
		Interest: Motorcycles (7%, 2.08)	US Politics: Very Conservative (14%, 0.50)
	Indian	Interest: Online games (9%, 2.04)	Away from hometown (22%, 0.51)
		Interest: Ecotourism (6%, 1.96)	Primary OS Mac OS X (7%, 0.56)
		US Politics: Very Conservative (14%, 5.19)	African American affinity (17%, 0.15)
	White	US Politics: Conservative (17%, 3.77)	US Politics: Very Liberal (12%, 0.16)
		Interest: Hiking (11%, 2.27)	Interest: Online games (9%, 0.20)

Table 4 :

 4 Examples of free-form attributes that can be targeted by advertisers. In the parenthesis, we show the number of audience that can be targeted or excluded with the attribute.

	Attribute Potential Target (PT) PT Audience (%) US Audience (%)
	Marie Claire	Female	90%	54%
	myGayTrip.com	Man interested in Man	38.6%	0.38%
	BlackNews.com	African American affinity	89%	16%
	Hoa hoc Tro Magazine Asian American affinity	95%	3.4%
	Nuestro Diario	Hispanic affinity	98%	16%
	Topic	Free-form attributes
	Religion	Islam (5.7M), Catholic Church (6.5M), Evangelicalism (5.6M)
	LGBT

Unfortunately, Facebook was found half a year later to still accept discriminatory ads, despite the fixes it claims were put in place(Angwin et al., 2017a).

https://developers.facebook.com/docs/ marketing-api

https://en.wikipedia.org/wiki/Matt_Dallas

Acknowledgements

We thank the anonymous reviewers for their helpful comments. This research was supported in part by the Alexander von Humboldt Foundation, CAPES, Fapemig, and NSF grants CNS-1563320 and CNS-1616234.