Stephanie Assad
email: assads@econ.queensu.ca

Emilio Calvano
email: emilio.calvano@unibo.it

Giacomo Calzolari
email: giacomo.calzolari@eui.eu

Robert Clark
email: clarkr@econ.queensu.ca

Vincenzo Denicolo
email: vincenzo.denicolo@unibo.it

Daniel Ershov
email: daniel.ershov@tse-fr.eu

Justin Pappas

Sergio Pastorello
email: sergio.pastorello@unibo.it

Andrew Rhodes
email: andrew.rhodes@tse-fr.eu

Lei Xu
email: lei.xu2@gmail.com

Vincenzo Denicolò

Justin Danielershov

Sergio Johnson

Andrew Pastorello

Lei Rhodes

Matthijs Xu

Wildenbeest

Justin Johnson
email: justin.johnson@cornell.edu

Matthijs R Wildenbeest

Vincenzo

Denicolò

Samuel Curtis Johnson

Autonomous algorithmic collusion: Economic research and policy implications

Keywords: Algorithmic Pricing, Antitrust, Competition Policy, Artificial Intelligence, Collusion, Platforms. JEL Classification: D42, D82, L42

la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

In the last fifteen years the drastic reduction of the cost of computation and data storage has (re-) activated general interest and significant developments in Artificial Intelligence (AI) and its market applications. In this paper we investigate the use and consequences of algorithms for pricing decisions that rely on Artificial Intelligence, "AI-powered algorithms", using both experimental tools involving such algorithms and empirical techniques. We argue that, in line with earlier suggestions in the law literature but in contrast to what many economists have previously argued, the growing use of such algorithms may increase the likelihood of collusion in some markets. But we also suggest new methods that may help fight such collusion.

Algorithms are not a new phenomenon in markets. At least since the 1980s, industries like airlines, hotels and financial markets have relied on these tools for pricing and trading decisions. Pricing algorithms for "revenue or yield management" can be thought as (possibly very long) lists of prespecified instructions to act in specific ways for specific contingencies, that the algorithm then executed (such as for example with Expert Systems). The novelty nowadays is a new generation of AI-powered algorithms. Their "intelligence" lies in the ability to autonomously learn how to reach a pre-specified objective in unknown environments without human intervention. Firms who want to deploy a pricing algorithm do not need to input information about demand or the strategic context in which this algorithm operates. Given a set of potential actions (today's price) for each possible observation (say, previous quantities and prices), the algorithm is capable of autonomously discovering the profit-maximizing mapping between what they observe and the price they choose.

The application of autonomously learning algorithms to price goods and services brings about important policy challenges that are becoming more relevant as pricing algorithms spread in online and traditional brick and mortar marketplaces. Amazon stresses the possibility and the benefits of pricing automation in its marketplace with a Selling Partners API service, 2 and Chen et al. (2016) document that more than one third of the best-selling items on Amazon.com were priced by pricing bots in 2014/2015. The European Commission's 2017 "Final report on the E-commerce Sector Inquiry" concludes that "A majority of retailers track the online prices of competitors. Two thirds of them use software programs that autonomously adjust their own prices based on the observed prices of competitors." Offline usage of pricing algorithms is spreading as well, for example, among gasoline retailers in northern Europe. 3 There is a growing new industry of software intermediaries offering automated pricing services, from turn-key options that even small sellers can afford to fully customized pricing software for large companies. 4 Many of these repricing companies, such as Kalibrate.com, a2i.com, and Kantify, explicitly rely on AI as a key characteristic of their algorithms.

The widespread adoption of algorithmic pricing reflects obvious benefits. Algorithms guarantee faster and potentially "better" decisions while saving costs. They are more responsive to changes in supply and demand conditions, which implies better inventory management and reduced waste, especially for perishable goods. They can also exploit consumer information, providing potentially highly personalized offers that could increase allocative efficiency. There is a general consensus that algorithmic pricing has the potential to generate significant efficiency gains and reduce transaction costs.

However, given the key allocative role that prices play in markets, algorithmic pricing can generate unintended consequences. Autonomous learning algorithms may learn to price discriminate on the basis of race or gender or fail to learn effective competitive strategies, resulting in higher market prices. Algorithms may also end up learning that the best way to guarantee maximal profits is to decrease competition by, for instance, coordinating with rival algorithms. Algorithms can make collusive outcomes easier to sustain due to increased ease of monitoring and quicker detection and punishment of deviations (Ezrachi and Stucke 2015; Mehra 2016). This is especially a concern in markets with high price transparency and near perfect monitoring like gasoline retail or the Amazon Marketplace. Algorithmic pricing can also affect competition if a single intermediary software provider sells their product to multiple competitors. Such adoption could lead to hub-and-spoke (where the provider acts as the hub of the sellers, Ezrachi and Stucke 2015) or parallel-use scenarios, with competitors coordinating to higher prices by delegating choices or relaying information to the same third party. These concerns are warranted by the statements and observed behaviour of software providers. Some providers promote their products by suggesting that they optimize for long-term revenues and avoid price wars (see for example Kantify). In Germany, advertisements show that at least one company offers their software to multiple stations and brands in the retail gas market.

In this paper we focus on algorithmic collusion. The fact that pricing algorithms may learn to collude autonomously, without being instructed to do so, and possibly without communication, opens up new challenging scenarios for market players, platforms and antitrust authorities. . Antitrust law and enforcement identify violations when colluding parties communicate explicitly. Currently, algorithms learning to tacitly collude (algorithmic collusion) is not a violation of antitrust or competition laws. It is crucial to study whether algorithms can learn to tacitly collude, whether algorithmic collusion does arise in practice, and potential policy responses to it.

The possibility of algorithmic collusion has not gone unnoticed by competition authorities. The OECD, the EU Competition Commissioner Vestager, the FTC in the US, the Competition Market Authority (CMA) in the UK, and the French, German and Canadian competition authorities all raised concerns about this risk and the need for additional information and monitoring. 5 More recently, authorities have also started to envision policy interventions to address algorithmic collusion. The FTC issued a guidance paper on the use of AI in markets with indications of desirable properties that AI tools should have to avoid unintended consequences. 6 The so called New Competition Tool currently being discussed in the European Union to cope with digital markets should be designed to account for "oligopolistic market structures with an increased risk for tacit collusion, including markets featuring increased transparency due to algorithm-based technological solutions (which are becoming increasingly prevalent across sectors)." 7The interest in algorithmic collusion by market authorities was anticipated by academic research. Early accounts of the possibility of algorithmic collusion were discussed by legal scholars, in particular Ezrachi and Stucke (2016) and Mehra (2016). However, it is only recently that economists have started to work on this topic. Common wisdom among economists was initially that algorithmic collusion is not possible or unlikely to arise in practice without explicit communication. 8 Theory models of adaptive learning suggest that tacit collusion is not possible (Milgrom and Roberts 1990). Some economists suggested that even if tacit algorithmic collusion is theoretically possible, it is unlikely to arise under dynamic real world conditions (Schwalbe 2018).

We present recent experimental evidence that autonomous collusion between algorithms can arise in synthetic environments (Calvano et al. 2020). We look at a new generation of reinforcement learning algorithms (Q-learning) that experiment with random actions as part of their learning. The dynamic systems induced by such algorithms are very hard to fully characterize using abstract modelling, except for very simple environments that are not realistic descriptions of markets. 9 Experimentally, however, it is possible to set up a testing environment to study how algorithms evolve and interact over time. Experiments allow for perfect identification in controlled albeit synthetic environments. Our setting features (i) algorithms that are representative of those likely to be used in practice, and (ii) a realistic simulation of actual marketplaces, i.e. a virtual market populated with consumers and pricing-algorithms. Using this approach we observe both market outcomes and their determinants. We find that reinforcement learning algorithms generate supra-competitive prices and that these higher prices are the result of tacit autonomous algorithmic collusion: without explicit communication, algorithms learn to engage in retaliatory pricing.

Having established that algorithms can learn to collude in synthetic environments, we present the first real empirical evidence of widespread algorithmic adoption raising margins and prices (Assad et al. 2020). Empirically, there are substantial challenges to identifying a causal link 9 Modelling of learning algorithms with a theoretical approach could offer deep insights on what to expect out of a repeated interaction among autonomous pricing algorithms. Authors in other disciplines (Brafuss et al 2019, for example) have attempted this approach using stochastic approximation methods. A recent theoretical literature has addressed the impact on market outcomes of algorithms that are "hard-coded", thus having no ability to explore and learn their behavior with market interactions. See for example, Miklos-Thal and Tucker (2019), Brown and MacKay (2019). between adoption and collusion. Pricing technology is often highly proprietary and adoption of new algorithms is rarely observed. Adoption choices are also not random and establishing causality is important. Moreover, even a causal link between adoption and observable markers of collusion such as higher prices and margins does not necessarily recover the intentions (i.e., strategies). Algorithmic adoption can affect competition but also other factors that change market prices (i.e., better demand discovery). We use comprehensive high frequency pricing data from German gasoline retailers to identify the adoption dates of algorithmic pricing technology by individual gas stations. We then use an instrumental variables approach to establish causality. We recover the effects of adoption on competition by focusing on pre-existing market structure: looking at the effects of adoption in monopoly vs. non-monopoly markets, and looking at the effects of market-wide adoption in duopoly markets. We find that algorithmic adoption increases margins and prices only for non-monopoly stations. In duopoly markets, margins and prices only increase if both stations adopt. Together, this suggests that algorithmic pricing has an effect on competition.

After showing that algorithmic collusion is a credible concern, we explore possible policies that can mitigate it (Johnson, Rhodes and Wildenbeest 2020). Because of aforementioned concerns with abstract modelling and the lack of empirical variation in policy, this is also done in a fully controlled experimental environment. The experimental approach allows a researcher to run a large set of experiments in a fully controlled environment, under many different market conditions and different algorithmic designs. This potentially allows her to identify factors that may make it harder for algorithms to collude. These factors may have to do with the design of the algorithms or with rules governing the marketplace. For example, many marketplaces have control over which products consumers consider, and could use this power to guide the behaviour of algorithms towards procompetitive outcomes. We show that changes in platform design, such as rewarding firms that cut prices with additional exposure to consumers, may help curb algorithmic collusion. We also show that policies raising consumer surplus can also raise platform profits. Overall, thoughtful marketplace design decisions may combat anti-competitive forces even when perpetrated by algorithms.

The paper proceeds as follows. In Section 2, we describe the experimental approach of algorithmic collusion has been explored in Calvano et al. (2020). In Section 3 we present the main findings of Assad et al. (2020) on the actual risk of algorithmic collusion. In Section 4, we discuss the remedies that platforms may put in place as a reaction of seller's algorithmic collusion, as investigated in Johnson et al. (2020). Although we think it is too early to provide concrete policy recipes as more research is needed, the concluding Section 5 distills some observations from the existing research to develop a robust policy design. Calvano et al. (2020) experiment with algorithms within the context of a workhorse model of competition in the economics literature: repeated oligopolistic competition where several firms compete over time with differentiated products. Each firm delegates its pricing to algorithms whose objective is to maximize the firm's discounted profit over an indeterminate time horizon. In each period, the algorithm observes and thus reacts to prices effectively charged in previous periods by all market participants. After making its choice, it observes the resulting profits realized in that period. The idea of the experimental approach is to study the behavior that these AI-powered pricing algorithms learn over time by observing them repeatedly interacting in this virtual market.

Experimental Analysis for Algorithmic Collusion

In particular, Calvano et al. (2020) perform experiments using a type of AI called reinforcement-learning, specifically Q-learning. Q-learning is an adaptive approach that allows algorithms to learn about the strategic environment over time based on their own actions. Although firms do not need to use AI algorithms to set prices, such algorithms are used in many other areas of application and it may be reasonable to think that firms may adopt these effective techniques in the future, if they haven't done so yet.

A small number of "hyperparameters" characterize the design of the algorithms. One parameter controls the rate of learning, that is the balance between what the algorithm has learned to date and the new observations. The experimentation rate governs the attitude to explore, that is the probability that in a given period the algorithm sets a possibly suboptimal price (given available knowledge) just to check the reaction of the market, that is of consumers and rivals. Few other parameters are then used to initialize the algorithms in each simulation, to set the discount factor embedded in the algorithms, the memory of the algorithms (typically one or two past periods) and, finally, prices are discretized in a finite price grid. The virtual environment is then completed with the economic parameters, that specify the number of firms active in the market, each firm's cost of production, the preferences of consumers (e.g. with logistic or linear demand functions) and the degree of differentiation, from homogeneous products to less substitutable ones. Notice that, while in principle these algorithms can react to a wide variety of inputs / observations, complexity increases rapidly as one expands the size of those potential information sets. For this reason, the baseline setup of Calvano et al. (2020) are able to condition their current price only on the previous period prices.Within this setup it is then possible to run many simulations for each parametric configuration and for different values of the economic and hyper-parameters. In this environment, algorithms explore and then learn by mutually interacting in what is called Multi Agent Reinforcement Learning. Giving the algorithms enough time to learn their strategies (in other terms, to converge according to some specific convergence criterion), one can investigate several outcomes. First, the actual prices set by the algorithms. Second, one can "dig" into the algorithms and study the embedded strategies they have learnt.

Experimental Results

Figure 1 illustrates the price distribution observed across 1000 representative simulations, with two firms and for given and reasonable economic and hyperparameters. 10 The variability reflects the fact that different simulations lead to quantitatively different, although as we shall see qualitatively similar, outcomes. It is instructive to compare these observed prices with two benchmarks, the monopolist price and the competitive price. The latter is the price that maximizes each firm's profit, given the rival's price. For differentiated products each firm prices above cost, equating marginal costs to marginal revenues. Crucially, in doing so firms ignore the negative impact that lowering their price has on their rival's profit. In the limit case of firms carrying identical products, the competitive price equals the marginal cost of production. The former is the price that a hypothetical monopoly owner of all firms in the market would set. Or, equivalently, is the price that a cartel would agree to charge to maximize the members' joint profits. The fact that a monopolist internalizes the impact of lowering prices on all firms is the reason why the monopoly prices exceed competitive one, to the benefit of consumers. The distribution shows that the algorithms set prices significantly higher than those in the competitive benchmark and not too far from the monopolist's prices. Clearly, these prices imply higher profits. In the simulations of figure 1, the algorithms manage to secure about 80% of the additional profits that they could make beyond the competitive benchmark if they were to behave as a hypothetical monopolist. 11 But how are these high prices obtained? A first possibility is that the pricing algorithms failed to learn to compete. If this were the case, a better algorithm or also a human player would promptly realize that by slightly reducing one's price it would be possible to exploit the rivals' high prices and attract most of the consumers in the market. In this case we should not worry about the results of simulations like those illustrated in Figure 1. High prices would be just a temporary phenomenon, wiped away by standard competitive pressure. The alternative is that the algorithms did instead learn to coordinate their prices and support them with some form of retaliatory pricing.

To test this conjecture, Calvano et al. (2020) perform experiments with the trained algorithms, meant to document how these tend to react to rival's price cuts. Specifically, at the end of each session, that is after concluding the learning phase, they override one of the pricing algorithms by forcing it to set a lower price for just one period and then report the behavior in the periods that follows this "shock." Figure 2 illustrates the typical pattern. Upon observing the firm 1's price cut, firm 2 substantially reduces its price in subsequent periods. Firm 1 follows suit as if it were expecting firm 2's reaction. This temporary "price war" exhibiting significantly lower prices gradually comes to an end, with both firms returning to the high prices they were charging before the exogenous shock. This property of reward (keeping prices high unless a price cut occurs), retaliatory pricing (for undercutting) and eventual forgiveness (increasing prices back to pre-deviation) is the hallmark of collusion. The algorithms have learned that undercutting the other firm's prices brings forth a war with low profits which ultimately makes any attempt to deviate from the spontaneous cartel price unprofitable. 12 Figure 2. After the two algorithms have learned their way to collusive prices, an attempt to "cheat" so as to gain market share is simulated by exogenously forcing one of the two algorithms to cut its price. From the "shock" period onwards, the algorithm regains control of the pricing. The deviation is punished by the other algorithm, so firms enter into a price war that lasts for several periods and then gradually ends as the algorithms return to pricing at a collusive level. [Source: Calvano et al. (2020). Copyright American Economic Association; reproduced with permission of the American Economic Review.] This finding is very robust as we will briefly discuss next, and it is remarkable that algorithms display such a stubborn ability to autonomously learn such a fairly sophisticated collusive strategy.. In fact, the observed pattern is very much consistent with what theoretical economic analysis of collusion among rational agents generally predicts.

Robustness. Calvano et al. (2020) first consider the economic parameters. When a larger number of independent algorithms maximizing their own profits interact in the market equilibrium prices reduce, but they are still significantly higher than the competitive level and supported by collusive strategies. Algorithmic collusion also persists, albeit to a smaller extent when firms differ in terms of cost efficiency and or quality of their product, exactly as theory predicts. Similarly, a smaller discount factor reduces prices and profits, in this case down to the competitive level when the algorithms become dynamically myopic with a nil discount factor. Algorithmic collusion persists also with different degrees of product differentiation and also with perfectly substitutable products. A stochastic demand reduces the ability to coordinate on high prices, but still algorithmic collusion prevails, as well as with a variable market structure where some firms unpredictably enter and exit the market as they would to for example responding to their inventories. Calvano et al. (2021) also show that algorithmic collusion copes with more complex economic environments with imperfect information and imperfect monitoring. In the former case algorithms privately know the costs of their firms but not those of the other firms which may differ. In the latter, instead, algorithms are not able to perfectly observe the prices chosen by the other algorithms as they only observe an imperfect signal. Surprisingly, algorithmic collusion also adapts to these much more complicated environments.

Algorithmic collusion is also robust to changes in the hyper parameters. That is changes in the design of the algorithms. Clearly, too short experimentation inhibits learning with algorithms failing to converge, as with extreme values of the learning rate. However, it is clear that algorithmic collusion is not the product of a fortuitous choice of these parameters and prevails over a very broad range.

Other interesting experiments can then be performed with the flexibility of the simulated virtual market. For example, one can show what happens when a new algorithm enters a market populated by algorithms that have already performed their learning and ended up with algorithmic collusion. The question is whether the new entrant learns to exploit the high prices of the "experienced algorithms" or rather it learns to adapt to their collusive behavior. Interestingly, it is possible to show that what happens is rather the second possibility with the market ending up in a new equilibrium with collusive prices (possibly reduced by the presence of an extra firm).

It is also possible to verify to what extent the learnt collusive strategies are specific to the episodes that the algorithms face in their learning history. This can be done taking algorithms that have performed their learning in different virtual markets and putting them together in the same market. Interestingly, in this case their behavior is clearly perturbed showing price wars for a certain number of initial periods, but very quickly they learn to restore a new collusive equilibrium with high prices supported with the expectation of punishments of deviations.

Policy Issues and Implications.

Collusion is by no means a new phenomenon. Antitrust authorities have been investigating and fighting cartels organized by managers all over the world for more than a century and we only became aware of cartels that get discovered. With algorithmic collusion, there are at least two important novelties. The first is that algorithms' ability to autonomously learn to collude is possible and seems very robust, as discussed above. The second, and probably even more important observation, is that algorithms autonomously learn to collude, without any instruction to do so, and they do it silently, without any form of communication. Since managers' intention to collude and explicit communications have been the key elements to proving unlawful collusion with humans, algorithmic collusion poses a fundamental legal challenge. If authorities discover algorithmic collusion, currently this would not constitute a violation of competition law.

We think that the current state of matters could and probably should be addressed. Here we mention the possible difficulties that will be confronted with.

The type of collusive strategies that algorithms easily learn as discussed above, could be in principle adopted by humans too. In fact, the tacitly agreed reward-punishment scheme discussed above is the typical model of collusion that is taught in economics textbooks as the canonical mechanism for sustaining a collusive agreement, that clearly cannot rely on explicit contractual obligations. A crucial difference relates to the potential for gathering evidence in the two scenarios. With humans there is no way courts and authorities could unveil a tacit collusive agreement, as they cannot read into the managers' minds. This is why the current application of antitrust law is administered on the basis of hard evidence of communication, such as emails and phone calls. With algorithmic collusion, it would instead in principle be possible to document the learnt strategies, performing experiments along the lines of the experiment depicted in figure 2. This is not to say that it is going to be an easy task, as we further discuss in Section 5, but it is at least a potential promising avenue to cope with algorithmic collusion.

Empirical Analysis of Algorithmic Collusion

Despite growing theoretical and experimental evidence that commonly used pricing algorithms can reach tacitly collusive equilibria, a question remains about how real this risk in practice is. The answer will influence the extent to which competition authorities oversee the adoption of these technologies (see for instance the UK Digital Competition Expert Panel 2019 Report pp 109-111). Therefore, empirical work investigating the impact of the adoption of algorithmic-pricing software is essential. However, any empirical analysis must overcome three important challenges. First, adoption decisions are typically not publicly observed. Second, adoption is endogenous because the decision to adopt is correlated with factors that are unobserved to researchers. Finally, even if adoption can be causally linked with higher prices or margins, it is not clear whether these can be attributed to changes in competition intensity rather than to other factors, such as an improved ability to price discriminate. Assad et al. (2020) address these challenges and provide the first empirical analysis of the impact of wide-scale adoption of algorithmic pricing solutions, complementing existing theoretical and experimental works. They take advantage of high-frequency retail gasoline price data from Germany, where advertising by a leading algorithmic software provider, Danish company a2i Systems, suggests that algorithmic pricing software has been widely available for adoption since 2017.

Algorithmic software providers claim that their products can help gasoline station owners "master market volatility with AI-powered precision pricing, respond rapidly to market events and competitor changes" (Kalibrate.com) and take advantage of "superhuman expertise" (a2i.com). Software providers stress the ability of their algorithms to incorporate market conditions and variables such as own and competitor prices, sales volumes, costs and weather and traffic events. For a given station, an algorithm trains based on historical data. It uses these inputs and takes in additional "real-time" information such as current weather and traffic patterns to set prices that maximize station profits. Transactions resulting from these prices are fed back into the algorithm as new inputs.

Although all software providers focus on the speed and responsiveness of their pricing algorithms, the exact specifications of algorithms used in the retail gasoline market are unknown. For example, while most software providers claim to condition on historical own and competitor prices, it is not known how long their algorithms' memories are. 13 Even the type of machine learning used (adaptive vs. reinforcement learning) is mostly obfuscated. References in a recent paper that broadly describes one such algorithm, Derakhshan et al (2016), suggests that they use reinforcement learning techniques that experiment with random actions to learn the state space as in Calvano et al. (2020) and Johnson et al. (2020), but it is not stated explicitly.

Regardless of the type of learning algorithms used in this market, widespread adoption could still facilitate collusive behaviour. The German gasoline retail market is subject to price disclosure regulations and near perfect price transparency. In such an environment algorithms can make deviations from collusive conduct easier to detect and punish and help sustain supra-competitive prices. Advertisements in trade publications also suggest that multiple stations in a single local market could adopt identical pricing software, raising concerns of hub-and-spoke collusion, depending on how individualized the algorithms are for each customer.

Identifying Station-Adoption. 13 Results from the experimental literature suggest memory is short. State spaces become exponentially larger and the price optimization problem becomes increasingly more complex and unstable with longer memory. Calvano et al. (2020) and Johnson et al. (2020) both limit algorithmic memory to one period.

A first challenge is that the decision to adopt algorithmic-pricing software is not directly observed in the data. To identify adopters Assad et al (2020) test for structural breaks in pricing behaviours related to the use of sophisticated pricing software. The software is advertised to "rapidly, continuously, and intelligently react" to market conditions; automatically setting optimal prices in reaction to changes in demand or competitor behaviour; or, to maximize margins without affecting the behaviour of consumers or competitors. Therefore, following adoption stations should make more frequent and smaller price adjustments, and should react more quickly to changes in competitors' prices.

These measures of pricing behaviour line up with what is described in the economic and legal literature discussing algorithmic adoption. Ezrachi and Stucke (2015) point out the ability for algorithmic software to increase the capacity to monitor consumer activities and the speed of reaction to market fluctuations. Mehra (2016) notes the ability of AI pricing agents to more accurately detect changes in competitor behaviour and more quickly update prices accordingly. 14 Assad et al. ( 2020) use a Quandt-Likelihood Ratio test (Quandt 1960), a method standard in the economics literature, to identify possible breaks for each station. To minimize false positives, a station is classified as an algorithmic-pricing adopter if it experiences a structural break in at least two measures within a short time period (taken to be eight weeks, but robust to alternative specifications). A large number of breaks are found in all three measures. For example, many stations go from changing their prices five times per day to ten times per day. Approximately 30% of stations experience structural breaks in more than one of the measures. The majority of these breaks occur in mid-2017, just after algorithmic pricing software becomes widely available, suggesting that the measures capture algorithmic-pricing software adoption.

Identifying Causal Effects of Adoption on Margins and Prices.

Having identified which stations adopt algorithmic pricing software and when, Assad et al. (2020) compare outcomes (margins and prices) of adopting and non-adopting stations. The challenge faced at this stage is that adoption decisions and timing are likely correlated with station/time specific factors unobservable to the researcher. For example, stations that hire better managers could be more likely to adopt the new software, but also different in other dimensions than worse-managed stations, making it difficult to isolate the effects of adoption. A simple OLS regression, even one controlling for a large number of station-and time-specific characteristics, as well as changing local confounders such as weather and demographics, would yield biased and inconsistent estimates of the effect of adoption on outcomes. Assad et al. (2020) address this challenge with an instrumental variable (IV) approach. They find variables (instruments) that shift station incentives to adopt the software independently of their idiosyncratic unobservable characteristics. The instruments allow them to recover the "true" causal effect of adoption on outcomes. The main IV is the adoption decision by a station's brand (i.e., by brand-HQ). 15 As in other cases of corporate technology adoption (e.g., Tucker 2008), technology adoption in retail gasoline happens at two levels: at the brand-HQ level and at the individual station level. Brands make big-picture decisions about the technology they would like their stations to use, and provide stations with employee training, technical support and maintenance and subsidies. Individual station owners make adoption decisions specific to their stations. This involves incurring investment costs such as pump and Point of Sale (PoS) terminal upgrades. The costs can be substantial and are not necessarily fully subsidized by the brand. An example is the 1990s Exxon Mobil (Esso's parent company) brand-wide roll-out of the Mobil Speedpass, a contactless electronic payment system. BusinessWeek reports that to adopt the technology individual station owners had "to install new pumps costing up to $17,000--minus a $1,000 rebate from Mobil for each pump" (BusinessWeek). Partial investment subsidies by brands help explain staggered or delayed technology adoption in this market. Brand-level decisions should not be correlated with individual station-specific unobservables.

Since brand adoption decisions are also unobserved Assad et al. (2020) use a proxy for adoption to instrument: the fraction of a brand's stations that adopt AI pricing. If only a very small fraction of a brand's stations adopts AI, it is unlikely that the brand itself decided to adopt it. If a large fraction adopts, it is likely that the brand itself adopted and facilitated adoption by the stations.

Using brand-adoption as an IV, Assad et al. (2020) examine the effects of adoption on mean monthly prices and margins, as well as on the distribution of prices and margins. They show that mean station-level margins increase by 0.7 Euro cents per litre after adoption. Mean margins for non-adopting stations are approximately 8 Euro cents, so this is a 9% increase in margins. 16 Other moments of the margin distribution also generally increase after adoption. Adoption also causes a 0.5 Euro cents per litre increase in mean prices. There are over 47 million cars registered in Germany (EuroStat). Assuming that each car has an average tank size of 40 litres and fills up once a week, universal adoption of algorithmic pricing software could increase total consumer expenditures on fuel by nearly 500 million Euros per year.

Identifying Effects of Adoption on Competition.

There are many channels, other than through competition, that adoption of algorithmic-pricing software can change margins. For instance, an algorithm can better detect underlying fluctuations in wholesale prices or better predict demand. To isolate the effects of adoption on competition Assad et al. (2020) focus on the role of market structure, comparing adoption effects in monopoly (one station) markets and non-monopoly markets. If adoption does not change competition, effects should be similar for monopolists and non-monopolists. They also perform a more direct test of theoretical predictions by focusing on duopoly (two station) markets. Assad et al. (2020) compare market-level average margins in markets where no stations adopted, markets where one station adopted and markets where both stations adopted. In the first market type, competition is between human price setters. In the second it is between a human price setter and an algorithm, while in the last it is between two algorithms. By comparing all three market types they are able to identify the effect of algorithmic pricing on competition.

Findings in Assad et al. (2020) show that outcomes vary based on market structure. First, adopting stations with no competitors in their ZIP code see no statistically significant change in mean margins, while those with competitors experience an increase of 0.8 cents per litre and a rightward shift in the distribution of their margins. These results suggest that algorithmic pricing software adoption raises margins only through its effects on competition. Second, estimates in duopoly (two station) markets reveal that, relative to markets where no stations adopt, markets where both do experience a mean margin increase of 2.2 cents per litre, or roughly 28%. Markets where only one of the two stations adopts see no change in mean margins or prices. These results show that market-wide algorithmic-pricing adoption raises margins and prices, suggesting that algorithms reduce competition. The magnitudes of margin increases are consistent with previous estimates of the effects of coordination in the retail gasoline market (Clark and Houde 2013, 2014; Byrne and De Roos 2019).

Finally, Assad et al. (2020) explore the mechanism underlying the relationship between algorithmic pricing and competition by asking whether algorithms are unable to learn how to compete effectively, or whether they actively learn how not to compete (i.e., how to tacitly collude). If it is the former, immediate increases in margins should be visible. If it is the latter, algorithms should take longer to train and converge to tacitly-collusive strategies (Calvano et al. 2020). Assad et al (2020) find evidence that margins do not start to increase until about a year after market-wide adoption, suggesting that algorithms in this market learn tacitly-collusive strategies. These findings are in line with simulation results in Calvano et al. (2020).

Policy Issues and Implications.

The findings in Assad et al. (2020) provide the first systematic evidence of the effects of algorithmic pricing software adoption on competition. From the perspective of competition and antitrust authorities, they are troubling. Algorithmic pricing software can learn to coordinate, suggesting that widespread adoption of such software can facilitate tacit collusion and raise prices and markups. To the best of our knowledge, this occurs without explicit communication between competitors, making it legal according to current competition laws in many countries.

While the evidence in Assad et al (2020) is particular to retail gasoline markets in Germany, the same algorithmic pricing software has been adopted in gasoline retail markets around the world. At a minimum, their results suggest that competition authorities in Germany and elsewhere should undertake a census of retail-gasoline pricing software to understand the market structure of the algorithmic software market and the extent of adoption. Such a census can help separate whether the main effect of algorithmic pricing software on competition comes from multiple stations in a market adopting the same or different algorithms. Which algorithm competitors adopt is not directly observed and the two possibilities have different implications for regulators and policy-makers.

Platform Design for Algorithmic Collusion

Online marketplaces such as those operated by Amazon, eBay, and Walmart allow third-party merchants to set the prices of goods that they sell on the marketplace. The potential for collusive merchant behavior exists, and there is concern that the growing prevalence and sophistication of pricing algorithms may facilitate collusion.

What steps, if any, can online retail marketplaces take to fight collusion by third-party merchants and improve competitive outcomes? This is the question posed in Johnson et al. (2020). In that article, which we discuss and summarize here, we seek answers using both economic theory and algorithmic experiments, and use the resulting insights to identify relevant policy issues.

We now sketch the underlying economic scenario we are trying to capture (full details can be found in the above-mentioned article). Imagine a consumer arrives at an online marketplace and types a product descriptor into the search tool. Perhaps she is looking for a certain type of product but is unsure exactly which brands to consider. The platform can influence how many products she considers, and which products those are, in many ways. For instance, the platform controls the ranking of products on the search page and how many are on that page. Additionally, if she clicks on a product to gain more information about it, the platform chooses which additional products to present on that page, and so on. This overall process might be very complicated and so to capture the basic idea we suppose that the platform chooses how many products the consumer considers.

Specifically, there are a total of n differentiated products in a category (we assume a standard logit model of differentiated-product demand), of which k<n are shown to consumers. We consider two policies that determine the identities of these k products. The simpler of the two policies is called Price Directed Prominence (PDP). Under this policy, in each period the platform shows k of the products with the lowest prices. 17 The second policy, Dynamic PDP, is more subtle and is described below.

What happens when a platform steers demand using these policies? We seek to answer this question primarily using experiments on AI algorithms. However, as a preliminary step we use economic theory to frame some of the challenges and potential tradeoffs faced by a platform.

Theory Predictions for Price Directed Prominence in Competitive Markets.

If sellers are not colluding, then individual firms have a strong incentive to cut prices, because the n-k firms with the highest price are not shown to any consumers. Indeed, theory predicts all firms will set prices equal to marginal cost. Consumers benefit from these price decreases but are harmed by the loss of variety presented to them. Therefore, PDP induces a tradeoff for consumers.

We show that this tradeoff between lower prices and less variety benefits consumers as long as consumers are shown enough products, that is, so long as k is not too small (indeed, we find that consumers benefit even if almost two-thirds of firms are not presented to consumers). This simple and intuitive result is nonetheless powerful as it shows that steering techniques that limit consumer choice can nonetheless benefit consumers, at least when the market is competitive.

Theory Predictions for Price Directed Prominence in Cartelized Markets. Now suppose that the n firms in the industry have formed a cartel. We note that there are many prices on which these firms could collude. To draw a sharp contrast with competitive markets, we focus on collusion at the prices that maximize the overall profits of the n cartel members.

In stark contrast to what happens in competitive markets, theory predicts that PDP harms consumers when sellers collude both before and after the implementation of PDP. The reason is that, when the market is cartelized, PDP does not lead to dramatic price decreases. Instead, the cartel finds it optimal to reduce prices slightly, and this is not enough to compensate consumers for the variety loss.

There is a silver lining: showing fewer firms to consumers makes it somewhat harder to sustain a cartel, meaning that some (but likely not all) cartels may no longer be sustainable. To understand why, recall that cartels are sustainable when each firm's short-run gain of deviating from cartel pricing is smaller than the long-run cost to that firm of starting a retaliatory price war. When fewer firms are shown to consumers, the gains to deviating from a cartel are larger because there are fewer alternative options being offered to consumers. This silver lining aside, we also consider a different steering technique, Dynamic PDP, which is tailored to attack the foundations of collusion more directly than PDP. The basic idea of Dynamic PDP is to make it more attractive for a firm to deviate from a cartel. It accomplishes this by making it more difficult for a cartel to punish those who deviate from cartel pricing. Specifically, under Dynamic PDP a firm that cuts its price today is rewarded not only today (by being one of the firms shown to all consumers) but also in future periods. The future benefits come in the form of a "cushion" or "advantage" offered by the platform that makes it easier for that firm to be shown to all consumers in the future even if rivals retaliate with their own price cuts. The net effect is that a firm cutting prices today expects also to be shown to consumers in the future even if rivals undercut it somewhat. In equilibrium, for a properly sized cushion, this logic leads all firms to compete for the cushion and the final effect is a breakdown in collusion. Indeed, theory predicts marginal-cost pricing under Dynamic PDP, even when firms would otherwise form a cartel and even when that cartel would be robust to the simpler PDP technique.

Results of Algorithmic Experiments.

But how do actual AI algorithms behave? To investigate, in Johnson et al. (2020) we perform experiments using the same type of reinforcement-learning (Q-learning) algorithms discussed in Section 2.

Briefly, we run the experiments in the following manner. We specify the same demand system as in our theoretical analysis and then allow our algorithms to interact repeatedly with each other until their learning converges. The algorithms condition on prices from the previous period (in principle they might condition on a longer horizon, but our assumption of a single period keeps the state space to a manageable size).We look at how prices, consumer surplus, and platform profits are affected by implementing the policies of PDP and Dynamic PDP. We separately consider both a low and a high level of product differentiation.

Our experiments reveal that AI algorithms do not always behave as predicted by theory. Overall, however, we find support for the idea that platform-design policies that limit consumer choice can benefit consumers.

In more detail, our first results involve circumstances where firms value the future highly, that is, have high discount factors (theory suggests collusion among economic actors is easiest in this case). Consistent with this, and in line with our predictions, we find that PDP may cause algorithms to lower their prices but that consumers may still be harmed overall due to the loss of variety. However, contrary to our theoretical predictions, we find that when the level of product differentiation is low, PDP lowers prices enough that consumers do benefit.

Although it is encouraging that PDP sometimes benefits consumers when the future is valued highly, the bottom line is that in this case PDP exhibits mixed success in our algorithmic experiments. It seems that the AI algorithms we use are sufficiently flexible in their learning that they are able to maintain very high prices, even when PDP is in place. However, our second policy-design tool, Dynamic PDP, appears to work very well even when the future is valued highly. We find that AI algorithms drop their prices substantially and that consumers benefit across a wide array of parameter values.

We also perform experiments using lower discount factors, corresponding to situations in which the future is not valued as highly. Here we find that PDP by itself can achieve large consumer-surplus gains. This is in line with our theoretical predictions in which consumers typically benefit from PDP if markets are competitive rather than cartelized.

We also use our algorithmic experiments to investigate whether a platform benefits from adopting steering policies that lower prices. This is important because if not then we might instead expect platforms only to adopt harmful policies such as those that tend to display firms with higher prices. Our experiments reveal that platforms can benefit from the policies we consider. Specifically, when a platform receives a per-unit fee from merchants (as, for example, Amazon does when a merchant uses its fulfillment services) then the platform benefits when total sales are higher and we find that our techniques sometimes increase the total units sold. This is more likely when the same steering also benefits consumers, which makes sense: if prices fall enough to benefit consumers despite the variety loss, total demand is typically up.

On the other hand, at least for the parameterizations we consider, the total revenue generated by merchants goes down when we impose our steering policies. Because many platforms receive a share of revenue from their merchants as a fee, at first this suggests that a platform may hesitate to impose such policies. However, by lowering prices across its entire platform, we believe a platform may generate additional total demand; the market size of those who frequent the platform should increase. Our calculations suggest that the platform may often benefit from lower prices when this is true.

Aggregate merchant (supplier) profits decrease under these policies, both due to the fact that prices are falling but also because some consumers are only shown a subset of the available products. Although we do not systematically study the distribution of merchant profits, we can say that sometimes asymmetric outcomes are reached, with one merchant earning more than another. Thus, symmetric learning outcomes are not always reached.

Policy Issues and Implications.

As originally described in more detail in Johnson et al. (2020), several policy implications emerge from that research. First, steering techniques that limit consumer choice can benefit consumers, because of how such techniques influence the strategic decisions of firms. A platform that commits to a policy that limits variety can compel firms to lower their prices, thereby making consumers better off despite diminished variety. To be clear, this means that sometimes a platform does not display a particular product to a consumer even though the consumer would prefer that product to those that they are shown, and yet consumers still benefit. At the same time, this does not imply that limiting choice on its own is certain to benefit consumers-for consumers to benefit it is crucial that such a policy causes firms to make procompetitive decisions that they otherwise would not have taken.

Second, even when algorithmic collusion might otherwise emerge, platforms may have the tools to fight back and destabilize a cartel. However, doing so may require more subtle policies. In our analysis, that more subtle policy is Dynamic PDP, which is specifically designed to influence the intertemporal tradeoff firm face when deciding whether to remain in a cartel or instead act competitively.

Third, when more subtle policies are required, such policies may appear to be non-neutral and yet have positive effects on competition. For instance, under Dynamic PDP, in some periods particular firms receive preferential consideration from the platform. Importantly, however, today's preferential treatment is "earned" in earlier periods by cutting prices in those periods, and so effective policies may still be non-neutral when viewed from a longer-term perspective.

Finally, there has been debate about whether platforms (especially large, dominant platforms) should have a legal duty to promote competition on their marketplaces. But a pertinent question in this debate is whether and how platforms can reasonably achieve this outcome. One implication of our article is that platforms may indeed have some of the tools they need to do this. Moreover, in some cases these tools can be fairly simple and related to steering techniques already employed by most platforms.

Concluding Remarks

Economists' research conducted so far on algorithmic collusion suggests that concern over the use of pricing algorithms may be warranted. As detailed in Section 2, at least in simplified experimental settings algorithms can autonomously learn to tacitly collude, and as detailed in Section 3 there is at least suggestive empirical evidence from the retail gasoline market that the adoption of such algorithms indeed raises prices. Moreover, under current laws algorithmic collusion may not even be illegal. However, as shown in Section 4, research also suggests that it be possible to limit the harm associated with such collusion by changing the rules by which algorithms interact on online marketplaces.

The experimental evidence in Section 4 naturally leads to questions about whether there are broader policy initiatives that might fight any algorithmic collusion. Any such initiatives must be resilient to the fact that AI algorithms might yield many consumer benefits, for example by enhancing allocative or productive efficiency, not merely lead to collusive prices. Heavy handed policies such as banning pricing algorithms may be welfare reducing (as well as nearly un-enforceable).

Given the uncertainty about whether pricing algorithms primarily help or harm consumers, and the very early current stage of research on such questions, we believe it is prudent for regulators to move cautiously, but to continue moving and learning about the diverse uses of pricing algorithms in the (very complex) real world.

A better understanding of the market for pricing software may be extremely valuable to authorities. For example, in a specific market it might be highly informative to know whether most or all industry participants use the pricing algorithm of the same company, and what exactly led them to adopt the same software.

Perhaps most importantly, it would be tremendously valuable to authorities to understand in more detail how different algorithms function. It can often be difficult (especially for outside observers) to understand how algorithms make decisions. Small decisions by the designers of the algorithms, including hyperparameter selection, objective function, and the data on which algorithms are trained, can all have substantial effects on how the algorithms ultimately behave. There is therefore much scope for the exact functioning and intent of an algorithm to be obfuscated.

One intriguing possibility is that regulators could gain access to the underlying algorithms and training data. Such access might allow regulators to gain insights into the design decisions behind specific algorithms, and to experiment to see how they behave in various settings. A challenge for this approach is similar to challenges encountered in our studies of synthetic environments (Calvano et al. 2020 and Johnson et al. 2020). There is no standard "format" by which algorithms operate; instead they are often customized within a specific IT setting and for a particular problem faced by a firm. Outcomes may also vary depending on the specific environment faced by the algorithm (i.e., the pricing strategy and algorithms of their competitors).

We believe that any investigations into algorithmic pricing must acknowledge that pricing algorithms that operate on marketplaces cannot be understood in isolation. Instead, they must be studied jointly along with the rules that the marketplace imposes on the algorithms; these rules themselves are implemented by algorithms, further complicating the situation.18 Financial markets, for example, have their own specific trading rules and, being populated by algorithms that automate trading decisions, are a natural subject for further investigation.

We believe we are in the very early stages of both academic and applied research on pricing algorithms and collusion. Future research, perhaps collaborative research with computer scientists and others, is urgently needed.

8

  See discussion at the session on Machine Learning, Market Structure and Competition at the 2017 NBER Conference on AI: https://www.economicsofai.com/nber-conference-toronto-2017.

Figure 1 .

 1 Figure 1. The distribution of prices charged by reinforcement-learning price algorithms in the virtual market created in Calvano et al. (2020). The price that would maximize the firms' joint profit is just above 1.93. The algorithms routinely learned to collude.

See for example, https://web.archive.org/web/20180819175854/https://www.techemergence.com/ai-for-pricing-comparing-5-curre nt-applications/.3 See also Sam Schechner, "Why Do Gas Station Prices Constantly Change? Blame the Algorithms," The Wall Street Journal, May 8, 2017.2 See https://web.archive.org/web/20201101114000/https://developer.amazonservices.com/

For example, see pp.109-111 of the 2019 CMA Furman Report, "Unlocking Digital Competition, Report of the Digital Competition Expert Panel." Also CMA Research and Analysis Jan. 2021 states "collusion appears an increasingly significant risk if the use of more complex pricing algorithms becomes widespread." https://www.gov.uk/government/publications/algorithms-how-they-can-reduce-competition-and-harm-consumers /algorithms-how-they-can-reduce-competition-and-harm-consumers

Proposal for a Regulation by the Council and the European Parliament introducing a new competition tool, European Commission, ref. Ares (2020) 2877634.6 https://www.ftc.gov/news-events/blogs/business-blog/2020/04/using-artificial-intelligence-algorithms

The assessment of the algorithms is performed after experimentation has been concluded.

Precisely, letting P, Pc, Pm being respectively the average observed profit, the competitive profit and the monopolist's profit, the profit gain is measured as (P-Pc)/(Pm-Pc)*100. These measures would be 0% in case of competitive behavior and 100% in case of monopolistic behavior.

More generally,Calvano et al. (2020) also show that the algorithms learned to play equilibrium strategies. Given the learnt strategy and embedded in other algorithms, a given algorithm learns to set a price that (almost) systematically and perfectly best responds.

[START_REF] Chen | An empirical analysis of algorithmic pricing on Amazon marketplace[END_REF] identify algorithmic pricing users in Amazon Marketplace by measuring the correlation of user pricing with certain target prices, such as the lowest price of a given product in the Marketplace.

Estimates using alternative broadband availability IVs are qualitatively similar to the main estimates but quantitatively larger.15 As a robustness check,[START_REF] Assad | Algorithmic Pricing and Competition: Empirical Evidence from the German Retail Gasoline Market[END_REF] consider an alternative set of instruments: annual measures of local broadband internet availability and quality. Most algorithmic-pricing software are "cloud" based and require constant downloading and uploading of information. Without high speed internet, adoption is not particularly useful. Conditional on local demographic characteristics broadband quality should not depend on station-specific unobservables, but stations are more likely to adopt once their local area has access to reliable high speed internet.

In the article we more generally allow the n-k firms with higher prices to receive some demand rather than none, but the number of consumers who see these products is smaller than the number who see the k lowest-priced products.

Also, consumers may activate to counter algorithmic collusion. Active algorithmic consumers have been investigated in[START_REF] Gal | Algorithmic Consumers[END_REF].

1 Daniel Ershov and Andrew Rhodes would like to acknowledge support from ANR under grant ANR-17-EUR-0010 (Investissements d'Avenir program).


MD Romain Palich
email: romain.palich@aphp.fr

Elisa Teyssou

Sophie Sayon

Basma Abdi

Cathia Soulie

Lise Cuzin

Roland Tubiana

Marc-Antoine Valantin

Luminita Schneider

Sophie Seang

Marc Wirden

Valérie Pourcher

Christine Katlama

Vincent Calvez

Anne-Geneviève Marcelin

Kinetics of archived M184V mutation in treatment-experienced virally suppressed HIV-infected patients

Keywords: M184V, HIV, antiretroviral treatment, HIV reservoir, ultradeep sequencing

Sequencing with ultrasensitive technique showed that the M184V mutation could be progressively cleared in the HIV-DNA reservoir over time, in virally suppressed HIV-patients. However, this mutation was likely to persist in patients with sustained past replication under lamivudine or emtricitabine.

A c c e p t e d M a n u s c r i p t

INTRODUCTION

The M184V resistance-associated mutation (RAM) is selected by lamivudine (3TC) and emtricitabine (FTC), and induces high-level resistance to these two drugs [START_REF] Turner | Multiple effects of the M184V resistance mutation in the reverse transcriptase of human immunodeficiency virus type 1[END_REF]. M184V is rapidly selected in the setting of non-suppressive antiretroviral therapy (ART) and archived in the HIV reservoir [START_REF] Lambotte | The lymphocyte HIV reservoir in patients on longterm HAART is a memory of virus evolution[END_REF]. As lamivudine continues to be used extensively as part of all recommended antiretroviral regimens, M184V is commonly found in patients with virological failure [START_REF] Schuurman | Rapid changes in human immunodeficiency virus type 1 RNA load and appearance of drug-resistant virus populations in persons treated with lamivudine (3TC)[END_REF]. ART guidelines do not recommend the use of drugs impacted by RAMs as they have been shown to be a risk factor for virological failure [START_REF] Delaugerre | Re-occurrence of HIV-1 drug mutations after treatment re-initiation following interruption in patients with multiple treatment failure[END_REF][START_REF] Devereux | Reduction in human immunodeficiency virus type 1 mutations associated with drug resistance after initiating new therapeutic regimens in pretreated patients[END_REF]. Nevertheless, several studies have suggested that 3TC/FTC can retain activity even in the presence of M184V in the past [START_REF] Ciaffi | Boosted protease inhibitor monotherapy versus boosted protease inhibitor plus lamivudine dual therapy as second-line maintenance treatment for HIV-1-infected patients in sub-Saharan Africa (ANRS12 286/MOBIDIP): a multicentre, randomised, parallel, open-label, superiority trial[END_REF][START_REF] Olearo | Impact of the M184V/I Mutation on the Efficacy of Abacavir/Lamivudine/Dolutegravir Therapy in HIV Treatment-Experienced Patients[END_REF][START_REF] Gagliardini | 2021 A c c e p a n u s c r i p t 16 Virological Efficacy and Durability of Lamivudine-Based Dual Antiretroviral Regimens as Maintenance Therapy in Individuals With Suppressed HIV-1 RNA: A Cohort Study[END_REF][START_REF] Rial-Crestelo | Long-term efficacy of dolutegravir plus lamivudine for maintenance of HIV viral suppression in adults with and without historical resistance to lamivudine: Week 96 results of ART-PRO pilot study[END_REF].

Characterizing the viral resistance profile for each individual on ART is a key challenge for lifelong ART management. To detect archived RAMs, HIV-DNA resistance genotype tests can be performed on the blood cells of virally suppressed patients. As past RAMs, documented by RNA genotypes, have not been systematically found on DNA genotypes, researchers have concluded that DNA genotypes are less sensitive in detecting RAMs than cumulative RNA genotypes [START_REF] Delaugerre | Comparison of resistance mutation patterns in historical plasma HIV RNA genotypes with those in current proviral HIV DNA genotypes among extensively treated patients with suppressed replication[END_REF][START_REF] Wirden | Historical HIV-RNA resistance test results are more informative than proviral DNA genotyping in cases of suppressed or residual viraemia[END_REF]. One could also argue that mutations may have been cleared, as suggested by recent works using ultradeep sequencing (UDS) [START_REF] Nouchi | Dynamics of drug resistance-associated mutations in HIV-1 DNA reverse transcriptase sequence during effective ART[END_REF][START_REF] Gantner | Drug resistance and tropism as markers of the dynamics of HIV-1 DNA quasispecies in blood cells of heavily pretreated patients who achieved sustained virological suppression[END_REF]. Given the potential benefits of lamivudine in the ART armamentarium, we aimed to assess whether, after a long period of viral suppression, the M184V mutation present in the past in replicative HIV could be cleared in blood proviral DNA, using Sanger sequencing (SS) and UDS. We also aimed to determine the factors related to the persistence of this mutation.

A c c e p t e d M a n u s c r i p t MATERIAL AND METHODS

Study population

For this observational study, we selected all HIV-1 infected patients in care at Pitié-Salpêtrière Hospital (Paris, France) in 2019, with an HIV-RNA viral load <20 copies/mL for at least 5 years, in whom the M184V mutation had been identified at least once in a previous RNA genotype resistance test.

Data collection

Patients' characteristics (gender, age, birth country) and HIV medical history (transmission factor, Study endpoints were: 1) the proportion of patients with viruses harboring the M184V mutation in proviral DNA from peripheral blood cells from the last sample available in 2019, using SS, and 2) the proportion of M184V+ DRVs detected from blood cells in 2019 and 2016, using UDS (detection threshold: 1%). RAMs in the reverse transcriptase gene, other than M184V, were identified from past HIV-RNA genotypes and assessed by SS on HIV-DNA from 2019, in order to assess whether there was a similar kinetics of the M184V mutation and the other RAMs over time.

Procedures

SS was performed only from all samples of 2019. UDS was performed from samples of 2019 and 2016, if the M184V was not detected by SS from samples of 2019. SS and UDS were performed from frozen whole blood samples according to the ANRS-MIE (French National Agency for Research on AIDS and Emerging Infectious Diseases) consensus using Illumina technology (Illumina, San Diego, California, USA), as previously described [START_REF] Lapointe | HIV drug resistance testing by high-multiplex 'wide' sequencing on the MiSeq instrument[END_REF]. The sequence reads were analyzed with Geneious software v.2021.1.1. For UDS, the minimum coverage was set at 50 and the ambiguity filter at 1%. Cell-associated HIV-1 DNA was quantified from frozen whole blood samples by ultrasensitive realtime PCR (Generic HIV-DNA assay, Biocentric, Bandol, France) as previously described [START_REF] Avettand-Fènoël | LTR real-time PCR for HIV-1 DNA quantitation in blood cells for early diagnosis in infants born to seropositive mothers treated in HAART area (ANRS CO 01)[END_REF].

All HIV-RNA values were quantified using the Cobas AmpliPrep/CobasTaqMan HIV-1 assay (Roche Diagnostics, RischRotkreuz, Switzerland) with a lower detection limit of 20 copies/mL. Ultra-sensitive viral load in the range of 1-20 copies/ml was indicated qualitatively (presence or absence of a detectable signal) [START_REF] Lambert-Niclot | Ultrasensitive Human Immunodeficiency Virus Type 1 Viral Load as a Marker of Treatment Choice for Simplification Strategies[END_REF]. All reported values are medians with interquartile range (IQR) for continuous variables and frequencies for categorical variables. We defined two groups of patients according to M184V mutation detection in HIV-DNA in 2019 using SS and UDS: the "M184V+" group (M184V detected by Sanger sequencing or as a DRV >1% by UDS) and the "M184V-" group (M184V undetected by Sanger sequencing and UDS). Uni-and multivariable logistic regressions were used to analyze factors associated with M184V persistence. The multivariable model was constructed by step-by-step descending, initially including all characteristics related to M184V persistence and subsequently removing those with a p-value greater than 0.05. We also assessed the potential correlation between the proportion of M184V+ variants detected from samples of 2019 and the time from the last M184V detection in HIV-RNA, using a Pearson's test.

Ethics

All patients are routinely followed using the Nadis ® electronic medical record [START_REF] Pugliese | A large French prospective cohort of HIV-infected patients: the Nadis Cohort[END_REF], and they provided signed consent ("French National Information Technology and Civil Liberties Board, CNIL" registration number: 770134) for information on record to be used. All data were anonymized before analysis. Patients were systematically notified of any supplementary biological analyses on frozen samples, initially collected as part of routine clinical practice.

RESULTS

Study population

A total of 110 patients fulfilling the inclusion criteria were identified (Figure 1). The median time since detection of the M184V mutation in RNA genotypes was 9.1 years (IQR 8.1-11.1). The reverse transcriptase gene was successfully amplified from peripheral HIV-DNA in 79/110 (72%) patients.

Patients' characteristics are shown in Table 1. Their median duration on ART was 23.8 years (IQR 20.1-26.8) with a median viral suppression period of 8.9 years (IQR 6.8-10.7). Ongoing ART included 3TC/FTC in 39% of cases. Median level of total HIV-DNA was 3.44 copies/10 6 cells (IQR 3.10-3.70).

M184V detection by Sanger and ultradeep sequencing

The M184V mutation was detected in HIV-DNA from blood cells in 53/79 (67%) patients, including 42 using SS and 11 using UDS (Figure 1). In the remaining 26/79 (33%) patients, the M184V mutation was not detected, either by SS or by UDS. In the 37 patients sampled by UDS from samples of 2019, M184V+ DRVs were detected in the range of 1-25%. The proportion of patients with DRVs between 5-15% was 16% (6/37), and ≥15-25% was 14% (5/37). Of note, no patient had M184V+ DRVs between 1-5%. Between 2016 and 2019, the proportion of viral variants carrying the M184V mutation decreased from 40% in 2016 to 14% in 2019 (mean difference: -18.5%, 95%CI -31.0 to -6.0, p=0.005) (Figure 2). From samples of 2016, the proportion of patients with M184V+ DRVs <1% was 56% (19/33), between 1-5% was 6% (2/33), between 5-15% was 0% (0/33) and ≥15-25% was 36% (12/33). (IQR 0-5) in HIV-DNA (mean difference: -1.2, 95%CI -2.0 to -0.3, p=0.007), using SS.

Factors associated with M184V mutation persistence

In univariate analysis, patients with persistent M184V mutation had a longer history of both HIV infection and ART, a lower CD4 nadir and higher pretherapeutic HIV-RNA (Table 2). The duration and level of plasma HIV-RNA replication under 3TC or FTC were higher in patients with persistent M184V mutation. Duration of viral suppression, quantification of total HIV-DNA in blood cells, composition of ART and proportion of patients with residual viremia did not differ between the two groups.

There was a median difference of one RAM (except M184V) between cumulated HIV-RNA and HIV-DNA genotypes in both groups.

In multivariable analysis, the duration of viral replication and the level of HIV-RNA while taking 3TC or FTC were significantly associated with persistence of the M184V mutation, with odds ratios of 1.03 per month of replication (95%CI 1.02 to 1.05, p=0.0009) and 3.17 per log 10 copies/mL (95%CI 1.61 to 7.02, p=0.009), respectively (Table 2).

There was no correlation between the proportion of M184V+ variants detected from samples of 2019 and the time from the last M184V detection in HIV-RNA (ρ=0.04, 95%CI -0.29 to +0.36, p=0.81).

A c c e p t e d M a n u s c r i p t 10 DISCUSSION Given the importance of lamivudine and emtricitabine in all current antiretroviral strategies, determining whether M184V persists in the HIV reservoir in long-term virally suppressed patients is key for the choice of best therapeutic options. In this study, we found that while the M184V mutation persisted in 67% of tested patients over 5 years, clearance of the mutation was observed in 33% of patients. This high rate of M184V persistence can be partially explained by the restrictive definition of persistence, using a detection threshold of 1%. UDS allowed the proportion of detected M184V mutations to rise in a substantial number of patients, as previously described, whereas SS failed to detect viral variants present in less than 15-25% of the total viral population [START_REF] Tzou | Comparison of an In Vitro Diagnostic Next-Generation Sequencing Assay with Sanger Sequencing for HIV-1 Genotypic Resistance Testing[END_REF][START_REF] Ávila-Ríos | Next-Generation Sequencing for HIV Drug Resistance Testing: Laboratory, Clinical, and Implementation Considerations[END_REF].

The relevance of detecting DRVs for predicting virological failure is controversial [START_REF] Stella-Ascariz | The Role of HIV-1 Drug-Resistant Minority Variants in Treatment Failure[END_REF]. It has been shown to be clinically meaningful in three settings: detection of NNRTI-resistant minority variants prior to initiation of a first-line NNRTI-based regimen [START_REF] Li | Low-frequency HIV-1 drug resistance mutations and risk of NNRTI-based antiretroviral treatment failure: a systematic review and pooled analysis[END_REF], detection of NNRTI-resistant minority variants after exposure to a single dose of nevirapine [START_REF] Boltz | Role of low-frequency HIV-1 variants in failure of nevirapine-containing antiviral therapy in women previously exposed to single-dose nevirapine[END_REF] and detection of CXCR4-using variants prior to initiation of maraviroc-containing regimens [START_REF] Kagan | A genotypic test for HIV-1 tropism combining Sanger sequencing with ultradeep sequencing predicts virologic response in treatment-experienced patients[END_REF], with debated thresholds for detection (1-5%). The data in the literature are not all convergent concerning the most suitable threshold for detecting DRVs with UDS. The risk in lowering this threshold is to reflect sequencing errors.

However, several authors have shown that a detection threshold of 1% was relevant to improve the sensitivity of genotyping [START_REF] Mohamed | s c r i p t 18 sequencing detection of minority mutations on the HIV-1 drug resistance interpretations after virological failure[END_REF][START_REF] Rodriguez | Ultra-deep sequencing improves the detection of drug resistance in cellular DNA from HIV-infected patients on ART with suppressed viraemia[END_REF], which explains our choice for this work. We did not find any patient with M184V+ DRVs between 1-5% from samples we studied. Therefore, choosing a detection threshold of 5% would not have changed the result of our analysis.

Previous works have suggested poor sensitivity of Sanger sequencing for detecting archived RAMs in blood proviral DNA in comparison with past RNA genotypes [START_REF] Delaugerre | Comparison of resistance mutation patterns in historical plasma HIV RNA genotypes with those in current proviral HIV DNA genotypes among extensively treated patients with suppressed replication[END_REF][START_REF] Wirden | Historical HIV-RNA resistance test results are more informative than proviral DNA genotyping in cases of suppressed or residual viraemia[END_REF]. Two main hypothesizes could explain the lack of sensitivity of DNA genotyping: difficulty extracting and amplifying the genetic Consequently, it has been suggested that DNA genotype testing be performed in duplicate or triplicate, which has its limitations, however, in terms of feasibility and costs. Like others, we assume that UDS can improve the sensitivity of standard DNA genotyping [START_REF] Alidjinou | Routine drug resistance testing in HIV-1 proviral DNA, using an automated next-generation sequencing assay[END_REF], although limited selection of amplified archived variants cannot be excluded.

Sustained quantitative and qualitative alterations occurred in virally suppressed patients. Total HIV-DNA from blood, used as a surrogate for the HIV reservoir [START_REF] Avettand-Fènoël | Total HIV-1 DNA, a Marker of Viral Reservoir Dynamics with Clinical Implications[END_REF], sharply decreases in the first years on ART, and then much more slowly but continuously [START_REF] Golob | HIV DNA levels and decay in a cohort of 111 long-term virally suppressed patients[END_REF][START_REF] Bachmann | Determinants of HIV-1 reservoir size and long-term dynamics during suppressive ART[END_REF]. A recent study showed that intact proviral DNA (i.e. competent for replication) continues to decline between 7 and 11 years after ART initiation in virally suppressed patients [START_REF] Gandhi | Intact proviral DNA levels decline in people with HIV on antiretroviral therapy[END_REF]. Furthermore, different genetic mechanisms impairing the integrity of proviral DNA, including inversions, large deletions and hypermutations G to A by the APOBEC pathway, have been reported [START_REF] Bachmann | Determinants of HIV-1 reservoir size and long-term dynamics during suppressive ART[END_REF]. It is estimated that intact proviral DNA constitutes <10% of total proviral DNA in chronic virally suppressed patients on ART [START_REF] Bruner | Defective proviruses rapidly accumulate during acute HIV-1 infection[END_REF][START_REF] Imamichi | Defective HIV-1 proviruses produce novel protein-coding RNA species in HIV-infected patients on combination antiretroviral therapy[END_REF][START_REF] Ho | Replication-competent noninduced proviruses in the latent reservoir increase barrier to HIV-1 cure[END_REF].

Despite the presence of long-lived HIV-infected lymphocytes potentially harboring viral mutations and their maintenance due to persistent very low-level replication [START_REF] Siliciano | Long-term follow-up studies confirm the stability of the latent reservoir for HIV-1 in resting CD4+ T cells[END_REF][START_REF] Chomont | HIV reservoir size and persistence are driven by T cell survival and homeostatic proliferation[END_REF][START_REF] Lorenzo-Redondo | Persistent HIV-1 replication maintains the tissue reservoir during therapy[END_REF][START_REF] Palmer | Low-level viremia persists for at least 7 years in patients on suppressive antiretroviral therapy[END_REF], recent data argue for real clearance of RAMs in proviral DNA. In a pilot study, Nouchi et al. showed a progressive decrease in the proportion of NRTI-and NNRTI-resistant minority variants over five years [START_REF] Nouchi | Dynamics of drug resistance-associated mutations in HIV-1 DNA reverse transcriptase sequence during effective ART[END_REF] in patients virally suppressed for at least 5 years, on non NRTI or NNRTI-containing ART regimens. In patients with significant virological failures in the past, Gantner, et al. reported that NRTI-, NNRTI-and PIresistant minority variants were no longer detected in PBMC over time if the ART regimen excluded these drugs [START_REF] Gantner | Drug resistance and tropism as markers of the dynamics of HIV-1 DNA quasispecies in blood cells of heavily pretreated patients who achieved sustained virological suppression[END_REF]. Our study supports the progressive decrease in the M184V mutation over time, with a kinetic not affected by the therapeutic pressure of 3TC/FTC. Eleven out of 26 (42%) patients with cleared M184V were receiving 3TC/FTC at the time of analysis. One of the unresolved questions In addition to this finding on the clearance of RAMs, we found two factors -the duration of viral replication on 3TC/FTC and the level of replicative HIV-RNA -to be associated with M184V mutation persistence. This suggests that sustained seeding of the HIV reservoir with proviruses carrying RAMs leads to sustained persistence of RAMs over time, which corroborates data from Verhofstede, et al [START_REF] Verhofstede | Drug-resistant variants that evolve during nonsuppressive therapy persist in HIV-1-infected peripheral blood mononuclear cells after long-term highly active antiretroviral therapy[END_REF]. In this study, the proportion of resistant variants was correlated with the duration for which resistant variants had been able to replicate. Interestingly, we did not find any impact of either residual viremia or level of blood total HIV-DNA viral load, even though our patients had higher total HIV-DNA than in other studies [START_REF] Rodriguez | Ultra-deep sequencing improves the detection of drug resistance in cellular DNA from HIV-infected patients on ART with suppressed viraemia[END_REF][START_REF] Cuzin | Levels of intracellular HIV-DNA in patients with suppressive antiretroviral therapy[END_REF], which may be due to long and deep seeding of the HIV reservoir in these patients treated over 20 years. Aging patients with a long HIV history are those who could benefit the most from drug-reduced ART, in terms of preventing drug-drug interactions and long-term cumulative toxicities [START_REF] Katlama | Individualized antiretroviral therapeutic approaches: less can be more[END_REF]. Unfortunately, these are the patients in whom the M184V mutation could persist the longest.

One limitation of our work is the fact that we did not evaluate the non-circulating HIV reservoir, including the lymph nodes and gut. Although total blood HIV-DNA could reflect the overall level of the HIV reservoir [START_REF] Avettand-Fènoël | Total HIV-1 DNA, a Marker of Viral Reservoir Dynamics with Clinical Implications[END_REF], genetic evolution of viral sequences could be dissociated in these different compartments [START_REF] Lorenzo-Redondo | Persistent HIV-1 replication maintains the tissue reservoir during therapy[END_REF]. The study of the extra-circulating reservoir requires invasive techniques that are very difficult to implement in clinical practice. Another limitation could be not having performed UDS in patients who harbored the M184V using SS. We assumed that M184V would also be detected in UDS, which however we did not verify. We did not evaluate APOBEC editing and hypermutations from UDS data, because we amplified only a small segment of the RT gene, on either side of the position 184, not allowing this to be analyzed on a sufficient portion of HIV-DNA.

Better knowledge of the profile of patients with M184V clearance could make it possible to select candidates for clinical trials assessing the efficacy of antiretroviral regimens including 3TC, despite A c c e p t e d M a n u s c r i p t 13 previous virological failures under NRTIs. This question is crucial for dual therapies including 3TC, to accompany boosted darunavir or dolutegravir. The MOBIDIP study and its associated in-depth virological analysis suggested that an archived M184V mutation may not affect the viral efficacy of the darunavir/ritonavir/lamivudine two-drug therapy [START_REF] Ciaffi | Boosted protease inhibitor monotherapy versus boosted protease inhibitor plus lamivudine dual therapy as second-line maintenance treatment for HIV-1-infected patients in sub-Saharan Africa (ANRS12 286/MOBIDIP): a multicentre, randomised, parallel, open-label, superiority trial[END_REF][START_REF] Delaugerre | Deep sequencing analysis of M184V/I mutation at the switch and at the time of virological failure of boosted protease inhibitor plus lamivudine or boosted protease inhibitor maintenance strategy (substudy of the ANRS-MOBIDIP trial)[END_REF]. However, data are much less robust for the dolutegravir/lamivudine two-drug therapy, with a very limited number of patients included [START_REF] Gagliardini | 2021 A c c e p a n u s c r i p t 16 Virological Efficacy and Durability of Lamivudine-Based Dual Antiretroviral Regimens as Maintenance Therapy in Individuals With Suppressed HIV-1 RNA: A Cohort Study[END_REF][START_REF] Rial-Crestelo | Long-term efficacy of dolutegravir plus lamivudine for maintenance of HIV viral suppression in adults with and without historical resistance to lamivudine: Week 96 results of ART-PRO pilot study[END_REF]42].

In conclusion, our findings provide new data on the persistence and progressive clearance of the M184V mutation in the HIV reservoir. Recycling drugs with viral activity potentially impaired by past resistance is now a key challenge in ART management. Assessing the benefit of proviral DNA genotyping in a randomized trial could provide a robust answer to this question.

  time since HIV diagnosis, CDC stage C, CD4 nadir, pretherapeutic HIV-RNA, HIV sub-type, ART duration, number and duration of previous ART regimens, and level of HIV-RNA replication while receiving 3TC/FTC, CD4 count and CD4/CD8 ratio, duration of viral suppression, ongoing ART) were collected at the time of the last available blood sample in 2019. Past HIV-RNA resistance genotypes were collected and reinterpreted using the latest version of the ANRS algorithm (www.hivfrenchressistance.org). Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021 A c c e p t e d M a n u s c r i p t 6 Study endpoints

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021A c c e p t e d M a n u s c r i p t 8

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021A c c e p t e d M a n u s c r i p t 11 material from whole blood with the risk of selecting a small part of the circulating reservoir, and insufficient capture of resistant archived variants, potentially diluted into a large library of viral variants, integrated in the mononuclear cells and including a majority of wild type variants.

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021 A c c e p t e d M a n u s c r i p t 12 is the impact of persistent RAMs, given that they are carried by a large majority of defective proviruses.

A c c e p t e d M a n u s c r i p tFigure 1 .

 1 Figure 1. Study flowchart.

Figure 2 .

 2 Figure 2. Proportion of drug-resistant viral variants (DRVs) carrying the M184V mutation in blood

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021 RAMs other than M184V were more frequently detected in cumulative past RNA genotypes, in comparison with recent DNA genotype, with 4 RAMs in median (IQR 2-6) detected in HIV-RNA and 2

	M a n u s c r i p t
	A c c e p t e d
	9

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

	FOOTNOTE PAGE Last CD4, cells/mm 3 , median (IQR) Last CD4/CD8 ratio, median (IQR) Duration of viral suppression, years, median (IQR) M a n u s c r i p t 614 (453-866) 0.90 (0.56-1.18) 8.9 (6.8-10.7)	623 (503-866) 0.83 (0.65-1.23) 8.4 (6.0-10.0)	592 (380-877) 0.94 (0.56-1.19) 9.0 (7.3-10.9)
	M HIV-RNA <50 cp/mL but detectable by RT-PCR, n(%) A c c e p t e d HIV-DNA, log 10 cp/10 6 cells, median (IQR) Number of drugs in ongoing ART, n(%) a n u s c r i p t 26 (33) 3.44 (3.10-3.70) 1 2 3 7 (9) 35 (44) 33 (42) 4 4 (5)	8 (31) 3.40 (2.90-3.70) 3 (12) 11 (42) 11 (42) 1 (4)	18 (34) 3.49 (3.15-3.70) 4 (8) 24 (45) 22 (41) 3 (6)
	3TC/FTC as part of ongoing ART, n(%)	31 (39)
	A c c e p t e d			22
		14

Table 2 . Uni and multi-variable logistic regression of patients' characteristics in relation to the presence of M184V in 2019.

 2 Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

	Univariable		Multivariable
	OR (95%CI)	p-value	OR (95%CI)	p-value

  Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021 Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021 Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

	Figure 1 Figure 2
	M a n u s c r i p t M a n u s c r i p t
	A c c e p t e d A c c e p t e d
	28 29

(42) 20[START_REF] Verhofstede | Drug-resistant variants that evolve during nonsuppressive therapy persist in HIV-1-infected peripheral blood mononuclear cells after long-term highly active antiretroviral therapy[END_REF] Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

Funding

This study was supported by ViiV Healthcare and the ANRS-MIE, AC43 (French National Agency for Research on AIDS and Emerging Infectious Diseases).

Transparency declaration

RP, BA, CS, LC, RT, MAV, LS, SS, MW, VP, CK, VC and AGM have received travel grants and honoraria from Gilead, ViiV Healthcare, Janssen and Merck. ET and SS have no conflict of interest to declare. Downloaded from https://academic.oup.com/jid/advance-article/doi/10.1093/infdis/jiab413/6355381 by guest on 22 August 2021

//academic.oup.