Jérôme Bouchet

Iratxe Del Río--Iñiguez

Andrés Alcover
email: andres.alcover@pasteur.fr

Imaging vesicular traffic at the immune synapse

Keywords: 'immunological synapse', 'vesicle traffic', 'endosomes', 'cytoskeleton', 'T cell activation', 'TCR signaling

Immunological synapse formation is the result of a profound T cell polarization process that involves the coordinated action of the actin and microtubule cytoskeleton, as well as intracellular vesicle traffic. Endosomal vesicle traffic ensures the targeting of the T cell receptor (TCR) and various signaling molecules to the synapse, being necessary for the generation of signaling complexes downstream of the TCR. Here we describe the microscopy imaging methods that we currently use to unveil how TCR and signaling molecules are associated to endosomal compartments and deliver their cargo to the immunological synapse.

Introduction

Immunological synapses are characterized by the accumulation and clustering of TCRs, co-signaling receptors like CD4 and CD28, inhibitory receptors like CTLA4 and PD1, adhesion molecules like LFA--1, and a number of signaling molecules of the TCR signaling cascade, including the tyrosine kinases Lck and ZAP70, the adapter molecules LAT and SLP76, and the serine--threonine kinases PKCθ and Erk1/2. Dynamic clustering of these various molecules ensures TCR signal regulation, long term T 2 cell activation and effector functions, like cytokine production or the expression of surface ligands by T helper cells, or the release of cytotoxic granule content by cytotoxic T cells (1,2). In order to accumulate and cluster at the immunological synapse, those molecules need to be transported to and/or retained at the T cell--antigen presenting cell contact site. Two main mechanisms have been proposed to transport proteins to the synapse: cytoskeleton--based movement at the plasma membrane (3) and transport via intracellular vesicles (4). While the former mechanism was shown to transport the LFA--1 integrin, the latter was shown to transport the TCR--CD3 complex subunits (5--7), the tyrosine kinase Lck (8,[START_REF] Thoulouze | Human immunodeficiency virus type--1 infection impairs the formation of the immunological synapse[END_REF], the adapter LAT [START_REF] Bonello | Dynamic recruitment of the adaptor protein LAT: LAT exists in two distinct intracellular pools and controls its own recruitment[END_REF][START_REF] Larghi | VAMP7 controls T cell activation by regulating the recruitment and phosphorylation of vesicular Lat at TCR--activation sites[END_REF] and the inhibitory receptor CTLA4 [START_REF] Linsley | Intracellular trafficking of CTLA--4 and focal localization towards sites of TCR engagement[END_REF].

Different endosomal compartments and regulatory transport proteins are involved. These different endosomal compartments appear intermingled in the T cell cytoplasm, but they are distinct with regard to endosomal markers and are differentially regulated when targeting TCR and signaling molecules to the synapse (4,[START_REF] Soares | Regulated vesicle fusion generates signaling nanoterritories that control T cell activation at the immunological synapse[END_REF]. Thus, the TCR--CD3 complex was shown to be transported via transferrin receptor positive recycling endosomes (5), while the TCRζ subunit is associated to Rab8 and intraflagellar transport (IFTs) protein--regulated compartments (7,[START_REF] Soares | Regulated vesicle fusion generates signaling nanoterritories that control T cell activation at the immunological synapse[END_REF][START_REF] Finetti | The small GTPase Rab8 interacts with VAMP--3 to regulate the delivery of recycling T--cell receptors to the immune synapse[END_REF]. Moreover, Lck traffic is regulated by MAL [START_REF] Anton | An essential role for the MAL protein in targeting Lck to the plasma membrane of human T lymphocytes[END_REF][START_REF] Anton | MAL Protein Controls Protein Sorting at the Supramolecular Activation Cluster of Human T Lymphocytes[END_REF], Unc119 [START_REF] Gorska | Uncoordinated 119 protein controls trafficking of Lck via the Rab11 endosome and is critical for immunological synapse formation[END_REF], and Rab11 and its effector FIP3 [START_REF] Soares | Regulated vesicle fusion generates signaling nanoterritories that control T cell activation at the immunological synapse[END_REF] (Bouchet et al, submitted). Finally, LAT was found associated to Rab7, Rab27 and Rab37 endosomes and its targeting to the synapse regulated by the SNARE VAMP7, the calcium sensor synaptotagmin--7 and IFT20 [START_REF] Larghi | VAMP7 controls T cell activation by regulating the recruitment and phosphorylation of vesicular Lat at TCR--activation sites[END_REF][START_REF] Soares | Regulated vesicle fusion generates signaling nanoterritories that control T cell activation at the immunological synapse[END_REF][START_REF] Purbhoo | The function of sub--synaptic vesicles during T--cell activation[END_REF][START_REF] Vivar | IFT20 controls LAT recruitment to the immune synapse and T--cell activation in vivo[END_REF]. Intracellular vesicle traffic and actin and microtubule cytoskeleton rearrangements are tightly interconnected processes that ensure immunological synapse architecture, molecular clustering, signaling complex dynamics and ultimately the regulation of T cell activation (4,[START_REF] Lasserre | Ezrin tunes T--cell activation by controlling Dlg1 and microtubule positioning at the immunological synapse[END_REF][START_REF] Bouchet | The membrane--microfilament linker ezrin is involved in the formation of the immunological synapse and in T cell activation[END_REF].

Interestingly, the subcellular localization and activity of the GTPase Rac1, a master regulator of the actin cytoskeleton crucial for immunological synapse formation and function, is regulated by Rab11 and its effector FIP3 [START_REF] Bouchet | The membrane--microfilament linker ezrin is involved in the formation of the immunological synapse and in T cell activation[END_REF].

Here we describe the methods we have been developing to localize TCR and signaling molecules together with endosomal regulatory proteins at the immunological synapse and to decipher the functional importance of this endosomal transport.

Materials

Cells

1. Jurkat T cell leukemia cells, J77 Clone 20 cells (J77cl20) and Raji B cell lymphoma cells have been previously described (5,[START_REF] Thoulouze | Human immunodeficiency virus type--1 infection impairs the formation of the immunological synapse[END_REF]22). Cells are cultured in RPMI 1640 + GlutaMAX™ + Phenol Red medium (Gibco®, Life Technologies™, No. 61870--010) supplemented with 10% fetal calf serum and 10 mM Hepes. Jurkat and Raji cells are cultured at a density average of 0.5--1 x 10 6 cells/mL, splitting the cultures every 2--3 days. gift of M. Bornens (Institut Curie, France) is used at 1/400 dilution. Mouse IgG2b anti--β--tubulin, clone KMX1 (Millipore) is used at 10 µg/mL. Rabbit anti--phospho ZAP70 (Y319) (Cell Signaling) is used at 1/100 dilution and mouse monoclonal IgG2a anti--phosphoTCRζ (Y142), clone K25--407.69 (Becton Dickinson), are used at 5 µg/mL. 9. Secondary antibodies for immunofluorescence: highly cross--adsorbed Cy3--coupled goat antimouse IgG2a, anti--mouse IgG2b and anti--rabbit IgG (Jackson Immuno--Research Laboratories) are used at 1/100 dilution. FITC--coupled goat anti--mouse IgG1 (Southern Biotech) is used at 0.7 µg/mL. Alexa Fluor 488-coupled goat anti--fluorescein (Molecular Probes) is used at 1 µg/mL. 10. Antibody used for pseudo--synapses: mouse IgG1 anti--CD3ε, clone UCHT1 (Bio Legend) is used at 500 ng/mL. 11. Expression vectors: pEGFP--C1/FIP3 and FIP3--I738E mutant expression vectors [START_REF] Horgan | Tumor susceptibility gene 101 (TSG101) is a novel binding--partner for the class II Rab11--FIPs[END_REF]. pEGFPN1/Rab11 WT, Rab11--Q70L and Rab11--S25N mutants provided by Dr. A. Echard (Institut Pasteur, Paris). pEF--BOS/Rac1--cMyc WT, Rac1--G12V and Rac1--T17N mutants (Komuro et al, 1996).

12. siRNA oligonucleotides: We provide here an example of our recent work [START_REF] Bouchet | The membrane--microfilament linker ezrin is involved in the formation of the immunological synapse and in T cell activation[END_REF]. siRNA duplexes based on the human RAB11--FIP3 sequence had been described previously by others: (5ʹ′--AAGGGATCACAGCCATCAGAA--3ʹ′) [START_REF] Jing | FIP1/RCP binding to Golgin--97 regulates retrograde transport from recycling endosomes to the trans--Golgi network[END_REF] and (5ʹ′--AAGGCAGTGAGGCGGAGCTGTT--3ʹ′) [START_REF] Wilson | The FIP3--Rab11 protein complex regulates recycling endosome targeting to the cleavage furrow during late cytokinesis[END_REF]. Stock solutions are stored in aliquots at --20°C and used at 1 nM per transfection [START_REF] Bouchet | The membrane--microfilament linker ezrin is involved in the formation of the immunological synapse and in T cell activation[END_REF]

Methods:

T cell transfection by electroporation

For expression vectors transfection, cell synapses assays are done 24 h after T cell transfection. For siRNA treatment of T cells, two sequential transfections at 24 h interval are performed, and cell synapses are performed 72 h after the first siRNA transfection. In case of DNA plasmid transfection on siRNA--transfected cells, two sequential transfections at 24 h interval are performed for the siRNA, followed by DNA plasmid transfection 48 h after the first siRNA transfection. Synapses in this case are performed 24 h after transfection of expression vectors (see Note 7).

Transfection of T cell lines using the Neon™ Transfection System

Transfection is done with 5--10 x 10 6 cells. Use 10 µg of DNA plasmid or 1 nM of siRNA for 10 x 10 6 cells. Complete with Resuspension Buffer R to equilibrate different volumes for the same amount of DNA or siRNA when preparing Eppendorf tubes with plasmid DNA or siRNA.

1. Pre--incubate at 37°C one flask with 10 mL T cell culture medium (RPMI 1640 supplemented with 10% FCS and 10 mM Hepes) for each transfection. Prepare Eppendorf tubes with DNA plasmid or siRNA.

2. Harvest the amount of cells required, centrifuge at 290 xg, at 20°C for 4 min. Wash twice in PBS and aspirate supernatant.

Transfection of purified primary CD4 + T cells using Nucleofector™ 2b Device

Transfection is done with 5--10 x 10 6 cells. Use 5 µg of DNA plasmid or 30--300 nM of siRNA for 10 x 10 6 cells. Complete with Resuspension Buffer R to equilibrate different volumes for the same amount of DNA or siRNA when preparing Eppendorf tubes with plasmid DNA or siRNA.

1. Pre--incubate at 37°C one flask with 1 mL Primary CD4 + T cell culture medium (RPMI 1614 medium supplemented with 10% FCS, 1 mM sodium pyruvate and 1% MEM Non--essential aminoacids) per 10x10 6 cells transfected. Prepare 1.5 mL Eppendorf tubes with DNA plasmid or siRNA.

2. Harvest the amount of primary CD4 + T cells required, centrifuge at 453 xg at 20°C for 10 min. Wash twice in PBS.

Turn on Nucleofector™ 2b Device and select electroporation protocol (U014).

4. Add corresponding volume of DNA plasmid or siRNA to the Amaxa cuvette per transfection required.

5. Resuspend the cell pellet in 100 µL Amaxa buffer per 10 x 10 6 CD4 + cells. 6. Take 100 µL of resuspended cells and add them to the cuvette with DNA plasmid or siRNA. Mix gently, avoiding air bubbles.

7. Insert cuvette in the Nucleofector™ 2b Device and electroporate the cells using protocol U014.

8. Use single--use pipettes to recover cells and transfer to the pre--warmed flask with medium.

Incubate 10 min at 37°C. 9. Count cells and resuspend to a final concentration of 2 x 10 6 cells/mL in RPMI 1614 medium supplemented with 10% FCS, 1 mM sodium pyruvate and 1% MEM Non--essential aminoacids at 37°C, 5% CO 2 for 24--48 h (see Note 8). 13. Microscopy analysis: Cells are observed under a LSM 700 confocal microscope (Zeiss) equipped with an oil--immersion Plan--Apochromat 63× objective. Z--stack optical sections are acquired at different increments depending on image analysis performed afterwards: 0.2 μm depth increments for deconvolution and colocalization analysis and 1 μm depth increments for fluorescence intensity analysis. Green and red laser excitations are intercalated to minimize fluorescence spill over different channels. Image acquisition is done with ZEN software (Zeiss).

Immunological synapse formation between T cells and antigen presenting cells

Pseudosynapse formation on anti--CD3--coated coverslips.

1. Coating coverslips with anti--CD3 antibodies. Prepare 10 µg/mL anti--CD3 (UCHT--1) in PBS. Round poly--L--Lysine coated coverslips are turned over 50 µL drops of anti--CD3 solution and incubated 2 h at 37°C in an humidified chamber or overnight at 4°C (see Note 13). Coverslips are washed once in PBS and saturated with 200 µL RPMI 1640 with serum to prevent nonspecific binding to poly--L--Lysine. 13. Microscopy analysis. Cells are observed under a LSM 700 confocal microscope (Zeiss) equipped with an oil--immersion Plan--Apochromat 63× objective). Z--stack optical sections are acquired at different increments depending on desired image analysis: 0.2 μm depth increments for deconvolution and colocalization analysis and 1 μm depth increments for fluorescence intensity analysis. Green and red laser excitation are intercalated to minimize fluorescence spill over different channels. Images acquisition is done with ZEN software (Zeiss).

Staining of microtubules

1. Immunological synapses or pseudo--synapses are performed as in 3.2 and 3.3 until the fixation step.

2. For microtubules staining, 8% paraformaldehyde is mildly dropped onto the coverslips with cells and incubated for only 10 min at room temperature. Directly after incubation, coverslips are placed in 6-- or 24--well plates with ice--cold methanol and stored at --20°C for at least 1 h for further fixation and permeabilization. Before staining coverslips are washed once with PBS--BSA.

3. The remaining steps are performed as in 3.2 for immunological synapses with antigen presenting cells, or 3.3 for pseudo--synapses (Figure 1).

Quantitative image analysis

1. Deconvolution of confocal images is used to improve image rendering, especially for 3D reconstruction and prior to colocalization analysis. Deconvolution is performed on Z--stacks of confocal optical sections obtained at 0.2 µm depth increments, using Huygens Professional software (Scientific Volume Imaging).

Colocalization analysis is performed with FiJi software (open platform for scientific image analysis).

JaCoP plugin or Colocalization Threshold option of Fiji are used.

Pearson's correlation coefficient corresponds to the linear relationship between intensity of pixels in the two analyzed channels. Mander's coefficient is defined as the ratio of the summed intensities of pixels from one image for which the intensity in the second channel is above the threshold [START_REF] Bolte | A guided tour into subcellular colocalization analysis in light microscopy[END_REF].

Costes Automatic Threshold allows us to consider only the pixels in each channel showing statistical correlation, as explained in Costes et al [START_REF] Costes | Automatic and quantitative measurement of protein--protein colocalization in live cells[END_REF] (Figure 2).

3. Fluorescence intensity analysis. Using FiJi, in a chosen channel, create a Z--projection of the image (whole cell, or a number of optical sections corresponding to an intracellular compartment) and select a region of interest (i.e. immunological synapse, or an intracellular compartment), depending on experimental requirements. Then, in the other channel, measure fluorescence intensity of the selected area of interest (Figure 3).

Notes:

1. Density--Gradient centrifugation to isolate peripheral blood mononuclear cells (PBMC). Expression may be different in Jurkat and in primary T cells. 9. Cover with Parafilm a flat surface and place the poly--L--Lysine coated coverslips over it (poly--L--Lysine side of the coverslips facing up). Add cell suspension over the coverslip, trying to distribute it evenly over the surface. 10. Once fixed and washed, round or square coverslips containing cells may be stored in 6--or 24--well plates, respectively, in an excess of PBS--BSA, for 1--2 weeks at 4°C. Staining does not require to be performed immediately after fixation, except for protein phosphorylation staining, which requires freshly prepared cells for optimum results.

11. Cover with parafilm a flat surface and label areas for different staining solutions planned. Deposit 80 µL drops of primary antibody mix on the parafilm. Take the coverslips with fixed cells and drain the excess of PBS--BSA by setting the edge of the coverslip on absorbent paper. This prevents dilution of antibody staining solutions. Place coverslips over primary antibody drops on Parafilm, cells facing the drops. Create a humidified chamber for the incubation by placing wet pieces of paper around the parafilm, and cover everything with a lid or opaque surface, in order to protect from light.

12. Once slides have been analyzed, it is recommended to store them at --20°C, to prevent loss of fluorescence. While flourochrome fluorescence tends to be stable, GFP fluorescence is lost much faster.

13. Cover with Parafilm a flat surface and deposit 50 µL drops of anti--CD3 antibody mix on the parafilm. Create a humidified chamber for the incubation by placing wet pieces of paper around the parafilm, and cover everything with a lid.

3 .

 3 Insert a new Neon™ tube into the Neon™ Pipette Station and fill it with 3 mL Electrolytic buffer E2. Turn on Neon™ device, and select electroporation protocol (voltage: 1400 V, width: 10 ms, pulses: 3) 4. Resuspend the cell pellet in 100 µL Resuspension buffer R per required transfection. 5. Add 100 µL resuspended cells in each Eppendorf tube with DNA plasmid or siRNA, and mix gently. 6. Take 100 µL of the cells--plasmid mix using the Neon™ pipette and a 100 µL tip, avoiding air bubbles. Change Neon™ tip for every transfection with different DNA or siRNA. Insert the Neon™ pipette with the sample vertically into the Neon® tube and press start.

7 .

 7 Once electroporation is complete, remove Neon™ pipette and transfer the sample into the pre-warmed culture flask and incubate at 37°C, 5% CO 2 for 24--48 h (see Note 8).

1 .

 1 Pulse antigen--presenting cells with superantigen: Harvest Raji cells (5 x 10 6 cells/mL) by centrifuging at 290 xg at 20°C for 4 min. Resuspend in RPMI 1640 (without serum) supplemented with 10 μg/mL Staphylococcus enterotoxin E superantigen (SEE) and incubate for 30 min at 37°C. 2. Immunological synapse formation: Harvest Jurkat J77 Clone 20 cells by centrifuging at 290 xg at, 20°C for 4 min and resuspend at 5 x 10 6 cells/mL RPMI 1640 (without serum). Jurkat J77 Clone 20 cells are incubated with pulsed--Raji cells at 1:1 ratio, at 37°C during 5, 15 or 30 min in RPMI 1640 medium (without serum) in 1.5 mL Eppendorf tubes at 37°C in a water bath. 3. Plating cells on coverslips: 160 µL of conjugated cells are plated onto poly--L--lysine--coated square coverslips and kept for 3 min at room temperature (see Note 9). 4. Fixation: 160 µL of 8% paraformaldehyde are mildly dropped onto the coverslips containing cell suspensions (final concentration of paraformaldehyde 4%), and incubated for 20 min at room temperature. After incubation, paraformaldehyde is removed and coverslips washed with PBS, 1 % (w/v) bovine serum albumin (PBS--BSA) (see Notes 4 and 10).

5 .

 5 Saturation: Nonspecific binding is prevented by 15 min incubation, in PBS--BSA.6. First antibody preparation. Primary antibody (or mix of primary antibodies in case of multiple staining), at the recommended dilution, is suspended in PBS--BSA, 0.1% Triton X--100. 7. Cell immuno--staining: 80 µL drops of primary antibody solution are deposited on Parafilm® and coverslips with fixed cells are turned over on drops (see Note 11). Incubate for 1 h at room temperature, protected from light. 8. Washes: After primary antibody incubation, coverslips are washed twice in PBS--BSA by submerging them several times in a beaker using forceps to handle the coverslips. 9. Second antibody preparation: Secondary antibody (or mix of secondary antibodies in case of multiple staining) is resuspended in PBS--BSA. 10. Staining of coverslips: 80 µL drops of secondary antibody solution are deposited on Parafilm® and coverslips are turned over on drops, as in step 7. Incubate for 45 min -1 h at room temperature, protected from light. 11. Washes: After secondary antibody incubation, coverslips are washed twice in PBS--BSA by submerging in a beaker using forceps to handle the coverslips. Drain the washing solution by setting the edge of the coverslip on absorbent paper, in order to remove excess PBS--BSA before mounting. 12. Mounting: Drops of 20 µL of ProLong Gold Antifade mounting medium with DAPI are deposited on 76 x 26 mm slides and coverslips are turned over on the drops. Let dry mounted slides overnight at room temperature, protected from light. Once dried, slides can be stored at 4°C and used during 2--4 weeks (see Note 12).

2 .

 2 Cell resuspension. Jurkat J77 Clone 20 cells or CD4 + primary T cells are washed twice by centrifugation at 290 xg at 20°C for 4 min and resuspended at 2 x 10 6 cells/mL in RPMI 1640 medium without serum. 3. Plating of cells on coverslips. 100 µL of cells are plated onto anti--CD3--coated coverslips during 3, 5 or 15 minutes 37°C in a humidified chamber kept at 37°C during the assay. 4. Fixation. 100 µL of 8% paraformaldehyde are mildly dropped onto the coverslips containing cell suspensions (final concentration of paraformaldehyde 4%) and incubated for 20 min at room temperature. After incubation PFA is removed and coverslips washed with PBS--BSA (see Notes 4 and 10).

5 .

 5 Saturation. Nonspecific binding is prevented by 15 min incubation in PBS--BSA.6. First antibody preparation. Primary antibody (or mix of primary antibodies in case of multiple staining) at the recommended dilution is suspended in PBS--BSA, 0.1% Triton X--100. 7. Staining of coverslips. 30 µL drops of primary antibody solution are deposited on Parafilm® and coverslips with fixed cells are turned over on drops (see Note 11). Incubate for 1 h at room temperature, protected from light. 8. Washes. After primary antibody incubation, coverslips are washed twice in PBS--BSA by submerging several times in a beaker using forceps to handle the coverslips. 9. Second antibody preparation. Secondary antibody (or mix of secondary antibodies in case of multiple staining) is resuspended in PBS--BSA. 10. Staining of coverslips. 30 µL drops of secondary antibody solution are deposited on Parafilm® and coverslips are turned over on drops, as done in step 7. Incubate for 45 min--1 h, at room temperature, protected from light. 11. Washes. After secondary antibody incubation, coverslips are washed twice in PBS--BSA by submerging in a beaker using forceps to handle the coverslips. Drain the washing solution by setting the edge of the coverslip on absorbent paper, in order to remove excess PBS--BSA before mounting. 12. Mounting. Drops of 20 µL of ProLong Gold Antifade mounting medium with DAPI are deposited on 76 x 26 mm slides and coverslips are turned on the drops. Let mounted coverslips harden overnight at room temperature, protected from light. Slides are then stored at 4°C and may be used during 2--4 weeks (see Note 12).

  Ficoll separation: spin Ficoll tubes 5 min at 453 xg at room temperature. Lay 28 mL of blood per tube of 15 mL Ficoll--Hypaque--Lymphoprep. Centrifuge 30 min at 805 xg at room temperature with no brake. Aspirate plasma layer containing PBMCs and add it to new empty falcon tubes (15 mL recovered PBMCs per 50 mL falcon tube). Add 35 mL RPMI no FCS per 15 mL of recovered PBMCs per tube. Spin 10 min at 453 xg at room temperature. Discard supernatants and pull pellets together to one tube.Wash twice with 50 mL RPMI no FCS (10 min at 453 xg). Resuspend pellet in 50 mL of RPMI no FCS and count cells using cell counter or Trypan blue vital staining and Malassez chambers (Figure4).

2 .

 2 Purification of CD4 + T cells using MACS CD4 Cell Isolation Kit II: follow manufacturer's instructions.

3 . 5 .

 35 Coverslip coating with poly--L--lysine: prepare a 1:50 dilution of 0.01% poly--L--Lysine solution in water. Cover a flat surface with Parafilm, and place round or square coverslips over it. Add 500 µL for square coverslips or 150 µL for round coverslips of the 1:50 poly--L--Lysine dilution over the coverslips, covering the surface completely. Incubate 20 min at room temperature and remove excess. Wash once with water and dry the coverslips before use.4. Paraformaldehyde. We recommend purchasing a stock commercial solution, avoiding manipulation of paraformaldehyde powder, in order to prevent toxicity. Paraformaldehyde solution manipulation should be carried out under a chemical hood. Staphylococcus enterotoxins are a biohazard and should be manipulated as such. Inactivation of all solutions and materials in contact with the toxin should be performed at the end of experiments using an excess of 0.5% (w/v) sodium hypochlorite for at least 1 h. Dispose as biohazard.6. siRNA efficacy to inhibit protein expression depends on the messenger RNA stability and the turnover of each protein. A dose response assay is useful to set the appropriate experimental conditions for different proteins.7. Schematic representation of the transfection time line (Figure5).

8 .

 8 Different plasmids might need different expression times in order to reach enough protein levels.

Figure 1 :

 1 Figure 1 : Microtubule detection at T cell « pseudosynapses » A: Schematic representation of T cells spreading on anti--CD3--coated coverslips. B: Jurkat cells spread 3 min on an anti--CD3--coated coverslip. β--tubulin staining. Confocal image treated by deconvolution. A 4 μm Z--projection close to the coverslip is shown. Scale bar = 5 µm

Figure 2 :

 2 Figure 2 : Colocalization analyses of Rac1, Rab11 and Rab11--FIP3. A, B: Immunostaining of Jurkat T cells for endogenous Rab11 and Rac1 GTPases. A Z--stack of confocal images were treated by deconvolution. A 4 μm Z--projection of the area where the intracellular pericentrosomal compartment is located is shown. Pearson's correlation coefficient (R) in the region framed was measured and the graph is shown on the right panel. Examples of different experimental conditions. A: control cells. Colocalization between Rab11 and Rac1. B: Cells silenced for the expression of the Rab11 effector FIP3. Colocalization between Rab11 and Rac1. C: Effect of GFP--FIP3 (green) overexpression in endogenous Rac1 localization (red). Colocalization between FIP3 and Rac1. Scale bar = 5 µm

Figure 3 :

 3 Figure 3 : Quantification of fluorescence accumulation at the immunological synapse Immunological synapse of a Jurkat (J77cl20) lymphocyte with a Raji B lymphocyte pulsed with SEE superantigen (antigen presenting cell, APC). Confocal image was treated by deconvolution. A 4 μm Z-projection is shown. From left to right: Lck (green), CD3 surface staining (red), merge of fluorescence images and phase contrast image. Rectangles show the region of interest in which quantification of the intensity of fluorescence is performed. It can be then referred to the total fluorescence of the cell or to a region at the opposite side of the synapse. Scale bar = 5 µm.

Figure 4 :

 4 Figure 4: Scheme of a Ficoll gradient centrifugation tube after centrifugation.

Figure 5 . 3 Bouchet et al Figure 4 Day

 534 Figure 5. Schematic representation of the transfection time line.

(see Note 6). 2.4. Transfection systems

	1. Neon™ Transfection System and Neon™ Transfection 100 µL Kit, containing electrolytic buffer E2,
	resuspension buffer R, 100 µL Neon™ tips and Neon™ Electroporation tubes (Invitrogen™, Life
	Technologies™).
	2. Nucleofector™ 2b Device and Primary Cell Nucleofector™ Kit, containing Nucleofector™ Solution,
	single use pipette and Amaxa™ 100 µl aluminum electrode cuvettes (Lonza).

Acknowledgments

This work was supported by grants from the Agence Nationale de Recherche (ANR, No 11 BSV3 025 01), the Agence National de Recherche sur le SIDA et les Hepatitis Virales (ANRS), Institut Pasteur, CNRS and INSERM. JB has been supported by ANRS and Roux--Institut Pasteur postdoctoral fellowships. IdRI is a scholar in the Pasteur -- Paris University (PPU) International PhD program and is funded by the People Programme (Marie Curie Actions) of the European Union's Seventh Framework Programme FP7/2007--2013/ under the REA grant agreement n° 317057 HOMIN.


Alexander R Luedtke

Antoine Chambaz

Faster Rates for Policy Learning

Keywords: individualized treatment rules, personalized medicine, policy learning, precision medicine 1

This article improves the existing proven rates of regret decay in optimal policy estimation. We give a margin-free result showing that the regret decay for estimating a within-class optimal policy is second-order for empirical risk minimizers over Donsker classes, with regret decaying at a faster rate than the standard error of an efficient estimator of the value of an optimal policy. We also give a result from the classification literature that shows that faster regret decay is possible via plug-in estimation provided a margin condition holds. Four examples are considered. In these examples, the regret is expressed in terms of either the mean value or the median value; the number of possible actions is either two or finitely many; and the sampling scheme is either independent and identically distributed or sequential, where the latter represents a contextual bandit sampling scheme.

Introduction 1.Objective

We consider an experiment where a player repeatedly chooses and carries out one among two actions to receive a random reward. Each time, the reward depends on both the action undertaken and a random context preceding it, which is given to the player before she makes her choice. The law of the context and conditional law of the reward given the context and action are fixed throughout the experiment. The player's objective is to obtain as large a cumulated sum of rewards as possible.

In this framework, a policy is a rule that maps any context to an action. The value of a policy is the expectation of the reward in the experiment where the action carried out is the action recommended by the policy. Given a class Π of candidate policies, the regret of a policy π ∈ Π is the difference between the largest value achievable within Π and the value of π.

Learning the optimal policy within a class of candidate policies is meaningful whenever the goal is to make recommendations. This is, for instance, the case in personalized medicine, also known as precision medicine. There, the context would typically consist of the description of a patient, the actions would correspond to two strategies of treatment, and the policies are rather called individualized treatment rules.

The objective of this article is not to establish optimal regret bounds for optimal policy estimators. It is, rather, to show that rates faster than n -1/2 can be demonstrated under much more general conditions than have previously been discussed in the policy learning literature.

A Brief Literature Review

There has been a surge of interest in developing flexible methods for estimating optimal policies in recent years. Here we give a deeply abbreviated overview, and refer the reader to [START_REF] Athey | Efficient Policy Learning[END_REF] for a recent overview of the literature. Exciting developments in policy learning over the last several years include outcome weighted learning [START_REF] Zhao | Estimating individual treatment rules using outcome weighted learning[END_REF][START_REF] Zhang | Estimating optimal treatment regimes from a classification perspective[END_REF][START_REF] Rubin | Statistical issues and limitations in personalized medicine research with clinical trials[END_REF], ensemble techniques [START_REF] Alexander R Luedtke | Super-learning of an optimal dynamic treatment rule[END_REF], and empirical risk minimizers (ERMs) [START_REF] Athey | Efficient Policy Learning[END_REF], to name a few. Each of these works has established some form of regret bound for the estimator, with most of these regret rates slower than n -1/2 . Notable exceptions to this n -1/2 restriction occur for outcome weighted learning (OWL) methods under a hard margin [START_REF] Zhao | Estimating individual treatment rules using outcome weighted learning[END_REF] and rates attainable by plug-in estimation strategies [START_REF] Farahmand | Action-gap phenomenon in reinforcement learning[END_REF][START_REF] A R Luedtke | Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy[END_REF][START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF].

We give our results in Section 2. Our first result pertains to empirical risk minimization within a Donsker class, where the optimal policy is defined as the so-called "value" maximizer within this class. These estimators were recently studied in [START_REF] Athey | Efficient Policy Learning[END_REF], and were also discussed for continuous treatments in [START_REF] A R Luedtke | Comment[END_REF]. The second result links the optimal policy problem to the classification literature, which has shown that faster regret decay rates are attainable by plug-in estimators under certain margin conditions [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]. Section 3 gives three remarks, one relating our results to the pioneering results of Koltchinskii [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF], another studying which term in our regret bound dominates the rate, and the third relating our results to those of Athey and Wager [START_REF] Athey | Efficient Policy Learning[END_REF]. Section A proves our results regarding empirical risk minimization.

All of our results are for the regret under a fixed data generating distribution. The fast rate that we will establish for ERMs (independent of any margin assumption) will not transfer to the minimax setting unless some form of margin assumption is imposed. We do not give high probability results, though these are often a straightforward extension of convergence in probability results when working within Donsker classes. Throughout this work, when we refer to Donsker classes, we are referring to P -Donsker classes, where P is the data generating distribution, rather than universal/uniform Donsker classes [START_REF] Sheehy | Uniform Donsker classes of functions[END_REF]. While our ERM results are not useful for non-P -Donsker classes, we note that they also serve as an oracle guarantee for an ensemble procedure, such as [START_REF] Alexander R Luedtke | Super-learning of an optimal dynamic treatment rule[END_REF], so that one need not know in advance whether or not all of the classes over which they are optimizing are P -Donsker for the result to be useful.

We do not concern ourselves with measurability issues, with the understanding that some work may be needed to make these arguments precise [START_REF] Van Der Vaart | Weak convergence and empirical processes[END_REF].

Formalization

Let X ∈ X denote a vector of covariates describing the context preceding the action, A ∈ {-1, 1} denote the action undertaken, and Y denote the corresponding reward, where here larger rewards are preferable. We denote by P the distribution of O ≡ (X, A, Y ) and by E the expectation under P . For simplicity we assume throughout that, under P , Y is uniformly bounded and that there exists some δ ∈ (0, 1/2) so that P (A = 1|X) falls in [δ, 1δ] with probability one. The second assumption is known as the strong positivity assumption. While stronger than we need, these assumptions simplify our analysis.

Let P be the class of all policies, the subset of L 2 (P ) consisting of functions mapping X to {-1, 1}. The value of a policy π ∈ P is given by

V(π) ≡ E [E[Y |A = π(X), X]] . (1)
Under some causal assumptions that we will not explore here, V(π) can be identified with the mean reward if, possibly contrary to fact, action π(X) is carried out in context X. Now, let Π ⊂ P be the class of candidate policies. The regret (within class Π) of π is defined as the difference between V(π) and the optimal value V ⋆ ≡ sup π∈Π V(π): for all

π ∈ Π, R(π) ≡ V ⋆ -V(π). (2)
We extend the definition of R to P. Obviously, R(π) ≥ 0 for all π ∈ Π, but R is not necessarily nonnegative over P.

For every π ∈ P, V(π) can be viewed as the evaluation at P of the functional

P ′ → E P ′ [E P ′ [Y |A = π(X), X]]
from the nonparametric model of distributions P ′ satisfying the same constraints as P to the real line. This functional is pathwise differentiable at P relative to the maximal tangent space with an efficient influence function f π given by

f π (o) ≡ 1{a = π(x)} P (A = a|X = x) (y -E[Y |A = a, X = x]) + E[Y |A = π(x), X = x] -V(π). (3)
By the bound on Y and the strong positivity assumption, there exists a universal constant

M > 0 such that P (sup π∈Π |f π (O)| ≤ M) = 1.
We refer the interested reader to [START_REF] Van Der | Asymptotic statistics[END_REF]Section 25.3] and [START_REF] A R Luedtke | Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy[END_REF] for definitions and proofs of these facts. Although the class

F ≡ {f π : π ∈ Π} ⊂ L 2 (P ) (4)
will play a central role in the rest of this article, it is not necessary to master the derivation or properties of efficient influence functions to appreciate the contributions of this work.

Main Results

Preliminary

Suppose that we observe mutually independent data-structures

O 1 ≡ (X 1 , A 1 , Y 1 ), . . . , O n ≡ (X n , A n , Y n ) drawn from P . Let P n ≡ n -1 n i=1 Dirac(O i
) denote the empirical measure. For every function f mapping an observation to the real line, we set

P f ≡ E[f (X, A, Y )] and P n f ≡ n -1 n i=1 f (O i ).
Our theorem will attain fast rates under the assumption that one has available an estimator { V(π) : π ∈ Π} of {V(π) : π ∈ Π} that makes the following term small:

Rem n ≡ sup π∈Π V(π) -V(π) -(P n -P )f π . (5)
If empirical process conditions are imposed on the estimators for the reward regression, E[Y |A, X], and the action mechanism, P (A|X), then V could be defined using estimating equations [START_REF] Van Der Laan | Unified methods for censored longitudinal data and causality[END_REF] or targeted minimum loss-based estimation (TMLE) [START_REF] Van Der Laan | Targeted maximum likelihood learning[END_REF][START_REF] Van Der Laan | Targeted Learning: Causal Inference for Observational and Experimental Data[END_REF]. Without imposing empirical process conditions, one could use cross-validated estimating equations [START_REF] A R Luedtke | Corrigendum to: Targeted Learning of the Mean Outcome under an Optimal Dynamic Treatment Rule[END_REF][START_REF] Van Der Laan | Targeted learning of the mean outcome under an optimal dynamic treatment rule[END_REF], now also called double machine learning [START_REF] Chernozhukov | Double machine learning for treatment and causal parameters[END_REF], or a cross-validated TMLE [START_REF] Zheng | Targeted maximum likelihood estimation of natural direct effects[END_REF]. Note that cross-validated estimating equations in this scenario represent a special case of a crossvalidated TMLE, where one uses the squared cross-validated efficient influence function as loss [START_REF] Chaffee | Targeted minimum loss based estimation based on directly solving the efficient influence curve equation[END_REF]. For any of these listed approaches, obtaining the above uniformity over π is straightforward if Π is a Donsker class.

Empirical Risk Minimizers

Our first objective will be to establish a faster than n -1/2 rate of regret decay for a valuebased estimator π of an optimal policy, given by any ERM π ∈ Π satisfying

V( π) ≥ sup π∈Π V(π) -o P (Rem n ). ( 6
)
Note that ( 6) is a requirement on the behavior of the optimization algorithm on the realized sample, rather than a statistical or probabilistic condition. The o P (Rem n ) term allows us to obtain an approximate solution to the ERM problem, which is useful because the optimization on the right is non-concave.

We now present a result that is similar to the groundbreaking results for ERMs based on general losses given in [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF]. In Section 3.1, we discuss how our results relate to those in [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF].

The upcoming theorem uses Π to denote the closure of Π under L 2 (P ) norm. The set

Π ⋆ ≡ {π ⋆ ∈ Π : V(π ⋆ ) = V ⋆ } (7) = {π ⋆ ∈ Π : R(π ⋆ ) = 0} ⊂ {π ⋆ ∈ Π : R(π ⋆ ) ≤ 0}
also plays an important role. If Π is Donsker, then Lemma A.3 in Section A guarantees that Π ⋆ is nonempty and coincides with the RHS set in the above display. Finally, we define

EP n ≡ lim inf s↓0 inf π ⋆ ∈Π ⋆ sup π∈Π: π-π ⋆ ≤s (P n -P )(f π -f π ),
which roughly corresponds to the best approximation in Π of the closest optimal policy π ⋆ ∈ Π ⋆ to the estimated policy π. We will show that EP n = o P (n -1/2 ) under mild assumptions.

Theorem 1 (Main ERM Result). If π satisfies (6) and Π is Donsker, then

0 ≤ R( π) ≤ EP n + [1 + o P (1)] Rem n . If also Rem n = o P (n -1/2 ), then R( π) = o P (n -1/2 ).
The proof of Theorem 1 is given in Section A. The above rate on R( π) is faster than the rate of convergence of the standard error of an efficient estimator of the value V(π) of a policy π ∈ Π, which converges at rate n -1/2 in all but degenerate cases. Note that our proof of the first result of the above theorem only uses that Π is totally bounded in L 2 (P ) while that of the second one uses the stronger assumption that Π is Donsker. Three detailed remarks on this result are given in Section 3. We now show that faster rates are attainable under some conditions if one uses a different estimation procedure. We point this out because, in our experience, this alternative estimation strategy can sometimes yield better estimates than a value-based strategy [START_REF] Alexander R Luedtke | Super-learning of an optimal dynamic treatment rule[END_REF].

Plug-In Estimators

Let γ(X) ≡ E[Y |A = 1, X] -E[Y |A = -1
, X] denote the conditional average action effect. In this subsection, we assume that Π is an unrestricted class, i.e., that Π contains all functions mapping X to {-1, 1}. Audibert and Tsybakov [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF] presented the surprising fact that plug-in classifiers can attain much faster rates (faster than n -1 ). In our setting, the plug-in policy that we will study first defines an estimator γ of γ, and then determines the action to undertake based on the sign of the resulting estimate. Formally, π is given by

π(x) ≡ 1{ γ(x) > 0} -1{ γ(x) ≤ 0}.
Extensions of the result of [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF] have previously been presented in both the reinforcement learning literature [START_REF] Farahmand | Action-gap phenomenon in reinforcement learning[END_REF] and in the optimal individualized treatment literature [START_REF] A R Luedtke | Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy[END_REF][START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF]. We therefore omit any proof, and refer the reader to [10, Lemma 5.2] for further details.

The upcoming theorem uses • to denote the L 2 (P ) norm and • ∞ to denote the L ∞ (P ) norm. Let denote "less than or equal to up to a universal positive multiplicative constant" and consider the following "margin assumption": MA) For some α > 0, P (0 < |γ(X)| ≤ t) t α for all t > 0.

Theorem 2. Suppose MA holds. If γ -γ = o P (1), then 0 ≤ R( π) γ -γ 2(1+α)/(2+α) . If γ -γ ∞ = o P (1), then 0 ≤ R( π) γ -γ 1+α ∞ .
We now briefly describe this result, though we note that it has been discussed thoroughly elsewhere [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF][START_REF] A R Luedtke | Statistical inference for the mean outcome under a possibly non-unique optimal treatment strategy[END_REF][START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF]. Note that MA does not place any restriction on the decision boundary {x ∈ X : γ(x) = 0} where no action is superior to the other, but rather only places a restriction on the probability that γ(X) is near zero.

If α = 1 and γ(X) is absolutely continuous (under P ), then MA corresponds to γ(X)

having bounded density near zero. The case that α = 1 is of particular interest for the sup-norm result because then the regret bound is quadratic in the rate of convergence of γ to γ. As α → 0, more mass is placed near the decision boundary (zero) and the above result yields the rate of convergence of γ to γ. As α → ∞, a vanishingly small amount of mass is concentrated near zero and the above result recovers very fast rates of convergence when γ converges to γ uniformly.

Remark 3 (Estimating γ). Doubly robust unbiased transformations [START_REF] Rubin | A doubly robust censoring unbiased transformation[END_REF] provide one way to estimate γ (Section 3.1 of [START_REF] Van Der Laan | Targeted learning of an optimal dynamic treatment, and statistical inference for its mean outcome[END_REF]). In particular, one can regress (via any desired algorithm) the pseudo-outcome 3 Three Remarks on Theorem 1

Γ(O) ≡ A P (A|X) Y -E[Y |A, X] + E[Y |A = 1, X] -E[Y |A = -1, X] against X,
3.1 Relation to the Rates of Koltchinskii [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF] Our Theorem 1 is related to [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF]Theorem 4] and to the discussion of classification problems in Section 6.1 of this landmark article, although we (i) make no assumptions on the behavior of γ near the decision boundary ("margin assumptions"), (ii) give a fixed-P rather than a minimax result, (iii) do not require that the regret minimizer belongs to the set Π, and (iv) make a slightly weaker complexity requirement on the class Π (only requiring Π Donsker).

We thus have weaker conditions (only constrain the complexity of Π using a general Donsker condition) and, consequently, a weaker implication. Our result also differs from that of [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF] due to the need for us to control the extra remainder term arising from (5).

Our Lemmas A.4 and A.5 in Section A are key to giving this general "Donsker implies fast rate" implication, even when Π does not achieve the infimum on the regret. Once this crucial lemma is established and one has dealt with the existence of the Rem n remainder term, one could use simlar arguments to those used in [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF]Section 4] to control the regret (though, the proofs in our work are self-contained). We were not able to find a similar result in the classification literature that shows that Π Donsker suffices to attain a fast rate on misclassification error (the classification analogue of our regret). A result that makes a similarly weak complexity requirement (only uses a Donsker condition) was given for general result for ERMs in [START_REF] Koltchinskii | Oracle inequalities in empirical risk minimization and sparse recovery problems[END_REF]Theorem 4.5]. In particular, that result shows that, if Π is Donsker and Π ⋆ is a singleton, then one attains a fast rate. In the policy learning context, this is a weaker result than what we have shown: Π Donsker implies a fast rate, without any assumption on the cardinality of Π ⋆ . We note also that our result could readily be extended to standard classification problems: the lack of a remainder term Rem n only makes the problem easier.

Tightening Theorem 1

Tightening Theorem 1 would require careful consideration of the rate of convergence of two o P (n -1/2 ) terms, namely Rem n and EP n . On the one hand, if one uses a cross-validated estimator, then the rate of Rem n is typically dominated by the rate of a doubly robust term, which is in turn upper bounded by the product of the L 2 (P ) norm rates of convergence of the estimated action mechanism and reward regression. It is thus sufficient to assume that this product is o P (n -1/2 ) to guarantee that Rem n = o P (n -1/2 ). It is worth noting that the product is O P (n -1 ) in a well-specified parametric model. On the other hand, the magnitude of EP n is controlled by both the size of class Π and the behavior of γ near the decision boundary, hence the need for a margin assumption like MA.

We do not believe there is a general ordering between the rate of convergence of Rem n and EP n that applies across all problems, and so it does not appear that the size of Π nor the use of an efficient choice of influence function fully determines the rate of regret decay of π, even when Rem n = o P (n -1/2 ). A notable exception to this lack of strict ordering between

Rem n and EP n occurs when the action mechanism is known: in this case, the size of Π and the behavior of γ on the decision boundary fully control the rate of regret decay whenever a cross-validated estimator is used. We also note that, as far as we can tell, there does not appear to be any cost to using an efficient value function estimator when the model is nonparametric. It is not clear if using the efficient influence function is always preferred in more restrictive, semiparametric models: there may need to be a careful tradeoff between the efficiency of the influence function and the corresponding Rem n .

Relation to Results of Athey and Wager [1]

In the recent technical report [START_REF] Athey | Efficient Policy Learning[END_REF], Athey and Wager showed that policy learning regret rates on the order of O P (n -1/2 ) are attainable by ERMs. High-probability regret upper bounds were derived, with leading constants that scale with the standard error of a semiparametric efficient estimator for policy evaluation. The authors argue that this leading constant demonstrates the importance of using semiparametric efficient value estimators to define the empirical risk used by their estimator. As we discussed in Section 3.2, we fully agree with the importance of using efficient estimators to estimate the empirical risk. Nonetheless, the present work shows that the regret of ERMs decays faster than the standard error of an efficient estimator of the value and so, the regret of ERMs that use this empirical risk will not necessarily scale with the standard error of this estimator.

Like in [START_REF] Athey | Efficient Policy Learning[END_REF], our results are given under a fixed data generating distribution P . We implicitly leverage the behavior of γ near the decision boundary (zero) under this distribution P . Crucially, there is a problem-dependent constant that can be made arbitrarily large if one chooses a P for which γ(X) concentrates a large amount of mass near the decision boundary. Hence, the minimax rate is not faster than n -1/2 unless one constrains the class of distributions to which P can belong via a margin condition (see [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF][START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF]). The implication of this observation is encouraging: it would not be surprising to find that, without a margin condition, the minimax regret does in fact decay at the rate of the standard error of an efficient estimator of an optimal policy. The leading constant may also critically depend on the efficient variance of such an estimator. It is worth studying whether the high-probability, finite sample results in [START_REF] Athey | Efficient Policy Learning[END_REF] can be extended in this direction, thereby demonstrating a criterion for which efficient value estimation is mathematically indispensable for obtaining the optimal regret decay. Let Π be a class of policies, i.e., a set of mappings from X to A, and let Π be its L 2 (P ) closure. We request that Π is not too large, in the sense that Π is totally bounded with respect to • .

The value of a policy π ∈ Π is quantified via V(π). As in our earlier result, the regret is

defined as R(π) ≡ V ⋆ -V(π), where V ⋆ ≡ sup π∈Π V(π).
We also introduce the condition

V(•) is uniformly continuous on Π with respect to • . ( 9
)
We assume that there exists a class {f π : π ∈ Π} ⊂ L 2 (P ) of mappings from O to [-M, M] (M not relying on π) and an estimator { V(π) : π ∈ Π} of {V(π) : π ∈ Π} such that, for a (possibly stochastic) rate r n → +∞,

π → f π is uniformly continuous from Π to L 2 (P ), and (10)
sup π∈Π r n V(π) -V(π) -G n f π = o P (1). (11)
In [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF], both spaces are equipped with • . In [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF], G n ∈ ℓ ∞ (F ) is a stochastic process on F ≡ {f π : π ∈ Π} that may or may not be equal to the empirical process, but must satisfy

G n ; G P in ℓ ∞ (F ), (12)
where almost all sample paths of G P are uniformly continuous with respect to • .

Finally we also assume that, for the same rate r n as in [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF], π ∈ Π satisfies the ERM property

V( π) ≤ inf π∈Π V(π) + o P (r -1 n ). (13)
Like ( 6), ( 13) is a requirement on the behavior of the optimization algorithm on the realized sample, rather than a statistical or probabilistic condition. The following result generalizes Theorem 1.

Theorem 4 (More General ERM Result). If (8) through (13) hold, then R( π) = o P (r -1 n ).
A sketch of the proof is given in Appendix B. We only outline where the proof would deviate from that of Theorem 1.

Remark 5. If G P is equal to the zero-mean Gaussian process with covariance given by E[G P f G P g] = P f g -P f P g, then G P has almost surely uniformly continuous sample paths.

In particular, [START_REF] Van Der Vaart | Weak convergence and empirical processes[END_REF]Example 1.5.10] shows that G P has almost surely uniformly continuous sample paths with respect to the standard deviation semimetric, and Section 2.1 in that same reference shows that, for bounded F , one can replace this semimetric by that on L 2 (P ). 2

In the remainder of this section, we give a taste of the broad applicability of this result via three examples. The first example is a simple extension of the framework of Section 2 that allows for more than two possible actions. The second example substitutes the median reward for the mean reward [for a similar setting, see 27]. The third example focuses again on the mean reward, but considers a non-i.i.d., contextual-bandit-type setting in which contextspecific actions may be informed by earlier observations [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF].

Example 1: Maximizing the Mean Reward of a Discrete Action

In this example, there are card(A) ∈ [2, ∞) (finitely many) candidate actions to undertake (think of a discretized dose in the personalized medicine framework). Without loss of generality, we assume that A ⊂ [0, 1]. We assume that under the distribution P of O ≡ (X, A, Y ), the reward Y is uniformly bounded and there exists some δ > 0 so that min a∈A P (A = a|X) ≥ δ with probability one. Here too, the value of a policy π ∈ Π is given by (1), the (within class Π) optimal value is V ⋆ ≡ sup π∈Π V(π) and the regret of π ∈ Π is defined as in [START_REF] Zhao | Estimating individual treatment rules using outcome weighted learning[END_REF]. Finally, we observe

O 1 ≡ (X 1 , A 1 , Y 1 ), . . . , O n ≡ (X n , A n , Y n ) drawn i.i.d. from P .
For each π ∈ Π and o ∈ O, let f π (o) be given by [START_REF] Zhang | Estimating optimal treatment regimes from a classification perspective[END_REF]. Let

G n ≡ n 1/2 (P n -P ) ∈ ℓ ∞ (F )
be the empirical process on F = {f π : π ∈ Π}. Note the parallel between the LHS of [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF] with r n ≡ n 1/2 and n 1/2 Rem n from (5).

We prove the next lemma in Section B.2: Lemma 6. If Π is Donsker, then (8), ( 9), ( 10) and (12) are met.

Therefore, Theorem 4 yields the following corollary:

Corollary 7. In the context of Section 4.2, suppose that π satisfies (13) with r n ≡ n 1/2 and that

sup π∈Π n 1/2 V(π) -V(π) -G n f π = o P (1)
. We define the value of π as the median rather than the mean reward, i.e., as

If Π is Donsker, then R( π) = o P (n -1/2 ).
V(π) ≡ inf {m ∈ R : 1/2 ≤ F π (m)} . (14)
Let Π ⊂ P be the class of candidate policies. Recall that the regret (within class Π) of π ∈ Π takes the form R(π) ≡ sup π∈Π V(π) -V(π).

Let us now turn to [START_REF] A R Luedtke | Comment[END_REF]. Assume that there exists c > 0 such that, for each π ∈ Π, F π is continuously differentiable in the neighborhood [V(π) ± c] with derivative Ḟπ (m) at m in this neighborhood, where

0 < inf π∈Π inf m∈[V(π)±c] Ḟπ (m) ≤ sup π∈Π sup m∈[V(π)±c] Ḟπ (m) < ∞. (15)
In addition, assume that there exists a function ω

: [0, ∞) → [0, ∞) with lim m↓0 ω(m) = ω(0) = 0 such that sup π∈Π sup |k|≤c Ḟπ (V(π) + k) -Ḟπ (V(π)) -ω(|k|) ≤ 0. ( 16
)
Remark 8. A sufficient, but not necessary, condition for such an ω to exist is that F π is twice continuously differentiable with the absolute range of the second derivative Fπ on [V(π) ± c] bounded away from infinity uniformly in π ∈ Π. Indeed, Taylor's theorem then

shows that one can take

ω(m) ≡ m × sup π∈Π sup m∈[V(π)±c] | Fπ ( m)|. 2
The next lemma gives conditions under which (9) holds. Its proof is given in Appendix B.3.

Lemma 9. If ( 15) and ( 16) are met, then (9) holds.

For every π ∈ P, V(π) defined in ( 14) can be viewed as the evaluation at P of the functional

P ′ → inf m ∈ R : 1/2 ≤ E P ′ [P ′ (Y ≤ m|A = π(X), X)]
from the nonparametric model of distributions P ′ satisfying the same constraints as P to the real line. This functional is pathwise differentiable at P relative to the maximal tangent space with an efficient influence function f π given by

f π (o) ≡ 1{a = π(x)} P (A = a|X = x) Ḟπ (V(π)) [1{y ≤ V(π)} -P {Y ≤ V(π)|A = a, X = x}] + P {Y ≤ V(π)|A = π(x), X = x} -1/2 Ḟπ (V(π)) . (17)
Observe that above Ḟπ (V(π)) only enters f π as a multiplicative constant, and therefore an estimating equation-based estimator or TMLE for V(π) can be asymptotically linear for V(π) without estimating Ḟπ . We, in particular, suppose that we have an estimator satisfying [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF] with r n = n 1/2 and G n ≡ n 1/2 (P n -P ) ∈ ℓ ∞ (F ) with F ≡ {f π : π ∈ Π}, see Remark 11 at the end of the present section.

With this choice of G n , ( 12) is met and Π Donsker yields [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF], as seen in the proof of Theorem 1 (the same argument applies because the range of each π ∈ P is bounded in

[-1, 1]
). Theorem 4 yields the following corollary:

Corollary 10. In the context of Section 4.3, suppose that ( 15) and ( 16) are met. Let π satisfy (13) with r n ≡ n 1/2 and

sup π∈Π n 1/2 V(π) -V(π) -G n f π = o P (1).
If Π is Donsker and (10) holds, then R( π) = o P (n -1/2 ).

Since Π is Donsker, (10) can be derived under regularity conditions by using essentially the same techniques as in the proof of Theorem satisfies [START_REF] Koltchinskii | Local Rademacher complexities and oracle inequalities in risk minimization[END_REF]. In the above display, P denotes an estimate of (certain conditional probabilities under) P . One could use cross-validated estimating equations to avoid the need for any empirical process conditions on P . 2

Example 3: Sequential Decisions to Maximize the Mean Reward of a Binary Action

The sampling design. This section builds upon [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF]. Let {(X n , Z n (-1), Z n (1)} n≥1 be a sequence of random variables drawn independently from a distribution P such that, if { Ãn } n≥1 is a sequence of independent actions drawn from the conditional distribution of A (an action with support {-1, 1}) given X under the same P as in Section 1.3, then the resulting sequence {(X n , Ãn , Z n ( Ãn ))} n≥1 is an i.i.d. sample from P . For notational simplicity, we assume that all the random variables Z n (-1) and Z n (1) take their values in [0, 1].

In this example, however, the distribution ν n of (O 1 , . . . , O n ) does not write as a product because the actions A 1 , . . . , A n are not i.i.d. On the contrary, once sufficiently many observations have been accrued to carry out inference then, sequentially, each new randomized action A n is drawn conditionally on X n and an estimate derived from the previous obser-

vations O 1 ≡ (X 1 , A 1 , Y 1 ), . . . , O n-1 ≡ (X n-1 , A n-1 , Y n-1
) yielded by the previous actions, where Y i ≡ Z i (A i ) for i = 1, . . . , n-1. Let us describe how the data-adaptive design unfolds.

Let {t n } n≥1 and {ξ n } n≥1 be two nonincreasing sequences with t 1 ≤ 1/2, lim n t n > 0 and lim n ξ n > 0. For each n ≥ 1, let G n be a nondecreasing κ n -Lipschitz functions approximating

u → 1{u ≥ 0} in the sense that G n (u) = t n for u ≤ -ξ n , G n (0) = 1/2, G n (u) = 1 -t n for u ≥ ξ n . We also suppose that lim sup n κ n < ∞. For each n ≥ 1, let Q n ≡ {Q β : β ∈ B n } be a working model consisting of functions mapping {-1, 1} × X to [0, 1]. Each Q β ∈ Q n can
be viewed as a candidate approximation to the function (A, X)

→ 1{A = 1} E P [Z(1)|X] + 1{A = -1} E P [Z(-1)|X].
For some n 0 ≥ 1, A 1 , . . . , A n 0 are independent draws from the conditional distribution of A given X under P . For every i = 1, . . . , n 0 , let g i : {-1, 1} × X → [0, 1] be defined pointwise by g i (a, x) ≡ P (A = a|X = x). Set n > n 0 , suppose that we have fully specified how O 1 , . . . , O n-1 have been sampled through the description of how A 1 , . . . , A n-1 have been randomly drawn from Rademacher laws † with parameters g 1 (1, X 1 ), . . . , g n-1 (1, X n-1 ), respectively. Now, let us describe how O n is sampled by specifying how action A n is randomized, therefore completing the description of the sampling design. Based on O 1 , . . . , O n-1 , we define

β n ≡ arg min β∈Bn n-1 i=1 L(Q β )(O i ) g i (A i |X i )
where the least-square loss function L is given by

L(Q β )(O) ≡ (Y -Q β (A, X)) 2 . This yields the mapping g n : {-1, 1} × X → [t n , 1 -t n ] ⊂ [0, 1] given by 1 -g n (-1, x) ≡ g n (1, x) ≡ G n Q βn (1, x) -Q βn (-1, x) .
Once X n is observed, we sample A n conditionally on O 1 , . . . , O n-1 and X n from the Rademacher law with parameter g n (1, X n ), we then carry out action A n , observe

Y n ≡ Z n (A n ),
and form

O n ≡ (X n , A n , Y n ).
Remark 12. The sampling design looks for a trade-off between exploration and exploitation.

Here, exploitation consists in using the current best estimates Q βn (1, X n ) and Q βn (-1, X n ) of E P [Z(1)|X = X n ] and E P [Z(-1)|X = X n ] to favor the action which seems to have the larger mean reward in context X n . If the absolute value of the difference between the two estimates is larger than ξ n , then the supposedly superior action is carried out with probability (1t n ). Exploration consists in giving the supposedly inferior action a probability t n to be undertaken nonetheless. This allows, for instance, to correct a possibly poor estimation of Optimal policy estimation. We now define set of candidate policies Π. In particular Π is the set of policies such that, for each π ∈ Π, there exists β ∈ ∪ n≥1 B n for which

π(X) ≡ arg max a∈{-1,1} Q β (a, X) = 1{Q β (1, X) ≥ Q β (-1, X)} -1{Q β (1, X) < Q β (-1, X)},
where the choice to let

A = 1 when Q β (1, X) = Q β (-1, X)
in the second equality is a convention. The value V(π) of π ∈ Π is still given by ( 1) and its (within class Π) regret R(π) by ( 2).

Recall the definition of F characterized by ( 3) and ( 4). Let G n ∈ ℓ ∞ (F ) be such that,

for each f ∈ F , n 1/2 G n f ≡ n i=1 (f (O i , W i ) -E P g i [f (O i , W i )|O 1 , . . . , O i-1 ]) ,
where we use the fact that g i is deterministic when one conditions on O 1 , . . . , O i-1 (i.e., on nothing when i = 1). By exploiting the martingale structure of each n 1/2 G n f , it is possible to show that G n satisfies (12) under mild conditions (expressed in terms of the uniform entropy integral) on {Q n } n≥1 [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward, supplementary material[END_REF]Lemma B.3]. Theorem 4 yields a corollary for our sequential design setting. We discuss the conditions of the corollary following its statement.

Corollary 14. In the context of Section 4.4, suppose that {Q n } n≥1 is chosen in such a way that Π is separable and J 1 (η n , Π) = o(1) for any η n = o(1), where J 1 (η, Π) is the uniform entropy integral evaluated at η of Π wrt the envelope function constantly equal to one. Moreover, suppose that π satisfies (13) with r n ≡ n 1/2 and that

sup π∈Π n 1/2 V(π) -V(π) -G n f π = o P (1). Then R( π) = o P (n -1/2 ).
Lemma 6 shows that ( 9) and ( 10) are met. Conditions ( 8) and ( 12) follow from Lemma A.7

in Appendix A and from the maximal inequality stated in [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward, supplementary material[END_REF]Lemma B.3], whose assumptions drive the conditions on {Q n } n≥1 stated in Corollary 14. A concrete example of such a sequence can be found in [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF]Section 4.4] (see also Section 4.1 therein for a brief reminder about the uniform integral entropy). Additional mild assumptions guaranteeing that an estimator V can be defined as requested by Corollary 14 can be adapted from [START_REF] Chambaz | Targeted sequential design for targeted learning inference of the optimal treatment rule and its mean reward[END_REF]. Finally, we note for the ERM property ( 13) that each π ∈ Π is known once one knows the β ∈ ∪ n≥1 B n that indexes this π. Hence, it suffices to study the estimated value V(π) for each such π.

The appropriate algorithm for carrying out this optimization will depend on the particular choice of classes B n and the working model Q β .

Discussion

We have presented fast rates of regret decay in optimal policy estimation, i.e., rates of decay that are faster than the rate of decay of the standard error of an efficient estimator of the value of any given policy in the candidate class. Our method of proof for our primary result, Theorem 1, leverages the fact that the empirical process over a Donsker class converges in distribution to a Gaussian process, and that the sample paths of this limiting process are (almost surely) uniformly continuous. The downside of our analysis, namely passing to the limit and then studying the behavior of the limiting process, is that it does not appear to allow one to obtain a faster than o P (n -1/2 ) rate of convergence for the regret.

It would be of interest to replace our limiting argument by finite sample results that would allow one to exploit the finite sample equicontinuity of the empirical process to demonstrate faster rates. Nonetheless, we note that the problem-dependent margin condition will often have a major impact on the extent to which the rate can be improved. Furthermore, as we discussed in Section 3.2, the existence of the remainder term Rem n necessitates a careful consideration of whether or not the empirical process term represents the dominant error term, or whether the second-order remainder that appears due to the non-linearity of the value parameter represents the dominant error term. In restricted, semiparametric models, we suspect that, depending on the underlying margin, an inefficient estimator of the value may yield a faster rate of regret decay than an efficient estimator. Despite this surprising phenomenon, we continue to advocate the use of first-order efficient value estimators.

This proves the continuity of V. That of R follows immediately.

Recall the definition (7) of Π ⋆ .

Lemma A.3. If Π is totally bounded, then inf π∈Π R(π) = 0. Moreover, Π ⋆ = {π ⋆ ∈ Π : R(π ⋆ ) ≤ 0} and Π ⋆ = ∅.
Proof. By Lemma A.1, Π is compact. By Lemma A.2, R is continuous. Thus, R admits and achieves a minimum R(π) = inf π∈Π R(π) on Π. Since inf π∈Π R(π) = 0, we know that R(π) ≤ 0. In fact, a contradiction argument shows that R(π) = 0. In other words,

Π ⋆ = {π ⋆ ∈ Π : R(π ⋆ ) ≤ 0} and π ∈ Π ⋆ , hence Π ⋆ = ∅.
Indeed, assume that R(π) < 0. Because π ∈ Π, there exists a sequence {π m } m≥1 of

elements of Π such that π m -π → 0. By continuity of R (see Lemma A.2), R(π m ) → R(π) < 0. Thus, inf π∈Π R(π) < 0. Contradiction. Lemma A.4. If Π is totally bounded, then lim sup r↓0 sup π∈Π:R(π)≤r inf π ⋆ ∈Π ⋆ π ⋆ -π = 0. (CA)
Proof. We argue by contraposition. Suppose CA does not hold. Then there exists a sequence {π m } m≥1 of elements of Π and δ > 0 such that R(π m ) → 0 and, for all m ≥ 1, inf

π ⋆ ∈Π ⋆ π m -π ⋆ > δ. (A.3)
We now give a contradiction argument to show that {π m } m≥1 does not have a convergent subsequence. Suppose there exists a subsequence {π

m k } k≥1 such that π m k -π ∞ → 0 for some π ∞ ∈ L 2 (P ). Now, note that (i) π ∞ ∈ Π, the closure of Π, (ii) R(π m k ) → 0, and (iii)
R(π m k ) → R(π ∞ ) by Lemma A.2. Consequently, R(π ∞ ) = 0. Since π ∞ ∈ Π, this reveals that π ∞ ∈ Π ⋆ and inf π ⋆ ∈Π ⋆ π m k -π ⋆ ≤ π n k -π ∞ → 0,
in contradiction with (A.3). Thus, there does not exist a convergent subsequence {π m k } k≥1 of {π m } m≥1 , completing the contradiction argument.

We now return to the contraposition argument. The existence of a sequence {π m } m≥1 of elements of Π not having a convergent subsequence implies that Π is not sequentially compact and, therefore, that it is not compact. By Lemma A.1, Π is not totally bounded.

This completes the proof.

Recall the definition (4) of F . The next lemma uses the following definitions. We let ℓ ∞ (F ) denote the metric space of all bounded functions z : F → R, equipped with the supremum norm and, for any r > 0, Π r ≡ {π ∈ Π : R(π) ≤ r}. If Π ⋆ is nonempty (for instance, if Π is totally bounded by Lemma A.3) we let, for any π ⋆ ∈ Π ⋆ and s > 0,

B Π (π ⋆ , s) ≡ {π ∈ Π : π ⋆ -π ≤ s} denote the intersection of the radius-s L 2 (P ) ball centered at π ⋆ and the collection Π. Because Π ⋆ ⊂ Π, B Π (π ⋆ , s) is nonempty. Lemma A.5. Define g : ℓ ∞ (F ) × (0, ∞) → R as g(z, r) ≡ sup π∈Πr z(f π ) -lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ). Let z ∈ ℓ ∞ (F ) be • -uniformly continuous and let {(z m , r m )} m≥1 be a sequence with values in ℓ ∞ (F ) × (0, ∞) such that sup f ∈F |z m (f ) -z(f )| + |r m | → 0. If Π is totally bounded, then lim sup m→∞ g(z m , r m ) ≤ 0.
The following corollary will prove useful.

Corollary A.6. Recall the definition of g from Lemma A.5. Let h :

ℓ ∞ (F ) × [0, ∞) → R be such that, for all z ∈ ℓ ∞ (F ), h(z, r) ≡ max (g(z, r), 0) if r > 0 and h(z, 0) ≡ 0. Let z ∈ ℓ ∞ (F ) be • -uniformly continuous. If Π is totally bounded, then h is continuous at (z, 0).
Proof of Lemma A.5 and Corollary A.6. Fix a sequence {(z m , r m )} m≥1 satisfying the conditions of the Lemma A.5. Observe that, for every m ≥ 1,

|g(z m , r m ) -g(z, r m )| ≤ sup π∈Πr m z m (f π ) -sup π∈Πr m z(f π ) + lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z m (f π ) -lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ) ≤ 2 sup π∈Π |z m (f π ) -z(f π )|
where the above RHS expression is o(1) because z m → z in ℓ ∞ (F ). Therefore,

g(z m , r m ) = g(z m , r m ) -g(z, r m ) + g(z, r m ) = g(z m , r m ) -g(z, r m ) + sup π∈Πr m z(f π ) -lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ) . ≤ sup π∈Πr m z(f π ) -lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ) + o(1). (A.4)
Let us show now that the RHS expression in (A.4) is o(1).

For any r > 0, there exists a π r ∈ Π r such that

sup π∈Πr z(f π ) ≤ z(f πr ) + r. (A.5)
Furthermore, there exists a π ⋆ r ∈ Π ⋆ such that

π ⋆ r -π r ≤ inf π ⋆ ∈Π ⋆ π ⋆ -π r + r ≤ sup π∈Πr inf π ⋆ ∈Π ⋆ π ⋆ -π + r. (A.6)
Likewise, there exists πr ∈ B Π (π ⋆ r , r) such that

z(f πr ) ≤ inf π∈B Π (π ⋆ r ,r) z(f π ) + r ≤ sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,r) z(f π ) + r = lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ) + o r (1) + r, (A.7)
where the above equality holds by the definition of the limit superior (the o r (1) above represents the term's behavior as r → 0). In light of (A.5) and (A.7), we thus have Lemma A.7. If Π is Donsker, then F is also Donsker.

sup π∈Πr z(f π ) -lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ) ≤ z(f πr ) -z(f πr ) + 2r + o r (
Proof. Similar to (A.1), the key is the following remark: for every policy π :

X → {-1, 1}, 2f π (O) = |A + π(X)| Y -Q(A, X) P (A|X) +(1 + π(X))Q(1, X) + (1 -π(X))Q(-1, X) -2V(π). (A.10)
For future use, we first note that (A.1) and (A.2) imply, for any π 1 , π 2 ∈ Π,

|f π 1 (O) -f π 2 (O)| |π 1 (X) -π 2 (X)| + π 1 -π 2 hence f π 1 -f π 2 π 1 -π 2 . (A.
φ • f = fπ if f ∈ F writes as f = (f 1 , f 2 , π, f 4 , f 5
). In light of (A.10), observe now that, for

every f 1 = (f 1 , f 2 , π 1 , f 4 , f 5 ), f 2 = (f 1 , f 2 , π 2 , f 4 , f 5 ) ∈ F, it holds that |φ • f 1 (o) -φ • f 2 (o)| |π 1 (x) -π 2 (x)|
(the bound on Y implies that γ ∞ is finite). By [START_REF] Van Der Vaart | Weak convergence and empirical processes[END_REF]Theorem 2.10.6], whose conditions are obviously met, φ • F = { fπ : π ∈ Π} is Donsker. Because Λ ≡ {V(π) : π ∈ Π} (viewed as a set of constant functions with a uniformly bounded sup-norm) is also Donsker, [13, Example 1.10.7] yields that { fπλ : π ∈ Π, λ ∈ Λ} is Donsker, and so is its subset F .

Proof of Theorem 1. In this proof, we make the dependence of π on n explicit by writing π n .

By Lemma A.7, Π Donsker implies F Donsker. Consider the empirical process G n as the element of ℓ ∞ (F ) characterized by G n f ≡ n 1/2 (P n -P )f for every f ∈ F . We let G P ∈ ℓ ∞ (F ) denote the zero-mean Gaussian process with covariance given by E[G P f G P g] = P f g-P f P g.

Since Π Donsker implies Π totally bounded, Lemma A.3 guarantees the existence of a π ⋆ ∈ Π ⋆ . For any s > 0 and π ⋆ s ∈ B Π (π ⋆ , s) ⊂ Π, ( 6) then (5) combined with (A.2) yield in turn the first and second inequalities below:

0 ≤ R( π n ) = V ⋆ -V( π n ) = [V(π ⋆ s ) -V( π n )] + [V(π ⋆ ) -V(π ⋆ s )] = ( V -V)( π n ) -( V -V)(π ⋆ s ) + [ V(π ⋆ s ) -V( π n )] + [V(π ⋆ ) -V(π ⋆ s )] ≤ ( V -V)( π n ) -( V -V)(π ⋆ s ) + [V(π ⋆ ) -V(π ⋆ s )] + o P (Rem n ) n -1/2 G n f πn -G n f π ⋆ s + s + [
(n -1/2 ) imply 0 ≤ R( π n ) s + O P (n -1/2 ) + [1 + o P (1)]o P (n -1/2 )
where the random terms do not depend on s. By letting s go to zero, we obtain R( π n ) = O P (n -1/2 ). The remainder of the proof tightens this result to R( π n ) = o P (n -1/2 ).

Let us go back to (A.12). Since Rem n = o P (n -1/2 ), it also yields the tighter bound

0 ≤ R( π n ) n -1/2 lim inf s↓0 inf π ⋆ ∈Π ⋆ sup π ⋆ s ∈B Π (π ⋆ ,s) G n f πn -G n f π ⋆ s + [1 + o P (1)]o P (n -1/2
0 ≤ n 1/2 R( π n ) ≤ sup π∈Π R( πn)+tn G n f π -lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) G n (f π ) + o P (1) = g(G n , R( π n ) + t n ) + o P (1).
Above we used R( π n ) + t n rather than R( π n ) to avoid separately handling the cases where Π ⋆ ∩ Π is and is not empty.

The conclusion is at hand. Recall the definition of h from Corollary A.6. Clearly the previous display yields the bounds

0 ≤ n 1/2 R( π n ) ≤ h(G n , R( π n ) + t n ) + o P (1). (A.15)
We have already established that R( π n ) = o P (1), hence 0 < t n ≤ R( π n ) + t n = o P (1) as well.

Because F is Donsker, G n ; G P in distribution on ℓ ∞ (F ). Therefore, (G n , R( π n ) + t n ) ;

(G P , 0) in distribution on ℓ ∞ (F ) × [0, ∞).
(G n , R( π n ) + t n ) ; h(G P , 0) in distribution in ℓ ∞ (F )
. This convergence also occurs in probability since the limit is almost surely constant (zero). By (A.15), 0 ≤ R( π n ) = o P (n -1/2 ). This completes the proof.

B Additional Proofs B.1 Sketch of Proof of Theorem 4

The proof of Theorem 4 is very similar to that of Theorem 1. We only sketch it and point out the places where they differ.

Sketch of Proof of Theorem 4. Obviously (9) implies that R(•) is uniformly continuous on Π with respect to • . Assumption ( 8) then shows that the implication of Lemma A.3 also holds in our context, i.e., that inf π∈Π R(π) = 0 and that Π ⋆ is not empty. As R(•)

is uniformly continous and the implication of Lemma A.3 holds, (CA) from Lemma A.4 also holds. Furthermore, (10) yields Lemma A.5, for which Corollary A.6 remains a valid corollary. We will not make use of Lemma A.7: we will instead use directly [START_REF] Sheehy | Uniform Donsker classes of functions[END_REF].

We now have the tools needed to modify the proof of Theorem 1. Using the results we have obtained thus far, and replacing ( 6) by ( 13) and ( 5) by ( 11), we see that (A.13) from the proof of Theorem 1 can be replaced by

0 ≤ R( π n ) r -1 n G n f πn -G n f π ⋆ s + [V(π ⋆ ) -V(π ⋆ s )] + o P (r -1 n ) (B.16) ≤ 2r -1 n sup f ∈F | G n f | + [V(π ⋆ ) -V(π ⋆ s )] + o P (r -1 n ).
By [START_REF] Sheehy | Uniform Donsker classes of functions[END_REF] and the continuous mapping theorem [13, Theorem 1.

3.6], sup f ∈F | G n f | = O P (1).
Hence, the leading term in the final inequality is O P (r -1 n ), where this term does not depend on s. The final term also does not depend on s. By (9), the middle term above goes to zero as s ↓ 0, where this convergence is uniform in both

π ⋆ ∈ Π ⋆ and π ⋆ s ∈ B Π (π ⋆ , s). Thus, R( π n ) = O P (r -1 n ). We tighten this result to R( π n ) = o P (r -1
n ) as in the proof of Theorem 1. In particular, nearly identical arguments to those used in that proof show that for every π 1 , π 2 ∈ Π, hence [START_REF] A R Luedtke | Comment[END_REF]. We also generalize (A.10), which becomes for every π ∈ Π. The above equality and (B.17) imply f π 1f π 2 π 1π 2 for every π 1 , π 2 ∈ Π, hence [START_REF] Audibert | Fast learning rates for plug-in classifiers[END_REF]. The second part of the proof of Lemma A.7 can easily be generalized to derive that F is Donsker from (B.18) and the fact that Π is itself Donsker. Therefore, [START_REF] Sheehy | Uniform Donsker classes of functions[END_REF] is met and the proof is complete.

0 ≤ r n R( π n ) ≤ g( G n , R( π n ) + t n ) + o P (1) ≤ h( G n , R( π n ) + t n ) + o P (1
f π (O) =

B.3 Proofs for Section 4.3

We start by proving Lemma 9.

Proof of Lemma 9. Set arbitrarily π 1 , π 2 ∈ Π and m ∈ [V(π 2 ) ± c]. We suppose without loss of generality that V(π 1 ) ≤ V(π 2 ). We will use the fact that [START_REF] Van Der Laan | Unified methods for censored longitudinal data and causality[END_REF] implies that F π 1 (V(π 1 )) = F π 2 (V(π 2 )) = 1/2 several times in this proof.

Firstly, note that

|F π 1 (m) -F π 2 (m)| = E 1{A = π 1 (X)} -1{A = π 2 (X)} P (A|X) 1{Y ≤ m} ≤ 1 2 E |π 1 (X) -π 2 (X)| P (A|X) 1{Y ≤ m} ≤ k 1 π 1 -π 2 (B.19)
where k 1 is a finite, positive constant that only depends on the lower bound on P (A|X) from the strong positivity assumption.

Secondly, the continuous differentiability of Ḟπ (m).

Therefore, there exists a finite, positive constant k 2 such that

|V(π 2 ) -m| ≤ k 2 |F π 2 (V(π 2 )) -F π 2 (m)| . (B.20)
The remainder of this proof is broken into two parts: we will show that (i) V(π 2 )-V(π 1 ) ≤ c implies V(π 2 )-V(π 1 ) ≤ k 1 k 2 π 1π 2 , and (ii) π 1π 2 < c/k 1 k 2 yields V(π 2 )-V(π 1 ) ≤ c. By combining these two results, we will thus prove that V(π 2 ) -V(π 1 ) ≤ k 1 k 2 π 1π 2 for all π 1π 2 sufficiently small, and this will complete the proof (k 1 k 2 does not depend on π 1 or π 2 ).

Recall that F π 1 (V(π 1 )) = F π 2 (V(π 2 )) = 1/2. If V(π 2 ) -V(π 1 ) ≤ c, then combining (B. [START_REF] Van Der Laan | Targeted learning of the mean outcome under an optimal dynamic treatment rule[END_REF] and (B.20) at m = V(π 1 ) establishes (i):

V(π 2 ) -V(π 1 ) ≤ k 2 [F π 2 (V(π 2 )) -F π 2 (V(π 1 ))] = k 2 [F π 1 (V(π 1 )) -F π 2 (V(π 1 ))] ≤ k 1 k 2 π 1 -π 2 .
We argue (ii) by contraposition. Suppose that V(π 1 ) < V(π 2 )c. By the monotonicity of cumulative distribution functions, F π 2 (V(π 1 )) ≤ F π 2 (V(π 2 )c). Combining this with (B.20)

at m = V(π 2 ) -c, F π 2 (V(π 2 )) -F π 2 (V(π 1 )) ≥ F π 2 (V(π 2 )) -F π 2 (V(π 2 ) -c) ≥ k -1 2 c.
By (B. [START_REF] Van Der Laan | Targeted learning of the mean outcome under an optimal dynamic treatment rule[END_REF]) and the fact that F π 2 (V(π 2 )) = F π 1 (V(π 1 )) = 1/2, the LHS expression is smaller Second note that, for every ǫ > 0 small enough,

than k 1 π 1 -π 2 . Therefore, π 1 -π 2 ≥ c/k 1 k 2 .
F π 2 (m + ǫ) -F π 2 (m)-F π 1 (m + ǫ) + F π 1 (m) ≤ 1 2 E |π 1 (X) -π 2 (X)| P (A|X) 1{m < Y ≤ m + ǫ} E [|π 1 (X) -π 2 (X)| 1{m < Y ≤ m + ǫ}] = E [|π 1 (X) -π 2 (X)|P (m < Y ≤ m + ǫ|X)] ≤ Lǫ E [|π 1 (X) -π 2 (X)|] ǫ π 1 -π 2 .
Returning to the first display completes the proof.

Corollary B.9. Under the conditions of Lemmas 9 and B.8, and the additional assumption [START_REF] Van Der Laan | Targeted maximum likelihood learning[END_REF], the map π → Ḟπ (V(π)) is uniformly continuous on Π with respect to • .

Proof of Corollary B.9. Set π 1 , π 2 ∈ Π. By Lemma 9, we can choose π 2 sufficiently close to π 1 in L 2 (P ) (where "sufficiently close" does not depend on the choice of π 1 ) so that |V(π 1 ) -V(π 2 )| ≤ c, where c is the constant from [START_REF] Van Der Laan | Unified methods for censored longitudinal data and causality[END_REF]. For all such choices of π 1 , π 2 , Ḟπ 2 (V(π 2 )) -Ḟπ 1 (V(π 1 )) ≤ Ḟπ 2 (V(π 2 )) -Ḟπ 1 (V(π 2 )) + Ḟπ 1 (V(π 2 )) -Ḟπ 1 (V(π 1 ))

π 1 -π 2 + ω( π 1 -π 2 ),
where the constant on the right does not depend on π 1 , π 2 . The final bound used Lemma B.8 and ( 16). As ω(0) = 0 and ω is continuous at zero, one can choose π 1π 2 sufficiently small so that the right-hand side is less than any ǫ > 0. As "sufficiently small" does not depend on the choice of π 1 , π → Ḟπ (V(π)) is uniformly continuous on Π with respect to

• .

4. 3

 3 Example 2: Maximizing the Median Reward of a Binary Action In this example, we use the same i.i.d. (from P ) observed data structure as in Section 1.3, including the strong positivity assumption and the bounds on A and Y . For every policy π ∈ P, define F π : R → R pointwise by F π (m) ≡ E [P (Y ≤ m|A = π(X), X)]. Under some causal assumptions, F π is the cumulative distribution function of the reward in the counterfactual world where action π(x) is taken in each context x ∈ X .

Remark 11 .

 11 1. A slight modification to Lemma A.7 is needed. The main regularity condition consists in assuming that the real-valued function π → Ḟπ (V(π)) over Π equipped with • is uniformly continuous, see Corollary B.9 in Appendix B. It is tedious, though not difficult, to complement this main regularity condition with secondary conditions. It would suffice to restrict the (uniform in π) behavior for all P {Y ≤ v|A = π(x), X = x} across all real v. One can, for example, establish conditions under which the estimating equationbased estimator defined as a solution in v to 1{a = π(x)} 1{Y ≤ v} -P {Y ≤ v|A = π(X), X} P (A = a|X = x) + P {Y ≤ v|A = π(X), X} = 1/2,

E P [Z( 1 ) 2 Remark 13 .

 1213 |X] and E P [Z(-1)|X] in some strata of X . For any g : {-1, 1} × X → [0, 1], let P g be the distribution of O defined by (i) sampling (X, Z(-1), Z(1)) from P, (ii) sampling A from the Rademacher law with parameter g(1, X), (iii) setting O ≡ (X, A, Z(A)). Note that W ≡ g(A, X) is a deterministic function of g and O. Therefore, we can augment O with W , i.e., substitute (O, W ) for O, while still denoting (O, W ) ∼ P g . A balanced design corresponds to P b , where b : {-1, 1} × X → [0, 1] only takes the value 1/2. 2

  1). (A.8) By the • -uniform continuity of z, |z(f πr )z(f πr )| = o r (1) if f πrf πr = o r (1). Let us show that the latter condition is met. Because πr ∈ B Π (π ⋆ r , r), the triangle inequality, (A.6) and (A.11) imply thatf πrf πr ≤ f πrf π ⋆ r + f π ⋆ rf πr π rπ ⋆ r + π ⋆ rπr ≤ sup π∈Πr inf π ⋆ ∈Π ⋆ π ⋆π + 2r. (A.9)Because Π r ⊂ {π ∈ Π : R(π) ≤ r}, Lemma A.4 (which applies because Π is totally bounded)⋆ ππ ⋆ = 0, from which we deduce that the RHS of (A.9) is o r (1). In summary, f πr -f πr = o r (1), hence |z(f πr )z(f πr )| = o r (1)and consequently, by (A.8),sup π∈Πr z(f π )lim sup s↓0 sup π ⋆ ∈Π ⋆ inf π∈B Π (π ⋆ ,s) z(f π ) = o r (1).Taking r = r m and using the above result reveals that the RHS expression in (A.4) is o(1) indeed. This completes the proof of Lemma A.5.In the context of Corollary A.6, let {(z m , r m )} m≥1 be a sequence with values in ℓ ∞ (F ) × [0, ∞) and such that sup f ∈F |z m (f )z(f )| + |r m | → 0. Lemma A.5 implies that h(z m , r m ) → h(z, 0) ≡ 0. Since the sequence was arbitrarily chosen, h is indeed continuous at (z, 0) and the proof of Corollary A.6 is complete.

11 )

 11 Introduce φ : R 5 → R given by 2φ(u) = u 1 |u 2 + u 3 | + (1 + u 3 )u 4 + (1 + u 3 )u 5 and f 1 , f 2 , f 4 , f 5 be the function given by f 1 (o) ≡ (y -Q(a, x))/P (A = a|X = x), f 2 (o) ≡ x, f 4 (o) ≡ Q(1, x) and f 5 (o) ≡ Q(-1, x). Let F 1 , F 2 , F 4 , F 5 be the singletons {f 1 }, {f 2 }, {f 4 }and {f 5 }, each of them a Donsker class. Let F ≡ F 1 × F 2 × Π × F 4 × F 5 and note that

  a∈A θ(π(X)a)Q(a, X)for every policy π : X → A, yielding|V(π 1 ) -V(π 2 )| π 1π 2 (B.17)

  a∈A θ(π(X)a) θ(Aa) Y -Q(A, X) P (A|X) + Q(a, X) -V(π) (B.18)

F π 2

 2 and[START_REF] Van Der Laan | Unified methods for censored longitudinal data and causality[END_REF] imply the existence of m ∈ [m, V(π 2 )] such that |F π 2 (V(π 2 )) -F π 2 (m)| = |V(π 2 ) -m| × Ḟπ 2 ( m) ≥ |V(π 2 ) -m| × inf π∈Π inf m∈[V(π)±c]

Lemma B. 8 . 1 ǫ

 81 Suppose the existence of a deterministic L > 0 such that, for all m ∈ R and sufficiently small ǫ > 0, with P -probability one,P (m < Y ≤ m + ǫ|X) ≤ Lǫ.Then, for all m ∈ R andπ 1 , π 2 ∈ Π, | Ḟπ 2 (m) -Ḟπ 1 (m)| π 1π 2 .Proof of Lemma B.8. Set arbitrarily π 1 , π 2 ∈ Π and m ∈ R. In this proof, the universal positive multiplicative constants attached to the -inequalities do not depend on π 1 , π 2 , m.First, observe that Ḟπ 2 (m) -Ḟπ 1 (m) = lim ǫ↓0 |F π 2 (m + ǫ) -F π 2 (m) -F π 1 (m + ǫ) + F π 1 (m)| .

  both for double robustness and for efficiency gains. Above P (A|X) and E[Y |A, X]

	are estimators of P (A|X) and E[Y |A, X]. One can use cross-validation to avoid dependence
	on empirical process conditions for the estimators of P (A|X) and E[Y |A, X].	2

  Let g be defined as in Lemma A.5. Note that the second term in (A.[START_REF] Van Der | Asymptotic statistics[END_REF] does not depend on π n . Let {t n } n≥1 be a sequence with positive values such that t n ↓ 0. As π n trivially falls in Π R( πn)+tn , we can take a supremum over π ∈ Π R( πn)+tn . Multiplying both sides of (A.14) by n 1/2 , we see that

). (A.

[START_REF] Van Der | Asymptotic statistics[END_REF]

  ).The proof concludes by noting that[START_REF] Sheehy | Uniform Donsker classes of functions[END_REF] includes the condition that almost all sample paths of G P are uniformly continuous on F with respect to • , and thus the right-hand side aboveis o P (1). In conclusion R( π n ) = o P (r -1 n ).Proof of Lemma 6. If the bounded class Π is Donsker, then it is totally bounded in L 2 (P ), so (8) is met. By Lemma A.1, Π is then compact. Let θ be a Lipschitz function from [-2, 2] to [0, 1] such that θ(0) = 1 and θ(u) = 0 if |u| ≥ min a =a ′ ∈A |aa ′ |. Introducing θ is merely a trick to generalize Lemma A.2. In particular, (A.1) becomes

	B.2 Proof for Section 4.2

Q(π(X), X) =

† The Rademacher law with parameter p ∈ [0, 1] is the law of A ∈ {-1, 1} such that A = 1 with probability p.

Acknowledgements

Alex Luedtke gratefully acknowledges the support of the New Development Fund of the Fred Hutchinson Cancer Research Center. Antoine Chambaz acknowledges the support of the French Agence Nationale de la Recherche (ANR), under grant ANR-13-BS01-0005 (project SPADRO) and that this research has been conducted as part of the project Labex MME-DII (ANR11-LBX-0023-01).

Appendix A Proof of Main ERM Result

We begin this section with five lemmas and a corollary, whose proofs only require results from functional analysis. We then prove Lemma A.7 and Theorem 1, using results from empirical process theory.

In Lemmas A.1 through A. [START_REF] Rubin | Statistical issues and limitations in personalized medicine research with clinical trials[END_REF] to follow, all topological results make use of the strong topology on L 2 (P ). A set S ⊂ L 2 (P ) is called totally bounded if, for every ǫ > 0, there exists a finite collection of radius-ǫ open balls that covers S. When we refer to convergence in these lemmas, we refer to convergence with respect to • . A set S is sequentially compact if every sequence of elements of S has a convergent subsequence. A set S ⊂ L 2 (P ) is compact if every open cover has a finite subcover. Sequential compactness and compactness are equivalent for subsets of metric spaces. Lemma A.5 and the proof of Theorem 1 use several additional definitions, which are given immediately before Lemma A.5.

Lemma A.1. If Π is totally bounded, then Π is compact.

Proof. The Hilbert space L 2 (P ) is a complete metric space. Therefore, [START_REF] Browder | Mathematical analysis: an introduction[END_REF]Corollary 6.65] implies that Π is relatively compact. In other words, Π is compact.

For brevity, introduce

Lemma A.2. Both V and R are continuous on Π.

Proof. The key to this proof is the following simple remark: for every policy π :

Choose arbitrarily π 1 , π 2 ∈ Π. By (A.1) and the Cauchy-Schwarz inequality, it holds that