Audrey Astori

Gabriel Matherat

Isabelle Munoz

Emilie-Fleur Gautier

Didier Surdez

Yaël Zermati

Frédérique Verdier

Sakina Zaidi

Vincent Feuillet

Amir Kadi

Evelyne Lauret

Oliver Delattre

Carine Lefèvre

Michaela Fontenay

Evelyne Ségal-Bendirdjian

Isabelle Dusanter-Fourt

Didier Bouscary

Olivier Hermine

Patrick Mayeux

Ph.D Frédéric Pendino
email: frederic.pendino@inserm.fr

Oliver Delattre

Olivier Delattre

The epigenetic regulator RINF

Keywords:

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

In earlier work, we demonstrated that RINF loss-of-function affects granulopoiesis in normal and tumoral human hematopoietic cells (1). In keeping with its gene location at chromosome 5q31.2 (Supplementary Figure S1), a commonly deleted region associated with high risk in myeloid neoplasms (2), RINF loss of expression has been proposed to contribute to development or progression of (pre)leukemia, such as MDS (1,(3)(4)(5). We and others have demonstrated that RINF mRNA expression is an unfavorable (6,7) and independent (3) prognostic factor in acute myeloid leukemia (AML) as well as in solid tumors (8,9). However, its role in normal hematopoiesis has hitherto been poorly investigated and its contribution to the erythroid lineage and RBC expansion is unknown.

RINF contains a Nuclear Localization Signal (NLS) that has been functionally validated (10) and in most studies, its subcellular localization is reported to be mainly or exclusively nuclear and to act as a transcriptional cofactor (1,3,8,(10)(11)(12)(13)(14)(15)(16)(17). RINF associates strongly with chromatin (1) through its conserved zinc-finger domain (CXXC) that plays an essential role in providing the capacity to bind CpG islands (18,[START_REF] Long | ZF-CxxC domain-containing proteins, CpG islands and the chromatin connection[END_REF]. Interestingly, this domain is almost identical to the one harbored by TET1 and TET3, two epigenetic modulators involved in the erasure of DNA-methylation marks [START_REF] Melamed | Tet Enzymes, Variants, and Differential Effects on Function[END_REF], pointing to the possibility that RINF might interfere with TET-activities, hydroxymethylation, and gene transcription, as recently demonstrated in mouse (12,15). RINF has also been reported to bind ATM, mediate DNA-damage inducedactivation of TP53 (3,10) and inhibit the WNT-β-catenin signaling pathway (3,[START_REF] Andersson | CXXC5 is a novel BMP4-regulated modulator of Wnt signaling in neural stem cells[END_REF][START_REF] Kim | A novel Wilms tumor 1 (WT1) target gene negatively regulates the WNT signaling pathway[END_REF][START_REF] Kim | CXXC5 is a negative-feedback regulator of the Wnt/betacatenin pathway involved in osteoblast differentiation[END_REF][START_REF] Lee | The Dishevelled-binding protein CXXC5 negatively regulates cutaneous wound healing[END_REF] through a cytoplasmic interaction with Disheveled proteins DVL and DVL2 [START_REF] Andersson | CXXC5 is a novel BMP4-regulated modulator of Wnt signaling in neural stem cells[END_REF].

Transforming growth factor β (TGFβ) is a powerful and widespread cell growth inhibitor in numerous mammalian tissues [START_REF] Pardali | Actions of TGF-beta as tumor suppressor and pro-metastatic factor in human cancer[END_REF][START_REF] Siegel | Cytostatic and apoptotic actions of TGF-beta in homeostasis and cancer[END_REF]. In the hematopoietic system, TGFβ is known to regulate hematopoietic stem and progenitor cells (HSPCs) and is also described as a potent inducer of erythroid differentiation and inhibitor of cell proliferation [START_REF] Zermati | TGF-beta1 drives and accelerates erythroid differentiation in the epo-dependent UT-7 cell line even in the absence of erythropoietin[END_REF][START_REF] Blank | TGF-beta signaling in the control of hematopoietic stem cells[END_REF][START_REF] Krystal | Transforming growth factor beta 1 is an inducer of erythroid differentiation[END_REF][START_REF] Ruscetti | Autocrine transforming growth factor-beta regulation of hematopoiesis: many outcomes that depend on the context[END_REF]. TGFβ signals through cell surface serine/threonine kinase receptors, mainly TGFβRI and TGFβRII. Activated TGFβRI phosphorylates SMAD2 and SMAD3 that translocate into the nucleus and form complexes that regulate transcription of target genes. TGFβ can also elicit its biological effects by activation of SMAD-independent pathways [START_REF] Derynck | Smad-dependent and Smad-independent pathways in TGF-beta family signalling[END_REF][START_REF] Shi | Mechanisms of TGF-beta signaling from cell membrane to the nucleus[END_REF]. Inhibitory SMADs (SMAD6 and SMAD7) inhibit TGFβ signaling. Importantly, a reduced SMAD7 expression sensitizes cells to the antiproliferative effects of TGFβ and contributes to anemia in patients suffering from myelodysplastic syndrome (MDS), suggesting, firstly, that identifying transcriptional regulators of SMAD7 could enlighten our understanding of erythropoiesis and, second, that inhibiting TGFβ signaling could be a therapeutic strategy that would mitigate ineffective hematopoiesis in disease states [START_REF] Bhagat | miR-21 mediates hematopoietic suppression in MDS by activating TGF-beta signaling[END_REF][START_REF] Zhou | Reduced SMAD7 leads to overactivation of TGF-beta signaling in MDS that can be reversed by a specific inhibitor of TGF-beta receptor I kinase[END_REF][START_REF] Zhou | Inhibition of the TGF-beta receptor I kinase promotes hematopoiesis in MDS[END_REF].

In the present work, we use primary human CD34+ cells to demonstrate that RINF knockdown affects human erythropoiesis and mitigates RBC production through a mechanism that is mediated by SMAD7, the main inhibitor of TGFβ signaling.

Methods

Primary culture of hematopoietic cells. Mononuclear cells were isolated from cord blood (CRB Saint-Louis Hospital, Paris) or adult bone marrow donors with written informed consent for research use, in accordance with the Declaration of Helsinki.

The ethics evaluation committee of Inserm, the Institutional Review Board (IRB00003888), approved our research project (N°16-319). Mononucleated cells were separated by Ficoll-Paque (Life Technologies) and purified with human CD34 MicroBead Kit (Cat.N°130100453, Miltenyi Biotec). A two-step culture method for cell expansion and erythroid differentiation was used to obtain highly stage-enriched erythroblast populations (36, 37). Briefly, CD34 + cells (purity 94-98%) were cultured 7 days with IL6, IL3 (10 ng/mL), and SCF (100 ng/mL) to expand hematopoietic progenitors (37). CD36 + erythroid progenitors were purified using magnetic microbeads (CD36 FA6.152 from Beckman Coulter and anti-mouse IgG1 MicroBeads from Miltenyi Biotec). Then, erythropoietin (EPO) was added for 10-18 days to allow erythroid differentiation. For Colony-Forming Cells (CFC, counted at 11-14 days) assays, CD34 + hematopoietic stem cells were seeded in methylcellulose (H4034, StemCell Technologies). TGFβRI inhibitor SB431542 was from Selleckchem (Cat.N°S1067) (38). TGFβ was from Peprotech (Cat.N°100-21).

Flow Cytometry and differentiation analysis.

To monitor end-stage erythroid differentiation of primary cells, cells were labeled with anti-human PE/Cy7-conjugated glycophorin-A (GPA/CD235a) (Cat.N°A71564), PE-conjugated CD71 (IM2001U), and APC-conjugated CD49d (Cat.N°B01682) from Beckman Coulter, PE-conjugated CD233/Band3 from IBGRL (Cat.N°9439-PE). FACS (fluorescence-activated cell sorting) analysis was performed on an Accuri™ C6 Flow Cytometer (Becton-Dickinson). For morphological characterization cells were cytospun and stained with May-Grünwald-Giemsa method (MGG). For benzidine assay, 2.10 5 cells were incubated for 5 minutes with 0.01% benzidine and H 2 0 2 at 0.3% (Sigma-Aldrich).

Culture of cell lines and treatments. K562 (ATCC#CCL243

) and UT7 5.3 (39) cells were cultured in RPMI 1640 medium and minimum essential medium (MEMα) supplemented with 2.5 ng/mL of GM-CSF (Myltenyi Biotech), respectively. Both medium were supplemented with 10% of inactivated fetal bovine serum (FBS), 2 mM L-Glutamine, 50 units/ml penicillin G and 50 µg/mL streptomycin (Life Technologies).

To induce hemoglobin production, K562 cells were treated with hemin at 40 µM (Sigma-Aldrich). Erythroid maturation of UT7 5.3 was triggered by replacing GM-CSF with EPO 5U/ml.

Lentiviral and Retroviral transduction of hematopoietic cells.

For shRNAmediated RINF knockdown, we used the pTRIPDU3/GFP lentiviral vector (40) which drives the same shRNA sequences downstream of H1 promoter than our previously described pLKO.1/puroR vector (1). For RINF overexpression, the retroviral vector MigR/IRES-GFP was used with experimental conditions as previously described (1).

Green fluorescent protein (GFP) sorting was performed on a FACSAriaIII (BD Biosciences). For SMAD7 overexpression, the retroviral vector pBABE-puro-SMAD7-HA (Addgene plasmid#37044) was used as previously described (1,41). For doxycycline-inducible lentiviral expression, SMAD7-HA cDNA was inserted in pINDUCER21 vector by gateway technology (Addgene plasmid #46948) and shRNA sequences were inserted in the Tet-pLKO-GFP vector (modified from Tet-pLKO-puro, Addgene plasmid#21915, deposited by Dmitri Wiederschain).

Quantitative RT-PCR. Cells were collected and directly stored at -80°C for RNA preparation with TRIzol (Life technologies) extraction protocol as previously described (1). First-strand cDNA synthesis (RT) were carried out using Transcriptor First Strand cDNA Synthesis Kit (Cat.N°489703000, Roche). RINF mRNA expression was detected using Lightcycler® 480 ProbesMaster kit (Cat.N°4707494001, Roche).

Relative mRNA expressions were normalized to RLPL2, PPIA or ACTB gene expression in a two-color duplex reaction. For SMAD7, PU.1 and c-KIT mRNAs detection, quantitative PCRs were performed using SYBRGreen. Q-RT-PCR were performed on a Light Cycler 480 machine (Roche) and gene expression calculated by with the 2-ΔΔCT method. Primer sequences are available in a supplementary file.

Western-blot analysis. Total cell extracts and western blotting were carried out as previously described (1). Blots were incubated with primary polyclonal antibody and then with appropriate peroxidase-conjugated secondary antibody (anti-rabbit and anti-mouse IgG HRP antibodies from Cell Signaling, Cat.N°7074S, Cat.N°7076S) and anti-goat IgG HRP antibody from Southern Biotech (Cat.N°6160-05). Detection of proteins was performed using chemiluminescent detection system (clarity western ECL, Cat.N°170-5060, Biorad). Rabbit polyclonal antibodies were previously described for RINF (1), P85/PI3KR1 (42), or commercially purchased for anti-RINF, anti-SMAD7 (Cat.N°16513-1-AP, Cat.N°25840-1-AP, ProteinTech), anti-β-ACTIN (ACTB, Cat.N°A1978, Sigma-Aldrich), anti-phospho-SMAD2/3 (Cat.N°8828, Cell Signaling), and anti-HSC70 mouse monoclonal antibody (SC-7298, Santa Cruz Biotechnology).

Statistical

Results

RINF is expressed during early human erythropoiesis. We employed cytokineinduced erythroid differentiation of CD34+ progenitors isolated from human cord blood or adult bone marrow to model human erythropoiesis in vitro. To obtain highly enriched populations of differentiating erythroblasts, cells were grown in a two-phase liquid culture method (experimental design in Figure 1A) as previously described (36, 37). We found that RINF protein was present at the earliest stages of erythroid differentiation and reached its highest level in progenitors and proerythroblasts (ProE) (Figure 1B). After that, RINF protein level decreased in the basophilic erythroblast stage and was barely detectable in late erythroblasts (i.e. polychromatic and orthochromatic erythroblast stages). Similarly, RINF mRNA levels were high in erythroid progenitors but decreased at the basophilic stage and onwards (Figure 1C). B). We found that RINF was extinguished as early as two days posttransduction (i.e. at day 0 of EPO) with shRINF#4 at both mRNA (Figure 2B) and protein (Figure 2C) levels, without affecting cell viability (assessed by trypan blue cell counting, Figure 2D) or cell growth up to day 10 of EPO (as illustrated on Figure 2E, 2F).

This temporal expression pattern mirrors

left
Concomitantly with the size reduction of large BFU-Es (most immature ones), we noticed a slight but statistically significant increase in small-BFU-Es (most mature ones), in agreement with an accelerated maturation (Figure 2G).

Erythroid maturation is impacted by RINF. We next investigated if the reduced expansion observed from day 10 of EPO could be the consequence of an accelerated maturation. Indeed, we observed that erythroid differentiation was accelerated under conditions of RINF knockdown using benzidine staining quantification (Figure 3A), morphological analysis of MGG-stained cells (Figure 3B), and flow cytometry for erythroid markers GPA and CD71 (Figure 3C, upper panel), and CD49d and Band3 (Figure 3C, lower panel). These differences were statistically significant (p<0.001) for both cord blood and adult bone marrow derived CD34+ cells.

The shRNA-RINF-induced acceleration of erythropoiesis was also noticed by cell size reduction (not shown) and by a more rapid downregulation of c-KIT and PU.1 mRNAs in RINF knockdown cells (Figure 3D). The accelerated maturation was also characterized by a pronounced reduction of ProE cells (∼2.8-fold less) enumerated at day 11 (Figure 3B). Despite a weaker efficiency of shRINF#3 to knockdown RINF expression compared to shRINF#4 (Supplementary Figure S3B andFigure3E, left panel), we also confirmed that a similar phenotype (i.e. accelerated maturation (as noticed by an increased benzidine staining, Figure 3E, right panel) and a reduction in the total number of RBC produced at day 17, Figure 3F) was obtained with another shRNA sequence targeting RINF.

RINF controls erythroid maturation and RBC expansion in a TGFβ-dependent manner. We then wondered if TGFβ, a well-known inducer of erythroid maturation and inhibitor of RBC expansion, could be involved in the RINF-dependent phenotype.

In agreement with an important role for autocrine TGFβ signaling in the early stages of erythropoiesis, both TGFB1 and TGFBRI were highly expressed in erythroid progenitors (Figure 4A). To functionally validate an involvement of the TGFβ signaling pathway during RINF knockdown dependent acceleration of erythropoiesis, we performed liquid cell culture experiments of primary CD34+ cells in the presence of SB431542, a potent and selective inhibitor of TGFβRI (38) (Figure 4B). Strikingly, in the presence of SB431542, the maturation of RINF knockdown cells (evidenced by a faster acquisition of erythroid cell surface markers such as CD49d/Band3 at day 10 of EPO) was not accelerated and SMAD2 protein was not phosphorylated/activated after 3 days of EPO (Figure 4C, lanes 5 and 6), supporting the efficiency of the inhibitor. Conversely, in absence of inhibitor, RINF-silencing gives rise to a faster phosphorylation rate of SMAD2 protein, here noticed at 8 hours of EPO (compare lanes 1 and 2, or the kinetics of pSMAD2 waves in both conditions, right panel), indicating a higher sensitivity to autocrine TGFβ-signaling that would be mediated through SMAD2, as previously reported in hematopoietic cells [START_REF] Zhou | Inhibition of the TGF-beta receptor I kinase promotes hematopoiesis in MDS[END_REF]. A weaker RBC expansion was also noticed for RINF knockdown cells, even when TGFβ1 (5ng/mL) was added as late as 11 days of EPO treatment (Figure 4D). Moreover, even for the two donors where efficiency of RINF silencing was more subtle (donors 1 and 2, Figure 2E), RINF knockdown cells were more sensitive than control cells to TGFβ1 (Figures 4E-F). This was especially pronounced at low-doses (0.1 ng/mL), were we found a faster acquisition of GPA (at day 6 of EPO) and a reduced RBC production at 17 days of EPO (Figure 4F). Taken together, these data suggest that RINF knockdown cells were more sensitive to TGFβ and that the level of RINF in erythroid progenitors impacts the upcoming RBC production, at least in presence of TGFβ.

Identification of SMAD7 as a RINF target gene candidate. To identify molecular mechanisms by which RINF might modulate TGFβ-responsiveness, we reanalyzed our previously published gene expression microarray datasets of K562 and UT7 5.3 cells transduced with shRINF#4 or shControl (7). A relatively discrete set of 193 gene candidates (Supplementary Table S1) were downregulated by RINF knockdown in both cell lines. Strikingly, ingenuity pathway analyses (IPA) revealed that this gene list was enriched in genes belonging to the TGFβ signaling pathway (Supplementary

Figure S4C

). Considering its well-known inhibitory function on TGFβ-signaling and its low expression in MDS, SMAD7 appeared as a promising candidate for functional investigation [START_REF] Bhagat | miR-21 mediates hematopoietic suppression in MDS by activating TGF-beta signaling[END_REF][START_REF] Zhou | Reduced SMAD7 leads to overactivation of TGF-beta signaling in MDS that can be reversed by a specific inhibitor of TGF-beta receptor I kinase[END_REF]. Encouragingly, SMAD7 was also downregulated by RINF knockdown in a third hematopoietic cell line, MV4-11 (Supplementary Figure S4E, right panel) and the downregulation of SMAD7 was confirmed by quantitative RT-PCR analyses in K562 cells, even though the knockdown appeared moderate (~45%, p<0.001) in these independent experiments (Supplementary Figure S4D, right panel).

We next measured RINF levels in two cell line models of erythroid maturation, the K562 cell line treated with hemin and the UT7 5.3 cell line treated with EPO (39)

after GM-CSF withdrawal (Supplementary Figures S4A-B). In agreement with our findings in primary cells, RINF expression was downregulated at the protein level upon treatment-induced erythroid differentiation, as early as after ten hours of treatment with hemin for K562 cells, and after two days with EPO for UT7 5.3 cells (Supplementary Figure S4A). Moreover, RINF knockdown accelerated the erythroid maturation program (Supplementary Figure S4B) triggered by hemin in K562 cells (noticed after one day of treatment), or EPO in UT7 5.3 cells (at four days of treatment). We also asked if RINF overexpression could delay hemoglobinization. To this end, we used the retroviral system MigR/IRES-GFP vector (1) (Supplementary Figure S5A, upper panel) to driving ectopic expression from full-length RINF cDNA.

We found high constitutive expression of RINF at the protein level for both cell lines (Supplementary Figure S5A), without cell toxicity or consequences on cell proliferation (not shown). When we evaluated erythroid differentiation using benzidine staining (Supplementary Figure S5B) we found that RINF overexpression reduced hemoglobinization in both cell lines (p<0.001), consistent with our knockdown experiments (Figures S4A-B).

RINF directly controls SMAD7 transcription. Our q-RT-PCR analyses demonstrated a robust increase (approximately five-fold) of SMAD7 mRNA levels in cells overexpressing RINF (Figure 5A). To investigate if RINF-mediated induction of SMAD7 mRNA was direct and occurred at the transcriptional level, we performed

ChIP-experiments with anti-RINF antibodies. To this end, five sets of primers were designed for regions encompassing the SMAD7 promoter (Figure 5B). As shown in Supplementary Figure S5C, we found that RINF binds directly to several regions (R1 to R4) of the SMAD7 promoter and that this binding was increased in cells overexpressing RINF. Moreover, H3K4me3 histone marks (quantified by ChIP-qPCR, Figure 5C) were increased by RINF overexpression in four proximal regions (R2-R4)

of the SMAD7 transcription start site, indicating that chromatin was in a more active transcriptional state.

To investigate functional relationships between RINF and SMAD7, we next performed "rescue" experiments in K562 cells. We employed retroviral transduction to express SMAD7 ectopically (Figure 5D). Our data demonstrated that SMAD7 overexpression did not affect endogenous expression of RINF mRNA or protein (Figures 5E-F), but reduced hemoglobinization (Figure 5G, lane 3), mimicking the effect of ectopic RINF (Supplementary Figure S5B). We did not detect any change in cell viability or proliferation in the absence of hemin. Importantly, under conditions when SMAD7 was ectopically expressed, RINF knockdown did not result in acceleration of hemoglobinization (Figure 5G). Taken together, these data indicate that RINF directly binds SMAD7 promoter to control its expression, and that SMAD7 is an effector downstream of RINF during erythroid maturation.

RINF and SMAD7 mRNA expression are correlated in bone marrow from adult donors and patients with myelodysplastic syndrome. We asked if RINF and SMAD7 mRNA expression would correlate in primary human CD34+ cells from healthy donors and donors with myelodysplastic syndrome. To this end, we performed q-RT-PCR on CD34+ cells isolated from adult bone marrow donors (n=11) and found a highly significant correlation between RINF mRNA and SMAD7 mRNA (Pearson, rho=0.684, p=0.02) (Figure 6A). We confirmed these data by analyzing an independent microarray dataset performed by Pellagatti et al. [START_REF] Pellagatti | Gene expression profiles of CD34+ cells in myelodysplastic syndromes: involvement of interferon-stimulated genes and correlation to FAB subtype and karyotype[END_REF]. As shown in Figure 6B, RINF and SMAD7 mRNA levels strongly correlated in CD34+ cells isolated from healthy donors (Pearson's rho=0.784, p<0.001, n=17). Interestingly, this correlation also reached statistical significance in CD34+ cells from MDS-5q patients (Pearson's rho=0.603, p<0.001, n=47) but not in other MDS (without del(5q) or 5q-). We then compared the intensity of this correlation with the other 5q genes (n=48 genes) from the CDR associated with high-risk, and the CXXC5 probe sets were the one correlating the best with SMAD7 probe set in both the healthy control group (rho=0.819, and rho=0.784) and MDS patients' cohort with del(5q) (Supplementary Figure S6A). Reinforcing the previously reported relevance of SMAD7 in MDS pathophysiology [START_REF] Bhagat | miR-21 mediates hematopoietic suppression in MDS by activating TGF-beta signaling[END_REF][START_REF] Zhou | Reduced SMAD7 leads to overactivation of TGF-beta signaling in MDS that can be reversed by a specific inhibitor of TGF-beta receptor I kinase[END_REF], SMAD7 expression was not only extinguished in CD34+ MDS samples compared to normal samples (in the microarray dataset from Gerstung et al. [START_REF] Gerstung | Combining gene mutation with gene expression data improves outcome prediction in myelodysplastic syndromes[END_REF], Supplementary Figure S6C), but patients with the lowest SMAD7 expression have a significantly shorter overall survival (Supplementary Figure S6D).

RINF-silencing alters genome-wide hydroxymethylation. Since murine Rinf has recently been described as a necessary platform for Tet2 activity (15,[START_REF] Ravichandran | Rinf Regulates Pluripotency Network Genes and Tet Enzymes in Embryonic Stem Cells[END_REF], we wondered if RINF-silencing could impact the genome-wide 5hmC of human immature erythroid cells. As shown on Figure 6C, we observed a statistically significant loss of 5hmC detected by immunofluorescence or flow cytometry (Figure6D) in knockdown primary cells.

A RINF/SMAD7 axis controls erythroid maturation and RBC expansion in primary cells. We next asked whether RINF knockdown would alter SMAD7 expression in primary erythroid progenitors. Indeed, as shown in Figures 7A andB, a 60-70% knockdown of RINF led to a statistically significant knockdown of SMAD7 (by 40-45%) in CD34 positive cells isolated from cord blood (n=3 donors, paired-T-test student p<0.001). Accordingly, a 30% knockdown of SMAD7 protein level was detected by immunofluorescence in primary erythroid progenitors (p<0.0001) (Figure 7C). To directly demonstrate the relevance of this finely tuned regulation of SMAD7 mediated by RINF protein in primary HSPCs, we next executed experiments in which we knocked down RINF and induced SMAD7 expression ectopically. To this end, we first transduced CD34+ cells with the doxycycline-inducible vector pInducer21/SMAD7-HA (Figure 7D), and then with pTRIPUD3 vector that drives shRNA/RINF or shRNA/Control expression. Notably, in the presence of doxycycline, SMAD7 expression (Figures 7E-F) robustly prevented both the RINF knockdown dependent accelerated maturation (Figure 7G) and reduction of RBC numbers (Figure 7H).

Discussion

An increasing body of work suggest that RINF is involved in the maturation, function, or development of various cell types such as neural stem cells [START_REF] Andersson | CXXC5 is a novel BMP4-regulated modulator of Wnt signaling in neural stem cells[END_REF], endothelial cells [START_REF] Kim | CXXC5 is a transcriptional activator of Flk-1 and mediates bone morphogenic protein-induced endothelial cell differentiation and vessel formation[END_REF], myoblasts (14), myofibroblasts [START_REF] Lee | The Dishevelled-binding protein CXXC5 negatively regulates cutaneous wound healing[END_REF], osteoblasts [START_REF] Kim | CXXC5 is a negative-feedback regulator of the Wnt/betacatenin pathway involved in osteoblast differentiation[END_REF], as well as in kidney development [START_REF] Kim | A novel Wilms tumor 1 (WT1) target gene negatively regulates the WNT signaling pathway[END_REF], wound healing [START_REF] Lee | The Dishevelled-binding protein CXXC5 negatively regulates cutaneous wound healing[END_REF], and hair-regrowth [START_REF] Lee | Targeting of CXXC5 by a Competing Peptide Stimulates Hair Regrowth and Wound-Induced Hair Neogenesis[END_REF]. Here, we report that the epigenetic factor RINF is a transcriptional regulator of SMAD7 that fine-tunes TGFβ sensitivity of erythroid progenitors, findings that bring insight into molecular barriers that may prevent effective erythropoiesis.

Our [START_REF] Carrancio | An activin receptor IIA ligand trap promotes erythropoiesis resulting in a rapid induction of red blood cells and haemoglobin[END_REF][START_REF] Bewersdorf | Transforming growth factor (TGF)-beta pathway as a therapeutic target in lower risk myelodysplastic syndromes[END_REF][START_REF] Fenaux | Luspatercept for the treatment of anemia in myelodysplastic syndromes and primary myelofibrosis[END_REF][START_REF] Suragani | Modified activin receptor IIB ligand trap mitigates ineffective erythropoiesis and disease complications in murine beta-thalassemia[END_REF]. In this manner, RINF could be one of the players regulating the dynamic balance between long-term expansion of the erythroid pool versus fast production of red blood cells for immediate physiological needs.

Two recent studies suggest that murine Rinf could act as an anchoring platform necessary for Tet2 (that lacks CXXC domain) and its activity at CpG sites (5'-hydroxymethylation), in plasmacytoid dendritic cells (pDCs) (15) and mouse embryonic stem cells (mESCs) [START_REF] Ravichandran | Rinf Regulates Pluripotency Network Genes and Tet Enzymes in Embryonic Stem Cells[END_REF]. Conversely, an earlier study indicate that Rinf inhibits Tet2 and hydroxymethylation of genomic DNA in a caspase-dependent manner (in mESCs), and another one indicates that RINF binds to unmethylated (and not methylated) CpGs ( 18), probably reflecting a more complex mechanism of action (12). We here provide the first experimental evidence (at our knowledge) that RINF can regulate genome-wide hydroxymethylation in human cells. The role for RINF that we describe here could pave the way to future studies that will be necessary to better understand the complex molecular epigenetic events that occur during normal and ineffective erythropoiesis (12,15,[START_REF] Ravichandran | Rinf Regulates Pluripotency Network Genes and Tet Enzymes in Embryonic Stem Cells[END_REF][START_REF] Lio | Dysregulation of the TET family of epigenetic regulators in hematopoietic malignancies[END_REF] and that may involve TET enzymes [START_REF] Yan | Distinct roles for TET family proteins in regulating human erythropoiesis[END_REF].

Several correlative studies have implicated RINF as a candidate in pathogenesis of MDS/AML (1, 3, 5, 60). MDS is a complex and heterogeneous malignant hemopathy in which the loss of several genes is suspected to lead to the pathophysiology (4). CXXC5 is located at chromosome 5q31. S7).

Our findings point to a molecular mechanism in which RINF functions as a negative modulator of TGFβ signaling by upregulating and maintaining SMAD7.

Consequently, RINF loss of expression will sensitize human erythroid progenitors to autocrine/paracrine TGFβ-induced growth inhibition as a mechanism to prevent expansion of the erythroid pool. Our data also suggest that this signaling would be at least partly mediated through SMAD2 activation and phosphorylation (Figure 4C), in agreement with previous report in human bone marrow [START_REF] Zhou | Inhibition of the TGF-beta receptor I kinase promotes hematopoiesis in MDS[END_REF]. The in vivo relevance of these findings is strengthened by correlated RINF and SMAD7 mRNA expression in human BM CD34+ cells. This correlation was also observed in MDS patients with del(5q) or 5q-but not in other MDS, in which SMAD7 was silenced in most of patient samples, whatever the RINF expression level. This extinction of SMAD7 in other MDS could be explained by various mechanisms such as miRNA21 expression [START_REF] Bhagat | miR-21 mediates hematopoietic suppression in MDS by activating TGF-beta signaling[END_REF].

However, since RINF knockdown leads to global loss of 5hmC, it is tempting to speculate that the loss of RINF could lead to hypermethylation of SMAD7 promoter in HSPCs, a process that could have gone unnoticed in previous studies. Interestingly, SMAD7 promoter is known to be silenced by hypermethylation in non-hematopoietic tissues, and hypomethylating agents such as azacytidine (AZA) can revert this hypermethylation and alleviates TGFβ-induced diseases in several pathological models such as atherosclerosis [START_REF] Wei | SMAD7 methylation as a novel marker in atherosclerosis[END_REF], as well as renal-( 62) and liver-fibrosis [START_REF] Bian | Repression of Smad7 mediated by DNMT1 determines hepatic stellate cell activation and liver fibrosis in rats[END_REF].

Thus, further studies should investigate whether SMAD7 promoter is demethylated after treatment with hypomethylating agents (AZA or decitabine) in humans HSPCs isolated from MDS patients, and to what extend this mechanism could contribute to the efficacy of these epigenetic drugs to sustain erythropoiesis on the long-term way.

Supporting this hypothesis, TET2 mutation, is a good indicator of response to azacytidine treatment [START_REF] Itzykson | Impact of TET2 mutations on response rate to azacitidine in myelodysplastic syndromes and low blast count acute myeloid leukemias[END_REF].

Previous studies have underscored the importance of SMAD7 regulation in MDS [START_REF] Bhagat | miR-21 mediates hematopoietic suppression in MDS by activating TGF-beta signaling[END_REF][START_REF] Zhou | Reduced SMAD7 leads to overactivation of TGF-beta signaling in MDS that can be reversed by a specific inhibitor of TGF-beta receptor I kinase[END_REF]. Our present data complement this work by demonstrating, firstly, a direct transcriptional mechanism by which SMAD7 is regulated in hematopoietic cells and, second, that the residual level of SMAD7 in MDS is associated with the severity of the disease and patient survival (Supplementary Figure S6D). These data are particularly relevant in the context of understanding how TGFβ responsiveness in the bone marrow microenvironment is fine-tuned [START_REF] Blank | TGF-beta signaling in the control of hematopoietic stem cells[END_REF], and in a context-specific manner [START_REF] Ruscetti | Autocrine transforming growth factor-beta regulation of hematopoiesis: many outcomes that depend on the context[END_REF]. Even though this mechanism is unlikely to be solely responsible for TGFβhypersensitivity, our findings pave the way for novel research avenues in hemopathies characterized by ineffective erythropoiesis such as Myelofibrosis (65),

Fanconi Anemia (66), β-Thalassemia (57, 67), or MDS [START_REF] Bewersdorf | Transforming growth factor (TGF)-beta pathway as a therapeutic target in lower risk myelodysplastic syndromes[END_REF], and for which novel therapies based on TGFβ-inhibitors are particularly promising [START_REF] Fenaux | Luspatercept for the treatment of anemia in myelodysplastic syndromes and primary myelofibrosis[END_REF][START_REF] Verma | Biological basis for efficacy of activin receptor ligand traps in myelodysplastic syndromes[END_REF]. Finally, beyond its role during erythropoiesis, the RINF/SMAD7 regulation axis here described, could at least partly contribute to the pleiotropic effects of RINF described during wound healing [START_REF] Lee | The Dishevelled-binding protein CXXC5 negatively regulates cutaneous wound healing[END_REF], fibrosis [START_REF] Wei | SMAD7 methylation as a novel marker in atherosclerosis[END_REF][START_REF] Yang | Alcohol promotes renal fibrosis by activating Nox2/4mediated DNA methylation of Smad7[END_REF][START_REF] Bian | Repression of Smad7 mediated by DNMT1 determines hepatic stellate cell activation and liver fibrosis in rats[END_REF][START_REF] Cheng | CXXC5 Attenuates Pulmonary Fibrosis in a Bleomycin-Induced Mouse Model and MLFs by Suppression of the CD40/CD40L Pathway[END_REF], immunity (15), and tumor biology (1,3,(6)(7)(8)(9), and deserves to be investigated in these physiological processes. [START_REF] Pellagatti | Gene expression profiles of CD34+ cells in myelodysplastic syndromes: involvement of interferon-stimulated genes and correlation to FAB subtype and karyotype[END_REF] For each gene, a Pearson's correlation coefficient was calculated by comparing its expression to the one of SMAD7, and the list was ranked by descending order. Cell growth was also monitored, and the total number of RBC numerated at day 17 of EPO are presented on histograms (lower panel). cells were isolated from adult bone marrow (n=11). RINF and SMAD7 mRNA levels were measured by q-RT-PCR. In order to assess whether the expression of these two genes correlated in donors' samples, a correlation analysis was performed with the Pearson's (for non-parametric analysis) correlation coefficient method: rho (r= 0.684), P(two-tailed)<0.05 (B) RINF and SMAD7 gene expression data were extracted from a previously described microarray dataset performed by Pellagatti et al. [START_REF] Pellagatti | Gene expression profiles of CD34+ cells in myelodysplastic syndromes: involvement of interferon-stimulated genes and correlation to FAB subtype and karyotype[END_REF]. In this study, CD34+ cells were isolated from adult bone marrow (n=17). For mRNA detection, the 233955_x_at and 204790_at probesets were used for CXXC5 and SMAD7, respectively. A correlation analysis was performed with the Pearson's rho (for non-parametric analysis). A strong correlation coefficient was noticed for normal healthy controls (rho = 0.784), and MDS patients with del(5q), (rho=0.603), that where both highly significant P (two-tailed)<0.001. (C) CD34+ cells isolated from cord blood or healthy adult donors were transduced with pTripizi vector expressing a shRNA targeting RINF expression (shRNA/RINF#4) or a "non target" shRNA (shRNA/Control). A couple of days later (during the amplification phase, see

Primer sequences used for quantitative RT-PCR and ChIP-qPCR

Human PPIA (Cyclophilin A) endogenous control was obtained from Thermo Fisher Scientific (Cat.N°4333763T). The other primer sequences (5'-3' orientation) used for RT-qPCR and ChIP-qPCR are indicated in the following table.

Detection of SMAD7 protein or 5hmC by immunofluorescence.

CD34+ cells were washed once (5 minutes in ice-cold PBS 1X), and fixed in PFA 2% (8 minutes at room temperature), permeabilized with Triton X100 0.1% (5 minutes at 4°C), incubated 1 hour in blocking buffer (PBS 1X, 5% Fetal Bovine Serum, 2% BSA) before overnight staining with primary mouse monoclonal anti-SMAD7 antibody (1:200, Santa-Cruz, #sc-365846). For 5hmC detection, two additional steps were performed after the permeabilization step, one DNA denaturation step in 3M HCl (5 minutes at RT) followed by a five-minute neutralization step in TE buffer (10 mM Tris, 1 mM EDTA) at pH8. The primary anti-5hmC polyclonal rabbit antibody (1:2000) was purchased from Active motif (Cat.N°39769). Two secondary antibodies (1:1.000) coupled with AlexaFluor 647 fluorescent dye were purchased from Lifetechnologies, a goat antirabbit antibody (Cat # A32733) for 5hmC detection and a goat anti-mouse antibody (Cat # A32723) for SMAD7 detection. Images of 5hmC and SMAD7 staining were acquired on a wide-field Nikon Eclipse TE2000 microscope through a ×20 objective, with a Cascade CDD camera (Photometrics). The quantification of intensity was performed after background subtraction (Fiji). The fluorescent intensity was measured in 100-120 cells for the two conditions.

Cell culture with AML cell lines

Human leukemic cell lines were purchased from DSMZ (MV4-11; Braunschweig, Germany) and from the American Type Culture Collection (K562; Molsheim, France). UT7 5.3 cells were kindly provided by Isabelle Dusanter-Fourt (Cochin Institute, Paris, France). K562 and MV4-11 were cultured in RPMI 1640 medium supplemented with 10% fetal calf serum (FCS), 2 mM L-Glutamine, 50 U/ml penicillin G and 50 µg/ml streptomycin (Life Technologies, Saint-Aubin, France). UT7 5.3 cells were cultured in minimum essential medium (MEM) α medium containing 10% of FCS, 2 mM L-Glutamine, 50 U/ml penicillin G and 50 µg/ml streptomycin (Life Technologies, Saint-Aubin, France) and 2,5 ng/µl of GM-CSF (Miltenyi Biotec, France). Cells were grown in mycoplasma-free conditions, regularly tested.

Microarray and data analysis

Analysis of global gene expression profiles. After validation of RNA quality with Bioanalyzer 2100 (using Agilent RNA6000 nano chip kit), 400 ng of total RNA was reverse transcribed following the Genechip WT plus Reagent kit (Affymetrix). Briefly, the resulting double strand cDNA was used for in vitro transcription with T7 RNA pol.

After purification, 15 µg of cRNA was used for reverse transcription with random primers. The cDNA obtained was then purified and fragmented. After control of fragmentation using Bioanalyzer 2100, cDNA was end labeled with biotin using Terminal Transferase (using the WT terminal labeling kit of Affymetrix). cDNA was then hybridized to GeneChip® Human Transcriptome Analysis 2.0 (Affymetrix) at 45°C for 17 hours. After overnight hybridization, chips were washed on the fluidic station FS450 following specific protocols (Affymetrix) and scanned using the GCS3000 7G. The image was then analyzed with Expression Console software (Affymetrix) to obtain raw data (cel files) and metrics for Quality Controls. Minimum information about a microarray experiment (MIAME)-compliant documentation of the microarray experiments have been deposited in Gene Expression Omnibus under the accession number GSE140770.

The Affymetrix HTA2 dataset analysis was performed by GenoSplice technology (www.genosplice.com). Data were normalized using quantile normalization. Background corrections were made with antigenomic probes and probes were selected according to their %GC, cross-hybridization status and potential overlap with repeat region as previously described (2,3). Only probes targeting exons and exon-exon junctions annotated from FAST DB® transcripts (release fastdb_2014_1) were selected (4,5). Only genes expressed in at least one compared condition were analyzed. To be considered to be expressed, the DABG P-value had to be ≤0.05 for at least half of the gene probes. We performed an unpaired Student's ttest to compare gene intensities between shRINF and shControl cells. Genes were considered significantly regulated when fold-change was ≥1.5, a fold-change noticed for CXXC5 genes for which we have validated the knock-down by q-RT-PCR and western-blot analysis. cytokines (IL3, SCF, G-CSF, GM-CSF, and EPO) enabling the formation of erythroid colonies (both large and small BFU-Es), myeloid colonies (CFU-G, CFU-M, or CFU-GM), or mixed colonies (CFU-GEMM). Colony Forming Cells were counted at 14 days of culture. (A) Schematic representation of the pTRIPDU3 (lower panel) lentiviral vector used to express a short hairpin RNA. The specificity and the efficacy of the sequences were previously described (11). The pTRIPDU3 vector drives the expression of the shRNA under a H1 promoter and the GFP expression gene under an EF1 promoter. After two days of expansion in (IL3, SCF, IL6), human CD34+ cells were transduced with the pTRIPDU3 vector. Of note, pTRIPDU3-transduced cells were that detects a specific band at 33 kDa in total extracts. ACTIN (ACTB) was used as a loading control. (B) For hemoglobin production analysis, the benzidine test was used.

"

  analyses. All analyses were performed using GraphPad Prism software. When distribution was normal, the 2-tailed Student t test was used for group comparisons. Contingency tables were performed by using the Fisher's exact test, and Pearson's correlation coefficient was used to determine the correlation between the normally distributed RINF mRNA and SMAD7 mRNA expressions. Statistics were carried out on a minimum of 3 independent experiments. Significance of P values are indicated in Figures. Not significant (ns): p>0.05; *: p<0.05; **: p<0.01; ***: p<0.001.Error bars represent confidence intervals at 95% or standard deviations (SD) for q-Immunofluorescence", "Chromatin Immunoprecipitation (ChIP) experiments", and "Primer sequences used for quantitative RT-PCR and ChIP-qPCR" are described in the Supplemental Method section.

  RINF expression data extracted from three independent transcriptome datasets (microarray or RNA-seq) performed by Merryweather-Clarke et al. (43), An et al. (44), and Keller et al. (45) with adults or cord-blood CD34+ cells (Supplementary Figure S2). Taken together, our data shows that RINF expression peaks in erythroid progenitors and proerythroblasts during cytokine-induced erythropoiesis. RINF silencing results in diminished RBC production without impacting cell viability. We next investigated consequences of RINF knockdown on erythropoiesis and cell viability (see experimental design in Figure 2A). Knockdown experiments were performed with two previously validated short-hairpin RNAs (shRINF#4 and shRINF#3) sequences (1) driven by the lentiviral pTRIPDU3/GFP vector (Figures S3A and

Figure 1 .

 1 Figure 1. RINF is expressed in early erythropoiesis and downregulated during maturation of human HSPCs. (A) Schematic representation of cell culture experiments. CD34+ cells (from cord blood or adult bone marrow) were first amplified for 7 days (D-7 to D0, in black) in presence of SCF, IL3 and IL6. Cells were sorted for CD36 expression and EPO was added to the medium in order to trigger the latestage of erythropoiesis, a phase lasting 11-17 days (in red). (B) For morphological characterization, cells were cytospun and stained with May-Grünwald-Giemsa (MGG) and the repartition of cells among differentiation stages indicated in percentage (right panel): early progenitor (Prog), proerythroblast (ProE), early and late basophilic erythroblast (Baso), polychromatophilic erythroblast (Poly), orthochromatic erythroblast (Ortho), and Reticulocyte (Retic). Scale bar represents 20 µm. The experiment presented here is representative of four distinct experiments which were performed on cells isolated from human cord blood. RINF protein was detected by western-blot analysis (lower panel). HSC70 protein was used as a loading control. For each lane, 40 µg of protein was loaded. Bands intensities were quantified with the Multigauge Software and the relative expression of RINF protein (normalized to the loading control) is indicated above the image in percentage of Day 3. (C) CXXC5 mRNA expression was measured by quantitative RT-PCR. CXXC5 mRNA expression is downregulated from the basophilic erythroblast stage. Here, CD34+cells were isolated from adult human bone marrow, but a similar expression profile was observed with cells isolated from cord blood.

Figure 2 .

 2 Figure 2. RINF knockdown leads to a reduction in total number of red blood cells generated from human HSPCs. (A) Schematic representation of shRNAmediated RINF loss of expression experiment. CD34+ cells isolated from cord-blood were transduced at day 5 of the amplification step with the lentiviral vector pTRIPDU3/GFP expressing a shRNA targeting RINF mRNA expression (shRNA/RINF) or a non-target sequence (shRNA/Control). GFP-positive cells were sorted by FACS two days after transduction. At that time, cell culture kinetics with less than 80% of CD36+ cells or more than 5% of cell death (trypan blue) were not pursued. EPO was added after sorting and maintained for 17-18 days. (B) RINF

Figure 3 .

 3 Figure 3. RINF silencing accelerates erythroid maturation of human primary erythroblasts. Erythroid differentiation was monitored for six separate experiments performed with CD34+ cells isolated either from cord-blood (n=3) or adult bone marrow (n=3), by using different techniques. Despite a donor-to-donor variability in the maturation kinetics, we consistently observed an accelerated maturation for RINF knockdown conditions. Of note, data presented on Figures 3B and 3C correspond to two distinct donors. (A) Hemoglobin production of primary cells was estimated by benzidine assay. CD34+ cells isolated from cord-blood were transduced with the lentiviral vector pTRIPDU3/GFP expressing a shRNA targeting RINF mRNA expression (shRNA/RINF) or a non-target sequence (shRNA/Control). GFP-positive cells were sorted by FACS two days after transduction. The black arrow at day 6 indicates the kinetic point selected for one representative picture on the right panel. (B) Morphological analysis has been performed after May-Grünwald-Giemsa (MGG) staining. In order to know whether the observed acceleration of maturation was statistically significant, a Fisher's exact test was performed. For this, enumerated erythroid cells (n=454) were segregated into either early (Progenitors, ProE, Baso) or late (Polychro, Ortho, Retic) group. P value <0.0001. (C) Erythroid maturation was also monitored by flow cytometry by using anti-CD71 and anti-glycophorin-A (GPA/CD235a) labelled antibodies (upper panel) or using anti-CD49d and anti-Band3 antibodies (lower panel). Scattergrams of two representative experiments are shown (n=6), indicating a more mature population of cells for shRNA-RINF conditions either at day 6 of EPO (GPA positive, upper panel), or at day 10 of EPO (Band3 positive, lower panel). (D) RINF loss of expression in human CD34+ is also associated with an accelerated downregulation of cKIT and PU.1. Histogram of relative mRNA expression determined by quantitative RT-PCR for RINF (left panel), cKIT (middle panel) and PU.1 (right panel). PU.1 and c-KIT mRNA are known to be downregulated upon erythroid maturation and are here used as molecular markers. In RINF knockdown cells (black bars), the downregulation of c-KIT and PU.1 mRNA preceded of at least three days the one observed in control cells, in agreement with an

Figure 4 .

 4 Figure 4. RINF knockdown-induced phenotype is TGFβ-dependent in human primary cells. (A) TGFβ ligand and TGFBRI expression profiles were extracted from our proteomic analyses of cord-blood derived CD34+ cells (Gauthier et al. (37)). TGFB1 and TGFBRI were highly expressed at the Prog1-Prog2 stage (i.e. D1-D3 of EPO), suggesting an important autocrine signaling in early steps of erythroid maturation in our serum-free culture conditions. (B) Cord-blood derived CD34+ cells were transduced at day 5 of the amplification step, GFP-sorted (2 days posttransduction), and EPO was added for 8 to 12 days, in presence of TGFβ inhibitor (SB431542, at 10µM) or vehicle control (DMSO). Histograms (left panels) represent FACS analyses of the whole cell suspension for CD49d and Band3 antigens (here at 10 days of EPO). In order to know whether the observed acceleration of maturation was statistically significant, student-T tests were performed. Scattergrams (right panel), presents one representative experiment out of three separate experiments performed. (C) RINF (left panel) and phosphorylated SMAD2 (right panel) protein levels were estimated at 8h and 72h of treatment with EPO by Western-Blot analysis (see Methods), on the same kinetics than the one presented on Figure 2C (pool of five cord-blood units). As expected, the specific band for phospho-SMAD2 (Ser465/467), indicated by an arrow, was not detected in presence of TGFβ inhibitor

Figure 5 .

 5 Figure 5. SMAD7 is a direct transcriptional target of RINF. (A) The MigR1 retroviral system was used to constitutively express RINF in K562 cell line. A few days after transduction, cells were sorted for GFP expression and relative expression of RINF and SMAD7 mRNA levels were measured by quantitative RT-PCR. (B) Schematic representation of SMAD7 gene structure (deduced from ENCODE database). CpG islands and primer set regions (R0 to R5) used for ChIP-experiments are also indicated as well as the distance to transcription start site (TSS) of SMAD7. (C) ChIP-qPCR bars indicate the percentage of SMAD7 promoter coimmunoprecipitated by anti-RINF (left panel) or anti-H3K4me3 (right panel) antibody in K562 cell line. (D) Schematic representation of the pBABE retroviral vector used for SMAD7 overexpression or empty cassette control. (E) Histograms represent the RINF or SMAD7 mRNA levels detected by quantitative RT-PCR. Values are

Figure 6 .

 6 Figure 6. RINF and SMAD7 mRNA closely correlate in primary human CD34+ bone marrow cells isolated from healthy donors or MDS patients with del(5q) and RINF-silencing leads to global loss of 5-hydroxymethylation. (A) CD34+

Figure1A), cells

  Figure1A), cells were GFP-sorted and 5hmC was detected by immunofluorescence (left panel) or flow cytometry (right panel) by using an anti-5hmC antibodies labeled with a fluorescent dye. The intensity of the labeling was also quantified by cell

Figure 7 .

 7 Figure 7. RINF knockdown-induced phenotype is SMAD7-dependent in human primary cells. (A) Histogram represents RINF (left panel) and SMAD7 (right panel) mRNA expression levels of shRNA-transduced CD34+ cells isolated from 3 cordblood units. In these specific experiments, lentiviral transduction was performed after magnetic sorting of CD34+ cells (i.e. Day 0 of the amplification step) and cells were GFP-sorted between 2 and 5 days of the amplification step. Data shown are means ±SD for each donor (q-RT-PCR performed in triplicates). To assess whether the mRNA level was statistically different between shRNA-RINF and shRNA-Scramble samples, a student T test was performed for each donor. Considering the three biological replicates, both RINF and SMAD7 mRNA levels were statistically downregulated by shRNA-RINF (Paired Student T test, p<0.001). (B) This histogram represents the level of SMAD7 mRNA obtained with the two sequences targeting RINF (shRINF#3 or shRINF#4) and detected by quantitative RT-PCR. Values are expressed in percentage of the shControl condition. RINF mRNA levels can be visualized on Supplementary Figure S3B. (C) Since detection of SMAD7 protein level requires a large quantity of cells for western blot analysis and the active endogenous level of this protein can be below the detection threshold, SMAD7 protein levels was estimated by immunofluorescence. Images of stained cells were acquired on a widefield Nikon Eclipse microscope through a ×20 objective, with an Eclipse TE2000, a cascade CDD camera (Photometrics). Images of SMAD7 had to be acquired with a binning of one, and two images were averaged for each condition. For cell imaging analysis the quantification of the labeling, performed on two images after background subtraction, the mean fluorescent intensity was measured in 100-120 cells for the two conditions by using Fiji. Representative images from three independent experiments are shown. Scale bar = 50 µm. (D) Schematic representation of the pInducer21/SMAD7 lentiviral vector used for rescue experiments. This vector drives the expression of SMAD7/HA under a Tetracycline Response Element (TRE)promoter that is activated after doxycycline treatment. To perform rescue experiments, cord-blood derived CD34+ cells (n=3 donors) were first transduced with the pInducer21/SMAD7 lentiviral vector, sorted for a low GFP expression, and

FACS-sorted

  for GFP expression at two days after transduction. (B) This histogram represents the level of RINF mRNA obtained with the two sequences targeting RINF (shRINF#3 or shRINF#4) and detected by quantitative RT-PCR. Values are expressed in percentage of the shControl condition. (student T-test, * p< 0.05) (C) CD34+ cells were isolated from bone marrow of healthy donors (n=4). The histogram represents the total number of colonies enumerated (left panel) and the relative proportion of each type of colony (right panel). For each of the four donors, the proportions are also visualized by pairs of red dots. (student T-test, * p< 0.05) (D) CD34+ were isolated from bone marrow of healthy donors (n=4, left panel) or cord-blood (n=6, right panel). Histograms give the mean number of small and large BFU-E-derived colonies. The error bars indicate donor-to-donor variability for each vector (± SEM). The trend was not significant (student t-test). (E) Histogram represents the number of BFU-E-derived colonies enumerated from 1000 MEP (Megakaryocyte-Erythroid Progenitor)-enriched seeded cells (right panel). Data shown are means ± SD (n=3). The increased number of BFU-Es enumerated in the shRNA-RINF condition was statistically significant (student T-test, * p< 0.05). The gating strategy used for MEP-enrichment is shown (right panel). The protocol is adapted from Manz et al.(1) For this, CD34+ cells were isolated from primary human bone marrow and stained with CD34, CD38, CD10, CD123 and CD45RA (see Supplemental Methods). We have verified that these MEP cells only generate erythroid colonies (BFU-Es) in our experimental conditions.

Figure S4 .

 S4 Figure S4. RINF silencing alters the TGF signaling pathway in K562 and UT75.3 cells and accelerates cell hemoglobinization triggered by hemin and EPO, respectively. The pTRIPDU3 lentiviral system was used to constitutively express a

Figure S5 .

 S5 Figure S5. Ectopic overexpression of RINF delays hemoglobinization of human hematopoietic cell lines K562 and UT75.3 triggered by hemin and EPO, respectively. The MigR1 retroviral system (schematized in upper panel A) was used to constitutively express RINF in two cell lines. A few days after transduction, cells were sorted for GFP expression. Cells were then respectively treated with hemin (40µM) for K562 or EPO (5UI/ml) for UT75.3. (A) The relative expression of RINF was measured by Western-Blot analysis using a polyclonal rabbit antibody (see Methods)

Table 1 .

 1

	GENE NAME Probeset	Pearson's (correl coeff.)	Gene location at 5q
	CXXC5	222996_s_at	0,819	High-Risk CDR
	CXXC5	233955_x_at	0,784	High-Risk CDR
	HSPA9	200690_at	0,729	High-Risk CDR
	CXXC5	224516_s_at	0,705	High-Risk CDR
	ETF1	201574_at	0,640	High-Risk CDR
	IL17B	220273_at	0,554	Low-Risk CDR
	SPARC	212667_at	0,532	Low-Risk CDR
	ETF1	201573_s_at	0,508	High-Risk CDR
	MATR3	238993_at	0,452	High-Risk CDR
	CD74	1567627_at	0,424	Low-Risk CDR
	TGFBI	201506_at	0,422	High-Risk CDR
	PDGFRB	202273_at	0,391	Low-Risk CDR
	EGR1	201694_s_at	0,373	High-Risk CDR
	SIL1	218436_at	0,355	High-Risk CDR
	PSD2	223536_at	0,352	High-Risk CDR
	UBE2D2	201345_s_at	0,350	High-Risk CDR
	AFAP1L1	1555542_at	0,343	Low-Risk CDR
	SH3TC2	233561_at	0,337	Low-Risk CDR
	CSF1R	203104_at	0,335	Low-Risk CDR
	TMEM173	224916_at	0,312	High-Risk CDR
	PAIP2	222983_s_at	0,289	High-Risk CDR
	HNRNPA0	229083_at	0,271	High-Risk CDR
	CDX1	206430_at	0,259	Low-Risk CDR
	EGR1	201693_s_at	0,214	High-Risk CDR
	FAM53C	218023_s_at	0,193	High-Risk CDR
	AFAP1L1	226955_at	0,184	Low-Risk CDR
	MZB1	223565_at	0,180	High-Risk CDR
	CARMN	231987_at	0,180	Low-Risk CDR
	CDC23	223651_x_at	0,178	High-Risk CDR
	CTNNA1	200764_s_at	0,176	High-Risk CDR
	REEP2	205331_s_at	0,167	High-Risk CDR
	SPATA24	238027_at	0,137	High-Risk CDR
	RPS14	208645_s_at	0,134	Low-Risk CDR
	LRRTM2	206408_at	0,128	High-Risk CDR
	NRG2	242303_at	0,122	High-Risk CDR
	GFRA3	229936_at	0,114	High-Risk CDR
	BRD8	242265_at	0,096	High-Risk CDR
	CDC25C	216914_at	0,083	High-Risk CDR
	SH3TC2	240966_at	0,052	Low-Risk CDR
	HNRNPA0	201055_s_at	0,041	High-Risk CDR
	KDM3B	210878_s_at	0,041	High-Risk CDR
	GRPEL2	238427_at	0,040	Low-Risk CDR

Table 1 . List of genes located at 5q and their correlation with SMAD7 expression in primary human CD34+ isolated from bone marrow of healthy

 1

	donors (n=17). All the genes were selected because they are located within one of
	the two Commonly Deleted Region (CDRs) at chromosome 5q. Their mRNA
	expression level was extracted from GDS3795 microarray study performed by
	Pellagatti et al.

Target sequences of the shRNAs used to target RINF knockdown shRNA name Target mRNA Sequence shControl

	Q-RT-PCR			Probe sequence (5'-
	(Hydrolysis)	Forward	Reverse	3')
				FAM-
				aacccaaagctgccctctc
	RINF	TCCGCTGCTCTGGAGAAG	CACACGAGCAGTGACATTGC	c-BBQ
				Cy5-
				agctgaatggaaaaaaca
	RPLP2	GACCGGCTCAACAAGGTTAT	CCCCACCAGCAGGTACAC	ttgaagacgtc-BBQ
	Q-RT-PCR
	(SybrGreen)	Forward	Reverse
	cKIT	TATGCTCTCGCACCTTTCCA	TCTCAATGAAGTGCCCCTGA
	PU1	CACTGGAGGTGTCTGACGG	GCGGATCTTCTTCTTGCTGC
	SMAD7	CAATGACCACGAGTTTATGCA	GTTGAAGATGACCTCTAGCCA
	Murine Rinf	TCCGCTGCTCTGGAGAAG	CACGCGGGCAGTGACATTGC
	murineSmad7	TGCTGTGCAAAGTGTTCAGGTG CCATCGGGTATCTGGAGTAAGGA
	murineGapdh	TGTGTCCGTCGTCGATCTGA	TTGCTGTTG AAGTCGCAGGAG
	ChIP-qPCR			Distance to SMAD7
	(SybrGreen)	Forward	Reverse	TSS
	SMAD7 R0
	(neg.)	ATGGGTGTTTCAAGAGTGGAG	GAAGAGGAAAGCCGTGTAGAG	-4.7kb
	SMAD7 R1	CTAGCGCTTCATTCATTGGTTT	TCGGTCCAGTCCGGTATAA	-1962-1840
	SMAD7 R2	GCAAACAACAGATCGGGTTTC	CAATCCATTCTGGGAGCTTCT	-1695 -1556
	SMAD7 R3	GCCTCGGCTTCCTACATGGA	CTCCCCACCCCCAAATTAAG	-243 -122
	SMAD7 R4	CACAGCCTCTTGACTTCCGAG	TCGTCCTACTCGCTCCCCCT	+737 +819
	SMAD7-R5
	(neg.)	CATCCCATCTATTCTCACGCC	AGACAGCCCCTTAAAGTTAGC	+23.7kb
		CCUGGGCAAAGAAUGGACAAU
	shRINF#4	GAAUGGACAAUCAGUUUCCUU
	shRINF#3	CCUUUGAUUCUUUCCGACCAU

Acknowledgements

This work was supported by INSERM, Paris-Descartes University, Ligue Nationale Contre le Cancer (LNCC), Cochin Institute, and the Laboratory of Excellence GR-EX. AA was supported by LNCC, Société Française d'Hématologie, Fondation pour la Recherche Médicale, and Boehringer Society (travel grant). GM was supported by fellowship grant from the Ministère de l'Enseignement supérieur et de la Recherche and Société Française d'Hématologie. E.S-B is supported by the National Center for Scientific Research (CNRS) and her team was supported by LNCC and Fondation de France. We are also grateful for the support from the French-Norwegian exchange program (Aurora to F.P.). Authors thanks Pr. Jérôme Larghero and Thomas Domet from Centre de Ressources Biologiques / Banque de Sang de Cordon de l'AP-HP (Saint-Louis Hospital) for providing cord blood units, Anne Dubart-Kupperschmitt for the pTRIPDU3/GFP vector (40), Johan Lillehaug, Øystein Bruserud, Camille Humbert, Pierre de la Grange (Genosplice), Eric Nguyen, Sabrina Bondu, Alexandre Artus, Amandine Houvert, Nadège Bercovici, Emmanuel Donnadieu, Franck Letourneur, Brigitte Izac, Sebastien Jacques, and Alain Trautmann for technical assistance and scientific discussions, as well as Life Science Editors for reading and editing the manuscript. We are grateful to the staff of the IMAG'IC, CYBIO, and GENOM'IC facilities of the Cochin Institute.

Authorship Contributions

Conflicts of Interest Disclosures

The authors declare no potential conflict of interest. Frederic Pendino is the main coinventor of patents describing methods for RINF mRNA detection (WO2009/151337 and WO2012/010661).

SUPPLEMENTAL METHODS

Flow cytometric cell sorting of MEP cells

Freshly isolated CD34+ were labeled with the following labeled antibodies directed against the following markers: CD34 APC-Cy7 (clone 581), CD123 APC (6H6) from BioLegend; CD38 PE (clone HIT2) and CD45RA-FITC (HI100) from BD Biosciences, CD10 PE-Cy5 (eBioscience; clone CD-CALLA). Cells were sorted using BD FACSAriaIII (BD Biosciences) according to the gating strategy described by Manz et al. (1) Purification was obtained to at least 98% and only erythroid colonies were generated in CFC assay (i.e. no G-, M-, or GM-derived colony were noticed).

Chromatin Immunoprecipitation (ChIP) experiments

ChIP was performed with anti-RINF (ProteinTech, Cat.N°16513-1-AP) or anti-H3K4me3 (Diagenode, Cat.N°C15410003), antibodies using iDeal ChIP-seq Kit for Transcription Factors (Diagenode) following manufacturer's instructions and using the following options: K562 cells were crosslinked with 1% formaldehyde for 10 minutes in fresh media at room temperature and quenched with 0.2 M glycine for 5 additional minutes. Chromatin was sonicated using Diagenode Bioruptor (Power High, 20 cycles of 30''on-30''off and by changing ice cold water in the bath every 5 cycles). Sonication was validated by agarose gel electrophoresis (average DNA size ~150-250 bp) using an aliquot of purified sheared chromatin (proteinase K treated followed by and phenolchloroform extraction). Quantitative-PCR was performed using Power SYBR Green PCR Master Mix (Thermo Fisher Scientific, Cat.N°4367659) using pairs of primers encompassing six regions (R0-R5) of SMAD7 gene. R0 and R5 regions were used as negative controls. The efficiency of chromatin immunoprecipitation at these loci was determined as the recovery of that locus calculated as the percentage of the input (the relative amount of immunoprecipitated DNA compared to input DNA). The amount used for the input was 1% of the amount used for ChIP, the recovery was calculated as follows: % recovery = 2^(Ct input -Ct sample ). S1.

Supplementary Table

LEGENDS TO SUPPLEMENTAL FIGURES

. The CXXC5/RINF gene is located at chromosome 5q31.2 within the CDR associated with higher risk in MDS and AML. Genes, as well as sequence tag site markers, are listed in the order of their physical positions (megabase, Mb) along chromosome 5q from centromere (upper) to telomere (bottom), according to the most recent human genome assembly at http://ensembl.org (Human, GRCh38.p12). CXXC5

gene is located at less than 20kb from UBE2D2, the more distal gene of the high-risk CDR (grey) investigated by Liu et al. in 2007 (6). According to a SNP-array study published in 2012 by Jerez et al. (7), and according to the GRCH37/hg19 genome assembly version (released in 2009) that was probably used for this study, the CXXC5 gene is located inside the refined-CDR associated with high-risk.

Supplemental References


Ramona Merhej
email: merhejramona@gmail.com

Mohamed Chetouani
email: mohamed.chetouani@sorbonne-universite.fr

LIEF: Learning to Influence through Evaluative Feedback

Keywords: Multi-Agent Reinforcement Learning, Evaluative Feedback, Opponent shaping, Opponent modelling, Learning to Influence, Optimising for the Future

We present a multi-agent reinforcement learning framework where rewards are not only generated by the environment but also by other peers in it through inter-agent evaluative feedback. We show that our method allows agents to effectively learn how to use this feedback channel to influence their peers' goals and move from independent or conflicting objectives to more coinciding and concurring ones. We advance with this in the field of cooperative reinforcement learning; not by aiding agents in increasing performance in an already cooperative environment, but by giving them a tool to transform antagonistic environments into cooperative ones.

INTRODUCTION

In Reinforcement Learning (RL) an agent learns a task through trialand-error from feedback received from its environment. Generally, depending on the state and action of the agent, a feedback in the form of a positive or negative reward is attributed. This feedback, defined by a reward function, "is the most succinct, robust, and transferable definition of the task" [START_REF] Andrew Y Ng | Policy invariance under reward transformations: Theory and application to reward shaping[END_REF]. In Multi-Agent Reinforcement Learning (MARL), several agents interact with each other and with the environment to solve a common or disparate tasks (i.e., they have a common or disparate reward functions). In the general case, defined as a Markov Game, both the reward function of an agent as well as the transition function of the environment, can have explicit dependencies on the joint action of all agents. The expected return of an agent depends on the policy it follows as well as the policies of all its peers [START_REF] Nowé | Game theory and multi-agent reinforcement learning[END_REF]. We note however that, although agents influence the performance of one another, they do not alter the structure of the reward function itself.

Additionally, in the multi-agent setting, other more direct interaction mechanisms can be present between agents. Many MARL algorithms integrate and allow communication between the agents. Communication can take several forms. For example, agents may communicate by sending messages [START_REF] Jakob N Foerster | Learning to communicate with deep multi-agent reinforcement learning[END_REF], sharing intentions [START_REF] Kim | Communication in Multi-Agent Reinforcement Learning: Intention Sharing[END_REF] or experiences [START_REF] Christianos | Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning[END_REF], advising actions [START_REF] Omidshafiei | Learning to teach in cooperative multiagent reinforcement learning[END_REF] to one another etc. Integrating such feedback has shown to improve performance [START_REF] Christianos | Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning[END_REF][START_REF] Jakob N Foerster | Learning to communicate with deep multi-agent reinforcement learning[END_REF], speed up learning [START_REF] Christianos | Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning[END_REF][START_REF] Jakob N Foerster | Learning to communicate with deep multi-agent reinforcement learning[END_REF][START_REF] Omidshafiei | Learning to teach in cooperative multiagent reinforcement learning[END_REF], enhance coordination [START_REF] Kim | Communication in Multi-Agent Reinforcement Learning: Intention Sharing[END_REF][START_REF] Omidshafiei | Learning to teach in cooperative multiagent reinforcement learning[END_REF] etc. Nonetheless and again, in all of the above, this type of feedback does not alter other agents' task or reward function which remains defined by the environment only.

However, in real-life situations, punishments and remunerations are widely used. This additional reward function is generated by the humans and not the environment itself. It is particularly seen when the environment presents conflicting goals to its agents or incentives for exploitation. Consider as an example the taxing system, the healthcare system or other public goods. In all of these examples, the systems themselves are unstable and entice exploitation. Fines and sanctions are put in place by the state to modify the dynamics and make exploitation and fraud less desirable and encourage and stabilise socially beneficial behaviours. These measures incite people to diverge from what would have otherwise been their optimal strategy. Remunerations are also a mean of influencing others' behaviours. For instance, when the number of users sending funds over the bitcoin blockchain exceeds the number of transactions that can be processed, users encourage miners to validate their transaction by remunerating them with higher transaction fees. Likewise, some business owners use cash, store credit, discounts etc. to encourage their customers to recommend them to other friends. Remunerations also appear in personal relations, education or other similar interactions. Despite their effectiveness, both sanctions and remunerations are costly to implement and the benefits extracted from them need to outweigh their costs.

While cooperative behaviours in an opponent can sometimes be encouraged without explicit evaluative feedback, e.g., using the Tit-for-Tat strategy [START_REF] Axelrod | Effective choice in the prisoner's dilemma[END_REF] for the iterated prisoner dilemma, such a retaliating strategy is not available or optimal for all games. Consider Public Good Games where agents contribute to a common pool and receive in return a shared reward proportional to the collected pool. Punishing a non-contributing agent by not contributing oneself results in punishment for everyone instead of a targeted one. Furthermore, in the bitcoin blockchain example, no retaliation policy exists for the user that encourages miners to validate his transaction. Additionally, in real-life situations, punishing or rewarding through action selection instead of feedback can be more challenging to implement. For instance, boycotting a company for animal testing is more challenging to implement than fining such policies (e.g., when the company holds a near-monopoly on the market). Evaluative feedback is a universal form of punishment and remuneration and has the advantage of being simultaneously easy to target and simple to implement.

In our work, we present a Multi-Agent Reinforcement Learning framework where agents learn to influence each other, not by action selection or communication, but directly using costly remunerations or punishments, that we denote by evaluative feedback. The framework is useful when agents have conflicting goals. Rewarding each other is a way of sharing preferences about the opponent's behaviours. Through mutual sharing of these preferences, we hope that agents find arrangements and compromises that allow mutual co-existence instead of mutual destruction. The new arrangements emerge from the reshaped goals (i.e., the reshaped reward functions) of every agent by its opponent. We propose Learning to Influence through Evaluative Feedback (LIEF), an algorithm designed to learn how to adequately use evaluative feedback to reshape an opponent's goal and transform it from a conflicting to a more conforming one.

We contribute to multi-agent reinforcement learning literature by proposing a framework that introduces inter-agent evaluative feedback on top of environment rewards. Using our model, agents learn to reshape each other's reward functions and goals to move from conflicting environments to conforming and cooperative ones. The novelty in our approach is that 1) opponent shaping is done through reward shaping, 2) the punitive feedback strategies used for opponent shaping are learned and not handcrafted and 3) no apriori knowledge of the optimal opponent target policy is needed.

RELATED WORK

The influence that agents can exercise on each other has been taken into account in some MARL algorithms where agents adapt their choices with respect to their opponent. Notably, the learning algorithm of an opponent is sometimes integrated in one's own learning algorithm. One suggested method is gradient ascent with policy prediction [START_REF] Zhang | Multi-agent learning with policy prediction[END_REF]. Here the strategy of the opponent is forecasted based on the current policy parameters and the gradient is computed taking this change into account. Learning with Opponent Learning Awareness (LOLA) [START_REF] Foerster | Learning with opponent-learning awareness[END_REF] on the other hand, differentiates through the variations of the opponent to actively shape their learning and consequently, manages to reach cooperative equilibria in some multi-agent settings. Stable Opponent Shaping (SOS) [START_REF] Letcher | Stable opponent shaping in differentiable games[END_REF] incorporates both policy prediction and opponent shaping which increases stability while simultaneously escaping saddle points.

While the above mentioned algorithms, try to influence a gradientbased learning opponent, Learning and Influencing Latent Intent (LILA) [START_REF] Xie | Learning Latent Representations to Influence Multi-Agent Interaction[END_REF] influences a handcrafted opponent that can select for every episode to implement one of several pre-defined policies. The set of pre-defined policies is small and comprises 2-3 policies. The selection of the policy by the opponent at the beginning of an episode is not random but depends on the previous interaction. LILA models and predicts the opponent's next policy choice and ensures that the current interaction results in a favourable opponent policy in the next episode.

In the context of influence for long term future benefits, Prognosticator [START_REF] Chandak | Optimizing for the future in non-stationary mdps[END_REF] is an algorithm that allows an agent in a non-stationary environment to forecast future performances and hence select to minimise performance in some episodes if that results in a future increase in performance. Here the future changes neither result from a learning opponent, nor from a conditionally changing one but from a smoothly varying environment.

All these enumerated works try to influence the dynamics of an opponent or an environment by selecting influential actions. On the other hand, in our work, we aim to influence an opponent's behaviour using directly rewards instead of regular actions.

Recovering or optimising a reward structure is at the center of interest of many RL problems. Inverse Reinforcement Learning (IRL) [START_REF] Andrew | Algorithms for inverse reinforcement learning[END_REF] for example, tries to recover the reward function that could be a prior for a given optimal policy or set of trajectories. AutoRL and Evolution Strategies (ES) have also been used to explore and evolve different reward functions with the goal of finding the one that facilitates the learning of a predefined task [START_REF] Hao-Tien | Learning navigation behaviors end-to-end with autorl[END_REF][START_REF] Faust | Evolving rewards to automate reinforcement learning[END_REF]. Moreover, in some adversarial RL problems [START_REF] Rakhsha | Policy teaching via environment poisoning: Training-time adversarial attacks against reinforcement learning[END_REF][START_REF] Zhang | Adaptive rewardpoisoning attacks against reinforcement learning[END_REF], an attacker aims to poisons a learner's reward function with the goal of enforcing a predefined target policy on that learner.

Yet several differences are notable between these works and ours. First, while we propose a reinforcement learning method to learn how to reward an opponent, the problem in the above examples is usually defined as a control problem and standard optimisation techniques are used to extract the optimal solution. Second, while the target policy is predefined for adversarial attacks and IRL, it is never explicitly given in our framework.

To the best of our knowledge, the only other work that introduces inter-agent evaluative feedback is a paper that studies the concept of gifting [START_REF] Lupu | Gifting in multi-agent reinforcement learning[END_REF] in common pool resource environments. The agents however, using the classical reinforcement learning objective of maximising episodic returns do not learn how to effectively gift their peers and stop using this action when gifting is costly.

DEFINITIONS AND NOTATIONS

Our work concerns general 𝑛-player stochastic games defined by a tuple ⟨S, A, 𝑃, 𝑟, 𝜇 0 ⟩. Here, S is the set of states, 𝜇 0 the initial state distribution and A = Π 𝑛 𝑖=1 A 𝑖 the Cartesian product of the sets of actions of all individual players. At every timestep, each agent selects from his set of actions an action 𝑎 𝑖 yielding a joint action a = {𝑎 1 , . . . 𝑎 𝑛 }. The system in state 𝑠, then transitions to 𝑠 ′ following the transition probability function 𝑃 (𝑠 ′ |𝑠, a) : S × A × S → [0, 1]. Finally, the reward function 𝑟 representative of the underlying tasks, evaluates and distributes for every player 𝑖, a reward according to 𝑟 𝑖 (𝑠, a, 𝑠 ′ ) : S × A × S → R.

We extend this setting to allow for inter-agent evaluative feedback. In this model, agents do not only receive rewards from their environment, but also directly from other peers. Therefore, we endow every agent with an additional set of evaluative actions U 𝑖 and we denote by U, the Cartesian product of these sets U = Π 𝑛 𝑖=1 U 𝑖 . We suppose that such evaluative actions can be costly. Every agent 𝑖, now incurs a cost based on his selected action 𝑢 𝑖 and receives evaluative feedback based on his peers' selected actions u ¬𝑖 . The costs are determined by a cost function 𝑐 such that 𝑐 𝑖 (𝑠, 𝑢 𝑖 , 𝑠 ′ ) : S × U 𝑖 × S → R and the peers' feedback is calculated by the feedback function 𝑓 where for every agent 𝑖 we have 𝑓 𝑖 (𝑠, 𝑢 ¬𝑖 , 𝑠 ′ ) : S × U ¬𝑖 × S → R.

The original stochastic game represented by the tuple ⟨S, A, 𝑃, 𝑟, 𝜇 0 ⟩ is now extended with an additional evaluative action set U and two reward functions 𝑐 and 𝑓 representing respectively the cost of evaluating others and the additional feedback received. The resulting dynamics are now described by the tuple ⟨S, A, U, 𝑃, 𝑟, 𝑐, 𝑓 , 𝜇 0 ⟩.

METHODS

To solve the described problem, we equip every agent with two decoupled policies with different objectives: the classical game policy and the feedback policy. The task or game policy of agent 𝑖, denoted by 𝜋 𝑔 𝑖 , maps into the action space A 𝑖 while the feedback policy 𝜋 𝑓 𝑖 maps into the action space U 𝑖 . The objective of the game policy is a classical RL objective i.e., solving a task by maximising the expected return within an episode. Note however, that the task, modelled by a reward function, is not only defined by the environment, but also generated by the agent's peers. The original task defined by 𝑟 is reshaped into the function 𝑟 + 𝑓 from the tuple ⟨S, A, U, 𝑃, 𝑟, 𝑐, 𝑓 , 𝜇 0 ⟩.

The objective of the feedback policy is of a different nature. While the game policy is concerned with maximising the returns within an episode, the feedback policy tries to influence its opponent to make future interactions more beneficial. We note that in a multiagent setting, the returns of one agent depend on all other agents' actions. Therefore, the performance of a fixed agent can vary if its opponent changes policies. We consider a future interaction to be more beneficial for an agent, if keeping his own game policy fixed, the opponents' game policy update results in higher payoffs for said agent. The algorithm we propose for this task is called Learning to Influence through Evaluative Feedback (LIEF). Figure 1 depicts the new extended game dynamics.

The Game Policy

In classical RL, an agent searches for an optimal policy to solve a task assigned to it by its environment and defined by a reward function. The game policy in our framework is analogous to the classical RL policy. It assumes a stationary world and maximises the expected episodic return. However, in our framework, feedback is received simultaneously from the environment and other agents. The expected episodic return, defined by 𝑟 + 𝑓 , is conditioned on all agents' game policies 𝝅 𝑔 (that we parametrise with 𝜃 𝑔 ) and on the opponents' feedback policies 𝝅 𝑓 ¬𝑖 (parametrised with 𝜃 𝑓 ). Given an episode horizon 𝑇 , agent 𝑖's game policy objective function is defined as

𝐽 𝑔 𝑖 (𝜃 𝑔 𝑖 ; 𝜽 𝒈 ¬𝒊 , 𝜽 𝒇 ¬𝒊 ) = E 𝜽 𝒈 ,𝜽 𝒇 ¬𝒊 𝑇 𝑡 =0 𝛾 𝑡 𝑔 (𝑟 𝑖,𝑡 + 𝑓 𝑖,𝑡 ) (1)
where 𝛾 𝑔 is a discount factor. Parameters 𝜃 𝑔 𝑖 are updated using the update rule

𝜃 𝑔,𝑘+1 𝑖 = 𝜃 𝑔,𝑘 𝑖 + 𝜂 𝑔 ∇ 𝜃 𝑔 𝑖 𝐽 𝑔 𝑖 (𝜃 𝑔 𝑖 ; 𝜽 𝒈 ¬𝒊 , 𝜽 𝒇 ) (2)
where 𝜂 𝑔 is the learning rate for updating the game policy and 𝑘 the optimisation step.

The Feedback Policy

On the other hand, the goal of the rewarding or feedback policy is to shape the objective of the opponent to make it more compatible with its own. To achieve this target, the feedback network tries to

where 𝛾 𝑟 and 𝛾 𝑐 are discount factors for the reward functions 𝑟 and 𝑐 respectively. While the second term can be estimated from experience, the first term corresponds to the future and needs to be predicted. Using first order Taylor expansion we can replace the first term of equation

; 𝜽 𝒈 ) = ∇ 𝜃 𝑓 𝑖 Δ𝜽 𝒈 ¬𝒊 𝑇 ∇ 𝜽 𝒈 ¬𝒊 (R 𝑖 + C 𝑖 ) + ∇ 𝜃 𝑓 𝑖 ∇ 𝜽 𝒈 ¬𝒊 (R 𝑖 + C 𝑖 ) 𝑇 Δ𝜽 𝒈 ¬𝒊 . (6)
Using a similar derivation as in LOLA [START_REF] Foerster | Learning with opponent-learning awareness[END_REF], that we do not detail here, we have (10)

Δ𝜽 𝒈 𝒋 =𝜂 𝑔 ∇ 𝜃 𝑔 𝑗 E 𝜽 𝒈 ,
∇ 𝜃 𝑔 𝑗 (R 𝑖 + C 𝑖 ) = E 𝜃 𝑓 𝑖 ,𝜽 𝒈 𝑇 𝑡 =0 (𝛾 𝑡 𝑟 𝑟 𝑖,𝑡 + 𝛾 𝑡 𝑐 𝑐 𝑖,𝑡 )∇ 𝜃 𝑔 𝑗 𝑡 𝑙=0 log(𝜋 𝑔 𝑗 (𝑎 𝑗,𝑙 |𝑠 𝑙 ; 𝜃 𝑔 𝑗 )) (8)
∇ 𝜃 𝑓 𝑖 ∇ 𝜃 𝑔 𝑗 (R 𝑖 + C 𝑖 ) = E 𝜃 𝑓 𝑖 ,𝜽 𝒈 𝑇 𝑡 =0 𝛾 𝑡 𝑐 𝑐 𝑖,𝑡 ∇ 𝜃 𝑔 (9)

Training Procedure

The feedback and game policy are trained alternately for a number of 𝐾 𝑓 and 𝐾 𝑔 update-steps respectively. The equivalent DiCE objective [START_REF] Foerster | Dice: The infinitely differentiable monte carlo estimator[END_REF] of the feedback policy objective is constructed and the corresponding loss is used during training to mediate the errors in estimating second order derivatives of a surrogate loss.

The game policy can be trained using an actor-critic architecture and the feedback policy with the REINFORCE algorithm.

A pseudo-code is given in Algorithm 1.

EXPERIMENTS AND RESULTS

We propose to test LIEF in two different scenarios. We begin with an environment called Teacher-Student. Here, the student agent receives no rewards from the environment and its only feedback comes from the teacher. We make sure that, without the teacher, the learning curve of the student is flat. Learning is purely directed by the teacher agent which has to learn how to use evaluative feedback to bring the student to accomplish a task.

In the second case, we test our framework on the iterated prisoner's dilemma. In this game, mutual cooperation is more beneficial than mutual defection. However, with the environment rewards only, the point of mutual cooperation is unstable and naive learners eventually converge to a less optimal but stable defective point. We test if, with inter-agent feedback, agents can modify the game dynamics and stabilise the more beneficial and cooperative point.

Hyper-parameters

In all experiments, we train the feedback and game policy alternately for 𝐾 𝑓 = 35 and 𝐾 𝑔 = 5 update-steps, using a batch size of 4096 episodes. All actors or critics are linear functions of the state and all states are one-hot encoded vectors. The learning rates of both actor and critic of the game policy are set to 𝜂 𝑔 = 1 and we use a discount factor 𝛾 𝑔 = 0.8. The learning rate of the feedback policy is set to 𝜂 𝑓 = 0.1 and the discount factors are 𝛾 𝑟 = 0.99 and 𝛾 𝑐 = 0.9.

Teacher-Student Environment

We begin by testing LIEF in a simple Teacher-Student chain environment depicted in Figure 3. A student, physically present in the environment, can at every timestep, choose to either move left or right. However, the environment is uninteresting to the student and doesn't provide him any rewards (𝑟 𝑠 = 0 ∀ 𝑠 ∈ S). The teacher on the other hand, would like the student to reach the rightmost cell in the chain since that state yields a positive reward of 𝑟 𝑡 = +1 to the teacher. The goals of the agents are not at conflict but nevertheless, uncorrelated. The teacher has to motivate the student somehow to move right since without extrinsic intervention, actions right and left would be equally desirable for the student. The environment is a representation of situations where one individual has an interest in accomplishing a task but lacks the skills to do so. Another agent, with the necessary skills, has no personal interest in getting the job done. Depending on the relationship between the two individuals, the former needs to either motivate the latter by paying him a remuneration for getting the job done or punishing him for not getting the task done.

In our experiment, we provide the teacher with a binary punitive action set U = {0, 1}. Both the teacher and the student observe the state of the environment (i.e., the chain position of the student and whether the student selected action right or action left in the previous timestep). The student can then select to either move right or left while the teacher can select to either punish the student or not. Not punishing, or selecting 𝑢 = 0, returns no feedback to the student (𝑓 = 0) and incurs no cost on the teacher (𝑐 = 0). However, choosing to punish, i.e., 𝑢 = 1, returns a negative feedback to the student (𝑓 = -1) and is equally costly to the teacher (𝑐 = -1). Figure 4 illustrates the interactions and feedback flow in this setting. We set a fixed timestep of 𝑇 𝑒 = 4 per episode and run our experiment with 10 different seeds. From the results plotted in Figure 5, we can see that during feedback policy training, the teacher learns to punish the student when the latter chooses the non beneficial action of going Left (green line rises during the 35 feedback policy update-steps). Consequently, during student learning, the feedback policy of the teacher causes the student to increase the frequency with which he chooses to go right (blue line rises). Nevertheless, some shortcomings are visible in the algorithm. The teacher has trouble completely cancelling punishments of actions in his favour (the red line doesn't decrease to zero). Deeper examination of this cause and an improvement of the algorithm are needed. When training the student without any feedback from the teacher, the preference of the student for action Right or Left remains constant throughout training (all rewards are zero and hence all gradients are zero). The student's preference for action Right in Figure 5 is solely constructed by the teacher. We note that our definition of Teacher-Student differs from that commonly seen in MARL [START_REF] Leno | Agents teaching agents: a survey on inter-agent transfer learning[END_REF][START_REF] Fachantidis | Learning to teach reinforcement learning agents[END_REF][START_REF] Ilhan | Teaching on a budget in multi-agent deep reinforcement learning[END_REF][START_REF] Kim | Learning hierarchical teaching in cooperative multiagent reinforcement learning[END_REF][START_REF] Omidshafiei | Decentralized teaching and learning in cooperative multiagent systems[END_REF][START_REF] Omidshafiei | Learning to teach in cooperative multiagent reinforcement learning[END_REF] where both the teacher and the student interact with the environment. There, the rewards or feedback come exclusively from the environment. The teacher, generally more skilled, is more of a guide to help the student achieve a task defined by the environment. In our case, the teacher is not more skilled than the student but lacks the means to perform a task he'd like accomplished. He cannot, like in the classical Teacher-Student case, give any demonstrations or examples to guide the student. His goal is to construct a target goal for the student (through evaluative feedback) that is compatible with his own goal.

Iterated Prisoner Dilemma

In a second experiment, we propose to test the efficacy of LIEF in an antagonistic environment. We select for this purpose the iterated prisoner's dilemma with returns shown in Table 1.

Table 1: Prisoner's dilemma reward table Actions A2 -C A2 -D A1 -C (-1, -1) (-3, 0) A1 -D (0, -3) (-2, -2)
In the one-shot version of the game, the environment raises in every player a preference to defect no matter the opponent's strategy. In fact, for a defecting opponent, defection results in one point more than cooperation (-2 compared to -3). Similarly, for a cooperative opponent, defection results in one point more than cooperation (0 compared to -1). As a result, both players learn to defect and converge to the bottom right cell of the table with an average of -2 points per player. We note here that, both players can in fact increase their returns by switching simultaneously to cooperation. However, this point, although more beneficial, is unstable.

In the iterated version of the game, which is the one we adopt for our experiment, agents play the game repeatedly against one opponent. They are endowed with a memory and observe at every timestep the actions taken by themselves and their opponent in the previous timestep. Accessing this info, allows some strategies to converge to cooperative behaviours such as the Tit-for-Tat [START_REF] Axelrod | Effective choice in the prisoner's dilemma[END_REF] strategy. In Tit-for-Tat, an agent punishes at timestep 𝑡, an opponent that defected at the previous timestep 𝑡 -1. The punishment is implemented as a retaliation and the agent reciprocates an opponent's defection at 𝑡 -1 with a defection at 𝑡. Although effective, Tit-for-Tat is not trivial to be found and naive reinforcement learners generally converge to defective behaviours.

By introducing inter-agent feedback, agents can modify the reward table and hence change the game dynamics. Instead of retaliation, agents may use punishment to compel cooperative behaviours from their opponents. If every agent manages to make action defect less desirable to their rival (e.g., by punishing this action), they can converge to cooperative behaviours without the risk of being exploited. While cooperation is usually difficult to sustain in the IPD, Figure 7 shows that agents converge to cooperative behaviours. We can see that during the training of the feedback policy, the green lines, indicating the rate with which agents punish a defective behaviour increase while the red lines, indicating the rate at which they punish cooperative behaviours decrease. As a result, during game policy training, cooperative actions become more advantageous than defective ones and players converge to total cooperation.

DISCUSSION AND CONCLUSION

We present a multi-agent reinforcement learning framework extended to incorporate inter-agent evaluative feedback. Leveraging the fact that the reward function in RL is the fundamental definition of a task, we allow agents, through this framework, to construct or modify the tasks of their peers.

In the teacher-student case (see Section 5.2), the teacher, using evaluative feedback, was constructed a target goal for the student who originally had none. Moreover, LIEF did not construct a random reward function, but one that is specifically aligned with his own. The construction of a reward function is not a trivial question in RL. Augmenting a sparse reward function to speed up learning is widely studied in literature [START_REF] Marthi | Automatic shaping and decomposition of reward functions[END_REF][START_REF] Andrew Y Ng | Policy invariance under reward transformations: Theory and application to reward shaping[END_REF][START_REF] Xiao | Fresh: Interactive reward shaping in highdimensional state spaces using human feedback[END_REF]. In this context, it's interesting to investigate the differences between rewards given by humans, constructed systematically or learned by reinforcement learners.

In a more conflicting case, the iterated prisoner's dilemma (see Section 5.3), two LIEF agents were capable of mutually reshaping each other's reward functions to modify the whole game dynamics. The agents, through evaluative feedback, managed to make cooperative actions more beneficial from their opponent's perspective. The mutual and simultaneous feedback, assured that both agents cooperated and no one exploited the other. Here, more conflicting environments can be tested, especially those in which retaliation is not an effective tool to enforce cooperation but where evaluative feedback may be (e.g., Public Goods Games and Common Pool Resources).

Figure 1 :

 1 Figure 1: Extension of a stochastic game to incorporate interagent evaluative feedback.

Figure 2 :

 2 Figure2: Here LIEF predicts from current data, the changes in the game policy of its opponent. These changes incur changes in the expected rewards 𝑟 2 received from the environment and changes in the expected feedback 𝑓 1 given to agent 1 which in turn yields changes in the expected costs 𝑐 2 . LIEF makes sure that his feedback policy ameliorates the future of the agent, i.e., that his expected returns 𝑐 2 + 𝑟 2 increase between the expected look-ahead step 𝑘 + 1 and the current step 𝑘.

  𝑖,𝑙 |𝑠 𝑙 ; 𝜃 𝑓 𝑖 )) .

Figure 3 :

 3 Figure 3: Teacher-Student chain environment. The environment doesn't motivate the student to move either left or right (𝑟 𝑠 = 0 for all states). However, the teacher has an incentive to bring the student to the right most cell where he receives a reward 𝑟 𝑡 = 1.

Figure 4 :

 4 Figure 4: Feedback loop of a MARL setting with additional evaluative feedback flowing from a teacher to a student.

Figure 5 :

 5 Figure 5: Teacher and Student policy dynamics. Vertical lines delimit the zones where the teacher and student alternate in learning. The teacher performs 35 update-steps after which the teacher feedback policy is frozen and the student, under said teacher policy, learns for 5 update-steps. The cycle is repeated 5 times.

  Figure shows the flow of the added inter-agent feedback and the resulting costs on the agents. In our case, we use a binary punitive action set U 𝑖 = {0, 1} for each agent and the resulting feedback and costs are such that 𝑓 𝑖 (𝑢 ¬𝑖 ) = -3𝑢 ¬𝑖 and 𝑐 𝑖 (𝑢 𝑖 ) = -1𝑢 𝑖 .

Figure 6 :

 6 Figure 6: Inter-agent evaluative feedback

Figure 7 :

 7 Figure 7: Behaviour and feedback dynamics of two mutually rewarding LIEF agents in an Iterated Prisoner's dilemma.

  𝜽

									𝑇
						𝒇			𝛾 𝑡 𝑔 (𝑟 𝑗,𝑡 + 𝑓 𝑗,𝑡 )
						¬𝒋		𝑡 =0
						𝑇						𝑡
			=𝜂 𝑔 E 𝜽 𝒈 ,𝜽	𝒇 ¬𝒋	𝑡 =0	𝛾 𝑡 𝑔 (𝑟 𝑗,𝑡 + 𝑓 𝑗,𝑡 )∇ 𝜃 𝑔 𝑗	𝑙=0	log(𝜋	𝑔 𝑗 (𝑎 𝑗,𝑙 |𝑠 𝑙 ; 𝜃	𝑔 𝑗 ))
												(7)
									𝑇			𝑡
	∇ 𝜃	𝑓 𝑖	Δ𝜽	𝒈 𝒋 = 𝜂 𝑔 E 𝜽 𝒈 ,𝜽	𝒇 ¬𝒋	𝑡 =0	𝛾 𝑡 𝑔 𝑓 𝑗,𝑡 ∇ 𝜃 𝑔 𝑗	𝑙=0	log(𝜋	𝑔 𝑗 (𝑎 𝑗,𝑙 |𝑠 𝑙 ; 𝜃	𝑔 𝑗 ))
											𝑡
									• ∇ 𝜃	𝑖 𝑓	log(𝜋
											𝑙=0

𝑓 𝑖 (𝑢 𝑖,𝑙 |𝑠 𝑙 ; 𝜃 𝑓 𝑖 ))

Algorithm 1 :

 1 Learning to Influence through Evaluative Feedback Input: Learning rates 𝜂 𝑔 and 𝜂 𝑓 , discount factors 𝛾 𝑔 , 𝛾 𝑟 and 𝛾 𝑐 , update-steps 𝐾 𝑔 and 𝐾 𝑓 , total updates 𝐾 Initialise: for agents 𝑖, . . . 𝑁 , policy parameters 𝜃 𝜽 𝒈 ) for 𝑖 = 1, . . . 𝑁 for 𝑘 𝑔 = 1, 2, . . . 𝐾 𝑔 do Rollout episode trajectory under 𝜋 g and 𝜋 f

					𝑔 𝑖 and 𝜃	𝑓 𝑖
	for k=1,. . . 𝐾 do
	for 𝑘 𝑓 = 1, 2, . . . 𝐾 𝑓 do
	Rollout episode trajectory under 𝜋 g and 𝜋 f
	Update:
	𝜃 𝑖 ; Update: 𝑓 𝑖 ← 𝜃 𝑓 𝑖 + 𝜂 𝑓 ∇ 𝜃 𝑓 𝑖 𝐽 𝑓 𝑖 (𝜃 𝑓
	𝜃	𝑔 𝑖 ← 𝜃	𝑔 𝑖 + 𝜂 𝑔 ∇ 𝜃 𝑔 𝑖	𝐽 𝑖 (𝜃 𝑔	𝑔 𝑖 ; 𝜽

𝒈 ¬𝒊 , 𝜽 𝒇 ) for 𝑖 = 1, . . . 𝑁 Output : Policy parameters 𝜽 𝒈 and 𝜽 𝒇

ACKNOWLEDGMENTS

This work has received funding from the European Union's Horizon 2020 research and innovation program under grant agreement No 76595.