Alexandre Fagnan

Frederik Otzen Bagger

Maria-Riera Piqué-Borràs

Cathy Ignacimouttou

Alexis Caulier

Cécile K Lopez

Elie Robert

Benjamin Uzan

Véronique Gelsi-Boyer

Zakia Aid

Cécile Thirant

Ute Moll

Samantha Tauchmann

Amina Kurtovic-Kozaric

Jaroslaw Maciejewski

Christine Dierks

Orietta Spinelli

Silvia Salmoiraghi

Thomas Pabst

Kazuya Shimoda

Virginie Deleuze

Hélène Lapillonne

Connor Sweeney

Véronique De Mas

Betty Leite

Zahra Kadri

Sébastien Malinge

Stéphane De Botton

Jean-Baptiste Micol

Benjamin Kile

Catherine L Carmichael

Ilaria Iacobucci

Charles G Mullighan

Martin Carroll

Peter Valent

Olivier A Bernard

Eric Delabesse

Paresh Vyas

Daniel Birnbaum

Eduardo Anguita

Loïc Garçon

Eric Soler

Juerg Schwaller

Thomas Mercher

T M Contributed

Human erythroleukemia genetics and transcriptomes identify master transcription factors as functional disease drivers

Acute erythroleukemia (AEL or acute myeloid leukemia [AML]-M6) is a rare but aggressive hematologic malignancy. Previous studies showed that AEL leukemic cells often carry complex karyotypes and mutations in known AML-associated oncogenes. To better define the underlying molecular mechanisms driving the erythroid phenotype, we studied a series of 33 AEL samples representing 3 genetic AEL subgroups including TP53-mutated, epigenetic regulator-mutated (eg, DNMT3A, TET2, or IDH2), and undefined cases with low mutational burden. We established an erythroid vs myeloid transcriptome-based space in which, independently of the molecular subgroup, the majority of the AEL samples exhibited a unique mapping different from both non-M6 AML and myelodysplastic syndrome samples. Notably, >25% of AEL patients, including in the genetically undefined subgroup, showed aberrant expression of key transcriptional regulators, including SKI, ERG, and ETO2. Ectopic expression of these factors in murine erythroid progenitors blocked in vitro erythroid differentiation and led to immortalization associated with decreased chromatin accessibility at GATA1-binding sites and functional interference with GATA1 activity. In vivo models showed development of lethal erythroid, mixed erythroid/myeloid, or other malignancies depending on the cell population in which AEL-associated alterations were expressed. Collectively, our data indicate that AEL is a molecularly heterogeneous disease with an erythroid identity that results in part from the aberrant activity of key erythroid transcription factors in hematopoietic stem or progenitor cells.

Visual Abstract Introduction

Acute myeloid leukemia (AML) of the erythroid lineage (acute erythroleukemia [AEL] or AML-M6) accounts for 3% to 5% of AML patients and is inherently associated with poor outcome. Although AEL can occur at any age, the majority of patients are >65 years, and the disease often occurs secondary to other neoplasms, including myeloproliferative neoplasms (MPNs) or myelodysplastic syndrome (MDS), or after cytotoxic cancer treatment. Two major morphological subtypes have been proposed: pure erythroleukemia (PEL; AML-M6b, also known as Di Guglielmo disease) with >80% of blasts committed to the erythroid lineage and AML-M6a characterized by the presence of both erythroid precursors and myeloid blasts. The 2016 World Health Organization (WHO) classification integrated AML-M6a into MDSs or not otherwise specified AML (AML-NOS), but this classification remains a matter of debate.

Functional studies have suggested that 2 to 5 genetic driver lesions on a background of preexisting alterations in hematopoietic stem or progenitor cells (HSPCs) might be sufficient to induce AML. For AEL, earlier work showed that leukemic cells often have complex karyotypes, and targeted DNA sequencing revealed the presence of several known AML-associated mutations, but AEL-driving molecular mechanisms remain incompletely understood and erythroleukemia-specific mutations have seldom been functionally validated. Strikingly, single or multiple TP53 mutations have been shown to be a molecular hallmark of PEL.

Normal erythroid differentiation is controlled by the activity of both extrinsic signaling factors, including erythropoietin (EPO) mediating its effects through the EPO receptor (EPOR) signaling pathways, and intrinsic multimeric transcription complexes.

The latter includes hematopoietic master regulators like GATA-binding protein 1 (GATA1), T-cell acute lymphocytic leukemia protein 1 (TAL1), LIM domainonly 2 (LMO2), CBFA2/RUNX1 partner transcriptional corepressor 3 (CBFA2T3, also known as ETO2), and LIM-domain-binding protein 1 (LDB1), thereafter broadly named GATA1 complexes, which can activate or repress transcription of target genes. These GATA1 complexes contribute to terminal erythroid differentiation through binding to gene loci and transcription of essential erythroid genes (eg, hemoglobin). This process is also regulated by Krüppel-like factor 1 (KLF1), which binds DNA next to the GATA1 complexes to coregulate erythroid genes.

To establish the erythroid differentiation program, functional synergism between these transcriptional complexes and the EPO/EPOR signaling is mediated by the presence of phosphorylated STAT5 binding in the neighborhood of GATA1 and KLF1.

Accordingly, mutations in these factors have been associated with altered erythropoiesis.

For example, GATA1 mutations are associated with congenital erythroid hypoplasia (Diamond-Blackfan anemia [DBA]) or Xlinked dyserythropoietic anemia. Moreover, the identification of a NFIA-ETO2 fusion in pediatric PEL suggests that an altered activity of these complexes may contribute to human erythroid leukemogenesis.

To better understand the molecular mechanisms that control the erythroid feature, we characterized the genetic and transcriptional landscape in leukemic cells from 33 AEL patients. We identified distinct molecular subgroups composed of patients carrying (1) TP53 mutations, (2) various combinations of mutations previously found in AML and MDS such as DNMT3A, TET2 or IDH2, and (3) those with none of these recurrent alterations. Comparative transcriptomics established an erythro/myeloid differentiation expression signature space that distinguished the majority of AEL cases from MDS or other AML forms. Notably, leukemic cells from >25% of AEL patients showed aberrant expression of key transcriptional regulators including SKI, ERG, and ETO2, which interfere with the activity of the erythroid master regulator GATA1. Combinatorial experimental expression in HSPC fractions induced lethal erythroid or mixed erythroid/myeloid diseases in mice phenocopying several aspects of the human disease, underlining their importance in the molecular pathogenesis of AEL. Fifty-eight human patient samples were obtained with the informed consent of the patient and approved by the local ethics committees in accordance with national ethics rules. AEL patient diagnostics were established according to the WHO 2008 classification and criteria described recently. Cytogenetic risk groups were defined according to the revised International Prognostic Scoring System (IPSS-R). Mononuclear cell fractions were obtained from patient blood or bone marrow (BM) samples by Ficoll gradient, and frozen in fetal bovine serum (FBS; Gibco) supplemented with 10% dimethyl sulfoxide (DMSO). DNA and RNA extraction were done on fresh or frozen samples. DNA was extracted using bulk or sorted cells from patient samples (n = 7, CD36 for blast cell population and CD3 or CD19 for nonneoplastic cell populations) or from xenograft-amplified samples (n = 4). RNA was extracted from patient samples (n = 22) and xenograft-amplified samples (n = 7) from bulk or sorted cells (CD36 or CD45 cells), respectively. We obtained appropriate sequencing material for 33 patients (11 paired patient samples for exome sequencing and 29 patient samples for RNA sequencing).

Materials and methods

Patient samples

Murine models

C57BL/6JOlaHsd mice (named C57BL/6J) were purchased from Envigo and NOD.Cg-Prkdc Il2rg

/SzJ (NSG) mice from The Jackson Laboratory (005557). TP53 knock-in mice were described previously. To generate double transgenic TET2 /GATA1s mice, we intercrossed Tet2 and Gata1 (here named Gata1s) mice.

Mice were maintained at the Gustave Roussy preclinical facility and all experiments were approved by the French National Animal Care and Use Committee (CEEA 26: projects 2017-082-12726 and 2017-084-12799).

Flow cytometry and cell sorting

Antibodies used for flow cytometry are listed in supplemental Table 1 (available on the Blood Web site). Cells were stained in 1× phosphate-buffered saline (PBS) supplemented with 2% FBS at 4°C for 30 minutes and washed prior to analysis. Whole BM or spleen cells were analyzed without red blood cell lysis. For cell sorting, total BM cells underwent red blood cell lysis. To obtain HSPCs, total BM was depleted of all major hematopoietic cell lineage (Lin ) using the Mouse Hematopoietic Progenitor (Stem) Cell Enrichment Set (Becton Dickinson [BD]]. Progenitor populations were further purified by fluorescence-activated cell sorting (FACS) according to the following phenotypes: Hematopoietic stem cells (HSC) were defined as Lin /Sca1 /KIT /CD34 /CD48 , megakaryocytic-erythroid progenitors (MEPs) were defined as Lin Sca1 KIT /CD34 /CD16/32 and granulocyte-macrophage progenitors (GMP) were defined as Lin /Sca1 /KIT /CD34 /CD16/32 . To obtain mouse erythroid progenitors, BM cells were first depleted using biotin-conjugated antibodies against CD3, B220, Gr-1, and CD11b (BD) followed by FACS according to the population described as CD71 /Ter119 /KIT . Flow cytometric analysis was performed using ARIAII, CANTO-II, or CANTO-X instruments (BD), and data were analyzed using the FlowJo software (Flowjo 9.3.2).

Cell culture

Mouse erythroid progenitor cells were expanded in StemSpan serum-free expansion medium (SFEM; Stem Cell Technologies) supplemented with penicillin (100 U/mL)-streptomycin (100 μg/mL), murine stem cell factor (mSCF) (10 ng/mL), murine interleukin 3 (mIL3) (10 ng/mL), mIL6 (10 ng/mL), human EPO (hEPO) (2 U/mL), 0.4% cholesterol, and dexamethasone (10 M). Mouse erythroleukemia (MEL) cells were maintained in RPMI 1640 (Gibco) supplemented with 10% FBS, penicillin (100 U/mL)streptomycin (100 μg/mL) and 2 mM -Glutamine (Gibco). Murine G1E cells, a generous gift from M. Weiss, were maintained in Iscove modified Dulbecco medium (Gibco) supplemented with 15% FBS, penicillin (100 U/mL)-streptomycin (100 μg/mL), mSCF (10 ng/mL), hEPO (2 U/mL), monothioglycerol (4.5 × 10 M), and 2 mM -Glutamine (Gibco). Murine growth factor-dependent Ba/F3 cells was maintained in RPMI 1640 (Gibco) supplemented with 10% FBS, penicillin (100 U/mL)-streptomycin (100 μg/mL), mIL3 (10 ng/mL), and 2 mM L-Glutamine (Gibco). Human embryonic kidney (HEK-293T) cells were grown in Dulbecco modified Eagle medium (Gibco) supplemented with 10% FBS, penicillin (100 U/mL)-streptomycin (100 μg/mL), and 2 mM L-Glutamine (Gibco).

Retroviral constructs, particle production, and cell transduction

The SKI complementary DNA (cDNA) was a kind gift from Suzana Atansoski (Basel, Switzerland). The other cDNAs were synthesized. All cDNAs were cloned into retroviral pMSCV-IRES-EGFP or -mCherry backbones. GATA1 cDNA was cloned into lentiviral pLT3-GEPIR-IRES-EGFP expression vector. For retroviral or lentiviral particles production, HEK-293T cells were plated 1 day before cotransfection with the expression constructs coexpressing EGFP or mCherry and cDNA using the X-tremeGENE-9 DNA Transfection Reagent (Roche) or jetPRIME reagent (Polyplus transfection), respectively, according to the manufacturer's recommendations. Culture media were changed 24 hours posttransfection and supernatants containing viral particles were harvested 48 hours and 72 hours posttransfection. Murine cells were transduced by spinoculation (90 minutes at 2500 rpm, 33°C) with supernatants containing viral particles supplemented with 5 μg/mL polybrene in 7.5 mM HEPES buffer.

BM transplantation

Total BM (0.4 × 10 cells) and/or transduced progenitor cells were transplanted through IV injection in lethally (9.5 Gy) or sublethally (5 Gy) irradiated 8-to 10-week-old C57BL/6J recipient mice.

RNA extraction and RT-qPCR

RNA was extracted using a RNeasy Mini kit (Qiagen) or AllPrep DNA/RNA Mini kit (Qiagen), according to the manufacturer's recommendations and quantified using NanoDrop (ThermoScientific). Reverse transcription (RT) was performed using SuperScript II (Invitrogen). Quantitative polymerase chain reaction (qPCR) was performed using SYBR Select Master mix or TaqMan Gene Expression Master mix (Applied Biosystems) on a 7500HT Fast Real-Time PCR System (Applied Biosystems) following the manufacturer;'s recommendations. Primer sequences are listed in supplemental Table 2.

Whole-exome sequencing

Whole-exome sequencing was conducted as described previously on paired-samples from 11 patients. DNA from sorted CD3 or CD19 nonneoplastic cells was used for exome capture using SureSelect All Exon V4 or V5 kits (Agilent Technologies). We performed paired-end sequencing (100 bp) using HiSeq2000 sequencing instruments at Gustave Roussy genomic platform. Reads were mapped to the reference genome hg19 using the Burrows-Wheeler Aligner (BWA) alignment tool version 0.7.10. PCR duplicates were removed using Picard tools-Mark Duplicates (version 1.119). Local realignment around indels and base quality score recalibration was performed using GATK 3.3 (Genome Analysis Tool Kit). Reads with a mapping quality score < 30 < 20 were removed. Somatic single-nucleotide variations (SNVs) and indels were called in the leukemic sample using Varscan (v2.3.7) by comparison with the paired nonneoplastic samples for exomes, and by comparison with the reference genome for RNA-seq. For candidate somatic mutations, the variants were adopted as candidate mutations when P value was <.001 and allele frequency was <.1 in the reference sample. Variants were annotated with Annovar (v141112). We excluded synonymous SNVs, variants located in intergenic, intronic, untranslated regions and noncoding RNA regions. The mean coverage in the targeted regions was, respectively, 85,4× and 91,2× for leukemic and nonneoplastic samples. The functional variants were predicted using the open platform Cancer Genome Interpreter (CGI) and only known-variants or predicted driver variants were confirmed through visualization with IGV (2.3.88) and finally kept in this study.

RNA sequencing

RNA sequencing (RNA-seq) was performed as described. Sequences were aligned to the reference genome with TopHat2 version 2.0.9 using the following parameters: -bowtie1-fusion-search-library-type fr-firststrand-read-realign-edit-dist 0 -p 8 -r 50 (or 2.0.14 for mice data sets) and Bowtie1 version 1.0.0. The number of reads per genes (RefSeq database) was counted with HTSeq-count version 0.5.4p5 using the "union" mode. The counts were then normalized with the DESeq2 method, which takes into account the library size of each sample. ATAC-sequencing (ATAC-seq) analysis has been previously described Briefly, after lysis of 50 000 cells, transposition, and purification steps, the transposed DNA fragments were amplified by PCR (12 cycles) using adapters from the Nextera Index Kit (Illumina). PCR purification was performed using Agencourt AMPure XP magnetic beads (Beckman Coulter) to remove large fragments and remaining primers. Library quality was assessed using an Agilent 2100 Bioanalyzer using a High Sensitivity DNA Chip (Agilent Technologies). Libraries were sequenced using NovaSeq 6000 sequencer (Illumina; 50 bp paired-end reads). Quality control of reads was performed using FastQC 0.11.7 and multiQC 1.5. The reads were aligned to the reference genome mm10 with bwa (aln 0.7.17). After alignment, we removed reads mapping to the mitochondrial genome, PCR duplicate reads, and reads with a mapping quality lower than 20 using samtools (v 1.9). Final read counts for all mouse data sets ranged from 42 to 128 million reads. Mapped reads were normalized to bins per million and were converted to bigwig format using deeptools (v3.3.0). Peak calling, differential analysis, annotation, and motif analysis were performed using macs2 (v 2.1.1) and homer (v4.10.4, annotatePeak.pl and findMotifsGenome.pl).

ATAC sequencing

PCA of data from differentiation map

To define the hematopoietic space described (Figure 3A), we built a principal component analysis (PCA) of cell types from Differentiation Map [DMAP] (DMAP_PCA), excluding NK cells, B cells, T cells, and dendritic cells. As features we included ranks of differentially expressed genes (DMAP_DE) (FDR < 0.05, logFC > 2) determined with limma. A Loess regression line was fitted in PCA space to erythroid cells (all erythrocytes and MEP), and Myelocytes (HSC, CMP, GMP, GRAN, MONO, BASO, and EOS). New data points (NP) was projected into the DMAP_PCA space as a dot product between scaled NP vector and DMAP_PCA rotation. By these calculation, we applied the same transformation to NP and added them to DMAP_PCA without recalculation of principal components.

Transcription factor activity inference

For gene regulatory network inference, the ARACNe-AP software was used to infer a Gene Regulatory Network using scRNA-seq data from healthy human progenitors to predict a list of target genes for each transcription factor (TF). ARACNe was run over the log2 normalized counts in bootstrap mode (100 iterations), with a P value threshold of 1e-8 and a custom curated list of 2171 TFs. Therefore, the activity of each TF in a normal context was computed in a network. For each AEL sample, TF activities were inferred by interrogating this network with AEL transcriptome data and expressed as Normalized Enrichment Score (NES) using the R library viper, as described in the bioconductor package manual. NES were used to test differential activity by Student t test and P value correction by Benjamini-Hochberg (FDR cutoff at 0.05). Differentially activated gene lists were established by PCA analysis using predicted activated gene matrix (previously computed using ARACNE and VIPER algorithm), then genes driving PCA dimensions were identified and ranked by contribution (using FactoMineR v1.41 and factoextra v1.0.5 R packages). Finally, heatmap of activated genes was obtained by plotting the top 50 most contributed genes from the first PCA dimension (using pheatmap v1.0.12 R package).

Data sets

Sequencing data were deposited into EBI -Array-Express under the accession E-MTAB-9012 (ATAC-seq) and European Genome-Phenome Archive (EGA) under the accession EGAS00001004203 (Exome/RNAseq). Available GATA1 ChIPseq on mouse erythroblasts were obtained from ENCODE (GSE36029; SRA accession: SRR492437) and available ATAC-seq data sets from mouse MEP, CFU-E, and proerythroblasts were previously published.

Statistical analysis

Statistical significance was calculated using Prism (version 6.0a) and is indicated as P values (Student t test except when otherwise specified). *P < .05, **P < .01, ***P < .001. We collected samples from 58 AEL patients, including 34 adults >60 years, 14 between 40 and 59 years, 8 young adults (21-39 years), and 2 pediatric patients. According to the 2008 WHO classification, 33 patients were diagnosed with de novo AEL, including 29 AML-M6a and 4 AML-M6b; 20 patients were diagnosed with AML-M6a secondary to MDS/CML/ALL; 1 with AML-M6b secondary to plexus choroid carcinoma; and a more precise diagnosis was lacking for 4 patients (supplemental Figure 1A; supplemental Table 3). Thereafter, the term "AEL" was used for all patients. Several AEL samples lacking sufficient number of viable cells, were expanded by xenografting them in NOD.Cg-Prkdc Il2rg /SzJ (NSG) mice. This approach provided additional leukemic material to isolate RNA (7 patients) and DNA (4 patients). Together, we obtained appropriate sequencing material for 33 patients and performed exome sequencing on 11 paired leukemic and nonneoplastic (either CD3 or CD19 cells from the same patient) samples and RNA sequencing of 29 leukemic samples. Combining exome and RNA-seq data, we identified sequence variants with predicted functional consequences in 62 genes (Figure 1). These data, including high variant allele frequency (supplemental Figure 1B), support classification of patients into 3 molecular subgroups (Figure 1). Subgroup 1, presenting with TP53 mutations (n = 12, 36.3% of patients), had in average 4.41 mutations per sample and was associated with both a higher cytogenetic risk and a poorer outcome (Figure 1; supplemental Figure 1C-F). Subgroup 2 (n = 11, 33.3%) mostly presented with TET2 nonsense mutations (n = 8) and DNMT3A mutations (n = 5), including 2 patients with both TET2 and DNMT3A mutations, and had in average 5.72 mutations per sample. Several patients with TET2 and/or DNMT3A mutations also carried SRSF2 or IDH2 mutations. Of note, in the only sample presenting both a TET2 and an IDH2 mutation, the variant allele frequencies were 60% and 13%, respectively (data not shown), possibly reflecting 2 independent clones. Interestingly, 1 case (#17) of subgroup 2 harbored a TET2 loss-of-function mutation and a GATA1 mutation, predicted to encode the short isoform GATA1s. Additional mutations affected transcription factors (eg, WT1, RUNX1), epigenetic regulators (eg, ASXL1, EP300, BCOR), signaling mediators (eg, NOTCH2, IL7R), and other genes in this group of patients. Finally, subgroup 3 (n = 10, 30.4%) contained samples without TP53 or epigenetic variants. On average, these AEL showed 1.60 mutations per sample, a significantly lower value than for subgroups 1 and 2 (Figure 1; supplemental Figure 1G-H). Overall, our data confirmed that AEL is a molecularly heterogeneous disease characterized by a high prevalence of genetic variants in TP53 and epigenetic regulators comparable to other published cohorts.

Results

Molecular alterations in AEL patients

AEL gene-expression signatures correlate with erythroid differentiation

As the heterogeneous genetic alterations did not provide any strong rationale for the erythroid phenotype of these leukemia, we investigated the erythroid feature by comparing gene-expression signatures (GES). PCA did not reveal any significant correlation between the GES and the 3 previously identified molecular subgroups (Figure 2A). Similarly, the percentage of erythroblasts in the patient BM at diagnosis was poorly reflected by GES (Figure 2B).

Because the AEL WHO classification is based on the number of erythroid and myeloid blasts present in the BM, we used a digital cellular deconvolution method (xCell) to compute a GES-based enrichment in erythroid, myeloid, and other hematopoietic cell types (supplemental Figure 2A). The majority of samples had a prominent "erythrocyte" signature (n = 20), whereas some AEL samples presented a higher signal for the immature (MPP, CMP, GMP) or mature myeloid (monocyte, neutrophil) signatures (n = 9). To further explore the link between AEL transcriptomes and different stages of human erythroid maturation, we compared GES from the patients with those obtained experimentally after in vitro differentiation of human peripheral blood mononuclear cells into colony-forming unit erythroid (CFU-E; CD71 CD235 ), proerythroblasts (Pro-E; CD71 CD235 ), intermediate (Int-E; CD71 CD235

), and late erythroblasts (Late-E; CD71 CD235

) and observed clustering according to these maturation stages (Figure 2C-D). Importantly, transcriptomes from an independent larger cohort of AEL patients clustered similarly (Figure 2E-F).

Together, AEL gene expression programs are influenced by the erythroid differentiation stages rather than by the presence of particular genetic lesions, suggesting that the erythroid identity in human AEL relates to the cellular origin and the activity of transcriptional regulators driving cellular differentiation. As the 2016 WHO classification assigns most cases previously diagnosed as AEL to MDS or other AML, we aimed at designing a transcriptome-based space that is able to distinguish AEL from MDS and other non-AEL AML subtypes. To this end, we retrieved cellular signatures from the DMAP database and computed erythroid and myeloid differentiation expression trajectories (Figure 3A). As expected, relative differential gene expression clustered our AEL samples between the erythroid and myeloid trajectories. The majority of cases (n = 25) mapped closer to the erythroid axis whereas the rest mapped closer to the myeloid trajectory (n = 7) and closer to MDS transcriptomes (Figure 3B-C). Likewise, the samples from a recently published large AEL cohort mostly clustered apart from MDS samples (Figure 3C) and apart from non-M6 AML samples (Figure 3D). Notably, AEL samples mostly projected between HSC and mature erythroid cells supporting that only part of the maturation-associated erythroid program is expressed in these samples. Interestingly, among our AEL samples that mapped closer to the myeloid axis and other AML samples, sample 24 showed a high expression of SPI1 (Figure 3D; supplemental Figure 2B) also seen in other AML subtypes and was actually independently reclassified as AML-M5 by clinicians during the course of this study. These data support the idea that the transcriptional programs of the majority of AEL cases differs from those of MDS and other AML subtypes. They also support the existence of an overlapping continuum between these entities and the WHO-2016 reclassification of some AEL cases as AML-NOS.

Expression and activities of erythroid regulators in AEL

Myelo/erythroid differentiation is controlled by expression and activity of a relatively small group of transcription factors. Using the ARACNe and VIPER packages and a large data set from human healthy progenitor cell transcriptomes, we computed the activity of transcription factors and inferred lists of putative target genes (supplemental Figure 3A-B). Interestingly, we observed a gradual decrease in expression of erythroid transcription factors (eg, KLF1, GATA1, NFE2, TAL1, NFIA) and their predicted activity when going from the erythroid to the myeloid trajectories and an inverse correlation with myeloid factors (eg, CEPBA and SPI1) (Figure 3E; supplemental Figure 3C). This finding indicates that AEL is characterized by the transcriptional proximity to the normal erythroid lineage trajectory and by the relative activity of master transcription factors that control erythroid differentiation.

These data led us to hypothesize that some AEL cases might be driven by aberrant expression and activity of erythroid transcription factors. We focused on factors known to be predominantly expressed during erythroid differentiation and/or to control the activity of the GATA1 erythroid master regulator. Using a threshold of fourfold higher expression level than the average, we observed that some AEL patients indeed expressed abnormally high levels of ERG (n = 2), GFI1 (n = 1), RUNX1T1 (= 1), and ETO2 (n = 1) ( Figure 3F). GATA3, which enforced expression previously resulted in erythroid bias, was also highly expressed in 3 samples. Notably, we also found high expression of the transcriptional corepressor SKI (v-Ski avian sarcoma viral oncogene homolog) in 2 patients from molecular subgroup 3 (Figure 3F). Interestingly, v-Ski was previously reported to transform chicken erythroid cells, and to directly interact with GATA1 to repress erythroid differentiation.

Our findings suggest that SKI not only influences experimental erythroid differentiation but could also contribute to human AEL pathogenesis.

The search for fusion transcripts using RNA-seq data revealed additional alterations, including 1 in-frame BCR-ABL1 fusion gene (expected in this secondary-to-CML sample) and 3 novel out-of-frame fusion transcripts, notably 2 of them in a TP53-mutated context (supplemental Figure 4A-B). Sample 37 harbored an out-of-frame fusion of YWHAE (tyrosine 3-monooxygenase/tryptophan 5-monooxygenase activation protein ε) with EPO and sample ES3 showed an out-of-frame fusion of HSD17B11 (hydroxysteroid 17-β dehydrogenase 11) with B4GALNT3 (β-1,4-N-acetyl-galactosaminyltransferase 3). These fusions were associated with ectopic expression of EPO and B4GALNT3, respectively. Interestingly, the YWHAE-EPO sample also showed high EPOR, suggesting an autocrine mechanism of proliferation/survival in this case (supplemental Figure 4C). Although the role of B4GALNT3 in human erythropoiesis remains unclear, an out-of-frame fusion leading to overexpression of B4GALNT3 has been previously reported in thyroid carcinoma. Sample ES1 from subgroup 3 presented with an out-of-frame fusion targeting the middle of the DNMT3B locus and associated with lower DNMT3B expression as compared with other samples, supporting DNMT3B inactivation. Notably, Dnmt3a and Dnmt3b expression were previously reported to be tightly controlled during erythroid maturation in mice.

Overall, genetic and transcriptional alterations in erythroid regulators, including physical or functional interactors of the GATA1 transcriptional complexes were found in 9 of 33 patients (27%) (Figure 3G). Notably, these patients showed a trend toward poorer overall survival (Figure 3H), which became significant upon analysis of a larger AEL data set (Figure 3I). Together, transcriptome analysis revealed that the majority of AEL samples are significantly different from MDS and other AML subtypes and that AEL frequently presents with epigenomic alterations that converge on factors interfering with GATA1 activity.

Overexpression of AEL-associated GATA1-interfering factors transforms mouse erythroid progenitors

To functionally test whether the aberrant expression of GATA1-interfering factors identified in AEL samples may contribute to the transformation of the erythroid lineage, we explored the consequences of ectopic expression of SKI, ERG, ETO2, GATA1s, EPO, SPI1, and B4GALNT3 on murine erythroid progenitors. FACS-purified KIT CD71 Ter119 cells were transduced with retroviruses encoding these genes and grown in vitro (Figure 4A). In contrast to vector-transduced controls that proliferated for only ∼7 days, ectopic expression of ERG, SPI1, ETO2, SKI, and B4GALNT3 significantly maintained proliferation of erythroid cells presenting with an immature CD71 KIT Ter119 phenotype and a proerythroblast morphology for >30 days (Figure 4B; supplemental Figure 5A). Although the precise comparison between the overexpression level observed in human AEL samples and those achieved in murine models is technically challenging in this setting, a similar range of overexpression was observed for ERG, ETO2, SKI, and B4GALNT3 (supplemental Figure 5B). Notably, ectopic expression of EPO or GATA1s alone was not sufficient to expand erythroblasts longer than 10 days (Figure 4B).

To address whether a cooperation between Tet2-inactivating and Gata1s mutations could transform erythroblasts in vitro, we purified erythroid progenitors from wild-type, Tet2-deficient (thereafter named Tet2 ), Gata1 knock-in (thereafter named Gata1s), and double Tet2 +Gata1s transgenic mice and compared their proliferation (Figure 4C). Although Gata1s-or Tet2 -only erythroblasts did not expand for >10 to 15 days, Tet2 +Gata1s erythroblasts proliferated >2 months and exhibited an erythroid morphology (Figure 4D; supplemental Figure 5C).

Collectively, these data demonstrate that ectopic expression of ERG, ETO2, SKI, or the combination of Tet2 loss-of-function and Gata1s mutations can efficiently immortalize murine erythroblasts in vitro.

Aberrantly expressed AEL-associated transcriptional regulators interfere with GATA1 chromatin accessibility and function

To better understand how aberrantly expressed transcription factors (ERG, ETO2, SKI, SPI1) immortalize erythroblasts, we studied chromatin accessibility by ATAC-seq. Motif analysis revealed a lower representation of GATA1 and KLF1 (also known as EKLF) motifs and a global increase of ETS-associated motifs (including ERG and SPI1 motifs) in all transformed cells compared with Ctrl (Figure 4E-F; supplemental Figure 5D).

To investigate chromatin accessibility at erythroid GATA1-binding sites and at sites that are regulated at specific stages of healthy erythroid differentiation, we interrogated previously published data sets. Interestingly, ERG-, ETO2-, SKI-, and SPI1-overexpressing erythroblasts showed a decreased chromatin accessibility at erythroid GATA1-binding sites compared with vector-transduced control cells (Figure 4G). ERG-, ETO2-, SKI-, and SPI1-expressing erythroblasts also showed a decreased chromatin accessibility at sites open in healthy CFU-E and proerythroblasts, whereas there was no difference in chromatin accessibility at sites open in less differentiated MEP cells (Figure 4H; supplemental Figure 5E). Notably, these observations correlated with a decreased chromatin accessibility and mRNA expression at GATA1controlled erythroid genes such as nuclear factor erythroid 2 (Nfe2) or hemoglobin A1 (Hba-a1) (Figure 4I ; supplemental Figure 5F). Finally, we investigated the consequence of aberrant expression of these transcription factors on GATA1 activity in the GATA1-deficient G1E erythroid cell line in which terminal erythroid maturation can be induced by expression of exogenous Gata1 (Figure 4K). As expected doxycycline-induced Gata1 expression restored G1E erythroid differentiation with upregulation of Ter119 expression (supplemental Figure 5G). In contrast, ectopic expression of ETO2, ERG, SKI, or SPI1 significantly inhibited GATA1induced differentiation (Figure 4L-M).

Collectively, these data indicate that aberrant expression of ETO2, ERG, SKI, and SPI1 functionally interferes with GATA1 activity and restrains GATA1-dependent erythroid differentiation consistent with impaired differentiation observed in primary human AEL cells.

In vivo modeling of AEL from immortalized erythroblasts

We used complementary strategies to model in vivo the leukemogenic potential of AEL-associated alterations of the different molecular subgroups. First, to ascertain that in vitro-transformed erythroblasts can induce disease in vivo, we injected them into irradiated syngeneic recipients. ERG-, ETO2-, SKI-, or Tet2 +Gata1s-transformed, but not SPI1-or B4GALNT3-transformed, cells rapidly induced a fully penetrant fatal disease characterized mostly by the accumulation of CD71 Ter119 and few CD71 Ter119 blasts lacking expression of myeloid markers (supplemental Figure 6A-C). Histopathological analysis of symptomatic mice showed infiltration of BM, spleen, and livers by erythroblasts expressing nuclear GATA1 (supplemental Figure 6D-E). These results show that some epigenomic alterations found in human AEL have the potential to immortalize murine erythroblasts, which can then induce an AEL-like disease.

In vivo modeling of functional cooperation between AEL-associated alterations

Next, to define in vivo transforming capacities starting from healthy hematopoietic progenitors, we obtained oncogene-expressing Lin HSPCs (either by retroviral transduction or by breeding transgenic models) and assessed disease development upon engraftment into lethally irradiated recipients. Based on our observations that AEL subgroups 1 and 2 showed frequent cooccurrence of mutations (Figure 1B) and that GATA1 activity is targeted either directly (TET2+GATA1s and IDH2+GATA1 mutations in another cohort ) or through associated factors (TP53+ERG ), we investigated these 2 representative potential functional cooperation schemes.

Previous work has shown that both Tet2 loss-of-function and Gata1s alter erythroid differentiation but do not induce bona fide leukemia in vivo alone (supplemental Figure 7A). To address functional cooperation, we transplanted Tet2 +Gata1s Lin HSPCs into lethally irradiated recipients (Figure 5A). As opposed to recipients of Tet2 -only cells, recipients of Tet2 +Gata1s cells developed a rapid and fully penetrant lethal disease associated with high WBC, anemia, thrombocytopenia, and splenomegaly ( Figure 5B-C; supplemental Figure 7B). Flow cytometry analysis indicated that leukemic blasts were primarily CD11b Gr1 myeloid cells (Figure 5D) and histopathological analysis confirmed that BM and spleen were highly infiltrated by blasts with myeloid features (supplemental Figure 7C). Notably, we also observed emperipolesis that was previously described in murine GATA1s models. Together, these data demonstrate that Tet2 loss of function cooperates with Gata1s mutation to promote an AML-like phenotype in vivo.

TP53-mutated AEL samples are associated with other alterations, including aberrant expression of the transcription factor ERG (Figure 3F). Most AEL-associated TP53 alterations are DNA-binding missense mutations including TP53

. To address functional cooperation, we transplanted TP53 Lin HSPCs transduced with an ERG-expressing retrovirus. Because ectopic ERG expression in adult murine hematopoiesis was shown to primarily induce T-cell leukemia (supplemental Figure 7D-E), we assessed the long-term consequences of high ERG expression specifically in erythroid progenitors by transplanting purified ERG-transduced (GFP ) wild-type or TP53 erythroblasts obtained from primary recipients, into secondary recipients (Figure 5E). All recipients of TP53 erythroblasts overexpressing ERG developed a fatal leukemia with a median survival of 60 days, whereas recipients of ERG-expressing wild-type erythroblasts developed disease after 4 months (Figure 5F). The +ERG-induced disease was characterized by anemia, thrombocytopenia (Figure 5G), and the accumulation of CD71 Ter119 erythroid and to a lesser extent CD11b Gr1 myeloid progenitors in the BM (Figure 5H), with infiltration in spleen and liver (supplemental Figure 7F). These data indicate that an AEL-associated TP53 DNA-binding mutation cooperates with aberrantly high ERG expression to enhance the proliferative capacity of erythroid progenitors leading to leukemia with several features of the human disease.

Taken together, these results demonstrate that mutation associations in human AEL functionally cooperate to induce murine AML-like leukemia in vivo. Notably, although both combinations could readily induce an AEL-like phenotype when expressed in erythroid-restricted progenitors, their expression in HSPCs led to other mostly mixed leukemia phenotypes. Therefore, these data also suggest that the target cell in which these mutations are active impacts the disease phenotype.

In vivo modeling of erythroid transformation by aberrant SKI expression

To further explore the relevance of the cell context for consequences of the transcriptional alterations, we investigated in vivo disease development upon high expression of SKI, which was observed in 2 AEL samples of subgroup 3 (Figure 3G). First, we confirmed previous work showing that transplantation of Lin HSPCs retrovirally overexpressing SKI induced a lethal disease (supplemental Figure 8A) characterized by anemia, thrombocytopenia, and increased myeloid cells in the periphery (supplemental Figure 8B) associated with hypercellular BM and spleens showing high percentage of mostly myeloid or erythroid transgene-expressing cells (supplemental Figure 8C-E). Notably, GFP detection in all 3 myeloid, erythroid, and platelet lineages (supplemental Figure 8F) suggested that SKI overexpression may affect early multipotent stem or progenitor cells.

To investigate whether the transforming activity of SKI depends on the hematopoietic target cell, we purified, transduced and transplanted long-term multipotent HSC, erythroid-enriched (MEP), or myeloidcommitted granulocyte-macrophage (GMP) progenitors (Figure 6A). Three weeks posttransplant, transduced cells were detectable in the blood for all groups but engraftment in the BM was only observed in recipients from SKI-transduced HSCs and MEPs that later developed symptomatic diseases characterized by anemia, thrombocytopenia, (supplemental Figure 8H) and presenting with both CD11b Gr1 myeloid and CD71 Ter119 erythroid features (Figure 6B-E). SKI-transduced GMP recipients did not develop disease. Symptomatic recipients of HSCs or MEPs showed an increase in basophilic, polychromatophilic, and orthochromatic erythroblasts and in reticulocytes associated with a relative decrease in mature red cells (Figure 6F), suggesting that SKI delays but does not fully block erythroid differentiation in vivo. Histopathological analyses confirmed BM hypercellularity and revealed infiltration of erythroid cells in the spleen and liver (Figure 6G; supplemental 8I).

Taken together, these data indicate that high SKI expression transforms HSCs and MEPs, but not myeloidrestricted progenitors like GMPs. Although aberrant SKI expression in erythroid-restricted progenitors leads to pure erythroid proliferation, expression in more immature HSPCs resulted in increased selfrenewal capacity with aberrant differentiation toward both myeloid and erythroid lineages indicative of an AEL/MDS-like disease.

Discussion

AEL is an aggressive human cancer, often difficult to diagnose due to its close resemblance to other forms of hematopoietic malignancies presenting with variable compositions of cells with erythroid features, like MDS or certain AML subtypes. Here, we describe novel features of AEL that shed light on the pathophysiology of this disease. First, our data indicate that the majority of human AEL exhibit a unique erythroid transcriptional signature that differs from those found in patients with non-M6 AML or MDS without prominent erythropoiesis. Second, aberrant expression of various transcriptional regulators known to modulate GATA1 activity was frequently found in AEL and may represent a common molecular module that controls erythroid differentiation. Third, in vivo models demonstrate that the relative composition of the erythroid and myeloid features is strongly dependent on the hematopoietic target cell in which a driving oncogene is expressed, providing a basis for a better understanding of the highly heterogeneous clinical appearance of AEL. The genomic lesions described here are in line with previous reports, including the largest genetic landscape study of human erythroleukemia to date, and support classification of AEL patients into molecular subgroups. In our study, 3 subgroups were identified, including patients with TP53 mutations (36.3% of cases), patients with mutations in epigenetic regulators previously associated with clonal hematopoiesis of indeterminate potential (CHIP) and MDS (eg, DNMT3A, TET2, and IDH1/2 mutations) (33.3% of cases) and another group of patients presenting with none of these recurrent alterations (30.4% of cases). Although TP53 mutations and epigenetic mutations are not mutually exclusive, their frequencies within AEL samples are similar. In our sequenced AEL cases, we did not detect the other recently described subgroups, including those with NUP98, KMT2A, and other in-frame fusions, which could reflect the limited number of pediatric patients in our cohort. Also, consistent with the frequent association between FLT3 mutations and NPM1 or KMT2A alterations, our patient cohort lacked samples with FLT3 or NPM1 mutations. For samples sequenced with RNA-seq only, we cannot exclude the possibility that some structural variants or low expressed mutated transcripts remained undetected.

In contrast to previous studies, we also found out-of-frame fusion transcripts associated with altered expression of 1 of the partner genes. For example, the fusion between YWHAE and EPO in a TP53mutated patient was associated with ectopic expression of EPO, and the concomitant high expression of EPOR suggested an autocrine EPO/EPOR-signaling mechanism. Interestingly, alterations of multiple signaling intermediates, including downstream of EPO/EPOR, were recently found in up to 48% of human AEL samples and acquired activating KIT mutations were also essential to induce a bona fide erythroleukemia in a transgenic murine model, indicating the importance of signaling alterations for efficient oncogenic transformation of the erythroid lineage.

As the vast majority of the AEL-associated mutations are also found in a wide spectrum of human myeloid malignancies, it is essential to gain insights into their functional role in the erythroid phenotype that leads to a diagnosis of AEL. Together with the molecular alterations targeting EPO (YWHAE-EPO fusion) and the erythroid transcription factor GATA1 found here, the recently reported APLP2-EPOR and MYB1-GATA1 fusion genes further support the relevance of alterations in erythroid master regulators as underlying the erythroid phenotype in some AEL cases. However, most AEL do not present with erythroid-specific genetic alterations. Our patient-based transcriptional data, together with chromatin accessibility and functional analyses in cellular and in vivo models revealed that at least 25% of AEL cases present with transcriptional alterations ultimately interfering with GATA1 activity through direct or functional interaction within the GATA1 transcriptional complexes (eg, aberrantly expressed ETO2, ERG, SKI, SPI1). Although some of these transcriptional alterations (eg, ERG expression) were recently reported to have a genetic bases, the origin of some others remains to be determined (eg, SKI ). As reported previously, we noted that ectopic SPI1 expression was sufficient to immortalize erythroblasts in vitro but not to induce the disease in vivo, supporting the idea that cooperating alterations that have yet to be identified are required in SPI1 human leukemia.

Although the basis for the erythroid phenotype remains to be demonstrated in many cases, several epigenomic AEL alterations may also functionally converge on aberrant activity of erythroid master regulators. Indeed, a novel signaling pathway based on JAK2-mediated phosphorylation of TET2 leading to interaction with KLF1 was recently reported. Combined TET2 and DNMT3A inactivation was also reported to upregulate expression of KLF1 and EPOR in HSCs. Therefore, the concomitant TET2 and DNMT3A mutations observed in 2 AEL patients and the presence of TET2 and GATA1s mutations in another AEL sample support a functional synergism between alterations of KLF1 and GATA1 transcriptional programs leading to differentiation blockage. Based on these observations, we hypothesize that the erythroid phenotype in AEL results from a cooperation between genetic and transcriptional alterations. As proposed for other subtypes of leukemia, interference with the activity of altered erythroid master regulators, for example, through targeting of critical protein-protein interactions may therefore represent promising therapeutic strategies for AEL.

Our observations also have implications for the classification of AEL patients into molecular and/or prognosis subgroups. Comparative analysis of AEL expression signatures with normal erythroid and myeloid differentiation indicated that AEL is heterogeneously spread along a differentiation-associated trajectory with some patient samples clustering next to progenitors retaining myeloid features and other patient samples clustering closer to the erythroid trajectory. Also, although several oncogenes (eg, SKI) can transform restricted erythroid lineages, they led to mixed erythroid/myeloid hematopoietic malignancies upon expression in multipotent murine progenitors. These data indicate that the relative composition of myeloid vs erythroid elements at time of diagnosis is not solely based on the type of mutations but likely also reflects the type of progenitor targeted by these mutations. Notably, the relationship between geneexpression signatures and normal differentiation trajectories was not clearly visible when comparing the reported immunophenotypes of the blasts, and no correlation was found with the different molecular subgroups. These data strongly suggest that, in some AEL patients, the erythroid phenotype maybe initiated either by strong mutations that interfere with erythroid differentiation, or by mutations that provide advantages to erythroid-restricted progenitors. Alternatively, in others, the erythroid phenotype may originate from mutations in multipotent progenitors with a subsequent epigenetic drift toward the erythroid lineage.

Taken together, our work provided insights into the molecular mechanisms of the erythroid identity in AEL. Future studies need to resolve, likely at the single-cell level, the clonal genetic and epigenomic heterogeneous architecture in prospectively collected fresh samples as a further step toward the development of specific therapies. indicate patient samples presenting with genetic alteration (orange) or transcriptional alteration (blue) of GATA1-associated genes in 3 molecular subgroups of AEL: TP53-mutated, epigenetics, and others. (H) Kaplan-Meier survival plot of AEL patients grouped according to the presence (or absence) of genetic or transcriptional alterations defined in panel G. P value using log-rank Mantel-Cox test is indicated. (I) Kaplan-Meier survival plot of AEL patients from Iacobucci et al and grouped according to the presence (or not) of genetic or transcriptional alterations defined in panel G. P value using log-rank Mantel-Cox test is indicated. 67-70 dexamethasone). (D) A total of 3.5 × 10 sorted erythroblasts from WT, Tet2 , Gata1s (G1s), or Tet2 + Gata1s (Tet2 +G1s) mice were grown in liquid cultures over 15 days and viable cells were counted by trypanblue exclusion. Mean plus or minus SD (n = 3) is shown. (E) Dot-plot showing the log(fold changes) of the percentage of sequence for a given motif found under ATAC-seq peaks, between normal and transformed erythroblasts (expressing ERG, ETO2, SKI, and SPI1). (F) Histogram representation of the percentage of sequence with GATA1 (top) or ERG (bottom) motif found under ATAC-seq peaks of normal (Ctrl) and transformed erythroblasts. Statistical differences were calculated using the χ test. For GATA1 motif, Ctrl vs ERG: P = .0013; Ctrl vs ETO2: P = .0001; Ctrl vs SKI: P < .00001; Ctrl vs SPI1: P = .0032. For ERG motif, Ctrl vs ERG: P = .00001; Ctrl vs ETO2: P < .00001; Ctrl vs SKI: P = .0498; Ctrl vs SPI1: P = .0415. (G) Heatmap representing the hierarchical clustering of ATAC-seq signals, performed using normal (Ctrl) and transformed erythroblast by either ERG, ETO2, SKI, or SPI1, focused on GATA1-binding sites in normal Ter119 erythroblast (ENCODE). Heatmaps were focused on peak centers with ±5 kb. (H) Profile plot representing ATAC-seq signals performed using normal (Ctrl) and transformed erythroblast expressing either ERG, ETO2, SKI, or SPI1 on ATAC-seq specific peaks previously identified in mouse MEP, CFU-E, or proerythroblasts (proE). Profile-plot were focused on peaks centers with ±5 kb. (I) Visualization of GATA1 ChIP-seq peaks performed in normal erythroblast (ENCODE, first lane, gray) and ATAC-seq peaks of normal MEP, proerythroblast, orthochromatic erythroblast, and transformed erythroblasts expressing ERG, ETO2, SKI, or SPI1, focused on Nfe2, Hba-a1, and Hbq-a1 genes, using IGV software (v 2.3.88). (J) Quantitative Nfe2 mRNA expression measured by RT-qPCR in WT erythroblast (Ctrl) or erythroblast transformed with either SKI, ERG, ETO2, or SPI1 overexpression. Expression levels were also compared with the erythroleukemia MEL or the Ba/F3 myelolymphoid cell line. (K) Experimental design: G1E cell lines were cotransduced with a GATA1 doxycycline-inducible vector and with MSCV-vector expressing either ERG, ETO2, SKI, SPI1, or empty. (L) Flow cytometry histogram analysis of Ter119 expression in G1E cells expressing ERG (green), ETO2 (orange), SKI (blue), SPI1 (red), or empty control (gray), without (left) or with (right) induction of GATA1 expression. (M) Histogram representation of Ter119 expression detected by flow cytometry analysis of G1E cells expressing ERG, ETO2, SKI, SPI1, or empty control, without (gray) or with (red) induction of GATA1 expression. Statistical significance (in panels B, D, J, and M) is indicated as P values (Student t test except when otherwise specified). *P < .05; **P < .01; ***P < .001. Articles from Blood are provided here courtesy of The American Society of Hematology

Figures and Tables

+ - + + + + +

38 A

 38 transcriptome-based space maps AEL, MDS, and other AML to erythroid-and myeloidlineage trajectories

Figure 1 .

 1 Figure 1.

Figure 2 .

 2 Figure 2.

Figure 3 .

 3 Figure 3.

  , and CEBPA expression and predicted activity. (F) Histogram representation of ERG, GFI1, RUNX1T1, ETO2, GATA3, and SKI gene expression in AEL patients. Positive patient samples (red bars) were defined as presenting an expression above the threshold set as fourfold the average of AEL samples. Dotted bars represent the average expression of AEL samples. (G) Table

2 +Figure 5 .Figure 6 .

 256 Figure 5.

/ /

Acknowledgments

The authors are grateful to Olivier Hermine, Françoise Moreau-Gachelin, Michaela Fontenay, and Françoise Pflumio for expert advice and useful discussions. The authors also thank Suzana Atanasoski (Basel, Switzerland) for providing the human SKI cDNA, and Stéphanie Ranga and Michael Finnegan for handling human AEL samples and helping with cDNA cloning. One patient's samples were handled, conditioned, and stored by the FILOtheque (no. BB-0033-00073), Tumor Bank of the French Innovative Leukemia Organization (FILO) Group (Cochin Hospital, Paris, France). This work was supported by Institut National du Cancer (PLBIO-2014-176 and PLBIO-2018-169 [T.M.]), Ligue Contre le Cancer (PhD grant [A.F.], Équipe Labelisée [T.M.]), Institut National du Cancer (INCa)-PlanCancer ''Soutien pour la Formation'' 2009-2013 (C.I.), Cancéropôle Ile-de-France (2014-2017 [C.K.L.]), Sites de Recherche Intégrée sur le Cancer (SIRIC)-SOCRATE (INCa-DGOS-INSERM 6043 [T.M.] and INCa-DGOS-INSERM_12551 [O.A.B.]), Fondation pour la Recherche Médicale (C.I., Z.A. [FRM-ING20150532273], and C.K.L.), Fondation de France (FdF-00057925 [C.T. and T.M.]), and the Gustave Roussy Genomic Core Facility (Taxe d'Apprentissage TA2018-ALFA [A.F.]). J.S. was supported by grants from the Swiss Cancer League (KFS-3487-08-2014), the Gertrude von Meissner Foundation (Basel, Switzerland), the San Salvatore Foundation (201525; Lugano, Switzerland), the Wilhelm Sander Foundation (2017.035.1; Munich, Germany) and the Swiss National Science Foundation (SNF, 31003A_173224/1). U.M. was supported by the National Institutes of Health, National Cancer Institute (2RO1CA176647) and the Stony Brook Foundation (Carol Baldwin Foundation). P. Valent was supported by the Austrian Science Fund (FWF) grant F4704-B20 and a stem cell grant of the Medical University of Vienna. P. Vyas was supported by the Bloodwise and Children with Cancer Specialist Programme (grant 13001), the National Institute for Health Research (NIHR) Oxford Biomedical Research Fund, and the Medical Research Council Molecular Haematology Unit (MRC MHU; MC_UU_12009/11). E.A. was supported by the Fundación Hay Esperanza. C.M. was supported by the Leukemia & Lymphoma Society Translational Research Program, National Institutes of Health and National Cancer Institute Outstanding Investigator Award R35 CA197695, and the American Lebanese Syrian Associated Charities of St. Jude Children's Research Hospital. E.S. was supported by Fondation pour la Recherche Médicale (Equipe FRM DEQ20180339221), the ATIP-Avenir Program (Plan Cancer), and Labex EpiGenMed (Investissements d'Avenir Program, reference ANR-10-LABX-12-01). 94800 Villejuif, France; e-mail: thomas.mercher@inserm.fr; or Juerg Schwaller, University Children's Hospital Basel (UKBB), Department of Biomedicine (DBM), University of Basel, ZLF-Laboratory 202, Hebelstrasse 20, CH-4031 Basel, Switzerland; e-mail: j.schwaller@unibas.ch.

Footnotes

Contact the corresponding authors for original data.

The publication costs of this article were defrayed in part by page charge payment. Therefore, and solely to indicate this fact, this article is hereby marked "advertisement" in accordance with 18 USC section 1734.

Supplementary Material

The online version of this article contains a data supplement.


Hussein Chaaban

Michèle Gouiffès

Annelies Braffort

Automatic Annotation and Segmentation of Sign Language Videos: Base-level Features and Lexical Signs Classification

Keywords: LSF Videos, Annotations, Lexical Signs, Sign Segmentation

The automatic recognition of Sign Languages is the main focus of most of the works in the field, which explains the progressing demand on the annotated data to train the dedicated models. In this paper, we present a semi automatic annotation system for Sign Languages. Such automation will not only help to create training data but it will reduce as well the processing time and the subjectivity of manual annotations done by linguists in order to study the sign language. The system analyses hand shapes, hands speed variations, and face landmarks to annotate base level features and to separate the different signs. In a second stage, signs are classified into two types, whether they are lexical (i.e. present in a dictionary) or iconic (illustrative), using a probabilistic model. The results show that our system is partially capable of annotating automatically the video sequence with a F1 score = 0.68 for lexical sign annotation and an error of 3.8 frames for sign segmentation. An expert validation of the annotations is still needed.

INTRODUCTION

Sign languages (SL) are natural gestural languages that are used mainly by the deaf community. A SL discourse consists of a sequence of signs performed by the hands at a specific location and configuration, accompanied by non-manual components like facial expressions and upper body movements. These sequences of manual and non manual components make it possible to transmit information in parallel during the discourse.

Currently several studies are interested in the problem of the automatic processing of SL, and more particularly of the recognition of the lexical signs, which are only one of the type of signs present in a SL discourse. In the following, for simplification, we will use the term SL recognition for lexical sign recognition. Although a large part of the lexical signs are defined in a dictionary, there is a very large variability linked to the context during their realization. In addition, the signs are often separated by co-articulation movements (transitions). This extreme variability and the co-articulation effect represent an important problem in the automatic processing of SL research field. It is therefore necessary to have a huge number of SL videos annotated, to study linguistic and to build a https://orcid.org/0000-0002-7152-4640 b https://orcid.org/0000-0003-4595-7714 large data-sets for machine learning. SL are not well understood nor described formally. Linguists and researchers in Human Movement attempt to understand which movement, which gesture or which body or face components (from the eyebrows to the hands) are key in forming a given message.

Certain annotation levels of SL videos require to isolate each individual sign before describing the hands movement and the facial/body events that accompany it. To date, the temporal segmentation and motion descriptions of signs are carried out manually using annotation tools such as ANVIL [START_REF] Kipp | Anvil -a generic annotation tool for multimodal dialogue[END_REF] or ELAN [START_REF] Wittenburg | Multimodal annotations in gesture and sign language studies[END_REF]]. Though these manual annotations present many problems. In addition to the significant time required for this work, the results are extremely variable because each annotator has its own criteria to estimate the beginning and end of the sign. This leads us to question ourselves on the issue of temporal segmentation and to propose an original approach to speed up this stage of video processing to study the SL. Therefore, in this paper, we propose an automatic annotation system for face and body annotations such as mouthing, head direction, location of signs. Also, the system segments automatically signs, using hands movements without any prior learning phase. In other words, no huge manually annotated data are required for automatic annotation nor for segmentation. After that we propose an algorithm to classify the type of the segmented sign using the automatically generated annotations. In the next section we present some of the works done in the SL processing field. Sections 3, 4 and 5 describes our proposed method. Then, Section 6 discusses the evaluation method, the dataset on which we worked on and the results. In the last section we present our conclusions and the perspective of our work.

RELATED WORK

Nowadays, most of the works dedicated to the SL processing are interested in the recognition of lexical signs. Many attempts of SL recognition are conducted on isolated signs [START_REF] Rastgoo | Videobased isolated hand sign language recognition using a deep cascaded model[END_REF], [START_REF] Lim | Isolated sign language recognition using convolutional neural network hand modelling and hand energy image[END_REF]. However, in a natural speech of SL, it is often difficult to find precisely the beginning and the end of each sign because of the problems of coarticulation we mentioned before. In addition, most of these works focus on specific datasets, made in controlled environments (uniform background, signer with dark clothes) and dealing with a specific topic, such as weather [START_REF] Koller | Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers[END_REF]. But the real challenge in SL recognition remains in identifying dynamic signs, i.e. signs in continuous SL speech, and most importantly independently of the signer [START_REF] Liang | 3D Convolutional Neural Networks for Dynamic Sign Language Recognition[END_REF].

In the field of SL recognition, the methods which exploit Hidden Markov Models (HMM) remain among the most used [START_REF] Fatmi | American sign language recognition using hidden markov models and wearable motion sensors[END_REF], [START_REF] Wang | American sign language recognition using multidimensional hidden markov models[END_REF]. These approaches identify specific information in a signal. It is then a question of comparing the signal of a sign to a previously learned model. The recognition and segmentation processes are then carried out simultaneously. This poses a great limitation in recognizing many types of iconic structures such as the depicting signs, or classifiers [START_REF] Cuxac | La langue des signes franc ¸aise (lsf). les voies de l'iconicité[END_REF] which cannot be learned due to their excessively large number and their variability depending on the context. Recently, with the great rise of the Convolutional Neural Networks in the machine learning field, there was no exception of deploying these networks for the SL recognition. Many researchers adopted this technique in their works as it showed high recognition rates [START_REF] Rao | Deep convolutional neural networks for sign language recognition[END_REF]]. But the main problem of this approach is the need of enormous amount of annotated data to train the network. Unfortunately this amount of data is not easily available especially for non-American sign languages. Therefore many attempts of recognition were limited to a certain number of signs [START_REF] Pigou | Sign language recognition using convolutional neural networks[END_REF].

Concerning automatic annotation of SL, some works represent a sign as a series of movements and configuration and try to annotate the segments by describing facial and body events as mouthing, gaze, occlusion, hands locations, hand shapes, and movements without any information about the signs themselves [START_REF] Naert | Per channel automatic annotation of sign language motion capture data[END_REF], [START_REF] Yang | Efficient generation of large amounts of training data for sign language recognition: A semi-automatic tool[END_REF]. [START_REF] Lefebvre-Albaret | Une approche de segmentation de la langue des signes franc ¸aise[END_REF] worked on the automatic segmentation and annotation of signs to provide reusable data for virtual signers animation. Their algorithm is based exclusively on the 2D movements of the hands in the video. But it requires a human operator to point, during the viewing of the video, a frame (which they call primer) so that each sign has one and only one primer. (Gonzalez Preciado, 2012) proposed a better way to segment signs using hands movements and configurations. The annotations provided are exclusively describing the hands with no information about the other non manual components.

FEATURES ANNOTATION

To study LS and use machine learning methods, it is necessary to have loads of annotated videos. In annotations, the linguists extract visual features from the video. Using statistics on these collected manual and non-manual features, linguistic models can be built. These annotations are done manually by linguists or SL experts. They are subject to error, depending on the SL knowledge of the annotator. Furthermore, they are non-reproducible and extremely time consuming. Thus, automating this task would be a saving of time and robustness.

The experts of SL can annotate the manual and non manual components at various levels: from low levels (face and body features and events) to high levels (structure of the discourse). In our automatic annotation, we are interested in the automatic base level features annotation. In Sign languages, these features are the lowest level units are. They are meaningless on their own. But they can be interpreted at a higher level for annotating more complex linguistic units. For example lexical level annotation can be generated using features extracted at the base level. As far now, there is no predefined list of manual and non manual components to annotate. In the literature, experts try to annotate hand shapes, their locations, motion, direction, symmetry between them, mouthing, mouth gestures, gaze and eyebrows.

Body and Face Landmarks

The first step to annotate these base level features, is extracting the different body and face parts. Many works used image segmentation techniques to locate these articulators. Those methods usually require specific environments and tools such as uniform backgrounds, colored gloves,... In our work we use Open-Pose [START_REF] Cao | OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields[END_REF] (figure.1), a recent real time body pose estimation library, which provides the coordinates of different body landmarks with high success rates and OpenFace [START_REF] Baltrusaitis | Openface 2.0: Facial behavior analysis toolkit[END_REF] (figure .2) a toolkit capable of facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.

Controlled Vocabulary

Once we get the coordinates of the different body and face landmarks, we proceed to calculate further features such as mouthing, gaze, etc. In manual annotation software, the ensemble of these features are called controlled vocabulary. Till this date, there is no one standard controlled vocabulary list defined. Each linguist annotates the events that are relevant for his studies. With our automatic annotation system, we annotate the features that are frequently annotated in the literature: mouthing, gaze/head direction, bi-manual motion, signing space and hands-head distance. To annotate mouthing, we calculated the isoperimetric ratio (or circularity) of the interior of the lips using the coordinates provided by OpenFace and the formula: IR = 4πa p 2 where a is the area of the interior of the lips and p is its perimeter. The higher the ratio is, the more the mouth is open, which is an indication of a mouthing. The hands-head distance is simply calculated using the coordinates of both wrists ((x W Right , y W Right ),(x W Le f t , y W Le f t )) and nose (x Nose , y Nose ) (center of the face). For the bi-manual motion, we measure the speed and direction of each hand ((V W Right ,D W Right ),(V W Le f t ,D W Le f t )). The correlation of these two information between the two hands indicates whether these latter are symmetrical or moving in an opposite motion. To determine the location of the sign in the space, we compare the abscissa of the neck (x Neck ) to the abscissas of both wrists (x W Right , x W Le f t ) to decide if the sign is centered or not. As for the gaze/head direction, we use OpenFace directly for the task. The calculation of these features is more detailed in our previous article [START_REF] Chaaban | Towards an Automatic Annotation of French Sign Language Videos: Detection of Lexical Signs[END_REF].

TEMPORAL SIGN SEGMENTATION

Signs can be produced sequentially, separated by transitions and pauses. In order to study the grammar of the SL or to automatically recognise them, one possible strategy consists in segmenting the video before analyzing the resulting portions of videos. The temporal segmentation of a sign means finding its boundaries, that is when the sign starts and when it ends.

In the manual annotation software, the linguists segment the signs manually, which takes hours of work.

In addition, the resulting segmentation is, as mentioned previously, non-reproducible since it depends on the subjectivity of the annotator and his/her experience. We propose an algorithm that segments the signs automatically using only hands motions, pauses and hand shapes. The algorithm does not need any learning phase thus annotated data are not required.

For the segmentation, we use the coordinates of both wrists to calculate and draw the speed variation of each hand (see figure.3a). Then, a Wiener filter is applied to smooth-in the signal and the local maxima and minima of speed are detected (as shown by figure .3b).

When comparing the detected minima and maxima in the signal to the beginning and the end of manually annotated signs (disregarding the sign category which we will discuss later on) we can see that each sign starts with a maximum of speed of one or both hands and ends with a minimum of speed, as can be shown on Figure .4. This observation is the basis for our temporal segmentation algorithm. To avoid the over-segmentation of signs that include repetitive movements, we rely on the hand shapes. Since [START_REF] Stokoe | A dictionary of american sign language on linguistic principles[END_REF], linguists describe the lexical signs using three parameters that are hands shapes, their locations and their movements. While in some cases the hands shapes may change during a sign, they remain stable most of the time. Thus, after our initial segmentation using the hands speed, we scan the video sequence a second time and merge consecutive segmented signs of similar hand shape. These latter are mostly segments of a repeated movements forming one sign. For the hand shape detection, we use a pre-trained model that distinguish 60 different hand configurations developed by [START_REF] Koller | Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled[END_REF]. Our algorithm for sign segmentation is represented in the diagram of the figure 5.

SIGN TYPE ANNOTATION

In section 3, we presented the base level features annotation. As we described, at that level, the annotated features are meaningless on their own. In this section, Detect coordinates of both wrists ((x W R , y W R ) and (x W L , y W L )) using openpose Draw hands speed graphs (V W R ,V W L )

Smoothing using Wiener filter

Detect local minima and maxima

Segment signs using local minima and maxima of hands speed Merge consecutive signs with same hands shapes we use the automatically annotated manual and non manual components to annotate a more meaningful features on a higher level: the lexical level. For this level, we do not annotate the meaning of the signs as we do not have enough data for it, but rather their types. [START_REF] Cuxac | La langue des signes franc ¸aise (lsf). les voies de l'iconicité[END_REF] described two main types of signs: the lexical signs (LS) which are conventionalized signs that can be found in a dictionary and the highly iconic structures (HIS) (depicting signs or classifiers) which are signs that are partly conventionalized and partly context-dependant, their form changes to describe sizes, shapes, situations and role playing. With a conventionalized form, therefore easier to identify, we are more intrigued in annotating the lexical signs after the segmentation. The automatic annotation of lexical signs will definitely save time for the linguists to study the SL in comparison to manual annotation. Furthermore these annotations will accelerate the automatic recognition of signs since the dictionary of candidate signs will be reduced almost to half the size by dividing it into two classes: lexical signs and non-lexical signs.

In the literature, the automatic annotation of lexical signs is almost absent. Most of the works study the automatic recognition of signs. For that reason, we had to explore the field to discover which manual and non manual features allow us to identify the lexical signs. We started by an analysis study for each of the features that we automatically annotated. In this study we represented the occurrence of each of the features during the production of each type of signs. The type of the signs were annotated manually by an expert on the video sequences. The resulting histograms showed that the distribution of mouthing, head direction, location of signs in the space and the hand-head distance are normal distributions with two distinct peaks for each type of signs. We drew the histograms for the 4 features. As an example, figure.6 shows the mouthing distribution between two types of signs w.r.t the isoperimetric ratio of the mouth. The other histograms (of the other features) have a similar shape, i.e. two semi separated peaks. The correlation ratio between the various features, displayed in Table .1, shows that they are not correlated, in other words independent.

P(x = x k | C) = 1 2πσ 2 k e - (x i -µ k ) 2 2σ 2 k (1)
Where C is the class of the sign: lexical or not.

To annotate the new frames of the new SL video sequences as lexical sign frames, the models are combined into one decision rule, as follows:

P(Lexical | F1, F2, F3, F4) = P(Lexical) 4 ∏ i=1 P(x i | Lexical).
(2)

P(Lexical | F1, F2, F3, F4) = P(Lexical) 4 ∏ i=1 P(x i | Lexical).
(3) where F1 is Mouthing, F2 is head pose, F3 is sign location and F4 is hand-head distance ( the bimanual motion feature was dropped of our study as it did not add any weight to the classification of the signs). Then, we compare (2) and ( 3). If the result of ( 2) is higher than the result of (3) the new frame is part of a lexical sign, otherwise it is considered as non-lexical. After the temporal segmentation in section 4, a classifier is applied on every frame of each segmented sign. A segment is classified as lexical when the majority of frames are classified as lexical, otherwise it is considered as non-lexical.

EXPERIMENTAL METHOD

In this section, we present the performances of our system to automatically annotate base level features, segmentation and sign category (lexical or not). To our knowledge, there is no public dataset with similar automatic annotations, on which to compare our results. Thus, our evaluation is based on quantitative measures of the segmentation and classification of lexical signs only in comparison to a ground truth dataset manually annotated by an expert.

Dataset

The dataset used in our work is a portion of MOCAP 1 dataset: a collection of 2D RGB videos in French Sign Language. The dataset includes 49 videos with 4 different signers filmed from hip up face view and equipped with motion detectors for 3D representation (figure.8) used other study purposes since this dataset was not produced specifically for our work. We note that the presence of the motion detectors did not prevent the detection of face, hands and body keypoints in our 2D RGB image study. The length of the videos varies from 15 to 34 seconds with an average of 24 seconds, and the dataset contains a total 19.63 min of videos. In each video, the signer was asked to describe an image that represents a scene such as a living room (figure.9) or a forest. 25 images were chosen delicately to have a variation between lexical signs and HIS. The segmentation and the annotation of sign type (Lexical and HIS) were done manually by an expert: 1011 signs were annotated, 709 were lexical signs and 304 were HIS.

Evaluation

As described before, our algorithm starts by annotating features such as mouthing, head direction, location of sign. Then, using hand motions, the frames are scanned to find the boundaries of each sign segment. After that, the automatically generated features and a portion of manual annotations provided by an expert are used to create a probabilistic model. The latter classifies the segmented signs into lexical and non lexical signs.

In this section, we measure the performance of the algorithm to locate the lexical signs and find its boundaries in comparison to the ground truth. Since our ground truth is based on manual annotations which are dependent on the annotators knowledge and experience, the evaluation results have to be taken with caution.

As we described in section 6.1, our dataset is divided into training data and testing data. For the evaluation, we compare the automatically generated annotation files to the rest of the manual annotations which were not used in the learning phase to create the classifier. Using the testing dataset, we count the number of true positives (TP) of the detected signs, the false positive (FP), the true negative (TN) and the false negative (FN) ones. We consider that a lexical sign is correctly detected when 3 consecutive frames classified as lexical fall in the range of the manually annotated sign (figure. 10). Then we compute the TP and TN rates (TPR and TNR), i.e. the sensitivity and the specificity, the positive prediction value (PPV) (or precision) and F1-score: We calculated the same measures with frame by frame comparison were each frame from the automatic annotations is compared to the corresponding frame in the manual annotations to determine whether it's a TP, FP, TN, or FN frame to give an approximate measure on the precision of the classification in terms of segmentation. The performance of the proposed segmentation approach has been quantitatively evaluated as well. For each detected sign, we calculate the difference between the first frame of the automatically segmented lexical sign and the first frame of the manually segmented lexical sign. We do the same thing for the last frames of both automatically and manually segmented lexical signs. The mean of the sum of both differences over all the correctly detected lexical signs represents the segmentation error of our algorithm. The results of the segmentation using the hands motion alone and the hands motion corrected by the hands shapes are compared as well.

Results

The results of our method are evaluated for each signer individually and then combined to check if the classification is independent of the signer.

quences. The segmentation was carried out by considering only the information coming from the hands, which makes our method applicable to all sign languages. We first used the hands motion calculated using body landmarks provided by OpenPose. Then we corrected the segmentation using the hands shapes.

Once we got all signs isolated individually, we built a probabilistic model using a portion of MOCAP dataset which was annotated manually by an expert. The model was used as a classifier to distinguish lexical and non lexical signs. To evaluate our algorithm of classification we used the sensitivity, the specificity, the precision and the F1 score metrics. The results showed that our algorithm was capable of detecting the lexical signs with a F1 score = 0.68 and that the use of hands shapes for segmentation improved the detection (F1 score improved by 0.13). To evaluate the segmentation approach we calculate the average difference between the beginning and end frames of annotated and detected signs (3.8 frames). In the future, we will try to refine both of the segmentation and the classification results by including more features that could be useful for the task. On a parallel axis, we will use the set of the annotated features to create sub-categories of signs based on the similarities between features. Such categorisation would accelerate the process of automatic recognition, make it more efficient.

Figure 1 :

 1 Figure 1: Body landmarks using Openpose.

Figure 2 :

 2 Figure 2: Face landmarks using OpenFace.

  (a) Hands speed variation. (b) Filtered speed with local minima and maxima.

Figure 4 :

 4 Figure 4: Annotated signs compared to the hands speed.

Figure 5 :

 5 Figure 5: The sign segmentation algorithm.

Table 1 :

 1 Correlation between the different features (M: Mouthing, H-D: Head Direction, Bi-M: Bimanual Motion, S-L: Sign Location, HH-D: Head-Hand Distance).

Figure 6 :

 6 Figure 6: Histogram of mouthing occurrence (represented by the isoperimetric ratio of the mouth) during the production of lexical signs and HIS. Using these histograms, the average µ k and the variance σ 2 k of each feature x k , we built a probability distribution model (figure.7) with the equation for a normal distribution parameterized by µ k and : σ 2 k

Figure 7 :

 7 Figure 7: Probability distribution model for mouthing (using the isoperimetric ratio).

Figure 8 :

 8 Figure 8: One frame of the MOCAP dataset.

Figure 9 :

 9 Figure 9: Examples of scenes to be described by the signers.

Figure 10 :

 10 Figure 10: Counting of False/True Positives and False/True Negatives.

VISAPP 2021 -16th International Conference on Computer Vision Theory and Applications

The corpus is downloadable here (for research only): https://www.ortolang.fr/market/corpora/mocap1.

Intra-signer Study. For each signer, the videos and their corresponding manual annotations are divided into 3 subsets L 1 , L 2 and L 3 . Two of them (L i , L j ) = (L 1 , L 2 ), (L 1 , L3), (L 2 , L 3 ) are used for learning and the last one for testing. A cross-validation is performed, by collecting the results of each experiment. Inter-signer Study. Here, the videos are divided into 4 subsets L 1 , L 2 ,L 3 and L 4 , each subset includes all the videos from the same signer. Again we tried all the different combinations of subsets for learning and testing with three subsets for learning and one subset for testing. Firstly, the performance measures for intra and intersigner studies were calculated after a temporal segmentation using only the hands motion. Then we corrected the segmentation with the hands shapes and recalculated the performance measures. The averages and standard deviations of both experiments and for both studies are shown in the tables 2, 3, 4 and 5. The tables show that our algorithm is able at a certain level of annotating automatically the lexical signs. The similarity between the results obtained for intrasigner and for inter-signer experiments indicates that our algorithm is signer independent. When comparing the results of classification after segmentation using only hands motion and segmentation using both hands motion and shapes, we can see clearly that the use of the hands shapes to correct the segmentation was beneficial.

To evaluate the performance of our segmentation technique, we calculated the same metrics with frame by frame comparison between manual annotations and automated annotations. The results are shown in table 6. These results though do not reflect the real performance of the segmentation approach as it includes the false positive and the false negative detected signs. Therefore we isolated the true positive detected signs and measured the ability of our algorithm to detect precisely the beginning and the end of each correctly detected sign. We calculated for each sign the number of frames between the detected beginning frame and the annotated beginning frame as well as the difference between the detected end frame and the annotated end frame. The average of this differences was 3.8 frames which represents our error range. Our automatic annotation algorithm is not perfect, and we cannot pretend that our annotations can be used directly for other linguistic studies. An expert validation for the automatic annotations is needed after every processing. Thus our work can be considered as an assistance tool which can with no doubt simplify the annotation of face/body features and lexical signs for the linguists and reduce the time spent for the process. The limitations of our algorithm are due to several factors starting with the particularity of the sign language and its grammar, as some lexical signs may change form depending on the context and some were created from the same repeatedly used HIS. Another factor is the subjectivity of the annotations used for training which makes our results subjective as well. Some classification errors were also due to the errors in the detection of facial and body landmarks.

CONCLUSIONS

This paper has proposed a tool that will be useful for linguists to pre-annotate Sign Language videos. We started by annotating face and body features such as mouthing, sign location, etc. Then we detailed a temporal segmentation of lexical signs in video se-