Bertsy Goic

Nicolas Vodovar

Juan A Mondotte

Clément Monot

Lionel Frangeul

Hervé Blanc

Valérie Gausson

Jorge Vera-Otarola

Gaël Cristofari

Maria-Carla P Saleh

Gael Cristofari

RNA-mediated interference and reverse transcription control the persistence of RNA viruses in the insect model Drosophila

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

host 1,2 . The outcome of such host-pathogen interactions is highly variable and ranges from deleterious infections with lethal or permanent damage to completely innocuous infections 3 .

For example, acute viral infections are characterized by a high rate of viral replication and the production of a large number of progeny. Replication is transient and is limited either by death of the infected cells or by clearance of the virus by the host immune response. In contrast, persistent infections may be the result of an acute primary infection that is not cleared. In this case, the ability of the virus to be transmitted to other organisms or to the offspring of the host is maintained. Persistent infections are at the boundary that separates deleterious infections from innocuous infections. In this unique circumstance, the virus and host use attack and counterattack machinery to reach an equilibrium at which viral infection is controlled but not eliminated. Insect-virus interactions are useful models with which to delineate persistent infections, because many insect viruses develop a persistent infection without clear fitness costs to the host 4,5 . Furthermore, many persistently infected arthropods, and insects in particular, can act as vectors for emerging viral infectious diseases with a strong medical and economic effect, such as West Nile Virus or Dengue virus 6 .

Flock house virus (FHV) belongs to the Nodaviridae family and is a nonenveloped virus with a bisegmented genome (RNA1, 3,107 nucleotides; RNA2, 1,400 nucleotides) of positive single-stranded RNA with a 5 terminal methylated cap and a nonpolyadenylated 3 end. FHV is a useful viral model because it can produce acute and persistent infection in cell culture as well as in animal models 7,8 . Initial efforts to characterize persistent infections in vitro indicated that the FHV genome is unaltered during the establishment of persistence and that mutations on the viral genome begin to accumulate only after multiple passages on persistently infected cells 9 . Of note, mutations accumulate in RNA2, which encodes the coat protein, but not in RNA1, which encodes the viral RNA-dependent RNA polymerase and B2, a strong suppressor of RNA-mediated interference (RNAi) 10 . Those observations suggest that a change in the cellular physiology rather than the virus itself is responsible for establishing the persistent state.

However, the molecular and cellular mechanisms underlying this process have remained unsolved. Other studies have associated the appearance of defective interfering particles with persistent infection with FHV 9,11 or other RNA viruses [12][13][14][15] . Defective interfering particles are unable to complete a full replication cycle because of genome deletions and consequently need wild-type viruses to replicate their genomes. Such particles can also interfere with the replication of wild-type virus through competition for viral or host factors essential for replication, facilitated by their replicative advantage due to the smaller size of their genome 16 .

It has been suggested that during persistent infection of Drosophila melanogaster cells with FHV, the RNA derived from such particles is a chief contributor to the formation of virus-derived small interfering RNA (vsiRNA), because double-stranded RNA (dsRNA) from defective interfering particles could be processed more efficiently by the RNAi machinery than viral dsRNA replicative intermediates 11,17 . The RNAi machinery is also important in maintaining persistent infections in Drosophila cell lines 18 . That study suggests that direct dicing of the viral dsRNA replicative intermediate might be one mechanism that allows control of the viral infection, as the bulk of FHV-derived siRNAs are not loaded into the RNA-silencing effector proteins Argonaute 1 and Argonaute 2 (Ago2). Even if the 'dicing hypothesis' was able to explain how viral replication is controlled during long-lasting infections, it does not explain how the persistent state is established, mainly because that study used cells already persistently infected with FHV.

In this work, we sought to understand how viral persistence is established and maintained in insects. We found that Drosophila cells and flies infected with FHV or other positive single-stranded RNA viruses generated DNA of viral origin through endogenous reverse-transcriptase activity. We further demonstrated that those viral DNA forms were transcribed and produced vsiRNAs that 'fed' the RNAi antiviral machinery.

RESULTS

Characterization of persistently infected Drosophila S2 cells

To study how viral persistence is established and maintained in insects, we infected naive Drosophila S2 cells (S2n) by limiting dilution 9,11 with several RNA viruses, including the positive single-stranded RNA viruses FHV and Drosophila C virus (DCV), and the dsRNA virus Drosophila X virus (DXV; Supplementary Fig. 1a). Cells that survived the lytic infection proliferated and remained persistently infected even after 35 passages (Supplementary Fig. 1b). We further characterized the S2 cell lines persistently infected with FHV (S2p). Immunostaining of S2p cells with antibody to FHV capsid showed that all cells were homogeneously infected by FHV (Fig. 1a), which excluded the possibility of the presence of cells refractory to infection. Furthermore, S2p cells did not show a difference in proliferation (Fig. 1b) or death (Fig. 1c) relative to that of S2n cells, which indicated that persistent infection did not impose any 'fitness cost' on the S2p cell population. To exclude the possibility that selection of rare initial events contributed to the establishment of persistence independently of the virus, we tested the resistance of S2p cells to apoptosis. Both S2p and S2n cells were similarly sensitive to ultraviolet irradiation (Fig. 1d) indicating that S2p survival to infection was not due to a defect in apoptosis. We also tested the infectivity of the virus produced by S2p cells. Wild-type (w 1118 ) flies infected with 500 TCID 50 (half-maximal tissue culture infectious dose) of virus recovered from the supernatants of either S2p cells or acutely infected S2n cells died at a similar rate (Fig. 1e), which indicated that persistence was not established from a less-virulent virus population or from a loss of virulence during infection. As neither cell fitness nor FHV virulence was altered in S2p cells, we next hypothesized that the persistence could have resulted from the control of viral replication below a cytopathogenic threshold that would be accompanied by lower production of virus in S2p cells 11 . We compared viral titers after acute and persistent infection and observed that viral titers were significantly lower in S2p cells (Fig. 1f); accordingly, there was also less accumulation of viral RNA segments during persistent infection (Supplementary Fig. 1c). As viral titers varied in S2n cells versus S2p cells, we analyzed differences in the antiviral response. In insects, the main antiviral response acts through the canonical Dicer-2-Ago2 siRNA pathway 10,[19][20][21] . To assess the antiviral RNAi response in S2p and acutely infected S2n cells, we produced small RNA libraries and examined the vsiRNA profiles. vsiRNAs that mapped all along both FHV RNA1 and RNA2 segments were found (Supplementary Fig. 1d-g), which indicated that the RNAi machinery effectively processed the viral dsRNA in both conditions. Together these observations showed that persistently infected cells produced less virus because of control of viral replication by an unknown cellular mechanism.

New cellular synthesis of viral cDNA from viral RNA

RNA from non-retroviral RNA viruses can be reverse-transcribed into cDNA by retrotransposons or endogenous retroviruses [22][23][24][25][26][27][28][29][30] . The role of non-retroviral DNA forms of RNA viruses remains unresolved, although involvement in immunity has been proposed 25,26 . Hence, we investigated whether RNA viruses generated DNA forms in Drosophila cells and whether those DNA forms correlated with the establishment and maintenance of persistent infection. We extracted genomic DNA from S2 cell lines persistently infected with FHV, DCV or DXV and amplified the DNA with primers located in various regions of the viral genomes. All samples contained DNA sequences (Fig. 2a and Supplementary Fig. 2a,b), a result we further confirmed by sequencing.

Sindbis virus, an arbovirus that naturally produces persistent infection in insects, also produced a DNA form (Supplementary Fig. 2c). When we treated DNA samples with RNase III, a mixture of RNase A and RNAse I, DNase I or exonuclease I, only DNase I precluded the generation of a PCR product (Supplementary Fig. 2d and data not shown), which confirmed that the molecular template was a DNA molecule.

Through the use of 'genome walking', we extended the initially identified sequences, corresponding to the FHV RNA1. We reconstructed the FHV DNA forms present in S2p cell lines (Fig. 2a and Supplementary Fig. 3a). The DNA form was heavily reorganized, with a major recombinant RNA1 segment considerably shorter than the usual 3107 nucleotides (Supplementary Fig. 3a,b). Nonhomologous RNA recombination during negative-strand synthesis of FHV RNA1 and RNA2 could have been the template for those new DNA structures 31 . Alternatively, defective interfering particles could have served as a template 32,33 , as the DNA forms had breakpoints and rearrangements similar to those identified in RNA1 defective interfering particles 11 . Of note, we also identified DNA forms derived from FHV RNA2 that were similar in sequence to RNA2 defective interfering particles 32 (data not shown). We then infected S2n cells with FHV at a multiplicity of infection (MOI) of 0.5 and monitored the appearance of FHV DNA over time by PCR. The DNA form was detectable as early as 12 h after infection (Fig. 2b).

Because reverse transcriptases encoded by retrotransposons and endogenous retroviruses are widespread in insect genomes 34,35 , we determined if we could detect reversetranscriptase activity in S2 cells. We detected robust Mn 2+ -dependent reverse-transcriptase activity in extracts of S2n cells (Fig. 2c). Additionally, we found that such activity was sensitive in vitro to the nucleoside reverse-transcriptase inhibitor azidothymidine (AZT) triphosphate to a degree similar to that achieved by a recombinant retroviral reverse transcriptase (Fig. 2d). Next we determined whether treating S2n cells with AZT would inhibit the appearance of FHV DNA after infection with FHV. Indeed AZT triggered a dose-dependent inhibition of FHV DNA in S2n cells infected with FHV at an MOI of 0.5 (Fig. 2e). We confirmed that AZT did not impair S2 cell growth at the concentrations and time used (Supplementary Fig. 4a). Moreover AZT did not inhibit FHV replication in persistently infected cells in which the DNA form was already present (S2p cells; Supplementary Fig. 4b,c). Therefore, AZT seemed to be specifically blocking the generation of viral DNA rather than affecting cell or virus viability.

As mitochondrial dysfunction is a known potential side effect of AZT, and as FHV replicates on the mitochondrial external membrane, we also tested the effect of AZT treatment on other viruses, such as DCV and Sindbis virus, whose replication is not associated with mitochondria. In S2 cells, 5 mM AZT also inhibited the synthesis of a viral DNA form after infection with DCV or Sindbis virus at a dose of 0.5 MOI (Fig. 2f,g). Finally to rule out the possibility of any effects specific to S2 cells, we also analyzed the generation of FHV DNA forms and its inhibition by treatment with AZT in another Drosophila cell line, Kc167. The appearance of FHV DNA after infection of these cells (Fig. 2h) indicated that our results were not unique to S2 cells but were instead a general characteristic of insect cells. Thus, during the establishment of viral persistence, RNA viruses and/or their defective interfering particles were reversetranscribed into viral DNA forms by host reverse transcriptase(s) in cultured Drosophila cells of various origins and AZT inhibited that process.

The viral DNA form mediates persistent infection

To determine whether the absence of a viral DNA form affects the antiviral response during the establishment of persistence, we deep-sequenced vsiRNA from FHV-infected S2n cells treated with AZT or not. At similar amounts of viral RNA (Supplementary Fig. 4d), the accumulation of vsiRNA was strongly impaired in cells treated with AZT and thus in the absence of FHV DNA forms (Fig. 3a and Supplementary Fig. 4e). In contrast, the global amount of miRNA remained unchanged despite AZT treatment (Fig. 3a), which indicated that at the doses and time used, AZT did not have pleiotropic effects. We then characterized the sequence diversity of vsiRNA reads. Cells showing the DNA form have vsiRNAs that mapped to the junctions of the DNA form rearrangements, whereas those vsiRNAs were undetectable in cells treated with AZT (Table 1).

Those results suggested that the FHV DNA form was transcribed and processed into specific vsiRNAs. To determine whether inhibition of the synthesis of FHV DNA and the associated lower amount and diversity of vsiRNAS affected the ability of S2 cells to control FHV replication, we measured viral load after prolonged exposure to AZT. When the DNA form was inhibited, the viral load was up to 1,000-fold higher than that of infected cells in which the DNA form was present (Fig. 3b). That higher viral titer when the appearance of FHV DNA was prevented was accompanied by more cell death (Fig. 3c), which indicated that the DNA form was needed to establish persistence. Together these observations emphasized the requirement for the viral DNA form at early time points during acute infection to improve the antiviral response and to allow the establishment of persistence.

Retrotransposons provide reverse transcriptase activity

Having linked the appearance of FHV DNA forms to the establishment of persistent infection, we next defined the mechanism by which protection was conferred. We hypothesized that determining the structure and the genomic location of FHV DNA would suggest a mode of action. We thus analyzed the genome of S2p cells by deep sequencing. Analysis of chimeric paired-end reads showed that most viral DNA forms (nine of ten) were fused to fragments corresponding to long-terminal repeat (LTR) retrotransposons, mainly 297, blood, diver, micropia and invader2 elements (Fig. 4a and Supplementary Table 1). That result suggests that FHV RNA was reverse-transcribed by the reverse-transcriptase activity of a broad set of retrotransposons actively transcribed in S2 cells (Supplementary Fig. 5a). In some paired-end reads, we were able to identify the exact crossover point between micropia and FHV DNA (Fig. 4b and Supplementary Fig. 5b,c). That junction was one nucleotide distant from the end of the LTR of micropia, which would suggest a possible 'forced copy-choice' recombination mechanism, as has been proposed for the recombination between retrotransposons and lymphocytic choriomeningitis virus in mice 24 . Because of the repetitive and polymorphic nature of the retrotransposon sequences, we were unable to unambiguously assign chromosomal positions to these FHV DNA forms. Alternatively, the DNA-repair machinery can also process nuclear retroviral DNA to produce stable extrachromosomal circular molecules with a single LTR or two LTRs 36 ; thus, we cannot exclude the possibility that the viral DNA form was located on such extrachromosomal molecules. In summary, these results indicated that LTR retrotransposons were the likely source of the reverse-transcriptase activity that produced FHV DNA fragments that were embedded in LTR retrotransposon DNA.

Production of vsiRNAs from newly synthesized viral cDNA

The presence of chimeric DNA molecules consisting of viral cDNA and retrotransposon DNA is not sufficient by itself to explain the mechanism by which persistence is reached. We thus hypothesized that a transcript from the FHV-retrotransposon DNA chimera might produce small RNAs that mediate protection against acute infection through the RNAi machinery, as suggested by the greater number and diversity of vsiRNAs observed (Fig. 3a,b). To assess the involvement of RNAi in this process, we depleted S2p cells of Dicer-2 (a core component of RNAi) 18 or CG4572 (an uptake-spread component of RNAi) 37 by knockdown via RNAi and measured cell death. The equilibrium between viral replication and persistence was broken in cells in which those genes were silenced by RNAi, which shifted the persistent infection to an acute infection that induced cell death (Fig. 4c). To further confirm the involvement of the RNAi response, we generated small RNA libraries from S2p cell lines. As each vsiRNA could originate from either replicating viral dsRNA (profiles, Supplementary Fig. 1d-g) or virus-retrotransposon chimeric transcripts, the only way to discriminate small RNAs specifically from the transcription of the DNA form was to identify those small RNAs whose sequence mapped partly to the virus and partly to the Drosophila genome. We expected these chimeric virus-Drosophila small RNAs to be very infrequent. To improve detection, we treated the samples to -elimination (which prevents ligation on the 3' end of the RNAs unless they bear a 3' modification) to discriminate small RNAs loaded in Ago2 complexes and bearing a 3' 2'-O-methyl from the total small RNA background. The frequency at which such virus-Drosophila small RNA chimeras occurred ranged from 1.15 to 2.3 per 10,000 total unique sequences (Fig. 4d). Indeed, in S2p cells, we identified over 899 chimeric small-RNA reads unambiguously identified and loaded in Ago2 (241 and 427 for S2p 1 cell lines a and b, respectively, and 231 for the S2p 2 cell line) when we aligned small-RNA libraries with the FHV and the Drosophila genome reference sequence. We further confirmed the existence of those chimeric small RNAs by analyzing publicly available small-RNA libraries generated from persistently infected S2 cells in other laboratories (Supplementary Fig. 6). Of note, in the libraries analyzed, all the chimeric reads mapped to retrotransposons on their Drosophila part, and ~65% of their virus-derived sequences matched the positive strand of FHV. Thus, the presence of chimeric small RNAs mapping partly to Drosophila retrotransposons and partly to FHV further confirmed that the RNA was transcribed from FHV DNA templates and was processed by the siRNA machinery into vsiRNA.

Inhibiting virus DNA form increases viral load in vivo

To determine if mechanisms similar to those described above could be involved in viral persistence in vivo, we infected wild-type flies with 20 TCID 50 FHV and monitored the appearance of the FHV DNA form over time by PCR of single flies. We detected fragments of FHV DNA in vivo from day 4 onward (Fig. 5a). The characterization of the FHV RNA1 sequence of these DNA forms identified an almost complete full-length DNA as well as reorganized forms similar to those present in S2p cells (Supplementary Fig. 3c). The appearance of a DNA form in infected flies was a common event, with 84.7% of 200 FHV-infected flies having a DNA form and 58% of 200 Sindbis virus-infected flies having a DNA form at day 6 after injection.

To assess the effect of the DNA form on the antiviral response in vivo, we developed a protocol for natural inoculation with FHV by feeding. We maintained flies in the presence of 25% sucrose and 93 mM AZT from day 2 after eclosion. At day 4 after eclosion, we fed the flies overnight pure FHV stock (1 × 10 9 TCID 50 per ml), then monitored survival every day for 16 d.

After that natural infection protocol, we found that flies infected with FHV but not treated with AZT controlled viral infection (Fig. 5b) and had a death rate undistinguishable from uninfected flies (Fig. 5c). In contrast, when treated with AZT, infected flies were unable to contain virus replication as shown by the high viral titers at day 13 (Fig. 5b), and >75% of the flies died within 13 d of infection (Fig. 5c). In control experiments, uninfected flies treated with AZT had a low death rate over the course of the experiment, which excluded the possibility of considerable pleiotropic effects of AZT alone (Fig. 5c). Of note, we originally developed a double-injection protocol in which flies were injected daily intrathoracically with AZT and challenged them with FHV. This protocol proved to be lethal for the flies above 4 days because of repeated physical injury; however, when analyzed, this injection protocol yielded a similar result: in the absence of a DNA form, infected flies died because of an increase in viral replication (data not shown).

Collectively, these results confirmed that inhibition of FHV DNA synthesis affected the establishment of persistent infection and demonstrated a role for the DNA form in antiviral immunity in vivo. Together our data demonstrated that RNAi and retrotransposons acted together to establish and maintain viral persistent infection in insects; these results provide a mechanistic framework for understanding this process (Fig. 6).

DISCUSSION

The host-pathogen interaction triggers selection pressures on both organisms that drive the development of survival strategies. This survival sometimes indicates the incorporation or endogenization of the full parasitic organism by the host, as noted for the endosymbiont bacterium Wolbachia, which protects fruit flies and mosquitoes against infection with several viruses 38,39 . In other cases, only part of the parasitic genome is endogenized 26 ; for example, bees whose genomes have integrated fragments of Israeli acute paralysis virus are resistant to further challenge with that virus 26 . Our results have demonstrated that one possible root of viral persistence, commonly observed in insects and other arthropods 40 , is the endogenization of viral RNA sequences. Indeed, the establishment of persistent viral infection depends on the formation of viral cDNA fragments from which small RNAs are produced by the RNAi machinery.

We postulate that viral dsRNA, the canonical substrate of the antiviral RNAi machinery, is also generated from viral cDNA. The biogenesis of that dsRNA remains unknown and should be the subject of future research. However, we speculate that dsRNA might originate from a singlestranded viral transcript generated from the DNA form annealed to the viral genome (either the positive or negative strand, depending on the orientation of the transcript). Another possibility is that a single-stranded viral transcript generated from the DNA form folds back on itself and forms double-stranded secondary structures that could be recognized by Dicer and enters into the RNAi pathway, similar to endogenous siRNA. A third possibility is two complementary single-stranded viral transcripts generated from different loci or by convergent transcription.

When the DNA form is inhibited, dsRNA produced through one or several of these mechanisms 3 disappears, with a consequent decrease in vsiRNA.

Given our data, we propose the following model to explain the establishment and maintenance of persistent infection with RNA viruses in insects. After viral infection, ongoing viral replication is limited either by the death of the infected cell or by the antiviral RNAi response in the host. During that process, viral RNA is reverse-transcribed by endogenous reverse-transcriptase activity of LTR retrotransposons. The resulting DNA molecule can then be imported in the nucleus, where retrotransposon-mediated integration into the host genome takes place 34 . Alternatively, the DNA-repair machinery can produce stable extrachromosomal circular DNA molecules that are efficiently transcribed 36 . In all cases, the viral DNA is continuously transcribed and produces dsRNA, which is recognized and processed by the RNAi machinery that boosts the antiviral response. It is possible that such dsRNA molecules are more exposed to Dicer-2 than are viral replication intermediates, and then the resulting small RNAs are loaded into the RNA-induced silencing complex. When a small RNA that is transcribed and processed from a viral DNA form reaches that complex, the ongoing infection can be better contained and controlled, as the immune response is already primed. In this way, both cell and virus have time to reach a metastable equilibrium, or persistent infection.

In the model propose, the interactions between two parasites (transposon and virus) and the RNAi pathways that control them determine the outcome of the infection. In our model, the basal protection afforded by RNAi during viral infection and the priming of the RNAi response in uninfected cells 37 are key to providing the time the cell needs to initiate the persistence mechanism and to control viral infection. In this way, the virus-transposon interaction serves an important role in the modulation of the immune system: the characteristically massive virus production followed by cell death in acute infection is compromised, yet viral dissemination in the persistent state is still ensured. We also speculate that in the absence of the canonical production of secondary small RNAs by RNA-dependent RNA polymerase in insects, this mechanism of transformation of viral RNA into DNA, then into RNA and finally into small RNA could be amplifying and maintaining the antiviral immune response throughout the insect's life after primary exposure. By the mechanism proposed, the RNAi immune response is triggered by viral dsRNA replication intermediates and is amplified and boosted through newly generated viral cDNA-derived dsRNA molecules. As defective interfering particles could be the template for new viral DNA synthesis, a similarity to interferon activation in mammalian cells can be seen. Indeed, viruses such as paramyxoviruses can activate the interferon cascade independently of viral protein synthesis but by a mechanism dependent on defective interfering particles 41 . In this model, the integrity of the defective interfering particle genomic RNA seems to be required for efficient interferon induction. In insects, the considerable sequence similarity among FHV DNA forms and defective interfering particles in different S2p cell lines in vitro, as well as in vivo, suggests a link between defective interfering particles and the biogenesis of viral DNA. Our results are compatible with two possibilities. In one, defective interfering particle RNAs are used as a template by retrotransposon reverse transcriptases to generate viral DNA forms. In the other, viral DNA is the template for the production of defective interfering particles. Further studies addressing this aspect could connect defective interfering particles to persistent infections and explain how these are linked.

Until now, endogenization of DNA has been considered a rare event, as it has been assumed that only endogenization in the germ line has an effect on host evolution. However, somatic (or 'nontransmissible') endogenization may be much more frequent than expected, as the restrictions on genome integrity in the soma could be more relaxed. DNA forms of nonretroviral viruses have been described in a wide variety of eukaryotic organisms, from plants to mammals 24,25,[27][28][29][30] . Whether those DNA forms are also involved in immunity mediated by small RNA or other types of immune responses in other organisms is an open question that deserves further exploration. The model proposed here offers a new perspective on antiviral immunity that considers persistent infection the result of the concerted effort of the host's multiple defense pathways.

METHODS

Methods and any associated references are available in the online version of the paper.

Accession codes. NCBI Small Read Archive: sequences, SRA045427. After viral infection, viral genomes (viral RNA) or dsRNA intermediates (viral dsRNA) are propagated (red). Those viral forms are reverse-transcribed by cellular reverse-transcriptase activity into DNA forms (green) that may integrate into the host genome or be processed into extrachromosomal circular DNA. The sequences of viral origin, now in a DNA form, will produce transcripts (black) that form dsRNA, recognized by Dicer 2 (Dcr2) and further processed by a small RNA-related pathway. When a viral small RNA from those transcripts reaches the RNAinduced silencing complex (RISC), the ongoing infection is contained and the acute infection is controlled. In this way, both cell and virus progress into a metastable equilibrium that defines the state of persistent infection. For infection of flies by viral feeding, the following procedure was used for the AZT in vivo assay: w 1118 flies were fed 93 mM AZT in 25% sucrose daily from day 2 after eclosion or were not fed AZT. At day 4, flies were fed a pure stock of FHV (1 × 10 9 TCID 50 per ml) or Tris-HCl 10 mM, pH 7.5 (as a control), once overnight. Survival was monitored every day for 16 d. Flies were kept at 25 C. At various time points, flies were collected and viral titers were calculated as described above.

Note: Supplementary information is available in the online version of the paper.

For single-fly PCR, each fly was homogenized in 50 l squishing buffer (100 mM Tris-HCl, 25 mM NaCl, 1 mM EDTA, pH 8, and 0.2 mg/ml proteinase K) and incubated for 1 h at 37 C.

Proteinase K was inactivated for 2 min at 95 C. A portion of the homogenate (1 l) was treated for 30 min at 37 C with 10 units of DNase I (Roche) or not, followed by heat inactivation (72 C for 10 min). A portion of the sample (1 l, corresponding to 0.1 l of the original homogenate) was analyzed by PCR.

AZT treatment. S2n or S2p cells (3 × 10 6 ) were incubated for 6-48 h in the presence or absence of 5 mM AZT (Sigma-Aldrich). Cells were then inoculated with a tenfold serial dilution of FHV. At the time of inoculation, AZT was removed or maintained for the rest of the experiment. Viral titers were calculated by endpoint dilution as described above. Small-RNA and genomic libraries. Small-RNA libraries of S2 cells were constructed as described 44 . For -elimination, standard procedures were followed 45 . Small RNAs were treated for 10 min at room temperature with 25 mM sodium periodate, followed by -elimination for 90 min at 45 C in 50 L 1× borax buffer (30 mM borax, 30 mM boric acid and 50 mM NaOH, pH 9.5). Only small RNA with 2'-O-methyl at the extremity (a sign of Ago2 loading) that resisted the oxidation treatment were recovered by sequencing. Genomic DNA libraries of S2 cells were generated with Nextera Technology for Next-Generation Sequencing Library Preparation (Epicentre) with a fragment size centered around 400 base pairs. Libraries were sequenced (36 single reads for small RNAs and 2 × 54 paired-end reads for genomic DNA) on an Illumina Genome Analyzer IIx. Reads were analyzed with in-house Perl scripts.

Bioinformatics analysis of small-RNA libraries. For the detection of chimeric small RNAs, virusderived siRNA profiles were generated according a published method 17 . For the identification of chimeric siRNAs, small RNA reads 36 nucleotides in length were clipped for adapters with the FASTX-Toolkit suite (a collection of command line tools), with reads at least 18 nucleotides in length kept and reads in which the adaptor sequence could not be detected discarded. After removal of contaminating sequences (primers, adapters, ribosomal RNA and so on), reads in which the adaptor as clipped (~25 × 10 6 for S2n and S2p cells, and ~15 × 10 6 for S2p cells after -elimination) were grouped by unique sequence (~3.6 × 10 5 for S2n cells and ~1 × 10 6 for S2p cells) with an in-house script. Each unique sequence was given a unique identifier followed by the number of sequences that it represented in the library.

First, unique sequences were filtered with Blastall software (National Center for Biotechnology Information) with the parameters '-W 9 -F F -e 1-g F', first against the Drosophila melanogaster reference genome (ftp://ftp.flybase.net/releases/FB2010_03/dmel_r5.26) and then against FHV RNA1 and RNA2 (NC_004146.1 and NC_004144.1, respectively) and their defective interfering particles (GU393238.1 for RNA1, and GU393239.1, GU393240.1 and GU393241.1 for RNA2). At each filter, all the unique sequences with a high-scoring segment pair above 18 bases with 0 and 1 mismatches were eliminated. This first step eliminates all nonchimeric small RNAs with a good match in either the host genome or the virus genome.

Second, the Blastall software was used with the parameters '-e 100 -W 9 -F F -q -100 -g F' for selection, among the remaining unique sequences, those with 9-12 bases with the most similarity to FHV RNA1 or RNA2 and their defective interfering particles without gaps or mismatches. Unique sequences without similarity or with similarity of more than 12 bases were not selected. A second blast analysis was done on the selected unique sequences with the parameters '-e 10000 -W 9 -F F -q -100 -g F' for selection of the unique sequences with 9-13 bases of similarity with the Drosophila reference genome. Unique sequences without similarity or with similarity of more than 13 bases were not selected. Then, an 'in-house' script was used for comparison of the two Blastall series (versus FHV and versus Drosophila), with selection of only the unique sequence with at least 1 'hit' against FHV and 1 'hit' against Drosophila and whose positions were considered nonoverlapping. Results that allowed 0, 1 and 2 overlapping bases between the two hits were examined for all the unique sequences. Only unique sequences representing at least five reads on the original small-RNA library were retained. This second step was specifically designed to 'fish out' chimeric transcripts with high confidence.

Of note, to detect chimeric small RNAs that unambiguously map partly to Drosophila and partly to FHV, we applied very stringent mapping and filtering parameters. By doing this, we may have lost many chimeric reads that were unable to pass the filters, and thus the final numbers are low (Fig. 4d and Supplementary Fig. 6).

For comparison of small-RNA libraries with or without AZT treatment, the following procedures were used. For miRNA analysis, the mirBase database was used as reference.

Mapping was done with Bowtie software, with applying a seed of 21 nts with a maximum of 2 mismatches. Each miRNA was quantified in the presence or absence of AZT with SAMtools. FHV small RNAs were mapped with Bowtie software for the alignment of short DNA sequences, and a maximum of two mismatches was allowed. The mapping of siRNAs was annotated for each position of FHV RNA1.

Silencing assay. S2 cells (~1 × 10 6 ) were transfected with dsRNA with Effectene (QIAGEN). The dsRNA was generated by in vitro transcription from T7 promoter-flanked PCR products, with. 2 g dsRNA used per condition in six-well plates with a 2 ml final volume of Schneider's medium.

After 3 d of dsRNA treatment, cells stained with propidium iodide, followed by analysis with a

FACSCalibur and CellQuest Software.

RNA blot analysis. Total RNA was isolated with TRIzol (Invitrogen). RNA (24 g) was separated by electrophoresis through 1.5% denaturing agarose gels, then transferred to a Nytran SuperCharge membrane with the Turbo Blotter system (Whatman). RNA was crosslinked to membranes by ultraviolet irradiation (Stratalinker) and was prehybridized for 2 h at 39 C in ULTRAhyb-oligo buffer (Ambion) . DNA oligonucleotide probes with complementary to FHV RNA1 and RNA3 and to FHV RNA2 were end-labeled with 32 P with T4 polynucleotide kinase (Fermentas), then were added to the hybridization buffer, followed by incubation overnight at 39 C. Membranes were washed several times at 39 C in 0.1× saline-sodium citrate with 0.1%

Figure 1

 1 Figure 1 Characterization of persistently infected Drosophila S2 cells and their viral population. (a) Immunostaining of persistently infected (S2p) and naive (S2n) S2 cells with polyclonal antibody to FHV (red), and with the DNA-intercalating dye DAPI (blue). Scale bars, 25 m. (b) Cell-proliferation index of S2n (white) and S2p (gray) cells, quantified by staining with CFDA (carboxfluorescein diacetate succinymidyl ester) and flow cytometry, presented as the ratio of proliferation at 0 h to that at 48 h. (c) Quantification of death in S2n and S2p cells stained with propidium iodide and assessed by flow cytometry 5 min later. (d) Viability of S2n and S2p cells (three independent groups of 2 x 10 6 cells each) 72 h after treatment with ultraviolet (UV) irradiation (increasing doses, horizontal axis). (e) Survival of wild-type flies (w 1118 ) after intrathoracic injection of FHV produced during acute or persistent infection or after mock

Figure 2

 2 Figure 2 Biogenesis of viral DNA forms during infection. (a) Structure of the main FHV RNA1 DNA form present in one S2p cell line. Numbers indicate breakpoint coordinates. nt, nucleotide. (b) Kinetics of the appearance of FHV DNA forms in S2n cells left uninfected (UI) or infected with FHV (MOI, 0.5), followed by extraction of total DNA and RNA at 0-48 h after infection (above lanes) and treatment of DNA with DNase I (DNase I +) or not (DNase I -), then analysis by PCR and RT-PCR, respectively, with primers designed to amplify an FHV RNA1 sequence. Sequence encoding the ribosomal protein Rp49 serves as a control. (c) Endogenous reversetranscriptase activity in S2n cell extracts, assessed in the presence of Mg 2+ or Mn 2+ and various combinations of oligo(dT) and poly(rA), spotted onto DE81 paper (which retains poly(dT) products but not free [ 32 P]dTTP) and measured at initial velocity phase. SSII, Superscript II used in place of S2n cell extract; Heat, heat-inactivated cells. Below, quantification of the spots above, in arbitrary units (AU). (d) Endogenous reverse transcriptase activity of in S2n cell extracts (3.2 M) and Superscript II (1.8 M) treated with AZT-triphosphate in vitro, presented as the half-maximal inhibitory concentration (IC 50 ) determined by nonlinear regression. (e) Appearance of FHV DNA form in S2n cells infected with FHV (MOI, 0.5) in the presence (1, 5 or 10 nM) or absence (0 nM) of AZT, then treated as in (b) at 24 h after infection. (f-h) Kinetics of the appearance of the DCV (f), Sindbis virus (g) or FHV (h) DNA form in S2 cells (f,g) or Kc167 cells (h) infected with DCV (f), Sindbis virus (g) or FHV (h) and treated as in b. Representative data from eight experiments (b), three experiments (c, e), two experiments (f-h). Data from three experiments (d; mean and s.e.m.).

Figure 3

 3 Figure 3 The FHV DNA form improves the antiviral response via the RNAi machinery. (a) Accumulation of vsiRNA and cellular miRNA in S2 cells infected for 12 h with FHV in the presence (AZT(+)) or absence (AZT(-)) of AZT, assessed as mapping by small RNA corresponding

Figure 4

 4 Figure 4 Viral-retrotransposon DNA chimeras produce transcripts processed by the RNAi machinery. (a) Paired-end genomic DNA deep sequencing of S2p cells, presented as reads of Drosophila DNA-FHV DNA form chimeras (numbers in plots). (b) Paired-end genomic DNA 'read' with a defined crossover point between the micropia LTR retrotransposon DNA and FHV cDNA. Numbers indicate nucleotide coordinates (distance in kilobases (kb) in key); long gray arrows, aligned 'read' pair; gray font, mismatched nucleotides. Gag-Pol, sequence encoding group-specific antigen and polymerase; PBS, primer-binding site; PPT, polypurine tract. (c) Death of S2n and S2p cells treated with scrambled sequence or RNAi directed at the gene encoding Dicer-2 (Dcr-2) or CG4572, assessed by staining with propidium iodide and flow cytometry; results are presented relative to those in cells treated with RNAi directed at the gene Gal80 as a nonspecific control. (d) Frequency of small RNA in S2n cells, S2p cells (two independent lines, S2p 1 and S2p 2) and a technical replicate (S2p 1a and S2p 1b) in the presence (+) or absence (-) of -elimination, assessed by deep sequencing of RNA libraries for the presence of chimeric sequences mapping partly to the virus and partly to Drosophila and presented as chimeras per 1 × 10 6 total unique sequences (only unique sequences representing at least five reads were included in the analysis). Data from one representative experiment (a), or three independent experiments (c).

Figure 5

 5 Figure 5 The FHV DNA form is involved in viral persistence in vivo. (a) Kinetics of the formation of FHV DNA in single wild-type flies injected with Tris-HCl (-; control) or infected with FHV (20 TCID 50 ; +), followed by extraction of DNA and RNA, treatment DNase I and then PCR and RT-PCR (all as in Fig. 2b). (b) Viral titers in flies infected with FHV and treated with AZT (AZT(+)) or not (AZT(-)), then homogenized in PBS at 0, 3 and 13 d after infection, determined by end-point

Figure 6

 6 Figure 6 Model for the establishment and maintenance of viral persistent infection in insects.

Table 1 :

 1 Abundance of vsiRNA covering FHV-FHV junctions in the presence or absence of AZT. produced by the injection procedure and those data were excluded from further analysis. Mortality was monitored daily for 14 d, and every 3-4 d the flies were transferred to fresh food. For in vivo viral DNA detection, 4-day-old female wild-type flies were injected with FHV (20 or 200 TCID 50 per fly). The appearance of FHV DNA was analyzed by single-fly PCR at 6 and 12 h after infection and daily up to 8 d after infection. To determine the number of flies that generate a viral DNA form after infection with RNA virus, we infected flies with FHV or Sindbis virus or mock infected flies with 10 mM Tris-HCl, pH 7.5. The formation of viral DNA was monitored by single-fly PCR at 6 d after infection with the following pairs of primers: 69F and 1002R for FHV; and NSP1F and 913R for Sindbis virus.

S2

  cell extracts and assay of reverse-transcriptase activity. S2 cell pellets were washed once with PBS and lysed in CHAPS lysis buffer (10 mM Tris-HCl, pH 7.5, 400 mM NaCl, 1 mM MgCl 2 , 1 mM EGTA, 0.5% CHAPS and 10% glycerol, supplemented before use with Complete EDTA-free protease inhibitor 'cocktail' (Roche) and 1 mM DTT). After incubation for 10 min at 4 C, cell debris were removed by centrifugation at 16,000g for 10 min at 4 C. Supernatants were transferred to clean tubes. Total protein concentration was determined by Bradford assay (Biorad). Samples were 'snap-frozen' in liquid nitrogen and were stored at -80 C until use.Reverse-transcriptase assays were carried out for 15 min at 25 C in a reaction volume of 50 l containing 4 g S2 cell extracts or 0.05 U Superscript II (invitrogen), 320 ng oligo(dT) (Invitrogen), 500 ng poly(rA) (GE Healthcare) and 1 Ci [-32 P]dTTP (3,000 Ci/mmol; PerkinElmer) in 50 mM Tris-HCl, pH 7.5, 50 mM KCl, 5 mM MgCl 2 or 0.7 mM MnCl 2 , 5 mM DTT and 0.1% Triton X-100. Then, 5 l of each reaction was spotted in triplicate onto DE-81 paper (an ion-exchange paper that retains incorporated nucleotides but not free dNTP). Papers were washed five times with 100 ml of 2× saline-sodium citrate solution, followed by exposure to a PhosphorImager screen. The nucleotide analog and reverse-transcriptase inhibitor azidothymidine-triphosphate was from Bioron. Nonlinear regression and determination of halfmaximal inhibitory concentration were calculated with Prism 5 software (GraphPad).

  SDS and then exposed to a PhosphoImager screen. Probes were stripped by boiling of the membrane twice in 0.1% SDS for a second round hybridization with Rp49 as a 'housekeeping' control.Quantitative real-time RT-PCR. Total RNA was extracted from S2 cells with TRIzol (invitrogen), then 1 g total RNA was treated with DNase I according to the manufacturer's instructions (Roche). The cDNA was prepared by reverse transcription with iScript Reverse Transcriptase (BioRad) with oligodT and random hexamer primers. Roche Universal Sybr Green Master Mix (Rox) and a StepOne Plus (Applied Biosystems) were used for quantitative RT-PCR. The changein-threshold values were calculated within the log-linear phase of the amplification curve with the StepOne Plus V2.2.2 software (Applied Biosystems). Quantification was normalized to that of mRNA encoding the endogenous ribosomal protein Rp49. Oligonucleotide primers were as follows:DNA oligonucleotides (5 to 3)

ACKNOWLEDGMENTS

We thank members of the Saleh lab, the Antoniewski lab and the Vignuzzi lab for discussions and technical support; M. Vignuzzi, J. Chandler and R. van Rij for critical reading of the manuscript; A. Schneemann (The Scripps Research Institute, La Jolla, CA) for antibody to FHV; P. Vargas for assistance with flow cytometry; and A. Pelisson for advice on retrotransposition. Supported by the French Agence Nationale de la Recherche (ANR-09-JCJC-0045-01 to M.-C.S.), the European Research Council (FP7/2007-2013 ERC 242703 to M.-C.S.), Institut National de la Santé et de la Recherche Médicale (for work in the laboratory of G.C.) the European Research Council (ERC 243312, Retrogenomics; for work in the laboratory of G.C.), the French Ministry of Research (C.M.), La Fondation ARC pour la Recherche sur le Cancer (C.M.) and the AXA Research Fund (J.A.M.).

COMPETING FINANCIAL INTERESTS

The authors declare no competing financial interests.

ONLINE METHODS

Cells and cell assays. Drosophila S2 cells (Invitrogen) and Kc167 cells were cultured at 25 C in Schneider's Drosophila medium supplemented with 10% heat-inactivated FCS. For cellproliferation assays, S2 cells were incubated for 15 min at room temperature in the presence of 2 M CFDA (5-(and 6-) carboxyfluorescein diacetate; Invitrogen) and were washed twice with PBS. For cell-viability assays, 1 g/ml of propidium iodide was added to the cells, followed by incubation for 5 min at room temperature. Cellular fluorescence of 5 x 10 4 cells was analyzed at various times after staining with a FACSCalibur and CellQuest software. Alternatively (for Fig. 3d), for quantification of the viability of FHV-infected S2 cells in presence or absence of AZT, a portion of the infected cells was removed and stained for 5 min with 0.2% trypan blue (Sigma-Aldrich). For ultraviolet irradiation-induced DNA damage, cells were exposed to increasing doses of ultraviolet irradiation. At 72 h after irradiation, cell viability was measured by CellTiter-Glo Luminescent Cell Viability Assay (Promega).

Viruses. FHV, DCV and DXV viral stocks were prepared on low-passage S2 cells and titers were measured by end-point dilution. S2 cells (25 × 10 4 cells per well in a 96-well plates) were inoculated with tenfold dilutions of virus stocks. At 14 d after infection, cytopathic effects were analyzed. Viral titers were calculated as TCID 50 (median tissue culture infective dose) according to a published method 42 .

For quantification of viral titers in flies, flies homogenized at various time in 250 L PBS, and titers in the homogenate were calculated as described above.

Sindbis viral stock was prepared in BHK hamster kidney cells and titers were measured by plaque assay.

Fly infection.

For infection of flies by injection, w 1118 flies were used as wild-type controls; these were reared on standard medium at 25 C. Four-day-old female flies were injected intrathoracically with 50 nl of a FHV dilution in 10 mM Tris-HCl (pH 7.5) as described 43 , with a Nanoject II injector. For analysis of survival, FHV was injected at a dose of 500 TCID 50 per fly.

Mock-infected flies were injected with 10 mM Tris-HCl, pH 7.5. Fly mortality at day 1 was


Jo Ão

V F Lima
email: jvlima@inf.ufsm.br

Issam Raïs

Laurent Lef Èvre

Thierry Gautier

Performance and Energy Analysis of OpenMP Runtime Systems with Dense Linear Algebra Algorithms

Keywords: OpenMP, task parallelism, linear-algebra algorithms, NUMA, energy efficiency

In this paper, we analyse performance and energy consumption of five OpenMP runtime systems over a NUMA platform. We also selected three CPU level optimizations, or techniques, to evaluate their impact on the runtime systems: processors features Turbo Boost and C-States, and CPU DVFS through Linux CPUFreq governors. We present an experimental study to characterize OpenMP runtime systems on the three main kernels in dense linear algebra algorithms (Cholesky, LU and QR) in terms of performance and energy consumption. Our experimental results suggest that OpenMP runtime systems can be considered as a new energy leverage, and Turbo Boost, as well as C-States, impacted significantly performance and energy. CPUFreq governors had more impact with Turbo Boost disabled, since both optimizations reduced performance due to CPU thermal limits. A LU factorization with concurrent write extension from libKOMP achieved up to 63% of performance gain and 29% of energy decrease.

Introduction

Energy efficiency is one of the five major challenges that should be overcome in the path to exascale computing [START_REF] Bergman | Exascale computing study: Technology challenges in achieving exascale systems[END_REF]. Despite improvements in energy efficiency, the total energy consumed by supercomputers is still increasing due to the even quicker increase in computational power. High energy consumption is not only a problem of electricity costs, but it also impacts greenhouse emissions 1 Universidade Federal de Santa Maria, Santa Maria, Rio Grande do Sul, Brazil 2 Univ. Lyon, Inria, CNRS, ENS de Lyon, Univ. Claude-Bernard Lyon 1, LIP, France and dissipating the produced heat can be difficult. As the ability to track power consumption becomes more commonplace, with some job schedulers supporting tracking energy use [START_REF] Yang | Integrating dynamic pricing of electricity into energy aware scheduling for hpc systems[END_REF], soon users of HPC systems may have to consider both how many CPU hours they need and how much energy.

Energy budget limitation imposes a high pressure to the HPC community making energy consideration a prominent research field. Most of the gain will come from technology by providing more energy efficient hardware, memory and interconnect. Nevertheless, recent processors integrate more and more leverages to reduce energy consumption (e.g. classical DVFS, deep sleep states) and low level runtime algorithms provide orthogonal leverages (e.g. dynamic concurrency throttling). However few of these leverages are integrated and employed in today local level software stack such as middleware, operating system or runtime library. Due to the complexity of this statement, we restricted our investigation to local node energy consumption by HPC OpenMP applications.

OpenMP is an API standard to express parallel portable programs. Most of controls are implementation defined and rely on the specific OpenMP programming environment used. The OpenMP standard does not impose any constraint on implementations. Even if there are more precise specifications, e.g. mapping of threads to cores, it is very tricky to precisely control performance or energy consumption using what OpenMP specification proposes [START_REF] Bari | Arcs: Adaptive runtime configuration selection for power-constrained openmp applications[END_REF]. Previous works have dealt with a specific OpenMP runtime (Porterfield et al. 2013a;[START_REF] Nandamuri | Power and energy footprint of openmp programs using openmp runtime api[END_REF][START_REF] Su | Model-based, memory-centric performance and power optimization on numa multiprocessors[END_REF][START_REF] Marathe | High Performance Computing: 30th International Conference[END_REF][START_REF] Lively | Energy and performance characteristics of different parallel implementations of scientific applications on multicore systems[END_REF][START_REF] Li | Hybrid mpi/openmp power-aware computing[END_REF]) that may be difficult to generalize to other OpenMP runtime systems without strong development effort. To the knowledge of the authors, there is no related work comparing OpenMP runtime systems in order to analyse performance and energy consumption.

In this paper, we analysed performance and energy consumption of five OpenMP runtime systems over a NUMA system. We also selected three CPU level optimizations, or techniques, to evaluate their impact on the runtime systems: processors features Turbo Boost and C-States, and CPU DVFS through Linux CPUFreq governors. We restrict our experiments on three dense linear algebra algorithms: Cholesky, LU and QR matrix factorizations. Source codes are based on KASTORS [START_REF] Virouleau | Evaluation of OpenMP Dependent Tasks with the KASTORS Benchmark Suite[END_REF]) benchmark suite and the state of the art PLASMA library using its new OpenMP implementations [START_REF] Yarkhan | Porting the plasma numerical library to the openmp standard[END_REF]) that rely on OpenMP tasks with data dependencies.

The paper is an extended version of the paper presented at the 8th Workshop on Applications for Multi-core Architectures (WAMCA 2017). The contributions of this paper are:

• We present early experiments of performance and energy consumption over OpenMP runtime systems; • We report the impact of three CPU level optimizations in order to present the respective gains with different combinations; • We observed that a LU factorization with concurrent-write access mode achieved up to 63% in performance gain and 29% in energy over original LU algorithm; • In addition, our findings suggest that Turbo Boost and C-States had significant impact on performance and energy. CPUFreq governors had more impact with Turbo Boost disabled since Turbo Boost with the performance governor reduced performance due to CPU thermal limits.

The remainder of the paper is organized as follows. Section 2 presents the related work. Section 3 gives some details of the OpenMP task programming model and an overview about five runtime implementations. Section 4 details the experimental hardware and methodology used. Our experimental Prepared using sagej.cls results are presented in Section 5. Finally, Section 6 and Section 7, respectively, present the discussion and conclude the paper.

Related work

Multiple techniques, or leverages, dealing with the energy-performance trade-off are exposed and used in the literature. In [START_REF] Benoit | Shutdown policies with power capping for large scale computing systems[END_REF], authors demonstrate the challenge of using the shutdown and wakeup leverages for large scale HPC infrastructure without altering the throughput of needed computation. In [START_REF] Ribic | Energy-efficient work-stealing language runtimes[END_REF], Dynamic Voltage and Frequency Scaling (DVFS) is used to lower the speed of threads that are not in the critical path with a warranty on performance. In [START_REF] Etinski | Utilization driven power-aware parallel job scheduling[END_REF], authors bind DVFS with EASY backfilling job scheduling to answer possible system load variation in an energy-efficient way. Considering communications between node, the authors of [START_REF] Rountree | Adagio: Making dvs practical for complex hpc applications[END_REF]) reduce the frequency of tasks which would block for MPI communication. In [START_REF] Laros | Energy based performance tuning for large scale high performance computing systems[END_REF] the authors analyses the effects of both DVFS and network bandwidth scaling Other works use the simplicity proposed by OpenMP to vary the number of threads, for energy efficiency. Authors in [START_REF] Curtis-Maury | Online strategies for high-performance power-aware thread execution on emerging multiprocessors[END_REF]) and (Porterfield et al. 2013b) defend the Dynamic Concurrency Throttling (DCT) and underline the fact that using OpenMP to control the number of threads could be energy efficient, depending on the algorithm or the chosen hardware.

Previous works show that various energy behaviors of computing nodes are possible through various leverages (DVFS, DCT, etc). But none of the previous work focus on OpenMP runtime systems as a leverage. None of the previous work dealt with the energy-performance trade-off and thus underlined possible variability concerning energy and performance for existing runtime systems. Thus, to the knowledge of the authors, no related work were trying to compare several OpenMP runtime libraries together for various representative workloads, as presented in our paper.

We use state of the art PLASMA library (YarKhan et al. 2016), on three main kernels in dense linear algebra (Cholesky, LU and QR factorizations), that implements dependent tasks model. This model is new and never addressed in related works. In (Porterfield et al. 2013a;[START_REF] Nandamuri | Power and energy footprint of openmp programs using openmp runtime api[END_REF]) the authors based their experiments using the BOTS [START_REF] Duran | Barcelona openmp tasks suite: A set of benchmarks targeting the exploitation of task parallelism in openmp[END_REF]) benchmarks that require only the independent tasks.

OpenMP Task programming model and implementations

In 2013 the OpenMP Architecture Review Board introduced in the OpenMP revision 4.0 a new way of expressing task parallelism using OpenMP, through the task dependencies. This section introduces the task dependency programming model targeted by the selected benchmark suites. We also present how the model is implemented in various runtime libraries.

Dependent task model

OpenMP dependent task model allows to define dependencies between tasks using declaration of accesses to memory with in,out, or inout. Two tasks are independent (or concurrent) if and only if they do not violated the data dependencies of a reference sequential execution order ⇤ . [START_REF] Yarkhan | Porting the plasma numerical library to the openmp standard[END_REF]. The programmer declares tasks and the accesses in,inout they made to a memory region (here only lvalue or memory reference, i.e. pointer).

The OpenMP library computes tasks and dependencies at runtime, and schedules concurrent tasks on the available processors. The strategy for task dependencies and task scheduling depends on the runtime implementation. Nevertheless, their implementations impact the performance and the energy consumption. Moreover, the absence of precise OpenMP specification about the task scheduling algorithm is the key point to allow research to improve performance and energy efficiency with implementation concerns.

Runtime system implementations

Table 1 summarizes the properties of five OpenMP runtime systems.

Prepared using sagej.cls libGOMP is the OpenMP runtime that comes with the GCC compiler. Dependencies between tasks are computed through a hash table that map data (pointer) to the last task writing the data. Ready tasks are pushed into several scheduling dequeues. The main dequeue stores all the tasks generated by the threads of a parallel region. Tasks seem to be inserted after the position of their parent tasks in order to keep an order close to the sequential execution order. Because threads share the main dequeue, serialization of operations is guaranteed by a pthread mutex which is a bottleneck for scalability. To avoid overhead in task creation, libGOMP implements a task throttling algorithm that serialize task creation when the number of pending tasks is greater than a threshold proportional to the number of threads.

libOMP was initially the proprietary OpenMP runtime of Intel for its C, C++ and Fortran compilers. Now it is also the target runtime for the LLVM/Clang compiler and sources are open to community. libOMP manages dependencies in the same way that libGOMP by using a hash table. Memory allocation during task creation relies on a fast thread memory allocator. libOMP task scheduling is based on Cilk almost non blocking work stealing algorithm [START_REF] Frigo | The implementation of the cilk-5 multithreaded language[END_REF], but dequeue operations are serialized using locks. Nevertheless, it implies distributed deques management with high throughput of dequeue operations. libOMP also implements a task throttling algorithm by using bounded size dequeue.

OmpSs [START_REF] Bueno-Hedo | Productive Programming of GPU Clusters with OmpSs[END_REF]) is a runtime system developed at the Barcelona Supercomputing Center, compatible with the OpenMP specification. It has a specific compiler, called Mercurium, that transforms OpenMP directives to calls to Nanos++ runtime entrypoints. As libGOMP and libOMP, OmpSs computes task dependencies at task creation using hash map. In our experiments we select the breadth-first scheduler. In our experiments we selected the Socket-aware scheduler (socket) which is a work stealing based task scheduler with distributed deques implementations.

XKaapi [START_REF] Gautier | Xkaapi: A runtime system for data-flow task programming on heterogeneous architectures[END_REF]) is a task library for multi-CPU and multi-GPU architectures which provides binary compatible library with libGOMP [START_REF] Broquedis | Libkomp, an efficient openmp runtime system for both fork-join and data flow paradigms[END_REF]. Task scheduling is based on the almost non blocking work stealing algorithm from Cilk [START_REF] Frigo | The implementation of the cilk-5 multithreaded language[END_REF]) with extension to combine steal requests in order to reduce overhead in stealing [START_REF] Tchiboukdjian | Decentralized list scheduling[END_REF]. Moreover, XKaapi computes dependencies on steal request, which is a perfect application of the work first principle to report overhead in task creation to critical path. The XKaapi based OpenMP runtime also has support to some OpenMP extensions such as task affinity [START_REF] Virouleau | Using data dependencies to improve task-based scheduling strategies on numa architectures[END_REF]) that allows to schedule tasks on NUMA architecture, and to increase performance by reducing memory transfer and thus memory energy consumption.

libKOMP [START_REF] Gautier | New libkomp library[END_REF] is a redesign of [START_REF] Broquedis | Libkomp, an efficient openmp runtime system for both fork-join and data flow paradigms[END_REF]) on a top of the Intel runtime libOMP. It includes following features coming mainly from XKaapi: the dequeue management and work stealing with request combining; task affinity specific work stealing heuristic; a dynamically resized hash map that avoid high conflicts when finding dependencies for large tasks' graph; and tracing tool based on the OpenMP OMPT API; and finally a task concurrent write extension with a Clang modification † to provide the OpenMP directive clause. This latter extension allows better parallelism and was used in one of our LU benchmark and it very closed of the task reduction feature currently under discussion in the OpenMP architecture review board.

Discussion

In our study of the mentioned OpenMP runtime systems, none of them include energy leverage such as thread throttling or DVFS. Nevertheless, their different task scheduling algorithms may impact energy efficiency. The main dequeue accesses in libGOMP serialize threads using a POSIX mutex. On Linux the mutex will block waiting threads after short period of active polling which ensure that few core cycles will be waste in the synchronisation.

On the other hand, libOMP, XKaapi and libKOMP work stealing actively poll dequeues until the program ends or a task is found. In order to reduce activity during polling, libOMP and libKOMP may block threads after an unsuccessful search of work by 200ms (default value). Once work is found, all threads are waked up.

Tools and Methods

This Section details the hardware configuration we experimented on and the OpenMP runtime systems we compared. We also give hints about the methodology used to process the collected data using statistical tools R.

Evaluation platform

Our experimental platform was a SGI UV2000 machine composed of eight NUMA nodes with one Intel Xeon E5-4617 (Sandy Bridge) processor each (total 8 processors) and 6 cores per processor (48 cores total) running at 2.9 GHz or 3.2GHz with Turbo Boost, and 512 GB of main memory. The processor has Turbo Boost 2.0 technology, and six idle states of C-States available: POLL C1-SNB C1E-SNB C3-SNB C6-SNB C7-SNB. The operating system was a Debian with Linux kernel 4.9.0-1 with two CPUFreq governors available: powersave and performance. Both DVFS governors have frequency limits from 1.20 GHz to 3.40 GHz.

Software description

4.2.1. Benchmarks We used kernels from two benchmark suites: the KASTORS [START_REF] Virouleau | Evaluation of OpenMP Dependent Tasks with the KASTORS Benchmark Suite[END_REF]) benchmark suite and an OpenMP-parallelized PLASMA version (YarKhan et al. 2016). Both benchmark suites tackle the same computational problems but use different algorithms in some cases. KASTORS was built from PLASMA 2.6.0 (released in dec. 2013) at a time when PLASMA parallelism was supported by a specific task management library called QUARK.

We focused our study on three dense linear algebra kernels:

• A Cholesky factorization (dpotrf); • A LU factorization (dgetrf); • A QR factorization (dgeqrf).
Cholesky factorization algorithms in both the benchmark suites are the same. We compare OpenMP based PLASMA version 82f89ee ‡ . All these linear algebra kernels we used rely on the BLAS routines, we used the implementation of OpenBLAS version 0. In all experiments, we set OMP_WAIT_POLICY environment variable to passive, which means that threads should not consume CPU power while waiting. OmpSs has an equivalent option to Nano++ through --enabled-sleep. XKaapi runtime does not implement a waiting policy.

Energy measurement methodology

Since several metrics have to be considered depending on the objective, we consider performance (GFlop/s) and energy consumption (energy-to-solution). GFlop/s is measured by the each benchmark itself: it corresponds to the algorithmic count of the number of floating point operations over the elapsed time, using fact that matrix-matrix product does not rely on a fast algorithm such as Strassen like algorithm. Times are get using the Linux clock gettime function with CLOCK REALTIME clock.

We employed the Intel RAPL (Running Average Power Limit) feature as source of data acquisition for energy measurement. It exposes the energy consumption of several components on the chip (such as the processor package and the DRAM) through MSRs (Model Specific Registers). Due to access limitation of MSRs on the tested system, we designed a small tool querying periodically the RAPL counters based on LIKWID [START_REF] Treibig | LIKWID: lightweight performance tools[END_REF]): Energy consumption for the whole package (PWR_PKG_ENERGY), for the cores (PWR_PP0_ENERGY), for the DRAM (PWR_DRAM_ENERGY), as well as the core temperature (TEMP_CORE). The tool gets the counter values periodically and associate them with a timestamp.

Experimental methodology

All benchmarks are composed of two steps: the first allocates and initializes a matrix; the second step is the computation. We report execution time only from the computation step. Each experiment was repeated at least 30 times, each computation on a newly random matrix (as implemented by the benchmark). In parallel of the computation, we monitored the system by collecting various energy counters from RAPL. § Git repository: https://gitlab.inria.fr/openmp/libkomp ¶ Available at: http://kaapi.gforge.inria.fr k Available at: https://pm.bsc.es

We selected three CPU level optimizations [START_REF] Orgerie | A survey on techniques for improving the energy efficiency of large-scale distributed systems[END_REF]) in order to evaluate their impact on performance and energy over the runtime systems: processor features Turbo Boost and C-States [START_REF] Rotem | Power-management architecture of the intel microarchitecture code-named sandy bridge[END_REF], and CPU DVFS through Linux CPUFreq governors. Intel Turbo Boost is an overclocking mechanism that allows to the processor to raise core frequencies as long as the thermal and power limits are not exceeded. The C-States feature corresponds to CPU idle states. Deeper C-States offer more power savings, but at the cost of longer latency to enter and exit the C-State. On Linux, CPUFreq allows the control of P-States, which defines the frequencies at which a processor can operate, by governors that choose a frequency for the processor to use. The available governors on our experimental platform were: powersave that chooses the lowest frequency; and performance, which chooses the highest frequency.

For each computation we collected performance (GFlop/s) timestamped by the beginning and the end of the computation. This two timestamps were used in data post-processing to compute energy consumed by the computation between the two timestamps. Values were interpolated by linear function if missing in the collected energy values sampled periodically. Post-processing employs R script to compute energy per computation and to output basic statistic for each configuration. In our experimental results, energy values were the mean computed among the at least 30 computations of each configuration.

Experimental results

The goal of our experiments is to evaluate performance and energy consumption of OpenMP runtime systems and the impact of three CPU level optimizations. Our objectives are:

1. Evaluate the runtime impact on performance and energy, as well as the CPU level techniques (Sec. 5.1); 2. Analyse the correlation coefficient of CPU optimizations over performance and energy (Sec. 5.2); 3. Assess the impact of runtime extensions on performance of LU (Sec. 5.3).

The presented runtime systems have been experimented on the two benchmark suites presented in section 4.2.1. We build two configurations of libKOMP using two sets of options [START_REF] Gautier | New libkomp library[END_REF]. On the following komp refers to libKOMP configured with T.H.E Cilk work stealing queue and requests combining protocol; and komp+cw is the same configuration than libKOMP with addition to support concurrent write extension used in the KASTORS LU code dgetrf [START_REF] Virouleau | Evaluation of OpenMP Dependent Tasks with the KASTORS Benchmark Suite[END_REF].

Runtime impact

Figure 2 shows performance and energy results with a matrix size of 32768 ⇥ 32768 and over all machine resources available. Each configuration color represents the combination of the three CPU level optimizations Turbo Boost (on and off ), CPUFreq governor (performance and powersave), and C-States (all enabled and none). Table 2 summarises the best results over each metric collected, i.e. performance, DRAM energy counter, PKG energy counter, and DRAM+PKG energy counters.

In all cases libkomp attained the best performance results, followed by: xkaapi and gcc for Cholesky; xkaapi and likomp (without CW) for LU; plasma and xkaapi for QR. OmpSs had similar performance to plasma on Cholesky, and performed worst than others on LU. Nonetheless, we were not able to evaluate ompss with QR due to a runtime error. The performance gains of libkomp over GCC were 9.7% for Cholesky, 63.8% for LU, and 9.4% for QR. In addition, the CPU level optimizations for these results were: Turbo Boost enabled; powersave governor; and all C-States disabled. Clearly, the concurrent-write feature of LU contributed to the significant gain of libkomp.

Prepared using sagej.cls

In energy libkomp had better energy results with Cholesky and LU, while QR had lower energy consumption with plasma implementation and GCC. The CPU level optimizations for these results were: Turbo Boost disabled; performance governor; and all C-States enabled for Cholesky and QR, and all disabled for LU.

The cost of energy efficient cases in performance was not significant on the three benchmarks. Comparing the best cases of energy over the best cases in performance, Cholesky had a reduction of 5.12% in performance, while LU and QR had 1.76% and 9.31% reduction respectively. Although, energy reduction was of 8.51%, 10.88% and 11.71% for Cholesky, LU, and QR respectively.

Regarding the CPU level techniques, it seems that Turbo Boost and C-States contributed to the performance gains, and CPUFreq governors had more impact on energy than performance. It was expected that the best performance cases had Turbo enabled and all C-States disabled; still, those cases had powersave as CPU DVFS governor. Besides, energy efficient cases had in most cases Turbo disabled, all C-States enabled, and performance as CPU frequency governor. In order to investigate the performance gains of powersave governor, we analysed the core temperature counter from RAPL over Cholesky with libkomp runtime as illustrated in Figure 3. We collected each socket temperature through a series of Cholesky executions, including an interval between executions of 30 seconds, and computed the mean of all readings per socket. The performance governor had greater temperature readings than powersave on all sockets using Turbo Boost, while both governors had similar temperature readings without Turbo Boost. It seems that tunning all CPU level optimizations to target performance reached the thermal limits of the processor sockets, and degraded performance.

Correlation analysis

We used the Person correlation coefficient to test the correlation between two variables X and Y in order to identify the CPU level techniques and their relation with performance and energy. This coefficient has values between 1 and +1 where +1 means a perfect positive linear correlation, 0 is no linear correlation, and 1 a total negative linear correlation. In addition, we added a significance test of the correlation with confidence interval of 95%.

Figure 4 shows a correlation graph for the best performance cases on the three benchmarks. We employ the following numerical values for each CPU level optimization: Turbo Boost on (+1) and off ( 1); C-States all enabled (+1) and all disabled ( 1); CPUFreq governor performance (+1) and powersave ( 1).

It seems that Turbo Boost parameter had direct impact on performance and energy with an almost perfect correlation on Cholesky and QR. An exception was LU with a lower correlation of 0.31 on performance. In all cases, the negative correlation between Turbo Boost and RAPL DRAM means that it reduced DRAM energy when enabled while increased PKG consumption.

Prepared using sagej.cls The correlation results also show that CPUFreq governors and C-States had no relation with performance and energy. It appears that CPUFreq governor parameter was not relevant in our comparison of performance and energy in Table 2. On the other hand, C-States had a correlation of 0.33 with RAPL DRAM over LU, i.e. increasing DRAM energy consumption with all C-States enabled. It explains the best energy case of LU with C-States disabled in Table 2.

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • Turbo boost

Focus on LU factorization

Figure 5 shows performance results for LU factorization with different matrix sizes. We used as reference the GCC runtime to compute the difference over the other three runtime systems, represented on the bar plots by a percentage value. The LU algorithm with concurrent-write showed significant improvement compared to other runtime systems (up to 165.1% over gcc), followed by PLASMA LU algorithm. Thanks to the concurrent write, the LU algorithm with libkomp cw runtime had more parallelism than other runtime systems due to the CW algorithm extension based on KASTORS [START_REF] Virouleau | Evaluation of OpenMP Dependent Tasks with the KASTORS Benchmark Suite[END_REF]. Figure 6 illustrates a Gantt execution from the LU factorization using PLASMA, libkomp and libkomp with CW.

On the LU factorization, even if CW generates more parallelism, the algorithm has poor efficiency and threads are frequently idle. The Gantt diagram on all the 48 cores illustrates long periods of inactivity. GCC is the only runtime where threads lock common dequeue to get task. Linux will put these lightweight process idle. If we do not consider libkomp+cw's algorithmic variant, then PLASMA algorithm with GCC is the best runtime in terms of energy consumption for LU factorization. This is not true for runtime systems based on task scheduling by work stealing such as libKOMP, libOMP or XKaapi which have very active threads that consume energy.

Discussion

Majority of best configurations from Figure 2 were runtime systems using work stealing based scheduling. On fine grain problems, libkomp and xkaapi were generally better. These results can be explained by the smaller task creation overhead on xkaapi and libkomp than others.

The difference between libomp and libkomp is the new features we add into the original Intel libomp runtime: the lightweight work stealing algorithm from Cilk and the request combining protocol from xkaapi. These features not only impact performance but also the way tasks are scheduled: it suppresses the bounded dequeue limitation that may degenerate task creation into task serialization. It means that at runtime a thread may be forced to execute immediately tasks for which no or less affinity exist. Without bounded size dequeue, a thread that completes a task will always activate one of the successors following a data flow relationship producer-consumer, thus sharing a data resident into cache; or the thread becomes Prepared using sagej.cls idle and try to steal tasks. We will investigate by more finer experiments the exact impact of these additions in libkomp.

On LU factorization where algorithmic variant libkomp+cw was the best, it was followed by xkaapi and libkomp on performance. LU factorization is a relevant code with inactivity sections from the dependencies imposed by the algorithm, mainly due to a search of pivot and swap of elements. This optimized algorithm allowed to increase performance while energy is decreased due to libkomp+cw runtime and concurrent write OpenMP extension [START_REF] Virouleau | Evaluation of OpenMP Dependent Tasks with the KASTORS Benchmark Suite[END_REF]. Nevertheless, the platform characteristic, and especially its memory network, had also an impact on both performance and energy consumption.

Without these algorithm variants, LU factorization code consumes less energy using GCC libGOMP runtime. In GCC the synchronization between threads on the shared task dequeue resource wastes less cycles. A work stealing based runtime may have interest to incorporate part of [START_REF] Ribic | Energy-efficient work-stealing language runtimes[END_REF] in which is used to lower the speed of threads that are not in the critical path with a warranty on performance. One of the big challenges is the design of adaptive OpenMP runtime capable to saving energy on short delays of inactivity.

Our findings on CPU level optimizations lead us to believe that the three CPU level optimizations impacts performance and energy of OpenMP runtime systems. The two processors parameters, Turbo Boost and C-States, had significant influence over experimental results. The former increased performance significantly at energy cost, while the latter reduced energy at idle states. It was not clear the impact of C-States over work stealing based runtime systems since our correlation test (Fig. 4) showed no relation between C-States and the experimental results. Nevertheless, we acknowledge that runtime systems based on work stealing may take advantage of steal phases at the beginning and the end of the computation to enter in idle state and reduce energy consumption [START_REF] Tchiboukdjian | Decentralized list scheduling[END_REF].

CPUFreq governors had more impact at experimental cases with Turbo Boost disabled. Our experimental results suggest that tunning all three CPU level optimizations to target performance, mostly DVFS governor and Turbo Boost, degraded performance due to CPU thermal limitations. Still, the powersave governor was able to sustain performance while consuming less energy. Other works show experimental results on techniques using DVFS, as discussed in Section 2.

Conclusion

In this paper, experiments with five production based OpenMP runtime systems and three CPU level optimizations (Turbo Boost, C-States, and Linux CPUFreq governors) on the three main kernel in dense linear algebra were conducted on a NUMA platform. We showed that OpenMP runtime is a new leverage for controlling energy, and Turbo Boost, as well as C-States, impacted significantly performance and energy. Our experimental results suggest that small algorithmic and runtime improvements may allow performance gains up to 63% and thus reducing the energy by 29%. Besides, work stealing based runtime systems were more efficient in our experiments; although, it was not clear the impact of C-States in order to reduce energy consumption at steal phases.

Future works include an extension of our experimental comparison over a wide range of architectures, including Intel KNL many-core. In addition, we will evaluate the impact on performance and energy of idle states at steal phases of work stealing scheduler.

Figure 1 .

 1 Figure 1. LU factorization with OpenMP dependent task.

Figure 3 .

 3 Figure 3. Processor temperature of each socket with Cholesky and likbomp runtime. We compared the impact of CPUFreq governor and Turbo Boost. C-States were disabled.

Figure 4 .

 4 Figure 4. Correlation coefficients on the three benchmarks based on the best performance results. The ⇥ symbol mean that the significance level (or p-value) of the correlation is under 5%.

Figure 6 .

 6 Figure 6. Gantt of LU algorithm from PLASMA (top), KASTORS and libKOMP (middle), and KASTORS with libKOMP and concurrent write (bottom). The matrix size was 32768x32768 with 352x352 of block size.

Table 1 .

 1 Characteristics of OpenMP runtime systems.

	Name	Dependencies Task Scheduling	Remarks
	libGOMP	hash table	centralized list	task throttling
	libOMP	hash table	work stealing	bounded dequeue
	OmpSs	hash table	socket-aware work stealing
	XKaapi	hash table*	non blocking work stealing	task affinity
	libKOMP	resizable hash table	non blocking work stealing	task affinity concurrent write
	Figure 1 illustrates a LU factorization based on PLASMA

  Prepared using sagej.cls 4.2.2. Runtime Systems We compared the following runtime systems during our experiments:• LibGOMP -the OpenMP implementation from GNU that comes with GCC 6.3.0. • LibOMP -a port of the Intel OpenMP open-source runtime to LLVM release 4.0. • LibKOMP (Gautier and Virouleau 2015) -a research runtime system, based on the Intel OpenMP runtime, developed at INRIA. It offers several non-standard extensions to OpenMP. We evaluate the concurrent write (CW) feature in our experiments coupled with Cilk T.H.E work stealing protocol. We make experiments with version 54f7a28 § . • XKaapi (Gautier et al. 2013) -research runtime system developed at INRIA. It has lightweight task creation overhead, and it offers several non-standard extensions to OpenMP (Broquedis et al. 2012) We evaluate its version efa5fdf ¶ . • OmpSs (Bueno-Hedo et al. 2012) -a runtime system developed at the Barcelona Supercomputing Center. The reported results are based on the 17.12 version k .

2.19. ‡ Mercurial hash from https://bitbucket.org/icl/plasma

Table 2 .

 2 Overview of the best results. Higher is better for performance, and lower is better in energy.

	Method	Runtime	Turbo Boost	C-States	Governor GFlop/s	RAPL PKG	RAPL DRAM
	PERFORMANCE
	Cholesky	komp enabled	none	powersave	732.45	13.02	2.13
	LU	komp+cw enabled	none	powersave	383.40	42.83	7.98
	QR	komp enabled	none	powersave	433.77	83.36	15.31
	RAPL PKG+DRAM
	Cholesky	komp disabled	all performance	694.95	11.62	2.24
	LU	komp+cw disabled	none performance	376.64	37.12	8.16
	QR	plasma disabled	all performance	393.40	71.42	15.70
	RAPL PKG
	Cholesky	komp disabled	all performance	694.95	11.62	2.24
	LU	gcc disabled	all	powersave	225.41	36.60	12.53
	QR	plasma disabled	all performance	393.40	71.42	15.70
	RAPL DRAM
	Cholesky	xkaapi enabled	none	powersave	723.23	13.22	2.11
	LU	komp+cw enabled	none	powersave	383.40	42.83	7.98
	QR	plasma enabled	none	powersave	404.92	81.07	15.19

  Performance (GFLop/s) results of LU. All percentages on top of bar plots are the difference of current runtime over GCC libGOMP runtime.

													LU
		400																				63.8%
	GFlop/s	100 300 200	-33.1%	0.0%	2.3%	71.7%	7.7%	11.7%	165.1%	-39.5%	0.0%	5.3%	58.7%	11.7%	14.2%	113.3%	-46.2%	0.0%	8.6%	10.4%	15.8%	20.3%	Runtime ompss gcc omp plasma komp xkaapi komp+cw
		0
					8192						16384					32768
											Matrix size
	Figure 5.

† http://gitlab.inria.fr/openmp/clang Prepared using sagej.cls

Prepared using sagej.cls

Acknowledgements

This work is integrated and supported by the ELCI project, a French FSN ("Fond pour la Société Numérique") project that associates academic and industrial partners to design and provide a software environment for very high performance computing.

Author Biographies

João Vicente Ferreira Lima received a joint PhD degree in computer science by the Federal University of Rio Grande do Sul (UFRGS), Brazil, and the MSTII Doctoral School at the Grenoble University, France. He received a BSc degree in Computer Science in 2007 by the Federal University of Santa Maria (UFSM), Brazil, and a MSc degree in Computer Science in 2009 by he Federal University of Rio Grande do Sul (UFRGS), Brazil. He is associate professor at the Federal University of Santa Maria (UFSM), Brazil, since 2014. His research interests are high performance computing, runtime systems for HPC, parallel programming for accelerators, and distributed computing.