Marie Chancel

Clémentine Brun

Anne Kavounoudias

Michel Guerraz
email: michel.guerraz@univ-smb.fr

The kinaesthetic mirror illusion: how much does the mirror matter?

Keywords: kinaesthesia, mirror illusion, muscle proprioception, visual impoverishment

The reflection of a moving hand in a mirror positioned in the sagittal plane can create an illusion of symmetrical, bimanual movement. This illusion is implicitly presumed to be of visual origin. However, muscle proprioceptive afferents of the armreflected in the mirror might also affect the perceived position and movement of the other arm. We characterized the relative contributions of visual and proprioceptive cues by performing two experiments. In Experiment 1, we sought to establish whether kinaesthetic illusions induced using the mirror paradigm would survive marked visual impoverishment(obtained by covering between 0% and 100% of the mirror in 16% steps). We found that the mirror illusion was only significantly influenced when the visual degradation was 84% or more. In Experiment 2, we masked the muscle proprioceptive afferentsof the arm reflected in the mirror by co-vibrating antagonistic muscles. We found that masking the proprioceptive afferents reduced the velocity of the illusory displacement of the other arm. These results confirm that the mirror illusion is not a purely visual illusion but emerges from a combination of congruent signals from the two arms, i.e. visual afferents from the virtually moving arm and proprioceptive afferents from the contralateral, moving arm.

INTRODUCTION

Reflection of a moving hand in a mirror positioned in the sagittal plane (i.e. the plane that separates the left and right sides of the body) can give the illusion of symmetrical bimanual movements. The mirror paradigm was initially developed to treat phantom limb pain in unilateral amputees [START_REF] Ramachandran | Touching the phantom limb[END_REF] but hasalso been used over the last two decadesas a rehabilitation tool for promoting recovery from hemiparesis (RamachandranandAlthschuler2009; RosenandLundborg2005; Dohle etal.2009;[START_REF] Guerraz | Mirror paradigm and mirror therapy: what about the benefit of the virtual hand on motor behavior?[END_REF]. More recently, experiments conducted in healthy participants showed that mirror reflection of an arm moved passively by a motorized manipulandum induces consistent, vivid kinaesthetic illusions of movement of the hidden, static arm in the direction of the mirror displacement [START_REF] Guerraz | Integration of visual and proprioceptive afferents in kinesthesia[END_REF][START_REF] Tsuge | Interaction between vibrationevoked proprioceptive illusions and mirror-evoked visual illusions in an arm-matching task[END_REF][START_REF] Metral | Kinaesthetic mirror illusion and spatial congruence[END_REF]. The occurrence of this visually induced kinaesthetic illusion indicates that visual afferents might be of prime importance in sensing limb movement (i.e. kinaesthesia). However, kinaesthesia is not exclusively derived from visual afferents; muscle spindle afferents (notably type Ia and II sensory endings; [START_REF] Goodwin | the contribution of muscle afferents to kinesthesia shown by vibration induced illusionsof movement and by the effects of paralysing joint afferents[END_REF][START_REF] Teasdale | The role of proprioceptive information for the production of isometric forces and for handwriting tasks[END_REF] for a review, see [START_REF] Proske | The proprioceptive senses: their roles in signaling body shape, body position and movement, and muscle force[END_REF] and cutaneous afferents [START_REF] Collins | Movement illusions evoked by ensemble cutaneous input from the dorsum of the human hand[END_REF][START_REF] Blanchard | Combined contribution of tactile and proprioceptive feedback to hand movement perception[END_REF][START_REF] Blanchard | Differential contributions of vision, touch and muscle proprioception to the coding of hand movements[END_REF]) also make significant contributions. For instance, it has been shown that the mirror illusion is less intense when the unseen arm is not in the same position as the reflected arm [START_REF] Metral | Kinaesthetic mirror illusion and spatial congruence[END_REF]. Likewise, masking the proprioceptive afferents of the unseen armincreases the illusory velocity of displacement [START_REF] Guerraz | Integration of visual and proprioceptive afferents in kinesthesia[END_REF].

Although the muscle proprioceptive afferents of the arm subjected to the kinaesthetic illusion has received much attention, this is not the case for the proprioceptive afferents originating from the other (moved) arm. However, it has recently been reported that manipulating the muscle proprioceptive afferents of one arm affectsnot only the motor behaviour of the other arm [START_REF] Ridderikhoff | Bilateral phase entrainment by movement-elicited afference contributes equally to the stability of in-phase and antiphase coordination[END_REF],Brun et al. 2015, Brun and Guerraz 2015) but also the latter'sperceived position and perceived movement [START_REF] Izumizaki | The illusion of changed position and movement from vibrating one arm is altered by vision or movement of the other arm[END_REF][START_REF] Hakuta | Proprioceptive illusions created by vibration of one arm are altered by vibrating the other arm[END_REF][START_REF] Kuehn | On the bimanual integration of proprioceptive information[END_REF]. For instance, Izumaki and colleagues (2010) showed that stimulating the muscle proprioceptive afferents of one arm (by the application of tendon vibration to either the flexor or extensor muscles) modified the perceived position of the other arm. The "bimanual integration of proprioceptive afferents" [START_REF] Kuehn | On the bimanual integration of proprioceptive information[END_REF] might well contribute to the occurrence and intensity of the kinaesthetic illusions reported in the mirror paradigm.

Hence, the kinaesthetic illusions evoked in the mirror paradigm might well be of both visual and proprioceptive origin.

To determinethe relative contributions of visual and muscle cues, we performed two experiments. Experiment 1 consisted in testing whether the kinaesthetic illusions induced by the mirror paradigm would survive marked visual impoverishmentobtained by covering between 0%and 100% of the mirror in 16% steps.Conversely, Experiment 2 was designed to estimate the relative contribution of visual cues by masking muscle proprioceptive afferents (through co-vibration of antagonistic muscles) of the arm reflected in the mirror. Indeed, we know that when vibration is applied concurrently on a muscle that is passively lengthened or shortened, it degrades afferent proprioceptive responsiveness since the primary ending activity is then predominantly driven at the vibration frequency and any frequency modulation related to the imposed movement disappears [START_REF] Roll | Alteration of proprioceptive messages induced by tendon vibration in man: a microneurographic study[END_REF]). The masking effect of the vibration seems also responsible forthe impairmentin position and force perception observed during a full whole-body exposition to vibration [START_REF] Ribot | Comparative effects of whole-body vibration on sensorimotor performance achieved with a mini-stick and a macro-stick in force and position control modes[END_REF]). Finally, co-vibrating two antagonist muscles at the same frequency does not elicit any movement perception [START_REF] Gilhodes | Perceptual and Motor Effects of Agonist-Antagonist Muscle Vibration in Man[END_REF]) and alters sensorimotor tasks such as matching position task or haptic shape perception task [START_REF] Bock | A method to reversibly degrade proprioceptive feedback in research on human motor control[END_REF]. We therefore hypothesised that if muscle proprioceptive inputs from the moving arm influence the contralateral kinaesthetic illusion evoked in the mirror paradigm, the latter illusion would be less intense when the proprioceptive afferents of the reflected arm are mostly masked by co-vibration.

METHOD Participants

Nineteen healthy adult participants (14 females and 5 males; 16 right-handed; mean ± SD age: 21.7 ± 1.6 years) took part in Experiment 1 and eighteen healthy adult participants (16 females and 2 males; 15 right-handed; mean ± SD age: 20.9 ± 5.9 years) took part in Experiment 2. Three of the 19 participants in Experiment 1 and 3 of the 18 participants in Experiment 2 failed to experience a mirror illusion during the experiment's familiarization phase and were therefore excluded from the studies. None of the participants had a history of visual, proprioceptive or neuromuscular disease. All the participants provided their written, informed consent prior to initiation of the experiment.The study was performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and was approved by the local independent ethics committee (University Savoie Mont Blanc, Chambery, France; reference: UDS 2013025).

Material

Participants sat in front of a large, custom-built box. A mirror measuring 65 x 65 cm was positioned vertically in the middle of the box, with the reflective surface facing the participant's left arm and oriented parallel to his/her midsagittal axis. The participant'sforearms were positioned on each side of the mirror and were supported by two manipulanda. The distances between the manipulanda and the mirror were adjusted so that the mirror image of the left arm coincided with the position of the right arm. Each manipulandum consisted of a wooden arm (on which the participant positioned his/her forearm) and a hand grip at the end of the wooden arm. The right manipulandum did not move, while the left manipulandum was fitted with a low-noise DC synchronous motor (220 v, Crouzet ™ France)and could flex or extend (via a remote control) the participant's left forearmfrom the initial starting position (Fig. 1).The manipulandum's angularspeed was always 3.8°/s. The participant's forearm was adjusted on the manipulandum so that the motorized device's axis of rotation coincided exactly with the elbow joint.

Participants were told to lift their right foot to indicate the onset of illusory movement. To this end, the participant's right foot was taped to a foot pedal, the rotational axis of which was close to the heel. The displacements of the left manipulandum and the foot pedal were recorded with an electromagnetic motion capture system (Fastrak ™ , Polhemus, Colchester, VT, USA). A sensor was positioned on each device so as to continuously record the manipulandum and foot angles (sampling frequency: 60 Hz).

Procedure

Experiment 1

Throughout the experiment, the participants were required to look at a fixation cross in the centre of the mirror. The right arm was always hidden. Before each trial, the two arms were positioned at an angle of 30 degrees to the horizontal.Following a baseline, movement-free epoch of ~5seconds, the left forearm was passively flexedat a constant angular speed of 3.8°/s (total duration of movement: ~8 s). Participants were told not to resist this passive displacement.

Visual impoverishment was performed by affixing different plastic sheets to the mirror (Fig. 1). The sheetscontained variable densities of randomly positioned black pixels (1.85 x1.85mm). Sheets were generated with a VBA script in PowerPoint software (Microsoft Visual Basic 7.1®). Overall, the black pixels covered 0%, 16%, 33%, 50%, 66%, 84% or 100% of the mirror's surface area. Each of these seven visual conditions was repeated four times in pseudorandom order, giving a total of 28 trials per participant. Four sham trials were performed with the eyes closed. Participants performed active, synchronous, flexion-extension movements of both arms before each trial. This allowed the two arms to have similar immediate history of contraction and length changes before trials [START_REF] Gregory | Aftereffects in the responses of cat muscle spindles and errors of limb position sense in man[END_REF][START_REF] Proske | Thixotropy in skeletal muscle and in muscle spindles: a review[END_REF]. Following these movements, the two arms were re-positioned at an angle of 30 degrees to the horizontal before the next trial.

Insert figure 1 around here Experiment 2

In Experiment 2, participants were required to look at a fixation cross in the centre of the mirror. In contrast to Experiment 1, the mirror was not obscured by black pixels (i.e. it corresponded to the 0% impoverishment condition from Experiment 1).The right arm was always hidden behind the mirror. Before each trial, the two arms were positioned at an angle of 45degrees to the horizontal. Following a baseline, movement-free epoch of ~5 seconds, the left forearm was passively flexed or extended at a constant angular speed of 3.8°/s (total duration of movement:~8s). An electromechanical vibratory apparatus (Innovative Technology, France) was attached to the left biceps and triceps with elastic bands.

Microneurographic studies (the recording of afferent units via a microelectrode inserted in a human superficial nerve) have shown that low-amplitude vibration preferentially activates muscle spindle endings; this masks the spontaneous discharges recorded in the absence of vibratory stimulation [START_REF] Roll | Alteration of proprioceptive messages induced by tendon vibration in man: a microneurographic study[END_REF].In half of the trials, the muscle afferents of the left arm were masked by switching on the vibrators (frequency: 40Hz) immediately before the arm was passively moved. When the trial had finished, the vibrators were switched off. A frequency of 40Hz was chosen because it is an optimal stimulationto completely mask natural discharges of muscle spindle endings that would otherwise be observed in responseto the passively imposed movement of the arm, as evidenced by microneurographic and psychophysical studies [START_REF] Roll | Alteration of proprioceptive messages induced by tendon vibration in man: a microneurographic study[END_REF][START_REF] Cordo | Proprioceptive consequences of tendon vibration during movement[END_REF].This level of vibration frequency was also low enough to be bearable for the duration of the experiment. Some sham trials (n= 8) without a mirror were included; only the left arm was visible and the mirror was covered by an opaque board (with a fixation cross at its centre). The two movement conditions (flexion vs. extension) were paired with two masking conditions (masking vs. non masking), giving a total of four experimental conditions in a within-subjects design. Each condition was repeated four times in pseudo-random order, giving a total of 16 true trials (and 8 sham trials) per participant. No vibration was applied during sham trials. Participants were asked to move both arms freely and synchronously for a few seconds at the end of each trial.

Quantification of the kinaesthetic illusion

Subjective reporting: At the end of each trial, participants were required to verbally rate the speed of the illusory displacement of the right arm on an integer scale from 0 to 20: a rating of 0 corresponded to the absence of illusory displacement,10 corresponded to the same speed of displacement as for the passively moved left forearm, and 20 corresponded to twice the speed of the passively moved left forearm. In order to familiarize themselves with this subjective rating of displacement speed, the subjects rated several trials with passive displacement of the left forearm prior to the experimental session itself. Velocity of passive displacement was always set at 3.8 °/s. "Illusion onset": During each trial, the participant was told to use his/her right foot to indicate when he/she felt the sensation of right arm movement. To do so, the participant was required to lift his/her right foot (Fig. 2). The onset was defined as the time point at which the angular position of the foot was more than two standard deviations from the mean baseline position (calculated over a 1-second epoch prior to movement of the manipulandum).The onset includes both the delay for the illusion to occur and the time for initiating foot movement (motor response time). Given the unusual nature of the foot-matching task, participants performed familiarization trials prior to the experimental session. Despite this familiarization phase, a few participants almost always forgot to report the illusion onset. Hence, only subjective ratings could be analysed for these participants.

Insert figure 2 around here

Statistics

In Experiment 1, a chi-squared test was used to compare the various experimental conditionsin terms of the frequency of illusion occurrence. The onset latency and subjective rating were analysedin a one-way, repeated-measures analysis of variance (ANOVA) with seven modalities (0%, 16%, 33%, 50%, 66%, 84% and 100%)in Experiment 1 and a twoway,repeated-measures ANOVAwith movement (flexion vs. extension) and proprioceptive masking (masking vs. no masking) as independent variablesin Experiment 2. For these ANOVA, we considered the mean latency and mean rating from the trials in each experimental condition and for each participant. The reported values were Huynh-Feldtcorrected. A Holm post hoccorrection was applied for multiple comparisons.The threshold for statistical significance was set to p<.05.

RESULTS

EXPERIMENT 1 Occurrence of the mirror illusion

Reflection of the passively moving left arm in the mirror evoked in most individuals a kinaesthetic illusion of right arm displacement in the same direction, i.e. a mirror illusion.

Among 19 individuals who consented to participate, 16 experienced such an illusion in our pre-test phase. Since the purpose of the Experiment was to study this illusion phenomenon, only those latter 16 individuals could take part to Experiment 1. Results showed that even in those 16 participants, illusion did not occur in all trialswhen the mirror was not obscured at all (visual impoverishment: 0%) but did still occur in 90.6 percent of them (Table 1).Interestingly, the illusion occurrence rateremained high (>85%) in those participants in the other visual impoverishment conditions -even in the 84% condition. The illusion occurrence rate only dropped when the mirror was completely obscured byblack pixels. Even then (i.e. when no mirror feedback was available), an illusion was reported in 40.6% of the trials. This value is similar to the rate of 36% observed in the sham "eyes closed" trials. A statistical analysis indicated that only the illusion occurrence rate in the 100% condition and "eyes closed" conditions differed significantly from that observed in the 0% condition (p<.05).

Table 1: Occurrence and subjective rating for each "visual impoverishment condition" (0%, 16%, 33%, 50%, 66%, 84% and 100% eyes closed -sham-). Mean subjective ratings represent the mean values of the 16 participants involved in Experiment1. We performed post hoc multiple comparisons (with Holm correction) of subjective ratings. The threshold for statistical significance was set to p<.05.

The speed of illusory movement (subjectiverating)

In the 0% visual impoverishment condition, the mean ± SD subjective rating of illusory movement was 7.2 ± 2.8. Hence, the hidden right arm was perceived to be movingabout 30% slower than the left arm (and its mirror reflection). As can be seen in Table 1 andFigure3A, the mean rating remained fairly constant up to a visual impoverishment condition of 50%, decreased slightly in the 66% and 84%conditions and dropped to a value of 0.8 ± 1.05 for the 100% condition. An ANOVA confirmed thatvisual impoverishment had a significant effect on the subjective rating (F(6, 90)=55.6.p<.001, η 2 p =.77). A post hoc pairwise analysis indicatedthat the first four visual conditions (0% to 50%) did not differ from each other but differed significantly almost systematically from the other three visual conditions (66% to 100%)(see table 1). The 66% and 84%conditions did not differ from each other but both differed significantly from the 100%condition.

Insert figure 3 around here

Since the occurrence of illusion differed from one condition to another, we performed a supplementary analysis to ensure that the reduction in the perceived speed of the mirror illusion was not solely due to mixing trials in which illusion occurred and trials in which illusion did not occur (with a subjective rating of 0 in the latter cases, the number of which increased with the degree of visual degradation). We thus limited our analysis to trials in which a mirror illusion had occurred. Given our within-subjects design, participants who did not experience any illusory displacement in one of the seven visual conditions could not be included in the analysis. This additional ANOVA (involving9 of the 16 participants)confirmed the effect of visual impoverishmenton the speed of illusory movement(F(6, 48)=41.3p<.001 η 2 p =.83) (Table 2 and Fig. 4A).This analysis based only on trials in which an illusion occurred confirmed that the velocity of the mirror illusion decreases with visual impoverishment.

Table 2: Subjective rating and onset for each "visual impoverishment condition" (0%, 16%, 33%, 50%, 66%, 84% and 100%) for the 9 participants experiencing illusions in each of the seven visual conditions. For mean onset, only trials in which an illusion occurred were considered (since no onset could be estimated in the absence of illusion).

The latency of illusory movement onset

Only trials in which an illusion occurred could be considered in the latency analysis, and so only the 9 (out of 16) participants who experienced illusory displacement in all seven visual conditions were included in the statistical analysis. The ANOVA revealed a significant effect of visual impoverishment on the latency of illusory movement onset (F(6, 48)=23.4, p< .001, η 2 p = .74). Post-hoc analysis revealed that the latency of illusory movement onset was significantly longer in the 100% visual impoverishment condition (4.2s ± 1.4) than in the other six conditions (see Table 2). However, no difference occurred between the other six visual conditions, from 0% to 84% conditions(see Fig. 4B).

Insert figure 4 around here EXPERIMENT 2 Occurrence of the mirror illusion

Among 18 individuals who consented to participate, 15 experienced mirror illusion in the pretest phase and were therefore included in Experiment 2. In those 15 participants, reflection of the passively moving left arm in the mirror evoked a kinaesthetic illusion in the right arm in 92% of the trials under both flexion and extension conditions (though in the opposite direction) when no masking was applied. When proprioceptive masking was appliedto the moving arm, the illusion occurrence rate fell slightly (to 83% and 90% in the extension and flexion conditions, respectively). Statistical analysis indicated that the occurrence rate did not differ when comparing the four experimental conditions (p>.05). Few illusory displacements of the unseen right arm occurred in sham trials that lacked reflection of the left arm (i.e. no mirror and no masking), with rates of 18% and 23%for trials with the extension and flexion movements, respectively.

The speed of illusory movement (subjective rating)

As can be seen in Figure 5A, the mean ± SD subjective rating of illusory movement was lower when proprioceptive masking was applied to the passively moved left arm (5.52 ± 3.5) than in the absence of masking (7.21 ±2.7)(F(1,14) =4.3 p=.05 η 2 p =.23). In contrast, there was no main effect of the direction of the movement (F(1,14) < 1 p> .05 η 2 p =.0006). Indeed, when the data from the two masking conditions were pooled, the mean ± SD subjective rating of illusory movement in the extension condition (6.4 ± 3.3) was similar to that reported in the flexion condition (6.3 ± 3.5). The ANOVA showed no significant main effect of the movement (direction) factor (F(1,14) < 1p = .92 η 2 p =.00012) nor any interaction with the masking factor(F(1,14) < 1 p= .84 η 2 p =.0002).

Insert figure 5 around here

The latency of illusory movement onset

Only trials in which an illusion occurred could be considered in the latency analysis, and so, only the 10 (out of 15) participants who either experienced illusory displacement in the four experimental conditions or did not forget to indicate theonset of the illusion were included in the statistical analysis. The latency of illusion onset was longer in the masking condition (2.86 ± 1.35 s) than in the absence of masking (2.43 ± 1.21 s), though this effect was not statistically significant (F(1,9)=2.9 p=.1 η 2 p = .24). An ANOVA revealed that movement did not have a significant main effect (F(1,9)<1 p=.41 η 2 p = .01) or a significant interaction with masking (F(1,9) <1 p=.78 η 2 p = .001). The latency of illusion onset was therefore similar in the extension and flexion movement conditions.

DISCUSSION

Limited vision of the arm provides sufficient kinaesthetic cues in the mirror illusion

Reflection of the passively moving left arm in the mirror evoked in most individuals a kinaesthetic illusion of right arm displacement in the same direction. The results of Experiment 1 revealed that the occurrence, latency and speed of the mirror illusion were notgreatly influenced by visual impoverishment (except under extreme conditions). In fact, the occurrence and latency of the illusion were similar in 0% to the 84% conditions, whereas the speed of mirror illusion was only slightly lower under the most extreme visual impoverishment conditions (66% and 84%). Giving only 16% vision of the mirror (84% impoverishment) increased the illusion occurrence from 40% (no mirror feedback) to ~90%, rating speed from less than 1 °/s to 4.5°/s, and reduced onset from 4.2 to 2.5s. All these effects are large, clearly significant, and bigger than any other of the visual steps.These results attest of the great role of visual cues in the occurrence of mirror illusion in particular and in kinesthesia in general. They also indicate that even a limited amount of visual information is enough to provide cues for kinaesthetic purposes. These results are in line with a large body of literature data showing that a limited amount of visual information can provide a high amount of information on biological motion [START_REF] Johansson | Visual-Perception of Biological Motion and a Model for Its Analysis[END_REF]. For instance, the kinematics of point-light animations (moving dots that reflect the motion of some key points ona moving body)can reveal many details about the action itself (such as the weight of a box being lifted (Runeson andFrykholm 1981) or how far an object will bethrown (Munzert et al.2010;for a review, see [START_REF] Troje | What is biological motion?: Definition, stimuli and paradigms[END_REF]. What holds true for motion recognition (i.e. the recognition of external actions) might well hold true for the perception of self-motion in general and kinaesthesia in particular. Our results suggest that healthy individuals can easily make out their body segments -even when the visual stimulus is markedly degraded (i.e. a low proportion of "mirror pixels"reflecting parts of the left arm) -and integrate these visual cues to yield a unified perception of arm movement. The rules that govern the use of visual cues for the perception of self-motion might therefore be similar to those governing the perception of the motion of external objects.

Our results attest that a limited amount of visual information is enough to provide many cues for kinaesthetic purposes. In the same vein, it has been shown that a limited morphological matching between the reflected arm and the hidden one is although sufficient as long as position sense is concerned. Holmes et al, 2004 reported that a discrepancy between the visual-mirror feedback about the right arm position and its actual position has a profound effect on reaching performances with the hidden hand. Interestingly, the visual capture of arm position still happens when the reflection of the left arm is replaced by a rubber hand and even block of wood reflection (Holmes et al, 2006). Overall, these data, including those of Experiment 1, confirm that the amount of visual information sufficient to bias the kinesthetic and position perception of the hidden arm is quite low both in terms of quality (morphological visual matching is not required) and quantity (only 16% of the mirror surface is needed for the illusion to arise).

Kinaesthetic sensing of one arm is influenced by somaesthetic stimulation of the other arm.

In Experiment 1, some participants reported sensations of right arm displacement when the mirror was completely covered with black pixels (100%condition) and in the "eyes closed"sham condition, i.e. conditions in which participants did not have any mirror feedback.

However, these two conditions were associated with a markedly lower occurrence rate; lower velocity and longer latency, relative to conditions with at least some visual information.

Nevertheless, the slight, occasional, illusory displacements observed in the 100% degradation and "eyes closed" conditions are unquestionablyof somaesthetic origin and not of visual origin. As reported above, kinaesthesia is derived from both visual and somaesthetic afferents from the perceived segment andsomaesthetic afferents from the other arm. In a recent study by [START_REF] Kuehn | On the bimanual integration of proprioceptive information[END_REF], the participant's hands (either one hand at a time or both hands together) were moved passively to new positions. The participant's task was to indicate the perceived location of the tip of the index finger of the designated target hand by orienting a laser beam mounted on a cap. It was found that synchronized bimanual movements were associated with a significantly better position sense, relative to unimanual movements. These results are in line with reports by [START_REF] Izumizaki | The illusion of changed position and movement from vibrating one arm is altered by vision or movement of the other arm[END_REF], [START_REF] Hakuta | Proprioceptive illusions created by vibration of one arm are altered by vibrating the other arm[END_REF] and [START_REF] Kigawa | Proprioceptive interaction between the two arms in a single-arm pointing task[END_REF]. Izumizaki and colleagues (2010) reported that in the absence of vision, passive displacement of one arm alters the speed of the vibration-evoked illusion experienced withthe other arm (by ~30%). They concluded that the sensation of movement is related to the difference between the inputs from each arm, rather than the vibration-induced signal from the reference arm alone. Therefore, stimulating the somaesthetic afferents of one arm by moving it passively influences perception of the movement of the other arm; this might have occurred when no visual information was available in Experiment 1. It is likely that detection of these slight illusory displacements was facilitated by the experimental context, in which participants had become familiar with the detection of illusory displacement.

The mechanism underlying the bimanual integration of proprioceptive afferents (even for the purpose of unimanual perception)has not been characterized but may stem from the natural tendency to perform bimanual movements in everyday activities. According to [START_REF] Perez | Modulation of transcallosal inhibition by bilateral activation of agonist and antagonist proximal arm muscles[END_REF], the strength of interhemispheric coupling between the sensorimotor cortices is stronger for bimanual movements. However, Formaggio and colleagues. (2013) recently reported that passive displacement of only one arm can induce bilateral activation of the motor loci (i.e. event-related desynchronization). More generally, it has been found that actively performed unimanual motor tasks involve not only the contralateral primary motor cortex but also the ipsilateral primary motor cortex in asymmetrical way; in right-handed subjects, activation of the left hemisphere during left hand movements is more intense than earlier (with reduced latency) and more intensely (with a higher perceived speed) than in the absence of proprioceptive masking [START_REF] Guerraz | Integration of visual and proprioceptive afferents in kinesthesia[END_REF]. Similarly, [START_REF] Metral | Kinaesthetic mirror illusion and spatial congruence[END_REF] showed that a larger degree of spatial incongruence between the mirror arm and the somaesthetically specified position of the unseen arm subjected to mirror illusion(with a shift of between 0° and 90° in the sagittal plane) was associated with a less intense kinaesthetic illusion. Finally, the illusory displacement can be either strengthened or weakened by adding proprioceptive inputs through vibration of the antagonist or agonist muscles of the hidden arm respectively [START_REF] Guerraz | Integration of visual and proprioceptive afferents in kinesthesia[END_REF][START_REF] Tsuge | Interaction between vibrationevoked proprioceptive illusions and mirror-evoked visual illusions in an arm-matching task[END_REF]. Taken as a whole, our results show that somaesthetic cues from both arms exerted an influence on spatial coding of arm position; this influence may inhibit or facilitate the illusion evoked by visual manipulation.

Conclusion

The first major finding of the present experiments is that the marked impoverishment of mirror feedback has only a marginal impact on the use of visual cues for kinaesthesia. This study shows that (as observed for biological motion) a limited amount of visual information may be enough to provide cues for kinaesthetic illusions. Secondly, it appears that the kinaesthetic mirror illusion is not a purely visual phenomenon; it corresponds to the integration of mirror feedback with somaesthetic afferents and results in perceptual

FUNDING:

The work was funded by the University Savoie Mont Blanc (Chambéry, France).

Figures & captions

  activation of the right hemisphere during right hand movements (Ziemann andHallet 2001;Van der Berg et al. 2011;[START_REF] Beaulé | Interhemispheric control of unilateral movement[END_REF]. Given that (i) most of the participants in the present study were right-handed and (ii) only the non-dominant left arm was moved passively, our experimental conditions may have facilitated the emergence of illusory movement of the contralateral (right) arm.Is the kinaesthetic mirror illusion a purely visual illusion?If stimulation of the somaesthetic afferents of one arm is enough to induce kinaesthetic illusions (albeit limited ones) in the other arm, what is the role of those afferents when they are accompanied by congruent visual cues (such as in the mirror paradigm)? In other words, does the illusory displacement induced in the mirror paradigm have a visual origin (as implicitly suggested in the literature) or does it result fromthe integration of congruent somaesthetic and visual inputs from two arms that often move synchronously (i.e. bimanual coupling)? This question prompted us to devise Experiment 2, in which somaesthetic afferents of the passively moved arm were masked by synchronous co-vibration of antagonistic muscles (the biceps and triceps) of the left arm. The results showed that masking the somaesthetic afferents of the arm reflected in the mirror was associated with a significantly lower velocity of illusory displacement of the other arm. These findings confirmed our hypothesis whereby the mirror illusion, is not a purely visual illusion. In fact, it appears to result from the combination of congruent signals from the two arms: the visual afferents related to the virtually moving arm and the somaesthetic afferents of the contralateral arm. It must also be borne in mind that masking the afferent signals from the antagonistic muscles of the arm subjected to the illusion has exactly the opposite effect; the mirror illusion occurred

  ipsilateral to the moving body side depends on spatio-temporal task complexity and hemispheric specialization. PLoS One 6:e17742. doi: 10.1371/journal.pone.0017742 Ziemann U, Hallett M (2001) Hemispheric asymmetry of ipsilateral motor cortex activation during unimanual motor tasks: further evidence for motor dominance. Clin Neurophysiol 112:107-113. doi: http://dx.doi.org/10.1016/S1388-2457(00)00502-2 COMPETING INTERESTS: None of the authors have any conflicts of interests.

Figure

  Figure 1: The mirror box setup (lower panel) and the different pixel densities used to create the visual impoverishment conditions in Experiment 1 (from 0% to 100%).

  Figure 1: The mirror box setup (lower panel) and the different pixel densities used to create the visual impoverishment conditions in Experiment 1 (from 0% to 100%).

Figure 2 :

 2 Figure 2: Illusion onset quantification. The upper trace depicts the passive (motorized) displacement of the left arm intro flexion. The lower trace shows a typical foot response in response to illusion appearance. The width of the grey bar represents illusion latency.

Figure 3 :

 3 Figure 3: Mean subjective speed (n = 16) of the kinaesthetic illusion, as a function of the degree of visual impoverishment. The dashed line represents the mean subjective rating observed in the sham trials, that is, when participants closed their eyes. The error bars correspond to 95% confidence intervals.

Figure 4 :

 4 Figure 4: Mean subjective speed (A) and onset latency (B) of the kinaesthetic illusion, as a function of the degree of visual impoverishment in the 9 participants that experienced illusory displacement in each visual impoverishment condition. The error bars correspond to 95% confidence intervals.

Figure 5 :

 5 Figure 5: Mean subjective speed (n = 15) of the kinaesthetic illusion, as a function of movement (extension vs. flexion) and proprioceptive masking (masking vs. no masking). The error bars correspond to the 95% confidence intervals. * = p<.05

Acknowledgments

We thank Dr David Fraser (Biotech Communication, Damery, France) for improving the manuscript's English and Dr Jean-Luc Roulin for his help in setting-up Experiment 1.

facilitation of the contralateral arm what is in line with recent studies of the bilateral integration of proprioceptive information.


David Nicolas
email: dnicolas@gmx.net

David Nicolas Counting

come

Introduction

In a recent paper, [START_REF] Liebesman | We do not count by identity[END_REF] claims that we do not "count by identity". Counting some objects by identity is giving the cardinality of the plurality that has these objects as members. Under this conception, some objects are two just in case these objects comprise one object distinct from another and no other object. Liebesman presents two arguments for his claim: the bagel argument and the liter argument.

In my reply, I focus on the bagel argument, which is the most important. Liebesman first purports to show that in (1) Two and a half bagels are on the table, we do not count by identity. In particular he rejects possibilities like the following: the expression two and a half bagels would really be two bagels and a half bagel. He then argues that in (2) Exactly two bagels are on the table, we do not count by identity either. Taking a uniform semantics to be preferable, he concludes that we never count by identity.

As I explain below, I agree that in some cases, we do not count directly and use some kind of measurement instead. But contra Liebesman, I argue that this is the exception rather than the rule. In normal cases, we just count ("by identity"). Moreover, I provide an explicit semantics of the relevant sentences. Finally, I argue that even when we measure, this involves genuine counting.

1) What is the semantics of two bagels and two and a half bagels?

The starting point of Liebesman is the following observation. With some count nouns, like bagel, we understand expressions like a half bagel and (a) half of a bagel, and sentences like this one:

(1) Two and a half bagels are on the table. This is true, for instance, if on the table there are two bagels and a half bagel (half of a bagel).

If instead of two and a half we use a simple cardinal like two, the truth conditions of a similar sentence can be stated like this:

(3) Two bagels are on the table is true iff x (bagel(x) Ù card(x) = 2 Ù on_the_table(x))

{"at least" semantics} where bagel() applies to one or more bagels and card() counts the number of bagels. The sentence is true just in case there is a plurality x of bagels on the table whose cardinality is two. This "at least" semantics of cardinals just asserts the existence of two things. An "exact semantics" would assert the existence of exactly two things and no more [START_REF] Spector | Bare numerals and scalar implicatures[END_REF].

Whether one adopts an "at least" semantics or an "exact" semantics, these kind of truth conditions are inadequate for (1) for two reasons [START_REF] Salmon | Wholes, parts, and numbers[END_REF][START_REF] Liebesman | We do not count by identity[END_REF]): -Intuitively, half a bagel is not in the denotation of bagels, so it cannot be in the denotation of two and a half bagels if two and a half just restricts the denotation of bagels.

-The function card() returns the cardinality of a plurality, which can never be a fractional number.

So what are the truth conditions of the sentence and how do we get them? Liebesman says that it involves a kind of measurement, but he does not provide an explicit semantics for the relevant sentences. In the next section, I present a first type of account, which Liebesman rejects. Then I put forward my own, positive account, in which measurement is involved only in special sentences, like (1).

2) Against analyzing two and a half bagels as two bagels and a half bagel [START_REF] Ionin | The composition of complex cardinals[END_REF] claim that complex numerals "compose". Syntactically, due to something akin to ellipsis, twenty-two bagels is really twenty bagels and two bagels. Semantically: [twenty-two bagels] = lx y z (x=y+z Ù [twenty bagels](y) Ù [two bagels](z)) twenty-two bagels applies to a plurality x just in case x is the sum of a plurality y of twenty bagels and a plurality z of two bagels.

One could propose the same for two and a half bagels. Or more directly, one could suppose that (1) should be analyzed as (4) or a variant like (4'): (4) Two bagels and a half bagel are on the table. (4') Two bagels are on the table and a half bagel is on the table.

But this kind of account has the following problems:1 a) It does not explain why we can refer anaphorically to two and a half as a number:

(5) Two and a half bagels are on the kitchen table. Twice as many onions as that are on the dining room table. (Liebesman 2014: 7) However, this argument is not as strong as it seems. In (5') below, one seems able to refer anaphorically to five things (two bagels and three bialys), and something similar could be happening in (5) itself: (5') Two bagels were on the side table, three bialys were in the bread box, and twice as many kaiser rolls as that were on the counter. b) The account predicts that (1) Two and a half bagels are on the table is false if there are five half bagels, insofar as (3) Two bagels are on the table is false if there are four half bagels from four different bagels. It also predicts that (1) is false if there are two bagels and two quarters from two different bagels, insofar as (6) A half bagel is on the table is false if there are two quarters from two different bagels (Liebesman 2014: 7).

3) My proposal: bagels receives a novel meaning in special contexts

Consider (1) again: (1) Two and a half bagels are on the table. The idea is that in this sentence, the meaning of two and a half clashes with that of bagels, which demands to combine with a cardinal number. To avoid this, a new meaning is constructed (partly analogous to the meaning of a measure expression like two and a half pounds of chocolates):2 [two and a half bagels] = x (bagel'(x) Ù amount-bagel(x) = 2.5) Within this construction, bagels receives this novel meaning (which I abbreviate as Q): [bagels] = Q = ln lx (bagel'(x) Ù amount-bagel(x) = n) bagel'() is the crucial element in fixing the novel denotation of bagels. Plausibly, what bagel'() applies to depends on context. In the most typical context, bagel'() applies only to whole bagels and half bagels. But in other contexts, it may also apply to other fractions, like a quarter of a bagel.

amount-bagel() is a partial, additive function, i. e. for any things to which it applies: If x and y do not overlap, amount-bagel(x+y) = amount-bagel(x) + amount-bagel(y). The function is defined in terms of the units whole bagel and half bagel, and possibly further units. Whenever x is a whole bagel, amount-bagel(x) = 1. And whenever x is a half bagel, amount-bagel(x) = 0.5. In some contexts, further units may also be defined, e. g. if x is a quarter of a bagel, amount-bagel(x) = 0.25. Then in a given context, one uses the units defined in that context to find out whether amount-bagel() associates a number to a plurality of things. amount-bagel() returns a number just in case this plurality can be divided into subpluralities that correspond to the available units, e. g. one bagel here, two half bagels there, and three quarters from three different bagels there.

The novel meaning Q is available only in special contexts. One context is when bagels is combined with an expression like two and a half. Another is when we judge the following sentence false if on the table there are two bagels and a half bagel:3 (2) Exactly two bagels on the table. What we see or know makes half bagels relevant. Then in this special context, bagels receives the novel meaning Q and two stands in competition with two and a half. Demanding the most precise answer, exactly rules out two. By contrast, in normal contexts halves are not relevant, two just stands in competition with other cardinal numbers like three, and (2) excludes these alternatives: it says that there are two bagels, not three, nor more.

4) This is really a novel meaning

An alternative proposal would be that the meaning of bagels is always Q. Liebesman is not explicit about what semantics bagels should receive. But this view might be congenial to his overall position.

However, the intuitions concerning the interpretation of the following sentences make a strong case against this view: (7) There are some bagels on the table. (8) Some bagels are on the table. (9) What there is on the table are bagels. Intuitively, these sentences assert the existence of a cardinal number of whole bagels. This would be completely unexpected if Q were the usual denotation of bagels. The sentences should be true if several halves of bagels were on the table. But that is not the case. Moreover, intuitively, the sentence (3) Two bagels are on the table says that two whole bagels are on the table. If Q were the denotation of bagels, then the sentence should be true if on the table there were four halves from four different bagels. Again, this is not the case.

So in sentence (3) and in sentences ( 7) to ( 9), bagels has just its normal meaning. Inspired by [START_REF] Krifka | Nominal reference, temporal constitution and quantification in event semantics[END_REF] and [START_REF] Bale | Yet more evidence for the emptiness of plurality[END_REF], we can suppose, for concreteness, that the normal denotation of bagels is this (nothing important depends on this particular choice):

[bagels] = ln lx (bagel(x) Ù card(x) = n) where bagel() applies to one or more bagels and card() counts the number of bagels.

5) Even when using half, counting is involved

By contrast, in sentences like the following one, bagels receives the novel meaning Q: (10) One and a half bagels are on the table. In the most typical context, the partial, additive function amount-bagel() is defined in terms of two units: whole bagel and half bagel. So any number it outputs must be derivable from counting and a simple calculation based on these units. 4 Then two kinds of situation make (10) true:

-Situation 1: one counts one bagel and one half bagel and calculates thus (where x is the multiplication sign): amount-bagel(x) = 1x1 + 1x(0.5) = 1.5 -Situation 2: one counts three half bagels and calculates thus: amount-bagel(x) = 3x(0.5) = 1.5

In general, for each unit available in the context, one counts how many things correspond to (or instantiate) that unit. And the rest of the calculation proceeds similarly.

Liebesman claims that when we interpret sentences like (10), we do not count ("by identity"). What precedes shows that the interpretation involves genuine counting and multiplication, based on the units bagel and half bagel (and further units such as quarters if these are given in the context).

Conclusion

Following [START_REF] Salmon | Wholes, parts, and numbers[END_REF], Liebesman argues that in sentences like (1) Two and a half bagels are on the table, we do not count "by identity". He concludes from this that we never do. This second step, however, is not warranted. There is another account, which is plausible and attractive. We genuinely count in sentences like (3) Two bagels are on the table. It is only in special cases, like (1), that we use some kind of measurement. And moreover, even when we do measure, this involves genuine counting.

The account of Ionin & Matushansky faces another challenge. The semantics given for twenty-two bagels allows overlap between the twenty bagels and the two bagels. Overlap is ruled out pragmatically. So there should be contexts where overlap is possible. But it is never the case when we describe something as twentytwo bagels.

On measure expressions, see[START_REF] Krifka | Nominal reference, temporal constitution and quantification in event semantics[END_REF],[START_REF] Rothstein | Counting, measuring, and the semantics of classifiers[END_REF],[START_REF] Partee | Sortal, relational, and functional interpretations of nouns and Russian container constructions[END_REF].

Both Salmon (1997) and[START_REF] Liebesman | We do not count by identity[END_REF] claim that the sentence is false under these circumstances. Several native speakers I have consulted accept both this interpretation and another one under which the sentence is true since on the table there are two bagels and not three.

See Wiese (2003: 28) for a similar view concerning extensive measurement more generally.


Kamani Perera

Dinesh Chandra

HAL is

TQM being a systemaIc process, which focuses on understanding customer needs and improving customer services, involves the quesIon of whether products and services do correspond to customers' expectaIons and needs and whether both are harmonious with each other. Here, the customer means anyone who is impacted by the product or process. The main idea of using TQM methods for ETD repositories is to saIsfy and even excel the user needs by using effecIve scienIfic methods. TQM constantly supports for the improvement of ETDs performance in an efficient way. The objecIve is achieved by involving all employees who maintain ETD repository to suit the needs of research scholars. To maintain an effecIve ETD repository, it is important to meet TQM values such as providing the customers' needs, exact assessment, conInuous improvement, teamwork, and enthusiasm of the ETD repository builders.

The introducIon of TQM requires the Top Management to provide leadership and support for all quality iniIaIves. Quality goals are moving targets and therefore require commitment from the whole OrganizaIon. Following factors, in parIcular, need to be taken into account:

• TQM is a result--oriented approach.

• TQM involves a process of c o n s t a n t p r o g r e s s a n d therefore ensures that library personnel are ready to play a construcIve role in the process.

• TQM ensures the standard, performance, reliability, appearance, commitment to delivery Ime of the required informaIon.

• TQM also ensures that the effort necessary for its implementaIon are at the same Ime rewarding for both staff and the insItuIon.

It is apparent that TQM can play a significant role in ETD repositories and it may lead to a result--oriented approach by way of improving data skills of research scholars. TQM is not a programme for a specific period, but it is the way--of--life focused towards conInuous i m p r o v e m e n t o f t h e OrganizaIon.

Benefits of TQM in ETD Repositories

• Achieve quality presentaIon and enhancing quality review process through various scienIfic skills to improve the process.

• Excel user's expectaIon

• Upgrading the University community • Staff development


Claude Tadonki
email: claude.tadonki@mines-paristech.fr

Wilson-Dirac Operator Revisited on Multicore Vector Computers

We revisit the Wilson-Dirac operator, also refered as Dslash, on multicore vector machines. The Wilson-Dirac operator is the major computing kernel in Lattice Quantum ChromoDynamics (LQCD), which is the canonical discrete formalism for Quantum ChromoDynamics (QCD) investigations. QCD is the theory of sub-nuclear particles physics, aiming at modeling the strong nuclear force, which is responsible for the interactions of nuclear particles. Based on LQCD formalism, intensive simulations are performed following the Monte Carlo paradigm. Informative observations are expected from large-scale and numerically sensitive LQCD simulations. The corresponding computing demand is therefore tremendous, thus the serious consideration for powerful supercomputers. Designing efficient LQCD codes on modern (mostly hybrid) supercomputers requires to efficiently exploit all available levels of parallelism including accelerators. Since the Wilson-Dirac operator is a coarse-grain stencil computation performed on huge volume of data, any performance and scalability related investigation should skillfully address memory accesses and interprocessor communication overheads. In order the lower the latter, an explicit shared memory implementation should be considered at the node level, since this will lead to a less complex data communication graph. This the main focus of the current paper, where we provide, explain, and discuss a multi-threaded vector implementation, whose experimental results in double precision on the recently released INTEL BROADWELL based machine show a competitive absolute efficiency and a good scalability on one of its four NUMA nodes. An extension to all available nodes is currently under investigation through NUMA-awareness consideration.

I. INTRODUCTION

Quantum ChromoDynamics (QCD) [START_REF] Wilczek | What QCD Tells Us About Nature and Why We Should Listen[END_REF], the theory of the strong nuclear force, can be numerically simulated on massively parallel supercomputers using the Monte Carlo paradigm and the lattice gauge theory (LQCD) approach (see Vranas et al. [START_REF] Vranas | Massively parallel quantum chromodynamics[END_REF]).

A typical LQCD simulation workflow iteratively involves basic linear algebra computations on a huge number of variables. The major LQCD kernel is the inversion of the Dirac operator, which is an important step during the synthesis of a statistical gauge configuration sample. Indeed, in the Hybrid Monte Carlo (HMC) algorithm [START_REF] Urbach | HMC Algorithm with Multiple Time Scale Integration and Mass Preconditioning[END_REF], it appears in the expression of the fermionic force, used to update the momenta associated with the gauge fields along a trajectory. The Wilson-Dirac matrix is sparse and implicit, thus iterative solvers are the main option for its inversion. In addition, some sensitive scenarios bring up almost null eigenvalues, which fact exacerbates numerical instability and pushes far away the required number of iterations to reach convergence. Moreover, such a numerical sensitivity justifies the importance of double precision computations. Some authors consider socalled mixed-precision [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF], which sacrifies the precision of the core computation, while keeping the double precision constraint for the convergence criterion. In the presence of very small eigenvalues, thus a ill-conditioned Wilson-Dirac matrix, the iteration process will be likely to diverge or the way to convergence will be noticeably longer. The use of mixed precision is motivated by the strong desire to use single precision in order to have faster computations using GPUs for instance or (larger) vector units. However, the penalty from the numerical consequences might not be affordable in case of numerically sensitive LQCD scenarios like the ones related to very small pion mass. For all the aforementioned reasons, the need for efficient high-precision implementations of the Dirac operator is on the spotlight of both the HPC and the LQCD communities.

A common way to parallelize LQCD applications is to follow the domain decomposition paradigm, which means to partition the lattice into sublattices and then assign each sublattice to a computing node (see [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF], [START_REF] Luscher | Implementation of the lattice Dirac operator[END_REF]). This yields a standard SPMD model which is then mapped onto a given parallel machine. Thus, tuning an individual computing node to efficiently perform a critical part of the simulation is a genuine LQCD contribution. Indeed, the impact of optimizing the computation on a single compute node is that the communication graph related to data exchanges (mostly through MPI) will be of a smaller size, thus less complex. Thereby, the interprocessor communication overhead should be significantly lowered. This is very important for large scale LQCD on supercomputers, where each node has to communicate with its 8 "neighbors" (stencil computation), thus the unacceptable communication overhead usually observed in that context. Number of authors have studied LQCD implementation on various supercomputers [START_REF] Vranas | Massively parallel quantum chromodynamics[END_REF]. However, the relative efficiency of standard frameworks on large computing clusters is likely to be mixed, sometimes unacceptable. The main reason is that, current and future supercomputers potentially have a noticeable computing power (breathtaking for some of them), but all levels of parallelism need to be skillfully harnessed in order to harvest a significant fraction of the overall computing potential. In addition, memory accesses and data exchanges, never counted on the theoretical peak performance, are dominant in LQCD computations.

For the critical case of solving Wilson-Dirac system, a domain decomposition approach associated with the deflation technique (related to small eigenvalues) is studied by Luscher in [START_REF] Luscher | Implementation of the lattice Dirac operator[END_REF]. A mixed-precision solution accelerated with GPUs is proposed by Clark et al. [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF]. . A hybrid threaded-MPI approach is presented in [START_REF] Smelyanskiy | High-performance lattice QCD for multi-core based parallel systems using a cache-friendly hybrid threaded-MPI approach In[END_REF] by Smelyanskiy et al. QCD implementations on the IBM-CELL are reported and discussed in [START_REF] Belletti | QCD on the Cell Broadband Engine[END_REF], [START_REF] Ibrahim | Implementing Wilson-Dirac operator on the cell broadband engine[END_REF], [START_REF] Tadonki | An efficient CELL library for Lattice Quantum Chromodynamics[END_REF], an a dedicated cluster of CELLs is presented in [START_REF] Pleiter | QPACE: Power-efficient parallel architecture based on IBM PowerXCell 8i[END_REF] by Pleiter. An implementation on Intel Xeon Phi by Joo et al. can be found in [START_REF] Joó | Lattice QCD on Intel Xeon Phi[END_REF]. This panorama will be extended and detailed in the related work section.

The main argument of this paper is that the way to get closer to a significant global efficiency on supercomputers is to put a serious design effort at the level of the computing node, harnessing all aspects related to its potential performance. In addition to lowered the data communication overhead because of a less complex interprocessor exchanges, data redundancy is also reduced by this explicit shared memory implementation on local nodes. This is the basis of our main contribution from this work, where we provide efficient strategies for memory and data management, vector computing, and multithreading, all illustrated by very promising experimental results. We focus on one evaluation of the Wilson-Dirac operator, also called Dslah. Since Wilson-Dirac inversion is exclusively done through iterative approaches, making each iteration faster should certainly improve the overall performance, beside those approaches which try to reduce the number of iterations through purely numerical techniques (not our concern here). Beside our factual achievements, this paper aims at providing a pedagogical and instructive HPC material related to high performance LQCD.

The rest of the paper is organized as follows. The next section provided basic and computing LQCD backgrounds, followed by basic computing considerations. Next, we discuss important HPC facts related to large-scale LQCD. Related work is presented in section V, while our methodology and efforts are presented in section VI, followed by experimental results in section VII. Section VIII present some future works, and section IX concludes the paper.

II. LQCD BACKGROUND AND COMPUTATION

LQCD models the time-space universe as a four dimensional grid. In practice, a regular bounded grid is considered through a subset of N 4 , which can be represented as {0, 1,

• • • , L t - 1}×{0, 1, • • • , L x -1}×{0, 1, • • • , L y -1}×{0, 1, • • • , L z - 1}
, where L t , L x , L y , and L z are the size of each dimension respectively. The size of the lattice for a given scenario, commonly written in the form L t ×L x ×L y ×L z , is somehow correlated with the underlying space density. That's why largescale LQCD is a serious target for cutting-edge investigations in particle physics. Each point x of the lattice, commonly referred as a site, is connected to its height neighbors x ± e i , i = 1, 2, 3, 4, where e i are the vector of canonical basis of N 4 , and each ±e i operation is performed modulo L i at the i th component. This yields a regular symmetric graph.

Five 4 × 4 special matrices, called Dirac γ-matrices, are defined below

γ 0 =     0 0 -1 0 0 0 0 -1 -1 0 0 0 0 -1 0 0     γ 1 =     0 0 0 -i 0 0 -i 0 0 i 0 0 i 0 0 0     (1)
γ 2 =     0 0 0 -1 0 0 1 0 0 1 0 0 -1 0 0 0     γ 3 =     0 0 -i 0 0 0 0 i i 0 0 0 0 i 0 0     (2)
γ 5 =     1 0 0 0 0 1 0 0 0 0 -1 0 0 0 0 -1     (3)
The Wilson-Dirac operator can be expressed as follow:

Dψ(x) = Aψ(x) - 1 2 3 µ=0 { [(I 4 -γ µ ) ⊗ U x,µ ]ψ(x + e µ )+ [(I 4 + γ µ ) ⊗ U † x-μ,µ ]ψ(x -e µ )} (4)
where • I n is the identity matrix of order n,

• e µ is the µ th vector of the canonical basis of {0, 1} 4 ,

• A is a 12 × 12 complex matrix of the form αI 12 + β(ν ⊗ γ 5 ), where α, β are complex coefficients and ν a 3 × 3 complex matrix, • x is a given point of the lattice (a site), which is a finite subset of N 4 , • ψ is an array over the lattice (called quark field or Wilson vector), and for each site x, ψ(x) is a 12-components complex vector (called spinor) , • U x,µ is a 3 × 3 complex matrix (called gluon field matrix, gauge matrix, or SU3 matrix), which is associated to the link (x, x + e µ ), and also to the reverse link (x + e µ , x), • ⊗ is the matrix tensor product • A † = ĀT (i.e. transpose of the conjugate matrix) For a given quark field ψ, Dψ is obtained by computing Dψ(x) for all sites of the lattice, and the result is also a quark field of the same length. Equation [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF] shows that Dψ(x) is a linear combination of the components of ψ(x). Thus, it is consistent to see Dψ as a matrix-vector product, and thereby consider D as an implicit square matrix (the Wilson-Dirac matrix). This matrix-product, sometimes referred in the literature as Wilson Dslash operation (WD), is the most time consuming kernel as it involves a significant amount of floating point operations on larger lattices and is done very frequently. Solving a linear system, whose the principal matrix is the Wilson-Dirac matrix is an important LQCD operation that is performed several times along a trajectory. Since the Wilson-Dirac matrix is implicit and sparse from its specification, iterative solvers are so far the only considered approaches to solve the corresponding linear system, called the Wilson-Dirac equation. In addition, for some specific but important physics parameters, the matrix is ill-conditioned, which severely increases the number of iterations to reach an acceptable level of convergence. This noticeable repetition of the Dslash operation clearly justifies the need for very fast implementations.

III. BASIC AND TYPICAL COMPUTING CONSIDERATIONS

There is a number of important facts to know or consider when it comes to deal with LQCD implementations. We describe some of them.

A. Data complexity

All data structures are based on complex data type, which means a structure composed with two floating point numbers. Then, all arithmetic operations follow their corresponding specifications on complex numbers. The update of one spinor involves height input spinors and the height SU(3) matrices of the corresponding links. This yields a volume of data (in Bytes) given by

8(12 × 2 × p + 9 × 2 × p) = 336p, ( 5
)
where p is the size of the actual floating point number, which is typically 4 bytes (resp. 8 bytes) for single (resp. double) precision, 2 × p stands for the derived complex type. We will later see that the choice between single precision and double precision is not only a matter of data size. For a given lattice of size L = L t × L x × L y × L z , we see that 336 × L × p bytes of data will be moving within the computing system. The PetaQCD project [1], for instance, was targeting 256 × 128 3 double precision simulations, which means 336 × 256 × 128 3 × 16 bytes = 1.3125 × 2 50 bytes = 1.3125 petabytes of effective data transfer at various levels. This aspect sometimes appears as the main reason for using large clusters, since the aggregation of available (distributed) memories should be sufficient to house all working data.

B. Organization of the Computation

Computing Wilson-Dslash is typically done by traversing the whole lattice while updating the corresponding spinor at each site. This yields one dependence-free main loop, whose the body implements equation ( 4). The effective scheduling of this main loop, if different from the natural 4D lexicographic order, should be managed with the aim of addressing explicit data reuse or content sharing among the caches, following a skillful analysis of the data dependence graph. Unfortunately, the coarse granularity of the computation makes the potential of this effort rather marginal in practice, unless it goes along with an explicit mechanism that implements a specific data management strategy [START_REF] Smelyanskiy | High-performance lattice QCD for multi-core based parallel systems using a cache-friendly hybrid threaded-MPI approach In[END_REF]. Concerning the calculation of Dψ(x) following equation ( 4), some factorizations should be applied in order to put in common the major floating point operations. Indeed, let first observe that a relation of the form v = (I 4 -sγ µ )u, where u, v are two 4-components complex vectors and s = ±1, can be computed as follows (for s = 1, the case s = -1 is similar):

µ = 0 µ = 1 v 1 = u 1 + u 3 v 2 = u 2 + u 4 v 3 = v 1 v 4 = v 2 v 1 = u 1 + iu 4 v 2 = u 2 + iu 3 v 3 = -iv 2 v 4 = -iv 1 µ = 2 µ = 3 v 1 = u 1 + u 4 v 2 = u 2 -u 3 v 3 = -v 2 v 4 = v 1 v 1 = u 1 + iu 3 v 2 = u 2 -iu 4 v 3 = -iv 1 v 4 = iv 2 (6)
We see from ( 6) that only the first two components v 1 and v 2 need to be computed, and afterwards the remaining two components v 3 and v 4 are derived by considering a factor in {1, -1, i, -i}. This saving is the major computing benefit of scheme [START_REF]QDP++[END_REF], especially when a matrix-vector product is involved in between as we are going to illustrate in our main calculation. Considering the so-called normal factors decomposition of the tensor product and the associated commutativity [START_REF] Van Loan | Computational Framework for the Fast Fourier Transform[END_REF], we get

(I 4 -sγ µ ) ⊗ U = ((I 4 -sγ µ ) ⊗ I 3 )(I 4 ⊗ U ) (7) = (I 4 ⊗ U )((I 4 -sγ µ ) ⊗ I 3 ) (8)
In ( 4), each term of the form

[(I 4 -sγ µ ) ⊗ U ]ψ(x) becomes [(I 4 -sγ µ ) ⊗ U ]ψ(x) = (I 4 ⊗ U ) ((I 4 -sγ µ ) ⊗ I 3 )ψ(x) (9)
By (virtually) block partitioning the 12-components vector ψ(x) (we use ψ for simplicity) into 3-components vectors (sometimes referred as su3 vectors) ψ (k) = (ψ k , ψ k+1 , ψ k+2 ), for k = 1, • • • , 4, we get a 4-components block vector expressed by ψ = (ψ (1) , ψ (2) , ψ (3) , ψ (4) ). Using this block representation, we get ((I 4 -sγ µ ) ⊗ I 3 )ψ = (I 4 -sγ µ ) ψ, which can then be calculated using ( 6), provided we replace u k by ψ (k) . Having thereby evaluate ω = (I 4 -sγ µ ) ψ, we finally have to compute ϕ = (I 4 ⊗ U )ω, which is equivalent to (ϕ (1) , ϕ (2) , ϕ (3) , ϕ (4) ) = (U ω (1) , U ω (2) , U ω (3) , U ω (4) ).

C. Even-odd Partitioning

Noticing that the update of a given site (t, x, y, z) involves height sites (t, x, y, z)±e i , i = 0, 1, 2, 3, we see that summing up all the components of two dependent sites respectively yields a difference of 1. Therefore, we might think of partitioning the lattice into two subsets, based on the parity of the sum of their components, thus the odd (resp. even) sublattice. The main advantage of this partitioning is that data dependencies are only between the odd sublattice and the even sublattice, with the gauge matrices seated on the corresponding links. This organization simplifies the macroscopic data exchanges and improves read/write data locality. There is less attention in the literature for gauge matrices sharing, we address this in our work as we will shall see.

D. Gauge Matrices Storage and Management

There is one gauge matrix per link in the lattice, thus 4L gauge matrices for a lattice of length L, since each site has 8 neighbors and the graph is symmetric. This yields a huge amount of data that need to be stored and managed efficiently since there is poor reuse of gauge matrices. Indeed, each of them is used only two times (one time in some cases), whereas each spinor is used height times. Thus, gauge matrices are serious source of (compulsory) cache misses, waste of memory bandwidth and cache pollution. The later is due to the fact a gauge matrix, whose size is 9×2×{4, 8}, does not fully covers typical 64-bits cache lines. In order to reduce the impact of this, and maybe to simplify the indexing, the so-called gauge copy is applied. The idea is to store contiguously the 8 gauge matrices of each site, which explicitly doubles the volume of data but yields a significant (memory) performance improvement. Moreover, since memory accesses is dominant in any case, the so-called SU(3)-reconstruct might be considered. Indeed, the third row of a SU(3) matrix can be reconstructed (on the fly) by taking the complex conjugate of the cross product of the first two rows (i.e. u 3 = u 1 ∧ u 2 ). This 2-row gauge field compression is exploited in [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF] and in this work.

E. Important Numerical Aspects

High-precision LQCD simulations require a special attention regarding numerical issues, we point out two of them. First, the reversibility property, which can be seen as a kind of numerical determinism, aims at ensuring that the calculations made along a trajectory are predictable, and the consistency of the computed results remains whether flowing forward or backward. Thus, every computation scheme should preserve this reversibility, which restriction might prevent from considering whatever efficient but too specific or "black-box"-like subroutines. For several reasons including the reversibility and the quality of the results for better estimates of the targeted physical quantities, the need for highly accurate calculations is relevant, thus the use of double precision computations, which is strictly the case for our investigations in this paper. The temptation of single precision looks strong, as it reduces (by half) the volume of data and leads to higher processor performance as we will detailed later. A mixed precision [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF] approach might be an acceptable compromise. As previously mentioned, for some particular physics parameters, the Wilson-Dirac matrix is known to have almost null eigenvalues, which seriously complicates its inversion. The case would be certainly worst with single or mixed precision. Thus, using double precision (or higher if possible), even if more computationally challenging, is the price for robustness, accuracy and stability.

IV. TECHNICAL HPC CONSIDERATIONS ON THE WAY TO EFFICIENT LARGE-SCALE LQCD

Here we point out number of important facts that should be taken into account at the best in order to harvest an increasing fraction of the available computing power. How observations will be bottom-up following the level of parallelism. Let start by pointing this performance of 0.5 GFlops/core reported by G. Grosdidier [START_REF] Grosdidier | Scaling stories, PetaQCD Final Review Meeting[END_REF] when running tmLQCD [START_REF] Jansen | tmLQCD: a program suite to simulate Wilson Twisted mass Lattice QCD[END_REF] on 10,000 cores of the CURIE-FAT machine [START_REF]QDP++[END_REF]. The machine is based on Xeon X7560 8C 2.26GHz processor, thus a peak of 9 GFlops per core. We then see that each core is running at 5% if its theoretical peak performance, which is unacceptable.

Among the reasons why large-scale LQCD might show some inefficiencies with standard codes, first there is a lack of low-level parallelism, which thereby reduces the theoretical performance expectation by a factor 4, since most of modern processors now have at least 128-bytes vector registers (4 double precision components). Memory performance is also a bottleneck. Indeed, as we have previously explained, computing Wilson Dslash implies a noticeable memory activity with lot of redundant accesses and waste of memory bandwidth. Indeed, the volume of a single spinor (resp. SU(3) matrix) is 192 bytes (resp. 144 bytes). Thus, regarding the L1 cache with its typical 64-bytes cache line, there is no waste coming from spinors use since each of them perfectly fits into 3 cache lines, whereas for SU(3) matrices there is a waste of 192 -144 = 48 bytes per access (unless we are in the gauge copy mode). Considering other levels of the cache, which implies wider cache lines for some architectures, the situation gets worse. We later explain how our data packing, primarily designed for vector computing, also improves the memory efficiency. Another memory issue is cache pollution. Indeed, SU(3) matrices, which are heavily loaded during the computation, have a poor or no reuse. This is not the case, at least by specification, for the spinors, since each of them is used to compute 8 spinors. The benefit from this spinor reuse is likely to be hampered by the aforementioned SU(3) pollution. Another important source of performance penalty is the interprocessor communication overhead when running on distributed memory parallel machines. Indeed, in addition to the natural cost of data transfers, there is a strong gap between the ideal 8D-torus topology required for LQCD computations and the physical topology of existing supercomputers. Moreover, most of the time, there is less attention in providing a suitable virtual topology that will reduce this gap. Hybrid implementations are certainly a relevant approach to reduce the need for explicit data exchanges, but this requires to have an efficient intranode implementation, which is the essence of this paper. With the advent of multi-socket processors, thus with a significant number of cores, designing efficient scalable LQCD code is challenging because of NUMA side effects, whose illustrative case studies can be found in [START_REF] Li | NUMA-aware algorithms: the case of data shuffling[END_REF], [START_REF] Al-Omairy | Dense Matrix Computations on NUMA Architectures with Distance-Aware Work Stealing[END_REF].

V. RELATED WORK

LQCD is a major in both QCD and HPC communities. For the reasons previously explained, LQCD simulations can be computationally challenging for some interesting scenarios. Thus, this hot topic is so far being investigated in various directions.

The basis of LQCD computation are explained by Luscher in [START_REF] Luscher | Implementation of the lattice Dirac operator[END_REF]. The paper also discussed the so-called delfation technique, whose the main aim is to overcome the effects of almost null eigenvalues. Urbach describes in [START_REF] Urbach | HMC Algorithm with Multiple Time Scale Integration and Mass Preconditioning[END_REF] the hybrid Monte-Carlo algorithm with multiple time scale integration and mass preconditioning.

General implementations and experimentations on large computing clusters are discussed by Vranas in [START_REF] Vranas | Massively parallel quantum chromodynamics[END_REF], and also by Grosdidier [START_REF] Grosdidier | Scaling stories, PetaQCD Final Review Meeting[END_REF] within the scope of the PETAQCD project [1]. In [START_REF] Pleiter | QPACE: Power-efficient parallel architecture based on IBM PowerXCell 8i[END_REF], Pleiter presents the QPACE cluster based on IBM PowerXCell 8i and dedicated to LQCD. A hybrid threaded-MPI approach on multi-core based parallel systems is studied by Smelyanskiy et al. in [START_REF] Smelyanskiy | High-performance lattice QCD for multi-core based parallel systems using a cache-friendly hybrid threaded-MPI approach In[END_REF].

Accelerators-based solutions are provided for the IBM CELL by Belletti et al. [START_REF] Belletti | QCD on the Cell Broadband Engine[END_REF], Ibrahim and Bodin [START_REF] Ibrahim | Implementing Wilson-Dirac operator on the cell broadband engine[END_REF], and Tadonki et al. [START_REF] Tadonki | An efficient CELL library for Lattice Quantum Chromodynamics[END_REF]. The case of GPUs is studied by Clark et al. in [START_REF] Clark | Solving Lattice QCD systems of equations using mixed precision solvers on GPUs[END_REF], where a mixed precision is considered and analyzed.

A complete and operational LQCD framework named tmLQCD is provided by Urbach in [START_REF] Jansen | tmLQCD: a program suite to simulate Wilson Twisted mass Lattice QCD[END_REF]. Since LQCD computation kernels are built up from basic linear algebra routines with special data structures, dedicated computing libraries are released for generic use like QDP++ [START_REF]QDP++[END_REF], which provides a data-parallel programming environment suitable for Lattice QCD, and Chroma [START_REF] Edwards | The Chroma Software System for Lattice QCD[END_REF], an open source LQCD toolbox.

A systematic DSL code generation approach is provided by Barthou et al. in [START_REF] Barthou | QIRAL: A High Level Language for Lattice QCD Code Generation[END_REF]. The corresponding framework, named QIRAL, provides a high level language for LQCD code generation together with the associated engine.

VI. OUR PERFORMANCE IMPROVEMENT TECHNIQUES

A. Context and Scenario for our Primary Investigations

We consider a hyperthreaded quad-core machine based on 2.4 GhZ Intel Cori i7, 8 GB of DDR3 main memory, private L2 cache of 256 KB per core and a shared L3 cache of 6 MB. The processor is capable of running 256-bit-wide vector instructions with AVX intrinsics. This yield a theoretical peak performance of 2.4 × 4 = 9.6 GFlops per core in double precision, thus 38.4 GFlops for the whole processor. Our LQCD scenario is a 32 × 16 3 configuration, with the gauge matrices stored following the gauge copy strategy. There is no strong need to consider larger lattices on a node, as the domain decomposition approach will generally allocates a rather medium size sublattice to a single node when running on a large computing cluster. Starting with a standard Dslash implementation, with the main loop (fully inlined) equally distributed among (Pthread) threads following the sublattices partitioning, and compiled using gcc 4.9 with optimization level -O3, we obtain the results displayed in table Table We see that standard code yields 45% of the peak on a single core, which is the conjunction of the basic optimizations previously explained, in addition to the ones from the compiler and maybe the Turbo Boost . The performance drops down to 27% with all the four cores, and 37% with hyperthreading. We will appreciate the impact of our effort on this basis. We emphasize on the fact that our machine has only 4 physical cores, and the 8 threads case is for hyperthreading. The reader should keep this on while inspecting each piece of performance results in this section. In addition, double precision is considered in all cases.

B. Pool of Tasks Approach

A part from memory contention that we address further, the observed moderate scalability of the threads (beyond 2 threads) might come from threads starvation. Our first action in this direction is to create a pool of tasks (sublattices), and leave the threads dynamically provision themselves from that pool, task after task, with mutual exclusion managed by the mutex mechanism and thread-cpu binding applied in order to avoid threads migration at runtime. By doing this, we observe that all threads end up at nearly the same time, depending on the granularity of the pool. Note that, even if the threads have the same task load following the standard block distribution, there are several side effects which might yield unequal running times among the threads, this is more pronounced in the NUMA context. Table I We observe an improvement of the scalability, thus a better overall performance. Table III shows the performances with various number of tasks in the pool ranging from 4 to 4096. We see that a pool of 128 tasks is the most suitable.

C. Scheduling for both Spinor and SU3 reuses

Spinors data dependencies are only between odd-even sites, and two consecutive sites with the same parity share one of their respective height dependent spinors. Thus, when computing the spinors of an odd (resp. even) sublattice in the lexicographical order of its sites, spinors reuse is granted because of this recurrent memory footprint intersection. This is why, even if there is a factor 8 data redundancy related to spinors use, the corresponding impact on the effective performance does not appear to be so offending.

The case of SU(3) matrices is different. First, each SU3 is used twice by a pair of two connected sites, thus the best reuse one could achieve is bounded by a factor 2. A strict application of the gauge copy storage completely drops any reuse expectation. Since the odd sublattice shares exactly the same set of SU3 matrices with the even sublattice, we suggest to keep the gauge copy for the odd sites and pick up the required SU(3) matrices for the even sites from their previous locations. So, we sacrifice data locality for data sharing or reuse. But this just a compromise as it applies only on the half set of sites. We will later see that this counterpart disappears with the way we organize the data for vector computing.

Putting together the previous two analyses including the asymmetric management of the SU(3), we suggest to alternatively iterate over the odd and the even sublattices in order to target the maximum cache sharing for both spinors and SU(3) matrices. Figure Fig. 1 depicts the scheduling for a binary lattice. Thus, the horizontal walk is for spinors reuse, while the vertical walk is for SU(3) matrices reuse. The results obtained with this approach are given in table Table IV, were each even-odd switch is performed after a small chunk of iterations rather than a single one as originally presented.

# We see a slight improvement of both scalability and overall performance compared to the baseline. The most surprising here is that this irregular indexing of the SU(3) matrices for odd lattices is not so penalizing, this might be due to highlevel caches sharing as expected. We think that this aspect needs more investigation, we leave it for future work.

D. AoS to SoA Approach for Efficient Vectorization

AoS-SoA is a well-know data layout transformation aiming at creating regular data accesses depending on the target and the computation paradigm. Vector computing is an important candidate for this approach, since data to be processed should be prepared for vector accesses. In our case, considering double precision and 256-bit-wide vector registers, we just replace the original complex data type typedef struct { double re, im; } complex; by typedef struct { m256d re, im; } complex simd; within the data structures associated to spinors and SU(3) matrices. Afterwards, we design and implement a vector version (using AVX intrinsics) of the linear algebra kernels. Now comes the explicit data shuffling that is required in order to have the vector operands ready for the computation. For instance, if we consider four spinor structures s (j) = [s The s (j) are not required to be consecutive in memory, thus one might expect a penalty from the extra memory cost due to this explicit packing. The stencil nature of LQCD computation exacerbates this fear. Since SU(3) matrices are constant, they are packed once before the computation, thus should not be really counted as an extra dynamic processing. For the spinors, there is no choice other than doing this on the fly. The experimental performance of our effort in that direction are reported in table Table We see an important factor 2 improvement on a single core (and nearly the same on 2 cores), but a weak scaling beyond 2 cores. We now address this point by reducing the culprit memory bandwidth and the expense of extra floating point operations.

E. Two-rows SU3 reconstruct

We recall that the third row of a SU(3) matrix can be reconstructed by taking the complex conjugate of the cross product of the first two rows (i.e. u 3 = u 1 ∧ u 2 ). Thus, our data structure for SU3 matrices does no longer include the third row, which is then reconstructed on the fly each time needed. This strategy saves some memory bandwidth and reduces cache pollution. The performance results on our working scenario are shown in table Table We get an additional flops of 10% on a single core and 20% globally, with a relative improvement of 40% and a better scalability. This is also another evidence that the memory bandwidth associated to SU3 matrices is critical. We now move to the section devoted to our experimental results on the newly released INTEL-BROADWELL processor.

VII. EXPERIMENTAL PERFORMANCE MEASUREMENTS A. Platform Description

We consider the newly released INTEL-BROADWELL-BASED processor described in figure Fig. 2.

Fig. 2. INTEL-Broawell Main Specifications

Considering the available 256-bit-wide vector registers and the corresponding vector processing capability using AVX2, we obtain a peak performance of 2.2 × 4 = 8.8 GFlops/core in double precision, which might be higher with the Turbo Boost.

The numactl --hardware command gives the information displayed in figure Fig. 3. From this NUMA observation, we say that the appreciation of our experimental results should focus on the 11 cores of a single node, since our implementation at this stage is not explicitly numa-aware, with hyperthreading not activated at the time of our experimentations. We see that the performances on one core is impressive and the absolute efficiency slowly decreases as the number of threads is increasing. We end up with 50% of the peak on a full node with its 11 cores. We need to stress the fact our computations are performed in double precision. In [START_REF] Joó | Lattice QCD on Intel Xeon Phi[END_REF] for instance, where single precision is considered, a global performance of 250 GFlops is obtained on an Intel Xeon Phi 5110P, whose peak performance is 1.053 × 16 × 60 = 1010.88 GFlops in single precision, thus a sustained efficiency of 25%. As stated previously, most of authors consider single precision, as this reduces memory bandwidth and increases the potential of vectorization, among other reasons. Showing good performances in double precision is thus one of the strength of this paper. However, some important points need more investigation, and others are worth considering too. We now describe a few of them.

B. Experimental results on INTEL-BROADWELL

VIII. FUTURE WORK

We point out some important aspects that we think should be investigated deeply. We plan to keep investigating most of them.

First, NUMA-awareness needs a serious attention. Indeed, the trend in the design of modern high-performance processors is to increase the number of cores, each of them with a moderate clock frequency. The packaging of the processor cores into different chips and different sockets, all of them sharing the same efficient but complex memory system, yields several NUMA nodes which should be skillfully addressed in order to seek an acceptable level of scalability. The case of LQCD, with the stencil nature of its computation, looks challenging.

Because of the threads scalability issue, among other reasons, the message passing paradigm seems to be widely considered, even with shared memory systems. Although this pragmatic approach is understandable in general, we think that hybrid approaches are worth considering. Indeed, even if message passing implementations can run as such on a shared memory node, explicit data exchanges have a significant cost and might trigger noisy synchronizations. For such exchanges, as long as all communicating processes reside on the same compute node, we are limited by the memory bandwidth available. Once more nodes become involved, the network bandwidth, usually much lower than the memory bandwidth, becomes the limiting factor. The case applications that expose heavy data exchanges like LQCD are particularly sensitive to this aspect, whose intensive experimental studies should also help to assess any technical guess.

As explained and illustrated in this paper, optimizing the memory traffic associated to SU(3) matrices would certainly yield a noticeable global performance improvement. Beside existing approaches like the gauge copy and the two-rows compression, other strategies should be examined. Our even-odd alternating scheduling, intended to bring some level of SU(3) reuse, is still providing a marginal effect at this stage, thus needs a deeper exploration. In addition, explicit prefetching mechanisms could be studied, at least at higher level caches.

Data exchanges in the context of distributed memory machines is also of a central importance. Indeed, it is likely to consider large-scale computing clusters for LQCD simulations. Even when following the way to hybrid implementations, the number of communicating nodes remains important when considering large supercomputers, and the gap between the virtual topology (following the tasks graph) and the effective one (following the interconnect of the machine) is usually important and penalizing. Thus, in addition to considering a shared memory approach at the computing node level, studies that address interprocessor communication in the perspective of better performances are of a keen interest.

IX. CONCLUSION

We have presented our view and efforts on the Wilson Dslash operator of LQCD. Considering the challenging double precision case, we have been able to get more than 50% of the overall peak on one 11-cores chip of the INTEL BROADWELL based on Intel Xheon E5-2699 processors. Our initial efforts on a single core, considering AoS-to-SoA transformation followed by a fast corresponding AVX implementation, together with the two-rows gauge compression, yields a sustained performance beyond 80% of the peak. We think that, when it comes to large computing clusters, harvesting the potential of single node at the best, before moving to the message passing extension, will have a good impact on the overall performance. This is one of our ongoing work.

4

 4

Fig. 1 .

 1 Fig. 1. Alternating Even-Odd Scheduling

2 ,

 2 • • • ], j = 1, 2, 3, 4, the corresponding vector structure would be s = [s

Fig. 3 .

 3 Fig. 3. Numa Specifications of our Hardware

  I.

	#threads	t(s) GFlops %Peak speedup
	1 0.0506	4.17	45 %	1
	2 0.0257	8.20	45 %	1.97
	4 0.0213	9.91	27 %	2.38
	8 0.0154	13.70	37 %	3.29

TABLE I BASELINE

 I PERFORMANCES FROM THE FIRST STAGE CODE

TABLE IV IMPACT

 IV OF THE EVEN-ODD ALTERNATING SCHEDULING

  V.

	#threads	t(s) GFlops %Peak speedup
	1 0.0261	8.07	84 %	1
	2 0.0144	14.68	76 %	1.82
	4 0.0128	16.45	42 %	2.04
	8 0.0140	15.03	39 %	1.86

TABLE V PERFORMANCE

 V WITH AOS-TO-SOA

  VI, the extra flops counting for the SU(3) reconstructs are not considered.

	#threads	t(s) GFlops %Peak speedup
	1 0.0233	9.06	94 %	1
	2 0.0139	15.12	79 %	1.67
	4 0.0099	21.27	55 %	2.35
	8 0.0094	22.38	58 %	2.47

TABLE VI EFFECT

 VI OF THE TWO-ROWS RECONSTRUCT

  Table VII presents our experimental results up to 11 threads, where in each scenario, we bind the process to a single node for the previously explained reasons.

	#threads	t(s) GFlops Peak %Peak speedup
	1 0.0300	7.02	8.8	80 %	1
	2 0.0159	13.28 17.6	75 %	1.89
	3 0.0115	18.31 26.4	69 %	2.61
	4 0.0087	24.19 35.2	69 %	3.45
	5 0.0072	29.41 44.0	67 %	4.19
	6 0.0062	33.93 52.8	64 %	4.83
	7 0.0056	37.58 61.6	61 %	5.35
	8 0.0053	39.91 70.4	57 %	5.69
	9 0.0050	42.06 79.2	53 %	6.00
	10 0.0048	44.32 88.0	50 %	6.25
	11 0.0044	48.02 96.8	50 %	6.81

TABLE VII PERFORMANCE

 VII ON ONE INTEL-BROADWELL NODE

ACKNOWLEDGMENT

This work was initiated from the PetaQCD projet funded by a grant from ANR, the french national agency for research, through the program COSINUS. We also expresse our sincere gratitude to Philippe Thierry from INTEL to have provided us an access to their Broadwell-based machine.