François Févotte
email: francois.fevotte@edf.fr

Bruno Lathuilière
email: bruno.lathuiliere@edf.fr

Debugging and optimization of HPC programs in mixed precision with the Verrou tool

Keywords: Floating-point arithmetic, Verrou, mixed precision, Verification & Validation

The analysis of Floating-Point-related issues in HPC codes is becoming a topic of major interest. First, parallel computing and code optimization often break the reproducibility of numerical results across machines, compilers and even executions of the same program. Second, mixed precision is more and more used to reduce memory footprint and bandwidth wherever possible, thereby benefiting more from SIMD units.

This paper presents how the Verrou tool can help during all stages of the Floating-Point analysis of HPC codes: diagnostic, debugging and optimization. Recent developments of Verrou are presented, along with examples illustrating the interest of these new features for industrial codes.

As an example, Verrou allows debugging code aster, an industrial, parallel, mixedprecision code for the simulation of structural mechanics. Re-using existing infrastructure from the non-regression testing process, setting up Verrou requires limited effort. The first useful debugging information can be obtained after only 25 minutes, while the full automated debugging process takes 10 hours to complete.

Introduction

Floating-Point (FP) arithmetic is becoming a hotter and hotter topic in High-Performance Computing (HPC). First, high computational performance is often achieved at the expense of a loss of control over the order in which FP operations are executed. Vectorizing code is a crucial part of the necessary optimizations to leverage the full power of modern CPUs [START_REF] Plagne | Portable vectorization and parallelization of C++ multi-dimensional array computations[END_REF]. However, loop vectorization (especially for reductions) often requires using algorithms which have the same semantics as sequential ones for real, but not FP arithmetic.

Even without changing the algorithms and source code of a program, aggressive compiler optimizations often achieve greater performance by changing the order in which FP instructions are executed (e.g. automatic vectorization) or simplifying arithmetic expressions (using real arithmetic semantics).

Moreover, the exact same binary executable is not always executed in the same way on modern hardware: whether parallelism is used at the scale of multi-core processors, multi-processor nodes or multi-node clusters, it involves running concurrent computations, often without much control 1 on the order in which they are executed (the cost of a synchronization ensuring repeatability in the order of computation is often considered too large).

Since FP arithmetic is not associative, this change in the order of operations leads to a loss of the reproducibility of the results, which can in turn have a wide array of consequences, ranging from benign difficulties in the debugging process, to more serious issues with the Verification & Validation (V&V) process.

Second, optimizing the use of FP precision is often key to achieving high performance. Using smaller-precision FP numbers allows reducing the memory bandwidth usage as well as increasing the number of simultaneous FP operations performed by a single SIMD instructions. Therefore, high FP precision should only be used where it is needed to ensure the required results quality; FP precision should be reduced everywhere possible. With the current trend of ever increasing SIMD register widths (for example, AVX-512 introduces 256-bit registers), and the large number of available FP formats since IEEE 754-2008 [START_REF]IEEE Standard for Floating-Point Arithmetic. IEEE Std[END_REF], such code optimizations are not to be neglected. Over the last few years, the use of mixed precision has therefore become a point of major interest for the HPC community [START_REF] Michael | Fine-grained floating-point precision analysis[END_REF][START_REF] Rubio-González | Precimonious: Tuning assistant for floating-point precision[END_REF][START_REF] Nathan | Recycled Error Bits: Energy-Efficient Architectural Support for Floating Point Accuracy[END_REF]. However, it is important to keep in mind that optimizing mixed precision programs should always be considered as a search for the optimal balance between results accuracy and run times. The quantification of FP-related losses of accuracy is an essential part of this process.

In this paper, we present how the Verrou tool [START_REF] Févotte | VERROU: Assessing Floating-Point Accuracy Without Recompiling[END_REF] can help deal with these issues in the context of large, industrial, high-performance scientific computing codes such as the ones developed and used by leading actors in the industry. Verrou is a free tool based on Valgrind, available under an open-source licence at http://github.com/edf-hpc/verrou.

Verrou is mainly developed by EDF, the main French electric utility, which heavily relies on Scientific Computing Codes (SCC) and numerical simulation to operate its industrial processes. As such, Verrou primarily targets development teams which need techniques and tools to address FP-related issues, both for the development and debugging of SCCs and for their Verification & Validation.

Part 2 of this paper presents an overview of the FP analysis techniques and tools. In part 3, the Verrou tool and its architecture will be briefly presented. Emphasis will be put on the recent development of new features, released in version 2.1.0 and specifically targeting high-performance codes. This part will also concentrate on the core features of Verrou, which are related to the diagnostic of FP issues. This is in contrast to part 4, where the debugging features of Verrou and its ecosystem will be detailed. Part 5 will be devoted to more methodological topics and show how Verrou can be used to debug large industrial scientific computing codes and help optimize their use of mixed precision.

The interest of the various features described in this paper will be illustrated by examples taken from the analysis of code aster [START_REF]Code Aster: Structures and thermomechanics analysis for studies and research[END_REF], a structural mechanics simulation tool of more than 1.2M lines of code. A previous study [START_REF] Févotte | Studying the Numerical Quality of an Industrial Computing Code: A Case Study on code aster[END_REF] details the complete analysis and debugging of code aster, and shows how the FP analysis process with Verrou can greatly complement a more traditional Quality Assurance (QA) process. In the case of code aster, this QA process relies on around 4 000 test cases, of which around 2 000 are run daily.

Overview of FP analysis techniques & tools

Studying the impact of Floating-Point arithmetic on the quality of results computed by scientific computing codes is not a new topic. Even after it was standardized in IEEE-754, FP arithmetic has been considered a tricky and error-prone subject [START_REF] Goldberg | What every computer scientist should know about floating-point arithmetic[END_REF], and much work has therefore been devoted to tools and techniques helping to diagnose FP-related losses of accuracy in computing codes.

Guaranteed methods

Some of these techniques, such as Interval or Affine Arithmetic [START_REF] Revol | Introduction to the IEEE 1788-2015 Standard for Interval Arithmetic[END_REF][START_REF] Jacquemin | A reduced product of absolute and relative error bounds for floating-point analysis[END_REF] give strong guarantees on the results, but can produce largely overestimated error bounds and are often limited to small programs. Such techniques and tools are therefore much used for the verification of small but crucial parts of larger programs (such as for example the computation of mathematical function in the standard libm [START_REF] Florent De Dinechin | Certifying the floatingpoint implementation of an elementary function using Gappa[END_REF]) or of full programs which can have large impacts (such as all Cyber-Physical Systems [START_REF] Bak | Simulation-equivalent reachability of large linear systems with inputs[END_REF][START_REF] Ábrahám | Techniques and tools for hybrid systems reachability analysis[END_REF]).

An important advantage of these guaranteed methods is their ability to provide guarantees for a range of inputs, in contrast to other techniques presented hereafter, which often provide only unwarranted indications that are only valid for a single input dataset.

Comparison between several precisions

Other techniques can be used, such as the comparison between an analyzed result and a reference computed in higher precision. Such techniques offer far less guarantees, but are far simpler. For example, fpDebug [START_REF] Benz | A dynamic program analysis to find floating-point accuracy problems[END_REF] performs the Dynamic Binary Instrumentation (DBI) of an input executable, in order to compare each floating-point value in the program with one obtained with higher precision. FpDebug outputs a graph indicating, for each floating-point operation, whether inaccuracies come from input data or from the operation itself.

The rpe [START_REF] Dawson | rpe v5: an emulator for reduced floating-point precision in large numerical simulations[END_REF] library uses the same idea in a reversed way: it emulates the use of reduced floating-point precision in order to estimate whether some parts of the program might be redeveloped in reduced precision without entailing too much change in the results.

Tools like Precimonious [START_REF] Rubio-González | Precimonious: Tuning assistant for floating-point precision[END_REF] also rely on comparisons between single-, double-, and quadrupleprecision results in order to automatically determine which variables in a program can benefit from using a lower precision without perturbing the results too much. This results in an optimized mixed-precision implementation. In this case, program instrumentation is performed at the compilation level (based on LLVM).

Stochastic arithmetic

Between theses two classes of techniques, the broad family of stochastic methods give far less guarantees, but are much more adapted to large and complex programs such as those developed by the HPC community to meet the industrial need for high-fidelity simulations of physical systems. Stochastic methods model the inaccuracies introduced by round-off errors as uncertainties on the results of computations. These uncertainties are estimated by transforming inaccurate results into random variables, which can be sampled and studied in a statistical way. Two main stochastic methods have emerged.

On the one hand, the CESTAC methodology [START_REF] Vignes | A stochastic arithmetic for reliable scientific computation[END_REF] models round-off errors by random rounding: the result of every inexact computation in a program is randomly rounded upward or downward. This effectively transforms the global result of a complete computation into a random variable, of which the standard deviation can be linked (under some hypotheses) to the inaccuracy of the results. CADNA [START_REF] Jézéquel | A new version of the CADNA library for estimating round-off error propagation in Fortran programs[END_REF][START_REF] Lamotte | CADNA C: A version of CADNA for use with C or C++ programs[END_REF] is the most used library implementing Discrete Stochastic Arithmetic (DSA), a synchronous variant of CESTAC in which all random samples of a given operation are computed at once. Using CADNA requires accessing and modifying the source code of the analyzed program, which is not always easy or even possible for large code bases. Nevertheless, successful uses of CADNA have been reported on large simulation codes [START_REF] Montan | Sur la validation numérique des codes de calcul industriels[END_REF], which validates the adequateness of the CESTAC method in industrial contexts.

PROMISE [START_REF] Graillat | Auto-tuning for floating-point precision with discrete stochastic arithmetic[END_REF] is a mixed-precision optimization tool which leverages the same kinds of algorithms as Precimonious to find optimal sets of variables which could use lower-precision floating-point numbers. In contrast to Precimonious, PROMISE relies on CADNA to assess the validity of the mixed-precision configurations that it considers.

On the other hand, Monte-Carlo Arthmetic (MCA) [START_REF] Parker | Monte Carlo arithmetic: exploiting randomness in floating-point arithmetic[END_REF] models several types of FP-related problems while allowing the user to choose an arbitrary "virtual precision". This allows estimating the behavior of a given program with a different precision than what appears in its source code, in contrast to CESTAC, which always models round-off errors at the precision which is implemented in the program. MCA is implemented in various libraries such as mcalib [START_REF] Frechtling | Mcalib: Measuring sensitivity to rounding error with monte carlo programming[END_REF]. Tools such as Verificarlo [START_REF] Denis | Verificarlo: Checking Floating Point Accuracy through Monte Carlo Arithmetic[END_REF] allow leveraging the full power of MCA for the analysis of large, industrial code bases. Verificarlo presents itself as a custom compiler based on LLVM, and implements MCA by instrumenting programs at the end of the compilation, after most optimization passes.

This paper focuses on Verrou, an FP analysis tool which performs the Dynamic Binary Instrumentation (DBI) of any given program. This eliminates the need for an instrumentation of the program sources or even a recompilation.

Verrou originally addressed diagnostics and debugging, and implemented several forms of random rounding: an asynchronous CESTAC method, and a variant of MCA with virtual precision equal to the real precision. We describe hereafter how recent versions of Verrou have been extended to implement other forms of FP analysis, and also address mixed-precision-related issues.

Presentation of the Verrou tool

Verrou is a tool aiming at helping diagnose, debug and optimize FP-related issues in large, industrial scientific computing codes. An earlier version of the Verrou tool has been presented in [START_REF] Févotte | VERROU: Assessing Floating-Point Accuracy Without Recompiling[END_REF], where internal arithmetic-related algorithms were presented in detail. Only the most important aspects of Verrou will be recalled here, and this paper will rather focus on recent developments in Verrou, released in version 2.1.0. Specifically, the features related to the use of Verrou for the diagnosis of mixed precision programs will be detailed. Also, this paper will focus more on the presentation of the features themselves, and less on the internal implementation.

From a user standpoint, Verrou instruments the program (in its binary form), replacing each FP instruction in it with a variant implementing another type of FP arithmetic. Results of the computation are output like in any normal execution, except that they are affected by the cumulative effect of all perturbed FP instructions. Analyzing the observed change in the results allows estimating the global impact of FP arithmetic for the code.

In its recent versions, the architecture of Verrou closely follows this description, with 3 major components which will be described in more details in the remainder of this section.

• The front-end is the part of Verrou responsible for running the user-given program, replacing each FP instruction with a variant.

• The back-end implements variants of all FP instructions, effectively defining a new arithmetic to be used in place of the original one. Different back-ends can be used in Verrou, implementing various arithmetics.

• A post-processing stage is needed in order to analyze the effect resulting from the use of an alternate arithmetic. This is mostly left for the user to implement, but advice can be given as to what kind of post-processing should be performed.

This architecture of Verrou in three parts stems from the Interflop1 initiative, which aims at providing a uniform API allowing interoperability between front-ends and back-ends of several FP intrumentation tools such as Verrou and Verificarlo [START_REF] Denis | Verificarlo: Checking Floating Point Accuracy through Monte Carlo Arithmetic[END_REF].

Front-end

Verrou is based on Valgrind to perform a Dynamic Binary Instrumentation (DBI) of the given program, and replace each FP instruction by a variant implemented by the user-chosen back-end.

The main advantage of using Valgrind is the ease of use of Verrou for end-users: there is no need to manually change the program or even recompile it. All that is needed is to prefix the usual command-line with in order to invoke Valgrind with the Verrou tool:

valgrind --tool=verrou [VERROU ARGS] PROGRAM [ARGS]
The development teams of many scientific codes already rely on Valgrind for their memory debugging, so that defining such command-line prefixes might even already be part of the standard development-test-debugging routine.

It is also worth noting that relying on DBI makes the core parts of Verrou independent of the tools or languages used for the development of the program. Verrou is therefore languageagnostic: it works for multi-language applications such as code aster [START_REF] Févotte | Studying the Numerical Quality of an Industrial Computing Code: A Case Study on code aster[END_REF], which internally uses Fortran, C, C++ and Python. Successful uses of Verrou have also been reported with languages less commonly used by the HPC community, such as JAVA or Rust. For the same reasons, Verrou works with any compiler and set of compilation options, which allows instrumenting programs exactly as they will later be used for industrial purposes. A consequence of great importance for the HPC community is that Verrou is naturally compatible with any MPI implementation:

mpirun [MPI ARGS] valgrind --tool=verrou [VERROU ARGS] PROGRAM [ARGS]

Also, programs analyzed with Verrou are not restricted in the libraries that they use: Verrou can analyze any part of the program as a whole, even third-party libraries, even closed-source. Again, it might be of interest for the HPC community to note that Verrou can also instrument multi-threaded applications using any framework (pthreads, tbb, OpenMP. . . ). But an important limitation of Valgrind in this case is that threads will be "sequentialized", which may incur a large performance overhead.

In the commands above, VERROU ARGS allows changing several aspects of Verrou, particularly the arithmetic back-end to be used (described in the next paragraph). It is also worth noting that the Verrou front-end can be instructed to only instrument part of a program. This feature is key to the debugging methodologies described in part 4.

Back-ends

Several back-ends in Verrou define variants of FP arithmetic to be used in place of the standard FP arithmetic. The back-end to be used in a given analysis can be selected using the --rounding-mode command-line option to Verrou.

Before entering the details of what type of arithmetic is implemented in each back-end, we discuss here a generic feature which all back-ends implement. When a Not-a-Number (NaN) appears in the results, Verrou emits a Valgrind error, which is reported to the user, along with the current back-trace. When combined with the usual Valgrind debugging features, this allows for an integrated debugging process.

Another interesting point to mention here is the verification process of Verrou itself. Recent versions rely on the UCB Floating-Point tests database2 to check the correctness of all arithmetic backends in all cases. This includes tricky corner cases such as denormalized numbers, non-finite values, positive and negative zeros, etc. Provided that Verrou has access to Fused Multiplication and Addition (FMA) operations, its back-ends correctly pass all UCB tests.

The verification process of Verrou also relies on a set of small programs which include various types of operations (including FMA). For these programs, tests are performed to check that rounding modes emulated by Verrou produce the exact same results as when the program is run without Verrou and the corresponding roundings are performed by the hardware.

Deterministic Rounding (DR)

In order to debug Verrou itself and check for instrumentation-induced non-reproducibilities, a Deterministic Rounding back-end re-implements within Verrou any of the 4 standard rounding modes defined by IEEE-754. This back-end can also be used as a cheap way of assessing the impact of round-off errors in FP computations. As discussed in [START_REF] Kahan | How futile are mindless assessments of roundoff in floating-point computation?[END_REF], some insight on FP errors can be obtained from the comparison of a few results obtained with the same program, using different rounding modes.

As an extension to this scheme, an additional 5 th deterministic rounding mode can be emulated by Verrou. Named "farthest" (--rounding-mode=farthest), this mode always rounds inexact computations in the opposite way to the standard round-to-nearest mode.

Random Rounding (RR)

The historical Verrou backend implements a stochastic arithmetic variant in which the result of each inexact FP instruction is randomly rounded either upward or downward. Depending on the chosen probability law, this can be similar to an asynchronous CESTAC arithmetic [START_REF] Vignes | A stochastic arithmetic for reliable scientific computation[END_REF] or a variant of Monte-Carlo Arithmetic [START_REF] Parker | Monte Carlo arithmetic: exploiting randomness in floating-point arithmetic[END_REF].

• --rounding-mode=average: in this mode, the probability of rounding upward or downward is determined such that the expected value of the randomly rounded result equals the real mathematical result. Suppose an operation z = x • y is to be computed, where x and y are the (FP) operands, z is the real result, and • ∈ {+, -, ×, ÷} is the operator. The randomly rounded result Z is defined with probabilities given by

P[Z = z] = z -z z -z and P[Z = z] = z -z z -z , such that E[Z] = z.
In the equations above, z and z respectivey denote the value of z rounded downward and upward. With the terminology defined in [START_REF] Parker | Monte Carlo arithmetic: exploiting randomness in floating-point arithmetic[END_REF], this mode is equivalent to an "output precision bounding" MCA in which the virtual precision is set to the actual FP precision.

• --rounding-mode=random: in this mode, roundings are upward or downard equiprobably.

With the notations defined above, we have

P[Z = z] = P[Z = z] = 1 2 .
This is equivalent to an asynchronous CESTAC arithmetic.

- ------------------------------------ ------------------------------------ ------------------------------------ ------------------------------------ Changes to this back-end have recently been introduced in order to correctly verify mixedprecision programs: type conversions subject to rounding (e.g. double → float) are now instrumented and perturbed when applicable. To the authors' knowledge, Verrou is the first tool based on stochastic arithmetic which correctly handles type conversions.

------------------------------------- -------------------------------------- Operation Instruction count Operation Instruction count '-Precision '-Precision '-Vectorization '-Vectorization -
-------------------------------------- '-llo 313498574 '-vec2 1412796 '-vec4 1764 -
Figure 1 illustrates the need for correctly handling these conversions. This listing presents the number of FP operations, as output by Verrou at the end of the execution of a code aster calculation. This calculation (performed on the ttlv300a test case) makes use of complex third-party libraries, such as the PETSc double-precision fGMRES solver preconditioned by a single-precision direct linear solver from the MUMPS library. As can be seen in the number of operations, a significant fraction of the FP operations have been performed in single precision. For example, around 19% of the subtractions, and more than 99% of the FMAs (Fused Multiplication and Addition) have used single precision in this case. Around 28% of the type conversions are potentially inexact double → float conversions.

Reduced Precision (RP)

A Reduced-Precision back-end has been introduced in newer versions of Verrou. As its name suggests, this back-end emulates the use of an IEEE-754-compliant FP arithmetic with reduced precision. It is currently implemented in only one variant, which reduces double precision FP numbers to single precision. This is similar to what is performed in the rpe library [START_REF] Dawson | rpe v5: an emulator for reduced floating-point precision in large numerical simulations[END_REF], except that Verrou performs the analysis directly at the level of the executable binary, rather than by instrumenting the sources like what is needed for rpe.

For legacy Fortran codes, which often feature a monolithic architecture, such a feature can be an easy way to foresee any problem which might arise when reducing the FP precision of variables. For more modern codes, for example developed in C++ with templated FP type, developers are often better served by simply recompiling their project with the reduced precision.

This back-end really shines when it is combined with the Verrou features allowing to restrict the scope of perturbations to only part of a program: functions and/or source code lines. In this case, Verrou allows emulating any mixed-precision variant of the program, where an arbitrary part uses single-precision arithmetic, while the rest remains in double precision. Debugging and optimization methodologies relying on such techniques are detailed in part 5.

Post-processing

Due to the changes introduced by the arithmetic emulated by Verrou, each execution of a given program (with a given dataset) produces perturbed results. An analysis of these results should be performed in order to assess the impact of FP arithmetic on the results quality.

Such post-processing is highly dependent on the nature and format of the results and, as such, it is mostly left for the user to implement for their own code. However in most instances, only a limited amount of work is needed when a Quality Assurance process with automated nonregression testing has been put in place. For example, it is possible to simply check whether the non-regression test suite succeeds when running the program within Verrou with the reducedprecision arithmetic back-end. If so, this gives a good indication that the program can be changed to use single-precision FP arithmetic while remaining "valid".

Post-processing can get more complicated in the case of stochastic arithmetic, where several sampled results are needed to compute statistics. This aspect is detailed in [START_REF] Sohier | Confidence Intervals for Stochastic Arithmetic[END_REF]. It is shown there that several statistics can be computing, yielding different accuracy estimators along with the corresponding confidence intervals. In particular, if it can be assumed (or verified in practice) that perturbed results follow a Gaussian distribution, the authors recommend computing the quantity

ŝcnh = -log 2 σ μ -δ cnh ,
where μ and σ respectively denote the sample average and variance, and δ cnh is a tabulated quantity depending on the number of samples and the targeted level of confidence. The computed quantity ŝcnh gives a lower bound for the number of significant bits (i.e bits which can be expected to be found in agreement between successive runs of the same computation).

Debugging HPC codes with Verrou

The core features of Verrou which have been presented above aim at diagnosing and quantifying FP-related losses of accuracy. However, these features can also be leveraged to set up methodologies aiming at debugging FP-related issues in industrial codes.

This debugging methodology is based on two techniques, aiming at detecting different kinds of FP-related issues.

Unstable tests are tests for which the branch taken depends on previous round-off errors).

These can be detected by combining a Verrou analysis to a code coverage analysis. The analyzed program can be recompiled so that it generates code coverage data3 . After a standard run and another run perturbed by Verrou, code coverages can be compared in order to determine where execution flows differ. the Delta-Debugging [START_REF] Zeller | Why Programs Fail[END_REF] algorithm to perform a binary search in the source code. It heavily relies on the Verrou ability to restrict the scope of FP perturbations to a part of the source code. A dedicated utility in the Verrou ecosystem, verrou dd, automates most of the Delta-Debugging process.

No specific development has been made recently regarding unstable tests detection, and this feature will not be detailed further here; the interested reader can get the details in [START_REF] Févotte | Studying the Numerical Quality of an Industrial Computing Code: A Case Study on code aster[END_REF]. However, recent improvements have been incorporated into verrou dd. They are described in more details in the following paragraphs.

Delta-Debugging algorithms

The Delta-Debugging methodology [START_REF] Zeller | Simplifying and isolating failure-inducing input[END_REF][START_REF] Zeller | Why Programs Fail[END_REF] is built in the Verrou ecosystem thanks to the verrou dd utility, which automates the full process. The Delta-Debugging process is based on two key ingredients:

• A search space, i.e. a set ∆ of program parts in which instabilities are to be searched for.

In our case, this is a list of functions (or source code lines), which is automatically built by verrou dd.

• A test function, which determines whether introducing Verrou-induced perturbations in a subset of ∆ produces instabilities. Most of this testing process is automated, but two userdefined scripts are required in order to specify how to run the analyzed program and how to validate its results. Based on these two user-defined scripts, the automatically-generated test function either validates () a given subset of ∆, or detects instabilities () in it.

As an illustration, at the beginning of the Delta-Debugging process, the test function should fulfill the two following requirements:

• test (∆) = : perturbing the whole program leads to instabilities (i.e. there are problems to look for); Initial versions of the Delta-Debugging used the DDmax algorithm (cf. alg. 1), which returns a 1-maximal stable subset ∆ max ⊂ ∆, characterized by:

test(∆ max ) = , test(∆ max ∪ {δ}) = , ∀δ ∈ ∆.
In other words, all program parts (functions or source code lines) in ∆ max can be perturbed together, without producing invalid results. But as soon as another program part is perturbed, instabilities appear. For practical use, verrou dd returns the set ∆ -∆ max .

The DDmax algorithm is traditionally described in the literature, and implemented, in terms of its DDmin counterpart (see below). However, verrou dd implements a standalone DDmax algorithm, presented in algorithm 1, which avoids some of the work and is therefore a bit more efficient.

Recent versions of Verrou include extensions to the verrou dd tool, which leverage Zeller's DDmin algorithm (cf. alg. 2). This algorithm returns a 1-minimal subset of symbol ∆ min ⊂ ∆, characterized by

test(∆ min ) = , test (∆ min -{δ}) = , ∀δ ∈ ∆ min .
In other words, instabilities appear as soon as all parts in ∆ min are perturbed together; no single part of ∆ min can be left unperturbed without making instabilities disappear. DDmin is used by verrou dd in the following recursive process (cf. alg. 3):

1. search for a minimal subset generating instabilities, 2. remove this subset from the search space, Data: ∆: search space (set of symbols or lines)

∆ current ← ∆ ; R ← {} ; while test(∆ current ) = do ∆ min ← DDmin(∆ current ); R ← R ∪ {∆ min } ; ∆ current ← ∆ current -∆ min ; end return R; Algorithm 3: Recursive DDmin (rDDmin)
3. restart the process if there are still instabilities to find. This recursive DDmin (rDDmin) algorithm has nice properties. First, rDDmin produces a set of unstable subsets. This is a richer information than DDmax algorithm, which would in principle4 return the union of all these subsets. This is illustrated in Table 1, which shows the output of verrou dd using the rDDmin algorithm for code aster on the same test-case as in section 3.2.2 and Figure 1. In this case, a total of 32 functions (or rather, symbols) are detected as unstable (in the sense of 1-minimality). A DDmax algorithm could have returned the same 32 unstable symbols, but rDDmin results are structured in a finer way. 30 of these symbols are unstable by themselves, but 2 are coupled together, forming rDDmin subset #2: these two symbols produce instabilities only if they are perturbed together.

Second, each DDmin step produces results which are immediately useful; the user can start analyzing them even before the full algorithm terminates. In the example of Table 1, around 10 hours are needed for the whole rDDmin algorithm to terminate its search fro unstable symbols. However, the first unstable symbol (KSPFGMRESBuildSoln from PETSc) is found after only 25 minutes, which allows the user to start investigating much sooner.

These results also highlight the role played by third-party libraries and the interaction between them. Unsurprisingly, a significant fraction of the unstable symbols are found in linear algebra libraries such as PETSc or OpenBLAS, for algorithms related to dot products. In these cases, additional information like Call Site Paths (CSP) would be very useful to help determine whether which parts of the program itself (code aster in this case) should be investigated. Such information is already available with tools like Veritracer [START_REF] Chatelain | VeriTracer: Context-enriched tracer for floating-point arithmetic analysis[END_REF], and are an important perspective for the development of Verrou.

Delta-Debugging in stochastic contexts

The algorithms mentioned above were designed for deterministic test functions. When the test function involves Stochastic Arithmetic, as it is the case with Verrou, successes and failures get non-deterministic. A practical way to deal with this issue consists in performing several Verrou samples within each call to the test function. As soon as one Verrou sample fails the userdefined validation criteria, the test is considered to be failing. Conversely, the test succeeds if all Verrou samples meet the user-defined criteria. By increasing the number of samples5 the test function can approach a deterministic behavior. However, this often increases prohibitively the computational cost of the Delta-Debugging.

The numerical experience reported in table 2 aims at evaluating the effectiveness and robustness of the DDmax and rDDmin algorithms with respect to the non-determinism of the test function. In this experiment, the Delta-Debugging is performed with a fake Verrou, which simulates a program comprising 10 symbols. 3 of these symbols, randomly distributed in the ∆ list, independently produce failures with probability 0.5.

Table 2 • %success: fraction of experiments where the unstable set produced contains the 3 unstable symbols and only these;

• %fail: fraction of experiments where false positive are found. Finding a strict subset of the expected result is neither considered a fail nor success, which explains why %fail + %success = 100%.

The main result reported table 2 is that rDDmin performs almost always better that DDmax for all metrics. In terms of computational cost, rDDmin needs approximately 3 times less runs of the analyzed program before it produces its first useful result. Although both algorithms perform similarly for a relatively large numbers of samples [START_REF] Frechtling | Mcalib: Measuring sensitivity to rounding error with monte carlo programming[END_REF], the results relevance is improved by rDDmin when using a reduced number of samples (2 or 5). And this metric does not take into account the structure of the partition of 1-minimal set.

Filtering

A last improvement, which is valid for both rDDmin and DDmax, consists in reducing the search space (i.e. the ∆ set in the DD terminology). In previous versions of Verrou, the search space contained all symbols during the execution of Verrou. In recent versions of Verrou, the generation of the ∆ set only considers symbol which contain instrumented FP operations.

Double/Stochastic Reference

The "stochastic reference" methodology used by PROMISE [START_REF] Graillat | Auto-tuning for floating-point precision with discrete stochastic arithmetic[END_REF] can provide the basis for a first Verrou-based way of finding optimal uses of mixed precision in a given program.

This methodology relies on the assumption that the program produces "valid" results in double precision (i.e. results which meet user-defined quality criteria). The validity of these results can either be postulated (in which case double-precision results are taken as reference without further questioning), or verified using Verrou with its Random Rounding back-end (in which case reference results could be obtained by averaging the results produced by all doubleprecision Verrou samples).

The same test case then can be run again, this time using Verrou with its Reduced Precision back-end (see §3.2.3). A comparison between reduced-precision results and the reference allows determining whether more work needs to be done. If reduced-precision results meet the user-defined quality criteria, then the whole program could benefit from using single precision variables, without affecting results quality too much. Conversely, if reduced-precision results are too different from the reference, then the Delta-Debugging techniques described above can be used to determine which parts of the program have to keep using double-precision variables. Everything else should be considered for the introduction of single-precision variables.

Full Stochastic

A second optimization methodology can be devised, which is closer in spirit to the so-called "Full Stochastic" methodology of the PROMISE tool.

In contrast to the method presented above, this methodology is based on a single-precision version of the analyzed program. As such, it is probably more adapted to newer scientific computing codes, where the possibility to (uniformly) change the FP precision is often built-in (for example via typedefs or C++ templates).

This time, a direct application of the Verrou Delta-Debugging techniques exhibits the program parts which have to be switched back to double precision (or redesigned with more accurate algorithms).

Overview of a full debugging and optimization process

Figure 2 presents a full Verrou-based workflow which can be used to analyze, debug and optimize industrial HPC codes in mixed precision. Actions printed in orange in the figure correspond to setup work, which needs to be done once in order to take into account the specifics of the analyzed program. Actions printed in red correspond to manual steps which have to be performed by the user.

The process starts with a program to analyze and the associated dataset. The user may then proceed to the standard diagnostics procedure, based on the Random Rounding Verrou backend. At this stage, the only work required consists in extracting the results of interest for the code output, and decide on relevant analysis criteria. Usually, for industrial codes with an emphasis on Quality Assurance, much of this preliminary work has already been performed, for example in order to setup non-regression testing.

At the end of the diagnostics, the user should have enough insight to determine whether code results are sufficiently accurate. If not, a first good option might be to try and find another smaller unstable test case, which will simplify and speed-up the debugging process.

Once a simple enough test case has been identified, the user may proceed to a debugging stage. Experience shows that a simple analysis and comparison of the program output between Verrou samples sometimes allows to quickly find the origin of errors. This comparison might consider not only the results themselves, but also the full log of the computation (number of iterations. . . ) and the number of FP operations counted and output by Verrou. If such a simple analysis is not fruitful, the user may use more complex debugging methodologies, relying either on code coverage comparisons in order to find unstable tests, or on Delta Debugging in order to find unstable functions or source code lines. In both cases, some work is required in order to setup these debugging processes.

After the debugging stage, the user should be able to try and implement fixes in the program (whether by changing the code itself, its parameters, or the test-case).

Once all FP-related issues have been fixed and the program produces sufficiently accurate results, the optimization stage may begin. Depending on the program being analyzed, the user may choose between the "Full Stochastic" and "Double/Stochastic Reference" methodologies described above.

With the "Full Stochastic" methodology, some work needs to be done in order to reduce the FP precision in the program (or in a part of it). The same diagnostics & debugging process can then be started over with this reduced-precision code. However, the debugging process should be easier since a natural fix consists in increasing the precision for all detected unstable code parts. At the end of this process, the developer should have an optimized mixed-precision version of the program, which meets the accuracy criteria on the given test case.

The "Double/Stochastic Reference" methodology might be more adapted to legacy code, where it is often difficult -if possible at all -to change the FP precision. Indeed, no additional work is needed for this optimization methodology. The diagnostics process just needs to be started over, this time using the Reduced Precision Verrou backend. Unstable parts of the program that are detected this way are parts of the code which have to remain in double precision. At the end of this process, developers should have enough insight to guide the development of an optimized mixed-precision version of the program.

Conclusions

Verrou is an open-source tool which aims at helping development teams during the whole Floating-Point analysis process, from diagnosing FP-related issues, to debugging them, and finally optimize the code for mixed-precision. Verrou specifically targets large, industrial, highperformance codes, and is freely available under an open-source licence at http://github.com/edf-hpc/verrou

In this paper, we presented various new features which have been recently been implemented in Verrou and released in version 2.1.0. These features extend the original capabilities of Verrou, and target more specifically the HPC needs.

The diagnostics stage of the Verrou analysis mainly relies on Random Rounding. This feature has been adapted and extended correctly handle mixed precision codes, which nowadays represent a fair share of the HPC codes.

As for the debugging stage of the analysis, the Verrou ecosystem provide a dedicated verrou dd utility which automates most of the search for unstable code parts. This utility, which relies on the Delta-Debugging family of algorithms by Zeller, has been extensively reworked. It now includes a new Recursive DDmin (rDDmin) algorithm, which is more robust and provides faster and more accurate results than the original DDmax algorithm. This is especially advantageous in the case of FP-related issues which appear infrequently and are thus hard to reproduce. When searching for unstable parts of the source code (functions/symbols or source code lines), a prefiltering feature also allows dramatically reducing the size of the search space by considering only functions (or source code lines) which actually perform FP operations.

In terms of the mixed-precision optimization of HPC codes, Verrou now includes a new backend which emulates the use of single precision in the program (or any part of it). In combination with the debugging features of Verrou, this can be used to identify which parts of the source code are sensitive to FP issues and should remain in double precision.

The interest of these features was illustrated throughout the paper by examples taken from the analysis of code aster, an industrial structural mechanics simulation tool. As an example, the Delta-Debugging process now produces its first useful results after only 25 minutes, which allows developers to start analyzing and debugging the code much sooner.

These new features make it possible to use Verrou not only as a diagnostic tool, but also as an help and guide during the development.

  --------------------------------------------------------------------

  --------------------------------

-Figure 1 :

 1 Figure 1: Number of FP operations output by Verrou for a run of code aster on test case ttlv300a

2 .

 2 Unstable functions (or source code lines) are parts of the code which, when perturbed, produce large perturbations in the results. The detection of such unstable code parts uses DDmax(∆ search , ∆ fixed , min(2 g, size(∆ search ))) ;

	Data: ∆ search : search space	(set of symbols or lines)
	Data: ∆ fixed : list of validated deltas	(default: empty)
	Data: g: granularity	(default value 2)
	{∆ 1 , . . . , ∆ g } ← split(∆ search , g) ;
	for j ∈ 1, g do
	if test(∆ fixed + ∆ j ) = then
	return DDmax(∆ search -∆ j , ∆ fixed + ∆ j , g -1);
	end
	end
	if g = size(∆ search ) then
	return ∆ fixed ;
	end
	return Algorithm 1: DDmax, as implemented in verrou dd

  • test ({}) = : perturbing no part of the program should produce an acceptable result. {∆ 1 , . . . , ∆ g } ← split(∆ search , g) ;

	Data: ∆ search : search space	(set of symbols or lines)
	Data: g: granularity	(default value 2)
	for j ∈ 1, g do
	if test(∆ j ) = then
	return DDmin(∆ j , 2);
	end
	end
	for j ∈ 1, g do
	if test(∆ search -∆ j ) = then
	return DDmin(∆ search -∆ j , g -1);
	end
	end
	if g = size(∆ search ) then
	return ∆ search ;
	end
	return DDmin(∆ search , min(2g, size(∆ search ))) ;
	Algorithm 2: Zeller's DDmin

Table 1 :

 1 DDmin subsets output by verrou dd for code aster on test case ttlv300a with 10 Verrou samples.

	rDDmin	Symbol name	Library
	subset
	1	KSPFGMRESBuildSoln	libpetsc.so.3.8.2
	2	KSPFGMRESCycle	libpetsc.so.3.8.2
		KSPGMRESClassicalGramSchmidtOrthogonalization	libpetsc.so.3.8.2
	3	MatMult SeqAIJ	libpetsc.so.3.8.2
	4	VecMAXPY Seq	libpetsc.so.3.8.2
	5	VecMDot Seq	libpetsc.so.3.8.2
	6	smumps fac asm master m MOD smumps fac a...	asterd
	7	smumps fac front aux m MOD smumps fac mq...	asterd
	8	smumps fac front aux m MOD smumps fac mq...	asterd
	9	amumpm	asterd
	10	amumpp	asterd
	11	ascopr	asterd
	12	assvec	asterd
	13	daxpy k HASWELL	libopenblasp-r0.2.12.so
	14	ddot k HASWELL	libopenblasp-r0.2.12.so
	15	dfdm3d	asterd
	16	dscal k HASWELL	libopenblasp-r0.2.12.so
	17	elraga	asterd
	18	elrfdf	asterd
	19	elrfvf	asterd
	20	op0024	asterd
	21	sgemm kernel HASWELL	libopenblasp-r0.2.12.so
	22	smumps facto send arrowheads	asterd
	23	smumps gather solution	asterd
	24	smumps ldlt asm niv12 . omp fn.0	asterd
	25	smumps solve driver	asterd
	26	smumps solve node . omp fn.7	asterd
	27	strsm kernel LN HASWELL	libopenblasp-r0.2.12.so
	28	strsm kernel LT HASWELL	libopenblasp-r0.2.12.so
	29	te0051	asterd
	30	te0054	asterd
	31	te0057	asterd
	32	te0061	asterd

Table 2 :

 2 Comparison of ddmax and rddmin algorithm for an evaluation of test function with 2,5 and 10 samples

	Algorithm	Total		First
	(#samples)	#tests #runs	#tests #runs %Success %Failure
	DDmax(2)	17.0	24.2	17.0	24.2	8.6	70.6
	rDDmin(2)	18.2	29.4	6.9	10.9	30.6	49.0
	DDmax(5)	21.1	50.2	21.1	50.2	72.0	20.0
	rDDmin(5)	18.0	49.9	6.5	17.9	86.0	10.0
	DDmax(10)	22.3	81.5	22.3	81.5	99.4	0.4
	rDDmin(10)	17.7	75.6	6.1	26.4	99.6	0.2

  reports a comparison of the DDmax and rDDmin algorithms, each tested with 2, 5 and 10 Verrou samples per evaluation of the test function. The same experiment is performed 500 times, and averaged results are presented in the table columns:• Total #tests (resp. #runs): total number of test function calls (resp. number of individual runs of the analyzed program) for the whole DD algorithm;

	• First #tests (resp. #runs): number of test function calls (resp. number of individual
	program runs) needed to obtain the first useful result;

https://github.com/interflop/interflop

http://www.netlib.org/fp/ucbtest.tgz

With gcc, this is performed for example using the -fprofile-arcs -ftest-coverage command-line switches during compilation. Later, code-coverage data can be analyzed using the gcov utility.

The set of unstable program parts is not unique; it depends on the order in which subsets are tested.

The VERROU DD NRUNS environment variable allows controlling this parameter.

Code

For code aster, with the test case ttlv300a, this pre-filtering of the ∆ set leads to a drastic reduction of the number of symbols considered by the Delta-Debugging: from an initial list comprising 4459 symbols, the filtered set is reduced to only 154 symbols. With such a small resulting search space, one could wonder whether using a Delta-Debugging algorithm is still an optimal choice.

Optimizing mixed precision with Verrou

The debugging techniques presented above can be used to optimize mixed-precision implementations of a given code. Indeed, two complementary workflows can be devised, which help developers decide where high FP precision is needed and, conversely, for which variables a low FP precision might be sufficient.


Reorienting Participation, Distance and Positionality: Ethnographic Encounters

Keywords: methodology, queer, participant observation, distance, positionality, social research

In this article I reflect on how complementing an interview-based methodological approach with participationas an instrument for knowledgemay be conducive to creating nuanced knowledges about processes of identification, belonging and marginalisation. The thoughts I elaborate on here emerge from two experiences of fieldwork covering a time period that goes from 2011 to the present day with nonheteronormative migrants in the UK and France.

Introduction

Between 2011 and 2013 I conducted ethnography in London with lesbian, gay, bisexual and trans (LGBT) asylum seekers and refugees, as well as legal representatives, NGO workers whose work revolves around LGBT asylum cases, and activist networks (i.e. support, lobbying, and campaigning groups). This study confirmed my initial skepticism about the notion of 'neutrality' framing the savant social scientist figure-this has already been largely conceptualised within the Social Sciences, although, perhaps not fully implemented into our practices as social researchers. I translated my uneasiness about what I thought was becoming a sort of fixed ethnographer posture into critical standpoints to discuss the dynamic instrumentalisations of sexuality and gender at the hands of current socio-political discourses in the UK. In doing so I put at the centre of the analysis the contested experiences of a migrant group of sexual Others that is either 'excluded from traditional studies of human behavior' [START_REF] Halberstam | Female Masculinity[END_REF] or, perhaps more worryingly today, overtly sensationalised by the media, and obfuscated by the onset of sexual/gender-based humanitarian interventions [START_REF] Mai | Between Embodied Cosmopolitanism and Sexual Humanitarianism: The Fractal Mobilities and Subjectivities of Migrants Working in the Sex Industry[END_REF] at global levels. What partly drew me to this subject was my conviction that researching sexuality and gender as tools for gaining rights within neoliberal migration regimes offered great ethnographic possibilities. But, readily I became aware that it was not the 'attractiveness' of the topic that would proffer a fertile ground for heuristic ethnographic encounters and knowledge. Rather it was more about how much I was willing to give up of myself and 'let myself be affected' (Favret-Saada, 2009[START_REF] Saada | Being Affected in HAU[END_REF] in the process, namely, to give the self in in order to willingly receive from the subjects of analysis and make some sense of the messiness of their social lives. In this article I look closely at my ethnographic experiences with non-heteronormative asylum seekers and refugees and simultaneously I refer to the ethnographic approach of Jean Favret-Saada, a French ethnographer who in the 1970s studied witchcraft amongst rural communities in the Bocage, northwest of France (1977). Her work was perceived (and still is now for many) as unorthodox back in the day. First because of the nature of the topic which was largely dismissed as not worthy of scientific attention within 'enlightened' France, but also for her vocal criticism of the methods of ethnographic observation as well as her ideas about the need of the researcher of accepting 'without prejudice, to transport oneself as in nature or in love relationships' [START_REF] Saada | En marge du dossier sur l'empathie en anthropologie (interview with Jeanne Favret-Saada by Cyril Isnart)[END_REF]:5 translation mine) whilst conducting fieldwork. Her approach is present in this text inasmuch as it is present in my own ethnographic work. Here I link her destabilising emic critique of the disciplines of social research, where she was situated, with what I see as a queering sensibility toward methodology: More generally, my work calls into question anthropology's parochial emphasis on the ideal aspects of the human experience, on the cultural production of "understanding", to employ a term derived from classical philosophy. It seems to me that there is an urgent need to rehabilitate old-fashioned "sensibility", the more so as we are now better equipped to address it than seventeenth century philosophers were (Favret-Saada, 2012: 437).

Following Favret-Saada, let us indeed rehabilitate the notion of sensibility in social research through the prism of Queer scholarship. Certainly, a focus on sensibility is not queer per se, but I would suggest that a focus on sensibility 'oriented' by Queer epistemological standpoints is productive insofar as it challenges academic habitus about what it means to produce academic knowledge. I am not interested in looking for specifically queer 'experiential modes of emotional reflexivity' (Berlant, 2011: 13). This implies that my aim here is not to fix a notion of queer sensibility, yet it is clear that my thoughts, my writing, and my ethnographic practice, in large parts, emerge from my lived experience as a queer migrant as well as my critical commitment to theories such as queer theory, antiracist, and postcolonial ethnographic studies. Social researchers are fascinated, I would add rightly so, by how sensibility is socially produced. I consider this dimension when I refer to the term but mostly I am interested in examining how one's sensibility arranges one's senses in particular ways in the ethnographic space. This leads me to think through how these arrangements impact on one's understanding of an object/subject of analysis. The etymology of the term sensibility refers to that which can be perceived by the senses (Latin: sensibilitas, from sensibilis 'that can be perceived by the senses', OED). When we are in the fieldsite our senses are alert whilst observing; there, we (researchers) claim to be conducting 'observation' but we know that it isn't just about our sense of vision, yet the word we use to indicate our method reduces it to that. Sensibility, as I use it here, involves an orientation of all the senses. This raises questions about how one perceives or what sort of arrangement of perceptions occurs whilst being there, with the research respondents. My attempt in this article is to pay attention to how one activates her/his sensibility in the fieldwork by putting under critical scrutiny some of 'our' well-established ethnographic methods-namely the rigidity inherent in their names (i.e. participant observation, distance, positionality) and how such rigidity can eclipse important parts of the researcher's sensibility.

Participating in the Observation:

The Queerness of the Method By paying systematic and zealous attention to the dynamics of everyday life, as inspired by the scholarship of Veena [START_REF] Das | Life and Words: Violence and the Descent into the Ordinary[END_REF], I decided to immerse myself in the lives of the research respondents for an extended period of time (two years). When asking asylum seekers questions pertaining to sexuality, difference and migration trajectories, I realised that elicitation methods in isolation would not have been enough to do justice, both analytically and ethically, to the composite narratives of formations of the self (as sexual Others and asylum seekers). In order to inscribe the social discourse produced by the research respondents and to do justice to the complexities of their stories I adopted, not unproblematically, the 'participant observation' approach. Barbara [START_REF] Tedlock | From participant observation to the observation of participation: The emergence of narrative ethnography in[END_REF] talks about the shift in cultural anthropology from participant observation toward the observation of participation during the 70s in the US. She says that: 'during participant observation ethnographers attempt to be both emotionally engaged participants and coolly dispassionate observers of the lives of others. In the observation of participation, ethnographers both experience and observe their own and others' coparticipation within the ethnographic encounter ' (1991:69). Since the discourses that surfaced in the 70s, the ethnographer today, even the most positivistoriented type, must necessarily locate herself in the text that she produces. It is noteworthy that how ethnography has developed over the years cannot be understood in isolation from more general political-epistemological debates about writing and the representation of otherness [START_REF] Clifford | The Predicament of Culture: Twentieth-Century Ethnography, Literature, and Art[END_REF]. But the debates over participant observation seemed to have mostly focused on the processes of self-examination on the part of the social researcher when writing. I take issue with the expression 'participant observation' prior to the ethnographer's writing phase, that is, when it is applied during the ethnographic encounters. By the time I began the study with the asylum seekers in London, I had already started to play with the idea of what 'participation'-in the all-encompassing and oft-used phrase 'participant observation'might entail. This, for me, became an obsessive thought for quite some time; every social researcher I knew was talking about it with such unquestioned confidence yet I couldn't quite figure out how to 'participate'. Participate in what and where? It became clear to me then that before researching I needed to search for the sense of participation in my study. I will provide examples of my search for participating in my fieldwork later in the text, suffice it to say here that participating was the most suitable framing to investigate the fragmentary moments of people who live at the margins of the social field. This initially proved to be difficult but eventually opened up unexpected possibilities that allowed me to form 'organic relationships' [START_REF] Bourgois | Search of Respect: Selling Crack in El Barrio[END_REF] with respondents by becoming closely involved with them. As the discipline of Anthropology has taught us for many years (Malinowski, Geertz, Clifford), participant observation requires full commitment from the researcher's point of view in spending long hours with people, in order to immerse oneself into their social worlds, and to establish close relationships with them. The trust that is (supposed to be) established through such long-term relationship may, perhaps, lead to a more nuanced understanding of what is going on. It can also allow the friendly ethnographer to ask questions that one would not be able to ask through even the most complete and wellprepared interview. But can participant observation readily become a 'confusing comfortblanket' (Favret- [START_REF] Saada | Being Affected in HAU[END_REF]) that social researchers make use of when reflecting upon their methods? Is it a non-concept, given its oxymoronic nature? Or even, is it enough to summon the orthodox forces of the quintessential Anthropological method to be content with one's immersion in the research terrain? This, in my view, needed some queering. In her conceptualisation of the queer orientation of Phenomenology, Sara Ahmed tells us that 'we are affected by what we come in contact with. It's not just about approaching or arriving at an object but it's more about how we turn to that object' (Ahmed, 2006 emphasis mine). Following Ahmed, queering acts within a given methodology would perhaps direct themselves toward queer objects. But, certainly it's not just a matter of focusing on a queer object of research; a queering methodology might turn to social research methodology by asking questions about the very 'orientation' of social research, namely its sets of well-established truth-finding mechanisms. Thus, part of one's research questions becomes how to reorient social research methods through an internal critique; for instance, asking oneself in the field what participant observation 'really' means and involves (Am I really participating? If so, in what? Or am I just observing even when I claim that I'm doing PO). If we think about our orientations when we navigate the shaky terrain of our fieldworks, then also the term 'observation' of 'participant observation' turns out to be reductive. In fact, being orientated towards our objects is something that is prior and that goes beyond our visual field. Our senses are taken by our orientations, not exclusively our vision, it is a much more embodied and 'emplaced' (Pink, 2009) experience. It involves touch, hearing, the grain of the voice, the silences and the pauses, laughter and so on. Questioning these taken-for-granted methodological certainties is crucial, beyond the fact that one calls this act queering or not, but I think that if our method of research also questions the significance of the very tools we use, then such method is queering insofar as it does something to shake the normed methods from within. It produces an emic critique (from the inside), which Queer theory has taught us to assume to produce responsible knowledge. Let us now turn our attention to the notion of 'distance' between the researcher and the researched in the ethnographic encounter.

Queering the 'distance' and letting oneself being affected

In studying LGBT asylum seekers and refugees my queerness allowed for building organic relationships with the research respondents. Thus, through my 'positionality' (I will focus on this term later in the text), I could readily connect with respondents as there often was an underlying sense of sharing experiences amongst us-or of 'having already been there', as it were. In this sense, queer signified a fruitful emic sensibility emerging in the delicate act of exploring the lived experiences of the non-heteronormative subjects who partook in the study. In my research I found that queering the distance between the researcher and the researched enabled me to create close and informal relationships with the respondents, by meeting them in different everyday life settings, sharing joy and frustration, witnessing and participating in their suffering (without appropriating it), that is, becoming close to them. This proved difficult to sustain at times, particularly when the research subjects are exposed to high level of violence or precarity. In my view, often the 'fetish of the distance' between the researcher and the researched (Kulick and Willson, 1995) is an obstacle and becomes counterproductive in the process of finding one's feet in the unknown territory that one is supposed to map out. Rigidity should not be conflated with scientific value, particularly given the complexities and opacities of fieldwork interactions. For example, in my fieldwork with the asylum seekers I was faced with a methodological issue that I could only address by breaking the rigidity of the researcher-respondent safety-distance. The issue was how to deal with 'coherence', or lack thereof, both in its textual and lived experience forms. Coherence was an epistemological trap for me just as much as it was an ontological one for the gay, lesbian and trans asylum seekers I was speaking to. As for me, I was struggling with fitting into writing all the complexities that were emerging in the fieldwork, some things were going beyond the representational. For the research respondents, coherence was an institutional trap that would define their destiny in the UK or in France, namely, recognition of rights or deportation, therefore could they tell me a different story from the one they had told the UK or the French border control institutions? Along the asylum journey they had to provide coherent and consistent accounts of their experiences of discovering their difference (i.e. 'when did you find out that you were gay?'), the triad fear-violence-trauma, as well as intimate information about their love and sexual relationships. All this then had to be backed up by some relevant evidence, and the narrations of events had to follow a 'plausible' logic. When I started to meet respondents I readily became aware that I had to somehow disrupt the rigidity of the modes of narrativising themselves to which they were getting used by being exposed (very frequently) to the border control institutions. I was set out to queer the space and the affective mode through which these narratives were delivered to me in the role of ethnographer. Let me illustrate this though a note taken from my fieldwork notebook during my ethnography in Marseille: I've just met Karim for an interview. He is one of my first respondents in Marseille; a Tunisian gay refugee living here for 4 years. We go to a café near Gare St. Charles, we sit at a table inside the café; it's only early October and it's unusually cold for that time of the year in Marseille. Karim looks smart, very polite and willing to talk. It doesn't take too long to go deep into the complexities of his story-from Tunis to Morocco, to then Italy to finally end up in Marseille. He knows his story very well, I'm sure of that from the very moment he starts talking to me. He has repeated this story many times to the social workers, the OFPRA vi , the judges and his extended support network. With Karim I allowed myself to be dispossessed by undoing what he could expect of my role as an interviewer. Through humor, a look, a gesture, we willfully lingered in the opacity of that ethnographic moment where we could queer the ways in which the fixed stories asylum seekers had often been asked to tell could be relived differently. Queering was here a sensibility, which was expressed and shared within research encounters, which did something; it produced specific affects between the researcher and the respondent, thus allowing us to talk and be multiple, contradictory, and even willing to maintain opacity.

Thus far I have stressed that reconfiguring language and destabilising the academic research habitus (rigid languages and sensibilities) are two important conditions for a sound methodology-queering it. Yet, I am aware that ethnographic methods predicated upon queer scholarship are also implicated in the reproduction of hierarchies when speaking of the Others. My intention here is not to offer queering as a panacea to resolve the complex predicaments of ethnographic methodologies and epistemologies. However, it is my conviction that this analytical grammar can be productively mobilised to do autocritique as social researchers (on the methodological level) as well as to underscore the contradictory ways through which respondents' agency is enacted within the structures construed in our social worlds (on the epistemological level).

Finding one's feet OR the ethnographer's 'positionality' in the fieldsite

My intention is not to saturate the article with detailed accounts of the different places that I consciously occupied, or that I was seen to occupy, as a researcher during the fieldwork-this would be a tedious imposition on the reader, that is; the risk of an autobiographical account gone wrong is always around the corner. But I will partly take up the risk as I find it important to locate myself within the text that emerged from the study, focusing on how my situatedness as a researcher impacted upon the interactions with the subjects of the study. First, I want to express my perhaps banal, but relevant sentiment that during the study with LGBT asylum seekers, I found that being a queer migrant, whose native language was not English, turned out to be an advantage in terms of gaining what one could call 'trust' from research respondents. Often respondents would feel understood given the modes in which my identity was articulated within the space of the research settings. Further, growing up in a socio-cultural context where

LGBT embodiments are stigmatised in the public sphere I could quickly recognise the tensions in the narratives presented by respondents pertaining to formations of selfhood, without Othering their understandings and enactments of difference. Thus my 'positionality' would have been of the non-heteronormative, educated male, migrant from a lower middle-class background (perhaps one should add, and so on and so forth) who was intending to enter the lives of non-heteronormative asylum seekers and refugees. Since I started using the word 'positionality' when talking about the various places I was occupying in the study and the sensibility that accompanied my orientations in the fieldsite, the term 'positionality' started to feel very heavy-it seemed that it did not succeed in reflecting what I was trying to convey to my interlocutors. Today, as social researchers we all learn that positionality refers to the fact that a researcher does not 'operate' from a socio-cultural vacuum but that she or he is producing knowledge from a specific perspective, being immersed in a particular value-system.

Positionality does not hide the fact that I am speaking from a political-ethical position, long gone are the days of neutral 'objectivity'. Then we welcome the term positionality in the Social Sciences, long live positionality. But often, it seems to me that how the term is used (by us, social researchers) makes it devoid of its ethico-political significance and becomes just a word that indicates my fixity in the fieldsite. Positionality thus is primarily understood as the arrival at the fieldsite and the positioning of oneself in a specific way. This can inhibit dynamism, namely, the act of following different orientations or letting the self being oriented; somehow it indicates that which is static. Although positionality differs from the term 'position', which makes it sound as if one should be even more anchored to the ground, it still doesn't get rid of the roots to the posture of the savant ethnographer. Had it not been subsumed by the marketing sector, I would suggest using the word 'positioning'. Although one doesn't have to feel trapped by terminology, it is important to queer tradition-received words because their unquestioned use might readily trigger then reduce their meanings to their most face-value significations. It is important to be mindful of the inherent meaning of 'fixity', stillness (the quality of being stationary) present in the term and remember to always attach its ethico-political value to positionality. Let's let our positionalities in the fieldsite allow us to 'occupy', or be seen to occupy by respondents, different places when engaging in the fieldwork.

My study with non-heteronormative asylum seekers in the UK consisted of three main phases, each one marked by specific developments: i) establishing contacts with gatekeepers to gain access to the migrant population I aimed to interview and study, ii) working as an LGBTI asylum support worker, and iii) direct involvement in theatre workshops and performances by a group of lesbian, gay and bisexual asylum seekers and refugees. These phases did not follow a strict chronological order, in the sense that the interviewing did not stop when the 'participant observation' started, most of these activities occurred simultaneously. My participatory strategies within the group of asylum seekers and refugees I studied did not happen within the context of my work as an asylum support worker, but it mostly occurred during social activities outside of the workspace. I found that orienting myself toward multiple roles in the research fieldsite provided me with a wealth of information that I couldn't have accessed otherwise-had I been exclusively seen as the ethnographer. Oftentimes it also put me in contradictory situations. When interviewing asylum seekers I wanted to question the pre-fabricated story of the self including their abuses and their necessity of leaving their countries that they were compelled to produce when claiming protection. However, when I was the asylum support worker my job indicated that I was the person that needed to hear a certain story then people were often telling me: 'I'm telling you what you want to hear'.

Working as an LGBT asylum support worker and my 'participatory being-there'

Participating as a tool of knowledge (Favret-Sada, 2012) allowed me to see the specificities of the recurrent problems faced by the migrants I studied. For instance, the fact that one has to continue to live a double-life and not reveal her difference to their conational social networks. Generally, respondents who lived amongst the communities of origin had a fear of one's sexuality being exposed. Further, sexual and gender minority subjects were often deprived of kinship support, or such support was very frail and at times counterproductive, and their sense of isolation was acutely emphasised when having to face institutions such as the UK Border Agency.

In the summer of 2012 I started to work within a charitable organisation supporting

LGBTI asylum as a part-time support worker. I took up the opportunity, for the job involved talking to people who wanted to apply for asylum on the basis of their sexuality and gender identity. I had to provide practical support to those going through the determination process; to help them write the stories that had to be presented during interviews at the Home Office and in court, and to refer claimants to legal aid solicitors that worked in partnership with the organisation. This job at the charity contributed considerably to my understanding of the variety of stories and situations of people in vulnerable conditions as well as to my knowledge of how the system of institutions implicated in the asylum determination process operates in the UK. This information necessarily fed into my descriptions of the complexities of the structural injustice that marks the lives of many research respondents.

Here, I could see some contradictions emerging. As a support worker I was telling asylum claimants to fit their stories with demands of the Border Agency to be successful, at the same time I could see that this was reinforcing the standard narrative of humanitarian support for the sexual and gender minority person coming from those 'homophobic cultures', namely, one simplistic story that obfuscates the reality and therefore the complexities of people's migratory trajectories. This, however, was not just my concern as a split subject as the support worker/researcher at the charity, in fact other colleagues of mine were conscious that their jobs were built on an emergency model, that is, they had to get people out of detention as fast as possible or that they had to help them write their witness statements in a way that could be easily intelligible by the judges, which often meant to advise them to craft their genuineness (Giametta, forthcoming).

Being in daily contact with support workers became as important as being in contact with the asylum seekers and refugees, in order to better understand how humanitarian organisations operate, particularly in the field of sexual rights. My being-there, participating in the work, sharing my colleagues' worries, agreeing and disagreeing with them, witnessing the everyday struggles and difficulties of both support workers and the supported was a very enriching experience. But I also started to fear that it could narrow the analytical scope of my research. Firstly, as an asylum support worker I felt limited by the responsibilities of the role in interacting with people outside of institutional spaces. Guaranteeing confidentiality and establishing boundaries are imperative in the research process, but being directly involved in the institutional structure rendered those boundaries more defined. In fact, being an employee within the support organisation made me feel even more responsible towards respondents as I could relate to them from different standpoints. I could better situate the specificities of their material problems when confronted with the institutions.

In my day-to-day job I noted how people felt compelled to shape their identities and behaviours in order to be recognised as genuine sexual dissidents in need of rescuing. I realised that in a pervasive atmosphere of distrust and disbelief characterising the approach of many UKBA caseworkers and adjudicators, it is common to find that people feel the pressure to exaggerate their states of vulnerability, their experience of trauma and their suffering in order to fully fit in the 'victim category'. Initially, the pressure of being granted asylum puts people in a state of anxiety. When people came to the organisation the first piece of advice they were given was that in telling their story it is crucial to always ask oneself the question: 'How did I feel when such and such happened to me?'

This helps to stress the uniqueness of one's inner feelings, and it shifts the focus from the facts, which are often similar across narratives, to the claimant's subjective feelings. It was within this context that I became aware of how the immigration apparatus, including migrant support organisations, highly values the performative dimension in the telling of a migratory journey. This requirement of working on the inner self to find a 'truth', at times, resembled classical methods in theatre training (see below) to help the actor present a realistic life or event. Whenever witnessing these processes I was reminded of the notion of 'authenticity' and of how problematic a concept is-it is how one tells one's story that provides access to services and not what one's past experiences have been.

These, in fact, do not always find validation from the official powers. In this context, the asylum claimant needs to become an 'expert' in communicating her or his vulnerable inner world to the inquisitive ears of the law and institutions.

'

We're all in the same boat': participating in theatre workshops and performances '...They wanted me to enter into it as a partner, to stake the contours of my then existence in the process. Initially, I oscillated between these two pitfalls: if I "participated", fieldwork would become a personal activity, that is to say the opposite of work, but if I attempted to "observe", meaning keeping myself at a distance, I'd have nothing to "observe". In the first case, my scientific project was threatened, but in the second, it was ruined' (Favret-Saada, 2012: 440).

Alongside my experience as an asylum support worker it is important to talk about the place I gradually took as a participant of a theatre group, named International Rainbow, comprising of sexual minority asylum seekers and refugees. More precisely, I would like to discuss the methodological and epistemological effects that emerged from being part of this group, namely the 'how' I got there as well as 'what kind' of data such experience produced. This theatre group materialised from the recent work conducted by the charity organisation where I was a volunteer, prior to my part-time employment within it. Since its inception International Rainbow was a lively amateur community theatre group, which contributed highly to the social capital of the asylum seekers and refugees involved. I was a participant myself and as such I took part in the rehearsals and some of the public performances. Alongside the two volunteer directors who ran the workshops and rehearsals, I was the only non-refugee group member. Initially the other members saw me as 'Calo the charity worker', but after some time the label started to become more blurry and 'Calo' started to be seen more and more as a 'full-time' Rainbow member, this became apparent to me when two participants verbalised it by telling me: 'so now you're here full-time, aren't you? It's good getting to know you here'.

As part of the research methods this experience enabled me to imaginatively engage with respondents' social worlds, as Les Back suggests (2007). It also helped me distance myself from the mould of positivist epistemology as the only legitimate research paradigm [START_REF] Kaptani | Participatory Theatre as a Research Methodology: Identity, Performance and Social Action Among Refugees[END_REF]. This approach was in line with my epistemological intent of queering the study and it allowed for alternative ways of analysing and retelling narratives of identification and belonging. From a more specific methodological point of view, the experience of the theatre was crucial because it enabled me to discuss aspects of the research to which I had not managed to gain access. For instance, the theatre space became my entry point to information concerning respondents' experiences of confinement in detention centres. As a matter of fact, some of the theatre group attendees had been in detention for extended time periods and they were willing to share their stories both within the group and with me personally. This was very important as I had scarce data about the living conditions within UK detention centres. Hence, the theatre group unexpectedly added a considerable layer of complexity to the overall analysis. The non-threatening nature of the theatre space allowed people to feel safe and it became perceived to be a place where one could open up whilst creating stories and characters to perform. This always happened in a climate of creativity and mutual respect.

At this stage of the research, the main purpose was to immerse myself in the theatre space to get to know people's stories differently and understand them from a less formal perspective, namely from a point of view which exceeded both my roles of researcher and support worker. Through the common condition of unknowing and learning together -I was very new to the art of actingany subtle form of hierarchical structure that had started to form previously between me, the researcher, and the researched was challenged. I quickly managed to establish an amicable relationship with the theatre group participants. Having conducted ethnography in different settings and with different groups of asylum seekers and refugees, I now wanted to further trouble my ethnographer's role and reconfigure my relationship with the respondents. Therefore, as a researcher in the fieldsite I wanted to: i) adopt a different point of view (being an amateur actor as the rest of the group in an unknown context); ii) set up more friendly relationships with respondents (learning playfully together); and iii) trigger a different type of information (anecdotal knowledge and more personal accounts). Generally, participants, in their current political position of asylum seeker or refugee, had told and continued to tell their stories repeatedly in formal contexts, their stories being repeated as many times as to create the illusion of acquiring a life of their own. The theatre work they did challenged the notion of authenticity, with which they constantly struggle in the realm of legal recognition. In fact, participants are required to be 'authentic' when telling their experiences to decision makers and when showing them those intimate feelings that had to be suppressed to avoid public attention and stay safe.

Through the theatre practice they had the opportunity to act, namely they could finally tell their stories as well as other stories without feeling the oppressive imperative of being 'authentic' or showing authenticity. On the contrary authenticity had to be eschewedthis proved to be visibly liberating for participants.

Conclusion

I attempted to reconstruct the various stages of my experience as an ethnographer doing fieldwork. In so doing I elaborated on the multiple queerness of my methods when making sense of the fragmented knowledge I gathered in studying a group of nonheteronormative migrants' lives. I stressed the importance of how queering ethnographic interventions should involve a sensibility that goes beyond 'observation' seeking to truly be 'participatory', as well as critique rigid research terminology and practices that reinforce the social research canon. I insisted on the unexpected and unpredictable nature of data sources in the ethnographic process. For instance, I highlighted that through my access to the theatre group I unexpectedly managed to collate information pertaining to ex-detainees' lived experiences. The queering methodological approach had implications on the knowledge produced with my respondents. As a matter of fact, it had an epistemological impact on the overall study. I could not expand much on this point in the pages of this article, but suffice it to say that applying the verb queering in my research methods with LGBT asylum seekers and refugees generated knowledge that: i) destabilise preconceived notions based on conceptual binarisms around sexual minority refugees' lives, ii) problematise the heteroand homo-normative essentialism of legal reasoning and the Home Office approach when sexuality (in the context of migration) is analysed, and iii) challenge the ever-narrowing humanitarian narratives that are available to the research respondents (Giametta, forthcoming). Destabilising the rigidity of academic research practices does not automatically lead to a lack of responsibility toward research respondents. On the contrary, I believe that emic critique in Social Sciences practices is conducive to making one's study important to the respondents. By this I mean that the research methods should emerge from, and produce, a theory that is relevant to the insiders (emic theory, as Boellstorff suggests, 2010) through empirical work during the researcher's participatory phases. This approach allows for a destabilising of the safe and unquestioned 'academic habitus' [START_REF] Bourgois | Ethnography's Troubles and the Reproduction of Academic Habitus[END_REF] that enormously distances the knowledge producedthrough the research encounterfrom the subjects being researched. I attempted to use queering as an active verb to question from within the concepts and methods that partly feed our sensibilities as social researchers.

  He tells me things as if in a rush and I can't stop him. I'm getting a bit frustrated, but almost halfway through the interview, he starts telling me about his 'manières', his gestures, and how he always had to tone them down not to 'become effeminate'-as he says. As he refers to that, and then goes on giving me more details of his relationship with his dad and uncle, his body becomes stiffer, he puts his hands under his thighs and keeps sitting on them, lifting his upper body slightly away from the chair and leaning more towards the table. He keeps his body weight on both hands as if to stop them from giving away his 'difference' or lack of normative virility. Now, what does that do to me? In this very moment I seem to re-orient myself towards him in a way that makes me listen to him and be with him there as if we suddenly were in a different place. I look at him more intensely, I recognise that particular act, the act of controlling my gestures; perhaps I have done that in the past myself, perhaps I have used this strategy too for similar reasons. I look at him again and I say, softly, and with a bit of hesitation in my voice (I can hear that in my audio-recording):

Me: Well...Look how much I move mine! Karim: What do you mean? What do you move? Me: My hands! I mean Karim smiles, I smile also, and we both burst into laughter. There was a look. Now we start talking, differently.