Elina Thibeau-Sutre

Mauricio Diaz

Ravi Hassanaly

Alexandre M Routier

Didier Dormont

Olivier Colliot

Ninon Burgos

ClinicaDL: an open-source deep learning software for reproducible neuroimaging processing

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Deep learning has become one of the most used data analysis technique for medical image analysis. Unfortunately, this recent massive use of deep learning has also been associated with methodological flaws in many studies which results are contaminated by data leakage.

Moreover, the whole deep learning community faces a reproducibility crisis that discredits its results. Hence there is an urgent need in publishing open-source software, data sets and scripts that allow reproducing the methodologies described in deep learning studies.

Finally, deep learning users who are not neuroimaging specialists have difficulty in accessing properly formatted and pre- processed data sets. This issue has been partly tackled by a data set format established by the community: the Brain Imaging Data Structure (BIDS).

Rigorous validation Reproducibility Use of neuroimaging

• Software overview

• MAPS

The software ecosystem

1

  ClinicaDL: an open-source deep learning software for reproducible neuroimaging processing Elina Thibeau-Sutre 1 , Mauricio Diaz 1 , Ravi Hassanaly 1 , Alexandre Routier 1 , Didier Dormont 1,2 , Olivier Colliot 1 , Ninon Burgos 1 Sorbonne Université, Institut du Cerveau, Inserm, CNRS, AP-HP Pitié-Salpêtrière, Inria Équipe-projet ARAMIS, Paris, France 2 AP-HP Pitié-Salpêtrière, Département de Neuroradiologie,

Fix random seed

ClinicaDL is an open-source software for deep learning processing on neuroimaging data. With this software, we solve the three main issues encountered by deep learning users who are not specialist of the neuroimaging domain:

(1) the data management and preprocessing of neuroimaging data sets, (2) the contamination of results by data leakage, (3) the lack of reproducibility of deep learning experiments. Moreover, thanks to abstract templates, a great flexibility is given to the users.


Tristan Benoit
email: tristan.benoit@loria.fr

Jean-Yves Marion
email: jean-yves.marion@loria.fr

Sébastien Bardin
email: sebastien.bardin@cea.fr

Binary level toolchain provenance identification with graph neural networks

Keywords:

We consider the problem of recovering the compiling chain used to generate a given stripped binary code. We present a Graph Neural Network framework at the binary level to solve this problem, with the idea to take into account the shallow semantics provided by the binary code's structured control flow graph (CFG). We introduce a Graph Neural Network, called Site Neural Network (SNN), dedicated to this problem. To attain scalability at the binary level, feature extraction is simplified by forgetting almost everything in a CFG except transfer control instructions and performing a parametric graph reduction. Our experiments show that our method recovers the compiler family with a very high F1-Score of 0.9950 while the optimization level is recovered with a moderately high F1-Score of 0.7517. On the compiler version prediction task, the F1-Score is about 0.8167 excluding the clang family. A comparison with a previous work demonstrates the accuracy and performance of this framework.

I. INTRODUCTION

The problem Identifying the toolchain provenance, i.e. the compiler family (e.g. Visual Studio or GCC), the compiler version (e.g. 10.0, 12.0) and its optimization options (e.g. -O1, -O2), that have been used to produce a given stripped binary code is an important problem in at least two scenarios:

• Determination of security flaws inside binary codes. Applications are often built by linking together commercial off-the-shelf libraries (COTS) 1 . While allowing faster development cycles, developers do not have the source code of these COTS and do not know the compiling chain used to generate them. This is an important issue in software maintenance and long-term support as compilers may inject vulnerabilities that are discovered after the COTS released and after the deployment of the applications that used them [START_REF] Hohnka | Evaluation of compiler-induced vulnerabilities[END_REF]. For example, CVE-2018-12886 describes a vulnerability allowing an attacker to bypass stack protection in GCC 4.1 though 8. Hence, there is a need to be able to retrieve the compiling chain to assess whether an application may present a certain vulnerability; • Identification of known functions. Library function identification in a binary code is another primary issue for software maintenance and security, such as clone detection (see for example [START_REF] White | Deep learning code fragments for code clone detection[END_REF] using Deep Learning) or malware reverse engineering (see for example [START_REF] Calvet | Aligot: Cryptographic function identification in obfuscated binary programs[END_REF] using I/O relationship). The function name identification problem is readily solved when the binary code under analysis is well-behaved, that is when it contains enough information to disassemble it. For example, IDA disassembler proposes the F.L.I.R.T algorithm [START_REF] Guilfanov | Ida fast library identification and recognition technology (flirt technology): In-depth[END_REF] based on signature-patterns that recognizes known functions, while ByteWeight [START_REF] Bao | BYTEWEIGHT: Learning to recognize functions in binary code[END_REF] constructs a weight prefix tree of the function prologue by machine learning to identify known functions. These methods work well for regular binary codes, yet identification fails in many situations: unknown libraries, slightly modified binaries, stripped or obfuscated binaries. Compiler chain provenance identification may help in such cases [START_REF] Shirani | Binshape: Scalable and robust binary library function identification using function shape[END_REF].

We tested IDA Freeware edition to determine the compiler used to generate a binary code. For this, we performed a test manually on 15 unstripped and stripped binary codes. A stripped binary is a binary code without debugging symbols. On unstripped binary codes, IDA correctly distinguishes MinGW binary codes from Visual Studio ones. In the case of stripped binary codes, IDA is not able to find the correct compiler and assigns each binary code to Visual Studio, probably as a default setting. Moreover and in all cases, IDA was unable to retrieve the compiling options.

Goal and approach The goal of the paper is to devise a machine learning based solution to the toolchain provenance identification problem over stripped binary codes.

Rosenblum et al.'s pioneering work [START_REF] Rosenblum | Recovering the toolchain provenance of binary code[END_REF] introduced the problem and a first solution based on Support Vector Machine (SVM) over binary functions disassembled from a binary. Recent works on this topic [START_REF] Yang | Understand code style: Efficient cnn-based compiler optimization recognition system[END_REF], [START_REF] Chen | Himalia: Recovering compiler optimization levels from binaries by deep learning[END_REF] rely on neural networks, either Convolutional (CNN) or Recurrent (RNN). The extracted features of a binary code are embedded as a text or as an image to be fed into a neural network. Rahimian et al. [START_REF] Rahimian | Bincomp: A stratified approach to compiler provenance attribution[END_REF] takes a stratified approach: first, the compiler family is guessed and then, given the compiler family, the optimization level is determined. With a combination of features coming from the control graph and the instruction sequences, the toolchain provenance is attributed with hash kernels and fingerprints.

A recent work by Massarelli et al. [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF] extracts binary function Control Flow Graph (CFG). Their neural network can be decomposed into two parts: first, the raw binary sequences in each basic block are dealt with natural language processing (NLP) techniques, second, the graph is operated using convolutions -aggregating blocks to classify each binary function.

Instruction embedding is a topic in itself. Different embeddings have been adapted from the natural language processing setting such as asm2vec [START_REF] Viet Phan | Convolutional neural networks over control flow graphs for software defect prediction[END_REF], cross-architecture instruction embedding [START_REF] Redmond | A cross-architecture instruction embedding model for natural language processing-inspired binary code analysis[END_REF], or i2v [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF].

Claim We claim that the decision process can be done at the binary program level rather than at the binary function level like in most prior approaches. Going away from precise block level binary semantics, we suggest using a forgetting control flow graph (CFG) of the whole binary program. That is why we propose using Graph Neural Networks (GNN) [START_REF] Zhou | Graph neural networks: A review of methods and applications[END_REF]. As a result, relationships between basic blocks (nodes of CFG) are taken into account in the graph embedding, and the weight of a CFG node depends transitively on its neighbors. In doing so, we take the proviso that a program is not like a text or an image that can be projected into a regular Euclidean space, and consequently, it is worth keeping the program structure, at least partially.

Contribution Our paper makes the following contributions:

1) We develop a Graph Neural Network based framework that we call Site Neural Network (SNN) to determine the compiler family, compiler version and the optimization level that generate a given binary code. The overall architecture is displayed in Figure 1 and Figure 2. We extract a CFG from a binary code and then abstract it by preserving only the skeletal control flow. Then, this forgetting flow graph is chopped into fixed-size sub-graphs (sites) whose size is a parameter noted α in the remainder of the text. This preprocessing step is fully automatic and unsupervised. Next, the Site Neural Network takes the graph of all sites as input to classify the binary code. The overall architecture of our framework follows the approach of adapting Residual Neural Network (ResNet) to sites [START_REF] Zhao | When work matters: Transforming classical network structures to graph cnn[END_REF] and by refining the model with adaptive max pooling layers [START_REF] Shervashidze | Weisfeiler-lehman graph kernels[END_REF] to graphs. We also propose to use hierarchies of multiple SNN experts making binary decisions (e.g. 'Clang 8.0' versus 'Clang 8.5').

Our approach has at least three advantages.

• Compared to prior works [START_REF] Rosenblum | Recovering the toolchain provenance of binary code[END_REF]- [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF], the model is quite simple because it is reduced to sites. There is no instruction specific feature and no focus on function prologues/epilogues. So, we do expect that Site Neural Networks are more robust and generic. From our point of view, this approach creates a bias, which might be acceptable depending on the context. Indeed, it might be difficult to correctly identify function boundaries when we deal with obfuscated COTS or stripped binaries and the preprocessing time to find functions, for example in COTS, may be important. Moreover, most of the time, libraries, like DLL, are dynamically linked and so it is not necessary to determine the compiler toolchain for each function individually.

That is why we focus on binary codes. Hence, our dataset consists of full stripped binary codes without any knowledge about functions and their localization. 3) Some study suffers from a limited dataset variety (e.g., only 18 compiler configurations in Rosenblum et al. [START_REF] Rosenblum | Extracting compiler provenance from program binaries[END_REF]). Our study covers a large number of possible compiling toolchains on Linux and Windows (both 64bit systems). Indeed, the identification covers 23 different compiler versions of four major compilers: Clang, GCC, MinGW, Visual Studio. We take into consideration four classes of optimization -O0, -O1, -O2/-O3, and -Os -for a total of 92 compiler configurations. We evaluated our system in terms of detection accuracy on a broad dataset composed of about 36, 272 stripped binary codes compiled from 36, 272 different source code with four compiler families, 23 different compiler versions, and five optimization levels. Thus, we can train and test our approach with 121 distinct and well-balanced sets of binary codes, all having different code sources. We demonstrate that toolchain provenance identification at program level is feasible with good precision and accuracy together with an efficient learning phase -obtaining better classification results than the most recent prior functionlevel methods [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF] together with much smaller learning time (Section VI-A). Overall, we believe these results are promising and may offer new, more robust leads for toolchain provenance identification. II. RELATED WORKS Rosenblum et al. in a series of two seminal papers [START_REF] Rosenblum | Extracting compiler provenance from program binaries[END_REF] and [START_REF] Rosenblum | Recovering the toolchain provenance of binary code[END_REF] were the first to attempt to recover the compiler and compiler options using SVM -where features are composed of regular expressions (idioms) on the assembly program together with 3-vertex graphlets. While they report excellent precision O3). We show in Section VI-B that more diversity significantly impacts results, hence we carry out our experiments in a more diverse setting.

Rahimian et al. [START_REF] Rahimian | Bincomp: A stratified approach to compiler provenance attribution[END_REF] developed BinComp, with a complex model based on three layers -the last one being an Annotated Control Flow Graphs. All these features are embedded into a vector by applying a neighbor hash graph kernel. Some features required the binary codes to be unstripped.

More recently, three papers were published on toolchain provenance. Yang et al. [START_REF] Yang | Understand code style: Efficient cnn-based compiler optimization recognition system[END_REF] extracts 1024 bits from the object file and process them with a one-dimensional CNN. Chen et al. [START_REF] Chen | Himalia: Recovering compiler optimization levels from binaries by deep learning[END_REF] develop Himalia, which is a two-tier classifier. Features are extracted from binary functions and consist of a sequence of instruction types of fixed size, which are eventually completed by padding. Thus, Himalia focuses on the prologue and epilogue of functions, and as a result, can explain its classification. That said, the authors made the strong hypothesis to be able to determine the function prologue and epilogue.

Lastly, the closest related work is by Massarelli et al. [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF]. They propose a graph embedding neural network based on methods developed in the field of NLP. The learning phase is composed of two stages. The first stage transforms sequences of instruction in basic blocks using an instruction embedding called i2v and a Recurrent Neural Network. Then, the overall CFG is embedded into a graph, which is aggregated by a 2round convolution process. Our approach is shown to have a much better learning time for overall better precision and accuracy (Section VI-A).

III. BACKGROUND

In this paper, we use Graph Neural Networks (GNN) [START_REF] Zhou | Graph neural networks: A review of methods and applications[END_REF], [START_REF] Scarselli | The graph neural network model[END_REF], [START_REF] Micheli | Neural network for graphs: A contextual constructive approach[END_REF] and more precisely Graph Convolutionnal Networks (GCN). The architecture of a GCN is mostly based on a transformation of classical CNN architecture. For example, Zhao et al. [START_REF] Zhao | When work matters: Transforming classical network structures to graph cnn[END_REF] generalizes ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] and DenseNet [START_REF] Huang | Densely connected convolutional networks[END_REF] to graphs. Inputs of a graph neural network is the representation of a graph. The recent survey by Hamilton et al. [START_REF] Hamilton | Representation learning on graphs: Methods and applications[END_REF] discusses the different graph representations. Compared to unstructured data like texts (one-dimensional data) or images (two-dimensional data), the graph encoding must preserve certain properties like the shape or the connectivity.

A key feature is the pooling method. A pooling layer of a GNN does not depend on the input size. For this, the pooling can just take the sum of node values. There are other methods such as sort pooling selecting a fixed sized set of maximum node values [START_REF] Zhang | An end-to-end deep learning architecture for graph classification[END_REF] or such as adaptive max pooling by dividing the matrix at each convolution in a fixed number of parts [START_REF] Yan | Classifying malware represented as control flow graphs using deep graph convolutional neural network[END_REF], as illustrated in Figure 4.

Note that we name each node with an identifier to increase the predictivity of the model [START_REF] Seidel | Anonymous Distributed Computing: Computability, Randomization, and Checkability[END_REF]. The architecture of the preprocessing is shown in Figure 1. Inputs are binary codes. The Control Flow Graph (CFG) is extracted using a concolic disassembler framework [START_REF] Bonfante | CoDisasm: Medium Scale Concatic Disassembly of Self-Modifying Binaries with Overlapping Instructions[END_REF]. Then, there are two more steps that we call the forgetful phase and the chopping phase. They reduce drastically the dimension of the adjacency matrix and facilitate transmission during the convolution phase.

IV. OUR METHOD FOR TOOLCHAIN PROVENANCE IDENTIFICATION

A. Binary code preprocessing

B. The forgetful phase

The forgetful phase consists of simplifying a CFG by removing sequential instructions and by just keeping control flow instruction types. Figure 3 illustrates this reduction. For this, the phase runs in two stages :

1) All consecutive nodes labeled by a sequential instruction (like mov or add) are pruned in one single node that is removed. 2) All remaining nodes are relabeled based on the instruction type following Table I. The forgetful phase respects the underlying structure of the input CFG and maps it to a reduced graph that we call the forgetting CFG for convenient notation in the remainder.

C. The chopping phase

The chopping phase cuts a forgetting graph into a set of small disconnected subgraphs. These subgraphs are called sites and their size is at most of α nodes. Sites are obtained using a breadth-first search algorithm from the forgetting graph -the algorithm is presented in Algorithm 1. It performs a limited exploration of the graph using a breadth-first search. During that exploration, it collects from the graph multiple small subgraphs.

We associate with each node of a site two features composed of the instruction type (see Table I) and a unique identifier to solve the problem of anonymity.

Since each site has a small diameter of at most α, a small number of convolutions allows information to pass through all nodes. Notice that sites are directed graphs, but they are processed as undirected graphs during convolution computations.

D. Site Neural Networks

Inputs are graphs built from a binary code by the two previous phases. Take a graph composed of a set of nodes V and represented by the adjacency matrix A. We define X 0 as the matrix containing the nodes attributes, thus it has a dimension of n × 2.

a) Mini-batches: A single input is a set of sites that are collectively regrouped in a graph. In the training phase, the input graphs are partitioned into mini-batches. This allows us to normalize the data [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF]. Each mini-batch B is normalized by calculating batchNorm B (x) = x-µ B σ B , where µ B is the observed mean and σ B is the observed variance. Notice that, the observed means and variances are memorized because they are reused in test time. This process has been shown to be successful but is not yet understood in theory. The activation function is the rectified linear unit relu(x) = max(0, x).

such that (x, y) ∈ E do if |N | > α then break end if if y ∈ N then A ← A ∪ {(x, y)} break end if N ← N ∪ {y} A ← A ∪ {(x, y)} push y in q2 end for end for q1 ← q2 end while V ← V \ {N } E ← E \ {(u, v)|u, v ∈ N } r ← first vertex left in V V r ← V r ∪ N E r ← E r ∪ A n ← n -1 end while return G r
b) Dense Convolution: Now, the vector sequence of node values (Y k+1 ) k≥0 obtained after k+1 convolution(s) is defined as follows:

Y 1 = relu(batchNorm B ((A + I)X 0 W 0 + b 0 )) Y k+1 = (relu(batchNorm B ((A + I)Y k W k + b k ))|Y k )
The notation | is the matrix augmentation. Using dense convolutions introduced by Huang et al. [START_REF] Huang | Densely connected convolutional networks[END_REF] the output of one step is fed into every future step. c) Dimensions: Let d t be the hyperparameter corresponding to the second dimension of the matrix W t at convo-

k , for k > 0, is k-1 t=0 d t .
d) Output: The dimension of matrix Y k , for k > 0, is n× k t=0 d t where n is the number of nodes. We now perform a pooling, which reduces the matrix of the last convolution to some smaller fixed-size matrix.

e) Adaptive Max Pooling Layer(s): Following [START_REF] Shervashidze | Weisfeiler-lehman graph kernels[END_REF], a crucial point in our approach is the extraction of features based on the Weisfeiler-Lehman test of graph isomorphism. For this, we apply on the result of the convolutional layers an (AMP) step. This pooling operation is defined by an operator amp n,m that reduces a matrix to a matrix of smaller dimension n × m as follows: Take a matrix M of dimension u × v. M is cut into n × m matrices of kernel size u

x × v y . We take the maximum in each block. The figure 4 illustrates the adaptive max pooling computation. We iterate four times adaptive max pooling to extract a fixed-size representation of X.

f) Readout layer: At this point, we have obtained from the adaptive max pooling layers a fixed-size representation of our graph. The output of the adaptive max pooling layers is fed into a multilayer perceptron to predict the probability distribution of the class that the input graph should belong to.

V. EVALUATION A. Dataset

We evaluate the performance of our framework on a data set coming from CodeForces 2 .

The Codeforces Dataset: The dataset is made from 36,272 C/C++ source code examples, that solve 91 problems from Codeforces. We compiled them using clang 3.9. Compilation process: Each source code is compiled using a random compiler with a random optimization among options -O0, -O1, -O2, -O3, and -Os. Binaries are then stripped. We thus have a relatively well balanced dataset of 36, 272 binaries.

Optimization level similarities: There are huge similarities between programs compiled with -O2 and -O3. Over a random sample of 2920 MinGW binary codes, 561 binary codes compiled with -O2 are identical when compiled with -O3. This has already been reported [START_REF] Egele | Blanket execution: Dynamic similarity testing for program binaries and components[END_REF] and has been an issue for Chen et al. [START_REF] Chen | Himalia: Recovering compiler optimization levels from binaries by deep learning[END_REF] in their compiler optimization detection framework. Note that this is not the case for Visual Studio Compiler which has only one advanced optimization level. Due to the similarity between programs produced by -O2 and by -O3, we will consider -O2 and -O3 as a proper subclass.

B. Hierarchies of SNN

Prior works [START_REF] Rosenblum | Recovering the toolchain provenance of binary code[END_REF]- [START_REF] Chen | Himalia: Recovering compiler optimization levels from binaries by deep learning[END_REF], [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF], [START_REF] Rosenblum | Extracting compiler provenance from program binaries[END_REF] take each classification task, such as determining the optimization level, separately, and use a single neural network to classify at the level of functions. We choose to implement binaries hierarchical classifiers with a local expert classifier per parent node [START_REF] Silla | A survey of hierarchical classification across different application domains[END_REF]. In this way, the optimization level prediction depends on the compiler family. Moreover, we take advantage of our domain problem, for instance, we expect -O0 instances and -O1 instances to be closer to each other than -O0 instances and -O2/-O3 instances.

A site neural network expert is specialized to make the separation between two choices (e.g. between MinGW -O0/-O1 and MinGW -O2/-O3). The hierarchy for optimization level prediction (Figure 5) is built upon the prediction of the compiler family. There is one expert per parent node, thus 15 experts. Similarly, the hierarchy for version prediction (Figure 6) is built upon family prediction and contains 22 experts. Three experts for compiler family prediction are in both hierarchies. Therefore, we have in total 34 experts.

C. Implementation of SNN

We perform four convolutions with a dimension of 8 at each step. The hyper-parameter d t of the matrix W t is 8 for each convolution t. We use four pooling layers with amp 2,2 , amp 4,4 , amp 8,8 , and amp 16,16 operators. The multilayer perceptron has four layers. Their respective number of neurons are 384, 256, 128, and 2. As a result, each of our SNN has 286, 962 trainable parameters. Therefore the complete model, with two hierarchies, has 9, 756, 708 trainable parameters. The number of training epochs depends on each expert and is put under each node in Figures 5 and6. Batch size is 20, learning rate is 0.005 and the loss function is cross-entropy loss.

We implemented our neural network using the language python along with the machine learning library PyTorch. Prepossessing, forgetful, and chopping phases are implemented in C++. Data and tools are available at https://gitlab.inria.fr/tbenoit/saner-2021-binary-level-toolchainprovenance-identification.

D. Research questions

We investigate the following research questions, ending the two most crucial ones, to validate our framework and to see its current limits: RQ1 How does our framework evolve when the site size α increases in term of running time performance? RQ2 How does our framework evolve when the site size α increases in term of accuracy? RQ3 Does our framework have the capacity to predict the compiler and optimization level of binary codes? RQ4 Does our framework have the capacity to predict the compiler version of binary codes? E. RQ1: How does our framework evolve when the site size α increases in terms of running time performance ? a) Goal: The whole control flow graph has variable size and is not suitable as direct input of a machine-learning process. As a result, we chop the control flow graph in 100 smaller graphs of fixed size α. The dimension of the parameter α modifies the processing time necessary to perform the chopping phase itself along with the learning phase. We want to evaluate the impact of this parameter α in the learning phase process of both our hierarchies the one for the optimization level prediction and the one for the compiler version prediction.

b) Method: We vary α by taking values : 4, 8, 12, 16, 20, 24, 28, 32, 48 and 64. As shown in Figure 7, using those α in the chopping phase retains from 2% of the original CFG data to 12%. This is a good range of values as we seek to retain only a small part of the forgetting graphs data.

We select only a sample of 1000 binaries from our dataset. Using an identical computer sequentially for all values of alpha, we learn one epoch for each expert. We measure the learning phase processing time in function of α. We extrapolate from that the average learning phase processing time per binary if the number of epochs was correct. We also record the average processing time of the chopping phase of a binary in function of α. We present the average time in seconds of the process a binary coming from our dataset.

We run experiments on a computer equipped with an Intel i7-8665U, 4 cores, and a frequency of 2.11 GHz. c) Results: Graph extraction, which does not depend on α, takes an average processing time of 0.11 seconds per binary. The average time of the chopping phase goes from 0.12 seconds per binary to 0.38 seconds per binary depending on α. Chopping processing time is monotonic with α. The average time of the complete learning phase processing goes from 0.95 seconds per binary to 2.21 seconds per binary.

d) Conclusion: As α increases, so do the volume of the data to be classified. Thus one can adapt the process to its need using α. It can scale to dozens of thousands of binaries. We note that extraction and chopping phases can be parallelized efficiently. Due to the use of experts, the learning phase is also done in parallel.

F. RQ2: How does our framework evolve when the site size α increases in terms of accuracy ? a) Goal: We want to identify the impact of α on the accuracy of our framework. Indeed, since the parameter α gives the size of data extracted, at first glance, it must affect the accuracy of the machine learning process. However, it is a bit tricky because machine learning neural networks have an inherent variance in their results. To measure the performance of our hierarchies, we use as a single metric the macro average F1-Score. It is a simple metric that is unbiased by potential class imbalances.

b) Methodology: As in RQ1, we apply our framework with 4, 8, 12, 16, 20, 24, 28, 32, 48, and 64 as different possible values of α. We effectuate ten runs for each alpha. In each run, we sample 10% of our dataset as the train set. We sample a test set of size 2000. A different validation set composing 10% of the train set is used for each expert. For each specialized site neural network, we select the model neural network with the best accuracy on the validation split on the last epoch. We train our hierarchies for the optimization level prediction task and the compiler version prediction task.

To deal with the inherent variance, we use the mean value of each 10 runs as the estimated macro average F1-Score for a given α. Our data is then analyzed using linear models with the ordinary least square (OLS) method. This simple model assumes that our data follow a linear law with some noise that should be identically distributed. We evaluate the impact of α on the macro average F1-Score by the r 2 value of the statistical analysis. We can also give a probability that the model fits the data using the F1-Statistic. c) Results: On the optimization prediction task, a linear model does not fit the data using the ordinary least square (OLS) method (Figure 9). First, the probability that there is no relationship is too high as demonstrated by a p-value of 0.537. Even if there were a relation, α would explain less than 5 percent of the variance in the overall macro average F1-Score. On the version prediction task, a linear model is a better fit for the data using the ordinary least square (OLS) method.

First, the p-value is 0.00991. This probability may be lower as indicated by the Durbin-Watson test result of 2.699. Such a value higher than 2 predicates negative auto-correlation which could artificially increase the p-value by breaking an assumption of the ordinary least squares method. However, the probability of the Omnibus test about the distribution of errors is 0.375 which is too close to 0. With a good probability, the model fails to have normally distributed errors, a potential break of another assumption. The r 2 of the model is an intermediate value of 0.585, α could explain around 58% of the variation in the mean F1-Score.

d) Conclusion:

We have moderate confidence that increasing alpha does increase our capacity to predict the data in the version prediction task. We select α = 32 as the value to use in our next questions due to it having achieved the best mean F1-Score in both prediction tasks and extracting only 8% of the complete CFG data in megabytes.

G. RQ3: Does our framework have the capacity to predict the compiler and optimization level of binary codes? a) Methodology: We set the site size to be α = 32 (cf. RQ1-RQ2). We train experts of the first hierarchies to predict both the compiler family and the optimization level. To run this experiment, the dataset is split into a train set and a test set. The test set has a size of 2200 and is balanced along the dimension of the combination of compilers, compiler versions, and optimization levels. A different validation set composing 10% of the train set is used for each expert. The loss function is the cross entropy loss. For each specialized site neural network, we select the epoch with the best accuracy on the validation split on the last ten epochs. After training, the hierarchical classifier is evaluated thanks to the test set. b) Results: Table III presents F1-Score for each compiler family and the macro average F1-Score. We achieve a very high overall F1-Score of 0.9950 on the compiler family prediction. Table IV presents F1-Score for each optimization level and the macro average F1-Score. We achieve a high F1-Score of 0.7549 on the optimization level prediction.

c) Conclusion:

We achieve a very high F1-Score on the compiler family prediction. It is so high that nearly all errors are in predicting the optimization level. On this other task, we achieve a high F1-Score. We conclude that predicting the optimization level is harder than predicting the compiler family. We suppose this is in part due to the similarity of binaries produced by different compiler options. H. RQ4: Does our framework have the capacity to predict the compiler version of binary codes? a) Methodology: We set the site size to be α = 32 (cf. RQ1-RQ2). We use the same methodology as before with experts of the second hierarchy to predict the compiler version.

VI. DISCUSSION

We choose to compare our framework to the work of Massarelli et al. [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF]. As we said previously, it is one of the closest frameworks to ours. Indeed, they used graph convolution at the binary function level while we performed graph convolution at the binary level. Moreover, this framework is available in a repository. This is not the case with other works [START_REF] Rosenblum | Recovering the toolchain provenance of binary code[END_REF]- [START_REF] Rahimian | Bincomp: A stratified approach to compiler provenance attribution[END_REF] which makes it difficult for a fair comparison. Still, we also provide elements of comparison with Rosenblum et al. [START_REF] Rosenblum | Extracting compiler provenance from program binaries[END_REF].

A. Comparison with Massarelli et al. [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF] a) Methodology: Massarelli et al. [START_REF] Massarelli | Investigating graph embedding neural networks with unsupervised features extraction for binary analysis[END_REF] relies on radare2 [START_REF] Team | Radare2 book[END_REF] to extract CFG of binary functions in a binary. They propose an instruction embedding method called i2v to represent each instruction, and then feed it to an RNN.

We sample a set of 2843 binaries from our dataset that we transform to CFG of functions using the Massarelli et al. framework. Using Massarelli et al. framework, 227 of these binaries constitute the validation split, 601 constitute the test split and the rest belong to the train split. We take a much smaller dataset that for RQ3 and RQ4 because Massarelli et al. preprocessing phase is around 1300 time slower than ours for learning.

On our side, we train our framework using a dataset containing the same 2843 binaries with a random test set of 601 binaries. We perform the experiment 10 times to mitigate randomness.

To compare with our binary level approach, we take two distinct approaches:

• Function-level We evaluate Massarelli et al., denoted MA. We transform the output of our SNN framework from binary level to function level. To do so, the prediction of each function is the prediction for the binary containing the function. We use the information of MA to get the number of functions in each binary. We note this process SNN-F; • Program-level We already have evaluated our approach with a complete dataset. We note this evaluation SNN-Full. We transform the output of MA from function level to binary level. To do so, the prediction for a given binary is the majority prediction of each function extracted from the binary. We note this process MA-B; b) Results: Massarelli et al. preprocessing is complicated and time-consuming, as radare2 disassembly is a costly process. The average time per binaries is 665 seconds with a minority of binaries taking hours to complete. While this preprocessing is done in parallel, it takes 525 hours of computation to obtain 2842 binaries. It was done on computers equipped with two Intel Xeon silver 4110. The dataset for Massarelli et al. approach contains approximately 825,000 binary functions. Each epoch takes approximately 2200 seconds. Thus, the average learning time is 116 seconds per binary, using computers equipped with two Intel Xeon gold 5218R -while on an Intel i7-8665U our approach has an average learning time of 1.71 seconds per binary.

We report the result of each framework on the three tasks in Table VII. For the sake of brevity, we report only the overall macro average F1-Score with two digits along with standard deviation when available. Comparing SNN with a restricted dataset to MA-B, we are on the same range on family prediction. However, we outperform MA-B by 0.16 on optimization prediction, and 0.13 on version prediction. Using our full dataset, SNN-Full outperforms MA-B by 0.08 en compiler family prediction, 0.33 on optimization prediction and 0.33 on version prediction. This is expected as MA-B had less training data than SNN-Full. On the other side, comparing MA to SNN-F, we are again on the same range on family prediction. We outperform it by 0.24 on optimization prediction and 0.06 on version prediction.

c) Conclusion:

We are approximately 68× times faster during learning, and 1300× time faster during preprocessing. Moreover, we can use much more parallelism in the learning phase. This processing time allows us to enhance our classification performance by learning from more binaries.We are also able to consistently outperform in terms of accuracy MA except for the compiler family where we are in the same range.

B. Comparison with Rosenblum et al. framework [17]

Rosenblum et al. [START_REF] Rosenblum | Extracting compiler provenance from program binaries[END_REF] dataset and classifier are not available, so we give only elements of comparison. They report accuracy of 0.999 for compiler family, 0.999 for optimization level, and 0.918 for compiler version. However, their dataset consists of 2,686 programs compiled from only 175 different source code examples, while we consider 36,272 different source code examples (internal validity is ensured by random assignment of a code to a toolchain). They consider three compiler families (Visual Studio, Intel Compiler, and GCC), but only 9 compiler versions, and the optimization prediction is reduced to 'Low' (e.g. -O0) vs 'High' (e.g. -O2) -hence a total of 18 compiler configurations where we consider 92 such configurations.

We can observe trends when we move from our broad dataset to a more restricted one closer to Rosenblum et al. setting. We select compiler versions VS 10.0, VS 12.0, VS 2017, GCC 4.8, GCC 5.5 and GCC 7.5. Reproducing the setting of Rosenblum et al., we restrict optimization option prediction to a choice between 'Low' and 'High' optimization options. After a learning phase on this restricted dataset, our F1-Scores is 1.0 for compiler family, 0.97 for optimization level and 0.89 for compiler version. As expected, broadness of the dataset significantly impacts accuracy.

VII. LIMITATIONS

Our dataset is composed of small programs (most file sizes are around 30kb). It would be interesting to use a more diverse dataset. Nevertheless, our evaluation at least demonstrates the accuracy of SNN. Moreover, our approach was tested and validated on stripped binary codes. In adversarial contexts where binary are obfuscated or when we are dealing with malware, the situation is quite different. We believe this is a challenge worth working on.

VIII. CONCLUSION

We consider the problem of toolchain provenance identification. Our starting hypothesis is that binary code is not unstructured data and that semantics is important. Moreover, since libraries are more frequently dynamically linked, binaries are usually homogeneous in terms of toolchain provenance.

In this work, we explore the possibility of (i) extracting semantic features in the form of graphs, (ii) processing the neural networks of the graphs to propagate the information according to the topology of the graph, and (iii) using tailored hierarchies to fit a dataset. We demonstrate that binary-level toolchain provenance identification is feasible with both high precision/accuracy and fast learning -we outperform a recent function-level approach on these metrics.

This work opens several immediate questions. The combination of the forgetful phase followed by the chopped phase provides a simple and realistic feature graph model. That said, one could think of a first phase that would leave out less information. Also, the pooling layers play an important role. It should be worth looking at which features are useful and how they are intertwined to improve pooling. This question bounces off the question of semantics. Finally, CFG provides only a very shallow program semantics. An interesting question would be to automatically extract and take advantage of some sort of richer semantic features without too much extra cost.

Fig. 1 .

 1 Fig. 1. Overall architecture of the preprocessing phase

Fig. 3 .

 3 Fig. 3. The forgetful phase

Algorithm 1

 1 Graph chopping algorihtm Input: A forgetting graph G = (V, E), a root vertex r in V Parameters: n the number of sites to extract, α max node in a site Output: A graph containing a maximum of n sites Let G r = (V r , E r ) be a a graph. while |V | > 0 and n > 0 do Let q1 be a queue. Let q2 be a queue. Let g = (N, A) be a graph. push q1, r while |N | < α and |q1| > 0 do empty q2 for all x ∈ q1 do if |N | > α then break end if for all y

Fig. 4 .

 4 Fig. 4. An example of two adaptive max pooling on a matrix of dimension 22 × 8. The first adaptive max pooling is of dimension 2 × 2, it has a kernel size of dimension 11 × 4. The second adaptive max pooling is of dimension 4 × 4, it has a kernel size of dimension 6 × 2.

Fig. 5 .

 5 Fig. 5. The hierarchical classifier for optimization level prediction. A parent node is an expert trained during the number of epoch noted in blue.

Fig. 6 .

 6 Fig. 6. The hierarchical classifier for version prediction. A parent node is an expert trained during the number of epoch noted in blue.

Fig. 7 .

 7 Fig. 7. Percent of the CFG dataset in megabyte kept by the chopping phase.

Fig. 8 .

 8 Fig. 8. Chopping phase: average time to process a binary code

Fig. 9 .

 9 Fig. 9. Macro average F1-Score along alpha on the optimization level prediction task. Red dots are values obtained by a run. Black dots are mean values for an α. The blue line is a linear model using the ordinary least square method (OLS).

Fig. 10 .

 10 Fig. 10. Macro average F1-Score along alpha on the compiler version prediction task. Red dots are values obtained by a run. Black dots are mean values for an α. The blue line is a linear model using the ordinary least square method (OLS).

Fig. 11 .

 11 Fig. 11. Confusion matrix of SNN on the compiler family and option prediction. On the diagonal, the best value is 100 and elsewhere it is 0.

Fig. 12 .

 12 Fig. 12. Confusion matrix of SNN on the compiler version prediction. On the diagonal, the best value is 100 and elsewhere it is 0.

•

  From a methodological point of view, Site Neural Networks provides end-to-end graph based classifiers. As a result, decisions and classifications should be more easily based on the binary code semantics. • Our framework can easily be adapted to different contexts: chopping is parameterized by the subgraph size α, while our SNN hierarchy allows adding new experts in a modular way -for example to deal with a new compiler version or a new option. 2) Most prior works in toolchain provenance based on Machine Learning use as datasets a set of binary functions generated by different compilers and optimization levels.

  Table V presents F1-Score for each compiler version, and macro averages F1-Score. We achieve a moderate F1-Score of 0.6475 on the version prediction task. Table VI presents macro averages F1-Score of compiler version identification by compiler family. We notice the macro average F1-Score of 0.1856 for the Clang family. The Clang family version prediction stands out due to this very low accuracy. On the other hand, the macro average F1-Score of the MinGW family is about 0.9799. Table VI shows that if we omit Clang, we attain a macro average F1-Score if 0.8167. Moreover, if we concentrate on binaries compiled for Windows, we attain a very high F1-Score of 0.9502. c) Conclusion: We conclude that compiler version prediction is hard on the Clang family but easy for binaries coming from the Windows platform.

	Family	Precision	Recall	F1-Score Support
	Clang	0.2019	0.1883	0.1856	600
	GCC	0.5731	0.5567	0.5496	600
	MinGW	0.9832	0.9767	0.9799	600
	VS	0.9162	0.9275	0.9206	400
	GCC-MinGW-VS	0.8242	0.8202	0.8167	1600
	MinGW-VS	0.9497	0.9521	0.9502	1000
		TABLE VI
	F1-SCORE OF SNN FOR VERSION PREDICTION AMONG COMPILERS
		FAMILY
	b) Results:

ACKNOWLEDGMENTS

Experiments were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER as well as other organizations (see https://www.grid5000.fr).

This work is supported by (i) a public grant overseen by the French National Research Agency (ANR) as part of the "Investissements d'Avenir" French PIA project "Lorraine Université d'Excellence", reference ANR-15-IDEX-04-LUE, and (ii) has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 830927 (Concordia).

1 More than 70% of commercial applications used COTS (Gartner).


H Mercier
email: hugo.mercier@gmail.com

Jean Nicod

Nicolas Claidière
email: nicolas.claidiere@normalesup.org

DOES DISCUSSION MAKE CROWDS ANY WISER? DOES DISCUSSION MAKE CROWDS ANY WISER?

Keywords: Group decision making, wisdom of crowds, aggregation, majority rule, social learning

Does discussion in large groups help or hinder the wisdom of crowds? To give rise to the wisdom of crowds, by which large groups can yield surprisingly accurate answers, aggregation mechanisms such as averaging of opinions or majority voting rely on diversity of opinions, and independence between the voters. Discussion tends to reduce diversity and independence. On the other hand, discussion in small groups has been shown to improve the accuracy of individual answers. To test the effects of discussion in large groups, we gave groups of participants (N = 1958 participants in groups of size ranging from 22 to 212; mean 59) one of three types of problems (demonstrative, factual, ethical) to solve, first individually, and then through discussion. For demonstrative (logical or mathematical) problems, discussion improved individual answers, as well as the answer reached through aggregation.

For factual problems, discussion improved individual answers, and either improved or had no effect on the answer reached through aggregation. Our results suggest that, for problems which have a correct answer, discussion in large groups does not detract from the effects of the wisdom of crowds, and tends on the contrary to improve on it.

DOES DISCUSSION MAKE CROWDS ANY WISER? Does Discussion Make Crowds Any Wiser? Ancient Athens is famous for its reliance on democratic decision making. Laws were put forward by a council of 500, and voted by an assembly of 6000 citizens. Judicial decisions were made by courts of 200 jurors [START_REF] Hansen | The Athenian democracy in the age of Demosthenes: Structure, principles, and ideology[END_REF]. In each case, the assembled citizens would listen to the arguments of the different parties, and the issue would be resolved by a simple majority vote. Crucially, during these votes, discussion among citizens was not formally allowed. Was this a wise rule? If answering this question might have helped Athenians make better decisions, the generalization of democratic decision making means it is an even more pressing question today. Crowds-defined here as any large group, whether or not they are organized-play an increasingly important role, whether in politics-from mass protests to citizens' assemblies-in the creation and diffusion of knowledge-from scientific consortia to Wikipedia contributors-or in business, as companies try to make the best of their workforce's knowledge.

We start by reviewing arguments suggesting that discussion might hinder the wisdom of crowds, and thus that groups might be better off aggregating their answers without discussion, before turning to arguments suggesting instead that discussion might improve the individual performance of the group members, without taking away the added value of the wisdom of crowds. In the absence of empirical evidence directly bearing on this issue, we conduct a large-scale experiment in which 1958 participants in 33 groups with size ranging from 22 to 212 participants (mean 59), are confronted with a variety of problems, first without being able to discuss them, and then with discussion allowed. When an objective benchmark for performance is available, our results suggest that discussion consistently improves individual answers, and also often improves the answer reached through the wisdom of crowds.

In ancient Athens, rules limiting discussion between citizens before a vote were no doubt linked to the practical necessity of making a decision in a limited time frame (often half a day) [START_REF] Manville | A Company of Citizens: What the World's First Democracy Teaches Leaders About Creating Great Organizations[END_REF]. More recently, theoretical work has suggested that these constraints might have been wise, maximizing the chances that the citizens would vote for the best available alternative. The most fundamental result underpinning the efficacy of majority voting is the Condorcet Jury Theorem [START_REF] Condorcet | Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix[END_REF]. For a dichotomous choice, the theorem "states that the probability that a majority votes for the better alternative exceeds p [the probability that each voter selects the right option] and approaches 1 as n [the number of voters] goes to infinity" (Ladha, 1992, p. 34). The efficacy of majority voting has been demonstrated not only in models (e.g., [START_REF] Banks | Information aggregation, rationality, and the Condorcet jury theorem[END_REF][START_REF] Ladha | The Condorcet jury theorem, free speech, and correlated votes[END_REF], but also a variety of experiments (e.g., [START_REF] Hastie | The robust beauty of majority rules in group decisions[END_REF].

For the Condorcet Jury Theorem to apply, a set of constraints has to be respectedthat the voters are more likely than chance to vote for the best alternative, that they do not vote strategically, and, crucially here, that their decisions are independent of one another. If some voters imitated others, without thinking for themselves, the effective size of the assembly would be reduced, along with the chances that the majority supports the best alternative. During discussion, voters are likely to influence each other, thereby potentially losing some of their independence, and lessening the benefits of majority voting (although see, [START_REF] Estlund | Opinion leaders, independence, and Condorcet's jury theorem[END_REF].

Besides majority voting, the other main phenomenon responsible for the wisdom of crowds is averaging. At least since [START_REF] Galton | Vox populi[END_REF], it has been well established that measures of central tendency such as the mean typically have a lower error than the mean individual error. For instance, when considering a range of numerical estimates that deviate more or less from a correct answer, the error of the mean answer will always be either lower than the mean error (if the correct answer is within the range of all the answers provided), or the same as the mean error (otherwise) (see, e.g., [START_REF] Larrick | Intuitions about combining opinions: Misappreciation of the averaging principle[END_REF]. Moreover, for many distributions of answers, the error of the mean is uncannily small compared to the mean error, a phenomenon which has allowed averaging to improve performance on a variety of problems ranging from political predictions to medical diagnoses [START_REF] Surowiecki | The Wisdom Of Crowds[END_REF].

As in the case of majority voting, the risks of discussion for the benefits of averaging are clear. During discussion, individuals are likely to converge on a middle of the road answer, eliminating the most extreme views, which will reduce the diversity and the range of answers, and lower the potential benefits of averaging. Even increases in individual accuracy might not compensate for this loss of diversity (see, e.g., [START_REF] Hahn | How communication can make voters choose less well[END_REF][START_REF] Hong | Groups of diverse problem solvers can outperform groups of high-ability problem solvers[END_REF][START_REF] Lorenz | How social influence can undermine the wisdom of crowd effect[END_REF]. There are therefore good grounds to believe that discussion might hamper information aggregation in large groups, which are most likely to benefit from the wisdom of crowds. Indeed, the problem might be particularly acute in the type of densely connected topologies that we will study here [START_REF] Hahn | Truth tracking performance of social networks: How connectivity and clustering can make groups less competent[END_REF].

By contrast, other results suggest that discussion might play a positive role. Smallgroup discussion has been shown to improve the average performance of the group members on a wide range of problems, ranging from logical tasks to political predictions (e.g., [START_REF] Mellers | Psychological strategies for winning a geopolitical forecasting tournament[END_REF][START_REF] Moshman | Collaborative reasoning: Evidence for collective rationality[END_REF][START_REF] Trouche | Arguments, more than confidence, explain the good performance of reasoning groups[END_REF]for reviews, see, Laughlin, 2011;[START_REF] Mercier | The argumentative theory: Predictions and empirical evidence[END_REF][START_REF] Mercier | The Enigma of Reason[END_REF]. In some cases, discussion can even lead to answers that are superior to those reached by any of the group members (e.g., [START_REF] Laughlin | Groups perform better than the best individuals on letters-to-numbers problems: Informative equations and effective reasoning[END_REF].

The question remains open of whether this improvement in performance, typically observed in groups of at most five people (although see, [START_REF] Hastie | Inside the Jury[END_REF][START_REF] Hans | Deliberation and dissent: 12 angry men versus the empirical reality of juries[END_REF] for 12-person juries, with less clearly correct answers, and [START_REF] Mellers | Psychological strategies for winning a geopolitical forecasting tournament[END_REF] for larger groups interacting through an internet forum), would translate to larger groups, which make discussion less natural [START_REF] Fay | Group discussion as interactive dialogue or as serial monologue: The influence of group size[END_REF][START_REF] Krems | Why are conversations limited to about four people? A theoretical exploration of the conversation size constraint[END_REF], and which might create more opportunities for herding, or for the majority to impose its view regardless of its accuracy (e.g., [START_REF] Asch | Studies of independence and conformity: A minority of one against a unanimous majority[END_REF]).

Still, it is possible that the improvement in performance yielded by small-group discussion might also be observed in larger groups (on the difficulty for accurate answers to spread widely, see, [START_REF] Moussaïd | Reach and speed of judgment propagation in the laboratory[END_REF]. Improvements in individual performance might then be sufficient to compensate for the decrease in the diversity and independence of the answers, such that discussion will improve, or at least not deteriorate, the wisdom of crowds (be it obtained through majority voting, averaging, or other means of aggregation).

A few studies have tested whether discussion is detrimental to the wisdom of crowds in large groups. In an experiment, mi-sized groups of participants (N = 12) had to make numerical estimates (about, e.g., the population size in a city), and some participants were provided with the average group answer, and an opportunity to revise their estimate on that basis [START_REF] Lorenz | How social influence can undermine the wisdom of crowd effect[END_REF]. Although the average performance of these participants improved, several indicators of the strength of the wisdom of crowds decreased (e.g. the degree of diversity within the answers). Another study confirmed that receiving the average answer from other participants leads to a decrease in diversity, but it also found that, for some network configurations, the increase in individual accuracy more than compensated for this loss of diversity [START_REF] Becker | Network dynamics of social influence in the wisdom of crowds[END_REF]. Importantly, in this latter experiment the participants received the average group answer, but they were expressively forbidden from discussing with one another. Several studies have shown that the increases in accuracy following discussion are substantially larger than those following mere exposure to others' opinion (e.g., [START_REF] Liberman | Naïve realism and capturing the "wisdom of dyads[END_REF][START_REF] Minson | Two to Tango[END_REF]. This experiment might thus underestimate the benefits of discussion.

In another experiment, a very large crowd (N = 5180) also had to provide numerical estimates of various quantities [START_REF] Navajas | Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds[END_REF]. Crowd members were then provided with the opportunity to talk to each other in small groups (N = 5), for a very short amount of time (1 min), and to revise their initial answers on the basis of this discussion. In this case, discussion had an unambiguously positive effect, as it increased not only individual performance, but also the answer reached through the wisdom of crowds. However, this study relied on the well-established improvement in performance following small-group discussion, and does not directly address the question of whether a broader discussion within the crowd would also yield such positive effects.

To the best of our knowledge, the study that most directly tested the effect of discussion in medium sized groups (N = 11 to 25) used the following method-which we describe in greater details, since it is similar to the method of the present experiments [START_REF] Claidière | Argumentation and the diffusion of counterintuitive beliefs[END_REF]. In each group, participants were seated together in a room, following a grid pattern. The participants were shown a logical or mathematical problem to solve, and given five minutes to attempt to find an answer on their own. Participants then either had fifteen minutes to talk about the problem with their neighbors (Discuss Condition), or to see the response of their neighbors, without discussion (Silence Condition). Every minute, participants recorded their answers, which allowed measuring changes in the percentage of correct answers with time. After the initial five minutes of solitary reasoning, performance improved faster in the Discuss than in the Silence condition. Moreover, a reanalysis of these data shows that discussion vastly improved on the ability of the wisdom of crowds (here, majority voting) to select the best answer. At the end of the first phase of solitary reasoning, the correct answer was supported by the majority of the participants in only 3 out of 12 groups, while it was supported by the majority in all groups after discussion.

Even if this latter study shows that discussion improve individual answers and the aggregated answer yielded by the wisdom of crowds, it has several limitations. The group size, while larger than that used in most experiments on group decision making, was still modest. The problems used were known to yield massive improvement with small-group discussion [START_REF] Trouche | Arguments, more than confidence, explain the good performance of reasoning groups[END_REF]. The participants were a homogenous group of students.

Finally, only one method of aggregating opinions-majority voting-was tested. A measure of central tendency, for instance, might be more sensitive to a loss of diversity following discussion [START_REF] Hong | Groups of diverse problem solvers can outperform groups of high-ability problem solvers[END_REF][START_REF] Lorenz | How social influence can undermine the wisdom of crowd effect[END_REF]. This overview of the literature suggests that there is no clear existing answer to the question of whether large groups are better off discussing before their opinions are aggregated. To start answering this question, we took advantage of a science festival, the European Researchers' Night which would be attended by hundreds of people across 11 towns in France. In each town, a room was set up in which participants could take part in the present experiment, as an introduction to research. As in the Discuss condition of [START_REF] Claidière | Argumentation and the diffusion of counterintuitive beliefs[END_REF], after being presented with a problem, participants had five minutes to think about it on their own, before being able to discuss it with their immediate neighbors for 15 minutes, with their answers being recorded every minute.

We used three types of problems. First, two demonstrative problems, one of which being the bat and ball from the Cognitive Reflection Test [START_REF] Frederick | Cognitive reflection and decision making[END_REF]. Demonstrative problems have a solution that can be conclusively demonstrated using shared knowledge [START_REF] Laughlin | Demonstrability and social combination processes on mathematical intellective tasks[END_REF]. These problems constitute an extension and a replication to large, more diverse groups, of the experiment described above [START_REF] Claidière | Argumentation and the diffusion of counterintuitive beliefs[END_REF].

Second, we used two factual problems, drawn from [START_REF] Navajas | Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds[END_REF], such as "How many goals were scored in the XXX world cup?" If small-group discussion has been shown to improve performance on such problems [START_REF] Navajas | Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds[END_REF][START_REF] Sniezek | Accuracy and confidence in group judgment[END_REF], the effects of large-group discussion, and the repercussions of the discussion for the value of the wisdom of crowds, have not been established to the best of our knowledge.

Finally, we used two ethical problems, drawn from [START_REF] Thorndike | Valuations of certain pains, deprivations, and frustrations[END_REF], such as "How much money should be awarded to compensate someone who lost a little finger in a DOES DISCUSSION MAKE CROWDS ANY WISER? workplace accident?" Discussion in small groups on such problems typically does not lead to systematic changes of mind (Mercier et al., 2017). We did not expect that large-group discussion would lead to different outcomes. As a result, these problems were used as a control in which we did not expect discussion to have any systematic effect on the answers.

If we expect the effects of small-group discussion to also be observed in large groups (as in [START_REF] Claidière | Argumentation and the diffusion of counterintuitive beliefs[END_REF] for demonstrative problems), we can derive the following hypotheses:

H1a For demonstrative problems, discussion improves performance more than solitary thinking.

H1b For factual problems, discussion improves performance more than solitary thinking.

H1c For ethical problems, discussion does not have a larger impact than solitary thinking.

When it comes to demonstrative problems, previous results also lead to the prediction that discussion will improve both individual performance and the aggregate answer.

H2 For demonstrative problems, discussion leads to better aggregate answers, as selected through majority voting.

By contrast, for factual problems, it is unknown whether the loss of diversity and independence will compensate for any potential individual gain in accuracy. As a result, we formulate the following research question: RQ1 For factual problems, does discussion lead to worse, equivalent, or better aggregate answers, as selected through averaging?

Method

Participants

The experiment was part of the European Researchers' Night, a pan-European science fair organized by researchers to introduce the public to the world of science and research. In France, the organizing committee of the 2017 edition gave us the opportunity to organize a large participative experiment involving 11 cities and 1958 participants (1048 females).

Participants were visitors to the science fair, who came in a large room to take part in an experiment advertised as not being suitable for children younger than 12 (90% of participants reported an age between 13 and 60; median = 24). There were two to six consecutive groups in each city (totaling 33 groups ranging from 20 to 208 individuals [mean 58]), which led to a total of between four to seven groups (259 to 468 participants) per problem. More details can be found in the ESM.

Materials

The six problems we used as material were:

Paul and Linda (demonstrative problem 1). Paul looks at Linda; Linda looks at John; Paul is married; John isn't married; Is someone married looking at someone who isn't married? Little Finger (ethical problem 1). How much money should be awarded to compensate someone who lost a little finger in a workplace accident?

Worms (ethical problem 2). How much money should be awarded to compensate someone who finds they have been eating earthworms in their restaurant meal?

Procedure

The experiment took part in large rooms with chairs arranged in a grid pattern. As participants arrived, they were asked to sit close to each other so that their seating arrangement would be as close as possible to a square grid, with no empty seats. Once everyone was seated, a trained researcher explained to the participants that they were taking part in a real experiment, that they could leave the room at any time, that their anonymous data would be used in a scientific publication and that by giving us their response sheet at the end of the experiment they agreed to these conditions. Answer sheets were distributed that contained 15 rows, one row for each time step, with the space for an answer to the problem, some demographic questions that were answered immediately (group number, seat number, town, age, gender), and a white space for free writing. After a brief explanation of the Silence Phase of the experiment, and the importance of not talking, showing each other their answers, or using their phones to check the answer, the experiment started. The problem was displayed on a large screen so that all participants could start answering it at the same time. After 20s, the participants provided their first answer. Four more answers were gathered at succeeding 1-min intervals.

Participants were then told that they would now be able to discuss their answers with their neighbors (Discussion Phase). Neighbors were defined as the eight (maximum) participants surrounding them. Participants were told that the goal was for them to reach a consensus. After they were given the signal to start discussing, the participants had to write down their answer every minute, as in the Silence Phase, for 10 minutes. Time was kept by the experimenter who prompted everyone to write down their answer every minute. At the end of the experiment a 15 min debrief explained the state of the art in group decision making, the purpose of the experiment, and the hypotheses. Participants were also encouraged to advertise the experiment to other potential participants at the fair, but without revealing its purpose and proceedings. Importantly, we changed problems between the groups in each city in order to make sure that participants were completely naïve (i.e. even if they had been informed by a previous participant, they would face a different problem).

Data coding and analysis

Response sheet for demonstrative and factual problems were coded using a crowdsourcing platform. Three independent coders coded the responses of each participant and when available the modal response was retained. In cases in which three different coders disagreed, often due to mistyping from the coders, the experimenters returned to the original response sheet to determine the most likely response (less than 1% of the responses were reevaluated).

Regarding ethical problems, that required more judgment, one independent coder coded all responses from the 499 participants using four categories (for Little finger: a number, a monthly allowance, cost of medical intervention and other; for Worms: a number, the price of the meal, medical costs, and other).

Data exclusion and response variable

We excluded a total of 11% of responses from analysis. This percentage varied between problems, but, crucially, it did not vary with time (see ESM for detailed table). For Paul and Linda, we excluded responses that were not any of the three proposed options (<1%) and used as response variable a binary variable with 1 for correct response and 0 for any of the other two responses. For Bat and Ball, we excluded responses that were not 5 or 10 cents (6%) and used a similar binary variable, with 1 for correct response and 0 for the incorrect response. For the Elevators and World Cup problems we excluded responses that were not numeric, and responses above the 99% quantile to avoid extremely large values (such as "123456"; 7% and 12% of data were excluded resp.).

Finally, for the Worms and Little Finger problems, we excluded data from the "other" category (25% and 27% resp.) and re-coded responses as a binary response variable with 1 being the most frequent response at the end of the Silence Phase (i.e. the majority option before discussion) and 0 for all alternative responses. We should note, however, that our ethical problems, which had no correct answer, were quite different from the other problems and raised a number of issues, such that no strong conclusion can be drawn from them. Based on the advice of reviewers we decided to present the results of the ethics problems in the ESM only. To test H2, and answer RQ1, we turn to the effect of discussion on the aggregate answers. For demonstrative problems, we find that discussion leads to better aggregate answers. At the end of the Silence Phase, out of 13 groups, only two had a majority of correct responses (both for the Bat and Ball). By contrast, all groups had a majority of correct responses at the end of the Discussion Phase (a significant improvement, McNemar's chisquared = 9.10, df = 1, p = 0.003).

Statistical method

For factual problems we found that the error of the mean response (how the mean response in each group differed from the correct answer) decreased for the Elevators problem (Fig. 2; all six groups had a lower error of mean at the end of the Discussion Phase compared to the end of the Silence Phase; binomial test, p = 0.03). By contrast, there was no evidence of a decrease for the World Cup problem (two groups had a value that increased and two a value DOES DISCUSSION MAKE CROWDS ANY WISER? that decreased). A possible cause of this difference between the two problems is discussed below.

Discussion

H1a, b, and c were confirmed. For both demonstrative and factual problems, discussion improved performance over solitary thinking. The results also clearly supported H2: for demonstrative problems, discussion improved not only individual answers, but also the answers favored by majority voting, which went from two correct answers at the end of beginning of the Discussion Phase, to 13 out of 13 at the end.

Regarding RQ1, the answer is more equivocal. For one factual problem (Elevators), discussion consistently improved not only on individual answers, but also on the answers reached through averaging within each group. By contrast, for the other factual problem (World Cup), discussion improved on individual answers, but not on the answers reached through averaging.

To understand the differential impact of discussion on the wisdom of crowds in the two factual problems, it is useful to go back to Figure 2. As noted previously, the mean error of the participants decreased through time for both problems. Moreover, the wisdom of crowds effect was present throughout the experiment, with the error of the mean being always inferior to the mean error of individuals (Fig. 2A,2C). However, while the size of the gain through aggregation (i.e. the difference between the mean error and the error of the mean) stayed relatively constant during the Discussion Phase for Elevators (Fig. 2B), it decreased for World Cup (Fig. 2D).

To make sense of this difference, we can consider two ways for the mean error to decrease: (i) if most answers are distant from the correct answer, and there is a directional shift towards the correct answer, or (ii) if most answers aren't too distant from the correct answer, and there is a reduction of the variance in the answers, with the most extreme answers converging towards the correct answer. Overall, in Elevators, there is no decrease in variance (Fig. 2A), but there is a general shift towards the correct answer, which the overwhelming majority of participants had initially underestimated (Fig. 1). By contrast, in World Cup, there is no directional shift towards the correct answer, with the average answer being as distant from the correct answer at the beginning than at the end of the discussion (Fig 1 ); however, there is a reduction in the variance of the answers (Fig. 2C). Such a reduction in variance lowers the mean error, but not the error of the mean, thereby decreasing the difference between the two.

It is also worth noting that in all but one of the 10 groups facing factual problems, on average participants moved more towards the correct answer than towards what was the average group answer at the beginning of the discussion (see ESM, Table S3, and Fig. S2).

Indeed, on the whole participants barely moved towards the average answer (Elevators, 1.34; World Cup, 0.10), but they consistently moved towards the correct answer (Elevators,7.30;World Cup,36.03). This means that the improvement observed during discussion did not result from participants simply converging towards an answer corresponding to the average at the beginning of the Discussion Phase, as might be expected if participants felt the pull of the majority (see, e.g., [START_REF] Moussaïd | Social influence and the collective dynamics of opinion formation[END_REF]. Instead, in every group participants moved towards the correct answer. For factual problems (as for logical problems), in the course of discussion participants appear to have been pulled by arguments towards the correct answer (see, [START_REF] Claidière | Argumentation and the diffusion of counterintuitive beliefs[END_REF][START_REF] Mercier | The Enigma of Reason[END_REF].

Conclusion

Are crowds wiser with or without discussion? The literature makes conflicting predictions, and to answer this question we gave groups of medium to large size (N = 20 to 208) a problem to tackle individually first, and then through discussion with their neighbors.

When there were objective benchmarks, individual answers consistently improved with discussion, while aggregate answers improved in most cases and never consistently worsened.

When it comes to problems for which a correct answer exists, our results strongly argue in favor of discussion. First, for the four problems with correct answers studied heretwo logical, demonstrative problems, and two factual problems-discussion always improved the mean individual answer. Second, in three out of four cases, discussion led to better aggregate answers, aggregated either through majority voting (the two demonstrative problems), or through averaging (one factual problem). Third, in the last case with no improvement in aggregate answers, discussion was not detrimental to the aggregated answer because it had no effect. Thus, discussion had no detrimental effect on the wisdom of crowds for the problems examined here.

Our results also demonstrate the effectiveness of discussion in a more qualitative manner. For the two demonstrative problems, 15 minutes of discussion yielded enormous improvements in individual answers, which moved from 12% correct to 84% correct for Paul and Linda, and from 41% correct to 91% correct for the Bat and Ball. Remarkably, in the case of Paul and Linda, all groups reached at least 75% of correct answers, even though they had started with at best 17%. These results thus demonstrate the robustness of the 'truth-wins' scheme, by which a single individual with a correct answer to a demonstrative problem can convince a group, since we also observe its effects in large and diverse groups.

The positive effects of discussion are also clear for the two factual problems. In the Elevator problem, all groups correctly increased their average answer through discussion, moving from a mean error of 55 at the beginning of the discussion to a mean error of 48 at the end. In the World Cup problem, discussion nearly halved the mean error from 96 to 52. We also note that asking participants to estimate the number of goals scored in one specific world cup is a very high bar and it is remarkable that the average number of goals scored in the past six world cups is 160 goals, a difference of only 19 goals with the grand average reached at the end of the discussion. Moreover, in our experiments, participants were constrained in terms of who they could discuss the problems with. Giving people flexibility in network formation might further increase the advantages of discussion (see, e.g., [START_REF] Almaatouq | Adaptive social networks promote the wisdom of crowds[END_REF]. Alternatively, constraining networks to optimize the flow of information has also been shown to improve accuracy when discussion is not possible, but the same results might extend to situation in which discussion is possible [START_REF] Jönsson | The kind of group you want to belong to: Effects of group structure on group accuracy[END_REF].

Our results have theoretical and practical consequences. They support theoretical frameworks that postulate the power of discussion to change minds for the best [START_REF] Mercier | Why do humans reason? Arguments for an argumentative theory[END_REF], 2017), and they show that the loss in independence and diversity in the DOES DISCUSSION MAKE CROWDS ANY WISER? answers during discussion can be largely compensated by the increase in accuracy, contrary to what had been suggested (e.g., [START_REF] Hong | Groups of diverse problem solvers can outperform groups of high-ability problem solvers[END_REF][START_REF] Lorenz | How social influence can undermine the wisdom of crowd effect[END_REF]. Practically, our results show that discussion is a robust tool to improve not only individual, but also collective answers, even in large and diverse groups, at least for problems that have a correct answer.

  Answers provided: Yes [correct] / No / We can't tell. Bat and Ball (demonstrative problem 2). A candy and a baguette cost 1.10€ together. The baguette costs 1€ more than the candy. How much does the candy cost? Correct answer: 0.05€. World Cup (factual problem 1). How many goals were scored in the football world cup of 2010? Correct answer: 145 Elevators (factual problem 2). How many elevators are there in New York's Empire State Building? Correct answer: 73

Figure 1 :

 1 Figure 1: Evolution of the group response for each problem through the Silence (shaded area) and Discussion phases. Each colored line represents a unique group mean response and the black line represents the between group mean (+/-SE). The correct answer to the Elevators problem was 73 and to World Cup problem 145.

Figure 2 :

 2 Figure 2: Effect of discussion on the wisdom of crowds. Evolution of the mean error made by individuals (the mean of all the individual errors) and the error of the mean response (mean responses which are depicted in Figure 1) (A, C), as well as the difference between the two (B, D) for the Elevators and World Cup problems. Each colored line represents a unique group mean response and the black line represents the between group mean (+/-SE).

Acknowledgments

We thank all the staff of the French European Researchers' Night, the local and the national organizing committees and in particular Matteo Merzagora and Lionel Maillot for their help in setting up the "Grande Experience Participative". We also thank all the people that conducted the experiment, Sacha Altay, François Druelle, Justine Epinat, Annabelle Goujon, Julie Gullstrand, Sébastien Lerique, Heather McLeod, Mathilde Menoret, Virginie Postal-Le Dorse, Jean-Pierre Thibaut, Romain Trincherini, and Jean-Baptiste Van der Henst. The authors declare no competing interests. NC gratefully acknowledges financial support from ASCE (ANR-13-PDOC-0004) and LICORNES (ANR-12-CULT-0002). HM gratefully acknowledges financial support from FrontCog (ANR-17-EURE-0017), and PSL (ANR-10-IDEX-0001-02).

Analysis were carried out using R (R Core Team, 2020), mixed models were analyzed with the package lme4 [START_REF] Bates | Fitting Linear Mixed-Effects Models Using lme4[END_REF] and ggplot2 was used to produce the figures [START_REF] Wickham | ggplot2: Elegant graphics for data analysis[END_REF].

Data availability

All the data analyzed here are available at DOI: 10.17605/OSF.IO/CFWV2

Results

To test H1a, b, and c, we sought to determine whether discussion had a larger effect on the answers than solitary reflection. Figure 1 summarizes the evolution through time of the different groups with the average response for each problem (Supplementary videos 1 to 6 illustrate this evolution using the spatial layout of the rooms in which the experiment was carried out; the videos of each group are available in the public repository of the experiment).

Following [START_REF] Claidière | Argumentation and the diffusion of counterintuitive beliefs[END_REF], we used mixed models to study the interaction between the experimental phase (Silence vs. Discussion), and time during the first 10 timesteps (to maintain the same number of observations in the two phases: 5 in each of the Silence and Discussion phases). We report the models that combined the problems of each type; however, we also analyzed each problem independently and found that the results of the combined models also applied to each problem independently (full reporting of the models can be found in the Electronic Supplementary Materials). As in our previous study we found that discussion favored the dissemination of the correct response for the two demonstrative problems ( = 0.38, SE = 0.04, z = 8.37, p < 0.001). For the two factual problems, there was also a significant interaction between the Silence and Discussion phases, with a reduction in the distance to the correct response observed only during the Discussion Phase ( = -2.31, SE = 0.74, df = 6586, t = -3.12, p = 0.002; see Fig. 2).


domain: spi spi.opti sdv

text:
Motion-compensated Optical Coherence Tomography based on higher-order regression for real-time volumetric imaging

Ruizhi Zuo a , Kristina Irsch b,c , Jin U. Kang a,c a Johns Hopkins University, Whiting School of Engineering, Baltimore, MD, USA; b CNRS, Vision Institute, Paris, France; c Johns Hopkins University, School of Medicine, Baltimore, MD, USA Optical Coherence Tomography (OCT) has evolved into a powerful clinical tool, with a wide range of applications in ophthalmology. However, most OCT systems for real-time volumetric (3D) and in vivo imaging suffer from image distortion due to motion artifacts induced by involuntary and physiological movements of the living tissue. Several methods have been proposed to obtain motion-free images, yet they are generally limited in their applicability due to long acquisition times, requiring multiple volumes [START_REF] Lezama | Segmentation guided registration of wide field-of-view retinal optical coherence tomography volumes[END_REF], and/or the need for additional hardware [START_REF] Larocca | Handheld simultaneous scanning laser ophthalmoscopy and optical coherence tomography system[END_REF]. Here we propose and analyze a motion-compensated 3D-OCT imaging system that uses a higher-order regression analysis and show that it can effectively correct the motion artifacts within 0 to 5 Hz in real time without requiring additional hardware.

Method

The scanning protocol of the proposed motion-compensated 3D-OCT system is shown in Figure 1. Three reference images in the slow-axis are utilized to obtain reference images within the C-Scan volume and compensate for the motion artifacts. Note that mainly axial motion is considered in this analysis, such as the one induced by involuntary vascular pulsation and respiration. We define 𝑑 as the distance between adjacent references. 𝐿 is the width of the B-scan raw image. 𝛼 is defined as the ratio between 𝑑 and 𝐿. The system is implemented in an in-house built 100kHz swept-source OCT centered at 1060nm with 110 nm tuning range.

The flowchart of the system algorithm is shown in Figure 2. The algorithm consists of two main steps, namely preprocessing and motion compensation. First, we apply a gaussian filter to every B-Scan of the raw C-scan data and remove the saturation artifacts introduced by light reflected from a highly specular surface generating signals that are over the dynamic range of the data-acquisition system. We simplified the saturation removal method based on minimum variance mean-line subtraction (MVMS) [START_REF] Moon | Reference spectrum extraction and fixed-pattern noise removal in optical coherence tomography[END_REF][START_REF] Huang | Real-time reference A-line subtraction and saturation artifact removal using graphics processing unit for high-frame-rate Fourier-domain optical coherence tomography video imaging[END_REF], to reduce the computation cost. Specifically, the A-line value of position (x, y) will be assigned as subtraction of the mean value of raw data 𝐴 Ω (𝑥, 𝑦) from 𝐴 Ω (𝑥, 𝑦). After thresholding and the connect component analysis, we remove any noisy portion, including tissue inside the cornea, then generate the initial segmentation. We revise the surface using the pixel with the largest gradient around the initial segmentation and fit it with a polynomial curve. To test this preprocessing algorithm, we applied it to clinical OCT images of corneas with normal and abnormal surface topography (e.g. from bullous keratopathy and keratoconus). As shown in Figure 3, the results indicate that the 4th order polynomial is accurate enough to represent the topological surface of both the healthy and pathological corneas. We compare the B-scan surface detection result with the three reference images and obtain the motion during the three intersecting planes. Then, the movement of every A-scan within the B-scan is predicted and compensated by the polynomial fitting.

Results

Because most involuntary motion frequency of the human body ranges from 0 to 5Hz [5], we explored the movement over this frequency range in this study. The Root Mean Square Error (RMSE) is used as our quantitative metric of the result, to measure the difference between the reference and compensated sample in micrometers. To test the performance of the system and the algorithm, we compensated for the motion around 1Hz using different order polynomial fittings with four cornea samples as shown in Figure 5. The result shows that 4th order fitting achieves the best fitting accuracy, with minimal RMSE.

The compensation result could be affected by the distance between the adjacent reference planes, which is defined as 𝛼. Since the middle reference is fixed in the center of the x-direction, 𝛼 can range from 0 to 0.5. Figure 6 shows that 𝛼 around 0.37 achieves the best result, that is, a minimal RMSE, below around 2.6 𝜇𝑚.

In conclusion, we propose and analyze a motion-compensated 3D-OCT system and show that it can compensate motion over the frequency range from 0 to 5 Hz effectively with less than 2.6 𝜇𝑚 error. Implementing such motion compensation into clinical OCT systems has the potential to improve the reliability of 3D imaging and reduce associated computational cost.

Figure 1 .

 1 Figure 1. Schematic of the scanning protocol of the motion-compensated 3D-OCT system.

Figure 2

 2 Figure 2 Flowchart of system algorithm, with (left) pre-processing (a through d) and (right) motion-compensation steps.

Figure 3 .

 3 Figure 3. Polynomial fitting results of corneal boundary. a), b), and c) shows the different polynomial order fitting results and corresponding fitting error d), e) and f) of the normal cornea, cornea with bullous keratopathy, and cornea with keratoconus, respectively.

Figure 4 .

 4 a)-c) shows an example of the motion-compensation result during corneal C-scan volume imaging with axial movement around 1Hz. As another example, Figure 4. d)-f) shows volumetric imaging of a phantom with motion around 3.6Hz. The algorithm successfully corrects the motion artifacts in these two frequency settings.

Figure 4 .

 4 Figure 4. Motion-compensation result of (top) cornea and (bottom) phantom C-scan volume. a) is the static reference of the cornea, b) is the cornea with motion around 1Hz. c) is the reconstructed corneal image after motion compensation. The red line in b) and blue line in c) represents the B-scan location, representing the corneal surface before and after compensation, respectively. Both are also shown in the inset of c), which provides a direct comparison between pre-and post-compensation. d), e) and f) are the static reference of the phantom, phantom with motion around 3.6Hz, and reconstruction of phantom after motion compensation.

Proc. of SPIE Vol. 11630 116301N-1 Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 27 Mar 2021 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

Proc. of SPIE Vol. 11630 116301N-2 Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 27 Mar 2021 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

Proc. of SPIE Vol. 11630 116301N-3 Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 27 Mar 2021 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use