Member, IEEE Yiming Xiao

Member, IEEE Hassan Rivaz
email: hrivaz@ece.concordia.ca

Matthieu Chabanas
email: matthieu.chabanas@univ-grenoble-alpes.fr

Maryse Fortin
email: maryse.fortin@concordia.ca

Ines Machado

Yangming Ou

Mattias P Heinrich

Julia A Schnabel

Xia Zhong

Andreas Maier

X Zhang

Roozbeh Shams

Samuel Kadoury

David Drobny

Wolfgang Wein

Marc Modat

Ingerid Reinertsen
email: ingerid.reinertsen@sintef.no

Evaluation of MRI to Ultrasound Registration Methods for Brain Shift Correction: The CuRIOUS2018 Challenge

Keywords: Registration, brain, ultrasound, MRI, brain shift, tumor I

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

tumors. Evidences [START_REF] Jakola | Comparison of a strategy favoring early surgical resection vs a strategy favoring watchful waiting in low-grade gliomas[END_REF][START_REF] Schomas | Intracranial low-grade gliomas in adults: 30-year experience with long-term follow-up at Mayo Clinic[END_REF] have shown that early tumor resection can effectively improve the patient's survival rate. Imageguidance can be a useful tool to assist the surgeon in obtaining a maximal safe resection of the tumor. Image-guidance based on pre-operative MR images is in routine clinical use worldwide. These systems, however, do not account for the tissue shift and deformations that occur as the resection progresses. Due to brain shift, the surgical target and other vital structures (e.g., blood vessels and ventricles) will be displaced relative to the pre-surgical plan and resulting in inaccurate image-guidance. Multiple factors can contribute to brain shift, including but not limited to drug administration, intracranial pressure change, tissue resection. Often such tissue shift is not directly visible by the surgeon. Both intra-operative ultrasound (iUS) and intra-operative magnetic resonance imaging (iMRI) have been employed to track tissue deformation and surgical progress. Intra-operative US has gained popularity thanks to its low cost, high portability and flexibility. However, limited field of view and challenging image interpretation remain obstacles for widespread use. Together with iUS, automatic image registration algorithms can be used to update the surgical plan based on pre-operative MRI by re-aligning the pre-operative images with intra-operative images and offer more intuitive assessments of the extent of resection.

Previously, a number of algorithms and strategies [START_REF] Arbel | Automatic nonlinear MRI-ultrasound registration for the correction of intra-operative brain deformations[END_REF][START_REF] Reinertsen | Clinical validation of vessel-based registration for correction of brain-shift[END_REF][START_REF] Coupe | 3D Rigid Registration of Intraoperative Ultrasound and Preoperative MR Brain Images Based on Hyperechogenic Structures[END_REF][START_REF] De Nigris | Multi-modal image registration based on gradient orientations of minimal uncertainty[END_REF][START_REF] Wein | Global Registration of Ultrasound to MRI Using the LC2 Metric for Enabling Neurosurgical Guidance[END_REF][START_REF] Rivaz | Automatic Deformable MR-Ultrasound Registration for Image-Guided Neurosurgery[END_REF] have been developed to address iUS-MRI registration for brain shift correction. They range from new strategies to map image features to similar domains [START_REF] Arbel | Automatic nonlinear MRI-ultrasound registration for the correction of intra-operative brain deformations[END_REF][START_REF] Reinertsen | Clinical validation of vessel-based registration for correction of brain-shift[END_REF] to novel cost function [START_REF] De Nigris | Multi-modal image registration based on gradient orientations of minimal uncertainty[END_REF][START_REF] Wein | Global Registration of Ultrasound to MRI Using the LC2 Metric for Enabling Neurosurgical Guidance[END_REF], and from different deformation models [START_REF] Coupe | 3D Rigid Registration of Intraoperative Ultrasound and Preoperative MR Brain Images Based on Hyperechogenic Structures[END_REF][START_REF] De Nigris | Multi-modal image registration based on gradient orientations of minimal uncertainty[END_REF] to improved G optimization procedures [START_REF] Rivaz | Automatic Deformable MR-Ultrasound Registration for Image-Guided Neurosurgery[END_REF]. However, partially due to the lack of relevant clinical datasets, it has been difficult to directly compare different algorithms, thus potentially slowing the speed of technical translation to benefit surgeons and patients. The MICCAI Challenge 2018 for Correction of Brain shift with Intra-Operative UltraSound (CuRIOUS2018) was launched as the first public platform to benchmark the latest image registration algorithms for the task, and to bring the researchers together to discuss the technical and clinical challenges in iUSguided brain tumor resection. For the first edition of the challenge, we focused on MRI-iUS registration to correct preresection deformation after craniotomy, as it typically sets the tone of brain shift for the rest of the surgery.

The challenge was divided into the training and testing phases. While the publicly available REtroSpective Evaluation of Cerebral Tumors (RESECT) database [START_REF] Xiao | REtroSpective Evaluation of Cerebral Tumors (RESECT): A clinical database of pre-operative MRI and intra-operative ultrasound in lowgrade glioma surgeries[END_REF] was used in the training phase for algorithm development, in the testing phase, a private testing database was distributed to assess the participating teams. The distances between homologous anatomical landmarks between iUS and MRI were used to assess and rank the registration quality. The CuRIOUS2018 challenge received 8 initial submissions [START_REF] Hong | Non-linear Approach for MRI to intra-operative US Registration Using Structural Skeleton: International Workshops[END_REF][START_REF] Sun | Deformable MRI-Ultrasound Registration Using 3D Convolutional Neural Network: International Workshops[END_REF][START_REF] Drobny | Registration of MRI and iUS Data to Compensate Brain Shift Using a Symmetric Block-Matching Based Approach: International Workshops[END_REF][START_REF] Machado | Deformable MRI-Ultrasound Registration via Attribute Matching and Mutual-Saliency Weighting for Image-Guided[END_REF][START_REF] Heinrich | Intra-operative Ultrasound to MRI Fusion with a Public Multimodal Discrete Registration Tool: International Workshops[END_REF][START_REF] Shams | Intra-operative Brain Shift Correction with Weighted Locally Linear Correlations of 3DUS and MRI: International Workshops[END_REF][START_REF] Wein | Brain-Shift Correction with Image-Based Registration and Landmark Accuracy Evaluation: International Workshops[END_REF][START_REF] Zhong | Resolve Intraoperative Brain Shift as Imitation Game: International Workshops[END_REF]. Seven teams validated their methods on the testing data, and six participated in the final ranking. The submissions cover a wide variety of approaches, including the latest registration metrics [START_REF] Wein | Global Registration of Ultrasound to MRI Using the LC2 Metric for Enabling Neurosurgical Guidance[END_REF][START_REF] Machado | Deformable MRI-Ultrasound Registration via Attribute Matching and Mutual-Saliency Weighting for Image-Guided[END_REF], optimization approaches [START_REF] Drobny | Registration of MRI and iUS Data to Compensate Brain Shift Using a Symmetric Block-Matching Based Approach: International Workshops[END_REF], and deep learning techniques [START_REF] Zhong | Resolve Intraoperative Brain Shift as Imitation Game: International Workshops[END_REF].

This paper describes the organization, submitted algorithms, and results for the challenge, and further discusses the current challenges and potential future directions of tissue shift correction in US-guided brain tumor surgery.

II. MATERIALS

Two datasets were included for the training and testing phases of the CuRIOUS2018 challenge. The RESECT database [START_REF] Xiao | REtroSpective Evaluation of Cerebral Tumors (RESECT): A clinical database of pre-operative MRI and intra-operative ultrasound in lowgrade glioma surgeries[END_REF] was provided to the participants as the training dataset for development and fine-tuning of the algorithms. The database contains pre-operative MR and pre-resection iUS images from 22 patients who have received LGG resection surgeries at St. Olavs University Hospital, Trondheim, Norway. The testing dataset was comprised of imaging data from 10 additional patients with LGG obtained in the same setting as the RESECT database. The collection and distribution of both datasets were approved by the Regional Committee for Medical and Health Research Ethics of Central Norway, and all patients signed written informed consent.

For both training and testing databases, Gd-enhanced T1w MRI and T2w fluid-attenuated inversion recovery (FLAIR) MRI scans were acquired for each patient before surgery. Five fiducial markers were glued to the patient's head prior to scanning. The T1w and T2w MRIs were rigidly co-registered, and aligned to the patient's head position on the operating table via a fiducial-based image-to-patient registration. The positiontracked 3D iUS scans were acquired with the Sonowand Invite neuronavigation system (Sonowand AS, Trondheim, Norway), with either the 12FLA-L linear transducer or the 12FLA flat linear array transducer for smaller superficial tumors. 3D volumes were reconstructed from the raw iUS data using the built-in proprietary reconstruction method in the Sonowand Invite system, with a reconstruction resolution in the range of 0.14x0.14x0.14 mm 3 to 0.24x0.24x0.24 mm 3 depending on the probe types and imaging depth. Both ultrasound transducers were factory calibrated and equipped with removable sterilizable reference frames for optical tracking. A Polaris camera (NDI, Waterloo, Canada) built in the Sonowand system was used to obtain the position and pose of the ultrasound probe. Therefore, the iUS volumes reveal tissue position and deformation in the patient's head on the operating table. The number of landmarks and mean initial Euclidean distances between landmark pairs are shown, and the range (min ~ max) of the distances is shown in parenthesis after the mean value.

Homologous anatomical landmarks manually labeled by two raters (authors YX and MF as Rater 1 and 2, respectively) were provided to assess registration quality, using the software 'register' included in the MINC toolkit (http://bicmni.github.io). Typical landmarks include the edge of the tumor, deep grooves of sulci, corners of sulci, convex points of gyri and the horns of the lateral ventricles. After Rater 1 defined the landmarks in the T2w FLAIR MRIs as the references, Rater 1 and Rater 2 then tagged the corresponding landmarks independently within the corresponding US volumes twice. A 1~2-week interval was ensured between the repetitions. The final landmarks in both training and testing database were provided as the averaged results of two trials of landmark marked by both raters (four 3D points for each landmark). The details of the landmarks are listed in Table I for the testing datasets. Similar details for the training dataset can be found in the original publication for the RESECT database [START_REF] Xiao | REtroSpective Evaluation of Cerebral Tumors (RESECT): A clinical database of pre-operative MRI and intra-operative ultrasound in lowgrade glioma surgeries[END_REF]. For both sets, a wide range of brain shifts measured as mean initial distances between corresponding landmarks were included to properly examine the performance of registration algorithms.

We employed the mean Euclidean distance between two sets of corresponding landmark points for each patient to assess the intra-and inter-rater variability. For intra-rater variability, we calculated the metric between two trials of landmark picking for each rater; for inter-rater variability, the average of two trials for each rater was first computed and used to obtain the value between two raters. The intra-and inter-rater variability evaluations are presented in Table II

III. CHALLENGE SETUP

The CuRIOUS2018 challenge started on April 1 st , 2018 when the challenge website went live on curious2018.grandchallenge.org. In the next few days, several groups who were active in the field of MRI-iUS registration were identified by literature search and were invited to participate. The challenge was also widely advertised on mailing lists and on bulletin board of medical imaging conferences held in the first half of 2018. Another factor that leads to a good participation was the incentive of generous support of challenge sponsors, which provided a total of 2,100 € for the top three winners. The challenge consisted of two phases (see Fig. I). In phase I, all the teams were required to submit a short paper that elaborated the technique and results on the 22 patients in the RESECT database. These papers were then peer-reviewed and the final camera-ready conference papers were submitted in July 2018.

Phase II started in August 2018, when all the participants who had submitted reports and results on the training data were provided with MRI and iUS data from 10 additional patients (test data). These datasets had identical data curation and annotation protocols as the RESECT database. The location of landmarks in the MRI was provided to the teams, and the teams had to return the locations of those landmarks after MRI-iUS registration within 13 days of the data release. All teams presented their methods and results on the training data at the challenge event, which took place in conjunction to MICCAI 2018 in Granada, Spain.

The RESECT database remains public, and has been downloaded 267 times since its release in April 2017. The test datasets were only released to the participants and the locations of the ground truth landmarks in these datasets remains private. The organizers will continue the challenge in 2019 by adding iUS test data collected during and after tumor resection.

IV. EVALUATION

The evaluation metric and ranking system are key criteria for the success of a challenge. The metric should reflect the overall quality of the methods and the ranking system should be as fair as possible. It is worth noting that our evaluation method was published on the official website before the challenge took place and was not modified afterwards. Although such transparency in the evaluation process may seem obvious, [START_REF] Reinke | How to Exploit Weaknesses in Biomedical Challenge Design and Organization[END_REF] reported that this transparency was not guaranteed in about 40% of biomedical challenges, which could lead to controversy. The first component of the evaluation process is a metric to assess the quality of the registration methods. More than 80% of the tasks in biomedical challenges concern segmentation, with the Dice similarity coefficient as the most common evaluation metric [START_REF] Reinke | How to Exploit Weaknesses in Biomedical Challenge Design and Organization[END_REF]. However, challenges with image registration, especially from different modalities, are rarer and we could not find any standard metrics from these competitions. We thus chose to rely solely on the expert-labeled anatomical landmark pairs, by computing the Euclidian distances between the transformed MRI landmarks, after registration, and the ground-truth landmarks defined in the iUS images.

The second component of the evaluation process concerns how the results for each test case are aggregated to rank the teams.

The two main options are 1) aggregate the results on all test cases, then rank; or 2) rank by test case, then aggregate the ranks. In the first scenario, we would have ranked the teams based on the mean distance computed from all landmarks of all cases. Instead, we chose the second scenario because it is better fitted to handle missing cases. For each case, we also ranked fully-automatic methods over semi-automatic methods. To aggregate the case-by-case ranks, we simply computed the mean rank of each team.

The evaluation system was as follows:

1. For each test case and for each team, compute the Euclidian distances between landmark pairs after registration, i.e. between the transformed MRI landmarks and the ground truth iUS landmarks.

3. Compute the mean rank of every team, which gives the final ranks of the Challenge.

V. CHALLENGE ENTRIES

A. Team cDRAMMS

Machado et al. [START_REF] Machado | Deformable MRI-Ultrasound Registration via Attribute Matching and Mutual-Saliency Weighting for Image-Guided[END_REF] extended the Deformable Registration via Attribute Matching and Mutual-Saliency Weighting (DRAMMS) algorithm [START_REF] Ou | DRAMMS: Deformable registration via attribute matching and mutual-saliency weighting[END_REF], a general-purpose algorithm [START_REF] Ou | Comparative evaluation of registration algorithms in different brain databases with varying difficulty: results and insights[END_REF], specifically for the US-MRI registration problem, which they termed as correlation-similarity DRAMMS or cDRAMMS.

They released it at https://www.nitrc.org/projects/dramms/ (version 1.5.1). The original DRAMMS has two good properties for US-MRI registration. First, representing each voxel with multi-scale and multi-orientation Gabor attributes in DRAMMS offers a richer information than purely image intensities. This helps to establish more reliable voxel correspondences despite the different image protocols and different intensity profiles between US and MRI images. Second, the mutual-saliency module in DRAMMS automatically assigns low confidence or weights to regions that cannot establish reliable or cannot find counterparts across images. This potentially reduces the negative effects of the missing correspondences between US and MRI images. Different from the original DRAMMS, which uses the sum of square differences (SSD) between attributes for matching, the modified cDRAMMS uses correlation coefficient [START_REF] Avants | Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain[END_REF] and correlation ratio [START_REF] Roche | The correlation ratio as a new similarity measure for multimodal image registration[END_REF] on attributes for voxel matching. CC and CR on voxel attributes in cDRAMMS establish voxel correspondences at a higher accuracy and higher reliability than SSD in DRAMMS. Free-form deformation and discrete optimization are chosen as the deformation and optimization strategy, respectively.

B. Team DeedsSSC

Heinrich et al. [START_REF] Heinrich | Intra-operative Ultrasound to MRI Fusion with a Public Multimodal Discrete Registration Tool: International Workshops[END_REF] used DeedsSSC, which comprises a linear and a non-rigid registration that are both based on discrete optimization and modality-invariant image Specifically, self-similarity context features (SSC) are extracted for both MRI and ultrasound scans that are matched based on a dense displacement sampling. First, the similarity maps for each considered control point are used to extract correspondences for fitting a linear transform using least trimmed squares, similar as done in block-matching approaches. Second, new similarity maps are calculated for linearly aligned images and an efficient graphical model based discrete optimization (deeds) is used to estimate a nonlinear displacement field that avoids implausible warps and further improves the registration quality. All computations are performed for scans resampled to isotropic 0.5 mm resolution and using the default parameters (see https://github.com/mattiaspaul/deedsBCV) with an optimization over multiple grid-scales. Finally, the nonlinearly warped landmarks are again constrained to follow a rigid 6parameter transform for improved robustness. The algorithm is executed within less than 10 seconds per scan pair on a multicore CPU and ongoing work considers the huge potential for further speed-ups through parallelized GPU computations.

C. Team FAX Zhong et al. [START_REF] Zhong | Resolve Intraoperative Brain Shift as Imitation Game: International Workshops[END_REF] proposed a learning-based approach to resolve intraoperative brain as an imitation game. This point-based approach predicts the deformation vectors of key points to compensate the non-rigid brain-shift. For each key point, they extract a local 3D patch in iUS and model the key point distribution as the encoding of the current observation. A demonstrator is constructed providing the optimal deformation vector based on the current key point location and the ground truth. An artificial neural network is trained to imitate the behavior of the demonstrator and to predict the optimal deformation vector given current observation. To increase robustness, the proposed technique uses a multi-tasking network with a rigid transformation as auxiliary output. In addition, we use a non-rigid deformation to augment the 3D volume and 3D key points to facilitate the training.

D. Team ImFusion

The method [START_REF] Wein | Brain-Shift Correction with Image-Based Registration and Landmark Accuracy Evaluation: International Workshops[END_REF] is based on the multi-modal similarity metric LC 2 [START_REF] Wein | Global Registration of Ultrasound to MRI Using the LC2 Metric for Enabling Neurosurgical Guidance[END_REF] and has recently been used in a first live evaluation during surgery [START_REF] Iversen | Automatic Intraoperative Correction of Brain Shift for Accurate Neuronavigation[END_REF] (data NOT overlapping with challenge data). The similarity metric was maximized using a non-linear optimization algorithm with a parametric transformation model. In a pre-processing step specific to the challenge data set (cartesian 3D ultrasound volumes compounded by the SonoWand system), the volume sides facing the ultrasound probe is estimated and the outermost 4mm of content are cropped accordingly. The registration algorithm is implemented using GPU acceleration. For registration, the US volume was employed as the fixed image, and was resampled to 0.5mm, which is half of the MRI resolution. The final selected patch-size for computing similarity metric is 7×7×7 voxels, which was optimized in their prior work [START_REF] Iversen | Automatic Intraoperative Correction of Brain Shift for Accurate Neuronavigation[END_REF]. The algorithm employs a two-stage non-linear optimization strategy that successively operates on the transformation parameters. In the first stage, a global DIRECT (DIviding RECTangles) subdivision method [START_REF] Jones | Lipschitzian optimization without the Lipschitz constant[END_REF] searches on translation only, and is followed by a local BOBYQA (Bound Optimization BY Quadratic Approximation) algorithm [START_REF] Powell | The BOBYQA Algorithm for Bound Constrained Optimization without Derivatives[END_REF] on all six rigid transformation parameters. Afterwards, the local optimizer conducts another search on full affine parameters in order to accommodate shearing and non-uniform scaling of the data.

E. Team MediCAL

Multimodal deformable registration between the MRI and intraoperative 3DUS was achieved with a weighted version of the locally linear correlation metric (LC 2 ), correlating MRI intensities and gradients with ultrasound, while adapting both hyper-echoic and hypo-echoic regions within the cortex. The method [START_REF] Shams | Intra-operative Brain Shift Correction with Weighted Locally Linear Correlations of 3DUS and MRI: International Workshops[END_REF] was initialized with a global rotation of the US volume to match the orientation observed on the MRI. This was achieved using a PCA of the extracted inferior skull region, identifying the principal orientation vectors of the head, followed by a scaling and translation correction. This fusion step uses a patch-based approach of the US voxels, comparing intensity and gradient magnitudes extracted from the MRI with a linear relationship. The registration applies sequentially a rigid and non-rigid step, with the later integrating a weighting term and controlled by a cubic 5×5×5 B-Spline interpolation grid, distributed uniformly in the fan-shaped US volume. The weighting term uses pre-annotated labels on the MRI, representing both the hypoechoic (fluid cavities) and the hyperechoic (ex. choroid plexus) areas observed on ultrasound. This term is added only at the non-rigid step as it is highly specific to the internal areas in US such as the lateral ventricles, requiring a rigid pre-alignment. Registration optimization was performed using BOBYQA, which avoids computing the metric's derivatives.

F. Team NiftyReg

Drobny et al. [START_REF] Drobny | Registration of MRI and iUS Data to Compensate Brain Shift Using a Symmetric Block-Matching Based Approach: International Workshops[END_REF] suggest a method which uses a blockmatching approach to automatically align the pre-operative MRI with the iUS image. The registration algorithm is part of the NiftyReg open-source software package [START_REF] Modat | Global image registration using a symmetric block-matching approach[END_REF]. Their blockmatching registration technique iteratively establishes point correspondences between the fixed image and the warped floating image, and finally determines the transformation parameters through least trimmed squares (LTS) regression. A two-level pyramidal strategy was used for coarse-to-fine image registration. For block-matching, both fixed and floating images were divided into uniform blocks of 4 voxel edge length. The top 25% of blocks with the highest intensity variance in the fixed image were used while the rest were discarded. Each of these image blocks was then compared to all floating image blocks that have an overlap of at least one voxel. The best match between the blocks from the fixed and floating images was determined as the pair with maximum absolute normalized cross-correlation (NCC). After establishing the point-wise correspondences, the second step was the update of transformation parameters via LTS regression. At every iteration, the composition of the block-matching correspondence and the transformation of the previous step determined the new transformation by LTS regression. For each team, the first columns give the mean distances between landmark pairs after registration, computed over all landmark of all cases, for the training and test sets. The mean case-by-case rank, computed on the test set only, and the final challenge rank are then given. For comparison, the last line contains the mean initial distances, before registration. Teams cDRAMMS and NiftyReg were eventually ranked tied at third (=). Hong et al. sent results on the test data but did not attend the challenge event. Sun et al. sent only partial results (*) on the training set, but did not participate to the second phase of the challenge. These two teams were thus not ranked.

VI. RESULTS

A. Phase I: distances on the training data

The results obtained on the training dataset were reported by each team in their respective contribution to the challenge proceedings [START_REF] Stoyanov | Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation[END_REF]. Most authors reported their distances obtained after registration for each case, although some reported only averaged values. Table III summarizes the mean distance between landmark pairs after registration, over all landmark of all cases, computed by each team. All teams but one improved from the initial distances, with three teams achieving a mean distance under 1.75 mm and two more under 3.35 mm. Team cDRAMMS initial reported a mean distance between landmarks of 3.35 ± 1.39mm. With an updated version of their method, this error was later reduced to 2.28 ± 0.71mm. Sun et al. [START_REF] Sun | Deformable MRI-Ultrasound Registration Using 3D Convolutional Neural Network: International Workshops[END_REF] provided partial results on 4 cases only, since the other 18 cases were used to train their neural network. This team eventually did not participate to the second phase.

B. Phase II: distances on the test data

This section presents the results of the 6 teams that completed phase II of the challenge, on the test dataset. Figure II first shows the results per test case, aggregated across all teams. Test cases with the largest initial error (cases 1, 5, and 3) were the most difficult to treat. Results for test cases with the smallest initial error (4 and 9) were in average improved, also several teams obtained larger distances after registration. Finally, results were consistently improved for all other cases with an initial error in the 4-6 mm range. Regarding team-by-team results, mean distances between landmark pairs after registration are summarized in Table III while the distribution of these distances is detailed in Figure III. Teams ImFusion and DeedsSSC obtained a mean distance between landmark pairs well below 2 mm, respectively of 1.57 and 1.87 mm. These excellent results are consistent across all test cases, with a standard deviation around 1 mm for both teams, which confirmed the results reported on the training data set. Team cDRAMMS also consistently obtained very good results, with a mean error of 2.18 mm and a single large residual error of 4.3 mm for case 5. Results of team NiftyReg are more contrasted. As can be seen on the lower panel of Figure III, they obtained excellent results for all cases but two, cases 1 and 5, where the distance was only reduced from 15.7 to 5.9 mm and 13.2 to 12.8 mm, respectively. Without these two outliers, the mean distance over all cases would be reduced from 3.21 ± 3.57 mm to 1.70 ± 0.91 mm. Team FAX reported the best results on the training set, with a mean distance between landmark pairs of 1.21 ± 0.55 mm. However, this distance leaped to 5.70 ± 0.55 mm on the test data, which potentially shows their deep learning method overfitted the data during the training phase. Finally, team MedICAL obtained few or no improvements from the initial distances between landmark pairs.

C. Phase II: complementary criteria

All methods were fully automatic. Although it was not a factor in the evaluation, the average computational time per case for each team is reported in Table IV, and each team's choice of software implementation and hardware set-up vary. These values range from 1.8 sec for team FAX with a trained neural network to approximately 450 sec for team cDRAMMS using singe-thread CPU implementation. To demonstrate the data and the registration task, MR and iUS volumes of one patient was chosen from each of the training and test datasets, and are shown in Figures IV andV, respectively. The selected cases have a relatively large initial mTRE and a substantial variability between the teams. Also, note that these two cases do not necessarily directly reflect the overall ranking of the challenge, which was based on averaged rankings of all cases. As no quantitative measures of registration quality are available in a clinical setting, visual inspection of the images is important to obtain an impression of the registration quality. As shown in Figures IV andV, the registration accuracy can be evaluated by adapted visualization and identification of homologous features such as sulci, gyri and ventricles in the images.

E. CuRIOUS2018 Challenge ranks

Following the description in Section IV, all teams were ranked independently for each test case, based on the mean distances between landmark pairs. These case-by-case ranks are summarized in Figure VI, with the number of times each team was ranked at the i th place, with i from 1 to 6.

The winner and runner-up are teams ImFusion and DeedsSSC, which are perfectly consistent with their respective results reported in Figure III. Note that ImFusion obtained the best registration for 6 of the 10 tests cases. Despite a larger mean registration error, team NiftyReg was ranked third before team cDRAMMS as it obtained a better case-by-case rank (3.1 vs 3.4). However, team cDRAMMS also had very good results but consistently handled all cases, including the extreme ones. This specific situation pointed out the fact that the challenge metric favors accuracy over precision, with a limited penalty when low quality results are obtained on a single case or two. To overcome this limit, as we consider precision is a crucial factor for the surgeons' acceptance of a method, the challenge's organizers decided to declare a tie for third place. Both NiftyReg and cDRAMMS thus received the same third place prize. Finally, both teams FAX and MedICAL obtained a mean case-by-case rank of 5.3, and were ranked tied at the 5 th place in the challenge.

VII. DISCUSSION

In this challenge, the focus has been on MR-iUS registration in the context of brain tumor surgery. As both the training and test datasets exclusively contain data from LGG surgeries, there has been a special focus on this tumor type. The resection of LGGs is particularly challenging as the tumor tissue can be very similar to normal brain tissue. In LGG surgery, there are also fewer options for additional guidance as tools like 5-ALA fluorescence are not available. Intraoperative ultrasound is therefore an attractive solution in these cases. The optimization and benchmarking of available registration algorithms on data from these tumors is therefore particularly important for successful future clinical translation. Even though the emphasis has been on LGG, the results from the challenge will generalize well to other tumor types such as HGGs and metastasis as these tumors are more distinct from normal brain tissue and depict clearer boundaries than LGGs in ultrasound images.

An important obstacle for the widespread use of iUS is the challenging and unfamiliar image interpretation. The integration of iUS into the navigation system and the visualization of corresponding slices in pre-operative MR and iUS makes this interpretation considerably more intuitive. With accurate MR-iUS registration, the surgeon can perform the resection based on the MR images even after brain shift which makes the neuronavigation accurate and easy to interpret. MR-iUS registration also enables correction of other types of preoperative MR data such as fMRI and DTI [START_REF] Xiao | Nonlinear deformation of tractography in ultrasound-guided low-grade gliomas resection[END_REF].

Image registration techniques tailored for MRI-iUS registration in this challenge were landmark-, intensity-or learning-based. The performance of landmark-based methods in non-linear image registration depends on both finding enough landmarks that cover the entire volume, and correctly finding their corresponding landmarks in the second volume. The voxel-wise attribute-based method of Machado et al. [START_REF] Machado | Deformable MRI-Ultrasound Registration via Attribute Matching and Mutual-Saliency Weighting for Image-Guided[END_REF] (team cDRAMMS) did relatively well despite the fact that iUS and MRI have drastically different salient features, and ranked third in a tie with Drobny et al. [START_REF] Drobny | Registration of MRI and iUS Data to Compensate Brain Shift Using a Symmetric Block-Matching Based Approach: International Workshops[END_REF] (team NiftyReg). The top three algorithms in this challenge [START_REF] Drobny | Registration of MRI and iUS Data to Compensate Brain Shift Using a Symmetric Block-Matching Based Approach: International Workshops[END_REF][START_REF] Machado | Deformable MRI-Ultrasound Registration via Attribute Matching and Mutual-Saliency Weighting for Image-Guided[END_REF][START_REF] Heinrich | Intra-operative Ultrasound to MRI Fusion with a Public Multimodal Discrete Registration Tool: International Workshops[END_REF] were all intensity-based techniques, which calculated a dense transformation map by utilizing intensity values at all locations.

Deep learning has been successfully applied to image registration [START_REF] Shun | A CNN Regression Approach for Real-Time 2D/3D Registration[END_REF][START_REF] Hou | 3-D Reconstruction in Canonical Co-Ordinate Space From Arbitrarily Oriented 2-D Images[END_REF], but in nonlinear image registration tasks with high accuracy requirement, such as brain shift correction, further exploration is still needed [START_REF] Litjens | A survey on deep learning in medical image analysis[END_REF]. The two submissions that used DL in this challenge were from Sun et al. [START_REF] Sun | Deformable MRI-Ultrasound Registration Using 3D Convolutional Neural Network: International Workshops[END_REF], who did not participate to the second phase, and Zhong et al. [START_REF] Zhong | Resolve Intraoperative Brain Shift as Imitation Game: International Workshops[END_REF] (team FAX), who ranked first in the results reported on the training database. However, their method did not work well on the test database. A common culprit for such behavior is overfitting, where the model overfits the training data and therefore performs poorly on the unseen test data. As more training data becomes available, this method is expected to perform better in the future.

Symmetric image registration techniques provide unbiased estimates of the transformation field and are known to generally outperform their asymmetric counterparts [START_REF] Avants | A reproducible evaluation of ANTs similarity metric performance in brain image registration[END_REF]. Two of the top three methods in this challenge [START_REF] Drobny | Registration of MRI and iUS Data to Compensate Brain Shift Using a Symmetric Block-Matching Based Approach: International Workshops[END_REF][START_REF] Heinrich | Intra-operative Ultrasound to MRI Fusion with a Public Multimodal Discrete Registration Tool: International Workshops[END_REF] compared the performance of their techniques in symmetric and asymmetric settings. They both concluded that asymmetric transformations lead to a superior performance in this challenge. This is an interesting finding and is likely due to the vast differences in physics of US and MR imaging modalities.

It was also noticed by some of the challenge participants and discussed during the event that in some cases affine transformations outperformed non-linear elastic transformations. This might seem surprising as brain shift is often described as a non-uniform deformation. However, before resection a large component of the experienced mismatch between MRI and iUS is often due to inaccurate patient-MRI registration. This is a rigid registration most often based on anatomical landmarks, fiducials, surfaces or a combination of these. Consequently, an affine transformation might be sufficient to correct for most of the misalignment. After resection, the situation will be different with larger and highly non-linear deformations and affine transformations will likely not be sufficient to register the images. Some teams initialize their nonlinear registration with affine alignments first. This approach can ensure the overall registration robustness when only focusing on relatively local tissue deformation. However, in cases of updating other intra-operative data, such as preoperative tractography [START_REF] Xiao | Nonlinear deformation of tractography in ultrasound-guided low-grade gliomas resection[END_REF], direct estimation of nonlinear deformation will be preferred.

In both the training and test databases, we selected landmarks that cover a large part of the iUS volume with maximal distance between neighboring landmarks. This strategy provides a good benchmark for comparison of image registration techniques. However, the quality of the alignment closer to the tumor is more clinically important as it better helps the neurosurgeon to optimize the resection size and location. To further investigate the registration performance, registration errors per landmark for each case are analyzed for the testing dataset. More specifically, the registration errors per landmark across all teams for each case were plotted in Figure VII. As expected, a higher variance of registration errors was observed for Case 1 & 5 with large brain shift (>10mm). Case 12 (initial mTRE=4.89mm) also exhibits a relatively high variance in registration errors. This is likely due to blurry sulci patterns in MRI in comparison to iUS. Furthermore, we also computed the medians, minimums, and variances of the registration errors per landmark across all teams for each clinical case from the testing dataset, and the results are show in Figure VIII. With Tukey's method, we identified outlier landmarks based on the computed statistical measures. In general, there are only very small amount of outlier landmarks (<10 out of 173). While the minimums and medians generally point to the same outlier landmarks, variances did not show any. Based on visual inspection, these landmarks are typically anatomical features, which are close to the tissue and transducer interface and border of the iUS volume, as well as sulcus patterns that are less distinguishable in MRI scans in comparison to the iUS due to differences in resolution.

The distance between corresponding landmarks in the two images before and after registration is a well-established metric for evaluation of registration results in the absence of a ground truth. Despite being widely used, this metric has some limitations. As there is only a limited number of landmarks associated with each image, the registration error is only evaluated at a limited number of locations and will therefore not capture local displacements and deformations in other locations. The number of landmarks and their distribution in the image volume are therefore important. The landmarks in both the training and test sets have therefore been carefully placed in order to capture the displacements and deformations as good as possible. However, we noticed that in test case #5 the registration results were not accurate by visual inspection even though the mTREs indicated successful alignment. This emphasizes the need for both quantitative and qualitative assessment of registration results. Another limitation of this metric is the localization error associated with manual placement of points. Also, for the landmarks to be valid for evaluation of registration results, this localization error has to be significantly lower than the expected registration errors. We have measured the inter-and intra-rater variability in both the training and test data and shown that these are significantly lower than the registration errors. Even though the landmarks do not represent the absolute ground truth, they are valid for the evaluation of the registration results. The use of landmarks as the only metric for the challenge also represent a limitation as other important characteristics such as computation time are not measured. For implementation in a clinical setting, for example, other characteristics would also be of critical importance, such as registration precision, accurate mapping of tumor boundaries, smoothness of deformation field (no vanishing tissues), and ease of use. However, some of the factors can be very complex and difficult to quantify, and in the challenge setting, the use of a single well-defined metric is advantageous. A single metric enables a straightforward, comprehensible ranking scheme and an open, fair competition. With the use of multiple metrics, there will always be a discussion of the weighting of the different characteristics and how to aggregate the results. The rules for aggregation of the ranks in this challenge were outlined before the challenge and were not changed at any point. Still, the system used favors accuracy over precision. As discussed during the challenge event, for the clinical users, high precision and high accuracy are equally important and precision can even be more important than accuracy. This point should be re-designed and improved in future editions of the challenge.

VIII. FUTURE WORK

In the first edition, the registration task solely focused on MRI-iUS registration before dura-opening and after craniotomy. However, with the progress of tumor resection, tissue deformation is an on-going process, and accurate tracking can ensure the complete removal of cancerous tissues, preventing any additional surgeries. Intended as a recurrent open challenge to further improve the registration algorithms, we expect to introduce multiple sub-challenges in future CuRIOUS challenges to target brain shift correction at different stages of the surgery, especially during and after resection.

For clinical practices, besides accuracy and robustness, processing speed is an imperative factor. In the inaugural edition of the challenge, performance speed was not emphasized in scoring the teams because it can be affected by multiple factors, including implementation platforms, for prototype algorithms. In future challenges, we aim to place discussions and emphasis on this topic, as well as optimization algorithms to direct the results of the challenge towards more realistic clinical implementations.

IX. CONCLUSION

Holding great clinical values, MRI-iUS registration for correcting tissue shift in brain tumor resection is still a difficult task. As the first public image processing challenge to tackle this clinical problem, the CuRIOUS2018 Challenge provided a common platform to evaluate and discuss existing and emerging registration algorithms on this topic. The results of CuRIOUS2018 provided valuable insights for the current developments and challenges from both the technical and clinical perspectives. This is an important step forward to help translate research-grade automatic image processing into clinical practice to benefit the patients and clinicians.

Figure I .

 I Figure I. Timeline for CuRIOUS 2018 Challenge.

Figure

  Figure II. Results per test case: box plot distribution of the distances between landmark pairs. For each test case, the left box plot (blue) shows the initial distances before registration. while the right box plot (orange) shows the distribution after registration, aggregated over all teams.

Figure

  Figure III. Distribution of the distances between landmark pairs obtained by each team after registration, on the test set. For comparison, the last column contains the initial distances before registration. The upper panel shows the global results computed over all landmarks of all test cases. In the lower panel, these results are split by test case.

Figure

  Figure IV. Qualitative comparison of registration results for Training Case 25 across different teams. For each team, the ultrasound and deformed FLAIR MRI scan is overlaid together. The mTRE values for each team is listed at the right bottom corner of each image overlay. The arrows point to the sulcus patterns with varied registration quality among teams.

Figure V .

 V Figure V. Qualitative comparison of registration results for Test Case 3 across different teams. For each team, the ultrasound and deformed FLAIR MRI scan is overlaid together. The mTRE values for each team is listed at the right bottom corner of each image overlay. The arrows point to the sulcus patterns with varied registration quality among teams.

Figure

  Figure VI. Case-by-case ranks for each team. For example: over the ten test cases, team ImFusion was ranked first six times, second three times, and third one time.

Figure

  Figure VII. Registration errors per landmark across all teams shown as boxplots for each case from the testing dataset. The error is measured as the Euclidean distance between the deformed landmark in MRI and the corresponding one in ultrasound, and the landmarks are ordered from the left to the right for each case according to their number provided.

Figure

  Figure VIII. The medians, minimums, and variances of registration errors for each landmark across all teams, and the results are plotted as boxplots for each case from the testing dataset.

TABLE IV SUMMARY

 IV OF AVERAGE COMPUTATIONAL TIME PER CASE FOR EACH TEAM.

	Team	Mean computational	Implementation with
		time per case	GPU/CPU
	cDRAMMS	450 sec	CPU
	DeedsSSC	25 sec	CPU
	FAX	1.8 sec	CPU
	ImFusion	20 sec	GPU
	MedICAL	103 sec	CPU
	NiftyReg	115 sec	GPU

D. Qualitative results

For each test case, rank teams according to their mean distance between landmark pairs. Exceptions include:a. If one team could not provide results for a test case, or if these results could not be processed for any reason, then that team is ranked last for the test case. b. If two mean distances differ by less than 0.5 mm, a team with a fully-automatic method is ranked higher than a team with a semiautomatic method.

ACKNOWLEDGMENT The work is supported by Norwegian National Advisory Unit for Ultrasound and Image guided Therapy, NSERC Discovery RGPIN-2015-04136. The organizers would like to thank Medtronic and the French ANR within the Investissements d'Avenir program under references ANR-11-LABX-0004 (Labex CAMI) for their generous sponsorship. The work for the DeedsSSC algorithm was funded in part by the German Research Foundation (DFG) under grant number 320997906 HE 7364/2-1. Drobny et al. were supported by the UCL EPSRC Centre for Doctoral Training in Medical Imaging and Wellcome/EPSRC Centre for Interventional and Surgical Sciences [NS/A000050/1], and the Wellcome/EPSRC Centre for Medical Engineering [WT 203148/Z/16/Z] and EPSRC [NS/A000027/1].


Antoine Duparc

Mathieu Depetris

Laurent 1 Floch

Pascal Cauquil

Pascal Bach

Julien Lebranchu

DEVELOPMENT STATUS OF THE NEW TROPICAL TUNAS TREATMENT (T3) SOFTWARE

teaching and research institutions in France or abroad, or from public or private research centers.

Introduction

The multi-species nature of tropical tuna surface fisheries gives rise to a series of difficulties when estimating both catch by species and catch by size statistics. The T3 processing was built about 30 years ago in order to correct biases in the species composition reported in logbooks and to provide more accurate estimates of catch by species for the European purse seine fleet. However, the evolution of fishing practices and fishing stocks according to climate change have challenged the T3 methodology on some part of its processing. The different biases in the existing T3 processing were already presented and discussed [START_REF] Duparc | Assessment of accuracy in processing purse seine tropical tuna catches with the T3 methodology using French fleet data. Case of the French fleet in the Indian Ocean[END_REF]. In order to take these different biases into account, we developed a new statistical methodology [START_REF] Duparc | Assessment of the species composition of major tropical tunas in purse seine catches: a new modeling approach for the Tropical Tuna Treatment processing[END_REF]. The aim of this paper is to present the status of the new T3 implementation, which began in 2019 with a complete redesign of the code and the upcoming milestones.

A redesign for the T3 code

The original methodology developed in the Fortran language was previously described (Pallarés and Petit, 1998). The lack of a dedicated detailed documentation of the treatments performed was one of the major drawbacks. Reverse engineering work was carried out at the beginning of 2010 with the aim of replacing the tool with a documented program in a modern language, Java. A new version was developed reproducing the same methodology based on large sample areas, relying on T3 on the documentation of reverse engineering [START_REF] Cauquil | T3+ -Documentation des traitements -internal report[END_REF]. However, the version 2 of T3 could not succeed completely because it did not resolve all the biases of the first version.

Based on these facts, and in order to provide a functional version, we decided to completely review the program code.

The new version of T3 process is developed in R language and available through a package. One of the advantages of this language is that it is more understandable and manipulated by the fisheries' scientific community rather than Java language. Moreover, this development follows the principles of the open and reproducible science:

• Process transparency: the code is fully documented and the documentation of how works the program will be available. • Open access and Open source: the code source is hosted on a github repository (https://github.com/OB7-IRD/t3) under a GPL version 3 license. In the end, it would be available on the Comprehensive R Archive Network (CRAN).

• Referencing of the software : enable result reproducibility and versioning [START_REF] Depetris | OB7-IRD/t3: Beta version of T3 software[END_REF] : version 0.9.0 -DOI: 10.5281/zenodo.3878125.

Changes in the new version

The overall schema for the processing remains unchanged from the last version of T3 (fig. 1). As previously described [START_REF] Duparc | Assessment of accuracy in processing purse seine tropical tuna catches with the T3 methodology using French fleet data. Case of the French fleet in the Indian Ocean[END_REF], the first two steps are the standardization of the logbook reporting and the sampling data, and third step is the assessment of the species composition of the catches. Thus, the development of the T3 version aims at fixing, or reducing as much as possible, biases identified previously [START_REF] Duparc | Assessment of accuracy in processing purse seine tropical tuna catches with the T3 methodology using French fleet data. Case of the French fleet in the Indian Ocean[END_REF]. Upgrades of the process are summarized below. Step 1: Standardization of logbooks Step 2: Standardization of samples Step 3: Assessment of the species composition to non-sampled sets [START_REF] Duparc | Assessment of the species composition of major tropical tunas in purse seine catches: a new modeling approach for the Tropical Tuna Treatment processing[END_REF] accounting for:

○ Spatial and temporal structure of the fishery: location and date of the catch ○ Vessel specificity which impacts the species composition of the catch ○ Crew reporting information which enables more flexibility in the model when a marginal composition is caught ○ Several years of fishing in the training to increase the robustness of the estimates • Model Checking to ensure reliability of the results (see example in Appendix 1) • Computation of a confidence interval (bootstrap method) to quantify the accuracy of estimates of catch per species.

Planning

Initially, following the development of the new model, we established a six-month development program with the objective of presenting the new tool to the IRD partners, namely IEO 3 (Spain), CRODT 4 (Senegal) and SFA 5 (Seychelles) during the annual data preparatory meeting that takes place within the framework of the RCG LP 6 at the end of March 2020 (Fig. 2).

During this meeting, it was planned to test data from each partner in order to experiment the program and then to provide a first version in April to be presented at the next dedicated RFMOs meeting for the Atlantic Ocean (ICCAT) and the Indian Ocean (IOTC). 3).

3

Instituto Español de Oceanografía 4 Centre de Recherches Océanographiques de Dakar-Thiaroye 5 Seychelles Fishing Authority 6 Regional Coordination Group of Large Pelagic The main milestones up to October 2020 are:

• Make available on GitHub a release candidate of the package • Write the complete documentation (User guide and Dev guide)

• Analyse the historical data and publish the results

At this time, the beta version (0.9.0) is available [START_REF] Depetris | OB7-IRD/t3: Beta version of T3 software[END_REF] but the implementation of the new version is not fully achieved. Several tests remained to perform to check for the data traceability and to optimize the model parametrization. Then the last two decades could be revised and compared to the T3 v1 process. Historical data, prior to the 2000s, will require adaptation of the new model considering the partial availability of the information and the decrease of the data quantity.

Fig. 1 :

 1 Fig. 1: Activities diagram of the main step in T3 processing (source: Duparc et al. 2018). Sections corresponds to the paragraph in Duparc et al. (2018) which details the step.

Fig. 2 :

 2 Fig. 2: Gantt diagram of the initial work scheduleHowever, the COVID-19 pandemic has led to a significant delay in implementation due to the containment from March 17 in France. Although working remotely, the time available for the project was therefore impacted and we had to review the development program for the remaining steps (fig.3).

Fig. 3 :

 3 Fig. 3: Gantt diagram of revised work schedule

•

  Strict selection of samples used in the model training according to several objective criteria on fishing sets and wells: school type, location, date, sample quality. • Classification of unknown school type (no data available) by clustering according to major tuna species composition. • Development of a statistical model (random forest model described in

MARBEC, IRD, Univ Montpellier, CNRS, Ifremer, Sète, France

MARBEC, IRD, Univ Montpellier, CNRS, Ifremer, Abidjan, Côte d'Ivoire

Acknowledgments

We are grateful for all technical staff that collected data used in T3 processing and used in this paper. The data used were collected through the Data Collection Framework (Reg 2017/1004 and 2016/1251) funded by both IRD and the European Union.

Appendix 1: Model validity checking in the T3 processexample with the skipjack proportions in catches on school type associated with floating object

The model used was a random forest model with proportion of skipjack from landing samples as response variable and as explanatory variables: latitude, longitude, year, month, vessel id and skipjack proportion reported in the logbook (period: 2014 -2017, Nsample = 782). These graphs (Fig. A1-1) will be a part of the outputs of the T3 v3 which enable the evaluation of the model validity by users.


Mélanie Donnez

François-Xavier Carton

Florian Le Lann

Emmanuel De Schlichting

Matthieu Chabanas

Realistic synthesis of brain tumor resection ultrasound images with a Generative Adversarial Network

Keywords: Intraoperative Ultrasound, Image synthesis, Generative Adversarial Networks, Neurosurgery

The simulation of realistic ultrasound (US) images has many applications in image-guided surgery such as image registration, data augmentation, or educational purposes. In this paper we simulated intraoperative US images of the brain after tumor resection surgery. In a first stage, a Generative Adversarial Networks generated an US image with resection from a resection cavity map. While the cavity texture can be realistic, surrounding structures are usually not anatomically coherent. Thus, a second stage blended the generated cavity texture into a real patient-specific US image acquired before resection. A validation study on 68 images of 21 cases showed that three raters correctly identified 64% of all images. In particular, two neurosurgeons correctly labelled only 56% and 53% of the simulated images, which indicate that these synthesized images are hardly distinguishable from real post-resection US images.

INTRODUCTION

Intraoperative ultrasound (iUS) is commonly used during brain tumor resection to localize tissue and ensure than the tumor removal is as complete as possible. These iUS images can be used as is in neuronavigation systems, or could also be registered with preoperative Magnetic Resonance (MR) images to compensate for brain-shift deformation. [START_REF] Xiao | Evaluation of mri to ultrasound registration methods for brain shift correction: The curious2018 challenge[END_REF] The simulation of realistic iUS images, especially after tumor resection, is valuable in several applications such as surgical planning or medical training, to shorten the learning curve of young surgeons. [START_REF] Pigeau | Ultrasound image simulation with generative adversarial network[END_REF] Another potential application is the registration of preoperative MR images with intraoperative iUS images at the end of tumor resection, to guarantee that tumor tissue was optimally removed. Instead of multimodal registration, always a challenging task, one strategy is to simulate US images from the preoperative MR then perform monomodal registration between these simulated US and the actual iUS images. [START_REF] Mercier | Comparing two approaches to rigid registration of three-dimensional ultrasound and magnetic resonance images for neurosurgery[END_REF] However, this monomodal registration is still hindered by the fact that the resection cavity is visible in the real iUS images only. Simulating US images with a resection cavity, which would follow the tumor contours delineated in the MR, is expected to significantly simplify this registration problem. Finally, another targeted application is data augmentation to train deep neural networks. We recently proposed such a network to automatically segment resection cavities in iUS images. [START_REF] Carton | Automatic segmentation of brain tumor resections in intraoperative ultrasound images using u-net[END_REF] However, realistically simulating a large number of resection patterns would enable to enlarge the training set and thus improve the robustness of the segmentation network.

Many authors proposed to simulate the physics of ultrasonic waves propagation and backscattering on biological tissue, [START_REF] Mercier | Comparing two approaches to rigid registration of three-dimensional ultrasound and magnetic resonance images for neurosurgery[END_REF][START_REF] Bamber | Ultrasonic b-scanning: a computer simulation[END_REF][START_REF] Karamalis | Fast ultrasound image simulation using the westervelt equation[END_REF] typically to generate US images from CT or MR volumes. More recently, several groups used deep neural networks to simulate intravascular or kidney US images. [START_REF] Pigeau | Ultrasound image simulation with generative adversarial network[END_REF][START_REF] Tom | Simulating patho-realistic ultrasound images using deep generative networks with adversarial learning[END_REF] These works especially used Generative Adversarial Networks 8 (GAN) which consist in two neural networks: a generator learns while attempting to generate realistic ultrasound images, while a discriminator simultaneously learns while attempting to discriminate between real images and those simulated by the generator. These networks are typically trained with real US images along with segmented tissue-maps of anatomical structures. Other tissue-maps can then be input into the trained GAN to synthesize the corresponding US image.

In many contexts the same anatomical structures are visible in all images (for instance the kidney, spleen, and bones in Pigeau et al. [START_REF] Pigeau | Ultrasound image simulation with generative adversarial network[END_REF] ), which differ only due to anatomical variations and pathology, and image characteristics. A major challenge in our context is that intraoperative US images of the brain are extremely different from patient to patient: due to the variation of tumor locations, no common structures can be reliably found in the iUS images. Sulci are almost always present, but their number and shape also depend on the imaged area. For these reasons, learning the relations between iUS images and tissue-maps only seemed extremely challenging.

The proposed method simulates post-resection iUS images of the brain from two inputs: a patient-specific iUS image acquired before resection and a resection cavity map. This is realized in two stages: 1) a GAN network generates a pseudo iUS image from a resection cavity map. While the cavity texture can be realistic, surrounding structures usually do not correspond to coherent anatomical features; 2) the GAN-generated cavity texture is then merged into a real iUS image acquired before resection to simulate the final image. A validation study was carried out with three raters, including two neurosurgeons, to evaluate whether they could distinguish simulated images from real ones. In this study, we used cavity maps segmented from real post-resection iUS images, so that the synthesized images can be directly compared with these real images.

MATERIALS AND METHODS

Dataset

We used intraoperative ultrasound images of patients with low-grade gliomas from the public database RESECT. 9 21 of the 23 cases of the database could be processed in this study, with two 3D volumes per case: iUS before and iUS after acquired before and after tumor resection, respectively. Several major differences can be observed between these two volumes:

• the US probe position changed between the two acquisitions yielding to different fields of view, image textures (intensities, speckle, noise), and potentially different reconstruction artefacts. • the brain-shift phenomenon induces tissue deformation during surgery, due to several factors like gravity, loss of Cerebrospinal fluid, or drugs. Thus, anatomical structures in the two volumes do not exactly fit. • in the iUS after volume, the tumor was removed and a resection cavity is instead visible. This cavity can induce additional tissue deformation as well as artifacts in the images (typically a hyper-intense area at the bottom of the cavity and ultrasound shadows). The cavity borders could also be appear brighter due to bleeding.

To compensate for part of the brain-shift, iUS before and iUS after volumes were non-rigidly registered with thin plate splines between homologous landmarks available in the RESECT database. Overall, the mean distance between landmarks (i.e. the fiducial registration error) was reduced from 3.55 ± 1.76 to basically zero. 1857 2D images containing a tumor were extracted from these 3D volumes, and later used in this study.

To simulate complete tumor resections, we used the real resection cavity masks segmented from the iUS after images. [START_REF] Carton | Automatic segmentation of brain tumor resections in intraoperative ultrasound images using u-net[END_REF] Examples of registered iUS before and iUS after images as well as resection cavity masks are available in figure 3.

Stage I: GAN simulation of a resection cavity image

The pix2pix conditional GAN developed by Isola et al. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] was used as pictured in figure 1. The generator simulates an US image with resection from a resection cavity map, while the discriminator compares this image to the actual iUS after image to determine if its input is real or simulated. 15 of the 21 cases (1253 of the 1857 2D images) were used for training, plus 1 case (108 images) for validation. The remaining images were kept for testing.

After this first stage, the GAN-generated image contains a realistic resection cavity. However its surrounding looks like an iUS image except that the structures (sulci, ventricles...) may not be anatomically coherent.

Stage II: simulated image in a patient's context

The next stage was to blend the GAN-generated cavity image into a real iUS before image in order to get the cavity within a patient's anatomical features and thus simulate a realistic iUS image after resection.

Intensities of the two images to blend may significantly differ. Therefore, a first step was to normalize the GAN-generated resection image by equalizing its histogram to the histogram of the iUS before image. Histograms were computed on a region of interest (ROI) centered on the cavity, the ROI's size being twice the size of the cavity.

After this normalization, pixels of the simulated image were assigned as follows:

• if a pixel is located inside the cavity map, its intensity is taken from the GAN-generated resection image;

• if a pixel is outside the cavity map but at a distance d inferior to half the cavity size, the pixel intensity is linearly interpolated from the two images with a weighting factor depending on d (the closest image has more influence); • if a pixel is outside this blending area, its intensity is taken from the iUS before image.

Validation

Three raters participated in the validation study, a medical imaging expert (rater 1) and two neurosurgeons who have routinely used intraoperative US during tumor resection. Although none of the raters was involved in the methodological developments of this paper, raters 1 and 2 already worked on the RESECT database to manually segment resection cavities in iUS images. [START_REF] Carton | Automatic segmentation of brain tumor resections in intraoperative ultrasound images using u-net[END_REF] Despite this bias, these two raters were still included in the validation study as they worked on the images more than a year ago and as they never saw any simulated iUS image.

The first validation test was a form of Visual Turing Test used in Pigeau et al. [START_REF] Pigeau | Ultrasound image simulation with generative adversarial network[END_REF] to evaluate how well a person can differentiate between real and synthesized images, thus determining how convincing the synthesized images are. The dataset consisted in 68 2D images, 34 simulated and 34 real ones, selected from all cases and presented in a random order. Each rater had to label each image as simulated or real, and score his confidence in his choice on a scale of 1 to 5.

In a second test, images of 10 cases were presented with: 1) an initial iUS before image acquired before resection; 2) a resection cavity mask (segmented from the fourth image); 3) the image simulated from these inputs; 4) and the corresponding real iUS after image acquired after resection. Examples of these images are shown in figure 3. Note that the simulated and real iUS images after resection are always different, because the simulation was based on the image acquired before resection. The goal here was simply to qualitatively determine whether the simulated images were plausible, and the aspects that were correctly simulated or could be improved.

RESULTS AND DISCUSSION

Examples of GAN-generated resection and final simulated images are shown on figures 1 and 3, respectively. Results of the differentiation study are presented in table 1.

If real and simulated images were easy to distinguish, 100% of the images should be correctly labelled. Conversely, a score of 50% would mean that there is no way to discriminate real from synthesized images, suggesting simulated images appear realistic. Overall, 64% of the images were correctly labelled. Significant differences can be observed among raters: rater 1 who extensively manually segmented structures on the RESECT images could distinguish almost 80% of all images. However, only 56% and 53% of the simulated images were labelled as such by the two neurosurgeons which is an excellent result. Cases that were used in the training phase of the GAN network were more difficult to discriminate that test cases, which suggests the cavity appearance of new cases is slighlty less realistic. The qualitative analysis of 10 cases revealed that all raters were globally very satisfied with the simulated images. Several limits were nevertheless identified, mostly than the resection cavity margins can appear blurry when the texture of the GAN-generated cavity significantly differs from the iUS before texture. Also, images may lack variability since several features that could be observed in real images are never synthesized, like high intensity areas from blood clots on the resection cavity borders, or deep shadows below the cavity.

CONCLUSION

We simulated post-resection iUS images by generating an image of a resection cavity with a GAN network, then merging this image into a real patient-specific iUS image before resection. Despite blurriness on some cases, validation results showed than most simulated images were hardly distinguishable from real images. Future development will be to include the iUS before image as a GAN input, along with the cavity map, to generate resection images which intensity, texture, and speckle better match the patient image characteristics. Also, we plan to evaluate the simulation of iUS images from more anatomical maps of sulci, ventricles or the tumor, and directly from preoperative MR images.

Figure 1 .

 1 Figure 1. Stage I: GAN generation of an iUS with resection from a cavity map and real after-resection iUS images.

Figure 2 .

 2 Figure 2. Stage II: merging of the GAN-generated resection cavity image into a real before-resection iUS image.

Figure 3 .

 3 Figure 3. Images of 5 cases. From left to right: 1) the initial iUS image acquired before resection; 2) a resection cavity mask; 3) the result image simulated from these inputs; and 4) the corresponding real iUS image acquired after resection. In this study, resection cavity masks (column 2) correspond to the actual resection cavities segmented in the real iUS image after resection (column 4).

Table 1 .

 1 Percentages of correctly labelled images.

		Rater 1	Rater 2	Rater 3	All raters
		Imaging expert Neurosurgeon Neurosurgeon
	Simulated images	0.79	0.56	0.53	0.63
	train cases (GAN stage)	0.78	0.48	0.54	0.59
	test cases (GAN stage)	0.82	0.72	0.52	0.69
	Real images	0.76	0.79	0.41	0.65
	Overall	0.78	0.68	0.47	0.64
	Confidence score	2.74 ± 1.2	3.37 ± 0.8	3.32 ± 1.34	3.14 ± 1.2

ACKNOWLEDGMENTS

This work was supported by the French National Research Agency (ANR) through the frameworks Investissements d'Avenir Labex CAMI ANR-11-LABX-0004, ANR-15-IDEX-02, and MIAI ANR-19-P3IA-0003. The GAN computations were performed using the GRICAD infrastructure (gricad.univ-grenoble-alpes.fr), which is supported by Grenoble research communities.


Ryugu D Hallatt

H Leroux

P Roussel

PRELIMINARY RESULTS OF EXPLORING THE BEHAVIOUR OF PHYLLOSILICATES WHEN SUBJECT TO THERMAL CONDITIONS RELEVANT TO ASTEROID

Introduction:

As the targets of JAXA's recently returned Hayabusa2 and NASA's ongoing OSIRIS-REx sample return missions respectively, the near-Earth, C-group asteroids Ryugu and Bennu present an extraordinary opportunity to materially expose the history of our Solar System and the evolution of the bodies that it has comprised. It is from these missions' consistent detections of a recognizable near-infrared (NIR) spectral signature (the ~2.7 𝜇m absorption band), that both asteroids' surfaces are predicted to consist of at least one phyllosilicate component, despite both being extremely dark and otherwise generally featureless [1,2]. Across global and local scales, this phyllosilicate-like NIR absorption on Ryugu and Bennu has been reportedly widespread on the surface of both asteroids, and thus it is plausible that these familiar minerals could be present in the samples collected from either body.

Although such observations of phyllosilicates have not been unexpected (similar minerals have already been found on/from extraterrestrial bodies) their occurrence on small, near-Earth, carbonaceous asteroids may face particular attention. This is because such bodies are believed to have likely experienced a dynamical history, and are also, as their classification suggests, expected to contain carbonaceous materials in addition to the silicate minerals. It is from the balance of having been subject to a myriad of potentially material altering events while also being relative geologically inactive (being small and undifferentiated for example), that the use of hydrated silicates as messengers of past alteration may be particularly attractive for understanding the history of small asteroids such as Ryugu and Bennu.

This work, in particular, aims to report the preliminary efforts of a broad investigation (a doctoral thesis) that is based on questions concerning the effect that asteroidal impact has on the composition, structure, thermal behaviour, and detection of phyllosilicate structures related to Ryugu. Here focused on serpentine, this work will later be extended to similarly study saponite and organic-infused phyllosilicates, starting from a bulk investigation of their behaviour under relevant conditions, continuing with the use of a lightimpact gun and pulsed laser shock experiments, and concluding with an attempt to formulate a functional story of phyllosilicate behaviour from a multidisciplinary suite of experimental techniques including electron microscopy, x-ray diffraction, spectroscopy, thermal analysis, and evolved gas analysis.

Conveniently, a presently developing story from Ryugu is concerned with the interpretation of phyllosilicate(s) role in explaining an apparent trend in both the strength and position of the asteroid's only reported absorption band [3,4]. Consistent with a visible-wavelength spectral trend found between Ryugu's natural craters [3], it has been suggested from a trend in the asteroid's NIR signature near the artificial crater (where subsurface material was likely recently exposed/ejected) that there is/has been a surface-related phenomena that affects phyllosilicates in a way that changes their spectral absorption around 2.7 𝜇m. One of the suggested mechanisms for these spectral trends is the thermal decomposition of previously aqueously altered phyllosilicates, in particular from solar radiative heating [3,4]. It is here where an early extension of our impact-centric thermal annealing work may be applied.

Methods: A reference, magnesium-rich serpentine, known as UB-N SARM, was used in its as-received powder state. UB-N is from Col des Bagenelles, France, and is reported to be structurally dominated by the lizardite polymorph [5]. For bulk analysis of its hightemperature behaviour, High-Temperature Powder Xray Diffraction (HT-PXRD) was first performed. These XRD experiments employed a Bruker D8 Advance instrument operating with a Cu anode (λ = 1.5418 Å) in Bragg-Brentano geometry equipped with a LynxeEye XE 1D detector and an Anton Parr XRK 900 reactivity chamber. Using XRD allowed insights to be made into not only the serpentine phase-field with subsequent scans at points along a temperature ramp, but also the kinetics of specific phase transitions and the accuracy of ramped HT-PXRD by taking repeated scans at isothermal temperatures of interest. Each of the HT-PXRD temperature-ramp scans were performed between 9 to 64 ° (2𝜃) every 25 °C from ambient to 850 °C. Simultaneous ThermoGravimetric Analysis (TGA) and Differential Scanning Calorimetry (DSC) (also known as STA) was also performed at conditions complementary of the HT-PXRD experiments. In this way, STA was also performed both isothermally and in ramps between two temperatures informed by the HT-PXRD results. STA was performed using a Mettler-Toledo TGA/DSC 3+ instrument. In both the HT-PXRD and STA, a redox-controlling H2/N2 (3 vol.%) environment was usually used (along with an ambient control and a weak vacuum was also explored). Scanning electron microscopy and electron dispersive spectroscopy were also employed, to characterize the sample's morphology and composition respectively.

Results: The ramp HT-PXRD spectra summarized in the colour-clusters of Figure 1 identifies a potential thermal progression of UB-N serpentine while under H2/N2. The most dominant change is observed around 600 °C (turquoise) where a few of the remaining lizardite peaks (e.g. ~12 °) are in brief coexistence with high-temperature peaks believed to be representative of forsterite (e.g. ~25.3 ° and 33 °) before they mature (beige). Prior to this however, a presently unattributed peak at ~44.5 ° forms in accompaniment with the significant decay of most of the lizardite-interpreted peaks (e.g. ~24.4 ° and 60 °) at ~425 °C (green). From the level of calculated XRD background, Figure 1.c depicts evidence that a period of amorphization may start at the approximate onset of lizardite peak decomposition around 425 °C. At higher temperatures, the XRD background returns to a lower level, where recrystallization may be interpreted to be consistent with the maturation of high-temperature peaks from ~775 °C. Lastly, somewhere between 750 and 775 °C a distinct peak forms on the right shoulder of the forsterite-like ~29.6 ° peak, defining the (red) domain between 750 and 850 °C where this peak is only found before it disappears again before or at 850 °C.

The ramp STA performed under H2/N2 shows a progressive behaviour of lizardite as expected from previously reports, with three distinct stages of mass change up to 1000 °C [6]. The temperature of the first and third differential mass change (less than 600 °C and around 800 °C, respectively) is similar to literature, but the intermediate transition seems to occur at lower temperatures than reported under just N2 [6]. At the end of these three processes, UB-N serpentine has lost ~12% of its mass, similar to that also recently reported [6].

Discussion: No STA signature is clearly associated with the unidentified ~425 °C XRD precursor to lizardite decomposition. The temperature domain of such changes and the amorphization found in these preliminary results is noticeably similar to those previously proposed to be experienced from radiative heating on Ryugu [4]. Here may be an area to explore the possibility that the phyllosilicate trends seen on Ryugu may reside in the domain of amorphous complexity, inspiring further work to study the details of this realm under conditions relevant to Ryugu.

Figure 1 .

 1 Figure 1. HT-PXRD patterns collected from UB-N serpentine heated under H2/N2 (3 vol.%) from 25 to 850 °C at 5 °C min -1 . Individual plots are: a) the entire series of (ka2 subtracted) measured patterns, b) a peak intensity map in l-T space of those patterns minus a calculated background, and c) a heatmap of the associated background.

Figure 2 .

 2 Figure 2. Simultaneous TGA/DSC thermal analysis of ~39 mg of UB-N serpentine heated under H2/N2 (3 vol.%) from 25 to 1000 °C at 10 °C min -1 . Acknowledgments: The authors thank the Chevreul Institute (FR 2638) for access to the XRD and electron microscopy platforms, the I-SITE ULNE (Université Lille Nord Europe), and the MEL (Métropole Européenne de Lille). The authors also thank L. Burylo and N. Dielal, from the Université de Lille, for their help performing the XRD and STA respectively. References: [1] Hamilton V. E. et al. (2019) Nat. Astron., 3, 332-340. [2] Kitazato K. et al. (2019) Science., 364, 272-275. [3] Morota T. et al. (2020) Science, 368, 654-659. [4] Kitazato K. et al. (2021) Nat. Astron., doi/.org/10.1038/s41550-020-01271-2 [5] Gayk A. B. and Kleinschrodt C. D. (2000) JGR, 90, 1151-1154. [6] Pohl L. (2020) University of Central Florida-Doctoral Thesis, 269.