Matthew J Muckley

Member, IEEE Bruno Riemenschneider

Alireza Radmanesh

Sunwoo Kim

Member, IEEE Geunu Jeong

Jingyu Ko

Yohan Jun

Hyungseob Shin

Dosik Hwang

Mahmoud Mostapha

Simon Arberet

Dominik Nickel

Zaccharie Ramzi

Student Member, IEEE Philippe Ciuciu

Senior Member, IEEE Jean-Luc Starck

Jonas Teuwen

Dimitrios Karkalousos

Chaoping Zhang

Anuroop Sriram

Zhengnan Huang

Nafissa Yakubova

Yvonne W Lui

Florian Knoll

Results of the 2020 fastMRI Challenge for Machine Learning MR Image Reconstruction

Keywords: Challenge, Public Data Set, MR Image Reconstruction, Machine Learning, Parallel Imaging, Compressed Sensing, Fast Imaging, Optimization

Accelerating MRI scans is one of the principal outstanding problems in the MRI research community. Towards this goal, we hosted the second fastMRI competition targeted towards reconstructing MR images with subsampled k-space data. We provided participants with data from 7,299 clinical brain scans (de-identified via a HIPAA-compliant procedure by NYU Langone Health), holding back the fully-sampled data from 894 of these scans for challenge evaluation purposes. In contrast to the 2019 challenge, we focused our radiologist evaluations on pathological assessment in brain images. We also debuted a new Transfer track that required participants to submit models evaluated on MRI scanners from outside the training set. We received 19 submissions from eight different groups. Results showed one team scoring best in both SSIM scores and qualitative radiologist evaluations. We also performed analysis on alternative metrics to mitigate the effects of background noise and collected feedback from the participants to inform future challenges. Lastly, we identify common failure modes across the submissions, highlighting areas of need for future research in the MRI reconstruction community.

I. INTRODUCTION

Due to advances in algorithms, software platforms [START_REF] Chetlur | cuDNN: Efficient primitives for deep learning[END_REF], [START_REF] Abadi | TensorFlow: A system for large-scale machine learning[END_REF], [START_REF] Paszke | PyTorch: An imperative style, highperformance deep learning library[END_REF] and compute hardware, over the last five years there has been a surge of research of MR image reconstruction methods based on machine learning [START_REF] Yang | Deep ADMM-Net for compressive sensing MRI[END_REF], [START_REF] Hammernik | Learning a variational network for reconstruction of accelerated MRI data[END_REF], [START_REF] Schlemper | A deep cascade of convolutional neural networks for dynamic MR image reconstruction[END_REF], [START_REF] Yang | DAGAN: Deep dealiasing generative adversarial networks for fast compressed sensing MRI reconstruction[END_REF], [START_REF] Eo | KIKI-net: Crossdomain convolutional neural networks for reconstructing undersampled magnetic resonance images[END_REF], [START_REF] Aggarwal | MoDL: Model-based deep learning architecture for inverse problems[END_REF], [START_REF] Zhu | Image reconstruction by domain-transform manifold learning[END_REF], [START_REF] Knoll | Assessment of the generalization of learned image reconstruction and the potential for transfer learning[END_REF], [START_REF] Knoll | Deep-learning methods for parallel magnetic resonance imaging reconstruction: A survey of the current approaches, trends, and issues[END_REF], [START_REF] Yaman | Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data[END_REF], [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF]. Traditionally, research in MR image reconstruction methods has been conducted on small data sets collected by individual research groups with direct access to MR scanner hardware and research agreements with the scanner vendors. Data set collection is difficult and expensive, with many research groups lacking the organizational infrastructure to collect data at the scale necessary for machine learning research. Furthermore, data sets collected by individual groups are often not shared publicly for a variety of reasons. As a result, research groups lacking large-scale data collection infrastructure face substantial barriers to reproducing results and making comparisons to existing methods in the literature.

Such challenges have been seen before. In the field of computer vision, the basic principles of convolutional neural networks (CNNs) were proposed as early as 1980 [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition[END_REF] and became well-established for character recognition by 1998 [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. Following Nvidia's release of CUDA in 2007, independent research groups began to use GPUs to train larger and deeper networks [START_REF] Raina | Large-scale deep unsupervised learning using graphics processors[END_REF], [START_REF] Cires ¸an | Deep, big, simple neural nets for handwritten digit recognition[END_REF]. Nonetheless, universal acceptance of the utility of CNNs did not occur until the debut of the large-scale ImageNet data set and competition [START_REF] Deng | ImageNet: A large-scale hierarchical image database[END_REF]. The introduction of ImageNet allowed direct cross-group comparison using this well-recognized data set of a size beyond what most groups could attain individually. In 2012 a CNN-based model [START_REF] Krizhevsky | ImageNet classification with deep convolutional neural networks[END_REF] out-performed all non-CNN models, spurring a flurry of stateof-the-art results for image recognition [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], [START_REF] He | Deep residual learning for image recognition[END_REF], [START_REF] Szegedy | Rethinking the Inception architecture for computer vision[END_REF], [START_REF] Huang | Densely connected convolutional networks[END_REF].

Since 2018, the fastMRI project has attempted to advance community-based scientific synergy in MRI by building on two pillars. The first consists of the release of a large data set of raw k-space and DICOM images [START_REF] Zbontar | fastMRI: An open dataset and benchmarks for accelerated MRI[END_REF], [START_REF] Knoll | fastMRI: A publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning[END_REF]. This data set is available to almost any researcher, allowing them to download it, replicate results, and make comparisons. The second pillar consists of hosting public leaderboards [START_REF] Zbontar | fastMRI: An open dataset and benchmarks for accelerated MRI[END_REF] and open competitions, such as the 2019 fastMRI Reconstruction Challenge on knee data [START_REF] Knoll | Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge[END_REF]. The dimension of public competitions is not new to the MR community. Other groups have facilitated challenges around RF pulse design [START_REF] Grissom | Advancing RF pulse design using an open-competition format: Report from the 2015 ISMRM challenge[END_REF], diffusion tractography reconstruction [START_REF] Schilling | Challenges in diffusion MRI tractography-lessons learned from international benchmark competitions[END_REF], and ISMRM initiatives for reconstruction [START_REF] Maier | CG-SENSE revisited: Results from the first ISMRM reproducibility challenge[END_REF], [START_REF] Beauferris | Multi-channel MR reconstruction (MC-MRRec) challenge -comparing accelerated MR reconstruction models and assessing their generalizability to datasets collected with different coils[END_REF].

The 2020 fastMRI Challenge continues this tradition of open competitions and follows the 2019 challenge with a few key differences. First, our target anatomy has been changed to focus on images of the brain rather than knee. Second, for 2020 we updated the radiologist evaluation process, asking radiologists to rate images based on depiction of pathology rather than overall image quality, emphasizing clinical relevance in competition results. Lastly, we address a core traditional problem in MR imaging: the capacity of models to generalize across sites and vendors. We introduce a new competition track: a "Transfer" track, where participants were asked to run their models on data from vendors not included in training. This contrasts with the 2019 challenge, which only included data from a single vendor for both training and evaluation.

II. METHODS

This challenge focuses on MRI scan acceleration, a topic of interest to the MR imaging community for decades. MRI scanners acquire collections of Fourier frequency "lines", commonly referred to as k-space data. Due to hardware constraints on how magnetic fields can be manipulated, the rate at which these lines are acquired is fixed, which results in relatively long scan times and has negative implications with regard to image quality, patient discomfort, and accessibility. The major way to decrease scan acquisition time is to decrease the amount of data acquired. Sampling theory [START_REF] Whittaker | XVIII.-On the functions which are represented by the expansions of the interpolation-theory[END_REF], [START_REF] Nyquist | Certain topics in telegraph transmission theory[END_REF], [START_REF] Kotelnikov | On the carrying capacity of the 'ether' and wire in telecommunications[END_REF], [START_REF] Shannon | Communication in the presence of noise[END_REF] states that a minimum number of lines are required for image reconstruction. This minimum requirement can be circumvented by incorporating other techniques such as parallel imaging [START_REF] Sodickson | Simultaneous acquisition of spatial harmonics (SMASH): Fast imaging with radiofrequency coil arrays[END_REF], [START_REF] Pruessmann | SENSE: Sensitivity encoding for fast MRI[END_REF], [START_REF] Griswold | Generalized autocalibrating partially parallel acquisitions (GRAPPA)[END_REF] and compressed sensing [START_REF] Lustig | Compressed sensing MRI[END_REF]. More recently, machine learning methods have demonstrated further accelerations over parallel imaging and compressed sensing methods.

To promote the advancement of methods for accelerated MRI, we organized a public challenge. We applied retrospective downsampling to fully-sampled MRIs and provided the downsampled data to challenge participants. Challenge participants ran their models on the downsampled data and submitted it to the competition website at https://fastmri.org, where we quantitatively evaluated it using the fully-sampled data as gold standards. We selected six cases for each of the top three teams in each track of the challenge (three tracks total) and presented the cases to a group of six radiologists for qualitative evaluation. The challenge winner was selected based on the best depiction of pathology compared to the ground truth as judged by radiologists.

At a high level we describe the principles of our 2020 challenge as follows. Using knowledge we gained through the 2019 challenge, we identified a few key alterations for 2020. These include:

• A new imaging anatomy, the brain, the most commonlyimaged organ using MRI. • A focus on an evaluation of pathology depiction rather than overall image quality impressions to strengthen the connection between the challenge evaluation and clinical practice.

• An emphasis on generalization with the introduction of a new "Transfer" track where participants were asked to run their models on multi-vendor data.

• We removed the single-coil track and moved to a pure multi-coil challenge to increase the clinical relevance of the submitted models. • Due to easier practical implementation and removal of the single-coil track, we used pseudo-equispaced subsampling masks (i.e., equispaced masks with a modification for achieving exact 4X/8X sampling rates) rather than random. This follows more closely sampling patterns (and relaxation effects) that are used for parallel imaging in vendor sequences, facilitating easier clinical deployment. We maintained the fully-sampled center due to its utility for autocalibrating parallel imaging methods [START_REF] Sodickson | Simultaneous acquisition of spatial harmonics (SMASH): Fast imaging with radiofrequency coil arrays[END_REF], [START_REF] Griswold | Generalized autocalibrating partially parallel acquisitions (GRAPPA)[END_REF], [START_REF] Uecker | ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA[END_REF] and compressed sensing [START_REF] Lustig | Compressed sensing MRI[END_REF]. • In the 2019 challenge our baseline model was a U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF]; however, winning models [START_REF] Pezzotti | An adaptive intelligence algorithm for undersampled knee MRI reconstruction[END_REF], [START_REF] Wang | Pyramid convolutional RNN for MRI reconstruction[END_REF], [START_REF] Putzky | i-RIM applied to the fastMRI challenge[END_REF] of the 2019 challenge were variational network/cascading models [START_REF] Knoll | Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge[END_REF]. For the 2020 challenge, we provided a much stronger baseline model based on an End-to-End Variational Network [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF].

We kept the following principles from the 2019 challenge:

• We again used a two-stage evaluation, where a quantitative metric was used to select the top 3 submissions. These finalists were then sent to radiologists to determine the winners. We used the structural similarity index (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] as our quantitative image quality index for ranking submissions prior to submission to clinical radiologists [START_REF] Knoll | Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge[END_REF]. • We wanted to maintain realism for a straightforward, 2D imaging setting, and so all of the competition data was once again based on fully-sampled 2D raw k-space data. • For the ground truth reference, we had discussions on alternatives to the root sum-of-squares (RSS) method used for quantitative evaluation in 2019. Although there was some consensus on the drawbacks of RSS [START_REF] Roemer | The NMR phased array[END_REF], [START_REF] Walsh | Adaptive reconstruction of phased array MR imagery[END_REF], there was no consensus on a single best alternative. In the following sections we discuss the impact of this choice further.

A. Challenge Tracks

In the 2019 challenge we included three submission tracks: multicoil with four times acceleration (Multi-Coil 4X), multicoil with eight times acceleration (Multi-Coil 8X), and singlecoil with 4X acceleration (Single-Coil 4X). Among these tracks, the single-coil track garnered the most engagement, but due to its distance from clinical practice we decided to remove it from the 2020 challenge, replacing it with the Transfer track. For the standard multicoil tracks in the 2019 challenge, we observed that although there were many high-quality submissions at 4X, all of the submissions began missing pathology at 8X acceleration [START_REF] Knoll | Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge[END_REF]. Since this time, 4X machine learning methods have been validated for clinical interchangeability [START_REF] Recht | Using deep learning to accelerate knee MRI at 3T: Results of an interchangeability study[END_REF]. This suggests that the current upper limit of 2D machine learning image reconstruction performance remains between 4-fold and 8-fold acceleration rates. In order to provide participants with both an obtainable target and a "reach" goal, we kept the 4-fold and 8-fold tracks for the 2020 challenge.

One frequent feedback on the 2019 challenge was on generalizability: despite the size of the data set, all of the data and results were from studies performed on MRI scanners from a single vendor at a single institution. To address this, we created the new Transfer track at 4-fold acceleration (Transfer 4X). For the Transfer track, participants were asked to run their models on data from vendors outside the main fastMRI data set. There was a caveat: we also restricted participants in the Transfer track to train their models only using available fastMRI data to ensure evaluation of transfer capability. At the time of the 2020 challenge announcement, we stated that these data would come "from another vendor" but did not specify further. At the challenge launch time, we revealed that the challenge data for this track was a mix of data from GE and Philips, providing additional difficulty for participants. As a result, submissions in the Transfer track exhibited wide deviations in performance depending on vendor.

B. Data Set

For the 2020 challenge we used brain MRI data. The neuroimaging subset of the fastMRI data has been described in an updated version of the arXiv paper [START_REF] Zbontar | fastMRI: An open dataset and benchmarks for accelerated MRI[END_REF], with further information included in the supplemental material for this paper. It includes 6,970 scans (3,001 at 1.5 T, 3,969 at 3 T) collected at NYU Langone Health on Siemens scanners using T1, T1 post-contrast, T2, and FLAIR acquisitions. Unlike the knee challenge, this data set exhibits a wide variety of reconstruction matrix sizes. A summary of the data for the two main track splits is shown in Table I. Of these 6,970 scans, 565 were withheld for evaluation in the challenge. In addition to standard HIPAA-compliant anonymization practices, all scans were cropped at the level of the orbital rim, preserving only the top part of the head. For the challenge, the 565 scans were augmented further by 329 non-Siemens scans for the Transfer track. GE data were collected at NYU Langone Health and Philips data were collected on volunteers by clinical partner sites of Philips Healthcare of North America. Since the Philips data was collected on volunteers, this subsplit had no post-contrast imaging. One difficulty of the Transfer track was the fact that the GE data did not contain frequency oversampling. The lack of frequency oversampling was due to automatic removal during the analog-to-digital conversion process on the GE scanner.

In total the 2020 challenge had 6,405 scans available for training and validation (train, val, test) and there were 894 total scans for evaluation in the final challenge phase. This marked a substantial increase in scale from the 2019 challenge. For reference, the multicoil data from the 2019 challenge on knee data had 1,290 scans for training and validation (train, val, test) and 104 scans for the challenge, so the data for training increased by roughly 5-fold and the data for challenge evaluation increased by roughly 8-fold.

Participants were restricted to only use this data set for training the weights of their models. We did permit participants to use their own data as validation data, but not for backpropagating gradients.

C. Evaluation Process

Submissions were processed via https://fastmri. org. This site maintains a submission system for both the challenge and the public test leaderboard. Currently, there are no plans to make the challenge split of the data available in order to maintain the integrity of the results, but research groups may submit to the public leaderboard using the test set via the website at any time.

After submission, evaluation followed a two-stage process of comparisons to the fully-sampled "ground truth" images. For the ground truth images, we followed the previous convention [START_REF] Knoll | Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge[END_REF] to use root sum-of-squares images. The advantage of this approach is that it does not bias to any one method for coil sensitivity estimation. There are some drawbacks to RSS, including 1) discarding of the phase information and 2) RSS images can have substantial noise in the background. The phase is not typically used for anatomical evaluation. The issue with noise is more fundamental as it is treated as ground truth in our quantitative evaluation, and any deviations from it influence our ranking. This is counterbalanced by using radiologist evaluation for declaring the challenge winner. In planning for the challenge, we were unable to build consensus on an alternative ground truth calculation technique, but this topic could be re-examined in future challenges. For the quantitative evaluation metric, we chose to use SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], with a script showing the script used for evaluation in the fastMRI repository at https://github. com/facebookresearch/fastMRI. SSIM has several parameters. We investigated adjusting these parameters prior to challenge launch, but found that they generally did not alter the ranking of methods evaluated in our quality control phase, and so as a result we used the default parameters in scikit-image [START_REF] Van Der Walt | scikit-image: image processing in Python[END_REF].

For the qualitative assessment phase, a board-certified neuroradiologist selected six (two T1 post-contrast, two T2, and two FLAIR) cases from the challenge data set in each of the three tracks. Cases were specifically selected to represent a broad range of neuroimaging pathologies from intracranial tumors and strokes to normal and age-related changes. The selection process favored cases with more subtle pathologies for the 4X track and more obvious pathologies for the 8X track with the objective that this might yield better granularity for separating methods in the 4X track. Selected cases included both intraaxial and extraaxial tumors, strokes, microvascular ischemia, white matter lesions, edema, surgical cavities, as well as postsurgical changes and hardware including craniotomies and ventricular shunts. The Philips data set was constructed from images of volunteers. Therefore small age-related imaging changes were used for ranking in place of pathology.

Six radiologists with 9-16 years of experience (two of whom are radiology division chiefs) were asked to evaluate the 18 selected image volumes for each team, basing their overall ranking on the quality of the depiction of the pathology using the ground truth as a reference. Radiologists came from a wide set of institutions, including the Mayo Clinic, Baylor College of Medicine, NYU Langone Health, the University of Pittsburgh Medical Center, Stanford University, and the University of California, Los Angeles. None of these institutions had finalist submissions. All radiologists looked at all images in the selected cases during the qualitative evaluation phase, and results were averaged. Radiologists were aware of the overarching goals of the challenge but were blinded as to which teams submitted the images. In addition, we also asked radiologists to score each case in terms of artifacts, sharpness and contrast-to-noise ratio (CNR) using a Likert-type scale. On the Likert scale, 1 was the best (e.g., no artifacts) and 4 was the worst (e.g., unacceptable artifacts). A Likert score of 3 would affect diagnostic image quality.

D. Timeline

The 2020 challenge had the following timeline:

• December 19, 2019 -Release of the brain data set and update to the arXiv reference [START_REF] Zbontar | fastMRI: An open dataset and benchmarks for accelerated MRI[END_REF]. • July 9, 2020 -Announcement of the 2020 challenge.

• October 1-15, 2020 -Release of the challenge data set and submission window. • October 16-19, 2020 -Calculation of SSIM scores.

We selected the top 3 submissions for each track and forwarded them to a panel of radiologists for qualitative evaluation. • October 19-November 1, 2020 -Radiologists evaluated submissions. They were asked to complete a score sheet for each of the 3 tracks which included ranking the submissions for each individual case in terms of overall quality of depiction of pathology. • December 5, 2020 -Publication of the challenge leaderboard with results. • December 12, 2020 -Official announcement of the winners of the three tracks with presentations at the Medical Imaging Meets NeurIPS Workshop.

E. Overview of Submission Methodologies

Here we share a brief description of the methodologies behind each of the submissions that made it to the finalist round for radiologist evaluation. The developers of these submissions are included as co-authors on this paper.

A brief overview of finalist model properties is shown in Table II. (Team names: "AIRS" is AIRS Medical, "ATB" is ATB, "MRR" is MRRecon, "Nspin" is Neurospin, "Res" is ResoNNance.) The number of model parameters ranged from 841,000 in the case of ResoNNance to 200 million in the case of AIRS. Teams applied GRAPPA [START_REF] Griswold | Generalized autocalibrating partially parallel acquisitions (GRAPPA)[END_REF], ESPIRiT [START_REF] Uecker | ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA[END_REF], or simple zero-filled initializations. For coil estimation, teams used either ESPIRiT [START_REF] Uecker | ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA[END_REF] or a simple center-based estimation with U-Net refinement similar to that in the End-to-End Variational Network [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF]. Teams used 1-8 GPUs for training, and training time was between 7 and 21 days.

AIRS Medical: The AIRS Medical model used a combination of image-and k-space-domain processing in a fashion analogous (but distinct) from that used in KIKI-Net [START_REF] Eo | KIKI-net: Crossdomain convolutional neural networks for reconstructing undersampled magnetic resonance images[END_REF]. The model included a data consistency cascade with 4 U-Net stages. At each convolutional layer of the U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF], the multi-domain processing split the channels into one group that operated in image space and one group that operated in k-space [START_REF] Eo | KIKI-net: Crossdomain convolutional neural networks for reconstructing undersampled magnetic resonance images[END_REF]. Data consistency was enforced at each layer. The network was initialized with a GRAPPA estimate [START_REF] Grissom | Advancing RF pulse design using an open-competition format: Report from the 2015 ISMRM challenge[END_REF] (preprocessed into reconstruction + residual), and coil sensitivities were estimated using ESPIRiT [START_REF] Uecker | ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA[END_REF]. Since the sampling pattern was pseudo-equispaced, multiple GRAPPA kernels were used to calculate the GRAPPA images. AIRS optimized their model using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] over 20 epochs using a batch size of 4 at a learning rate of 10 -3 (decayed to 10 -4 after 15 epochs) with SSIM as the loss function. Optimization took approximately 7 days on four NVIDIA V100 GPUs. Code is not publicly available.

ATB: The ATB model, called "Joint-ICNet" [START_REF] Jun | Joint deep model-based MR image and coil sensitivity reconstruction network (Joint-ICNet) for fast MRI[END_REF], was a 10iteration unrolled algorithm with CNNs replacing regularization terms in a fashion similar to other recent methods [START_REF] Hammernik | Learning a variational network for reconstruction of accelerated MRI data[END_REF], [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF], [START_REF] Eo | KIKI-net: Crossdomain convolutional neural networks for reconstructing undersampled magnetic resonance images[END_REF]. Joint-ICNet used the U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF] at each convolutional layer with the dual-domain processing previously introduced in KIKI-net [START_REF] Eo | KIKI-net: Crossdomain convolutional neural networks for reconstructing undersampled magnetic resonance images[END_REF]. Joint-ICNet used a zero-filled reconstruction as the initial estimate and and coil sensitivities were calculated by refining a rough central k-space estimate with a U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF], [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF]. ATB optimized Joint-ICNet using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] over 50 epochs using a batch size of one at a learning rate of 10 -4 with SSIM as the loss function. Training took approximately 10 days using 8 NVIDIA TITAN GPUs. Code is not publicly available.

MRRecon: The MRRecon model, called "Momentum DIHN," unrolled the Nesterov momentum algorithm with CNN-based regularization for 12 cascades, 6 pre-cascades, and 0 or 1 post-cascade. The CNN module is a "Deep, Iterative, Hierarchical Network" (DIHN) that extends the Down-Up network [START_REF] Yu | Deep iterative down-up CNN for image denoising[END_REF] with a hierarchical block design, facilitating memory efficiency over a standard U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF]. Momentum DIHN used a zero-filled image as the initial estimate and ESPIRiT for calculating coil sensitivities. To improve transfer track performance, models with several hyperparameters were ensembled to generate the final images. MRRecon optimized Momentum DIHN using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] for less than 5 epochs using a batch size of one at a learning rate of 10 -4 with a compound L1/MS-SSIM loss function [START_REF] Pezzotti | An adaptive intelligence algorithm for undersampled knee MRI reconstruction[END_REF].

Training took approximately 14 days on an NVIDIA V100 GPU. Code is not publicly available.

Neurospin: The Neurospin model, called XPDNet [START_REF] Ramzi | Benchmarking MRI reconstruction neural networks on large public datasets[END_REF], [START_REF] Ramzi | XPDNet for MRI reconstruction: An application to the 2020 fastMRI challenge[END_REF], is a modular neural network unrolling the Chambolle-Pock algorithm [START_REF] Chambolle | A first-order primal-dual algorithm for convex problems with applications to imaging[END_REF] for 25 iterations. The model was inspired by the primal-only version of the Primal-Dual net [START_REF] Adler | Learned primal-dual reconstruction[END_REF], replacing the vanilla CNN with a multi-level wavelet CNN [START_REF] Liu | Multi-level Wavelet-CNN for image restoration[END_REF]. XPDNet used a zero-filled image as the initial estimate and calculated coil sensitivities using a rough central k-space estimate refined by a U-Net [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF], [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF]. Neurospin optimized XPDNet using Rectified Adam [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF] over 100 epochs using a batch size of one at a learning rate of 10 -4 with a compound L1/MS-SSIM loss function [START_REF] Pezzotti | An adaptive intelligence algorithm for undersampled knee MRI reconstruction[END_REF] (98% MS-SSIM weight). Training took approximately 7 days on an NVIDIA V100 GPU. Code is available at https://github.com/zaccharieramzi/ fastmri-reproducible-benchmark.

ResoNNance: ResoNNance used a Recurrent Inference Machine (RIM) that has been previously described [START_REF] Putzky | Recurrent inference machines for solving inverse problems[END_REF], [START_REF] Lønning | Recurrent inference machines for reconstructing heterogeneous MRI data[END_REF], [START_REF] Putzky | i-RIM applied to the fastMRI challenge[END_REF], [START_REF] Beauferris | Multi-channel MR reconstruction (MC-MRRec) challenge -comparing accelerated MR reconstruction models and assessing their generalizability to datasets collected with different coils[END_REF]. Coil sensitivities were calculating using the center of k-spaced followed by U-Net refinement [START_REF] Ronneberger | U-Net: Convolutional networks for biomedical image segmentation[END_REF]. RIM used ESPIRiT as the model input calculated from the BART toolbox [START_REF] Uecker | Berkeley advanced reconstruction toolbox[END_REF]. ResoNNance optimized RIM using Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] over 90 epochs using a batch size of one at a learning rate of 10 -3 with an SSIM loss function. Separate models were trained for every field strength (1.5 T, 3 T) and contrast (FLAIR, T1/T1PRE/T1POST, and T2). Code for the RIM, data loaders, and documentation can be found through the DIRECT repository at https://github.com/directgroup/direct.

III. RESULTS

A. Submission Overview

For the 2020 challenge we received a total of 19 submissions from eight different groups. Seven groups submitted to the Multi-Coil 4X and Multi-Coil 8X tracks. One of these groups chose not to submit to the Transfer track, while an eighth group submitted only to the Transfer track. As previously, we encourage all submitting groups to publish papers and code used to generate their results.

Figure 1 shows an overview of images submitted to the 4X track of the challenge with Siemens data that were forwarded to radiologists. All three top performing submissions were able to successfully reconstruct the T2 and FLAIR images with minimal artifact presentation. For some images in this track's evaluation, radiologists had difficulty perceiving substantive differences between the three top performing reconstructions in terms of their overall ability to depict the pathology. Overall, the results were better on the high signal-to-noise T2 and FLAIR contrasts compared with those on the T1POST. In the case in Figure 1, the ATB and Neurospin methods struggled with a strong susceptibility effect, introducing false vessels between the susceptibility and the lateral ventricular wall.

Figure 2 shows example images for radiologist evaluation from the 8X track with Siemens data. In this track, artifacts are seen to be more severe and pronounced. For some cases radiologists stated that they were hesitant to accept any of the submissions at 8X. Over-smoothing is readily apparent in T1POST reconstructions from all three of the top performers. We noticed at this acceleration level that so-called horizontal "banding" effects [START_REF] Defazio | MRI banding removal via adversarial training[END_REF] could be appreciated in the FLAIR images due to the extreme acceleration and the anisotropic sampling pattern.

Example images from the 4X Transfer track are shown in Figure 3. For this track, we observed the lowest SSIM values (Section III-B). Of note, there is a divergence between performance of methods on GE versus Philips data. This can be seen the image submitted by ResoNNance in Figure 3, which introduces artifacts in its reconstructions of the GE images (T1POST and T2 in Figure 3), but less so in its Philips reconstruction (FLAIR in Figure 3). Most participant models (trained on Siemens data) were able to reconstruct Philips data with higher fidelity than GE, likely due to the fact that Philips and Siemens followed the same protocol for writing frequency-oversampled data to their raw data files. An additional factor is that GE uses a T1-based FLAIR, whereas Philips and Siemens use a T2-based FLAIR.

B. Quantitative Results

Figure 4 shows an overview of SSIM scores across group rankings. SSIM values were highly clustered in the 4X track, with all top 4 participants scoring between 0.955 and 0.965. We observed greater variation between submissions in the 8X track, with the top participant scoring 0.952 and the others scoring below 0.944. The greatest variation occurred in the Transfer track. Many participants struggled to adapt their models to GE data. These data did not include frequency oversampling in the raw k-space data, which we have observed can decrease SSIMs for models by as much as 0.1-0.4 if no other adjustments are made. On the other hand, the Philips data did include frequency oversampling, so adaptation here was more straightforward.

Table III summarizes results by contrast for the finalists in each competition track with means and 95% confidence intervals based on 2.5% and 97.5% quantiles. The strongest SSIM scores were usually recorded on T1 post-contrast images (T1POST), while the weakest scores were typically on FLAIR images. The same participant recorded the top average SSIM score for every contrast in every track except the Transfer track for T1 contrast. In this case, two other participants posted higher SSIM scores. One team, HungryGrads, submitted to all tracks and received a very low SSIM score between 0.4 and 0.5. This team set the background air to nearly 0s, which led to a clinically irrelevant SSIM loss of approximately 0.3 for their submissions. The HungryGrads submission prompted our team to perform a post-hoc analysis where we masked both the submission and the reference RSS ground truth before calculating SSIM, with results plotted in Figure S1 in the supplementary material. Applying this mask markedly improved the SSIM scores of HungryGrads, although it would not have made this team a finalist. Applying the mask would have enabled ATB to enter the finalist round for the Transfer track. Our custom mask would not have changed finalist rankings otherwise.

C. Radiologist Evaluation Results

Radiologist rankings based on quality of pathology depiction were concordant with SSIM scores for the top submissions as shown in Figure 5. The second and third place performers for both 4X and 8X tracks were flipped between the quantitative ranking based on SSIM and the qualitative ranking based on radiologists. The SSIM difference between these two constructions methods was relatively small, out to the third A summary of the ranks and Likert scores with with means and standard deviations is shown in Table IV (AIRS = AIRS Medical, Nspin = Neurospin, MRR = MRRecon, and Res. = ResoNNance). We applied standard deviations for Table IV (instead of quantiles as used in Table III) to show more information on the variability. Across all metrics AIRS Medical separated itself from the other submissions with the highest SSIM and best image quality. Aside from this single team, differentiation among the other teams was not strong. Of note, both the Neurospin and ATB teams had nearly identical average SSIM scores for the quantitative evaluation, with ATB presenting a slightly higher score (0.960 vs. 0.959 in 4X, 0.944 vs. 0.942 in 8X). In the radiologist evaluation phase, these ranks flipped, with Neurospin receiving slightly higher ranks (1.94 vs. 2.22 in 4X, 2.25 vs. 2.28 in 8X).

A case-wise breakdown of the ranks for all 3 finalists and all rated cases is shown in Figure 5. For second and thirdplace metrics as rated by SSIM, radiologist assessment was discordant between the two methods. However, in 16 out of 18 cases the highest SSIM score within the finalists' batches also received the highest radiologists' rating. A similar relationnot shown here -was found for the other used metrics such as normalized mean-squared error (NMSE) and peak signal-to-noise ratio (PSNR).

Radiologist agreement according to Kendall's coefficient of concordance generally improved as SSIM scores diverged. We calculated the concordance using radiologist rankings of teams for quality of depiction of pathology vs. the ground truth. For each case in the radiologist evaluation phase, we evaluated Kendall's coefficient of concordance with tie correction, and then aggregated over all cases by averaging. This resulted in values of 0.457 for the 4X track, 0.386 for the 8X track, and 0.781 for the 4X Transfer track (where 0 indicates complete disagreement and 1 indicates complete agreement). In the 4X and 8X tracks, discordance was primarily driven by two submissions (Neurospin and ATB) that were very close in SSIM score. For the Transfer track, separation among the teams was more clear, and we observed corresponding increases in concordance.

Radiologists did take note of hallucinatory effects introduced by the submission models. Figure 6 shows hallucination examples from all three tracks. In some cases methods created artifact-mimics. In other examples, models morphed an abnormality into a more normal brain structure, such as a sulcus or vessel. Finally, we observed at least one example combining these two where an artifact was created at some intermediate layer of a model and then processed by the remaining portions of the network into a normal structure mimic.

IV. DISCUSSION

A. Submission Overview

In the 2019 challenge all three tracks were very closely contested, with little separation between teams either in the quantitative or the radiologist evaluation phases. We observed this pattern to be reversed in the 2020 challenge, with one team assertively scoring the best in all evaluation phases. For some images in the 4X track, multiple radiologists said that they did not observe major differentiating aspects affecting the depiction of pathology in the submissions. However, when averaging the radiologists' rankings, radiologists preferred the method that had the highest-scoring on SSIM from AIRS Medical. We further observed that the AIRS model scored highest on Likert-type ratings of artifacts, sharpness, and CNR. This model also provided improvement over the baseline [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF], which had previously been demonstrated for clinical interchangeability at 4X for knee imaging [START_REF] Recht | Using deep learning to accelerate knee MRI at 3T: Results of an interchangeability study[END_REF]. Outside of the AIRS model, in the 4X and 8X tracks the second and third-place models scored very close together in both the quantitative and the qualitative evaluation phase. In some cases the SSIM scores for these two models were identical out to three decimal places. We observed decreases in performance in the Transfer track. Many participants struggled to adapt their models to the GE data with its lack of disk-written frequency oversampling. Although technically the GE scanner did not operate in any majorly different way than Philips and Siemens scanners (all use frequency oversampling), this simple aspect rendered many models useless in this track without modification. Another factor was a divergence in FLAIR methodology: our Philips and Siemens data used T2 FLAIR images, whereas the GE data had T1 FLAIR images. Modifications for correcting these effects seem not to be straightforward. We note that as designed the Transfer track primarily evaluated one type of transfer: generalization across vendors. This was the most commonlycited type of transfer in feedback from the 2019 challenge, but future challenges may investigate other types of transfer.

In terms of radiologist evaluations, despite the drawbacks of SSIM and RSS ground truths, we observed a correlation between radiologist scores and SSIM scores for large SSIM separations. Multiple radiologists found images at 4X to be similar in terms of depiction of the pathology, although artifacts tended to be more problematic in T1POST images. When it came to the 8X and Transfer tracks, radiologist sentiment became more negative. Multiple radiologists in both of these tracks offered feedback that none of the submitted images would be acceptable, indicating that these two tasks may remain open problems going into 2021. We note that the results of this paper were observed within the regime of retrospective undersampling. Retrospective undersampling does not consider potential differences in signal relaxation along the echo trains. Even though echo trains can be designed for most sequences in such a manner that undersampling does not lead to a change in overall relaxation weighting along the phase dimension, we have not shown equivalency in our work. We would recommend that researchers confirm the results of the methods in this paper with prospective sampling prior to clinical use.

The results of the challenge suggests a few conclusions on approaches. The first: cascaded models with a data fidelity term and CNN regularization continue to dominate the submission field, as occurred in the previous challenge [START_REF] Knoll | Assessment of the generalization of learned image reconstruction and the potential for transfer learning[END_REF]. Second: AIRS, the team that won all three tracks, had the largest model. However, large models were not always better, with ATB having a model with similar performance to Neurospin despite having 87% less parameters. Lastly, we note the AIRS model used a normalization routine to get the data into a consistent format for all coil configurations, as well as being the only team to use a GRAPPA [START_REF] Griswold | Generalized autocalibrating partially parallel acquisitions (GRAPPA)[END_REF] reconstruction as the initialization.

B. Quantitative Evaluation Process

Discussions around the quantitative evaluation process primarily concerned the presence of background noise during both the planning and execution stage of the challenge. The influence of background noise on SSIM scores is substantial. One participant in the 2019 challenge had a dedicated style transfer model in order to add this noise back into the reconstructed images [START_REF] Hammernik | Σ-net: Systematic evaluation of iterative deep neural networks for fast parallel MR image reconstruction[END_REF]. Despite the drawbacks to SSIM, we were unable to agree on an alternative for the 2020 challenge.

In the 2020 challenge, the HungryGrads team submitted images with backgrounds of nearly zeroes, which penalized their scores. Prompted by this submission we investigated the effect a masked metric might have had on their scores in Section III-B. We opted for a masking algorithm that removes most background pixels and altered the algorithm parameters for low-SNR edge cases where it did not perform well. Due to the relatively small size of the challenge data set visual inspection of the validity of the masks was feasible. The of the challenge did not change dramatically due to masking, but masking made metrics less prone to a specific reconstruction method's impact on the background.

Another intriguing alternative would be to use alternative reconstruction techniques such as adaptive combine that implicitly suppress background noise [START_REF] Walsh | Adaptive reconstruction of phased array MR imagery[END_REF]. We considered using adaptive combine reconstructions in our evaluation, but concluded that the results were not particularly meaningful as most models had been trained with RSS backgrounds. Another alternative to adaptive combine would be to use other metrics that are more aware of the noise properties, such as Stein's Unbiased Risk Estimate (SURE).

One area lacking in our quantitative analysis was hallu-cination detection. This is an area of great interest to the community, but as of the end of our challenge we were unaware of automated, methods for detecting lesions or characterizing stability (although some methods can demonstrate instability qualitatively [START_REF] Antun | On instabilities of deep learning in image reconstruction and the potential costs of AI[END_REF]). Detection of automated stability/hallucination analysis remains a topic of great interest for future challenges.

C. Qualitative Radiologist Evaluation

For the 2020 challenge, we altered the radiologist questionnaire to focus their ranking on the depiction of pathologies rather than general image quality. Some radiologists found the focus helpful, commenting specifically that the images at 8X and in the 4X Transfer track might not be acceptable for clinical use. As this task aligns more closely with the normal clinical workflow, we would encourage future competition organizers to use this approach for their radiologist evaluation procedures.

In the 4X track, there were specific cases where the radiologist rankings were concordant and others where the rankings were discordant. Discordant cases tended, upon review, to show that the main abnormalities were similarly well depicted across the top 3 reconstructions, though there were oftentimes concordant estimations of differences between reconstructions in terms of artifacts, sharpness and CNR.

Radiologist sentiment was affected by hallucinations such as those in Figure 6. Such hallucinatory features are not acceptable and especially problematic if they mimic normal structures that are either not present or actually abnormal. These images had high SSIM scores, indicating that even though these images are considered well-optimized according to this metric, they are not optimized regarding hallucination features. Neural network models can be unstable as demonstrated via adversarial perturbation studies [START_REF] Antun | On instabilities of deep learning in image reconstruction and the potential costs of AI[END_REF]. Despite the lack of realism in some of these perturbations, our results indicate that hallucination and artifacts remain a real concern, particularly at higher accelerations. This topic is in major need for further development.

We note that we did not perform intensity correction for either the participant submissions or the ground truth reconstructions. None of the radiologists commented on intensity inhomogeneity in either the initial testing phase or the final evaluation phase. However, intensity correction is routinely done by vendor scanners and could have affected the evaluation, as some pathologies manifest via varying tissue intensities.

D. Feedback from Participants

We asked participants for feedback regarding challenge organization and future organization. Participants were generally enthusiastic about being able to participate in the challenge. We received positive feedback on our communication via the fastMRI GitHub repository at https://github.com/ facebookresearch/fastMRI and the forum associated with the web site at https://fastmri.org. We also received positive feedback around the challenge's realism in focusing on multi-coil data, as well as the challenge's generalizability initiative in focusing on the Transfer track.

Still, participants felt the realism could be improved in other areas. In particular, the sampling mask used for the challenge used pseudo-regular sampling in order to achieve exact 4X and 8X sampling rates. This sampling pattern is not equivalent to the perfectly equidistant sampling pattern used on MRI systems, which give acceleration rates slightly less than the target rate due to the densely-sampled center. As a result, challenge models are likely to require further fine-tuning training before application to clinical data.

Another point of feedback centered on the storage and compute resources necessary to participate in the challenge. In the 2019 challenge, the storage aspect was mitigated by the inclusion of the single-coil track (which had a smaller download size). The single-coil track attracted a lot of engagement, with 25 out of 33 groups submitting to it [START_REF] Knoll | Advancing machine learning for MR image reconstruction with an open competition: Overview of the 2019 fastMRI challenge[END_REF]. From the compute angle, the trend towards larger models requires costly hardware. Training the baseline End-to-End Variational Network [START_REF] Sriram | End-to-end variational networks for accelerated MRI reconstruction[END_REF] requires 32 GPUs, each with 32 GB of memory, for about 3.5 days. This level of compute power is not available at many academic centers. By comparison, multiple participants submitted models trained on only a single GPU. This was also topic of feedback from non-participants, with some telling us informally that they did not participate due to compute or storage requirements. For the future, researchers felt it would be helpful for the barriers to entry were lower, particularly for academic groups that might have innovative methods but less compute or storage.

As always, the selection of best quantitative evaluation metrics to use is extremely difficult and there are potential drawbacks to many or all. Participants did provide feedback concerning the use of SSIM and the use of RSS for ground truth images. Although groups acknowledged efforts to seek superior metrics, they felt that settling for this particular metric was disappointing. Some participants felt there was a tradeoff between optimizing for SSIM (which promotes smoothing) vs. radiologist interpretation. Most vendors have variations in their post-processing pipelines for precisely this reason. Some vendor post-processing methods even allow for radiologists to adjust the strength of the regularization. We did not allow secondary submissions from participants that might enhance the images for human perception, such as those based on noise dithering or inspired by stochastic resonance [START_REF] Moss | Stochastic resonance and sensory information processing: A tutorial and review of application[END_REF], [START_REF] Recht | Using deep learning to accelerate knee MRI at 3T: Results of an interchangeability study[END_REF]. Allowing secondary submissions for radiologist interpretation may be beneficial for future challenges, provided ground truth images are also included to allow radiologists to watch for hallucination. In compiling the results for this challenge we have attempted to investigate some other options that would at least mitigate the effects of background noise and feel that this is an important topic for further investigation. Consensus around evaluations-for ground truth calculations, metrics, and radiologist presentation-would substantially aid the organization of future challenges.

V. CONCLUSION

The 2020 fastMRI reconstruction challenge featured two core modifications from its 2019 predecessor: 1) a new competition Transfer track to evaluate model generalization and 2) adjusting the radiologist evaluation to focus on pathology depiction. In addition to these, we extended our competition to a new anatomy with much larger data sets for both training and competition evaluation. The competition resulted in a new state-of-the-art model. Our challenge confirmed areas in need of research, particularly those along the lines of evaluation metrics, error characterization, and AI-generated hallucinations. Radiologist sentiment was mixed for images submitted to the 8X and the Transfer tracks; these may remain open research frontiers going into 2021. We hope that researchers and future challenge organizers find the results of the 2020 fastMRI challenge helpful in their future endeavors.

Fig. 1 .

 1 Fig. 1. Examples of 4X submissions evaluated by radiologists with slicelevel SSIM scores. All methods reasonably reconstructed T2 and FLAIR images. The ATB and Neurospin methods struggled with a susceptibility region, exaggerating the focus of susceptibility and introducing a few false vessels between the susceptibility and the lateral ventricular wall. In other cases, radiologists observed mild smoothing of white matter regions on T1POST images.

Fig. 2 .

 2 Fig. 2. Examples of 8X submissions evaluated by radiologists with slicelevel SSIM scores. At this level of acceleration fine details are smoothed and obscured for all contrasts. On T1POST images, AIRS Medical was relatively more successful than ATB and Neurospin in showing fine details of the mass, particularly in its periphery. Noticeable on the FLAIR images are horizontal "banding" effects that arise from how neural networks interact with anisotropic sampling patterns.

Fig. 3 .

 3 Fig. 3. Examples of 4X Transfer submissions evaluated by radiologists with slice-level SSIM scores. The T1POST and T2 examples are from GE scanners, whereas the FLAIR example is from a Philips scanner. All methods introduced blurring to the images. Several methods had trouble adapting to the GE data while performing relatively well on the Philips data, as seen in the form of aliasing artifacts in one of the T1POST images.

Fig. 4 .

 4 Fig. 4. Summary of SSIM values across contestants. (top) Model perfomance for teams submitting to the main 4X and 8X Siemens competition tracks. (bottom) Model performance for teams submitting to the Transfer track (combination of GE and Philips data). The "AVG" model score for the Transfer track was a simple average across all volumes in the Transfer track.

Fig. 6 .

 6 Fig. 6. Examples of reconstruction hallucinations among challenge submissions with SSIM scores over residual plots (residuals magnified by 5). (top) A 4X submission from Neurospin generated a false vessel, possibly related to susceptibilities introduced by surgical staples. (middle) An 8X submission from ATB introduced a linear bright signal mimicking a cleft of cerebrospinal fluid, as well as blurring of the boundaries of the extra-axial mass. (bottom) A submission from ResoNNance introduced a false sulcus or prominent vessel.

  Scatter plot of mean radiologist rank across cases. The horizontal axis has a separate tick for each case evaluated by the radiologist cohort. The scatter plot markers indicate whether that method was from the team with the highest, middle, or lowest SSIM scores. We generally observed radiologists awarding the best ranks to models with the best SSIM score.

				Overall Rating
		1
	Mean Radiologist Rating	2 3	Determined Case-Wise: Highest SSIM
				Middle SSIM
				Lowest SSIM
			0	4X Main Reviewed Individual Cases | 8X Main | 4X Transfer
	Fig. 5.

MUCKLEY, RIEMENSCHNEIDER et al.: PREPARATION OF PAPERS FOR IEEE TRANSACTIONS ON MEDICAL IMAGING

IEEE TRANSACTIONS ON MEDICAL IMAGING, VOL. XX, NO. XX, XXXX 2020

ACKNOWLEDGMENT

We would like to thank all research groups that participated in the challenge. We thank the scientific advisors that gave feedback during challenge organization. We thank Girish M. Fatterpekar, Nael Kambiz, Amy L. Kotsenas, Christie Mary Lincoln, Joseph Mettenburg, and Max Wintermark for judging the submissions and the contribution of their clinical expertise and time. We would also like to thank Tullie Murrell, Mark Tygert, and C. Lawrence Zitnick for many valuable contributions to the launch and execution of this challenge. We give special thanks to Philips North America and their clinical partner sites for the data they contributed for use as part of the Transfer track. The NYU authors thank NIH grants P41EB017183, R01EB024532, and R21EB027241 for financial support. The ResoNNance team's submission was conducted under the STAIRS project under the TKI-PPP program. The ResoNNance collaboration project is co-funded by the PPP Allowance made available by Health Holland, Top Sector Life Sciences & Health, to stimulate public-private partnerships.

We acknowledge grant support from the NIH under awards R01EB024532, P41EB017183, and R21EB027241.


A 2D X-ray imaging crystal spectrometer (XICS) was installed on WEST [1] A remote-controlled rotating stage allows to choose one of 3 crystals, focusing on: • Intrinsic FeXXV K-alpha spectrum (~1.86 A) (intrinsic Fe, low S/N) • Injected Ar XVII K-alpha spectrum (~3.97 A) (injected Ar, good S/N) • Injected ArXVIII Lyman-alpha spectrum (~3.73 A) (for high Te, low S/N otherwise) Designed to indirectly measure: Te (line ratio), Ti (Doppler broadening), vi (Doppler shift)

INTRODUCTION

First experimental spectra and extra lines from W

Crystal box

Detector box

(PILATUS)
Rotating table

 Very accurate [2]  Remote ccontrol
Rotating laser (for horizontality checking)

Fig. 01 :

 01 Fig. 01: Sketch of the XICS set-up on WEST and alignment apparatus. The detector camera is located tangentially to a best fit of the three crystals' Rowland circles. Horizontality was checked with a rotating laser, micro-screws were used for fine-positioning of the boxes and of each crystal to determine the best focalization of a slit source, and lines of sight were finally embodied by retro-lighting using a panel of diodes on a mock detector frame.

Fig. 04 :

 04 Fig. 04: In campaign C5, one half of each crystal was covered by an Al plate, leading to the disappearance of the line doubling => experimental confirmation of the hypothesis

CEA, IRFM, F-13108 Saint Paul-lez-Durance, France. (didier.vezinet@cea.fr)

Conclusion:

• Spectrometer installed, aligned, operated • Spectra as expected except:

• W extra lines (identified by another team in [START_REF] Rice | [END_REF])


Zaccharie Ramzi

Alexandre Vignaud

Jean-Luc Starck

Philippe Ciuciu

Is good old GRAPPA dead?

We perform a qualitative analysis of performance of XPDNet, a state-of-theart deep learning approach for MRI reconstruction, compared to GRAPPA, a classical approach. We do this in multiple settings, in particular testing the robustness of the XPDNet to unseen settings, and show that the XPDNet can to some degree generalize well.

Main findings

XPDNet, a state-of-the-art deep learning approach for MRI reconstruction, can generalize well when compared to GRAPPA on unseen settings.

Introduction

Deep Learning has recently overwhelmed the field of MR image reconstruction from under-sampled data. From AUTOMAP [START_REF] Zhu | Image reconstruction by domain-transform manifold learning[END_REF] to the more recent unrolled neural networks [START_REF] Schlemper | A Deep Cascade of Convolutional Neural Networks for MR Image Reconstruction[END_REF][START_REF] Hammernik | Learning a Variational Network for Reconstruction of Accelerated MRI Data[END_REF][START_REF] Sriram | End-to-End Variational Networks for Accelerated MRI Reconstruction[END_REF][START_REF] Ramzi | XPDNet for MRI Reconstruction: an Application to the fastMRI 2020 Brain Challenge[END_REF], all of them showed the great potential of deep learning for the field, achieving ground-breaking results in quantitative metrics such as PSNR and SSIM for high acceleration factors.

However, there is a lack of comparison of these approaches to the current gold standards for periodic under-sampling, typically implemented in the MRI scanners. In this paper, we compare a state-of-the-art approach, the XPDNet [START_REF] Ramzi | XPDNet for MRI Reconstruction: an Application to the fastMRI 2020 Brain Challenge[END_REF], to GRAPPA (Generalized Autocalibrating Partially Parallel Acquisitions) [START_REF] Mark | Generalized Autocalibrating Partially Parallel Acquisitions (GRAPPA)[END_REF] on the task of reconstructing periodically under-sampled MR images in different 1 arXiv:2106.00753v1 [eess.IV] 1 Jun 2021 qualitative settings. GRAPPA is used in all the Siemens scanners, the most distributed in the world, as the default method for image reconstruction in the case of periodic under-sampling (and similar approaches for other manufacturers). However, very few deep learning papers for MRI reconstruction include a comparison to GRAPPA.

Methods

Network. The XPDNet is a type of unrolled network that secured the second place in the 2020 fastMRI brain reconstruction challenge [START_REF] Muckley | State-of-the-art Machine Learning MRI Reconstruction in 2020: Results of the Second fastMRI Challenge[END_REF]. Very basically, it unrolls the PDHG [START_REF] Chambolle | A first-order primal-dual algorithm for convex problems with applications to imaging[END_REF] algorithm using an Multi-level Wavelet CNN (MWCNN) [START_REF] Liu | Multi-level Wavelet-CNN for Image Restoration[END_REF] as the learned proximity operator. It has 25 unrolled iterations, and also features a sensitivity maps refinement module. Two networks were trained for acceleration factors 4 and 8, using retrospectively under-sampled data from the fastMRI dataset [START_REF] Zbontar | fastMRI: An Open Dataset and Benchmarks for Accelerated MRI[END_REF] with equidistant Cartesian masks1 . We chose to use non fine-tuned versions of the networks (i.e. trained on the four available imaging contrasts).

GRAPPA. We used the vanilla version of GRAPPA without noise handling. We use kernels that span 5 points in the readout direction and 2 in the phase direction. We manually set the regularisation parameter λ > 0 to obtain the best compromise between quantitative and qualitative evaluation, therefore biasing the analysis towards GRAPPA2 . We leave the analysis of a smart setting of λ for future works.

Data. We used 3 data sets to perform our comparison on:

• a brain slice from the fastMRI validation data set [START_REF] Zbontar | fastMRI: An Open Dataset and Benchmarks for Accelerated MRI[END_REF] (the state-of-the-art network was trained on the training data set), with T2 contrast, retrospectively under-sampled at acceleration factors 4 and 8;

• a brain slice acquired at a different resolution (0.25mm × 0.25mm) using a different magnetic field strength (7T), orientation and acceleration factor than the fastMRI brain data set and featuring the cerebellum (not present in the fastMRI brain data set), with T2 contrast, prospectively undersampled at acceleration factor 2 -this allows us to test the robustness of the network to somewhat unseen settings [START_REF] Marrakchi-Kacem | Robust imaging of hippocampal inner structure at 7T: in vivo acquisition protocol and methodological choices[END_REF];

• a NIST phantom, prospectively under-sampled at acceleration factor 8, acquired at 3T with 64 coils and a matrix size of 256 × 256.

All the data is periodically under-sampled with an Autocalibration Signal (ACS).

Code. We use the code linked in [START_REF] Ramzi | XPDNet for MRI Reconstruction: an Application to the fastMRI 2020 Brain Challenge[END_REF] for the network3 . We used our own implementation of GRAPPA with a TensorFlow backend4 .

Results

On the fastMRI brain slice. At acceleration factor 4, the quantitative results seem to show that the XPDNet has an overwhelmingly better image quality than GRAPPA. However, upon visual inspection of the images available in Fig. 1, we see that the image reconstructed by GRAPPA is only degraded by some noise not deteriorating its interpretability. At acceleration factor 8, the quantitative metrics once again show a clear advantage of the XPDNet over GRAPPA. This time, it is clearly confirmed by the visual inspection of the images presented in Fig. 2.

On the out-of-distribution brain slice. The image reconstructed with the XPDNet shows some faint smoothing in the cerebellum as shown in the bottom row of Fig. 3. However, the overall image is artifacts-free and very difficult to distinguish from the GRAPPA-reconstructed one.

On the NIST phantom. The phantom reconstructed using GRAPPA and XPDNet at acceleration factor 8 are very poor. For GRAPPA, the noise completely obfuscates the signal while for XPDNet the artifacts are present everywhere as can be seen in Fig. 4.

Conclusion and Discussion

The Deep Learning techniques seem ready for a substitution test at acceleration factor 4, however, they do not seem to provide an overwhelming advantage over GRAPPA visually. The acceleration factor of 8 looks like an attainable target, and it would drastically improve the image quality when compared to GRAPPA, even when using the latest noise handling techniques. We also showed that the XPDNet is robust enough to be adapted to settings relatively different from the training distribution. However, if trained on brains a network can not reconstruct objects that are too dissimilar, like phantoms. We conclude that it is therefore important to test visually the results of a reconstruction network at low acceleration factors to measure the difference compared to GRAPPA, and that the high acceleration factors are currently the real target for Deep Learning. This work demands other types of robustness and sanity tests such as (but not limited to) receiver array coil design, SNR level, contrasts, organs, and orientation.

Figure 1 :

 1 Figure 1: Magnitude reconstruction results for a specific fastMRI slice (Find which slice and which file and which contrast, part of the validation set) at acceleration factor 4. The top row represents the reconstruction using the different methods, while the bottom one represents the error when compared to the reference.

Figure 2 :Figure 3 :

 23 Figure 2: Magnitude reconstruction results for a specific fastMRI slice (Find which slice and which file and which contrast, part of the validation set) at acceleration factor 8. The top row represents the reconstruction using the different methods, while the bottom row represents the error when compared to the reference.

Figure 4 :

 4 Figure 4: Magnitude reconstruction results for a phantom acquired at acceleration factor 8. The top row represents the reconstruction using the different methods, while the bottom one represents the error when compared to the reference.

facebookresearch/fastMRI/issues/54

youtube.com/watch?v=PngT6chFy6c

github.com/zaccharieramzi/fastmri-reproducible-benchmark

github.com/zaccharieramzi/grappa 3