Malogorzata Niespodzinska

Jean-Paul Gaudet

Pascale Royer

J.-P Gaudet

J.-L Auriault

Instabilities during dispersion flow through porous media : pore-size effect

INTRODUCTION

Instabilities in dispersion flow, known as a finger ing effect, occur in many different engineering fields or technological processes which involve fluid flow through porous media (oil recovery, pollution and remediation of soils, groundwater flow, saline water intrusions, geothermal wells, filtration). It was first studied by [START_REF] Hill | Channelling in packed columns[END_REF] who carried out experiments on flow of sugar liquors due to water in columns of granu lar media. He explained the appearance of instabilities by the pressure difference due to small perturbations over the interface between the miscible fluids. [START_REF] Perrine | The Development of Stability Theory for Miscible Liquid-Liquid Displacement[END_REF] and Heller (1966) studied the effect of dif fusion on this phenomenon. [START_REF] Dumore | Stability consideration in down ward miscible displacement[END_REF] and [START_REF] Schowalter | Stability Criteria for Miscible Dis placement of Fluids from Porous Medium[END_REF] found out that both the density and the viscosity contrasts with respect to the concentra tion played an important role in fluid flow stability. [START_REF] Wooding | The Stability of a Viscous Liquid in a Vertical Tube Containing Porous Material[END_REF] established the first stability crite rion for two miscible fluids. Later, new techniques were developed to describe this phenomenon and it was studied more precisely taking into account differ ent aspects like: influence of temperature or dispersion anisotropy (Quintard, 1979[START_REF] Routaboul | Influence de l'anisotropie de disper sion sur la stabilite des deplacements miscibles en milieu poreux[END_REF][START_REF] Quintard | Stabilite des deplacements miscibles en milieu poreux homogene: injection d 'un fluid chaud dans un massif poreux sature par ce meme fluid froid[END_REF][START_REF] Routaboul | Influence de l'anisotropie de disper sion sur la stabilite des deplacements miscibles en milieu poreux[END_REF][START_REF] Routaboul | Influence de l'anisotropie de dispersion sur la stabilite des deplacements rniscibles verticaux en milieu poreux homogene[END_REF]). More recently, numerical simulations were performed [START_REF] Tan | Stability of miscible displace ments in porous media: Rectilinear flow[END_REF][START_REF] Homsy | Viscous Fingering in Porous Media[END_REF][START_REF] Zimmerman | Nonlinear viscous fingering in miscible displacement with anisotropic dis persion[END_REF][START_REF] Manickam 0 | Fingering instabilities in vertical miscible displacement flows in porous media[END_REF].

This work is aimed towards investigating experimentally and theoretically the occurrence of instabilities during vertical flow of two miscible fluids through a column packed with glass beads.

Our goal is to determine the pore size influence, which is related to the bead size. Then, we analyze experimental results and observations on the basis of the c1iterion proposed by [START_REF] Wooding | The Stability of a Viscous Liquid in a Vertical Tube Containing Porous Material[END_REF] and for the critical values calculated by [START_REF] Bues | Stabilite des deplace ments miscibles verticaux dans un milieu poreux en regime de dispersion mecanique[END_REF] by means of a perturbation method.

The fingering effect which occurs in the porous medium during unstable fluid flow is a function of different parameters such as: solute density p, vis cosity µ, flow velocity v and diameter of pores d. For simplicity, a vertical isothermal movement of two miscible incompressible fluids at constant velocity in a column packed with beads is considered. Stable and unstable flows during a vertical ascending and descending movement were investigated. The porous medium is assumed to be non-de formable, homoge nous and isotropic, and no chemical or biological reaction occurred.

EXPERIMENTS

Experimental set-up

The experimental set-up ( Simultaneously, solute samples were collected by the fraction collector at equal time. The weight of the sat urated column was measured before each experiment and checked afterwards, in order to verify that the water content volume remains unchanged.

Results

The breakthrough curves (BTC) that describe the evolution of concentration with respect to time are deduced from the conductivity values which have been measured during the experiments and also from the chemical analyses which have been afterwards per formed on the collected solute samples [START_REF] Pallud | Etude multi-echelles du fonctionnement hydrodynamique et microbiologique d 'un sol soumis Ii un apport de solute: application Ii !'ammonium et au 2[END_REF].

Let def i the relative concentration C/C0 by:

(1)

The normalized BTC-curves that describe the evolu tion of the relative concentration C/C0 with respect to the relative volume VN0 are plotted on Figure 2. The analysis of these breakthrough curves is performed by applying the moment method [START_REF] Jury | Transfer Functions and Solute Movement through Soil[END_REF]Schoen et al., 1999): for each curve the successive temporal moments of order 0 and 1 are numerically calculated in order to determine the mass balance MB, the average residence time t, of the solute in the column and the retardation factor R [START_REF] Jury | Transfer Functions and Solute Movement through Soil[END_REF]. The dispersion coefficient D can be determined by comparing the experimental breakthrough curves with the theoretical breakthrough curves (see Fig. 2). These latter are obtained by solving the convection dispersion model (CD). In case a, the curve 2a is obtained whatever the flow direction. Therefore, the flow is stable. It can be seen from curve 2a that the experimental results are in good agreement with the theoretical results obtained by solving Eq. 2. The mass balance is equal to MB1 = 0.98 (MB = 1 means that the total injected mass was gathered at the exit of the column). The retardation factor R � 1.04 denotes that there is no chemical reac tion between the solutes and the porous medium. The dispersion coefficient is found to be D1 = 2.5 cm2/h. When the bead size is d2 = 4mm (cases b and c), the flow direction has an influence (Fig. 2b,2c). Both curves are significantly different: on Fig. 2b a tailing effect appear whereas the experimental curve 2c is oscillating. In both cases, the mass balance is low (MB2 = 0.75) and the experimental curves are non-symmetrical. The dispersion coefficient deter mined from the rising part of curve 2b is found to be D2 = 4cm2/h.

THEORETICAL ANALYSIS

.1 Stability criterion

To analyse the above experimental results, let consider the non-dimensional criterion for hydrodynamic sta bility established by [START_REF] Wooding | The Stability of a Viscous Liquid in a Vertical Tube Containing Porous Material[END_REF] and studied later by [START_REF] Bachmat | Hydrodynamic Instability of Miscible Fluids in a Vertical Porous Column[END_REF]:

g•k•d 2 •(D.p/L) 2 c < {T n •µ,•Dr ' (3)
in which g is the earth's gravitation, k is the intrinsic permeability of the porous medium, de is the col umn diameter, D.p is the density contrast between the miscible fluids, Lm is the mixing length, n is the cin ematic porosity equal to the water contents es. µ, is the dynamic viscosity, Dr is the transversal disper sion coefficient and a2 is the critical parameter value for stable fluid flow. For critical density contrast, Eq. 3 reduces to:

K•d'1 ---< a 2 , n•L•Dr (4)
in which K denotes the hydraulic conductivity of the medium.

For investigating the problem of instabilities, Bues (1987) applied a method of perturbations and deter mined the possible values of a 2 . From these results and from Eq. ( 4), critical values for the mixing length Lm can be determined.

Results

The values of K were estimated to be approximately 10 cm/h and 40 cm/h for the glass beads of diameter 1 mm and 4 mm, respectively. When the bead size is of 4 mm, we obtained a critical value for the mix ing length of Lmin = 35.23 cm. The critical length Lmin is therefore greater than the length of the column L, which means that the stability conditions cannot be met. This is in agreement with the experimental results. For beads of diameter 1 mm, Lm i n is found to be 1.85 cm: the stability conditions can therefore be reached.

DISCUSSION OF RESULTS

In order to explain the occurrence of instabilities, we examine the stabilizing and destabilizing mechanisms. The stabilizing mechanisms are: viscosity, dispersion and increase of fluid velocity, while the destabiliz ing mechanisms include gravitation, decrease of fluid velocity and increase of pore size (related to the bead size). When the most concentrated solute B pushes the less concentrated solute A, the destabilizing force which is acting is gravity. In this case, the pore size is small, and the viscosity forces and dispersion effects are suf ficiently large to balance the destabilizing effect of gravity. As a consequence, instabilities which may occur in the mixing zone between both miscible fluids are naturally reduced. Even if some local instabilities appear over the interface, the global flow remains sta ble whatever the flow direction and the configurations of solutes (the most concentrated solute above the less concentrated solute or inversely).

4.2 Unstable upward fluid flow (Figure 2b)

In this case, the most dense solute B is injected into the column and pushes up the less dense solute A. In this configuration, the fluid flow is stable (the rising part of the breakthrough curve of Figure 2b fits well the the oretical curve calculated with the CD model) despite large size of glass beads. When the injection of solute B is stopped, solute A is then injected. Solute B is thus pushed by the less dense solute A. The viscosity forces and dispersion effects are insufficient to balance the gravity force. Instabilities appear which can be seen by the occurrence of small fingers and by the tail ing effect on the descending part of the breakthrough curve on Figure 2b. Fingering occurs as a result of solute configuration: the most concentrated solute is located above the less concentrated solute.

Unstable downward fluid flow (Figure 2c)

In this case, instabilities occur on the rising part of the breakthrough curve (the densest solute is located above the less dense solute). However, the flow remained stable on the descending part of the curve when solute A pushes down solute B.

CONCLUSIONS

On the basis of the above experimental and theoretical results, we conclude that instabilities may occur when the most concentrated solute is located above the less concentrated solute and when the pore size is quite large.

In order to accurately interpret non-uniformities on breakthrough curves, a rectangular impulse should be injected during both ascending and descending flow: obtaining identical curves for both flow directions ensures the absence of instabilities.

  Figure l. Experimental set-up.

Figure 2

 2 Figure 2 shows three different experimental results corresponding to different flow directions and bead sizes. a) stable (ascending or descending) flow; bead size: Imm b) unstable ascending flow; bead size: 4 mm c) unstable descending flow; bead size: 4 mm. Comparison of the results calculated from the CD model (solid line) with the experimental breakthrough curves (points) shows the influence of pore size.In case a, the curve 2a is obtained whatever the flow direction. Therefore, the flow is stable. It can be seen from curve 2a that the experimental results are in good agreement with the theoretical results obtained by solving Eq. 2. The mass balance is equal to MB1 = 0.98 (MB = 1 means that the total injected mass was

Figure 2 .

 2 Figure 2. Experimental and calculated breakthrough curves during: a) bead size: d1 =Imm; stable flow, b) bead size: d2 = 4mm; unstable ascending flow, c) bead size: d2 = 4mm; unstable descending flow.

4. 1

 1 Stable fluid flow (Figure2a)


Sharib Ali
email: sharib.ali@eng.ox.ac.uk

Mariia Dmitrieva

Noha Ghatwary

Sophia Bano

Gorkem Polat

Alptekin Temizel

Adrian Krenzer

Amar Hekalo

Yun Bo Guo

Bogdan Matuszewski

Mourad Gridach

Irina Voiculescu

Vishnusai Yoganand

Arnav Chavan

Aryan Raj

Nhan T Nguyen

Dat Q Tran

Duy Le

Huynh

Nicolas Boutry

Shahadate Rezvy

Haijian Chen

Yoon Ho Choi

Anand Subramanian

Velmurugan Balasubramanian

Xiaohong W Gao

Hongyu Hu

Yusheng Liao

Danail Stoyanov

Christian Daul

Stefano Realdon

Renato Cannizzaro

Dominique Lamarque

Terry Tran-Nguyen

Adam Bailey

Barbara Braden

James East

Jens Rittscher

Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Endoscopy is a widely used imaging technique for both diagnosis and treatment of patients with complications in hollow organs such as esophagus, stomach, colon, bladder, kidney and nasopharynx. During the endoscopic procedure, an endoscope, a long thin tube with a light source and a camera at its tip, is inserted into the organ cavity. The imaging procedure is usually displayed on a monitor on-the-fly and is often recorded for post analysis. Each organ imposes very specific constraints to the use of endoscopes, but the most common obstructions in all endoscopic surveillance consists of artefacts caused by motion, specularities, low contrast, bubbles, debris, bodily fluid and blood. These artefacts hinder the visual interpretation of clinical endoscopists [START_REF] Ali | An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy[END_REF] . Missed detection rates of precancerous and cancerous lesions are another limitation. Gastrointestinal (GI) cancer (especially colorectal cancer) has high mortality rates and 5-year relative survival rates for stage IIB is around 65% [START_REF] Rawla | Epidemiology of colorectal cancer: incidence, mortality, survival, and risk factors[END_REF] . In general, the missed detection rates in endoscopic surveillance is considerably high, at over 15% [START_REF] Lee | Risk factors of missed colorectal lesions after colonoscopy[END_REF] . Therefore, the requirement for technology that can be effectively used in clinical settings during endoscopy imaging is necessary.

While a dedicated endoscopic procedure is followed for each specific organ, often these procedures are very similar, in particular for the GI tract organs like the esophagus, stomach, small intestine, colon and rectum. Notably, some precancerous abnormalities such as inflammation or dysplasia and even cancer lesions in these GI organs naturally look very similar. Often automated methods are only trained for a specific abnormality, organ and imaging modality [START_REF] Zhang | Real-time gastric polyp detection using convolutional neural networks[END_REF] , whereas multiple different types of abnormalities can be present in different organs and several imaging protocols are used during endoscopy. Also, methods that are built for colonoscopy cannot be used during a gastroscopy (in the esophagus, stomach and small intestine), despite the nature and occurrence of many abnormalities being similar in these organs. Artefacts are prevalent in all endoscopy surveillance and are usually confused with lesions, which can lead to unreliable outcomes.

A pathway to develop and reliably deploy methods in clinical settings is by benchmarking methods on a curated multicenter, multi-modal, multi-organ and multi-disease dataset and through a thorough evaluation of built methods using standard imaging metrics and metrics that can test their clinical applicability, for example ranking based on accuracy, robustness and computational efficiency [START_REF] Ali | An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy[END_REF] . Most publicly available datasets are specific to a particular organ, modality or a single abnormality class, e.g., polyp detection and segmentation challenges [START_REF] Bernal | Comparative validation of polyp detection methods in video colonoscopy: results from the miccai 2015 endoscopic vision challenge[END_REF][START_REF] Jorge | GastrointestinalG imageI analysisANA challenge[END_REF] . While dedicated organ specific challenges help to identify one particular disease type, they do not resemble the clinical workflow where the endoscopists are interested in biopsy and treatment of such abnormalities when of potential threat. For polyp class, it is required to identify different stages of polyp such as benign, dysplastic or cancer. Recently, it was shown that polyps and artefacts can be confused mostly due to specularity [START_REF] Soberanis-Mukul | A learning without forgetting approach to incorporate artifact knowledge in polyp localization tasks[END_REF] . Artefacts are the fundamental and inevitable issue in endoscopy that often add confusion in detecting tissue abnormalities in these organs. It is therefore vital to accelerate research in identifying these classes and restore frames where possible [START_REF] Ali | A deep learning framework for quality assessment and restoration in video endoscopy[END_REF] or reduce the false detections by adding uncertainties for such confusions [START_REF] Soberanis-Mukul | A learning without forgetting approach to incorporate artifact knowledge in polyp localization tasks[END_REF] . Other ways to address artefact problems in the endoscopy data is by using synthetically generated frames [START_REF] Mahmood | Unsupervised reverse domain adaptation for synthetic medical images via adversarial training[END_REF][START_REF] Formosa | Nonlinear Dynamic Modeling of a Robotic Endoscopy Platform on Synthetic Tissue Substrates[END_REF][START_REF] Incetan | Vr-caps: A virtual environment for capsule endoscopy[END_REF] . 9 used self-regularized transformer network that allowed to transform the real images into synthetic-like images with preserved clinically-relevant features. This allowed the authors to estimate depth in colonoscopy data robustly without being affected by adverse artefact problems. [START_REF] Incetan | Vr-caps: A virtual environment for capsule endoscopy[END_REF] demonstrated the use of a virtual active capsule environment that can simulate wide range of normal and abnormal tissue conditions such as inflated, dry and wet; organ types and endoscopy camera designs in capsule endoscopy. This allowed to optimize the analysis software for varied real conditions.

The Endoscopy Computer Vision Challenge (EndoCV2020) [START_REF] Ali | An objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy[END_REF] is another crowd-sourcing initiative to address fundamental problems in clinical endoscopy and consists of: 1) Endoscopy artefact detection and segmentation (EAD2020), and 2) samples. Both single frame samples (top) and sequence frames (bottom) were released. While detection annotations involve 8 classes, segmentation classes were limited to 5 distinct class instances, mostly large indefinable shapes that include specularity, saturation, imaging artefact, bubbles and instrument. It can be observed that for sequence data most artefact instances follow upto few sequential frames so it is desirable to achieve such training datasets. 4th sample in the single frame data for segmentation shows that even though bounding boxes for detection are provided for all specular regions, some segmentation labels were missing. This shows the presence of annotator variability in the data. (b) Endoscopy disease detection and segmentation training samples for sub-challenge EDD2020. First four samples belong to esophageal endoscopy while the last two frames were acquired during colonoscopy. It can be observed that disease classes in esophagus confuse often, mostly the patient choice here is Barrett's where clearly suspected and high-grade dysplasia appear jointly. Similarly, for colonoscopy data protruded polyps can easily be confused with the surrounding ridge-like openings and specular areas.

Endoscopy disease detection and segmentation (EDD2020). EndoCV2020 releases diverse datasets that include multi-center, 3/32 multi-modal, multi-organ, multi-disease/abnormality, and multi-class artefacts. Among the two sub-challenges, EAD2020 is an extended sub-challenge of EAD2019 [START_REF] Ali | Endoscopy artifact detection (EAD 2019) challenge dataset[END_REF] , however, unlike EAD2019 it includes both frame and sequence data with an addition of nearly 500 frames and a total of 41,832 annotations for detection task and 10,739 for segmentation task.

In this paper, we summarise and analyse the results of the top 17 (out of 43) teams participating in the EndoCV2020 challenge. Additionally, we benchmark these methods with the current state-of-the-art detection and segmentation methods. Each method is also evaluated for its efficacy to detect and segment multi-class instances. In addition to the standard computer vision metrics used to evaluate methods during the challenge, we perform a holistic analysis of individual methods to measure their clinical applicability.

Related work

With the advancements in deep learning for computer vision, object detection and segmentation algorithms have shown rapid development in recent years. This is due to the hidden feature representations provided by Convolutional Neutral Networks (CNNs) that show significant improvement over hand-crafted features. CNN-based methods quickly gained the attention of the Medical Imaging community and are now widely used for automating the diagnosis and treatment for a range of imaging modalities, e.g. radiographs, CT, MRI, and endoscopy imaging. Below we present an overview of the recent deep learning-based object detection and segmentation techniques and discuss the related work in the context to medical image analysis with a particular focus on endoscopy imaging applications.

Detection and localization

Object detection and localization refers to determining the instances of an object (from a list of predefined object categories) that exist in an image. Object detection approaches can be broadly divided into three categories: single-stage, multi-stage and anchor-free detectors. A brief survey of these is presented below.

Single-stage detectors Single-stage networks perform a single pass on the data and incorporate anchor boxes to tackle multiple object detection on the same image grid such as in YOLO-v2 [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF] . Similarly, Liu et al. [START_REF] Liu | Ssd: Single shot multibox detector[END_REF] proposed the Single Shot MultiBox Detector (SSD) with additional layers to allow detection of multiple scales and aspect ratios. RetinaNet was introduced by Lin et al. [START_REF] Lin | Focal loss for dense object detection[END_REF] where the authors introduced focal loss that puts the focus on the sparse hard examples enabling a boost in performance and speed.

The domain of Gastroenterology has started to benefit from the success of single-stage object detectors. Wang et al. [START_REF] Wang | Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy[END_REF] proposed a model that is based on SegNet 17 architecture to detect polyps during colonoscopy. Urban et al. [START_REF] Urban | Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy[END_REF] used YOLO to detect polyps from colonoscopy images in real-time. Horie et al. [START_REF] Horie | Diagnostic outcomes of esophageal cancer by artificial intelligence using convolutional neural networks[END_REF] used SSD to detect superficial and advanced esophagal cancer. RetinaNet was the most popular detector in the first EAD challenge held in 2019. RetinaNet detector with focal loss was used by some top performing teams [START_REF] Kayser | Focal loss for artefact detection in medical endoscopy[END_REF][START_REF] Oksuz | Artefact detection in video endoscopy using retinanet and focal loss function[END_REF] Multi-stage detectors Multi-stage detectors use a region proposal network to find regions of interest for objects and then a classifier to refine the search to get the final predictions. A two-stage architecture R-CNN using the classical region proposal method was proposed by Girshick et al. [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] whose speed was improved later by integrating an end-to-end trainable region proposal network (RPN), widely known as Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] . Due to the high precision of the Faster R-CNN, its architecture has become the base for many successful models in the object detection and segmentation domains, such as Cascade R-CNN [START_REF] Cai | Cascade r-cnn: Delving into high quality object detection[END_REF] and Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] . Although these two-stage networks have shown successful results on public datasets such as Pascal VOC [START_REF] Everingham | The PASCAL Visual Object Classes Challenge[END_REF] and COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] , they are slow compared to the single-stage object detectors due to their region proposal mechanism.

In the field of Gastroenterology, Yamada et al. [START_REF] Yamada | Development of a real-time endoscopic image diagnosis support system using deep learning technology in colonoscopy[END_REF] used Faster R-CNN with VGG16 as the backbone to detect challenging lesions which are generally missed by colonoscopy procedures. Their reported prediction speed was not suitable for real-time examination. Shin et.al. [START_REF] Shin | Automatic colon polyp detection using region based deep cnn and post learning approaches[END_REF] detected Polyps using the Fast R-CNN architecture with a region proposal network and an inception ResNet backbone. The two-stage detectors tend to yield better results than their single-stage contemporaries and have performed better at medical image analysis challenges. In the EAD2019 challenge, the top performing team 30 used a Cascade R-CNN with a feature pyramid network (FPN) module and a ResNet backbone. Similarly, Pengyi Zhang et.al. [START_REF] Zhang | Ensemble mask-aided r-cnn[END_REF] who used Mask aided R-CNN with an ensemble of different ResNet backbones finished second.

Anchor-free detectors A newly emerging detector type are the anchor-free detectors. Single and multi-stage detectors rely on the presence of anchors. Anchor free architectures claim to detect objects while skipping the process of anchor definition. They rely on different geometrical characteristics like the center or corner points of objects [START_REF] Law | Cornernet: Detecting objects as paired keypoints[END_REF][START_REF] Duan | Centernet: Keypoint triplets for object detection[END_REF] . Duan et al. [START_REF] Duan | Centernet: Keypoint triplets for object detection[END_REF] utilized the upper left and lower right corner to mark an object. The authors used classical backbones to generate a heatmap from the feature map showing potential spots of the object corners. A corner pooling technique was then used to create the classic bounding box of object detection. Zhou et al. [START_REF] Zhou | Bottom-up object detection by grouping extreme and center points[END_REF] used a similar approach but instead they used a single point as the center of the bounding box.

The EndoCV challenge: Dataset, evaluation and submission

In this section, we present the dataset compiled for the EndoCV2020 challenge, the protocol used to obtain the ground truth for this data, evaluation metrics that were defined to assess participants methods and a brief summary on the challenge setup and ranking procedure.

Dataset and challenge tasks

The EndoCV2020 challenge consists of two sub-challenges critical in clinical endoscopy. The EAD20204 sub-challenge comprises of diverse endoscopy video frames collected from seven institutions worldwide, including three different modalities and five different human organs (see Figure 2). Endoscopy video frames were annotated for detection and localization of eight different artefact class occurrences identified by clinical experts in the challenge team. These include specularity, saturation, misc. artefacts, blur, contrast, bubbles, instrument and blood. A total of 280 patient videos from multiple organs and institutions have been used for curating this dataset. Over 45,478 annotations were performed for this challenge on both single frame and sequence video data. Example annotations are shown in Figure 1. Training data for the detection task consisted of total 2,531 frames with 31,069 bounding boxes while 643 frames with 7,511 binary masks were released for the segmentation task (except for blur, blood and contrast). The sequence data were sampled by manually observing the amount of changes in artefact categories in the selected sequence. Sequences were required to change from large areas of artefacts to small or no artefact frames and vice versa mimicking natural occurrence in endoscopic procedures. Sequence data for training included 5 sequences (232 frames) for detection and 2 sequences (70 frames) for semantic segmentation tasks sampled from 3 videos of 3 different patients. For the test set, two sequence (80 frames) for detection task were used from 2 independent patient videos. As observed in Figure 2, due to the nature of occurrence of various artefact classes, the proportion of annotations for each class is different (Figure 3). However, the proportion of training and test samples per-class were matched in the test data (also see Table 1). Separately, EDD2020 [START_REF] Bernal | Comparative validation of polyp detection methods in video colonoscopy: results from the miccai 2015 endoscopic vision challenge[END_REF] is a new disease detection and segmentation sub-challenge that consists of five disease categories [START_REF] Ali | Endoscopy disease detection challenge[END_REF] . The provided training set consisted of total 385 video frames comprising of 137 different patients used in this study with a total of 817 individual annotations. The annotations included non-dysplastic Barrett's esophagus (NDBE), suspicious, high-grade dysplasia (HGD), cancer, and polyp categories (also see Figure 1). These disease classes were from three different endoscopic modalities (white light, narrow-band imaging, and chromoendoscopy) acquired from four different clinical centers, investigating four different GI organs. By including varied range of endoscopy data acquired from multiple organs like GI tract and liver in EAD sub-challenge and both upper and lower GI tract data for EDD sub-challenge, EndoCV2020 challenge aimed at developing more general methods that can potentially be applied in different endoscopy routine procedures independent to organ type. To our knowledge, this is the first comprehensive dataset for the multi-class detection and segmentation tasks. More details on the dataset are provided in Figure 2. The detailed breakdown of training set and test set for each specific task is provided in Table 1.

EndoCV2020 posed three specific challenge tasks (see Figure 4) that included: 1) detection and localization task, 2) semantic segmentation task and 3) out-of-sample generalization task. For detection and generalization tasks, participants were provided with both frame label annotations for single and sequence images for the EAD2020 challenge while only single frames were released for EDD2020. The generalization task was only evaluated for the EAD2020 and only consisted of test data from an unseen institution that was not present in any training set. It is to be noted that test samples for all other tasks were taken from different patients as well even though they were collected from the same centers as that in the training set. EAD2020 attracted nearly 700 participants with 29 teams on the leaderboard and EDD2020 recorded nearly 550 participants with 14 teams on the leaderboard. Participation was permitted in either one or both sub-challenges. Both challenge datasets are publicly available for research and education. EAD2020 challenge data is available at Mendeley Data (10.17632/c7fjbxcgj9.3) and EDD2020 dataset is available at IEEE dataPort (http://dx.doi.org/10.21227/f8xg-wb80).

Ethical and privacy aspects of the data

Data for EAD2020 were collected from 7 different centers while for EDD2020 were from 4 centers. Each center was responsible for handling the ethical, legal and privacy of the relevant data sent to the challenge organizers. The data collection from each center included either two or all essential steps described below:

1. Patient consenting procedure at the home institution (required)

2. Review of the data collection plan by a local medical ethics committee or an institutional review board 3. Anonymization of the video or image frames (including demographic information) prior to sending to the organizers (required)

Table 2 illustrates the ethical and legal processes fulfilled by each center along with the endoscopy equipment and recorders used for the data collected for this challenge.

Annotation protocol

A team of two clinical experts and one post-doctoral researcher determined the class labels for the artefact detection challenge while for the disease detection challenge we consulted with four senior Gastroenterologists (over 20 years experience) regarding the class labels in the GI tract endoscopy. For each sub-challenge senior Gastroenterologists sampled the video frames from a small sub-set of video data collected from various institutions and multi-patient data cohort (see Figure 2). These frames were then taken as reference to produce bounding box annotations for the remaining train-test dataset by four experienced postdoctoral fellows. Finally, further validation by three clinical endoscopists independently was carried out to assure the reference standard. The ground-truth labels were randomly sampled (1 per 20 frames) during this process. However, after the completion of this phase the entire annotation was discussed and reviewed together with the team of senior Gastroenterologists. Priority was given to indecisive frame annotations to have a collective opinion from experts. Following general annotation 9/32 strategies were used by clinical experts and researchers:

8/32
• For the same region, multiple boxes (for detection/generalization) or pixel-wise delineation (for semantic segmentation) were performed if the region belonged to more than 1 class

• The minimal box sizes were used to describe the class region and similarly possible small annotation areas for semantic segmentation were merged instead of having multiple small boxes/regions

• Each class type was determined to be distinctive and general across all datasets

For EAD dataset, defined class categories used included below descriptions [START_REF] Ali | A deep learning framework for quality assessment and restoration in video endoscopy[END_REF] . Related samples are presented in Fig. 1 (a 1. NDBE or BE → non-dysplastic Barrett's esophagus determined by a squamo-columnar junction above the gastric fold in the esophagus [START_REF] Eluri | Barrett's esophagus: diagnosis and management[END_REF] 2. HDG → high-grade dysplasia or early adenocarcinoma determined by irregular mucosal appearance [START_REF] Wang | Diseases of the esophagus : official journal of the international society for diseases of the esophagus[END_REF] 3. suspected → aka low-grade dysplasia, an early sign of pathology [START_REF] Eluri | Barrett's esophagus: diagnosis and management[END_REF] 4. cancer → abnormal growth [START_REF] Boland | Infection, inflammation, and gastrointestinal cancer[END_REF] 5. polyp → abnormal protrusion of the mucosa [START_REF] Williams | Management of the malignant colorectal polyp: Acpgbi position statement[END_REF] For the annotations of disease classes, pathology reports were also used to validate the class category for non-dysplastic Barrett's esophagus (BE), high-grade dysplasia (HGD), suspected (dysplasia or low-grade dysplasia), and cancer categories. That is, expert annotations (three senior gastroenterologists) were taken and supported with the pathology report for most disease categories including some indecisive cases. However, for the polyp class, both the protruded and flat polyps were marked by two experienced post-doctoral researchers and checked by a senior lower-GI specialist (no further categorization based on pathology report was done except for cancer cases).

Evaluation metrics

The challenge problems fall into three distinct categories. For each there already exist well-defined evaluation metrics used by the wider imaging community which we use for evaluation here. Codes related to all evaluation metrics used in this challenge are also available online [START_REF] Jorge | GastrointestinalG imageI analysisANA challenge[END_REF] .

10/32

Spatial localization and classification task

Metrics used for multi-class disease detection:

• IoU -intersection over union: This metric measures the overlap between two bounding boxes A and B, where A is segmented region and B is annotated GT. It is evaluated as the ratio between the overlapped area A ∩ B over the total area A ∪ B occupied by the two boxes:

IoU = A ∩ B A ∪ B (1)
where ∩, ∪ denote the intersection and union respectively. In terms of numbers of true positives (TP), false positives (FP) and false negatives (FN), IoU (aka Jaccard JC) can be defined as:

IoU/JC = T P T P + FP + FN (2)
• mAP -mean average precision: mAP of detected class instances is evaluated based on precision (p) defined as p = T P T P+FP

and recall (r) as r = T P T P+FN . This metric measures the ability of an object detector to accurately retrieve all instances of the ground truth bounding boxes. Average precision (AP) is computed as the Area Under Curve (AUC) of the precision-recall curve of detection sampled at all unique recall values (r 1 , r 2 , ...) whenever the maximum precision value drops:

AP = ∑ n (r n+1 -r n ) p interp (r n+1 ) , (3)
with p interp (r n+1 ) = max r≥r n+1 p(r). Here, p(r n ) denotes the precision value at a given recall value. This definition ensures monotonically decreasing precision. The mAP is the mean of AP over all N classes given as

mAP = 1 N N ∑ i=0 AP i (4)
This definition was popularised in the PASCAL VOC challenge [START_REF] Everingham | The PASCAL Visual Object Classes Challenge[END_REF] . The final mAP (mAP d ) was computed as an average mAPs for IoU from 0.25 to 0.75 with a step-size of 0.05 which means an average over 11 IoU levels is used for 5 categories in the competition (mAP @[.25 : .05 : .75] ).

Participants were finally ranked on a final mean score (score d ), a weighted score of mAP and IoU represented as:

score d = 0.6 × mAP d + 0.4 × IoU d (5)
Standard deviation between the computed mAPs (±σ score d ) are taken into account when the participants have the same score d .

Scores on both single frame data and sequence data were first separately computed and then averaged to get the final score d of the detection task.

Segmentation task

Metrics widely used for multi-class semantic segmentation of disease classes have been used for scoring semantic segmentation.

The final semantic score score s comprises of an average score of F 1 -score (Dice Coefficient, DSC), F 2 -score, precision (PPV), recall (Rec) and accuracy (Acc). Precision, recall, F β -scores: These measures evaluate the fraction of correctly predicted instances. Given a number of true instances #GT (ground-truth bounding boxes or pixels in image segmentation) and number of predicted instances #Pred by a method, precision is the fraction of predicted instances that were correctly found, PPV = #TP #Pred. where #TP denotes number of true positives and recall is the fraction of ground-truth instances that were correctly predicted, Rec = #TP #GT . Ideally, the best methods should have jointly high precision and recall. F β -scores gives a single score to capture this desirability through a weighted (β ) harmonic means of precision and recall,

F β = (1 + β 2 ) • PPV •Rec (β 2 •PPV )+Rec .
Participants are ranked based on the value of their semantic performance score given by: score s = 0.25

× (p + r + F 1 + F 2 ) (6)
Standard deviation between each of the subscores are computed and averaged to obtain the final ±σ score s which is used during evaluation for participants with same final semantics score. We have also used provided accuracy of each semantic method in this paper for scientific completeness. Accuracy (Acc) can be defined as Acc =

T P+T N T P+T N+FP+FN .

11/32

Out-of-sample generalization task

Out-of-sample generalization of disease detection is defined as the ability of an algorithm to achieve similar performance when applied to a completely different institution data. To assess this, participants were challenged to apply their trained models on video frames that were neither included in the training nor in the test data of the other tasks. Assuming that participants applied the same trained weights, the out-of-sample generalization ability was estimated as the mean deviation between the mAP score of the detection and out-of-sample generalization test datasets of each class i for deviation greater than a tolerance of {0.1 × mAP i d }.

dev g = 1 N ∑ i dev g i ( 7
)

dev g i = 0, for |mAP d i -mAP g i |/mAP d i ≤ 0.1 |mAP d i -mAP g i |, for |mAP d i -mAP g i |/mAP d i > 0.1 (8)
The best algorithm should have high mAP g and low dev g (→ 0). Participants were finally ranked using a weighted ranking score for out-of-sample generalization as R gen = 1/3 • Rank(dev g ) + 2/3 • Rank(mAP g ) where Rank(mAP g ) is the rank of a participant when sorted by mAP g in ascending order.

Challenge setup, and ranking procedure

The challenge proposal was submitted to the IEEE ISBI challenge organisers and was peer-reviewed by two reviewers. Upon the acceptance, the challenge website [START_REF] Soberanis-Mukul | A learning without forgetting approach to incorporate artifact knowledge in polyp localization tasks[END_REF] was launched on 1st November 2019. Training datasets for each sub-challenge (EAD and EDD) were first provided (via AWS amazon S3 for EAD data and IEEE data portal for EDD data [START_REF] Ali | A deep learning framework for quality assessment and restoration in video endoscopy[END_REF] ). The test data was released nearly 20 days before the leaderboard closing through a docker container set-up. A docker based online leaderboard was established separately for EAD2020 [START_REF] Mahmood | Unsupervised reverse domain adaptation for synthetic medical images via adversarial training[END_REF] and EDD2020 [START_REF] Formosa | Nonlinear Dynamic Modeling of a Robotic Endoscopy Platform on Synthetic Tissue Substrates[END_REF] where each participating team was allowed to submit a maximum of 2 submissions per day on the final test data. A wiki-page [START_REF] Incetan | Vr-caps: A virtual environment for capsule endoscopy[END_REF] was set-up for the submission guidelines and a code repository with evaluation metrics used in the challenge was also provided [START_REF] Ali | Endoscopy artifact detection (EAD 2019) challenge dataset[END_REF] .

For the ranking of different task categories, we used the metrics described in Section 3.2. The participants were able to see only the final score in the leaderboard and all other sub-scores were hidden for the final test data. This was done to avoid any class specific refinement on the released test set. Notably, the detection task was bounded by two IoU thresholds (mAP @ IoU thresholds [.25 : .05 : .75]) and the overall IoU scores itself. For the detection task, participants were ranked on a final weighted score of mAP and IoU (see Eq. ( 5)), while for the segmentation task, participants were ranked based on a final weighted average of DSC or F1-score, F2-score, precision and recall (see Eq. ( 6)). For the generalization task, both the mAP score gap dev g and mAP on generalization data mAP g were taken into account.

Method summary of the participants

In this Section, we present summary of top participating teams for both EAD2020 and EDD2020 sub-challenges. Each of these teams has participated in either detection task or segmentation task or both.

EAD2020 Participating teams

• Team polatgorkem [START_REF] Polat | Endoscopic artefact detection with ensemble of deep neural networks and false positive elimination[END_REF] The team used an ensemble of three object detectors: Faster R-CNN (ResNet50 with FPN), Cascade R-CNN (ResNet50 with FPN), RetinaNet (ResNet101 with FPN). Class-agnostic NMS operation, where the model predictions were passed through the NMS procedure together for all classes, was applied to the output of each individual model. During ensemble, only the bounding boxes for which majority of the models agree were kept. False-positive elimination was applied as a post-processing step to eliminate same-type predicted boxes located close to each other. For each class, an IoU threshold was determined.

• Team CVML 76 CVML team's model was inspired by DeepLabV3+. The team experimented with several changes including the backbone, the global pooling, the dilated kernels and the convolution kernels with dilation rates. Moreover, the squeeze-and-excitation module is added behind the balanced ASPP module to introduce attention gating at the output of the original encoder to better utilize the information available in the computed feature maps. In addition, the original 7 https://endocv.grand-challenge.org 8 https://ieee-dataport.org/competitions/endoscopy-disease-detection-and-segmentation-edd2020 9 https://ead2020.grand-challenge.org/evaluation/leaderboard/ 10 https://edd2020.grand-challenge.org/evaluation/leaderboard/ 11 https://github.com/sharibox/EndoCV2020/wiki 12 https://github.com/sharibox/EndoCV2020 12/32 Table 3. Endoscopy artefact detection and segmentation (EAD2020) method summary for top 13 teams (out-of 33 valid submissions). multi-class classifier is replaced with 5 binary classifiers to enable segmentation of the overlapping objects. At test time, they used some post-processing techniques such as rotation, holes filling and removal of objects from the image boundary.

• Team mouradai_ox [START_REF] Gridach | OXENDONET: A dilated convolutional neural networks for endoscopic artefact segmentation[END_REF] The team proposed a novel neural network called OxEndoNet to tackle the segmentation challenge.

The network uses the pyramid dilated module (PDM) consisting of multiple dilated convolutions stacked in parallel. For each input image, pre-trained ResNet50 (on ImageNet) was used as the backbone to extract the feature map followed by multiple PDM layers to form an end-to-end trainable network. In the final architecture, they used four PDM layers; each layer used four parallel dilated convolutions with a filter size of 3 × 3 and dilation rates of 1, 2, 3, and 4. They fed the final PDM layer to a convolution layer followed by a bilinear interpolation to up-scale the feature map to the original image size.

• Team mimykgcp [START_REF] Prakash | A submission note on EAD 2020: Deep learning based approach for detecting artefacts in endoscopy[END_REF] The team re-trained the ResNeXt101 backbone with the cardinality parameter set to 64. To enable detection of artefacts at different scales, an FPN was integrated into the object detectors. Data-Augmentation techniques based on RandAugment [START_REF] Cubuk | Practical data augmentation with no separate search[END_REF] were incorporated to improve the generalization capability. For the segmentation task, a U-Net with an ImageNet pre-trained ResNext50 backbone was used.

• Team DuyHUYNH 80 For segmentation, the team exploited a model based on U-Net++ using pre-trained EfficientNet on ImageNet as the backbone. The model was trained to minimize F2-loss using the Adam optimizer. At the test-time the team used five transformations: horizontal, vertical flipping, and three rotations. For detection, the team used the bounding boxes deduced from the results of their segmentation model on the EDD dataset, while for EAD, they used YOLOv3 pre-trained on COCO.

• Team mathew666 [START_REF] Hu | Endoscopic artefact detection in mmdetection[END_REF] The team used Cascade RCNN architecture with the ResNeXt backbone in a FPN based feature extraction paradigm. Data augmentation with probability of 0.5 for horizontal flip was applied. The team also utilised multi-scale detection to tackle with variable sized object detection.

• Team arnavchavan04 [START_REF] Jadhav | Multi-plateau ensemble for endoscopic artefact segmentation and detection[END_REF] For the object detection task, the team used an ensemble of three models: Faster R-CNN (ResNext101 + FPN), RetinaNet (ResNet101 + FPN) and Faster R-CNN (ResNext101 + DC5). For the segmentation task, an ensemble of multiple depth EfficientNet models with FPN trained on multiple optimization plateaus (DSC, BCE, IoU) was designed. Data augmentation techniques like horizontal and vertical flip, cutout (random holes), random contrast, gamma, brightness, rotation along with CutMix 83 strategy for the segmentation task were incorporated to improve generalization capability.

14/32

• Team anand_subu [START_REF] Subramanian | Exploring deep learning based approaches for endoscopic artefact detection and segmentation[END_REF] The team used RetinaNet with ResNet101 backbone. For the segmentation task, the team used an ensemble network with U-Net with a ResNet50 backbone and DeepLabV3. However, the team reported U-Net with ResNet101 as their best architecture of choice. All the backbones were pre-trained on the ImageNet. Real-time augmentation techniques like rotation, shear, random-image-flip, image contrast, brightness, saturation, and hue variations were incorporated while training to improve the generalization capability of the network.

• Team higersky [START_REF] Chen | Endoscopy artefact detection and segmentation using deep convolutional neural network[END_REF] The team implemented Hyper Task Cascade and Cascade R-CNN with ResNeXt101 backbone as a feature extractor and FPN module for multi-scale feature representation for the object detection task. They applied Soft-NMS [START_REF] Bodla | Soft-nms-improving object detection with one line of code[END_REF] to avoid mistakenly discarded bounding-boxes. For the semantic segmentation task, the team incorporated DeepLabV3+ with ResNet101 backbone and trained with BCE and DICE losses. The backbones for both tasks were pre-trained on ImageNet.

• Team MXY [START_REF] Yu | Endoscopic artefact detection using cascade R-CNN based model[END_REF] The team used a Cascade R-CNN with an ImageNet pre-trained ResNet101 backbone and a FPN module.

Post-detection, soft-NMS was added to remove false predictions. The dataset was augmented by random resizing technique to improve the final output scores. The team used more weight for the losses of specularity, artefact, and bubbles classes to overcome classification difficulties between those classes.

• Team StarStarG The team used Cascade-RCNN as network architecture and adopted COCO2017 pre-trained ResNeXt as backbone with FPN and multi-stage RCNN framework. The authors also integrated Deformable Convolutional Networks in backbone to improve the model performance.

• Tesam xiaohong1 [START_REF] Gao | Artefact detection and segmentation based on a deep learning system[END_REF] The team built their detection and segmentation method upon Yolact-based instance segmentation system. Yolact 89 adds a segmentation component to the RetinaNet to ensure the tasks of detection, classification and delineation which are performed simultaneously. The network uses ResNet101 as an imageNet pretrained backbone.

EDD2020 Participating teams

• Team Adrian [START_REF] Krenzer | Endoscopic detection and segmentation of gastroenterological diseases with deep convolutional neural networks[END_REF] The team compared two different models: YOLOv3 with darknet-53 backbone and Faster R-CNN with ResNet-101 backbone. For post-processing, both algorithms in the final architecture were combined. For the second task, the team leveraged the state-of-the-art Cascade Mask R-CNN with ResNeXt-151 as a backbone. The team trained YOLOv3 using categorical cross-entropy for classification and default localization loss, while for Cascade Mask-RCNN, they used binary cross entropy for classification and mask, and L1 smooth for boundary box regression.

• Team Shahadate [START_REF] Rezvy | Transfer learning for endoscopy disease detection & segmentation with mask-rcnn benchmark architecture[END_REF] The team implemented a modified benchmark Mask R-CNN infrastructure model on the EDD2020 dataset. They used COCO trained weights and biases with the ResNet101 backbone as an initial feature extractor. The network head of the backbone model was replaced with new untrained layers that consisted of a fully-connected classifier with five classes and an additional background class. Non-maximum suppression was used to reduce overlapped detection. Finally, the team merged multiple bounding boxes for the same class label as one bounding box to match with the mask annotation.

• Team VinBDI [START_REF] Nguyen | Detection and segmentation of endoscopic artefacts and diseases using deep architectures[END_REF] For the object detection task, the team designed an ensemble of six EfficientDet models (with BiFPN modules) trained on six different EfficientNet backbones. A total of eleven augmentation techniques were incorporated to increase the output prediction scores of the model. For the segmentation task, an ensemble of U-Net and EfficientNet-B4 and BiFPN with the ResNet50 backbone was devised. The same team also participated in the EAD2020 sub-challenge.

• Team YH_Choi [START_REF] Choi | Centernet-based detection model and u-net-based multi-class segmentation model for gastrointestinal diseases[END_REF] The team implemented a CenterNet-based model with the PASCAL VOC pretrained ResNet50 backbone for the object detection task. A similar backbone with U-Net was devised for the segmentation task. The dataset was randomly duplicated to tackle class-imbalance. To improve generalization performance, each image was augmented 86 times by randomly choosing augmentation techniques from the pool of rotation, flipping, contrast enhancement and brightness adjustment.

• Team drvelmuruganb [START_REF] Balasubramanian | Semantic segmentation, detection AND localisation of mucosal lesions from gastrointestinal endoscopic images using SUMNET[END_REF] For the segmentation of disease classes the team used an encoder-decoder based SUMNet architecture with the ImageNet pretrained VGG11 backbone. The authors also applied several augmentation strategies including variable brightness and HSV values, multiple crops and geometric transformations such as rotation, affine, scaling and projective were also applied to improve the accuracy.

15/32 For the EAD2020 sub-challenge, we present the results of 12 participating teams for multi-class artefact detection task and 8 teams for segmentation task. Similarly, for EDD2020 sub-challenge, we have included top 6 teams for detection and 7 teams for segmentation of multi-class diseases. In this section we present the quantitative and qualitative results for each team based on the evaluation metrics discussed in Section 3.2. For the EAD2020 sub-challenge, 3 different test dataset were released: 1) single-frame data for detection and segmentation, 2) sequence dataset for detection only and 3) out-of-sample data for generalization task only. For the detection task, the average of the aggregated sum of the detection scores for the single frame data and the sequence data were considered for final scoring. While, for the EDD2020 challenge only single frame detection and segmentation data were released. Below we present the result for each sub-challenges separately.

Quantitative results

EAD2020 sub-challenge

In this section, the results of the participant teams in the EAD2020 challenge to detect and segment artifacts are presented.

Detection task for EAD2020 Table 5 and Table 6 present the mAP values computed at different IoU thresholds (i.e., 25%, 50%, and 75%), overall mAP, overall IoU, and the final score for the detection of the artefacts from single frame and sequence data, respectively. Additionally, we also provide results of baseline methods that include YOLOv3 and RetinaNet with darknet53 and ResNet101 backbones, respectively. In Furthermore, Table 7 shows the overall ranking for the teams in terms of Score (R score d ), mAP (R mAP ), and generalizability performance (R g ) in addition to, mAP d , mAP seq , score d , mAP g and dev g . The baseline RetinaNet recorded the least deviation but also the least mAPs. On considering the mAP g and dev g together for the final ranking of the generalization task, teams VinBDI and StarStarG secured the first place. On observing at the class-wise performance in Figure 5 (a) (i.e., single frame), it can be seen that there was a high detection score (score d ) and AP for larger artefact instances such as saturation and contrast. Similarly, most of the teams had a high IoU with the ground truth when detecting the instrument class. On the other hand, the detection and localization of smaller artefact instances such as bubble and saturation showed the degraded performances by all the participating teams and by the baseline methods.

Segmentation task for EAD2020 Table 8 presents the JC, DSC, F2, PPV, recall, and accuracy obtained by each team and baseline methods. As shown, the method proposed by team arnavchavan04 and team VinBDI had the best performance in terms of JC (> 62%), DSC (> 67%), F2 (> 67%) and PPV (> 80%) proving the ability to segment less false positive regions. However, the method suggested by team qzheng5 and team DuyHUYNH segmented more true positive regions compared to other teams obtaining top recall values of 0.8352 and 0.828. The baseline methods showed a low performance in terms of final score compared to the methods proposed by the participants. Furthermore, Figure 6 (a) shows class-wise scores for DSC, PPV and Recall. Similar to detection, segmenting larger instances like the saturation and the instrument obtained the high scores. Specularity, bubble and the artefact classes were among least performing classes for many teams and baseline methods.

EDD2020 sub-challenge

In this section, we report the performance of the participating teams in the EDD2020 challenge for the detection and segmentation.

Detection task for EDD2020 In Table 9, the team adrian achieved the highest score among other participants and the baseline methods with a final score d of 33.602 ± 8.523 with the highest overall mAP (37.594) and the second highest overall mIoU (27.614). The best localization score was obtained by the team sahadate but with nearly 5% lower mAP than the top scorer team. Furthermore, the baseline method RetinaNet with the ResNet101 backbone performed better than most of the participating teams. From Table 10, it is evident that most teams and baselines failed to detect suspicious class instance while most teams performed comparatively better on polyp and NDBE classes. Only the winning team adrian and RetinaNet (ResNet101) Table 7. EAD2020 team ranking based on different metric criteria for detection and generalization task. Overall mAPs (mAP d and mAP seq ) computed on single frame and sequence data are averaged. Final score d is then computed as the weighted value between the final IoU d and the averaged mAP. Rankings for each metric are also provided based on ascending order of the scores except for deviation score for out-of-sample data. Three best scores for each metric criteria are in bold. provided a descent score for cancer class with most teams recording mAP below 10. For HGD class category, top performing teams were adrian andVinBDI with mAP over 25.

Team

Segmentation task for EDD2020 From Table 11, it can be observed that the three teams (Adrian, sahadate and nhanthanhn-guyen94) achieved a DSC over 0.80. Moreover, they maintained the high performance for other metrics as well that include JC (>0.78), F2 (>0.81), and PPV (>0.85) securing first, second and third ranks, respectively. Teams VinBDI and DuyHUYNH were able to segment more true positive regions reaching the top recall values. Fig. 6 (b) represents per-class metric values. It can be observed that unlike detection task, most teams reported high performance for cancer class. Also, most teams showed higher DSC, PPV and recall for BE class instance as well (> 0.8 for top three teams). However, similar to the detection task, most team and baseline methods reported least values for the suspicious class.

Qualitative results

Detection task

Figure 7 shows the best (panel a) and the worse (panel b) performing frames from single frame dataset for EAD2020. It can be observed that specularity and artefacts are detected and well localized by top teams (see Figure 7 a). Similarly, in the bottom example, saturation is also detected by all the participants. Even though, blur is not present for this sample, most methods also detected it. While for the worse performing frame (see Figure 7 b), instrument class is confused with contrast or artefact on the top sample, while in the bottom sample instrument is detected by some teams but often either detected only partially or overlapped by different classes such as saturation or artefact.

For out-of-sample generalization task, it can be seen in Figure 8 (a) that besides YOLOv3 baseline method, all the baselines and teams detected saturation class. While some teams (mathew666, VinBDI, higersky) detected multiple bounding boxes for the same class, they also detected blur class for this frame. While for worse performing frame (see Figure 8 (b)), instrument class (at the center of the image) is well localized only by the team xiahong1 while most teams either partially detected the instrument (e.g., team qzheng5) or could not detect the instrument class at all (e.g., team polatgorkem). In both cases, the three teams VinBDI, higersky and StarStarG produced multiple overlapping and different size bounding boxes. Qualitative results for the EDD2020 challenge is shown in Figure 9. The best performing samples in Figure 9 7).

Teams

Discussion

Deep learning methods are rapidly being translated for the use of computer aided detection (CADe) and diagnosis (CADx) of diseases in complex clinical settings including endoscopy. However, the amount of data variability particularly in endoscopy is significantly higher than in natural scenes which possess a significant challenge in the process. It is therefore vital to determine an effective translational pathway in endoscopy. Majority of challenges in endoscopy are due to its complex surveillance that lead to severe artefacts that may confuse with disease. Similarly, a system designed for a particular organ may not generalize to be used in the other. Most deep learning methods that were used in the EndoCV2020 challenge can be categorised into multiscale, symbiotic, ensemble, encoder-decoder and cascading nature, or a combination of these (see Table 3 andTable 4). Figure 12 presents the overview of the used methods for the detection (a) and segmentation (b) challenge tasks based on the architecture usage. It can be observed that the majority of detection methods used two-stage Faster-RCNN with 4/7 teams combining it with one-stage RetinaNet or YOLOv3 or a combination of all. Cascade R-CNN which is built upon Faster R-CNN cascaded architecture was exploited by 4 teams. Similarly, U-Net-based architectures were utilised by most teams for semantic segmentation task with 4 teams exploring pyramid module-based architectures and 2 teams used Deeplabv3+ architecture. Faster RCNN-based model was also explored with additional thresholding (e.g., team adrian) or per pixel prediction heads (e.g., team sahadate). Even though similar techniques were used in EAD2019 challenge 1 , a direct comparison is not possible. This is due to the inclusion of more data for EAD2020 in both train and test sets. Also, EAD2020 includes sequence data which was not provided in EAD2019 challenge. For the detection task, the top performing teams on the challenge metric in both EAD (team polatgorkem) and EDD (team adrian) were those using ensemble networks, i.e., maneuvering outputs from multiple architectures. However, these networks sacrifice the speed of detection which can be observed from the computational time which were significantly higher than teams that used a single architecture (see Table 7 andTable 9). Other teams that used such an approach included team arnavchavan04 and mimykgcp who combined Faster R-CNN with RetinaNet but both teams were respectively on 10th and 11th ranking. Just using Faster R-CNN alone with ResNet101 backbone, teams qzhang5 and mathew666 were able to detect both small and large size bounding boxes with sub-optimal accuracy that put them at 2nd and 4th positions, respectively. Similarly, team sahadate claimed 2nd position on EDD detection task using Mask R-CNN which is based on the Faster R-CNN architecture. For EAD2019 challenge 1 , team yangsuhui also used an ensemble network with Cascade RCNN and FPN approach for the detection task similar to the EAD2020 top scorer team polatgorkem.

An intelligent choice for improved speed and accuracy using a scalable network was presented by the teams xiahong1 (used YOLACT) and VinBDI (used EfficientDet D0) which were placed 3rd and 5th, respectively, on the final detection score of the EAD2020. On the sequence data, team VinBDI was the 2nd best method demonstrating the reliability of the used EfficientNet and FPN architectures. However, for almost all team methods the standard deviation was higher than for single frame data. No team exploited the sequence data provided for training. Team VinBDI was also ranked 3rd on the EDD detection task. Teams higerssky, StarStarG and MXY that used cascaded R-CNN were ranked respectively on 6th, 7th and 9th positions. Additionally, the team StarStarG was ranked 1st and team higersky was ranked 2nd on the overall mAP. However, it is to be noted that taking only mAP scores into account for detection could lead to over detection of the bounding boxes that increases the chance of finding a particular class but at the same time weakens the localization capability of the algorithm (see Figure 7). Similar observations were found for the EDD dataset where the team mimykgcp obtained an overall mAP of 20.742 but only 2.270 for the overall IoU (see Table 9). As a result, over detection of the bounding boxes can be seen in Figure 9. In order to deal with the over detection of the bounding boxes, YOLACT architecture used by xiahong1 suppressed the duplicate detections using already-removed detections in parallel (fast NMS). Similarly, teams such as polatgorkem from the EAD and adrian from the EDD were able to eliminate the duplicate detections using ensemble network and a class agnostic NMS.

Hypothesis I: In the presence of multiple class objects, object detection methods may fail to precisely regress the bounding boxes. Methods need better penalisation on the bounding box regression or a technique to perform effective non-maximal suppression.

suspicious class by most of the team. However, Figure 11 (b) (top) shows that while majority of teams were able to segment NDBE class area, the teams either missed the HGD area or miss classified HGD as suspicious class instance. It is to be noted that there is a very subtle difference between the HGD and the suspicious region even for the expert endoscopists. Similar observation can be found for the segmentation of protruded structures (Figure 11 (b), bottom) where most methods confused the class with the polyp class and the top two teams (adrian, sahadate) classified it as cancer class. Looking up into our expert consensus notes we found that these samples had hard to reach agreement cases (i.e., suspicious and HGD classes; and cancer and polyp region).

Hypothesis V: Instead of hard scoring of predicted mask classes that penalizes the method performance heavily in presence of marginal visual difference between classes and variability due to existing expert consensus in the dataset, probability maps can be used to mitigate such problem. Additionally, teams should be encouraged to report results for different batch size and learning rates for obtaining better insight regarding performance especially when datasets are prone to have some incorrect labels.

Conclusion

We provided a comprehensive analysis of the deep learning methods built to tackle two distinct challenges in the gastrointestinal endoscopy: a) artefact detection and segmentation and b) disease detection and segmentation. It has been possible by the crowd-sourcing initiative of the EndoCV2020 challenges. We have provided the summary of the methods developed by the top 17 participating teams and compared their methods with the state-of-the-art detection and segmentation methods. Additionally, we dissected different paradigms used by the teams and presented a detailed analysis and discussion of the outcomes. We also suggested pathways to improve the methods for building reliable and clinically transferable methods. In future, we aim towards more holistic comparison of the built techniques for clinical deployability by testing for hardware and software reliability in clinical settings.

Figure 1 .

 1 Figure 1. EndoCV2020 train data samples. (a) Endoscopy artefact detection and segmentation sub-challenge (EAD2020) samples. Both single frame samples (top) and sequence frames (bottom) were released. While detection annotations involve 8classes, segmentation classes were limited to 5 distinct class instances, mostly large indefinable shapes that include specularity, saturation, imaging artefact, bubbles and instrument. It can be observed that for sequence data most artefact instances follow upto few sequential frames so it is desirable to achieve such training datasets. 4th sample in the single frame data for segmentation shows that even though bounding boxes for detection are provided for all specular regions, some segmentation labels were missing. This shows the presence of annotator variability in the data. (b) Endoscopy disease detection and segmentation training samples for sub-challenge EDD2020. First four samples belong to esophageal endoscopy while the last two frames were acquired during colonoscopy. It can be observed that disease classes in esophagus confuse often, mostly the patient choice here is Barrett's where clearly suspected and high-grade dysplasia appear jointly. Similarly, for colonoscopy data protruded polyps can easily be confused with the surrounding ridge-like openings and specular areas.

Figure 2 .

 2 Figure 2. Endoscopy computer vision EndoCV2020 challenge dataset details. (a) Multi-center, multi-modality and multi-organ dataset for EAD and EDD sub-challenges. For EAD2020, 2532 frames with 8 class bounding boxes for the detection task out-of which 573 included ground truth masks for segmentation task were provided. Participants were assessed on 317 frames for detection and 162 frames for segmentation tasks. An additional 99 frames were used to test out-of-sample generalization task for EAD sub-challenge. While EDD2020 consisted of 384 train samples and 43 test samples for 5 disease classes. (b-c) The distribution of 8 artefact classes of EAD and 5 disease classes of EDD w.r.t. their size compared to their height and width of image is provided. Each class size variability is also shown on right as blobs with mean at center and radius as standard deviation.

Figure 4 .

 4 Figure 4. EndoCV2020 challenge task descriptions for each sub-challenge. The three tasks of the EndoCV2020 challenge includes: (a) The "detection" task aimed at the coarse localization and classification. Given an input image (left) a detection model (middle) outputs the artefact/disease class and coordinates of the containing bounding box. (b) The "segmentation" task is aimed at precise delineation of artefact/disease object boundaries. The model predicts binary output images denoting the presence ('1') or absence ('0') of each class. (c) The "out-of-sample generalization" task is aimed at assessing the ability of a model trained on different dataset to generalize on an unseen dataset usually coming from a different center.

). 1 .

 1 blur → fast camera motion 2. bubbles → a thin film of liquid with air that distorts tissue appearance 3. specularity → mirror-like reflection 4. saturation → overexposed bright pixel areas 5. contrast → low contrast areas from underexposure 6. misc. artifact → chromatic aberration, debris etc. 7. instrument → biopsy or any other instrument 8. blood → flow of red colored liquid due to biopsy or surgery For EDD dataset, both upper-GI (gastroscopy) and lower-GI (colonoscopy) data were used with below defined class categories (please refer to the samples in Fig. 1 (b)):

3 aFigure 5 .

 35 Figure7shows the best (panel a) and the worse (panel b) performing frames from single frame dataset for EAD2020. It can be observed that specularity and artefacts are detected and well localized by top teams (see Figure7 a). Similarly, in the bottom example, saturation is also detected by all the participants. Even though, blur is not present for this sample, most methods also detected it. While for the worse performing frame (see Figure7 b), instrument class is confused with contrast or artefact on the top sample, while in the bottom sample instrument is detected by some teams but often either detected only partially or overlapped by different classes such as saturation or artefact.For out-of-sample generalization task, it can be seen in Figure8(a) that besides YOLOv3 baseline method, all the baselines and teams detected saturation class. While some teams (mathew666, VinBDI, higersky) detected multiple bounding boxes for the same class, they also detected blur class for this frame. While for worse performing frame (see Figure8 (b)), instrument class (at the center of the image) is well localized only by the team xiahong1 while most teams either partially detected the instrument (e.g., team qzheng5) or could not detect the instrument class at all (e.g., team polatgorkem). In both cases, the three teams VinBDI, higersky and StarStarG produced multiple overlapping and different size bounding boxes. Qualitative results for the EDD2020 challenge is shown in Figure9. The best performing samples in Figure9(a) shows polyp class (at the top); non-dysplastic Barrett's esophagus (NDBE) and suspicious classes on the bottom. It can be observed that polyp class is

Figure 6 .

 6 Figure 6. Semantic segmentation for EAD and EDD sub-challenges: Error bars with overlayed swarm plots for dice similarity coefficient (DSC), positive predictive value (PPV) or precision and recall are presented for each team and baseline methods for the EAD2020 (a) and EDD2020 (b) challenges. 6 different baseline methods are also provided for comparison.

Figure 7 .Figure 8 .

 78 Figure 7. EAD2020 best and worse performing samples for the detection task. a) Best performing samples for 6 top ranked team results. b) Worse performing samples for the same teams in (a). Results with baseline methods are also included together with ground truth sample.

Figure 9 .

 9 Figure 9. EDD2020 best and worse performing samples for the detection task. a) Best performing samples for 6 top ranked team results. b) Worse performing samples for the same teams in (a). Results with baseline methods are also included together with ground truth sample.

Figure 10 .

 10 Figure 10. EAD2020 best and worse performing samples. a) Best performing samples for 5 top ranked team results. b) Worse performing samples for the same teams in (a). Results with baseline methods are also included together with ground truth sample (top). Single class samples are chosen at the top and multi-class samples are at the bottom in each category.

Table 1 .

 1 Breakdown of data: Number of samples and annotations released for EndoCV2020 challenge.

	EndoCV	Tasks	# of classes	# of frames Train Test	# of annotations Train Test
	EAD2020	Detection task Segmentation task	8 5	single: 2299 seq.: 232 643	single: 237 seq.: 80 162	31069 7511	7750 3228
		Generalization task	8	na	99	na	3013
	EDD2020	Detection task Segmentation task	5 5	386 386	43 43	749 749	68 68
	robotic scene segmentation 68 , and computer aided detection and segmentation for polyps 5, 69 and Barrett's cancer detection 2
	have been initiated under MICCAI EndoVis challenge 3 . Endoscopy artefact detection (EAD2019) is another challenge which
	was first initiated in 2019 and launched in conjunction with IEEE International Symposium on Biomedical Imaging (ISBI)
	2019 1 .

6/32 99 frames 287 frames 2530 frames

	Train data	EAD2020 Test data		Out-of-sample test data	EDD2020 Train and test data
								386 + 48
								frames
	Institutes (Outer circle)					Modality (Middle circle)	Organ (Inner circle)
	John Radcliffe Hospital, Oxford, UK	Ambroise Paré Hospital, Paris, France	White light (WL)		Oesophagus
	University Hospital Vaudois, Switzerland	ICL Cancer Institute, Nancy, France		Narrow band imaging (NBI)	Stomach
	Centro Riferimento Oncol., IRCCS, Italy	Istituto Oncologico Veneto, Padova, Italy	Chromo endoscopy		Small intestine
	Botkin Clinical City Hospital, Moscow, Russia							Colon
									Other
	Train data				Test data		Mean box width and height
	specularity	saturation	artifact	blur	contrast	bubbles	instrument	blood

Mean box width and height

	a. EndoCV2020 multi-center data cohort: Train and test data for each sub-challenge
	b. EAD2020 train and test sample with per class width and height for detection dataset
	c. EDD2020 train and test sample with per class width and height for detection dataset
	Train data		Test data	Mean box width and height
	BE	suspicious	HGD	cancer	polyp

Table 2 .

 2 Data collection information for each center: Data acquisition system and patient consenting information.

Table 4 .

 4 Endoscopy disease detection and segmentation (EDD2020) method summary for top 7 teams (out-of 14 submission).

	Team EDD2020 Algorithm	Preprocessing	Nature	Basis-of-choice Backbone	Data aug.	Pretrained	Computation	code
	Detection								GPU	Test time
	Adrian	YOLOv3+ Faster R-CNN	Resize	Ensemble	Accuracy+ , speed+	Darnet53 ResNet101	Yes (F, D) †	COCO dataset public polyp	Tesla P100	0.41	Adrian398/EDD
	shahadate	Mask R-CNN	Resize Normalise	Multiscale	Accuracy , speed+	ResNet101	Yes (Sc, R, F, Cr, S, N) †	COCO	RTX2060	NA	EDD-Mask-rcnn
	VinBDI	EfficientDet D0	Resize (512x512)	Ensemble	Speed++	EfficientNet B0	Yes (S, Sc, R, N, MU) †	COCO	RTX 2080TI NA	endocv2020-seg
	YH_Choi	CenterNet	NA	Context	Accuracy++	ResNet50	Yes(Du, R, F, C, B) †	PASCAL VOC2012	RTX 2080	2	NA
	DuyHUYNH (LRDE)	U-Net++	Normalise	Encoder-decoder	Speed	EfficientNet B1	Yes (R, S, F, Sc, LD, TTA) † ImageNet	GTX1080 Ti 1.53	dhuynh/endocv2020
	mimykgcp (vishnusai)	Faster RCNN + RetinaNet	Resize (256x256) normalise	Ensemble	Accuracy+ , speed+	ResNeXt101	Yes (RA) †	COCO	GTX1080Ti	0.58	NA
	Segmentation
	Adrian	YOLOv3 + Faster R-CNN + Cascade RCNN	Resize	Ensemble	Accuracy++	Darnet53 ResNet101	Yes (F, D) †	COCO dataset public polyp	Tesla P100		Adrian398/EDD2020
	shahadate	MaskRCNN	Resize Normalise	Multiscale	Accuracy , speed+	ResNet101	Yes (Sc, R, F, Cr, S, N) †	COCO	RTX2060		EDD-Mask-rcnn
	VinBDI	U-Net + BiFPN	Resized (512x512)	Ensemble Endcoder-decoder	Accuracy++ , speed+	EfficientNet B4 ResNet50	Yes (S, Sc, R, F) †	COCO ImageNet	RTX 2080 Ti NA	endocv2020-seg
	YH_Choi	U-Net	NA	Encoder-decoder	Accuracy+	ResNet50	Yes(Du, R, F, C, B) †	PASCAL VOC2012	RTX 2080	7	NA
	DuyHUYNH (LRDE)	U-Net++	Normalise	Encoder-decoder	Accuracy+ , speed+	EfficientNet B1	Yes (R, S, F, Sc, LD, TTA) † ImageNet	GTX1080 Ti 1.53	endocv2020
	drvelmuruganb	SUMNet	NA	Encoder-decoder	Accuracy+ , speed++	VGG11	Yes(R, A, Sc, P, and Cr) †	ImageNet	GTX1080 Ti 0.16	drvelmuruganb/EDD2020
	mimykgcp	U-Net	Resize Normalise	Encoder-decoder	Accuracy+	ResNeXt50	Yes (RA) †	ImageNet	RTX2070	1.25	NA

† A: affine, B: brightness, C: contrast, Cr: cropping, D: distortion, Du: duplication, F: flip, H: hue, LD: local deformation, Mu: mixup, N: noise, P: perspective transformation, R: rotation, RA: RandAugment library, S: shift, Sc: scaling, Sh: shear, St: saturation, T: translation, TTA: test-time augmentation

Table 5 .

 5 EAD2020 results for the detection task on the single frame dataset. mAP at IoU thresholds 25%, 50% and 75% are provided along with overall mAP and overall IoU computations. Overall scores are computed at 11 IoU thresholds and averaged. Weighted detection score score d is computed between overall mAP and IoU scores only. Three best scores for each metric criteria are in bold.

	Team names	mAP 25 mAP 50 mAP 75	overall mAP d	overall mIoU d	mAP δ	score d ± δ
	polatgorkem	26.886	17.883	5.608	17.486	36.579	7.124	25.123 ± 7.124
	qzheng5	33.134	20.084	5.570	19.720	27.185	8.820	22.706 ± 8.820
	xiahong1	30.627	19.384	4.935	18.512	26.388	8.428	21.663 ± 8.428
	mathew666	20.360	19.440	7.783	18.091	32.692	5.617	23.931 ± 5.617
	VinBDI	38.429	25.426	7.053	24.069	12.644 10.291 19.499 ± 10.291
	higersky	36.920	25.770	9.452	24.771	17.298	8.707	21.781 ± 8.707
	StarStarG	41.800	29.984	10.733	28.380	16.250 10.042 23.528 ± 10.042
	anand_subu	29.755	19.893	5.271	18.886	24.029	7.619	20.943 ± 7.619
	arnavchavan04 38.752	27.247	9.858	26.021	21.165	9.342	24.079 ± 9.342
	MXY	25.373	18.967	7.171	17.82	28.056	5.754	21.914 ± 5.754
	mimykgcp	39.897	26.296	6.839	25.082	10.209 10.765 19.133 ± 10.765
	DuyHUYNH	20.512	12.234	2.978	11.894	27.063	5.671	17.962 ± 5.671
	baselines
	YOLOv3	22.798	13.736	2.804	13.249	24.883	6.525	17.903 ± 6.525
	RetinaNet (ResNet101)	15.270	8.927	2.061	8.754	23.202	4.275	14.533 ± 4.275
	5 Results

Table 5

 5 (i.e., single frame detection), it can be observed that the team polatgorkem that implemented ensemble technique with Cascaded RCNN, Faster-RCNN and RetinaNet surpassed the other teams by achieving the highest final score on the leaderboard (score

d , Eq. 5) of 25.123 ± 7.124 with the best overall mIoU of 36.579 providing a high overlap ratio between the generated bounding box with ground truth per frame. The method proposed by the team arnavchavan04 comes in the second place with score d of 24.079 ± 9.342 with 9% more mAP than the winning team but large sacrifice in the mean IoU. Similarly, for sequence data in Table

6

, team polatgorkem maintained the first position with a final score of 25.529 ± 10.326. While the second scorer team VinBDI suggested a method that obtained a better balanced between mAP and mIoU scores.

16/32

Table 6 .

 6 EAD2020 results for the sequence dataset. mAP at IoU thresholds 25%, 50% and 75% are provided along with overall mAP and overall IoU computations. Overall scores are averaged with 11 IoU thresholds. Weighted detection score score d is computed between overall mAP and IoU scores only. Three best scores for each metric criteria are in bold.

	Team names	mAP 25 mAP 50 mAP 75	overall mAP seq	overall mIoU seq	mAP δ	score d ± δ
	polatgorkem	38.464	24.803	4.138	23.137	29.117	10.326 25.529 ± 10.326
	qzheng5	48.210	25.717	3.997	25.665	20.949	14.222 23.779 ± 14.222
	xiahong1	46.087	25.813	2.684	25.136	18.398	15.128 22.441 ± 15.128
	mathew666	31.599	21.878	3.053	19.623	20.858	9.718	20.117 ± 9.718
	VinBDI	45.295	26.723	4.396	25.285	23.426	13.972 24.542 ± 13.972
	higersky	47.716	29.841	4.473	28.334	12.865	14.579 22.147 ± 14.579
	StarStarG	46.965 30.202	5.432	28.107	8.371	13.367 20.213 ± 13.367
	anand_subu	38.352	25.535	3.843	23.014	20.703	10.859 22.089 ± 10.859
	arnavchavan04 34.511	21.524	4.886	20.700	11.827	9.839	17.151 ± 9.839
	MXY	31.391	19.838	3.620	18.601	21.504	8.688	19.762 ± 8.688
	mimykgcp	44.972	26.780	4.400	25.937	6.892	13.697 18.319 ± 13.697
	DuyHUYNH	28.632	15.524	0.815	15.468	16.968	9.381	16.068 ± 9.381
	baselines
	YOLOv3	32.199	18.473	1.137	17.176	16.351	10.596 16.846 ± 10.596
	RetinaNet (ResNet101)	17.646	6.447	0.767	8.079	10.000	5.151	9.252 ± 5.151

Table 8 .

 8 Evaluation of the artefact segmentation task. Top three best scores for each metric criteria are in bold.

Table 9 .

 9 EDD2020 results for the detection task on the single frame dataset. mAP at IoU thresholds 25%, 50% and 75% are provided along with overall mAP and overall IoU computations. Overall scores are computed at 11 IoU thresholds and averaged. Weighted detection score score d is computed between overall mAP and IoU scores only. Three best scores for each metric criteria are in bold.

	Team Names	JC	DSC	F2	PPV	Rec	Acc	Score s R score s
	qzheng5	0.477 0.532 0.561 0.556 0.835 0.973 0.621	8
	VinBDI	0.628 0.673 0.670 0.837 0.738 0.978 0.730	2
	higersky	0.529 0.579 0.587 0.675 0.758 0.975 0.650	5
	anand_subu	0.304 0.354 0.361 0.430 0.747 0.975 0.473	14
	arnavchavan04	0.622 0.673 0.683 0.800 0.767 0.977 0.731	1
	DuyHUYNH	0.502 0.557 0.583 0.593 0.829 0.974 0.640	6
	mimykgcp	0.531 0.576 0.579 0.723 0.726 0.977 0.651	4
	mouradai_ox	0.581 0.632 0.647 0.711 0.800 0.974 0.697	3
	baselines
	FCN8	0.500 0.548 0.550 0.670 0.708 0.976 0.619	9
	UNet-ResNet34 0.310 0.364 0.373 0.419 0.766 0.974 0.481	13
	PSPNet	0.497 0.541 0.534 0.698 0.680 0.975 0.613	10
	DeepLabv3 (ResNet50)	0.448 0.495 0.492 0.599 0.704 0.974 0.572	12
	DeepLabv3+ (ResNet50)	0.485 0.533 0.535 0.646 0.726 0.976 0.610	11
	DeepLabv3+ (ResNet101)	0.501 0.547 0.546 0.683 0.718 0.973 0.624	7

  numerous bounding boxes failing to optimally localize adherent disease classes. For the worse performing frames (Figure 9 (b)), cancer class (top) in the ground truth is confused with the polyp class instance for most of the teams and the

	Team names	mAP 25 mAP 50 mAP 75	overall mAP d	overall mIoU d	mAP δ	score d ± δ
	adrian	48.402	33.562	27.098	37.594	27.614	8.523	33.602 ± 8.523
	sahadate	37.612	23.284	15.837	26.834	32.420	8.325	29.068 ± 8.325
	VinBDI	43.202 26.981	17.001	30.219	17.773	9.478	25.241 ± 9.478
	YHChoi	23.183	11.082	8.800	15.783	24.623	6.216	19.319 ± 6.216
	DuyHUYNH	23.959	9.587	5.659	12.479	13.829	6.284	13.019 ± 6.284
	mimykgcp	34.884	20.982	4.463	20.742	2.270	9.359	13.353 ± 9.359
	drvelmuruganb	31.018	18.421	11.768	21.790	7.322	7.424	16.002 ± 7.424
	baselines
	YOLOv3	34.305	21.227	14.650	22.980	24.351	6.456	23.528 ± 6.456
	RetinaNet (ResNet50)	26.833	14.441	9.907	17.552	25.580	6.464	20.763 ± 6.464
	RetinaNet (ResNet101) 42.579	27.000	11.194	27.974	26.434	11.949 27.358 ± 11.949

produced

  ). It can be observed that polyp is segmented well by all the baselines and most

	a. Best performing Ground truth YOLOv3	RetinaNet (ResNet50)	RetinaNet (ResNet101)	adrian	b. Worse performing Ground truth YOLOv3	RetinaNet (ResNet50)	RetinaNet (ResNet101)	adrian
	sahadate	VinBDI	YHChoi	drvelmuruganb	mimykgcp	sahadate		VinBDI	YHChoi	drvelmuruganb	mimykgcp
	Ground truth	YOLOv3	RetinaNet (ResNet50)	RetinaNet (ResNet101)	adrian	Ground truth	YOLOv3	RetinaNet (ResNet50)	RetinaNet (ResNet101)	adrian
	sahadate	VinBDI	YHChoi	drvelmuruganb	mimykgcp	sahadate		VinBDI	YHChoi	drvelmuruganb	mimykgcp
			NDBE	suspicious	HGD	polyp	cancer
											22/32

https://endovissub-barrett.grand-challenge.org

https://endovis.grand-challenge.org

https://ead2020.grand-challenge.org

https://edd2020.grand-challenge.org

https://github.com/sharibox/EndoCV2020

Acknowledgments

The research was supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. S. Ali, B. Braden, A. Bailey and JE. East is supported by NIHR BRC and J. Rittscher by Ludwig Institute for Cancer Research and EPSRC Seebibyte Programme Grant (EP/M013774/1). We want to acknowledge Karl Storz for co-sponsoring the challenge workshop. We would also like to acknowledge the annotators and our EndoCV2020 challenge workshop proceedings reviewers and IEEE International Symposium on Biomedical Imaging 2020 challenge committee Michal Kozubek and Hans Johnson.

The choice of networks from each team depended on their ambition of either obtaining very high accuracy without focusing on speed or a trade-off between the speed and the accuracy or focusing on both and thinking out-of the box to use more recent developed methods which beats faster networks (such as YOLOv3) that included EfficientDet D0 architecture used by the team VinBDI (see Table 3). Due to the efficiency of the EfficientDet D0 network that used biFPN and efficientNet backbone, team VinBDI achieved second least deviation in mAP (i.e., dev g = 5.607) with competitive mAP g (= 24.140) and won the generalization task together with the team StarStarG who had slightly higher mAP g (= 25.340) but larger mAP deviation between detection and generalization datasets. Most methods for the detection task on both the EAD and EDD dataset performed better than the baseline one-stage methods (YOLOv3 and RetinaNet). However, it was found that even though team polatgorkem won the detection task, the method failed on generalization data where the team was ranked only last. The main reason behind this could be because the generalization gap mAP g was estimated between two mAP's (mAP d and mAP g ) and not IoU. Also, the final ranking was done taking into account the rank of dev g and mAP g only. It can be observed in Figure 8 that the bounding box localization of team polatgorkem is precise in (a) while it misses instrument area at the center in (b). However, the winning teams VinBDI and StarStarG both over detect the boxes. The generalization ability of the methods were not explored for EDD dataset.

Hypothesis II: Metrics are critical but using a single metric does not always gives the right answer. Weighted metrics are desired in object detection task to establish a good trade-off between detection and precise localization.

A major problem in the detection of EDD dataset was class confusion mostly for suspicious, HGD and cancer classes. This could be because of smaller number of samples for each of these classes compared to NDBE and polyp (see Figure 3). While most methods were able to detect and localize NDBE and polyp class in general (3/7 teams with an overall mAP > 45 and 4/7 teams with > 50), all teams failed in suspicious class (overall mAP < 5.0) and most teams for cancer class (overall mAP < 15.0) (see Table 10). Figure 9 shows that polyp is detected and localized very well by most teams (a, top). Similarly, NDBE is localized by most methods, however, in this case suspicious class is confused mostly with the HGD. Also, in Figure 9 (b, top), it can be observed that the cancer class instance is confused with mostly polyp class.

Hypothesis III: Detection bounding boxes confuse with classes that have similar morphology and smaller number of samples failing to learn the contextual features. To improve detection, such samples need to be identified and more data demonstrating such attributes need to be injected (both positive and negative samples).

Similar to the detection task, teams that used ensemble techniques were among the best performing teams for the 25/32 In the EDD2020 segmentation task, the team adrian combined predictions from three object detection architectures where the YOLOv3 and Faster R-CNN class predictions were used to correct the instance segmentation masks from Cascade R-CNN. A direct instance segmentation approach used by the team sahadate secured second position (score s = 0.856) on the same while ensemble network of the team VinBDI secured the third position (score s = 0.847). Direct usage of a single existing state-of-the-art methods utilising different augmentation techniques (e.g., DuyHUYNH) or different backbones (e.g., mimykgcp, qzheng5) resulted in improved results compared to the original baseline methods, however, much lower than the top performing methods (see Table 8 and Table 11).

Hypothesis IV: The choice of combinatorial networks that well synthesizes width, depth and resolution to capture optimal receptive field, and a domain agnostic knowledge transfer mechanism are critical to tackle heterogeneous (multi-center and variable size) multi-class object segmentation task.

From Figure 6 it can be observed that the top three performing teams of the EAD2020 segmentation task (arnavchavan04, VinBDI, mouradai_ox) has high DSC value (0.538, 0.548 and 0.492 respectively) compared to most methods for the specularity class instance. It is to be noted that the specularities are often confused with either artifact or bubbles which makes them hard to differentiate. For the instrument, saturation and bubbles class instances (see Figure 10 a.), most methods obtained high performance compared to other classes (e.g., the top three teams obtained 0.853, 0.844, 0.848 for the instrument; 0.722, 0.758, 0.703 for the saturation; and 0.738, 0.693, 0.693 for the bubbles class instance, respectively), artefact (DSC < 0.520) was among the worst class for most teams and for the baseline methods. This is mostly due to the variable size of artefacts; and the bubbles class instance is predominantly confused with either artefact or the specularity class (see Figure 10 b.). Additionally, due to small sized and sparsely scattered specularity or bubble regions in some cases (for e.g., 4th image from left in Fig. 3 (a)), the annotator variability for these samples can have affected method performances for these classes. While checking for such biases is beyond the conducted study, we refer to the work by [START_REF] Rolnick | Deep learning is robust to massive label noise[END_REF] . The authors suggested that in general deep learning models are capable of generalizing from training data where the correct labels are outnumbered by the incorrect ones. However, the authors also acknowledged that a decrease in performance is inevitable and necessary steps such as using larger batch size and downscaling learning rate can help mitigate these issues.

Unlike the EAD2020, the EDD2020 segmentation task comprised of larger shaped regions and only a few classes confused (see 1 b.). Most methods scored comparably high DSC values with over 75% for most of the disease classes except for

Author contributions

S. Ali conceptualized the work, led the challenge and workshop, prepared the dataset, software and performed all analyses. M. Dmitrieva, N. Ghatwary, and S. Bano served as organising committee and participated in annotations. A. Bailey, B. Braden, J.E. East, R. Cannizzaro, D. Lamarque, S. Realdon were involved in the validation and quality checks of the annotations used in this challenge. G. Plolat, A. Temizel, A. Krenzer, A. Hekalo, YB. Guo, B. Matuszewski, M. Gridach, V. Yoganand assisted in compiling the related work and method section of the manuscript. S. Ali wrote most of the manuscript with inputs from M. Dmitrieva, N. Ghatwary, S. Bano and all co-authors. All authors participated in the revision of this manuscript and provided substantial input.