S Park

M K Hong

Min-Kook Choi

Arnaud Huaulmé
email: arnaud.huaulme@univ-rennes1.fr

Kanako Harada

Quang-Minh Nguyen

Bogyu Park

Seungbum Hong

Michael Peven

Yunshuang Li

Yonghao Long

Qi Dou

Satyadwyoom Kumar

Seenivasan Lalithkumar

Ren Hongliang

Hiroki Matsuzaki

Yuto Ishikawa

Yuriko Harai

Satoshi Kondo

Manoru Mitsuishi

Pierre Jannin
email: pierre.jannin@univ-rennes1.fr

• Challenge of surgical workflow recognition with different modality • Comparison of multiple deep learning based recognition methods PEg TRAnsfer Workflow recognition challenge report: Do multimodal data improve recognition?

Keywords: Surgical Process Model, Workflow recognition, Multimodal, OR of the future

HAL is

Introduction

To fully integrate computer-assisted surgery systems in the operating room, a complete and explicit understanding of the surgical procedure is needed. A surgical process model (SPM) is a "simplified pattern of a surgical process that reflects a predefined subset of interest of the surgical process in a formal or semi-formal representation" [START_REF] Jannin | Modeling surgical procedures for multimodal image-guided neurosurgery[END_REF], thus allowing for the surgical procedure to be rigorously modeled and described. The SPM methodology consists of decomposing a surgical procedure into five increasingly-coarse levels of granularity: dexeme, surgeme, activity, step, and phase [START_REF] Lalys | Surgical process modelling: a review[END_REF][START_REF] Despinoy | Unsupervised trajectory segmentation for surgical gesture recognition in robotic training[END_REF]. A dexeme, the lowest granularity level, is a numeric representation of the motion. A surgeme represents a surgical motion with an explicit semantic interpretation of the immediate motion (e.g., pulling). An activity describes the motion's overall action (action verbs; e.g., cut) performed on a specific target (e.g., the pouch of Douglas) by a specific surgical instrument (e.g., a scalpel). A step is the succession of these activities which together achieve a specific surgical objective (e.g., resection of the pouch of Douglas). Finally, a phase is the succession of steps that constitute a main period of the intervention (e.g., resection). SPM's are used for learning and expertise assessment [START_REF] Huaulmé | Sequential surgical signatures in micro-suturing task[END_REF][START_REF] Forestier | Surgical skills: Can learning curves be computed from recordings of surgical activities?[END_REF], robot assistance [START_REF] Ko | Surgery task model for intelligent interaction between surgeon and laparoscopic assistant robot[END_REF], operating room optimization and management [START_REF] Sandberg | Deliberate Perioperative Systems Design Improves Operating Room Throughput[END_REF][START_REF] Bhatia | Real-time identification of operating room state from video[END_REF], decision-making support [START_REF] Quellec | Real-Time Task Recognition in Cataract Surgery Videos Using Adaptive Spatiotemporal Polynomials[END_REF], and quality supervision [START_REF] Huaulmé | Offline identification of surgical deviations in laparoscopic rectopexy[END_REF].

The primary limitation of the state-of-the-art in SPM's [START_REF] Despinoy | Unsupervised trajectory segmentation for surgical gesture recognition in robotic training[END_REF][START_REF] Huaulmé | Sequential surgical signatures in micro-suturing task[END_REF][START_REF] Forestier | Surgical skills: Can learning curves be computed from recordings of surgical activities?[END_REF][START_REF] Sandberg | Deliberate Perioperative Systems Design Improves Operating Room Throughput[END_REF][START_REF] Quellec | Real-Time Task Recognition in Cataract Surgery Videos Using Adaptive Spatiotemporal Polynomials[END_REF][START_REF] Huaulmé | Offline identification of surgical deviations in laparoscopic rectopexy[END_REF] is their need to be manually interpreted by human observers, which is observer-dependent, time-consuming, and subject to error [START_REF] Huaulmé | Automatic annotation of surgical activities using virtual reality environments[END_REF]. Thus, the proposed solutions can not be directly used to bring context-awareness into computer-assisted surgery applications in the operating room. To overcome this limitation, automatic workflow recognition methods have been developed for multiple granularity levels, including phase [START_REF] Bhatia | Real-time identification of operating room state from video[END_REF][START_REF] Padoy | Statistical modeling and recognition of surgical workflow[END_REF][START_REF] Andru | EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos[END_REF], step [START_REF] Bouarfa | Discovery of high-level tasks in the operating room[END_REF][START_REF] James | Eye-Gaze Driven Surgical Workflow Segmentation[END_REF], and activity [START_REF] Ko | Surgery task model for intelligent interaction between surgeon and laparoscopic assistant robot[END_REF][START_REF] Lalys | Automatic knowledge-based recognition of low-level tasks in ophthalmological procedures[END_REF]. With the emergence of deep learning, most of these recent automatic workflow recognition methods are based on convolutional neural networks, such as AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] or ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]; on recurrent neural networks, such as LSTM [START_REF] Hochreiter | Long Short-Term Memory[END_REF] or gated recurrent unit (GRU) [START_REF] Cho | On the properties of neural machine translation: Encoder-decoder approaches[END_REF]; and more recently on transformers [START_REF] Vaswani | Attention is all you need[END_REF].

Along with what methodology to use, it is also an open question as to which data modalities should be used as input for this task. In robot-assisted surgery and virtual reality training environments, video and kinematic data are both readily available. Despite this, most state-of-the-art workflow recognition methods are based on a single modality, such as only video [START_REF] Sarikaya | Surgical Gesture Recognition with Optical Flow only[END_REF][START_REF] Funke | Using 3D Convolutional Neural Networks to Learn Spatiotemporal Features for Automatic Surgical Gesture Recognition in Video[END_REF] or only kinematic data [START_REF] Despinoy | Unsupervised trajectory segmentation for surgical gesture recognition in robotic training[END_REF][START_REF] Dipietro | Automated Surgical Activity Recognition with One Labeled Sequence[END_REF]. Few studies have used workflow recognition method based on both video and kinematic data [START_REF] Huaulmé | MIcro-surgical anastomose workflow recognition challenge report[END_REF][START_REF] Long | Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery[END_REF][START_REF] Qin | Learning Invariant Representation of Tasks for Robust Surgical State Estimation[END_REF]. However, with the exception of the study by Long et al. [START_REF] Long | Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery[END_REF], they do not compare the results obtained based on the number and type of input modalities.

Semantic segmentation of surgical video is also essential for surgical understanding and is an active area of research. For example, in five editions of the EndoVis MICCAI Challenge (2015 to 2020), six of the 19 proposed sub-challenges were dedicated to this topic. However, to the best of our knowledge, semantic segmentation has rarely been used as a supplementary task paired with, or as additional input for, surgical workflow recognition.

Therefore, the "PEg TRAnsfer Workflow recognition by different modalities" (PETRAW) sub-challenge, which is part of EndoVis, provided a unique data set for automatic recognition of surgical workflows containing video, kinematic, and segmentation data on 150 peg transfer training sequences. Participants were asked to develop model(s) to recognize phases, steps, and activities using one or several of the available modalities.

Methods: Challenge Design

This section describes the challenge design, organization, objective, data set, and assessment methods.

Challenge organization

The PETRAW challenge was a one-time event organized as part of EndoVis during the online 2021 international conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI2021). Four people were involved in the organization: Arnaud Huaulmé and Pierre Jannin from the University of Rennes 1 (France), and Kanako Harada and Mamoru Misthuishi from Tokyo University (Japan). Complete information about the challenge was made available to participants using the Synapse platform: www.synapse.org/PETRAW. Challenge participants were subject to the following rules:

• Participants had to submit a fully automatic method that could recognize phases, steps, and activities on the same model using one or several modalities; and

• Only data provided by the organizers and publicly available data sets, including pre-trained networks, were authorized for use in training. The publicly available data sets must have been open or otherwise available to all participants at the time the PETRAW data set was released.

The results of all participating teams were announced publicly during the challenge day. Challenge organizers and people from the organizing institutions could also participate in the challenge but were excluded from the competitive rankings. Participating teams were encouraged (but not required) to provide their code as open access.

For a valid submission, the participating teams had to provide the following elements: a write-up, a Docker image allowing the organizers to compute the results, and a pre-recorded talk to limit technical issues during the challenge day (online event). Multiple Docker images could be submitted, but only the last submission was officially used to generate the evaluation results. No leaderboard or evaluation results were provided prior to the challenge day.

The challenge schedule was as follows: The training data set, including videos, kinematic data, and workflow annotations, was released on June 1, 2021; corresponding semantic segmentation data was released on June 9, 2021; submissions were accepted until September 12, 2021 (23:59 PST); and the evaluation results were announced on October 1, 2021, during the online MICCAI2021 event. Some teams obtained unexpectedly poor results (i.e., workflow recognition rates inferior to 50%), which made further analysis of the results not relevant. Therefore, each team was allowed to provide a new submission before October 31, 2021. The teams that made a new submission are identified in Section 3.2. The challenge test data set and the organizers' evaluation scripts were released with this paper at www.synapse.org/PETRAW

Challenge objective

The objective of the PETRAW challenge was to study the contribution of each modality (either alone or in combination) to surgical workflow recognition.

To achieve this goal, participants were asked to create a single classification model to determine the surgical task at three levels of granularity (phase, step, and action). Five different tasks were offered as part of the challenge: three concerned the development of unimodal models (i.e., video-based, kinematic-based, or semantic segmentation-based models); and two concerned multimodal-based models. The unimodal-based models were used as a baseline for comparison with the multimodal-based models. In order to keep to a reasonable number of tasks, not all multimodal configurations could be studied. For models based on semantic segmentation data (and to reflect the fact that clinically this modality can be only obtained through a trained segmentation model), participants were asked to use the output of such model as input for PETRAW.

Challenge data set

The challenge data set was composed of 150 sequences of peg transfer training sessions. The objective of the peg transfer session was to transfer six blocks from the left peg to the right and then back. Each block needed to be extracted from the peg using a grasper (operated by one hand), transferred to the other grasper (in the other hand), and finally inserted onto the peg on the opposite side of the board.

All sequences were acquired by a non-medical expert at the LTSI Laboratory, University of Rennes 1, France. The data set was divided into training data (n=90 sequences) and test data (n=60 sequences). Each sequence included kinematic data, video, semantic segmentation of the video for each frame, and workflow annotations at each level of granularity. Only the training data set was provided to participants.

Data acquisition

The challenge data was acquired on a virtual reality simulator (Figure 1) developed at the Department of Mechanical Engineering, University of Tokyo, Japan [START_REF] Heredia Perez | Haptic As-sistance for Robotic Surgical Simulation[END_REF], consisting of a laptop (i7-700HQ, 16Go RAM, GTX 1070), a 3D rendering setup (3D screen: 24 inches, 144Hz; and 3D glasses), and two haptic user interfaces (3D system Touch TM ). For data acquisition, a single operator performed a series of five consecutive peg transfer tasks followed by a break of at least 5 hours to limit fatigue. This was repeated 30 times to yield a total of 150 peg transfer task sequences. The COVID-19 crisis (acquisition made in 2020-2021) did not allow us to recruit multiple participants. To limit the effect of immediate learning or fatigue in a single session, three sequences from each series were randomly chosen for training, and the remaining two for testing.

The kinematic data and videos were synchronously acquired at 30 Hz during each peg transfer task. Each video had a resolution of 1920x1080 pixels and semantic segmentation was performed for each frame off-line following the task. Kinematic data included the position, rotation quaternion, forceps aperture angle, linear velocity (obtained from simulation, not derived from position), and angular velocity (obtained from simulation, not derived from orientation) of the left and right instruments (i.e., graspers). The position and linear velocity were measured in centimeters and centimeters per second, respectively. The angle and angular velocity were measured in degrees and degrees per second, respectively.

The semantic segmentation included six classes (shown in Figure 2): back-ground (black, hexadecimal code:#000000), base (white, #FFFFFF), left instrument (red, #FF0000), right instrument (green, #00FF00), pegs (blue, #0000FF), and blocks (magenta, #FF00FF). The workflow annotations were automatically computed using the scene information and the ASURA method [START_REF] Huaulmé | Automatic annotation of surgical activities using virtual reality environments[END_REF]. The challenge organizers had previously demonstrated in [START_REF] Huaulmé | Automatic annotation of surgical activities using virtual reality environments[END_REF] that ASURA is more accurate and robust than manual annotation on peg transfer tasks. Two phases, twelve steps, six action verbs, two targets, and one surgical instrument were identified to describe the workflow (Table 1). Each phase corresponded to the transfer of all of the blocks in one direction (e.g. "L2R" for left to right). Each step (six per phase) corresponded to the transfer of a single block (e.g."Block1 L2R" for the transfer of the first block from the left to the right). For the activities, two targets were differentiated: "block" and "other block". "Block" corresponds to the one that is currently being transferred. "Other block" is an additional target used to differentiate when the user accidentally interacts with any block other than the one to be transferred.

One limitation of the method presented by [START_REF] Huaulmé | Automatic annotation of surgical activities using virtual reality environments[END_REF] was the inability to accurately differentiate between the action verbs "catch" and "touch", as each tool tip was considered as a unique virtual object. The virtual reality simulator was updated to include four separating regions rather than one, allowing these actions to be readily differentiated. Accordingly, the workflow annotations were manually examined and corrected to ensure annotation quality.

Data pre-processing

The original workflow annotations were formatted in terms of start and finish time, expressed in milliseconds. These annotations were sampled to provide a discrete sequence at 30Hz, synchronized with the kinematic, video, and segmentation data to allow for frame-by-frame annotation. Due to their lack of variability, the two targets and the tool were not included in the workflow annotation.

Furthermore, when no phase, step, or activity occurred, the term "idle" was used.

For each timestamp, the following information was provided: timestamp_number, phase_value, step_value, verb_Left_Hand, verb_Right_Hand.

Ground truth uncertainties

The semantic segmentations were the primary source of uncertainty in the ground truth. Due to the transformation of 3D meshes into 2D images, some pixels were attributed to the wrong class, especially at boundaries between the right instrument/peg, left instrument/peg, left instrument/block, and left/right instruments (Figure 3). We estimated this uncertainty by counting the number of mis-segmented pixels on 10 images that included many boundary regions, such as those between surgical instruments, pegs, and blocks. On each image, the number of mis-segmented pixels represents less than 0.25% of the total image.

To take into account the fact that this manual assessment was not representative of the whole data set, we estimated that this mis-segmentation represents less than 0.5% of pixels.

Workflow annotations were another source of uncertainty. Although the ASURA method is consistent (i.e., it generates the same result in two identical situations) and a manual check was performed to limit inaccuracies, some components could not be recognized with complete certainty. Two particular instances were identified. First, in sequence 130 of the training data set, the block in step "Block 1 R2L" was inserted in a non-standard way. Specifically, the block was released by the operator, and while falling became inserted in the peg. Therefore, the insert action was absent. The other instance concerned sequence 79 of the test data set. This time, the operator caught a block before the previous one had been fully inserted, leading to an overlap between the steps "Block 5 R2L" and "Block 6 R2L". The second was chosen as the sole annotation to maintain the true beginning of the step.

Data set characteristics

The training and test data sets presented similar characteristics. The mean and standard deviation duration was 140.2±18.9 seconds for the training data set and 141.7 ± 18.0 seconds for the test data set. Figure 4 Another important characteristic of the data sets was the high class unbalance of at least one vocabulary term for each granularity level. For the phases, the term "idle" represented less than 4% of all data, whereas the other phase terms accounted for more than 47% (L2R and R2L). For the steps, the term "idle" represented less than 4%, whereas the non-idle steps accounted for approximately more than 7.5% of each data set(Figures 4a-4d). This unbalance was more pronounced at the action level, where the least represented verb (i.e., "touch") represented approximately 0.6% of the data set, whereas the verb "idle" accounted for more than 53%. The detailed distribution values for each granularity level in both data sets are provided in supplementary material.

Assessment method 2.4.1. Metrics

To assess the participants' workflow recognition models and to take into account the high class unbalance, balanced versions of accuracy, precision, recall, and F1 were used.

In practice, however, some small variations in surgical task recognition are not clinically meaningful and do not constitute a true error. Motivated by this, Dergachyova et al. [START_REF] Dergachyova | Automatic data-driven real-time segmentation and recognition of surgical workflow[END_REF] proposed a re-estimation of these classic frame-by-frame scores, called application-dependent scores, to take into account an acceptable delay d. When a predicted transition occurs within a transition window (2d) centered on the ground truth transition, all frames between the two transitions are considered correct if it is the same transition type ( e.g. transition for verb "catch" or verb "extract"). Therefore, the balanced application-dependent accuracy (AD-Accuracy) was used and the acceptable delay was fixed at 250 ms.

To assess the participants' segmentation models, the mean Intersection-Over-Union (IoU) over all classes was also used, also known as the Mean Jaccard Index over all classes. The IoU is the area of overlap between the predicted segmentation (P red) and the ground truth (GT ), divided by the area of union between the P red and the GT . In our cases, there was a multi-class segmentation problem, therefore the mean IoU value of the image was calculated by taking the IoU of each class and averaging it over the classes:

M eanIoU f rame = 1 6 class IoU class = 1 6 class |GT ∩ P red| class |GT ∪ P red| class (1) = 1 6 class T P class T P class + F P class + F N class ,
where T P (True Positives) is the number of pixels inside the GT area that are correctly predicted, F P (False Positives) is the number of pixels outside the GT area but predicted as belonging to the class, and F N (False Negatives) is the number of pixels inside the GT area that are incorrectly predicted.

Ranking method

The ranking of the participating methods used only the surgical task recognition metrics. Metrics computed for evaluating the segmentation models were provided for information purposes only.

A metric-based aggregation method using the AD-Accuracy values across all test sequences was used for the ranking. Metric-based aggregation was used according to the recommendations made in [START_REF] Maier-Hein | Why rankings of biomedical image analysis competitions should be interpreted with care[END_REF], which show it to be one of the most robust. As all tasks consisted of recognizing the phase, step, and the actions of the left and right hands (i.e., the left and right verbs), the ranking score for the algorithm a i was computed as follows:

s(a i ) = s phase (a i ) + s step (a i ) + s verb_lef t (a i ) + s verb_right (a i ) 4 (2)
with,

s phase (a i ) = T t=0 phase_balance_accuracy_case_t T , ( 3
)
where T is the number of sequences to test. Similar equations were used for the other terms (s step (a i ), s verb_lef t (a i ) and s verb_right (a i )) with a numerator specific to each, i.e., T t=0 step_balance_accuracy_case_t for s step (a i ), etc.

If a participant method did not produce a prediction for one or several granularity levels, the accuracy given for each missing granularity level was that expected for uniformly random predictions. For example, if a model did not predict the phase, s phase would be set to 1/3 corresponding to the phase having 3 potential values. In practice, this was not encountered and each evaluated model produced results for each level of granularity.

Ranking stability was assessed by testing different ranking methods: mean-ThenRank, medianThenRank, rankThenMean, rankThenMedian, and testBased.

MeanThenRank was chosen for the ranking. MedianThenRank differs from the previous method because it used the median instead of the mean in equation 3. For rankThenMean and rankThenMedian, first, the results of each sequence were ranked among participants, and then the final results were the mean or median of all ranks. The testBased method is based on bootstrapping. The ranking was considered stable if a team was ranked in the same position with the majority of ranking methods. If the ranking was not stable according to the chosen methods, a tie between teams was pronounced. The ranking computation and analysis were performed with the ChallengeR package provided by [START_REF] Wiesenfarth | Methods and open-source toolkit for analyzing and visualizing challenge results[END_REF].

Online recognition compatibility

To be online compatible, the proposed methods must satisfy two conditions:

• to produce predictions faster than the duration between the two samples (i.e., faster than 30 Hz); and

• to be causal (i.e., not use data from a future time point to make predictions).

The computation time was not studied because it could not be assessed fairly for all teams. Indeed, the teams provided a unique Docker image for all tasks, and some teams did not write the output file to standard output as it was received, which did not allow for their durations to be precisely measured.

To verify that the methods were causal, the online availability of the frames was mimicked. One additional sequence of 10 seconds, corresponding to the transfer of the first block from the left to the right, was recorded. This sequence was used to generate 300 sub-sequences, each one a frame longer than the previous. Thus, the first sequence only contained the information of the first frame, the second one contained the information of the two first frames, etc.

The models were run on the 300 sub-sequences and the last prediction of each sub-sequence to create a definitely causal prediction sequence. A method was considered causal if and only if this definitely causal prediction sequence was identical to the prediction sequence given by the full 300 frames. This causalitytesting method is fully automated and also takes into account the complete pipeline used to perform the prediction, such as pre-and post-processing steps, which could lead to a non-causal method even if the network only uses causal components. For reasons of computation time and environmental responsibility, this test was not performed on a whole sequence or the whole test data set. By testing the entire data set, we could be more confident in the causality of the proposed methods, but this would quickly display diminishing returns.

Additional analyses

To further analyze the impact of using multimodal instead of unimodal models, we performed two additional analyses that were not initially included in the challenge design: the statistical significance to use multimodal models instead of unimodal models, and the execution time. These additional analyses only concerned the teams that participated in the multimodal tasks (4 and 5)

with a combination of the same or similar models used for the unimodal tasks.

Comparison between unimodal and multimodal models

To assess the impact of each modality and its combinations on automatic workflow recognition, we performed a statistical analysis with the Wilcoxon test.

The difference was significant if the p-value was inferior to 0.05.

Execution time

Performance is not the only important factor when developing automatic recognition models. Indeed, environmental aspects must also be taken into account [START_REF] Jannin | Towards responsible research in digital technology for health care[END_REF]. To answer this question, we examined the execution time to compute the results of the 60 test sequences. These durations were interpolations that assumed the predictions in each task were computed independently and not the real execution time. Indeed, one team (Hutom, see section 3.2.1) used the predictions from tasks 1 to 3 as input for those of tasks 4 and 5, so the interpolation for the multimodal tasks took into account the execution time for the unimodal ones.

Results: Reporting of the Challenge Outcomes

Challenge submission

By September 12, 2021, 29 participants had registered for the PETRAW challenge: 17 were members of one of the six competing teams. The organizers also submitted results as a non-competing team to provide a baseline. As explained in Section 2.1, some teams obtained unexpected results and three teams resubmitted results for at least one task.

Information on the participating teams and their methods

This section describes each team, the methods they used, and the tasks in which they participated. Competing teams are presented in alphabetical order and not in terms of their ranking.

Hutom

The They used a similar baseline architecture for tasks based on the same modality.

They computed segmentation data from the video recording using a DeepLabV3+ architecture [START_REF] Liang Chieh Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF]. They used a 3D ResNet network [START_REF] Hara | Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet[END_REF] for workflow recognition based on the video modality. For the segmentation modality, they used a SlowFast50 network [START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF] for segmentation-based recognition and a 3D ResNet network for video/kinematic/segmentation-based workflow recognition. They inputted kinematic data on a bi-directional long short-term memory (Bi-LSTM) network [START_REF] Schuster | Bidirectional Recurrent Neural Networks[END_REF]. For multimodal recognition tasks, they used a convolutional feature fusion layer to efficiently perform the fusion of the feature output of each modality. They obtained embedding features with individual modal inputs from each model trained accordingly. Then, they compared the embedding features of each modality with those of other modalities to learn the different representations of each modality. They used the stop gradient-based SimSiam method [START_REF] Chen | Exploring Simple Siamese Representation Learning[END_REF] to compare representations between embedding features. Concomitantly, they stacked embedding features by modality into one block as a chunk and fused them into one embedding through a convolution operation. The approach assumed that feature elements for each modality in the same column have similar temporal information in similar positions. For all networks, they used the Adam optimizer and an initial learning rate of 1e -3 , with a combination of Equalization loss v2 [START_REF] Tan | Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection[END_REF] and Normsoftmax Loss [START_REF] Zhai | Classification is a Strong Baseline for Deep Metric Learning[END_REF] as long-tail recognition for addressing data imbalance.

JHU-CIRL

The JHU-CIRL team (Michael Peven and Gregory D. Hager; Johns Hopkins University) participated in the kinematic-based workflow recognition task.

They performed an under-sampling of the kinematic data to reduce the time dimension size in order to prevent vanishing gradient issues during training.

For the test, they used the same under-sampling. The JHU-CIRL team did not perform any other pre-processing because they considered that besides the positional data, the addition of velocity data was sufficient for the recognition.

They used a unidirectional LSTM network [START_REF] Dipietro | Recognizing surgical activities with recurrent neural networks[END_REF] to recognize the four workflow components. They trained the model using traditional cross-entropy loss and the Adam optimizer. They paid special attention to the selection of the following hyperparameters: sampling rate, learning rate, LSTM hidden dimension size, and the number of layers in the LSTM. They ran 5-fold cross-validation to obtain results from each of these hyperparameters. Then, they selected the best set of hyperparameters for the final training: 15Hz sampling rate, 1e -3 learning rate, 256 LSTM Hidden dimension, and 2 LSTM layers.

MedAIR

The for steps and verbs. Finally, they used three transformer layers to combine spatial and temporal features to obtain the final output for the three labels.

Phases were not directly predicted by the networks, but identified based on the predicted step. They used a stochastic gradient descent (SGD) optimizer with a cross-entropy loss and a learning rate of 5e -4 .

For multimodal-based workflow recognition (video/kinematic-based task), they used a multi-modal relational graph network (MRG-Net) [START_REF] Long | Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery[END_REF]. Like for unimodal-based workflow recognition, they used two CNNs to extract features from each frame in the video for steps and verbs. Then, they obtained the step labels using the original MRG-Net structure, which was the result of the fully connected layer with the output of three nodes in the graph. For the verb labels, the MedAIR team used fully connected layers to produce outputs k l t and k r t , the final label prediction for left and right verb labels. They identified phases based on the predicted step. They used an Adam optimizer with cross-entropy loss and learning rate of 1e -4 .

MMLAB

The MMLAB team was composed of Satyadwyoom Kumar, Lalithkumar Seenivasan, and Hongliang Ren from the Netaji Subhas University of Technology, National University of Singapore, and the Chinese University of Hong Kong.

They participated in the video/kinematic-based recognition task.

MMLAB team proposed a multi-task learning model to perform the recognition. First, each video frame was resized to 224 × 224 pixels. A ResNet 50 [START_REF] He | Deep residual learning for image recognition[END_REF] pre-trained on ImageNet was used to extract visual features for each video frame.

These features were passed with the frame-specific kinematic data through four label-specific networks (one per component). Each label-specific network was composed of two LSTMs [START_REF] Hochreiter | Long Short-Term Memory[END_REF], one for each modality, to capture the temporal features. The sequential length was set to 5, allowing the model to infer based on the current and past 4 temporal information sets. The resulting temporal features were then passed through a single linear layer for recognition. Each label-specific network was trained independently with cross-entropy loss, Adam optimizer, and a learning rate of 1e -3 for phase and step recognition, and 1e -2 for hand verbs.

NCC NEXT

The NCC NEXT team (Hiroki Matsuzaki, Yuto Ishikawa, Kazuyuki Hayashi, Yuriko Harai, and Nobuyoshi Takeshita, National Cancer Center Japan East Hospital) participated in all proposed tasks. They resubmitted a Docker image for all tasks except the kinematic-based recognition task.

They resized the initial video frames to a resolution of 512 × 256 pixels for video-based workflow recognition and of 480 × 270 pixels for segmentation-based workflow recognition. This was followed by normalization. They did not perform any preprocessing of kinematic data.

For video-based workflow recognition they used Xception networks [START_REF] Chollet | Xception: Deep Learning with Depthwise Separable Convolutions[END_REF] pretrained on ImageNet, one per component. They used the Radam optimizer [START_REF] Liu | On the Variance of the Adaptive Learning Rate and Beyond[END_REF] with different learning rates with a batch size of 4, 1e -3 for phases and steps, and 1e -4 with a cosine decay scheduler for hand verbs. They also used cross-entropy loss.

For kinematic-based workflow recognition, the NCC NEXT used the light gradient boosting machine (LightGBM) framework [START_REF] Ke | LightGBM: A Highly Efficient Gradient Boosting Decision Tree[END_REF]. Like for the previous task, they did the training and tuning of hyperparameters (i.e., learning rate, minimum data in leaf, number of iterations, and number of leaves) separately for each component ( The segmentation was performed by a Deeplabv3+ architecture [START_REF] Liang Chieh Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF] with an Xception backbone pre-trained on the Pascal visual object classes (PascalVOC)

data set [START_REF] Everingham | The Pascal Visual Object Classes (VOC) Challenge[END_REF]. With the predicted segmentation, they trained a multi-output classification model, based on the EfficientNetB7 architecture [START_REF] Tan | EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks[END_REF], with Radam optimizer, cross-entropy loss function, a learning rate of 0.0001 with a cosine decay scheduler, and a batch size of 16.

For the multimodal workflow recognition tasks, the NCC NEXT team selected the method used in the three previous tasks that displayed the highest accuracy for each component. Specifically, for video/kinematic-based workflow recognition task, they used the video-based architecture for phase and step recognition and the kinematic-based architecture for hand verb recognition.

For the video/kinematic/segmentation-based model, they used the video-based architecture for phase recognition, the segmentation-based architecture for step recognition, and the kinematic-based architecture for hand verb recognition.

SK

The SK team (Satoshi Kondo, Muroran Institute of Technology) participated in all proposed tasks.

For preprocessing, the SK team resized the images to 640 × 353 pixels and then used random shifting (maximum shift size of 10% of the image size), scaling (0.9 to 1.1 times), rotation (-5 to 5 degrees), color jitter (-0.9 to 1.1 times for brightness, contrast, saturation, and hue), and Gaussian blurring (maximum sigma value = 1.0) for data augmentation. Finally, the images were normalized and the kinematic data were normalized in each dimension.

For the video-based workflow recognition task, the SK team used an 18-layer

ResNet network [START_REF] He | Deep residual learning for image recognition[END_REF], pre-trained on ImageNet. The SK team omitted the final fully-connected layer of ResNet and fed its input 512-dimensional feature vector into two fully-connected layers to obtain a prediction of the step and hand verbs.

Between these fully-connected layers, they inserted one ReLU and Dropout layers. The team used an Adam optimizer, with learning rate changes with cosine annealing with an initial value of 7.2e -4 , and a batch size of 96. The team optimized the initial learning rates for each task with the Optuna library [START_REF] Akiba | Optuna: A Next-generation Hyperparameter Optimization Framework[END_REF].

The team chose cross-entropy loss as the loss function, with weights for each class depending on the class frequency for hand verbs. Phases were not directly predicted from the image, but identified based on the predicted step.

The SK team used a stacked LSTM [START_REF] Hochreiter | Long Short-Term Memory[END_REF] with two layers and 28 hidden layers for the kinematic-based workflow recognition task. The LSTM output was fed into three fully connected layers as done for the previous task. The same optimizer and loss function were used. The initial learning rate was 1.5e -3 with a batch size of 6 and the number of data in a sequence was 30.

Image segmentation was done using the U-Net architecture [START_REF] Ronneberger | U-Net: Convolutional Networks for Biomedical Image Segmentation[END_REF] with ResNet18 as encoder with the summation of cross-entropy loss and dice loss. The SK team exploited the same model used for the video-based workflow recognition task and for the segmentation-based task. Both models were trained separately with an Adam optimizer and an initial learning rate of 2.4e -5 with a batch size of 32 for segmentation, and a learning rate of 1e -4 with a batch size of 6 for recognition.

For the video/kinematic-based task and video/kinematic/segmentation-based task, the SK team ensembled the previously trained dedicated modality networks to obtain a new prediction. As the SK team used the network parameters trained for the previous task, they did not train any network for these tasks.

MediCIS: non-competing team

The MediCIS team was a non-competing team due to the presence of challenge organizers (Quang-Minh Nguyen and Arnaud Huaulmé, University of Rennes 1).

The team participated in all proposed tasks.

For the preprocessing step, they resized the frames to 256 × 512 pixels.

Additionally, to train the segmentation model, they down-sampled the data to 6

Hz. They z-normalized the kinematic data.

For the video-based workflow recognition task, the MediCIS team used a hierarchical RestNet50 network [START_REF] He | Deep residual learning for image recognition[END_REF] pre-trained on ImageNet to extract spatial features. Then, they used a Multi-Stage Temporal Convolutional Network called MS-TCN++ [START_REF] Li | MS-TCN++: Multi-Stage Temporal Convolutional Network for Action 51 Segmentation[END_REF], with two stages, trained from scratch.

For the kinematic-based workflow recognition task, they directly used data as features for a two-stage MS-TCN++.

They selected as their segmentation model a U-Net [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] network trained from scratch with the Adam optimizer, cross-entropy loss, learning rate of 1e -4 , and batch size of 10. Like for the video-based task, workflow recognition was done by hierarchical ResNet50 followed by a two-stage MS-TCN++.

For the video/kinematic-based and video/kinematic/segmentation-based tasks, the MediCIS team extracted unimodal spatial features using a hierarchical ResNet50 network for video and segmentation data, followed by concatenation.

Then, they trained a two-stage MS-TCN++.

They trained all workflow recognition models with the Adam optimizer, cross-entropy loss, learning rate of 1e -4 , and batch size of 2. For the hierarchical ResNet50 network, they emphasized the training for granularities that are harder to recognize using the following weights in the loss: 1 for phases, 2 for steps, and 5 for both action verbs. They set the number of dilated convolutional layers in MS-TCN++ to 10, except for the first layer where it was 11. The number of feature maps for each layer was 64.

Workflow recognition results

All results were computed on the organizers' hardware via the provided Docker images. This section only presents the results used for the ranking (balanced AD-Accuracy). Other results, such as application-dependent scores for each sequence and task, for each participating team, are available as supplementary material and at www.synapse.org/PETRAW.

Task 1: Video-based workflow recognition

Task 1 consisted of recognizing phases, steps, and hand verbs using video data only. Table 3 summarizes the algorithms used by the five teams that submitted models for this task. Causal X Table 3: Algorithms used for task 1. Teams that resubmitted models are highlighted with an asterisk. An "X" means that the method performed preprocessing, data augmentation, or is causal.

Comparison of the mean AD accuracy values for each test sequence (all models) (Figure 5) showed only a slight performance decrease (from 95.1% to 82.2%), but sequences 79 and 54 displayed the lowest performance (77.7% and 72.9%, respectively). Moreover, for all the test sequences, one model displayed lower AD-Accuracy values than the other models. Comparison of the mean AD-Accuracy value for each model (Figure 6 showed that team SK and team Hutom, obtained the highest values (>90%), followed by team MediCIS and team NCC NEXT (>87%). MedAIR obtained the lowest results (≈84%). Team ranking was not influenced by the chosen method (Figure 7), except for the ranking of the SK and Hutom teams using the rankThenMedian and testBased methods.

Task 2: Kinematic-based workflow recognition

Task 2 consisted of recognizing phases, steps, and hand verbs using kinematic data only. Table 4 summarizes the methods used by the six participating teams for this task.

As with task 1, the performance per sequence slightly decreased (Figure 8).

The highest AD-Accuracy values were superior to 90% for all teams. Three sequences (including sequences 79 and 54) had mean AD-Accuracy values inferior to 80%. Unlike task 1, the majority of sequences did not have outliers.

-3 1e -3 5e -4 1e -1 & 5e -2 1.5e -3 1e -4
Causal X X X Table 4: Summary of the models used for task 2. An "X" means that the method performed preprocessing, data augmentation, or is causal. Results were very similar among teams (Figure 9). Four had a mean AD-Accuracy value of between 89.7% and 90.7%, and the other two displayed mean AD-accuracy values of 86.4% and 84.3%, respectively. Ranking was not stable for team SK and team MediCIS (Figure 10). As

559

MediCIS was a non-competing team, SK was ranked third for this task. Task 3 consisted of recognizing phases, steps, and hand verbs using semantic segmentation data only. First, the results of the segmentation models provided by the participants will be described, and then the workflow recognition models.

Segmentation models:

Table 5 summarizes the methods used by the four participating teams to perform semantic segmentation. Table 5: Segmentation models used for task 3. Teams that resubmitted models are highlighted with an asterisk. An "X" means that the method performed preprocessing, data augmentation, or is causal.

Comparison of the IoU values for each class independently and for all classes (Macro) (Table 6) showed that, the IoU varied between 94.0% and 91.1% for Macro. Pegs were the least recognized structure (IoU between 83.9% and 82.3%).

Specific sequences with lower performance were not identified.

Comparison of the mean IoU values of each team for all classes (Macro) and for each class independently ( Table 8 summarizes the methods used by the four participating teams to perform the workflow recognition.

Comparison of the mean AD-Accuracy values for each test sequence (Figure 12) showed that performance decreased from 87.5% to 76.6%. The same two sequences (79 and 54) displayed very low results (67.4% and 65.5%, respectively).

Moreover, for all test cases, one model had results lower than 70%. Comparison of the mean AD-Accuracy value for each model indicated that three teams obtained results between 88.5% and 87.2%, whereas the Hutom team had a mean AD-Accuracy value of 60.3% (Figure 13). Causal X Teams that resubmitted models are highlighted with an asterisk. An "X" means that the method performed preprocessing, data augmentation, or is causal. The choice of method did not influence the team ranking, except for the 588 second (NCC NEXT) and the third (MediCIS) rank (Figure 14).

Task 4: Video/kinematic-based workflow recognition

Task 4 consisted of recognizing phases, steps, and hand verbs using video and kinematic data. Table 9 summarizes the methods used by the six participating teams.

AD-Accuracy values for each sequence were similar to those of the previous tasks (Figure 15). Indeed, performance slightly decreased from 95.1% to 83.1% for most sequences, and was again low for sequences 79 and 54 (81.2% and 76.5%). For this task, the number of outliers was limited.

-3 1e -4 1e -3 & 1e -2 1e -1 & 5e -2 & 1e -3 & 1e -4 7.2e -4 & 1.5e -3 1e -4
Causal X Table 9: Summary of the models used for task 4. Teams that resubmitted models are highlighted with an asterisk. An "X" means that the method performed preprocessing, data augmentation, or is causal. The NCC NEXT team obtained the best results (Figure 16), with a mean AD-Accuracy of 93.1%, followed by SK, Hutom, and MediCIS teams with results of between 91.6% and 90.2%. For the last two teams, the AD-Accuracy was above 84.5%. The ranking is stable according to the ranking method chosen (Figure 17).

-3 1e -1 , 5e -2 , 1e -3 & 1e -4 7.2e -4 , 1.5e -3 & 1e -4 1e -4
Causal Table 10: Models used for task 5. Teams that resubmitted models are highlighted with an asterisk. An "X" means that the method performed preprocessing, data augmentation, or is causal. The teams' mean AD-Accuracy values ranged between 93.1% and 89.8%

612

(Figure 19). The SK and Hutom teams displayed very similar results, with 91.4%

Additional analyses

The additional analyses concern four of the seven participating teams: Hutom, NCC Next, SK, and MediCIS. They were the only teams to participate with a combination of the same or similar models used for the unimodal tasks. Although MedAIR team participated in task 4 and the two corresponding unimodal tasks

(1 and 2), the models used were too different to allow a model comparison.

Comparison between unimodal and multimodal models

Table 12 presents the results of the statistical analysis. For the four teams, the combination of video and kinematics (task 4) is statistically different than the use of only one modality (tasks 1 and 2). The same statistical differences exist between the combination of the three modalities (task 5) and each modality individually (tasks 1, 2, and 3), with the exception of task 2 and task 5 for the MediCIS team. However, the addition of the segmentation modality (task 5) to the video/kinematic-based (task 4) models was only significant for the MediCIS team.

Discussion

Accurate surgical workflow recognition is necessary for context-aware computerassisted surgical systems. The proposed methods obtained good results but were not perfect and the PETRAW data set itself presented several limitations.

Specifically, the peg transfer task is significantly easier than a real surgical intervention due to the simpler environment, clearly identifiable objects, static field of view, and constant lighting. In addition, each sequence was performed by the same operator resulting in lower data set variability.

By analyzing the performance of the methods across individual sequences, we observed a gradual decrease in performance, except for two sequences (54 and 79) that displayed very low AD-Accuracy compared to the others regardless of modality. We analyzed these two sequences in detail to understand this poor performance. In sequence 54, the block was dropped twice during the transfer between hands, forcing the operator to catch the block for a second time. In addition, one block got stuck on the peg, forcing the operator to reposition it.

Sequence 79 is one of the sequences identified as containing uncertainty (see Section 2.3.3). However, the overlapping steps (by 0.5 seconds) could not entirely explain the low performance, as the overlap was partially compensated by the delay of 0.25 seconds used to compute the AD-Accuracy. In addition, a block got stuck on a peg in this sequence and the order in which the blocks were caught did not correspond to the one used in most sequences. These deviations from the most common workflow might explain the low performance. For tasks 1 to 4, at least one team submitted a method that could be truly causal. It is important to note that several proposed methods were provably non-causal due to their preprocessing steps and not the core network such as with NCC NEXT (task 3), SK (task 1, 3, 4, 5), and MediCIS (task 2, 4 and 5). Causal methods generally have lower performance than non-causal models.

With the exception of task 4, the causal methods displayed performances that were surprisingly close to that of the best method. For example, for task 2, the AD-Accuracy of the best method was 90.7%, compared to 90.3% and 89.7%

for the causal methods by NCC NEXT and SK, respectively. Obviously, it is not possible to conclude that causal methods give similar results to acausal models: i) because during the challenge we did not have the two versions of a similar method, ii) due to data simplicity. Nevertheless, the results of the causal methods are promising for developing applications, such as the implementation of automatic reports after training sessions on a virtual simulator.

Among the seven participating teams, four (Hutom, NCC Next, SK, and MediCIS) participated in the multimodal tasks (4 and 5) with a combination of the same or similar models used for the unimodal tasks. In all cases, recognition was improved when several modalities were used (Table 11); however, the addition of segmentation modality decreased the performance. The statistical analysis (Table 12) confirmed a significant performance improvement when using multimodal models, with the exception of tasks 2 and 5 for the MediCIS team.

The performance decrease experienced with the addition of the segmentation modality to the video/kinematic-based models was only significant for the MediCIS team.

Therefore, the combination of video and kinematic (task 4) data gives significantly better results compared with other modality combinations. The results obtained by the MedAIR team could contradict this point because they obtained better results for the kinematic-based recognition task than for the video/kinematic-based one. However, the models they used were very different:

a Trans-SVNet and an MRG-Net combined with a CNN respectively. So, in this case, it is difficult to determine if the performance modifications were due to the model or to the modalities used. However, task 4 was more time-consuming than task 2 (53 vs. 3 minutes for SK, 57 vs. less than 1 for Hutom, and 203 vs. less than 1 for MediCIS). One may ask whether it is wise to spend 2,000% to 20,000% more computing time for less than a 3% improvement. The training time should also be taken into account, as it is much more time-consuming [START_REF] Patterson | Carbon Emissions and Large Neural Network Training[END_REF][START_REF] Strubell | Energy and Policy Considerations for Deep Learning in NLP[END_REF], but we did not have access to this information. Data storage should also be considered.

Video can require a lot of storage space, especially for long surgical interventions.

Conversely, kinematic data are less voluminous.

Figure 1 :

 1 Figure 1: The virtual reality simulator used for data acquisition.

Figure 2 :

 2 Figure 2: Representative segmentation mask with the six classes: background (black), base (white), left instrument (red), right instrument (green), pegs (blue) and blocks (magenta).

Figure 3 :

 3 Figure 3: Zoom of 219x123 pixels from Figure 2 to highlight segmentation errors. Right instrument/block (green/magenta) and left/right instruments (red/green) errors are shown where pixels are labeled as background (black). On this zoom, only 51 pixels were misssegmented (around 0.2%).

  Training verb left hand. (f) Test verb left hand. (g) Training verb right hand. (h) Test verb right hand.

Figure 4 :

 4 Figure 4: Distribution of each term at each granularity level in the training and test data sets.The y-axis represents the percentage of frames. In (a) and (b), "L2R" means transfer left to right and "R2L" means transfer right to left. In (c) and (d), "B1 L2R" means block 1 left to right, "B2 L2R" means block 2 left to right.

  Hutom team (Bogyu Park, Seungbum Hong, and Minkook Choi from VisionAI hutom) participated in all proposed tasks. They resubmitted a Docker image for all tasks except the kinematic-based recognition task. Before training, they performed a simple pre-processing step. To preserve temporal information, they split data into clips of 8 frames. They normalized kinematic data by standardizing the raw input without data augmentation. They resized video data to 256 × 256 pixels, followed by random cropping (224 × 224 pixels) and normalization. The cropping was limited to preserve the spatial information in each frame of the clip. They resized segmentation data to 512×512 pixels.

  MedAIR team (Yunshuang Li, Yonghao Long, and Qi Dou, Zhejiang University and the Chinese University of Hong Kong) participated in three tasks: video-based, kinematic-based, and video/kinematic-based workflow recognition. They resubmitted a Docker image for the video-based workflow recognition task. The MedAIR team resized videos to 224 × 224 pixels and then augmented the data using a random horizontal flip and a random rotation of 5 • . For kinematic data, they used a linear layer to obtain 2048 dimensions from the 28 dimensions to enrich the information. For unimodal-based workflow recognition (video-based and kinematic-based tasks), the MedAIR team used a Trans-SVNet model [41]. First, they trained two different convolutional neural networks (CNN) to extract spatial features, one for steps and another for left and right verbs. Then, they trained three multi-stage temporal convolutional networks (TCN) to obtain temporal features

Figure 5 :

 5 Figure 5: Task 1 recognition AD-Accuracy values (%) for each sequence. Each dot represents the AD-Accuracy of one model. The x-axis represent the test sequence id.

Figure 6 :

 6 Figure 6: Mean task 1 recognition AD-Accuracy for each model. Each dot represents the AD-Accuracy for one sequence.

Figure 7 :

 7 Figure 7: Task 1 recognition ranking stability using different ranking methods. Rank 1 indicates the best method.

Figure 8 :

 8 Figure 8: Task 2 recognition AD-Accuracy for each sequence. Each dot represents the AD-Accuracy of one model.

Figure 9 :

 9 Figure 9: Mean task 2 recognition AD-Accuracy for each model. Each dot represents the AD-Accuracy for one sequence.

  560

Figure 10 :

 10 Figure 10: Task 2 recognition ranking stability using the indicated ranking methods.

29 3. 3 . 3 .

 2933 Task 3: Segmentation-based workflow recognition

Figure 11 :

 11 Figure 11: Ground truth (a) and segmentation results for each team (b to e) for one frame.

Figure 12 :

 12 Figure 12: Task 3 recognition AD-Accuracy for each sequence. Each dot represents the AD-Accuracy of one model.

Figure 13 :

 13 Figure 13: Mean recognition AD-Accuracy for each model for task 3. Each dot represents the AD-Accuracy for one sequence.

  589

Figure 14 :

 14 Figure 14: Task 3 recognition ranking stability using the indicated ranking methods.

Figure 15 :

 15 Figure 15: Task 4 recognition AD-Accuracy values for each sequence. Each dot represents the AD-Accuracy for one model.

Figure 16 :

 16 Figure 16: Mean task 4 recognition AD-Accuracy for each team. Each dot represents the AD-Accuracy for one sequence.

Figure 18 :

 18 Figure 18: Task 5 AD-Accuracy for each sequence. Each dot represents the AD-Accuracy for one model.

613 3 . 3 . 6 .

 336 Workflow recognition results summary

For task 1 (

 1 video-based recognition), ResNet-based models gave the best results, and the simplest model was ranked first. For task 2 (kinematic-based recognition), LSTM-based methods presented the worst results. For task 3, the two segmentation models used (DeepLabV3 and U-Net), displayed similar IoU values and the differences were probably due to differences in the training characteristics. For workflow recognition, the EfficientNetB7 and ResNet models obtained similar results. For Tasks 4 and 5, the NCC NEXT team's strategy (i.e., using the modality that gave the best results in the unimodal tasks for each workflow component) provided the best result.For the segmentation-based recognition task (task 3), the segmentation quality seemed to influence workflow recognition up to a certain threshold. Indeed, the workflow recognition performances of the three teams with Macro IoU values superior to 94.0% were similar (AD-Accuracy between 88.5% and 87.2%), but the ranking was inverted for the two first teams. Conversely, the workflow recognition performance with a Macro IoU value of 85% dropped drastically (60.3%). Additional research is required to fully quantify and understand the degree to which segmentation quality influences workflow recognition since, in this challenge, teams used different combinations of models for the segmentation and workflow recognition components.

Future work should focus

  on overcoming the limitations of the current data set by including peg transfer sequences performed by several operators in different systems. Moreover, tests on more realistic data are necessary to validate the finding that kinematic data display the best performances in recognition rate and have less environmental impact thanks to the lowest computation time and storage cost.

Table 1 :

 1 Peg-transfer vocabulary.

	Phases	Steps	Verb	Activities Target	Tool
		Block 1 L2R	Catch	Block	Grasper
		Block 2 L2R	Drop	Other block
	Transfer Left	Block 3 L2R	Extract
	To Right (L2R)	Block 4 L2R	Hold
		Block 5 L2R	Insert
		Block 6 L2R	Touch
		Block 1 R2L
		Block 2 R2L
	Transfer Right	Block 3 R2L
	To Left (R2L)	Block 4 R2L
		Block 5 R2L
		Block 6 R2L

  presents the distribution of every vocabulary component for each granularity level in the training data set (Figures4a, 4c, 4e, 4g) and the test data set (Figures4b, 4d, 4f, 4h). Even for underrepresented components, the distribution was very similar in both data sets. For instance, the verb "touch" (left hand) represented 0.59% and

0.60% of the samples in the training and test data sets, respectively, and "touch" (right hand) represented 0.62% and 0.48%, respectively. The distribution of each vocabulary component between each data set is only statistically different (Mann-Whitney test) for two steps: "Block 1 L2R" and "Block 6 L2R", with p=0.045 and p=0.036 respectively.

Table 2

 2

	). They chose gradient boosting as a predictor
	optimizer and the mean absolute error (MAE) as loss of function.

Table 2 :

 2 Hyperparameters for the kinematic based model developed by the NCC NEXT team

Table 7

 7

	) showed similar Macro results for the NCC
	Next, SK and MediCIS teams ( 96.9%, 96.4%, and 94.0%, respectively). The
	Hutom team's Macro IoU was the lowest (85.0%), mainly due to the IoU for
	pegs (63.3%). Figure 11 presents the ground truth and the segmentation results
	of each team for one frame.
	Workflow models

Table 6 :

 6 Mean Intersection-Over-Union values for all classes of each sequence independently

		Hutom * NCC Next *	SK	MediCIS
	Background	97.7	99.5	99.2	98.9
	Base	91.4	98.4	98.4	96.1
	Pegs	63.3	92.1	92.0	85.3
	Blocks	82.8	96.0	96.0	92.2
	Left tool	89.3	98.1	96.1	96.0
	Right tool	85.5	97.8	96.7	95.8
	Macro	85.0	96.9	96.4	94.0

Table 7 :

 7 Mean Intersection-Over-Union values for all the classes of each team. Teams that resubmitted models are highlighted with an asterisk and best results are in bold.

Table 8 :

 8 Summary of the models used for task 3 (segmentation-based workflow recognition).

Table 11

 11 summarizes the results of each team for the five tasks. All the best methods displayed mean AD-Accuracy superior to 90%, except for task 3.

	Team	Task 1 Task 2 Task 3	Task 4	Task 5
	Hutom	90.51 * 84.31	60.28 * 91.33 *	91.27 *
	JHU-CIRL		86.45
	MedAIR	84.31 * 90.72		86.98
	MMLAB				84.80
	NCC NEXT 87.77 * 90.32	87.71 * 93.09 * 93.09 *
	SK	90.77	89.66	88.51	91.61	91.37
	MediCIS	89.15	89.71	87.22	90.18	89.81

Table 11 :

 11 Mean AD-Accuracy of each team for the five tasks. The best results are highlighted in bold for each task. Resubmitted models are highlighted with an asterisk.

Acknowledgements

Authors thanks IRT b<>com for providing the "Surgery Workflow Toolbox

[annotate]" software, used for this work.

Task 5: Video/kinematic/segmentation-based workflow recognition

In task 5, teams recognized phases, steps, and hand verbs using video, kinematic and segmentation data. Table 10 summarizes the recognition methods used by the four participating teams. The models to create the segmentation were the same as those described in Table 5.

As for the previous tasks, the mean AD-Accuracy values per sequence (Figure 18) highlighted a slight performance decrease (from 97.2% to 85.9%). Sequences 79 and 54 again displayed the lowest performances (80.8% and 78.0%, respectively).

the final rank (Figure 20). Team

T4 <> T5 X Table 12: Significant performance differences between unimodal and multimodal tasks. T1 <> T4: comparison of task 1 and task 4; X: significant performance variation (p-value < 0.05).

Execution time

Statements of ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain patient data.

Conflict of interest statement

The authors declare that they have no conflict of interest.


Christophe Serra
email: serrac@ecpm.u-strasbg.fr

Nicolas Berton

Michel Bouquey

Laurent Prat

Georges Hadziioannou

A predictive approach of the influence of operating parameters on the size of polymer particles synthesized in a simplified microfluidic system

published or not. The documents may come

A predictive approach of the influence of the operating parameters on the size of polymer particles synthesized in a simplified microfluidic system

Introduction

Microfluidic devices have been recently developed for the generation of monodisperse emulsions and the synthesis of polymer particles in the range of few microns to several hundreds of microns. Such particles are usually prepared by suspension polymerization or by precipitation. However in this specific range of particle size, these processes induce polydispersity in size. Polymer spheres, 1,2,3,4 rods, 3 disks 3 and plugs 3,4,5 were obtained in microfluidic devices by polymerizing monomer emulsions. The dispersion in particle size is quite low as the coefficient of variation, defined as the standard deviation of the particle size distribution divided by the mean particle size, is typically lower than 5%. From multiphase droplets, 6 it is also possible to synthesize polymer hemispheres and other partially spherical particles, 7 Janus 8 and even ternary 9 polymer particles. Thus microfluidic systems are a promising way for the synthesis of polymer particles of various size, shape and morphology.

By forcing a to-be-dispersed phase into a continuous phase through a membrane, emulsions with coefficient of variation below 10 % 10 can be produced. Microchannel emulsification 11,12 and straightthrough microchannel emulsification processes 13 were able to produce monodisperse microdroplets with diameters in the range of 1 to 100 µm. Emulsions were also generated at a tapered capillary tip in a surfactant-laden continuous phase. 14 In all of these systems, the droplet diameter was found to vary with respect to a characteristic dimension of the system (pore size, channel width, capillary inner diameter) as well as dispersed phase flow rate and outer flow velocity.

Recently a great attention has been paid to T-shaped microchannels 15 and to microfluidic flowfocusing devices (MFFDs). 16 They were extensively used to generate monodisperse droplets or polymer particles, 3,15,16,17,18,19,20,21 and bubbles 22,23,24 in the size range from tens to several hundreds micrometers. However for these planar microsystems, the to-be-dispersed phase is directly in contact with the channels walls. Therefore the material constituting these channels should be carefully chosen to avoid unexpected inverse emulsion. Inverse emulsions occur when the dispersed phase wet preferentially the channels walls. To overcome this problem, it is possible to modify the wettability properties of the material at the very location where the droplets are formed by an appropriate surface pre-treatment 25 or to use selected surfactants. Another option was considered by Takeuchi et al. 25 who developed an axisymmetric microfluidic flow-focusing device (AFFD) which prevents direct contact of the to-bedispersed phase droplets with the channels walls. However the fabrication of such systems necessitates lithography processes which usually require a clean room. In addition, the microchannels can be clogged easily by solid particles.

Gañán-Calvo et al. 26 developed an axisymetric fluidic platform, so-called Flow-Focusing (FF), to generate either microparticles or microbubbles. In this FF, a focused fluid is delivered trough a capillary tube in a chamber pressurized with the focusing fluid and immerged in a large volume of focusing fluid. A very small orifice, directly facing the tip of the capillary tube, serves as the chamber's exit. The focusing fluid stream forces the focused liquid to flow through the orifice producing a microjet which diameter is much smaller than that of the orifice. Then the microjet undergoes some capillary instabilities and break up past the orifice into monodisperse droplets which size is independent of the capillary tube inner diameter but varies with respect to the focused and focusing liquid flow rates. 26a,26b This mechanism is referred as the jetting mode. 33 In jetting, droplet size can be much smaller than the orifice size. Quevedo et al. 27,28 conceived a simplified microfluidic system delivering droplets in the center of an outlet tubing via a thin needle transpiercing the tubing wall. This needle/tubing system exhibited a behavior similar to that of standard microfluidic devices 27 and solved the clogging problem. 28 It is now well established from the literature that the diameter of droplets in microfluidic emulsions is a function of the flow rates 3,8,18,20,24,27,29,30 , viscosities 31,32 and interfacial tensions 29,33,34 of continuous and dispersed phases. However all these parameters have been studied separately and to the best of our knowledge no attempt has been reported to express the variations in droplet diameter with respect to a dimensionless group based on these parameters. Then the diameter of droplets decreases when the flow rate of the dispersed phase decreases or when the flow rate of the continuous phase increases. By varying the flow rate, Q d , of the dispersed phase at different fixed values of the flow rate, Qc, of the continuous phase, it was observed that the ruling parameter was actually the ratio of the flow rates. 20,24 Concerning the viscosity, an early work of Köhler et al. 31 followed by the work of Husny et al. 32 have shown that in T-shaped microchannels smaller droplets can be produced by increasing the viscosity of the continuous phase. However in the particular case of the emulsification of monomers, the effect of the viscosity was only reported for the dispersed phase 3,20,21 and was imposed by the nature of the monomers. It was concluded that the viscosity did not influence droplet size, analogously with the formation of droplets in quiescent fluid. 26a,26c,33 Finally it was found that the smaller the interfacial tension between the dispersed and continuous phase, the smaller were the droplets.

Here, O/W monomer droplets were formed without surfactant or pre-treatment in an axisymmetric needle/tubing microfluidic device. Droplets were polymerized in-situ under UV irradiation to form spherical polymer particles. The effect of the viscosity of the continuous phase was investigated and it was found that a high viscosity of the continuous phase has a double effect. Firstly it allows for the synthesis of smaller particles for a given flow rate of the dispersed and continuous phases. Secondly it slows down the time of migration of the droplets toward the walls to the point that the droplets are transformed into solid particles before wetting can occur. Surfactants are usually necessary to stabilize droplets, but a sufficiently viscous continuous phase allows the production of stable droplets without surfactant. It could be of particular interest for the synthesis of particles with functionalized surface.

Experimental Section Microfluidic system. Figure 1 shows the schematic of the microfluidic system. A needle was inserted inside a T-junction (SS-100-3, Swagelok) along its main axis. The needle tip exits the T-junction at the center of an outlet PTFE tubing. Two syringe pumps (PHD 2000, Harvard Apparatus) were used to deliver the continuous and dispersed phases at a specific flow rate. The liquid-to-be-dispersed phase was injected via the needle. The outlet PTFE tubing length was 60 cm. Experiments reported here were run using gauge-25 and gauge-32 needles with inner diameters of 260 µm and 110 µm respectively and two outlet PTFE tubings with inner diameters of 1.6 mm and 1.06 mm.

The outlet tubing was wrapped in an aluminum foil into which lightguides were introduced. We used a UV source (Lightningcure LC8, Hamamatsu) operating at λ = 365 nm, which corresponds to the maximum of absorbance of the photoinitiator added to the monomer phase.

Direct observation of the droplet formation was made by coupling a CCD camera (Pike F-032B, AVT) with a Nikon microscope (Optishot). The camera captures up to 200 fps at a full resolution of 648 x 488 pixels.

Materials An aqueous solution of methyl cellulose in distilled water was used as the continuous phase. The viscosity was measured using a rheometer (RV8 from Rheo, spindle 6, 50 rpm) and varied with the amount of methyl cellulose from 350 cP to 1450 cP.

The dispersed phase consisted of: Methyl Methacrylate (MMA) 87 wt%; Glycidyl Methacrylate (GMA), 3 wt% ; Dimethacrylate Ethylene Glycol (DIMAEG), 5 wt% ; a photoinitiator (Irgacure 907), 5 wt%. The photoinitiator was provided by Ciba, DIMAEG and GMA were purchased from Aldrich. MMA was purchased from Acros Organics and was used as received. The viscosity of the dispersed phase was 0.72 cP as measured with a Ubbelohde capillary viscosimeter (type 531 10 I, Schott Geräte). The free radical polymerization of dispersed phase droplets under UV irradiation led to insoluble and infusible particles thanks to the presence of a crosslinker (DIMAEG).

Particle size measurement. The diameter of particles was measured using an optical microscope (Optishot, Nikon). The polymerized particles were collected at the outlet of the PTFE tubing. Average diameter, D p , and standard deviation of the particle diameter distribution, σ, were determined by measuring the diameter of at least 50 polymer particles. It should be noticed that monomer droplets shrink when polymerized since the polymer has a higher density (1.18 g.cm -3 ) than the monomer (0.94 g.cm -3 ). Thus polymer particles had diameters 7% smaller than those of monomer droplets.

Results and Discussion

The Reynolds number and the capillary number are two dimensionless numbers that ordinary describe the hydrodynamic conditions in microfluidics. The Reynolds number Re that is considered in the present work is the Reynolds number of the continuous phase. The capillary number, which represents the ratio of viscous forces to interfacial tension forces, usually refers to the dispersed phase (Ca d ). In our system, since the continuous phase is a co-flowing fluid, we also consider a capillary number for the continuous phase (Ca c ). Thus, for the continuous phase, we use Re c =ρ c .V c .D tubing /µ c and Ca c =µ c .V c /γ, where ρ c is the density of the fluid, µ c the viscosity and V c the average velocity of the continuous phase inside the outlet tubing, D tubing the diameter of the outlet tubing and γ the interfacial tension. The capillary number of the dispersed phase is defined as Ca d =µ d .V d /γ, where µ d is the viscosity and V d the average velocity of the dispersed phase in the needle. Experiments were run at low Reynolds (10 -3 ≤ Re c ≤10 -2 ), which is usual in microfluidics.

As reported by other groups, the diameter of droplets decreases when the flow rate of the dispersed phase decreases or when the flow rate of the continuous phase increases. In this work similar values of Q c /Q d led to similar diameters, independently of the total flow Q c +Q d (Figure 2). Two regimes were distinguished for droplet formation. The first one corresponds to a decreasing zone, where particle diameter decreases with increasing flow rates ratio. At high values of the ratio Q c /Q d , a threshold zone is reached where particle diameter remained constant. Snapshots of the droplet formation in each zone (Figure 3) clearly indicate that the mechanism under which droplets are formed is quite different. In the decreasing zone, droplets of uniform diameter are formed at a constant frequency. Since we worked at high viscosity ratios, µ c /µ d , as well as at a relatively low velocity of the continuous and dispersed phases, the droplets were formed accordingly to the work of Cramer et al. 33 in the dripping mode. Thus the long threads characteristic of the jetting mode that can occur under specific conditions in microfluidic devices 3,7,26,33 were not observed. In the dripping mode, droplets are formed at the needle tip. There is no formation of satellite droplets 33,34 and the polydispersity in droplet size is therefore reduced. In the threshold zone, it was observed that several small droplets are released before the formation of the main droplet. The small droplets have a diameter far below the inner diameter of the needle as usually observed in the jetting mode while the main drop has roughly the same size that the needle inner diameter. It can be concluded that the mode of droplet formation observed in the threshold zone is somehow intermediate between the dripping and jetting modes. The effect of the viscosity of the continuous phase, µ c , on particle size was investigated. The value of the average particle diameter in the threshold zone varies with µ c (Figure 4). Higher viscosity of the continuous phase leads to smaller particle diameter. This is due to the relative increase in the shear force exerted on the dispersed phase by the continuous phase over the interfacial force. However one observes on Figure 3 that for µ c > 800 cP, the average particle diameter in the threshold zone does not decrease any more. It is due to the fact that the average particle diameter has reached the needle inner diameter (260 µm). This is consistent with previous observations reported by Cygan et al. 18 or by Quevedo et al. 27 where droplet diameter approaches the dimension of the system while increasing the continuous phase flow rate. Using a thinner capillary, smaller particles were produced (Figure 7). With a gauge-32 needle, we obtained particles of 130 µm average diameter, which is again close to the needle inner diameter (110 µm). Generally the capillary number accounts for the viscous forces and droplet diameter was found to vary with this dimensionless parameter. 21 Furthermore, simulations conducted by Zhang and Stone for droplet formation in viscous flows 34 had already shown that both capillary numbers of continuous and dispersed phases should influence droplet size. Similar to the ratio Q c /Q d , the ratio of the capillary numbers Ca c /Ca d was defined as:

d d c c d c V µ V µ Ca Ca . . = (1)
By plotting the data of Figure 4 versus Ca c /Ca d , a master curve was obtained (Figure 5). We conclude that the ratio of capillary numbers of the continuous and dispersed phases is a relevant parameter which controls the particle diameter. Figure 6. Variation in the average diameter of polymer particles plotted as a function of the capillary numbers ratio between the continuous and dispersed phases for two different outlet tubing diameters using a gauge-32 needle. Variation in particle diameter corresponds to standard deviation of particle diameter, σ.

Experiments were run using two different outlet tubing diameters, 1.6 and 1.06 mm. For both tubings, the experimental data fit the same master curve (Figure 6). We conclude that, for a given average velocity of the continuous phase, the particle size is independent of the outlet tubing inner diameter. However this master curve was obtained using a given needle. Because thinner needles lead to smaller particles, a specific master curve was obtained for each needle (Figure 7). Thus it seems interesting to study the normalized particle diameter defined as D p /D cap , where D p is the particle average diameter and D cap the inner diameter of the needle. By plotting data of Figures, 5, 6 and 7 expressed as the variation of the normalized particle diameter versus Ca c /Ca d , we found an overall master curve (Figure 8). Similar trends were recently reported for droplet formation in a T-shaped microfluidic device. 32 In this study, the experiments were run at constant dispersed phase flow rate and the viscosity of the continuous phase varied. Several master curves of normalized drop diameter versus Q c were obtained for several viscosity ratios. We think that these different master curves may have fallen onto one master curve if the data had been plotted versus the capillary number ratio or at least versus an appropriate number including the viscosity ratio. Although these results were obtained in a smaller and sharper range of viscosity ratios (1< µ c /µ d <50) and by varying only the continuous phase flow rate, the comparison suggests that our simplified microfluidic system and cross-flow shear microfluidic devices exhibit similar behaviors. Data of Figure 8 plotted on a logarithmic scale (Figure 9) arrange linearly. It finally gives a new empirical law describing particle diameter in such microfluidic system:

22 . 0 . -         = d c cap p Ca Ca K D D (2)
The value of K in equation ( 2) is equal to 7.92. The validity of equation ( 2) only extends to the decreasing zone, in which monodisperse particles are obtained. The limit between the decreasing zone and threshold zone can be easily determined from the plot of the coefficient of variation of polymer particles CV=σ/D p , where σ is the standard deviation and D p the particle average diameter, versus the normalized particle diameter, D p /D cap (Figure 10). CV remains below 5% for D p /D cap smaller than 1.5 and tends to increase quite rapidly for higher values of D p /D cap . According to Figure 8, this limiting value of 1.5 corresponds to a value of Ca c /Ca d equal to 3000. This is consistent with observations previously reported that the polydispersity increased above a certain value of the flow rates ratio. It should be noted that the surface tension between the dispersed and continuous phase also influences the droplet diameter. However it is expected that the interfacial tension would not influence the exponent in equation (2). Indeed several models of drop formation have been reported in the literature. All of them predict a droplet diameter varying proportionally to interfacial tension γ. 29,34 In addition Cramer et al. 33 have studied the variation of droplet diameter with the flow rate of the continuous phase for two different values of γ and at constant flow rate of the dispersed phase. Their experimental data gave two parallel lines when plotted on a logarithmic scale. Consequently, the factor K in equation ( 2) should be proportional to γ and we would get parallel lines on Figure 8 for different values of interfacial tension.

CONCLUSION Polymer particles were produced from monomer emulsions without surfactant or pre-treatment using a simplified axisymmetric microfluidic device. Delivering droplets at a needle tip in a viscous co-flowing continuous phase avoids that droplets wet the walls. Particle size can be controlled through the regulation of the viscosity of the continuous phase. We found a new empirical law which predicts particle diameter as a function of the needle inner diameter and the ratio of Capillary numbers of continuous and dispersed phases.

Figure 1 .

 1 Figure1. Schematic of the microfluidic system for the synthesis of controlled-size polymer particles. The monomer dispersed phase and aqueous continuous phase are delivered by syringe pumps. The dispersed phase is injected via a thin needle positioned along the main axis of a 1/16" T-junction by means of a 1/16" OD stainless steel (SS) tubing. The needle tip exits from the T-junction at the center of an outlet PTFE tubing. The continuous phase is injected perpendicular to the main axis of the Tjunction.

Figure 2 .Figure 3 .

 23 Figure 2. Evolution of the average diameter, Dp, of polymer particles as a function of the flow rate ratio between the continuous and dispersed phases for two different continuous phase flow rates (viscosity µc 1450 cP, gauge-25 needle, outlet tubing 1.6 mm i.d.). Variation in particle diameter corresponds to standard deviation of particle diameter, σ.

Figure 4 .

 4 Figure 4. Evolution of the average diameter of polymer particles as a function of the flow rate ratio between the continuous and dispersed phases for different viscosities of the continuous phase: 350 cP, 500 cP, 800 cP, 1050 cP and 1450 cP (gauge-25 needle, outlet tubing 1.6 mm i.d.). For clarity, variation in particle diameter is not represented.

Figure 5 .

 5 Figure 5. Data of Figure4plotted as a function of the capillary numbers ratio between the continuous and dispersed phases (gauge-25 needle, outlet tubing 1.6 mm i.d.). Variation in particle diameter corresponds to the standard deviation of particle diameter, σ.

Figure 7 .Figure 8 .

 78 Figure 7. Master curves of the average diameter of polymer particles obtained using two different needles. Variation in particle diameter corresponds to standard deviation of particle diameter, σ.

Figure 10 .

 10 Figure 10. Evolution of CV versus normalized particle diameter. Insets give a typical plot of the particle size distribution in the decreasing and threshold zones.

ACKNOWLEDGMENT Authors acknowledge the French Ministry of Higher Education and Research for having funded this work through the grant ANR n° NT05-1_45715 as well as Prof. E. Kumacheva for fruitful discussions. Particular thanks are directed to S. Bouilhol for images recording of droplet formation.