William Power
email: william.power@temple.edu

Martin Pavlovski
email: martin.pavlovski@temple.edu

Daniel Saranovic
email: daniel.saranovic@temple.edu

Ivan Stojkovic
email: ivan.stojkovic@temple.edu

Zoran Obradovic
email: zoran.obradovic@temple.edu

Autonomous Navigation for Drone Swarms in GPS-Denied Environments using Structured Learning

Keywords: Machine learning, Adaptive control, Swarm intelligence and decision-making

Drone swarms are becoming a new tool for many tasks including surveillance, search, rescue, construction, and defense related activities. As their usage increases, so does the possibility of adversarial attacks on their contribution to these use cases. One possible avenue, whether deliberate or not, is to deny access to the position feedback offered by the Global Positioning System (GPS). Operating in these 'GPS denied' environments poses a new challenge; both in navigation, and in collision avoidance. This study proposes two novel concepts; a structural model of environmental deviance to aid in autonomous navigation, and a method to use the output of said model to implement a collision avoidance system. Both of these concepts are developed and tested in the framework of a simulated environment that mimics a GPS-denied scenario. Using data from hundreds of simulated swarm flights, this work shows structured learning can improve navigational accuracy without the need for externally provided position feedback.

Introduction

Drone swarms are collections of remotely or autonomously controlled aerial vehicles which maintain some form of internal structure between the individuals. These swarms have utility in many domains and tasks; surveillance (coverage), search and rescue, construction, and defense related activities [START_REF] Hayat | Survey on Unmanned Aerial Vehicle Networks for Civil Applications: A Communications Viewpoint[END_REF][START_REF] Tahir | Swarms of Unmanned Aerial Vehicles -A Survey[END_REF]. Swarms typically rely on some kind of position feedback to maintain the prescribed structure of the swarm. Referred to as localization, this task is typically achieved by utilizing GPS data and other sensor modalities [START_REF] Kokić | Enhanced drone swarm localization using gps and trilateration based on rf propagation model[END_REF][START_REF] Yanmaz | Drone networks: Communications, coordination, and sensing[END_REF]. Without position feedback, maintaining a safe, collision-free flight within a crowded airspace becomes an increasingly difficult task [START_REF] Barbeau | Geocaching-inspired Resilient Path Planning for Drone Swarms[END_REF][START_REF] Wang | Bearing-only Visual SLAM for Small Unmanned Aerial Vehicles in GPS-denied Environments[END_REF].

In domains where adversaries or technical issues might disable or impede the efficacy of position feedback, drone swarms are at risk of losing the position information prerequisites that allow for safe, collision-free flight. In these GPS-Denied environments, other methods must be used to provide position information to the drones, or to provide other navigation schemes that allow for safe flight. This paper seeks to investigate a method for coping with a GPSdenied environment without incurring these systemic costs of complex sensor fusion and navigational calculation. Given a set of drones with the most basic of intra-drone communication, and with a limited computational budget, are there still navigation methods that can cope with the loss of GPS provided position feedback? One of the simplest methods for dealing with the lack direct positional feedback operate is the technique of dead reckoning. In such an approach, the current estimate of a drones position is taken as the 'true' position, and new control signals are calculated based on the relative location of this position to a desired position.

Applying the method of dead reckoning to a drone swarm introduces a new source of error that might lead to collisions of drones within a swarm due to environmental deviations. Dead reckoning assumes the position estimate at the prior time-step is accurate, and only influenced by the action of the drone. It lacks a model of the influences of the environment. However, additional sensors, if available in the GPS-denied environment, provide intra-drone communication and distance keeping. Radio frequency sensors, or simple optical systems may provide such feedback [START_REF] Faigl | Low-cost embedded system for relative localization in robotic swarms[END_REF][START_REF] Lockspeiser | Radio Frequency Ranging for Swarm Relative Localization[END_REF]. While this is useful for collision avoidance, ultimately, they cannot also solve the navigation issue. This work assumes the existence of a simple inter-drone communication channel. This can be used to craft a nearest neighbor graph of the swarm. The history of drone locations over time, coupled with this graph, provides a spatio-temporal data-set.

This work hypothesizes that such a spatio-temporal data-set can be used to train a model which will improve upon the efficacy of dead reckoning in GPS-denied environments by predicting future environmental deviations based on changes in the structure of the swarm network. We hypothesize that such a network encodes some information about the environmental deviations that introduce the accumulated errors of dead reckoning. By using appropriate structural machine learning tools, this information could be leveraged to correct the estimated position of each drone within the swarm. In turn, this will reduce the accumulation of error, and lead to position estimates that are closer to their true value when the model is used versus pure dead reckoning.

The novelty of the proposed procedure is that it provides a framework to safely navigate a swarm of drones in a GPS-denied environment with a purposefully simplified set of sensor modalities and computational capabilities. This framework is shown to provide course maintenance and collision avoidance via structural learning of environmental deviance's in an online manner. To support this, in this study we answer the following research questions: 1) Can a prediction-based approach properly estimate a drones trajectory after communication loss? 2) If so, do structure-based models improve this prediction capability? 3) Can a method for avoiding collisions between swarms be implemented, using the variance of the structured model as input?

The past decade has seen substantial work on the issues of collision avoidance, localization, and mapping [START_REF] Chung | A Survey on Aerial Swarm Robotics[END_REF]. The prevalence of these swarms has incentivized the creation of high fidelity models of both individual drones [START_REF] Fernando | Modelling, simulation and implementation of a quadrotor UAV[END_REF] and of swarms of these individuals [START_REF] Belkhouche | Modeling and controlling 3D formations and flocking behavior of unmanned air vehicles[END_REF][START_REF] Watson | Simulation of unmanned air vehicle flocking[END_REF].

Due to their prevalence, the security and consistency of communication within a swarm becomes paramount [START_REF] Brust | Defending against Intrusion of Malicious UAVs with Networked UAV Defense Swarms[END_REF]. The manner of communication that enables the localization of drones within the swarms becomes an attack surface. In systems where GPS is the communication channel for localization, entering a 'GPS-denied' environment poses a threat to the swarm. Work has been done to produce systems that leverage visual input to accommodate localization in such settings. On board visual sensors have been utilized to localize drones with respect to observations of visual indicators applied to each drone [START_REF] Saska | System for deployment of groups of unmanned micro aerial vehicles in GPS-denied environments using onboard visual relative localization[END_REF]. The bearing of the swarm were combined with observed landmarks to implement a sufficient simultaneous localization and mapping (SLAM) framework. It is also shown that other modalities can be utilized [START_REF] Singh | Landmarks based path planning for UAVs in GPS-denied areas[END_REF]. Radio frequency ranging has been investigate. This can provide a means of inter-drone localization within a swarm by observing the received signal strength of RF signals from neighboring drones [START_REF] Lockspeiser | Radio Frequency Ranging for Swarm Relative Localization[END_REF].

When position feedback is lost, the technique of dead reckoning can be used. This manner of control simply assumes the current position estimate of the drone location is correct, and flies accordingly. Such a system is prone to quick accumulation of error when used in isolation. However, work has been done to improve upon these estimates. The combination of dead reckoning, and observations using a Kalman filter can still allow for improved trajectory tracking [START_REF] Zhou | Dead reckoning and Kalman filter design for trajectory tracking of a quadrotor UAV[END_REF]. Dead reckoning can also be combined with other localization methods to improve its accuracy. Such a method is proposed in which the received signal strength of neighboring drone broadcasts is combined with a dead reckoning scheme to enable accurate localization [START_REF] Wu | Precise Localization and Formation Control of Swarm Robots via Wireless Sensor Networks[END_REF].

Method

Problem Statement

Assume a swarm of D drones that is heading towards a certain destination in a stochastic environment. Given that global communication (e.g., GPS communication) is lost on the way towards the destination, the objective is to predict the positions of all drones at all subsequent time-steps.

Formally, a swarm

S [T ] = {D [T ] 1 , . . . , D [T ]
D } is defined as a collection of D individual drones, where D

[T ] i = {p 1 i , . . . , p T i } represents the trajectory of the i-th drone containing its positions p t i ∈ R K for each time-step t = 1, . . . , T . Over the course of a swarm's flight during which global communication is lost at a certain time-step t c , the objective is to learn a swarm-level function F S = (f 1 , . . . , f D ) : R D×ωK → R D×K . Each f i : R ωK → R K is a drone-level function that, when at any time-step t > t c , predicts the position of the i-th drone at the next time-step p t+1 i based on a ω-sized window of past positions ω t i = [p t-ω+1 i , . . . , p t i ]. The above statement assumes that none of the individual drones is aware of its true position upon communication loss since any position feedback will be unavailable (e.g., after a GPS-denied regime was initiated). Nevertheless, an intra-swarm communication method can be leveraged to allow the drones to share their true and predicted positions before and after communication loss, respectively. This assumption is supported by the fact that low cost and complexity methods are available for enabling such communication [START_REF] Lockspeiser | Radio Frequency Ranging for Swarm Relative Localization[END_REF][START_REF] Faigl | Low-cost embedded system for relative localization in robotic swarms[END_REF]. In addition, the task of distributing this information falls within a well researched field [START_REF] Campion | Uav swarm communication and control architectures: a review[END_REF][START_REF] Chmaj | Distributed processing applications for uav/drones: a survey[END_REF], which is out of the scope of this paper.

Structured Learning for Trajectory Prediction

Given a swarm S [tc] that lost global communication at time-step t c , we propose to use a Multi-Target Gaussian Conditional Random Field (MT-GCRF) model to predict the drones' positions at each subsequent time-step t > t c by leveraging the available positional data from all time-steps prior to t c . To that end, an input matrix W = [W 1 , . . . , W tc-ω ] is constructed for each previous timestep t. For a specific time-step t, W t contains the ω-sized windows of all D drones before t, i.e.

W t = [ω t+ω-1 1 , . . . , ω t+ω-1 D
] . Similarly, a target matrix P = [P 1 , . . . , P tc-ω ] is obtained, where P t = [p t+ω 1 , . . . , p t+ω D ] contains the positions of all D drones at time-step t. Moreover, to capture the betweentarget similarities, a similarity matrix S was additionally constructed as S t ij = 1-

d t ij maxi,j d t ij
, where d t ij is the Euclidean distance between p t i and p t j . The resulting S 1 , . . . , S tc-ω are diagonally placed in a single supra matrix S.

The input and target matrices W and P, along with the similarity matrix S, are then used to learn the parameters of an MT-GCRF model by maximizing the probability of the drones' target positions conditioned on their corresponding windows of previous points, P (P|W). Finally, the learned MT-GCRF model is used to estimate the positions for all drones at each time-step following t c .

MT-GCRF Overview. Let X = [x 1 , . . . , x N ] and Y = [y 1 , . . . , y N ] denote an input matrix of explanatory variables and a target vector of response variables, respectively, such that y i ∈ R K is associated with x i ∈ R M , for each i = 1, . . . , N . A conventional Continuous Conditional Random Field [START_REF] Lafferty | Conditional random fields: Probabilistic models for segmenting and labeling sequence data[END_REF] models the conditional probability over all target vectors Y given X as

P (Y|X) = 1 Z exp    - K k=1   α N i=1 (y i,k -f k (x i )) 2 + β i∼j S ij (y i,k -y j,k ) 2      , (1)
where [α, β] are the model parameters, f k is a parameterized function that outputs an estimate of the k-th dimension of y i given the i-th input vector, and Z is a normalization constant. The first term in the exponent is the association potential that models the pairwise relationship between the k-th target of y i and its estimate. In the seconds term, S ij describes the similarity between y i and y j . The entire similarity matrix S = [S ij ] N ×N can be thought of as an adjacency matrix of a weighted similarity graph. The structure of such a graph reflects the similarities among its nodes.

MT-GCRF Learning. To allow for efficient parameter learning, the conditional probability in Eq. 1 can be transformed to a Multivariate Gaussian form [START_REF] Radosavljevic | Continuous conditional random fields for regression in remote sensing[END_REF]:

P (Y|X) = 1 (2π) N/2 |Σ| 1/2 exp - 1 2 (Y -µ) Σ(Y -µ) . (2)
The above Gaussian-form model is referred to as a Multi-Target Conditional Random Field (MT-GCRF). The parameters of MT-GCRF are determined by maximizing the conditional likelihood of Eq. ( 2). MT-GCRF Inference. Given an input matrix

X ∈ R J×M , inference is car- ried out by calculating µ = αΣ[b 1 , . . . , b J ] , where b i = [f 1 (x i ), . . . , f K (x i )] .

Variance Estimation and Collision Avoidance

To address the third research question, a method for swarm-based collision avoidance was developed. "Swarm-based" refers to the fact that this is a procedure that allows a set of swarms to model each others environmental deviation. In the proposed approach each swarm will manage and train its own structure-based model. Then, recalling the assumption of an existing means of distributed communication, we extend the assumption to imply swarms will be close enough to share models.

Simply sharing the models does not provide a collision avoidance system. To leverage the models in creation of one, this work chooses to interpret the uncertainty of predicted future locations as output variance of the structured model, and in turn uses this to set the radius of sphere representing the 'possible collision' volume for the swarm. Since the swarms will have a copy of each others' models, this radius can be deduced for each swarm. More specifically, we take the mean of the predicted positions of all drones in a swarm to be the swarm's estimated 'center'. Consequently, we define a swarm's variance as the distance of the swarm's center to the predicted position of the furthest drone, plus the maximum per-dimension variance of that drone.

During inference, each swarm uses the shared models to determine if their radius intersects that of another swarm since in such a situation a possible collision might occur soon. To avoid collision, the swarms update the target positions of their control systems such that their revised targets are at positions opposite one-another along the line connecting the centers of the swarms variance radius. The trajectories update distance along this line is set to keep the radii of the possibly colliding swarms from intersecting while simultaneously minimizing change from the intended mission plan. This is done by setting the distance from the center points along the connecting line for each swarm to be half the value of the overlap of the swarm radii. Once these new safe positions have been reached, the swarms reset their targets to the original location.

We have conducted two sets of experiments. The first set is aimed to characterize the efficacy of the model, as measured by its ability to correct the errors induced by environmental deviations when communication is lost and the swarm utilizes dead reckoning without position feedback. In these experiments, the distance from a ground truth flight path (where no feedback is lost) is used to compare the errors from pure dead reckoning, dead reckoning corrected by an unstructured model, and dead reckoning corrected by the proposed structural MT-GCRF model. Parameters of the models were learned from simulated training data gathered while drones locations were available and the prediction were utilized to augment location estimates in a GPS-denied environment. In the second set of experiments, the qualitative efficacy of the proposed collision avoidance mechanism is tested.

Drone-level trajectory prediction

To compare the efficacy of dead reckoning (DR) with no model, an unstructured regression model (UR), and a structured model (MT-GCRF in this case); four sets of data were collected for each simulated flight path. Each of these data sets is collected from a simulation that shares the same set of randomly generated gusts. This is accomplished by managing the random state for the simulations, so that they are all initialized with the same random seed.

The first simulation for each 'run' of the experiment is the baseline, or ground truth run. In this, no communication loss occurs, and the swarm proceeds in navigating with full position feedback for the entire flight path. This provides the basis of comparison for the three methods.

The second, third, and fourth simulations make use of dead reckoning, the unstructured model UR, and the structured model MT-GCRF, respectively. In the case of DR, once communication is lost the navigation of the drone swarm becomes based on an estimate of the current position. Without feedback, this estimate is updated by adding the current instantaneous velocity of each drone to its previous position estimate. In the third simulation, the drones' positions after communication loss are predicted by UR, which are later used by MT-GCRF. In a sense, this is as if we set the β parameter of MT-GCRF to 0. In the fourth simulation, the full MT-GCRF model is used, which learns the relevance of the UR's outputs and the swarm's structure, represented by α and β, respectively.

In both sets of trajectory prediction experiments, the models were ran on simulated flight paths consisting of 150 time steps, with communication loss occurring at the 100-th timestep; meaning that 100 timestep were used for training, while the remaining 50 were used to evaluate the models. All prediction-based models used a ω-sized window, with ω being 10, to construct their training data. The UR model (and therefore the underlying model in MT-GCRF) was a Multi-Target Neural Network with a single hidden layer containing 30 hidden units, trained using the L-BFGS optimization method.

The root mean squared error (RMSE) between the true and predicted drones' positions, along a certain dimension, was used. For a swarm S of D drones, over a flight-path of T time-steps, the error along the k dimension is computed as

RM SE(S, k) = 1 T -tc T t=tc+1 1 D D i (p t i,k -pt i,k ) 2
where pt i,k is the predicted position of the i-th drone at timestep t after and p t i,k is its true position at timestep t.

This set of experiments was split into two groups. One to investigate the lower-complexity swarms, and another to investigate higher-complexity swarm topologies. This was done to aid in highlighting the effect of structure in the efficacy of the prediction. More complex layouts provide more interesting structure, and in turn show a different response to the structured learning method.

Experiments for Lower-Complexity Swarms. In the lower-complexity swarms, the layouts was a four-drone planar swarm, and an eight-drone cubic swarm. In each case, the swarms start flights at one corner of a volume, and move towards the diagonal corner, with a set critical time in the center of the flight path. Each flight had at least one gust present in the training and inference stages. A set of 100 random simulations were conducted, collecting the previously described data.

Experiments for Higher-Complexity Swarms. The higher-complexity swarm experiments were conducted in the same manner as the lower-complexity swarm experiments, but using nine-drone planar, and 27-drone cubic layouts. This is of particular interest because these configurations contain 'internal' drones, whereas the low complexity configurations do not. This is hypothesized to increase the influence of the structure (the similarity matrix S) within the structural model, improving its efficacy over the unstructured model when compared to the lower complexity layouts.

Multi Swarm Collision Avoidance Experiments

To explore the utility of the proposed collision avoidance procedure, a simulated mission was constructed that places two swarms at opposite corners of a volume, with target flight paths that intersect. The critical time for GPS denial was set prior to the ostensible collision time of the swarms. Each swarm uses a separate model that will be shared with the other swarm. Each swarm's model was bagged five times using bootstrap aggregation to obtain its uncertainty (prediction variance). All other experimental settings remain the same as those described in Section 4.1. The resulting behaviour of the swarms is gathered from observation of animations of the flights, as well as from calculating the number (if any) of collisions.

Simulation Design

To test the hypothesis that a structural model can aid in navigation, a simulator must be able to model the behaviour of the drones in a deterministically reproducible way, under a reproducible set of environmental deviations. This must be done while tracking the spatio-temporal data set encoded in the nearest-neighbor graph of the drones. To accomplish this, a simulator was written in Python 3.6.9.

To simulate the use of GPS, and the subsequent loss of GPS feedback, the program tracks an actual and estimated position for each member of the swarm. When in a GPS-enabled regime, the actual position is used, while in a GPSdenied regime the estimated position is used.

The simulation introduces environmental deviation by use of a simple gusting wind model. A gust of wind is represented as a single vector of a random magnitude, pointing a randomly selected direction on the plane. For the duration of the gust (also selected from a random distribution), this vector is used to move the drone position within the simulated environment.

The wind deviation vector is not applied to each drones position identically. To introduce more realistic behaviour, and to partially encode some information about the deviation within the network structure, a model of 'drafting' was introduced. In this model, the relative location of each drone to others, along the axis in which the gust is pointing, will decrease the magnitude of the deviation applied to that drone. That is, drones 'behind' other drones along this gust vector will feel less deviation. To achieve this, a convex hull is calculated, and used to quickly determine the influence of drones on the hull onto those inside, with respect to the wind direction. This process is done recursively until the drafting effect of each drone on each other is calculated, yielding a scaled wind deviation for each drone.

Results

Drone-level trajectory prediction

For the lower and higher complexity trajectory prediction tasks described in Section 4.1, the error is reported using the RMSE measure. The behaviour typical of all models in a single run is illustrated in Figure 1. The unstructured model has a decreased stability compared to the structured model (2 of the 4 drones have erratic flight paths). When the structured model is used, the similarity matrix, and the learned association potentials, can effectively 'smooth out' the impact of the instability of the underlying regressors.

Lower-complexity Swarms. Results of repeating trajectory prediction experiments 100 times for the same flight plan of 4 and 8 drones and a variety of environmental conditions are summarized in Table 1. In this, we can see that both MT-GCRF and UR outperformed pure DR. Additionally, the structured model MT-GCRF outperformed the unstructured model UR.

Higher-complexity Swarms. Trajectory prediction experiments were also repeated 100 times for more complex swarms of 9 and 27 drones. Results are summarized in Table 2. Similar to the low-complexity experiments, the strutured model (MT-GCRF) outperforms the unstructured regressors (UR), which in turn outperforms dead reckoning (DR), in most instances.

Parameter Analysis. The parameters learned by MT-GCRF, over all 100 runs, for both low and higher-complexity swarms are presented in Figure 2. There are two main insights that the figure suggests. First, the α/β ratio decreases for more complex swarms, suggesting that the larger/complex a swarm is, the more relevant the structure is to the model (i.e. the structural term in (1) has Recall that α and β represent the importances of the unstructured predictor and the swarm structure, respectively (refer to Eq. ( 1) for more details). a higher relative value). Moreover, the ratio of the structural terms exhibits a higher stability over multiple runs for larger/complex swarms (i.e. MT-GCRF is more stable w.r.t. its parameters). In the trajectory prediction experiments it was observed that MT-GCRF was more accurate than its alternatives, therefore, MT-GCRF was selected as the prediction model for each of two swarms. Each swarm's MT-GCRF was ran on 150-timestep flight path, with t c set at the 100-th timestep.

Multi Swarm Collision Avoidance

To investigate the efficacy of the avoidance procedure, 100 such simulations were performed and 0 collisions were observed. As a reference, the states of a two-swarm collision procedure, from a single simulation, before, during, and after collision avoidance was triggered, are illustrated in Figure 3.

The results of our study provide evidence that the prediction-based approaches can sufficiently model deviations in drone swarms trajectories after communication loss without using additional sensors. In hundreds of conducted experiments both unstructured and structured predictive models showed significantly improved swarm-level RMSE over dead reckoning when compared to the ground truth. Spatio-temporal patterns of environmental deviations observed before GPS signal was lost were exploited the most effectively by the proposed MT-GCRF structured learning method for autonomous navigation in both the lower and higher complexity swarm configurations. Finally, the proposed structured regression based autonomous navigation method was able to be leveraged to form a collision avoidance framework that successfully managed and avoided a collision between two swarms in GPS-denied environment.

Fig. 1 .

 1 Fig. 1. Example of a simulated flight trajectories of a 4 drone planar swarm. Subfigures contain the ground truth trajectory and the predicted trajectories by DR (left), UR (middle), and MT-GCRF (right) upon communication loss.

Fig. 2 .

 2 Fig.2. The ratio of α to β learned for each swarm size and shape, across all runs. Recall that α and β represent the importances of the unstructured predictor and the swarm structure, respectively (refer to Eq. (1) for more details).

Fig. 3 .

 3 Fig. 3. Multi-Swarm Collision Avoidance. Positions of each swarm's drones as well as the swarm's estimated variances are displayed for a time-step: Left: Prior to triggering the collision avoidance mechanism. Middle: During collision avoidance. Right: After the collision avoidance mechanism has terminated.

Table 1 .

 1 Trajectory prediction for lower-complexity swarms (4 and 8 drones) by DR, UR and MT-GCRF. RMSE and its 90% confidence interval in three dimensions (x,y,z) is reported for 100 random flights over 50 inference timesteps in a GPS-denied environment.

	Model	x	RMSE y	z
			4 Drone Planar
	DR	2.4290 ± 0.0298 1.4102 ± 0.0141 0.3813 ± 0.0230
	UR	0.5854 ± 0.0677 0.3095 ± 0.0333 0.3327 ± 0.0426
	MT-GCRF 0.4280 ± 0.0447 0.2531 ± 0.0233 0.2264 ± 0.0292
			8 Drone Cubic
	DR	2.4204 ± 0.0265 1.4094 ± 0.0141 0.3401 ± 0.0224
	UR	0.5994 ± 0.0713 0.3155 ± 0.0210 0.3582 ± 0.0264
	MT-GCRF 0.4228 ± 0.0374 0.2395 ± 0.0130 0.2875 ± 0.0198

Table 2 .

 2 Trajectory prediction for higher-complexity swarms (9 and 27 drones) by DR, UR and MT-GCRF. RMSE and its 90% confidence interval in three dimensions (x,y,z) is reported for 100 random flights over 50 inference timesteps in a GPS-denied environment.

	Model	x	RMSE y	z
			9 Drone Planar
	DR	2.3408 ± 0.0245 1.3720 ± 0.0145 0.4002 ± 0.0239
	UR	0.7208 ± 0.1019 0.3826 ± 0.0480 0.4850 ± 0.0709
	MT-GCRF 0.5171 ± 0.0628 0.3476 ± 0.0370 0.2753 ± 0.0449
			27 Drone Cubic
	DR	2.2934 ± 0.0224 1.3701 ± 0.0127 0.3748 ± 0.0212
	UR	0.6932 ± 0.0591 0.4127 ± 0.0385 0.4810 ± 0.0425
	MT-GCRF 0.5014 ± 0.0282 0.3637 ± 0.0160 0.3965 ± 0.0238

Acknowledgements This research was supported in part by the NSF grants IIS-1842183,SES-1659998 and AFRL Contract No. FA8750-18-C-0041, subcontact 555027-78056.


Changhee Han
email: han@lpixel.net

Leonardo Rundo

Kohei Murao

Takafumi Nemoto

Hideki Nakayama

Bridging the Gap between AI and Healthcare Sides: Towards Developing Clinically Relevant AI-Powered Diagnosis Systems

Keywords: Translational research, Computer-aided diagnosis, Generative adversarial networks, Data augmentation, Physician training

Despite the success of Convolutional Neural Network-based

Computer-Aided Diagnosis research, its clinical applications remain challenging. Accordingly, developing medical Artificial Intelligence (AI) fitting into a clinical environment requires identifying/bridging the gap between AI and Healthcare sides. Since the biggest problem in Medical Imaging lies in data paucity, confirming the clinical relevance for diagnosis of research-proven image augmentation techniques is essential. Therefore, we hold a clinically valuable AI-envisioning workshop among Japanese Medical Imaging experts, physicians, and generalists in Healthcare/Informatics. Then, a questionnaire survey for physicians evaluates our pathology-aware Generative Adversarial Network (GAN)-based image augmentation projects in terms of Data Augmentation and physician training. The workshop reveals the intrinsic gap between AI/Healthcare sides and solutions on Why (i.e., clinical significance/interpretation) and How (i.e., data acquisition, commercial deployment, and safety/feeling safe). This analysis confirms our pathology-aware GANs' clinical relevance as a clinical decision support system and non-expert physician training tool. Our findings would play a key role in connecting interdisciplinary research and clinical applications, not limited to the Japanese medical context and pathology-aware GANs.

Introduction

Convolutional Neural Networks (CNNs) have enabled accurate/reliable Computer-Aided Diagnosis (CAD), occasionally outperforming expert physicians [START_REF] Hwang | Development and validation of a deep learning-based automatic detection algorithm for active pulmonary tuberculosis on chest radiographs[END_REF][START_REF] Wu | Deep neural networks improve radiologists' performance in breast cancer screening[END_REF][START_REF] Mckinney | International evaluation of an AI system for breast cancer screening[END_REF].

However, such research results cannot be easily transferred to a clinical environment. Artificial Intelligence (AI) and Healthcare sides have a huge gap around technology, funding, and people [START_REF] Allen | A road map for translational research on artificial intelligence in medical imaging: From the 2018 National Institutes of Health/RSNA/ACR/The Academy Workshop[END_REF]. In Japan, the biggest challenge lies in medical data sharing because each hospital has different ethical codes and tends to enclose collected data without annotating them for AI research. This differs from the US, where National Cancer Institute provides annotated medical images [START_REF] Clark | The Cancer Imaging Archive (TCIA): Maintaining and operating a public information repository[END_REF].

Therefore, a Research Center for Medical Big Data was launched in November 2017: collaborating with 6 Japanese medical societies and 6 institutes of informatics, we collected large-scale annotated medical images for CAD research. Using over 60 million available images, we achieved prominent research results, presented at major Computer Vision [START_REF] Tokunaga | Adaptive weighting multifield-of-view CNN for semantic segmentation in pathology[END_REF] and Medical Imaging conferences [START_REF] Kanayama | Gastric cancer detection from endoscopic images using synthesis by GAN[END_REF]. Moreover, we published 6 papers [START_REF] Han | GAN-based synthetic brain MR image generation[END_REF][START_REF] Han | Infinite brain MR images: PGGAN-based data augmentation for tumor detection[END_REF][START_REF] Han | Combining noise-to-image and image-toimage GANs: Brain MR image augmentation for tumor detection[END_REF][START_REF] Han | Learning more with less: Conditional PGGAN-based data augmentation for brain metastases detection using highlyrough annotation on MR images[END_REF][START_REF] Han | Synthesizing diverse lung nodules wherever massively: 3D multi-conditional GAN-based CT image augmentation for object detection[END_REF][START_REF] Han | Learning more with less: GAN-based medical image augmentation[END_REF] on Generative Adversarial Network (GAN)-based medical image augmentation [START_REF] Goodfellow | Generative adversarial nets[END_REF]. Since the GANs can generate realistic samples with desired pathological features via many-to-many mappings, they could mitigate the medical data paucity via Data Augmentation (DA) and physician training.

Aiming to further identify/bridge the gap between AI and Healthcare sides in Japan towards developing medical AI fitting into a clinical environment in five years, we hold a workshop for 7 Japanese people with various AI and/or Healthcare background. Moreover, to confirm the clinical relevance for diagnosis of the pathology-aware GAN methods, we conduct a questionnaire survey for 9 Japanese physicians who interpret Computed Tomography (CT) and Magnetic Resonance (MR) images in daily practice. Fig. 1 outlines our investigation. Contributions. Our main contributions are as follows:

-AI and Healthcare Workshop: We firstly hold a clinically valuable AIenvisioning workshop among Medical Imaging experts, physicians, and Healthcare/Informatics generalists to bridge the gap between AI/Healthcare sides. -Questionnaire Survey for Physicians: We firstly present both qualitative/quantitative questionnaire evaluation results for many physicians about research-proven medical AI. -Information Conversion: Clinical relevance discussions imply that our pathology-aware GAN-based interpolation and extrapolation could overcome medical data paucity via DA and physician training.

Pathology-aware GAN-based Image Augmentation

In terms of interpolation, GAN-based medical image augmentation is reliable because acquisition modalites (e.g., X-ray, CT, MR) can display the human body's strong anatomical consistency at fixed position while clearly reflecting inter-subject variability [START_REF] Hsieh | Computed tomography: Principles, design, artifacts, and recent advances[END_REF][START_REF] Brown | Magnetic resonance imaging: Physical principles and sequence design[END_REF]. This is different from natural images, where various objects can appear at any position; accordingly, to tackle large intersubject/pathology/modality variability, we proposed to use noise-to-image GANs (e.g., random noise samples to diverse pathological images) for (i ) medical DA and (ii ) physician training [START_REF] Han | GAN-based synthetic brain MR image generation[END_REF]. While the noise-to-image GAN training is much more difficult than training image-to-image GANs [START_REF] Tmenova | CycleGAN for style transfer in X-ray angiography[END_REF] (e.g., a benign image to a malignant one), it can increase image diversity for further performance boost. Regarding the DA, the GAN-generated images can improve CAD based on supervised learning [START_REF] Frid-Adar | GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification[END_REF][START_REF] Madani | Chest X-ray generation and data augmentation for cardiovascular abnormality classification[END_REF][START_REF] Konidaris | Generative adversarial networks as an advanced data augmentation technique for MRI data[END_REF]. For the physician training, the GANs can display novel desired pathological images and help train medical trainees despite infrastructural/legal constraints [START_REF] Finlayson | Towards generative adversarial networks as a new paradigm for radiology education[END_REF]. However, we have to devise effective loss functions and training schemes for such applications. Diversity matters more for the DA to sufficiently fill the real image distribution, whereas realism matters more for the physician training not to confuse the trainees.

So, how can we perform GAN-based DA/physician training with only limited annotated images? Always in collaboration with physicians, for improving 2D classification, we combined the noise-to-image and image-to-image GANs [START_REF] Han | Infinite brain MR images: PGGAN-based data augmentation for tumor detection[END_REF][START_REF] Han | Combining noise-to-image and image-toimage GANs: Brain MR image augmentation for tumor detection[END_REF]. Nevertheless, further DA applications require pathology localization for detection and advanced physician training needs the generation of images with abnormalities, respectively. To meet both clinical demands, we proposed 2D/3D bounding box-based GANs conditioned on pathology position/size/appearance. Indeed, the bounding box-based detection requires much less physicians' annotation effort than segmentation [START_REF] Rundo | GTVcut for neuro-radiosurgery treatment planning: an MRI brain cancer seeded image segmentation method based on a cellular automata model[END_REF].

In terms of extrapolation, the pathology-aware GANs are promising because common and/or desired medical priors can play a key role in the conditioningtheoretically, infinite conditioning instances, external to the training data, exist and enforcing such constraints have an extrapolation effect via model reduction [START_REF] Stinis | Enforcing constraints for interpolation and extrapolation in generative adversarial networks[END_REF]. For improving 2D detection, we proposed Conditional Progressive Growing of GANs that incorporates rough bounding box conditions incrementally into a noise-to-image GAN (i.e., Progressive Growing of GANs [START_REF] Karras | Progressive growing of GANs for improved quality, stability, and variation[END_REF]) to place realistic/diverse brain metastases at desired positions/sizes on 256×256 MR images [START_REF] Han | Learning more with less: Conditional PGGAN-based data augmentation for brain metastases detection using highlyrough annotation on MR images[END_REF]. Since the human body is 3D, for improving 3D detection, we proposed 3D Multi-Conditional GAN that translates noise boxes into realistic/diverse 32 × 32 × 32 lung nodules [START_REF] Al-Shabi | Lung nodule classification using deep local-global networks[END_REF] placed at desired position/size/attenuation on CT scans [START_REF] Han | Synthesizing diverse lung nodules wherever massively: 3D multi-conditional GAN-based CT image augmentation for object detection[END_REF]. Interestingly, inputting the noise box with the surrounding tissues has the effect of combining the noise-to-image and image-to-image GANs.

We succeeded to (i ) generate images even realistic for physicians and (ii ) improve detection using synthetic training images, respectively; they require different loss functions and training schemes. However, to exploit our pathologyaware GANs as a (i ) non-expert physician training tool and (ii ) clinical decision support system, we need to confirm their clinical relevance for diagnosis-such information conversion [START_REF] Honda | Multi-aspect mining of complex sensor sequences[END_REF] techniques to overcome the data paucity, not limited to our pathology-aware GANs, would become a clinical breakthrough.

Methods

AI and Healthcare Workshop

-Subjects: 2 Medical Imaging experts (i.e., a Medical Imaging researcher and a medical AI startup entrepreneur), 2 physicians (i.e., a radiologist and a psychiatrist), and 3 Healthcare/Informatics generalists (i.e., a nurse and researcher in medical information standardization, a general practitioner and researcher in medical communication, and a medical technology manufacturer's owner and researcher in health disparities)

-Experiments: As its program shows (Table 1), during the workshop, we conduct 2 activities: (Learning) Know the overview of Medical Image Analysis, including state-of-the-art research, well-known challenges/solutions, and the summary of our pathology-aware GAN projects; (Thinking) Find the intrinsic gap and its solutions between AI researchers and Healthcare workers after sharing their common and different thinking/working styles. This workshop was held on March 17th, 2019 at Nakayama Future Factory, Open Studio, The University of Tokyo, Tokyo, Japan.

Questionnaire Survey for Physicians

-Subjects: 3 physicians (i.e., a radiologist, a psychiatrist, and a physiatrist) committed to (at least one of) our pathology-aware GAN projects and 6 project non-related radiologists without much AI background. This paper's authors are surely not included.

-Experiments: Physicians are asked to answer the following questionnaire within 2 weeks from December 6th, 2019 after reading 10 summary slides written in Japanese about general Medical Image Analysis and our pathologyaware GAN projects along with example synthesized images. We conduct both qualitative (i.e., free comments) and quantitative (i.e., five-point Likert scale [START_REF] Allen | Likert scales and data analyses[END_REF]) evaluation: Likert scale 1 = very negative, 2 = negative, 3 = neutral, 4 = positive, 5 = very positive. 4 Results

Workshop Results

We show the clinically-relevant findings from this Japanese workshop.

Gap between AI and Healthcare Sides

Gap 1: AI, including Deep Learning, provides unclear decision criteria, does it make physicians reluctant to use it for diagnosis in a clinical environment?

-Healthcare side: We rather expect applications other than diagnosis. If we use AI for diagnosis, instead of replacing physicians, we suppose a reliable second opinion, such as alert to avoid misdiagnosis, based on various clinical data not limited to images-every single diagnostician is anxious about their diagnosis. AI only provides minimum explanation, such as a heatmap showing attention, which makes persuading not only the physicians but also patients difficult. Thus, the physicians' intervention is essential for intuitive explanation. Methodological safety and feeling safe are different. In this sense, pursuing explainable AI generally decreases AI's diagnostic accuracy [START_REF] Adadi | Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)[END_REF], so physicians should still serve as mediators by engaging in high-level conversation or interaction with patients. Moreover, according to the medical law in most countries including Japan, only doctors can make the final decision. The first autonomous AI-based diagnosis without a physician was cleared by the Food and Drug Administration in 2018 [START_REF] Abràmoff | Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices[END_REF], but such a case is exceptional.

-AI side: Compared with other systems or physicians, Deep Learning's explanation is not particularly poor, so we require too severe standards for AI; the word AI is excessively promoting anxiety and perfection. If we could thoroughly verify the reliability of its diagnosis against physicians by exploring uncertainty measures [START_REF] Nair | Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation[END_REF], such intuitive explanation would be optional.

Gap 2: Are there any benefits to actually introducing medical AI?

-Healthcare side: After all, even if AI can achieve high accuracy and convenient operation, hospitals would not introduce it without any commercial benefits. Moreover, small clinics, where physicians are desperately needed, often do not have CT or MR scanners [START_REF] Jankharia | Commentary-radiology in India: The next decade[END_REF].

-AI side: The commercial deployment of medical AI is strongly tied to diagnostic accuracy; so, if it can achieve significantly outstanding accuracy at various tasks in the near future, patients would not visit hospitals/clinics without AI. Accordingly, introducing medical AI would become profitable in five years.

Gap 3: Is medical AI's diagnostic accuracy reliable?

-Healthcare side: To evaluate AI's diagnostic performance, we should consider many metrics, such as sensitivity and specificity. Moreover, its generalization ability for medical data highly relies on inter-scanner/inter-individual variability [START_REF] O'connor | The healthy brain network serial scanning initiative: A resource for evaluating inter-individual differences and their reliabilities across scan conditions and sessions[END_REF]. How can we evaluate whether it is suitable as a clinically applicable system?

-AI side: Generally, alleviating the risk of overlooking the diagnosis is the most important, so sensitivity matters more than specificity unless their balance is highly disturbed. Recently, such research on medical AI that is robust to different datasets is active [START_REF] Rundo | USE-Net: incorporating squeeze-andexcitation blocks into U-Net for prostate zonal segmentation of multi-institutional MRI datasets[END_REF].

How to Develop Medical AI Fitting into a Clinical Environment in Five Years

Why: Clinical significance/interpretation -Challenges: We need to clarify which clinical situations actually require AI introduction. Moreover, AI's early diagnosis might not be always beneficial for patients.

-Solutions: Due to nearly endless disease types and frequent misdiagnosis coming from physicians' fatigue, we should use it as alert to avoid misdiagnosis [START_REF] Vandenberghe | Relevance of deep learning to facilitate the diagnosis of HER2 status in breast cancer[END_REF] (e.g., reliable second opinion), instead of replacing physicians. It should help prevent oversight in diagnostic tests not only with CT and MR, but also with blood data, chest X-ray, and mammography before taking CT and MR [START_REF] Li | Medical data stream distribution pattern association rule mining algorithm based on density estimation[END_REF]. It could be also applied to segmentation for radiation therapy [START_REF] Agn | A generative model for segmentation of tumor and organs-at-risk for radiation therapy planning of glioblastoma patients[END_REF], neurosurgery navigation [START_REF] Abi-Aad | Machine learning as a potential solution for shift during stereotactic brain surgery[END_REF], and pressure ulcers' echo evaluation. Along with improving the diagnosis, it would also make the physicians' workflow easier, such as by denoising [START_REF] Yang | Low-dose CT image denoising using a generative adversarial network with Wasserstein distance and perceptual loss[END_REF]. Patients should decide whether they accept AI-based diagnosis under informed consent.

How: Data acquisition -Challenges: Ethical screening in Japan is exceptionally strict, so acquiring and sharing large-scale medical data/annotation are challenging-it also applies to Europe due to General Data Protection Regulation [START_REF] Rumbold | The effect of the general data protection regulation on medical research[END_REF]. Considering the speed of technological advances in AI, adopting it for medical devices is difficult in Japan, unlike in medical AI-ready countries, such as the US, where the ethical screening is relatively loose in return for the responsibility of monitoring system stability. Moreover, whenever diagnostic criteria changes, we need further reviews and software modifications. For example, the Tumor-lymph Node-Metastasis (TNM) classification [START_REF] Sobin | TNM classification of malignant tumours[END_REF] criteria changed for oropharyngeal cancer in 2018 and for lung cancer in 2017, respectively. Diagnostic equipment/target changes also require large-scale data/annotation acquisition again.

-Solutions: For Japan to keep pace, the ethical screening should be adequate to the other leading countries. Currently, overseas research and clinical trials are proceeding much faster, so it seems better to collaborate with overseas companies than to do it in Japan alone. Moreover, complete medical checkup, which is extremely costly, is unique in East Asia, thus Japan could be superior in individuals' multiple medical data-Japan is the only country, where most workers aged 40 or over are required to have medical checkups once a year regardless of their health conditions by the Industrial Safety and Health Act [START_REF] Nawata | Evaluation of the distribution and factors affecting blood pressure using medical checkup data in Japan[END_REF]. To handle changes in diagnostic criteria/equipment and overcome dataset/task dependency, it is necessary to establish a common database creation workflow [START_REF] Mansour | Visual charting method for creating electronic medical documents[END_REF] by regularly entering electronic medical records into the database. For reducing data acquisition/annotation cost, AI techniques, such as GAN-based DA [START_REF] Han | Synthesizing diverse lung nodules wherever massively: 3D multi-conditional GAN-based CT image augmentation for object detection[END_REF] and domain adaptation [START_REF] Ghafoorian | Transfer learning for domain adaptation in MRI: Application in brain lesion segmentation[END_REF], would be effective.

How: Commercial deployment -Challenges: Hospitals currently do not have commercial benefits to actually introduce medical AI.

-Solutions: For example, it would be possible to build AI-powered hospitals [START_REF] Chen | Feasibility study for implementation of the AI-powered Internet+ Primary Care Model (AiPCM) across hospitals and clinics in Gongcheng county, Guangxi, China[END_REF] operated with less staff. Medical manufacturers could also standardize data format [START_REF] Laplante-Lévesque | Hearing device manufacturers call for interoperability and standardization of internet and audiology[END_REF], such as for X-ray, and provide some AI services. Many IT giants like Google are now working on medical AI to collect massive biomedical datasets [START_REF] Morley | Google Health and the NHS: Overcoming the trust deficit[END_REF], so they could help rural areas and developing countries, where physician shortage is severe [START_REF] Jankharia | Commentary-radiology in India: The next decade[END_REF], at relatively low cost.

How: Safety and feeling safe -Challenges: Considering multiple metrics, such as sensitivity and specificity [START_REF] Rossini | Diagnostic accuracy and measurement sensitivity of digital models for orthodontic purposes: A systematic review[END_REF], and dataset/task dependency [START_REF] Huang | Medical knowledge constrained semantic breast ultrasound image segmentation[END_REF], accuracy could be unreliable, so ensuring safety is challenging. Moreover, reassuring physicians and patients is important to actually use AI in a clinical environment [START_REF] Krittanawong | The rise of artificial intelligence and the uncertain future for physicians[END_REF].

-Solutions: We should integrate various clinical data, such as blood test biomarkers and multiomics, with images [START_REF] Li | Medical data stream distribution pattern association rule mining algorithm based on density estimation[END_REF]. Moreover, developing biasrobust technology is important since confounding factors are inevitable [START_REF] Li | Fully convolutional network ensembles for white matter hyperintensities segmentation in MR images[END_REF].

To prevent oversight, prioritizing sensitivity over specificity is essential while maintaining a balance [START_REF] Jain | Addressing class imbalance problem in medical diagnosis: A genetic algorithm approach[END_REF]. We should also devise education for medical AI users, such as result interpretation, to reassure patients [START_REF] Wartman | Reimagining medical education in the age of AI[END_REF].

Questionnaire Survey Results

We show the questions and Japanese physicians' response summaries. Concerning the following Questions 1,2,3, Fig. 2 visually summarizes the expectation scores on medical AI (i.e., general medical AI, GANs for DA, and GANs for physician training) from both 3 project-related physicians and 6 project nonrelated radiologists.

Conclusion

Our first clinically valuable AI-envisioning workshop between people with various AI and/or Healthcare background reveals the intrinsic gap between both sides and its preliminary solutions. Regarding clinical significance/interpretation, medical AI could play a key role in supporting physicians with diagnosis, therapy, and surgery. For data acquisition, countries should utilize their unique medical environment, such as complete medical checkup for Japan. Commercial deployment could come as AI-powered hospitals and medical manufacturers' AI service.

To assure safety and feeling safe, we should integrate various clinical data and devise education for medical AI users. We believe that such solutions on Why and How would play a crucial role in connecting inter-disciplinary research and clinical applications. Through a questionnaire survey for physicians, we confirm our pathologyaware GANs' clinical relevance for diagnosis as a clinical decision support system and non-expert physician training tool: many physicians admit the urgent necessity of general AI-based diagnosis while welcoming our GAN-based DA to handle the lack of medical images. Thus, GAN-powered physician training is promising only under careful tool designing.

We find that better DA and expert physician training require further generation of images with abnormalities. Therefore, for better GAN-based extrapolation, we plan to conduct (i ) generation by parts with coordinate conditions [START_REF] Lin | COCO-GAN: Generation by parts via conditional coordinating[END_REF], (ii ) generation with both image and radiogenomic conditions [START_REF] Xu | Correlation via synthesis: End-to-end nodule image generation and radiogenomic map learning based on generative adversarial network[END_REF], and (iii ) transfer learning among different body parts and disease types. Whereas this paper only explores the Japanese medical context and pathology-aware GANs, our findings are more generally applicable and can provide insights into the clinical practice in other countries.

Fig. 2 :Question 2 :Question 3 :Question 4 :

 2234 Fig. 2: Bar chart of the expectations on medical AI, expressed by five-point Likert scale scores, from 9 Japanese physicians: 3 project-related physicians and 6 project non-related radiologists, respectively. The vertical rectangles and error bars denote the average scores with 95% confidence intervals.

Table 1 :

 1 Workshop program to (i) know the overview of Medical Image Analysis and (ii) find the intrinsic gap and its solutions between AI researchers/Healthcare workers. * indicates activities given by a facilitator (i.e., the first author), such as lectures. Any comments/suggestions about our projects towards developing clinically relevant AI-powered systems based on your experience?

	Time (mins) Activity
		Introduction
	10	1. Explanation of the workshop's purpose and flow*
	10	2. Self-introduction and explanation of motivation for participation
	5	3. Grouping into two groups based on background*
		Learning: Knowing Medical Image Analysis
	15	1. TED speech video watching: Artificial Intelligence Can Change the future of Medical Diagnosis*
	35	2. Lecture: Overview of Medical Image Analysis including state-of-the-art research, well-known
		challenges/solutions, and the summary of our pathology-aware GAN projects*
		(its video in Japanese: https://youtu.be/rTQLknPvnqs)
	10	3. Sharing expectations, wishes, and worries about Medical Image Analysis
		(its video in Japanese: https://youtu.be/ILPEGga-hkY)
	10	Intermission
		Thinking: Finding How to Develop Robust Medical AI
	25	1. Identifying the intrinsic gap between AI/Healthcare sides after sharing their common and different
		thinking/working styles
	60	2. Finding how to develop gap-bridging medical AI fitting into a clinical environment in five years
	10	Intermission
		Summary
	25	1. Presentation
	10	2. Sharing workshop impressions and ideas to apply obtained knowledge
		(its video in Japanese: https://youtu.be/F31tPR3m8hs)
	5	3. Answering a questionnaire about satisfaction and further comments
	5	4. Closing remarks*
	-Question 2: What do you think about using GAN-generated images for
		DA? (five-point Likert scale)
	-Question 3: What do you think about using GAN-generated images for
		physician training? (five-point Likert scale)
	-Question 4:

-Question 1: Are you keen to exploit medical AI in general when it achieves accurate and reliable performance in the near future? (five-point Likert scale) Please tell us your expectations, wishes, and worries (free comments).

Question 1 :

 1 Are you keen to exploit medical AI in general when it achieves accurate and reliable performance in the near future? -Response summary: As expected, the project-related physicians are AIenthusiastic while the project non-related radiologists are also generally very positive about the medical AI. Many of them appeal the necessity of AIbased diagnosis for more reliable diagnosis because of the lack of physicians. Meanwhile, other physicians worry about its cost and reliability. We may be able to persuade them by showing expected profitability (e.g., currently CT scanners have an earning rate 16% and CT scans require 2-20 minutes for interpretation in Japan). Similarly, we can explain how experts annotate medical images and AI diagnoses disease based on them (e.g., multiple physicians, not a single one, can annotate the images via discussion).


I Maglogiannis

A Zlatintsi

A Menychtas
email: amenychtas@unipi.gr

D Papadimatos

P P Filntisis

N Efthymiou
email: nefthymiou@central.ntua.gr

G Retsinas
email: gretsinas@central.ntua.gr

P Tsanakas

P Maragos
email: maragos@cs.ntua.grdpapadimatos@gmail.com

An intelligent cloud-based platform for effective monitoring of patients with psychotic disorders

Keywords: Healthcare platforms, Biosignal collection, Patient monitoring, Psychotic disorders, Motion analysis

The therapy of patients with psychotic disorders (i.e., bipolar disorder and schizophrenia) could benefit from the constant monitoring of their physiological and motor parameters. In this paper, we present an innovative and advanced cloud based platform that facilitates the effective monitoring of such patients. A commodity smartwatch is used for biosignal and motion data collection at a 24/7 basis. The paper describes the technical details of the implemented application both on the smartwatch and the cloud server side. Technical challenges regarding the upload, the storage and the battery constraints of the smartwatch are also discussed, along with the initial results regarding data visualization and processing.

Introduction

A key issue in the therapy of patients with psychotic disorders (i.e., bipolar disorder and schizophrenia) is the ability to constant monitor their physiological and mental status, even 24/7 if that is possible. In this research work, we present an innovative and advanced computer-based platform that facilitates the effective monitoring and can support the relapse prevention in such patients. This research work is done within the e-Prevention research and development project that is coordinated by the National Technical University of Athens (https://eprevention.gr/) and it is co-funded by the European Commission and Greek National funds.

The innovative key offerings of the envisioned system, are as follows: 1) longterm continuous recordings of biometric signals and motion data through simple Fig. 1: The overall architecture of the proposed system. wearable sensors (i.e., smartwatch), 2) a portable device (tablet) installed in the patient's residence, which records short-term audio-visual videos of the patient while communicating with the doctor; and by using affective computing methodologies is able to understand the emotional status of the patient, and 3) automatic and systematic storing and management of the captured data in a Cloud Infrastructure. The stored data are then processed through machine learning and signal processing techniques in order to detect changes and patterns, to facilitate the prediction of clinical symptoms and side effects of the patient's medication.

The final user of the platform, who is the attending physician, is able to continuously monitor and optionally also annotate the data and receive information about the patient's daily mental and physiological status by observing the data from the wearable sensors and the emotion analysis of the short videos. Additionally, the computational intelligence algorithms incorporated in the platform enable big data processing and statistical analysis and prediction of salient events regarding the patient's daily routine along with the corresponding visualizations. The overall architecture of the proposed e-Prevention system is illustrated in Fig. 1.

Related Work

The evolution and rapid dissemination of mobile and wearable consumer technology, such as smartwatches and fitness trackers, has created unique opportunities for personalized data collection in an unobtrusive and even affordable way. Furthermore, the enormous technological advances have enabled the reliable recording and quantification of a large number of behavioral and biometric indexes through their sensors [START_REF] Menychtas | Automated integration of wireless biosignal collection devices for patient-centred decision-making in point-of-care systems[END_REF][START_REF] Panagopoulos | Utilizing a homecare platform for remote monitoring of patients with idiopathic pulmonary fibrosis[END_REF]. Such sensors usually include accelerometers, gyroscopes and heart rate monitors among others for measuring and detecting the user's motion/kinetic activity, sleep patterns and autonomic function [START_REF] Boletsis | The use of smartwatches for health monitoring in home-based dementia care[END_REF][START_REF] Patel | A review of wearable sensors and systems with application in rehabilitation[END_REF], opening this way the possibility for non-intrusive acquisition of activity data.

Over the last years, wearable technology due to the inclusion of these sensors has in fact become more proximal to human activity, having as an eminent result to become increasingly acceptable even in healthcare. As early as 2006, the Institute of Medicine recognized the potential to transform mental health services by providing more continuous and precise information on patient-specific behavior, symptoms, and medication side effects. In order to do that, novel methods are demanded to be developed, to transform raw data into knowledge and then use this in turn to support personalized interventions [START_REF] Aung | Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies[END_REF] so as to transform hospitalcentered healthcare practice to proactive, individualized care. Behavioral and biometric indexes have been already used in general medicine and sports and nowadays the evidence shows that they could be introduced into clinical psychiatry [START_REF] Aung | Sensing behavioral symptoms of mental health and delivering personalized interventions using mobile technologies[END_REF], as well. Specifically, psychosis is a spectrum of disorders that underlie different etiopathogenic mechanisms acting on the Central Nervous System (CNS), leading to common symptoms [START_REF] Van Os | Schizophrenia. seminar[END_REF]. Despite extensive research over the last 60 years in neurobiology and neurophysiology of psychotic disorders, their cause remains unclear and reliable biometric indexes for the diagnosis and prediction of the course of the psychotic symptomatology have not yet been found. Based on the fact that the process of psychosis is continuous and relapse is a "biological" process that evolves over time [START_REF] Gaebel | Early neuroleptic intervention in schizophrenia: are prodromal symptoms valid predictors of relapse?[END_REF][START_REF] Mccandless-Glimcher | Use of symptoms by schizophrenics to monitor and regulate their illness[END_REF][START_REF] Wiersma | Natural course of schizophrenic disorders: a 15-year followup of a dutch incidence cohort[END_REF], it would be expected to observe changes in such indexes that are related and likely precede the onset and/or worsening of psychotic conditions. To mention a few, some of the typical early warning signs of mental illnesses include parkinsonian type symptoms, i.e., rigidity, tremor, jerking arm movements, or involuntary movements of the limps, awkward gait, unusual gestures or postures, decreased physical activity and social interactions, abnormal sleep patterns [START_REF]Schizophrenia[END_REF][START_REF] Association | Desk reference to the diagnostic criteria from DSM-5®[END_REF][START_REF] Maxhuni | Classification of bipolar disorder episodes based on analysis of voice and motor activity of patients[END_REF]. The continuous monitoring assessed through such wearables, opens the way for more precise and personalized digital interventions, and complement by enabling detection of early signs of illness relapse, medication adherence or even treatment efficacy and thus may help increase the number of positive clinical outcomes in mental healthcare [START_REF] Collins | Grand challenges in global mental health[END_REF][START_REF] Reinertsen | A review of physiological and behavioral monitoring with digital sensors for neuropsychiatric illnesses[END_REF].

Major research areas in psychiatry [START_REF] Bauer | Self-reporting software for bipolar disorder: validation of chronorecord by patients with mania[END_REF][START_REF] Blum | M-psychiatry: Sensor networks for psychiatric health monitoring[END_REF][START_REF] Koutsouleris | Early recognition and disease prediction in the at-risk mental states for psychosis using neurocognitive pattern classification[END_REF][START_REF] Mcgorry | Biomarkers and clinical staging in psychiatry[END_REF] have suggested employing technological advancement for accurate and continuous monitoring of patients to reduce the impact of mental illness on a patient's daily activities, for early diagnosis and prevention of psychotic relapses so as to increase the effectiveness of treatments. For instance, sensor technology, has been used in order to understand patterns of daily behavior that may be indicative of trends or changes in factors related to mental health, including sleep, mood, and stress [START_REF] Javelot | Telemonitoring with respect to mood disorders and information and communication technologies: overview and presentation of the psyche project[END_REF][START_REF] Mohr | Personal sensing: understanding mental health using ubiquitous sensors and machine learning[END_REF], or to characterize a diverse array of psychopathology, from schizophrenia and depression to general mental health [START_REF] Cornet | Systematic review of smartphone-based passive sensing for health and wellbeing[END_REF]. In [START_REF] Saeb | Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: an exploratory study[END_REF] a combination of accelerometer and location sensors could detect relapse in depressive symptoms before patients reported such changes. It has also been found that such sensor technology could predict changes in affect and behavior (for a review, please see [START_REF] Javelot | Telemonitoring with respect to mood disorders and information and communication technologies: overview and presentation of the psyche project[END_REF]).

System Architecture

The availability of commodity consumer devices (i.e., smartwatches) has eased motion data collection but in the context of the e-Prevention project the requirement to collect and transmit constantly large amounts of raw biosignal data did not fit the normal use of available consumer devices. We evaluated a number of available devices and selected the Samsung Gear S3 Frontier as the user data collection device. Compared to other smartwatches that were tested, the Samsung Gear has the ability to store and send data from acceleration, angular velocity and pulse sensors, providing also the heartbeat interval period value (R-R interval), while it has a large storage capacity (4 GB) capable of storing data for a few days and a WiFi interface removing the need for a smartphone for connectivity.

In addition, the smartwatch has the ability to run specific applications to collect data and upload them to a server. An application was developed in the Tizen Studio environment using technologies to implement Tizen applications (Tizen Hybrid Application) [2]. Hybrid Application applications consist of sections implemented either with native C code to achieve efficient operation or with javascript code to implement interfaces with available Tizen Advanced UI (TAU) libraries. The application implements smartwatch data collection functions and transmission to a cloud infrastructure. Due to the large volume of data produced, various compression techniques without loss of information were considered taking into account the limited computational and energy resources of the smartwatch. Analytics were also incorporated to gather critical smartwatch operating parameters such as power levels, free storage and network availability.

Architecture of the smartwatch application

The architecture of the smartwatch application is illustrated in Fig. 2 and consists of the following modules:

1. The Sensor Service module that interacts with the smartwatch sensors to collect data and store them temporarily on the device. Due to performance requirements it is implemented as a Tizen native service. 2. The Data Transmission Module (Network Service) that compresses and transfers the data to the server; also implemented as a Tizen native service. 3. The Control Service that coordinates the data collection and transmits units taking into account the state of the internet connection and the smartwatch energy status. It also aggregates the smartwatch operating parameters (analytics) for sending to the server. 4. The Tizen Web App interface that provides the user with basic information on the functionality of the application. The implementation utilizes advanced web design technologies based on TAU wearable interface libraries.

Biosignal data collection and energy consumption of smartwatch

The sensor data are provided by the Tizen operating system in various ways dependent on the characteristics of the measured parameters. Extensive testing during the pilot development of the system leaded to the optimum data sampling rate in relation to the optimum time between successive battery charges. Table 1 shows the sampling frequencies that were selected according to the requirements of the project's medical team to ensure continuous recordings between charges of approximately 24 hours. Step data and sleep duration are also stored as calculated by the smartwatch, while GPS tracking was not used as it significantly increased power consumption and reduced the time required between successive loads. The actual output data amounts to about 300 MB/user/day, where the day of use includes approximately 22 hours of continuous recording. The data is compressed at about 100-120 MB/user/day. About 20 users require about 2.5 GB of storage per day. In total, 24 months of compressed logging require approximately 1.7 TB of storage.

Data collection module

The requirements for collecting large volumes of data from smartwatch sensors require an appropriate software architecture based on efficient event-driven sensor management. Tizen uses an asynchronous programming interface (API) for sensors based on function callbacks that allow data transfer when available. The data collection module initializes the sensors required and attaches appropriate management routines to transfer the data and store it along with a timestamp in the smartwatch storage.

Data are organized into separate folders that contain files depending on the sensor type as well as timing information. Each data structure corresponds to a smartwatch usage period between two charging events. The above organization allows smartwatch data to be cached if there is no network connection or data upload problems on the server. Also independent storage in separate structures reduces the risk of data loss.

Data transmission module

The watch has advanced connectivity to local wireless WiFi networks but the volume of data requires efficient management of the data upload. The upload process is demanding on watch resources especially concerning energy, so it only takes place when the smartwatch is in the charging state.

Due to the high volume, the data collection module ensures efficient storage with minimal processing (Fig. 3). The data are stored in the smartwatch's storage in chronological order and are organized into groups corresponding to the intervals between device charging events. The available smartwatch storage capacity (approx. 4 GB) allows continuous recording and storage of measurements corresponding to 8 days. If the smartwatch is connected to a network, then as described below the data are transferred to the cloud and this time is renewed.

Transmission is organized into packets and compressed with the DEFLATE algorithm as implemented by the zlib library [3]. This library is a reliable solution with efficient use of the limited smartwatch computing resources and it is supported by a middleware on the node.js server platform. Running the decompression processes on the server concurrently allows data to be received from multiple smartwatches simultaneously.

Architecture of the biosignal collection server

The server uses the '∼ okeanos' national public infrastructure-as-a-service (IaaS), see Fig. 4 for the server architecture. The application is implemented with node.js technologies and uses the NoSQL MongoDB database. The application runs in an isolated Docker container and is structured as follows:

1. Authentication Controller that implements system resources access policies according to user roles. 2. Device Controller that implements the process of registering new devices in the system and gathering watch information on the functionality of the connected devices. 3. Biosignals Controller that manages the efficient transfer of biosignal data. 4. Admin Controller that implements the system management and monitoring. 5. The WEB UI module that constitutes the interface of the administrators with the application described in Sec. 4.1. 6. Web REST interface (REST interface) that routes REST calls from the devices and Web UI application to the appropriate module. [START_REF] Abarbanel | Obtaining order in a world of chaos [signal processing[END_REF] The system in practice

Web UI for system administrators

The Administrator Web UI interface (Fig. 5) allows devices to monitor and access downloaded data. It consists of a client subsystem implemented with front-end responsive frameworks such as Bootstrap and Vue.js and a server subsystem implemented with node.js. The portal features are: i) access to user log data, ii) new device registration, iii) e-mail notification system to administrators when a device has not uploaded data for more than 1 day, and iv) watch analytics.

The alert system uses information from the Device Controller module concerning device connectivity to generate alerts to administrators. The administrator interface allows access to functionality concerning activity alerts, user data and smartwatch analytics. It also provides access to data collected from the accompanying Android application used to collect activity data, which are manually annotated by the users.

Data management

As mentioned above, for efficient use of limited smartwatch computing resources, data is stored as provided by the available sensors. Motion and pulse tracking Fig. 5: Administrator Web UI.

sensors produce data at regular intervals depending on the sampling rate, while step and sleep data are provided by the smartwatch operating system comprising a very small amount of data compared to the original sensor data.

Various methodologies for storing the original sensor data were tested, but prior to the development of the system, priority was given to recording all available information in a way that allowed for thorough processing to eliminate errors caused by sensor operating problems and smartwatch software. In addition, in the pilot phase, the storage of the .csv format was chosen to save the processing time so as not to limit the processing capabilities.

The sheer volume of data produced and the requirement for flexible ways of securely accessing data create high storage requirements; thus was mitigated by the use of proven data compression techniques. Structuring the application of the server as a set of "micro services" for data transfer, without loss of information, entails the transfer of biosignals from the efficient processing and storage process. As part of the pilot process, the data is stored either uncompressed in the form of .csv files for immediate local processing or as compressed packets for a specific data collection time period ready for off-server transfer.

The need to store a large volume of data led to the creation of a new virtual machine in the ∼okeanos infrastructure for the ultimate storage of the compressed data, and it is available for further processing through an appropriate sftp interface.

Initial data processing and results

Qualitative Analysis

We performed an exploratory data analysis (i.e., kinetic and heart rate data), employing both traditional signal processing techniques, such as short-time analysis, as well as more sophisticated non-linear methods; i.e., multi-scale fractal and non-linear dynamics analysis, in order to extract descriptors that efficiently convey behavioral and biometric information. Such non-linear descriptors have been shown to be of importance for the modeling of a series of other 1D signals, see as for instance [START_REF] Kokkinos | Nonlinear speech analysis using models for chaotic systems[END_REF][START_REF] Maragos | Fractal dimensions of speech sounds: Computation and application to automatic speech recognition[END_REF][START_REF] Pitsikalis | Analysis and classification of speech signals by generalized fractal dimension features[END_REF][START_REF] Zlatintsi | Multiscale fractal analysis of musical instrument signals with application to recognition[END_REF]). Our first preliminary results validate the efficiency of the features, thus they could be employed for building tools that Fig. 6: Visualization of user movement and heart rate data, i.e., accelaration data (top row), gyroscope data (middle row) for all three axis (x, y, z-axis) and data collected by the heart rate motitor (i.e., heart rate and RR-interval data). Fig. 7: Short Time Energy of the acc and gyr signals (x-axis) for two different states (walking and other, e.g., resting, standing), using a 10 sec sliding window. can be used for more complex pattern extraction, i.e., behavioral changes, that can be correlated with mental health issues or relapses. Figure 6 shows examples of the raw data from the accelererometer (acc), gyroscope (gyr) and heart rate monitor (hrm) collected form a control user. All data is shown for a 24 Hour interval (where the horizontal blue line indicates when the user was asleep), so obvious differences can be observed.

Short-time energy analysis: We extracted the short-time energy, to explore differences when the user is walking, sleeping, or doing other activities. Figure 7 presents the calculated short-time energy, on the acc and gyr signals (x-axis), showing two 10-min. intervals corresponding to the different states of walking vs. other activity. The short-time energy is computed using a 400-sample window applied on the signal and provides insights for the signal content. We can observe that the short-time energy presents a multitude of peaks in the acc and gyr data during walking, and fewer peaks with higher values of energy when doing other activities. On the contrary, during sleep, the energy was very low, mainly including the energy of the noise produced by the sensors, thus we omitted showing the respective figures. Concluding, the results are intuitive, and show good discriminative abilities for the different states.

Multi-scale fractals: Regarding non-linear analysis, we explored an efficient algorithm [START_REF] Maragos | Fractal signal analysis using mathematical morphology[END_REF] that measures the short-time fractal dimension, based on the Minkowski-Bouligand dimension [START_REF] Falconer | Fractal geometry: mathematical foundations and applications[END_REF]. Fractal dimension D is between 1 and 2 for 1D signals; and the larger the D is, the larger the amount of geometrical fragmentation of the signal. We conducted our analysis at multiple time scales, since real-world signals do not have the same structure over all scales [START_REF] Maragos | Fractal signal analysis using mathematical morphology[END_REF], and we measured the multiscale fractal dimension (MFD) profile as a descriptor for 10-min. intervals of continuous data. Figure 8 shows MFD profiles measured for the three different states, i.e., walking, other and sleeping for the acc and the hrm data. The various profiles show quite distinctive patterns for every state and each type of data. Specifically, we note high fractal dimension up to D = 1.2 and D = 1.4 for the acc data during walking and other, while it gets up to D = 1.6 for the acc-sleep data. Regarding the hrm data, the higher fractal dimension is noticed for sleeping (up to D = 1.4), while walking is around D = 1 getting higher and up to D = 1.3 for larger scales s.

Nonlinear dynamics and attractor reconstruction: Next, we considered that the human movements, as well as the human heart rate system, constitute separate nonlinear dynamics systems. The two variables that must be calculated in order to unfold the attractor is the time delay T d and the dimension D e . The most common methods for calculating the lag T d constitute of either selecting the first local minimum of the mutual information between the original signal s(n) and its time-delayed version s(n + T d ) or the first zero of the autocorrelation function. The dimension D e is increased until the false neighbours vanish, in which case we consider the attractor has unfolded [START_REF] Abarbanel | Obtaining order in a world of chaos [signal processing[END_REF].

Figure 9 shows the reconstructed attractors for short 1-min. intervals. During walking we observe that the reconstructed phase space exhibits a quite clear topology, due to the fact the user performs specific movements. During sleeping, due to the low amount of movement, the sensor noise is prevalent, with few random movements, while during other activities the phase space is dominated by seemingly random movements without a pattern. We generally notice that a nonlinear dynamical systems approach could be proven to be useful for deducing activities and other behavior throughout everyday life. The reconstruction of the attractor is a necessary step towards computing the nonlinear dynamics of the system. Such values are for example the correlation dimension of the reconstructed attractor [START_REF] Grassberger | Measuring the strangeness of strange attractors[END_REF], and the sample entropy [START_REF] Richman | Physiological time-series analysis using approximate entropy and sample entropy[END_REF]. In Figure 9 we also show those values; D2 is the correlation dimension and SE the sample entropy.

Quantitative Analysis

Database Description: In order to perform the quantitative analysis, we created a small dataset by randomly sampling 10 volunteers. We then selected from each volunteer 120 time intervals of 10-min. length, i.e., 40 intervals during which the subject was sleeping, 40 intervals during walking/running, and 40 intervals during other activities (rest, standing, working). The selected intervals were random, and we only made sure that the data collected during these intervals were valid, since in some cases, sensor malfunctions resulted in data loss, or the heart rate monitor was unable to accurately detect the heart rate. The analysis that follows use this dataset and attempts to identify useful cues/features, that can be used for accurately detecting the state of a person (sleeping, walking, running), so that later on can generalize to other physiological states. It should be finally mentioned that according to the the act for data protection a written consent was collected for all participating users providing motion data that are describe and analyzed in Sec. 5.

Experimental Evaluation: In order to validate the performance of the features explored in Sec. 5.1, we employed a Support Vector Machine (SVM) with an RBF kernel, using 5-fold cross validation. Table 2 presents the results for different subset combinations.

With STE we denote the statistics (mean and std) of short-time energy analysis performed on the: 3 accelerometer (x,y,z-axis), 3 gyroscope (x,y,z-axis), and 1 heart rate signals. By taking the mean and std of the short-time energy calculated over each interval we have a total of 14 features for each interval. CH denotes the correlation dimension and the sample entropy calculated by performing delay embedding with D e = 4 for the kinetic data and D e = 2 for the heart rate data. MFD-G, MFD-A, and MFD-H denote features from the multiscale fractal analysis of the gyroscope (x-axis), the accelerometer (x-axis), and the heart rate, respectively. In order to reduce the number of features extracted by the MFD analysis (900 dimensions for kinetic data and 300 for heart rate), we perform linear sampling, picking a total of 30 points for the heart rate and 32 points for the accelerometer and gyroscope MFD analysis.

As we can see from the accuracy (%) results in Table 2, both the traditional speech processing methods, such as the STE analysis, as well as the multi-scale fractal analysis can be used in order to discriminate between the different states. The same can be observed using the CH features which combined with STE, increase the accuracy of the system. Our results show that the extracted descriptors examined in this paper can provide useful insights and can be used in order to build a tool able to assess the state of the user.

Conclusions

Concluding, the novelty of this work is the implementation of a cloud based platform for the extraction of key markers and indicators that can possibly assist the medical personnel in monitoring patients with mental disorders and predict psychotic symptom's relapses and adverse medicine side effects. This is done initially by long-term recordings of biosignal and motion related indexes and further processing. As this project evolves the collected data will be combined with the short-time videos and their affective analysis as well as with responses of patients to standard questionnaires. Nevertheless, the results so far prove the feasibility of our research effort in the continuous collection of biometric and data along with their prognostic value.

Fig. 2 :

 2 Fig. 2: Smartwatch application architecture.

Fig. 3 :

 3 Fig. 3: Flowchart of data collection and transmission.

Fig. 4 :

 4 Fig. 4: Biosignal Collection server architecture.

Fig. 8 :

 8 Fig. 8: Raw signals and mean and std (error bars) of the multiscale fractal dimension distribution of the acc (x-axis) and the hrm data for walking, other and sleep (the MFDs are shown for 60-sec analysis windows, updated every 30 sec).

Fig. 9 :

 9 Fig. 9: Reconstructed attractors for the three states. Each additional dimension above 3 and up to 6 is shown using different colors. D2 is the correlation dimension, and SE the sample entropy.

Table 1 :

 1 Biosignal collection parameters.

	Data Parameter/hr Sampling frequency Data volume
	Linear Acceleration	20 Hz	ca. 7 MB/hr
	Angular Velocity	20 Hz	ca. 7 MB/hr
	Pulse data	5 Hz	ca. 1.5 MB/hr

Table 2 :

 2 Results on the collected data for various linear and non-linear features.

	Features Acc(%) Features	Acc(%)
	STE	72.17 STE+CH	74.58
	MFD-G	72.08 STE+MFD-G	76.92
	MFD-A	74.16 STE+MFD-GA	77.66
	MFD-H	60.50 STE + MFD-GAH	77.33
	CH	65.30 STE + MFD-GAH+CH 77.58

This research has been co-financed by the European Regional Development Fund of the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH -CREATE -INNOVATE (project code: T1EDK-02890, acronym: e-Prevention).