Wei Ouyang
email: weio@kth.se

Florian Mueller

Martin Hjelmare

Emma Lundberg

Christophe Zimmer

ImJoy: an open-source computational platform for the deep learning era

setting up the required computational environments, which involve many hardware and operating system-dependent libraries with complex dependencies, can be daunting. Furthermore, although these implementations allow prototyping by developers, they use command line instructions and are thus difficult to employ for researchers lacking adequate programming skills. Cloud computing services can alleviate some of these difficulties [START_REF] Haberl | CDeep3M-Plug-and-Play cloud-based deep learning for image segmentation[END_REF] , but require users to upload their data, which can cause privacy and confidentiality issues, or conflict with regulations such as GDPR, especially for biomedical data [START_REF] Tang | Protecting genomic data analytics in the cloud: state of the art and opportunities[END_REF] . Thus, empowering biomedical researchers to take full advantage of recent advances in deep learning currently remains an important challenge.

To address this challenge, we developed ImJoy ( https://imjoy.io ), an open-source platform designed to deliver advanced, yet easy-to-use data analysis tools, especially based on deep learning. The core of ImJoy is a browser based application (i.e. a software running in a web browser) that offers a rich and interactive user interface and leverages modern web technologies ( Fig 1a, Supplementary Figure 1 ). ImJoy's data analysis functionalities are provided by a flexible system of plugins that operate independently of each other but can be organized into workspaces and workflows ( Supplementary Figure 2) . Depending on the need, ImJoy can access various computational resources by performing computations either in the browser itself or using 'plugin engines' that can run either locally or remotely ( Fig 1a, Supplementary Figure 1 ). Below, we highlight ImJoy's main features, along with illustrative plugins with installation links and descriptions being provided in the Methods. A detailed documentation on how to use ImJoy and its plugins and how to develop new ones is provided online ( https://imjoy.io/docs ).

ImJoy provides self-contained plugin development support with a built-in code editor for developing and testing plugins. It permits the use of plugins written in different programming languages (JavaScript, HTML, and Python) within the same analysis workflow. ImJoy's API enables bidirectional data exchange and allows plugins to use functions from other plugins ( Supplementary Figure 2) . Plugins are stored as separate files, can be hosted on GitHub, and distributed with a single link, e.g. by email or social media ( Fig 1b ). This link directly opens ImJoy and, importantly, allows developers to control and automate complex installations, e.g. to set up platform-dependent deep learning libraries (see Methods for more details). This makes installation transparent to the user and removes a major obstacle for tool deployment and reuse by researchers. ImJoy also makes it easy to develop plugins that leverage a large variety of available advanced user-interface libraries. As a result, ImJoy plugins can be optimized for responsiveness and flexibility, making them suitable for different screen sizes including smartphones. Furthermore, these libraries provide vast possibilities for interactive data-visualization. As an illustration, we implemented a plugin for manual annotation of ImJoy offers flexible access to multiple resources for computational analysis, which can be adapted to the computational requirements of each task. For many applications, computations can be performed efficiently within the browser itself. Indeed, beyond their original purpose of navigating the Internet, web browsers are now increasingly used as computational platforms.

Thanks to sandboxing inherent to browser computing, ImJoy also offers higher levels of security and stability than non-browser based software. ImJoy supports several browser-based libraries that offer advanced data processing capabilities. For example, TensorFlow.js allows to run deep learning models on graphical processing units (GPUs), while WebAssembly allows to run Python code in the browser. Note that browser-based computing does not require an Internet connection. As an example, we implemented a plugin to classify images of potentially cancerous skin lesions [START_REF] Tschandl | The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions[END_REF] using deep convolutional networks ( Methods : Skin-Lesion-Analyzer plugin). Thanks to the interface versatility mentioned above, the plugin can run on a smartphone and directly analyze skin images taken with the camera ( Fig 1b ). Importantly, the actual classification task is performed in the phone's browser without sending any data over the Internet, thus mitigating privacy concerns. Another example is a plugin to classify protein localization patterns by deep learning ( Methods : HPA-classification plugin). In addition to predicting localization classes from images, this plugin also shows a class activation map [START_REF] Selvaraju | Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization[END_REF] , which highlights the image regions most important for the classification. Such information is valuable not only for developers aiming to fine-tune the algorithm and improve classification performance, but also for users seeking to interpret the output, e.g. for medical diagnosis [START_REF] Castelvecchi | Can we open the black box of AI?[END_REF] .

To perform more demanding or computationally intensive tasks, such as training or retraining a deep net on large data sets, browser-based computation may not always suffice. Therefore, ImJoy can connect to a plugin engine ( Fig 1a, Supplementary Figure 1 ), which can be launched on different hardware, ranging from a local computer to private servers or public cloud services. The plugin engine has access to the entire Python ecosystem, and uses Conda to automatically solve plugin dependencies and set up virtual environments to avoid dependency conflicts. This relieves users from setting up complex software dependencies and provides reproducible analysis workflows, which is crucial for data science [START_REF] Grüning | Bioconda: sustainable and comprehensive software distribution for the life sciences[END_REF] . Importantly, the unified plugin interface of ImJoy ensures that remote procedure calls to the plugin engine are transparent, i.e. the user will be exposed to the same interface independently of where computations actually occur. As an illustration, we implemented two ImJoy plugins for deep learning based image transformation methods: a plugin implements ANNA-PALM, a recent method to create dense super-resolution images from sparse localization data using neural networks [START_REF] Ouyang | Deep learning massively accelerates super-resolution localization microscopy[END_REF] ( Methods : ANNA-PALM plugin), and another implements CARE, which uses deep learning for 3D image restoration [START_REF] Weigert | Content-aware image restoration: pushing the limits of fluorescence microscopy[END_REF] to fine-tune the algorithm -for example by adding regularization or perform early stopping to remedy overfitting. Together with appropriate computational resources (typically GPUs), ImJoy thus enables users to train these powerful methods on their own data or on other public data sets, which was not easily possible before.

Finally, we note that ImJoy is ideally suited to integrate with online databases and model repositories, providing a versatile front-end to efficiently leverage large numbers of data samples and methods. In addition to the HPA-UMAP plugin mentioned above, which already illustrated interactive exploration of a large image database, we also provide a plugin that implements DeepBind 10 , a deep learning method to predict the sequence-specific binding affinities of proteins ( Methods : DeepBindScan plugin, Fig 2b ). This plugin is based on Kipoi.org -a collaborative initiative that provides ready-to-use trained models for genomics [START_REF] Schindelin | Fiji: an open-source platform for biological-image analysis[END_REF] . Unlike the original website and command line tool [START_REF] Alipanahi | Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning[END_REF] , this plugin allows to analyze any user-defined DNA or RNA sequence from a graphical user interface. Thousands of other models available on Kipoi.org can now easily be implemented through ImJoy.

We presented ImJoy, a new computational platform that permits the development of powerful data processing tools with rich and interactive interfaces, and allows computations to be performed in the browser, on local machines or devices, and remote servers. We highlighted plugins that illustrate interactive visualization capabilities, the use of pre-trained deep learning models, operation on smartphones, retraining of deep models on user-provided data, real-time monitoring of learning, applications to imaging and genomics data, and analysis of large public data repositories. ImJoy enables flexible hybrid computation modes, where analysis tasks can be distributed to a combination of local and remote computational environments. An interesting application is the processing of confidential medical data: raw images could be pre-processed ImJoy -Ouyang et al. - May 29, 2019 and anonymised locally, for example to calculate feature vectors which could then be sent to a remote server for further analysis, e.g for comparison against a database. Although ImJoy was developed with a focus on deep learning, it can be used to implement and deploy a much broader range of computational methods. We foresee several extensions of ImJoy beyond its current implementation and plugins. Among these are the possibility to incorporate scripts already developed for other widely used data processing platforms, for example ImageJ/Fiji [START_REF] Schindelin | Fiji: an open-source platform for biological-image analysis[END_REF] , allowing ImJoy to benefit from their functionalities and, conversely, confer new capabilities to these tools, such as easy deployment in the cloud. We also envision plugins that leverage hybrid computing to provide real-time feedback of freshly trained models to the user in response to annotations or other interactions. Such interactivity (human-in-the-loop) can result in better fine-tuning of the algorithm and improved learning [START_REF] Sommer | Ilastik: Interactive learning and segmentation toolkit[END_REF][START_REF] Holzinger | Interactive machine learning: experimental evidence for the human in the algorithmic loop[END_REF] .

In sum, we believe that ImJoy will greatly accelerate the adoption of deep learning technique in biomedical research by reducing the gap between developers and researchers and thereby open the door to new scientific discoveries, while also promoting reproducible research and open science. Images can be annotated with a dedicated plugin, allowing to generate masks for training. The pre-trained model can be retrained on these data and applied for segmentation of new images using two other plugins. d) The "DeepBindScan" plugin uses all available DeepBind 10 models to predict protein binding affinities against a user defined DNA-or RNA sequence. ImJoy is a browser application developed with modern web technologies (top). Its functionalities are implemented with plugins that can run either directly in the browser or on a dedicated plugin engine (bottom). Browser plugins allow interacting with the user, visualizing data with a large variety of web libraries, performing computational tasks in webworkers, which enable computational tasks to run in a dedicated thread, or using Python through WebAssembly. Native Python plugins are supported through plugin engines implemented in Python and have access to the entire Python ecosystem, allowing them to perform demanding computational tasks with powerful scientific computing libraries such as Numpy, Scipy, or Scikit-image and to leverage deep learning tools such as Tensorflow and Pytorch. Virtual environments and dependent packages are managed with Conda.

Local workstation

ImJoy is open-source (MIT license) and entirely hosted on GitHub in several repositories for the browser app, the plugin engine, the desktop app, and different plugin repositories (more details on https://imjoy.io ).

ImJoy web application

The core of ImJoy is a set of static HTML/Javascript/CSS code which runs completely in the browser and can be served with any static web server. The official ImJoy website ( https://imjoy.io ) is hosted on GitHub and served through GitHub pages. When the user opens the ImJoy app with the browser, these static web pages will be loaded to the browser. No local data will be sent to the imJoy.io server. All data will remain local or exchanged only with the connected plugin engine. This so-called serverless design makes ImJoy suitable to work with sensitive data and offers maximum scalability. Lastly, this design guarantees that the user always obtains the latest ImJoy version upon relaunching the app.

The Desktop App ( https://github.com/oeway/ImJoy-App ) allows to use ImJoy locally without Internet connection, which is achieved by running a web server locally. The user can then access this locally served version of ImJoy (instead of https://imjoy.io ). While the ImJoy core can run offline entirely, some plugins may still require the Internet to fetch remote resources, e.g. javascript libraries or trained models for inference.

To keep the ImJoy application and plugins up to date, users are encouraged to use the online version. Note that this does not imply that the data will be sent to the server, since the ImJoy.io static web server does not accept any user data, and plugins process data either in the browser itself or through the local plugin engine.

ImJoy plugin system

Different plugin types are available to implement user interface, data display, or data processing in the browser or on the plugin engine. Developers can use a consistent API (application program interface) across different programming languages (JavaScript, Python), and `Asynchronous programming`to coordinate plugin processes that run in parallel. Plugins are individual html files, and can be directly developed in ImJoy with the built-in code editor. ImJoy provides isolated workspaces, where plugins for a given data processing task can be grouped.

Dependencies can be defined for each plugin, and will be automatically resolved by ImJoy. A plugin can run in an isolated virtual environment to avoid conflicts with other plugins. Further, plugin tags provide configurable modes for their dependency installation and execution, e.g. to specify operating system dependent versions of Tensorflow, or whether the plugin will use GPU or CPU for computation. This is especially important for modern deep learning libraries, which frequently require operating system and hardware specific dependencies.

Plugin files can be stored in any GitHub repository, which can then serve as a plugin repository.

Plugin developers are thus clearly identified and can use ImJoy as a tool to facilitate usage of their code. Once deployed in the Github plugin repository, plugins can then be installed via a single web link. This will open ImJoy in the browser and allow the user to install the plugin.

Several url parameters further allow to control how the plugin is installed and started. For example, a GitHub commit tag can be provided to control which version of the plugin is installed, thereby guaranteeing high reproducibility.

Example plugins

We designed these example plugins to demonstrate the versatility of ImJoy. For these demonstration plugins we deliberately chose to simplify the user experience, for example by hiding many hyperparameters for neural network training from the interface. However, these plugins can be easily adapted, extended or tailored for different use cases and for increased functionalities.

Below we list all example plugins mentioned in the paper. This link will open ImJoy and display the plugin installation dialog. After confirmation, the plugin will be installed in a dedicated workspace 'imjoy-examples'. The plugin will show up in the plugin list on the left side of the ImJoy interface. Installation progress is shown in the plugin-specific progress bar. More details can be found in the plugin log (accessible by clicking on the symbol left of the plugin name). Once a plugin is correctly installed, its name will turn blue and the plugin can be executed by pressing on it. Note that some plugins may install "helper" plugins which provide additional functionalities, e.g. a dashboard to monitor training progress. Such helper plugins will be shown in the plugin list with a grayed-out name.

Plugins running in the browser (indicated below with "Browser" in the title) can be used directly.

The installation link of these plugins is configured such that they start automatically upon successful installation.

ANNA-PALM (Engine)

Plugin installation link: https://imjoy.io/#/app?w=imjoy-examples&plugin=oeway/ImJoy-Plugins:ANNA-PALM This plugin implements the ANNA-PALM method [START_REF] Ouyang | Deep learning massively accelerates super-resolution localization microscopy[END_REF] , which uses deep learning to reconstruct super-resolution views from sparse, rapidly acquired localization images and/or widefield images. The user can choose between training or prediction from a central launch pad.

Training/prediction can be performed either on provided images of microtubules, or on user-supplied data in the csv format of ThunderSTORM. The launch pad also provides access to a detailed user manual, which is the recommended starting point for a new user.

More information on ANNA-PALM can be found at https://annapalm.pasteur.fr/#/ . Code is ported from https://github.com/imodpasteur/ANNA-PALM

CARE (Engine)

Plugin installation link: https://imjoy.io/#/app?w=imjoy-examples&plugin=oeway/ImJoy-Plugins:CARE This plugin implements an example of the deep learning-based CARE method (Content-aware image restoration) for denoising of 3D fluorescence microscopy images [START_REF] Weigert | Content-aware image restoration: pushing the limits of fluorescence microscopy[END_REF] .

As for the ANNA-PALM plugin, the user can choose between training or prediction from a central launch pad and training can be performed either on provided example data (tribolium

  structures of interest in images, a key requirement for defining training targets in image segmentation ( Methods : Image Annotator plugin). Another example plugin uses uses high dimensional features previously extracted from deep learning models to analyze the localization of proteins based on data from the Human Protein Atlas 17 (HPA, https://proteinatlas.org , Methods : HPA-UMAP plugin, Fig 2a ). Using this plugin, researchers can visualize this entire data set as scatter plots organized by feature-similarity (using dimensionality reduction techniques) and interactively inspect the raw images associated to individual data points. Such interactive visualization capabilities will be instrumental to explore large image databases.

(

  Methods : CARE plugin, Fig 2a ). Other machine learning applications often require annotations performed by an expert before training, a task that is greatly facilitated by ImJoy's afore-mentioned user-interactivity. As an example, we implemented a deep learning nuclei segmentation framework. This framework is the winning entry on the 2018 Kaggle Data Science Bowl for nuclear semententation and based on the popular U-net architecture 2 ( Methods : DPNUnet plugin). From a central plugin, the user can choose between plugins for annotation, mask generation, training, or prediction ( Fig 2b ). The annotation plugin allows researchers to manually outline nuclei, using a computer mouse or touch-screen device, these annotations can then be used for mask generation. The training plugin trains the U-net on the local or remote plugin engine and the prediction plugin uses the trained U-net to segment new images. For all these examples, training can be monitored in real time, e.g. by inspecting the loss function or outputs for individual samples ( Fig 2a ), which allows

Figure 1 .

 1 Figure 1 . Overview of ImJoy.

Figure 2 .

 2 Figure 2 . ImJoy example plugins .

Figure 2 Supplementary Figure 1 :

 21 Figure 1

Please note that ImJoy runs best on the latest version of Google Chrome (version 74.0+) or Firefox (version 66.0+).

  To install a plugin simply open the provided links starting with http://imjoy.io/#/app in a web browser.

Acknowledgements

This work was funded by Institut Pasteur. W.O. was a scholar in the Pasteur-Paris University (PPU) International PhD program and was partly funded by a Fondation de la Recherche Médicale grant to C.Z. (DEQ 20150331762). W.O. is a postdoctoral researcher supported by the Knut and Alice Wallenberg Foundation (2016.0204) and Swedish Research Council (2017-05327). We also acknowledge Investissement d'Avenir grant ANR-16-CONV-0005 for 8 ImJoy -Ouyang et al. -May 29, 2019 funding a GPU farm used for testing ImJoy. We thank the IT department of Institut Pasteur, in particular Stéphane Fournier and Thomas Menard, for providing access to the kubernetes cluster and DGX-1 server for running and testing the ImJoy plugin engine and for technical support. We thank Quang tru Huynh for maintaining the GPU farm and for advice and assistance during the development of ImJoy. We also thank Anna Martinez Casals, Peter Thul, Hao Xu, Andrey Aristov, Anthony Cesnik, Christian Gnann, Alex Hu, Jyotsana Parmar, Kyle M. Douglass, Nico Stuurman, Xian Hao, Shubin Dai, David Guo, Kyrie Zhou for testing and helping with the ImJoy plugin development. We thank Elena Rensen for proofreading the manuscript.

We thank Juan Nunez-Iglesias, Shalin Mehta, Bryant Chhun, Joshua Batson, Loic Royer, Nicholas Sofroniew, and Maxime Woringer for useful advice and discussion.

Code availability

ImJoy ( https://ImJoy.io ) is open-source software released under MIT license. Code is hosted on GitHub: https://github.com/oeway/ImJoy .

Code was ported from https://github.com/vbookshelf/Skin-Lesion-Analyzer .

HPA-Classification (Browser)

Data for training the network is downloaded from https://www.kaggle.com/c/data-science-bowl-2018

Supplementary Figure 2: Plugin interactions in ImJoy . a) Workspaces. ImJoy allows to define different workspaces, each of which can contain different plugins. While keeping these plugins inside their own sandboxes or processes, the ImJoy API permits these plugins to call each other's API functions and exchange data within the same workspace. The same API can be used across programming languages or processes running on another computer. For example, a JavaScript plugin rendering a user interface can call functions of a Python plugin for data-processing and then display the results. In the example code snippets shown, a JavaScript plugin ( getPlugin-demo ) uses the ImJoy API function api.getPlugin to call the function calc_exp of a Python plugin ( calculator ) and calculate the exponential of the provided input. The result is then reported to the user. All API function calls are asynchronous to achieve concurrent execution of plugins. This is implemented by using async/await syntax with the function call as in the example plugin 2. b) Workflows. Plugins within a workspace can also be composed into a workflow, where multiple plugins are chained to perform more complex data analysis and visualization tasks. Note that these workflows can also be stored and shared with other users via a single URL. In the example shown, a JavaScript plugin asking the user to provide a number is executed first, then this number is used by a second plugin to perform calculations.

Methods

Here, we provide a brief overview of the general ImJoy software design and its plugin system.

For a more detailed and regularly updated description, we refer to the ImJoy documentation ( https://imjoy.io/docs ). We also list all example plugins with a brief description and installation links.

ImJoy software design

ImJoy is developed as a minimal and flexible plugin-powered browser application built with modern web technologies (HTML5, JavaScript, WebAssembly). ImJoy provides a server-less solution with offline support, and runs on mobile devices. By design, ImJoy itself provides very minimal task-specific functionality. It offers a flexible plugin system with plugins allowing to provide user interfaces and perform computational tasks. ImJoy consists of three parts, and each can be extended with plugins (detailed below): (1) User Interface, where any existing web design library can be used to interact with the user; (2) Web Computational Backend to execute computational tasks directly inside web browsers; (3) Native Computational Backend (Plugin Engine) to perform more demanding computational tasks either locally or remotely.

We provide a detailed online documentation ( https://imjoy.io/docs ), which includes a user manual and description of all API functions with examples, dedicated demos to illustrate how to implement different plugin types, and a section with frequently asked questions (FAQ). Further, ImJoy is a community partner on image.sc, where questions for how to use ImJoy and process data can be asked in a dedicated forum ( https://forum.image.sc/tags/imjoy ).

Plugins requiring the plugin engine are indicated below with "Engine" in the title. To run these plugins, the plugin engine has to be installed as described in the ImJoy user manual ( https://imjoy.io/docs/#/user_manual?id=imjoy-app-and-plugin-engine ).

Once ImJoy is connected to this engine, the plugins can be assigned to it using the menu to the left of the plugin name. The specified workspace folder will then be created on the engine and the virtual environments and software dependencies of the plugin will be installed. More details are provided in the plugin log, which is accessible by pressing on the inverted exclamation mark next to the plugin name. Large data sets can be loaded into the plugin for annotation as folders. For more details on the required file organization we refer to the plugin documentation. Opening data with existing annotations, will result in an automatic display of these annotations.

HPA-UMAP (Browser)

Plugin installation link: https://imjoy.io/#/app?w=imjoy-examples&plugin=imjoy-team/example-plugins:HPA-UMAP&start

=HPA-UMAP&fullscreen=1

This plugin allows to visualize results of a classification of protein localization patterns from microscope images of the entire HPA cell atlas [START_REF] Pontén | The Human Protein Atlas-a tool for pathology[END_REF] the annotation plugin ("annotate images") permits to browse and annotate training images by manually outlining nuclei; the mask generation plugin ("generate masks") can then be used to create masks based on these annotations; the training plugin ("train with data from the engine") then uses these masks as targets to re-train the DPNUnet model; finally, the trained model can be used for segmenting user-supplied images with the prediction plugin ("predict").

Code for segmentation is ported from https://github.com/selimsef/dsb2018_topcoders

DeepBindScan (Engine)

Plugin installation link: https://imjoy.io/#/app?w=imjoy-examples&plugin=imjoy-team/example-plugins:DeepBindScan

This plugin uses a deep learning approach DeepBind to predict the binding affinities of transcription factors (TFs) and RNA binding proteins (RBPs) in different species to a user-provided DNA-or RNA-sequence [START_REF] Alipanahi | Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning[END_REF] . It is implemented with Kipoi 28 ( https://kipoi.org/ ), which provides ready-to-use trained models for genomics and a corresponding API. The user can specify a sequence, which will then be used as input to all available pre-trained DeepBind models ( https://kipoi.org/groups/DeepBind/ ), each of which predicts the binding affinity of a single TF or RBP from a specific species. The models used require input sequences to be 101 base pairs long. Predicted binding affinities are shown in a vertical bar plot and updated dynamically. Final prediction results can also be exported as csv files.

Please note that for some organisms (e.g. homo sapiens), models for many proteins are provided, and the prediction will then take several minutes.


Lucas Nussbaum
email: lucas.nussbaum@loria.fr

An overview of Fed4FIRE testbeds -and beyond?

Introduction and motivations

We have now reached a point where there is a large number of testbeds available for experimenters throughout the world, each with its own features and capabilities. This is a good thing of course, but also a bad thing, because it is getting increasingly difficult for potential users to identify which testbeds would better match their needs. It is not unusual at all to encounter researchers that were not aware that the experiments they performed could have been done much more efficiently on a publicly available testbed. Also, one could also argue that this results in useless duplication of efforts between testbed efforts, even if there is some value to exploring different ways to solve the same issues.

In the context of the Fed4FIRE+ project, we have been working on presenting a better overview of Fed4FIRE testbeds -the current state being the Fed4FIRE website 1 . Section 2 provides the current state of this overview (and can serve as a technical introduction to Fed4FIRE testbeds), while Section 3 discusses possible future work in that area, in the context of GEFI.

Fed4FIRE testbeds

17 testbeds (14 from Fed4FIRE, 3 outside Fed4FIRE) were analyzed, and are described in Table 1 andTable 2. 13 are wireless testbeds, while 4 testbeds address wired networking, cloud, HPC, or nodes distributed at large.

Wireless testbeds (Table 1) differ on various aspects:

• The environment they are deployed in: anechoic chambers, various indoor or outdoor environments

• The protocols they support

• The hardware they rely on, and the services they provide

The other testbeds (Table 2) are much more diverse, even if Grid'5000 and VirtualWall (which is based on the Cloudlab software stack) are similar in terms of features, as it was pointed in the past [START_REF] Nussbaum | Testbeds Support for Reproducible Research[END_REF].

Plans for collaboration

This work is still at a preliminary stage. And despite discussions about surveys during the previous GEFI meetings, we have not made much progress towards building an overview of the available testbeds offerings. The GEFI workshop could be a good opportunity to discuss ways to make concrete progress. One idea that could be explored is to start small, and aim for a Wikipedia comparison page2 . But that still requires us to refine comparison criterias.

Table 1 :

 1 Wireless testbeds

	Name & website	Categories (F4F web) Patron, Country		Hardware	Environment	Typical protocols	Services
	FuSeCo	Wireless	Fraunhofer, Berlin
	IRIS	Wireless	Trinty College Dublin, Ireland 20x USRP N210	Indoor	5G, any (SDR)
	LOG-a-TEC	Wireless	JSI, Slovenia				Outdoor	6LoWPAN, ZigBee
	NETMODE	Wireless	NTUA, Greece				Indoor + roof	Wi-Fi
	PerformLTE	Wireless	UMA, Spain				Indoor ?	4G / LTE
	Smart Santander	Wireless	UC, Spain				outdoor	802.15.4	access to collected data only
	w-iLAB.t	Wireless	imec, Belgium		embedded PCs with sev-	Indoor (utility room + of-	Wi-Fi + 802.15.4 + BT +	bare-metal access to nodes
					eral wireless NICs	fices)	some USRPs (SDR)
	CityLab	Wireless, IoT	imec, Belgium		32 nodes. embedded PCs	Outdoor (city)	Wi-Fi + 802.15.4 + BT +	bare-metal access to nodes
					with several wireless NICs		LORA
	Portable Wireless Testbeds Wireless, IoT	several		embedded PCs with sev-	depends on deployment	Wi-Fi, 802.15.4, spectrum	bare-metal access to nodes
					eral wireless NICs		scanners
	NITOS	Wireless, OpenFlow	CERTH, Greece		embedded PCs with sev-	indoor and outdoor	Wi-Fi, WiMAX, LTE
					eral wireless NICs + some
					USRPs
	FIT IoT-Lab	Not in F4F	SU / Inria / others, France	MSP430 or STM32 micro-	indoor (several deploy-	802.15.4. some BLE
					controllers, Cortex A8	ments, dedicated rooms or
					linux board. some mobile	offices)
					robots
	FIT CortexLab	Not in F4F	SU / Inria / others, France	29 x USRP N2932, Nutaq	indoor (anechoic cham-	any (SDR)
					PicoSDR	ber)
	FIT R2Lab	Not in F4F (yet)	SU / Inria / others, France	Nitos X50, USRP B210,	indoor (anechoic cham-	Wi-Fi, any (SDR)	bare metal access
					USRP N210, USRP 2,	ber)
					and USRP 1, LimeSDR,
					smartphones
	Name & Website	Categories on F4F website	Patron, Country	Type of hardware
	Tengu	Big Data		imec, Belgium		Deployment of Big Data software stacks
	OFELIA I2CAT island OpenFlow		i2CAT, Spain		OpenFlow-enabled switches (NEC IP8800) + end hosts (VMs)
	PL-LAB	Wired		PSNC, Poland		Traffic generator + NetFPGA (as Programmable switches) + routers + some bare-metal servers
	PlanetLab Europe	Wired		UPMC, France	nodes distributed over the Internet
	ExoGENI	OpenFlow		UvA, Netherlands OpenStack cloud
	Virtual Wall	Wired, Cloud		imec, Belgium		Two clusters of nodes with network emulation
	Grid'5000	Wired, OpenFlow, Cloud, Big Data Inria, France		About 30 clusters of nodes, on 8 sites

Table 2 :

 2 Other testbeds

https://www.fed4fire.eu/testbeds/

example: https://en.wikipedia.org/wiki/Comparison_of_text_editors


Portable Online Prediction of Network Utilization Problem

Predict network utilization in near future to enable optimal utilization of spare bandwidth for low-priority asynchronous jobs co-located with an HPC application

Goals

High accuracy, low compute overhead, learn on-the-fly without previous knowledge

Proposed solution

Dynamic sequence-to-sequence recurrent neural networks that learn using a sliding window approach over recent history Evaluate the gain of a tree-based meta-data management INRIA, The Univ. of Tennessee, Exascale Comp. Proj., UC Irvine, Argonne Nat. Lab.

Grid'5000 experiments

Monitor and predict network utilization for two HPC applications at small scale (30 nodes) Easy customization of environment for rapid prototyping and validation of ideas (in particular, custom MPI version with monitoring support) Impact: Early results facilitated by Grid'5000 are promising and motivate larger scale experiments on leadership class machines (Theta@Argonne)

Energy proportionality on hybrid architectures 2

Hybrid computing architectures : low power processors, co processors, GPUs. . . Supporting a "Big, Medium, Little" approach : the right processor at the right time

Damaris

Scalable, asynchronous data storage for large-scale simulations using the HDF5 format The creation of a dedicated working group within the OpenStack community that deals with Fog/Edge challenges (now managed by the foundation with key actors such as ATT, Verizon, CISCO, China mobile etc.) Several presentations / publications (see the DISCOVERY website) France has the main academic actor in the worldwide community (Inria/IMT Atlantique) thanks to the G5K testbed in particular.

A leadership position A strong expertise for experiments related to performance, scalability of OpenStack components (concrete actions with RedHat, ongoing actions with Huawei, etc.)

Reconfiguring resources

s

  (10K+) write on disk at the same time synchronously Problems: 1) I/O jitter, 2) long I/O phase, 3) Blocked simulation during data writing Solution Aggregate data in dedicated cores using shared memory and write asynchronously Grid'5000 used as a testbed Access to many (1024) homogeneous cores Customizable environment and tools Repeat the experiments later with the same environment saved as an image The results show that Damaris can provide a jitter-free and wait-free data storage mechanism G5K helped prepare Damaris for deployment on top supercomputers (Titan, Pangea (Total), Jaguar, Kraken, etc.) Inria Project Lab: Discovery (contd)

efficient, reliable and flexible: 200 nodes deployed in ~5 minutes

	Key numbers: users and publications Grid'5000 organization and governance
	Operating System reconfiguration with Kadeploy: Groupement d'Intérêt Scientifique (GIS)
	Provides a Hardware-as-a-Service cloud infrastructure Year 2010 2011 2012 2013 2014 2015 2016 2017 2018 Director -Frédéric Desprez
	it e A Enable users to deploy their own software stack & get root access Active users 564 553 592 514 528 458 573 Publications 154 141 101 134 106 143 122 Bureau (6 members: FD, LN, PhD & HDR 14 20 9 27 24 30 27 Usage rate 50% 56% 58% 63% 63% 63% 55% 53% 70% default VLAN 600 564 151 127 23 22 1313 active users over the last 3 years Christian Perez, Adrien Lebre, Laurent Lefevre, David Margery) Comité des responsables de sites Technical Director -Lucas Nussbaum institutional and scientific steering Scalable, Avoid network pollution Technical team (8.2 FTE) technical steering 3769 active users since 2003 routing between Grid'5000 sites 2007 publications that benefited from Grid'5000 in our HAL collection 4 Architects committee (6 members) Create custom topologies global VLANs all nodes connected at level 2, no routing SSH gw local, isolated VLAN only accessible through a SSH gateway connected to both networks By reconfiguring VLANS Computer Science: 96%, Mathematics: 2.4%, Physics: 2.4% Conseil de groupement almost no overhead Inria, CNRS, RENATER, CEA, Since 2015: LORIA: 23%, IRISA: 23%, LIG: 19%, LIP: 13%, LS2N: 13%, CRISTAL: 5%, CPU, CDEFI, IMT advisory and LIRMM: 5%, LIP6: 3% (≈ Allistène + RENATER) evaluation bodies
	Conseil scientifique 10 international members	routed VLAN separate level 2 network, reachable through routing	s it e B

Customize networking environment with KaVLAN

Protect the testbed from experiments (Grid/Cloud middlewares)

Lucas Nussbaum SILECS/Grid'5000

V. Villebonnet, G. Da Costa, L. Lefèvre, J.-M. Pierson and P. Stolf. "Big, Medium, Little" : Reaching Energy Proportionality with Heterogeneous Computing Scheduler", Parallel Processing Letters, 25 (3), Sep. 2015 Lucas Nussbaum SILECS/Grid'5000

https://www.grid5000.fr/w/Hardware Lucas Nussbaum SILECS/Grid'5000

https://hal.archives-ouvertes.fr/GRID5000Lucas Nussbaum SILECS/Grid'5000

Toward a resource management system for Fog/Edge infras.

Inria Project Lab: Discovery

Design a resource management system (a.k.a. a cloudkit) for Fog/Edge infrastructures A four year project started in 2015 with Inria, Orange (and initially Renater) Designing from scratch such a system cannot be envisioned (OpenStack 13 Millions of LOCs)

Contributions

Implementation of a complete workflow to evaluate OpenStack WANWide


Aarón Ayllón-Benítez

Fleur Mougin

Manuel Quesada Martinez

Jesualdo Tomás

Fernández Breis

Romain Bourqui

Patricia Thebault

Deciphering functional annotation of multiple gene sets with MOTVIS

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Deciphering functional annotation of multiple gene sets with MOTVIS

A. Ayllón-Benítez 1,2 , F. Mougin 1,2 , M. Quesada-Martínez 3 , J.T. Fernández-Breis 4 , R. Bourqui 1 , P. Thébault

-Context

The advances in omics technologies, such as single-cell, RNA-SEQ or microarray facilitate understanding the link between expression profiling and phenotype.

Statistical approaches or clustering aim at grouping genes according to their expression levels [START_REF] Huang | Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene lists[END_REF] to detect gene signatures and understand the biological processes.

For that, annotating gene sets will be crucial to: Elucidate the biological role of these specific cells. Highlight their specificity.

Problem:

-Managing a large number of annotation terms associated with a gene set is very difficult. -Enrichment methods have been proposed [START_REF] Huang | Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene lists[END_REF] but they show an important pitfall related to redundancy in the results [START_REF] Thebault | Advantages of mixing bioinformatics and visualization approaches for analyzing sRNA-mediated regulatory bacterial networks[END_REF] due to the lack or under-exploitation of semantic relations between terms.

Objective

To illustrate the usefulness of the MOTVIS workflow to decipher the main roles in immune cell signatures

-Workflow

We proposed three main steps to compute gene sets annotations before presenting the visualization system [3]: 1. Gene sets are annotated using an enrichment approach. g:Profiler has been chosen because it uses several annotation databases. This enables to combine complementary knowledge for enriching functional information about gene sets. 2. Lexical analysis is done to infer relations between terms coming from different sources to eliminate redundant terms. The OntoEnrich framework [START_REF] Ínez | Ontoenrich: A platform for the lexical analysis of ontologies[END_REF] has been integrated for associating annotation terms with GO terms according to the following strategy: Decomposing annotations into words Searching groups of consecutive words that correspond to a GO term or its synonyms Removing words included in other ones. 3. Because of the large size of GO, we selected only the most relevant terms that synthesize the functional information of the input gene sets. The most informative parent terms of each GO term found at the previous step are recursively processed until the root term is reached.

-Case study

To demonstrate the efficiency and reproducibility of the pipeline, the signature profiling of different types of cells has been analyzed using the data from The immunome compendium of immune cell subpopulations [START_REF] Bindea | Spatiotemporal dynamics of intratumoral immune cells reveal the immune landscape in human cancer[END_REF]. They isolated 28 subpopulations of innate and adaptive immune cells, including normal mucosa and colon cancer cell lines. Each cell type presents different transcriptional profiles that can be considered as gene sets.

323 annotations for 24 gene sets using a hierarchical filter proposed in the tool. 98 annotations would have been obtained for only 16 gene sets.

After using the lexical mapping, 264 out of the 323 annotations were kept. 5 out of the 24 gene sets were ignored by our pipeline. Hierarchy simplification decreased the number of annotations from 264 to 119.


Louis Lecrosnier

Rémi Boutteau

Pascal Vasseur

Xavier Savatier

Friedrich Fraundorfer

Camera pose estimation based on PnL with a known vertical direction

Keywords: Computer Vision for Other Robotic Applications, Sensor Fusion, Localization I. INTRODUCTION

In this paper, we address the problem of camera pose estimation using 2D and 3D line features, also known as PnL (Perspective-n-Line) with a known vertical direction.

) we propose a new efficient RANdom SAmple Consensus (RANSAC) scheme for both feature pairing and outliers rejection based solely on rotation estimation from 2 line pairs. This greatly diminishes the computational cost compared to a RANSAC3 or RANSAC4 scheme.

We evaluate our algorithms on synthetic data and on our own real dataset. Experimental results show state of the art results in term of accuracy and runtime, when facing 2D noise, 3D noise and vertical direction sensor noise.

3

possible combinations for 3 pairs of lines and only 2. C m 2 . C n 2 combinations for 2 pairs of lines, i.e. for a 20 2D and 3D lines dataset, we have 7,797,600 possible line combinations for 3 pairs of lines and 72,200 for 2 pairs of lines. We easily understand that reducing from 3 to 2 the number of pairs required in the pairing process, and thus the number of combinations to evaluate, can greatly improve the speed of the overall process. Feature pairing is computationally expensive. Performing this task online is challenging for embedded CPUs.

Increasing the overall speed of the process could leverage this issue.

Many modern vision-based systems are equipped with relatively cheap and accurate inertial measurement units (IMU), which provide useful information about the system orientation, i.e. give a prior knowledge on two of the three rotations of the camera. This vertical direction vector, or up-vector, can be accurate from 0.5°for the cheapest embedded sensors to 0.02°for less affordable sensors [START_REF] Albl | Rolling shutter absolute pose problem with known vertical direction[END_REF]. Some PnP and PnL methods rely on the provided up-vector to reduce the number of potential pose solutions, to reduce the general problem complexity in case of minimal formulation, or to increase performance [START_REF] Albl | Rolling shutter absolute pose problem with known vertical direction[END_REF] [START_REF] Lee | Relative pose estimation for a multi-camera system with known vertical direction[END_REF] [10] [START_REF] Saurer | Homography based egomotion estimation with a common direction[END_REF].

In this paper, our first contribution is a Linear-formulationbased PnL with a known vertical direction. Our formulation relies on the Plücker representation of lines inspired by [START_REF] Bartoli | The 3d line motion matrix and alignment of line reconstructions[END_REF] and [START_REF] Přibyl | Camera pose estimation from lines using plucker coordinates[END_REF], but we benefit the known up-vector of the system to reduce the PnL problem from 6 DoF to 4 DoF, and to a unique solution. We also propose a pose refinement process based on a Gauss-Newton method to sequentially optimize the estimated rotation and translation.

Our second contribution is a new RANSAC2 algorithm capable of either line-pairing or outliers rejection by simply changing its inputs. With our PnL formulation, the translation estimation requires at least three lines to be solved but the remaining rotation R can be computed with only 2 pairs of lines. Thereby, we present a RANSAC2 scheme that is solely based on the rotation estimation. Our approach doesn't rely on a prior pairing, and takes advantage of our PnL formulation to proceed simultaneously and efficiently to feature-pairing and pose estimation.

We bring forward a study of the robustness of our PnL formulation and our outliers removal module with both synthetic and real data.

II. RELATED WORK

All PnL methods formulate the pose estimation problem as a set of linear or polynomial equations and minimize algebraic or geometric error (such as reprojection error).

[1] and [START_REF] Chen | Pose determination from line-to-plane correspondences: existence condition and closed-form solutions[END_REF] introduced PnL research in 1989. They formulated the problem with a minimum of three 2D-3D line correspondences, known as P3L. [START_REF] Dhome | Determination of the attitude of 3d objects from a single perspective view[END_REF] were the first to propose a closed-form solution to the PnL problem with a polynomial approach. [START_REF] Chen | Pose determination from line-to-plane correspondences: existence condition and closed-form solutions[END_REF] stated that in the general case, at least 8 solutions exist for the PnL problem.

A decade later, [START_REF] Ansar | Linear pose estimation from points or lines[END_REF] reduced the number of solutions using 4 or more line-correspondences, and employed polynomial lifting to convert a polynomial system to a linear system of the elements from the rotation matrix. [START_REF] Mirzaei | Globally optimal pose estimation from line correspondences[END_REF] proposed more recently a PnL solution using a minimum of 3 linecorrespondences, more robust to noise, but that spans 23 solutions. [START_REF] Zhang | Robust camera pose estimation from unknown or known line correspondences[END_REF] presented the RLPnL algorithm, requiring at least 4 line correspondences. This algorithm uses subsets of 3 line correspondences and identifies a solution in the derivatives of a 16 th order polynomial. [START_REF] Xu | Pose estimation from line correspondences: A complete analysis and a series of solutions[END_REF] proposed a modified RLPnL method named ASPnL (Accurate Subset based PnL), that is more efficient on small noisy datasets, but is very sensitive to outliers. ASPnL estimates the complete pose in homogeneous coordinates, and includes a pose refinement module based on a Gauss-Newton optimization.

[5] [START_REF] Přibyl | Absolute pose estimation from line correspondences using direct linear transformation[END_REF] proposed a PnL algorithm based on the DLT (Direct Linear Transform) algorithm of [START_REF] Hartley | Multiple view geometry in computer vision[END_REF], using the Plücker line representation. While we use the same coordinates parameterization, we solve the PnL problem differently. They recover rotation and translation by solving a homogeneous system with singular value decomposition, where we rely on a linear least squares solver. Their method is efficient and accurate with a high number of lines (up to a few thousands), but is inaccurate for small datasets [START_REF] Xu | Pose estimation from line correspondences: A complete analysis and a series of solutions[END_REF].

On the topic of PnL with a known vertical direction (or upvector), [START_REF] Horanyi | Multiview absolute pose using 3d-2d perspective line correspondences and vertical direction[END_REF] proposed two PnL algorithms relying on modified Plücker coordinates, known as NPnLUpL and NPnLUpC (Ncamera PnL Up-vector Linear and Cubic). The first one solves the orientation and position from a single system of linear equations producing a unique solution, while the later recovers the pose from a cubic polynomial, thus yielding 3 solutions. To perform accurately these algorithms require a calibrated multicamera setup, and known correspondences between 2D-lines. These assumptions are not suitable for a lightweight robotic application (eg. UAV with a monocular camera).

Several outliers rejection schemes exist within the litterature. A first category is based on the PnP outliers rejection scheme by [START_REF] Ferraz | Very fast solution to the pnp problem with algebraic outlier rejection[END_REF], and tends to minimize an algebraic error [6] [17]. These methods struggle with small and noisy datasets, but handle efficiently large datasets.

On the other hand, RANSAC methods using 3 or 4 line correspondences are more efficient with smaller datasets, but tend to have an unsuitable runtime for real-time applications with an increasing number of lines in the dataset [START_REF] Xu | Pose estimation from line correspondences: A complete analysis and a series of solutions[END_REF].

Diminishing the number of line correspondences required for the pairing process is necessary to reduce the runtime of the latter methods. For this reason, we address in this paper

III. THE PERSPECTIVE-N-LINE PROBLEM

We assume in this section that we have a Z-axis-facing calibrated pinhole camera, with an associated IMU (inertial measurement unit) giving us the vertical direction of the camera (i.e. two of the three rotations are known).

A. 3D Line parameterization

As seen in Fig. 1, Plücker coordinates define a 3D line L in the world frame {W} from its direction V and a plane U passing through the line and the origin, in the form of a homogeneous 6-vector. Given two distinct 3D points in homogeneous coordinates A = (a 1 , a 2 , a 3 , a 4 ) T and B = (b 1 , b 2 , b 3 , b 4 ) T , we define the homogeneous 6-vector L = (U T , V T ) T such as:

L = (L 1 , L 2 , L 3 , L 4 , L 5 , L 6 ) T (1)
where

U = (a 1 , a 2 , a 3 ) × (b 1 , b 2 , b 3 ) V = a 4 .(b 1 , b 2 , b 3 ) -b 4 .(a 1 , a 2 , a 3 ), (2)
and '×' being the cross-product. It is important to note that L must satisfy the bilinear constraint U T .V = 0. Let L W be the expression of L in the world frame {W }. L W can be expressed in the camera frame {C} with the line motion matrix M 6×6 as:

L C = M.L W (3) with M = R -R.T [×] 0 3×3 R . (4)
'[×]' represents the skew symmetric matrix of a 3-vector. T = (t x ,t y ,t z ) T is a translation vector and R is a 3 × 3 rotation matrix.

B. Line projection

The projection of the 3D line L onto the image plane Π is referred as l. l is defined by the intersection of Π and the projection plane u passing through the 3D line L and the origin of the camera frame {C}. For this reason, the 3-vector u is sufficient to fully represent l. Let l C be the expression of l in the camera frame. We obtain l C by projecting L W with the upper part of the line motion matrix M from equation (4), which is a projection matrix P, with

P = R -R.T [×] (5)
and

l C ∼ P.L W , (6)
where l C = (l 1 , l 2 , l 3 ) T . We use here the same parameterization as [START_REF] Přibyl | Camera pose estimation from lines using plucker coordinates[END_REF], derived from [START_REF] Bartoli | The 3d line motion matrix and alignment of line reconstructions[END_REF]. This framework provides a linear projection, that can easily be used as input for a linear least squares pose solver. Note that the equality ( 6) is true up to a non-zero scale factor.

C. Pose estimation with a known vertical direction

Let l i = (l i1 , l i2 , l i3 ) T be the 2D projection of the 3D line

L i = (L i1 , L i2 , L i3 , L i4 , L i5 , L i6
) T , l i and L i being the i th element of a n pairs of lines dataset. We have L i = (U T i , V T i ) T . Solving the complete P3L problem requires estimating three rotations ρ, θ , ψ and three translations t x , t y , and t z . IMU sensors provide a prior knowledge on two out of the three rotations, which reduces the problem to a 4 DoF problem.

By definition, L C i lies on the projection plane of the 2D line l i , leading to the constraint

(R CW .V W i ) T .l C i = 0, (7)
where R CW is the rotation matrix composed with the remaining rotation to be determined, such as

R CW = R z .R y .R x =   c z -s z 0 s z c z 0 0 0 1   .   c y 0 s y 0 1 0 -s y 0 c y   .   1 0 0 0 c x -s x 0 s x c x   (8) where c x = cos(ρ) , s x = sin(ρ), c y = cos(θ ), s y = sin(θ ), c z = cos(ψ), s z = sin(ψ).
Prior knowledge on the vertical direction provides the values for ρ and θ , i.e. provides the product matrix R y .R x , meaning that only two unknowns c z and s z remain. For two line correspondences, we obtain a linear system that can be solved for ψ, with a unique solution, through a linear least squares solver.

In a noiseless case, we directly obtain R CW . However, when proceeding with real data, we often obtain a badly scaled rotation matrix R, that requires further refinement. Having recovered c z and s z , we can recompose a matrix R with equation [START_REF] Albl | Rolling shutter absolute pose problem with known vertical direction[END_REF]. Since c z and s z are estimated separately, they might not satisfy the trigonometric constraint c 2 z + s 2 z = 1. We enforce this constraint with a singular value decomposition of the recomposed R matrix, such as

U.Σ.V * = R R CW = U.V * T (9)
In order to estimate the translation vector T CW , we reformulate (6) into

l C [×] .P.L W = 0 3×1 . ( 10
)
For a 2D/3D line correspondence, we obtain from (10) a system of three equations. However, for only one line correspondence, this system is rank deficient [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. We need to stack the equations from at least three 2D/3D line correspondences to obtain a rank 3 linear system and recover the three remaining unknowns t x , t y and t z . We rearrange this linear system such as

M.T CW = N, (11)
each column of M being respectively expressed in term of t x , t y and t z . T CW is the translation matrix such as T CW = (t x ,t y ,t z ) T , and N contains all terms independent from t x , t y and t z . We finally solve (11) using a linear least squares method, i.e.

T CW = (M T .M) -1 M T .N. (12)
We refer to this method as VPnL LS (Vertical Perspective-n-Lines).

D. Gauss-Newton optimization for the rotation and the translation

In our PnL formulation, the rotation and translation are sequentially estimated. We propose rotation and translation refinement methods that can also be sequentially applied to the estimated pose.

a) Rotation optimization: Our PnL method relies on the known up-vector to estimate the complete pose. However, the up-vector is susceptible to sensor error. For this reason, we propose a rotation optimization scheme to refine the estimated camera orientation. Let A W i and B W i be two 3D points in the world frame defining the i th 3D line L i of a n line dataset. Let l i be the projection of L i onto the image plane. V i W and V i

C

are respectively the 3D line directions in the world and camera frames. For this set of n lines, we express the constraint [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] as a set of n functions f = ( f 1 , ..., f n ) depending on the three rotations ρ, θ and ψ defined as in [START_REF] Albl | Rolling shutter absolute pose problem with known vertical direction[END_REF], so that

f i (R CW ) = (R CW .V W i ) T .l i = 0. ( 13
)
We derive all f i equations into a jacobian matrix J R such as

J R =      ∂ f 1 ∂ ρ ∂ f 1 ∂ θ ∂ f 1 ∂ ψ ... ... ... ∂ f n ∂ ρ ∂ f n ∂ θ ∂ f n ∂ ψ      . ( 14
)
We then iteratively refine the rotation from a step s to the next step s + 1 with

β β β s+1 = β β β s -(J T R .J R ) -1 .J T R .f, (15)
and

β β β = [ρ, θ , ψ] T .
b) Translation optimization: Any 3D point A i lying on the 3D line L i can be projected onto the 2D line l i . We express this constraint similarly to [START_REF] Chen | Pose determination from line-to-plane correspondences: existence condition and closed-form solutions[END_REF] in a set of n functions g = (g 1 , ..., g n ) to refine the translation estimation in

g i (T CW ) = (R CW .A W i + T CW ) T .l i = 0. ( 16
)
We also express the jacobian matrix of ( 16), which we use as in [START_REF] Mirzaei | Globally optimal pose estimation from line correspondences[END_REF] to optimize the translation. When coupled to our PnL algorithm, these two optimizations are referred as VPnL GN.

Because of the non-linear nature of the functions f and g, the Gauss-Newton algorithm does not guarantee convergence to the global minimum with a random initialization [START_REF] Mascarenhas | The divergence of the bfgs and gauss newton methods[END_REF]. However, convergence to optimal parameters can be achieved with an initialization close to the global minimum. We observed that both our methods generally converge within 2 iterations, when initialized with our linear least square solver VPnL LS. For this reason, we stop the algorithm when the refined parameters reach a stationary value over two consecutive iterations, i.e. convergence is attained, or when a maximum of 20 iterations is reached, i.e. the parameters are extremely close to a local minimum and evolution becomes almost stationary. N : maximum number of iteration 7:

E. Line pairing and outliers rejection with RANSAC2

x : size of each set of data between each quantile ε ε ε : error vector such as l i T .R CW .V j 13:

ε : first quantile of ε ε ε 14:

Algorithm :

15:

do 16:
Random sample two 2D-3D pairs 17:

Estimate R CW with a linear least square solver 18:

Measure

ε ε ε = l i T .R CW .V j , ∀ i = 1..n, j = 1..m 19:
Split the error vector ε ε ε into m.n x + 1 quantiles 20:

Extract ε 21:

while ε > criterion AND iterations counter < N

22:

Return inliers for ε ε ε i < criterion Algorithm 1 presents the line pairing process with outliers rejection. First, a random sample of two 2D-3D pairs of lines is used to estimate a rotation matrix R from the world frame to the camera frame. An error vector ε ε ε is then computed for all possible pairs of lines. We split ε ε ε into (m.n)/(x + 1) quantiles, m.n being the number of unique 2D/3D line combinations and x number of lines separated by each quantile. We observe the value of the first quantile of the error vector. A low value for the first quantile of the error vector implies that a subset of x lines generated a low error according to our error function. In this case, we can assume that the estimated rotation is correct, and that the line producing an error value under our threshold are inliers. If not, we iterate until this condition is met, or the maximum authorized number iterations is obtained. In the latter case, we still return the pairs that generated the smallest error over all iterations. Note here that the number of iterations N is defined as

N = log(1 -p) log(1 -(1 -ε) s ) , ( 17
)
where p is the probability of selecting a random sample containing only inliers (usually set to 0.99), ε the percentage of outliers in the data, and s the number of data required for the algorithm to operate (i.e. s = 2 here).

Regarding the number of quantiles (m.n)/(x + 1), experiments suggest that x should be set close to 35% the theoretical number of inliers in the dataset.

IV. RESULTS

A. Synthetic data

In this section, we evaluate quantitatively the performance and robustness of the different steps of our method.

a) Synthetic dataset setup: For the entire synthetic dataset, we simulate a monocular perspective camera setup with a 640×480 pixel resolution, a focal length of 655 pixels, and no radial and tangential distortion.

We generate a set of random 2D lines, each being at least 70 pixels long and defined initially by two endpoints on the image plane. These lines are given in the camera frame {C}. We then express these lines in metric coordinates, and add a random depth from 1 to 3 meters to each endpoint. Finally we express these 3D points in the world frame {W } by applying the transform Tr CW = [R CW (ρ, θ , ψ)|T CW ] with ρ, θ , ψ being random rotations between 0 rad and 2π rad, and T CW being a random translation between 0 to 5m around the camera optical center.

We evaluate the robustness of our method to 2D and 3D noises. A normally distributed noise is added on both 2D endpoints of the lines so that each line endpoint is displaced of a given distance σ in pixels. This modifies both the position and direction of the line on the image plane. We add a gaussian 3D noise to the endpoints of the 3D lines so that the displacement of each 3D endpoint caused by the noise has a σ 3D mm norm.

Unless otherwise specified, all tests are run with 20 2D-3D correspondences, and we run 2000 tests per noise step. Our formulation requires at least three pairs of lines to work, but is compatible with a higher number. For this reason, we evaluate the impact of the number of lines on our algorithms.

We refer to different metrics to evaluate our algorithms. Since the vertical direction is assumed known, the rotation error refers to the absolute angular difference between the estimated angle ψ est and the initially applied angle ψ. The translation error is the Euclidean distance between the estimated location of the camera and the real camera location divided by the norm of the original translation vector, and is expressed in percent.

In the synthetic dataset, 2D and 3D lines are expressed from points. For this reason, we can express the reprojection error as the mean Euclidean distance between all 2D points in pixel coordinates and the corresponding 3D points projected on the image plane using the estimated pose.

For the pairing algorithm, we include the recall rate, i.e. the number of inliers recovered over the total number of inliers. We also show the precision rate, i.e. the number of inliers recovered over the number of pairs recovered. Both these metrics are evaluated with an increasing outliers rate.

b) Linear PnL with Plücker coordinates formulation: We begin our tests with the pose estimation method. We present the robustness to 2D and 3D gaussian noises in Fig. 2, and we evaluate the impact of the number of lines as input in Fig. 3.

We compare both our algorithms with the ASPnL algorithm from [START_REF] Xu | Pose estimation from line correspondences: A complete analysis and a series of solutions[END_REF], NPnLUpL and NPnLUpC from [START_REF] Horanyi | Multiview absolute pose using 3d-2d perspective line correspondences and vertical direction[END_REF]. While the two latter are designed for multiview pose estimation, they accept a single view configuration as input. ASPnL estimates the 6 DoF pose, and both our algorithm and NPnLUpL consider a 4 DoF problem. For a fair comparison between these algorithms, we introduce a 0.5°constant noise on one of the two rotations of the up-vector, simulating this way a common noise from a low-cost orientation sensor.

In term of rotational error, results in Fig. 2 show that when facing 2D noise, both our algorithm perform similarly. With noise on the up-vector, we outperform NPnLUpL and NPnLUpC for small 2D noises, but our methods tend to similar results facing heavy 2D noises. All these four algorithms outperform ASPnL when confronting a moderate to high 2D noise. Regarding small 3D noise, our two algorithms outperform ASPnL, NPnLUpC and NPnLUpL. From 100mm to higher noise, our VPnL LS is outperformed by NPnLUpC and NPnLUpL, but VPnL GN competes with the two latter.

Regarding translation error, ASPnL performs better for small 2D and 3D noise, because not suffering from errors on the up-vector, but for medium to high noise, our VPnL GN algorithm outperforms the others, and greatly improves the translation error when facing 3D noise.

Fig. 3 presents how the number of lines impacts the pose estimation in presence of noisy 2D or 3D lines. Here, 2D noise on line endpoints is set to 10 pixels and 3D noise is set to 100mm. Noise on the up-vector is set to 0.5 °. We vary the number of lines from 4 to 40. The result shows that our algorithms VPnL LS and VPnL GN systematically outperform all other methods, independently of the dataset size. Note here that NPnLUpC recovers the rotation with a similar accuracy than our two algorithms, but as a median translation error above 100% in our test case, for any dataset size we tested.

We also show that in presence of 2D noise, all methods tend to perform only slightly better when using more than 25 lines, while increasing the computational complexity. In case of 3D noise, we observe a similar phenomenon, apart from ASPnL whose performances decrease when using more than 17 lines. c) RANSAC2 as outliers removal module: We present here the results of our RANSAC2 algorithm used as outliers Fig. 2: Rotation and translation errors in case of 2D and 3D noises. VPnL LS is our PnL formulation with a known vertical direction and a linear least squares solver only. VPnL GN is the same method with Gauss-Newton optimization on the complete pose. ASPnL refers to [START_REF] Xu | Pose estimation from line correspondences: A complete analysis and a series of solutions[END_REF] method, and NPnLUpL is the PnL linear formulation of [START_REF] Horanyi | Multiview absolute pose using 3d-2d perspective line correspondences and vertical direction[END_REF] workin on a single-camera setup. Left column evaluates robustness against 2D noise, right column evaluates robustness against 3D noise. Left column evaluates the impact of the number of lines with 2D line endpoints having a 10 pixel noise. Right columns evaluates the impact of the number of lines with a 100mm 3D noise on all 3D line endpoints.

removal module. We evaluate the recall and precision rates of this outliers rejection module, as well as the runtime. We combine our RANSAC2 algorithm with both our methods VPnL LS and VPnL GN to evaluate the overall runtime. We compare our outliers removal algorithm to RANSAC3, RANSAC4 and RLPNL Enull from [START_REF] Xu | Pose estimation from line correspondences: A complete analysis and a series of solutions[END_REF]. RANSAC3 and RANSAC4 propose a combined outliers rejection and pose estimation scheme. RANSAC3 is P3L-based, and RANSAC4 is ASPnL-based. RLPnL Enull is a notably outlier-resistant pose estimation algorithm.

For these tests, we imposed a 0.5°noise on the up-vector used in our methods, and used a set of 40 lines with a fixed 5 pixels 2D noise and 50mm for the 3D noise. We measured the runtime for each method for a range of outliers from 0% to 60%, and run 2000 tests for each outlier step.

We finally computed the mean runtime over the entire outlier range, 2D and 3D noise for each method, since preliminary results showed that for 40 lines, all methods tend to similar runtimes independently of the outlier rate. For each outlier step, we also computed the mean recall and precision rates. All our runtime results are mesured on a laptop with an Intel Core i5-6440HQ CPU running at 2.6 GHz with a single threaded matlab R2017a script, on an Ubuntu 18.04 operating system.

Runtime results in table I show that even with our rotation and translation optimization enabled, our RANSAC2 algorithm competes with the non-RANSAC RLPnL Enull method with a 3.2 ms runtime, while our RANSAC2+VPnL LS is more than 10 times faster with a 0.21 ms runtime. RANSAC3 and RANSAC4 have a respective runtime of 341.8 ms and 1,666.7 ms, which is significantly slower than our two methods. For a fair comparison, we forced the number or iterations for RANSAC3, RANSAC4 and RANSAC2 to be 10 times the number of pairs of lines set as input, i.e. 400 iterations here.

RANSAC3 and RANSAC4 find solutions to the PnL problem in the roots of an 8 th degree polynomial, which is computationally more expensive than the linear formulation used in both our methods and in RLPnL Enull. This could explain the important runtime difference.

The outliers removal module in RLPnL Enull relies in the analysis of the nullspace of a homogeneous linear system and a Gauss-Newton scheme to find an optimal solution, while our RANSAC2 algorithm simply relies on a linear least squares method to solve a 4-lines linear system in a RANSAC scheme. This explains the runtime difference of the two methods, and also why our RANSAC2 coupled with our Gauss-Newton optimization tends to the same runtime than RLPnL Enull. Fig. 4 shows the recall and precision rate of our RANSAC2 algorithm. We observe a mean recall rate of about 65% in the 2D noise case, and 45% in the 3D noise case. The mean precision rate is 100% in both cases, independently of the outlier rate.

We observe here that while our algorithm can be improved in term of recall rate, we obtain a perfect precision. This means that all inlier propositions returned by our algorithm are valid, and thus lead to a pose estimation with an accuracy that is shown in Fig. 2 and Fig. 3.

B. Experiment on real data a)

Experimental setup: To validate our methods with real data, we created an experimental dataset composed of images of an indoor scaffolding structure and corresponding camera poses in a known 3D environment. The 3D environment is represented in the form of point cloud acquired with a Leica ScanStation C10 LiDAR. Ground truth for the poses is provided with a millimetric accuracy by a Vicon motion capture system, calibrated with the 3D point cloud and the camera. 3D and 2D lines are extracted respectively from the 3D point cloud and the images of our dataset. Because we perform the extraction process manually, we limited our experiment to 6 images acquired from various locations and orientations.

To acquire images with 6 DoF poses, we mounted a calibrated 640×480 pixel Logitech Quickcam 4000 camera on a rigid metallic frame (see Fig. 6 (a)), denoted camera support. We moved and tilted this hand-held camera-setup in the laboratory, around the scaffolding structure seen in Fig. 6(d) and Fig. 7. Note here that the camera support to camera transform Tr CamSupport , as seen in Fig. 5, is initially unknown.

b) Calibration: To calibrate the camera with respect to the Vicon system, we acquire images of a chessboard equipped with Vicon markers (see Fig. 6 (b)) with our camera. We obtain here Tr CamChess and Tr WChess .

For multiple images and associated poses, we can now recompose Tr CamSupport with:

Tr CamSupport = Tr CamChess .Tr WChess -1 .Tr WSupport . ( 18
)
We estimate the rigid transform between the Vicon frame {W } and the Leica frame {L} using known point features available in the two frames. We placed several Vicon markers into the laboratory, and manually selected their correspondences in the point cloud. We obtained two sets of 23 3D points A i and B i (i = 1..n) respectively in Vicon and Leica frame, linked by a rigid transform Tr LW with

Tr LW .A = B (19)
Having calibrated the complete system, we know the complete pose of the camera in the world frame. c) Pose estimation tests and results: For each of the 6 images of our dataset, we estimated the camera pose using manually matched 2D lines from the images, and 3D lignes for the point cloud. Results are shown in table II. Regarding the rotation error, result show that ASPnL has the lowest accuracy with a median error above 5°. All methods perform similarly with an accuracy around 0.2°. This result can be explained, since ASPnL recovers the complete rotation where all the other method considered benefit the up-vector to only recover one rotation.

TABLE II: Mean, median and standard deviation for the translation error (%) and rotation error (°) over 6 poses. 2D and 3D lines are manually extracted and paired. The camera is displaced in a 5m radius around the Vicon frame origin. Regarding the translation error, VPnL LS achieves the best results with a 2.72% median translation error, followed by VPnL GN with 2.78%. NPnLUpL and NPnLUpC are designed for a multi-camera system, which explains a greater translation error, respectively about 3.24% and 5.28%. ASPnL is here the less accurate, with 5.76 % translation error. Reprojection with the parameters estimated with VPnL GN are shown in Fig. 7.

d) Outliers removal module tests and results: To test the robustness against outliers with real data, we select a pose from the 6 poses available in our dataset and randomly introduce 40% of outliers in the line correspondences. For a given test, each algorithm is given the same data as input. The selected pose has 25 line correspondences and the camera is placed 4.3m far from the Vicon frame origin. We evaluate the rotation and translation errors, as well as the runtime, for RANSAC3, RANSAC4, RLPnL Enull, RANSAC2+VPnL LS and RANSAC2+VPnL GN. Results shown in table III are median values for 1000 tests. The TABLE III: Outliers rejection and pose estimation test on real data. We show the median, mean and standard deviation for the translation error (%), rotation error (°) and runtime (ms). The camera is placed 4.3m far from the Vicon frame origin and tilted. For each test, 40% of the 26 line correspondences are shuffled to introduce outliers. The result shows that when coupled, our outliers rejection and pose estimation algorithms successfully recover a valid pose in most of the cases, where RANSAC3+P3L and RANSAC4+ASPnL fail. We note that RLPnL Enull performs better than RANSAC3 and RANSAC4 methods for a much smaller runtime, but is still greatly outperformed by our methods in term accuracy. We observe a runtime respectively two and three orders of magnitude faster when comparing VPnL GN to RANSAC3 and RANSAC4, when RLPnL Enull has a runtime comparable to both our methods.

Note that for this pose, VPnL GN does not perform better than VPnL LS, because we are in presence of low 2D and 3D noises combined with noise on the up-vector. This is consistent with the simulations results shown in Fig. 2.

Our experiment validates the simulation results, and proves the advantages of a known vertical direction for outliers rejection and pose estimation, in term of accuracy and runtime.

V. CONCLUSION

In this paper, we first present a line-based pose estimation scheme with known vertical direction. Our algorithm follows a Plücker coordinates formulation, and a unique solution is estimated with a linear least squares solver. We derive a rotation and translation optimization scheme based on the Gauss-Newton algorithm. We also present a pairing and outliers removal module based on the RANSAC2 algorithm, relying only on the rotation estimation.

Our PnL formulation is well suited for small and noisy datasets, especially in mobile robotics, where the vertical direction is often known from integrated sensors (IMU). However, the use of a linear least squares solver involves a low robustness to outliers. Our outliers removal module achieves a 100% precision rate, up to 60% of outliers, in presence of 2D and 3D noise. Its very low runtime enables its use for real-time robotic applications or as a pairing module.

Fig. 1 :

 1 Fig. 1: The 3D line L is parameterized by its direction V and a normal U intersecting L and the origin of the world frame {W}. l is the projection of L onto the image plane Π in the camera frame {C}.

  normalized 2D line, i = 1..n 4: V j : normalized direction of the 3D line, j = 1..m 5: criterion : loop break threshold 6:

  : Rotation matrix World → Camera 12:

Fig. 3 :

 3 Fig.3: Impact of the number of lines on the rotation and translation errors for 2D and 3D noise. Comparison between ASPnL, VPnL LS, VPnL GN, NPnLUpL and NPnLUpC. Left column evaluates the impact of the number of lines with 2D line endpoints having a 10 pixel noise. Right columns evaluates the impact of the number of lines with a 100mm 3D noise on all 3D line endpoints.

Fig. 4 :

 4 Fig. 4: Evaluation of RANSAC2 as outliers removal module. Recall and precision rates with 5 pixels 2D noise and 50mm 3D noise. Constant 0.5°noise on the up-vector. 2000 tests per outlier step.

Fig. 5 :

 5 Fig. 5: Experimental setup for our dataset. The camera and camera-support are rigidly attached, but only the camera support pose is initially known in the Vicon frame. Ground truth for the camera pose is provided by a calibration step.

Fig. 6 :Fig. 7 :

 67 Fig. 6: a. camera setup equipped with Vicon markers, b. calibration target template, c. point cloud with extracted 3D line endpoints, d. image with extracted 2D lines.

TABLE I :

 I Mean runtime for combined outliers rejection and pose estimation methods. For the RANSAC-based methods, the maximal number of iterations is fixed to 400. Results are calculated for 2000 tests per outlier step.

			100							100
			90					Recall rate, Precision rate,	=5p, =5p,	90				Recall rate, Precision rate,	3D =50mm, 3D =50mm,
			80							80
		Recall and Precision rate (%)	30 40 50 60 70						Recall and Precision rate (%)	30 40 50 60 70
			20							20
			10							10
			0							0
			0	10	20	30	40	50	60	0	10	20	30	40	50	60
						Outlier rate (%)							Outlier rate (%)
	Method	Runtime(ms)
	RANSAC2 (ours) + VPnL LS	0.21
	RANSAC2 (ours) + VPnL GN	3.25
	RLPnL Enull	2.90
	RANSAC3	341.80
	RANSAC4	1,666.7

ACKNOWLEDGMENT

The authors would like to thank Romain Rossi, Pierre Merriaux and Sophie Ladet for helpful discussions.

This work was carried out as part of the COPTER research project, and is co-funded by the European Union and the Région Normandie. Europe is involved in Normandy with the European Regional Development Fund (ERDF).