domain: info spi sdv

text:
Vladimir A Baulin
email: vladimir.baulin@urv.cat

Yves Usson

Xavier Le Guével

Deep learning: step forward to high-resolution in vivo shortwave infrared imaging

In vivo optical imaging is a fast growing field that offers great perspectives for biomedical applications. In particular, imaging in the shortwave infrared window (SWIR: 1000-1700 nm) represents major improvement compared to the NIR-I region (700-900 nm) in terms of temporal and spatial resolutions in depths down to 4 mm. SWIR is a fast and cheap alternative to more precise methods such as X-ray and optoacoustic imaging. Main obstacles in SWIR imaging are the noise and scattering from tissues and skin that reduce the precision of the method. We demonstrate the combination of SWIR in vivo imaging in the NIR-IIb region (1500-1700 nm) with advanced deep learning image analysis allows to overcome these obstacles and making a large step forward to high resolution imaging: it allows to precisely segment vessels from tissues and noise, provides morphological structure of the vessels network, with learned 1 pseudo-3D shape, their relative position, dynamic information of blood vascularization in depth in small animals and distinguish the vessels types: artieries and veins. For demonstration we use neural the network IterNet that exploits structural redundancy of the blood vessels (L. Li, et.al., The IEEE WACV, 2020), which provides a useful analysis tool for raw SWIR images.

Introduction

The field of in vivo optical imaging for biomedical applications is expanding rapidly over the last two decades leading to more precise diagnostic of early stage diseases and to advanced image-guided-surgery system already available for clinical applications. [START_REF] Cao | Recent Progress in NIR-II Contrast Agent for Biological Imaging[END_REF] One of these breakthroughs is related to the development of innovative imaging systems in the shortwave infrared (SWIR) spectral window, called also NIR-II, between 900 and 1700 nm. SWIR has demonstrated a major improvement in terms of spatial and temporal resolution, reaching deep in tissue down to 4 to 6 mm compared to the Visible (400-700 nm) and NIR-I (700-900 nm) regions. The benefit moving forward from NIR-I to SWIR is mainly associated to the weak auto-fluorescence and reduced scattering from the living tissues at longer wavelengths. [START_REF] Hong | Near-infrared fluorophores for biomedical imaging[END_REF] For instance, it was shown recently the striking improvement of detection with higher signal-to-noise ratio selecting the SWIR sub-windows NIR-IIb (1500-1700 nm) for in vivo imaging. [START_REF] Zhong | In vivo molecular imaging for immunotherapy using ultra-bright nearinfrared-IIb rare-earth nanoparticles[END_REF][START_REF] Wang | Fluorescence molecular tomography in the second near-infrared window[END_REF][START_REF] Musnier | Optimization of spatial resolution and scattering effects for biomedical fluorescence imaging by using sub-regions of the shortwave infrared spectrum[END_REF] The concomitant progress of the sensor technology in the SWIR range and of the formulation of new bright and biocompatible SWIR emitting organic and inorganic contrast agents 6-9 has enabled to use these optical systems for intra-operative surgery in small animals [START_REF] Zhu | Repurposing Cyanine NIR-I Dyes Accelerates Clinical Translation of Near-Infrared-II (NIR-II) Bioimaging[END_REF][START_REF] Du | Highly Stable and Bright NIR-II AIE Dots for Intraoperative Identification of Ureter[END_REF] and recently in human. [START_REF] Hu | First-in-human liver-tumour surgery guided by multispectral fluorescence imaging in the visible and near-infrared-I/II windows[END_REF] One of the most appealing field of applications for SWIR imaging concerns the (micro)vascularization, with success to monitor in real time non-invasively different pathologies such as vascular disorders, (neo)angegiogenosis in cancer, wound healing, implants. [START_REF] Zhu | Near-Infrared-II Molecular Dyes for Cancer Imaging and Surgery[END_REF][START_REF] Saif | Non-invasive monitoring of chronic liver disease via near-infrared and shortwave-infrared imaging of endogenous lipofuscin[END_REF][START_REF] Wang | Light-sheet microscopy in the near-infrared II window[END_REF][START_REF] Ma | Near-Infrared IIb Fluorescence Imaging of Vascular Regeneration with Dynamic Tissue Perfusion Measurement and High Spatial Resolution[END_REF] Despite these major steps, we are still far to reach the spatial resolution below one micron at high depth achieved by X-ray imaging. [START_REF] Wang | Sensitive imaging of intact microvessels in vivo with synchrotron radiation[END_REF] Other recent optical imaging systems based on full field optical coherence [START_REF] Mazlin | In vivo high resolution human corneal imaging using full-field optical coherence tomography[END_REF] and high-resolution opto-acoustic imaging [START_REF] Deán-Ben | Advanced optoacoustic methods for multiscale imaging of in vivo dynamics[END_REF] lead to spatial resolution down to 1.7 um but with a quite limit field of view that requires long time acquisition to image the whole animal. A promising strategy to overcome this issue relies on the image analysis using deep neural networks. This field shows exponential growth in recent years due to growth in computational power of modern parallel computers, and the quality of feature extraction, detection and segmentation made by deep neural networks has made a major step forward. In particular, several networks built on a popular fully connected convolutional neural network (CNN) U-Net, [START_REF] Ronneberger | Convolutional Networks for Biomedical Image Segmentation[END_REF] were very successfully applied in the context of a classical problem of the segmentation of retina vessels using as a training set a series of open annotated databases for retina vessels. As a result, several developed deep neural networks from the leaderboard show the performance above 97% precision tested against ground truth. Thus, a logical extension of these networks would be the application of these developments for SWIR images. Although the nature of the images is different: shadows of the visible light passing through the tissue versus fluorescent NIR signal directly from the vessels, the structure of the vessels representing continuous interconnected lines with a certain redundancy enables the generalization of the developed networks to the case of SWIR images. It allows segment the vessels from the background, reduce scattering light originated from the tissues, and detect 3D blood vessels structures, thus providing essential information for a full structural analysis via skeletonization of the vessel network and enhanced statistics: number of branching points, average length of the vessels, the thickness of the vessels, relative length of vessels of different categories, etc. Thus, combination of fast and relatively cheap SWIR method with advanced deep learning image analysis may contribute to fill the gap in resolutions between SWIR and X-ray.

Recent study has demonstrated the significant improvement of contrast and spatial resolution of SWIR in mice using Monte Carlo Restoration which enabled to perform segmentation analysis of small animal presenting vascular disorder. [START_REF] Yu | High-Resolution Shortwave Infrared Imaging of Vascular Disorders Using Gold Nanoclusters[END_REF] In this work we explore the advantages of using deep neural networks specially designed to extract blood vessels structure trained on large datasets of retina vessels in order to predict the structure of vessels in SWIR images. We use one of the best neural networks in prediction of vessels structures IterNet [START_REF] Li | IterNet: Retinal Image Segmentation Utilizing Structural Redundancy in Vessel Networks[END_REF] on in vivo SWIR NIR-IIb imaging to demonstrate the high potential of this method to go one-step further to high-resolution optical imaging that could be easily transferred in clinics and hospitals.

Deep Neural network for SWIR images

Fully connected convolutional neural network is a popular neural network U-Net 19 using strong data augmentation to significantly reduce the number of training images. Its successor, deep neural network IterNet 21 is build on U-Net and combines iteratively N -1 mini-networks U-Net after one segmentation with U-Net. It goes further in precision and uses the structural redundancy or self-similarity of blood vessels that allows the network to find obscured details of the vessel from the segmented vessel image itself, rather than from the raw input image. In fact, IterNet can learn from as few as 10-20 annotated images (ground truth) to provide a good accuracy. To test the performance of the network for in vivo SWIR images of nude mice, we have performed several setups with different cameras, lenses and different distances from the sample. The quality of prediction was then tested on ex vivo post-mortem sample with removed skin. This allows to evaluate the structure of vessels with high magnification camera in order to confirm the prediction of microvessels.

SWIR in vivo images

SWIR imaging was performed using a Princeton camera 640ST (900-1700 nm) coupled with a laser excitation source at λ = 808 nm (100 mW/cm 2 ). We used a short-pass excitation filter at 1000 nm (Thorlabs) and a long-pass filter on the SWIR camera from Thorlabs (LP1500 nm). 25 mm or 50 mm lenses with numerical aperture (n.a) = 1.4 (Navitar) were used to focus on the mice placed at 30 cm working distance. 25 mm and 50 mm lenses provide a theoretical spatial resolution of 400 microns and 150 microns respectively. NMRI nude mice (Janvier, France) were anesthetized (air/isofluorane 4% for induction and 1.5% thereafter) and were injected intravenously via the tail vein (200 µL of Indocyanine Green (ICG) at 500 µM in PBS). In vivo SWIR imaging was performed using 25 mm or 50 mm lenses and LP1500 nm at different exposure times from 100 ms to 1 s.

Training of the neural network

We use IterNet [START_REF] Li | IterNet: Retinal Image Segmentation Utilizing Structural Redundancy in Vessel Networks[END_REF] for prediction of vessels in SWIR images (i) with the released by the authors universal pre-trained weights trained across multiple datasets. [START_REF] Li | IterNet: Retinal Image Segmentation Utilizing Structural Redundancy in Vessel Networks[END_REF] They were used without additional fine-tuning; (ii) weights from specially trained network on manually annotated SWIR images used as a ground truth for detection, example is Fig. 1. These images were added to the training set of DRIVE database, which were gray scaled. The manual annotation of SWIR training examples was not complete and it was done for the test of robustness of the prediction of the network. The training was performed with the following parameters: batch size=32,repeat=10, minimum kernel=32, epochs=200, iteration=3, crop size=128, stride size=3 and the resulting performance is shown in Fig. 1C): An example of the original image, an example of the annotated image and the resulting ROC curve showing the performance on SWIR images. Although the annotated images were not as elaborated as in the vessel databases and despite their small number, the network predictions have 0.9 AOC score on images and manually annotations unseen during training. This result demonstrates that the network reliably predicts the vessels at least comparable with the human annotation results on SWIR images.

We use the universal pre-trained weights of IterNet neural network. SWIR images have 640x512 pixels dimensions and the vessels are smaller than in the original training database. In order to achieve a good accuracy, SWIR images (dimensions of vessels and brightness/contrast) should be as close as possible to the training set used for the network training. SWIR images contain a large number of micro-vessels, almost invisible by eye.

They were not annotated in the training set and in order to segment them the stride size and the crop size should be reduced to minimum. We use the following parameters that give the best results for the inference of the vessels with pre-trained weights: Activation='ReLU', dropout=0.1, minimum kernel=32, batch size=128, epochs=600, iteration=3, stride size=1, crop size=16. The images were processed on a desktop computer equipped with AMD Ryzen 5 5600X processor with 64 GB RAM, Nvidia RTX 3090 graphical card with 24 GB memory, Ubuntu 20.04 and Tensorflow 1.15 [START_REF] Abadi | TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems[END_REF] through Nvidia-TensorFlow Horovod 26 compilation to ensure back compatibility with RTX 3090 graphical card. With this setup, the speed of processing of a SWIR image with all GPU overheads, pre-processing, post-processing and writing the resulting files to the disk is presented in Table 1.

Neural network prediction

The performance and the number of vessels predicted by the network largely depends on the quality of SWIR images, the distance of the camera from the mouse skin and the brightness/noise of the vessels. The example of the predicted vessels is shown in Fig. 2. The network can not only segment vessels from the background, but it can also detect microvessels, predict their correct connectivity and relative position, thus, giving the impression of pseudo-3D vision.

Figs. 2A andB shows the vascular network of the ventral side of a whole mouse after the injection of ICG thanks to the detection of its photoluminescence signal above 1500 nm. It The use a 50 mm lens on the SWIR imaging system allows to reach higher spatial resolution and gain more information on the blood vessels morphology.

Validation of the vessels morphology

Detection of SWIR signal from vessels below 4 mm depth and keeping a high spatial resolution remains highly challenging. In fact, upper skin is still a main obstacles for SWIR signal, where the light can be scattered and adsorbed. Thus, to confirm microvessels structure predicted by the neural network and inferred from the original SWIR images, one of the direct methods would be the removal of the skin post-mortem and validation of small vessels with high magnification optical camera on ex vivo images of the inner skin.

For that purpose, the mice skin flap of 2 to 3 mm thick were soaked in formaldehyde just after the mice were sacrificed. High magnification optical images were taken on the inner side of the flap with an Andor Ikon-M CCD camera at 1 s exposure time with WD 112 mm lens from Leica (zoom x0.8) under white light illumination. Andor camera under the white light (neon light which has a broad excitation). A selected area in the center of the inner skin flap (Fig. 5B) was used for inference of the blood vessels with the same IterNet neural network and the same parameters that were used for SWIR images segmentation. The result of the predicted vessels is shown in Fig. 5C. This image shows main blood vessels and also a number of microvessels. To confirm the existence of these microvessels, three regions of the predicted vessels were selected to compare with separately taken optical images with even higher magnification. The results of this direct comparison are shown in insets with an outline of the same color: red region in Fig. 5C is compared with an optical image Fig. 5D; green region Fig. 5E is compared with optical image Fig. 5F and blue region Fig. 5G is compared with Fig. 5H. It is noteworthy, that optical images that were taken for comparison, are from different images taken on the inner skin sample regions with higher magnification from x1.7 (Fig. 5B) to x15.4 (fig. 5F), while neural network was applied on Fig. 5B.

This indicates a good confidence to use such neural network for processing SWIR images for inference of blood vessels structure and blood vessel mapping.

3 Blood vessels analysis The detection and segmentation of blood vessels are not the only possibilities for the neural network analysis. The propagation of the contrast agent across the blood vessels after the first injection follows non-trivial haemodynamics that can be visualized with an appropriate setup. To demonstrate the capabilities of the method, the initial vascularization of the contrast agent was recorded on the flank of the mouse in the first second after i.v.

injection. Very quickly after injection, the constant increase of photoluminescence signal across the vessels in site with high blood vessel density has converted in overexposed spot, where single vessel cannot be visualized. In contrast, the regions, where the contrast agent still has not arrived, are underexposed and there is not enough signal to visualize the vessels, upper raw, Fig. 6A.

The neural network applied to each of the frames does not pick up the whole structure of the vessels, because this information is not present in each image, Fig. 6B. However, Another important feature that can be extracted by neural network is related to the differentiation between arteries and veins. The distinction between veins and arteries is considered critical in angiogenesis and related to the analysis of the couple veins/arteries. This information can be inferred from the shape and the surrounding of the vessels, e.g. the thick veins are accompanied by thin arteries going parallel to the veins. This information can be learned by the neural network carefully trained on well-annotated examples, where the composition of the vessels is known. This was implemented in the network SeqNet, [START_REF] Li | Joint Learning of Vessel Segmentation and Artery/Vein Classification with Post-processing[END_REF] specially trained to distinguish between veins and arteries.

For this purpose we applied SeqNet with provided pre-trained weights to SWIR images, that were not optimized and were not included in the training set. extraction by open source Python library Skan. [START_REF] Nunez-Iglesias | A new Python library to analyse skeleton images confirms malaria parasite remodelling of the red blood cell membrane skeleton[END_REF] The result is shown in Fig. 8 and the corresponding video is available in SI.

Conclusion

We demonstrated the potential of deep learning applied to the IR optical imaging in general and in the NIR-IIb (1500-1700 nm) region in particular. This analysis allows to (i) segment vessels from the tissues and noise due to scattering and adsorption of light in tissues; (ii) distinguish vessels overlap and junctions; (iii) to get a morphology and depth estimation with a pseudo-3D shape of blood vessels; (iv) distinguish different hierarchies of vessels types, such as veins and arteries; (v) obtain a information about haemodynamics and kinetics of blood flow; and (vi) perform all types of statistics on vessels junctions, connections, branching

Figure 1 :

 1 Figure 1: A) Original SWIR image used for training of the network and obtained with a 50 mm lens (n.a.= 1.4) at 100 ms exposure; B) Example of the annotation of ground truth, microvessels are not annotated; C) Result of the training in form of the receiver of operating characteristic (ROC) curve showing the performance of the training on SWIR images. Area under the curve: 0.90; Area under precision-recall curve: 0.57, Jaccard similarity: 0.89.

Table 1 :Figure 2 :

 12 Figure 2: A) SWIR image of a whole body mouse taken at 30 cm working distance before the injection of the contrast agent without long pass filter; B) few seconds after the injection of the contrast agent; C) The resulting prediction of vessels from the image of the whole body mouse in B and D) zoomed image of the segmented blood vessels structure in C.

Figure 3 :

 3 Figure 3: A fragment of the original SWIR image showing complex topology of the blood vessel network with corresponding zoomed insets. It illustrates the predicted blood vessel structure with junctions and cross-sections of the vessels.

Fig. 3

 3 top raw depicts the ventral side of the mouse (with consecutive insets) with a high density of blood vessels of different sizes up to micrometer resolution and showing a complex vessels topology. Bottom raw demonstrates the corresponding predicted vessels structure obtained by neural network and provides extremely detailed information not accessible from the raw image by naked eye with junctions and overlaps of the vessels as seen in the insets Figs.3B and C. Examining the same segmented image Fig. 3A in inverted colors, Fig. 4 allows to see a complex morphology of blood vessels structure, their relative position, intersection and branching, hence leaving an impression of pseudo-3D shape and depth, as illustrated in the insets of Fig. 4.

Figure 4 :

 4 Figure 4: Predicted vessels structure from SWIR image of the whole body of the mouse, Fig. 2 and two fragments in the insets with the detail of the original inverted SWIR images and the corresponding segmented images.

Figure 5 :

 5 Figure 5: A) An overview of the removed inner skin region; B) Closer view of the removed inner skin region used for inference (zoom x1.7 of image A); C) Predicted vessels structure inferred from B); D) Optical view with higher resolution of the red fragment (zoom x5.6 of image A); E) A green fragment of the predicted vessels; F) The same region in optical image with higher magnification (zoom x15.4 of image A); G) A blue fragment of the predicted vessels; H) The same region in optical image with higher magnification (zoom x8.8 of image A).

Fig. 5A corresponds

  Fig. 5A corresponds to the optical image of the inner skin of the mouse made with the

Figure 6 :

 6 Figure 6: A) Consequent frames (one frame per 500 ms) of the original inverted SWIR image, where the contrast agent propagates through the blood network. B) the corresponding predicted vessels by IterNet network. Vessels in overexposed regions are not detected, thus allowing to follow the blood propagation. C) consequent sum of predicted vessels from individual frames ("SWIR HDR" effect): red corresponds to first frame vessels detection, next frames are superimposed images from previous frames, each with its own color.

Figure 7 :

 7 Figure 7: A) Inverted original SWIR image of a mouse ear after ICG injection and B) predicted vessels structure inferred from A. C) Fragment of the original image A. D) Overlay of predicted vessel structure and the original fragment image c. Arteries (red) and veins (blue) predicted by SeqNet 27 neural network; E) Segmented arteries and veins. Predicted arteries (red) and veins (blue) by SeqNet neural network.

Figure 8 :

 8 Figure 8: Skeletonization of the predicted blood vessels network. A) Predicted blood vessels from Fig. 1A; B) Euclidean skeleton of the vessels, which connects directly the branching points; C) Vessel network overlapped on the original SWIR image.

  Fig.7Ashows the original inverted SWIR image of the ear mice and Fig.7Bshows the corresponding segmented image obtained by SeqNet, which segmentation is essentially the same as IterNet. SeqNet network post-processing enables to predict artery and veins based on their size and their distance between each other with high confidence even for images, which are very different from the training set. The resulting distinction in Fig.7C-Eshows a potential to have a much better differentiation with specially trained networks.The ability to obtain an accurate prediction of the blood vessel network structure by neural networks facilitates further post-analysis of the blood vessels structure tortuosity with more confidence than on the same skeletonization performed on the raw images. This analysis includes extraction of the skeleton of the network and, consequently, deep statistical analysis of network structure: the position and number of branching points, measuring the length of skeleton branches, paths statistics along vessels, density of junctions, vessels, distances between different kinds of structures, comparison of skeletons in order to detect new vessels. As an example, we use a segmentation result of Fig.1Aas an input for skeleton

Acknowledgments XLG would like to thanks Maxime Henry and Benjamin Musnier for the animal experiments and Plan cancer (C18038CS) for their financial support. The Optimal imaging platform is supported by France Life Imaging (French program "Investissement d'Avenir" grant; "Infrastructure d'avenir en Biologie Santé", ANR-11-INBS-0006) and the IBISA French consortium "Infrastructures en Biologie Santé et Agronomie".

Ethics statement

Authors contribution

Computational analysis and methodology: Vladimir Baulin. Analysis and interpretation of data: Vladimir Baulin and Xavier Le Guevel. Drafting of the manuscript: Vladimir Baulin and Xavier Le Guevel. Animal experiments and imaging: Xavier Le Guevel, Yves Usson. Critical revision of the manuscript for important intellectual content: all authors. Obtained funding: Xavier Le Guevel.


Franck Michel
email: franck.michel@inria.fr

Antonia Ettorre

Catherine Faron

Julien Kaplan

Olivier Gargominy

Biodiversity Knowledge Graphs: Time to move up a gear!

Harnessing worldwide biodiversity data requires integrating myriad pieces of information, often sparse and incomplete, into a global, coherent data space. To do so, projects like the Global Biodiversity Information Facility, Catalog of Life and Encyclopedia of Life have set up platforms that gather, consolidate, and centralize billions of records from multiple data sources. This approach lowers the entry barrier for scientists willing to consume aggregated biodiversity data but tends to build silos that hamper cross-platform interoperability.

The Web of Data embodies a different approach underpinned by the Linked Open Data (LOD) principles [START_REF] Heath | Linked Data: Evolving the Web into a Global Data Space[END_REF]. These principles bring about the building of a large, distributed, cross-domain knowledge graph (KG), wherein data description relies on vocabularies with shared, formal, machine-processable semantics. So far however, little biodiversity data have been published this way. Early efforts focused primarily on taxonomic registers, such as NCBI, VTO and AGROVOC. More recent efforts have started paving the way for the publication of more diverse biodiversity KGs [START_REF] Page | Ozymandias: a biodiversity knowledge graph[END_REF][START_REF] Penev | OpenBiodiv: A Knowledge Graph for Literature-Extracted Linked Open Data in Biodiversity Science[END_REF][START_REF] Michel | A Model to Represent Nomenclatural and Taxonomic Information as Linked Data. Application to the French Taxonomic Register[END_REF].

Today, we believe that it is time for more biodiversity data producers to join in and start publishing connected KGs spanning a much broader set of domains, far beyond just taxonomic registers. In this talk, we wish to present an on-going endeavor in line with this vision. In a previous work, we published TAXREF-LD [START_REF] Michel | A Model to Represent Nomenclatural and Taxonomic Information as Linked Data. Application to the French Taxonomic Register[END_REF]

, a LOD ‡ ‡ ‡ ‡ §
representation of the French taxonomic register developed and maintained by the French National Museum of Natural History. We modeled nomenclatural information as a thesaurus of scientific names, taxonomic information as an ontology of classes denoting taxa, and additional information such as ranks and vernacular names. Recently, we have extended the scope of TAXREF-LD to represent and interlink data as various as geographic locations, species interactions, development stages, trophic levels, as well as conservation, biogeographic, and legal status (regulations, protections, etc.).

We put a specific effort into working out a model that accurately accounts for the semantics of the data while respecting knowledge engineering practices. For instance, a common design shortcoming is to attach all information as properties of a taxon. This is a rightful choice for some properties like a scientific name or conservation status, but properties that actually pertain to biological individuals themselves, e.g. habitat and trophic level, should better be attched to class members. With the presentation of this work, we wish to advance the discussion about integration scenarios based on knowledge graphs with the different biodiversity data stakeholders.

Keywords

linked data, modeling, modelling

Presenting author

Franck Michel

Presented at

TDWG 2021